Explains consistency assessment in relation to assessment reliability, covering scope, evidence, decision authority, material exceptions and continuing assurance.
Current consideration of consistency in the assessment of assessment reliability is informed by the international comparative assessment practice, with consequences for governance, evidence and the treatment of affected learners. The central issue is the meaning of the expectation in practice, including its scope, the evidence needed to demonstrate it and the circumstances in which it may not apply. The control response should be sufficient to protect learners while avoiding burdens not justified by the evidence.
Meaning in practice
International comparative assessment practice provides the reference point for this analysis. Its relevance to consistency in the assessment of assessment reliability should be assessed against the affected jurisdiction, learner population and form of provision.
For assessment reliability, the applicable expectation should be capable of consistent application. Evidence is sufficient when it is current, attributable, representative of the relevant scope and capable of being reconciled with other available records. Operational definitions should be precise enough to support consistent consequential decisions and explain justified variation.
The principal risks in relation to the applicable expectation are uncontrolled changes to assessment, tasks that do not assess the stated outcome, results used beyond the evidence they support, and weak assurance of authorship or performance. When examining assessment reliability, a weakness in one part of the control environment may obscure a related failure elsewhere. Review should follow the sequence of decisions and records rather than assess documents in isolation.
Consistency does not require identical decisions regardless of context. For assessment reliability, it requires comparable matters to be treated on the same principles, with material differences explained by relevant evidence and recorded criteria. A conclusion concerning assessment reliability should identify both its evidential basis and the part of the stated scope for which assurance cannot be given.
The evidential record for the control should permit a reviewer to trace the matter from decision to outcome. This may require analysis of results and differential outcomes, appeal and correction records, moderation and exception records, and approval and change-control records, supported by authorship and identity controls proportionate to risk and assessment maps to learning outcomes.
Responsibilities and material risks
Proportionality in relation to consistency in the assessment of assessment reliability does not mean reduced protection for learners exposed to greater risk. Reliability without validity produces consistent but potentially irrelevant results. Validity without adequate consistency may expose learners to unequal judgement. For the applicable requirement, a prescribed method should not be treated as the only acceptable method where another approach establishes the same outcome with equivalent evidence.
Within the scope under review, decisions concerning the matter should remain traceable to the information available for the stated reference period.
- Calibrate assessors.
- Review differential and anomalous results.
- Moderate material variation.
- Align tasks and criteria with learning outcomes.
- Retain evidence sufficient for review.
Basis for a reliable conclusion
The review method for consistency in the assessment of assessment reliability should be reproducible. Responsible bodies should use common definitions and decision criteria, calibrate responsible staff, review outliers and compare outcomes across locations and groups. Where variation is justified, retain the reason and verify that it is applied without arbitrary disadvantage. The retained analysis should be reproducible from the selected evidence, decision rule and recorded reasons for accepted exceptions.
The final record on the assurance conclusion should identify the applicable expectation, the relevant scope, the evidence examined, the sampling basis, material exceptions and the reason for the conclusion. As regards assessment reliability, if an alternative method is accepted, the record should demonstrate that it achieves the same required outcome. A limitation preventing a complete conclusion should remain visible and unresolved until suitable evidence is obtained.
- Does review correct inconsistent treatment?
- Where are outcomes materially different?
- Is the reason relevant and documented?
- Are common criteria in use?
- Have decision-makers been calibrated?
Maintaining effective oversight
Accountability for consistency in the assessment of assessment reliability should follow decision-making authority. Evidence of material risk should be placed before the body with authority to act, together with a traceable decision.
For assessment reliability, assessment should provide valid and sufficiently consistent evidence that the stated learning outcomes have been achieved by the learner receiving the result.
Where evidence concerning assessment reliability cannot support assurance, the limitation should be reported and corrective work should remain open.