Sets out the matters that should be established when applying consistency in the assessment of assessment reliability, including scope, responsibility and the basis for a reliable conclusion.
Current consideration of consistency in the assessment of assessment reliability is informed by the international comparative assessment practice, with consequences for governance, evidence and the treatment of affected learners. A decision concerning the relevant requirement should recognise that the central issue is the meaning of the expectation in practice, including its scope, the evidence needed to demonstrate it and the circumstances in which it may not apply. The control response should be sufficient to protect learners while avoiding burdens not justified by the evidence.
Purpose and present context
The historical reference basis is the international comparative assessment practice. Its relevance to consistency in the assessment of assessment reliability should be assessed against the affected jurisdiction, learner population and form of provision. International developments provide context; decisions affecting learners require evidence that is current and representative of the setting concerned.
The governing expectation for the stated expectation should be capable of consistent application. Oversight of the stated expectation should reflect the principle that evidence is sufficient when it is current, attributable, representative of the relevant scope and capable of being reconciled with other available records. Operational definitions should be precise enough to support consistent consequential decisions and explain justified variation.
The principal risks in relation to the stated expectation are uncontrolled changes to assessment, tasks that do not assess the stated outcome, results used beyond the evidence they support, and weak assurance of authorship or performance. A weakness in one part of the control environment may obscure a related failure elsewhere. Review should follow the sequence of decisions and records rather than assess documents in isolation.
The technical issue within the matter under review concerns the basis on which a conclusion is reached. The analysis of the assurance matter proceeds on the basis that consistency does not require identical decisions regardless of context. It requires comparable matters to be treated on the same principles, with material differences explained by relevant evidence and recorded criteria. A conclusion should identify both its evidential basis and the part of the stated scope for which assurance cannot be given.
The evidential record for the control should permit a reviewer to trace the matter from decision to outcome. This may require analysis of results and differential outcomes, appeal and correction records, moderation and exception records, and approval and change-control records, supported by authorship and identity controls proportionate to risk and assessment maps to learning outcomes. The sample should be extended when records conflict, a material group is missing or earlier corrective action may not have been sustained.
Responsibilities and material risks
Proportionality in relation to consistency in the assessment of assessment reliability does not mean reduced protection for learners exposed to greater risk. The analysis of the assurance matter proceeds on the basis that reliability without validity produces consistent but potentially irrelevant results. Validity without adequate consistency may expose learners to unequal judgement. For the relevant requirement, a prescribed method should not be treated as the only acceptable method where another approach establishes the same outcome with equivalent evidence. An exception is to remain time-limited, approved and subject to a stated review point.
Decisions concerning the matter under review should remain traceable to the information available for the stated reference period. Any revised finding should identify precisely what has changed and why the earlier conclusion no longer applies. Users should not be left to infer a change in performance where the observed movement results from revised reporting.
- Calibrate assessors within a defined period and review the result.
- Review differential and anomalous results, including material exceptions and unequal effects.
- Moderate material variation and retain evidence sufficient for independent review.
- Align tasks and criteria with learning outcomes, including material exceptions and unequal effects.
- Retain evidence sufficient for review, including material exceptions and unequal effects.
Basis for a reliable conclusion
The review method for consistency in the assessment of assessment reliability should be reproducible. In reviewing the stated expectation, responsible bodies should use common definitions and decision criteria, calibrate responsible staff, review outliers and compare outcomes across locations and groups. Where variation is justified, retain the reason and verify that it is applied without arbitrary disadvantage. The retained analysis should be reproducible from the selected evidence, decision rule and recorded reasons for accepted exceptions.
The final record on the assurance matter should identify the applicable expectation, the relevant scope, the evidence examined, the sampling basis, material exceptions and the reason for the conclusion. If an alternative method is accepted, the record should demonstrate that it achieves the same required outcome. A limitation preventing a complete conclusion should remain visible and unresolved until suitable evidence is obtained.
- Does review correct inconsistent treatment?
- Where are outcomes materially different?
- Is the reason relevant and documented?
- Are common criteria in use?
- Have decision-makers been calibrated?
Conditions for responsible implementation
Accountability for consistency in the assessment of assessment reliability should follow decision-making authority. Evidence of material risk should be placed before the body with authority to act, together with a traceable decision. The operating function may change, but responsibility for oversight and learner protection should remain clear.
The quality significance of the matter under review follows from a basic distinction between availability and effective provision. A decision concerning the stated expectation should recognise that assessment should provide valid and sufficiently consistent evidence that the stated learning outcomes have been achieved by the learner receiving the result. A single entry control or reported outcome cannot demonstrate consistent operation across the learner journey.
The appropriate response to the stated expectation is therefore one of controlled implementation and review. Neither administrative activity nor general assurance should obscure the intended result or its effect on learners. Where evidence cannot support assurance, the limitation should be reported and corrective work should remain open.