Review of consistency in evaluating assessment reliability sets out the evidence, authority and controls needed to reach and maintain a defensible conclusion.
Evidence relevant to consistency in evaluating assessment reliability
Its relevance to consistency in the assessment of assessment reliability should be assessed against the affected jurisdiction, learner population and form of provision.
The principal risks in relation to the applicable expectation are uncontrolled changes to assessment, tasks that do not assess the stated outcome, results used beyond the evidence they support, and weak assurance of authorship or performance. When examining assessment reliability, a weakness in one part of the control environment may obscure a related failure elsewhere.
Consistency does not require identical decisions regardless of context. For assessment reliability, it requires comparable matters to be treated on the same principles, with material differences explained by relevant evidence and recorded criteria. A conclusion concerning assessment reliability should identify both its evidential basis and the part of the stated scope for which assurance cannot be given.
The evidential record for the control should permit a reviewer to trace the matter from decision to outcome. This may require analysis of results and differential outcomes, appeal and correction records, moderation and exception records, and approval and change-control records, supported by authorship and identity controls proportionate to risk and assessment maps to learning outcomes.
Application to consistency in evaluating assessment reliability
Proportionality in relation to consistency in the assessment of assessment reliability does not mean reduced protection for learners exposed to greater risk. For the applicable requirement, a prescribed method should not be treated as the only acceptable method where another approach establishes the same outcome with equivalent evidence.
Across the defined scope, decisions concerning the matter should remain traceable to the information available for the stated reference period.
- Calibrate assessors.
- Review differential and anomalous results.
- Moderate material variation.
- Align tasks and criteria with learning outcomes.
- Retain evidence sufficient for review.
Controls for consistency in evaluating assessment reliability
The review method for consistency in the assessment of assessment reliability should be reproducible. Responsible bodies should use common definitions and decision criteria, calibrate responsible staff, review outliers and compare outcomes across locations and groups. Where variation is justified, retain the reason and verify that it is applied without arbitrary disadvantage.
The final record on the conclusion should identify the applicable expectation, the relevant scope, the evidence examined, the sampling basis, material exceptions and the reason for the conclusion. For assessment reliability, if an alternative method is accepted, the record should demonstrate that it achieves the same required outcome.
- Does review correct inconsistent treatment?
- Where are outcomes materially different?
- Is the reason relevant and documented?
- Are common criteria in use?
- Have decision-makers been calibrated?
Review of consistency in evaluating assessment reliability
Accountability for consistency in the assessment of assessment reliability should follow decision-making authority.
For assessment reliability, assessment should provide valid and sufficiently consistent evidence that the stated learning outcomes have been achieved by the learner receiving the result.
Where evidence concerning assessment reliability cannot support assurance, the limitation should be reported and corrective work should remain open.