Records required to substantiate assessment reliability

In 2013, consideration of assessment reliability must take account of the international comparative assessment practice and the responsibilities it places before education systems. The analysis of the stated expectation proceeds on the basis that the requirement should be read as an assurance obligation: the provider must be able to explain the control, show its operation and account for material exceptions. A reliable review extends beyond the central process to material variation across programmes, sites, delivery arrangements and learner groups. Evidence of formal policy should not be treated as evidence of uniform implementation.

Public-interest context

The historical reference basis is the international comparative assessment practice. Its relevance to assessment reliability should be assessed against the affected jurisdiction, learner population and form of provision. The wider development does not remove the need to establish the position through attributable evidence from the relevant jurisdiction or institution.

Implementation of the relevant requirement should be organised around a decision that can be tested. In reviewing the control, evidence is sufficient when it is current, attributable, representative of the relevant scope and capable of being reconciled with other available records. Resources and activity should be reconciled with the operating evidence and result for which the responsible function is accountable.

The principal risks in relation to the control are uncontrolled changes to assessment, inconsistent judgement between markers or locations, reasonable adjustment altering the assessed outcome, and weak assurance of authorship or performance. The risks are interdependent; failure of one control may conceal or disable another. Review should follow the sequence of decisions and records rather than assess documents in isolation.

A focused examination of the stated expectation requires a clear analytical discipline. A decision concerning the relevant requirement should recognise that a reliable record should identify what occurred, when it occurred, who was responsible, the authority for the action and any later correction. Records should remain protected against unauthorised alteration while legitimate amendments remain visible. An imprecise scope or measure may produce a credible-looking record that does not answer the relevant decision question.

Relevant evidence for the matter under review will normally include marking criteria and calibrated judgement, appeal and correction records, moderation and exception records, approval and change-control records, and assessment maps to learning outcomes. Evidence outside the relevant period or scope should be identified and given no more weight than its limitations permit. Contradictory evidence should be investigated and resolved, not omitted from the record.

Operational significance

Care is required in drawing conclusions about assessment reliability. For the assurance matter, reliability without validity produces consistent but potentially irrelevant results. Validity without adequate consistency may expose learners to unequal judgement. Oversight of the assurance matter should reflect the principle that interpretive guidance should not create an obligation that is absent from the governing instrument or applicable law. Limitations should be prominent wherever the finding may influence a consequential decision.

Accountability and effective correction both depend on a record that can be followed from evidence to decision. For the assurance matter, the responsible body should be able to identify the evidence considered, the judgement made, the person or body authorised to make it and the action that followed. Material changes require a traceable effective date and explanation so that prior reliance can be reviewed fairly.

• Review differential and anomalous results and retain evidence sufficient for independent review.
• Define the decision each assessment must support, with responsibility, scope and timing recorded.
• Retain evidence sufficient for review within a defined period and review the result.
• Calibrate assessors, identifying the accountable function and affected scope.
• Moderate material variation before it is relied on for a decision with material effect.

Basis for a reliable conclusion

Implementation of assessment reliability can be tested without imposing unnecessary reporting. For the stated expectation, the reviewer should specify mandatory fields, source ownership, access rights, retention and correction procedures. Test a sample from creation through use, amendment, reporting and disposal, including records created during disruption or by a delivery partner. Reuse of existing information is appropriate only where its purpose, scope and reliability correspond to the decision under review.

The final record on the matter under review should identify the applicable expectation, the relevant scope, the evidence examined, the sampling basis, material exceptions and the reason for the conclusion. Equivalent methods should be assessed by demonstrated result, with the basis for acceptance retained. The affected scope should remain open where a material limitation prevents assurance.

• Can records be retrieved throughout the required period?
• Is the record attributable?
• Are partner records subject to equivalent controls?
• Can an amendment be distinguished from the original?
• Are access rights proportionate?

Matters requiring continuing review

Accountability for assessment reliability should follow decision-making authority. Evidence of material risk should be placed before the body with authority to act, together with a traceable decision. The operating function may change, but responsibility for oversight and learner protection should remain clear.

The quality significance of the matter under review follows from a basic distinction between availability and effective provision. The analysis of the stated expectation proceeds on the basis that assessment should provide valid and sufficiently consistent evidence that the stated learning outcomes have been achieved by the learner receiving the result. Review should cover the stages at which learners receive information, provision, assessment, support and remedy.

No individual measure is sufficient to establish effective operation of the stated expectation across the affected scope. A conclusion should be revised when stronger evidence materially changes the assessment of implementation, outcome or risk.