Consistency in the assessment of student progression data

The present attention to consistency in the assessment of student progression data follows the international education indicators in 2007 and requires a careful distinction between public commitment, institutional practice and demonstrated result. For the relevant requirement, the requirement should be read as an assurance obligation: the provider must be able to explain the control, show its operation and account for material exceptions. Review should cover the complete affected scope and preserve material differences between locations, programmes, delivery modes and learner groups. A policy approved at the centre is insufficient where local implementation has not been tested.

The stated reference is International education indicators in 2007. Interpretation should preserve the unit and population represented in the data collection. A national or international pattern may justify closer review of the stated expectation, but provider-level action requires evidence relating to the affected provision. The comparability record should identify material variation in coverage, period and classification.

Purpose and present context

The intended substantive result should remain the starting point for review. The analysis of consistency in the assessment of student progression data proceeds on the basis that assessment should provide valid and sufficiently consistent evidence that the stated learning outcomes have been achieved by the learner receiving the result. The existence of an approved measure or completed activity is not evidence of educational effect. Assurance should address actual effect and provide a means of correcting disadvantage that the arrangement did not intend.

A focused examination of the matter under review requires a clear analytical discipline. Oversight of the matter under review should reflect the principle that consistency does not require identical decisions regardless of context. It requires comparable matters to be treated on the same principles, with material differences explained by relevant evidence and recorded criteria. The distinction matters because evidence may appear sufficient while addressing a different population, period or outcome.

Failure in relation to the matter under review may arise even where the stated policy is reasonable. Material concerns include uncontrolled changes to assessment, weak assurance of authorship or performance, reasonable adjustment altering the assessed outcome, and results used beyond the evidence they support. Materiality depends on the consequence and extent of an exception, not only on how often it appears in sampled records.

The substantive quality question

Assurance of consistency in the assessment of student progression data should draw on more than one form of evidence. Useful records include appeal and correction records, moderation and exception records, assessment maps to learning outcomes, analysis of results and differential outcomes, and authorship and identity controls proportionate to risk. Policy and records should be tested against actual practice, including evidence from learners where appropriate. A selected successful case does not establish effectiveness across the system.

Records relating to the control should preserve both the conclusion and its limits. New evidence should trigger a traceable correction and review of decisions materially affected by the earlier conclusion. The correction process should identify prior users and decisions where published information has had material effect.

• Review differential and anomalous results within a defined period and review the result.
• Align tasks and criteria with learning outcomes before using it to determine a learner or provider outcome.
• Moderate material variation before using it to determine a learner or provider outcome.
• Control changes and retain evidence sufficient for independent review.
• Retain evidence sufficient for review before using it to determine a learner or provider outcome.

Evidence and assurance

A proper review of consistency in the assessment of student progression data should establish the intended outcome before selecting controls or indicators. A decision concerning the control should recognise that the assessment question is whether the control operates across the relevant sites, programmes, delivery modes and learner groups, including material exceptions. The basis for selection, authority for exceptions and timing of reassessment should remain traceable.

The review method for the assurance matter should be reproducible. A competent review of the assurance matter should use common definitions and decision criteria, calibrate responsible staff, review outliers and compare outcomes across locations and groups. Where variation is justified, retain the reason and verify that it is applied without arbitrary disadvantage. The retained analysis should be reproducible from the selected evidence, decision rule and recorded reasons for accepted exceptions.

Assurance concerning the stated expectation should be expressed at the level established by the evidence. A sample may support a conclusion about the sampled process, but not automatically about every location or programme. Where reliance is placed on central controls, testing should confirm that local operation and exceptions are reported accurately to the centre.

Accountability for the stated expectation should follow decision-making authority. Oversight is effective only if the responsible body receives the evidence and records its decision on resources, policy and residual risk. Where work is delegated, the record should continue to identify who is accountable for material consequences to learners.

Limitations and safeguards

Proportionality in relation to consistency in the assessment of student progression data does not mean reduced protection for learners exposed to greater risk. For the relevant requirement, reliability without validity produces consistent but potentially irrelevant results. Validity without adequate consistency may expose learners to unequal judgement. In reviewing the stated expectation, the volume of documentation is not a measure of conformity. Relevance, integrity and coverage are more important than the number of records produced. No exception should continue without a documented basis, accountable approval and scheduled review.

The measure of progress on the matter under review is not the amount of policy or documentation produced. Performance should be judged by outcomes and timely response to shortfalls, not by the volume of administrative activity.