Sets out quality controls as an evidence-led approach to learning-outcome measurement, covering responsibility, outcome evidence and sustained effect.
In 2010, consideration of learning-outcome measurement must take account of the international assessment and indicator development and the responsibilities it places before education systems. The method set out here treats improvement as a controlled cycle of diagnosis, action, measurement and review. The control response should be sufficient to protect learners while avoiding burdens not justified by the evidence.
International assessment and indicator development provides the reference point for this analysis. Its relevance to corrective action should be assessed against the affected jurisdiction, learner population and form of provision.
For learning-outcome measurement, responsibility should be identifiable at the point where consequential decisions are made. The corrective action should be tested on a scale proportionate to the risk before wider implementation, unless immediate system-wide action is necessary to protect learners. A decision should not be closed at the operating level where material impact, conflict or a significant evidential gap remains unresolved.
Defining the problem
The system and institutional dimensions of learning-outcome measurement should be considered together. Education indicators should support decisions by describing outcomes and variation with definitions and limitations that permit responsible interpretation. The regulatory setting is determined by public authorities, but responsibility for controlled provision remains with the provider.
Review of corrective action should be based on a stated method rather than general assurance. For decisions concerning learning-outcome measurement, effectiveness is the demonstrated change in the condition the action was intended to address. Completion of training, publication of guidance or installation of a system is an output and should not be reported as an outcome without further evidence.
Failure in relation to the intended improvement may arise even where the stated policy is reasonable. Material concerns include data revisions not carried through to published conclusions, small differences overstated, averages concealing distribution, and incomplete coverage. As regards learning-outcome measurement, an exception should be assessed by effect, duration, recurrence and reach, including possible exposure beyond the initial sample.
The evidential record for the corrective action should permit a reviewer to trace the matter from decision to outcome. This may require disaggregated results, coverage and missingness analysis, revision and comparability records, and population and sampling information, supported by indicator definitions and metadata and uncertainty estimates where relevant. Within the scope under review, conflicting records, absent populations and uncertain follow-through require additional testing.
Improvement method
Authorities and providers reviewing learning-outcome measurement should proceed in a defined sequence. The method for the intended improvement is to set a baseline and success measure before intervention, define the review period, compare the result with the intended outcome and examine adverse or unequal effects. Continue monitoring long enough to determine whether the improvement is sustained. The record for learning-outcome measurement should distinguish a finding that requires action from an observation that supports no formal conclusion.
Improvement of learning-outcome measurement should proceed through controlled tests where risk permits.
When examining learning-outcome measurement, records relating to the matter should preserve both the conclusion and its limits. New evidence should trigger a traceable correction and review of decisions materially affected by the earlier conclusion.
Measures and review
The analysis of learning-outcome measurement should remain within the limits of the evidence. Correcting an individual record does not establish that the process which produced the error has been corrected. Measurement can reveal where outcomes differ; it does not by itself establish why they differ or which intervention will work.
Where responsibilities for delivery relating to learning-outcome measurement are shared with partners, suppliers or several public bodies, responsibility should be mapped across the complete service. The division of responsibilities should cover records, communication, escalation and the power to require correction. Within the scope under review, multiple delivery partners do not justify fragmented accountability or remedy.
For learning-outcome measurement, assessment of corrective action should reconcile more than one source of evidence and control. The final judgement on learning-outcome measurement should connect the applicable expectation to implementation and outcomes while identifying unresolved risk.
Effectiveness should be reassessed after the defined review period and against the position recorded under International assessment and indicator development. Later improvement does not retrospectively remove the original deficiency or the need for a traceable corrective record. The assessment should show whether benefit is sustained and whether material adverse effects remain.