Examines learning outcome reporting, addressing evidence for policy decisions, source definitions, coverage, comparability, uncertainty and limits on inference.
In 2006, consideration of learning outcome reporting must take account of the expansion of comparable education indicators and the responsibilities it places before education systems. The analysis proceeds on the basis that comparable indicators can support public decision-making, but they do not remove the need to examine variation within systems and institutions. Proportionality should be assessed against effects on access, learning, fair treatment and the accuracy of learner information.
The reference basis—the expansion of comparable education indicators—is evidential rather than self-executing. Patterns in the material may justify enquiry, although they do not by themselves determine legal position or cause. In applying it to the analysis, users should review the source definitions, population coverage, reference period and stated limitations before transferring a system-level finding to an individual provider or learner group.
For learning outcome reporting, the intended substantive result should remain the starting point for review. Education indicators should support decisions by describing outcomes and variation with definitions and limitations that permit responsible interpretation. The existence of an approved measure or completed activity is not evidence of educational effect. Assurance should address actual effect and provide a means of correcting disadvantage that the arrangement did not intend.
Evidence and method
Review of learning outcome reporting should be based on a stated method rather than general assurance. For the available evidence, the subject should be examined as a connected system of policy, people, resources, decisions and evidence. A review should test the interfaces between functions, not assume that sound component controls ensure a sound end-to-end process.
For decisions concerning learning outcome reporting, the applicable expectation should be capable of consistent application. For the analysis, where an indicator is used as a proxy, the relationship between the proxy and the underlying educational outcome should be stated and tested. Criteria affecting learners should not permit materially different interpretation without an evidenced reason.
Patterns requiring examination
Risk assessment of learning outcome reporting should give particular attention to proxy measures treated as direct outcomes, changes in definition presented as changes in performance, and incomplete coverage. A provider should also consider small differences overstated and averages concealing distribution.
- Avoid causal claims unsupported by the design, identifying the accountable function and affected scope.
- Document numerator and denominator.
- Report uncertainty and revisions before it is relied on for a decision with material effect.
- Analyse missing information.
- Test comparability.
Implications for decision-makers
Relevant evidence for learning outcome reporting will normally include disaggregated results, indicator definitions and metadata, coverage and missingness analysis, triangulation with administrative and qualitative evidence, and population and sampling information. The record for learning outcome reporting should retain disagreement between sources until its cause and effect are understood.
Authorities and providers reviewing the available evidence should proceed in a defined sequence. Within the scope under review, responsible bodies should map the complete process, identify the intended result and responsible authority at each stage, and test normal cases together with exceptions.
Decision-makers using evidence on the issue should be told what the data cannot establish as clearly as what it can. Users should be able to distinguish descriptive, comparative and evaluative findings and understand their proper level of application. Application in another setting depends on a separate examination of context and comparability.
Limits of inference
Interpretation of learning outcome reporting should avoid two errors: treating a formal commitment as proof of effect, and treating one adverse case as proof that every part of the system has failed. Measurement can reveal where outcomes differ; it does not by itself establish why they differ or which intervention will work. A single indicator rarely provides an adequate account of quality. Quantitative evidence should be considered with implementation records and the experience of affected learners.
For learning outcome reporting, decisions concerning the measure should remain traceable to the information available for the stated reference period. The reason for revision should be explicit, including whether it arises from new evidence, a methodological change or a different interpretation.
Where responsibilities for delivery relating to learning outcome reporting are shared with partners, suppliers or several public bodies, responsibility should be mapped across the complete service. Governance between participating bodies should make information duties and corrective authority explicit. Protection should operate across the complete service, irrespective of how delivery is divided.
As regards learning outcome reporting, complete assurance concerning the available evidence cannot rest on a single indicator or isolated control.