Learning outcome reporting: evidence for policy decisions

In examining learning outcome reporting: evidence for policy decisions, the analysis proceeds on the basis that comparable indicators can support public decision-making, but they do not remove the need to examine variation within systems and institutions.

Patterns in the material may justify enquiry, although they do not by themselves determine legal position or cause. In applying it to the analysis, users should review the source definitions, population coverage, reference period and stated limitations before transferring a system-level finding to an individual provider or learner group.

For learning outcome reporting, the intended substantive result should remain the starting point for review.

Application to learning outcome reporting

In examining learning outcome reporting: evidence for policy decisions, for the analysis, where an indicator is used as a proxy, the relationship between the proxy and the underlying educational outcome should be stated and tested.

Controls for learning outcome reporting

Risk assessment of learning outcome reporting should give particular attention to proxy measures treated as direct outcomes, changes in definition presented as changes in performance, and incomplete coverage. A provider should also consider small differences overstated and averages concealing distribution.

• Avoid causal claims unsupported by the design, identifying the accountable function and affected scope.
• Document numerator and denominator.
• Report uncertainty and revisions before it is relied on for a decision with material effect.
• Analyse missing information.
• Test comparability.

Review of learning outcome reporting

Relevant evidence for learning outcome reporting will normally include disaggregated results, indicator definitions and metadata, coverage and missingness analysis, triangulation with administrative and qualitative evidence, and population and sampling information.

Authorities and providers reviewing the available evidence should proceed in a defined sequence. Across the defined scope, responsible bodies should map the complete process, identify the intended result and responsible authority at each stage, and test normal cases together with exceptions.

Decision-makers using evidence on the issue should be told what the data cannot establish as clearly as what it can.

Implications for learning outcome reporting

For learning outcome reporting, decisions concerning the measure should remain traceable to the information available for the stated reference period.

For learning outcome reporting, complete assurance concerning the available evidence cannot rest on a single indicator or isolated control.

In examining learning outcome reporting: evidence for policy decisions, users should be able to distinguish descriptive, comparative and evaluative findings and understand their proper level of application.

Application in another setting depends on a separate examination of context and comparability.