Public performance reporting: definitions and comparability

The 2017 accountability agenda provides the immediate context for public performance reporting. For the available evidence, the available evidence should be interpreted with close attention to definitions, population coverage, collection methods and the limits of comparison.

For public performance reporting, the applicable expectation should be capable of consistent application. Reported averages should be accompanied by sufficient distributional information to identify material differences between learner groups, locations and forms of provision. Terms governing eligibility, support, assessment, reporting or review should prevent materially different treatment without recorded justification.

Evidence base for public performance reporting

The stated reference is the 2017 accountability agenda. Application to public performance reporting depends on evidence from the relevant jurisdiction or institution. Authorities and providers should distinguish established fact, policy expectation and matters left to institutional judgement. That distinction should remain visible in the decision record, public reporting and later review.

In the context of public performance reporting, education indicators should support decisions by describing outcomes and variation with definitions and limitations that permit responsible interpretation.

• Define the decision the indicator will inform.
• Analyse missing information.
• Report uncertainty and revisions before using it to determine a learner or provider outcome.
• Avoid causal claims unsupported by the design.
• Test comparability.

Coverage and comparability

Comparison requires more than the use of a common label. When examining public performance reporting, definitions, reference periods, population coverage, institutional boundaries and collection practices must be sufficiently aligned for the observed difference to have a stable meaning. The decision record for public performance reporting should distinguish the scope supported by evidence from any scope that remains unresolved.

The principal risks in relation to the analysis are averages concealing distribution, incomplete coverage, small differences overstated, and data revisions not carried through to published conclusions. As regards public performance reporting, a weakness in one part of the control environment may obscure a related failure elsewhere. Within the scope under review, review should follow the sequence of decisions and records rather than assess documents in isolation.

In the context of public performance reporting, readily available material should not define the enquiry if it cannot answer the relevant decision question. For the available evidence, the most relevant material is likely to include disaggregated results, revision and comparability records, triangulation with administrative and qualitative evidence, and coverage and missingness analysis.

• Do the reference periods align?
• Is the remaining difference educationally material?
• Has a classification changed?
• Are the populations defined on the same basis?
• Are exclusions and missing records comparable?

Responsible interpretation

Authorities and providers reviewing public performance reporting should proceed in a defined sequence. A competent review of the comparison should prepare a comparability table before analysing results. Record common elements, material differences, breaks in series and the direction in which each limitation may affect the conclusion; do not rank systems where those limitations remain material.

Publication of findings on public performance reporting should distinguish observed values, estimates and interpretation.

Interpretation of the measure should avoid two errors: treating a formal commitment as proof of effect, and treating one adverse case as proof that every part of the system has failed. For decisions concerning public performance reporting, measurement can reveal where outcomes differ; it does not by itself establish why they differ or which intervention will work. Missing or delayed information may be patterned rather than random.

The assurance record for the comparison should retain the date of the evidence, the source responsible for it, the scope examined and the version of any instrument or definition applied. In the context of public performance reporting, traceable source and version information allow genuine improvement to be distinguished from administrative revision. Revision should not remove an earlier conclusion from the record where reliance has occurred.

Public reporting on public performance reporting should distinguish established fact, analytical judgement and planned action. If definitions, coverage or evidence alter an earlier conclusion, the reason should be stated so that revision is not mistaken for changed performance.

As regards public performance reporting, no individual measure is sufficient to establish effective operation of the available evidence across the affected scope.