Data governance in monitoring post-PISA improvement planning

The immediate international context is the PISA 2022 follow-up during 2024. Its significance for post-PISA improvement planning lies in the quality of implementation rather than in formal acknowledgement alone. Evidence concerning post-PISA improvement planning should inform action without implying a level of precision, coverage or causal certainty that the underlying data cannot support. A proportionate arrangement should protect learners and decision integrity without controls unrelated to the identified risk.

Analytical scope

For post-PISA improvement planning, the first PISA 2022 results were released on 5 December 2023, with mathematics as the principal assessment domain and reading and science also reported. The cycle was conducted after major disruption to education systems. Comparisons require attention to participation, coverage and the exceptional context; changes from earlier cycles should not be attributed to a single cause without further evidence.

For post-PISA improvement planning, education indicators should support decisions by describing outcomes and variation with definitions and limitations that permit responsible interpretation. Review should cover the stages at which learners receive information, provision, assessment, support and remedy.

• Define the decision the indicator will inform before using it to determine a learner or provider outcome.
• Document numerator and denominator.
• Avoid causal claims unsupported by the design.
• Test comparability.
• Analyse missing information.

Definitions and data coverage

PISA 2022 follow-up during 2024 provides the reference point for this analysis. Its relevance to post-PISA improvement planning should be assessed against the affected jurisdiction, learner population and form of provision. The international development warrants attention, but a consequential conclusion still requires current, attributable and representative evidence for the affected scope.

For decisions concerning post-PISA improvement planning, analysis should make its decision rule explicit. An improvement plan should connect a verified problem with a specific intervention, accountable ownership, resources, milestones and a measure of effect. Broad intentions should be converted into decisions capable of review. This supports consistent review and reduces the risk of redefining the basis of judgement after an adverse result appears.

Use of the findings

Implementation of post-PISA improvement planning should be organised around a decision that can be tested. Reported averages should be accompanied by sufficient distributional information to identify material differences between learner groups, locations and forms of provision. The implementation record should link purpose, authority, resources, operation and reported result.

Risk assessment of the available evidence should give particular attention to averages concealing distribution, proxy measures treated as direct outcomes, and small differences overstated. A provider should also consider incomplete coverage and changes in definition presented as changes in performance.

• Who confirms sustained effectiveness?
• What measure will establish success?
• Does each action address a stated cause?
• Is the problem defined by evidence?
• Are dependencies and resources identified?

Uncertainty and safeguards

Evidence concerning post-PISA improvement planning should be selected against a clearly defined question. The most relevant material is likely to include uncertainty estimates where relevant, triangulation with administrative and qualitative evidence, indicator definitions and metadata, and population and sampling information.

For operational review, authorities and providers should proceed in a defined sequence. Responsible bodies should prioritise actions by learner impact and control weakness, establish dependencies, test implementation at suitable intervals and retain unresolved items until effectiveness is verified. In reviewing post-PISA improvement planning, amend the plan where evidence does not support the original causal assumption.

Uncertainty and safeguards

Decision-makers using evidence on post-PISA improvement planning should be told what the data cannot establish as clearly as what it can. Users should be able to distinguish descriptive, comparative and evaluative findings and understand their proper level of application. The basis for applying the result elsewhere should be established rather than assumed.

Proportionality in relation to the issue does not mean reduced protection for learners exposed to greater risk. For post-PISA improvement planning, measurement can reveal where outcomes differ; it does not by itself establish why they differ or which intervention will work. International comparison can identify variation, but institutional and policy context remains necessary before a practice is transferred from one setting to another.

The assurance record for post-PISA improvement planning should retain the date of the evidence, the source responsible for it, the scope examined and the version of any instrument or definition applied. This enables later review to separate substantive change from correction, reclassification or expanded coverage. A superseded conclusion should be retained where it formed the basis of a material decision.

Public reporting on the measure should distinguish established fact, analytical judgement and planned action. For decisions concerning post-PISA improvement planning, revision history should remain available where users have relied on the earlier conclusion.

In work concerning post-PISA improvement planning, progress should not be assessed by the amount of policy or documentation produced. The measure is demonstrated public benefit, including detection and correction of material variation.