Examines pISA 2022, addressing the decline in mathematics and reading performance, source definitions, coverage, comparability, uncertainty and limits on inference.
Consideration of PISA 2022 should retain the date and status of PISA 2022 results released on 5 December 2023. Later developments should not be read into the position available at publication. Its status should be distinguished from the jurisdiction-specific evidence required for implementation. The principal analytical task is to separate an observed difference from a conclusion about its cause. The appropriate administrative form will depend on the jurisdiction and the allocation of lawful responsibility.
Consideration of the issue should retain the date and status of PISA 2022 results released on 5 December 2023. The cycle was conducted after major disruption to education systems. Comparisons require attention to participation, coverage and the exceptional context; changes from earlier cycles should not be attributed to a single cause without further evidence.
In the context of pISA 2022, the applicable expectation should be capable of consistent application. A sound interpretation should identify the unit of analysis, reference period, denominator, exclusions, missing values and any change in definition or collection practice. Criteria affecting learners should not permit materially different interpretation without an evidenced reason.
Evidence base for pISA 2022
The reference basis—the PISA 2022 results released on 5 December 2023—is evidential rather than self-executing. The source can frame further questions without supplying either a binding direction or proof of causation. In applying it to the measure, users should review the source definitions, population coverage, reference period and stated limitations before transferring a system-level finding to an individual provider or learner group.
For the analysis, the public interest is not confined to institutional compliance. In this case, education indicators should support decisions by describing outcomes and variation with definitions and limitations that permit responsible interpretation. In work concerning pISA 2022, where learners rely on published information or support decisions, errors should be identifiable and capable of prompt, fair correction.
- Analyse missing information.
- Report uncertainty and revisions, identifying the accountable function and affected scope.
- Document numerator and denominator.
- Define the decision the indicator will inform.
- Disaggregate material results.
Coverage and comparability
For the available evidence, the subject should be examined as a connected system of policy, people, resources, decisions and evidence. Transfer of decisions or records can expose weaknesses not visible in separate reviews of individual controls. A conclusion concerning pISA 2022 should identify both its evidential basis and the part of the stated scope for which assurance cannot be given.
Failure in relation to the available evidence may arise even where the stated policy is reasonable. Material concerns include small differences overstated, data revisions not carried through to published conclusions, averages concealing distribution, and changes in definition presented as changes in performance. Within the scope under review, review should consider whether an exception is prolonged, recurring or capable of affecting learners outside the cases examined.
The evidential record for the analysis should permit a reviewer to trace the matter from decision to outcome. This may require triangulation with administrative and qualitative evidence, population and sampling information, uncertainty estimates where relevant, and disaggregated results, supported by revision and comparability records and indicator definitions and metadata. For pISA 2022, further cases should be examined when the initial sample does not represent the affected scope or confirm sustained correction.
- What outcome is intended?
- Which evidence establishes operation?
- Who controls each stage?
- Where do exceptions occur?
- What action is required by the finding?
Responsible interpretation
Authorities and providers reviewing PISA 2022 should proceed in a defined sequence. A competent review of the analysis should map the complete process, identify the intended result and responsible authority at each stage, and test normal cases together with exceptions. The conclusion should identify whether further sampling or system-level action is required. Findings should state the affected scope and required action; an observation should not be represented as evidence of conformity or effectiveness.
The analytical record for pISA 2022 should state the research question, data source, unit of analysis, reference period, coverage, exclusions, treatment of missing values and principal limitations.
The basis and limits of any conclusion concerning the available evidence should be explicit. For decisions concerning pISA 2022, measurement can reveal where outcomes differ; it does not by itself establish why they differ or which intervention will work. For the measure, association should not be presented as causation, and statistical significance should not be treated as evidence of educational importance without further analysis. Decision-makers and affected users should receive the conclusion together with its material evidential limits.
The assurance record for the comparison should retain the date of the evidence, the source responsible for it, the scope examined and the version of any instrument or definition applied. When examining pISA 2022, a later reviewer should be able to identify whether the condition changed or the evidential record was corrected. A superseded conclusion should be retained where it formed the basis of a material decision.
Public reporting on the available evidence should distinguish established fact, analytical judgement and planned action. In work concerning pISA 2022, a revised conclusion should distinguish a change in the underlying condition from a change in method, coverage or evidence.
As regards pISA 2022, progress should not be assessed by the amount of policy or documentation produced.