Evidence relevant to PISA 2022 is assessed for currency, coverage and comparability, with material uncertainty stated alongside the finding.
Consideration of PISA 2022 should retain the date and status of PISA 2022 results released on 5 December 2023.
Consideration of the issue should retain the date and status of PISA 2022 results released on 5 December 2023. The cycle was conducted after major disruption to education systems. Comparisons require attention to participation, coverage and the exceptional context; changes from earlier cycles should not be attributed to a single cause without further evidence.
In the context of pISA 2022, the applicable expectation should be capable of consistent application.
Evidence base for pISA 2022
The source can frame further questions without supplying either a binding direction or proof of causation. In applying it to the measure, users should review the source definitions, population coverage, reference period and stated limitations before transferring a system-level finding to an individual provider or learner group.
For the analysis, the public interest is not confined to institutional compliance. In this case, education indicators should support decisions by describing outcomes and variation with definitions and limitations that permit responsible interpretation. For pISA 2022, where learners rely on published information or support decisions, errors should be identifiable and capable of prompt, fair correction.
- Analyse missing information.
- Report uncertainty and revisions, identifying the accountable function and affected scope.
- Document numerator and denominator.
- Define the decision the indicator will inform.
- Disaggregate material results.
Controls for pISA 2022
A conclusion concerning pISA 2022 should identify both its evidential basis and the part of the stated scope for which assurance cannot be given.
Failure in relation to the available evidence may arise even where the stated policy is reasonable. Material concerns include small differences overstated, data revisions not carried through to published conclusions, averages concealing distribution, and changes in definition presented as changes in performance. Across the defined scope, review should consider whether an exception is prolonged, recurring or capable of affecting learners outside the cases examined.
The evidential record for the analysis should permit a reviewer to trace the matter from decision to outcome. This may require triangulation with administrative and qualitative evidence, population and sampling information, uncertainty estimates where relevant, and disaggregated results, supported by revision and comparability records and indicator definitions and metadata. For pISA 2022, further cases should be examined when the initial sample does not represent the affected scope or confirm sustained correction.
- What outcome is intended?
- Which evidence establishes operation?
- Who controls each stage?
- Where do exceptions occur?
- What action is required by the finding?
Review of pISA 2022
Authorities and providers reviewing PISA 2022 should proceed in a defined sequence. A competent review of the analysis should map the complete process, identify the intended result and responsible authority at each stage, and test normal cases together with exceptions. The conclusion should identify whether further sampling or system-level action is required.
The analytical record for pISA 2022 should state the research question, data source, unit of analysis, reference period, coverage, exclusions, treatment of missing values and principal limitations.
The basis and limits of any conclusion concerning the available evidence should be explicit. For decisions concerning pISA 2022, measurement can reveal where outcomes differ; it does not by itself establish why they differ or which intervention will work. For the measure, association should not be presented as causation, and statistical significance should not be treated as evidence of educational importance without further analysis.
The assurance record for the comparison should retain the date of the evidence, the source responsible for it, the scope examined and the version of any instrument or definition applied. When examining pISA 2022, a later reviewer should be able to identify whether the condition changed or the evidential record was corrected.
Public reporting on the available evidence should distinguish established fact, analytical judgement and planned action. For pISA 2022, a revised conclusion should distinguish a change in the underlying condition from a change in method, coverage or evidence.