Assesses the evidence concerning PISA 2018, including comparability, uncertainty and limits on interpretation.
At the publication date, PISA 2018 results released on 3 December 2019 provides the relevant international context for the reported measure. Any consequential application still requires evidence from the affected jurisdiction or institution. Oversight of the reported measure should reflect the principle that evidence should inform action without implying a level of precision, coverage or causal certainty that the underlying data cannot support. Data used for PISA 2018 should be interpreted against stable definitions and an identifiable population. A revision or break in series should not be reported as a change in performance.
The contemporaneous reference point for the analytical question is PISA 2018 results released on 3 December 2019. Its status should be distinguished from the jurisdiction-specific evidence required for implementation. The level of inference should not extend beyond the level at which the source data are valid. The evidential basis for the reported measure should identify source, period, coverage and material limitations. Corroboration is required where a single record cannot support the decision. Data used for the reported measure should be interpreted against stable definitions and an identifiable population. Reporting should identify a break in comparability before describing movement over time.
Public-interest context
Consideration of PISA 2018 should retain the date and status of PISA 2018 results released on 3 December 2019. Later developments should not be read into the position available at publication. Analysis of the evidence under review should state the unit of analysis, reference period, coverage, exclusions and treatment of missing information. Material differences in population, setting or method should remain explicit in any comparison. Any indicator used in relation to the evidence under review should distinguish description from causal explanation. The result should be accompanied by distributional information, uncertainty and limits on transfer to another setting.
Analysis of the reported measure should state the unit of analysis, reference period, coverage, exclusions and treatment of missing information. Comparative findings should not conceal differences capable of changing their meaning. Review should cover the stages at which learners receive information, provision, assessment, support and remedy.
Assurance should not overlook failures arising at the boundary between otherwise adequate controls. A formally complete record is not reliable if its scope or measure does not correspond to the decision being made.
Responsibility for the matter examined should be visible at the point where consequential decisions are made. Data used for the evidence under review should be interpreted against stable definitions and an identifiable population. Apparent movement caused by revision should not be attributed to educational performance. The matter should be escalated when evidence is incomplete, a conflict is present, affected learners are not represented or the likely effect is material.
Assurance of the analytical question should draw on more than one form of evidence. A comparison is reliable only if material differences remain visible. A positive example may illustrate operation, but it cannot demonstrate coverage or consistency.
- Review policies that create avoidable exclusion, identifying the accountable function and affected scope.
- Analyse barriers across the learner journey within a defined period and review the result.
- Target support transparently before it informs a consequential decision.
- Report limitations in available data, including material exceptions and unequal effects.
- Evaluate differential outcomes, including material exceptions and unequal effects.
Operational significance
Risk assessment of PISA 2018 should give particular attention to financial or geographic barriers, completion measures detached from learning, and admission without adequate support. The position on the analytical question should be established through proportionate evidence and should remain open to correction when material new information becomes available. Preventive safeguards are particularly important when harm is difficult to detect or cannot be fully corrected after the event.
The review method for the evidence under review should be reproducible. Review of the analytical question should give particular attention to adverse cases, unequal effects and errors that learners may be unable to identify or remedy after the event. Repeated or connected failures should be addressed through the relevant system control, not only through individual remedies. Risk assessment for the evidence under review should consider severity, reach, duration, recurrence and detectability, with escalation where learner impact may be material.
Decision-makers using evidence on the comparison should be told what the data cannot establish as clearly as what it can. The conclusion should state whether the result is descriptive, comparative or evaluative, the matter examined whether it applies to a system, institution, programme or learner group. Analysis of the analytical question should state the unit of analysis, reference period, coverage, exclusions and treatment of missing information.
Proportionality in relation to the matter examined does not mean reduced protection for learners exposed to greater risk. In reviewing The reported measure, equal treatment does not necessarily produce equitable access or outcomes. Different support may be justified where it addresses a documented barrier without changing the required educational outcome. Review of the reported measure should give particular attention to adverse cases, unequal effects and errors that learners may be unable to identify or remedy after the event.
Decisions concerning the analytical question should remain traceable to the information available for the stated reference period. The reason for revision should be explicit, including whether it arises from new evidence, a methodological change or a different interpretation. A break in method or coverage must not be presented as if it demonstrated a change in educational performance.
The principal risks associated with the analytical question should be assessed as connected conditions. A failed safeguard may conceal another weakness or prevent timely correction. Governance of the evidence under review requires a clear allocation of authority, information and follow-through. Material matters should be referred to the body authorised to act or accept residual risk. Closure requires evidence that the condition has changed; completion of planned activity is not sufficient.
Data used for the analytical question should be interpreted against stable definitions and an identifiable population.