PISA 2018: performance, equity and responsible international comparison

At the publication date, PISA 2018 results released on 3 December 2019 provides the relevant international context for the measure. Any consequential application still requires evidence from the affected jurisdiction or institution. Evidence concerning pISA 2018 should inform action without implying a level of precision, coverage or causal certainty that the underlying data cannot support. Data used for PISA 2018 should be interpreted against stable definitions and an identifiable population. A revision or break in series should not be reported as a change in performance.

The contemporaneous reference point for the analysis is PISA 2018 results released on 3 December 2019. Its status should be distinguished from the jurisdiction-specific evidence required for implementation. The level of inference should not extend beyond the level at which the source data are valid. The evidential basis for the measure should identify source, period, coverage and material limitations. Corroboration is required where a single record cannot support the decision. Data used for the measure should be interpreted against stable definitions and an identifiable population. Reporting should identify a break in comparability before describing movement over time.

Evidence base for pISA 2018

Consideration of PISA 2018 should retain the date and status of PISA 2018 results released on 3 December 2019. Later developments should not be read into the position available at publication. Analysis should state the unit of analysis, reference period, coverage, exclusions and treatment of missing information. Material differences in population, setting or method should remain explicit in any comparison. Any indicator used in relation to the available evidence should distinguish description from causal explanation. The result should be accompanied by distributional information, uncertainty and limits on transfer to another setting.

In the context of pISA 2018, analysis should state the unit of analysis, reference period, coverage, exclusions and treatment of missing information. Comparative findings should not conceal differences capable of changing their meaning. Review should cover the stages at which learners receive information, provision, assessment, support and remedy.

Assurance should not overlook failures arising at the boundary between otherwise adequate controls. In work concerning pISA 2018, a formally complete record is not reliable if its scope or measure does not correspond to the decision being made.

As regards pISA 2018, responsibility should be identifiable at the point where consequential decisions are made. Data used for the available evidence should be interpreted against stable definitions and an identifiable population. Apparent movement caused by revision should not be attributed to educational performance.

Assurance of the analysis should draw on more than one form of evidence. In reviewing pISA 2018, a comparison is reliable only if material differences remain visible. Within the scope under review, a positive example may illustrate operation, but it cannot demonstrate coverage or consistency.

• Review policies that create avoidable exclusion, identifying the accountable function and affected scope.
• Analyse barriers across the learner journey.
• Target support transparently before it informs a consequential decision.
• Report limitations in available data.
• Evaluate differential outcomes.

Coverage and comparability

Risk assessment of PISA 2018 should give particular attention to financial or geographic barriers, completion measures detached from learning, and admission without adequate support. The position on the analysis should be established through proportionate evidence and should remain open to correction when material new information becomes available.

The review method for the available evidence should be reproducible. Review of the analysis should give particular attention to adverse cases, unequal effects and errors that learners may be unable to identify or remedy after the event. Repeated or connected failures should be addressed through the relevant system control, not only through individual remedies. Risk assessment for the available evidence should consider severity, reach, duration, recurrence and detectability, with escalation where learner impact may be material.

Decision-makers using evidence on the comparison should be told what the data cannot establish as clearly as what it can. The conclusion should state whether the result is descriptive, comparative or evaluative, the issue whether it applies to a system, institution, programme or learner group. In reviewing pISA 2018, analysis should state the unit of analysis, reference period, coverage, exclusions and treatment of missing information.

Proportionality in relation to the issue does not mean reduced protection for learners exposed to greater risk. As regards pISA 2018, equal treatment does not necessarily produce equitable access or outcomes. Different support may be justified where it addresses a documented barrier without changing the required educational outcome. Review of the measure should give particular attention to adverse cases, unequal effects and errors that learners may be unable to identify or remedy after the event.

For pISA 2018, decisions concerning the analysis should remain traceable to the information available for the stated reference period. The reason for revision should be explicit, including whether it arises from new evidence, a methodological change or a different interpretation.

The principal risks associated with the analysis should be assessed as connected conditions. Governance of the available evidence requires a clear allocation of authority, information and follow-through. As regards pISA 2018, material matters should be referred to the body authorised to act or accept residual risk. Closure requires evidence that the condition has changed; completion of planned activity is not sufficient.

Data used for the analysis should be interpreted against stable definitions and an identifiable population.