The analysis of PISA 2018 identifies what the evidence establishes, where comparison is limited and which qualifications must be reported.
At the publication date, PISA 2018 results released on 3 December 2019 provides the relevant international context for the measure. Any consequential application still requires evidence from the affected jurisdiction or institution. Evidence concerning pISA 2018 should inform action without implying a level of precision, coverage or causal certainty that the underlying data cannot support. Data used for PISA 2018 should be interpreted against stable definitions and an identifiable population.
The contemporaneous reference point for the analysis is PISA 2018 results released on 3 December 2019. The evidential basis for the measure should identify source, period, coverage and material limitations. Corroboration is required where a single record cannot support the decision. Data used for the measure should be interpreted against stable definitions and an identifiable population.
Evidence base for pISA 2018
Consideration of PISA 2018 should retain the date and status of PISA 2018 results released on 3 December 2019. Any indicator used in relation to the available evidence should distinguish description from causal explanation.
In the context of pISA 2018, analysis should state the unit of analysis, reference period, coverage, exclusions and treatment of missing information.
For pISA 2018, a formally complete record is not reliable if its scope or measure does not correspond to the decision being made.
Data used for the available evidence should be interpreted against stable definitions and an identifiable population.
Assurance of the analysis should draw on more than one form of evidence. In reviewing pISA 2018, a comparison is reliable only if material differences remain visible. Across the defined scope, a positive example may illustrate operation, but it cannot demonstrate coverage or consistency.
- Review policies that create avoidable exclusion, identifying the accountable function and affected scope.
- Analyse barriers across the learner journey.
- Target support transparently before it informs a consequential decision.
- Report limitations in available data.
- Evaluate differential outcomes.
Controls for pISA 2018
Risk assessment of PISA 2018 should give particular attention to financial or geographic barriers, completion measures detached from learning, and admission without adequate support. The position on the analysis should be established through proportionate evidence and should remain open to correction when material new information becomes available.
The review method for the available evidence should be reproducible. Review of the analysis should give particular attention to adverse cases, unequal effects and errors that learners may be unable to identify or remedy after the event. Risk assessment for the available evidence should consider severity, reach, duration, recurrence and detectability, with escalation where learner impact may be material.
Decision-makers using evidence on the comparison should be told what the data cannot establish as clearly as what it can. The conclusion should state whether the result is descriptive, comparative or evaluative, the issue whether it applies to a system, institution, programme or learner group. In reviewing pISA 2018, analysis should state the unit of analysis, reference period, coverage, exclusions and treatment of missing information.
Proportionality in relation to the issue does not mean reduced protection for learners exposed to greater risk. For pISA 2018, equal treatment does not necessarily produce equitable access or outcomes. Review of the measure should give particular attention to adverse cases, unequal effects and errors that learners may be unable to identify or remedy after the event.
For pISA 2018, decisions concerning the analysis should remain traceable to the information available for the stated reference period.
The principal risks associated with the analysis should be assessed as connected conditions. Governance of the available evidence requires a clear allocation of authority, information and follow-through. For pISA 2018, material matters should be referred to the body authorised to act or accept residual risk.
Data used for the analysis should be interpreted against stable definitions and an identifiable population.