Evidence gaps affecting reading, mathematics and science outcomes

In 2019, consideration of reading, mathematics and science outcomes must take account of the PISA 2018 results released in December 2019 and the responsibilities it places before education systems. The comparison, the principal analytical task is to separate an observed difference from a conclusion about its cause. The control response should be sufficient to protect learners while avoiding burdens not justified by the evidence.

Evidence base for evidence gaps affecting reading, mathematics and science outcomes

The reference basis—the PISA 2018 results released in December 2019—is evidential rather than self-executing. Patterns in the material may justify enquiry, although they do not by themselves determine legal position or cause. In applying it to reading, mathematics and science outcomes, users should review the source definitions, population coverage, reference period and stated limitations before transferring a system-level finding to an individual provider or learner group.

For reading, mathematics and science outcomes, the PISA 2018 results, released on 3 December 2019, give primary emphasis to reading and also report mathematics and science. They include evidence on performance and its distribution across participating systems. Users should distinguish the assessed constructs from broader claims about education quality, retain uncertainty in comparisons and examine whether aggregate results conceal material differences between learner groups.

Coverage and comparability

For reading, mathematics and science outcomes, the public interest is not confined to institutional compliance. Analysis should state the unit of analysis, reference period, coverage, exclusions and treatment of missing information. A comparison is reliable only if material differences remain visible.

Evidence concerning the issue should be current, attributable and representative of the affected scope. In work concerning reading, mathematics and science outcomes, material gaps or contradictions should remain visible in the conclusion. A conclusion on the analysis should extend no further than the available evidence permits. Missing populations, inconsistent records and unresolved exceptions should be reported with the finding. Assurance should examine the complete process, including each change in custody, authority or decision ownership.

• Which evidence establishes operation?
• Where do exceptions occur?
• What outcome is intended?
• What action is required by the finding?
• Who controls each stage?

Responsible interpretation

Responsibility for reading, mathematics and science outcomes should be visible at the point where consequential decisions are made. Data used for the available evidence should be interpreted against stable definitions and an identifiable population. A revision or break in series should not be reported as a change in performance. Incomplete evidence, unmanaged conflict, absent learner groups or material learner impact require a higher level of review.

For decisions concerning reading, mathematics and science outcomes, analysis should state the unit of analysis, reference period, coverage, exclusions and treatment of missing information. Material differences in population, setting or method should remain explicit in any comparison. A provider should also consider small differences overstated and incomplete coverage. Stronger controls are required where learners may not detect an error or where later correction cannot restore the lost opportunity.

Limitations and reporting

Assurance of reading, mathematics and science outcomes should draw on more than one form of evidence. Useful records include revision and comparability records, triangulation with administrative and qualitative evidence, disaggregated results, population and sampling information, and uncertainty estimates where relevant. A positive example may illustrate operation, but it cannot demonstrate coverage or consistency.

The review should follow a stated and reproducible method, including the decision rule, sampling basis, treatment of exceptions and threshold for escalation. Review of the comparison should give particular attention to adverse cases, unequal effects and errors that learners may be unable to identify or remedy after the event. In work concerning reading, mathematics and science outcomes, an isolated incident and a recurring or systemic condition require different findings and responses. A competent reviewer should be able to follow the record from source selection to conclusion and exception handling.

The analytical record For comparative analysis, the available evidence should state the research question, data source, unit of analysis, reference period, coverage, exclusions, treatment of missing values and principal limitations.

Limitations and reporting

Interpretation of reading, mathematics and science outcomes should avoid two errors: treating a formal commitment as proof of effect, and treating one adverse case as proof that every part of the system has failed. Data used for the analysis should be interpreted against stable definitions and an identifiable population. The result should be accompanied by distributional information, uncertainty and limits on transfer to another setting. Where reading, mathematics and science outcomes is under review, changes in method, definition or series should remain separate from changes in the underlying result.

Responsibility for the available evidence should be identifiable at each consequential decision point. As regards reading, mathematics and science outcomes, delegating operational work does not transfer accountability for its effect on learners. A later reviewer should be able to identify whether the condition changed or the evidential record was corrected. Earlier conclusions should remain traceable if they affected a learner, provider or public decision.

Any indicator used in relation to the available evidence should distinguish description from causal explanation. In the context of reading, mathematics and science outcomes, material variation and uncertainty should be reported together with any restriction on wider application. A reported result should state how outcomes are distributed and where transfer beyond the observed setting is not supported.

For the available evidence, the implementation record should distinguish binding duties, policy expectations and institutional choices, including any transition or jurisdictional limitation. The decision record for reading, mathematics and science outcomes should state the unsupported element and the further work required.