Evidence gaps affecting reading, mathematics and science outcomes

The comparison, the principal analytical task is to separate an observed difference from a conclusion about its cause.

Evidence base for evidence gaps affecting reading, mathematics and science outcomes

Patterns in the material may justify enquiry, although they do not by themselves determine legal position or cause. In applying it to reading, mathematics and science outcomes, users should review the source definitions, population coverage, reference period and stated limitations before transferring a system-level finding to an individual provider or learner group.

In examining evidence gaps affecting reading, mathematics and science outcomes, for reading, mathematics and science outcomes, the PISA 2018 results, released on 3 December 2019, give primary emphasis to reading and also report mathematics and science. They include evidence on performance and its distribution across participating systems. Users should distinguish the assessed constructs from broader claims about education quality, retain uncertainty in comparisons and examine whether aggregate results conceal material differences between learner groups.

Controls relevant to evidence gaps affecting reading, mathematics and science outcomes

For reading, mathematics and science outcomes, the public interest is not confined to institutional compliance.

Evidence concerning the issue should be current, attributable and representative of the affected scope. For reading, mathematics and science outcomes, material gaps or contradictions should remain visible in the conclusion. A conclusion on the analysis should extend no further than the available evidence permits. Missing populations, inconsistent records and unresolved exceptions should be reported with the finding.

• Which evidence establishes operation?
• Where do exceptions occur?
• What outcome is intended?
• What action is required by the finding?
• Who controls each stage?

Review criteria for evidence gaps affecting reading, mathematics and science outcomes

Responsibility for reading, mathematics and science outcomes should be visible at the point where consequential decisions are made. Data used for the available evidence should be interpreted against stable definitions and an identifiable population.

For decisions concerning reading, mathematics and science outcomes, analysis should state the unit of analysis, reference period, coverage, exclusions and treatment of missing information. A provider should also consider small differences overstated and incomplete coverage. Stronger controls are required where learners may not detect an error or where later correction cannot restore the lost opportunity.

Implications for evidence gaps affecting reading, mathematics and science outcomes

Assurance of reading, mathematics and science outcomes should draw on more than one form of evidence. Useful records include revision and comparability records, triangulation with administrative and qualitative evidence, disaggregated results, population and sampling information, and uncertainty estimates where relevant.

The review should follow a stated and reproducible method, including the decision rule, sampling basis, treatment of exceptions and threshold for escalation. Review of the comparison should give particular attention to adverse cases, unequal effects and errors that learners may be unable to identify or remedy after the event. For reading, mathematics and science outcomes, an isolated incident and a recurring or systemic condition require different findings and responses.

The analytical record For comparative analysis, the available evidence should state the research question, data source, unit of analysis, reference period, coverage, exclusions, treatment of missing values and principal limitations.

Evidence considered for evidence gaps affecting reading, mathematics and science outcomes

Interpretation of reading, mathematics and science outcomes should avoid two errors: treating a formal commitment as proof of effect, and treating one adverse case as proof that every part of the system has failed. Data used for the analysis should be interpreted against stable definitions and an identifiable population. Where reading, mathematics and science outcomes is under review, changes in method, definition or series should remain separate from changes in the underlying result.

For reading, mathematics and science outcomes, delegating operational work does not transfer accountability for its effect on learners. A later reviewer should be able to identify whether the condition changed or the evidential record was corrected.

Any indicator used in relation to the available evidence should distinguish description from causal explanation. In the context of reading, mathematics and science outcomes, material variation and uncertainty should be reported together with any restriction on wider application.

For the available evidence, the implementation record should distinguish binding duties, policy expectations and institutional choices, including any transition or jurisdictional limitation. The decision record for reading, mathematics and science outcomes should state the unsupported element and the further work required.