Monitoring the effectiveness of reading, mathematics and science outcomes

In 2019, consideration of the effectiveness of reading, mathematics and science outcomes must take account of the PISA 2018 results released in December 2019 and the responsibilities it places before education systems. For the corrective action, corrective action, the purpose of an improvement method is not to produce an action plan; it is to change a material condition and verify that the change is sustained.

The stated reference is PISA 2018 results released in December 2019. The findings should be interpreted only at the level represented by the underlying data. For reading, mathematics and science outcomes, corrective action should be proportionate to the identified condition and tested where risk permits. Wider implementation should follow evidence of benefit and acceptable unintended effects. Comparisons and public reporting should retain material differences in coverage, timing and classification.

For reading, mathematics and science outcomes, the PISA 2018 results, released on 3 December 2019, give primary emphasis to reading and also report mathematics and science. They include evidence on performance and its distribution across participating systems. Users should distinguish the assessed constructs from broader claims about education quality, retain uncertainty in comparisons and examine whether aggregate results conceal material differences between learner groups.

Scope of the improvement

In the context of reading, mathematics and science outcomes, the central objective should not be obscured by the form of the administrative response. Oversight of the effectiveness of reading, mathematics and science outcomes should reflect the principle that education indicators should support decisions by describing outcomes and variation with definitions and limitations that permit responsible interpretation. The existence of an approved measure or completed activity is not evidence of educational effect. Assurance should address actual effect and provide a means of correcting disadvantage that the arrangement did not intend.

Improvement work on corrective action should begin with a verified problem, defined baseline and measurable outcome. In work concerning reading, mathematics and science outcomes, completion should depend on evidence of effect rather than completion of planned activity. Corrective action concerning the intended improvement should address the identified cause, assign responsibility and set a review period. Residual risk should remain open until sustained improvement is demonstrated. Completion of training, publication of guidance or installation of a system is an output and should not be reported as an outcome without further evidence.

Risk assessment of the relevant practice, the corrective action should give particular attention to proxy measures treated as direct outcomes, data revisions not carried through to published conclusions, and small differences overstated. A provider should also consider incomplete coverage and averages concealing distribution.

Implementation responsibilities

Relevant evidence for the effectiveness of reading, mathematics and science outcomes will normally include population and sampling information, disaggregated results, revision and comparability records, uncertainty estimates where relevant, and triangulation with administrative and qualitative evidence. Evidence outside the relevant period or scope should be identified and given no more weight than its limitations permit. Where reading, mathematics and science outcomes is under review, conflicting records require reconciliation before a complete assurance conclusion is reached.

Reporting on the matter should distinguish established fact, analytical judgement and planned action. In work concerning reading, mathematics and science outcomes, material revisions should retain their reason and effective date. Changes in condition, evidence, method and interpretation should be recorded separately when a conclusion is revised. Transparent treatment of reporting changes prevents artificial movement from being read as substantive progress or decline.

• Define the decision the indicator will inform.
• Avoid causal claims unsupported by the design.
• Analyse missing information.
• Disaggregate material results.
• Test comparability, with responsibility, scope and timing recorded.

Testing effectiveness

Responsibility for the effectiveness of reading, mathematics and science outcomes should be visible at the point where consequential decisions are made. Corrective action concerning corrective action should address the identified cause, assign responsibility and set a review period.

A competent review of the corrective action, the matter should set a baseline and success measure before intervention, define the review period, compare the result with the intended outcome and examine adverse or unequal effects. In the context of reading, mathematics and science outcomes, continue monitoring long enough to determine whether the improvement is sustained.

When examining reading, mathematics and science outcomes, analysis should state the unit of analysis, reference period, coverage, exclusions and treatment of missing information. Comparative findings should not conceal differences capable of changing their meaning. The result should be accompanied by distributional information, uncertainty and limits on transfer to another setting. Completion of planned activity should remain distinct from evidence that the underlying condition has improved. Unfinished work and remaining exposure should be reported rather than absorbed into a general statement of progress.

Risk assessment for the matter should consider severity, reach, duration, recurrence and detectability, with escalation where learner impact may be material. In work concerning reading, mathematics and science outcomes, the action record should identify who is responsible and when implementation is due. Evidence of outcome, rather than completion of tasks, should determine whether corrective work can close.

Maintaining the result

Proportionality in relation to the effectiveness of reading, mathematics and science outcomes does not mean reduced protection for learners exposed to greater risk. Oversight of the intended improvement, the corrective action should reflect the principle that measurement can reveal where outcomes differ; it does not by itself establish why they differ or which intervention will work. Review of the intended improvement should give particular attention to adverse cases, unequal effects and errors that learners may be unable to identify or remedy after the event. The record for reading, mathematics and science outcomes should identify the reason, approving authority, period of operation and date for reconsideration.

For reading, mathematics and science outcomes, progress is demonstrated when the intended educational result is achieved, adverse variation is identified and responsible bodies act where it is not.