{"schema_version":"ICEQC_NEWS_ARTICLE_V1","id":"iceqc-news-4a7324b86e88c6ab","slug":"2019-12-09-monitoring-the-effectiveness-of-reading-mathematics-and-science-outcomes","language":"en","publication_status":"PUBLISHED","publication_date":"2019-12-09","last_modified_date":"2019-12-09","title":"Monitoring the effectiveness of reading, mathematics and science outcomes","summary":"Describes how improvement in reading, mathematics and science outcomes can be planned, measured and maintained through clear responsibility and follow-up evidence.","category":{"code":"QUALITY_IMPROVEMENT_METHODS","label":"Quality Improvement Methods"},"article_type":"Quality improvement method","publisher":"International Council for Education Quality Certification (ICEQC)","jurisdictional_scope":"International","historical_reference_basis":"PISA 2018 results released in December 2019","reference_authority":"Organisation for Economic Co-operation and Development","sections":[{"heading":null,"paragraphs":["In 2019, consideration of the effectiveness of reading, mathematics and science outcomes must take account of the PISA 2018 results released in December 2019 and the responsibilities it places before education systems. For the intervention, the corrective programme, the purpose of an improvement method is not to produce an action plan; it is to change a material condition and verify that the change is sustained.","The stated reference is PISA 2018 results released in December 2019. The findings should be interpreted only at the level represented by the underlying data. Intervention in the corrective programme should be proportionate to the identified condition and tested where risk permits. Wider implementation should follow evidence of benefit and acceptable unintended effects. Comparisons and public reporting should retain material differences in coverage, timing and classification.","For reading, mathematics and science outcomes, the PISA 2018 results, released on 3 December 2019, give primary emphasis to reading and also report mathematics and science. They include evidence on performance and its distribution across participating systems. Users should distinguish the assessed constructs from broader claims about education quality, retain uncertainty in comparisons and examine whether aggregate results conceal material differences between learner groups."]},{"heading":"Scope of the improvement","paragraphs":["In the context of reading, mathematics and science outcomes, the central objective should not be obscured by the form of the administrative response. Oversight of the effectiveness of reading, mathematics and science outcomes should reflect the principle that education indicators should support decisions by describing outcomes and variation with definitions and limitations that permit responsible interpretation. When examining reading, mathematics and science outcomes, the existence of an approved measure or completed activity is not evidence of educational effect. For decisions concerning reading, mathematics and science outcomes, assurance should address actual effect and provide a means of correcting disadvantage that the arrangement did not intend.","Improvement work on the corrective programme should begin with a verified problem, defined baseline and measurable outcome. Completion should depend on evidence of effect rather than completion of planned activity. Corrective action concerning the improvement objective should address the identified cause, assign responsibility and set a review period. Residual risk should remain open until sustained improvement is demonstrated. In work concerning reading, mathematics and science outcomes, completion of training, publication of guidance or installation of a system is an output and should not be reported as an outcome without further evidence.","Risk assessment of the practice, the intervention should give particular attention to proxy measures treated as direct outcomes, data revisions not carried through to published conclusions, and small differences overstated. A provider should also consider incomplete coverage and averages concealing distribution."]},{"heading":"Implementation responsibilities","paragraphs":["Relevant evidence for the effectiveness of reading, mathematics and science outcomes will normally include population and sampling information, disaggregated results, revision and comparability records, uncertainty estimates where relevant, and triangulation with administrative and qualitative evidence. As regards reading, mathematics and science outcomes, evidence outside the relevant period or scope should be identified and given no more weight than its limitations permit. Where reading, mathematics and science outcomes is under review, conflicting records require reconciliation before a complete assurance conclusion is reached.","Reporting on the issue under review should distinguish established fact, analytical judgement and planned action. Material revisions should retain their reason and effective date. For the assessment of reading, mathematics and science outcomes, changes in condition, evidence, method and interpretation should be recorded separately when a conclusion is revised. For reading, mathematics and science outcomes, transparent treatment of reporting changes prevents artificial movement from being read as substantive progress or decline."],"bullets":["Define the decision the indicator will inform.","Avoid causal claims unsupported by the design.","Analyse missing information.","Disaggregate material results.","Test comparability, with responsibility, scope and timing recorded."]},{"heading":"Testing effectiveness","paragraphs":["Responsibility for the effectiveness of reading, mathematics and science outcomes should be visible at the point where consequential decisions are made. Corrective action concerning the corrective programme should address the identified cause, assign responsibility and set a review period.","A competent review of the intervention, the issue under review should set a baseline and success measure before intervention, define the review period, compare the result with the intended outcome and examine adverse or unequal effects. In the context of reading, mathematics and science outcomes, continue monitoring long enough to determine whether the improvement is sustained.","When examining reading, mathematics and science outcomes, analysis should state the unit of analysis, reference period, coverage, exclusions and treatment of missing information. For decisions concerning reading, mathematics and science outcomes, comparative findings should not conceal differences capable of changing their meaning. The result should be accompanied by distributional information, uncertainty and limits on transfer to another setting. Completion of planned activity should remain distinct from evidence that the underlying condition has improved. Unfinished work and remaining exposure should be reported rather than absorbed into a general statement of progress.","Risk assessment for the issue under review should consider severity, reach, duration, recurrence and detectability, with escalation where learner impact may be material. In work concerning reading, mathematics and science outcomes, the action record should identify who is responsible and when implementation is due. As regards reading, mathematics and science outcomes, evidence of outcome, rather than completion of tasks, should determine whether corrective work can close."]},{"heading":"Maintaining the result","paragraphs":["Proportionality in relation to the effectiveness of reading, mathematics and science outcomes does not mean reduced protection for learners exposed to greater risk. Oversight of the improvement objective, the intervention should reflect the principle that measurement can reveal where outcomes differ; it does not by itself establish why they differ or which intervention will work. Review of the improvement objective should give particular attention to adverse cases, unequal effects and errors that learners may be unable to identify or remedy after the event. The record for reading, mathematics and science outcomes should identify the reason, approving authority, period of operation and date for reconsideration.","For the assessment of reading, mathematics and science outcomes, progress is demonstrated when the intended educational result is achieved, adverse variation is identified and responsible bodies act where it is not."]}],"word_count":997,"content_hash":"sha256-3c256f6c05202318ed15c15af6d93cce5143c7aed039d868c4fb06dac7992468","seo_keywords":["monitoring the effectiveness of reading, mathematics and science outcomes","education quality improvement","education quality","ICEQC"],"schema_type":"TechArticle"}