{"schema_version":"ICEQC_NEWS_ARTICLE_V1","id":"iceqc-news-3bd382cd4958fb4a","slug":"2023-12-08-pisa-2022-interpreting-the-decline-in-mathematics-and-reading-performance","language":"en","publication_status":"READY_FOR_IMPORT","publication_date":"2023-12-08","last_modified_date":"2023-12-08","title":"PISA 2022: interpreting the decline in mathematics and reading performance","summary":"Examines pISA 2022: interpreting the decline in mathematics and reading performance, defining the evidence base, coverage and limits that should govern comparison and use.","category":{"code":"DATA_RESEARCH","label":"Data Research"},"article_type":"Data and research analysis","publisher":"International Council for Education Quality Certification (ICEQC)","jurisdictional_scope":"International","historical_reference_basis":"PISA 2022 results released on 5 December 2023","reference_authority":"Organisation for Economic Co-operation and Development","sections":[{"heading":null,"paragraphs":["Consideration of PISA 2022 should retain the date and status of PISA 2022 results released on 5 December 2023. Later developments should not be read into the position available at publication. Its status should be distinguished from the jurisdiction-specific evidence required for implementation. In reviewing the measure, the principal analytical task is to separate an observed difference from a conclusion about its cause. In relation to pISA 2022, the appropriate administrative form will depend on the jurisdiction and the allocation of lawful responsibility.","Consideration of the issue should retain the date and status of PISA 2022 results released on 5 December 2023. The cycle was conducted after major disruption to education systems. Comparisons require attention to participation, coverage and the exceptional context; changes from earlier cycles should not be attributed to a single cause without further evidence.","The governing expectation for the analysis should be capable of consistent application. In reviewing the available evidence, a sound interpretation should identify the unit of analysis, reference period, denominator, exclusions, missing values and any change in definition or collection practice. Criteria affecting learners should not permit materially different interpretation without an evidenced reason."]},{"heading":"Evidence base for pISA 2022","paragraphs":["The reference basis—the PISA 2022 results released on 5 December 2023—is evidential rather than self-executing. The source can frame further questions without supplying either a binding direction or proof of causation. In applying it to the measure, users should review the source definitions, population coverage, reference period and stated limitations before transferring a system-level finding to an individual provider or learner group.","For the analysis, the public interest is not confined to institutional compliance. In this case, education indicators should support decisions by describing outcomes and variation with definitions and limitations that permit responsible interpretation. Where learners rely on published information or support decisions, errors should be identifiable and capable of prompt, fair correction."],"bullets":["Analyse missing information.","Report uncertainty and revisions, identifying the accountable function and affected scope.","Document numerator and denominator.","Define the decision the indicator will inform.","Disaggregate material results."]},{"heading":"Coverage and comparability","paragraphs":["The technical issue within PISA 2022 concerns the basis on which a conclusion is reached. For the available evidence, the subject should be examined as a connected system of policy, people, resources, decisions and evidence. Transfer of decisions or records can expose weaknesses not visible in separate reviews of individual controls. A conclusion should identify both its evidential basis and the part of the stated scope for which assurance cannot be given.","Failure in relation to the available evidence may arise even where the stated policy is reasonable. Material concerns include small differences overstated, data revisions not carried through to published conclusions, averages concealing distribution, and changes in definition presented as changes in performance. Review should consider whether an exception is prolonged, recurring or capable of affecting learners outside the cases examined.","The evidential record for the analysis should permit a reviewer to trace the matter from decision to outcome. This may require triangulation with administrative and qualitative evidence, population and sampling information, uncertainty estimates where relevant, and disaggregated results, supported by revision and comparability records and indicator definitions and metadata. In relation to pISA 2022, further cases should be examined when the initial sample does not represent the affected scope or confirm sustained correction."],"bullets":["What outcome is intended?","Which evidence establishes operation?","Who controls each stage?","Where do exceptions occur?","What action is required by the finding?"]},{"heading":"Responsible interpretation","paragraphs":["For operational review of PISA 2022, authorities and providers should proceed in a defined sequence. A competent review of the analysis should map the complete process, identify the intended result and responsible authority at each stage, and test normal cases together with exceptions. The conclusion should identify whether further sampling or system-level action is required. Findings should state the affected scope and required action; an observation should not be represented as evidence of conformity or effectiveness.","The analytical record for the comparison should state the research question, data source, unit of analysis, reference period, coverage, exclusions, treatment of missing values and principal limitations.","The basis and limits of any conclusion concerning the available evidence should be explicit. A decision concerning the available evidence should recognise that measurement can reveal where outcomes differ; it does not by itself establish why they differ or which intervention will work. For the measure, association should not be presented as causation, and statistical significance should not be treated as evidence of educational importance without further analysis. Decision-makers and affected users should receive the conclusion together with its material evidential limits.","The assurance record for the comparison should retain the date of the evidence, the source responsible for it, the scope examined and the version of any instrument or definition applied. A later reviewer should be able to identify whether the condition changed or the evidential record was corrected. A superseded conclusion should be retained where it formed the basis of a material decision.","Public reporting on the available evidence should distinguish established fact, analytical judgement and planned action. A revised conclusion should distinguish a change in the underlying condition from a change in method, coverage or evidence.","The measure of progress on the comparison is not the amount of policy or documentation produced."]}],"word_count":860,"content_hash":"sha256-e2d776b1166ec904af55b601196adfc2c90521aced031d0f7e864fd1c6bd8e5d","seo_keywords":["pISA 2022","international education data research","education quality","ICEQC"],"schema_type":"AnalysisNewsArticle"}