{"schema_version":"ICEQC_NEWS_ARTICLE_V1","id":"iceqc-news-dc9e57c272ed5dda","slug":"2016-12-08-pisa-2015-science-performance-equity-and-the-limits-of-league-table-interpretation","language":"en","publication_status":"READY_FOR_IMPORT","publication_date":"2016-12-08","last_modified_date":"2016-12-08","title":"PISA 2015: science performance, equity and the limits of league-table interpretation","summary":"Reviews pISA 2015: science performance, equity and the limits of league-table interpretation and identifies the conditions required for sound interpretation, responsible.","category":{"code":"DATA_RESEARCH","label":"Data Research"},"article_type":"Data and research analysis","publisher":"International Council for Education Quality Certification (ICEQC)","jurisdictional_scope":"International","historical_reference_basis":"PISA 2015 results released on 6 December 2016","reference_authority":"Organisation for Economic Co-operation and Development","sections":[{"heading":null,"paragraphs":["At the publication date, PISA 2015 results released on 6 December 2016 provides the relevant international context for the comparison. Any consequential application still requires evidence from the affected jurisdiction or institution. Oversight of the comparison should reflect the principle that evidence should inform action without implying a level of precision, coverage or causal certainty that the underlying data cannot support.","Records concerning the issue should remain traceable from source evidence to decision and follow-up. Superseded conclusions should be retained where they informed a material outcome. They provide internationally comparable evidence for the sampled population, together with contextual information relevant to equity and learning conditions. The results can direct attention to material variation, but they do not replace provider-level evidence or establish that one observed policy feature caused the reported outcome.","Implementation of the available evidence should be organised around a decision that can be tested. Oversight of the measure should reflect the principle that trend claims require comparable observations over time and a documented account of revisions, breaks in series and changes in coverage. Governance of the comparison requires a clear allocation of authority, information and follow-through. Escalation should place material evidence before the authority capable of an effective response."]},{"heading":"Evidence base for pISA 2015","paragraphs":["The reference basis—the PISA 2015 results released on 6 December 2016—is evidential rather than self-executing. The source can frame further questions without supplying either a binding direction or proof of causation. In applying it to the available evidence, users should review the source definitions, population coverage, reference period and stated limitations before transferring a system-level finding to an individual provider or learner group.","The quality significance of the measure follows from a basic distinction between availability and effective provision. A decision concerning the comparison should recognise that education systems should examine not only who enters education, but who can participate effectively, progress and complete with the intended learning outcomes. In relation to pISA 2015, review should cover the stages at which learners receive information, provision, assessment, support and remedy."],"bullets":["Evaluate differential outcomes before it is relied on for a decision with material effect.","Review policies that create avoidable exclusion.","Analyse barriers across the learner journey.","Identify groups not represented in aggregate results, identifying the accountable function and affected scope.","Monitor progression at useful intervals before using it to determine a learner or provider outcome."]},{"heading":"Coverage and comparability","paragraphs":["A focused examination of PISA 2015 requires a clear analytical discipline. A decision concerning the available evidence should recognise that the subject should be examined as a connected system of policy, people, resources, decisions and evidence. A control framework may fail at its interfaces even where each component appears satisfactory in isolation. The decision question, affected scope and measure should align; otherwise the conclusion may be unsupported despite substantial documentation.","Risk assessment of the issue should give particular attention to completion measures detached from learning, attrition detected too late, and admission without adequate support. A provider should also consider support allocated without evidence of need and aggregate gains concealing excluded groups. Stronger controls are required where learners may not detect an error or where later correction cannot restore the lost opportunity.","The evidential record for the comparison should permit a reviewer to trace the matter from decision to outcome. This may require records of barriers and support, resource allocation by need, progression and early-warning information, and clearly defined access and completion indicators, supported by disaggregated participation and outcome data and learner feedback and complaints. In relation to pISA 2015, further cases should be examined when the initial sample does not represent the affected scope or confirm sustained correction."],"bullets":["Which evidence establishes operation?","What outcome is intended?","What action is required by the finding?","Where do exceptions occur?","Who controls each stage?"]},{"heading":"Responsible interpretation","paragraphs":["A proportionate method is available for PISA 2015. Review of the comparison should map the complete process, identify the intended result and responsible authority at each stage, and test normal cases together with exceptions. The conclusion should identify whether further sampling or system-level action is required. Adverse cases and unresolved contradictions should be retained because they may reveal limitations concealed by an average result.","Decision-makers using evidence on the comparison should be told what the data cannot establish as clearly as what it can. The nature of the result and its applicable unit—system, institution, programme or learner group—should be explicit. Use in a different context requires an independent judgement that the settings are materially comparable.","Proportionality in relation to the issue does not mean reduced protection for learners exposed to greater risk. For the available evidence, equal treatment does not necessarily produce equitable access or outcomes. Different support may be justified where it addresses a documented barrier without changing the required educational outcome. A decision concerning the issue should recognise that a single indicator rarely provides an adequate account of quality. Quantitative evidence should be considered with implementation records and the experience of affected learners.","The evidential trail should allow an affected decision to be identified, examined and corrected. Data used for the issue should be interpreted against stable definitions and an identifiable population. Reporting should identify a break in comparability before describing movement over time. A material amendment should record its reason and effective date, preserving the information basis of earlier decisions.","Public reporting on the analysis should distinguish established fact, analytical judgement and planned action. A revised conclusion should distinguish a change in the underlying condition from a change in method, coverage or evidence.","The appropriate response to the comparison is therefore one of controlled implementation and review. Arrangements for the issue should provide accurate information, timely support and an accessible route for correction or review without adverse treatment. Any indicator used in relation to the analysis should distinguish description from causal explanation. A reported result should state how outcomes are distributed and where transfer beyond the observed setting is not supported."]}],"word_count":974,"content_hash":"sha256-efedfccbc43cea6b2325559c5d90cd3ef18ce6ce2ea6981d120736bfcb9793b4","seo_keywords":["pISA 2015","international education data research","education quality","ICEQC"],"schema_type":"AnalysisNewsArticle"}