{"schema_version":"ICEQC_NEWS_ARTICLE_V1","id":"iceqc-news-ad5b87e62ff46461","slug":"2010-12-10-pisa-2009-interpreting-performance-equity-and-system-level-variation","language":"en","publication_status":"READY_FOR_IMPORT","publication_date":"2010-12-10","last_modified_date":"2010-12-10","title":"PISA 2009: interpreting performance, equity and system-level variation","summary":"Examines pISA 2009: interpreting performance, equity and system-level variation, defining the evidence base, coverage and limits that should govern comparison and use of.","category":{"code":"DATA_RESEARCH","label":"Data Research"},"article_type":"Data and research analysis","publisher":"International Council for Education Quality Certification (ICEQC)","jurisdictional_scope":"International","historical_reference_basis":"PISA 2009 results released on 7 December 2010","reference_authority":"Organisation for Economic Co-operation and Development","sections":[{"heading":null,"paragraphs":["The contemporaneous reference point for PISA 2009 is PISA 2009 results released on 7 December 2010. Its status should be distinguished from the jurisdiction-specific evidence required for implementation. A decision concerning the issue should recognise that the principal analytical task is to separate an observed difference from a conclusion about its cause. The response should be proportionate to risk while preserving access, learning, fair treatment and reliable learner information.","Data used for the available evidence should be interpreted against stable definitions and an identifiable population. Changes in method, definition or series should remain separate from changes in the underlying result. They allow systems to examine performance, distribution and the relationship between outcomes and selected background conditions. Comparisons should remain within the assessment design: the data can identify patterns and disparities, but causal claims require additional evidence and policy transfer requires attention to institutional context.","Implementation of the available evidence should be organised around a decision that can be tested. the analysis proceeds on the basis that reported averages should be accompanied by sufficient distributional information to identify material differences between learner groups, locations and forms of provision. In relation to pISA 2009, the implementation record should link purpose, authority, resources, operation and reported result."]},{"heading":"Evidence base for pISA 2009","paragraphs":["The stated reference is PISA 2009 results released on 7 December 2010. Interpretation should preserve the unit and population represented in the data collection. A national or international pattern may justify closer review of the issue, but provider-level action requires evidence relating to the affected provision. The comparability record should identify material variation in coverage, period and classification.","The quality significance of pISA 2009 follows from a basic distinction between availability and effective provision. Oversight of the available evidence should reflect the principle that education systems should examine not only who enters education, but who can participate effectively, progress and complete with the intended learning outcomes. Assurance should follow the learner journey and test more than a single access point or aggregate result."],"bullets":["Identify groups not represented in aggregate results.","Review policies that create avoidable exclusion.","Target support transparently.","Analyse barriers across the learner journey.","Evaluate differential outcomes."]},{"heading":"Coverage and comparability","paragraphs":["A focused examination of PISA 2009 requires a clear analytical discipline. A decision concerning the issue should recognise that the subject should be examined as a connected system of policy, people, resources, decisions and evidence. Transfer points should be tested because responsibility and information may be lost between otherwise sound functions. A formally complete record is not reliable if its scope or measure does not correspond to the decision being made.","Risk assessment of the comparison should give particular attention to attrition detected too late, financial or geographic barriers, and admission without adequate support. A provider should also consider support allocated without evidence of need and completion measures detached from learning. Stronger controls are required where learners may not detect an error or where later correction cannot restore the lost opportunity.","Relevant evidence for the available evidence will normally include records of barriers and support, disaggregated participation and outcome data, evaluation of interventions, clearly defined access and completion indicators, and progression and early-warning information. In relation to pISA 2009, the conclusion should rely on evidence whose date, source and coverage are sufficient for the decision. Contradictory evidence should be investigated and resolved, not omitted from the record."],"bullets":["Which evidence establishes operation?","What action is required by the finding?","Who controls each stage?","Where do exceptions occur?","What outcome is intended?"]},{"heading":"Responsible interpretation","paragraphs":["The review method for PISA 2009 should be reproducible. Review of the analysis should map the complete process, identify the intended result and responsible authority at each stage, and test normal cases together with exceptions. The retained analysis should be reproducible from the selected evidence, decision rule and recorded reasons for accepted exceptions.","Publication of findings on the available evidence should distinguish observed values, estimates and interpretation.","Conclusions concerning the comparison require careful treatment of scope and evidential limits. The analysis of the measure proceeds on the basis that equal treatment does not necessarily produce equitable access or outcomes. Different support may be justified where it addresses a documented barrier without changing the required educational outcome. In reviewing the available evidence, association should not be presented as causation, and statistical significance should not be treated as evidence of educational importance without further analysis. A finding should not be separated from limitations capable of changing how it is understood or applied.","Traceability is necessary for accountable decision-making and fair correction. In this case, the responsible body should be able to identify the evidence considered, the judgement made, the person or body authorised to make it and the action that followed. A material amendment should record its reason and effective date, preserving the information basis of earlier decisions.","For The comparison, governing bodies should receive a concise account of the intended result, affected scope, principal risks, evidence limitations and unresolved exceptions. The action record should identify who is responsible and when implementation is due. Closure requires evidence that the condition has changed; completion of planned activity is not sufficient.","The measure of progress on the measure is not the amount of policy or documentation produced. Performance should be judged by outcomes and timely response to shortfalls, not by the volume of administrative activity."]}],"word_count":873,"content_hash":"sha256-249e2193f58eb8ebc7f214eac62e47f4f6c152947793dca0447510a037fd642a","seo_keywords":["pISA 2009","international education data research","education quality","ICEQC"],"schema_type":"AnalysisNewsArticle"}