The reliability of evidence on PISA 2009 is examined together with the limits that apply when findings inform consequential decisions.
The contemporaneous reference point for PISA 2009 is PISA 2009 results released on 7 December 2010.
Data used for the available evidence should be interpreted against stable definitions and an identifiable population. For pISA 2009, changes in method, definition or series should remain separate from changes in the underlying result. They allow systems to examine performance, distribution and the relationship between outcomes and selected background conditions. Comparisons should remain within the assessment design: the data can identify patterns and disparities, but causal claims require additional evidence and policy transfer requires attention to institutional context.
Implementation of the available evidence should be organised around a decision that can be tested. The analysis proceeds on the basis that reported averages should be accompanied by sufficient distributional information to identify material differences between learner groups, locations and forms of provision. In the context of pISA 2009, the implementation record should link purpose, authority, resources, operation and reported result.
Evidence base for pISA 2009
Interpretation should preserve the unit and population represented in the data collection. A national or international pattern may justify closer review of the issue, but provider-level action requires evidence relating to the affected provision. The comparability record should identify material variation in coverage, period and classification.
When examining pISA 2009, education systems should examine not only who enters education, but who can participate effectively, progress and complete with the intended learning outcomes.
- Identify groups not represented in aggregate results.
- Review policies that create avoidable exclusion.
- Target support transparently.
- Analyse barriers across the learner journey.
- Evaluate differential outcomes.
Controls for pISA 2009
Risk assessment of the comparison should give particular attention to attrition detected too late, financial or geographic barriers, and admission without adequate support. A provider should also consider support allocated without evidence of need and completion measures detached from learning. In reviewing pISA 2009, stronger controls are required where learners may not detect an error or where later correction cannot restore the lost opportunity.
Relevant evidence for the available evidence will normally include records of barriers and support, disaggregated participation and outcome data, evaluation of interventions, clearly defined access and completion indicators, and progression and early-warning information. Across the defined scope, contradictory evidence should be investigated and resolved, not omitted from the record.
- Which evidence establishes operation?
- What action is required by the finding?
- Who controls each stage?
- Where do exceptions occur?
- What outcome is intended?
Review of pISA 2009
The review method for PISA 2009 should be reproducible. Review of the analysis should map the complete process, identify the intended result and responsible authority at each stage, and test normal cases together with exceptions.
Publication of findings on pISA 2009 should distinguish observed values, estimates and interpretation.
Conclusions concerning the comparison require careful treatment of scope and evidential limits. For pISA 2009, equal treatment does not necessarily produce equitable access or outcomes.
For decisions concerning pISA 2009, traceability is necessary for accountable decision-making and fair correction.
For comparative analysis, governing bodies should receive a concise account of the intended result, affected scope, principal risks, evidence limitations and unresolved exceptions. For pISA 2009, the action record should identify who is responsible and when implementation is due. Across the defined scope, closure requires evidence that the condition has changed; completion of planned activity is not sufficient.
Performance in relation to pISA 2009 should be judged by outcomes and timely response to shortfalls, not by the volume of administrative activity.