Science performance data and the responsible interpretation of system differences

The PISA 2006 results released in December 2007 provides the immediate reference point for consideration of science performance data and the responsible interpretation of system differences in 2007. A decision concerning the matter examined should recognise that evidence should inform action without implying a level of precision, coverage or causal certainty that the underlying data cannot support. The scope should include every materially affected setting, with differences in location, programme, delivery mode and learner population kept visible. The conclusion remains incomplete unless central requirements are reconciled with evidence of local practice.

Purpose and present context

The reference basis—the PISA 2006 results released in December 2007—is evidential rather than self-executing. Patterns in the material may justify enquiry, although they do not by themselves determine legal position or cause. In applying it to science performance data and the responsible interpretation of system differences, users should review the source definitions, population coverage, reference period and stated limitations before transferring a system-level finding to an individual provider or learner group.

The PISA 2006 cycle gives particular emphasis to science while also assessing reading and mathematics among 15-year-old students. The results provide a comparative account of performance and its distribution across participating systems. Interpretation should take account of the sampled population, uncertainty and contextual information; a system-level association does not establish the cause of an individual learner’s result or the effectiveness of a particular provider.

Implementation of the analytical question should be organised around a decision that can be tested. The analysis of the reported measure proceeds on the basis that a sound interpretation should identify the unit of analysis, reference period, denominator, exclusions, missing values and any change in definition or collection practice. Oversight requires a traceable line from the approved objective through responsible action to evidence of outcome.

The principal risks in relation to the comparison are small differences overstated, proxy measures treated as direct outcomes, incomplete coverage, and averages concealing distribution. A weakness in one part of the control environment may obscure a related failure elsewhere. The evidential trail should be examined from initial decision to outcome, including transfers of responsibility.

The technical issue within the evidence under review concerns the basis on which a conclusion is reached. A decision concerning the matter examined should recognise that the subject should be examined as a connected system of policy, people, resources, decisions and evidence. The assessment should follow authority and information across functional boundaries and verify completion of required action. The decision record should distinguish the scope supported by evidence from any scope that remains unresolved.

Assurance of the comparison should draw on more than one form of evidence. Useful records include coverage and missingness analysis, revision and comparability records, disaggregated results, uncertainty estimates where relevant, and triangulation with administrative and qualitative evidence. Policy and records should be tested against actual practice, including evidence from learners where appropriate. Evidence of effectiveness should represent the declared scope, including adverse and exceptional cases.

The substantive quality question

Care is required in drawing conclusions about science performance data and the responsible interpretation of system differences. For the matter examined, measurement can reveal where outcomes differ; it does not by itself establish why they differ or which intervention will work. The analysis of the comparison proceeds on the basis that missing or delayed information may be patterned rather than random. Conclusions should account for the possibility that excluded learners or providers differ from those observed. A finding should not be separated from limitations capable of changing how it is understood or applied.

The assurance record for the analytical question should retain the date of the evidence, the source responsible for it, the scope examined and the version of any instrument or definition applied. The retained record should show whether later movement reflects changed conditions or a change in the information reported. The evidential history should preserve conclusions that were operative when a material decision was made.

• Analyse missing information within a defined period and review the result.
• Report uncertainty and revisions, including material exceptions and unequal effects.
• Disaggregate material results, including material exceptions and unequal effects.
• Avoid causal claims unsupported by the design, identifying the accountable function and affected scope.
• Document numerator and denominator and retain evidence sufficient for independent review.

Testing implementation and effect

The review method for science performance data and the responsible interpretation of system differences should be reproducible. The method for the comparison is to map the complete process, identify the intended result and responsible authority at each stage, and test normal cases together with exceptions. Review should establish the reach of the condition before determining the corrective response. The retained analysis should be reproducible from the selected evidence, decision rule and recorded reasons for accepted exceptions.

Decision-makers using evidence on the analytical question should be told what the data cannot establish as clearly as what it can. Users should be able to distinguish descriptive, comparative and evaluative findings and understand their proper level of application. A finding should not be transferred beyond its setting without testing the relevant contextual differences.

• Who controls each stage?
• Where do exceptions occur?
• What outcome is intended?
• Which evidence establishes operation?
• What action is required by the finding?

Limitations and safeguards

Where science performance data and the responsible interpretation of system differences involves partners, suppliers or several public bodies, responsibility should be mapped across the complete service. Governance between participating bodies should make information duties and corrective authority explicit. Learner safeguards should remain continuous where provision is delivered by several bodies.

For the matter examined, the public interest is not confined to institutional compliance. A decision concerning the evidence under review should recognise that education indicators should support decisions by describing outcomes and variation with definitions and limitations that permit responsible interpretation. Material arrangements should be communicated clearly, with an accessible route to correct error or unfair treatment.

The appropriate response to the matter examined is therefore one of controlled implementation and review. The objective should be explicit, the evidence proportionate and learner impact visible. Assurance should be withheld for the affected scope until the limitation is resolved.