Science performance data and the responsible interpretation of system differences

The PISA 2006 results released in December 2007 provides the immediate reference point for consideration of science performance data and the responsible interpretation of system differences in 2007. Evidence concerning science performance data and the responsible interpretation of system differences should inform action without implying a level of precision, coverage or causal certainty that the underlying data cannot support.

Evidence base

The reference basis—the PISA 2006 results released in December 2007—is evidential rather than self-executing. Patterns in the material may justify enquiry, although they do not by themselves determine legal position or cause. In applying it to science performance data and the responsible interpretation of system differences, users should review the source definitions, population coverage, reference period and stated limitations before transferring a system-level finding to an individual provider or learner group.

For science performance data and the responsible interpretation of system differences, the PISA 2006 cycle gives particular emphasis to science while also assessing reading and mathematics among 15-year-old students. The results provide a comparative account of performance and its distribution across participating systems. Interpretation should take account of the sampled population, uncertainty and contextual information; a system-level association does not establish the cause of an individual learner’s result or the effectiveness of a particular provider.

Implementation of the analysis should be organised around a decision that can be tested. In the context of science performance data and the responsible interpretation of system differences, a sound interpretation should identify the unit of analysis, reference period, denominator, exclusions, missing values and any change in definition or collection practice. Oversight requires a traceable line from the approved objective through responsible action to evidence of outcome.

The principal risks in relation to the comparison are small differences overstated, proxy measures treated as direct outcomes, incomplete coverage, and averages concealing distribution. For decisions concerning science performance data and the responsible interpretation of system differences, a weakness in one part of the control environment may obscure a related failure elsewhere.

In work concerning science performance data and the responsible interpretation of system differences, the subject should be examined as a connected system of policy, people, resources, decisions and evidence. The assessment should follow authority and information across functional boundaries and verify completion of required action. The decision record for science performance data and the responsible interpretation of system differences should distinguish the scope supported by evidence from any scope that remains unresolved.

Assurance of the comparison should draw on more than one form of evidence. Useful records include coverage and missingness analysis, revision and comparability records, disaggregated results, uncertainty estimates where relevant, and triangulation with administrative and qualitative evidence. Within the scope under review, policy and records should be tested against actual practice, including evidence from learners where appropriate. For science performance data and the responsible interpretation of system differences, evidence of effectiveness should represent the declared scope, including adverse and exceptional cases.

Coverage and comparability

Care is required in drawing conclusions about science performance data and the responsible interpretation of system differences. In this case, measurement can reveal where outcomes differ; it does not by itself establish why they differ or which intervention will work. Missing or delayed information may be patterned rather than random. A finding should not be separated from limitations capable of changing how it is understood or applied.

The assurance record for science performance data and the responsible interpretation of system differences should retain the date of the evidence, the source responsible for it, the scope examined and the version of any instrument or definition applied. The evidential history should preserve conclusions that were operative when a material decision was made.

• Analyse missing information.
• Report uncertainty and revisions.
• Disaggregate material results.
• Avoid causal claims unsupported by the design, identifying the accountable function and affected scope.
• Document numerator and denominator.

Responsible interpretation

The review method for science performance data and the responsible interpretation of system differences should be reproducible. The method for the comparison is to map the complete process, identify the intended result and responsible authority at each stage, and test normal cases together with exceptions. Review should establish the reach of the condition before determining the corrective response. The retained analysis should be reproducible from the selected evidence, decision rule and recorded reasons for accepted exceptions.

As regards science performance data and the responsible interpretation of system differences, decision-makers using evidence on the analysis should be told what the data cannot establish as clearly as what it can. Users should be able to distinguish descriptive, comparative and evaluative findings and understand their proper level of application. A finding should not be transferred beyond its setting without testing the relevant contextual differences.

• Who controls each stage?
• Where do exceptions occur?
• What outcome is intended?
• Which evidence establishes operation?
• What action is required by the finding?

Limitations and reporting

As regards science performance data and the responsible interpretation of system differences, where responsibilities for delivery are shared with partners, suppliers or several public bodies, responsibility should be mapped across the complete service. Within the scope under review, governance between participating bodies should make information duties and corrective authority explicit. Learner safeguards associated with science performance data and the responsible interpretation of system differences should remain continuous where provision is delivered by several bodies.

In this case, the public interest is not confined to institutional compliance. For decisions concerning science performance data and the responsible interpretation of system differences, education indicators should support decisions by describing outcomes and variation with definitions and limitations that permit responsible interpretation.

The objective for science performance data and the responsible interpretation of system differences should be explicit, the evidence proportionate and learner impact visible. Assurance should be withheld for the affected scope until the limitation is resolved.