Evidence concerning science learning outcomes is reviewed for source definitions, coverage, comparability, uncertainty and limits on inference.
For the measure, the principal analytical task is to separate an observed difference from a conclusion about its cause.
For science learning outcomes, the PISA 2006 cycle gives particular emphasis to science while also assessing reading and mathematics among 15-year-old students. The results provide a comparative account of performance and its distribution across participating systems. Interpretation should take account of the sampled population, uncertainty and contextual information; a system-level association does not establish the cause of an individual learner’s result or the effectiveness of a particular provider.
A proper review of the comparison should establish the intended outcome before selecting controls or indicators. In the context of science learning outcomes, where an indicator is used as a proxy, the relationship between the proxy and the underlying educational outcome should be stated and tested.
Application to science learning outcomes
In reviewing science learning outcomes, interpretation should preserve the unit and population represented in the data collection. A national or international pattern may justify closer review of administrative data to examine science learning outcomes, but provider-level action requires evidence relating to the affected provision. Variation in population coverage, reference period or classification should accompany the reported comparison.
In the context of science learning outcomes, education indicators should support decisions by describing outcomes and variation with definitions and limitations that permit responsible interpretation.
- Analyse missing information.
- Disaggregate material results.
- Test comparability.
- Define the decision the indicator will inform.
- Document numerator and denominator.
Controls for science learning outcomes
A proxy is useful only where its relationship with the intended outcome is sufficiently understood. Participation, activity and expenditure may support learning, but none constitutes direct evidence of learning without an explicit and tested connection. For decisions concerning science learning outcomes, any condition preventing complete assurance should appear with the evidence on which the judgement relies.
A narrow control over the analysis may create false assurance. In the present context, data revisions not carried through to published conclusions, changes in definition presented as changes in performance and proxy measures treated as direct outcomes may produce acceptable aggregate reporting while individual learners remain exposed to material disadvantage.
Relevant evidence for the issue will normally include triangulation with administrative and qualitative evidence, revision and comparability records, population and sampling information, uncertainty estimates where relevant, and coverage and missingness analysis. For science learning outcomes, contradictory evidence should be investigated and resolved, not omitted from the record.
- For which groups may it fail?
- Is direct evidence available?
- Would the decision change if the proxy were inaccurate?
- What outcome does the proxy represent?
- What evidence supports the relationship?
Review of science learning outcomes
The review method for administrative data to examine science learning outcomes should be reproducible. For comparative analysis, the reviewer should state the construct to be measured, explain why the proxy is expected to represent it, test that relationship against direct evidence and identify circumstances in which the proxy may fail. Do not allow convenience to determine the measure.
Interpretation of the analysis should avoid two errors: treating a formal commitment as proof of effect, and treating one adverse case as proof that every part of the system has failed. For decisions concerning science learning outcomes, measurement can reveal where outcomes differ; it does not by itself establish why they differ or which intervention will work. For the measure, association should not be presented as causation, and statistical significance should not be treated as evidence of educational importance without further analysis.
For science learning outcomes, records relating to the issue should preserve both the conclusion and its limits.
No individual measure is sufficient to establish effective operation of the analysis across the affected scope. In the context of science learning outcomes, a reasoned conclusion should reconcile the governing requirement, evidence of operation, learner outcomes and residual risk, and remain open to better evidence.