Evidence gaps affecting science learning outcomes

The PISA 2006 results released in December 2007 provides the immediate context for science learning outcomes. For comparative analysis, the principal analytical task is to separate an observed difference from a conclusion about its cause. The appropriate administrative form will depend on the jurisdiction and the allocation of lawful responsibility.

Evidence base for evidence gaps affecting science learning outcomes

The reference basis—the PISA 2006 results released in December 2007—is evidential rather than self-executing. For decisions concerning science learning outcomes, the source can frame further questions without supplying either a binding direction or proof of causation. In applying it to science learning outcomes, users should review the source definitions, population coverage, reference period and stated limitations before transferring a system-level finding to an individual provider or learner group.

In the context of science learning outcomes, the PISA 2006 cycle gives particular emphasis to science while also assessing reading and mathematics among 15-year-old students. The results provide a comparative account of performance and its distribution across participating systems. Interpretation should take account of the sampled population, uncertainty and contextual information; a system-level association does not establish the cause of an individual learner’s result or the effectiveness of a particular provider.

When examining science learning outcomes, a proper review of the comparison should establish the intended outcome before selecting controls or indicators. Trend claims require comparable observations over time and a documented account of revisions, breaks in series and changes in coverage. A chosen approach should be justified against its context, with departures and review points under documented control.

Failure in relation to the comparison may arise even where the stated policy is reasonable. Material concerns include incomplete coverage, data revisions not carried through to published conclusions, averages concealing distribution, and changes in definition presented as changes in performance. In work concerning science learning outcomes, an exception should be assessed by effect, duration, recurrence and reach, including possible exposure beyond the initial sample.

The analysis proceeds on the basis that the subject should be examined as a connected system of policy, people, resources, decisions and evidence. For science learning outcomes, gaps may emerge when authority, records or action pass between responsible bodies. A formally complete record is not reliable if its scope or measure does not correspond to the decision being made.

Assurance of the comparison should draw on more than one form of evidence. Useful records include uncertainty estimates where relevant, revision and comparability records, coverage and missingness analysis, indicator definitions and metadata, and disaggregated results. As regards science learning outcomes, documents should be reconciled with observed practice and, where relevant, the experience of affected learners. Within the scope under review, system-wide assurance cannot be inferred from a favourable case chosen after the event.

Coverage and comparability

The analysis of science learning outcomes should remain within the limits of the evidence. A single indicator rarely provides an adequate account of quality. Quantitative evidence should be considered with implementation records and the experience of affected learners. Measurement can reveal where outcomes differ; it does not by itself establish why they differ or which intervention will work. Decision-makers should not extend assurance beyond the point supported by the available evidence.

The assurance record for science learning outcomes should retain the date of the evidence, the source responsible for it, the scope examined and the version of any instrument or definition applied. This enables later review to separate substantive change from correction, reclassification or expanded coverage. The evidential history should preserve conclusions that were operative when a material decision was made.

• Document numerator and denominator before it informs a consequential decision.
• Avoid causal claims unsupported by the design before it is relied on for a decision with material effect.
• Test comparability before using it to determine a learner or provider outcome.
• Define the decision the indicator will inform.
• Report uncertainty and revisions before it is relied on for a decision with material effect.

Responsible interpretation

Authorities and providers reviewing science learning outcomes should proceed in a defined sequence. The method for the comparison is to map the complete process, identify the intended result and responsible authority at each stage, and test normal cases together with exceptions. Findings should state the affected scope and required action; an observation should not be represented as evidence of conformity or effectiveness.

In work concerning science learning outcomes, decision-makers using evidence on the measure should be told what the data cannot establish as clearly as what it can. Reporting should state whether a result describes, compares or evaluates, together with the level at which it is valid. The basis for applying the result elsewhere should be established rather than assumed.

• What action is required by the finding?
• What outcome is intended?
• Who controls each stage?
• Which evidence establishes operation?
• Where do exceptions occur?

Limitations and reporting

In reviewing science learning outcomes, where responsibilities for delivery are shared with partners, suppliers or several public bodies, responsibility should be mapped across the complete service. Agreements governing science learning outcomes should allocate information exchange, incident escalation, learner communication, record custody and corrective authority. Within the scope under review, protection should operate across the complete service, irrespective of how delivery is divided.

The system and institutional dimensions of the comparison should be considered together. In work concerning science learning outcomes, education indicators should support decisions by describing outcomes and variation with definitions and limitations that permit responsible interpretation. The allocation of responsibility should prevent gaps between system oversight and institutional operation.

Governance of science learning outcomes requires a clear allocation of authority, information and follow-through. Material matters should be referred to the body authorised to act or accept residual risk. Public confidence cannot be separated from an institution's ability to identify responsibility and substantiate its conclusions.