Evidence gaps affecting science learning outcomes

For comparative analysis, the principal analytical task is to separate an observed difference from a conclusion about its cause.

Evidence base for evidence gaps affecting science learning outcomes

For decisions concerning science learning outcomes, the source can frame further questions without supplying either a binding direction or proof of causation. In applying it to science learning outcomes, users should review the source definitions, population coverage, reference period and stated limitations before transferring a system-level finding to an individual provider or learner group.

In the context of science learning outcomes, the PISA 2006 cycle gives particular emphasis to science while also assessing reading and mathematics among 15-year-old students. The results provide a comparative account of performance and its distribution across participating systems. Interpretation should take account of the sampled population, uncertainty and contextual information; a system-level association does not establish the cause of an individual learner’s result or the effectiveness of a particular provider.

When examining science learning outcomes, a proper review of the comparison should establish the intended outcome before selecting controls or indicators.

Failure in relation to the comparison may arise even where the stated policy is reasonable. Material concerns include incomplete coverage, data revisions not carried through to published conclusions, averages concealing distribution, and changes in definition presented as changes in performance. For science learning outcomes, an exception should be assessed by effect, duration, recurrence and reach, including possible exposure beyond the initial sample.

The analysis proceeds on the basis that the subject should be examined as a connected system of policy, people, resources, decisions and evidence. For science learning outcomes, gaps may emerge when authority, records or action pass between responsible bodies.

Assurance of the comparison should draw on more than one form of evidence. Useful records include uncertainty estimates where relevant, revision and comparability records, coverage and missingness analysis, indicator definitions and metadata, and disaggregated results. For science learning outcomes, documents should be reconciled with observed practice and, where relevant, the experience of affected learners. Across the defined scope, system-wide assurance cannot be inferred from a favourable case chosen after the event.

Controls for evidence gaps affecting science learning outcomes

The analysis of science learning outcomes should remain within the limits of the evidence.

The assurance record for science learning outcomes should retain the date of the evidence, the source responsible for it, the scope examined and the version of any instrument or definition applied.

• Document numerator and denominator before it informs a consequential decision.
• Avoid causal claims unsupported by the design before it is relied on for a decision with material effect.
• Test comparability before using it to determine a learner or provider outcome.
• Define the decision the indicator will inform.
• Report uncertainty and revisions before it is relied on for a decision with material effect.

Review of evidence gaps affecting science learning outcomes

Authorities and providers reviewing science learning outcomes should proceed in a defined sequence. The method for the comparison is to map the complete process, identify the intended result and responsible authority at each stage, and test normal cases together with exceptions.

For science learning outcomes, decision-makers using evidence on the measure should be told what the data cannot establish as clearly as what it can. Reporting should state whether a result describes, compares or evaluates, together with the level at which it is valid. The basis for applying the result elsewhere should be established rather than assumed.

• What action is required by the finding?
• What outcome is intended?
• Who controls each stage?
• Which evidence establishes operation?
• Where do exceptions occur?

Implications for evidence gaps affecting science learning outcomes

In reviewing science learning outcomes, where responsibilities for delivery are shared with partners, suppliers or several public bodies, responsibility should be mapped across the complete service. Agreements governing science learning outcomes should allocate information exchange, incident escalation, learner communication, record custody and corrective authority. Across the defined scope, protection should operate across the complete service, irrespective of how delivery is divided.

Review of the comparison should address both system-level conditions and institutional practice. For science learning outcomes, education indicators should support decisions by describing outcomes and variation with definitions and limitations that permit responsible interpretation.

Governance of science learning outcomes requires a clear allocation of authority, information and follow-through. Material matters should be referred to the body authorised to act or accept residual risk. Public confidence cannot be separated from an institution's ability to identify responsibility and substantiate its conclusions.