Sets out a controlled approach to improving science learning outcomes: a structured review method, covering diagnosis, responsible action, outcome evidence and sustained effect.
Against the background of the PISA 2006 results released in December 2007, education authorities and providers should review how science learning outcomes is defined, implemented and evidenced. Effective improvement requires ownership, a time-bound intervention and independent confirmation that the intended result has been achieved. The assessment addresses decisions capable of affecting learners, institutions or the proper use of entrusted educational resources.
The PISA 2006 cycle gives particular emphasis to science while also assessing reading and mathematics among 15-year-old students. Analysis of improving the intended improvement should state the unit of analysis, reference period, coverage, exclusions and treatment of missing information. For science learning outcomes, a comparison is reliable only if material differences remain visible. Data used for the intended improvement should be interpreted against stable definitions and an identifiable population. A revision or break in series should not be reported as a change in performance.
Scope of the improvement
The reference basis—the PISA 2006 results released in December 2007—is evidential rather than self-executing. For decisions concerning science learning outcomes, the material may reveal patterns or evidential gaps, but it neither directs a legal outcome nor establishes causation. Analysis of science learning outcomes should state the unit of analysis, reference period, coverage, exclusions and treatment of missing information. Comparative findings should not conceal differences capable of changing their meaning.
For corrective action, the public interest is not confined to institutional compliance. Corrective action concerning the matter should address the identified cause, assign responsibility and set a review period. For science learning outcomes, residual risk should remain open until sustained improvement is demonstrated.
Corrective action concerning corrective action should address the identified cause, assign responsibility and set a review period. Data used for improving corrective action should be interpreted against stable definitions and an identifiable population. For science learning outcomes, broad intentions should be converted into decisions capable of review. The decision record for science learning outcomes should distinguish the scope supported by evidence from any scope that remains unresolved.
When examining science learning outcomes, the applicable expectation should be capable of consistent application. Effectiveness should be judged against an agreed outcome and reference period, not against completion of activities alone. Public information on the corrective action should state the applicable scope and limitations in terms that affected users can understand, including the basis for any later correction.
Failure in relation to the corrective action may arise even where the stated policy is reasonable. Material concerns include tasks that do not assess the stated outcome, results used beyond the evidence they support, inconsistent judgement between markers or locations, and reasonable adjustment altering the assessed outcome. For decisions concerning science learning outcomes, review should consider whether an exception is prolonged, recurring or capable of affecting learners outside the cases examined.
Improvement work on the corrective action should begin with a verified problem, defined baseline and measurable outcome. In the context of science learning outcomes, completion should depend on evidence of effect rather than completion of planned activity. For the matter, the most relevant material is likely to include approval and change-control records, authorship and identity controls proportionate to risk, appeal and correction records, and moderation and exception records. Each source has limitations; confidence depends on corroboration between independent records and transparent treatment of uncertainty.
Implementation responsibilities
Authorities and providers reviewing science learning outcomes should proceed in a defined sequence. Responsible bodies should prioritise actions by learner impact and control weakness, establish dependencies, test implementation at suitable intervals and retain unresolved items until effectiveness is verified. Amend the plan where evidence does not support the original causal assumption. Improvement work on improving the relevant practice should begin with a verified problem, defined baseline and measurable outcome.
Improvement of the corrective action should proceed through controlled tests where risk permits. Corrective action concerning the relevant practice should address the identified cause, assign responsibility and set a review period.
As regards science learning outcomes, analysis should remain within the limits of the evidence. Measures should remain in place long enough to detect recurrence and unintended effects. Reliability without validity produces consistent but potentially irrelevant results. Within the scope under review, validity without adequate consistency may expose learners to unequal judgement. Material uncertainty should result in further enquiry or an expressly limited finding.
The assurance record for the matter should retain the date of the evidence, the source responsible for it, the scope examined and the version of any instrument or definition applied. In work concerning science learning outcomes, corrective action should be proportionate to the identified condition and tested where risk permits. Wider implementation should follow evidence of benefit and acceptable unintended effects. Records concerning the relevant practice should remain traceable from source evidence to decision and follow-up. Superseded conclusions should be retained where they informed a material outcome.
Review of the corrective action should give particular attention to adverse cases, unequal effects and errors that learners may be unable to identify or remedy after the event. For science learning outcomes, the action record should identify who is responsible and when implementation is due. Closure requires evidence that the condition has changed; completion of planned activity is not sufficient.
No individual measure is sufficient to establish effective operation of the relevant practice across the affected scope. Examination of the matter should give particular attention to adverse cases, unequal effects and errors that learners may be unable to identify or remedy after the event.