{"schema_version":"ICEQC_NEWS_ARTICLE_V1","id":"iceqc-news-688e4e7f261cb713","slug":"2007-12-12-evidence-gaps-affecting-science-learning-outcomes","language":"en","publication_status":"READY_FOR_IMPORT","publication_date":"2007-12-12","last_modified_date":"2007-12-12","title":"Evidence gaps affecting science learning outcomes","summary":"Assesses evidence on evidence gaps affecting science learning outcomes, with particular attention to definitions, population coverage, comparability and uncertainty.","category":{"code":"DATA_RESEARCH","label":"Data Research"},"article_type":"Data and research analysis","publisher":"International Council for Education Quality Certification (ICEQC)","jurisdictional_scope":"International","historical_reference_basis":"PISA 2006 results released in December 2007","reference_authority":"Organisation for Economic Co-operation and Development","sections":[{"heading":null,"paragraphs":["The PISA 2006 results released in December 2007 provides the immediate context for science learning outcomes. For the comparison, the principal analytical task is to separate an observed difference from a conclusion about its cause. In relation to evidence gaps affecting science learning outcomes, the appropriate administrative form will depend on the jurisdiction and the allocation of lawful responsibility."]},{"heading":"Evidence base for evidence gaps affecting science learning outcomes","paragraphs":["The reference basis—the PISA 2006 results released in December 2007—is evidential rather than self-executing. The source can frame further questions without supplying either a binding direction or proof of causation. In applying it to science learning outcomes, users should review the source definitions, population coverage, reference period and stated limitations before transferring a system-level finding to an individual provider or learner group.","The PISA 2006 cycle gives particular emphasis to science while also assessing reading and mathematics among 15-year-old students. The results provide a comparative account of performance and its distribution across participating systems. Interpretation should take account of the sampled population, uncertainty and contextual information; a system-level association does not establish the cause of an individual learner’s result or the effectiveness of a particular provider.","A proper review of the comparison should establish the intended outcome before selecting controls or indicators. In reviewing the issue, trend claims require comparable observations over time and a documented account of revisions, breaks in series and changes in coverage. A chosen approach should be justified against its context, with departures and review points under documented control.","Failure in relation to the comparison may arise even where the stated policy is reasonable. Material concerns include incomplete coverage, data revisions not carried through to published conclusions, averages concealing distribution, and changes in definition presented as changes in performance. An exception should be assessed by effect, duration, recurrence and reach, including possible exposure beyond the initial sample.","A focused examination of the analysis requires a clear analytical discipline. the analysis proceeds on the basis that the subject should be examined as a connected system of policy, people, resources, decisions and evidence. Gaps may emerge when authority, records or action pass between responsible bodies. A formally complete record is not reliable if its scope or measure does not correspond to the decision being made.","Assurance of the comparison should draw on more than one form of evidence. Useful records include uncertainty estimates where relevant, revision and comparability records, coverage and missingness analysis, indicator definitions and metadata, and disaggregated results. Documents should be reconciled with observed practice and, where relevant, the experience of affected learners. System-wide assurance cannot be inferred from a favourable case chosen after the event."]},{"heading":"Coverage and comparability","paragraphs":["The analysis of science learning outcomes should remain within the limits of the evidence. In reviewing the analysis, a single indicator rarely provides an adequate account of quality. Quantitative evidence should be considered with implementation records and the experience of affected learners. In reviewing the available evidence, measurement can reveal where outcomes differ; it does not by itself establish why they differ or which intervention will work. In relation to evidence gaps affecting science learning outcomes, decision-makers should not extend assurance beyond the point supported by the available evidence.","The assurance record for the measure should retain the date of the evidence, the source responsible for it, the scope examined and the version of any instrument or definition applied. This enables later review to separate substantive change from correction, reclassification or expanded coverage. The evidential history should preserve conclusions that were operative when a material decision was made."],"bullets":["Document numerator and denominator before it informs a consequential decision.","Avoid causal claims unsupported by the design before it is relied on for a decision with material effect.","Test comparability before using it to determine a learner or provider outcome.","Define the decision the indicator will inform.","Report uncertainty and revisions before it is relied on for a decision with material effect."]},{"heading":"Responsible interpretation","paragraphs":["For operational review of science learning outcomes, authorities and providers should proceed in a defined sequence. The method for the comparison is to map the complete process, identify the intended result and responsible authority at each stage, and test normal cases together with exceptions. Findings should state the affected scope and required action; an observation should not be represented as evidence of conformity or effectiveness.","Decision-makers using evidence on the measure should be told what the data cannot establish as clearly as what it can. Reporting should state whether a result describes, compares or evaluates, together with the level at which it is valid. The basis for applying the result elsewhere should be established rather than assumed."],"bullets":["What action is required by the finding?","What outcome is intended?","Who controls each stage?","Which evidence establishes operation?","Where do exceptions occur?"]},{"heading":"Limitations and reporting","paragraphs":["Where science learning outcomes involves partners, suppliers or several public bodies, responsibility should be mapped across the complete service. Agreements should allocate information exchange, incident escalation, learner communication, record custody and corrective authority. Protection should operate across the complete service, irrespective of how delivery is divided.","The system and institutional dimensions of the comparison should be considered together. Oversight of the available evidence should reflect the principle that education indicators should support decisions by describing outcomes and variation with definitions and limitations that permit responsible interpretation. The allocation of responsibility should prevent gaps between system oversight and institutional operation.","Governance of science learning outcomes requires a clear allocation of authority, information and follow-through. Material matters should be referred to the body authorised to act or accept residual risk. Public confidence cannot be separated from an institution's ability to identify responsibility and substantiate its conclusions."]}],"word_count":925,"content_hash":"sha256-be8e75c666825f24b398e5fee7ee86712e22dd729b8205fe756bc4ff8c0ca4d0","seo_keywords":["evidence gaps affecting science learning outcomes","international education data research","education quality","ICEQC"],"schema_type":"AnalysisNewsArticle"}