{"schema_version":"ICEQC_NEWS_ARTICLE_V1","id":"iceqc-news-2eb04261a3c46a3b","slug":"2007-12-27-using-administrative-data-to-examine-science-learning-outcomes","language":"en","publication_status":"READY_FOR_IMPORT","publication_date":"2007-12-27","last_modified_date":"2007-12-27","title":"Using administrative data to examine science learning outcomes","summary":"Reviews using administrative data to examine science learning outcomes and identifies the conditions required for sound interpretation, responsible comparison and defensible.","category":{"code":"DATA_RESEARCH","label":"Data Research"},"article_type":"Data and research analysis","publisher":"International Council for Education Quality Certification (ICEQC)","jurisdictional_scope":"International","historical_reference_basis":"PISA 2006 results released in December 2007","reference_authority":"Organisation for Economic Co-operation and Development","sections":[{"heading":null,"paragraphs":["The present attention to administrative data to examine science learning outcomes follows the PISA 2006 results released in December 2007 and requires a careful distinction between public commitment, institutional practice and demonstrated result. For the measure, the principal analytical task is to separate an observed difference from a conclusion about its cause.","The PISA 2006 cycle gives particular emphasis to science while also assessing reading and mathematics among 15-year-old students. The results provide a comparative account of performance and its distribution across participating systems. Interpretation should take account of the sampled population, uncertainty and contextual information; a system-level association does not establish the cause of an individual learner’s result or the effectiveness of a particular provider.","A proper review of the comparison should establish the intended outcome before selecting controls or indicators. The analysis of the available evidence proceeds on the basis that where an indicator is used as a proxy, the relationship between the proxy and the underlying educational outcome should be stated and tested. In relation to using administrative data to examine science learning outcomes, the record should explain why the approach suits the affected context, how material departures are authorised and when review will occur."]},{"heading":"Analytical scope","paragraphs":["The stated reference is PISA 2006 results released in December 2007. Interpretation should preserve the unit and population represented in the data collection. A national or international pattern may justify closer review of administrative data to examine science learning outcomes, but provider-level action requires evidence relating to the affected provision. Variation in population coverage, reference period or classification should accompany the reported comparison.","The quality significance of the available evidence follows from a basic distinction between availability and effective provision. The analysis of the available evidence proceeds on the basis that education indicators should support decisions by describing outcomes and variation with definitions and limitations that permit responsible interpretation. Review should cover the stages at which learners receive information, provision, assessment, support and remedy."],"bullets":["Analyse missing information.","Disaggregate material results.","Test comparability.","Define the decision the indicator will inform.","Document numerator and denominator."]},{"heading":"Definitions and data coverage","paragraphs":["The technical issue within administrative data to examine science learning outcomes concerns the basis on which a conclusion is reached. Oversight of the analysis should reflect the principle that a proxy is useful only where its relationship with the intended outcome is sufficiently understood. Participation, activity and expenditure may support learning, but none constitutes direct evidence of learning without an explicit and tested connection. Any condition preventing complete assurance should appear with the evidence on which the judgement relies.","A narrow control over the analysis may create false assurance. In the present context, data revisions not carried through to published conclusions, changes in definition presented as changes in performance and proxy measures treated as direct outcomes may produce acceptable aggregate reporting while individual learners remain exposed to material disadvantage.","Relevant evidence for the issue will normally include triangulation with administrative and qualitative evidence, revision and comparability records, population and sampling information, uncertainty estimates where relevant, and coverage and missingness analysis. Contradictory evidence should be investigated and resolved, not omitted from the record."],"bullets":["For which groups may it fail?","Is direct evidence available?","Would the decision change if the proxy were inaccurate?","What outcome does the proxy represent?","What evidence supports the relationship?"]},{"heading":"Use of the findings","paragraphs":["The review method for administrative data to examine science learning outcomes should be reproducible. For the comparison, the reviewer should state the construct to be measured, explain why the proxy is expected to represent it, test that relationship against direct evidence and identify circumstances in which the proxy may fail. Do not allow convenience to determine the measure. In relation to using administrative data to examine science learning outcomes, a competent reviewer should be able to follow the record from source selection to conclusion and exception handling.","The analytical record for the measure should state the research question, data source, unit of analysis, reference period, coverage, exclusions, treatment of missing values and principal limitations.","Interpretation of the analysis should avoid two errors: treating a formal commitment as proof of effect, and treating one adverse case as proof that every part of the system has failed. In reviewing the issue, measurement can reveal where outcomes differ; it does not by itself establish why they differ or which intervention will work. For the measure, association should not be presented as causation, and statistical significance should not be treated as evidence of educational importance without further analysis.","Records relating to the issue should preserve both the conclusion and its limits. New evidence should trigger a traceable correction and review of decisions materially affected by the earlier conclusion. The correction process should identify prior users and decisions where published information has had material effect.","Accountability for the analysis should follow decision-making authority. Evidence of material risk should be placed before the body with authority to act, together with a traceable decision. Operational tasks may be delegated, but accountability for material effects on learners must remain identifiable.","No individual measure is sufficient to establish effective operation of the analysis across the affected scope. A reasoned conclusion should reconcile the governing requirement, evidence of operation, learner outcomes and residual risk, and remain open to better evidence."]}],"word_count":863,"content_hash":"sha256-ee9e19716560db2f9078213af136ea05895980d38cfbfe235fe356e8cfd36bba","seo_keywords":["using administrative data to examine science learning outcomes","international education data research","education quality","ICEQC"],"schema_type":"AnalysisNewsArticle"}