{"schema_version":"ICEQC_NEWS_ARTICLE_V1","id":"iceqc-news-8e41e0b88eb15213","slug":"2006-08-21-learning-outcome-reporting-evidence-for-policy-decisions","language":"en","publication_status":"READY_FOR_IMPORT","publication_date":"2006-08-21","last_modified_date":"2006-08-21","title":"Learning outcome reporting: evidence for policy decisions","summary":"Assesses evidence on learning outcome reporting: evidence for policy decisions, with particular attention to definitions, population coverage, comparability and uncertainty.","category":{"code":"DATA_RESEARCH","label":"Data Research"},"article_type":"Data and research analysis","publisher":"International Council for Education Quality Certification (ICEQC)","jurisdictional_scope":"International","historical_reference_basis":"Expansion of comparable education indicators","reference_authority":"Relevant public authorities and official international sources","sections":[{"heading":null,"paragraphs":["In 2006, consideration of learning outcome reporting must take account of the expansion of comparable education indicators and the responsibilities it places before education systems. the analysis proceeds on the basis that comparable indicators can support public decision-making, but they do not remove the need to examine variation within systems and institutions. Proportionality should be assessed against effects on access, learning, fair treatment and the accuracy of learner information.","The reference basis—the expansion of comparable education indicators—is evidential rather than self-executing. Patterns in the material may justify enquiry, although they do not by themselves determine legal position or cause. In applying it to the analysis, users should review the source definitions, population coverage, reference period and stated limitations before transferring a system-level finding to an individual provider or learner group.","In relation to learning outcome reporting, the intended substantive result should remain the starting point for review. A decision concerning the comparison should recognise that education indicators should support decisions by describing outcomes and variation with definitions and limitations that permit responsible interpretation. The existence of an approved measure or completed activity is not evidence of educational effect. Assurance should address actual effect and provide a means of correcting disadvantage that the arrangement did not intend."]},{"heading":"Evidence and method","paragraphs":["In practical terms, learning outcome reporting should be reviewed against a stated method rather than general assurance. For the available evidence, the subject should be examined as a connected system of policy, people, resources, decisions and evidence. A review should test the interfaces between functions, not assume that sound component controls ensure a sound end-to-end process.","The governing expectation for the measure should be capable of consistent application. For the analysis, where an indicator is used as a proxy, the relationship between the proxy and the underlying educational outcome should be stated and tested. Criteria affecting learners should not permit materially different interpretation without an evidenced reason."]},{"heading":"Patterns requiring examination","paragraphs":["Risk assessment of learning outcome reporting should give particular attention to proxy measures treated as direct outcomes, changes in definition presented as changes in performance, and incomplete coverage. A provider should also consider small differences overstated and averages concealing distribution."],"bullets":["Avoid causal claims unsupported by the design, identifying the accountable function and affected scope.","Document numerator and denominator.","Report uncertainty and revisions before it is relied on for a decision with material effect.","Analyse missing information.","Test comparability."]},{"heading":"Implications for decision-makers","paragraphs":["Relevant evidence for learning outcome reporting will normally include disaggregated results, indicator definitions and metadata, coverage and missingness analysis, triangulation with administrative and qualitative evidence, and population and sampling information. In relation to learning outcome reporting, the record should retain disagreement between sources until its cause and effect are understood.","For operational review of the available evidence, authorities and providers should proceed in a defined sequence. In reviewing the analysis, responsible bodies should map the complete process, identify the intended result and responsible authority at each stage, and test normal cases together with exceptions.","Decision-makers using evidence on the issue should be told what the data cannot establish as clearly as what it can. Users should be able to distinguish descriptive, comparative and evaluative findings and understand their proper level of application. Application in another setting depends on a separate examination of context and comparability."]},{"heading":"Limits of inference","paragraphs":["Interpretation of learning outcome reporting should avoid two errors: treating a formal commitment as proof of effect, and treating one adverse case as proof that every part of the system has failed. A decision concerning the issue should recognise that measurement can reveal where outcomes differ; it does not by itself establish why they differ or which intervention will work. The analysis of the measure proceeds on the basis that a single indicator rarely provides an adequate account of quality. Quantitative evidence should be considered with implementation records and the experience of affected learners.","Decisions concerning the measure should remain traceable to the information available for the stated reference period. The reason for revision should be explicit, including whether it arises from new evidence, a methodological change or a different interpretation.","Where the available evidence involves partners, suppliers or several public bodies, responsibility should be mapped across the complete service. Governance between participating bodies should make information duties and corrective authority explicit. Protection should operate across the complete service, irrespective of how delivery is divided.","Complete assurance concerning the available evidence cannot rest on a single indicator or isolated control."]}],"word_count":728,"content_hash":"sha256-ae2f40f2fd2d6e4d2efcd031b4020fabc9eba0371f1af2490c96583c115f81d4","seo_keywords":["learning outcome reporting","international education data research","education quality","ICEQC"],"schema_type":"AnalysisNewsArticle"}