{"schema_version":"ICEQC_NEWS_ARTICLE_V1","id":"iceqc-news-4ddae312e6dd297b","slug":"2010-06-29-learning-outcome-measurement-an-evidence-led-improvement-method","language":"en","publication_status":"READY_FOR_IMPORT","publication_date":"2010-06-29","last_modified_date":"2010-06-29","title":"Learning-outcome measurement: an evidence-led improvement method","summary":"Sets out a controlled approach to learning-outcome measurement: an evidence-led improvement method, linking diagnosis, accountable action, outcome measures and verification.","category":{"code":"QUALITY_IMPROVEMENT_METHODS","label":"Quality Improvement Methods"},"article_type":"Quality improvement method","publisher":"International Council for Education Quality Certification (ICEQC)","jurisdictional_scope":"International","historical_reference_basis":"International assessment and indicator development","reference_authority":"Relevant public authorities and official international sources","sections":[{"heading":null,"paragraphs":["The international assessment and indicator development provides the immediate reference point for consideration of learning-outcome measurement in 2010. A decision concerning the improvement objective should recognise that the purpose of an improvement method is not to produce an action plan; it is to change a material condition and verify that the change is sustained. The unit of review should correspond to the full reach of the decision, including significant differences in provision and population.","The historical reference basis is the international assessment and indicator development. Its relevance to the issue under review should be assessed against the affected jurisdiction, learner population and form of provision."]},{"heading":"Scope of the improvement","paragraphs":["For learning-outcome measurement, the public interest is not confined to institutional compliance. For the issue under review, education indicators should support decisions by describing outcomes and variation with definitions and limitations that permit responsible interpretation.","The technical issue within the intervention concerns the basis on which a conclusion is reached. For the intervention, effectiveness is the demonstrated change in the condition the action was intended to address. In relation to learning-outcome measurement, completion of training, publication of guidance or installation of a system is an output and should not be reported as an outcome without further evidence. The decision record should distinguish the scope supported by evidence from any scope that remains unresolved.","The principal risks in relation to the corrective programme are incomplete coverage, small differences overstated, proxy measures treated as direct outcomes, and data revisions not carried through to published conclusions. A weakness in one part of the control environment may obscure a related failure elsewhere. In relation to learning-outcome measurement, review should follow the sequence of decisions and records rather than assess documents in isolation."]},{"heading":"Implementation responsibilities","paragraphs":["The evidential record should be limited to material that can answer the question under review. For learning-outcome measurement, the most relevant material is likely to include indicator definitions and metadata, coverage and missingness analysis, disaggregated results, and revision and comparability records. Confidence is strengthened by corroboration, not by the volume of records drawn from the same underlying source.","Records relating to the practice should preserve both the conclusion and its limits. A changed evidential position should be applied to the affected scope, including prior decisions that may no longer be reliable. Where reliance has occurred, correction may require review of affected decisions as well as amendment of published information."],"bullets":["Define the decision the indicator will inform before it is relied on for a decision with material effect.","Document numerator and denominator.","Avoid causal claims unsupported by the design before it is relied on for a decision with material effect.","Analyse missing information before it informs a consequential decision.","Test comparability."]},{"heading":"Testing effectiveness","paragraphs":["Responsibility for learning-outcome measurement should be visible at the point where consequential decisions are made. Oversight of learning-outcome measurement should reflect the principle that effectiveness should be judged against an agreed outcome and reference period, not against completion of activities alone. Escalation should follow whenever the available record cannot support a safe conclusion for the affected learners.","Implementation of the issue under review can be tested without imposing unnecessary reporting. For the improvement objective, the reviewer should set a baseline and success measure before intervention, define the review period, compare the result with the intended outcome and examine adverse or unequal effects. Continue monitoring long enough to determine whether the improvement is sustained. Existing records may be used if reliable and relevant, but data collected for another purpose may not answer the assurance question.","A decision to close improvement work on the corrective programme should be made by a person with authority and sufficient independence from implementation.","Accountability for the issue under review should follow decision-making authority. Evidence of material risk should be placed before the body with authority to act, together with a traceable decision. Delegation of delivery does not remove the need for a named authority to oversee material learner impact."]},{"heading":"Maintaining the result","paragraphs":["The analysis of learning-outcome measurement should remain within the limits of the evidence. A decision concerning the practice should recognise that improvement data should not be selected only because it is readily available. Oversight of the improvement objective should reflect the principle that measurement can reveal where outcomes differ; it does not by itself establish why they differ or which intervention will work. Decision-makers should not extend assurance beyond the point supported by the available evidence.","The measure of progress on the intervention is not the amount of policy or documentation produced."]}],"word_count":738,"content_hash":"sha256-c0c3582e8cf515d4bf7e2294070ec198592e080700430dd460c7993786f6a194","seo_keywords":["learning-outcome measurement","education quality improvement","education quality","ICEQC"],"schema_type":"TechArticle"}