{"schema_version":"ICEQC_NEWS_ARTICLE_V1","id":"iceqc-news-4090195c7f25e9e4","slug":"2010-07-28-evidence-boundaries-in-the-evaluation-of-learning-outcome-measurement","language":"en","publication_status":"PUBLISHED","publication_date":"2010-07-28","last_modified_date":"2010-07-28","title":"Evidence boundaries in the evaluation of learning-outcome measurement","summary":"Explains the application of learning-outcome measurement, including scope, evidential sufficiency, accountable decisions and justified exceptions.","category":{"code":"STANDARDS_INTERPRETATION","label":"Standards Interpretation"},"article_type":"Standards interpretation","publisher":"International Council for Education Quality Certification (ICEQC)","jurisdictional_scope":"International","historical_reference_basis":"International assessment and indicator development","reference_authority":"Relevant public authorities and official international sources","sections":[{"heading":null,"paragraphs":["The international assessment and indicator development provides the immediate reference point for consideration of learning-outcome measurement in 2010. Interpretation should begin with the intended outcome, then identify the controls and evidence needed to show that the outcome is achieved across the declared scope.","For the relevant requirement, the public interest is not confined to institutional compliance. For the assurance question, education indicators should support decisions by describing outcomes and variation with definitions and limitations that permit responsible interpretation. For learning-outcome measurement, learner protection requires intelligible information and a timely means of reviewing consequential mistakes or unfair decisions."]},{"heading":"Scope and application","paragraphs":["The evidential record for learning-outcome measurement should permit a reviewer to trace the matter from decision to outcome. This may require disaggregated results, uncertainty estimates where relevant, population and sampling information, and triangulation with administrative and qualitative evidence, supported by indicator definitions and metadata and revision and comparability records.","This analysis is informed by international assessment and indicator development. Its relevance to the assurance question should be assessed against the affected jurisdiction, learner population and form of provision. In the context of learning-outcome measurement, the international development warrants attention, but a consequential conclusion still requires current, attributable and representative evidence for the affected scope.","When examining learning-outcome measurement, analysis should make its decision rule explicit. For decisions concerning learning-outcome measurement, effectiveness is the demonstrated change in the condition the action was intended to address. In work concerning learning-outcome measurement, completion of training, publication of guidance or installation of a system is an output and should not be reported as an outcome without further evidence. As regards learning-outcome measurement, the method should prevent an unfavourable result from being dismissed through an unrecorded change in interpretation.","The principal risks in relation to the control are averages concealing distribution, changes in definition presented as changes in performance, incomplete coverage, and proxy measures treated as direct outcomes. Where learning-outcome measurement is under review, a weakness in one part of the control environment may obscure a related failure elsewhere. For the assessment of learning-outcome measurement, review should follow the sequence of decisions and records rather than assess documents in isolation."]},{"heading":"Evidence required","paragraphs":["For learning-outcome measurement, the applicable expectation should be capable of consistent application. In the context of learning-outcome measurement, a provider should be able to trace the expectation from approved policy through implementation, monitoring, identified exceptions and corrective action. When examining learning-outcome measurement, terms governing eligibility, support, assessment, reporting or review should prevent materially different treatment without recorded justification.","For the assurance question, governing bodies should receive a concise account of the intended result, affected scope, principal risks, evidence limitations and unresolved exceptions.","Records relating to the stated expectation should preserve both the conclusion and its limits. For decisions concerning learning-outcome measurement, new evidence should trigger a traceable correction and review of decisions materially affected by the earlier conclusion. In work concerning learning-outcome measurement, this is material where learners, authorities or institutions relied on information that cannot be corrected by replacing the current text alone."],"bullets":["Avoid causal claims unsupported by the design before it informs a consequential decision.","Define the decision the indicator will inform.","Document numerator and denominator, with responsibility, scope and timing recorded.","Analyse missing information.","Disaggregate material results."]},{"heading":"Decision criteria and exceptions","paragraphs":["The method for the relevant requirement is to set a baseline and success measure before intervention, define the review period, compare the result with the intended outcome and examine adverse or unequal effects. As regards learning-outcome measurement, continue monitoring long enough to determine whether the improvement is sustained. Where learning-outcome measurement is under review, adverse cases and unresolved contradictions should be retained because they may reveal limitations concealed by an average result.","The final record on the issue under review should identify the applicable expectation, the relevant scope, the evidence examined, the sampling basis, material exceptions and the reason for the conclusion. If an alternative method is accepted, the record should demonstrate that it achieves the same required outcome. Unresolved limitations should be stated with the conclusion and carried forward for action.","Particular care is required when interpreting evidence about the issue under review. Measurement can reveal where outcomes differ; it does not by itself establish why they differ or which intervention will work. For the issue under review, interpretive guidance should not create an obligation that is absent from the governing instrument or applicable law. For the assessment of learning-outcome measurement, material limitations should be stated with the finding presented to decision-makers and affected learners.","Neither one indicator nor one control can establish the complete position on the stated expectation. A reasoned conclusion should reconcile the governing requirement, evidence of operation, learner outcomes and residual risk, and remain open to better evidence."]}],"word_count":775,"content_hash":"sha256-a2b50e989921ac4a3dce90394aaaaf3661d0cb9254c772e1741062a518e45585","seo_keywords":["evidence boundaries in the evaluation of learning-outcome measurement","education quality standards","education quality","ICEQC"],"schema_type":"TechArticle"}