{"schema_version":"ICEQC_NEWS_ARTICLE_V1","id":"iceqc-news-d38a9473b35b6506","slug":"2009-02-13-assessment-moderation-an-international-evidence-note","language":"en","publication_status":"READY_FOR_IMPORT","publication_date":"2009-02-13","last_modified_date":"2009-02-13","title":"Assessment moderation: an international evidence note","summary":"Reviews assessment moderation: an international evidence note and identifies the conditions required for sound interpretation, responsible comparison and defensible conclusions.","category":{"code":"DATA_RESEARCH","label":"Data Research"},"article_type":"Data and research analysis","publisher":"International Council for Education Quality Certification (ICEQC)","jurisdictional_scope":"International","historical_reference_basis":"Cross-system comparability and academic standards","reference_authority":"Relevant public authorities and official international sources","sections":[{"heading":null,"paragraphs":["The cross-system comparability and academic standards provides the immediate context for assessment moderation. A decision concerning the issue should recognise that the principal analytical task is to separate an observed difference from a conclusion about its cause.","The principal risks in relation to the available evidence are inconsistent judgement between markers or locations, weak assurance of authorship or performance, reasonable adjustment altering the assessed outcome, and uncontrolled changes to assessment. The control environment should be assessed as a connected system rather than as unrelated individual risks."]},{"heading":"Evidence and method","paragraphs":["The governing expectation for assessment moderation should be capable of consistent application. Oversight of the issue should reflect the principle that a sound interpretation should identify the unit of analysis, reference period, denominator, exclusions, missing values and any change in definition or collection practice. Operational definitions should be precise enough to support consistent consequential decisions and explain justified variation.","The historical reference basis is the cross-system comparability and academic standards. Its relevance to the comparison should be assessed against the affected jurisdiction, learner population and form of provision.","A focused examination of the comparison requires a clear analytical discipline. In this case, comparison requires more than the use of a common label. Definitions, reference periods, population coverage, institutional boundaries and collection practices must be sufficiently aligned for the observed difference to have a stable meaning."],"bullets":["Control changes.","Define the decision each assessment must support before it informs a consequential decision.","Moderate material variation.","Calibrate assessors.","Align tasks and criteria with learning outcomes."]},{"heading":"Patterns requiring examination","paragraphs":["Interpretation of assessment moderation should avoid two errors: treating a formal commitment as proof of effect, and treating one adverse case as proof that every part of the system has failed. Oversight of the analysis should reflect the principle that reliability without validity produces consistent but potentially irrelevant results. In relation to assessment moderation, validity without adequate consistency may expose learners to unequal judgement. A decision concerning the issue should recognise that missing or delayed information may be patterned rather than random.","Relevant evidence for the comparison will normally include marking criteria and calibrated judgement, authorship and identity controls proportionate to risk, approval and change-control records, analysis of results and differential outcomes, and appeal and correction records. Currency, provenance and representativeness should be established before evidence is used for assurance. In relation to assessment moderation, conflicting records require reconciliation before a complete assurance conclusion is reached.","Decisions concerning the available evidence should remain traceable to the information available for the stated reference period. A revision should state whether the change concerns the underlying condition, the evidence, the method or the interpretation. Without this distinction, a reporting change may be mistaken for improvement or deterioration in educational practice."]},{"heading":"Implications for decision-makers","paragraphs":["A proportionate method is available for assessment moderation. A competent review of the available evidence should prepare a comparability table before analysing results. Record common elements, material differences, breaks in series and the direction in which each limitation may affect the conclusion; do not rank systems where those limitations remain material. Adverse cases and unresolved contradictions should be retained because they may reveal limitations concealed by an average result.","Decision-makers using evidence on the analysis should be told what the data cannot establish as clearly as what it can. Users should be able to distinguish descriptive, comparative and evaluative findings and understand their proper level of application. A finding should not be transferred beyond its setting without testing the relevant contextual differences."],"bullets":["Has a classification changed?","Are the populations defined on the same basis?","Is the remaining difference educationally material?","Do the reference periods align?","Are exclusions and missing records comparable?"]},{"heading":"Limits of inference","paragraphs":["The intended substantive result should remain the starting point for review. For assessment moderation, assessment should provide valid and sufficiently consistent evidence that the stated learning outcomes have been achieved by the learner receiving the result. The existence of an approved measure or completed activity is not evidence of educational effect. Implementation evidence should be sufficient to identify unequal consequences and assign corrective responsibility.","Accountability for the issue should follow decision-making authority. Delegation of delivery does not remove the need for a named authority to oversee material learner impact.","The measure of progress on the analysis is not the amount of policy or documentation produced. Progress is demonstrated when the intended educational result is achieved, adverse variation is identified and responsible bodies act where it is not."]}],"word_count":723,"content_hash":"sha256-36976630c5affc52ea4895c0f488f0e42af013dd83aa79364b673ff3d9e33bcb","seo_keywords":["assessment moderation","international education data research","education quality","ICEQC"],"schema_type":"AnalysisNewsArticle"}