{"schema_version":"ICEQC_NEWS_ARTICLE_V1","id":"iceqc-news-7b806d707f99233f","slug":"2013-06-13-testing-whether-improvements-in-assessment-reliability-are-sustained","language":"en","publication_status":"READY_FOR_IMPORT","publication_date":"2013-06-13","last_modified_date":"2013-06-13","title":"Testing whether improvements in assessment reliability are sustained","summary":"Describes how improvement in testing whether improvements in assessment reliability are sustained can be planned, measured and maintained through clear responsibility.","category":{"code":"QUALITY_IMPROVEMENT_METHODS","label":"Quality Improvement Methods"},"article_type":"Quality improvement method","publisher":"International Council for Education Quality Certification (ICEQC)","jurisdictional_scope":"International","historical_reference_basis":"International comparative assessment practice","reference_authority":"Relevant public authorities and official international sources","sections":[{"heading":null,"paragraphs":["The immediate international context is the international comparative assessment practice. Its significance for assessment reliability lies in the quality of implementation rather than in formal acknowledgement alone. The analysis of the issue under review proceeds on the basis that effective improvement requires ownership, a time-bound intervention and independent confirmation that the intended result has been achieved.","Implementation of the intervention should be organised around a decision that can be tested. The analysis of the practice proceeds on the basis that follow-up should determine whether the change is embedded in ordinary operations and whether it has created new risks or unequal effects. In relation to testing whether improvements in assessment reliability are sustained, resources and activity should be reconciled with the operating evidence and result for which the responsible function is accountable."]},{"heading":"Scope of the improvement","paragraphs":["The stated reference—the international comparative assessment practice—establishes the contemporaneous context. Any conclusion about assessment reliability still requires evidence from the setting concerned. Decision-makers should state which matters are evidenced, which express policy and which require authorised judgement. Later review should not obscure whether the earlier position rested on fact, policy or judgement.","The quality significance of the improvement objective follows from a basic distinction between availability and effective provision. A decision concerning the improvement objective should recognise that assessment should provide valid and sufficiently consistent evidence that the stated learning outcomes have been achieved by the learner receiving the result. Assurance should follow the learner journey and test more than a single access point or aggregate result."],"bullets":["Control changes.","Moderate material variation.","Align tasks and criteria with learning outcomes.","Retain evidence sufficient for review.","Define the decision each assessment must support."]},{"heading":"Implementation responsibilities","paragraphs":["In practical terms, assessment reliability should be reviewed against a stated method rather than general assurance. Oversight of the corrective programme should reflect the principle that effectiveness is the demonstrated change in the condition the action was intended to address. Completion of training, publication of guidance or installation of a system is an output and should not be reported as an outcome without further evidence. Decision-makers should receive an intelligible account of how the result was reached and where it should not be applied.","The evidential record for the corrective programme should permit a reviewer to trace the matter from decision to outcome. This may require assessment maps to learning outcomes, analysis of results and differential outcomes, appeal and correction records, and moderation and exception records, supported by marking criteria and calibrated judgement and authorship and identity controls proportionate to risk.","Where the intervention involves partners, suppliers or several public bodies, responsibility should be mapped across the complete service. Governance between participating bodies should make information duties and corrective authority explicit. Division of delivery responsibilities must not create gaps in learner protection.","A decision to close improvement work on the corrective programme should be made by a person with authority and sufficient independence from implementation."]},{"heading":"Testing effectiveness","paragraphs":["A proportionate method is available for assessment reliability. For the improvement objective, the reviewer should set a baseline and success measure before intervention, define the review period, compare the result with the intended outcome and examine adverse or unequal effects. In relation to testing whether improvements in assessment reliability are sustained, continue monitoring long enough to determine whether the improvement is sustained. The review record should preserve exceptions capable of showing a weakness in design, implementation or coverage.","A narrow control over the improvement objective may create false assurance. In the present context, results used beyond the evidence they support, uncontrolled changes to assessment and weak assurance of authorship or performance may produce acceptable aggregate reporting while individual learners remain exposed to material disadvantage.","Decisions concerning the corrective programme should remain traceable to the information available for the stated reference period. The reason for revision should be explicit, including whether it arises from new evidence, a methodological change or a different interpretation.","The analysis of the intervention should remain within the limits of the evidence. In reviewing the improvement objective, methods should be proportionate to the significance and recurrence of the problem; low-risk local issues and systemic learner-protection failures require different levels of control. The analysis of the practice proceeds on the basis that reliability without validity produces consistent but potentially irrelevant results. Validity without adequate consistency may expose learners to unequal judgement. If uncertainty could change a consequential decision, additional evidence or a narrower conclusion is required.","Any response to the present development should test the evidential connection between the improvement objective, its implementation and the outcome claimed. Improvement should be supported by evidence and an accountable decision record capable of public scrutiny."]}],"word_count":763,"content_hash":"sha256-b664244c955fd94989d7c0b58ad1a19c2d9f8c4f1800722283a7a3f6a7390a45","seo_keywords":["testing whether improvements in assessment reliability are sustained","education quality improvement","education quality","ICEQC"],"schema_type":"TechArticle"}