{"schema_version":"ICEQC_NEWS_ARTICLE_V1","id":"iceqc-news-1c245c620a6d422d","slug":"2013-04-01-consistency-in-the-assessment-of-assessment-reliability","language":"en","publication_status":"READY_FOR_IMPORT","publication_date":"2013-04-01","last_modified_date":"2013-04-01","title":"Consistency in evaluating assessment reliability","summary":"Explains how consistency in evaluating assessment reliability should be applied in practice, including scope, evidential sufficiency, accountable decisions and justified.","category":{"code":"STANDARDS_INTERPRETATION","label":"Standards Interpretation"},"article_type":"Standards interpretation","publisher":"International Council for Education Quality Certification (ICEQC)","jurisdictional_scope":"International","historical_reference_basis":"International comparative assessment practice","reference_authority":"Relevant public authorities and official international sources","sections":[{"heading":null,"paragraphs":["Current consideration of consistency in the assessment of assessment reliability is informed by the international comparative assessment practice, with consequences for governance, evidence and the treatment of affected learners. A decision concerning the relevant requirement should recognise that the central issue is the meaning of the expectation in practice, including its scope, the evidence needed to demonstrate it and the circumstances in which it may not apply. The control response should be sufficient to protect learners while avoiding burdens not justified by the evidence."]},{"heading":"Meaning in practice","paragraphs":["The historical reference basis is the international comparative assessment practice. Its relevance to consistency in the assessment of assessment reliability should be assessed against the affected jurisdiction, learner population and form of provision.","The governing expectation for the stated expectation should be capable of consistent application. Oversight of the stated expectation should reflect the principle that evidence is sufficient when it is current, attributable, representative of the relevant scope and capable of being reconciled with other available records. Operational definitions should be precise enough to support consistent consequential decisions and explain justified variation.","The principal risks in relation to the stated expectation are uncontrolled changes to assessment, tasks that do not assess the stated outcome, results used beyond the evidence they support, and weak assurance of authorship or performance. In relation to consistency in evaluating assessment reliability, a weakness in one part of the control environment may obscure a related failure elsewhere. In relation to consistency in evaluating assessment reliability, review should follow the sequence of decisions and records rather than assess documents in isolation.","The technical issue within the issue under review concerns the basis on which a conclusion is reached. The analysis of the assurance question proceeds on the basis that consistency does not require identical decisions regardless of context. It requires comparable matters to be treated on the same principles, with material differences explained by relevant evidence and recorded criteria. A conclusion should identify both its evidential basis and the part of the stated scope for which assurance cannot be given.","The evidential record for the control should permit a reviewer to trace the matter from decision to outcome. This may require analysis of results and differential outcomes, appeal and correction records, moderation and exception records, and approval and change-control records, supported by authorship and identity controls proportionate to risk and assessment maps to learning outcomes."]},{"heading":"Responsibilities and material risks","paragraphs":["Proportionality in relation to consistency in the assessment of assessment reliability does not mean reduced protection for learners exposed to greater risk. The analysis of the assurance question proceeds on the basis that reliability without validity produces consistent but potentially irrelevant results. Validity without adequate consistency may expose learners to unequal judgement. For the relevant requirement, a prescribed method should not be treated as the only acceptable method where another approach establishes the same outcome with equivalent evidence.","Decisions concerning the issue under review should remain traceable to the information available for the stated reference period."],"bullets":["Calibrate assessors.","Review differential and anomalous results.","Moderate material variation.","Align tasks and criteria with learning outcomes.","Retain evidence sufficient for review."]},{"heading":"Basis for a reliable conclusion","paragraphs":["The review method for consistency in the assessment of assessment reliability should be reproducible. In reviewing the stated expectation, responsible bodies should use common definitions and decision criteria, calibrate responsible staff, review outliers and compare outcomes across locations and groups. Where variation is justified, retain the reason and verify that it is applied without arbitrary disadvantage. The retained analysis should be reproducible from the selected evidence, decision rule and recorded reasons for accepted exceptions.","The final record on the assurance question should identify the applicable expectation, the relevant scope, the evidence examined, the sampling basis, material exceptions and the reason for the conclusion. If an alternative method is accepted, the record should demonstrate that it achieves the same required outcome. A limitation preventing a complete conclusion should remain visible and unresolved until suitable evidence is obtained."],"bullets":["Does review correct inconsistent treatment?","Where are outcomes materially different?","Is the reason relevant and documented?","Are common criteria in use?","Have decision-makers been calibrated?"]},{"heading":"Maintaining effective oversight","paragraphs":["Accountability for consistency in the assessment of assessment reliability should follow decision-making authority. Evidence of material risk should be placed before the body with authority to act, together with a traceable decision.","The quality significance of consistency in evaluating assessment reliability under review follows from a basic distinction between availability and effective provision. A decision concerning the stated expectation should recognise that assessment should provide valid and sufficiently consistent evidence that the stated learning outcomes have been achieved by the learner receiving the result.","The appropriate response to the stated expectation is therefore one of controlled implementation and review. Where evidence cannot support assurance, the limitation should be reported and corrective work should remain open."]}],"word_count":787,"content_hash":"sha256-65c63fdb1c6674f0d9f81973cc008151a3a9452a6590c43fd9b470d6b2dd6810","seo_keywords":["consistency in evaluating assessment reliability","education quality standards","education quality","ICEQC"],"schema_type":"TechArticle"}