{"schema_version":"ICEQC_NEWS_ARTICLE_V1","id":"iceqc-news-49ff9a22820790f2","slug":"2009-01-23-the-use-of-proxy-measures-in-analysing-assessment-moderation","language":"en","publication_status":"READY_FOR_IMPORT","publication_date":"2009-01-23","last_modified_date":"2009-01-23","title":"The use of proxy measures in analysing assessment moderation","summary":"Reviews the use of proxy measures in analysing assessment moderation and identifies the conditions required for sound interpretation, responsible comparison and defensible.","category":{"code":"DATA_RESEARCH","label":"Data Research"},"article_type":"Data and research analysis","publisher":"International Council for Education Quality Certification (ICEQC)","jurisdictional_scope":"International","historical_reference_basis":"Cross-system comparability and academic standards","reference_authority":"Relevant public authorities and official international sources","sections":[{"heading":null,"paragraphs":["The cross-system comparability and academic standards provides the immediate context for assessment moderation. A decision concerning the available evidence should recognise that evidence should inform action without implying a level of precision, coverage or causal certainty that the underlying data cannot support. The assessment addresses decisions capable of affecting learners, institutions or the proper use of entrusted educational resources. The assessment does not depend on adoption of one institutional or administrative design.","The governing expectation for the available evidence should be capable of consistent application. Oversight of the comparison should reflect the principle that a sound interpretation should identify the unit of analysis, reference period, denominator, exclusions, missing values and any change in definition or collection practice. Operational definitions should be precise enough to support consistent consequential decisions and explain justified variation."]},{"heading":"Evidence and method","paragraphs":["The position at publication is informed by the cross-system comparability and academic standards; evidence from the affected setting remains necessary before reaching a conclusion on assessment moderation. The decision basis should identify what is evidenced, what reflects policy and what depends on authorised discretion. In relation to the use of proxy measures in analysing assessment moderation, later review should not obscure whether the earlier position rested on fact, policy or judgement.","For the available evidence, the public interest is not confined to institutional compliance. A decision concerning the analysis should recognise that assessment should provide valid and sufficiently consistent evidence that the stated learning outcomes have been achieved by the learner receiving the result."],"bullets":["Moderate material variation.","Retain evidence sufficient for review.","Define the decision each assessment must support.","Align tasks and criteria with learning outcomes.","Review differential and anomalous results before it informs a consequential decision."]},{"heading":"Patterns requiring examination","paragraphs":["In practical terms, assessment moderation should be reviewed against a stated method rather than general assurance. Oversight of the issue should reflect the principle that a proxy is useful only where its relationship with the intended outcome is sufficiently understood. Participation, activity and expenditure may support learning, but none constitutes direct evidence of learning without an explicit and tested connection.","Relevant evidence for the comparison will normally include authorship and identity controls proportionate to risk, appeal and correction records, assessment maps to learning outcomes, analysis of results and differential outcomes, and marking criteria and calibrated judgement. Evidence outside the relevant period or scope should be identified and given no more weight than its limitations permit. Contradictory evidence should be investigated and resolved, not omitted from the record.","In this case, governing bodies should receive a concise account of the intended result, affected scope, principal risks, evidence limitations and unresolved exceptions. Management should assign each material action to an accountable owner and completion date.","Decision-makers using evidence on the measure should be told what the data cannot establish as clearly as what it can. The finding should identify its analytical character and the system, institution, programme or learner population to which it applies. Application in another setting depends on a separate examination of context and comparability."]},{"heading":"Implications for decision-makers","paragraphs":["For operational review of assessment moderation, authorities and providers should proceed in a defined sequence. A competent review of the comparison should state the construct to be measured, explain why the proxy is expected to represent it, test that relationship against direct evidence and identify circumstances in which the proxy may fail. Do not allow convenience to determine the measure.","Risk assessment of the issue should give particular attention to tasks that do not assess the stated outcome, inconsistent judgement between markers or locations, and results used beyond the evidence they support. A provider should also consider weak assurance of authorship or performance and uncontrolled changes to assessment.","The assurance record for the issue should retain the date of the evidence, the source responsible for it, the scope examined and the version of any instrument or definition applied. In relation to the use of proxy measures in analysing assessment moderation, traceable source and version information allow genuine improvement to be distinguished from administrative revision. Earlier conclusions should remain traceable if they affected a learner, provider or public decision.","Particular care is required when interpreting evidence about the issue. The analysis of the issue proceeds on the basis that reliability without validity produces consistent but potentially irrelevant results. Validity without adequate consistency may expose learners to unequal judgement. For the measure, association should not be presented as causation, and statistical significance should not be treated as evidence of educational importance without further analysis. Material limitations should be stated with the finding presented to decision-makers and affected learners.","The current development provides a basis for examining whether the issue is supported by responsible action and demonstrable result."]}],"word_count":769,"content_hash":"sha256-5a39a84a46ff91fd1e482037e0e0880a74e06bf2714b2accbec0b5832a1dd758","seo_keywords":["the use of proxy measures in analysing assessment moderation","international education data research","education quality","ICEQC"],"schema_type":"AnalysisNewsArticle"}