{"schema_version":"ICEQC_NEWS_ARTICLE_V1","id":"iceqc-news-e20eef72d2c56562","slug":"2012-12-12-large-scale-open-online-provision-evidence-expectations-for-identity-assessment-and-learner-support","language":"en","publication_status":"READY_FOR_IMPORT","publication_date":"2012-12-12","last_modified_date":"2012-12-12","title":"Large-scale open online provision: evidence expectations for identity, assessment and learner support","summary":"Explains how large-scale open online provision: evidence expectations for identity, assessment and learner support should be applied in practice.","category":{"code":"STANDARDS_INTERPRETATION","label":"Standards Interpretation"},"article_type":"Standards interpretation","publisher":"International Council for Education Quality Certification (ICEQC)","jurisdictional_scope":"International","historical_reference_basis":"Expansion of massive open online courses in 2012","reference_authority":"Relevant public authorities and official international sources","sections":[{"heading":null,"paragraphs":["The policy and evidence context for large-scale open online provision has been materially shaped by the expansion of massive open online courses in 2012. For the assurance question, interpretation should begin with the intended outcome, then identify the controls and evidence needed to show that the outcome is achieved across the declared scope. Attention is directed to decisions with material consequences for learners, institutional responsibility and educational resources. The assessment does not depend on adoption of one institutional or administrative design."]},{"heading":"Meaning in practice","paragraphs":["The position at publication is informed by the expansion of massive open online courses in 2012; evidence from the affected setting remains necessary before reaching a conclusion on large-scale open online provision.","A proper review of the assurance question should establish the intended outcome before selecting controls or indicators. In reviewing the control, evidence is sufficient when it is current, attributable, representative of the relevant scope and capable of being reconciled with other available records. Suitability, authorised variation and the date for reconsideration should be established when the arrangement is approved.","Failure in relation to the stated expectation may arise even where the stated policy is reasonable. Material concerns include weak assurance of authorship or performance, inconsistent judgement between markers or locations, reasonable adjustment altering the assessed outcome, and uncontrolled changes to assessment. Review should consider whether an exception is prolonged, recurring or capable of affecting learners outside the cases examined.","A focused examination of the assurance question requires a clear analytical discipline. In reviewing the control, the subject should be examined as a connected system of policy, people, resources, decisions and evidence. Assurance should examine the complete process, including each change in custody, authority or decision ownership. In relation to large-scale open online provision, the decision question, affected scope and measure should align; otherwise the conclusion may be unsupported despite substantial documentation.","The evidential record for the control should permit a reviewer to trace the matter from decision to outcome. This may require moderation and exception records, marking criteria and calibrated judgement, analysis of results and differential outcomes, and assessment maps to learning outcomes, supported by authorship and identity controls proportionate to risk and appeal and correction records. Conflicting records, absent populations and uncertain follow-through require additional testing."]},{"heading":"Responsibilities and material risks","paragraphs":["Proportionality in relation to large-scale open online provision does not mean reduced protection for learners exposed to greater risk. In reviewing the assurance question, reliability without validity produces consistent but potentially irrelevant results. Validity without adequate consistency may expose learners to unequal judgement. For the relevant requirement, an isolated example cannot establish consistent operation, and an isolated failure should be evaluated for materiality, recurrence and systemic effect. In relation to large-scale open online provision, no exception should continue without a documented basis, accountable approval and scheduled review.","Decisions concerning the stated expectation should remain traceable to the information available for the stated reference period. The reason for revision should be explicit, including whether it arises from new evidence, a methodological change or a different interpretation."],"bullets":["Align tasks and criteria with learning outcomes before it is relied on for a decision with material effect.","Define the decision each assessment must support before it is relied on for a decision with material effect.","Review differential and anomalous results.","Moderate material variation.","Control changes."]},{"heading":"Basis for a reliable conclusion","paragraphs":["Implementation of large-scale open online provision can be tested without imposing unnecessary reporting. Review of the assurance question should map the complete process, identify the intended result and responsible authority at each stage, and test normal cases together with exceptions. Reuse of existing information is appropriate only where its purpose, scope and reliability correspond to the decision under review.","Interpretation of the assurance question should produce a test that another competent reviewer can apply to comparable evidence."],"bullets":["Who controls each stage?","Where do exceptions occur?","What outcome is intended?","What action is required by the finding?","Which evidence establishes operation?"]},{"heading":"Maintaining effective oversight","paragraphs":["Public reporting on large-scale open online provision should distinguish established fact, analytical judgement and planned action.","The system and institutional dimensions of the assurance question should be considered together. Oversight of the stated expectation should reflect the principle that assessment should provide valid and sufficiently consistent evidence that the stated learning outcomes have been achieved by the learner receiving the result. Authorities and providers hold different responsibilities, both of which must be discharged for the arrangement to operate reliably. Each level should be able to demonstrate the decisions and controls for which it is accountable.","Complete assurance concerning the issue under review cannot rest on a single indicator or isolated control. The final judgement should connect the applicable expectation to implementation and outcomes while identifying unresolved risk."]}],"word_count":768,"content_hash":"sha256-64dc6259e8ec8075b0b4906a0c780b196e1c6ae7ea53a7fa4c11b6d1b9faf8d1","seo_keywords":["large-scale open online provision","education quality standards","education quality","ICEQC"],"schema_type":"TechArticle"}