{"schema_version":"ICEQC_NEWS_ARTICLE_V1","id":"iceqc-news-374be4e742b1d622","slug":"2012-09-24-evidence-boundaries-in-the-evaluation-of-large-scale-open-online-learning","language":"en","publication_status":"READY_FOR_IMPORT","publication_date":"2012-09-24","last_modified_date":"2012-09-24","title":"Evidence boundaries in the evaluation of large-scale open online learning","summary":"Clarifies the practical meaning of evidence boundaries in the evaluation of large-scale open online learning and the evidence required to support a reliable assurance conclusion.","category":{"code":"STANDARDS_INTERPRETATION","label":"Standards Interpretation"},"article_type":"Standards interpretation","publisher":"International Council for Education Quality Certification (ICEQC)","jurisdictional_scope":"International","historical_reference_basis":"Rapid expansion of open online courses in 2012","reference_authority":"Relevant public authorities and official international sources","sections":[{"heading":null,"paragraphs":["The present attention to large-scale open online learning follows the rapid expansion of open online courses in 2012 and requires a careful distinction between public commitment, institutional practice and demonstrated result. For the stated expectation, interpretation should begin with the intended outcome, then identify the controls and evidence needed to show that the outcome is achieved across the declared scope.","The governing expectation for the stated expectation should be capable of consistent application. In reviewing the relevant requirement, a provider should be able to trace the expectation from approved policy through implementation, monitoring, identified exceptions and corrective action. In relation to evidence boundaries in the evaluation of large-scale open online learning, criteria affecting learners should not permit materially different interpretation without an evidenced reason."]},{"heading":"Meaning in practice","paragraphs":["The historical reference basis is the rapid expansion of open online courses in 2012. Its relevance to large-scale open online learning should be assessed against the affected jurisdiction, learner population and form of provision.","In practical terms, the issue under review should be reviewed against a stated method rather than general assurance. In reviewing the issue under review, the subject should be examined as a connected system of policy, people, resources, decisions and evidence. Transfer of decisions or records can expose weaknesses not visible in separate reviews of individual controls.","Failure in relation to the relevant requirement may arise even where the stated policy is reasonable. Material concerns include inaccessible content or interaction, assessment methods that do not support valid judgements, reduced opportunities for timely support, and unclear identity and participation records. Materiality depends on the consequence and extent of an exception, not only on how often it appears in sampled records.","Assurance of the issue under review should draw on more than one form of evidence. Useful records include assessment validity and integrity reviews, teacher capability and workload information, supplier performance and exit arrangements, learner access and participation information, and delivery-mode design and approval records. System-wide assurance cannot be inferred from a favourable case chosen after the event."]},{"heading":"Responsibilities and material risks","paragraphs":["Implementation of large-scale open online learning can be tested without imposing unnecessary reporting. Review of the stated expectation should map the complete process, identify the intended result and responsible authority at each stage, and test normal cases together with exceptions. In relation to evidence boundaries in the evaluation of large-scale open online learning, existing records may be used if reliable and relevant, but data collected for another purpose may not answer the assurance question.","Decisions concerning the stated expectation should remain traceable to the information available for the stated reference period. The reason for revision should be explicit, including whether it arises from new evidence, a methodological change or a different interpretation. Without this distinction, a reporting change may be mistaken for improvement or deterioration in educational practice."],"bullets":["What outcome is intended?","Which evidence establishes operation?","Where do exceptions occur?","Who controls each stage?","What action is required by the finding?"]},{"heading":"Basis for a reliable conclusion","paragraphs":["Accountability for large-scale open online learning should follow decision-making authority.","The final record on the relevant requirement should identify the applicable expectation, the relevant scope, the evidence examined, the sampling basis, material exceptions and the reason for the conclusion. Departure from an illustrative method may be justified where equivalent outcome and evidence are established. Unresolved limitations should be stated with the conclusion and carried forward for action."],"bullets":["Support staff and learners before using it to determine a learner or provider outcome.","Define the educational purpose of the technology.","Test access before requiring use.","Assure assessment validity before it is relied on for a decision with material effect.","Provide alternative routes for material barriers."]},{"heading":"Maintaining effective oversight","paragraphs":["The quality significance of large-scale open online learning follows from a basic distinction between availability and effective provision. A decision concerning the control should recognise that a change in delivery mode should not weaken the defined learning outcomes, learner protection, accessibility or reliability of assessment. Review should cover the stages at which learners receive information, provision, assessment, support and remedy.","Proportionality in relation to the issue under review does not mean reduced protection for learners exposed to greater risk. For the issue under review, digital participation data should not be treated as a direct measure of learning. Log-ins, connection time and activity counts require interpretation alongside assessment and learner experience. For the relevant requirement, an isolated example cannot establish consistent operation, and an isolated failure should be evaluated for materiality, recurrence and systemic effect. Each exception should record its basis, authorisation, duration and review date.","The measure of progress on the assurance question is not the amount of policy or documentation produced. The relevant measure is demonstrated public benefit, including detection and correction of material variation."]}],"word_count":773,"content_hash":"sha256-8abd9d8206066345c9df9c4ecd2bafa575065ae3766c9c9365de77420d82d92d","seo_keywords":["evidence boundaries in the evaluation of large-scale open online learning","education quality standards","education quality","ICEQC"],"schema_type":"TechArticle"}