Sets out the matters that should be established when applying large-scale open online provision, including scope, responsibility and the basis for a reliable conclusion.
The policy and evidence context for large-scale open online provision has been materially shaped by the expansion of massive open online courses in 2012. For the assurance matter, interpretation should begin with the intended outcome, then identify the controls and evidence needed to show that the outcome is achieved across the declared scope. Attention is directed to decisions with material consequences for learners, institutional responsibility and educational resources. The assessment does not depend on adoption of one institutional or administrative design.
Scope of this analysis
The position at publication is informed by the expansion of massive open online courses in 2012; evidence from the affected setting remains necessary before reaching a conclusion on large-scale open online provision. Implementation should proceed on a clear distinction between factual position, public policy and institutional judgement. Decisions and public statements should preserve the distinction, including when the matter is reconsidered.
A proper review of the assurance matter should establish the intended outcome before selecting controls or indicators. In reviewing the control, evidence is sufficient when it is current, attributable, representative of the relevant scope and capable of being reconciled with other available records. Suitability, authorised variation and the date for reconsideration should be established when the arrangement is approved.
Failure in relation to the stated expectation may arise even where the stated policy is reasonable. Material concerns include weak assurance of authorship or performance, inconsistent judgement between markers or locations, reasonable adjustment altering the assessed outcome, and uncontrolled changes to assessment. Review should consider whether an exception is prolonged, recurring or capable of affecting learners outside the cases examined.
A focused examination of the assurance matter requires a clear analytical discipline. In reviewing the control, the subject should be examined as a connected system of policy, people, resources, decisions and evidence. Assurance should examine the complete process, including each change in custody, authority or decision ownership. The decision question, affected scope and measure should align; otherwise the conclusion may be unsupported despite substantial documentation.
The evidential record for the control should permit a reviewer to trace the matter from decision to outcome. This may require moderation and exception records, marking criteria and calibrated judgement, analysis of results and differential outcomes, and assessment maps to learning outcomes, supported by authorship and identity controls proportionate to risk and appeal and correction records. Conflicting records, absent populations and uncertain follow-through require additional testing.
The substantive quality question
Proportionality in relation to large-scale open online provision does not mean reduced protection for learners exposed to greater risk. In reviewing the assurance matter, reliability without validity produces consistent but potentially irrelevant results. Validity without adequate consistency may expose learners to unequal judgement. For the relevant requirement, an isolated example cannot establish consistent operation, and an isolated failure should be evaluated for materiality, recurrence and systemic effect. No exception should continue without a documented basis, accountable approval and scheduled review.
Decisions concerning the stated expectation should remain traceable to the information available for the stated reference period. The reason for revision should be explicit, including whether it arises from new evidence, a methodological change or a different interpretation. A break in method or coverage must not be presented as if it demonstrated a change in educational performance.
- Align tasks and criteria with learning outcomes before it is relied on for a decision with material effect.
- Define the decision each assessment must support before it is relied on for a decision with material effect.
- Review differential and anomalous results within a defined period and review the result.
- Moderate material variation within a defined period and review the result.
- Control changes and retain evidence sufficient for independent review.
Evidence and assurance
Implementation of large-scale open online provision can be tested without imposing unnecessary reporting. Review of the assurance matter should map the complete process, identify the intended result and responsible authority at each stage, and test normal cases together with exceptions. The review should determine whether correction of an individual case is sufficient or broader action is required. Reuse of existing information is appropriate only where its purpose, scope and reliability correspond to the decision under review.
Interpretation of the assurance matter should produce a test that another competent reviewer can apply to comparable evidence. The test should separate mandatory conditions, recommendations and illustrative methods. A finding should describe the evidence and affected scope; it should not rely on undefined terms such as adequate, appropriate or effective without explaining the basis of judgement.
- Who controls each stage?
- Where do exceptions occur?
- What outcome is intended?
- What action is required by the finding?
- Which evidence establishes operation?
Conditions for responsible implementation
Public reporting on large-scale open online provision should distinguish established fact, analytical judgement and planned action. The record should preserve every revision capable of affecting a prior decision. Users should be told when apparent movement results from revision rather than substantive improvement or deterioration.
The system and institutional dimensions of the assurance matter should be considered together. Oversight of the stated expectation should reflect the principle that assessment should provide valid and sufficiently consistent evidence that the stated learning outcomes have been achieved by the learner receiving the result. Authorities and providers hold different responsibilities, both of which must be discharged for the arrangement to operate reliably. Each level should be able to demonstrate the decisions and controls for which it is accountable.
Complete assurance concerning the matter under review cannot rest on a single indicator or isolated control. The final judgement should connect the applicable expectation to implementation and outcomes while identifying unresolved risk.