Explains evidence expectations for identity, assessment and learner support in relation to large-scale open online provision, with attention to decision authority.
The policy and evidence context for large-scale open online provision has been materially shaped by the expansion of massive open online courses in 2012. For the assurance conclusion, interpretation should begin with the intended outcome, then identify the controls and evidence needed to show that the outcome is achieved across the declared scope. Attention is directed to decisions with material consequences for learners, institutional responsibility and educational resources. The assessment does not depend on adoption of one institutional or administrative design.
Meaning in practice
The position at publication is informed by the expansion of massive open online courses in 2012; evidence from the affected setting remains necessary before reaching a conclusion on large-scale open online provision.
A proper review of the assurance conclusion should establish the intended outcome before selecting controls or indicators. As regards large-scale open online provision, evidence is sufficient when it is current, attributable, representative of the relevant scope and capable of being reconciled with other available records. Suitability, authorised variation and the date for reconsideration should be established when the arrangement is approved.
Failure in relation to the applicable expectation may arise even where the stated policy is reasonable. Material concerns include weak assurance of authorship or performance, inconsistent judgement between markers or locations, reasonable adjustment altering the assessed outcome, and uncontrolled changes to assessment. In the context of large-scale open online provision, review should consider whether an exception is prolonged, recurring or capable of affecting learners outside the cases examined.
When examining large-scale open online provision, the subject should be examined as a connected system of policy, people, resources, decisions and evidence. Assurance should examine the complete process, including each change in custody, authority or decision ownership. The decision question, affected scope and measure should align; otherwise the conclusion may be unsupported despite substantial documentation.
The evidential record for the control should permit a reviewer to trace the matter from decision to outcome. This may require moderation and exception records, marking criteria and calibrated judgement, analysis of results and differential outcomes, and assessment maps to learning outcomes, supported by authorship and identity controls proportionate to risk and appeal and correction records. In work concerning large-scale open online provision, conflicting records, absent populations and uncertain follow-through require additional testing.
Responsibilities and material risks
Proportionality in relation to large-scale open online provision does not mean reduced protection for learners exposed to greater risk. Reliability without validity produces consistent but potentially irrelevant results. Validity without adequate consistency may expose learners to unequal judgement. For the applicable requirement, an isolated example cannot establish consistent operation, and an isolated failure should be evaluated for materiality, recurrence and systemic effect. Within the scope under review, no exception should continue without a documented basis, accountable approval and scheduled review.
For large-scale open online provision, decisions concerning the applicable expectation should remain traceable to the information available for the stated reference period. The reason for revision should be explicit, including whether it arises from new evidence, a methodological change or a different interpretation.
- Align tasks and criteria with learning outcomes before it is relied on for a decision with material effect.
- Define the decision each assessment must support before it is relied on for a decision with material effect.
- Review differential and anomalous results.
- Moderate material variation.
- Control changes.
Basis for a reliable conclusion
Implementation of large-scale open online provision can be tested without imposing unnecessary reporting. Review of the assurance conclusion should map the complete process, identify the intended result and responsible authority at each stage, and test normal cases together with exceptions. Reuse of existing information is appropriate only where its purpose, scope and reliability correspond to the decision under review.
Interpretation of large-scale open online provision should produce a test that another competent reviewer can apply to comparable evidence.
- Who controls each stage?
- Where do exceptions occur?
- What outcome is intended?
- What action is required by the finding?
- Which evidence establishes operation?
Maintaining effective oversight
Public reporting on large-scale open online provision should distinguish established fact, analytical judgement and planned action.
The system and institutional dimensions of the assurance conclusion should be considered together. For decisions concerning large-scale open online provision, assessment should provide valid and sufficiently consistent evidence that the stated learning outcomes have been achieved by the learner receiving the result. Authorities and providers hold different responsibilities, both of which must be discharged for the arrangement to operate reliably. Each level should be able to demonstrate the decisions and controls for which it is accountable.
Complete assurance concerning the matter cannot rest on a single indicator or isolated control. The final judgement on large-scale open online provision should connect the applicable expectation to implementation and outcomes while identifying unresolved risk.