Requirements for large-scale open online provision are considered through evidence sufficiency, accountable decisions, corrective action and follow-up.
In examining large-scale open online provision: evidence expectations for identity, assessment and learner support, for the conclusion, interpretation should begin with the intended outcome, then identify the controls and evidence needed to show that the outcome is achieved across the declared scope.
Application to large-scale open online provision
The position at publication is informed by the expansion of massive open online courses in 2012; evidence from the affected setting remains necessary before reaching a conclusion on large-scale open online provision.
A proper review of the conclusion should establish the intended outcome before selecting controls or indicators. For large-scale open online provision, evidence is sufficient when it is current, attributable, representative of the relevant scope and capable of being reconciled with other available records.
Failure in relation to the applicable expectation may arise even where the stated policy is reasonable. Material concerns include weak assurance of authorship or performance, inconsistent judgement between markers or locations, reasonable adjustment altering the assessed outcome, and uncontrolled changes to assessment. In the context of large-scale open online provision, review should consider whether an exception is prolonged, recurring or capable of affecting learners outside the cases examined.
The evidential record for the control should permit a reviewer to trace the matter from decision to outcome. This may require moderation and exception records, marking criteria and calibrated judgement, analysis of results and differential outcomes, and assessment maps to learning outcomes, supported by authorship and identity controls proportionate to risk and appeal and correction records. For large-scale open online provision, conflicting records, absent populations and uncertain follow-through require additional testing.
Controls for large-scale open online provision
For the applicable requirement, an isolated example cannot establish consistent operation, and an isolated failure should be evaluated for materiality, recurrence and systemic effect. Across the defined scope, no exception should continue without a documented basis, accountable approval and scheduled review.
For large-scale open online provision, decisions concerning the applicable expectation should remain traceable to the information available for the stated reference period.
- Align tasks and criteria with learning outcomes before it is relied on for a decision with material effect.
- Define the decision each assessment must support before it is relied on for a decision with material effect.
- Review differential and anomalous results.
- Moderate material variation.
- Control changes.
Review of large-scale open online provision
Review of the conclusion should map the complete process, identify the intended result and responsible authority at each stage, and test normal cases together with exceptions.
Interpretation of large-scale open online provision should produce a test that another competent reviewer can apply to comparable evidence.
- Who controls each stage?
- Where do exceptions occur?
- What outcome is intended?
- What action is required by the finding?
- Which evidence establishes operation?
Implications for large-scale open online provision
Review of the conclusion should address both system-level conditions and institutional practice. For decisions concerning large-scale open online provision, assessment should provide valid and sufficiently consistent evidence that the stated learning outcomes have been achieved by the learner receiving the result.
In examining large-scale open online provision: evidence expectations for identity, assessment and learner support, complete assurance concerning the matter cannot rest on a single indicator or isolated control.