Evidence boundaries in the evaluation of large-scale open online learning

In examining evidence boundaries in the evaluation of large-scale open online learning, for the applicable expectation, interpretation should begin with the intended outcome, then identify the controls and evidence needed to show that the outcome is achieved across the declared scope.

Application of the evidence to evidence boundaries in the evaluation of large-scale open online learning

For decisions concerning large-scale open online learning, this analysis is informed by rapid expansion of open online courses in 2012. Its relevance to large-scale open online learning should be assessed against the affected jurisdiction, learner population and form of provision.

In examining evidence boundaries in the evaluation of large-scale open online learning, review of the matter should follow a stated and reproducible method.

Failure in relation to the applicable requirement may arise even where the stated policy is reasonable. Material concerns include inaccessible content or interaction, assessment methods that do not support valid judgements, reduced opportunities for timely support, and unclear identity and participation records. For large-scale open online learning, materiality depends on the consequence and extent of an exception, not only on how often it appears in sampled records.

When examining large-scale open online learning, assurance of the matter should draw on more than one form of evidence. Useful records include assessment validity and integrity reviews, teacher capability and workload information, supplier performance and exit arrangements, learner access and participation information, and delivery-mode design and approval records. Across the defined scope, system-wide assurance cannot be inferred from a favourable case chosen after the event.

Controls relevant to evidence boundaries in the evaluation of large-scale open online learning

Implementation of large-scale open online learning can be tested without imposing unnecessary reporting. Review of the applicable expectation should map the complete process, identify the intended result and responsible authority at each stage, and test normal cases together with exceptions.

Decisions concerning the applicable expectation should remain traceable to the information available for the stated reference period. For large-scale open online learning, the reason for revision should be explicit, including whether it arises from new evidence, a methodological change or a different interpretation.

• What outcome is intended?
• Which evidence establishes operation?
• Where do exceptions occur?
• Who controls each stage?
• What action is required by the finding?

Review criteria for evidence boundaries in the evaluation of large-scale open online learning

Accountability for large-scale open online learning should follow decision-making authority.

When examining large-scale open online learning, the final record on the applicable requirement should identify the applicable expectation, the relevant scope, the evidence examined, the sampling basis, material exceptions and the reason for the conclusion.

• Support staff and learners before using it to determine a learner or provider outcome.
• Define the educational purpose of the technology.
• Test access before requiring use.
• Assure assessment validity before it is relied on for a decision with material effect.
• Provide alternative routes for material barriers.

Implications for evidence boundaries in the evaluation of large-scale open online learning

For large-scale open online learning, a change in delivery mode should not weaken the defined learning outcomes, learner protection, accessibility or reliability of assessment.

Proportionality in relation to the matter does not mean reduced protection for learners exposed to greater risk. For the matter, digital participation data should not be treated as a direct measure of learning. For large-scale open online learning, log-ins, connection time and activity counts require interpretation alongside assessment and learner experience. For the applicable requirement, an isolated example cannot establish consistent operation, and an isolated failure should be evaluated for materiality, recurrence and systemic effect.

For large-scale open online learning, the measure is demonstrated public benefit, including detection and correction of material variation.