Digital education: methodological considerations

In 2016, consideration of digital education must take account of the expansion of technology-supported learning and the responsibilities it places before education systems. A decision concerning the evidence under review should recognise that evidence should inform action without implying a level of precision, coverage or causal certainty that the underlying data cannot support. The chosen response should address the risk without weakening access, educational quality or fair treatment.

Scope of this analysis

The historical reference basis is the expansion of technology-supported learning. Its relevance to digital education should be assessed against the affected jurisdiction, learner population and form of provision. The international development warrants attention, but a consequential conclusion still requires current, attributable and representative evidence for the affected scope.

Responsibility for the matter examined should be visible at the point where consequential decisions are made. A decision concerning the evidence under review should recognise that trend claims require comparable observations over time and a documented account of revisions, breaks in series and changes in coverage. The matter should be escalated when evidence is incomplete, a conflict is present, affected learners are not represented or the likely effect is material.

The technical issue within the matter examined concerns the basis on which a conclusion is reached. In reviewing the reported measure, the subject should be examined as a connected system of policy, people, resources, decisions and evidence. Assurance should not overlook failures arising at the boundary between otherwise adequate controls. The judgement should state its supporting evidence and any condition limiting application to the declared scope.

Assurance of the evidence under review should draw on more than one form of evidence. Useful records include learner access and participation information, service availability and incident records, assessment validity and integrity reviews, accessibility and usability testing, and supplier performance and exit arrangements. Documentary conformity alone is insufficient where operation or learner experience indicates a material difference. System-wide assurance cannot be inferred from a favourable case chosen after the event.

The substantive quality question

The relevant outcome should be capable of direct and consistent explanation. A decision concerning digital education should recognise that a change in delivery mode should not weaken the defined learning outcomes, learner protection, accessibility or reliability of assessment. Assurance should not stop at adoption, resourcing or completion of administrative tasks. Authorities and providers require evidence of operation and effect, with a route to identify and correct unequal or unintended consequences.

Failure in relation to the analytical question may arise even where the stated policy is reasonable. Material concerns include unclear identity and participation records, technology access determining educational access, supplier dependency without continuity controls, and inaccessible content or interaction. Materiality depends on the consequence and extent of an exception, not only on how often it appears in sampled records.

• Define the educational purpose of the technology and retain evidence sufficient for independent review.
• Test access before requiring use and retain evidence sufficient for independent review.
• Assure assessment validity, including material exceptions and unequal effects.
• Provide alternative routes for material barriers and retain evidence sufficient for independent review.
• Maintain continuity and supplier exit controls and retain evidence sufficient for independent review.

Evidence and assurance

For operational review of digital education, authorities and providers should proceed in a defined sequence. The method for the reported measure is to map the complete process, identify the intended result and responsible authority at each stage, and test normal cases together with exceptions. The review should determine whether correction of an individual case is sufficient or broader action is required. A finding must identify its evidential basis, reach and required response, without giving informal observations a status they do not have.

Publication of findings on the analytical question should distinguish observed values, estimates and interpretation. Revisions, breaks in series and changes in classification should be visible. Where disaggregation creates small or unstable groups, confidentiality and uncertainty should be managed without concealing a material disparity that requires further investigation.

Proportionality in relation to the comparison does not mean reduced protection for learners exposed to greater risk. Oversight of the reported measure should reflect the principle that digital participation data should not be treated as a direct measure of learning. Log-ins, connection time and activity counts require interpretation alongside assessment and learner experience. The analysis of the evidence under review proceeds on the basis that a single indicator rarely provides an adequate account of quality. Quantitative evidence should be considered with implementation records and the experience of affected learners. Each exception should record its basis, authorisation, duration and review date.

Accountability and effective correction both depend on a record that can be followed from evidence to decision. For the matter examined, the responsible body should be able to identify the evidence considered, the judgement made, the person or body authorised to make it and the action that followed. Material changes require a traceable effective date and explanation so that prior reliance can be reviewed fairly.

Conditions for responsible implementation

Where digital education involves partners, suppliers or several public bodies, responsibility should be mapped across the complete service. The division of responsibilities should cover records, communication, escalation and the power to require correction. Division of delivery responsibilities must not create gaps in learner protection.

A complete conclusion on the reported measure requires evidence extending beyond an individual measure or safeguard. A conclusion should be revised when stronger evidence materially changes the assessment of implementation, outcome or risk.