Completion and retention: measurement and statistical uncertainty

Evidence relevant to completion and retention

A proper review of the analysis should establish the intended outcome before selecting controls or indicators. When examining completion and retention, where an indicator is used as a proxy, the relationship between the proxy and the underlying educational outcome should be stated and tested.

Risk assessment of the analysis should give particular attention to aggregate gains concealing excluded groups, admission without adequate support, and attrition detected too late. A provider should also consider completion measures detached from learning and support allocated without evidence of need. In the context of completion and retention, stronger controls are required where learners may not detect an error or where later correction cannot restore the lost opportunity.

In the context of completion and retention, analysis should make its decision rule explicit. An estimate should be reported with the uncertainty arising from sampling, measurement, missing information and model choice. A precise number may still support only a cautious conclusion if the underlying assumptions are weak.

For decisions concerning completion and retention, each source should have a stated purpose in supporting or limiting the conclusion. For the available evidence, the most relevant material is likely to include evaluation of interventions, clearly defined access and completion indicators, resource allocation by need, and progression and early-warning information.

Application to completion and retention

Across the defined scope, different support may be justified where it addresses a documented barrier without changing the required educational outcome.

For completion and retention, records relating to the available evidence should preserve both the conclusion and its limits.

• Analyse barriers across the learner journey.
• Report limitations in available data before it informs a consequential decision.
• Evaluate differential outcomes.
• Monitor progression at useful intervals, identifying the accountable function and affected scope.
• Identify groups not represented in aggregate results.

Controls for completion and retention

For the available evidence, the reviewer should report intervals or sensitivity ranges where appropriate, test whether the conclusion changes under plausible assumptions, and distinguish statistical detectability from educational significance. Record unresolved uncertainty beside the principal finding.

Decision-makers using evidence on the analysis should be told what the data cannot establish as clearly as what it can.

• What are the principal sources of uncertainty?
• Would a reasonable alternative assumption change the conclusion?
• Are small groups adequately represented?
• How should uncertainty affect the decision?
• Is the apparent difference larger than the measurement error?

Review of completion and retention

For decisions concerning completion and retention, education systems should examine not only who enters education, but who can participate effectively, progress and complete with the intended learning outcomes.

For completion and retention, a complete conclusion on the issue requires evidence extending beyond an individual measure or safeguard.