Risk-based improvement planning for AI-supported assessment

In examining risk-based improvement planning for AI-supported assessment, improvement of AI-supported assessment should begin with a defined problem, a credible account of its causes and a measure capable of showing whether the response has worked.

For AI-supported assessment, the intended substantive result should remain the starting point for review. Assessment should provide valid and sufficiently consistent evidence that the stated learning outcomes have been achieved by the learner receiving the result.

Application of the evidence to risk-based improvement planning for AI-supported assessment

Assurance of AI-supported assessment should draw on more than one form of evidence. Useful records include authorship and identity controls proportionate to risk, analysis of results and differential outcomes, appeal and correction records, marking criteria and calibrated judgement, and approval and change-control records.

Its relevance to risk-based improvement planning for AI-supported assessment should be assessed against the affected jurisdiction, learner population and form of provision.

For AI-supported assessment, materiality should be judged by the possible effect on learning, safety, rights, recognition, public resources and the reliability of a consequential decision.

Risk assessment of the corrective action should give particular attention to tasks that do not assess the stated outcome, weak assurance of authorship or performance, and results used beyond the evidence they support. A provider should also consider uncontrolled changes to assessment and inconsistent judgement between markers or locations. For AI-supported assessment, the control response should reflect whether an affected learner can identify the error and obtain an effective remedy in time.

Controls relevant to risk-based improvement planning for AI-supported assessment

In examining risk-based improvement planning for AI-supported assessment, across the defined scope, operational definitions should be precise enough to support consistent consequential decisions and explain justified variation.

In examining risk-based improvement planning for AI-supported assessment, accountability for AI-supported assessment should follow decision-making authority.

When examining AI-supported assessment, decisions concerning the corrective action should remain traceable to the information available for the stated reference period.

• Control changes.
• Align tasks and criteria with learning outcomes.
• Moderate material variation.
• Review differential and anomalous results.
• Retain evidence sufficient for review.

Review criteria for risk-based improvement planning for AI-supported assessment

The review method for AI-supported assessment should be reproducible. For the corrective action, the reviewer should define escalation thresholds before reviewing cases, consider severity, reach, duration, recurrence and detectability, and record the reason for the final classification.

The improvement record for the corrective action should contain the verified problem, affected scope, immediate containment, causal analysis, selected intervention, accountable owner, resources, milestones and effectiveness measure. When examining AI-supported assessment, the action record should separate administrative completion from verification of the intended change.

The basis and limits of any conclusion concerning corrective action should be explicit. For decisions concerning AI-supported assessment, reliability without validity produces consistent but potentially irrelevant results. Across the defined scope, limitations should be prominent wherever the finding may influence a consequential decision.

Neither one indicator nor one control can establish the complete position on risk-based improvement planning for AI-supported assessment. A conclusion concerning AI-supported assessment should be revised when stronger evidence materially changes the assessment of implementation, outcome or risk.