Automated decision oversight: what constitutes adequate evidence

In examining automated decision oversight: what constitutes adequate evidence, the relevant concern is the effect of consequential decisions on learners, institutions and resources entrusted for education.

Application to automated decision oversight

The position at publication is informed by the expanding use of AI-supported education decisions; evidence from the affected setting remains necessary before reaching a conclusion on automated decision oversight.

Review of the control should address both system-level conditions and institutional practice. In reviewing automated decision oversight, technology may support teaching, administration and access, but consequential educational decisions must remain accountable, explainable and open to effective review.

• Review incidents and supplier changes.
• Test performance across relevant groups.
• Classify uses by effect on learners.
• Retain accountable human decision-makers.
• Notify users of material limitations.

Controls for automated decision oversight

For the applicable expectation, evidence should be relevant to the stated requirement, sufficiently complete for the affected scope, current for the decision period and attributable to a source with knowledge or control of the matter. Volume does not cure a gap in relevance. For automated decision oversight, any condition preventing complete assurance should appear with the evidence on which the judgement relies.

The principal risks in relation to the matter are opaque use of personal or inferred data, unclear responsibility between providers and suppliers, automation bias in consequential decisions, and loss of meaningful human review. For automated decision oversight, a weakness in one part of the control environment may obscure a related failure elsewhere. Across the defined scope, review should follow the sequence of decisions and records rather than assess documents in isolation.

For automated decision oversight, assurance of the matter should draw on more than one form of evidence. Useful records include supplier change and incident records, an inventory of systems and their intended uses, pre-deployment and periodic performance testing, learner information and accessible challenge routes, and records of human review and overrides.

• What would require expanded testing?
• What fact must be established?
• Does it cover the material scope?
• Do independent sources agree?
• Is the evidence current and attributable?

Review of automated decision oversight

The method for the conclusion is to define the proposition to be established, identify the minimum combination of records, test authenticity and reconcile contradictions. Expand the sample where an exception, complaint or material unexplained variation indicates that the initial evidence may not be representative.

Proportionality in relation to the applicable requirement does not mean reduced protection for learners exposed to greater risk. For the applicable requirement, a technical capability is not evidence that a use is educationally justified. When examining automated decision oversight, accuracy measured in one setting may not transfer to another population, language, curriculum or decision context. For the applicable expectation, a prescribed method should not be treated as the only acceptable method where another approach establishes the same outcome with equivalent evidence.

For automated decision oversight, decisions concerning the applicable expectation should remain traceable to the information available for the stated reference period.

Public reporting on the control should distinguish established fact, analytical judgement and planned action. Across the defined scope, if definitions, coverage or evidence alter an earlier conclusion, the reason should be stated so that revision is not mistaken for changed performance.

The present development should inform review of the applicable expectation, with attention to the relationship between commitment, implementation and demonstrated outcome.