Longitudinal evidence on artificial intelligence in education policy

The Beijing Consensus adopted in May 2019 provides the immediate context for artificial intelligence in education policy. Evidence concerning artificial intelligence in education policy should inform action without implying a level of precision, coverage or causal certainty that the underlying data cannot support.

Evidence base

For artificial intelligence in education policy, the Beijing Consensus on Artificial Intelligence and Education was adopted in May 2019. It addresses policy planning, management, teaching, learning, skills, lifelong learning, inclusion, gender equality, data and research. It promotes human-centred and equitable use rather than technology adoption as an end in itself. Authorities and providers should therefore connect each proposed use to an educational purpose, governance responsibility and evidence of benefit and risk.

The system and institutional dimensions of artificial intelligence in education policy should be considered together. For the analysis, technology may support teaching, administration and access, but consequential educational decisions must remain accountable, explainable and open to effective review. Authorities and providers hold different responsibilities, both of which must be discharged for the arrangement to operate reliably. Each level should be able to demonstrate the decisions and controls for which it is accountable.

• Classify uses by effect on learners.
• Retain accountable human decision-makers.
• Prohibit uses for which evidence or authority is insufficient.
• Control personal and confidential information.
• Review incidents and supplier changes.

Coverage and comparability

The Beijing Consensus adopted in May 2019 provides a policy reference for artificial intelligence in education policy. This distinction protects learners from overstated claims and enables providers to plan against a defined obligation.

Review of the measure should be based on a stated method rather than general assurance. Trend analysis depends on stable definitions and repeated observation of comparable populations. For decisions concerning artificial intelligence in education policy, a change in policy, coverage or recording practice can create an apparent movement that is not a change in the underlying educational condition.

Responsible interpretation

In work concerning artificial intelligence in education policy, the applicable expectation should be capable of consistent application. Reported averages should be accompanied by sufficient distributional information to identify material differences between learner groups, locations and forms of provision. Operational definitions should be precise enough to support consistent consequential decisions and explain justified variation.

The principal risks in relation to the available evidence are automation bias in consequential decisions, opaque use of personal or inferred data, unverified outputs entering teaching or assessment, and unequal performance across learner groups. Risk assessment should account for dependencies between controls and the possibility that one failure masks the next.

• Were earlier values revised?
• Is the period long enough to show sustained movement?
• What external condition may explain the change?
• Is the baseline still comparable?
• Have coverage or definitions changed?

Limitations and reporting

Relevant evidence for artificial intelligence in education policy will normally include data provenance and access controls, records of human review and overrides, documented authority for each consequential use, pre-deployment and periodic performance testing, and an inventory of systems and their intended uses. Within the scope under review, conflicting records require reconciliation before a complete assurance conclusion is reached.

Authorities and providers reviewing the measure should proceed in a defined sequence. The review should establish a baseline, annotate every material change in definition or collection, compare like periods and retain revised series. In work concerning artificial intelligence in education policy, where comparability is interrupted, begin a new series or present the break clearly rather than joining unlike observations. The record for artificial intelligence in education policy should distinguish a finding that requires action from an observation that supports no formal conclusion.

Limitations and reporting

Publication of findings on artificial intelligence in education policy should distinguish observed values, estimates and interpretation.

Conclusions concerning the comparison require careful treatment of scope and evidential limits. For comparative analysis, a technical capability is not evidence that a use is educationally justified. For artificial intelligence in education policy, accuracy measured in one setting may not transfer to another population, language, curriculum or decision context. A single indicator rarely provides an adequate account of quality. Quantitative evidence should be considered with implementation records and the experience of affected learners. Material limitations should be stated with the finding presented to decision-makers and affected learners.

Records relating to the comparison should preserve both the conclusion and its limits. For decisions concerning artificial intelligence in education policy, a changed evidential position should be applied to the affected scope, including prior decisions that may no longer be reliable. Where reliance has occurred, correction may require review of affected decisions as well as amendment of published information.

Accountability for the comparison should follow decision-making authority.

Governance of the available evidence requires a clear allocation of authority, information and follow-through. For artificial intelligence in education policy, the responsible body should receive matters requiring resources, policy change or formal risk acceptance. Public confidence cannot be separated from an institution's ability to identify responsibility and substantiate its conclusions.