AI competency frameworks: evidence for policy decisions

The international frameworks for students and teachers released in 2024 provides the immediate reference point for consideration of AI competency frameworks in 2024. Comparable indicators can support public decision-making, but they do not remove the need to examine variation within systems and institutions. Suitability should be judged within the relevant system rather than against a presumed universal administrative model.

The formal status of the international frameworks for students and teachers released in 2024 should be preserved in any public account. For the measure, the instrument should be used to identify the intended direction, the actors addressed and the implementation measures that remain necessary.

For AI competency frameworks, international artificial-intelligence competency frameworks for students and teachers were released in September 2024. They organise capability around human-centred understanding, ethics, techniques and application, with teacher responsibilities also covering pedagogy and professional development. Competency frameworks guide curriculum and workforce planning; they do not establish that competence has been achieved without suitable learning and assessment evidence.

For the measure, the public interest is not confined to institutional compliance. In reviewing AI competency frameworks, technology may support teaching, administration and access, but consequential educational decisions must remain accountable, explainable and open to effective review.

Evidence and method

In the context of AI competency frameworks, the subject should be examined as a connected system of policy, people, resources, decisions and evidence. The assessment should follow authority and information across functional boundaries and verify completion of required action. An imprecise scope or measure may produce a credible-looking record that does not answer the relevant decision question.

For decisions concerning AI competency frameworks, responsibility should be identifiable at the point where consequential decisions are made. Where an indicator is used as a proxy, the relationship between the proxy and the underlying educational outcome should be stated and tested. Escalation should follow whenever the available record cannot support a safe conclusion for the affected learners.

Patterns requiring examination

The principal risks in relation to AI competency frameworks are loss of meaningful human review, unverified outputs entering teaching or assessment, unclear responsibility between providers and suppliers, and automation bias in consequential decisions. Risk assessment should account for dependencies between controls and the possibility that one failure masks the next.

• Review incidents and supplier changes.
• Classify uses by effect on learners.
• Notify users of material limitations, with responsibility, scope and timing recorded.
• Retain accountable human decision-makers.
• Prohibit uses for which evidence or authority is insufficient.

Implications for decision-makers

Assurance of AI competency frameworks should draw on more than one form of evidence. Useful records include data provenance and access controls, supplier change and incident records, records of human review and overrides, learner information and accessible challenge routes, and an inventory of systems and their intended uses. Documents should be reconciled with observed practice and, where relevant, the experience of affected learners. Evidence of effectiveness should represent the declared scope, including adverse and exceptional cases.

The review method for the measure should be reproducible. Review of the available evidence should map the complete process, identify the intended result and responsible authority at each stage, and test normal cases together with exceptions. Review should establish the reach of the condition before determining the corrective response. Within the scope under review, working papers should allow another competent reviewer to understand the evidence, judgement and treatment of material exceptions.

The analytical record for AI competency frameworks should state the research question, data source, unit of analysis, reference period, coverage, exclusions, treatment of missing values and principal limitations.

Limits of inference

Interpretation of AI competency frameworks should avoid two errors: treating a formal commitment as proof of effect, and treating one adverse case as proof that every part of the system has failed. A technical capability is not evidence that a use is educationally justified. Accuracy measured in one setting may not transfer to another population, language, curriculum or decision context. Association should not be presented as causation, and statistical significance should not be treated as evidence of educational importance without further analysis.

In the context of AI competency frameworks, traceability is necessary for accountable decision-making and fair correction. For comparative analysis, the responsible body should be able to identify the evidence considered, the judgement made, the person or body authorised to make it and the action that followed. Material changes require a traceable effective date and explanation so that prior reliance can be reviewed fairly.

Accountability for the measure should follow decision-making authority. For decisions concerning AI competency frameworks, relevant evidence should reach the body authorised to commit resources, amend policy or accept residual risk, and its judgement should be recorded. Where work is delegated, the record should continue to identify who is accountable for material consequences to learners.

Data used for the comparison should be interpreted against stable definitions and an identifiable population. As regards AI competency frameworks, a revision or break in series should not be reported as a change in performance. Improvement of AI competency frameworks should be supported by evidence and an accountable decision record capable of public scrutiny.