AI competency frameworks: evidence for policy decisions

The international frameworks for students and teachers released in 2024 provides the immediate reference point for consideration of AI competency frameworks in 2024. Oversight of the comparison should reflect the principle that comparable indicators can support public decision-making, but they do not remove the need to examine variation within systems and institutions. Attention is directed to the practical conditions in which decisions have consequences for learners, institutions and entrusted resources. Suitability should be judged within the relevant system rather than against a presumed universal administrative model.

The formal status of the international frameworks for students and teachers released in 2024 should be preserved in any public account. Adoption records an agreed instrument or policy position; it does not necessarily make every provision directly enforceable in every jurisdiction. For The reported measure, the instrument should be used to identify the intended direction, the actors addressed and the implementation measures that remain necessary. Domestic law and authorised guidance continue to determine specific legal duties.

International artificial-intelligence competency frameworks for students and teachers were released in September 2024. They organise capability around human-centred understanding, ethics, techniques and application, with teacher responsibilities also covering pedagogy and professional development. Competency frameworks guide curriculum and workforce planning; they do not establish that competence has been achieved without suitable learning and assessment evidence.

For The reported measure, the public interest is not confined to institutional compliance. In reviewing The reported measure, technology may support teaching, administration and access, but consequential educational decisions must remain accountable, explainable and open to effective review. Material arrangements should be communicated clearly, with an accessible route to correct error or unfair treatment.

The present position

A focused examination of AI competency frameworks requires a clear analytical discipline. In reviewing The evidence under review, the subject should be examined as a connected system of policy, people, resources, decisions and evidence. The assessment should follow authority and information across functional boundaries and verify completion of required action. An imprecise scope or measure may produce a credible-looking record that does not answer the relevant decision question.

Responsibility for the reported measure should be visible at the point where consequential decisions are made. In reviewing The comparison, where an indicator is used as a proxy, the relationship between the proxy and the underlying educational outcome should be stated and tested. Escalation should follow whenever the available record cannot support a safe conclusion for the affected learners.

The substantive quality question

The principal risks in relation to AI competency frameworks are loss of meaningful human review, unverified outputs entering teaching or assessment, unclear responsibility between providers and suppliers, and automation bias in consequential decisions. Risk assessment should account for dependencies between controls and the possibility that one failure masks the next. The evidential trail should be examined from initial decision to outcome, including transfers of responsibility.

• Review incidents and supplier changes, and retain the basis, responsible function and affected scope.
• Classify uses by effect on learners and retain evidence sufficient for independent review.
• Notify users of material limitations, with responsibility, scope and timing recorded.
• Retain accountable human decision-makers within a defined period and review the result.
• Prohibit uses for which evidence or authority is insufficient and retain evidence sufficient for independent review.

What should be examined

Assurance of AI competency frameworks should draw on more than one form of evidence. Useful records include data provenance and access controls, supplier change and incident records, records of human review and overrides, learner information and accessible challenge routes, and an inventory of systems and their intended uses. Documents should be reconciled with observed practice and, where relevant, the experience of affected learners. Evidence of effectiveness should represent the declared scope, including adverse and exceptional cases.

The review method for the reported measure should be reproducible. Review of the evidence under review should map the complete process, identify the intended result and responsible authority at each stage, and test normal cases together with exceptions. Review should establish the reach of the condition before determining the corrective response. Working papers should allow another competent reviewer to understand the evidence, judgement and treatment of material exceptions.

The analytical record for the matter examined should state the research question, data source, unit of analysis, reference period, coverage, exclusions, treatment of missing values and principal limitations. Results should be reproducible from the retained data and method. Any causal explanation should be identified separately from descriptive findings and supported by an appropriate design.

Limitations and safeguards

Interpretation of AI competency frameworks should avoid two errors: treating a formal commitment as proof of effect, and treating one adverse case as proof that every part of the system has failed. Oversight of the comparison should reflect the principle that a technical capability is not evidence that a use is educationally justified. Accuracy measured in one setting may not transfer to another population, language, curriculum or decision context. A decision concerning the evidence under review should recognise that association should not be presented as causation, and statistical significance should not be treated as evidence of educational importance without further analysis.

Traceability is necessary for accountable decision-making and fair correction. For The comparison, the responsible body should be able to identify the evidence considered, the judgement made, the person or body authorised to make it and the action that followed. Material changes require a traceable effective date and explanation so that prior reliance can be reviewed fairly.

Accountability for the reported measure should follow decision-making authority. Relevant evidence should reach the body authorised to commit resources, amend policy or accept residual risk, and its judgement should be recorded. Where work is delegated, the record should continue to identify who is accountable for material consequences to learners.

Data used for the comparison should be interpreted against stable definitions and an identifiable population. A revision or break in series should not be reported as a change in performance. Improvement should be supported by evidence and an accountable decision record capable of public scrutiny.