{"schema_version":"ICEQC_NEWS_ARTICLE_V1","id":"iceqc-news-c3ea6b62b233d598","slug":"2024-09-20-ai-competency-frameworks-evidence-for-policy-decisions","language":"en","publication_status":"READY_FOR_IMPORT","publication_date":"2024-09-20","last_modified_date":"2024-09-20","title":"AI competency frameworks: evidence for policy decisions","summary":"Examines aI competency frameworks: evidence for policy decisions, defining the evidence base, coverage and limits that should govern comparison and use of the findings.","category":{"code":"DATA_RESEARCH","label":"Data Research"},"article_type":"Data and research analysis","publisher":"International Council for Education Quality Certification (ICEQC)","jurisdictional_scope":"International","historical_reference_basis":"International frameworks for students and teachers released in 2024","reference_authority":"United Nations Educational, Scientific and Cultural Organization","sections":[{"heading":null,"paragraphs":["The international frameworks for students and teachers released in 2024 provides the immediate reference point for consideration of AI competency frameworks in 2024. Oversight of the comparison should reflect the principle that comparable indicators can support public decision-making, but they do not remove the need to examine variation within systems and institutions. Suitability should be judged within the relevant system rather than against a presumed universal administrative model.","The formal status of the international frameworks for students and teachers released in 2024 should be preserved in any public account. For the measure, the instrument should be used to identify the intended direction, the actors addressed and the implementation measures that remain necessary.","International artificial-intelligence competency frameworks for students and teachers were released in September 2024. They organise capability around human-centred understanding, ethics, techniques and application, with teacher responsibilities also covering pedagogy and professional development. Competency frameworks guide curriculum and workforce planning; they do not establish that competence has been achieved without suitable learning and assessment evidence.","For the measure, the public interest is not confined to institutional compliance. In reviewing the measure, technology may support teaching, administration and access, but consequential educational decisions must remain accountable, explainable and open to effective review."]},{"heading":"Evidence and method","paragraphs":["A focused examination of AI competency frameworks requires a clear analytical discipline. In reviewing the available evidence, the subject should be examined as a connected system of policy, people, resources, decisions and evidence. The assessment should follow authority and information across functional boundaries and verify completion of required action. In relation to aI competency frameworks, an imprecise scope or measure may produce a credible-looking record that does not answer the relevant decision question.","Responsibility for the measure should be visible at the point where consequential decisions are made. In reviewing The comparison, where an indicator is used as a proxy, the relationship between the proxy and the underlying educational outcome should be stated and tested. Escalation should follow whenever the available record cannot support a safe conclusion for the affected learners."]},{"heading":"Patterns requiring examination","paragraphs":["The principal risks in relation to AI competency frameworks are loss of meaningful human review, unverified outputs entering teaching or assessment, unclear responsibility between providers and suppliers, and automation bias in consequential decisions. Risk assessment should account for dependencies between controls and the possibility that one failure masks the next."],"bullets":["Review incidents and supplier changes.","Classify uses by effect on learners.","Notify users of material limitations, with responsibility, scope and timing recorded.","Retain accountable human decision-makers.","Prohibit uses for which evidence or authority is insufficient."]},{"heading":"Implications for decision-makers","paragraphs":["Assurance of AI competency frameworks should draw on more than one form of evidence. Useful records include data provenance and access controls, supplier change and incident records, records of human review and overrides, learner information and accessible challenge routes, and an inventory of systems and their intended uses. Documents should be reconciled with observed practice and, where relevant, the experience of affected learners. Evidence of effectiveness should represent the declared scope, including adverse and exceptional cases.","The review method for the measure should be reproducible. Review of the available evidence should map the complete process, identify the intended result and responsible authority at each stage, and test normal cases together with exceptions. Review should establish the reach of the condition before determining the corrective response. In relation to aI competency frameworks, working papers should allow another competent reviewer to understand the evidence, judgement and treatment of material exceptions.","The analytical record for the issue should state the research question, data source, unit of analysis, reference period, coverage, exclusions, treatment of missing values and principal limitations."]},{"heading":"Limits of inference","paragraphs":["Interpretation of AI competency frameworks should avoid two errors: treating a formal commitment as proof of effect, and treating one adverse case as proof that every part of the system has failed. Oversight of the comparison should reflect the principle that a technical capability is not evidence that a use is educationally justified. Accuracy measured in one setting may not transfer to another population, language, curriculum or decision context. A decision concerning the available evidence should recognise that association should not be presented as causation, and statistical significance should not be treated as evidence of educational importance without further analysis.","Traceability is necessary for accountable decision-making and fair correction. For The comparison, the responsible body should be able to identify the evidence considered, the judgement made, the person or body authorised to make it and the action that followed. Material changes require a traceable effective date and explanation so that prior reliance can be reviewed fairly.","Accountability for the measure should follow decision-making authority. Relevant evidence should reach the body authorised to commit resources, amend policy or accept residual risk, and its judgement should be recorded. Where work is delegated, the record should continue to identify who is accountable for material consequences to learners.","Data used for the comparison should be interpreted against stable definitions and an identifiable population. A revision or break in series should not be reported as a change in performance. Improvement should be supported by evidence and an accountable decision record capable of public scrutiny."]}],"word_count":842,"content_hash":"sha256-3d79f684daee4e330207c4700483f0d7243e6603b9f390bcb90d97e08ca0ae9f","seo_keywords":["aI competency frameworks","international education data research","education quality","ICEQC"],"schema_type":"AnalysisNewsArticle"}