Testing whether improvements in governance of generative artificial intelligence are sustained

Its significance for governance of generative artificial intelligence lies in the quality of implementation rather than in formal acknowledgement alone. Learner effect, institutional duty and proper resource use should inform the judgement.

For governance of generative artificial intelligence, international guidance on generative artificial intelligence in education and research was released in September 2023. It calls for a human-centred approach, protection of data privacy, age-appropriate use, validation and institutional capacity. Immediate provider controls should address authorised uses, assessment, disclosure, information security, unequal access and human review while evidence on educational benefit and risk continues to develop.

In the context of governance of generative artificial intelligence, implementation of the intended improvement should be organised around a decision that can be tested.

Application and scope

The relevance of the international guidance released in September 2023 is contextual. Consequential findings on governance of generative artificial intelligence require current, attributable evidence for the scope concerned.

For governance of generative artificial intelligence, technology may support teaching, administration and access, but consequential educational decisions must remain accountable, explainable and open to effective review.

• Prohibit uses for which evidence or authority is insufficient.
• Classify uses by effect on learners.
• Test performance across relevant groups.
• Notify users of material limitations.
• Control personal and confidential information.

Controls relevant to whether improvements in governance of generative artificial intelligence are sustained

The criteria applied to governance of generative artificial intelligence should be settled and recorded before the evidence is assessed. Effectiveness is the demonstrated change in the condition the action was intended to address. Across the defined scope, a stated decision rule enables comparable examination and limits retrospective explanations of adverse evidence.

Failure in relation to the intended improvement may arise even where the stated policy is reasonable. Material concerns include unequal performance across learner groups, loss of meaningful human review, opaque use of personal or inferred data, and automation bias in consequential decisions. For governance of generative artificial intelligence, review should consider whether an exception is prolonged, recurring or capable of affecting learners outside the cases examined.

For governance of generative artificial intelligence, each source should have a stated purpose in supporting or limiting the conclusion. For the intended improvement, the most relevant material is likely to include learner information and accessible challenge routes, records of human review and overrides, an inventory of systems and their intended uses, and documented authority for each consequential use.

• What condition should change?
• Has the improvement been sustained?
• What was the baseline?
• When should an effect be visible?
• Did the effect reach the intended group?

Review criteria for whether improvements in governance of generative artificial intelligence are sustained

The review method for governance of generative artificial intelligence should be reproducible. Responsible bodies should set a baseline and success measure before intervention, define the review period, compare the result with the intended outcome and examine adverse or unequal effects.

Improvement of governance of generative artificial intelligence should proceed through controlled tests where risk permits.

For the matter, a technical capability is not evidence that a use is educationally justified. Across the defined scope, accuracy measured in one setting may not transfer to another population, language, curriculum or decision context.

Public reporting on the corrective action should distinguish established fact, analytical judgement and planned action. For governance of generative artificial intelligence, material revisions should be traceable to their reason and effective date.

When examining governance of generative artificial intelligence, any response to the present development should test the evidential connection between whether improvements in governance of generative artificial intelligence are sustained, its implementation and the outcome claimed. Improvement of governance of generative artificial intelligence should be supported by evidence and an accountable decision record capable of public scrutiny.