{"schema_version":"ICEQC_NEWS_ARTICLE_V1","id":"iceqc-news-c41e5e86b16f0b8c","slug":"2023-02-09-a-disaggregated-analysis-of-ai-supported-assessment","language":"en","publication_status":"READY_FOR_IMPORT","publication_date":"2023-02-09","last_modified_date":"2023-02-09","title":"A disaggregated analysis of AI-supported assessment","summary":"Reviews a disaggregated analysis of AI-supported assessment and identifies the conditions required for sound interpretation, responsible comparison and defensible conclusions.","category":{"code":"DATA_RESEARCH","label":"Data Research"},"article_type":"Data and research analysis","publisher":"International Council for Education Quality Certification (ICEQC)","jurisdictional_scope":"International","historical_reference_basis":"Rapid adoption of generative AI tools","reference_authority":"Relevant public authorities and official international sources","sections":[{"heading":null,"paragraphs":["The present attention to AI-supported assessment follows the rapid adoption of generative AI tools and requires a careful distinction between public commitment, institutional practice and demonstrated result. A decision concerning the issue should recognise that evidence should inform action without implying a level of precision, coverage or causal certainty that the underlying data cannot support. Learner effect, institutional duty and proper resource use should inform the judgement. System context should determine the appropriate administrative arrangement within the governing requirements.","The historical reference basis is the rapid adoption of generative AI tools. Its relevance to the analysis should be assessed against the affected jurisdiction, learner population and form of provision.","Implementation of the comparison should be organised around a decision that can be tested. In reviewing The comparison, a sound interpretation should identify the unit of analysis, reference period, denominator, exclusions, missing values and any change in definition or collection practice. In relation to a disaggregated analysis of AI-supported assessment, the implementation record should link purpose, authority, resources, operation and reported result."]},{"heading":"Analytical scope","paragraphs":["The quality significance of AI-supported assessment follows from a basic distinction between availability and effective provision. Any indicator used in relation to the measure should distinguish description from causal explanation. Interpretation should retain uncertainty, distributional differences and limits on generalisation.","The technical issue within the available evidence concerns the basis on which a conclusion is reached. For the measure, an average may improve while a material group experiences no improvement or a worse outcome. Disaggregation should follow a defined public-interest question and should protect confidentiality where small numbers could identify individuals. The judgement should state its supporting evidence and any condition limiting application to the declared scope.","The principal risks associated with the issue should be assessed as connected conditions. A provider should also consider tasks that do not assess the stated outcome and reasonable adjustment altering the assessed outcome. The control response should reflect whether an affected learner can identify the error and obtain an effective remedy in time.","Evidence collection should be designed around the decision question rather than administrative convenience. For the analysis, the most relevant material is likely to include assessment maps to learning outcomes, appeal and correction records, approval and change-control records, and authorship and identity controls proportionate to risk. In relation to a disaggregated analysis of AI-supported assessment, each source has limitations; confidence depends on corroboration between independent records and transparent treatment of uncertainty."]},{"heading":"Definitions and data coverage","paragraphs":["Implementation of AI-supported assessment can be tested without imposing unnecessary reporting. For the analysis, the reviewer should examine results by relevant learner, programme, location and delivery characteristics; compare both levels and rates of change; and test whether observed gaps persist after differences in coverage and prior conditions are considered. Reuse of existing information is appropriate only where its purpose, scope and reliability correspond to the decision under review.","The analytical record for the analysis should state the research question, data source, unit of analysis, reference period, coverage, exclusions, treatment of missing values and principal limitations.","Records relating to the analysis should preserve both the conclusion and its limits. The correction record should state what the new evidence changes and which earlier conclusions or decisions require review. Where reliance has occurred, correction may require review of affected decisions as well as amendment of published information."]},{"heading":"Use of the findings","paragraphs":["Proportionality in relation to AI-supported assessment does not mean reduced protection for learners exposed to greater risk. In reviewing The comparison, reliability without validity produces consistent but potentially irrelevant results. Validity without adequate consistency may expose learners to unequal judgement. The analysis of the measure proceeds on the basis that missing or delayed information may be patterned rather than random. No exception should continue without a documented basis, accountable approval and scheduled review.","Accountability for the comparison should follow decision-making authority. Where work is delegated, the record should continue to identify who is accountable for material consequences to learners.","Neither one indicator nor one control can establish the complete position on the issue.","The reference period and version of the source identified in Rapid adoption of generative AI tools should remain traceable. A later correction, expanded dataset or revised classification may justify a new conclusion, but it should not reconstruct the earlier record without stating what changed and how the change affects comparability."]}],"word_count":708,"content_hash":"sha256-ad2081e289a3cda581e5a9c5953537ea48f5cda581f7e83ba098a45c831ec81e","seo_keywords":["a disaggregated analysis of AI-supported assessment","international education data research","education quality","ICEQC"],"schema_type":"AnalysisNewsArticle"}