Using root-cause analysis to improve multilingual literacy assessment

The policy and evidence context for root-cause analysis to improve multilingual literacy assessment has been materially shaped by the international literacy measurement guidance available in 2006. For the intervention, improvement should begin with a defined problem, a credible account of its causes and a measure capable of showing whether the response has worked. The scope should include every materially affected setting, with differences in location, programme, delivery mode and learner population kept visible. A policy approved at the centre is insufficient where local implementation has not been tested.

The stated reference is the international literacy measurement guidance available in 2006. Application to the corrective programme depends on evidence from the relevant jurisdiction or institution. Verified fact, policy expectation and discretionary institutional choice should remain distinct in the record. Decisions and public statements should preserve the distinction, including when the matter is reconsidered.

Public-interest context

The relevant outcome should be capable of direct and consistent explanation. Oversight of root-cause analysis to improve multilingual literacy assessment should reflect the principle that assessment should provide valid and sufficiently consistent evidence that the stated learning outcomes have been achieved by the learner receiving the result. Formal adoption, expenditure and activity do not in themselves establish the intended result. Implementation evidence should be sufficient to identify unequal consequences and assign corrective responsibility.

The technical issue within the corrective programme concerns the basis on which a conclusion is reached. A decision concerning the corrective programme should recognise that cause analysis should explain why the condition occurred and why existing controls did not prevent or detect it. Restating the event, assigning blame or identifying the last person involved does not establish a root cause. The decision record should distinguish the scope supported by evidence from any scope that remains unresolved.

The principal risks in relation to the corrective programme are inconsistent judgement between markers or locations, results used beyond the evidence they support, tasks that do not assess the stated outcome, and weak assurance of authorship or performance. Risk assessment should account for dependencies between controls and the possibility that one failure masks the next. A reliable conclusion requires examination of the connected decision record, not a series of separate document checks.

Operational significance

The evidential record for root-cause analysis to improve multilingual literacy assessment should permit a reviewer to trace the matter from decision to outcome. This may require analysis of results and differential outcomes, authorship and identity controls proportionate to risk, moderation and exception records, and appeal and correction records, supported by marking criteria and calibrated judgement and assessment maps to learning outcomes. Conflicting records, absent populations and uncertain follow-through require additional testing.

The assurance record for the intervention should retain the date of the evidence, the source responsible for it, the scope examined and the version of any instrument or definition applied. The retained record should show whether later movement reflects changed conditions or a change in the information reported. The evidential history should preserve conclusions that were operative when a material decision was made.

• Moderate material variation within a defined period and review the result.
• Retain evidence sufficient for review and retain evidence sufficient for independent review.
• Define the decision each assessment must support and retain evidence sufficient for independent review.
• Calibrate assessors before using it to determine a learner or provider outcome.
• Align tasks and criteria with learning outcomes within a defined period and review the result.

Testing implementation and effect

Responsibility for root-cause analysis to improve multilingual literacy assessment should be visible at the point where consequential decisions are made. The analysis of the affected practice proceeds on the basis that the intervention should be tested on a scale proportionate to the risk before wider implementation, unless immediate system-wide action is necessary to protect learners. Incomplete evidence, unmanaged conflict, absent learner groups or material learner impact require a higher level of review.

For operational review of the corrective programme, authorities and providers should proceed in a defined sequence. Review of the intervention should reconstruct the sequence, test contributing conditions, compare affected and unaffected cases, and identify the control failure that can be changed. Confirm the explanation against evidence before selecting corrective action. Findings should state the affected scope and required action; an observation should not be represented as evidence of conformity or effectiveness.

The improvement record for the corrective programme should contain the verified problem, affected scope, immediate containment, causal analysis, selected intervention, accountable owner, resources, milestones and effectiveness measure. The action record should separate administrative completion from verification of the intended change. Closure reporting should not obscure unresolved action or risk retained by the responsible authority.

For the corrective programme, governing bodies should receive a concise account of the intended result, affected scope, principal risks, evidence limitations and unresolved exceptions. The action record should identify who is responsible and when implementation is due. The matter should remain open until the intended effect is demonstrated across the relevant scope.

Matters requiring continuing review

The analysis of root-cause analysis to improve multilingual literacy assessment should remain within the limits of the evidence. The analysis of the improvement priority proceeds on the basis that a short-term increase in activity may not represent sustained improvement. Measures should remain in place long enough to detect recurrence and unintended effects. In reviewing the affected practice, reliability without validity produces consistent but potentially irrelevant results. Validity without adequate consistency may expose learners to unequal judgement. A conclusion should be qualified where unresolved uncertainty may affect the decision.

The measure of progress on the intervention is not the amount of policy or documentation produced. A credible measure shows whether the intended result is present across the affected scope and what action follows when it is not.