Explains evidential boundaries in relation to learning-outcome measurement, with attention to decision authority, material exceptions and continuing assurance.
The international assessment and indicator development provides the immediate reference point for consideration of learning-outcome measurement in 2010. Interpretation should begin with the intended outcome, then identify the controls and evidence needed to show that the outcome is achieved across the declared scope.
For the applicable requirement, the public interest is not confined to institutional compliance. For the assurance conclusion, education indicators should support decisions by describing outcomes and variation with definitions and limitations that permit responsible interpretation. For learning-outcome measurement, learner protection requires intelligible information and a timely means of reviewing consequential mistakes or unfair decisions.
Scope and application
The evidential record for learning-outcome measurement should permit a reviewer to trace the matter from decision to outcome. This may require disaggregated results, uncertainty estimates where relevant, population and sampling information, and triangulation with administrative and qualitative evidence, supported by indicator definitions and metadata and revision and comparability records.
This analysis is informed by international assessment and indicator development. Its relevance to the assurance conclusion should be assessed against the affected jurisdiction, learner population and form of provision. In the context of learning-outcome measurement, the international development warrants attention, but a consequential conclusion still requires current, attributable and representative evidence for the affected scope.
When examining learning-outcome measurement, analysis should make its decision rule explicit. Effectiveness is the demonstrated change in the condition the action was intended to address. Completion of training, publication of guidance or installation of a system is an output and should not be reported as an outcome without further evidence. The method should prevent an unfavourable result from being dismissed through an unrecorded change in interpretation.
The principal risks in relation to the control are averages concealing distribution, changes in definition presented as changes in performance, incomplete coverage, and proxy measures treated as direct outcomes. Within the scope under review, a weakness in one part of the control environment may obscure a related failure elsewhere. For learning-outcome measurement, review should follow the sequence of decisions and records rather than assess documents in isolation.
Evidence required
For learning-outcome measurement, the applicable expectation should be capable of consistent application. A provider should be able to trace the expectation from approved policy through implementation, monitoring, identified exceptions and corrective action. Terms governing eligibility, support, assessment, reporting or review should prevent materially different treatment without recorded justification.
For the assurance conclusion, governing bodies should receive a concise account of the intended result, affected scope, principal risks, evidence limitations and unresolved exceptions.
Records relating to the applicable expectation should preserve both the conclusion and its limits. For decisions concerning learning-outcome measurement, new evidence should trigger a traceable correction and review of decisions materially affected by the earlier conclusion. This is material where learners, authorities or institutions relied on information that cannot be corrected by replacing the current text alone.
- Avoid causal claims unsupported by the design before it informs a consequential decision.
- Define the decision the indicator will inform.
- Document numerator and denominator, with responsibility, scope and timing recorded.
- Analyse missing information.
- Disaggregate material results.
Decision criteria and exceptions
The method for the applicable requirement is to set a baseline and success measure before intervention, define the review period, compare the result with the intended outcome and examine adverse or unequal effects. As regards learning-outcome measurement, continue monitoring long enough to determine whether the improvement is sustained. Within the scope under review, adverse cases and unresolved contradictions should be retained because they may reveal limitations concealed by an average result.
The final record on the matter should identify the applicable expectation, the relevant scope, the evidence examined, the sampling basis, material exceptions and the reason for the conclusion. In the context of learning-outcome measurement, if an alternative method is accepted, the record should demonstrate that it achieves the same required outcome. Unresolved limitations should be stated with the conclusion and carried forward for action.
Particular care is required when interpreting evidence about the matter. In the context of learning-outcome measurement, measurement can reveal where outcomes differ; it does not by itself establish why they differ or which intervention will work. For the matter, interpretive guidance should not create an obligation that is absent from the governing instrument or applicable law. Material limitations should be stated with the finding presented to decision-makers and affected learners.
Neither one indicator nor one control can establish the complete position on the applicable expectation. In the context of learning-outcome measurement, a reasoned conclusion should reconcile the governing requirement, evidence of operation, learner outcomes and residual risk, and remain open to better evidence.