{"schema_version":"ICEQC_NEWS_ARTICLE_V1","id":"iceqc-news-73e2f706997538f9","slug":"2006-10-31-cross-system-comparability-in-multilingual-literacy-assessment","language":"en","publication_status":"READY_FOR_IMPORT","publication_date":"2006-10-31","last_modified_date":"2006-10-31","title":"Cross-system comparability in multilingual literacy assessment","summary":"Assesses evidence on cross-system comparability in multilingual literacy assessment, with particular attention to definitions, population coverage, comparability and uncertainty.","category":{"code":"DATA_RESEARCH","label":"Data Research"},"article_type":"Data and research analysis","publisher":"International Council for Education Quality Certification (ICEQC)","jurisdictional_scope":"International","historical_reference_basis":"International literacy measurement guidance available in 2006","reference_authority":"Relevant public authorities and official international sources","sections":[{"heading":null,"paragraphs":["The international literacy measurement guidance available in 2006 provides the immediate context for multilingual literacy assessment. In reviewing the measure, the available evidence should be interpreted with close attention to definitions, population coverage, collection methods and the limits of comparison.","The quality significance of the analysis follows from a basic distinction between availability and effective provision. The analysis of the measure proceeds on the basis that assessment should provide valid and sufficiently consistent evidence that the stated learning outcomes have been achieved by the learner receiving the result. In relation to cross-system comparability in multilingual literacy assessment, review should cover the stages at which learners receive information, provision, assessment, support and remedy."]},{"heading":"Analytical scope","paragraphs":["Relevant evidence for multilingual literacy assessment will normally include appeal and correction records, approval and change-control records, moderation and exception records, authorship and identity controls proportionate to risk, and analysis of results and differential outcomes. Evidence outside the relevant period or scope should be identified and given no more weight than its limitations permit. Contradictory evidence should be investigated and resolved, not omitted from the record.","The historical reference basis is the international literacy measurement guidance available in 2006. Its relevance to the measure should be assessed against the affected jurisdiction, learner population and form of provision.","The analysis of the issue should make its decision rule explicit. For the measure, comparison requires more than the use of a common label. Definitions, reference periods, population coverage, institutional boundaries and collection practices must be sufficiently aligned for the observed difference to have a stable meaning. Comparable evidence should be assessed against criteria settled before the result is known.","A narrow control over the available evidence may create false assurance. In the present context, tasks that do not assess the stated outcome, results used beyond the evidence they support and reasonable adjustment altering the assessed outcome may produce acceptable aggregate reporting while individual learners remain exposed to material disadvantage."]},{"heading":"Definitions and data coverage","paragraphs":["A proper review of multilingual literacy assessment should establish the intended outcome before selecting controls or indicators. Oversight of the available evidence should reflect the principle that where an indicator is used as a proxy, the relationship between the proxy and the underlying educational outcome should be stated and tested. The basis for selection, authority for exceptions and timing of reassessment should remain traceable.","Public reporting on the analysis should distinguish established fact, analytical judgement and planned action. Revision history should remain available where users have relied on the earlier conclusion. In relation to cross-system comparability in multilingual literacy assessment, if definitions, coverage or evidence alter an earlier conclusion, the reason should be stated so that revision is not mistaken for changed performance.","Records relating to the available evidence should preserve both the conclusion and its limits. A changed evidential position should be applied to the affected scope, including prior decisions that may no longer be reliable. Where reliance has occurred, correction may require review of affected decisions as well as amendment of published information."],"bullets":["Retain evidence sufficient for review.","Moderate material variation.","Align tasks and criteria with learning outcomes, identifying the accountable function and affected scope.","Review differential and anomalous results.","Control changes."]},{"heading":"Use of the findings","paragraphs":["A proportionate method is available for multilingual literacy assessment. The method for the issue is to prepare a comparability table before analysing results. Record common elements, material differences, breaks in series and the direction in which each limitation may affect the conclusion; do not rank systems where those limitations remain material. Adverse cases and unresolved contradictions should be retained because they may reveal limitations concealed by an average result.","Publication of findings on the analysis should distinguish observed values, estimates and interpretation.","Proportionality in relation to the analysis does not mean reduced protection for learners exposed to greater risk. A decision concerning the available evidence should recognise that reliability without validity produces consistent but potentially irrelevant results. Validity without adequate consistency may expose learners to unequal judgement. In reviewing the issue, association should not be presented as causation, and statistical significance should not be treated as evidence of educational importance without further analysis. The record for an exception should identify the reason, approving authority, period of operation and date for reconsideration.","Any indicator used in relation to the issue should distinguish description from causal explanation. Interpretation should retain uncertainty, distributional differences and limits on generalisation."]}],"word_count":718,"content_hash":"sha256-56bd99ed083eae519d437ad033615f6aa72909f4f0ed9c86c1048c3d6dd7cde1","seo_keywords":["cross-system comparability in multilingual literacy assessment","international education data research","education quality","ICEQC"],"schema_type":"AnalysisNewsArticle"}