{"schema_version":"ICEQC_NEWS_ARTICLE_V1","id":"iceqc-news-e193f86924d05176","slug":"2017-07-26-public-performance-reporting-definitions-and-comparability","language":"en","publication_status":"READY_FOR_IMPORT","publication_date":"2017-07-26","last_modified_date":"2017-07-26","title":"Public performance reporting: definitions and comparability","summary":"Examines public performance reporting: definitions and comparability, defining the evidence base, coverage and limits that should govern comparison and use of the findings.","category":{"code":"DATA_RESEARCH","label":"Data Research"},"article_type":"Data and research analysis","publisher":"International Council for Education Quality Certification (ICEQC)","jurisdictional_scope":"International","historical_reference_basis":"2017 accountability agenda","reference_authority":"Relevant public authorities and official international sources","sections":[{"heading":null,"paragraphs":["The 2017 accountability agenda provides the immediate context for public performance reporting. For the available evidence, the available evidence should be interpreted with close attention to definitions, population coverage, collection methods and the limits of comparison.","The governing expectation for the available evidence should be capable of consistent application. A decision concerning the available evidence should recognise that reported averages should be accompanied by sufficient distributional information to identify material differences between learner groups, locations and forms of provision. In relation to public performance reporting, terms governing eligibility, support, assessment, reporting or review should prevent materially different treatment without recorded justification."]},{"heading":"Evidence base for public performance reporting","paragraphs":["The stated reference is the 2017 accountability agenda. Application to public performance reporting depends on evidence from the relevant jurisdiction or institution. Authorities and providers should distinguish established fact, policy expectation and matters left to institutional judgement. That distinction should remain visible in the decision record, public reporting and later review.","The quality significance of the available evidence follows from a basic distinction between availability and effective provision. In reviewing the analysis, education indicators should support decisions by describing outcomes and variation with definitions and limitations that permit responsible interpretation."],"bullets":["Define the decision the indicator will inform.","Analyse missing information.","Report uncertainty and revisions before using it to determine a learner or provider outcome.","Avoid causal claims unsupported by the design.","Test comparability."]},{"heading":"Coverage and comparability","paragraphs":["The technical issue within public performance reporting concerns the basis on which a conclusion is reached. In reviewing the available evidence, comparison requires more than the use of a common label. Definitions, reference periods, population coverage, institutional boundaries and collection practices must be sufficiently aligned for the observed difference to have a stable meaning. The decision record should distinguish the scope supported by evidence from any scope that remains unresolved.","The principal risks in relation to the analysis are averages concealing distribution, incomplete coverage, small differences overstated, and data revisions not carried through to published conclusions. A weakness in one part of the control environment may obscure a related failure elsewhere. In relation to public performance reporting, review should follow the sequence of decisions and records rather than assess documents in isolation.","Readily available material should not define the enquiry if it cannot answer the relevant decision question. For the available evidence, the most relevant material is likely to include disaggregated results, revision and comparability records, triangulation with administrative and qualitative evidence, and coverage and missingness analysis."],"bullets":["Do the reference periods align?","Is the remaining difference educationally material?","Has a classification changed?","Are the populations defined on the same basis?","Are exclusions and missing records comparable?"]},{"heading":"Responsible interpretation","paragraphs":["For operational review of public performance reporting, authorities and providers should proceed in a defined sequence. A competent review of the comparison should prepare a comparability table before analysing results. Record common elements, material differences, breaks in series and the direction in which each limitation may affect the conclusion; do not rank systems where those limitations remain material.","Publication of findings on the analysis should distinguish observed values, estimates and interpretation.","Interpretation of the measure should avoid two errors: treating a formal commitment as proof of effect, and treating one adverse case as proof that every part of the system has failed. Oversight of the analysis should reflect the principle that measurement can reveal where outcomes differ; it does not by itself establish why they differ or which intervention will work. The analysis of the measure proceeds on the basis that missing or delayed information may be patterned rather than random.","The assurance record for the comparison should retain the date of the evidence, the source responsible for it, the scope examined and the version of any instrument or definition applied. Traceable source and version information allow genuine improvement to be distinguished from administrative revision. Revision should not remove an earlier conclusion from the record where reliance has occurred.","Public reporting on the comparison should distinguish established fact, analytical judgement and planned action. If definitions, coverage or evidence alter an earlier conclusion, the reason should be stated so that revision is not mistaken for changed performance.","No individual measure is sufficient to establish effective operation of the available evidence across the affected scope."]}],"word_count":693,"content_hash":"sha256-f8f47f38aac6a9752e1d943a24ee62c4248e9b8c47f8b624130106797e382423","seo_keywords":["public performance reporting","international education data research","education quality","ICEQC"],"schema_type":"AnalysisNewsArticle"}