The Scottish government has announced that Scotland will re-join the main international surveys which investigate pupils’ understanding of mathematics, science and reading. This is good news. The data from the surveys will cast light on the effectiveness of Scottish policy and professional practice in ways that have been missing since Scotland withdrew a decade and a half ago. But there is a problem: data from the surveys will not start to become available until 2027 at the earliest, after the next Scottish parliament election. The purpose of this blog is to suggest a pragmatic way in which the Scottish government could fill the gap.
The full titles of the surveys are the Trends in International Mathematics and Science Study (TIMSS) and Progress in International Reading Literacy Study (PIRLS). They have been run by the International Association for the Evaluation of Educational Achievement (IEA) since 1995 (TIMSS) and 2001 (PIRLS). They measure pupils’ attainment at ages 8-9 (Primary 5) for all three domains, and 12-13 (Secondary 2) for mathematics and science. As well as this information, they also gather data on pupils’ attitudes to their studies, on their home circumstances, and on their teachers’ methods (for example. whether homework is given, and whether teaching is in large or small groups). It is likely that, when Scotland re-joins, each survey will sample around 3-4,000 pupils from around 130 schools at each of the specified stages. Since Scottish withdrawal, the only international survey in which Scotland participates has been the study of 15-year-olds (Secondary 3) by the Programme for International Student Assessment (PISA).
The decision to re-join has been widely welcomed, and indeed came after much pressure from researchers and opposition political parties. Reform Scotland’s Commission on School Reform has been to the fore in that campaigning, arguing that these surveys provide a reliable means by which Scotland could learn from good educational practice in the more than 70 countries that now take part in them. The Commission has also recommended that Scotland resumes its previously annual surveys of attainment, which were abolished by the Scottish government after 2017. That was the final year of the annual Scottish Survey of Literacy and Numeracy, which itself had replaced the annual Scottish Survey of Achievement in 2011. Understanding current policies requires annual frequency. It also requires questionnaires and attainment tests that are tailored to the Scottish curriculum and pedagogy.
The delay in getting data from the international surveys is because they run on quite lengthy cycles. TIMSS has a four-year cycle, and there is currently a wave of it in the field. So the next one will not be till 2027, reporting probably in 2028. PIRLS is five-yearly, and the next scheduled wave is 2026 with reports in 2027. The problem is even worse than this. These first new waves will be able to measure changes since Scotland last took part (2007 for mathematics and science, 2006 for reading). But if an aspect of the purpose is to measure trends in attainment that might reflect on policies that are currently being implemented, the earliest date by which current change could be detected would be the wave after next – 2031 for each series, which is two Scottish parliamentary elections away, and a quarter of a century since Scotland last took part.
Here is a pragmatic proposal for what to do in the meantime, achieving two aims in one move – resurrecting an annual survey, and filling the gap until Scotland gets the first results of the international surveys. Scotland should operate what might be thought of as a shadow version of the international surveys, starting next spring and running annually. The TIMSS and PIRLS questionnaires for pupils’ attitudes and circumstances and for teachers’ practices are fully in the public domain, and so could be readily applied here. Access to the testing materials is more complex, because the IEA keeps confidential a portion of the test questions for repeating in each survey so as to measure trends over time. Nevertheless, there is a mechanism by which researchers can apply to the IEA for permission to use the questions that are not reserved in this way. It seems likely that researchers in the Scottish government would readily be given that permission as part of a process of applying to re-join. Because the test items and questionnaires are already used in England and Northern Ireland, there would be no need for special piloting in Scotland, because conditions throughout the UK are similar enough to ensure that questions which work in one part will work in another.
If these shadow surveys could be started in 2024, then trends would already have started to be available by the time of the next Scottish parliament elections. More to the point in a practical sense, the annual data could be used to evaluate the developing impact of the Scottish curriculum. The Scottish government could choose to extend the investigation beyond the IEA’s target ages to, for example, Primary 7 for all three domains, and Secondary 2 for reading. It could add questionnaire items to measure aspects of current policies, such as universal free meals, free laptops, and policies for pupils with additional support needs.
It would also be possible to use these annual surveys to calibrate the results of the National Standardised Assessments, which were introduced by the present government from 2017. These assessments are potentially valuable as aids to teachers, and have been designed to a high standard. It would be interesting to know how the standards which they represent compare with the international criteria of the surveys. It would also be informative to set the national assessments in the same kind of contexts of pupils’ homes and teachers’ practices as are recorded in the international surveys. This could all be done by electronically linking the survey results for each pupil in the sample to the results of the most recent national assessments. There are reliable mechanisms by which such linkage can be done in a way that protects the anonymity of pupils, teachers and schools.
Beyond 2026-27, if this scheme seems to be working well, the proposed annual surveys could continue to run in parallel to the periodic international surveys, thus giving a better-grounded explanation of Scotland’s development in a global context.
This proposal needs to be tested in debate, and I’m sure that I’ve failed to notice some complications that would need to be ironed out. But if something like the structure proposed here could be implemented, it would re-establish what used to be Scotland’s rich environment of educational statistics. It could be a welcome long-term consequence of the government’s forward-looking decision to re-join the international surveys.
Lindsay Paterson is emeritus professor of education policy at Edinburgh University.