The latest edition of the Oxford Journal of Education has a study looking at reactions to Scotland’s declining results in the international PISA education survey. People in government, councils, and semi-official bodies were interviewed giving their take.
It’s worth looking at this in some detail, to get an insight into the response of official Scotland to PISA.
“And that reaction has been, with some notably exceptions, a collective shrug of the shoulders”
One argument, for example, is that PISA doesn’t match up with the aspirations of the Curriculum for Excellence to create broad-minded, well-formed young people.
Take a parent representative quoted arguing that ‘Scottish education values more than just what is measured in PISA such as being more compassionate, having better communication skills, and continuing to learn regularly.’
Or a union leader pondering if ‘we want our young people to pass these tests, or do we want them to engage with a wider, more socially just curriculum?.’
This is fascinating. Scotland used to have a real sense of exceptionalism founded on academic quality. Our highers were hard and our teachers were harder.
Now we have a different kind of exceptionalism, focused on values.
It’s tempting to point out that PISA is also pretty underwhelming for the soft stuff, too. Our pupils are less lonely at school than the international average, for example – but then the rate of children who are frequently bullied at school is roughly 20% higher than other countries. More compassionate, eh?
But that kind of misses the point. Why not have academic excellence and broad-minded citizens keen on social justice? Is it really a choice between the two?
You can see that point in the tests themselves. Far from being super-abstract, the PISA tests are geared towards real-world examples. In fact, PISA’s tests describe a minimum level of competence to take part in society – the very definition of social inclusion. In Scotland, about a third of our pupils don’t meet that standard for maths. In reading it’s one-fifth, and science, one-quarter. The first two of those numbers have declined since 2018.
So is there a choice between good PISA scores and a socially just curriculum? No. Only if you think taking part in society is not a matter of justice.
Surely no-one thinks that – and we should assume that all the people quoted in this paper are speaking in good faith. But the perceived issues with PISA feel a bit like people convincing themselves we’re somehow exempt from this kind of international survey.
One central government official, for example, argues that ‘some of the things appear in the PISA assessment are not taught in Scottish maths programme until children are older than 15.’
Who knows if this is true. It has the feel of anecdata. But you can plainly read this the other way round: if other countries’ children are getting to the hard bits before ours, isn’t that…a sign that our children are behind others?
Or listen to another parent organisation arguing that the ‘PISA data consist of a small sample of students. Data are also cross-sectional, which means different students are assessed in every wave’ and therefore cannot be trusted.
As the paper authors point out, this is just factually not true. That’s not how PISA works. It’s not how any social research works.
Or a teacher representative arguing that ‘if teachers in Scotland had 20 in their classes as other countries do, maybe the results would be different.’
Except PISA does have data about class sizes and resources. That also allows us to see some interesting stuff – such as the evidence about the impact of class sizes being really very mixed. It also shows a really interesting result: the UK does have larger class sizes at secondary (25 pupils vs 23 average). But PISA also asks headteachers if they think teacher shortages are undermining teaching. In Scotland, relatively fewer pupils are in schools where that is the case.
These comments build up a picture of not engaging with PISA. Or not wanting to. As if it just doesn’t apply: oh, you wouldn’t know our curriculum – it goes to a different school.
Let’s again, forgive that. People are busy. PISA methodology is hardly bedtime reading.
But it’s hard not to worry when it’s education specialists. Three central officials sent a combined response to the paper researchers, pointing to the Scottish Government’s official position which takes PISA seriously. Good for them. But look at this quote from another, about whether PISA could be used to measure CfE’s success:
“I don’t think it’s [PISA] a valid measure. And I would say the same of the Scottish Survey of Literacy and Numeracy. It’s exactly the same thing because of its no accountability at local level. People didn’t really understand what it was. […] They [teachers] are not accountable for it. They [students] won’t be told whether they passed or failed, the teachers won’t be told whether they passed or failed and then this data came out and you know press are telling us our education system’s rubbish. And it’s not. You know if you look at our national qualifications they are going up. [Central education official (1)]”
It’s hard to know where to start with this. The whole point of surveys like PISA and SSLN is that they are not national exams and are not an accountability mechanism. They are a standardised test, on a robust sample, with no pass or fail. Every sensible system treats this as a useful data point.
And even if you think international comparisons are somehow invalid, PISA also shows how a country has changed in its own right, and how different types of pupil within a country perform. I’m personally less vexed by the country rankings, than the fact Scottish students today do worse than those 20 years ago. Or that the gap between the least and most deprived pupils in Scotland has got bigger. And as Lindsay Paterson points out, it is very plausible indeed that it’s due to the Curriculum for Excellence.
This independence and ability to make a range of comparisons is what matters. Nobody serious thinks PISA is the only measure. But ‘not valid’ is a kind of denial.
And that’s probably the most troubling thing here. PISA shows outcomes. Those outcomes are not just measurements of schools but of school systems, including lots of ineffable things about expectations and factors outside the classroom. Some interviewees make this point. It’s a fair one. Culture matters.
But the irony is that saying ‘PISA is invalid because of culture’ is an amazingly acute expression of our own culture. It’s one of complacency. It’s one of looking inwards. It’s one of buying our own spin.
One final quote sums up this situation:
So culturally you know the only people were using PISA at that kind of level are politicians. […] And nobody, nobody actually implementing it really cares.
Nobody really cares? Speak for yourself.
Gordon Hector is a policy consultant and former Director of Policy and Strategy for the Scottish Conservatives