What do the PISA results tell us about Scottish education? Co-authored by Professor Mark Priestley and Dr Marina Shapira, University of Stirling
“the media hype about Scotland’s decline in maths and science is not especially warranted, as the evidence is actually pretty underwhelming.”
December saw the triennial publication of the results of the 2018 PISA survey, including the much awaited country rankings in reading, mathematics and science. In Scotland, these results have been much anticipated, following the ‘PISA shock’ of 2015. The 2018 results have shown a modest improvement in the reading score – from 493 in 2015 to 504, effectively bringing the country’s performance back to the levels recorded in 2006, 2009 and 2012 (but below earlier scores in 2000 and 2003). Scores in mathematics and science have remained stagnant since 2005, with marginal declines in performance (from 491-489 in Maths and from 497 to 490 in Science). Inevitably, these results were used to score political points in the immediate run-up to the election. Deputy First Minister and Education Cabinet Secretary John Swinney tweeted: “PISA has its limitations but Scotland’s performance in reading has risen sharply. Just 5 countries are now significantly higher than Scotland. The Scottish Attainment Programme started with an emphasis on literacy – the foundation of so much other learning. That is bearing fruit. (3 December 2019)”
18
The Scottish Educational Journal
The reaction to the results has been more negative in the media. Holyrood Magazine stated that, “Scotland’s score for reading improved in the latest PISA report, returning to a level similar to 2012 after a drop in 2015, but for maths and science there has been a decline in scores with each PISA survey since 2003 for maths and 2006 for science”. According to the Sun, “Pupils are now performing slightly worse than they did before she [Nicola Sturgeon] started improving our school system. And the figures are clear that, despite the up-tick in reading, performance in maths and science has continued to fall.” The Times has been similarly critical, stating that “performance levels in science and maths slipped to a record low in the Pisa test.” So who is right? What do the PISA results tell us about Scotland? Is there really evidence of decline in standards, and can this be attributed to Curriculum for Excellence and the SNP, as many are claiming? Prior to discussing the results it is important to note that the PISA study is based on a sample[1] and as such the measures it produces have
sampling errors and therefore cannot be automatically generalised towards the entire population of 15 year old students in Scotland. Therefore, prior to describing a change in a PISA score in 2018 (compared to the previous years) as an ‘increase’ or a ‘decrease’, it should be first checked whether the change is statistically significant. ‘Statistical significance’ means that measures estimated from a sample can be generalised for the entire research population. We can never know a true population parameter unless the measurement is based on entire population. Yet, we can estimate the risk of making a mistake if we use the sample estimate. Usually 5% is deemed to be an acceptable level of risk. This means that we can be 95% confident that the estimates are true for the population. Thus, the alleged ‘decrease’ in Maths and Sciences attainment in Scotland compared with 2015 is not statistically significant. In other words, the difference in numbers falls within the margin of error in this sort of survey, and the best that can be said is that there is no change between 2015 and 2018 in these subjects. Moreover, in international comparative terms, these performances
