Taking the PISA – or how dodgy data was used to support breakneck education reform
Nearly five years ago my Christmas shopping was interrupted by a Daily Mail headline: Travesty of our Stagnating Schools. This ‘damning indictment’ of British education under Labour contained a graph showing plummeting PISA scores between 2000 and 2009. I’d never heard of PISA – until that day I thought it was a leaning tower. So I started to dig. I found PISA stood for Programme for International Student Assessment administered by the Organisation for Economic Cooperation and Development. I found PISA results for 2009 but not those for 2000. I also discovered the UK Briefing Paper accompanying the 2009 results. And it contained a prominent warning saying the 2000 and 2003 UK results hadn’t met response standards and shouldn’t be used for comparison with 2009. I also found a DfE research brief dated December 2010 warning the data wasn’t comparable. But many papers were comparing PISA 2000 with 2009. Was it deliberate deception? Or an honest mistake? And where had the graph come from? I dug some more. And I found the source. It was a press release from the Department of Education. There was the graph. There were quotes from the Briefing Paper. How was it possible they missed the warning on top of the first page? And there was a comment from Education Secretary Michael Gove: ‘Today’s PISA report underlines the urgent need to reform our school system.’ But they did no such thing. The comparison was false. I contacted FullFact. They covered the story and wrote to the DfE for an explanation. I found this site and wrote my first article. I wrote to my MP. FullFact and I eventually received an flimsy explanation early in 2011. The DfE had used a report from Southampton University published in 2006 which suggested the bias in 2000 made little difference to the results. But the Southampton report contained a caveat: their research was ‘neither the last word’ in analysing the response pattern to PISA 2000 or 2003. Nor was it ‘comprehensive in its analysis of actual or potential adjustments to the data for response bias’. And an OECD spokesperson told FullFact ‘the Southampton report doesn’t directly address the issue of comparability between 2000 and 2006.” The response bias wasn't the main reason the PISA Technical Advisory Group cautioned against long-term comparisons in the data, FullFact wrote. So Gove, other Tory politicians and much of the media kept on repeating the dodgy data. In October 2012, two years later, the UK Statistics Authority expressed concern about DfE use of these figures. Most politicians and most papers got the message. But a few continued to use the zombie statistics. Incredibly, Education Secretary Nicky Morgan repeated the discredited figures in her CBI speech this month. It could be argued it doesn’t matter what politicians say, it’s what they do that counts. But this deception underpinned all education reform of the last few years. It was the theory behind the practice. It was the justification for the breakneck speed of reform. So it matters. It’s been suggested in comments on another thread that Gove might just have made an error. But I don’t buy that. Problems with the 2000 data were known in 2006 – Southampton University’s report discussed them. In 2007, Channel 4 Factcheck criticised Gove, then shadow Education Secretary, for quoting the flawed data in the Commons: ‘In 2000 too few schools took part in the survey, and in 2003, too few students took part... Result: UK data from before 2006 aren't reliable, and can't be accurately compared.’ Factcheck gave Gove’s Commons statement a score of 4/5 where 5 indicates no basis in truth. It concluded: ‘So a man of Gove's legendary intelligence really has no excuse for trotting out these obviously misleading stats one more time.’ But the trotting didn’t stop – it speeded up into a gallop. A constant repetition of the fiction that the UK was ‘plummeting’ down league tables. Based on a false comparison. A deception. A myth – one that’s been blasted. Somebody tell Nicky Morgan. More myths are exposed in our newly-published LSN book The Truth About Our Schools: Exploding the Myths, Exploring the Evidence. It can be ordered here. More details here. UPDATE 30 November 2015, 11.38am From Commons Library Blog, FAQs re PISA tests written by Paul Bolton, author of the House Of Commons' 'Statistical literacy guide: How to spot spin and inappropriate use of statistics' 'So are we getting better or worse [in PISA tests]? Neither. Performance in all three subjects was broadly similar in 2006, 2009 and 2012.' 'I heard that our performance had fallen since 2000. Is that true? The UK’s 2000 and 2003 samples did not meet minimum response-rates, “…so data from the United Kingdom cannot be used for comparisons.”' It appears that neither Nicky Morgan nor Nick Gibb have read this item from the Commons Library. Perhaps they should also read Bolton's guide to the inappropriate use of statistics.