Stories + Views
Statistics watchdog expresses concern about DfE use of the PISA 2000 figures
After a campaign lasting nearly two years, FullFact has seen a letter from Andrew Dilnot, Chair of the UK Statistics Authority, which gives concerns about the use by the Department for Education (DfE) of the flawed figures from the Programme for International Student Assessment (PISA) tests for the UK in 2000.
In December 2010, FullFact published its misgivings about a DfE press release timed to coincide with the publication of the PISA 2009 test results. This press release was widely churned in the media including by the BBC, the Mail and in a Telegraph blog. FullFact pointed out at the time that the Organisation for Economic Cooperation and Development (OECD), which administers the triennial PISA tests, had warned that the 2000 PISA results for the UK were flawed and should not be used for comparison.
Despite this warning the DfE defended its use of the figures and FullFact asked the UK Statistics Authority to intervene. This request was followed by a letter from David Miliband, Schools Minister from June 2002 to December 2004, expressing his concern about the use of the flawed figures by Sir Michael Wilshaw, Chief Inspector of Schools, in September.
Andrew Dilnot, Chair of the UK Statistics Authority, in his reply to Miliband (downloadable here), expresses concern that OECD warnings were not highlighted when trend comparisons using the 2000 data were published. He singled out the DfE press release of December 2010 for detailed criticism:
“I was concerned to review the Department for Education’s press release of 7 December 2010 in which headline results for England from the PISA study, alongside relative international rankings, were not accompanied by detailed advice or caveats to help the reader in making comparisons over time, nor were the statistical implications of an increase in the number of reporting countries in later PISA studies noted.”
Dilnot said that readers might misunderstand the trend comparisons if they were presented without these warnings. He noted that the PISA data was contradicted by other evidence, including the Trends in International Mathematics and Science Study (TIMSS), and concluded that ‘it may be difficult to treat an apparent decline in secondary school pupils’ performance as “a statistically robust result”’.
FullFact points out that it isn’t ‘necessarily wrong’ to compare the 2000 PISA results with those of 2009, but any comparison should be accompanied by the OECD caveat. Dilnot wrote:
“These uncertainties and weaknesses are not just a technical footnote; they are themselves an important part of the evidence, and affect interpretation and meaning. League tables and the presentation of international rankings can be statistically problematic, and require clear and careful commentary alongside them.”
“Clear and careful commentary” has been missing in the trend comparisons which have been used repeatedly by Secretary of State for Education, Michael Gove, Parliamentary Under-Secretary of State for Schools, Lord Hill, and ex-schools minister, Nick Gibb, to justify the Government’s education policies. At the same time, contradictory evidence, such as TIMSS, has been ignored.
“Clear and careful commentary” has been missing from innumerable media articles which used these trend comparisons despite many commentators, including FullFact and this site, warning that these comparisons are based on flawed data. Instead, these comparisons have been used to paint a bleak picture of UK state education – “plummeting down international league tables” is now accepted as “truth”.
FullFact noticed that the DfE has not used these trend comparisons since the Statistics Authority has been looking into the case. However, this does not mean that these comparisons will not be made again in the future without the necessary warnings. FullFact says it will be vigilant in spotting any “slips”.