Data spinning minister ignores warning about comparing results over time

Janet Downs's picture

Free schools had ‘particular success’ in key stage 1 Assessments, says schools minister Nick Gibb in the Department for Education media blog.  They outperformed ‘other type of schools for the fifth year in a row’.

He doesn’t give actual figures but the statistics appear to back him up.   87% of free school pupils reached the expected standard in phonics in year one compared with 83% in converter academies and 82% in local authority maintained schools.   At the end of key stage one, 79% reached the expected standard in reading, writing and maths compared with 77% in converter academies and 76% in LA maintained schools.

Although the difference is small, it’s still an achievement, you might say.  That’s until we consider sample size.  There were just 188 free schools where pupils took the phonics test and 180 which assessed pupils at the end of KS1. 

A sample of 180+ schools is too small to come to any reliable conclusion about their performance as a group.  This is a tiny number is compared with 10,800+ LA schools.  Such comparisons, then, should be used with caution. 

Worse than ignoring the small sample size, Gibb appears to take no notice of advice on comparability over time for key stage 1 data on the statistical release.  It makes it clear that this year’s results are ‘not directly comparable’ to those of previous years.  This is because changes had been made both to frameworks and expected standards in the last three years.

It would therefore be incorrect and misleading to make direct comparisons showing changes over time,’ the DfE statisticians warn.

Yet Gibb’s boast implies free schools outperformed other schools in both phonics and the end of KS1 assessments for the last five years.  This implication is wrong.  This year’s KS1 assessment results can’t be compared with earlier years.

What, then, is the data showing free schools outperformed other schools for five years consecutively?   It can’t be KS1 figures for reasons given above.  It must year 1 phonics.  But five years ago, the number of free schools entering pupils for this test would have been even tinier than today.

Gibb has, of course, given the more favourable figures: phonics in year 1 not year 2.  By the end of year 2*, the proportion reaching the expected standard in free schools is only slightly higher (94%) than in converter academies (93%) and LA schools (92%).  Hardly any difference at all.

The schools minister has implied a comparison which is ‘incorrect and misleading’.   He has ignored the importance of sample size.  He has trumpeted free school phonics scores when the difference between them and other schools is negligible one year later.

Spinning data in this way makes ministers look either disingenuous or incompetent.

*Phonics table downloadable here

AMENDMENT: 28 September, 08.34: The words 'for the last five years' have been added to the sentence which previously said, 'Gibb’s boast implies free schools outperformed other schools in both phonics and the end of KS1 assessments' to make it clear I was not referring to Gibb's comparison of results for 2019 only but to his implication that free schools had outperformed other schools over five years.

Share on Twitter Share on Facebook

Be notified by email of each new post.


Roger Titcombe's picture
Fri, 27/09/2019 - 16:39

There is another factor. Free schools are very national-testing focussed, with results driving performance bonuses for executives and job threats to those lower in status. It is part of the business culture that drives these schools. This means that in such schools test results are even more high stakes than in LA schools. This is especially true in schools that adopt Gibb's favoured knowledge based instruction, strict discipline, approach where there is lots of bulls**t PR to be lived up to.

There is a general pattern throughout our test-obsessed system in which high stakes targets at each Key Stage tend to be met through extreme cramming/revision approaches. Even in the LA infant/junior/secondary system there is a tendency for SATs results to be inflated to the disadvange of the flawed progress measure of the schools to which the pupils transfer.

Within secondary, high social disadvantage, low performing 11-16 schools for whom GCSE results are very high stakes, get inflated GCSE results which is bad news for the KS5 schools that sign their pupils up for A level and even more seriously for FE Colleges that act as recruiting agents for nursing and midwifery degrees. This was bad enough in 2018 and is worse still this year where grade threshold marks have been even more depressed.

In my study of Perry Beeches school, which claimed astonishing GCSE results, my research showed that this did not translate into academic A  level uptake. It will be interesting to see the progression pattern of this year's Michaela School leavers (if we ever find out).

These factors render our school testing system, designed to drive the marketised education ideology, completely unfit for purpose, with many negative side effects including the creation of 'attainment gap' illusions. John Mountford and I have been researching the link between Free School Meals eligibility and SATs and CATs scores. CATs data have been hard to come by in the north of England as there are comparatively few schools that take them. However John was able to find a lot of data from schools along the generally prosperous M4 corridor. The relative affluence of school catchment areas makes little difference to the general pattern, which is that SATs scores are inflated compared to CATs and especially so in schools that have high proportions of FSM pupils. It was almost always the case that the CATs scores compared to SATs, for FSM pupils were lower (and often much lower) than for non-FSM pupils. This is discussed in my article refuting the claims made by Dr Rebecca Montacute of the Sutton Trust. Some of the SATs/CATs data, that we obtained from schools using Freedom of information, are set out in this article.  

Add new comment

Already a member? Click here to log in before you comment. Or register with us.