No, Mr Gove, there is still no evidence of academy overperformance

Henry Stewart's picture
 0
You have to admire the chutzpah of the Secretary of State. First he warns MPs not to be "selective" in their use of evidence around academies, and in the very next breath makes very selective use of statistics on academy GCSE results:

"In sponsored academies, open for three years ... the proportion of pupils who achieve five good GCSEs including English and maths has increased by an average of 12.1 percentage points. Over the same time, results in all state-funded schools have gone up, which is good, but only by 5.1 percentage points." (Hansard, col 953)

Very selective use of facts



Note how Gove selects his facts. For sponsored academies he never compares absolute results with other schools overall (as the latter outperform sponsored academies by 61% to 53%). For converter academies he only uses absolute results and never compares growth (as GCSE results for converter academies fell in 2012). And he only ever compares sponsored academies with all schools, never with schools similar to them.

Sponsored academies were made into academies because of their previous low results. To judge whether the academies experiment has been a success, in terms of GCSE results, they must be compared to other schools with similar low results - whose results grow at a far faster rate than schools with previously high results.

No evidence of academies performing better than other schools



LSN analysis has previously shown that - when similar schools are compared - non academies did as well as similar academies in 2011-12; that the same was true for 2010-2011 and non-academies did slightly better over 2008-2011. This time Gove's figures appear to be for growth in the GCSE benchmark (5 GCSEs including English and Maths) from 2009 to 2012.

The chart below bands schools according to their GCSE benchmark results in 2009, the beginning of this period. Thus academies are compared with schools overall that are similar in that they started with a similar level of GCSE results.

For those schools in the 0-20% band in 2009, academy results grew by 18%, while the average growth for all schools was 20% - though numbers were small (21 schools). In the 20%-40% band academies did do slightly better, growing 17% to an overall 15%. The gap was larger (13% to 8%) for the 40%-60% band but again numbers were small - just 24 academies.

image

Academies far more likely to use GCSE equivalents



So the gap is nowhere near what Gove described but growth in academy results appears a little higher in these two bands. Growth can result from several things: it can come from better performance by students, can come from a more capable student intake* or can come from, as the DfE described it, by "inflating results" through the use of GCSE equivalents.

It was the Daily Telegraph that first pointed out that academies were far more likely to be "gaming" the system in this way, with an average 12% of academy GCSE results being down to GCSE equivalents, compared to 6% in maintained schools. So what proportion of the figures achieved above are down to equivalents. For the 20%-40% range, academy results fall by 16% compared to 13% in non academies, for results without equivalents. (Most equivalents will no longer be counted towards the GCSE benchmark from 2014.)

image

For the 40%-60% band, the gap is bigger. For academies in this band, 14% of the results are down to equivalents compared to just 6% for non academies.

Conclusion: no evidence of academy overperformance



Michael Gove, showing highly selective use of data, claimed that academy GCSE growth outperformed schools overall by 12% to 5%. When compared to similar schools this gap falls (to 2% in the 20-40% band and 5% in the 40-60% band). In both cases this can be entirely explained by the greater use of GCSE equivalents.

If the effect of GCSE equivalents is removed then academies figures fall, to a point where they appear to have done slightly less well than schools overall. The overperformance, that Michael Gove sought to demonstrate, disappears.

 

 

* An example of securing a more capable intake can be seen at Mossbourne, which changed its admission criteria in 2012. Admission, by lottery, is now based on capability bands reflecting national, not local Hackney, abilities. This means that Mossbourne now ensures its intake is at the national average, and above the Hackney average. However securing a more capable intake is normally only possible once results are strong and a strong reputation has been achieved.

 
Share on Twitter

Add new comment

Already a member? Click here to log in before you comment. Or register with us.