The DfE Question: Which Schools "Artificially Inflate" Their Results?
The DfE responded this week
to the analysis on this web site on the GCSE results in academies and other maintained schools. My post in response
pointed out that any greater growth in GCSE results could probably be explained by GCSE equivalents, the use of which is heavily criticised by the Education Minister.
(What are GCSE Equivalents: See footnote)
In a subsequent exchange on twitter Sam Freedman (@Samfr), policy adviser at the Department for Education, argued that the "problem with stripping out equivalences is that some are good (and will still be equivalent post-14) even if many aren't." He continued "of course from 2014 we'll see whether schools academy or maintained have been using equivalents to artificially inflate."
The good news is that we don't need to wait until 2014 to find which schools are "artificially inflating" their results, as the DfE has published school-level data
on what each school's 2011 GCSE results would look like if the 2015 reforms had been applied.
How Academy & Non-Academy Results Are Affected by 2015 Reforms
2011 figure for 5 GCSEs including English and Maths, according to this table, is 47.0%. After applying the 2015 reforms this falls to 35.6%, a reduction of 11.4%.
Other maintained schools:
2011 figure: 59.3%. After 2015 reforms this falls to 54.5%, a reduction of 4.8%.
For some schools the difference is far more dramatic. Let's take one specific example quoted by Michael Gove in his Spectator speech
- South Norwood Academy: Gove quoted 100% achieving 5 GCSEs, the DfE table gives 75% for 5 GCSEs including Englisha nd Maths with equivalents, but only 49% on the 2015 basis.
The picture is clear. Using the DfE analysis and the DfE term, overall it is academies that have been artificially inflating their results, using equivalents seen by the DfE as invalid to achieve an extra 6.6% gain (11.4% less 4.8%) over non-academies.
The DfE this week published
long-term analysis of growth in results from 05/06 to 10/11. The DfE claims that long-standing academy results grew by 27.7% over this period compared to 21.3% for a group of similar schools (p 18). The statistical significance is questionable, as it is based on just 33 academies and also no list of who the similar schools are is available for checking.
But how do we create a fair comparison that takes account of the greater use by academies of equivalents? If we simply strip out the equivalents that the DfE regard as invalid, and compare these net figures to the 05/06 ones we get growth of 16.3% in academies and 16.5% in similar schools.
This is an approximation as we don't know the use of equivalents in these specific schools (though the DfE will have that data), or have any data for use of equivalents in 2005/06. The figures we have are for use of equivalents across all academies and non-academies in 2011. But if academies, on average, inflate their figures by 6.6% more than non-academies it seems fair to reduce academy results by 6.6% to enable a fair comparison. And then the results, effectively on core GCSEs, are pretty much equal.
Footnote: What are GCSE equivalents?
Some other qualifications, such as Btecs or diplomas, are currently given a GCSE equivalence for school results. Thus the Btec ICT qualification is currently equivalent to 4 GCSEs. A student who achieved Maths and English GCSEs and Btec ICT would be included in those achieving 5 GCSEs including Maths & English. This was heavily criticised by Michael Gove in his Spectator speech and, while Btec ICT may be an appropriate exam for some students, it is hard to argue that it is equivalent to 4 GCSEs.
School GCSE results have improved considerably in recent years, particularly for those schools with previous low results. A large part of that improvement is due to genuine improvement in teaching and learning, and real achievements from students. However it has been argued that some schools, facing pressure to increase results, have chosen to 'game' the system and put students in for equivalent qualifications that have little benefit to the students but improve the key 5 GCSE (with English and Maths) figure. As a result the DfE has announced that, from 2014, no qualification will be equivalent to more than one GCSE and, form 2015, many qualifications will no longer have a GCSE equivalance - as explained here
What the analysis in this article reveals is that if gaming the system is taking place, it is far more prevalent among academies than among other schools.
The DfE spread-sheet on 2015-style results, available here
(headed Equality Impact assessment annex C
), has one oddity. The first column of % figures is stored as text and so calculations based on it lead to error messages. To use the spread-sheet you need to create a new column, and use the =VALUE() function to convert these text %s to numbers.h2