The DfE Question: Which Schools "Artificially Inflate" Their Results?
The DfE
responded this week to the analysis on this web site on the GCSE results in academies and other maintained schools. My
post in response pointed out that any greater growth in GCSE results could probably be explained by GCSE equivalents, the use of which is heavily criticised by the Education Minister.
(What are GCSE Equivalents: See footnote)
In a subsequent exchange on twitter Sam Freedman (@Samfr), policy adviser at the Department for Education, argued that the "problem with stripping out equivalences is that some are good (and will still be equivalent post-14) even if many aren't." He continued "of course from 2014 we'll see whether schools academy or maintained have been using equivalents to artificially inflate."
The good news is that we don't need to wait until 2014 to find which schools are "artificially inflating" their results, as the DfE has published
school-level data on what each school's 2011 GCSE results would look like if the 2015 reforms had been applied.
How Academy & Non-Academy Results Are Affected by 2015 Reforms
Academies: 2011 figure for 5 GCSEs including English and Maths, according to this table, is 47.0%. After applying the 2015 reforms this falls to 35.6%, a reduction of 11.4%.
Other maintained schools: 2011 figure: 59.3%. After 2015 reforms this falls to 54.5%, a reduction of 4.8%.
For some schools the difference is far more dramatic. Let's take one specific example quoted by Michael Gove in his
Spectator speech:
- South Norwood Academy: Gove quoted 100% achieving 5 GCSEs, the DfE table gives 75% for 5 GCSEs including Englisha nd Maths with equivalents, but only 49% on the 2015 basis.
The picture is clear. Using the DfE analysis and the DfE term, overall it is academies that have been artificially inflating their results, using equivalents seen by the DfE as invalid to achieve an extra 6.6% gain (11.4% less 4.8%) over non-academies.
The DfE
this week published long-term analysis of growth in results from 05/06 to 10/11. The DfE claims that long-standing academy results grew by 27.7% over this period compared to 21.3% for a group of similar schools (p 18). The statistical significance is questionable, as it is based on just 33 academies and also no list of who the similar schools are is available for checking.
But how do we create a fair comparison that takes account of the greater use by academies of equivalents? If we simply strip out the equivalents that the DfE regard as invalid, and compare these net figures to the 05/06 ones we get growth of 16.3% in academies and 16.5% in similar schools.
This is an approximation as we don't know the use of equivalents in these specific schools (though the DfE will have that data), or have any data for use of equivalents in 2005/06. The figures we have are for use of equivalents across all academies and non-academies in 2011. But if academies, on average, inflate their figures by 6.6% more than non-academies it seems fair to reduce academy results by 6.6% to enable a fair comparison. And then the results, effectively on core GCSEs, are pretty much equal.
Footnote: What are GCSE equivalents?
Some other qualifications, such as Btecs or diplomas, are currently given a GCSE equivalence for school results. Thus the Btec ICT qualification is currently equivalent to 4 GCSEs. A student who achieved Maths and English GCSEs and Btec ICT would be included in those achieving 5 GCSEs including Maths & English. This was heavily criticised by Michael Gove in his Spectator speech and, while Btec ICT may be an appropriate exam for some students, it is hard to argue that it is equivalent to 4 GCSEs.
School GCSE results have improved considerably in recent years, particularly for those schools with previous low results. A large part of that improvement is due to genuine improvement in teaching and learning, and real achievements from students. However it has been argued that some schools, facing pressure to increase results, have chosen to 'game' the system and put students in for equivalent qualifications that have little benefit to the students but improve the key 5 GCSE (with English and Maths) figure. As a result the DfE has announced that, from 2014, no qualification will be equivalent to more than one GCSE and, form 2015, many qualifications will no longer have a GCSE equivalance - as explained
here.
What the analysis in this article reveals is that if gaming the system is taking place, it is far more prevalent among academies than among other schools.
Data Note: The DfE spread-sheet on 2015-style results,
available here (headed E
quality Impact assessment annex C), has one oddity. The first column of % figures is stored as text and so calculations based on it lead to error messages. To use the spread-sheet you need to create a new column, and use the =VALUE() function to convert these text %s to numbers.h2
Comments
Mossbourne: 82%
Average number of exam entries per pupil: 11.9
Average number of GCSE entries per pupil: 8.2
Burlington Danes: 75%
Average number of exam entries per pupil: 10.2
Average number of GCSE entries per pupil: 8.
Walworth Academy: 69%
Average number of exam entries per pupil: 10.8
Average number of GCSE entries per pupil: 6
Perry Beeches (community school), Birmingham: 75%
Average number of exam entries per pupil: 13.1
Average number of GCSE entries per pupil: 6
Arthur Terry (community school), Sutton Coldfield: 70%
Average number of exam entries per pupil: 12.6
Average number of GCSE entries per pupil: 10.6
This school has only 10% low attainers.
Ormiston Victory Academy, Norwich: 64%
Average number of exam entries per pupil: 12.1
Average number of GCSE entries per pupil: 7.6
Paddington Academy, London: 69%
Average number of exam entries per pupil: 12
Average number of GCSE entries per pupil: 7
Outwood Grange Academy, Wakefield: 77%
Average number of exam entries per pupil: 15.6
Average number of GCSE entries per pupil: 6
Of course, there are many secondary schools who use vocational exams - and it's not always done to boost scores but to offer the most suitable exam for the pupil. However, the Government has rubbished vocational exams. It follows, therefore, that it should not boast about exam pass rates in schools where the percentage includes the same equivalent exams that the Government condemns.
Do you know whether equivalents deemed to be 'worth' two or four GCSEs are represented in the figures on the performance tables according to their 'worth'?
For instance, if the average number of entries were 12 and the average GCSEs 8, does that mean the average person is taking four different vocational courses, or might it mean they are taking one course worth 4 GCSEs or two courses worth 2?
However, the Government has rubbished vocational exams.
Huh????
Vocational qualifications are hugely important, and will continue to be so – so we are working hard to ensure that they are just as ambitious and useful as academic courses.Michael Gove
one of the unhappy trends which actually grew in force over the past 13 years was the assumption that the purely academic route was really always the preferred one – and unless you’d secured a place on leaving school to study at university for three years you were somehow a failure. These assumptions undermine social cohesiveness because, in a big society, unless each feels valued and all feel valued, then the conferral of value is imperfect. And they also limit opportunity.Michael Gove
if we are to ensure more and more students are capable of benefitting from a growth in apprenticeship numbers we have to take action to improve vocational education before people leave school. We have to have courses, qualifications and institutions during the period of compulsory schooling which appeal to those whose aptitudes and ambitions incline them towards practical and technical learning.Michael Gove
it’s crucial to note that securing this core base of knowledge would not preclude the study of technical or vocational subjects as some have suggested. It’s not either/or but both/and. I’m absolutely clear that every child should have the option of beginning study for a craft or trade from the age of 14 ...Michael Gove
The government has fostered University Technical Colleges and Studio Schools and although useless qualifications have been dropped, 150 different vocational courses remain recognized for performance tables purposes.
Have any of those words created anything real on the ground?
References would be nice by the way - you can start a new post for each one to avoid the spamometer.
Since it takes a minimum of 18 months and possibly years to sack a teacher and only about three have been sacked since the dawn of time, I'm pretty sure the guy working down the pet shop is more likely to be Elvis than a victim of a non-existent EBacc purge.
I thnk the dread is very real. I know many teachers in local schools who are fearing the worst and some who have already been told that because they teach "vocational" and life skills subjects, their courses will be phased out, rendering their position redundant.
But to get back to Henry's original point - it seems rather hypocritical of Gove to want to obliterate these subjects when it suits his orations about "rigour" at the same time as keeping them in as equivalents to bolster Academy success. He really should be publishing data which strips out the equivalents he doesn't like to prove that Academies are outperforming other schools but as Henry has shown, he can't do this because Academies aren't outperforming and that must be embarrassing for him and his political ambitions.
"For a decade now we have steered hundreds of thousands of young people towards courses and qualifications which are called vocational even though employers don't rate them and which have been judged to be equivalent in league tables to one - or sometimes more – GCSEs, even though no-one really imagines they were in any way equivalent."
"Whether they were called Level 2 Btecs or Diplomas, these qualifications and courses lacked rigour, they were not externally assessed, they did not provide a route onto other qualifications, they did not confer skills which employers valued and they were overwhelmingly taught to those students marked down at an early age as under-achievers."
But he's quite happy to accept these vocational exams when they appear as equivalents in the schools he wants to praise.
There has never been a problem getting rid of teachers Ricky - there are many ways. Especially is you're actually prepared to pay them their redundancy pay if their services are no longer required. Heaven forbid anyone should do that though.
Were you there in the 80s when teaching massively demanned? I suspect not.
This stuff about it being hard to get rid of teachers is just another of Gove's pipe dreams. The reason it takes a while is because in many cases it's appropriate to go through the steps involved in turning the situation round - because very often the situation is turned around and you can't tell which situation is which until you go through the process. Often situations of apparent failure are caused by short term issues in the life of the teacher which they need some extra support get through or by failures in the schools system beyond the teachers' control which need to be addressed. It doesn't take 18 months to sack a teacher who is proven to be failing in a way which can't be addressed. It takes 18 months from there being a concern about the teacher to see if that concern is serious and can't be rectified or not. And in most cases it can be rectified. Often during this time the teacher will accept major changes, loss of responsibility, going part time or whatever and will then prove they can do an excellent job again under less stressful conditions.
There is no hypocrisy here at all. No one blames schools for making rational choices to achieve the performance targets they are set. If you want to blame someone, blame those who set up a system of targets that offered perverse incentives - i.e. the last Labour government.
It would be grossly unfair to judge schools against criteria imposed ex post facto rather than against the performance indicators in place. That's why EBacc stats were deployed as a 'nudge' rather than as a stick to beat schools with.
The schools that offered these inferior vocational courses did so in good faith (they were accredited courses) and were not on constructive notice of how useless they were until the the Wolf Report told them so.
Gove has kept something like 150 different vocational courses, while scrapping those that were no good.
Harris Academy, Peckham 50%
Average number of exam entries per pupil: 10.1
Average number of GCSE entries per pupil: 4.6
(Note: intake skewed towards bottom end: only 7% high attainers, 37% low)
Harris Academy, Bermondsey 64%
Average number of exam entries per pupil: 13.2
Average number of GCSE entries per pupil: 7.3
Harris Academy, Falconwood 57%
Average number of exam entries per pupil: 15.9
Average number of GCSE entries per pupil: 6.5
Harris Academy, Merton 75%
Average number of exam entries per pupil: 14.8
Average number of GCSE entries per pupil: 8.1
Harris Academy, Purley 61%
Average number of exam entries per pupil: 12.3
Average number of GCSE entries per pupil: 7
Harris Academy, Crystal Palace 95%
Average number of exam entries per pupil: 13.2
Average number of GCSE entries per pupil: 10.2
(Note: intake skewed towards top end: 56% high attainers, only 4% low. Only 11% FSM)
Harris Academy, South Norwood 75%
Average number of exam entries per pupil: 13.9
Average number of GCSE entries per pupil: 6
Harris Girls’ Academy, East Dulwich 67%
Average number of exam entries per pupil: 14.2
Average number of GCSE entries per pupil: 6.2
*Data from DfE School Performance Tables 2011. Percentage reaching the benchmark 5+ GCSEs A*-C (including Maths and English) is given after the name of the school.
ARK Globe 45%
Average number of exam entries per pupil: 11.1
Average number of GCSE entries per pupil: 5.6
(10% high attainers, 42% low)
Charter Academy 39%
Average number of exam entries per pupil: 9.1
Average number of GCSE entries per pupil: 4.4
(9% high attainers, 41% low)
Walworth Academy 69%
Average number of exam entries per pupil: 10.8
Average number of GCSE entries per pupil: 6.0
(10% high attainers, 47% low)
Have you had any joy finding out the answer to my question at 28/06/12 at 1:16 ?
Skegness Academy 45% achieved 5+ A*-C including Maths and English
Average number of exam entries per pupil: 12
Average number of GCSE entries per pupil: 2
http://www.greenwoodacademies.org/images/ssat.pdf
Totally agree with your post. I expect you know John O'Farrell's book May Contain Nuts, which was about a woman who coduln't rely on her child to pass entrance exams, so sat them for her. It's only a mild exaggeration of the grades more important than integrity' mentality you mention. I know course work helps those children who are not brilliant in exams; but if the Gov are so keen to look at other European models for school systems, maybe they'll look at their assesment techniques too. That mix seems a much fairer way and, as you dare to say, more fun.
Add new comment