On DfE Criteria, Academies GCSE Results are "Artificially Inflated"

14
The DfE Question: Which Schools "Artificially Inflate" Their Results?

The DfE responded this week to the analysis on this web site on the GCSE results in academies and other maintained schools. My post in response pointed out that any greater growth in GCSE results could probably be explained by GCSE equivalents, the use of which is heavily criticised by the Education Minister.

(What are GCSE Equivalents: See footnote)

In a subsequent exchange on twitter Sam Freedman (@Samfr), policy adviser at the Department for Education, argued that the "problem with stripping out equivalences is that some are good (and will still be equivalent post-14) even if many aren't." He continued "of course from 2014 we'll see whether schools academy or maintained have been using equivalents to artificially inflate."

The good news is that we don't need to wait until 2014 to find which schools are "artificially inflating" their results, as the DfE has published school-level data on what each school's 2011 GCSE results would look like if the 2015 reforms had been applied.

Academies: 2011 figure for 5 GCSEs including English and Maths, according to this table, is 47.0%. After applying the 2015 reforms this falls to 35.6%, a reduction of 11.4%.

Other maintained schools: 2011 figure: 59.3%. After 2015 reforms this falls to 54.5%, a reduction of 4.8%.

For some schools the difference is far more dramatic. Let's take one specific example quoted by Michael Gove in his Spectator speech:

• South Norwood Academy: Gove quoted 100% achieving 5 GCSEs, the DfE table gives 75% for 5 GCSEs including Englisha nd Maths with equivalents, but only 49% on the 2015 basis.

The picture is clear. Using the DfE analysis and the DfE term, overall it is academies that have been artificially inflating their results, using equivalents seen by the DfE as invalid to achieve an extra 6.6% gain (11.4% less 4.8%) over non-academies.

The DfE this week published long-term analysis of growth in results from 05/06 to 10/11. The DfE claims that long-standing academy results grew by 27.7% over this period compared to 21.3% for a group of similar schools (p 18). The statistical significance is questionable, as it is based on just 33 academies and also no list of who the similar schools are is available for checking.

But how do we create a fair comparison that takes account of the greater use by academies of equivalents? If we simply strip out the equivalents that the DfE regard as invalid, and compare these net figures to the 05/06 ones we get growth of 16.3% in academies and 16.5% in similar schools.

This is an approximation as we don't know the use of equivalents in these specific schools (though the DfE will have that data), or have any data for use of equivalents in 2005/06. The figures we have are for use of equivalents across all academies and non-academies in 2011. But if academies, on average, inflate their figures by 6.6% more than non-academies it seems fair to reduce academy results by 6.6% to enable a fair comparison. And then the results, effectively on core GCSEs, are pretty much equal.

Footnote: What are GCSE equivalents?

Some other qualifications, such as Btecs or diplomas, are currently given a GCSE equivalence for school results. Thus the Btec ICT qualification is currently equivalent to 4 GCSEs. A student who achieved Maths and English GCSEs and Btec ICT would be included in those achieving 5 GCSEs including Maths & English. This was heavily criticised by Michael Gove in his Spectator speech and, while Btec ICT may be an appropriate exam for some students, it is hard to argue that it is equivalent to 4 GCSEs.

School GCSE results have improved considerably in recent years, particularly for those schools with previous low results. A large part of that improvement is due to genuine improvement in teaching and learning, and real achievements from students. However it has been argued that some schools, facing pressure to increase results, have chosen to 'game' the system and put students in for equivalent qualifications that have little benefit to the students but improve the key 5 GCSE (with English and Maths) figure. As a result the DfE has announced that, from 2014, no qualification will be equivalent to more than one GCSE and, form 2015, many qualifications will  no longer have a GCSE equivalance - as explained here.

What the analysis in this article reveals is that if gaming the system is taking place, it is far more prevalent among academies than among other schools.

Data Note: The DfE spread-sheet on 2015-style results, available here (headed Equality Impact assessment annex C), has one oddity. The first column of % figures is stored as text and so  calculations based on it lead to error messages. To use the spread-sheet you need to create a new column, and use the =VALUE() function to convert these text %s to numbers.h2
Category:

Be notified by email of each new post.

Thu, 28/06/2012 - 10:44
In his Spectator speech, Gove praised the results of several named schools for having a good GCSE pass rate. But most of these entered pupils for a large number of equivalents. I give the details below (the percentage of pupils reaching the benchmark 5 A*-C including Maths and English appears first).

Mossbourne: 82%
Average number of exam entries per pupil: 11.9
Average number of GCSE entries per pupil: 8.2

Burlington Danes: 75%
Average number of exam entries per pupil: 10.2
Average number of GCSE entries per pupil: 8.

Average number of exam entries per pupil: 10.8
Average number of GCSE entries per pupil: 6

Perry Beeches (community school), Birmingham: 75%
Average number of exam entries per pupil: 13.1
Average number of GCSE entries per pupil: 6

Arthur Terry (community school), Sutton Coldfield: 70%
Average number of exam entries per pupil: 12.6
Average number of GCSE entries per pupil: 10.6
This school has only 10% low attainers.

Average number of exam entries per pupil: 12.1
Average number of GCSE entries per pupil: 7.6

Average number of exam entries per pupil: 12
Average number of GCSE entries per pupil: 7

Average number of exam entries per pupil: 15.6
Average number of GCSE entries per pupil: 6

Of course, there are many secondary schools who use vocational exams - and it's not always done to boost scores but to offer the most suitable exam for the pupil. However, the Government has rubbished vocational exams. It follows, therefore, that it should not boast about exam pass rates in schools where the percentage includes the same equivalent exams that the Government condemns.

Thu, 28/06/2012 - 13:16
Janet

Do you know whether equivalents deemed to be 'worth' two or four GCSEs are represented in the figures on the performance tables according to their 'worth'?

For instance, if the average number of entries were 12 and the average GCSEs 8, does that mean the average person is taking four different vocational courses, or might it mean they are taking one course worth 4 GCSEs or two courses worth 2?

Thu, 28/06/2012 - 15:12
Janet

However, the Government has rubbished vocational exams.

Huh????

Vocational qualifications are hugely important, and will continue to be so – so we are working hard to ensure that they are just as ambitious and useful as academic courses.Michael Gove

one of the unhappy trends which actually grew in force over the past 13 years was the assumption that the purely academic route was really always the preferred one – and unless you’d secured a place on leaving school to study at university for three years you were somehow a failure. These assumptions undermine social cohesiveness because, in a big society, unless each feels valued and all feel valued, then the conferral of value is imperfect. And they also limit opportunity.Michael Gove

if we are to ensure more and more students are capable of benefitting from a growth in apprenticeship numbers we have to take action to improve vocational education before people leave school. We have to have courses, qualifications and institutions during the period of compulsory schooling which appeal to those whose aptitudes and ambitions incline them towards practical and technical learning.Michael Gove

it’s crucial to note that securing this core base of knowledge would not preclude the study of technical or vocational subjects as some have suggested. It’s not either/or but both/and. I’m absolutely clear that every child should have the option of beginning study for a craft or trade from the age of 14 ...Michael Gove

The government has fostered University Technical Colleges and Studio Schools and although useless qualifications have been dropped, 150 different vocational courses remain recognized for performance tables purposes.

Thu, 28/06/2012 - 17:55
Meanwhile I chat to the teachers of vocational subjects who are now working in our pet shop having been sacked and replaced with french and geography teachers.

Have any of those words created anything real on the ground?

References would be nice by the way - you can start a new post for each one to avoid the spamometer.

Thu, 28/06/2012 - 18:14
Since it takes a minimum of 18 months and possibly years to sack a teacher and only about three have been sacked since the dawn of time, I'm pretty sure the guy working down the pet shop is more likely to be Elvis than a victim of a non-existent EBacc purge.

Thu, 28/06/2012 - 18:40
Rebecca -

I thnk the dread is very real. I know many teachers in local schools who are fearing the worst and some who have already been told that because they teach "vocational" and life skills subjects, their courses will be phased out, rendering their position redundant.

But to get back to Henry's original point - it seems rather hypocritical of Gove to want to obliterate these subjects when it suits his orations about "rigour" at the same time as keeping them in as equivalents to bolster Academy success. He really should be publishing data which strips out the equivalents he doesn't like to prove that Academies are outperforming other schools but as Henry has shown, he can't do this because Academies aren't outperforming and that must be embarrassing for him and his political ambitions.

Fri, 29/06/2012 - 07:49
Allan - Gove's own words reveal his hypocrisy. In the Spectator's speech he criticised vocational courses offered in schools:

"For a decade now we have steered hundreds of thousands of young people towards courses and qualifications which are called vocational even though employers don't rate them and which have been judged to be equivalent in league tables to one - or sometimes more – GCSEs, even though no-one really imagines they were in any way equivalent."

"Whether they were called Level 2 Btecs or Diplomas, these qualifications and courses lacked rigour, they were not externally assessed, they did not provide a route onto other qualifications, they did not confer skills which employers valued and they were overwhelmingly taught to those students marked down at an early age as under-achievers."

But he's quite happy to accept these vocational exams when they appear as equivalents in the schools he wants to praise.

Fri, 29/06/2012 - 19:53
It's very easy to get rid to teachers Ricky. Are you not aware how many are on insecure contracts? Are you aware what it's like for a secondary teacher who suddenly finds they're teaching a non-specialist subject, that they've got grotty sets in it and that Ofsted inspectors demand to see the non-specialists?

There has never been a problem getting rid of teachers Ricky - there are many ways. Especially is you're actually prepared to pay them their redundancy pay if their services are no longer required. Heaven forbid anyone should do that though.

Were you there in the 80s when teaching massively demanned? I suspect not.

This stuff about it being hard to get rid of teachers is just another of Gove's pipe dreams. The reason it takes a while is because in many cases it's appropriate to go through the steps involved in turning the situation round - because very often the situation is turned around and you can't tell which situation is which until you go through the process. Often situations of apparent failure are caused by short term issues in the life of the teacher which they need some extra support get through or by failures in the schools system beyond the teachers' control which need to be addressed. It doesn't take 18 months to sack a teacher who is proven to be failing in a way which can't be addressed. It takes 18 months from there being a concern about the teacher to see if that concern is serious and can't be rectified or not. And in most cases it can be rectified. Often during this time the teacher will accept major changes, loss of responsibility, going part time or whatever and will then prove they can do an excellent job again under less stressful conditions.

Fri, 29/06/2012 - 09:02
Janet

There is no hypocrisy here at all. No one blames schools for making rational choices to achieve the performance targets they are set. If you want to blame someone, blame those who set up a system of targets that offered perverse incentives - i.e. the last Labour government.

It would be grossly unfair to judge schools against criteria imposed ex post facto rather than against the performance indicators in place. That's why EBacc stats were deployed as a 'nudge' rather than as a stick to beat schools with.

The schools that offered these inferior vocational courses did so in good faith (they were accredited courses) and were not on constructive notice of how useless they were until the the Wolf Report told them so.

Gove has kept something like 150 different vocational courses, while scrapping those that were no good.

Fri, 29/06/2012 - 08:26
The figures* below for Harris Academies that entered pupils for exams in 2011 reveal that in most cases the academies entered pupils for a large number of equivalent examinations. This might have been the right course for the pupils but while Mr Gove slams these exams yet at the same time praises certain academies or academy chains for their overall GCSE results then it is important to know how many of these results contain the criticised equivalents.

Average number of exam entries per pupil: 10.1
Average number of GCSE entries per pupil: 4.6
(Note: intake skewed towards bottom end: only 7% high attainers, 37% low)

Average number of exam entries per pupil: 13.2
Average number of GCSE entries per pupil: 7.3

Average number of exam entries per pupil: 15.9
Average number of GCSE entries per pupil: 6.5

Average number of exam entries per pupil: 14.8
Average number of GCSE entries per pupil: 8.1

Average number of exam entries per pupil: 12.3
Average number of GCSE entries per pupil: 7

Average number of exam entries per pupil: 13.2
Average number of GCSE entries per pupil: 10.2
(Note: intake skewed towards top end: 56% high attainers, only 4% low. Only 11% FSM)

Average number of exam entries per pupil: 13.9
Average number of GCSE entries per pupil: 6

Harris Girls’ Academy, East Dulwich 67%

Average number of exam entries per pupil: 14.2
Average number of GCSE entries per pupil: 6.2

*Data from DfE School Performance Tables 2011. Percentage reaching the benchmark 5+ GCSEs A*-C (including Maths and English) is given after the name of the school.

Fri, 29/06/2012 - 08:54
Here are the figures for ARK academies that entered pupils for exams in 2011. In all cases the cohort was skewed towards the bottom end. This suggests that vocational examinations might have been the most appropriate examinations for these pupils. However, as before, if Mr Gove praises ARK academies then it’s necessary to know how much the pass rate relied on those equivalent exams that Gove criticises. Two of the academies would fail the 2015 benchmark of 50%.

ARK Globe 45%

Average number of exam entries per pupil: 11.1
Average number of GCSE entries per pupil: 5.6
(10% high attainers, 42% low)

Average number of exam entries per pupil: 9.1
Average number of GCSE entries per pupil: 4.4
(9% high attainers, 41% low)

Average number of exam entries per pupil: 10.8
Average number of GCSE entries per pupil: 6.0
(10% high attainers, 47% low)

Fri, 29/06/2012 - 09:03
Janet

Have you had any joy finding out the answer to my question at 28/06/12 at 1:16 ?

Fri, 29/06/2012 - 17:45
Skegness academy, sponsored by Greenwood Dale Foundation Trust, has just been given an award by SSAT Ltd (aka The Schools Network, aka the Specialist Schools and Academies Trust) for "added value at GCSE by scoring 10 points or more on the SSAT value-added scale for achieving 5 or more GCSEs at grade A*-C". The 2011 results are below. They include a very large number of equivalent exams.

Skegness Academy 45% achieved 5+ A*-C including Maths and English

Average number of exam entries per pupil: 12
Average number of GCSE entries per pupil: 2