Could someone please explain statistics to the Education SoS?

Janet Downs's picture
 8
Education Secretary Michael Gove has made another speech. As you might expect it hypes Government education policies to show there’s been a “renaissance” in education in England.

But when Gove starts quoting data he‘s on even dodgier ground. He seems unaware of the dangers of generalising from a small sample and doesn’t understand that if improvement rates are calculated from a low base they are likely to be greater than those calculated from a high one. It's unwise to imply a greater improvement rate is a sign of superiority over schools with a lower improvement rate.

Academies sponsored by Greenwood Dale Foundation Trust, Gove said, “are achieving fantastic results. Last year, on average, the proportion of pupils achieving 5 or more good GCSEs including English and maths rose more than twice as fast in Greenwood Dale Trust academies as in local authority schools across the country.”

Greenwood Dale had five secondary academies which posted GCSE results for 2013. Only two showed an increase in GCSE results in and one, Nottingham Girls’ Academy, showed a decline from 56% to 30%. Department for Education number crunchers said Greenwood Dale’s results showed “an average improvement of 4.8 percentage points.” But the average results for the five academies were just 48%. If equivalent** exams are removed, the average drops to 30% which doesn’t look quite so “fantastic”.

But it would be wrong to criticise these academies because of results (something Gove should bear in mind). Ofsted has inspected four and judged them all Good.

Gove compared Greenwood Dale academies with “all local authority schools” which only improved by 1.8%. But what he didn’t make clear was that sponsored academies, mostly previously low performing schools, started from a lower base of 48.8%. In 2013 results rose to 51.1% (an improvement rate of 2.3%). Results at non-academies rose from 57.4% to 59.2% (an improvement rate of 1.8%). Gove used these two improvement rates to praise sponsored academies.

But if Gove’s logic is applied consistently then he should be condemning converter academies for only improving GCSE results by 1.6%.

What Gove should have done was compared sponsored academies with similar non-academies. But he’s unlikely to do that because there’s no difference in results.

The Harris chain was also praised. Gove focussed on just one school: Harris Primary Academy Philip Lane better known by its previous name, Downhills. Gove used it to attack those who had opposed the schools’ enforced conversion:

“When Downhills was under the control of local politicians, it failed its pupils year after year. For almost a decade it drifted in and out of the very lowest category of performance: ‘special measures’.”

Note the derogatory term “drifted in and out”. This implies Downhills was barely adequate for most of a decade. But from 2005 to 2011 Downhills was not in special measures. In 2008, Inspectors said: “Downhills is an improving school that is providing a satisfactory education...”

It wasn’t until January 2011 that Downhills was judged Inadequate. But monitoring in September 2011 found a “clear trend of improvement”. Gove ignored this and said Downhills was failing. The same lead inspector returned and declared the school was still inadequate (despite the “clear trend of improvement” noted just a term before). But Sats results were rising and continued to rise after the Inadequate verdict. Art work from Downhills was chosen for display in the National Gallery before conversion.

Harris built on a “clear trend of improvement” and Harris Primary Academy Philip Lane was judged Good in June 2014. Like so many academies which post higher results, the school was already on an upward path. So much the better for sponsors taking such schools over – they can claim credit for any future improvement.

* 5 GCSEs A*-C including Maths and English

**Equivalent examinations are non-GCSE exams which are given a GCSE “equivalent” of, say, four GCSEs.

NOTE: Figures comparing GCSE results in sponsored academies, converter academies, non-academies and all state-funded secondary schools is on page 53 of the Academies Annual Report 2012/13 downloadable here. I shall write about this piece of PR puff at a later date.
Share on Twitter

Comments

Janet Downs's picture
Sun, 13/07/2014 - 08:08

See Henry's latest post about how applying the new rulings re the use and value of equivalent exams would have affected last year's GCSE results. This is especially pertinent because many of the academies and academy chains praised by the DfE in the Academies Annual Report 2012/13 for their GCSE results made heavy use of equivalent exams.


Arthur Harada's picture
Sun, 13/07/2014 - 12:58

Research methodology and statistical analysis is not a strong point at the DfE or for that matter Ofsted. Time and time again both trundle out so called findings that e.g. 1 in 5 pupils have special educational needs, 1 in 5 schools offer poor teaching and learning, 1 in 5 teachers are below average in teaching skills,1 in 5 schools are in a poor state of repair, 1 in 5 schools have below sport activities. In a representative and balanced sample of whatever ,provided the information is graphed as the normal curve of distribution there will always be a 1 in 5 below the average just as there will be 1 in 5 above average. In the field of medicine 1 in 5 men in the age group 65 to 70 are suffering from terminal prostate cancer and 1 in 5 are free of that disease. Should the HMCI of Schools be worried?


Trevor Fisher's picture
Sun, 13/07/2014 - 16:44

comments about the use of stats in the department are tilting at the wrong windmill. The misuse of statistics is consistent in proving that their policies are right. The point has to be hammered home that the government cherry picks stats to claim a policy is justified, not that it is correct. The attempt to control media reporting is the central point, and in this it has been wholly successful.

Question is, why are the critics unable to get through to the media? And how to stop being sidelined and ignored?

Trevor FIsher.

Patrick Hadley's picture
Sun, 13/07/2014 - 23:03

The UK Statistics Authority should be the place to take complaints of the government misusing official statistics. Back in 2012 they ticked Gove off about his distortion of the Pisa results.http://www.localschoolsnetwork.org.uk/2012/10/statistics-watchdog-expres...


Arthur Harada's picture
Mon, 14/07/2014 - 16:46

I know there are certain bodies to which one may express concerns as to how so called statistical data can be so manipulated to support or dismiss this or that alleged finding. The one indisputably objective body to analyse data is the Royal Statistical Society. The Society is rarely used by government departments simply because it declines to put a spin on information that might be "uncomfortable" for the latter. One of the great travesty's in analysing school data is by the likes of Ofsted inspectors many of whom obtained a C in the lower tier of the GCSE. That tier requires neither knowledge nor use of even the basic statistical technique of calculating an average.


Brian's picture
Mon, 14/07/2014 - 17:34

' .... in analysing school data is by the likes of Ofsted inspectors many of whom obtained a C in the lower tier of the GCSE.'

And your evidence for that statement is .... ? I can't see the Royal Statistical Society accepting that statement on face value!

Andy V's picture
Tue, 15/07/2014 - 11:45

On a lighter note perhaps this approach may help Mr G overcome his lack of ability with facts and figures:


http://games.usvsth3m.com/slap-michael-gove/

In excess of 21,600,000 have had a go so far and they can't all be wrong ... :-)

Janet Downs's picture
Tue, 15/07/2014 - 13:14

Thanks, Andy. The number slapping has now reached 21,698,023.


Add new comment

Already a member? Click here to log in before you comment. Or register with us.