Gibb’s selective use of ‘reliable data’ paints misleading picture of ‘stagnation’

Janet Downs's picture
 14
Turn the clock back to the start of the last Government. Remember the propaganda? The UK was plummeting down league tables, we were told. But this was based on a false comparison between PISA* results for 2009 with flawed results from 2000 – results which the OECD* warned shouldn’t be used for comparison.

Eventually, after the UK Statistics Watchdog censured the Government’s use of these figures, politicians stopped talking about plummeting and used ‘stagnation’ instead. It didn’t matter that at the time England was the top European nation in Maths and Science in TIMSS** tests – England was ‘stagnating’.

Schools minister Nick Gibb regurgitated ‘stagnation’ again yesterday in the debate on the Education and Adoption Bill. ‘Reliable data’, he said, proved the Coalition had ‘inherited a legacy of stagnation’. This was because ‘our schools were failing to progress’ against ‘international benchmarks’ in 2010. Four countries, ‘Poland, Germany, Austria and Estonia’, were examples of countries leaving the UK behind, Gibb said.

 

But was this entirely true? Not quite:

1Estonia’s 15 year-olds outperformed those in the UK in all three subjects

2Germany outscored the UK in maths and science but was not statistically significantly different in reading (Germany scored just 3 points more).

3Poland outscored the UK in reading but the scores for maths and science were not statistically significantly different – the UK actually scored 6 points more than Poland in science.

4The UK outperformed Austria in reading and science. Austria was statistically significantly below the OECD average in these two subjects. Austria’s score in maths (496) was not statistically different from the UK’s (492).

5The UK was at the OECD average for reading and maths and above average in science.

Gibb’s interpretation of the ‘reliable data’ from 2010 is a little shaky. And he was, of course, ignoring those much more favourable TIMSS results.

There has been another round of PISA and TIMSS since then. PISA 2012 showed a slight improvement for the UK although Estonia, Germany and Poland pulled ahead. Austria scored above the UK in maths but not in reading or science. TIMSS 2011, however, tells a different story with English 10 and 15 year-olds doing well***.

Gibb trotted out the same statistic about 100,000 more six-year-olds being ‘on track’ to becoming confident readers. This, he said, was because of ‘our focus on phonics’. Gibb seems to have forgotten the number of pupils in primary schools is rising but even if it were not, research commissioned by the Department for Education found teachers were supplementing phonics with other methods.

One million more children are in good and outstanding schools than in 2010, Gibb said. But it’s in the primary sector where the proportion of good and outstanding schools has risen – and the primary sector is predominantly non-academies. In the secondary sector, where more than half of schools are academies, the proportion of good and outstanding schools has stayed the same.

Gibb quoted Ofsted’s annual report: ‘Overall, sponsor-led academies have had a positive and sustained impact on attainment in challenging areas’. It's on page 31 of the 2013/14 report. Wilshaw noted:

'As Figure 10 shows, these academies had the greatest impact in the first few years that they were open.'

These would have been Labour academies. Machin and Vernoit's research into these early academies said becoming an academy had a 'significant improvement in the quality of pupil intake, [and] a significant improvement in pupil performance.' It's hardly surprising improved intake results in improved results.

Wilshaw also noted the rate of improvement in these academies was beginning to slow as they got nearer to national levels of attainment and results at some sponsored academies were declining. It should be remembered that many sponsored academies boosted their results by including 'equivalents' and the DfE admitted sponsored academies did no better than similar non-academies when these equivalents were removed.

Gibb said the Chief HMI's comment was ‘backed up by results that show that sponsored academies are improving their performance faster than maintained schools.’ I didn’t know whether to react to this statement with weariness or a snort of derision. Gibb knows – he’s been told often enough – that the rate of improvement in sponsored academies starts at a lower base than maintained schools. And a rate of improvement calculated from a low base is bound to be larger than one calculated from a higher base. But, as noted above, when sponsored academies are compared with similar non-academies, the DfE admitted in the High Court that the former did no better than the latter when equivalent exams are stripped out.

It appears the debate on the Bill has been debased by shaky analysis of data, misleading statements and exaggeration about academy success. This is something the Education Select Committee warned against. It seems that Nick Gibb has decided to ignore this warning just as he and the then Secretary of State Michael Gove brushed aside a warning five years ago. The earthquake which has undermined state education in England was caused by disregarding a warning: an after-shock is likely following the Government’s insistence on flouting another one.


UPDATE 24 June. The original article has been corrected. I hadn't been able find the Chief HMI quote in an Ofsted Annual Report and asked anyone who did find it to let me know. agov found it so I've removed the sentence. I have also added comments about what the Chief HMI also said after the quote. These are the two paragraphs after the quotation beginning 'As Figure 10 shows...'.

NOTES

*Programme for International Student Assessment set every three years by the Organisation for Economic Cooperation and Development to test 15 year-olds in reading, maths and science.

**Trends in Maths and Science Survey set every five years to test 10 year-olds and 14 year-olds.

***Estonia didn’t take part in TIMSS. Results 2011:

10 year-olds maths: England 9th out of 50, Germany 16th, Austria 23rd and Poland 34th.

10 year-olds science: Austria 13th, England 15th, Germany 16th, Poland 30th. Note: rankings are misleading. Austria’s score was 532, England’s 529, and Germany’s 528. The difference in scores was statistically insignificant.

15 year-olds maths and science. Austria, Germany and Poland did not take part in these. England’s score for maths was 507, not significantly different from the centre point of the international scale (500) and ranked tenth among participating nations. England’s score for science was 533, above the centre point of the international scale (500) and ranked ninth among participating nations.
Share on Twitter

Comments

rogertitcombe's picture
Tue, 23/06/2015 - 16:25

Janet - You know I disagree with you about the international PISA tests. As stated in my last post, I argue that marketisation lowers standards. I set out the learning theory mechanisms by which this lowering of standards takes place. I would therefore expect marketised systems such as the US, England and most recently Sweden, to do comparatively poorly in PISA, and they do. I will stick my neck out once again and predict that in the next PISA round this pattern will continue, despite the ever increasing proportion of schools being graded as 'good' or better on the basis of the English shallow and flawed GCSE exams system.

When it comes to TIMMS, this is a different animal to PISA. I discuss this here.

http://www.localschoolsnetwork.org.uk/2013/06/pisa-and-timms/

My judgement on the basis of the test items that I studied was that TIMMS assessments were of questionable quality and did not test deep learning. PISA does test deep learning. Marketisation incentivises shallow learning. I have written a book about it. Marketised systems do badly in PISA. This is not proof but it does add to the evidence base.

As I argued in my last post, just because our government uses poor PISA results to justify its misguided market-based reforms does not mean that the PISA conclusions lack validity.

The second part of your post in which you draw attention once again to the false claims made for the success of Academies is spot on. Your correct second argument does not depend on or follow from your dismissal of the validity of PISA.

rogertitcombe's picture
Wed, 24/06/2015 - 05:02

PiqueABoo has just posted this on my 26 June 2013 thread.

"I compared the maths a couple of years ago and arrived at much the same conclusion as Roger. TIMSS was largely recipe-following curriculum fare and PISA was wordier, multi-step stuff that expects the child to think more and know enough to select an appropriate method. One thing we can do is compare domestic performance with performance in in international tests e.g. if the apparent improvements in GCSE performance were entirely genuine we should be way out in front in PISA/TIMSS, but that isn’t the case."

PiqueABoo is right. I invite other teachers to adopt the same independent professional attitude. Of course I am not blaming teachers. They are as much victims of the marketisation ideology as are pupils, parents and our society as a whole, but it would be nice if the teachers unions were brave enough to recognise this. I post a consistent argument on this site and have written my book. I am a retired member of the NUT. They have to support all teachers including those that have to do as they are told in schools whose Executive Principals make them put results focussed shallow learning before proper teaching. Many such teachers post on LSN but they all do so anonymously, for obvious reasons.

However if bodies like the NUT want the respect of politicians, other professionals and the public they really do need to be a little braver. They could make a start by reviewing 'Learning Matters' in their member magazines. I welcome professional argument and people disagreeing and debating with me.

Janet Downs's picture
Wed, 24/06/2015 - 09:05

Roger - it doesn't really matter whether you or I agree whether PISA or TIMMS are reliable, valid etc. The point is that Gove, Gibb etc cited PISA to justify their reforms. But what they said PISA said wasn't what PISA actually said. Gove made a comparison which the OECD warned should NOT be made.

Gibb did the same in the recent debate. He said four countries were steaming ahead of 'stagnating' UK (actually England). But a look at PISA data showed it wasn't entirely true - only one, Estonia, 'beat us' in all three subjects.

The point about quoting TIMSS (reliable or not) is that there are other international tests which Gibb could have cited. He chose not to because they show England in a more favourable light. A cynic might say that if TIMSS had showed England doing poorly, Gibb et all would shriek it from the rooftops. Similarly with PIRLS - England upped its score in the 2011 round but little has been said about that.

Your argument that marketised school systems do worse in PISA doesn't quite hold. The Netherlands is committed to 'choice' and was the third highest scoring European nation behind Liechtenstein and Switzerland in PISA 2012. Australia, too, is moving towards a marketised system - Australia did well in the last PISA round.

agov's picture
Wed, 24/06/2015 - 09:04

"Overall, sponsor-led academies have had a positive and sustained impact on attainment in challenging areas."


Page 31, https://www.gov.uk/government/uploads/system/uploads/attachment_data/fil...

Janet Downs's picture
Wed, 24/06/2015 - 09:24

Thanks, agov. I'll amend the article asap. Can't see how I missed it now it's pointed out. It's clearly there on page 31. I note Wilshaw said:

'As Figure 10 shows, these academies had the greatest impact in the first few years that they were open.'

These would have been Labour academies. Machin and Vernoit's research into these early academies said becoming an academy had a 'significant improvement in the quality of pupil intake, [and] a significant improvement in pupil performance.' It's hardly surprising improved intake results in improved results.

Wilshaw also noted that rate of improvement in these academies was beginning to slow as they got nearer to national levels of attainment and results at some sponsored academies were declining. It should be remembered that many sponsored academies boosted their results by including 'equivalents' and the DfE admitted sponsored academies did no better than similar non-academies when these equivalents were removed.

agov's picture
Thu, 25/06/2015 - 09:12

"Can’t see how I missed it"

Could it be that you looked at the annual report rather than the annual schools report?

rogertitcombe's picture
Wed, 24/06/2015 - 10:09

It wasn't just 'some' sponsored Academies that 'boosted' their results in this way. They all appeared to have done this to some degree. The worst offenders were those Academies located in 'poor' areas that did not have banded admissions based on CATs scores. Some of these are still in difficulties now despite changes in sponsorship. This is all described in detail with supporting evidence in Part 4 of 'Learning Matters' and in my paper.

http://www.wwwords.co.uk/pdf/freetoview.asp?j=forum&vol=50&issue=1&year=...

rogertitcombe's picture
Wed, 24/06/2015 - 10:10

PWC were also taken in.


rogertitcombe's picture
Wed, 24/06/2015 - 10:23

But what PISA actually said still reveals that the English system is, like that in America and (now) Sweden, performing relatively poorly. Not only that, but the marketisation of the English system has now been in full flow for over a decade without much impact on our PISA performance.

I stand to be corrected about the Netherlands, but I don't believe their education system is remotely similar to ours, that of the US, or to the Swedish Free School model. An element of parental choice in relation to schools has always been part of the post 1945 British education system. As for Australia, we will have to wait and see if the educational consequences follow those in Sweden.

Janet Downs's picture
Wed, 24/06/2015 - 10:39

Roger - PWC weren't as taken in as reported. Their 2008 academies report said there is "insufficient evidence to make a definitive judgement about the Academies as a model for school improvement” and “the process of change was complex and varied and could not be ascribed to a “simple uniform ‘Academy effect’”. It also found that when academies improved they used similar methods to improving non-academies.

See faq above 'The Government cites a 2008 report by PriceWaterhouseCoopers (PwC) to justify its academy conversion programme. Does this report wholeheartedly endorse academy conversion?' for more detail.

Janet Downs's picture
Wed, 24/06/2015 - 10:42

Roger - it's true the PISA results of Sweden and America have fallen but the results for the UK show the UK is still performing at the OECD average in reading and maths and ABOVE the OECD average in Science. That isn't 'performing relatively poorly'.

And, as I keep on saying, if the Government wants to cite PISA it should cite it correctly and not say PISA says things it does not say.

Janet Downs's picture
Thu, 25/06/2015 - 11:32

Roger - PISA shows the UK's results below those of some European countries eg, Finland, Austria, Netherland (which pushes 'choice' but pupils can only go to the secondary schools which will accept them); not statistically different from others eg France, Latvia, Norway; and better than others eg Italy, Spain, Sweden, Hungary.

This shows the danger of using PISA to prove either that education in the UK (actually England) is 'poor' due to education received when Labour was in power (the Government view) or that education in England is poor due to marketisation.

PISA neither proves one or the other: according to PISA England is middling in maths and reading and above average in science.

That's not to say there isn't a great deal wrong with education in England due to all the reasons you mention: high-stakes tests, marketisation which increases competition between schools and reduces cooperation, marketisation which results in superficial pseudo-improvement which in turn reduces deep learning.

But wheeling in PISA to 'prove' deterioration is misleading because PISA data doesn't show a decline in UK performance. Neither does it show UK (England) is at the bottom of the European PISA league. It isn't. Better to concentrate on showing where education in England is going wrong than rely on PISA to prove something which the figures don't show.

rogertitcombe's picture
Wed, 24/06/2015 - 12:35

England performs well below the average level of non-marketised European education systems, let alone the non-marketised education systems of Asia.

The government is right to point to this as a concern. What we should be arguing is that our poor performance is without doubt CAUSED by aspects of our marketised system in relation to the many perverse incentives that are almost universally admitted and very well documented by everybody involved in the education system except the government.

I go further than this in arguing that marketisation of public services and especially education and the NHS INEVITABLY lowers standards because the perverse incentives are structurally embedded in the systems. Here I refer to the 1988 Education Reform Act, the Academies legislation and the NHS Foundation Trust Act.

There are many other examples. Here in the Furness peninsula we continue to suffer greatly from the privatisation and fragmentation of our rail services. My daughter in law recently returned from London on a weekday evening (Virgin Trains - overcrowded) to find that her connection at Lancaster consisting of a single coach train (Northern Rail) was so full it was physically impossible for her and our two grandchildren to get past the entrance door. They had to wait for the next train.

The same Northern Rail operates services on the Furness line and on the little known and under-promoted Yorkshire Dales line to Leeds, where two of our other grandchildren live. These services cross at Carnforth but they do not connect in any meaningful sense. In some cases the Leeds train departs from Carnforth one minute before the Furness line train arrives at the opposite platform.

There is more. The Furness line trains are also operated by First TransPennine Express, who operate the main stations on the line. This company promotes its own services to Leeds via Manchester. These are longer in distance and time, much less scenic and the fares are 30 percent higher. The stations display a railway route map of northern England but the Northern Rail line from Carnforth to Leeds via Skipton is not even shown on the map! If you want to buy a ticket by the best route to Leeds from Ulverston or Barrow you will be sold the more expensive First Transpennine Express route even if you actually travel via Skipton, unless you specifically ask for a ticket via Skipton. You can save even more money if you buy split tickets - Ulverston to Carnforth and Carnforth to Leeds, but non-one tells you that either.

OK, I admit to being a lifelong train nerd.

rogertitcombe's picture
Wed, 24/06/2015 - 12:38

Janet - thank you for this. I stand corrected. Apologies to PWC.


Add new comment

Already a member? Click here to log in before you comment. Or register with us.