Stories + Views

Avatar Image

grandparent

Posted on

09/05/13

go to 32 comments

DfE digs up more “survey’s” – but do they support Gove’s statement that teenagers have “disturbing historical ignorance”?

Today’s young people are uninformed about history – that’s what Education Secretary, Michael Gove, says. In a Mail article, he cited “survey after survey” which displayed “disturbing historical ignorance” among teenagers. But the Department of Education (DfE) could only find one survey when asked*. That survey, by TV Gold, turned out to be targeting all ages not just teenagers. And the survey’s questions contained inaccuracies.

Gove’s “survey after survey” implies more than one. So the DfE has dug deeper. What “survey’s” (sic) did it find?

First up is Lord Ashcroft’s poll (2012) which found a lack of factual knowledge about WW2 among 11-17 year-olds but showed that young people appreciated the human cost of war.

So far, then, one survey seems to support Gove. But what about the rest?

The DfE provided no link to its next citation: A survey of 2000 11-16 year-olds undertaken by Premier Inn. An internet search found that three newspapers, the Mail, Independent and Metro, had reproduced “shock-horror” articles citing the survey. The articles were remarkably similar which suggests the papers churned a press release without considering the reliability of the research. Premier Inn’s Facebook page (scroll down to 22 March) claimed the survey showed British teenagers were “clueless” about history and urged parents to explore historic sites. But the ridiculous answers suggest that a large number of teenagers were having a joke at Premier Inn’s expense. These included Delia Smith being one of Henry VIII’s wives and the plague being a heavy metal band.

Next up was a survey by Professor Robert Tombs for the think tank Politeia. Again, there was no link. An internet search found a Politeia study co-authored by Professor Tombs but nothing about a survey. The study could, of course, have contained a survey but no details were given in the press release and the study itself is not freely available.

An article in London Mums Magazine was the next source. This was based on research by an exam revision service.

The final survey was, again, not a survey but a 2010 Telegraph article about a poll done by Sea Cadets to discover the extent of knowledge about England’s maritime past.

So, the DfE “survey’s” included only one, Lord Ashcroft’s, which could be described as a properly-conducted poll. One was a marketing exercise taken at face value by gullible journalists. One wasn’t a survey and the final two were articles about surveys. An internet search found no details of either so we don’t know whether there was a representative sample, what the questions were or how the survey was conducted.

Gove says he is in favour of “evidence-based” policies. However, it’s unclear what value there is in “evidence” which includes surveys of dubious reliability – unless, of course, they’re chosen simply because they support Gove’s point-of-view.

*Freedom of Information request via WhatDoTheyKnow.com.  Download here.

UPDATE 13 May 2013

The original post contained a typo: WhatDoTheyKnolw.Com.  This should have read WhatDoTheyKnow.com.  It has been corrected.  My thanks to Simon for pointing out the error.

UPDATE 14 May 2013

1  Nick, commenting below on 13 May (1.10pm and 2.08pm) told us that OnePoll did the Premier Inn survey.  OnePoll provides a panel which comprises different demographics (eg teenagers, although parental consent is required for those under 18).  The surveys are posted on OnePoll’s website and panellists can choose which questionnaires to complete.  They are paid for each completed questionnaire.  OnePoll does not write the questions – that is done by the group commissioning the survey.  The New Statesman has discovered that OnePoll also did the Sea Cadets and UK TV Gold surveys.

2  Kathy Bramley has left an annotation at the foot of the FoI response which says the survey cited on the London Mums website had not been found on the Education Quizzes site but had been sent in a press release pack to London Mums.

 

UPDATE 24 May 2013

Radio 4’s More or Less discussed the surveys on 17 May.  It confirmed they were “non-rigorous polls … with the aim of turning the results into press releases designed to create publicity”.  The programme’s presenter, Tim Harford, spoke to Michael Marshall  (Merseyside Skeptics Society, BadPR) who explained how the PR survey industry operated:

1      These lightweight surveys are commissioned by companies to generate media stories.

2      The firms doing the surveys post them online for registered users to complete.

3      Registered users are paid for each completed survey.

4      There’s no monitoring of who is completing the surveys.

5      Those who answer the questions don’t necessarily care too much about accuracy.

The clip can be heard here at 11.36 mins.

 

 

Share this page:

Comments, replies and queries

  1. A Cooper says:

    A great article. However, could you remove the apostrophe from ‘survey’s’ – the pedants will have a field day.

    Thank you.

  2. Brian says:

    Just read a report of Gove’s speech about history teaching, in Brighton today.

    http://www.bbc.co.uk/news/education-22454219

    If that’s the best he can do then one can only conclude that he is becoming increasingly desperate in finding appropriate ‘evidence’ for his crusade.

    • A Cooper says:

      He has resorted to personal attacks on those he considers to be the ‘enemies of promise’ including the authors Stephanie Myer, Roger Hargreaves and Michael Rosen. The sign of a very poor statesman indeed.

      • A Cooper – it’s one of Gove’s strategies to make ad hominem attacks. At the same time he pours fullsome praise over the heads of those who agree with him. He then ranges those he praises (his “good guys”) against those he attacks (the “blob”, writers for the Guardian, unions, parents who oppose academy conversion).

        As wll as his use of dubious use “survey’s”, there are other dubious strategies:

        1 soundbites about academies which aren’t confirmed by evidence,
        2 misrepresentation of international exam data (the UK Stats Watchdog censured him on that one),
        3 his supposed support of evidence while ignoring evidence that doesn’t fit his theories,
        4 his profligate use of taxpayers’ money on his pet schemes (£1 billion overspend on academies, and the Audit Office is going to investigate spending of free schools),
        5 his misrepresentation of exam questions (eg picking out the simple part of a more complex question and producing it as an example of “dumbing-down”),
        6 his waving-away of criticism with, “Yada, yada”.

        NB My use of “survey’s” was prompted by the DfE. It appeared in the FoI response.

  3. Brian says:

    The apostrophe in survey’s is copied directly from the DfE response to the FoI request. It says ‘ …The other survey’s ….’.

  4. Adrian Elliott says:

    “But the ridiculous answers suggest that a large number of teenagers were having a joke at Premier Inn’s expense. These included Delia Smith being one of Henry VIII’s wives and the plague being a heavy metal band.”

    This is an interesting point. How many people are actually taking the mickey out of these questions? Possibly too few to make a significant difference but no one can be sure.

    I recall a Sunday Times survey about 15/20 years ago purporting to show up how ignorant teachers were. The questions were ludicrously simple – what is the capital of Poland etc. I met someone afterwards who had been surveyed and said he answered every question wrongly just to mess the exercise up.

    The most interesting aspect of Gove’s speech yesterday is how he is increasingly personalising his attacks to a quite extreme degree.Perhaps the job is getting too much for him. There must be a lot of vacancies at News International.

    • Adrian – I can’t believe that journalists were taken in by the Premier Inn poll. Perhaps they weren’t but decided to run with it because (a) it was yet another story knocking teenagers, (b) the journalists were pandering to their readers’ prejudices about state education in order to sell papers and (c) they held their readers in such contempt that they were sure the readers would be taken in.

      The ridiculous answers included:

      Jerry Hall and the Duchess of Cornwall were married to Henry VIII;
      TV builder Nick Knowles built the Pyramids;
      Shakespeare was the chairman of the BBC;
      The Emirates stadium was built before Westminster Abbey;
      WW2 Prime Ministers included Rod Stewart, Bruce Forsyth and Sir Alan Sugar;
      Emmeline Pankhurst was an X-Factor Finalist.

      A Premier Inn spokesperson told the Mail that teenagers’ lack of historical knowledge was “something that can be rectified by visiting all the fantastic landmarks and places of interest the UK has to offer.”

      While staying at Premier Inn, of course.

  5. Keith Turvey says:

    Yes great work Janet. A colleague of mine also pointed out the absurdity of all of this when contrasted with Gove’s call for RCTs in education e.g. https://www.gov.uk/government/news/new-randomised-controlled-trials-will-drive-forward-evidence-based-research

  6. Helen Marsh says:

    I find it deeply ironic that we teach children to use evidence and cite it correctly in their work, yet the DfE doesn’t think it is necessary to do so. Although they have been selective, they haven’t checked the provenance of their sources – essential for historians. No, not ironic – infuriating.

  7. Simon Whitaker says:

    When you say “WhatDoTheyKnolw” do you by any chance mean “WhatDoTheyKnow”? At least you didn’t put an apostrophe in there, eh? ;-)

    • Simon – thanks for pointing out the typo. In all fairness to the unfortunate DfE agent* who typed “survey’s” in the FoI response, I suspect that was also a typo.

      Pedantic explanatory paragraph follows – skip it if you want: If you look closely at my post you will see that every time I typed “survey’s” it was in speech marks. This shows it was a quotation. The first time I typed “survey’s” I followed it with the word “sic” in brackets. This is used to indicate that the person writing is quoting something which contains an error. It shows the writer is aware of the error but because it’s a quote s/he’s keeping the error intact. (End of pedantic bit.)

      In retrospect, I wish I hadn’t. This is for two reasons:

      1 It drew attention from the main point – Gove’s use of dodgy surveys (oops, nearly put in an apostrophe).

      2 It was petty. I apologise.

      I shall, however, correct the typo in WhatDoTheyKnolw and acknowledge the alteration.

      *Yes, the DfE does call its staff “agents” – just ring it up and listen to the recorded message. Are some of them recruited from MI5, do you think?)

  8. Janet didn’t put the apostrophe in. It was Francis Quinn from the Ministerial and Public Communications division of FOI who did.

    https://www.whatdotheyknow.com/request/details_of_surveys_underpinning

  9. Harry says:

    Congratulations to Janet for pursuing this. It’s interesting to see that the Guardian picked it up today – http://www.guardian.co.uk/politics/2013/may/13/michael-goves-claim-teenagers-ignorance , with the education secretary, in a nice touch, labelled Mr Sloppy.

  10. Nick says:

    haha, I believe I may have been one of those 2000 ‘Teenagers!’ The Premier Inn survey was one conducted by an online polling company called Onepoll: http://www.72point.com/coverage/delia-smith-married-henry-viii-nick-knowles-built-pyramids-teens/

    They pay 50p a survey, and payout at £40 – so you have to complete a lot of surveys for your money. No doubt, the system encourages people to be a little economical with the truth – especially when surveys come with screening questions which subtley hint at the direction you need to respond with.

    You can, however, have a lot of fun trying to answer the questions in such a manner that there’s no easy PR angle for them to write about.

    • Thank you for that Nick – very useful. I’ve just been on the phone to OnePoll and discovered that they just provide the panel (ie the people who respond to surveys) and post their surveys on line. As you say, respondents get paid for each completed survey. They tell me that they check to ensure that respondents in a particular demographics (eg teenagers) have to prove they are the age they claim. If the respondent is under 18 then parental consent is required.

      OnePoll said it’s up to each panel member to decide whether to respond to a survey. But they don’t provide the survey – the questions are decided by the group commissioning the survey.

      Thanks again for your help.

      • Nick says:

        No problem! I wold like to add a couple more points however:

        “they tell me that they check to ensure that respondents in a particular demographics (eg teenagers) have to prove they are the age they claim. If the respondent is under 18 then parental consent is required.”

        Hmm I’d say this depends very much on your definition of proof! The extent of the proof demanded by Onepoll is always couple of questions at the begining of a survey that ask
        1) Do you have a child between the ages of (x) living with you? and
        2) Are you happy for that child to complete the rest of the survey?

        With these two questions answered, no further checks are made and they don’t seem to bother cross-referencing their data from other surveys (so you’ll never get into trouble for being a single mother of 1 in one survey or a married father of 3 in the next.)

        I’m also a little dubious about their claims that they “just provide the panel (ie the people who respond to surveys) and post their surveys on line.” While that might strictly be true of onepoll.com, it’s worth noting that they share phone numbers and some of their website material with sister site 72-point, that offer News Copy Writing as one of their services (and who also wrote the Premier Inn story) http://www.72point.com/services/news-copy-writing/

      • Jonathan Brown says:

        Janet, My name is Jonathan Brown and I am a reporter for The Independent (not the one who churned out the Premier Inn poll – although there but for the grace of God …). Would you please give me a call/email to discuss your excellent FoI requests? I’m on 07740 582279. I will call you straight back. Many thanks.

  11. Mike Duncan says:

    Excellent dogged persistence!

  12. Congratulations, Janet, for a thorough, evidence based investigation. It’s a shame that the Secretary of State for education, who received the benefit of the best possible education chooses to fulfill his duties in such a dangerous and intellectually dishonest manner, and your comments (10th May ) re. his modus operandi are spot on. The Huffington post has picked this up, with comments from others including Russell Tarr and Tristram Hunt who says the SofS risks coming across as “Mr Sloppy” http://tinyurl.com/cah9hvv

  13. Annetta Blakemore says:

    I haven’t laughed so readily and with such spontaneity since I read Richmal Compton’s “William” books (or book’s) years ago. I defended Gove to my history graduate grandson; thank you, Janet, for giving him such a delightful and effective riposte !

  14. FullFact has commented on my FoI request in an opinion piece in which it describes how two government departments have been caught out in one week over dodgy data.

    http://fullfact.org/articles/education_surveys_benefit_statistics_two_government_departments_caught_out_one_week-28929

    • Keith Turvey says:

      Yes interesting Janet but I find the last paragraph of the Fullfact article a bit disappointing in implying that policy based on poor evidence in education is a bit of fun but employment statistics are far more serious. Bad education policy has potential to impact significantly on employment/unemployment statistics affecting real,people’s lives and aspirations. What could be more serious?

      Good that you scoop is getting so much coverage though.

  15. I’m not a tweeter but according to MSN #govepolls has provoked a flurry of tweets. MSN quotes two in particular with spoof surveys (at least I think they’re spoofs):

    John Prescott: 10 out of 10 tweets have made #govepolls the top trend on Twitter.

    Kevin Brennan MP: According to #govepolls 10.66% of pupils think that the Battle of Hastings was won by will.i.am the Conqueror.

    http://news.uk.msn.com/trending-blog/education-secretary-michael-gove-ridiculed-over-facts-765410

  16. Kathy Bramley says:

    I wonder about a further request asking for details on department polices relating to evidence standards. Can you ask for the text of the memos where the information was sought and received?

    It might also be interesting if this opens a conversation about what’s expected of ministers that Michael Gove will listen to without his political rage filters up. I’m sure he finds it embarrassing, especially the comparisons made: Gove wouldn’t look favourably on A Level history students quoting unsourced facts from the internet. So he shouldn’t be surprised by criticism when he does the same. (PR week)

    I can understand people wanting to go into government and shoot from the hip, to play it with your heart. I think that might be a bit sympathetic, and maybe Gove was instinctively or deliberately trying to light up recollections others have regarding these surveys. Variants have been circulating in the media for a while. Whilst all well done and reliable department surveys can be a bit dry and limited in audience, the very familiarity of the of surveys I think increases our confidence in them in psychological mechanisms related to the ways brand recognition and ‘no such thing as bad publicity’ are supposed to work. I haven’t of course right now got evidence. Go go gadget google! And anyway, I’m just a slacktivist with mental health issues.

    I like to think that in intent, veiw of other people, effect and action generally I have more positive agendas than Michael Gove does, though I am sure he doesn’t see it that way. Although rather than guess what he thinks, I’d like politicians to take down their shields and talk with grass-roots out in his field; to listen and have ears to hear. And yet remain rooted in something.

    (PR Week – retrieved Fri 17 May 2013- http://www.prweek.com/uk/news/1182280/hit-miss-michael-gove-cites-pr-surveys-criticising-teenagers-knowledge/) — Not a standard format of reference but I was always a bit rubbish at them.

  17. Kathy Bramley says:

    Oops – so that was two points for potential action at the beginning of my post: department evidence standards; internal memos relating to the request. I’m uncertain about how many pops you get. But only you can make further reply/request linked in. It would be good to have it all together in-line on the one request, although I could make a separate one. I don’t want to simply Gove bash; that’s not me. Rather I want to seek the all round best interests in mind in a satyagraha style conversation; to Occupy my democracy. Neither do I wish to tread on your toes!

    http://en.wikipedia.org/wiki/Satyagraha#Satyagraha_theory

  18. […] well targeted Freedom of Information request to the UK Department for Education and its consequent report hit the news here recently. It turns out that a claim by Minister for Education, Michael Gove, that […]

  19. […] see: 1 Second half of this thread. 2 Myths promoted by the DfE here. 3 The now notorious dodgy surveys used by Gove. 4 Concerns about how the DfE “misunderstands” and “misuses” evidence. 5 How […]

Want to follow comments on this post? Use the RSS feed or subscribe below

Reply


four + 5 =

DfE digs up more “survey’s” – but do they support Gove’s statement that teenagers have “disturbing historical ignorance”?

2 Trackbacks