Stories + Views

Posted on

23/11/12

go to 49 comments

Mixed messages from Sweden

An interesting insight into developments in Swedish education when I participated yesterday in a seminar on Swedish free schools and their continuing impact on Britain’s education, hosted by the Swedish embassy. As LSN readers will recall, Michael Gove made much of the success of Sweden’s free schools prior to the 2010 election; as Rachel Wolf, CEO of the New Schools Network – the free school organisation over here – said, at yesterday’s seminar, the Conservative party thoroughly enjoyed invoking Sweden – Guardian readers’ egalitarian dream – in their new schools policy, just as they have been thrilled by the adoption of Charter schools by Democrat dream Barack Obama.

But, of course, as Warwick Mansell pointed out on this site, Gove has not spoken much of Sweden’s free schools since 2010, because, in fact, the evidence on their achievements and social impact has not been particularly positive.( Gove has since moved on to Poland, and the liberal quoting of African proverbs.)

At yesterday’s seminar, there was much discussion of the latest research findings on the impact of free schools ( which now amount to about 10% of secondary schools in total.) While previous studies have recorded marginal gains for these new schools, the latest study seems to confirm a slight rise in achievement in the free schools, and again, unlike previous studies which showed no long term effect of attendance at these schools ( as in increased take up of university education) there does now seem to be continuing impact . They do not however comment on the social make-up of these schools, or deal with the suggestion that benefits of free schools are largely to the children of the already well educated.

Not surprisingly, the pro free school lobby have been making much of this study, with Fraser Nelson of The Spectator – who also participated in the day’s discussion on Sweden’s economic and social policies – using the study to urge the government to move quickly in favour of for-profit schools over here.

With everyone focussing largely on statistical results – or ‘product’ – it was left to me to point out that other evidence on free schools indicates growing social and ethnic segregation, and intensifying issues regarding admissions that will be familiar to anyone who understands the English school system. In Sweden, there are long ‘pizza queues’ as parents – or often, their au-pairs – queue to get a place at so called ‘open admissions’ schools.

Three things struck me as a result of yesterday’s discussion. The first is that Sweden’s free schools brought new providers into what was essentially a comprehensive system, although a system now becoming distorted, English fashion, by competition for school places. But this makes the function and impact of free schools very different from the one that it is having here, where free schools enter an already highly stratified school system, and judging from all the various early findings on free school admissions policies, aired often on this site, seem likely to add to that.

The second thing that became very clear is the now almost unstoppeable growth of the for profit sector, worldwide. This is, of course, the essence of GERM ( the Global Educational Reform Movement) and, of course, in human terms, there are now hundreds and thousands of people and companies who are dependent on, or keen to enter, the new schools market. I suspect that mere governments – elected by those unimportant creatures ‘ the people’ – are increasingly vulnerable to, and powerless in the face of, the smooth and powerful power-point claims of the for-profit sector in education. Let’s hope that Labour holds out in its current resistance to the incursion of the for-profit sector: not least because these companies – whatever their results – create very different kinds of schools and school communities – or lack of them.

Finally, it became clear during the afternoon that Sweden’s recent plunge down the PISA tables may be less to do with free schools, in fact, and everything to do with the dramatic deregulation and liberalisation of education in Sweden that has occurred post 1992 – with the introduction of school vouchers and increased school choice, individual teachers pay and the reduction in central government regulation of standards. At the same time, there have been rapid changes in education policy from the centre, with ministers constantly ‘interfering’ in matters such as the curriculum.

Sound familiar?

Share this page:

Comments, replies and queries

  1. Thanks for posting this Melissa. It sounds like an interesting discussion although I am puzzled by how a discussion about the performance of Swedish free schools could take place without some evidence of the intake of these schools. If they are indeed educating a largely advantaged group, possibly with high prior attainment, that may well explain their results.

    I still think the for profit lobbyists have got a long way to go if the debate in which I took part at the London Festival of Education was anything to go by. The Festival attracted people with a wide range of views but on this issue there seemed to be a strong consensus against introducing for profit.

    Interestingly my opponent, Lucy Lee , from Policy Exchange didn’t attempt to make the argument on the grounds of higher quality in for profit schools ( presumably because the evidence doesn’t support this) but because for profit providers could help supply the extra school places needed in some areas with no extra cost to the state.

    • Yes I agree Fiona. I’m sure if the Policy Exchange thought this report offered robust evidence of higher quality learning they’d have seized it.

    • I was also at the debate at the London Festival of Education about for-profit schools. It was interesting that Lucy Lee from Policy Exchange didn’t argue outright for for-profit schools but suggested a half-way “social-enterprise” model. I agree with Fiona’s remark in the debate that “social-enterprise” in this context (allowing those involved – teachers and so on – to take a share of any excess) to be a smoke screen.

      As a parent and grand-parent, I would be concerned if I thought any teacher stood to make a profit out of the education provided. There would be a conflict of interest. How could I be sure that the resources were the most suitable or just chosen because they were the cheapest? Would I be expected to contribute? Would classes be larger than they needed to be? Would money be skimped on classroom support?

      http://www.localschoolsnetwork.org.uk/2012/06/is-social-enterprise-the-acceptable-face-of-making-money-from-providing-state-education/

  2. Paul Reeve says:

    .

  3. Paul Reeve says:

    I didn’t mean to say nothing! I thought I was asking for an e-mail to keep me informed about any comments on this interesting article.

  4. Yes the report from the Swedish IFAU is interesting but there are some things I don’t get from it. Most importantly I’m not convinced by how they controlled for potentially significant variables, particularly grade inflation. We can look at grade inflation from different perspectives e.g. within a cohort (within year), across cohorts (historically).

    From what I can determine from the report they appear to have looked at grade inflation within cohort as opposed to historically over a number of years. Their concern presumably being to refute any claims that there may have been some manipulation or ‘skim creaming’ in the admissions process. A significant paragraph I think is this one:

    ‘In Sweden, the average grade scores determine admittance to specific high school programs whenever there is an excess demand for slots. Although the scores on the national standardized tests guide the teachers’ grade setting in some core subjects (math, English and Swedish), the concern is that differential grading standards might have developed in municipalities with more or fewer independent schools. We might expect the schools to compete for students not only with high-quality education but also by inflated grades.’ P.23

    They go on to give a defense of their methods in ruling out grade inflation. First they comparing grades in core subjects where there are national standardised tests with subjects where there are no standardised tests and recognised grade inflation. This seems quite robust for within cohort comparison. Secondly they try to rule out historical grade inflation by comparing grades with post compulsory university attendance claiming ‘we find positive effects also for high school grades and university attendance.’ It should be noted also that the outcome measure here was ‘At least 1 semester of university studies at age 22.’ This to me would seem to be an inadequate way of ruling out year-on-year grade inflation. It is not comparing like for like or coming anywhere near it. I can’t as yet see any evidence ruling out potential year-on-year grade inflation. Interestingly, in the footnotes on p.24 they acknowledge referring to Vlachos:

    ‘He stresses, however, that there is uncertainty due to the fact that there is no perfectly objective measure of knowledge to compare the grades with. The results in Vlachos (2010) are very much in line with what we find.’

    They seem here to be acknowledging that there is no objective measure of knowledge that would allow them to rule out absolutely year-on-year ‘grade inflation’ over time. If this interpretation of this study is accurate (I would need much more time to study it in more detail) then the conclusions they are drawing about educational performance after the reforms seem bold and potentially misleading.

    Increased performance within a bounded system is not necessarily synonymous with increased learning or increased intellectual/academic capability. Indeed they go on to say ‘Our positive estimates might appear surprising given Sweden’s relative decline in scores on international tests such as PISA since the mid 1990s………Either way, we do not find any support for the belief that an increase in the share of independent school students provides an explanation for Sweden’s relative decline.’

    It’s worth remembering that a lack of evidence is not evidence itself of a phenomenon. And I would also point out that this report has not undergone rigorous scientific peer review as the authors and IFAU point out themselves at the start:

    ‘Papers published in the Working Paper Series should, according to the IFAU policy, have been discussed at seminars held at IFAU and at least one other academic forum, and have been read by one external and one internal referee. They need not, however, have undergone the standard scrutiny for publication in a scientific journal. The purpose of the Working Paper Series is to provide a factual basis for public policy and the public policy discussion.’

    Am interested to know what others think about how they have controlled for ‘grade inflation.’

    • Keith – thanks for pointing out that the paper “need not… have undergone the standard scrutiny for publication in a scientific journal.” It’s a “Working Paper” which is intended to provide facts to inform discussion and is full of statistical information much of which is impenetrable to a non-statistician.

      For the sake of argument, though, let’s presume the statistics are sound and the methodology robust. The researchers looked at test scores in Maths and English. They didn’t use grades in Swedish because separate classes and grading scales applied to native Swedes and immigrants. So, the researchers base their conclusions on a limited set of test scores which neglects other subjects and other kinds of educational achievement.

      • Quite. However you are too generous in presuming the methodology is robust. I still have significant concerns about the methodology and the confidence with which they rule out grade inflation.

  5. Ricky Tarr says:

    Gabriel Sahlgren cites a paper by Tegle that may have been overlooked for the reason he suggests:

    …..missed a study published in April 2010 (it is available in Swedish only). Whereas all previous research relies on a sample of students/municipalities, Tegle (2010) analyses the educational achievement of all 9th grade students in 2006. Data from graduates in 2005 are used as a control for model specification, but the results are almost identical. Tegle finds a significant effect of competition on municipal school students’ GPAs for students from all socio-economic backgrounds: a 10% increase of students in independent schools increases the municipal school average GPA by up to 2%, while increasing the performance on the standardised test in mathematics by up to 5.9%. Furthermore, Tegle shows that students in independent schools do significantly better than peers in municipal schools. The effect of attending an independent school equates to a 21% increase in the GPA and, even more astonishingly, a 33% increase in scores on the standardised mathematics test.

    Doesn’t the fact that the relative difference in the overall GPA is smaller than the difference in the standardized test suggest that accusations of over-generosity in teacher assessments in free schools may be unfounded?

    • Ricky Tarr says:

      The IFAU document (page 24) is not a relevant consideration in assessing the contribution of free schools to system-wide outcomes:

      We find a statistically insignificant association between the change in average “grade inflation” in the municipality between 1992 and 2009 and the change in the share of independent-school students between the same years. If anything, the negative point estimates indicate that there is less grade inflation in areas with more independent-school students.We therefore conclude that differential grade inflation does not drive our positive results for educational achievement outcomes.

    • The reason Rebecca Allen overlooked this ‘study’ Ricky I would suggest is not because it’s in Swedish but because it’s merely a pdf document supported by the confederation of Swedish enterprise. If there was any worthwhile truth in this so-called ‘study’ it would have been translated by now and validated academically. However, I suspect it’s just another bogus bit of ‘grey’ literature that is m ore propaganda than serious evidence.

  6. One interesting comment in the paper concerned “ownership structure” of Sweden’s independent schools. The authors say, “Forward-looking owners of for-profit schools might develop successful and competitive schools to maximize their long-term profits. However, they might also start schools that turn out to be of low quality if their focus is on short-term profits.”

    So, the authors say a focus on “short-term profits” could lead to poor quality schools but a focus on “long-term profits” might, in the authors’ view, be more likely to result in successful schools. But is this necessarily so? The Swedish education minister has doubts – he’s just established an enquiry into the motivation of the for-profit providers of Swedish free schools. He told the BBC that bringing in the profit motive resulted in a conflict of interest – those of the child versus the needs of shareholders for a financial return.

  7. The IFAU paper makes it clear that there’s a “sizeable fraction” of Swedish municipalities where there are no independent (ie non municipal) schools. Not all parts of Sweden, therefore, want non municipal schools.

    It’s important to remember that the IFAU paper looked at all types of Swedish independent schools – whether religious, non-profit cooperatives, or for-profit corporations. And the authors found no difference between different types of independent schools as to the effect they had on pupils.

    So, this paper which looked at test scores in only two subjects and which included all types of publicly-funded, independently run schools in Sweden (not just those run by for-profit organisations) is being used to support for-profit schools in England. Fraser Nelson tried to deflect criticism on this point (see PS on his blog) by saying he called for for-profits because they “get help fastest, to the communities that most need it.”

    Really? Can’t see a for-profit company being very interested in an area which is unlikely to bring in much profit such as disadvantaged or sparsely-populated areas.

    Message to Fraser – you put the term “bog-standard comprehensive” in quotes. It doesn’t appear at all in the IFAU document.

  8. In the conclusion, the authors of the IFAU papers cite research from Chile and by Clark (2009) in the UK.

    According to the authors, Clark looked at GM schools and found “large positive effects for these schools, but little spillover effects on the neighborhood schools.” That’s probably because the intake of GM schools changed – they had fewer disadvantaged and low attaining pupils than previously. This led to suspicions that many GM schools were using their admission policy to dissuade pupils whose performance would bring down overall results.

    And Chile. Research published in 2006 won’t reflect the current unrest – students are protesting against for-profit schools. These have been going on for more than a year – I’m surprised the authors haven’t noticed.

  9. Tony Chuan says:

    Presumably the queuing by parents to get their children into particular schools must be a good thing, both in Sweden and in England?

    It certainly provides a much simpler way of estimating the performance of a particular school rather than exhausting statistical studies using various metrics, as you suggest.

    Having said that, both UNICEF and OECD studies give Sweden very high ratings in their child well being indexes.

    Of course these indexes do not measure social and ethnic integration, no doubt because these are not considered relevant to a child’s essential well being.

    Additionally, the most recent study of Swedish educational outcomes suggests that although Swedish education has suffered a decline in PISA related rankings, that decline is relative and in no way due to an increase in independent schools.

    ‘Our positive estimates might appear surprising given Sweden’s relative decline in
    scores on international tests such as PISA since the mid 1990s. However, we do not find
    significant positive effects for the earlier years, when Swedish relative test scores declined most dramatically.

    Additionally, because we look at variation across municipalities it can very
    well be the case that the municipalities with few or no independent schools have contributed most to this decline.

    Either way, we do not find any support for the belief that an increase in the share of independent school students provides an explanation for Sweden’s relative decline’

    Independent Schools and Long-Run Educational Outcomes: Evidence from Sweden’s Large Scale Voucher Reform June 2012 Anders Böhlmark Mikael Lin

    • Tony – can you please explain what is meant by “relative decline”?

      • Tony Chuan says:

        I am sure you can provide an explanation for yourself.

        Relative decline is, of course, decline relative to other performers.

        Thus Sweden and Britain have both suffered relative declines in PISA rankings over the last decade or so.

        However, Sweden performs well in child well being indexes.

        Britain does not.

        • Thank you, Tony. I wanted to be absolutely sure you understood the term because all you had done was cut-and-pasted from an article. However, your reply shows a misunderstanding about the PISA UK results which is hardly surprising since Gove has contributed to the misunderstanding by using figures (ie the 2000 PISA UK data) which are flawed. The UK Statistics Watchdog recently said that the use of these figure by the DfE caused “concern” namely:

          “I was concerned to review the Department for Education’s press release of 7 December 2010 in which headline results for England from the PISA study, alongside relative international rankings, were not accompanied by detailed advice or caveats to help the reader in making comparisons over time, nor were the statistical implications of an increase in the number of reporting countries in later PISA studies noted.”

          In other words, the DfE had failed to take heed of an OECD warning that the 2000 figures were flawed and it was not safe to say that the UK had fallen “over the last decade” as you did. At the same time, the DfE had failed to stress the implications of more countries taking part.

          As it was, PISA scores for the UK hardly changed (not enough to be statistically significant) between 2006 and 2009 (the only two years where it is safe to make a comparison).

          http://fullfact.org/articles/statistics_watchdog_education_international_school_league_table-28392

          • Tony Chuan says:

            Heavens to Betsy and suffering catfish!

            Is there some reason why cutting and pasting from a referenced article in order to illustrate/reference a point of view is frowned upon here?

            I had always, perhaps naively, assumed it was standard practice to provide referenced quotations to provide at least some evidence of researched back up for a particular point of view in order to stimulate debate?

            For example, in support of your comments that actual PISA scores hardly changed 2006-2009 and in support of mine, (not necessarily conflicting with yours) that this represented a relative rather than an absolute decline:

            ‘This leads to the following conclusions:

            • Both PISA and TIMSS are problematic for studying change in average test 
            performance in England over time. 
            • Statements such as those made by the policymakers cited above are based upon 
            flawed interpretations of the underlying data.
            • England’s movement in the international achievement league tables neither 
            supports nor refutes policymakers’ calls for change.

            The PISA study has been conducted four times (2000, 2003, 2006 and 2009) with all 
            OECD countries taking part in every survey wave.

            The total number of countries in PISA has, however, risen from 43in 2000 to 65 in 2009 (a large number of non‐OECD members have been added). 

            The implication of this is that one of the reasons England has “plummeted” down the 
            international rankings is because more countries are now included (i.e. it is easier 
            to come tenth in a league of 43 than it is in a league of 65)’

            At least consider it possible therefore that the above comments, admittedly cut and pasted (?), from a University of London working paper (DoQSS Workings Papers often represent preliminary work and are circulated to encourage discussion. Citation of such a paper should account for its provisional character. A revised version may be available directly from the author.) must, equally, apply to the relative decline of Sweden in PISA rankings?

            Describing such a relative decline for Sweden as ‘a plunge down the PISA rankings’ is therefore every bit as exaggerated as the OECD claim that British State education has failed to benefit at all from massive increases in spending over the last decade.

            The OECD study – Education at a Glance – found that expenditure on UK primary and secondary schools and colleges as a percentage of GDP increased from 3.6 per cent in 1995 to 4.5 per cent in 2009, higher than the OECD average of 4.0 per cent.

            At the same time, there has been ‘no improvement (based on PISA scores) in student learning outcomes’, the report said.

            http://www.oecd.org/education/preschoolandschool/educationataglance2011oecdindicators.htm

            In the words of the great Frank Sinatra:

            ‘You can’t have one without the other’

    • Tony, I did say in the post that the overall decline in Sweden’s performance might well be due to system wide changes since 1992 – brought in by a right of centre/austerity driven government; changes that subsequent social democratic governments have tried, in part, to reverse.

      • Tony Chuan says:

        Thank you for your response.

        Yes, you did.

        However the paper that I reference: ‘Independent Schools and Long-Run Educational Outcomes: Evidence from Sweden’s Large Scale Voucher Reform’, written in June 2012 by Anders Böhlmark and Mikael Lin, seems to be the most recent research on this subject.

        What they suggest is that declines in educational achievement in Sweden relative to other tested countries, evidenced by PISA, have been turned around by Swedish educational reform:

        ‘This paper evaluates average educational performance effects of an expanding independent- school sector at the compulsory level by assessing a radical voucher reform that was implemented in Sweden in 1992.

        Starting from a situation where all public schools were essentially local monopolists, the degree of independent schools has developed very differently across municipalities over time as a result of this reform.

        We regress the change in educational performance outcomes on the increase in the share of independent-school students between Swedish municipalities.

        We find that an increase in the share of independent-school
        students improves average performance at the end of compulsory school as well as long-run educational outcomes.

        We show that these effects are very robust with respect to a number of
        potential issues, such as grade inflation and pre-reform trends.

        However, for most outcomes, we do not detect positive and statistically significant effects until approximately a decade after the reform.

        This is notable, but not surprising given that it took time for independent schools
        to become more than a marginal phenomenon in Sweden.

        We do not find positive effects on school expenditures. Hence, the educational performance effects are interpretable as positive effects on school productivity.

        We further find that the average effects primarily are due to external effects (e.g., school competition), and not that independent-school students gain significantly more than public-school students.’

        • Tony to count as reliable research this paper should be peer reviewed to ensure the claims that are being made in the conclusion – which you are citing uncritically – are based on a robust methodology.

          The paper isn’t peer-reviewed and as I point out in my comments further up there are aspects of the methodology that do not add up. Completion of one semester at University by age 22 is not a very robust way to control for grade inflation for example, which is one approach they used.

          • Tony Chuan says:

            The paper I quote from makes no bones about its provenance:

            ‘IZA Discussion Papers often represent preliminary work and are circulated to encourage discussion. Citation of such a paper should account for its provisional character.’

            I don’t believe I am citing it uncritically. That is why I use qualifying words like ‘seems’ and ‘suggest’.

            I simply offer it up as the latest research on the subject.

            Readers here will make up their own minds based on their own experience and research.

            However, clearly a fair amount of review and discussion has already taken place regarding the paper:

            ‘This is a substantially revised and updated version of the working paper “Does School Privatization Improve Educational Achievement? Evidence from Sweden’s Voucher Reform,” (IZA DP 3691).

            We are grateful to Anders Björklund, Peter Fredriksson, Erik Grönqvist, Eric Hanushek, Lena Hensvik, Caroline Hoxby, Markus Jäntti, Matthew Lindquist, Dan-Olof Rooth and Jonas Vlachos for valuable comments and suggestions. We thank Eskil Forsell and Arvid Olovsson for excellent research assistance.

            We would also like to thank participants at the CESifo/PEPG economics of education
            conference in Munich, the IFN workshop on privatization in Vaxholm, and at seminars in Gothenburg, Stockholm (SOFI) and Uppsala (IFAU)…….

            Mikael Lindahl is a Royal Swedish Academy of Sciences Research Fellow’

            For me, perhaps the most telling comment of the paper is: ‘However, for most outcomes, we do not detect positive and statistically significant effects until approximately a decade after the reform. This is notable, but not surprising given that it took time for independent schools to become more than a marginal phenomenon in Sweden.’

            This seems, intuitively, to be a statement of the blindingly obvious.

            Counter arguments against coalition government education reforms seem to fall mainly into the category: ‘I may not always be right but I’m never wrong’

            I believe in keeping an open mind.

            The curse of education policy, as with so many other areas of policy, is the two party system of this country which rarely allows reforms to take their course: ‘order, counter order, disorder’.

            These coalition reforms enjoy cross party support. We will not know their efficacy with any certainty until 2020.

        • Yes my main issue Tony is that we shouldn’t call this research as its methodology has not been rigorously peer-reviewed. Alot of stuff gets called research inferring ‘academic research’ and I admit I have done this myself in the past. Clearly the paper is an informed discussion paper, but I wonder why the authors haven’t submitted their work for peer review. It maybe that as well as their conclusions, their methodology is also provisional.

  10. johnbolt says:

    Have people noticed that IES – the Swedish company that runs Breckland Free School – has been bought by US private equity firm TA Associates. As Mary Bousted put iut “IES is no longer an educational start up business; it’s part of a global firm that exists to sweat their assets”

    • johnbolt – it’s not just IES: 90% of the John Bauer Organisation, Sweden’s biggest provider, was sold in October 2008 to the Danish investment company Axcel which is a venture capital company. Axcel had no experience in education and was involved in the business of selling and buying businesses. The company’s particular interest in home styling and dog food.

      Presumably this is why Sweden’s education minister has started an enquiry into the motivation of the for-profit firms running Sweden’s free schools.

  11. Adrian Elliott says:

    Janet……….’dog food’!

    At least we know now how savings can be made, given the ending by Gove of any nutritional standards in academies. Will certainly make for cheap and wholesome school dinners.

    • Michael Uttley says:

      Many of my fellow undergraduates, particularly rugby players and lawyers, enjoyed cans of PAL (Prolongs Active Life) at university, at the expense of my dog, whom they were supposed to be looking after.

      Some slipshod research seems to indicate that this practice survives to this very day:

      ‘Nutritionist Mike Konowalski has lived on a diet of dog food alone for the last month.
      He claims to have lost over 7lbs in three days from wolfing down tins of cans of dog food and crunching bone shaped biscuits.

      Doctors have warned him he faces intestinal problems but will not suffer from any long term medical issues from eating foods he usually serves to his pet boxer dog.

      Konowalkski has agreed to go on the doggie diet to show that lifestyle choices are to blame for people eating too much.

      In an earlier experiment he lived on fast food for a month and piled on the pounds.
      The Las Vegas based nutritionist stocked up on tins of organic chicken, beef and turkey from his local pet store.

      To supplement the meat he munches dry biscuits known as Milk Bones.

      Konowalski said he will stick to the stomach turning diet for a month to prove that any type of food can be healthy and people do not have to diet to lose weight.

      ‘Anybody who’s choosing the new lifestyle, we have to suffer. There’s no other way around,’ said the Polish-born nutritional consultant.

      ‘I think the dog food that I’m eating right now is more healthy than what I’ve had in a year.’

      http://www.sugarscape.com/main-topics/celebrities/745593/exclusive-perrie-edwards-eats-dog-food

  12. Interesting also is the fact that when you do a Google scholar search for Bohlmark and Lindahl all you seem to get is ‘discussion papers.’ I’ve also tried searching various academic databases like Scopus, sciverse, isi web of knowledge ….

    • Bohlmark and Lindahl published a discussion paper in 2008 on the same theme (“Does School Privatization Improve Educational Achievement? Evidence from Sweden’s Voucher Reform”) for the Institute for the Study of Labor (IZA) in Bonn. It contained this caveat:

      “IZA Discussion Papers often represent preliminary work and are circulated to encourage discussion.Citation of such a paper should account for its provisional character”

      Their conclusion in this provisional paper was:

      “We find that an increase in the private school share moderately improves short-term educational outcomes such as 9thgrade GPA and the fraction of students who choose an academic high school track. However, we do not find any impact on medium or long-term educational outcomes such as high school GPA, university attainment or years of schooling. We conclude that the first-order short-term effect is too small to yield lasting positive effects.”

      They also wrote, “…despite these arguments in favor of choice, the evidence from the literature evaluating general effects of school choice has been mixed.”

      http://ftp.iza.org/dp3691.pdf

      Of course, this was four years ago (and it was provisional). However, their conclusion about literature evidence was confirmed by a major review which took part in 2010 (see faqs above “Do market forces in education increase achievement and efficiency?” ).

  13. It appears, then, that an earlier working paper by Bohlmark and Lindahl found no impact on medium or long-term educational outcomes of an increase in private school share. Of course, that paper was only provisional. It was published to stimulate discussion as was the IFAU paper which is, nevertheless, being treated as if it were a paper which had been subjected to rigourous academic peer review.

    As stated in faqs, a review of the evidence in 2010 found that the evidence linking market forces with educational outcomes was “inconclusive”. In 2011, the OECD said the evidence was “uncertain” (Economic Review of the UK 2011). In November 2011, two Harvard professors also looked at evidence from 1998 to 2011 and wrote:

    “Market-based reforms such as school choice or school vouchers have, at best, a modest impact on student achievement…This suggests that competition alone is unlikely to significantly increase the efficiency of the public school system.”

    http://www.economics.harvard.edu/faculty/fryer/files/effective_schools.pdf

    • Thanks for these Janet. While entering caveats etc., is standard academic practice to make sure that one is not over claiming based on the evidence and methodology, I’m intrigued by the claim to provisionality. When one reads the conclusion of the latest IFAU paper it seems far less about stimulating discussion than drawing quite firm conclusions. I find this quite ambiguous. It’s almost as if the authors are content for the ‘discussion paper’ to be read as research (in the peer-reviewed sense) but have entered the various caveats as an ultimate defense should they be challenged.

      • And another thing. Is there a place for “conclusions” in a ‘provisional discussion paper’ ?

        • Tony Chuan says:

          Yes.

          Working Papers and their conclusions, obviously, represent preliminary work and are circulated to encourage discussion.

          Mikael Lindahl is a Royal Swedish Academy of Sciences Research Fellow.

          Citation of working papers should account for their provisional character. A revised version may be available directly from the author. In the case of the paper I quote clearly a great deal of discussion and consultation has taken place, since it has already been revised:

          ‘We are grateful to Anders Björklund, Peter Fredriksson, Erik Grönqvist, Eric Hanushek, Lena Hensvik, Caroline Hoxby, Markus Jäntti, Matthew Lindquist, Dan-Olof Rooth and Jonas Vlachos for valuable comments and suggestions. We thank Eskil Forsell and Arvid Olovsson for excellent research assistance.

          We would also like to thank participants at the CESifo/PEPG economics of education
          conference in Munich, the IFN workshop on privatization in Vaxholm, and at seminars in Gothenburg, Stockholm (SOFI) and Uppsala (IFAU)…….’

          If you have any concrete points to make with the regard to the revised discussion paper “Does School Privatization Improve Educational Achievement? Evidence from Sweden’s Voucher Reform,” (IZA DP 3691). and its conclusions, I am sure its author would be delighted to hear from you.

          That is, of course, the point of a discussion or working paper, to permit peer review to take place.

          By the way, criticisms of sources referred to on this very non academic forum would carry a lot more weight if they were applied across the board in a balanced fashion rather than simply directed at papers with which commentators do not happen to agree.

          I’m sure all will agree that objectivity and impartiality requires nothing less?

      • Keith – I have in the past quoted Bohlmark’s and Lindahl’s preliminary findings in 2008 but when I did I made it clear that it was a working paper for discussion purposes. Such working papers are a useful addition to the debate but shouldn’t be considered as definitive as, say, an extensive view of the research literature undertaken for the OECD (see faqs above). Even in the latter case the OECD wrote:

        “OECD Education Working Paper Series. This series is designed to make available to a wider readership selected studies drawing on the work of the OECD Directorate for Education…The opinions expressed in these papers are the sole responsibility of the author(s) and do not necessarily reflect those of the OECD.”

  14. The IFAU working paper is likely to be quoted again and again and again to support competition between schools and profit-making schools. Here’s a Telegraph blog in praise of Michael Gove’s “art of politics”:

    “A massive study in Sweden has just clearly established that competition between state schools lifts standards in all.”

    So one working paper which looked at results in just two subjects is publicised heavily while at the same time reviews (eg on behalf of the OECD, or by Harvard professors) of a wide range of existing research into market forces and educational outcomes which shows the evidence is actually “inconclusive” are ignored.

  15. Tony Chuan says:

    CESifo Working Paper No. 3866, dated 19 June 2012, is a substantially revised and updated version of the working paper “Does School Privatization Improve
    Educational Achievement? Evidence from Sweden’s Voucher Reform,” (IZA DP 3691).

    It has been peer reviewed in some detail both by individual scholars and at a number of different conferences and seminars.

    There is no reason, therefore, why its findings are likely to be of any lesser value than research conducted by the OECD, particularly since CESifo Working Paper No. 3866 draws on the very latest information available.

    Furthermore, one of the main conclusions that it draws, that only recent evidence of the efficacy of Swedish educational reforms is useful due to the inevitable time lag in those reforms taking effect, is so obvious as to be incontestable.

    It is plain common sense.

    The jury is out on coalition education reforms in this country and will remain so until 2020.

    • Tony you clearly don’t understand peer review for academic publication. There are different approaches but essentially it’s a double-blind process where a paper is sent out to 3 internationally recognised experts in the same field as the research paper. That is experts who have published themselves in the field. They then give detailed feedback and recommendations for major or minor revisions. Even minor revisions can mean significant changes required before the paper will be considered for publication.There are some variants on this model but the paper in question here has not undergone such rigour. Inviting your academic mates to a symposium at a conference to discuss your work is quite different. Of course this too can be quite involved and get quite critical. The key difference is that as an author or organisation self-publishing, you can choose whether or not to revise your research paper in the light of peer critique at a conference. However, to get your paper published in a high quality scientific journal if there was any doubt that your methodology and data did not support your claims or conclusions, it would not be acceptable for publication.

      Of course I acknowledge that there has been published, peer reviewed research that has slipped through this process and should not have. However, the paper in question it would seem has not even been submitted to this process as yet.

      With regards your smokescreen about the efficacy of coalition education policy requiring ten years to take effect, again I beg to differ with such an uncritical view. Many schools are finding their budgets cut whilst others are being funded to open new schools with unproven demand for places; you don’t need to wait ten years to conclude this is wasteful. Also thousands of young people were in Gove’s own words treated unfairly over the marking of their GCSEs last year on his watch. Parents and children won’t accept being told that they have to wait ten years to find out if education reforms have worked. I expect for my own children and other people’s to know that the examination system works fairly now. Not in ten years time. I also expect a broad, balanced and engaging curriculum for our children now. Not in ten years time. Clearly the full effect of some reforms can take time but applying this argument to all reforms is anything but common sense.

      • Keith – it appears that the “ten years time” argument is ignored when it suits. Tony said “it’s plain common sense” to wait for a time lag and he’s correct in some cases (eg an early years policy can only really be judged when the pupils have passed through compulsory education). However, when it comes to judging the academies programme then politicians were claiming success after a very short time as I point out here:

        http://www.localschoolsnetwork.org.uk/2012/03/deception-about-academies-has-been-going-on-since-they-first-opened/

        Gove et al consider the academy programme a resounding success despite there being very few academies which are ten, even five, years old (and the success is not borne out when the stats are analysed – see Henry’s faq above). Few converter academies are more than two years old but I expect the “wait ten years” argument will be ignored when the 2012 school performance tables are published if converter academies, which were mainly outstanding or good schools before conversion (and many are selective), are found to have performed better than non-academy schools. These same politicians will forget that the converter academy 2012 cohort taking GCSEs (the results of which are disputed) spent most of their secondary education in non-academy schools.

      • Tony Chuan says:

        Things move a bit more quickly these days than you are perhaps prepared to acknowledge.

        Guides to peer review, for example from the Research Information Network, are, of course, readily available online:

        ‘The internet has brought new ways of doing research, and communicating and evaluating its results……papers are circulated and read before they have been subject to peer review. In most (but not all) cases, the papers are then peer reviewed – and often revised – before being published in a scholarly journal……Some journal publishers are
        now using web technologies to enable readers to add comments, notes and ratings to individual articles, as signals to subsequent readers. Such developments are
        welcomed by many, and have led some to suggest that peer review prior to publication should be abandoned.’

        Be that as it may, the heavily revised and updated paper that I refer to is still within the review process and so you are welcome to make any criticisms of it direct to the author.

        If you had any such criticisms, I’m sure you would have communicated them by now.

        Use of pejorative words such as ‘smokescreen’ betray a certain partiality on your side and do you no credit.

        It is clearly impossible to categorically state that the coalition reforms are wasteful until the end product is clear and a detailed, and indeed properly reviewed, report issued.

        You are happy to dismiss research until it has been rigorously reviewed.

        You should apply the same standards to a set of detailed educational reforms that enjoy cross party consensus and have only recently commenced.

        Otherwise some might accuse you of a lack of academic rigour and that would never do.

        You will, I know, share my delight that my children are thriving on the current examination system in this country.

        • So according to you ‘things move a bit more quickly these days than you [I] are perhaps prepared to acknowledge.’ And yet we have to wait ten years before we are allowed to critically examine the outcomes of coalition education policy. Hmm. I’d say you’ve just illustrated Janet’s point above perfectly.

          As for the quotation from the RIN the whole piece is actually about the issues brought about by self-publishing and the Internet. Note the careful use of language in the sentence ‘have led some to suggest that peer review prior to publication should be abandoned.’ ie this is not a widespread view and on the whole, whilst coming under pressure peer-review pre-publication of academic research in respected journals remains the norm.

          • Tony Chuan says:

            I think you are getting in a bit of a muddle here.

            I am not suggesting that ‘the “ten years time” argument’ should be ‘ignored when it suits’, quite the reverse.

            In similar fashion, I am not suggesting that ‘peer review prior to publication should be abandoned'; simply that working papers are now much more widely disseminated and commented on than hitherto thanks to the information revolution.

            A working paper that has been disseminated widely for comment over a four year period, revised and updated with the latest research over that period, is no longer lightly to be dismissed.

            Regarding Academies, I very much agree with commentators on this forum who continually point out that it is far too early to make any sensible judgements about their performance.

            Expecting politicians to be consistent is to inhabit a world created by Charles Lutwidge Dodgson.

            The rest of us must live by higher standards, must we not?

  16. “…if this report makes nothing else clear, it is that no such magic bullets exist at an international level – or at least that they cannot, as yet, be statistically proven.”

    Major report, published by Pearson and written by the Economist Intelligence Unit, commenting on what factors contribute to a successful education system from an economic and social point of view (not just relying on test scores in two subjects).

    I doubt this will stop individual politicians and supporters from claiming they know the answer.

    http://thelearningcurve.pearson.com/the-report/executive-summary

    • Tony Chuan says:

      Very interesting, sensible, report which very much confirms what would seem, intuitively, to be obvious:

      ‘The understanding of what inputs lead to the best educational outcomes is still basic, which is not surprising given that robust international benchmarking figures are few and often of recent date. Moreover, education remains an art, and much of what engenders quality is difficult to quantify.

      General lessons to be drawn, then, are often still basic as well. Dr Finn says of studies looking at high-performing school systems, “I don’t detect many similarities other than high standards, solid curriculum, competent teachers and a supportive culture that is education-minded.” Other research might point to the importance of school choice and school autonomy…..

      What Professor Hanushek says of improving autonomy and choice applies generally: “Local countries and institutions are extraordinarily important. Each country has its own system. It is difficult to take any of the specifics and apply them elsewhere.”

      In seeking those solutions, officials also need a dose of humility, remembering that formal education can do only so much.

      As Professor Woessmann notes, “a lot of these things [determinants of academic success] are not amenable to government action. They are really within families and how society operates.” Moreover, as the differing approaches of Finland and South Korea show, there are diverse paths to success…..

      This study, like others, ends with an appeal for more research’

      The academic jury is out on what kind of educational reforms are most likely to benefit the state educational system in this country.

      Cross party support for coalition education reforms currently in progress dictates that some time, will elapse before any kind of revision is either likely or practicable.

  17. [...] Also something to consider is “other evidence on free schools indicates growing social and ethnic segregation, and intensifying issues regarding admissions” Source  [...]

  18. [...] Also something to consider is “other evidence on free schools indicates growing social and ethnic segregation, and intensifying issues regarding admissions”  Source  [...]

Want to follow comments on this post? Use the RSS feed or subscribe below

Reply

CAPTCHA Image
*