New research claims to show "for-profit" schools are great, but can it be trusted?

Francis Gilbert's picture
 19
The Institute of Economic Affairs (IEA) new research publication, Schooling for Money: Swedish Education Reform and the Role of the Profit Motive looks at the role of for-profit schools in Sweden and for the first time attempts to provide valid quantitative evidence regarding how these schools perform. The results claim to show that the competition that drove improvements in the Swedish system was only possible because of the high number of for-profit schools that were established.

However, examination of these statistics reveals that they are probably too clever for their own good. The IEA has used very sophisticated statistical models to show that for-profit schools significantly improve children's results. Many statisticians would question their results because they have used "control mechanisms" which claim to equalise children from different backgrounds in an attempt to compare “like for like”. The trouble is that when you compare a child from a "for-profit" school, it's very difficult to find his/her equivalent in a "non-profit" school. For example, boys of the same age and same social class in the two types of school may look like they are comparable, but they may not be because one may come from a highly aspirational background (as many children in "free" schools do) and the other may come from a severely "depressed" background. This research attempts to eradicate such problems by sprinkling some statistical "magic dust" over the data in the form of a series of formulas which attempt to iron such issues. In other words, they have used puzzling methods in their attempt to compare children from different social backgrounds and different schools. They claim to have statistically resolved the differences between these children so that their comparisons between the achievements of children in for-profit schools can be validly compared with children in non-profit schools. However, have they really done this? The methods they’ve done this by are very sophisticated indeed and must be taken with a serious pinch of salt. The trouble is that these sophisticated statistics tend to manipulate the data in ways in which make the ensuing results unreliable.

Other reports into the Swedish free school system tend to show that for-profit schools lead to increased social segregation and indifferent results across the board. The fourth report on Pisa (Programme for International Student Assessment) published this month concluded: “Countries that create a more competitive environment in which many schools compete for students do not systematically produce better results…While students who attend schools that compete with other schools for student enrolment perform better than students who attend schools that do not compete with other schools, the cross-country analysis suggests that systems as a whole do not benefit from higher rates of school competition.” Research from the IOE on Sweden's 'free school' reforms suggests that the entry of new schools had a positive effect on pupils' academic achievements. But according to a survey of the evidence by Rebecca Allen, the benefits are small, they are predominantly focused on children from highly educated families and they do not persist: scores are no higher in the end-of-school exams.

Perhaps it comes as no surprise that the IEA is financed partly by donations from corporate firms and always attempts to promote the cause of the free market in whatever sphere it’s investigating. I appeared on Jeff Randall's News Programme on Sky News talking about this issue with Mark Littlewood,  Director General of the IEA tonight at 7.50pm. He attempted to claim the research was valid, but had to concede when I challenged him that the Swedish free school system didn't compare favourably with ones like Finland in the recent Pisa survey. The presenter, Anna Jones, asked me what I thought of allowing private companies to make a profit in the education sector; I replied that it was highly problematic for taxpayer's hard-earned cash going into the pockets of fat cats. Littlewood admitted that there was a gut instinct in the British public against this idea but said that we are not so emotional about private firms making profits out of food. I countered by saying that our children are not cans of beans -- a phrase taken from one of Fiona Millar's excellent Guardian Education columns. That was all I had time to say, but I felt it made an important point about why the free-market won't work in schools. Unlike cans of beans which can be returned to the factory if they are defective, children and teachers can't be binned; it is very difficult to close down failing schools, even for-profit ones, because real lives, not products, are affected.
Share on Twitter

Comments

Gabriel H. Sahlgren's picture
Thu, 16/12/2010 - 16:37

As the author of the paper that Mr Gilbert discusses, I am rather flattered by his accusation, which appears to be that my models are ‘too sophisticated’ – but if there is any problem, it is that the models I employ might not be sophisticated enough. On page 16 in the paper, I discuss the problems of ‘endogeniety’ - it might be the case that even after employing a vast range of control variables, students in free schools may be better (or worse!) than students attending municipal schools, have more/less motivation etc. That is, if we fail to control for important uncontrollable variables that affect performance, our estimates will be biased. However, the most recent study I refer to (Tegle 2010) suggests that not taking this into account significantly underestimates the effect of free schools – the effect is much stronger when utilizing more sophisticated models that explicitly are supposed to take into account the problem of endogeniety. My paper is very clear on that point – the estimates presented are likely to err on the side of caution.

The reason I didn’t attempt to employ models accounting for endogeniety, which according to previous research would show even stronger effects of for-profit/non-profit schools, was three-fold. Firstly, I was mostly interested in analyzing whether the ‘deterioration thesis’, - that the profit motive drives down educational quality – held up to empirical scrutiny. As endogeniety was highly likely to bias the estimates against free schools, finding that my models displayed positive effects was thus enough to strongly indicate that the ‘deterioration thesis’ was not accurate (especially since 65% of all free schools are for-profit). Secondly I didn’t have enough time. Assembling the statistics for about 1,500 schools over 5 years, and coding all schools by ownership structure, was time-consuming. Employing more advanced models necessitate further variables. Thirdly, as I analysed school-level evidence, I was concerned that it might be difficult to control for endogeniety entirely.

However, after I had completed the paper, I analysed this further. Using models specifically designed to take into account endogeniety (so called Instrumental-Variable models), I find that, in line with Tegle’s (2010) findings, post-reform for-profit and non-profit schools, on average increase the GPA by about 34 points , which is much stronger than the effects I find in my paper (around 5 points in the model with the overall estimates). For this analysis, I have only used data in 2009 due to time constraints. Again, this confirms my argument that the estimates presented in the paper err on the side of caution. The models employed there simply underestimate the effect of for-profit schools – but this is something I anticipated. I was mainly interested in whether for-profit schools drove down quality and found no evidence that they were.

Mr Gilbert argues that ‘Many statisticians would question [the IEA’s] results because they have used “control mechanisms” which claim to equalise children from different backgrounds in an attempt to compare “like for like”.’ However, statisticians would not question the use of control variables (which is essential to statistical evidence), but potentially that I have not controlled for enough variables. But the statistical models described above is specifically designed to control for the variables, which are difficult to control for (such as motivation, ability etc.). And when one does, the effects of for-profit schools are even more positive.

Thirdly, regarding the PISA results to which Mr Gilbert refers one must take into account other reasons for why countries with more competition do not perform better on average – again, we meet the problem of not controlling for enough variables. For example, the Swedish voucher programme in 1992 was introduced during an economic crisis. Residential segregation and unemployment increased during the 1990s while school budgets were cut. Meanwhile, we introduced a new grading system, which removed a ‘relative system’, which was based on a bell-curve. Furthermore, we began to transfer more responsibility from the teacher to the student (allowing more students to work on their own), which the international research shows has been negative. A perfect storm of causes contributed to problems in Swedish education during the 1990s.

My argument is that given that hitherto presented research – which I have extended to include for-profit free schools – finds positive effects of school competition in Sweden, these problems would probably have had a larger negative impact on educational performance had it not been for school competition. The evidence of declining performance in international ratings during the 1990s is bad news for those who argue that school competition is a panacea. Most of us who do not would still conclude that competition was beneficial on the margin.

The next step for those wanting to question my findings is to provide new research showing that for-profit free schools in fact are bad. So far, no evidence showing that for-profit free schools in fact are bad has been presented, and I therefore conclude that fears of the profit motive in education reform, at least in the Swedish context, are unfounded.

Gabriel H.Sahlgren

Author of the paper ‘Schooling For Money: Swedish Education Reform and the role of the Profit Motive’

Francis Gilbert's picture
Thu, 16/12/2010 - 17:55

Thanks for this Gabriel. You have made some fascinating points here. I suppose we're running up against the problems of "quantitative methods" assessing pupils' attainment: I am happy to concede that there are issues with the Pisa as well as your research. Controlling the variables is fiendishly difficult. Perhaps combining quantitative and qualitative research better informs people of the truth of the situation. The Pisa, of course, does have the advantage of being a much bigger, cross cultural survey. Its findings are fairly clear that a free school system doesn't drive up standards.

I am interested in your comment: "The evidence of declining performance in international ratings during the 1990s is bad news for those who argue that school competition is a panacea. Most of us who do not would still conclude that competition was beneficial on the margin." What do you mean it's "bad news" for people who argue school competition is a "panacea"?

Philip's picture
Thu, 16/12/2010 - 17:18

This is a strange post. You should note that the IEA, by a long way, does not get most of its funding from corporate firms. I do not know where you have got your figure from but it is wrong. Secondly, I am surprised that you find the statistical methods sophisticated and if the studies you quote use less sophisticated methods then it is those you should call into question. Thirdly, Finland is a well-known outlier in all education surveys and an odd country with which to make comparisons. Fourthly, most surveys of Swedish schools find benefits for all - some surveys have found relatively few benefits. It is strange to interpret such a set of results negatively. If, on a scale of -10 (poor outcomes) to +10 (good outcomes) the surveys tend to produce results for free schools around the -2 to 8 mark it is ridiculous to interpret this negatively. Finally, your baked beans analogy is very wide of the mark. When most firms go out of business, the owners change, the management changes, the capital stays in place but it is managed differently. Your analogy does not even work for baked beans. When Safeways was taken over by Morrisons nobody sent the baked beans back to the factory. The stores were not closed down. The customers were not sent home. The main potential for free schools to work is with the poor in Anglo-Saxon systems that are so state dominated and middle class sharp elbows are best able to get the best out of the system. This is what the randomised statistical surveys show in the states.

Francis Gilbert's picture
Thu, 16/12/2010 - 17:45

Thanks for this Philip. Out of interest, does the IEA get any funding from organisations that are connected to, or run by "for-profit" schools? Perhaps my analogy doesn't stand up to sustained scrutiny, but the general point I feel is still good: it's proved very difficult to close down or improve "failing" schools. They enter a cycle of decline where there's an unbridled free market. The success of the London Challenge, where schools were encouraged to co-operate, shows that co-operation drives up standards more successfully than competition.

Francis Gilbert's picture
Thu, 16/12/2010 - 18:05

To Philip, I have changed the wording to say that the IEA is financed "partly" by corporate firms. Is this correct?

Philip's picture
Thu, 16/12/2010 - 18:10

Francis - we keep the programme entirely separate from fund raising (except where it is through charitable trusts) and accept no tied corporate funding. As such, though I meet individual donors, I never see a list of all donors. I do know that the number of corporate donors is small, that the proportion of our income from corporate donors is small and I have never come across any corporate donor connected with for-profit schools. In the interests of integrity I will not go quite as far as saying that there are none, but I would be surprised if there were. I certainly had no knowledge of any when I asked Gabriel to work on this subject.

Regarding the closing down issue, this is probably less of a problem if you have profit making schools. Trusts can be very slow to react to signals until it is too late (and as for the state...!). One can point to individual examples of state schools improving by cooperation but, clearly, after several decades of trying, this approach does not generalise as a way of sorting out poor schools. The point about price signals is that we see them and react to them before we become bankrupt. Even if we become bankrupt, if there is physical capital there (and human capital including good teachers etc) a takeover is much more likely than closure. Closure, change, change of management etc is an issue whatever the ownership model - it does not become more difficult under proprietary ownership. It is rare for a company in any sector to shut down, demolish its buildings and leave people around wondering what used to be there - this is likely to be even less so in the schools sector. By the way, there are quite a few for-profit private schools and there used to be a lot before the war (I believe that Inheritance Tax made them difficult to pass on within families), I am not aware of serious problems arising with regard to closure etc. But, as I say, the problem does not go away whether schools are run by charities or the state.

Philip's picture
Thu, 16/12/2010 - 18:32

It is financed partly by corporate firms - not a penny of government money!

Andrew Old's picture
Thu, 16/12/2010 - 20:22

Hmmmmm. Difficult with research like this to work out what the effect is. It appears to be a few points on a GPA that is on average over 200, but I can't seem to find anything about the variance and can't help but think of John Hattie's claim that education research finds a significant large positive effect size for almost all educational changes.

I suppose that, even if the benefits are marginal, it does, nevertheless, undermine the idea that the policy is harming achievement in Sweden, or that competitition or profit is generally harmful, but that is quite far from justifying whatever the costs involved in introducing the system were. (Are costs mentioned in the report? I haven't the time to read it all.)

More importantly for discussion in the UK, if the benefits from a move to a centralised monopoly to very open competition in Sweden have been modest, then why would we expect much benefit at all in a country where there is already a lot of competition and a significant private sector?

I don't want to repeat Francis' ad hominems, but I can't help but think the data falls short of the report's conclusions and, had it been in a peer-reviewed journal rather than published by a free-market think tank, it might reasonably have concluded that the small size of the effects, rather than their mere existence, was the real importance of the research.

Gabriel H. Sahlgren's picture
Fri, 17/12/2010 - 10:37

Francis,

1.I wouldn’t say that the cross-country survey method is an advantage as you then has to take into account various country characteristics, country trends, economic situation etc – making the enterprise increasingly difficult. You say that it is difficult to control for variables, which is true. However, it is much more difficult to do so in cross-national surveys for the reasons I just stated. Therefore, assessing whether competition has been successful should not be done through international assessments like PISA, UNLESS they are specifically designed to compare assess competition (PISA is not), but rather in within-country studies.

2.Sure, combining qualitative evidence with quantitative evidence is fine in most cases. But when it comes to attainment, it is difficult to assess scientifically without utilizing quantitative techniques.

3.What I meant by the comment is that given that since Sweden’s performance in international ratings appears to have declined since the 1990s, it is difficult to argue that competition can do miracles amid the exceptionally difficult situation Sweden was in – which I outlined in the comment above. That is what I meant by a panacea – that competition would increase all students results radically amid all problems and changes that occurred. However, since all academic studies find positive effects, and no study finds negative effects, we can safely conclude that competition has been beneficial on the margin.

Francis Gilbert's picture
Fri, 17/12/2010 - 15:14

The "perfect storm" of conditions in Sweden -- residential segregation, changing schools systems, cuts in school budgets etc -- has happened in London recently, but a recent Ofsted report shows that the London Challenge, which encouraged co-operation between schools, significantly improved results, despite all these adverse conditions. I wonder if anyone has any thoughts on that report. I've blogged about it previously.

Francis Gilbert's picture
Sun, 19/12/2010 - 18:17

For Gabriel and Philip, I've been researching this matter still further and would like to offer these points to question the nature of this research.

1. Chapman and Chapman (1973) stated that there is no statistical method
that can address the question of whether two groups that differ on variable A would differ on variable B if they did not differ on variable A" (quote from Miller and Chapman, 2001). If we were to interpret B as GPA, and A as one of the variables you are trying to "control" for in your research, such as people per km2.

2. Fleiss and Tanur, 1973, cited in Miller and Chapman, 2001 state: "No amount of statistical manipulation can tell what might have been had certain differences been non-existent.... the overwhelming weight of logic is on the side of those who warn that neither analysis of covariance nor any other statistical technique can undo systematic differences which were outside the investigator's control."

I'd like to suggest that correlation is not the same as causation, and that no statistical technique, however sophisticated, can prove otherwise.

I would either of you had any thoughts on these important quotations which do question the way you have controlled the variables in the research.

Philip's picture
Sun, 19/12/2010 - 21:02

I am still a bit puzzled (to say the least). We have a position that argues that parents should be denied autonomy (the existence of a private sector in the UK does not falsify that - the private sector is not realistically available to more than 15% of the population at most and is distinct from the public sector rather than competing with it) and the best that the proponents of that position can come up with (as far as I can see) is that some experiments in cooperation worked (competitive systems involve cooperation too) and that the very convincing results that profit making schools and parental autonomy produce might not be (in the worst case scenario) quite as convincing as some would suggest. If you want to deny parents autonomy then I think you have to produce convincing results that it does serious harm. The evidence suggests the opposite (especially in the US which is more like the UK than Sweden). What is the harm that parental autonomy in Belgium, Netherlands and so on also does? The position of people like Fiona Miller is a radical social engineering position; most of the other proponents of the (broadly defined) status quo seem to be simply conservatives (with a very small c).

Francis Gilbert's picture
Sun, 19/12/2010 - 22:06

I think there's a great deal of evidence that free schools cause quite a bit of harm, increase social segregation and lower standards. You only have to look at the blogs on this site to see how divisive the free school movement is in the areas that it's being set up. Even Michael Gove has drawn the line at allowing "for profit" schools because he recognises that allowing private companies to make profits out of taxpayers' money is not going to solve anything and will cause even more divisions in the system than there are at the moment. What is needed is more co-operation between schools to raise standards and a fair admissions' system.

Fiona Millar's picture
Mon, 20/12/2010 - 12:18

Could you define what you mean by 'parental autonomy'?

Gabriel's picture
Mon, 20/12/2010 - 18:14

First, quoting two psychologists (certainly no experts regarding quantitative methods) discussing psychological experiments is not exactly the references you want when arguing that an econometric specification in a study on free schools is wrong. Second, even if statistical evidence is completely worthless (a statement that would invalidate pretty much all of mainstream economics and a significant portion of sociology and political science), I don’t understand how this reference can be used in defence of a position stating that free schools are bad? You still haven’t supplied any evidence that for-profit free schools (or any free schools) in fact are lowering standards.

You argue that ‘I think there’s a great deal of evidence that free schools cause quite a bit of harm, increase social segregation and lower standards.’ Okay, please supply this evidence and we can discuss it. Arguing that free schools and school competition are divisive and that this is evidence of their negative impact is the same as arguing that free schools and school competition are negative because people think that they have a negative impact. I wouldn’t call this evidence, but rather unsubstantiated fears.

Of course, correlation does not prove causation. This applies to qualitative as well as quantitative evidence. But when there is no correlation whatsoever, most people would argue that a hypothesis can be rejected (or at the very least supply an argument for why the correlation may not show up). From what I understand from the arguments supplied here, the Local Schools Network claims that for-profit schools are bad and that they will drive down quality. Apparently, no statistics can be supplied as evidence against this claim. Now, I want to relate this argument to your comment to your post on reading standards. There, you argue that ‘The SATS results are “raw data”, which have not been statistically manipulated at all’ and that the PISA study (which you very recently supplied as evidence against my paper), which shows declining performance for the UK, is biased because it ‘manipulates its test data’ by controlling for family background, ethnicity etc (but not country-specific variables as I pointed out earlier). I take this as you prefer to look at the data directly and determine whether improvements in reading standards have occurred.

Now, applying this argument to Swedish free schools and school competition, as both the descriptive statistics, i.e. the ‘raw data’ (which shows a stronger impact than my models), and the models (accounting for endogeniety or not) in my paper, as well as in all other studies on the subject of Swedish school competition, show that there is no evidence for a negative impact of for-profit schools or school competition (but rather a positive impact), why should we conclude that for-profit schools and school competition drive down quality? The answer is that there is no logical argument for why we should.

I conclude that you use different standards regarding what you consider valid evidence. In the case of reading performance in the UK, looking at the raw data of SATS is apparently enough to validate your argument that reading performance has increased. But in the case of Swedish school competition and for-profit schools, the raw data is suddenly not enough, and neither are statistical techniques, regardless how sophisticated. Suddenly, all statistics is flawed and cannot disprove the position that school competition is bad. Therefore, appears to be the conclusion, you are right. Such a position is untenable.

Philip's picture
Thu, 23/12/2010 - 15:23

Francis - i think your arguments are getting more bizarre over time. "You only have to look at this blog to see how divisive free schools are in the areas in which they are set up"? Well, I suppose if you supply all parents with a single school and prevent them from making any difference to the schooling they receive they will be compliant in a certain sort of way but not in a very satisfactory way. Surely the argument you use is just as strong an argument in favour of stopping people from complaining about free schools (ie having free speech) as it is from stopping free schools. If people could not complain they could not behave in a way that sows seeds of division. If this blog did not exist, perhaps people would find less cause for complaint. People are going to disagree, that is a fact of life. But you have moved on from saying that free schools should be stopped because they do harm to saying that they should be stopped because they upset people and people think they may do harm (even if they don't). Perhaps when free schools are run of the mill so that people do not even think about the fact that they are "free" (extraodindary that we have come to the state that it is something odd for a school to be free) that divisiveness will disappear.

Fiona Miller- by parental autonomy I meant giving parents control over their childrens' education rather than having children divided up by where they happen to live and allocated to a school by bureaucratic fiat. All children are different and parents should be allowed to take account of that.

Francis Gilbert's picture
Thu, 23/12/2010 - 16:26

I think it's fair enough to hold these schools to account in the ways that we see fit and exercise our right to freedom of speech, Philip. I think what people are getting upset about is the fact that it seems very unfair that certain schools get more money than their maintain peers, can gerrymander their admissions' procedures (cf WLFS), seem to be accountable to no one, can employ untrained teachers, and effectively take resources away from other schools. That's what people are complaining about on the blog.

Philip's picture
Thu, 23/12/2010 - 21:25

well, let me be clear. I simply want a system where all funding is directed through parents - for all parents - who are then free to educate their children according to basic principles set out in primary legislation. If that funding is higher for poorer parents I have no particular problem with that as long as the basis is transparent. I should also be clear that I believe in the free schools policy only as a possible route to that objective (and it would not have been the route I would have designed myself). It is quite possible that the policy will lead to some of the problems you suggest - with a half way house you often get the worst of all worlds rather than the best. It is also quite possible that the policy enacted half-heartedly will lead to free schools only for a few, which is the main reason I would like profit making schools to be able to enter. If this slow, pragmatic policy is botched it could set the agenda of parental autonomy back decades. So I quite accept that some of the problems you suggest are problems caused by a half way house - that is possibly a point of agreement.

Rob Billson's picture
Tue, 06/09/2011 - 09:54

Its easy to cherry pick evidence that supports ones prejudices. Some evidence from Sweden appears to support for profit schools, although educational outcomes in Sweden are only a bit better than the UK. The Finnish education system produces the best educational outcomes in europe and yet private and for profit schools play very little part in the system. Finnish authorities haven't outsourced school management to for-profit or non-profit organizations, or introduced performance related pay, or league tables. Instead they have invested heavilly in the public education system, reduced class size, boosted teacher pay, and introduced a rigorous master's program that teachers must complete. That being said there is a lot of autonomy for teachers, much less political interference. see : http://www.oecd.org/document/61/0,3746,en_32252351_32235731_46567613_1_1...

Add new comment

Already a member? Click here to log in before you comment. Or register with us.