Reading standards in our pupils have risen massively in the last decade

Francis Gilbert's picture
 16
Pupils' reading skills have improved massively during the last fifteen years thanks to the great work our primary schools have been doing. In 1995, 49% of pupils attained a Level 4 in reading in their SATS; this year it was 81%. Since Level 4 is the grade that teachers would expect 11-year-olds to get, this shows that standards have improved very dramatically. This news shouldn't be ignored amidst the furore stirred up by Michael Gove about the low standards amongst our most socially disadvantaged communities. It's clear that our poorest pupils are still struggling to improve. The fact that he has abolished one-to-one reading catch-up help won't help things in this regard. In my borough Tower Hamlets where my son goes to school, as with much of inner-London, the percentage of boys attaining below a Level 3 is not as high as many similar boroughs because resources have been used to address the problem.

However, let's not forget the vast majority of pupils are doing very well. From the reports that the government is issuing you'd think that all our state schools were failing their children. Nothing could be further from the truth. Our schools are doing extremely well at improving reading standards. These results are more reliable than the PISA -- which uses questionable techniques for comparing countries against each other. Instead of knocking our schools for failing our children, we should be celebrating their great success and helping them address the continuing problem of under-achievement in our most disadvantaged communities. These results show that schools that are supported achieve great things.
Share on Twitter
Category: 

Comments

Michael Tidd's picture
Sun, 19/12/2010 - 23:14

And significantly - as Warwick Mansell pointed out this week - some 45% of 11-year-olds are apparently reading at the level expected of a 14-year-old!

Andrew Old's picture
Mon, 20/12/2010 - 07:34

SATS results are more reliable than PISA?

In which parallel universe?

Francis Gilbert's picture
Mon, 20/12/2010 - 09:57

They are much more reliable than the PISA results because they are "raw" data, tried and tested over twenty years of marking, unlike PISA which has only been running a few years. The PISA data -- as we saw from a previous post -- use various complex statistical techniques to control the variables invovled in order to compare pupils across a range of countries: it is notoriously unreliable statistically speaking. Many statisticians won't accept PISA as valid at all. The comments made at the end of my post on "for-profit" schools apply to PISA as much as the dodgy stats provided by the IEA.

Andrew Old's picture
Mon, 20/12/2010 - 12:48

I think you are confusing the use of PISA for comparisons between countries and its use to track performance within countries. As I understand it PISA has the advantage in that it doesn't change much each time allowing direct comparison between different cohorts. By comparison SATs change every year and are generally thought to have got easier.

Francis Gilbert's picture
Mon, 20/12/2010 - 13:08

I was talking about the ways in which PISA manipulates its test data in order to make "like for like" comparisons; for example, PISA controls its variables by factoring in things like ethnicity and gender when comparing results. The SATS results are "raw data", which have not been statistically manipulated at all. I accept that the SATS tests change from year to year, but the levelling process has remained the same since they were introduced 15 years ago.

Gabriel's picture
Mon, 20/12/2010 - 18:13

First, quoting two psychologists (certainly no experts regarding quantitative methods) discussing psychological experiments is not exactly the references you want when arguing that an econometric specification in a study on free schools is wrong. Second, even if statistical evidence is completely worthless (a statement that would invalidate pretty much all of mainstream economics and a significant portion of sociology and political science), I don’t understand how this reference can be used in defence of a position stating that free schools are bad? You still haven’t supplied any evidence that for-profit free schools (or any free schools) in fact are lowering standards.

You argue that ‘I think there’s a great deal of evidence that free schools cause quite a bit of harm, increase social segregation and lower standards.’ Okay, please supply this evidence and we can discuss it. Arguing that free schools and school competition are divisive and that this is evidence of their negative impact is the same as arguing that free schools and school competition are negative because people think that they have a negative impact. I wouldn’t call this evidence, but rather unsubstantiated fears.

Of course, correlation does not prove causation. This applies to qualitative as well as quantitative evidence. But when there is no correlation whatsoever, most people would argue that a hypothesis can be rejected (or at the very least supply an argument for why the correlation may not show up). From what I understand from the arguments supplied here, the Local Schools Network claims that for-profit schools are bad and that they will drive down quality. Apparently, no statistics can be supplied as evidence against this claim. Now, I want to relate this argument to your comment to your post on reading standards. There, you argue that ‘The SATS results are “raw data”, which have not been statistically manipulated at all’ and that the PISA study (which you very recently supplied as evidence against my paper), which shows declining performance for the UK, is biased because it ‘manipulates its test data’ by controlling for family background, ethnicity etc (but not country-specific variables as I pointed out earlier). I take this as you prefer to look at the data directly and determine whether improvements in reading standards have occurred.

Now, applying this argument to Swedish free schools and school competition, as both the descriptive statistics, i.e. the ‘raw data’ (which shows a stronger impact than my models), and the models (accounting for endogeniety or not) in my paper, as well as in all other studies on the subject of Swedish school competition, show that there is no evidence for a negative impact of for-profit schools or school competition (but rather a positive impact), why should we conclude that for-profit schools and school competition drive down quality? The answer is that there is no logical argument for why we should.

I conclude that you use different standards regarding what you consider valid evidence. In the case of reading performance in the UK, looking at the raw data of SATS is apparently enough to validate your argument that reading performance has increased. But in the case of Swedish school competition and for-profit schools, the raw data is suddenly not enough, and neither are statistical techniques, regardless how sophisticated. Suddenly, all statistics is flawed and cannot disprove the position that school competition is bad. Therefore, appears to be the conclusion, you are right. Such a position is untenable.

Francis Gilbert's picture
Mon, 20/12/2010 - 18:27

I think on the whole it's best to have the data "uncooked" but obviously this isn't always possible. I find the SATs data persuasive because of the nature of the test, compared with the PISA. I guess I'm trying to show there's a "hierarchy of reliability", the SATs for are more reliable than PISA, but the PISA -- after weighing up you and they are doing -- seems a bit more reliable than what you've been doing.

My main quantitative data base for showing that free schools lower standards is from the Stanford Credo report. Here's the press release for starters: http://credo.stanford.edu/reports/National_Release.pdf
Here's a quote from the report: "While the report recognized a robust national demand for more charter schools from parents and local communities, it found that 17 percent of charter schools reported academic gains that were significantly better than traditional public schools, while 37 percent of charter schools showed gains that were worse than their traditional public school counterparts, with 46 percent of charter schools demonstrating no significant difference."

Andrew Old's picture
Mon, 20/12/2010 - 19:07

What I can't work out is what, other than wishful thinking, makes you think that SATs have remained at a consistent level of difficulty.

Gabriel's picture
Thu, 23/12/2010 - 13:00

Francis,

First, I note that you have not supplied evidence against for-profit (or non-profit) free schools in Sweden - which is the voucher programme I was analysing.

Second, you fail to mention key things in the study on US charter schools to which you refer:

1. "For students that are low income, charter schools had a larger and more positive effect than for similar students in traditional public schools." (page 2)

Would you then admit, given that I am quoting the same research you seem to claim is valid, that charter schools favour pupils from disadvantaged backgrounds, and that such schools therefore are important for lifting their achievement?

2. "The report found that the academic success of students in charter schools was affected by the individual state policy environment. States with caps limiting the number of charter schools reported significantly lower academic results than states without caps limiting charter growth." (page 1)

This is just confirmation that the DESIGN of school choice programmes is extremely important for whether it is successful (Arkansas, Colorado (Denver), Illinois (Chicago), Louisiana and Missouri have seen improvements due to school choice). And in this case, it seems that if you want charter schools to work, you don't want caps on them. I suspect, however, that this is not an argument you are prepared to make.

Third, the methodology of this attempt to compare students across different charter schools programmes throughout the US has been questioned by Dr Hoxby at Stanford: http://www.nber.org/~schools/charterschoolseval/memo_on_the_credo_study.pdf

Fourth, the most comprehensive study to date, analysing the New York charter programme (a gold standard study since it is randomised) find that charter schools there increase achievement significantly: http://www.nber.org/~schools/charterschoolseval/how_NYC_charter_schools_...

Finally, I still wonder one thing: Why do you view this quantitative research, which attempts to compare students in different voucher and charter school programs throughout the US (a very difficult statistical exercise indeed), suddenly valid? They employ statistical modelling, which you just have claimed cannot be used for evaluating school performance.

Francis Gilbert's picture
Thu, 23/12/2010 - 16:21

Thanks for this Gabriel. Firstly, to respond to your last point: I think that quantitative research needs to be interrogated and put into context, which is what I was attempting with your research, though I have to admit not extremely well for the simple fact that while I have done a statistics as part of my PhD, I am no expert in the field, using mainly qualitative research methods in my PhD. To justify my critiques to a certain, I would just say that you have to judge each case separately; it's a fine judgement call. Both your study and PISA clearly have "issues" connected with them -- I think you more or less admit this yourself -- as do the SATS; I think it's fair enough to look at them and say, "Well, I have these doubts, but that looks about right.." I liked Andrew's point earlier that perhaps your research needs proper peer reviewing in academic journal (apologies if this has been done, but I couldn't see it on the IEA site). Your findings are obvious very important and could have a major impact upon future education policy so it's going to need this I think.

1. Yes, I concede the point you make about these charter schools, although I have my doubts about the methods that some of these charter schools have attained their results; I personally find the KIPP schools methods of extracting great results quite troubling. But there's no doubt they get them. http://www.francisgilbert.co.uk/2010/04/why-do-our-politicians-think-ame... Please feel free to respond as I find your comments thought-provoking and important to respond to.

2. I absolutely agree with the second point. This is the key issue for me; it doesn't matter what we call these schools, what is crucial is that they have fair admissions and are properly accountable to their communities. Where this happens, as I feel, on the whole, has happened with Academies in Hackney (though not always) this kind of system works; I perhaps differ from Fiona on this point; we're not a political organisation! What doesn't work and is unpalatable in my view is when fat cats make a profit out children's education and are not accountable to the taxpayer in any recognisable way, and institute unfair procedures such as overt and covert selection, unjust exclusions etc to gerrymander their results.

I haven't look at the NY research yet, but will do so.

Thanks for your comments I have found them very useful; they have given me a lot to think about. I guess we're all in the game of delivering a fair education that improves the lives of children from all backgrounds and we mustn't lose sight of that. This kind of debate needs to be had so that fairness is at the heart of the system.

Nick Cowen's picture
Fri, 24/12/2010 - 16:27

Francis: it is funny that you mention the troubling methods for extracting results in US charter schools when that is exactly what seems to have been happening in the UK to achieve those SATS results you've been raving about in this article.

Teachers, independent measures and several experts all seem to indicate that the current SATS regime produces unreliable measures (over time at least) and is damaging to children's general education:

http://www.civitas.org.uk/pdf/deWaal_artificialachievement_June07.pdf
http://www.civitas.org.uk/pdf/FastTracktoSlowProgress.pdf

It seems strange that you use results that are widely recognised to be unreliable (far from "raw") in this discussion while criticising Gabriel for at least attempting something approaching a like-for-like comparison in another.

Francis Gilbert's picture
Fri, 24/12/2010 - 17:27

Thanks for this point Nick. I do think it's important to have a proper discussion about statistics, which, as you know, are highly problematic for lots of reasons. I don't think SATs are perfect, far from it, I was pointing out in this context that the SATs are a more reliable reading test than the PISA; I really felt that the PISA questions were very reductive, and didn't test high-level reading skills. I agree that in some schools there is far too much teaching to the test, but I don't this invalidates it completely as a test. It's not perfect at all. Some of the top achieving schools don't teach to the test though:
http://www.telegraph.co.uk/education/leaguetables/8201090/Primary-school...


I'm quite enjoying teaching "controlled conditions" coursework, which seems to me to be a good half-way house between exam and coursework, avoiding the cheating of much coursework and the narrowness of exams.

My criticism of Gabriel's statistics was slightly different; I was bothered by the way he controlled the variables in his scheme, enabling him to match pupils in for-profit schools with those in maintained schools. The SATs results are not "leavened out" in this way.

Once you get into comparing statistics in this way, you're always going to get bogged down in methodologies etc. Statistics are like blurred, filtered snapshots of a moving target; it's difficult to make out what's going on and you have to use your instincts and prior knowledge to inform your reading them rather than seeing what you're told is there by the person producing the research. Size and scope counts for quite a bit in my view. The SATs results show a massive improvement in reading scores over the last fifteen years; it's a very big picture were getting here, much bigger than Gabriel's research. There's a definite sense that things have improved hugely.

Francis Gilbert's picture
Fri, 24/12/2010 - 17:31

Nick, I will look at your articles at Civitas and think about what you've written here. I haven't seen it before and it looks like it requires a bit of thought!

Nick Cowen's picture
Sat, 25/12/2010 - 14:08

Thanks for such a kind response, Francis. We are certainly all grappling with the same difficulties to work out what works for best for children's education. I've never seen support for choice as inconsistent with supporting great local schools. Our aim is always to try and have a good local school for every child, it is just a question of which mechanisms work best at the level of policy to achieve that.

Francis Gilbert's picture
Sat, 25/12/2010 - 15:10

Yes, I'm interested in this comment: "I’ve never seen support for choice as inconsistent with supporting great local schools." Again, it's getting a balance between choice and co-operation. What's increasingly interesting me is not parental choice about where to go to school but parental/pupil choice about what happens in school: that there is real choices for them to choose the teachers and courses that meet their needs. You can go to a so-called great school and yet have a miserable time because you've got teachers who don't "do it" for you. Realistically, even in a totally free-market system, you're never going to have that much choice about where to go to school, but surely it's possible to empower children/parents to make real choices about the teachers/courses they have. I think this would drive up standards more than a free-market between schools...

In the New Year, I'm interested in blogging about this matter in more depth because it seems to me to be so important.

Nick Cowen's picture
Sun, 26/12/2010 - 13:22

That is an interesting idea, allowing children to choose courses within a school. The question is 'what would be more likely to achieve that at the level of public policy?'. Because if such a scheme is preferable for many parents and children and isn't much more costly, it is the sort of thing that school choice might produce. On the other hand, I can't imagine how Whitehall or even an lea would have the leverage or know-how to mandate a choice of courses within a school.

In fact your idea isn't too far off what some free schools in Sweden offer, such as kunskapskolan amongst others. I describe their school briefly at the beginning of a pamphlet I wrote a couple of years ago: http://www.civitas.org.uk/pdf/SwedishLessons.pdf

Add new comment

Already a member? Click here to log in before you comment. Or register with us.