The stupidity of school statistics

Michael Dix's picture
It has been pointed out in other articles here how the recent GCSE fiasco creates a paradox for the government.

Put simply, the government wants the percentage of children achieving the highest grades to stop rising each year, something it has signalled to the exam boards if not directly instructing them. At the same time the government want the percentage of children achieving the highest grades in each school to go up. So, unless the schools already achieving a good number of A to Cs start doing less well, any increase will increase the numbers getting higher grades. There's the paradox.

But this confused thinking has been around for much longer than this summer. Take satisfactory schools for example. Maybe the new schedule means the judgements are moving away from their reliance on data, but as we all know, satisfactory no longer means satisfactory. It means not good enough. Only good or outstanding are satisfactory!

However, to be good or outstanding, schools have had to achieve above average results, either in attainment or progress. Now my year 6s could tell you that an average is a middle amount and that it is mathematically impossible for everyone to be above average. Unfortunately the government and Ofsted aren't quite as clever as my year 6s. We want all schools to be good they chorus, whilst following schedules that perversely work directly against this. An HMI told a head's conference I was at recently told us that many schools had moved from satisfactory to good in the last year. Then, under his breath, he muttered that a similar number had moved the other way because that's the way that norm referencing works.

The final paradox is the advice being given on how to spend the pupil premium. The newspapers were quick to condemn schools for not following the government's advice on the most effective ways to raise attainment as laid out by Durham University in their Pupil Premium Toolkit. Now put aside the fact that the top three strategies are in that position because they don't cost a great deal to implement and so are unlikely to be the main area of spending for schools. Even if they were, these strategies: effective feedback, meta-cognition and self regulations strategies and peer tutoring, are all actions that schools would implement for all children. My teachers are hardly going to differentiate the feedback they give, depending on whether a child falls into a particular category. 'Sorry dear, you're not pupil premium so I'm not going to tell you what you need to do to improve.'

The consequence of adopting these strategies would be that all pupils would improve. If all pupils improve then the gap the government rightly wants to close will remain: it's the way the maths works - another paradox.
Share on Twitter Share on Facebook

Be notified by email of each new post.


Ricky-Tarr's picture
Thu, 13/09/2012 - 12:01

Put simply, the government wants the percentage of children achieving the highest grades to stop rising each year....

That's putting it over-simply. What the government (and almost everyone else) wants is an end to grade inflation, which is 'a rise in the percentage achieving the highest grade without a corresponding rise in performance.

The government (along with almost everybody else) would be only too delighted if genuine improvements in teaching and learning led to more A*-C at GCSE. Which is why crude norm-referencing is not used.

it is mathematically impossible for everyone to be above average

In a dynamic system with a historical referent, it is possible for everyone to be above last year's average. Unlikely maybe, but possible.

If all pupils improve then the gap the government rightly wants to close will remain

But if the rate of improvement of the disadvantaged is higher than the general rate, the gap will narrow. Which is the point of the pupil premium.

Rebecca Hanson's picture
Thu, 13/09/2012 - 13:18

Well done Ricky,

Your post shows that you understand the issues much more clearly than you did a couple of weeks ago.

Now what about dealing with the kids who didn't get their apprenticeships and other opportunities they deserved because their GCSEs were marked down by people who did not understand these issues?

Michael Dix's picture
Sat, 15/09/2012 - 19:37

The pupil premium is a genuine attempt by government to try and close the long tail of underachievement that plagues our education system. My gripe is that schools are likely to get criticised by government, the media and Ofsted if we don't use the money in the way that they recommend - the Durham University Pupil Premium Toolkit.

The toolkit takes a look at all the research into raising pupil attainment and then presents it as a spectrum from highest impact/lowest cost to lowest impact/highest cost. The research it examines is not into raising achievement in socially deprived pupils, but all pupils. Therefore, even if the top three approaches do have a greater than average impact on the progress of this group, there is still likely to be a rise in attainment with all other pupils and a corresponding rise in the average. The gap may narrow a bit but not as much as is needed.

Most schools will use the money on focussed interventions with these pupils such as one to one tuition and small group work. These are lower down the Durham list and schools will be slated for following what they consider to be the best ways forward.

There are also other views on this that are as relevant as the Durham research. The Cambridge Educational Review finds that until the gap between the poorest in society and the rest is narrowed, it is unlikely that any measures will significantly reduce the gap.

The Joseph Rowntree Foundation have also identified factors that can improve attainment such as involving parents far more in their children's learning and convincing pupils and parents that they can affect their own futures through education where there is a culture of passive defeatism.

In my school, taking an approach over many years that looks at all pupils as individuals, those under-attaining receive a range of interventions and other strategies. Consequently, the percentage of pupil premium pupils under-achieving in my school is lower than all children nationally. We currently have no gap to close. We are looking into the top three recommended approaches from the toolkit but they are low cost so will not use much of the money we have received.

There are also some other issues with the premium. Our free school meals number is half the national average yet 20% of our children come from areas ranked in the 10% most deprived. Free school meals are not the most accurate way of measuring deprivation.

Also, whilst the number of under-achieving pupils is low, the number of higher attaining children is also low and that's our challenge: to identify which of them has the potential to achieve higher but currently do not due to background and early years development.

Rebecca Hanson's picture
Sat, 15/09/2012 - 20:58

I would like to assume that a short written account from a head of your calibre justifying the way in which you are choosing to use your pupil premium money should be acceptable Michael, whether or not it correlates with the Durham list.

Do you feel this is the case? I'm genuinely concerned to know. I'm not sure what the tracking processes are. I'm sure you agree there should be some but also that they should not preclude the professional freedom of those with the ability to make and justify intelligent decisions. I'm on the committee of the Lib Dems Education Association so any feedback you give me will be passed on.
(obvious note - my opinions expressed online are my own - not those of any body I sit on or organisation I represent or work for unless I specifically state they are).

Beyond that first, timely question comes the longer term and much more profound question of how things could be done better.

Please understand that I already know that things can be done much better under the custodianship of good LAs and through cooperative redistribution of funds. I also see the flaws in the pupils premium you have mentioned (more clearly because of your excellent analysis) and more. But we are where we are and given that - if you have any thoughts on where we could move which would be better I'd be interested to hear them.

Michael Dix's picture
Sun, 16/09/2012 - 19:13

I would hope that my analysis of the best way to use the pupil premium in my school with my pupils will be acceptable when Ofsted come calling - time will tell. I have discussed it with staff and will be presenting it to governors; it is also on the school website as required by the new legislation. I suppose that I am in a good position because I don't have that tail of under-achievement with the pupil premium children in my school because we have spent the premium, and before that other grants and funding from the LA, on the areas where we felt it would have most impact.

I get frustrated with the DfE who should know better and the media who could do better when they present the way schools are using the premium as a negative story. It is unfortunate that the premium has arrived in schools at the very same time that budget cuts have taken effect. This will result in it being used to plug holes rather than targetting in on the individuals.

I am not having a go at the Durham toolkit which is a very useful tool to help schools use research to impact on whole school improvement. There are strategies in the toolkit which can be focussed on those pupil premium children who are under-achieving and if schools are using the money for these, this should be celebrated rather than damned.

In my school a lot of the children in this group have benefitted from short focussed interventions with a Higher Level Teaching Assistant. Therefore some of my money will be used to maintain this level of support. However, the issue I have is middle achieving pupils who might have the potential for higher achievement but whose start in life or background has led to a false ceiling appearing. Here, I want to use the money to provide a rich curriculum to enable these children to engage in many different experiences, to raise aspirations and to work with parents far more than we have done in the past. Consequently, some of my money is replacing the now defunct funding for extended schools, funding visits and residentials for these children, and some of it is supporting work with parents. These don't appear in the toolkit but to my mind they are equally valid approaches.

I would urge the government to examine other indicators of deprivation and not purely rely on free school meals, an entitlement that not all parents take up. My school exemplifies this issue with only half the national average of fsm but twice as many children coming from the 10% most deprived areas.

Rebecca Hanson's picture
Sun, 16/09/2012 - 20:55

Thank you for all your comments Michael.

I'm also very concerned about the quality of FSM as way of allocating this money. Do you know of any other statistics which could be used or can you think of a way a better indicator could be constructed?

I'm secondary so I'm used to seeing the kids who demand attention or are borderline getting attention the other miss out on. I would very much like to see all children who are lacking working role models and/or people to fight for them getting the same attention.

I absolutely agree with you that we are not sufficiently addressing parenting as a society. I it's useful to think about this in two ways - firstly about the crucial experiences which some young children miss out on - which you are identifying and trying to address. But then we have the elements of parenting which are nurtured by the Finnish system - all families have more time together. Children have more time to grow and play and parents learn as they interact with their children. Do we take children away from their families earlier to ensure they don't miss out or do we work with families and allow them time? Most families would respond to the latter but some wouldn't. It's difficult. I'd be delighted to read anything you want to write.

Ricky-Tarr's picture
Thu, 13/09/2012 - 13:56

Now what about dealing with the kids who didn’t get their apprenticeships and other opportunities they deserved because their GCSEs were marked down by people who did not understand these issues?

There's a problem with that, Rebecca. I don't think any progress can be made there until we have a straight answer to a question that was posed again and again by the Education Committee (and in particular by Labour's Pat Glass) but which never really got answered. None of the Ofqual lot answered it. Gove didn't really answer it either.

I genuinely don't think any of them were being deliberately evasive. Perhaps they just didn't know the answer. Perhaps they didn't quite get the import of the question.

The question is this:

Were the changes in the grade boundaries for the June exams just enough to make those June exams fair & comparable with previous years, or or were they raised higher than would be needed to meet that limited objective in order to compensate for the over-generous regime in January?

Glenys Stacey has repeatedly said that the grade boundaries in the summer were "right" - implying that were right for the exams concerned and could serve as a useful guide for next year. That seems to me a denial that there was any 'over-compensation'.

If January students got a luck break and were marked more leniently....and Ofqual now thought it best just to shrug, put it down to experience, and let them keep their inflated grades; while being duly more stringent about June, then I'd see that as being 'unfair' but not really 'unjust', if you get the distinction.

What would certainly be *unjust* is if summer candidates were disadvantaged by a perceived need to massage the stats for the year as a whole such that they were not only marked more severely than the January cohort, but were also hit by a double whammy in order to offset the inflationary effects of January's leniency. That would offend against natural justice.

But, as I say, no one seems to be giving a straight yes/no answer here. What do you think?

Rebecca Hanson's picture
Thu, 13/09/2012 - 21:51

"Were the changes in the grade boundaries for the June exams just enough to make those June exams fair & comparable with previous years, or or were they raised higher than would be needed to meet that limited objective in order to compensate for the over-generous regime in January?"

In the chaos of the email deletions, the general top down bullying in state education, the obvious Mandy Rice-Davies comment which could be applied to Glenys and, conversely, the general anger which could inappropriately focus on this topic it's hard to find the truth. The only way to do that is through the direct testimonies of very experienced teachers who taught a lot of borderline children for years and taught many this year. Quite a few such people have stepped up and spoken out about this through all sorts of media.

Janet Downs's picture
Thu, 13/09/2012 - 15:07

The GCSE fiasco is the subject of other threads. This thread is about the incompatibility of asking for more rigorous exams (with more failures) while at the same time demanding that schools get more pupils to pass. It is also about the misuse of the word "average" and the paradox around spending the Pupil Premium.

Ways in which to tackle disadvantage include ensuring pupils attend lessons regularly: the OECD found that "learning time in school is one of the strongest predictors of which students will outperform their peers." So perhaps part of the Pupil Premium could be spent on tackling absenteeism:

OECD also found that the language spoken at home reduced pupil achievement. So perhaps some of the Pupil Premium money could be spent on increasing pupil proficiency in speaking English.

Or perhaps the answer is in the whole school system. OECD has found that the best-performing countries tend to be the most equitable.

Or perhaps the money could be spent on encouraging carers to read to their children:

The academies system is widely touted as being the answer to raising achievement among disadvantaged children. But the National Audit Office (NAO) 2010 found that "on average, the gap in attainment between more disadvantaged pupils and others has grown wider in academies than in comparable maintained schools." And recent research commissioned by (but not widely published by) the DfE found the City Challenge (later London Challenge) was more successful that the academies programme in raising attainment in poor performing schools and of disadvantaged children.

So perhaps the Pupil Premium could be spent on emulating the successful City Challenge strategies.

Rebecca Hanson's picture
Thu, 13/09/2012 - 21:57

There an even more fundamental insight here which is that children need role models and they need personal attention to thrive. Most get that at home but our most disadvantaged often do not. Many schools employ high quality professional mentors for their challenging children. These mentors are often brilliant people - such as retired head teachers - doing a little work in their retirement that they passionately believe in for just a little money. I think it would be wise to spend the pupil premium on ensuring all the target children receive such mentoring, perhaps allying it to the ASDAN cope qualification which provides an appropriate structure for such mentors to work with nice children who do not have behaviour issues to help them thrive and to raise their aspirations.

Such mentors could make the school more clearly aware of the further needs of the target children of the kind you described Janet. Often it's not so much the money that's a barrier to them getting the support they need - it's having someone to fight for them.

Adrian Elliott's picture
Fri, 14/09/2012 - 08:38

The schools I have spoken to or heard about all seem very clear that their results were not simply down on the January exams (which most had not entered pupils for anyway) but were significantly down on what they might have anticipated the same pupils getting in 2011, 2010 and so on if they had taken the examination then.

Ricky-Tarr's picture
Fri, 14/09/2012 - 14:30

Yes, Adrian... I'm hearing the same thing. But then when one looks at the actual figures, they don't clearly support that. Overall, A*-C is down only 0.4%, which means schools should only be seeing tiny numbers in their cohort doing worse than expected. Even in English, where the results are down 1.5% - that should mean (if the problem were universal and evenly distributed) just two or three students per cohort. Yet some schools are reporting 30, 40 + students affected despite ALL exam boards supposedly having taken the same action.

Rebecca Hanson's picture
Fri, 14/09/2012 - 15:34

It's the English that's the problem although I've noticed there are some specific issues in maths to do with the twin pair pilot this year which worry me but I think that's a different issue.

The issue is concentrated in specific cohorts and specific papers so it's much more evident in schools with a lot of borderline students which took the problem papers. Hence some schools will have had no problems while others had significant problems and within the schools with significant problems those problems would be further concentrated in the borderline sets. So some teachers would see very substantial effects.

Michael Dix's picture
Sat, 15/09/2012 - 18:55

There is always going to be a paradox when you require the end of compulsory education examination to fulfill several competing functions. It is meant to identify the most able, hence the concern about more and more gaining the higher grades. However, it is also being seen as the minimum expected for all students, which is why Mr Gove is calling on schools to ensure all bar those with special needs to attain 5 A to Cs, rightly aspirational but surely diametrically opposed to its first function. Then it is used to hold schools to account. Fair enough, but as soon as exams become high status for schools then schools will naturally look for ways to present themselves in the best possible ways.
Couple this with exam boards competing for business and you have the perfect recipe for grade inflation.

However, this may not be the only cause of more students attaining higher grades. What if, shock horror, teaching and learning is actually improving? How would we know? My experience of marking KS2 SATs since the beginning and as a primary headteacher illustrate the difficulties in identifying exactly what is causing grades to rise.

The first year I marked it was a doddle. Most pupils failed to complete more than about half the test and spent half my time drawing red lines across reams of empty pages. Two years later and an empty page had become a rarity. Schools, encouraged by government, given additional money and held to account by Ofsted, introduced teaching to the test, revision, coaching, boosters, holiday classes and exam techniques. It meant that the curriculum became increasingly narrowed in many schools, but the results improved.

However, the quality of teaching and learning improved in English and maths over this time, something that is borne out by Ofsted findings. So the increase in results was a combination of factors. Could not the same be said of the rise in A to Cs?

If any increase is simply put down to grade inflation then it doesn't matter how much teaching and learning improves, there will never be any acknowledgement of this.

Add new comment

Already a member? Click here to log in before you comment. Or register with us.