More evidence-based argument on the ‘attainment gap’ fallacy

Roger Titcombe's picture
 1

I refer to the work of Professor Rebecca (Becky) Allen, Director of the Centre for Education Improvement Science, UCL Institute of Education, who has published a set of three articles analysing the fallacies peddled by DfE, the Sutton Trust, the Social Mobility Foundation, the Education Policy Institute and others that are so frequently and uncritically echoed in the mainstream media. The following selection of quotes (in italics) from her articles are my attempt to summarise her arguments. My comments are in square brackets.

Part 1  The pupil premium is not working: Do not measure attainment gaps

We think about attaching money to free school meal students as a Coalition policy, but the decision to substantially increase the amount going to schools serving disadvantaged communities came during the earlier Labour Government. However, [this] icing on the cake turned out to have a slightly bitter taste, for it came with pretty onerous expenditure and reporting requirements.

The money must be spent on pupil premium students, and not simply placed into the general expenditure bucket. Schools must develop and publish a strategy for spending the money. Governors and Ofsted must check that the strategy is sound and that the school tracks the progress of the pupil premium students to show they are closing the attainment gap.

Using school free school meal eligibility as an element in a school funding formula is a perfectly fine idea [actually it’s not – see later], but translating this into a hypothecated grant attached to an actual child makes no sense. The first reason why is that free school meals eligibility does not identify the poorest children in our schools. [my bold]

This was well known by researchers at the time the pupil premium was introduced thanks to a paper by Hobbs and Vignoles that showed a large proportion of free school meal eligible children (between 50% and 75%) were not in the lowest income households. One reason why is that the very act of receiving the means-tested benefits and tax credits that in turn entitle the child to free school meals raises their household income above the ‘working poor’.

Children who come from households who are time-poor and haven’t themselves experienced success at school often do need far more support to succeed at school, not least because their household financial and time investment in their child’s education is frequently lower, their child’s engagement in school and motivation could be lower and the child’s cognitive function might lead them to struggle [my bold]. These are social, rather than income, characteristics of the family. [Except for cognitive function, which mine and John Mountford’s research shows to be the major factor.]

Yes, there are mean average differences by pupil premium status in attendance, behaviour and attainment. However, the group means mask the extent to which pupil premium students are almost as different from each other as they are from the non-pupil premium group of studentsWhy do we like these ‘gap stories’? We like them because we humans like the pattern forming that group analysis facilitates, and having formed the [false] gap story, we are then naturally drawn to thinking of pupil cases that conform to the stereotypesYour school’s gap depends on your non-Pupil Premium demographic. Tracking whether or not ‘the gap’ has closed over time is largely meaningless, even at the national level [my bold].

Part 2  Reporting requirements drive short-term, interventionist behaviour

Equally, we want these schools to provide rich cultural experiences that the students might not otherwise afford. And yet, many of these things we’d like schools to spend money on aren’t central to the question of how we should spend money to raise attainment (remember, the pupil premium is supposed to be used to raise attainment) [But can we be sure of that? It depends what they are.] Beyond the obvious provision to help make home life matter less to education (e.g. attendance and homework support), we struggle to make highly evidenced and concrete recommendations, in part because ‘money’ has a poor track record in raising educational standards in general. The Education Endowment Foundation was established alongside the pupil premium with the expectation they would identify effective programmes or widgets that schools could then spend money on. Unfortunately, most [but not all] trials have shown that [such] programmes are no more effective than existing school practice, and in any case free school meal eligible children do not disproportionately benefit from them [my bold].

Ofsted comment on pupil premium expenditure and attainment more often than not, even during short inspections. In a sample of 663 Ofsted reports we reviewed from the 2017/18 academic year, 51% mention the pupil premium and well over half of these assert that inspectors can see the monies are being spent effectively! Where their comments are critical of pupil premium expenditure, they rarely make concrete recommendations that could be useful to anyone [my bold], except to the industry of consultants and conferences that help schools solve the riddle of how to spend the pupil premium [in a way that satisfies OfSTED]. These are example quotes from inspection reports (with the one mentioning external review appearing regularly):

 “The school does not meet requirements on the publication of information about the pupil premium spending plan on its website”

 “The leaders and managers do not focus sharply enough on evaluating the amount of progress in learning made by the various groups of pupils at the school, particularly the pupils eligible for the pupil premium …”

 “An external review of the school’s use of the pupil premium funding should be undertaken in order to assess how this aspect of leadership and management may be improved”

To be clear, as a parent whose own children are educated in one of the most poorly funded counties in England, I am gravely concerned about how the current funding crisis is damaging both the quality of the experiences they have and the well-being of their teachers. But equally, as a researcher in this field, I would not be able to give a school well-evidenced advice about how to use money to close the attainment gap [my bold]. I think this is because improved classroom instruction isn’t something it is easy to buy. Is it [even] possible to teach in a way that disproportionately benefits those in the classroom from disadvantaged backgrounds? [my bold]

 [The answer to this is no, but it is possible to enhance the cognitive ability of all students, which is of greatest benefit to those where the starting point is lowest.]

Part 3  Can within-classroom inequalities ever be closed?

This last part of Becky’s Pupil Premium analysis is so important that it warrants detailed consideration. She raises very important issues about the culture of classrooms and whether extreme approaches to directing the attention of students (that rightly concern her) are justified. They are not!

I will return to this in my next article.

So how should schools best combat social and educational disadvantage?

Becky has pointed to the answers to this question.

  1. Higher proportions of educationally disadvantaged pupils in school intake cohorts should result in enhanced general funding for the school to reflect the increased costs of the effective teaching and learning methods (for all pupils) that are needed. 
  1. Forget all the nonsense about ‘Closing Gaps’. 
  1. Use approaches to teaching and learning for all pupils that are proven to be effective. For example, those recommended by EEF for science teaching, that in fact work across the curriculum.

In terms of how to determine the amount of extra funding to be delegated to schools, together with how to hold schools accountable for outcomes, I believe the answer to both questions was provided by the Cumbria LEA in the early 1990s. I was one of the Cumbria heads that served on the LEA Working Party that devised the approach, so I know in detail how it worked. In those days there were substantial ‘Non-Statutory Special Needs’ funds to be distributed to schools through the funding formula of each LEA.

Unlike most LEAs, Cumbria rejected a formula based on FSM in favour of Cognitive Ability Tests (CATs) data obtained from screening all Y7 pupils in October of the intake year. The Cumbria formula delegated additional funds to schools, not individual pupils, through a formula driven by the numbers of pupils with CATs scores at various threshold levels below 85 (-1SD). Where there were significant differences in the scores on the three sub-tests (Verbal, Non-Verbal and Quantitative Reasoning) the CATs profiles for each pupil should prompt further testing for Specific Learning Difficulties (SLD), so enabling specific intervention for individual students, reflecting Becky’s important point about the diversity of needs.

My school had a strong relationship with the South Cumbria Dyslexia Association, which led to our providing rent free accommodation for the Barrow Dyslexia Centre in return for dyslexia awareness training for our teachers and free specialist tuition for our pupils in the Centre, which provided for pupils from other schools on a fee-paying basis. We found that such tuition benefited many more pupils than just those with a positive dyslexia diagnosis, with those with more general literacy problems also being greatly helped by the same approach. This relationship was highly praised in successive OfSTED inspections.

All this has to be seen in the context of the very proactive LEA wide approach to SEN, which was called One in Five, this being the proportion of Cumbria pupils estimated to have SEN in some form. This led to the formation of a brilliant team of area SEN advisers and an area Educational Psychologist to support school staff. Cumbria was a national leader in adopting a modified form of the Reading Recovery programme pioneered in New Zealand and implemented in area ‘Literacy Centres’ which took local primary children during the school day for short-term  programmes.

The Cumbria formula provided for much more effective support than the present ‘Pupil Premium’ for generally cognitively challenged pupils as well as those with diagnosed SLD because it could lead to higher quality teaching and learning across the curriculum in which in our school the example of Cognitive Acceleration through Science Education (CASE) played a prominent role. There were many (too many to cover in this article) other significant initiatives in our school that supported a co-operative learning culture that has been consistently endorsed by EEF. These included our elected School Council and the Alfred Barrow School Association (parents). Members of both attended Governors Sub Committees alongside governors and teachers. It was clear that parents living in some of the most socially and economically deprived electoral wards in England were learning from their children about effective approaches to teaching and learning and positive social values.

This is an appropriate point to explain the demographic profile of Cumbria, one of the largest counties in England, but one in which average ‘shire county’ affluence masks a narrow coastal strip running from Carlisle in the north to Barrow-in-Furness in the south characterised by white working class communities lacking in ethnic and cultural diversity that continue to suffer greatly from the austerity policies of the post 2010 Coalition and Conservative governments.

Visitors to the Lake District are often unaware that this coastal strip had a major role in the industrial revolution (with the invention of the Bessemer Converter) and remained heavily industrial until the decline that began in the early 1970s that closed heavy industries and has ever since blighted huge swathes of northern England. The Victorians developed  iron ore and coal mining that supported an iron and steel works at Workington that produced railway rails for the UK and much of the world, together with an even larger one at Barrow that provided steel for the shipbuilding, armaments and general engineering complex that is now BAE Systems, manufacturing the nuclear submarines that provide for the UK nuclear deterrent. Another other  main industry is the Sellafield nuclear waste processing site near Whitehaven, a major employer from Barrow to Carlisle.

The Cognitive Ability Test (CATs) profiles for Cumbria schools match this demographic, with very low mean intake scores along the coastal strip. My school, Alfred Barrow, was consistently less than 85 (16th percentile), with mean intake scores steadily rising as you move east towards the M6 corridor that connects the prosperous communities of Kendal and Penrith. The LEA produced this chart annually, showing the mean GCSE attainment of each secondary school spread along the X axis of mean intake CATs scores.

The LEA never made as much use as it should of this CATs-based analysis. Political control of Cumbria County Council tends to swing from Labour to Conservative with Lib Dems often holding the balance of power. Sadly, during a period of Labour control some years ago, the LA county wide system of CATs testing was abandoned. I was told by a Labour councillor that there was no way they were going to have the voters in the coastal towns that voted Labour ‘smeared’ as being ‘dim’. You will have to take my word for it that nearly all of the schools with a mean intake score below 95 (37th percentile) were indeed in this coastal band. However with the loss of universal CATs testing, the basis for a statistically sound approach to school accountability was also lost. Schools towards the left of the chart, with lower intake CATs scores automatically received higher levels of extra funding. The regression line divides all the schools into those performing above, from those performing below the Cumbria average (above and below the regression line) providing a sound basis (unlike that currently used by OfSTED) for a discussion with school leadership teams about the efficacy of the teaching and learning arrangements in their schools.

Alfred Barrow School had the lowest intake CATs score in the county by far. Two thirds of our pupils were in the bottom third  of the national cognitive ability range. A typical intake of 100 pupils would usually contain only about 12 with CATs scores of 100 or above with two or three above 120. You might think that such a school would be a basket case, but this was far from the case.

The school received successively improving inspection reports in 1990 (pre-OfSTED HMI), 1995 and 1998 and also did well in 2004, a year after I retired, despite never once exceeding the floor target introduced by the new Labour government. The inspectors had sound, irrefutable data on which to base their judgements. Y11 GCSE results were listed student by student in descending order of CATs scores. It was not just those with low CATs scores on admission that did relatively well, the proportion of A and A* grades also rose steadily. This excellent attainment pattern matched to CATs scores strongly suggested that general cognitive ability had been significantly raised across the ability range in accordance with the school’s belief in the plasticity of general intelligence.

We often had students in the ‘top 5’ nationally in a number of subjects. One visiting HMI commended our ‘business model’, which was based on maximising additional formula funding to enhance curriculum provision and provide pupil support initiatives that other schools could not afford. We soon became oversubscribed and in 2002 the LEA had to enlarge our capacity by 150 pupils with a £multi-million expansion/refurbishment  project, all of which has since been demolished.

Unlike other schools we used Local Management to reverse all the outsourcing inherited from the LEA, directly employing all of our own non-teaching staff including cleaners and catering staff on union rates and the Local Government pension scheme. Large numbers of these staff , along with teachers (including me) enrolled their own children in the school. A new Head of Maths moved her child from a posh private school in the east of the county.

I recall being summoned to a heads meeting in London with David Blunkett, the new Secretary of State for Education. He started by asking the assembled heads why parents should send their children to ‘failing schools’ where less than 25% of students were achieving 5+A*-Cs at GCSE, where none of them would send their own children! New Labour was soon to address this through the GNVQ scam. About the only thing Michael Gove did right was to rid our schools of this corruption and curriculum degradation.

The great majority of our students did well on leaving us at 16, with excellent career progression including many achieving good A levels at the Sixth Form College and progressing to top universities. In September 2018, I attended an Alfred Barrow reunion. Many ex-students (some now in their 30s) with an astonishing range of successful careers had travelled long distances from as far as London and Scotland.

Sadly, Alfred Barrow School, along with the two other non-religious secondaries on the town mainland were closed in 2009 as part of a disastrous Academy re-organisation, from which the Barrow school system is yet to fully recover, with hundreds of Barrow pupils now travelling by bus and train (when running) to Local Authority schools in the nearby towns of Dalton and Ulverston, while BAE Systems has taken control of the Academised secondary schools.

The Alfred Barrow site is to become a Health Centre. Two other secondary schools have been demolished and replaced  by posh housing estates, while the excellent co-ordinated school system provided by the former LEA has been destroyed at the secondary level with the primaries increasingly threatened by aggressive Academisation.

Share on Twitter

Be notified by email of each new post.





Comments

John Mountford's picture
Tue, 30/10/2018 - 17:13

There is so much to comment on here, it is a challenge to know where to begin. With that in mind, I will begin with the FSM issue which now affects the working poor more than at any time previously.

The use of FSM to identify the most disadvantaged pupils is and always was problematic. Added to what Prof Allen has to say on this, one anecdote from my own experience as an Ofsted inspector is illuminating. Some time before inspecting a school in Bedfordshire one of the major employers in the local area went out of business. Prior to this event, the school had below average FSM pupil numbers. Almost overnight this was transformed as skilled engineers and technicians became unemployed and their children became eligible for FSM. Though an extreme event, and in this case fairly short-lived, it serves to remind us of the fact that even when the pupil premium was first developed it was well documented that, "a large proportion of free school meal eligible children (between 50% and 75%) were not in the lowest income households." However, it is important to recognise that, this said, most low attaining children will still be brought up in homes where "the educational and social disadvantage of families" is a formative factor in early development. I could not agree more strongly with Professor Allen that it is time to re-examine this particularly ineffective policy.

Another issue I feel it necessary to comment on is Ofsted's role in perpetuating the myth that schools and inspectors are able "to assert that the monies (PP funding) are being spent effectively." As Becky Allen points out, "It isn’t possible for a school to conduct the impact analysis required by DfE and Ofsted to ‘prove’ that their pupil premium strategy is working for all the usual reasons. Sample sizes in schools are usually far too small to make any meaningful inferences about the impact of expenditure, and no school ever gets to see the counterfactual (what would have happened without the money)."

In this event, the pointlessness of the exercise is only surpassed by the effort required to carry it out and it has to be hoped that under its new leadership and strategy for inspection, Ofsted will take a long hard look at this area. For sanity's sake, the government needs to understand that PP funding, based on FSM and other relevant factors, be rolled into the general school allocation for school leaders to allocate according to the needs of the most disadvantaged pupils.

The detailed analysis of 'the gap' carried out by Prof Allen and developed further by Roger go the the very heart of what really needs to change in our schools. "Why do we like these ‘gap stories’? We like them because we humans like the pattern forming that group analysis facilitates, and having formed the gap story, we are then naturally drawn to thinking of pupil cases that conform to the stereotypes." It doesn't have to stay this way.

The professor's advice is clear "We should stop measuring or monitoring school attainment gaps because it is a largely statistically meaningless exercise that doesn’t help us identify what is and isn’t working in our school. In any case, ‘gaps’ matter little to students from poorer backgrounds; absolute levels of attainment do."

As we keep hearing from Roger, there are effective, tried and tested ways to raise attainment but not unless we have the courage to challenge some old established taboos. There is nothing more powerful than an idea whose time has come.


Add new comment

Already a member? Click here to log in before you comment. Or register with us.