Coasting Schools – the DfE marches blindfold across a statistical minefield

Roger Titcombe's picture

Henry Stewart has exposed the serious shortcomings of the latest Department for Education (DfE) proposals in his post on Local Schools Network. The gist of his argument is that schools with ‘high attaining intakes’ are very unlikely to be defined as coasting. Henry questions whether the government intends never to tackle ‘underperforming schools’ in leafy areas, or does it really not understand the measures that it is choosing to use? The latter seems unlikely. The DfE employs highly qualified statisticians. Given that OfSTED behaves as if it is an executive arm of the DfE rather than a truly independent regulator of education standards, the result will be huge numbers of secondary schools serving less affluent communities that produce lower intake attainment scores being lined up for privatisation through the compulsory academisation mechanism. In November 2006, the NfER journal ‘Practical Research for Education‘, published my article, ‘Cognitive Ability and School Improvement’. This argued for the use of standardised school intake ability scores for assessing the effectiveness of secondary schools through an LEA managed approach based on producing regression data as in the example for Cumbria Schools in 2002. I haven't mastered getting getting complex charts to appear satisfactorily on LSN posts so to view it you will need to visit my website, where you will find a version of this article. The named school with by far the lowest GCSE Average points and average intake Cognitive Ability Test Scores (CAT) was The Alfred Barrow School where I was headteacher at the time (2002). The chart shows all the Cumbria secondary schools along the horizontal axis according to the average CAT score for the children admitted in 1997. The vertical axis gives the school GCSE performance in 2002 for the same year group of children. The former DfES calculation of GCSE point score is used but the chart has a very similar form if the percentage of pupils with 5 or more A*-C grades (5+A*-C) is substituted. The line shows the relationship between average intake CAT score and subsequent GCSE performance. This regression line gives a correlation of 0.84. This means that school GCSE performance is very strongly predicted by the CAT. The significant features of this chart are as follows. There is a vast difference between the average intake cognitive abilities of the lowest and highest performing schools. The lowest performing school, Alfred Barrow had an average intake pupil CAT score at the 16th percentile. This means that only 16 percent of 11 year-olds in England had a lower score than that of the average for pupils admitted to this school in 1997. The highest performing school, Cumbria’s only selective grammar, admitted pupils with an average cognitive ability score much higher than the national average of 100. The chart, with its very high correlation, takes no account of social deprivation or any other factor that might constrain or promote pupil and hence school performance. This suggests that individual cognitive ability is a much stronger factor in school achievement than social deprivation, or else that cognitive ability and social deprivation are related. Both possibilities challenge the DfE model of school achievement and effectiveness. The regression line shows the average performance of Cumbria schools in relation to the intake cognitive abilities of pupils. Schools below the line had a worse than average performance and schools above the line had a better than average performance. When schools were ranked in order of distance from the regression line the resulting rank order bore no relation to the published GCSE league table of Cumbria. Alfred Barrow was bottom of the Cumbria GCSE league table in 2002 but in the top half of the Cumbria rank order of effectiveness in relation to intake cognitive ability, without making any allowance for the acute social deprivation suffered in its catchment area. This means that when its 2002 leavers were placed in rank order of Cognitive Ability Test (CAT) score as measured on entry to the school in 1997, the GCSE results of each pupil compared favourably, on average, with pupils of the same CAT score in the school with the highest GCSE score, which was top of the DfES GCSE league table for Cumbria in 2002. This was a statistical necessity resulting from the positions of the two schools on the chart with respect to the regression line. This may well have been true throughout the ability range, the chief difference being that Alfred Barrow had far fewer (but still some) high CATs score pupils, but far more low CATS score pupils. There is nothing in these data to suggest that the minority of Alfred Barrow pupils with high intake CATs scores underperformed compared with pupils with similarly high intake CATs scores in other schools. Is this a model for a much better definition of ‘Coasting Schools? I believe it is. At that time the Cumbria Education Service was beginning to use such regression based data for evaluating the performance of Cumbria Secondary schools. A local Cumbria definition of a ‘coasting’ school could well have been falling below the regression line. It should be noted that some of the highest performing schools at GCSE fell into this category. However, at that time in Cumbria as elsewhere, LEAs were in the process of being replaced by ‘Children’s Services’ reorganisation and the political climate was anticipating the eventual dismantling of the LEA based national education system created by the 1944 Education Act. In Part 5 of my book, ‘Learning Matters' I argue for the restoration of the equivalent of LEAs and democratic local control of the school system. While such ideas will not appeal to the present Conservative government they are very much on the agenda of the Labour opposition led by Jeremy Corbyn. There are two problems with my model in relation to a national policy for identifying and challenging ‘coasting schools’. The first is that it relies on universal CATs testing. In ‘Learning Matters’ I argue that KS2 SATs should be abolished in favour of universal Y6 CATs screening of all pupils as now happens in the Hackney LA. However, because CATs drive the outcomes of all tests that measure the development of cognition, there is a good correlation between CATs and KS2 points scores. Therefore GCSE/KS2 regression charts could perform the same function. The second is that the DfE is seeking to establish a basis for judging school performance based on the national rather than local norms. So the national equivalent chart would contain the data for all the secondary schools in England. Those appearing below this national regression line could be fairly judged to be producing less academic progress for their pupils than the schools above the line. LEA specific regression charts would still have a useful function as they could be a sound basis for comparing the effectiveness of LAs in bringing about school improvement. Nothing is ever perfect and even on my regression based model there is still room for factors like the mix of GCSE subjects offered to and taken up in different schools to distort the comparisons. And none of this invalidates the greatest statistical health warning of all. There is much more to quality education and a good school than just maximising GCSE performance per pupil.

Share on Twitter Share on Facebook

Be notified by email of each new post.


Barry Wise's picture
Thu, 26/11/2015 - 14:40

How close is the match between CATs administered as a baseline test in Y7 and KS2 SATS taken the previous year?

Roger Titcombe's picture
Thu, 26/11/2015 - 16:15

Barry - In Hackney the CATs are taken early in Y6. My understanding is that there is good correlation. This is the information from GL Assessment, who provide the CATs tests in Hackney.

Roger Titcombe's picture
Thu, 26/11/2015 - 14:53

Henry's post

raises a parallel issue. He shows that pupils with 4c at KS2 are unlikely to hit either the 60% 5+A*-C floor target or the expected progress at GCSE. So why is 4c such a high stakes target for primary schools? What is the point of a high stakes target that consistently fails to result in another high stakes target at KS4 being hit?

Have no doubt about how high stakes these floor targets are for headteachers and all the other stakeholders in their schools that include parents, pupils and their teachers. CATs data show that less affluent postcode communities are characterised by lower than average mean cognitive abilities. Don't shoot the messenger - this is a fact. See Section 4.8 of 'Learning Matters'.

The Primary Schools that serve such communities will be permanently at risk of failing the floor target, so who could blame their heads and teachers for concentrating their efforts at hitting the 4c target at all costs for the maximum proportion of pupils, at the expense of developing the cognitive development of all their pupils? This means concentrated revision type cramming.

The most effective versions of this approach will concentrate on ensuring that pupils are trained to get the required number of marks from questions that test what they already understand, rather than to explore more cognitively demanding material that would result in the cognitive development needed for pupils to progress readily through KS3 & KS4. So it is likely that such schools are being incentivised in a way that will result in fewer 4bs, 4as and 5s than could otherwise have been the case if the school was more relaxed about 4cs, which don't do the job for secondaries anyway.

If we now shift our attention to the secondary school that takes these children, the same reasonable paranoia about getting C at GCSE grades will apply. Such a school will be incentivised to get C grades rather than Bs or As so passing unnecessary underachievement into KS5.

The only conclusion is that high stakes floor targets LOWER rather than raise standards.

This is of course a major theme of 'Learning Matters'.

Add new comment

Already a member? Click here to log in before you comment. Or register with us.