Pupil Premium accountability

Roger Titcombe's picture
 4

Professor Becky Allen of the UCL Institute of Education has written important articles about the Pupil Premium (PP) that I discuss here and here.

It is perfectly reasonable for OfSTED to demand accountability for additional funding generated by the PP. Professor Allen draws attention to the frequency of critical comments like this in OfSTED reports.

“The leaders and managers do not focus sharply enough on evaluating the amount of progress in learning made by the various groups of pupils at the school, particularly the pupils eligible for the pupil premium …”

My articles support the reservations of Professor Allen about the perverse outcomes that arise from attempts to determine such accountability. However, the determining of accountability remains necessary.

This is my solution to the problem in secondary schools. It is based on relating GCSE attainment to Y7 cognitive ability data for all pupils  including those identified as PP using the current DfE criteria.

 

1. Screen all intake Y7 pupils with Cognitive Ability Tests (CATs).

This will establish the same general patterns in relation to SATs, CATs and Social and Economic Status (SES) data that John Mountford found in his research that is reported here.

These data have been further summarised by John as follows.

 

SchoolABDEFG
% FSM12%10%9%7%7%22%
SATs Y7 mean105.68103.45105.91106.42107.07105.22
CATs Y7 mean100.2696.76105.92104.76109.3897.86
FSM SATs mean102.41100.33104.00106.05104.4498.18
FSM CATs mean92.9190.8098.83102.1397.2889.93
Cohort – FSM SATs 3.283.121.910.37 2.63 7.04
Cohort – FSM CATs 7.535.96 7.09 2.63 12.1 .93

 

SchoolHIJKL
% FSM44%15%15%4%15%
SATs Y7 mean100.68105.42105.42109.14106.57
CATs Y7 mean95.58103.35103.35107.61105.71
FSM SATs mean100.32101.83101.83104.33102.37
FSM CATs mean94.7792.8592.8597.7797.25
Cohort – FSM SATs 0.36 3.593.595.114.2
Cohort – FSM CATs– 1.19 10.510.59.848.46

 

The data for school C are omitted because of comparability issues.

 

2. Produce a ‘scatter-chart' showing the GCSE attainment of each pupil against the Y7 CATs score for that pupil

The following chart appears in a number of my articles illustrating how to validly compare the GCSE attainment data for different schools taking proper account of differences in mean intake cognitive ability.

Graph2

However, instead of the X axis showing the mean intake CATs scores of schools, it should be used for the Y7 CATs score of every individual pupil. The data points of PP pupils on the chart, would be labelled (eg with a different style of data point). The measure chosen for the Y axis could be the DfE defined ‘Attainment 8’, or other measure chosen by the school to recognise more attainment in technical and creative subjects. The school would have to make the case for the chosen measure with OfSTED.

As in the above chart, the regression line (that Excel can produce for you) shows the average performance of pupils in the school in relation to their individual CATs scores. Students appearing above the regression line have done better than the school average and  those appearing below the regression line have done worse.

On the basis of John’s and national data published by the CATs provider GL Assessment (p10), it would be expected that PP pupils would be bunched towards the left hand side of the chart on account of their lower mean cognitive abilities. However any such students that appear on or above the regression line have closed any SES gap between them and their non PP peers. The general pattern for the school will be obvious from the chart.

 

3. The school should then reflect the outcomes in terms of its curriculum and teaching and learning policies

How should schools best combat social and educational disadvantage? This is addressed in this article.  Becky (Professor Allen) has also pointed to the answers to this question in her articles.

We both agree that a change in DfE policy should result in higher proportions of educationally disadvantaged pupils in school intake cohorts generating enhanced general funding for the school to reflect the increased costs of the effective teaching and learning methods (for all pupils) that are needed. 

Even in the absence of any change in DfE policy, schools can and should use approaches to teaching and learning for all pupils that are proven to be effective. For example, those recommended by EEF for science teaching, that in fact work across the curriculum.

Primary schools can help themselves and their secondary colleagues, by also using cognitively enhancing approaches such as P4C. Further cognitive enhancement can be stimulated by changing school cultures as explained here. Don’t be put off by the title of the article – it really is relevant.

In terms of how the DfE should determine the amount of extra funding to be delegated to schools, together with how to hold schools accountable for outcomes, I believe the answer to both questions was provided by the Cumbria LEA in the early 1990s. I was one of the Cumbria heads that served on the LEA Working Party that devised the approach, so I know in detail how it worked. In those days there were substantial ‘Non-Statutory Special Needs’ funds to be distributed to schools through the funding formula of each LEA. Unlike other LEAs, Cumbria rejected a formula based on FSM in favour of Cognitive Ability Tests (CATs) data obtained from screening all Y7 pupils in October of the intake year. The Cumbria formula then delegated additional funds to schools, not individual pupils, through a formula driven by the numbers of pupils with CATs scores at various threshold levels below 85 (-1SD). Where there were significant differences in the scores on the three sub-tests (Verbal, Non-Verbal and Quantitative Reasoning), the CATs profiles for each pupil should prompt further testing for Specific Learning Difficulties (SLD), so enabling specific intervention for individual students, reflecting Becky’s important point about the diversity of learning needs.

Such charts have further uses. They are an even better way of presenting data for schools to evaluate their own standards and progress than the Cumbria CATs/school GCSE performance scatter-graph. This is because they can provide fine detail information over time for the school. The regression line enables the mean GCSE performance for the school to be calculated for sub-groups of cognitive abilities. For example, schools like Mossbourne Academy have ‘fair banding’ admissions policies, in which there are admission limits for each CAT band. Mossbourne. like many Academies have quartile bands.

A: 110 and above

B: 100-109

C:  90 – 99

D: below 90

The mean GCSE performance of the school for each quartile boundary can be read from the regression line, including the mean school performance for pupils of national average cognitive ability (CATs score = 100)

If in successive years the school becomes more effective in terms of teaching and learning then the school GCSE performance at each quartile boundary will increase (and vice-versa). Finer detail is also available. For example there may be differences in improvement/deterioration between the ability bands.

So GCSE attainment /pupil CATs score scatter-graphs and regression lines can not only provide sound pupil premium accountability, but much else besides for driving school improvement.

Share on Twitter

Be notified by email of each new post.





Comments

John Mountford's picture
Thu, 22/11/2018 - 21:43

Given a chance, Roger, what you propose offers a very workable and positive way out of the current situation where testing is a real blight on education. It also tackles a gross unfairness where schools with very different intake profiles are having to 'compete' with eachother for the education spoils.

You are right about accountability. It isn't going to disappear when the next government (or subsequent ones) takes power. We have come, and rightly so, to expect accountability from all sectors except, of course, for the political elite who, despite claims to the contrary, are not run out of town at elections unless their personal track record is blemished or their party slumps at the poles. Our first-past-the-post system makes sure that duff politicians can find their way to a 'safe seat' if they climb the greasy pole well enough. (Enough of that old chestnut!!)

So, part of the task must be to tackle the issue of accountability especially in relation to the PP and the vexed issue of progress. As you point out, that can only be justified if there is a level playing field for all schools to operate under and the measures used to establish accountability are valid. Professor Allen's suggestion to allocate the extra funding in the general budget works for me. Also, what you propose by way of replacing the flawed and highly discredited Key Stage 2 National Curriculum tests with Cognitive ability testing has huge merit. Despite my personal support for what you propose, I have one concern.

Few professional educationalists are prepared to even consider this as an option because of their flat refusal to accept that such tests are valid or that they should be used at all, presumably for moral or ethical reasons that escape me. This, of course, does not include those engaged in running programmes for grammar school selection and those secondary schools using CATs to supplement the meaningless evidence from SATs. As I say, apart from those exceptions, the whole issue of genetics, intelligence and the measurement of cognitive ability is anathema to most in education. Unless we can reverse that view and promote a more balanced one through engagement and debate by exploring the available evidence, your excellent proposal is likely, very sadly, to remain on the shelf. That said, I intend to copy your article to all the schools engaged in our research into SATs/CATs carried out earlier this year.

In the hope of chipping away at the prejudice and hysteria associated with the perceived heresy of deigning to raise the issue of genes again, I recommend readers take a look at this article from a recent edition of New Scientist.
https://www.newscientist.com/article/mg23931870-500-the-truth-about-inte...


Roger Titcombe's picture
Fri, 23/11/2018 - 09:39

Thank you for your comments John. I did not realise the full potential of regression analysis myself until I began to think about the Pupil Premium issues raised by Professor Allen. I used the same approach in my analysis of PISA scores, which is one of the most read (especially internationally) of all my articles.

https://rogertitcombelearningmatters.wordpress.com/2016/12/18/national-i...

I also note that Micheal Shayer uses it in his CASE evaluation paper.

http://www.education.uwa.edu.au/tsa/__data/assets/pdf_file/0014/1025150/...

https://rogertitcombelearningmatters.wordpress.com/2018/09/22/definitive...

Reurning to the Pupil Premium issue, our own work, supported by national data from GL Assessment shows the link between FSM and lower mean cognitive ability. It is worth repeating that my regression analysis is the way of proving once and for all whether the 'Attainment Gap' claimed by the Sutton Trust and the DfE really exists. If in general, Pupil Premium pupils do just as well as pupils with the same Y7 CATs score without Social and Economic Disadvantage (SES), then there is no SES-based 'attainment gap'. So come on the DfE and Sutton Trust,  here is an important research opportunity to prove me wrong.

On the IQ heritability issue, you are right to be frustrated by the spurious and ridiculous smearing of any consideration of the obvious implications for education of 'general intelligence' with 'sympathies with Nazism'.

However we must still be cautious with the conclusions of Plomin, Herrnstein & Murray and other genetics enthusiasts that are ignorant of the potential power of schools to significantly raise cognitive ability with the right kind of teaching and learning. It is here that I take issue with this statement on the New Scientist article.

 “Genetics gives us a blueprint – it sets the limits. But it is the environment that determines where within those limits a person develops,” says psychologist Russell Warne at Utah Valley University."

Not so, as the 'Growth Mindset' movement rightly argues, it is impossible to define a limit to the intellectual potential of anybody. See

https://rogertitcombelearningmatters.wordpress.com/2017/01/30/the-growth...

We must never estimate the power of the right sort of schooling to have a huge effect. Look at the Cumbria schools regression chart for the CATs score of 95 (37th percentile), where school GCSE mean points scores range from 30 to 45, a huge and potentially life changing difference.

The issues raised by Professor Allen's questioning of the Pupil Premium are not trivial.


Roger Titcombe's picture
Fri, 23/11/2018 - 13:17

Corrections - Really sorry to readers but there are two corrections needed in the table. Both apply to the bottom row 'Cohort - FSM CATs'. The figure for School G should be 7.93 and for School H should be 0.81. 


Roger Titcombe's picture
Fri, 23/11/2018 - 13:38

Another correction - Really, really sorry, but School A needs correcting as well.  'Cohort - FSM SATs' should be 3.27 and 'Cohort - FSM CATs' should be 7.35


Add new comment

Already a member? Click here to log in before you comment. Or register with us.