How to use data badly: levels of progress
Good data can illuminate and help a school effectively tackle areas of underperformance. Bad data can confuse and mislead and cause schools to focus on the wrong areas. The increasing use of flat levels of progress across a school is a classic case of bad use of data.
However the use of 3 levels of progress (defined by the DfE as “expected progress”) is now widespread, being included in the DfE data tables, being the focus of the Ofsted data dashboard and often used in school comparisons by local authorities. (3 levels of progress means going from a level 5 at age 11 to a B at GCSE, or a level 4 to a C or a level 3 to a D.)
If somebody presents you with a table showing the % of students making 3 levels, or 4 levels, of progress across a school, the only appropriate response is to ask them why they have such a lack of understanding of school measurement that they could think this is in any way useful. (If it is an Ofsted inspector please point out that they are acting in direct breach of Ofsted guidelines – see notes below.)
I think I am not alone in recognising this, as my previous articles on the subject (Why 3 Levels of Progress is a Very Silly Measure & “Making Expected progress”: A Deeply Flawed Measure) remain in our top 10 most viewed posts month after month. A simple glance at how the % achieving 3 levels of progress varies with the pupil starting point should make this clear:
Levels of progress is presented as a neutral value added measure. In fact it is massively biased in favour of schools with academically strong intakes. While 99% of students arriving with a 5a achieve what the DfE defines as “Expected progress”, only 15% of those on a 3c do so.
“Expected progress” is not a useful measure
This is a dangerous measure. If the message form the DfE is that “expected progress” for all students is 3 levels than a level 5 student is only expected to get a B at GCSE. When I’ve pointed this out, the response from otherwise sensible education professionals has been “Ah, but we are increasingly setting 4 levels of progress as the target for all students”. This response misses the point that progress differs with the starting point. In reality a 5a student should have a target of 5 levels of progress (to an A*), while 3 levels is a real stretch for a student starting with 3c.
Let me emphasise this: Setting an across the board 3 levels or 4 levels of progress target is posed as setting high expectations. But it actually means that those we now call the “more able” are not being asked to stretch themselves.
I know of one young person who left teaching in frustration after working at a school where virtually no students got A grades at GCSE. This establishment, highly lauded by the previous Secretary of State for its success on the 5 A-Cs (with English and Maths) measure, had focused its curriculum solely on getting students to a C. That is the danger with bad data measures. Schools working to maximise “expected progress” need only to focus on getting students to a B.
It can also create especially misleading data feedback within schools. If classes are set at different levels of ability, then a teacher of the higher set can achieve all Greens (on target) without stretching their students, while a teacher of a lower set could have mainly Reds (behind target) even if their students are achieving well above national average levels of progress for their starting point.
Note the variation within the broad levels. While 85% of 4a students achieve 3 levels of progress in Maths, only 39% of 4c students do so. The reason is simple. For both 4a and 4c students, 3 levels of progress means achieving a C at GCSE. But achieving a C requires a much greater progress for a 4c student than for a 4a student, as the latter is further ahead at age 11.
Levels of progress vary with pupil starting points
The 2014 transition tables for Maths and English give the following % achieving 3, 4 and 5 levels of progress, for each KS2 starting point:
The colour coding is to help with target setting. Green indicates the target to be adopted if the aim is to be in the top 25% for progress. Yellow is the level for the top 50%.
Implications for Targets
Below are an indication of how targets might vary in a school if you were expecting your students to progress at the level that 50% of students nationally achieve, or that 25% achieve. If you wish to be more ambitious and go for, say, top 10% then you can use the table above to calculate the resulting target grades.
Some will say these targets are not ambitious enough for those entering at lower levels. I am only using 25% and 50% thresholds as examples. However I am arguing for consistency. If a student with 4c in Maths is set a target of 4 levels of progress to get a B at GCSE (which only 5% of 4c students nationally achieve) are you also setting your 5c students a target of an A* (which 9% of 5c students achieve)?
At the same time I am wary of the trend to more and more ambitious target setting. One school near me has set a target of one full level a year for all students. I heard reports of another setting 4 sub levels (ie, one and a third level) a year. The former would mean all level 3 students achieving a B at GCSE. The latter would mean them achieving an A.
Maybe some magical new educational method will enable this level of progress. But to me setting targets entirely unrelated to what the data tells us is more reminiscent of Soviet Stakhonovite targets or those of the Chinese Great Leap Forward than targets rooted in sound educational knowledge.
I hope this data is useful. Thoughts and comments are welcome.
Caveat: I accept that this article is entirely based on the assumption that KS2 scores are a useful basis for GCSE targets. I am aware of the evidence that age 11 CAT scores are better predictors and the recent analysis from Education Datalab that shows that individual student progress is actually very variable. However this article is a response to the fact that most schools, local authorities, the DfE and Ofsted do use KS2 scores as the basis for GCSE targets.
The GCSE transition tables are produced each year, and are available for all subjects. I found the 2014 figures in the Raise Online Document Library, under "Transition matrices, target setting": “2014 Key Stage 2 to 4 National subject transition matrices unamended”)
Note: Ofsted guidance
Ofsted guidelines: After one of my previous posts, criticising Ofsted’s focus on levels of progress, an Ofsted Director got in touch to point out that Ofsted understands that levels of progress are related to a student’s starting point and that an inspector should never use a figure for 3 levels of progress or 4 levels of progress across a school. Indeed Raise Online breaks down the % of pupils making 3 or 4 levels of progress by their KS2 starting point, and compares each separately to the national figures.
The Ofsted Handbook for Inspectors specifies clearly that progress must be judged relating to pupil’s starting points. For instance:
185: “When judging achievement, inspectors must have regard for pupils’ starting points in terms of their prior attainment and age.”
208: “Inspectors must consider the proportions of children who have made at least typical or better progress from their starting points”