How to use data badly: levels of progress

Henry Stewart's picture
 16

Good data can illuminate and help a school effectively tackle areas of underperformance. Bad data can confuse and mislead and cause schools to focus on the wrong areas. The increasing use of flat levels of progress across a school is a classic case of bad use of data.

However the use of 3 levels of progress (defined by the DfE as “expected progress”) is now widespread, being included in the DfE data tables, being the focus of the Ofsted data dashboard and often used in school comparisons by local authorities. (3 levels of progress means going from a level 5 at age 11 to a B at GCSE, or a level 4 to a C or a level 3 to a D.)

If somebody presents you with a table showing the % of students making 3 levels, or 4 levels, of progress across a school, the only appropriate response is to ask them why they have such a lack of understanding of school measurement that they could think this is in any way useful. (If it is an Ofsted inspector please point out that they are acting in direct breach of Ofsted guidelines – see notes below.)

I think I am not alone in recognising this, as my previous articles on the subject (Why 3 Levels of Progress is a Very Silly Measure & “Making Expected progress”: A Deeply Flawed Measure) remain in our top 10 most viewed posts month after month. A simple glance at how the % achieving 3 levels of progress varies with the pupil starting point should make this clear:



Levels of progress is presented as a neutral value added measure. In fact it is massively biased in favour of schools with academically strong intakes. While 99% of students arriving with a 5a achieve what the DfE defines as “Expected progress”, only 15% of those on a 3c do so.

“Expected progress” is not a useful measure



This is a dangerous measure. If the message form the DfE is that “expected progress” for all students is 3 levels than a level 5 student is only expected to get a B at GCSE. When I’ve pointed this out, the response from otherwise sensible education professionals has been “Ah, but we are increasingly setting 4 levels of progress as the target for all students”. This response misses the point that progress differs with the starting point. In reality a 5a student should have a target of 5 levels of progress (to an A*), while 3 levels is a real stretch for a student starting with 3c.

Let me emphasise this: Setting an across the board 3 levels or 4 levels of progress target is posed as setting high expectations. But it actually means that those we now call the “more able” are not being asked to stretch themselves.

I know of one young person who left teaching in frustration after working at a school where virtually no students got A grades at GCSE. This establishment, highly lauded by the previous Secretary of State for its success on the 5 A-Cs (with English and Maths) measure, had focused its curriculum solely on getting students to a C. That is the danger with bad data measures. Schools working to maximise “expected progress” need only to focus on getting students to a B.

It can also create especially misleading data feedback within schools. If classes are set at different levels of ability, then a teacher of the higher set can achieve all Greens (on target) without stretching their students, while a teacher of a lower set could have mainly Reds (behind target) even if their students are achieving well above national average levels of progress for their starting point.

Note the variation within the broad levels. While 85% of 4a students achieve 3 levels of progress in Maths, only 39% of 4c students do so. The reason is simple. For both 4a and 4c students, 3 levels of progress means achieving a C at GCSE. But achieving a C requires a much greater progress for a 4c student than for a 4a student, as the latter is further ahead at age 11.

Levels of progress vary with pupil starting points



The 2014 transition tables for Maths and English give the following % achieving 3, 4 and 5 levels of progress, for each KS2 starting point:



The colour coding is to help with target setting. Green indicates the target to be adopted if the aim is to be in the top 25% for progress. Yellow is the level for the top 50%.

Implications for Targets



Below are an indication of how targets might vary in a school if you were expecting your students to progress at the level that 50% of students nationally achieve, or that 25% achieve. If you wish to be more ambitious and go for, say, top 10% then you can use the table above to calculate the resulting target grades.



Some will say these targets are not ambitious enough for those entering at lower levels. I am only using 25% and 50% thresholds as examples. However I am arguing for consistency. If a student with 4c in Maths is set a target of 4 levels of progress to get a B at GCSE (which only 5% of 4c students nationally achieve) are you also setting your 5c students a target of an A* (which 9% of 5c students achieve)?

At the same time I am wary of the trend to more and more ambitious target setting. One school near me has set a target of one full level a year for all students. I heard reports of another setting 4 sub levels (ie, one and a third level) a year. The former would mean all level 3 students achieving a B at GCSE. The latter would mean them achieving an A.

Maybe some magical new educational method will enable this level of progress. But to me setting targets entirely unrelated to what the data tells us is more reminiscent of Soviet Stakhonovite targets or those of the Chinese Great Leap Forward than targets rooted in sound educational knowledge.

I hope this data is useful. Thoughts and comments are welcome.

 

Caveat: I accept that this article is entirely based on the assumption that KS2 scores are a useful basis for GCSE targets. I am aware of the evidence that age 11 CAT scores are better predictors and the recent analysis from Education Datalab that shows that individual student progress is actually very variable. However this article is a response to the fact that most schools, local authorities, the DfE and Ofsted do use KS2 scores as the basis for GCSE targets.

Data Source



The GCSE transition tables are produced each year, and are available for all subjects. I found the 2014 figures in the Raise Online Document Library, under "Transition matrices, target setting": “2014 Key Stage 2 to 4 National subject transition matrices unamended”)

Note: Ofsted guidance



Ofsted guidelines: After one of my previous posts, criticising Ofsted’s focus on levels of progress, an Ofsted Director got in touch to point out that Ofsted understands that levels of progress are related to a student’s starting point and that an inspector should never use a figure for 3 levels of progress or 4 levels of progress across a school. Indeed Raise Online breaks down the % of pupils making 3 or 4 levels of progress by their KS2 starting point, and compares each separately to the national figures.

The Ofsted Handbook for Inspectors specifies clearly that progress must be judged relating to pupil’s starting points. For instance:

185: “When judging achievement, inspectors must have regard for pupils’ starting points in terms of their prior attainment and age.”
208: “Inspectors must consider the proportions of children who have made at least typical or better progress from their starting points”

Contact: henry@happy.co.uk

Share on Twitter Share on Facebook

Be notified by email of each new post.





Comments

John Mountford's picture
Sun, 29/03/2015 - 16:29

Henry, I've recently been talking with friends whose children are in primary schools about the changes to levels. I'm no longer a governor and have not kept up with what is happening about levels. With this in mind and in response to this article, I have some questions/comments.

I was recently asked what is happening to the old levels and I am stumped how to reply. Friends tell me that the staff are now working independently within their school to agree about what represents suitable progress for different pupils. If I have this correct, how is this going to fit in with 'expected levels of progress'? How will the changes affect the analysis of data henceforward?

Another set of conversations with friends has thrown up something I feel far more concerned about. Two families, quite independently of eachother, told me roughly the same story about their children's everyday experiences of being educated across Key Stages 1-2. It goes like this-
Child 1 has reached top of KS 2 -
Child 2 is in Reception/early KS1-
The gap in ages is about four years (give or take) in both instances -
This is their concern, as expressed to me -
Child 2 is having a very different experience from their older sibling- in general not as pleasant for the child and increasingly stressful for the family.

I went on to ask them why they felt this and was told-
Consistent pressure on Child 2 to undertake more formal learning tasks eg. being expected to learn to record maths operations and learn times tables earlier than Child 1 had, getting less access to expressive/creative play opportunities, sitting at a desk for longer periods - you get the idea.

What I was left with was a sense that parents believe the focus in these schools has changed in just a few short years. Clearly, I feel the focus on data may be having a direct negative impact on children's experience of schooling. Is this more widespread than my limited encounter with the phenomenon might suggest? Are teachers and school leaders aware, if this is a more widespread experience, of the changes and what is to be done about it?

I trust I have not strayed too far from your original focus, Henry.

Henry Stewart's picture
Sun, 29/03/2015 - 17:52

John, thats a really thoughtful comment. Its a long time since my primary governor days so I'm a bit our of touch. Can any other readers answer John on changes in primary schools?


Barry Wise's picture
Mon, 30/03/2015 - 11:27

Consistent pressure on Child 2 to undertake more formal learning tasks eg. being expected to learn to record maths operations and learn times tables

Times tables in Reception? Surely this cannot be true?

John Mountford's picture
Mon, 30/03/2015 - 13:07

Who would know, Barry? More to the point, who SHOULD know and if it's happening, what do we think of this kind of development?

Surely, if the government expects 'more' (whatever that is) by the end of primary ed. then the pressure to perform has to begin in earnest at some point earlier than previously.

Brian's picture
Sun, 29/03/2015 - 20:54

The change to assessment without levels (AWL) is causing considerable confusion in many, maybe all, primary schools. Gove removed levels but offered nothing concrete as an alternative, preferring to let schools find their own way. Schools are now working with new descriptors and seem to be focusing on ' emerging' , 'expected' and 'exceeding' requirements for each year group. So a child who is performing at 'expected' outcomes at the end of Y3 will be performing at 'emerging' after the summer break because they will then be assessed against Y4 expectations. This is, apparently, going to be much clearer for parents. All the schools I work with are aware of the national commission looking into AWL and take the view that the DfE has not set this up at this late point because Ofsted is reporting things are going well in schools. What constitutes good or better progress under AWL remains debatable. With the government's suggestion that end of KS2 tests will have a simple pass / fail outcome I guess data analysis is sorted in the most simplistic and damaging way possible.

In terms of expectations of younger children ... the government view is that higher standards are achieved simply by requiring children to do and understand concepts at a younger age. It takes no account of child development and requires schools to accept that rote learning in maths is synonymous with good learning. A child who can compete a Maths operation by memory and get the right answer without any understanding age 6 is better educated than a child who completes this aspect of learning, with understanding, age 7!

It's educational nonsense and we're stuck with it. Have a look at the current media commentaries on what are the 'key issues' in the election. I haven't seen education mentioned in one of them yet.

Brian's picture
Sun, 29/03/2015 - 21:29

I should also have said that, in my opinion, AWL has the potential to provide a more secure base for assessing pupil progress and next steps that assessment through levels. However this is significantly at risk because of the continued political imperative to have a single assessment process which supports pupil learning AND allows a simple way of preparing league tables. This isn't possible and I suspect the league table imperative will always triumph.

It will fall at the same hurdle as levels ... assess KS2 schools' effectiveness by the percentage of pupils attaining Level 4, league-table on the same criteria, measure schools' effective support for disadvantaged pupils in the same way ... and then berate the schools for focusing on the Level 4 boundary.

Goodness me, I am in a cynical frame of mind this evening!

John Mountford's picture
Sun, 29/03/2015 - 21:43

Thanks, Brian. I made an effort to dig a little deeper into the issue of life without levels after posting above. It seems to me that a system already subject to broad interpretation, dare I say confusion, has been further complicated. At least while the levels were in use there was something tangible to relate judgements about standards to. Do we have here yet another example of seeking to raise standards by dictat (and by shifting the goal-posts)? I hope parents are brighter than me when it comes to understanding the new regime and more trusting than I am of the government's view that there is clearly more to be squeezed out of our children and young people in the name of international education league-table blagging.

Henry, your observation about schools apparently opting to set "more and more" ambitious targets is of great concern to me as a grandparent and raises important questions about the role of teachers and school leaders in creating undue pressure on the young. One of the issues I failed to mention in my initial response above, was homework. This is something my grandson never fails to comment on (he's just nine) as he sees time out of school as 'his time', and so it is and should be, to play and have fun. It is alarming to hear friends airing their concerns about the type and amount of homework set for ever younger children. What has happened to the idea of balance and why are professionals seemingly aiding and abetting successive governments in turning education into little more than a number-crunching exercise where shallow, rote learning is apparently valued so highly ? Or am I just foolishly unable to go along with the crowd? No. To me, the fact is the emperor is as close to being naked as I can accept!!

Brian's picture
Sun, 29/03/2015 - 21:57

' .... why are professionals seemingly aiding and abetting successive governments in turning education into little more than a number-crunching exercise where shallow, rote learning is apparently valued so highly'

Ofsted.

agov's picture
Mon, 30/03/2015 - 09:05

Effectively, so I gather, the 2014 national curriculum requires that children now do things at an earlier age than was previously the case. So John's friends are not imagining things.

Not really sure that schools are treating AWL as anything other than a change of name with the new name not yet known.

Brian's picture
Mon, 30/03/2015 - 09:29

One of the things Ofsted has made clear is that the change to AWL must not be levels under a different name. All the schools I work with have largely abandoned levels, except in Y6 and Y2 where they still apply for this year. They are trying to make sense of a 'post-levels' confused picture. They aren't substituting 'emerging', 'expected' and 'exceeding' for, for example, Level 3c, 3b and 3a. Of course there is no national agreement about what terms such as 'exceeding' mean and schools are interpreting this differently.


Janet Downs's picture
Mon, 30/03/2015 - 09:41

Brian, agov, John - the Government consulted on 'Performance descriptors for key stage 1 and 2 statutory teacher assessment'. The response was not enthusiastic. Respondents thought the descriptors were ambiguous and confusing. You can judge for yourself here.




Brian's picture
Mon, 30/03/2015 - 09:53

Ambiguous and confusing indeed and, in maths, designed to give an impression of improved standards by assuming that ability to repeat a maths operation successfully by rote is the same as a deeper understanding of fundamental maths concepts.

It's as secure as telling children that multiplying by 10 is easy, you just add a 0 to the end of the number.

Janet Downs's picture
Mon, 30/03/2015 - 10:02

Henry - I know that if I were still teaching, my low-to-bottom set English groups would be coloured Red. What 'progress' had they made with me? Very little as judged by external tests. That obviously makes me a very bad teacher.

So what did they achieve? They spent two years studying Shakespeare, modern plays ('The Long and The Short and The Tall was always popular), poetry, rhetoric, novels (selected from such popular choices as 'Animal Farm', 'To Kill a Mockingbird' and, yes, 'Of Mice and Men') and short stories. They discussed them, acted bits out, watched videos, designed stage sets, drew storyboards, wrote about them (the least popular requirement). We compared texts about the same subject in different genres. For example, one Set 3 group compared Keats's 'Isabella, or the Pot of Basil' with two pre-Raphaelite paintings. This wasn't easy stuff; neither was it dumbed-down. And I read aloud to them all the time.

But judged on results, my pupils weren't making 'progress' since their GCSE results were around the F/E mark. I can't tell you how angry that makes me.

Martin Richardson's picture
Mon, 30/03/2015 - 11:16

Henry, 'expected progress' is indeed a poor measure. If we plot the KS2 intake score for secondary schools against the 'Best 8' score there is a very clear pattern reflecting what you say.

A school with a KS2 intake score of 25 (Level 4c) will on average get a Best 8 score of 248 (a little below 8 Ds at GCSE). So on average fewer than 2 'levels' of progress.

It is only when a school's KS2 score reaches 30 (so just the 200 or so secondary schools with the highest KS2 intakes) that the average Best 8 score of 368 (8 Bs) reflects about 3 levels of progress. (KS2 score of 30 is borderline 4a/5c)

When a school's KS2 score is 32 or higher is the average Best 8 score is 416 (average score of A), so 4 'levels' of progress from 'level' 5.

At present the value added score for KS4 is a much better measure, as it takes account of a school's intake. I understand the planned Progress 8 score will also take account of expectations relative to KS2 scores.

'Expected progress' should wither be calculated taking account of KS2 scores, or scrapped.

PiqueABoo's picture
Mon, 30/03/2015 - 22:57

Y7 Sprogette's maths department is aiming for three levels of progress across KS3, which for her with shiny KS2 SATs is a good thing.

T

PiqueABoo's picture
Mon, 30/03/2015 - 23:05

I accidently hit return :(

The traffic light problems are with the subjects beyond English and maths. For instance she did Spanish at primary, but in Y7 it is (new to her) French and the end year target is the KS2 RWM average plus two sub levels.

Add new comment

Already a member? Click here to log in before you comment. Or register with us.