Pupil progress measure is ‘not just flawed but is a fabrication’ says data analyst

Janet Downs's picture
 24
What if measuring progress is nonsense? That’s the question asked by James Pembroke, data analyst with expertise in drilling school data.

Progress, he says, is not uniform. It depends on the child. Progress is uneven. One pupil may struggle to grasp the basics but will surge ahead when they are secure; another may absorb the basics quickly but then plateau. Progress isn’t an unwavering, forever upward gradient. And it can’t be broken down into ‘uniform blocks of equal difficulty’.

The progress measure is ‘based on an arbitrary scale between two ill-define, best fit and somewhat spurious end points’.

What teachers can do, James says, is ‘teach the curriculum’, judge whether pupils have achieved curriculum objectives and then ‘make a broad assessment’ of where they stand in relation to what could be expected at their age – ‘below, at or above’ at that point in time.

But teachers have ‘lost the plot’ if they develop data merely to satisfy senior leaders, governors, local authorities, academy chain head office, Ofsted and the Department for Education data crunching machine. All that’s needed is a single report which shows where pupils stand in relation to age-related expectations and which can be used to highlight gaps in learning. Formative assessment, in other words, used for the benefit of pupils, not summative assessment produced to placate the demand for data by people who forget children are individuals but who view them as widgets to be measured using a standardized ruler. As Russell Hobby, NAHT general secretary, says, ‘…perhaps the time has come to slay the sacred cow of progress.’
Share on Twitter

Comments

John Mountford's picture
Tue, 21/04/2015 - 21:21

I am always puzzled how anyone can expect individual pupil progress to be objectively measured, as if on a well-oiled production line, and even more puzzled that it is often implied, if not directly stated, that it is an ever upward journey. Even if it's not your particular experience to work with real children in 'real-time' and might be forgiven for making facile comments about the nature of their progress in a particular subject, ones own experience of life ought to appreciate that,
"Progress isn’t an unwavering, forever upward gradient. And it can’t be broken down into ‘uniform blocks of equal difficulty’." That's simply not how it works.

Take, for example, teaching someone to drive. I'm sure many a promising relationship has been wrecked at that fateful moment, when having negotiated, with ever increasing accuracy, reversing around a corner, the rear wheels mount the kerb and the car grinds to a standstill, only coat of paint away from a lamp-post. The cause of this lapse of performance from the expected can be anything from tiredness, to a lack of concentration, feeling like crap or simply not giving a toss for once. It happens. We are not machines and neither are children.

So what of your remark, Janet? - "But teachers have ‘lost the plot’ if they develop data merely to satisfy senior leaders, governors, local authorities, academy chain head office, Ofsted and the Department for Education data crunching machine."

It's not just teachers. It's the whole system. I believe we have lost our way in education and won't recover unless school leaders do just that, and lead their colleagues in revolt. What ever happened to the idea of solidarity? Sure, a united stand against this madness would require courage, but it would not be the first time such action has been taken when the cause demanded. If we truly believe "The progress measure is ‘based on an arbitrary scale between two ill-defined, best fit and somewhat spurious end points’, why aren't we taking the required action to challenge this folly?

Janet Downs's picture
Wed, 22/04/2015 - 08:19

John - teachers have been crushed by the data-crunching madness for the simple fact they must earn a living. Refusing to give the data required by senior management, governors (who are judged by Ofsted whether they have been robust in monitoring 'progress') etc could mean the sack. Head teachers who take a laid-back view to league tables, 'progress' measures etc can find themselves quickly removed if Ofsted give the school a low rating - academy brokers would be beating down the school door before the ink was dry on the Ofsted report.

Some heads and governors have, of course, been complicit in this by publicising their school as the 'best-performing' in the area (or even the country if they've earned the dubious approval of a SoS or junior school minister).

Perhaps unions could play a part. There was a nationwide boycott of SATs in 2010 - something similar could be done to stop the insatiable demand for unreliable and misleading data.

Janet Downs's picture
Wed, 22/04/2015 - 08:40

John - parents are also responsible if they take too much notice of these flawed measures and unnecessary tests such as SATS. An article in The Times Weekend supplement (18 April 2015) about how to help children through exams said this:

'Parental fretting isn't reserved for GCSEs and A levels either, now that SATs in the last year of primary school have assumed such importance: parents report that schools are starting revision in January and pupils are being given extra tuition to bag the elusive Level 6 so they are assured a top-set place at secondary school.'

I didn't know whether to laugh or cry. I think it's the latter. Extra tuition - at age 10/11 in order to get an 'elusive' Level 6 (obviously 5 is no longer good enough for these pushy parents). The way we test our children - in a way unknown in most of the rest of the developed world - is almost becoming child abuse.

The best thing parents of Year 6 pupils can do is recognise these tests have no educational value and are only used to judge schools.

PiqueABoo's picture
Wed, 22/04/2015 - 15:17

Nice distracting cliché re. “pushy parents” and L6, which is precisely the kind of thing that bothers me.

Y7 Sprogette is very whizzy and L5 curriculum content was simply not enough for her at a single-form entry, thus mixed ability, Middling State Primary. She's not pushed by us e.g. the closest we came to explicit school-stuff at home was reading integrated into the bed-time routine. I think we'd have had a much tougher time with this modest, astute and increasingly cynical child if Gove hadn't revived L6 or the school had ignored it. L6 meant she learnt some useful new things instead of having to redundantly repeat something for the 100th time for the benefit of the majority in the middle.

I loathe the sub-level progress and targets twaddle for various reasons, but suspect removing that would make it worse for a child like her. In my local world too many expectations are quite pitiful now and I don't know whether to laugh or cry about some of them. A few good and typically quite old teachers aside, I think the only things currently preventing her innate enthusiasm for learning being crushed are those levels/targets and to a lesser extent a shiny CAT score.

Take away the admittedly dire bureaucratic nonsense from a system that has long been selecting for people who are content to stay there (or even thrive) then what are the consequences for whom?

Janet Downs's picture
Wed, 22/04/2015 - 09:06

I believe ex-SoS Michael Gove didn't make expected progress when learning to drive - it took several attempts before he eventually passed his test. This was one of the embarrassing details about Gove which so horrified the Daily Mail. That was, of course, before Sarah Vine (aka Mrs Gove) went off to write for that illustrious tabloid.



rogertitcombe's picture
Sun, 26/04/2015 - 11:30

The real tragedy is a culture whereby teachers think they can only introduce demanding material to children if is both on the prescribed curriculum and there is a Level for it. I remember a young teacher who got his Y7 class to devise and carry out experiments to judge the absorbency of different brands of disposable nappies. If some pupils then go on to produce a valid 'value for money' comparison parameter then this involves far higher cognitive demand and Piagetian level than you can reasonably expect from an average Y7 pupil? So what? Some of them will get a long way, most less so, there will be loads of high quality pupil - pupil and pupil - teacher discussion, much practical ability gained in weighing and measuring and a good, very messy, time had by all, pupils and teachers alike.

Teachers - you are or should be a professional with a high degree of responsible autonomy and the confidence to try out your ideas and fearlessly discuss them with your colleagues.

Janet Downs's picture
Mon, 27/04/2015 - 10:12

Roger - the absorbency of disposable nappies lesson reminds me of the 'eggsperiment' we used to do on Industry Day. As part of a session on packaging, the pupils had to devise an attractive container which could protect an egg. They had a choice of materials and had to take into account protection, attractiveness and size (too big a container would mean fewer could be packed on the delivery lorry).

Not sure how this would have fitted with Levels. And was it maths or marketing or both? Didn't matter - it was great fun. I did a similar thing in Business Studies - the pupils had to produce a container to keep six edible snails (represented by screwed up paper) in one piece. It had to be attractive, be stackable, contain a product name and a recipe.

PiqueABoo's picture
Thu, 30/04/2015 - 21:29

"The real tragedy is a culture whereby teachers think they can only introduce demanding material to children if is both on the prescribed curriculum and there is a Level for it."
--

Yes, but I get the impression many primary teachers would argue that they're already struggling to get their class through the curriculum. Sprogette's primary did throw in some "fun maths" with an HTLA in place of occasional Numeracy lessons, but you had to be ahead of the curve to qualify.

Barry Wise's picture
Thu, 23/04/2015 - 17:11

Yes, PiqueaBoo....My experience with 2 children was exactly as you describe.


John Mountford's picture
Fri, 24/04/2015 - 14:25

PiqueABoo, your comment interested me because it says so much about the school system we have allowed to evolve in 2015:

"L6 meant she learnt some useful new things instead of having to redundantly repeat something for the 100th time for the benefit of the majority in the middle."

There's a lot about schooling that is not working well for our children's futures, including what Janet has unearthed here about progress. Would that it ended there!

However, I regard it as a scathing indictment of the profession if Michale Gove's intervention over L6 really ensured your daughter was taught something useful at the end of her primary schooling. And what, I wonder of the majority in the middle who required 100 repetitions in order to learn what was prescribed for them or, God forbid, those struggling at the "bottom" end?

Trevor Fisher's picture
Tue, 28/04/2015 - 12:45

back in the days when people actually understood children and schooling, it was recognized that children develop at different rates.

This went out the window with targets.

While its now not recognized that Albert Einstein would have failed the 11 plus and that Mark Rylance did not speak till age 9 (both statements are true) we can't get through to people in the bubble who think that schools are factories.

To break through we will need to attack the exam factory concept. No one actually agrees with it. But all politicians who back targets believe that that is what schools are.

trevor fisher.

Barry Wise's picture
Tue, 28/04/2015 - 13:12

Trevor

both statements are true

Actually, on the basis of a cursory check, both statements seem to be false - nearly all the reports about Mark Rylance being a late speaker cite 6 or "nearly six" as the age, not 9.

As for Einstein - according to the not infallible Wikipedia, he went to a Gymnasium (broadly a German equivalent of a grammar school) at aged 8. He did fail an entrance exam to a Swiss school when aged 16, though he apparently scored highly in the Maths and Physics papers. No real reason to think he would have failed the 11+.........just another urban myth really.

Trevor Fisher's picture
Tue, 28/04/2015 - 13:24

thanks important to get the facts right, for a blog site I don't check in detail, so it is good to get the fine detail correct. Both Einstein and Rylance had educational problems, yet ended up at the top of their professions. I have no idea about Rylance's career, and it is good he got to be a top actor. Yet as there is a major discussion in the theatrical world currently over the dominance of the public schools over the theatrical profession, there is a massive problem of selection in this area.

Rylance did have problems in speaking as I did myself. My speech therapy clinic north of the Hockley flyover in Birmingham (the building is still there... I have fond memories) saved me. Would the facility still be there in the modern world?

As for Einstein, urban myth says he was a patents clerk when he wrote his first major papers, not an academic. Would he have survived in the current world of physics, where to get into a top university like one of the Golden 5 (Oxbridge, LSE, Imperial, Kings College London) requires parents with a second mortgage?

lets get the facts right on individuals. But neither of these corrections falsify the argument about late and variable developers, which is the point. A society which tests kids at 4, 7 and 11 has got it wrong.

Or would you not agree?

Trevor Fisher

Barry Wise's picture
Tue, 28/04/2015 - 13:53

Trevor,

I don't buy the "high-stakes" claim made about KS2 SATS, let alone baseline or KS1 assessments. They may be "high-stakes" for schools (and consequently for Heads and even individual classroom teachers) but NOT for the children. The 11+, of course, is different in that it does have consequences for children and it is certainly true that a significant number of late developers will be missed by a selection test at 11.

On your other point, I do rather rate the new wave of posh actors such as Dominic West (Eton), Damian Lewis (Eton) and Laurence Fox (Harrow), am not sure about Tom Sturridge (Winchester), but doubt whether there is much new in all this, given John Gielgud (Westminster), James Robertson Justice (Marlborough) via Hugh Grant (Latymer) and Alan Rickman (Latymer). All it shows is that the UK's permanent ruling class like dressing up.

rogertitcombe's picture
Tue, 28/04/2015 - 14:16

Barry - Any parent or grandparent that has had their children pass through primary school will put you right about the high stakes nature of KS2 SATs preparation. Of course SATs are of no value to the pupils but this does not reduce their high stakes nature.

Parents, and I suspect most pupils, are well aware of the importance of SATs to their teachers, the head and the school. In the case of my granddaughter, she loves her school and would everything possible to avoid trouble for her teachers and the head.

This makes the pressure put upon our children even more unacceptable.

John Mountford's picture
Tue, 28/04/2015 - 18:46

You can buy what you like about testing of younger children, Barry. I utterly disagree with you and would never go shopping with you looking for a sound education experience for my grandson or any other young child for that matter. But, as we all agree on this site, it's a matter of opinion!!


Janet Downs's picture
Wed, 29/04/2015 - 08:39

Roger and Barry - Year 6 SATS are, as both of you say, high-stakes for the schools. This in turn puts pressure on the children. A recent TV programme about the biggest primary in the UK showed Y6 pupils getting SAT results in envelopes like GCSE pupils with some parents present. It was clear by the attitude of the school and the parents that SATs were high-stakes for the children (or made out to be). In reality they are of no educational value.

This school may, or may not, be typical in making such a fuss. But the Times Weekend supplement a couple of weeks ago ran an article about how parents could reduce exam pressure. It claimed SATs were important especially getting an 'elusive' Level 6 in order, the article said, for one's super-clever sprog to get into the top stream at secondary school.

UK children, especially in England, are among the most tested in the developed world. It subjects our children to pressure which is deemed unnecessary elsewhere. It's time it stopped.

Trevor Fisher's picture
Tue, 28/04/2015 - 19:40

in response to Barry, the high stake system is so deeply damaging that it is being questioned even in South Korea and China. Because of the very high suicide rates among teenagers in those countries.

And for primary schools, the pressures on children, who start school earlier than anywhere else in Europe are simply unacceptable, as is the level of obesity. This is the first generation ever which will die before their parents. Too much too young. Its amazing that this is not the biggest issue in the election.

And for the theatre, the debate among the experts is how much damage is being done to the theatre, and you can find many good articles in the last six months on the BBC website on this, its now a high profile issue. No doubt about the quality of the top actors, they have had fabulous sums spent on developing their talent. But this is now a TV and Cinema driven profession, drawing on multi ethnic talent across the globe.

What are we doing in the UK? Drawing on a small pool of upper class white kids. The experts are unanimous. This cannot be the way to run a world class drama system.

FOr John, perhaps an even more fundamental problem for this site and all blog sites. The damage to children cannot be a matter of opinion. Either it is empirically verifiable, and it seems to me absolutely to be so (even at the level of cognitive development, hence the debate about international tables for all their faults) or we are wasting our time.

Indeed, as so much blogging is merely about opinion, it has no political effect. As with football, where opinion is plentiful but the league tables really don't lie, ultimately the real world decides on the basis of performance. In my case, I am told my team Aston Villa are too good to go down.

Reality is they are not- and realilty is what we have to be about if we are having public discussions. And for education, I don't think it is a false comparison to say that international league tables are important, if currently flawed. We have to know whether the billions we spend on education are well spent or not. It can't just be a matter of opinion.

Trevor Fisher

rogertitcombe's picture
Tue, 28/04/2015 - 20:46

Trevor - We are in agreement about much in relation to education, including international league tables. Also, it appears, Aston Villa, in which we are joined by David Cameron (if only occasionally when his memory serves him), Prince William and of course Nigel Kennedy. And they are not too good to go down - what a worry.


John Mountford's picture
Tue, 28/04/2015 - 21:21

Trevor, my point was that no matter what the evidence, not all commentators will agree. As for the strength of evidence about the damage to children from international tables, the impact on their cognitive development tells just a small part of the story.

During SATs time, I have sat alongside children supporting the efforts of those that function below the threshold for disapplication and felt their vulnerability. Very often the cause of their panic is little more than their appreciation that the clock is ticking and they panic. With all the practice and coaching in the world, such youngsters struggle under these conditions and seldom produce their best.

One of the biggest problems for education is the league table mentality that has gripped the system, especially here. This has led to inappropriate teaching and learning as schools are expected to defy the natural distribution pattern of attainment.

rogertitcombe's picture
Wed, 29/04/2015 - 07:35

You are right John. In England politicians have misused PISA results and conclusions to force onto schools reforms that have damaged children. However, if English education was taken out of the hands of politicians then this would not happen. It would be left to a 'National Commission for Education' or something similar to interpret PISA data. The OECD PISA system is nothing more or less than serious international educational research. We should not blame PISA for its misuse.

Your last paragraph is absolutely correct. Market forces cannot 'close attainment gaps' and the imposition of such an ideology onto schools damages schools and teachers, but especially pupils.

Trevor Fisher's picture
Wed, 29/04/2015 - 04:10

I agree with John that league tables are flawed, and in a historical context I would liken this to the Soviet union centralized planning. However the original comment was that its a matter of opinion what effect changes have. This can't be right. We need to have general agreement on what counts as evidence. Obviously league tables do not count.

BBC radio 4 had a very good programme on this with experts at the time of the last PISA results but then the TOday programme ran an item which went entirely counter to what the experts had said. They had not listened to it.

That I think is the real problem. OECD do produce useful data. But journalists don't understand it, and there is no simple crib they can use. Ditto for January results tables in England.

As for the Villa, its a private grief, best left to those of us who will be there on Saturday. But on evidence, Prince Will has been photographed in the trinity road stand. No problem. On Nigel, I met him outside the North stand in 2013. no problem. But David Cameron.... we are back to what counts as evidence.

Trevor Fisher

PiqueABoo's picture
Wed, 29/04/2015 - 18:08

A few children in Sprogette's Y6 class were tutored for maths because the secondary set that based on the


PiqueABoo's picture
Thu, 30/04/2015 - 21:20

Belated take two.

A few children in Sprogette’s Y6 class were tutored for maths because the secondary set that based on the KS2 maths SAT. Some of those parents might have inflated views of their child’s brilliance, but I can see their point of view. In the very real world here there was no question that Sprogette’s maths SAT results and a place in top set were very important, but she could have passed the L6 in her sleep so we weren’t going to risk disrupting that with superfluous parental pressure. Perhaps some of the children who had to fight hard to try and reach a floor-level experienced pressure from the school, but I didn't detect any misery in the very best friend who was one of the floor-level borderliners.

I’m not sure what you expect from parents of the brightest children in terms of sacrifices for other children, but Sprogette routinely reports significant thumb twiddling waiting for the majority of the class in the Y7 maths top set lessons. She still loves the subject so I’m content enough, but it would be quite insane to want her working at a significantly slower pace in a lower set or, [$deity help me] in a mixed ability class with no credible differentiation and most likely going backwards. I can rip the mixed-ability English to shreds if you want me to justify the latter view.

Of course KS2 SATs often create secondary school “flight paths” and they’re one of the few weak defences we have against low expectations in some of the mixed-ability subjects e.g. we can point to the red stuff (alleged current level is behind the KS2 SATs based end-Y7 target) on the reports we’re given and ask when she might be taught the parts of the curriculum she needs to learn to turn that green. If nothing else it helps uncover teachers who are worth talking to i.e. the ones who then decide to drop the bullshit and discuss pragmatic reality.

Add new comment

Already a member? Click here to log in before you comment. Or register with us.