The myth of measurement -- the problem with high stakes exams

Francis Gilbert's picture
 9
"A school that is ridden by examination incubus is charged with deceit. All who become acclimatized to the influence of the system, pupils, teachers, examiners, employers of labour, parents, MPs and the rest, fall victims and are content to see themselves with outward and visible signs, class lists, orders of merit, as being of quasi-divine authority." (Edmund Holmes, 1911)

I'm reading a brilliant book at the moment, An Introduction To Assessment by Patricia Broadfoot which mounts the most convincing case I've read against standardised tests. Broadfoot's thesis is that for all standardised tests' apparent objectivity, they are inevitably subjective tests because of the narrow range of skills and knowledge that they assess. Skills such as problem-solving, personal effectiveness, thinking skills, and willingness to accept change are very difficult to quantify in a standardised test. She makes the argument that standardised tests arose in the 19th century as a form of centralised social control -- a more effective form than even a system like feudalism. Napoleon in France and Frederick the Great of Prussia used standardised tests to set the agenda as to what was taught in schools. She goes on to show that far from testing intelligence most standardised tests are actually assessments of social class and only serve to exacerbate social inequalities. She points out that the 11-plus is one of the most socially divisive exams in history with just 5.8% of children on Free School Meals attending grammar schools. Her quotation of Mary-Lee Smith's summary in 1991 of the problems with tests is worth quoting in full:

"The teacher may not have covered what is tested in the examination.

The test may be too long for students to concentrate on it.

Students may guess if the formate is a multiple-choice one.

The test may be incorrectly normed for a given population of students.

Perhaps one summative grade is offered which conceals substantial differences in the quality of performance in understanding contributory aspects of achievements.

The test may measure performance on a given day which has no connection with long-term retention by the student.

Students may be bored and disaffected and not engage to the best of their ability with the test or examination.

They may find the questions confusing or ambiguous.

They may not be able to apply their knowledge because of the limitations of handwriting or other mechanical abilities.

Many achievement tests merely measure endurance or persistence rather than performance.

Some students who are divergent thinkers may read too much into the test terms.

Some students become frightened and 'freeze up' in the testing situation, especially those who have no self-confidence or some kind of emotional or family disturbance."

Furthermore, research in the psychology of assessment has consistently demonstrated these factors all have an effect on performance that:


  • small changes in task presentation;

  • in response mode;

  • in the conditions under which assessment takes place;

  • in the relations between assessor and assessed and within students on different occasions

Share on Twitter

Comments

Ben Taylor's picture
Wed, 27/06/2012 - 23:52

So presumably there is no point in the process of being professionally assessed as being a teacher in a protected role...

Actually I am in some agreement with the problems of assessment, but rather than abandoning it would seek to mitigate these problems. Some examples of coping: rehearsal; emotional state as a function of effective culture; learning and assessment as a journey to a goal rather than one instant in time; the test is just one trial of many. When I think about it this is how the armed forces work.

I can't believe you really think this because otherwise why bother having the professional status of teacher and study towards a PhD?

I agree that assessment is a muddle, because at GCSE for example we have a system which is trying to prepare for further and higher study, be a final assessment, assess functional competence, assess both skills and knowledge without always distinguishing which, cover a very wide ability and has too limited a window for multiple attempts at success. In particular many of the problems you speak about relate to an attitudinal problem with our society seeing assessment as being one off, whereas I think you are right to emphasise the process.

Perhaps a return to O level is some attempt to separate functions of assessment. Would it perhaps be wise to emphasise A level as a preparation for university, or let universities set their own admissions exams, and offer other "lower stakes" pathways for post 16 learning?

Rebecca Hanson's picture
Thu, 28/06/2012 - 07:08

Here is how we replace standardised high stakes assessment with something much more effective and intelligent for students up to the age of 14:
http://mathseducationandallthat.blogspot.co.uk/2012_03_01_archive.html

Once we've down that the ways in which we can positively reform GCSEs and A-levels (starting from where we are moving from there in directions which are clearly evidenced) will be much more obvious

Janet Downs's picture
Thu, 28/06/2012 - 07:37

OECD warned last year in its Economic Survey of the UK that the excessive emphasis on exam grades in England risked distorting what was taught in schools and carried the risk of, among other things, teaching-to-the-test and neglecting important skills necessary for pupils' life after school. Universities have complained that undergraduates seem unable to cope with self-directed study and blame "spoon feeding" for exams.

Most other high-performing countries have moved to a graduation diploma. This can comprise end-of-upper-secondary external tests but doesn't necessarily have to do so. Some imaginative graduation diplomas work like a portfolio which gathers together pupils' achievements in many areas: tests (vocational and academic), continuous assessment, coursework, extended projects, practical work, activities outside school similar to Duke of Edinburgh Award.

University degrees no longer rely on just one, sudden death examination - pupils are expected to submit work to deadlines throughout their courses. And something like a PhD requires research, analysis and a dissertation not just regurgitation of knowledge. Vocational training doesn't rely on paper tests - trainee teachers demonstrate skill in the classroom; trainee doctors meet patients; trainee plumbers demonstrate that they have the skills not just the theory.

I've only discussed secondary and post-secondary assessment. Rebecca's post links to a possible, more effective method of assessing the attainment of primary children.

Rebecca Hanson's picture
Thu, 28/06/2012 - 08:11

I've been looking at assessment up to the age of 14 Janet. Improving the system requires that teachers be given some responsibility in assessing students in a world where they have been gradually but eventually profoundly stripped of their input into the formative assessment of students.

This culture can't be turned around instantly. You need coherent starting points. One of the biggest things schools can do is to look again at coursework. Understanding and exploring the possibilities of integrated formative and summative assessment will bring new possibilities.

But we can change the realities of competitive job and university places.

Ricky-Tarr's picture
Thu, 28/06/2012 - 08:22

Rebecca

Please would you explain (perhaps with some examples) what you mean here?

they {teachers} have been gradually but eventually profoundly stripped of their input into the formative assessment of students.


How can this be? Surely teachers are engaged in some kind of formative assessment all the time - seeing how students answer questions or perform tasks in the classroom, monitoring scores on online learning areas, marking homework etc. ?

Rebecca Hanson's picture
Thu, 28/06/2012 - 09:17

Thank you for your question Ricky.

If we go back to the 1980s, into pre-National Curriculum Culture, teachers and schools used to be responsible for writing their own curriculums. At this time schools were capable and competent in writing their own examination syllabuses up to the age of 16. Mode 3 CSE was used extensively by schools to create recognised qualifications which reflected the needs and strengths within their own communities. Here's a link which makes reference to that: http://www.nationalstemcentre.org.uk/elibrary/resource/2138/examining-at...

You can imagine how teachers would become very highly skilled - with what we would now consider to be masters level understanding of curriculum planning and and assessment - as they went through the process of learning to devise and deliver these exams.

When GCSEs were first introduced this culture was evident in the way they operated and were assessed. For example we had the 'coursework only GCSE maths' which was highly respected and led to both students and teachers becoming much more inventive, flexible and highly skilled. When you meet maths teachers who taught this qualification they are always outstanding teachers. They may not have been when they started with it but the became so because maths was constantly being 'invented' in the classroom and they had to learn very highly professional skills not only in the ways in which they managed students, examinations and curriculum but also in the much higher level of understanding of mathematics they developed compared with teachers who had not taught this syllabus.

We had problems because this success was rolled to too far to fast. When I did early GCSE we had to do 6 investigative projects of our own devising. Teachers received little or no CPD for this and it happened at at time when they were experiencing very serious cuts and no-one had any free lessons. It was hell for them and they rightly complained. But it had a very powerful effect on us. I was in a set of students who would in the past have done O-level in year 10, OA-level in year 11, then A-level in on or two years with the option for Further maths in year 13. Logically this structure was gone - we spent year 10 and 11 fussing around with our projects and playing cards a lot. How could we cope with A-level in 1 year after not having done the old content of O-level and AO-level? It was logical that the system would collapse. Teachers rightly protested that we needed less coursework and it was reduced. But in amongst all this the reality of what happened went rather unnoticed. For example from my set we did cope with A-level in 1 year. In fact 8 of us did it where in the past it had usually only been 2...... But as always insights which seem illogical crumble away in the face of hard rational opposition which makes obvious sense but doesn't actually reflect reality.

Anyway the amount of teacher assessed coursework reduced. It became more predictable and easier to cheat. Cheating was always a possible issue which needed to be addressed but the sudden abolition of the whole lot was to do with the issues associated with cheating and despite the benefits coursework brings to education and to the professionalisation of teachers.

Maths teacher became more intelligent and better at maths themselves because they had to interact with maths that students were inventing. In maths I'm particularly worried that the stats coursework has gone. Even though it had degraded to a level where is was much more predictable and led than it should have been, students and teachers were still doing a heck of a lot of stats which is one of the reasons why England's attainment in this area is so high.

So I accept that my use of the word 'formative' here was untidy as you could argue that mode 3 CSEs and coursework are summative rather than formative assessment. In fact they are both. I could instead have focused on the ways in which teachers have been stripped of other aspects of their powers in formative assessment of students - how they now have to give deeply flawed and meaningless level increments and pre-defined report sentences instead of being able to personally assess and describe the progress of the child but it's not actually the main point I want to make. This circumstance would also be substantially improved by the creation of the kind of systems of integrated formative and summative assessment I've describe in my blog.


Perhaps after reading this you might have some insight into the reality that the biggest challenge we have faced in improving standards in maths has been doing it in the context of the deprofessionalisation of staff due to changes in the assessment system. We have made real and significant progress but the benefits have been countered by force from outside maths education.


I envisage a future which is different to what has gone before because of the capacities the internet brings. Many qualifications can be criteria based, making them much more flexible and they can be assessed by teachers with robust QA processes run by new types of examination companies which audit their practice. It's already happening with small scale qualifications such as INGOTS but we need a coherent policy environment in order to persuade the large education companies to make the very substantial investments needed to create the kinds of large systems which would work for the mainstream.

In essence we need a government commitment to ending SATS in about 5 years for schools which are instead using robustly accredited systems of integrated formative and summative assessment. There would be no loss of central insight into data, in fact much more would be available allowing better quality insight.

Francis Gilbert's picture
Thu, 28/06/2012 - 09:51

http://www.fairtest.org/ is good website I think for highlighting the problems with standardized tests.

Janet Downs's picture
Thu, 28/06/2012 - 07:58

There's also the problem of cheating. In the US, the public education director of Fair Test told TES last year that he had confirmed cases of cheating in 27 states and the District of Columbia. The largest scandal was in Atlanta where teachers held cheating "parties" where answers were corrected before being submitted for marking. One child who sat under the table throughout a test still managed to pass.

It's worrying that Gove and his admirers should so admire the US system when such underhand practices occur in over half of the states.

http://www.tes.co.uk/article.aspx?storycode=6111394

Janet Downs's picture
Thu, 28/06/2012 - 10:06

Thanks, Francis, for your link to Fair Test. Its fact sheet on Tests, Cheating and Education Corruption is especially illuminating.

http://www.fairtest.org/sites/default/files/Cheating_Fact_Sheet_8-17-11.pdf

Add new comment

Already a member? Click here to log in before you comment. Or register with us.