PISA versus SATs -- which assess reading the best?

Francis Gilbert's picture
 7
The key question that arises from much of the data that's generated about pupils' attainment is what test best assesses pupils' ability in reading. We know that the current government like the PISA (Programme for International Student Assessment) a great deal and are suspicious of Key Stage 2 tests, which they don't like because they show our teachers are doing a great job in improving children's reading -- pass rates have doubled in fifteen years.

The PISA reading test is predominantly multiple choice and highly problematic in my view. If we examine a sample question, we'll begin to see why:

"Read the text and answer the questions which follow.

IN POOR TASTE

from Arnold Jago

Did you know that in 1996 we spent almost the same amount on chocolate as our Government spent on overseas aid to help the poor? Could there be something wrong with our priorities? What are you going to do about it?

Yes, you.

Arnold Jago,

Mildura

Source: The Age Tuesday 1 April 1997

Arnold Jago's aim in the letter is to provoke:

•Guilt

•Fear

•Amusement

I suppose the obvious answer is guilt, but you could see how some children with reasonably reading abilities might answer "amusement" or "fear" because there are elements of the letter which are faintly ridiculous and/or menacing. Other reading questions ask pupils to fill in the details for a Warranty Card, read an old fashioned fairytale, an informative booklet about bees and a library timetable. Very dry stuff indeed. It struck me that many students might not know what a "Warranty" card is, particularly if they are from more socially disadvantaged backgrounds. The questions that are asked are either multiple choice question or require very "closed" answers; there's little room for broader readings of the texts, certainly nothing that allows for a personal response. It's a very mechanical test that means computers can mark it.

Let's compare with the Key Stage 2 reading SATS. In my view, although flawed, they are much better than the reductive PISA reading test. For the reading test, pupils read a selection of different texts and then answer some closed, short questions -- quite similar to the type that PISA ask -- but then are required to answer questions like this, which appeared in the 2009 Key Stage 2 reading SATs:

"15a. The idea of a child living in a tree is quite humorous.

Explain what else is funny about Norman’s situation."

In my view, there's no question that the Key Stage 2 SATS are a much better test of reading; they check to see if pupils have understood the underlying meanings behind texts, have understood high level concepts like irony and are able to articulate personal responses. The PISA does none of this. That's why we need to trust our home-grown test results much more than the flawed ones devised by quantitative researchers who are looking for correct, mechanical responses rather than personal, thoughtful ones. Furthermore, the SATs' levelling system is quite robust; pupils attaining a Level 4 overall in reading will be competent readers and will have done well on this test. Attainment below Level 3 indicates pupils are having great difficulty reading.

For all their faults, the SATs tests are much more enlightened than the PISA. Let's hope that Gove doesn't bring in tests like the PISA and allows room for personal responses. Otherwise, he won't be testing reading ability in any meaningful fashion.
Share on Twitter

Comments

Andrew Old's picture
Mon, 20/12/2010 - 18:11

I think you are missing the point here. SATS questions are changed every year and students are massively prepared for the test and it has huge implications for their school. PISA questions, as I understand it, change less from year to year and aren't prepared for and are designed for ease of comparison. For this reason PISA is likely to be a more reliable measure when making comparisons.

Francis Gilbert's picture
Mon, 20/12/2010 - 18:16

I think I'm making a very important point that the PISA simply doesn't assess reading in a meaningful fashion. You need human markers to assess pupils' work, not computers. The PISA allows no room for personal response.

Andrew Old's picture
Mon, 20/12/2010 - 18:41

No test is perfect, but your condemnation of PISA seems to be based on its objectivity, which is the whole point. It is not trying to create a complete profile of a child's abilities, it is simply trying to judge them well enough to compare with other countries and over time, something SATs simply aren't very good at.

Francis Gilbert's picture
Mon, 20/12/2010 - 21:01

I don't think the PISA test is objective at all, it's a highly subjective test which has been written by statisticians searching for objectivity. It doesn't test reading skills in the more searching, meaningful ways that the SATs do.

Andrew Old's picture
Mon, 20/12/2010 - 21:47

Okay, I'm lost. A moment ago you were saying it was marked by computers. Now you are saying it is highly subjective.

Which is it?

Francis Gilbert's picture
Mon, 20/12/2010 - 21:57

The test is subjective in the way it is devised. Computers mark the test: however the answers those computers are looking for must be within the parameters devised by the people who made the test. Hence, its subjectivity.

A genuine reading test examines whether pupils read for the underlying meanings of texts. Only human marking can discover that.

Andrew Old's picture
Mon, 20/12/2010 - 22:17

Setting the questions is, of course, a matter of subjective judgement, but when I was talking about objectivity I was talking about the marking, because it is consistency in marking over years that will make it useful for comparisons over time.

Add new comment

Already a member? Click here to log in before you comment. Or register with us.