Why CAT scores are better than SATs for judging a school's performance

Bogstandard's picture
 0
The Cognitive Abilities Test (CAT) is an assessment of reasoning skills. It consists of nine short subtests within three 'batteries' or sections. The batteries are labelled as verbal, quantitative and non-verbal reasoning.

The verbal reasoning battery assesses reasoning processes using the medium of words; the quantitative test battery assesses the same processes but use numbers as the symbols. The non-verbal battery again tests reasoning processes but use shapes and figures.

The quality of prior teaching, opportunities to learn, parental support, pupils' educational experience, and their emotional and physical well-being, including nutrition, may affect pupils' performance on all educational tests. However tests of the taught curriculum - reading, mathematics, spelling etc. - are likely to be influenced by these factors to a much greater degree than reasoning tests.



Chart 1 is for Cumbria schools in 2002. It plots Average GCSE points (using the scale that applied at the time) against mean intake CAT scores. This chart shows the usual very high correlation between CAT scores and GCE performance. Each school is represented by a point on the chart. It is clear that schools above the regression line are doing better than the Cumbria average and schools below are doing worse. By choosing 'average points' as the measure of performance all GCSE grades count and the chart is not distorted by cramming for C grades.



Such a chart can only be produced in LAs like Cumbria and Hackney that screens all their intake pupils with the CAT tests. It seems to me to be a powerful tool for evaluating school effectiveness, however as with all statistical measures there are health warnings. This is a simple version of the chart  that does not show confidence limits. These are needed to indicate the significance of the distance of the school from the regression line. For this and other reasons such data cannot be reduced to an easily comprehensible 'Value Added' index for each school. However, such an approach seems to me to provide a very sound basis for a discussion between school leaders and an LA team charged with raising standards in its schools.

Note that 'Gas Street Comprehensive' is doing pretty well for its pupils even though the mean intake CAT score (16th percentile) is very low and its GCSE results fall well below floor targets.  Hundreds of such highly effective schools have fallen victim to crude league table driven market forces and floor targets.

Some of the problems with SATs are shown by Chart 2. Both charts are from my 2006 paper, Cognitive Ability and School Improvement, Practical Research for Education, Issue 36.



Unlike CATs, SATs are not consistent from year to year. Although SATs also correlate well with GCSE scores, the CATs correlation is higher. Of much more importance however is the high stakes nature of SATs. A primary school can have a small but significant influence on its mean Y6 CATs score through the best kind of developmental teaching. However, as we have seen, huge gains can be made in SATs scores when primary schools are threatened with closure and heads are sacked because of failure to meet floor targets. If these inflated SATs scores are used as the baseline for secondary Value Added then secondary schools admitting pupils from 'improved' primaries stand to be seriously damaged by SATs driven Value Added measures, unless of course they inflate their GCSE results by the same methods. Cramming for KS2 SATs clearly works, but the CATs scores will not rise with the SATs scores and it is Cognitive Ability that counts in the long term throughout the school system. There is growing evidence that the reality could be even worse: cramming for SATs and GCSE success may actually be depressing cognitive ability.

It is therefore very important that CATs and not SATs are used as the baseline for secondary Value Added, so it's time all pupils took them.

 
Share on Twitter

Add new comment

Already a member? Click here to log in before you comment. Or register with us.