Twenty countries took part in the first international Adult Skills Survey – but only four reached the target response rate of 70%. There’s always the possibility of bias in results when not enough people respond so the Organisation for Economic Cooperation and Development (OECD) asked countries where the response was insufficient to do a Non-Response Bias Analyses (NRBA).
Two countries failed to complete the NRBA: England and Northern Ireland.
I wrote to OECD asking if the figures for England/Northern Ireland should have been withheld because the NRBA hadn’t been done. I also asked if it was likely that the data for England/Northern Ireland would be redacted.
This was OECD’s reply*:
“Despite the fact that certain elements of the non-response bias analysis were not completed, the overall conclusion was that the data from England and Northern Ireland was of sufficient quality to be published.”
There followed an explanation in dense technical language (see Appendix) most of which went over my head. However, OECD provided a summary:
1 Basic non-response bias analysis: England/Northern Ireland performed all required analysis.
2 Extended non-response bias analysis: England/Northern Ireland did NOT perform all required analysis.
The OECD said the response rate for England was 59% and for Northern Ireland it was 65%. OECD warned that “data users need to be cautioned that the analysis is based on assumptions about the range of proficiency scores for sampled cases that have no scores (41% of the sample).”
So, the Reader’s Companion to the Adult Skills Survey urged caution. The technical notes urged caution because the analysis was “based on assumptions”.
But no such caution was exercised in media and politicians’ soundbites. Important insight has been lost in the desire to use the findings to score political points and make simplistic generalizations.
So, what did cause the poor showing of English and Northern Irish 16-24 year-olds? Possible answers include:
1 Very few pupils study Maths and Literacy beyond 16.
2 The teaching received in school doesn’t prepare pupils for applying these skills in different contexts.
3 Schools focus too much on getting pupils through exam hoops rather than concentrating on deep, long-lasting learning. This is a consequence of school league tables.
It’s important to remember the survey did not exclude those educated privately. So if the education system in England/Northern Ireland failed the 16-24 year-olds as many commentators have said then this reflects on the whole system not just that provided by the state.
for further details of OECD findings and recommendations following analysis of the adult skills data. The survey’s insights go far beyond where countries lie on the league tables.
*email dated 17 October 2013 from William Thorn, Senior Analyst, OECD Directorate for Education and Skills, Programme for the International Assessment of Adult Competencies
APPENDIX: OECD information about Sampling Error
“Sampling error: The design effect due to unequal weights is 1.35 for England for a sample size of 5 131; and 1.54 for Northern Ireland for a sample size of 3 761. The effective sample size, which is the sample size needed to achieve the same sampling variance as a simple random sample, is 2 425 for England and 2 773 for Northern Ireland. The effective sample size was computed as the number of cases with plausible values divided by the overall design effect (using the literacy component first plausible value), which accounts for both unequal weights and clustering. The United Kingdom’s address sample was an equal probability sample in both England and Northern Ireland. Variation in the selection probabilities was introduced from (a) subsampling households for addresses containing multiple households, and (b) the within-household selection at the person level. Further variation in the weights was added through nonresponse and calibration adjustments, although the Consortium followed standard procedures to balance bias and variance.”