Last year's PISA test results, which show where New Zealand students' reading, maths and science skills rank compared to other countries, are expected to be skewed, the Ministry of Education says. Photo: pixabay
A skewed sample is likely to have boosted New Zealand's scores in the most recent PISA tests of reading, maths and science.
There were 4682 15-year-olds who participated in the OECD's Programme for International Student Assessment (PISA) last year. But the group was not a representative sample and was more likely to do well in the tests than a fully representative sample, the Education Ministry says.
The problem happened because only 72 percent of the schools and students invited to participate did so, short of the benchmark of 85 percent of invited schools and 80 percent of invited students.
RNZ reported at the time that some principals refused to participate because their students were already under pressure from Covid-related disruption to their learning.
The ministry said its analysis found the results due to be published in early December would probably not be an accurate representation of New Zealand students' performance.
"Our student sample was found to be slightly biased. The analysis showed slightly more high-achieving students participated. This means the results will likely show Aotearoa New Zealand in a more positive light than if we had an unbiased sample, potentially by about 10 PISA points."
- Read the full report: PISA 2022: Aotearoa New Zealand Non-Response Bias Analysis.
It said students who went to private schools and schools in richer neighbourhoods (high-decile schools) were over-represented, as were students who regularly went to their classes.
"Unsurprisingly, students who were chronically absent around the time PISA 2022 was conducted in New Zealand were less likely to take part, and students who were regular attenders were more likely to take part."
The ministry said readers would have to keep the bias in mind when considering New Zealand's results.
"If the data shows better outcomes than in the past, on performance or in survey data, we cannot be certain of the extent to which it is due to the full population improving or higher achievers being over-represented in the data.
"Where results show worse outcomes than in the past, we can be fairly confident that the outcomes are truly poorer.
"We cannot be certain that if there are changes, what the size of the change is. For example, country 'rankings' become difficult to conclude."
New Zealand's scores in the tests have dropped by 22 to 29 points over a period of nearly 20 years and in the previous round of testing in 2018 ranged from 508 in science to 494 in maths.
The ministry said sample bias might have affected previous years' results and the results of other countries.
"It is important to note that the bias may not affect trend or country comparisons to its full extent, because a similar (although, likely, smaller) bias may also affect past results and the results of other countries and economies."
But the Education Ministry said New Zealand's results would still be included in the PISA report due in early December.
"New Zealand's results, as well as 12 other countries (including Canada, Hong Kong (China) and Latvia) who did not meet the PISA sampling standards, will be included in the international PISA reports published by the OECD. Each volume of the international report will include a 'Readers Guide' which describes each country's data limitations in detail, and readers are alerted to these limitations throughout the OECD volumes via annotations," the ministry's hautū (leader) operations and integration, Sean Teddy, said.
Teddy said New Zealand had done well to include as many schools as it had in last year's tests.
"The adversity that families and students have gone through in the past few years was immense, and so were the impacts of attending school in the context of a public health crisis. In that context, our response rate of 72 percent was a success, given that 40 percent of secondary students attended school regularly over the survey period, and is a true credit to the hard work of participating schools."
University of Auckland associate professor of education Dr Aaron Wilson said PISA had become increasingly high-stakes for many countries and the ministry's findings meant people would have to be very cautious interpreting the results. He said even without the bias in the 2022 sample, the results should not be considered in isolation from previous years.
"If we're looking to see a pattern or a trend over time in results, even without that [bias], you need more data points anyway. You can't conclude much from a single data point. A lot will depend on - because it's an international comparison - knowing exactly what the level of sample bias was for other jurisdictions."
Wilson said he was not surprised that a lot of schools opted out of the tests, particularly those in lower and mid-decile schools that had struggled most with Covid-19.
New Zealand Initiative researcher Dr Michael Johnston said a 10-point difference was "a reasonably big deal" given that reading scores had fallen 25 points in 18 years. He said it would be difficult to interpret the results with any certainty, even allowing for the difference.
"They can put a bound of 10 points on the likely extent of the error, but within that it's very had to say where things sit."
Johnston said schools had a lot to deal with due to Covid-19 and it was understandable that some had opted out of the tests. He said the ministry could compare the 2022 results with the results of students who sat the 2018 assessment and had similar characteristics to the 2022 sample in order to find out if achievement had changed.