On Sunday 25 October, the Sunday Times reported that education economist Professor Martin Gustafsson had uncovered a fundamental error in the 2016 Progress in International Reading Literacy Study (PIRLS) data for South Africa. 

PIRLS had initially found there was no improvement in Grade 4 reading levels in South Africa between 2011 and 2016. Gustafsson found this to be incorrect.

Among all participating countries, South Africa had in fact seen some of the largest gains during this period. 

Gustafsson’s findings led the International Association for the Evaluation of Educational Achievement, which runs PIRLS every five years, to remove all references to South Africa’s lack of improvement from its website and report, and declare the data unreliable.

Outside our annual matric results, PIRLS is one of a handful of measurements – alongside the Trends in International Mathematics and Science Study and the Southern and Eastern Africa Consortium for Monitoring Educational Quality – that South Africa uses as a “reliable” and “objective” measure of how learners are doing, both at a national level and against international benchmarks.

Two findings stood out from the 2016 PIRLS report: first, that the schooling system had basically stagnated in literacy gains, exhibiting no improvements in reading at Grade 4 level; and second, that 78% of learners are not able to read for meaning in any language by the end of Grade 4. 

These two findings have, in the main, driven the rapid growth of national campaigns and largely dictated the focus of the system in the last few years, with reading declared a national imperative.

Although Gustafsson has proved the stagnation in reading improvement false, assuming the rest of the 2016 data is correct, the finding of persistently poor reading levels still stands. 

This is important, because even with large improvements in reading levels, South Africa continues to lag behind other countries that were part of PIRLS 2011 and 2016.

Importantly for South Africa, the PIRLS mishap has further highlighted that we have a severe measurement problem, one that has resulted in us at times outsourcing our ability to measure the impact of our interventions at a large scale. 

We are constantly searching for “what works”, without a clear sense of how we are doing until it is generally too late (i.e. the release of the NSC results). 

We simply do not have a robust and reliable national view of how learners are performing in earlier grades.

Improvements in the schooling system are often small and gradual, and assessments that would monitor this improvement would need to be sufficiently rigorous and conducted at strategic grades in order to be effective. 

A robust and reliable national view, based on standardised assessments, would allow for system-wide early interventions and help to determine the success of such interventions.

Of course, this is not to suggest that when the Annual National Assessments still existed – providing systematic evaluations at grades 3, 6 and 9 – we had full clarity of the problems in the system and how to fix them, but we did have a sense of how learners were progressing in the system before their point of exit in Grade 12.

Since the release of the PIRLS report in 2017, to date there has been a large surge in literacy interventions in the country, driven both at a state and non-state level. 

But with all this time and investment, when and how will we know the system is beginning to move in the right direction? 

If the answer is that we will wait until the next PIRLS report, or wait to see improvements in the next NSC results, we are setting ourselves up for failure. 

If the system is to continue to develop and eventually stabilise, we need to understand the gaps, challenges and successes much earlier.

Finally, the ability to perform large-scale assessment is important because without internal measures and existing data in place to test such results against, it  becomes difficult to refute findings like those from the IEA, and with this data already out for three years and widely disseminated, the damage has largely been done both nationally and internationally.

Fortunately, in response to the need for a more rigorous national assessment system, the Department of Basic Education (DBE) is piloting different elements of a set of national systemic evaluations for grades 3, 6 and 9, which, in line with international trends, focus on the assessment of mathematics and languages. 

Additional proposed elements, which would provide a more holistic view of the health of the system, are the assessment of schools through a whole-school evaluation, and the quality of district support. Given delays caused by Covid-19, a full pilot is planned for 2021.

It is critical to remember that despite their limitations, studies like PIRLS provide a fairly good indication of the general problem with literacy levels in the country, in all languages. 

They also expose other systemic and largely racialised socioeconomic realities that we are unable to escape – the large difference in performance of African learners and African languages compared with other groups.

These studies also serve the purpose of highlighting less obvious elements that require support from social investors, broadly classified as interventions that work towards capacitating the system to deliver. 

While teachers and learners form the core of education, at the nexus of their interactions are the structures that support the functioning of the system as a whole, thereby enhancing or restricting the meaningfulness of these interactions. 

Like any good organisation, the DBE requires the appropriate systems, processes and technology to support the delivery of quality and relevant education. With regard to literacy, one such systemic element that requires support is the development of African language reading benchmarks. 

Strikingly, because of limited focus on and investment in African languages, there is an understanding of the overall comprehension problem through studies like PIRLS, but there is limited understanding of where the specific gaps are and what could be expected from a typical learner at each grade level. 

Comprehension is the endpoint of the reading process, and there is a need to better understand and measure which components of reading learners are struggling with and why. For African languages, we have not been able to sufficiently articulate where learners should be performing at the end of each grade and concurrently develop standardised benchmarks to assess this.

A recent publication in the South African Journal of Childhood Education offers ways in which reading benchmarks may be developed for early-grade reading and underscores underlying contributors to “comprehension” and the different components that promote or hinder reading in African languages. 

However, this work is at different stages of development for different languages and there is an urgent need for increased funding if such benchmarks are to be developed. 

Such benchmarks are critical to ensure learners are developing at the right pace, to hold different players in the system accountable, and ensure a reliable and accurate measurement of progression.

The development of African language benchmarks presents an important and tangible way in which social investors can support the improvement of literacy levels in the country at systems level. There are multiple current initiatives in the space, however there is a need for more collaboration to ensure funding is effectively targeted to ensure coverage of all languages and resulting reliable benchmarks that can be used nationally to set targets and gauge the health of the system.

The uncovering of improvement in South Africa’s performance in PIRLS is a small ray of light which, to some extent, illustrates that we are doing something right in the system and making strides towards improving literacy (largely understood as the bedrock for all other improvements in education).

In short, things do seem to be improving, but the improvement of literacy must remain a national concern. 

Professor Gustafsson’s findings should drive further prioritisation and increased, consistent and more deliberate investment in literacy. DM