On Tuesday, Statistics South Africa released the latest results from its Quarterly Labour Force Survey (QLFS), telling a depressing story: in the last quarter of 2020, employment was nearly 9% lower than a year earlier, corresponding to 1.4 million fewer people employed. This stands in stark contrast to the results of the NIDS-CRAM survey, released a week earlier, which showed a far stronger recovery in employment. The NIDS-CRAM data suggested that employment in October 2020 had recovered close to the levels of February 2020. Why are the estimates so different, and what does this suggest about a possible employment recovery? 

The NIDS-CRAM employment results in brief

The NIDS-CRAM survey followed more than 5,000 adults via three rounds of telephone interviews in 2020, recording labour market status in February (pre-pandemic), April (lockdown Level 5), June (Level 3) and October (Level 1). Bassier, Budlender and Zizzamia showed that, compared to February, standard employment for working-age adults was 17% lower in April, 15% lower in June, but only 1% lower in October – a stronger than expected recovery. This extremely surprising result was consistent with several internal and external checks, such as Statistics South Africa’s monthly sales data, and other researchers using NIDS-CRAM found the same large recovery. NIDS-CRAM results for April and June also matched the corresponding QLFS results. In stark contrast,  as the figure below shows, the QLFS recorded very little recovery between quarters 3 and 4.

Figure: Employment patterns in QLFS compared with NIDS-CRAM

The measured employment proportion, in levels, is actually higher in NIDS-CRAM than QLFS. We present the trends relative to February for ease of comparison.

Possible explanations for QLFS and NIDS-CRAM differences

Differences in measurement

One possible explanation is that the two surveys measured different things. NIDS-CRAM asked about employment in October, whereas the QLFS conducted interviews across the fourth quarter (ie, October, November, and December combined). This could be important because the QLFS may include some effects of the Covid-19 second wave, as cases started to rise and people restricted their economic activity, and restrictions were imposed in hotspots in December. This could be particularly important if the QLFS conducted most of their interviews towards the end of the quarter. On the other hand, the QLFS report shows similarly large declines in employment across most provinces, which suggests the second wave is probably only a partial explanation. 

Additionally, NIDS-CRAM asked respondents if they were employed at any point in October, whereas QLFS asked about employment in the last seven days only. It is possible that part-time temporary work increased, overstating the NIDS-CRAM employment recovery compared to QLFS. However, this is unlikely to be a major issue given that “days worked in a usual week in October” also showed a strong recovery in NIDS-CRAM. 

Random sampling error

Two different surveys, even if they’re representative of the same population, are generally expected to give slightly different results. While a random sample of individuals should on average reflect the broader population, there is always a chance that the random sample may be quite different. Imagine we each flip a coin 10 times: we know we should both get tails about half the time, but it’s quite possible that you get tails four times and another person gets tails seven times.  

The “confidence interval” or “margin of error” around an estimate is meant to indicate such uncertainty, due to what is called random “sampling error”. However, even if we take into account the margins of error on both surveys, the NIDS-CRAM and QLFS employment estimates are just too far apart for this to fully explain the divergence. The QLFS estimate (a 9% decline in employment) is very far below the conventional lower bound of the NIDS-CRAM confidence interval (a 5% decline).  

Non-sampling error

The confidence intervals around the estimates assume that individuals were randomly sampled from the population. One potential flaw in this assumption is that even if people are randomly chosen for interviews, those who choose to respond may be different from those who do not respond – so we could end up with a biased sample, which is not representative of the broader population

This could be particularly problematic in 2020 because both NIDS-CRAM and the 2020 QLFS had to conduct interviews by telephone, due to the pandemic. It is difficult to figure out exactly what types of people respond to telephone interviews, how they differ from the broader population, and what kind of bias this could introduce.

The NIDS-CRAM and QLFS sampling teams adjusted for this issue by assuming that respondents who are similar to non-respondents (based on observable characteristics such as race, gender, language, income and area) can stand in for these non-responders. The problem is that non-respondents may be different even compared to respondents who are similar to them – they might be unobservably different – and if these unobservable differences are also related to a person’s employment status, then the weights cannot fully correct for these response biases. 

This “non-sampling error” has the potential to be a serious issue, and could explain a large part of the NIDS-CRAM and QLFS divergence. However, because it is by definition about unobserved biases, it is difficult to judge, 1) whether it is indeed a significant cause of divergence and 2) whether NIDS-CRAM or the QLFS deals with it better. The methods used to adjust for non-response do differ in the two surveys, but to investigate this further would require more detail than is available in the public QLFS technical documentation. 

Differences in representativeness 

Aside from non-random sample selection, one clear difference between the surveys is the population they aim to represent. While the representativeness of NIDS-CRAM is not straightforward, it is drawn from a representative sample of South Africa in 2008, which was then reweighted and topped up to reflect Statistics South Africa’s estimates of the population by gender, race, age and province in 2017. This means that no adjustments were made for changes in the population between 2017 and 2020; and further, no adjustments were made for changes not reflected in the reweighting. This was made clear in the NIDS-CRAM documentation. For example, international migrants are under-represented in NIDS-CRAM, a group perhaps less likely to see an employment recovery.

The QLFS is superior to NIDS-CRAM in this regard: As we understand it, the QLFS results are designed to be representative of the South African population in 2020. On the other hand, it’s unclear how important this difference in representativeness is. As per the figure above, NIDS-CRAM and the QLFS seemed to match fairly well in quarters 2 and 3 of 2020.

So how much did employment recover?

We have discussed a number of factors that could explain some part of the NIDS-CRAM and QLFS divergence, but we can’t point to any single factor as the definitive issue, and we also can’t definitively say which survey provides the better picture. Given that the QLFS was designed to have a more representative pre-pandemic sample, we think the QLFS estimate of a 9% decline between Q1 and Q4 employment is probably closer to the true story than the NIDS-CRAM estimate of a 1% decline. The QLFS also has a much larger sample size than NIDS-CRAM, though it is unclear how this helps if there is systematic non-sampling error. 

There is thus some uncertainty about what the labour market really looked like at the end of last year. This article is an attempt to transparently explain some of the issues.  A collaboration between the Statistics South Africa QLFS team and the NIDS-CRAM team would be invaluable in helping understand why the surveys diverged when measuring employment. This would be useful not only for understanding what happened at the end of last year but would also help make sense of the employment numbers going forward, as we track South Africa’s path out of the pandemic. DM

Ihsaan Bassier and Joshua Budlender are Economics PhD candidates at the University of Massachusetts, Amherst. Andrew Kerr is an Associate Professor in the School of Economics at the University of Cape Town. All authors were involved with the NIDS-CRAM survey: Ihsaan Bassier and Joshua Budlender published employment analysis, and Andrew Kerr is part of the sampling team. We write in our personal capacities. We thank the NIDS-CRAM labour and data production researchers for several insights discussed in this article, without implicating them in our conclusions.