Statisticians on the frontline: How are humanitarian disasters quantified?

By Simon Allison 2 April 2014

Numbers are the foundation of the humanitarian world: how many died, how many were displaced, how many need assistance. But getting these numbers is a messy, complicated business – and, consequently, often unreliable. SIMON ALLISON examines how statisticians and humanitarian organisation count the cost of conflict and disaster.

In a world of suffering, journalists play uncomfortable games with humanitarian disasters. Just how many people must die before a disaster makes the front page? How many must flee their homes and countries before it is a crisis of sufficient gravity to interest readers, viewers and listeners?

The answer is subjective, as each editor and audience is different. But the underlying principle is the same: tragedies can be measured and compared; suffering is quantifiable.

To do this, journalists need numbers: body counts, displaced person counts, refugee counts. So do international organisations such as the United Nations. They rely on numbers to figure out how much money and which resources to spend on a particular disaster. Humanitarian groups need the numbers to coordinate their responses and determine where to dispatch relief efforts first. Advocacy groups need the numbers to raise awareness of the disaster and funds to pay for their supplies and efforts.

These numbers are the foundation upon which the edifice of humanitarian intervention rests. They are the currency in the world of disaster response. Without them, organisations would be going in blind: unaware of the scale of the problem and therefore ill-equipped to deal with it appropriately.

Here are a few recent examples of the type of statistics thrown around: The UN High Commissioner for Refugees (UNHCR) says 5.9m people worldwide were forced to leave their homes in the first six months of 2013, compared with 7.6m in 2012. In Africa, the UNHCR counted 3.7m refugees as of June 2013. More than 1,000 South Sudanese have fled their country every day into neighbouring Ethiopia, Kenya and Uganda since the outbreak of violence there in December 2013, reports Doctors Without Borders (MSF). About 500,000 people have left northern Mali to become either Internally Displaced Persons (IDPs) or refugees in neighbouring countries since fighting began in January 2012, according to research by Refugees International, a US-based humanitarian organisation. Human Rights Watch claims that 55,000 Somali exiles are living in Nairobi, Kenya’s capital.

Obviously, none of these numbers is exact. They are nice and round, rough calculations rather than precise counts. This makes sense, given the context in which the figures are generated. The people listed above are all fleeing or moving through war zones and natural disaster areas, places where the normal order has disintegrated so dreadfully that people are forced out of their homes. Record keeping is not a priority and often not possible.

This begs the following questions: where do these numbers come from? Axelle Ronsse, an epidemiologist with MSF, is grappling with these questions in Bangui, the capital of the Central African Republic. Her job is to count people and bodies in the sprawling displaced persons’ camp around the city’s main airport. The numbers she gauges will determine MSF’s response to the crisis. “We should look at where the problem is and the sectors where we have an epidemic or something like that,” she told Africa in Fact. “It’s also important to plan the number of structures we put in this camp and the numbers of staff we have to hire, and if we have to plan a vaccination or anything.”

To estimate the number of people in the camps, Ronsse uses three different techniques. First, she taps into the camp’s leadership structure. The camp is divided into 88 sectors, each headed by a leader. Ronsse liaises with each sector leader and asks them for an accurate count. This is not straightforward and is subject to manipulation. Sector leaders are motivated to inflate their figures because food distribution is dependent on the number of people living in each sector. “When people see you counting them and asking them, they always think about food distribution and people try to increase their number,” Ronsse explained. “I understand. If it was me I would do the same in their situation.”

To compensate, the MSF epidemiologist uses two other statistical methods based on random sampling of the camp’s population. Both rely on a global positioning system (GPS) survey of the camp’s area, which Ronsse generates herself with a handheld GPS device.

With this information, a computer is asked to identify 50 random points within the camp. At each of these points, she goes into the nearest house and counts the inhabitants. As Bangui’s airport camp is an open, transient camp, she must also ask if anyone else will return at night. “Ideally I should do it at night but for security reasons I cannot.” She then puts this data into special software, which uses carefully designed algorithms to extrapolate the sample into an overall estimation.

The third method is similar, but instead of 50 points the computer will specify 15 areas of 25 metres squared. Ronsse and two of her team will cordon off each area with red and white construction tape, and survey all the houses that fall within. Again, the raw data is fed into the software, which comes up with an overall number. Taken together, these three methods should provide an accurate picture of the camp’s IDPs. If Ronsse had access to satellite images of the area, she would incorporate this method too. Satellite images used alone, however, can produce unreliable figures, she warned. “It’s not enough by itself,” she said. “I know that there are some empty houses and I cannot see what house is empty or not.”

This explains the 10,000 fewer persons counted by MSF in the Bangui camp than the 100,000 identified in early January by the UNHCR, which has relied on satellite imagery, for the most part.? As a rule, the UNHCR maintains the most comprehensive database of refugee statistics. It is the first port of call for anyone looking for reliable, big-picture information. It monitors and tracks refugees all over the world. It makes this information public through an online population database and yearbook. This task is just as daunting as it sounds.

“Data collections remains a complex process involving many actors— governments, NGOs, implementing and operating partners, and the United Nations country teams, among others,” according to the 2012 yearbook, the most recent available. “This process is often a collaborative effort, in many cases requiring agreement by all parties involved. In the refugee context, data collection is typically coordinated by UNHCR and the government concerned.”

The UNHCR has carefully refined this process over the 64 years of its existence. “Over the past decade, the methods for collecting refugee-related data have remained virtually unchanged, with the main techniques continuing to be censuses, registrations, and surveys or estimations,” according to the UNHCR yearbook. “Some countries rely on a single method while others use combinations to provide refugee statistics… registration has remained the main method used by UNHCR.” By registration, the UNHCR is referring to the forms filled in by refugees or asylum seekers. This information is then fed into the proGres database, a specially-designed computer programme which allows UNHCR analysts to crunch through larger datasets than was possible previously.

To improve its accuracy, the agency also trains dozens of staff members every year in collection techniques and data management in the field, and has partnered with Statistics Norway, a Norwegian government unit, which provides the UNHCR with more skilled staff.

While the UNHCR is the leading source of refugee data, the Internal Displacement Monitoring Centre (IDMC), a Geneva-based advocacy group, responded to a UN call to track internally displaced people, an even more daunting task. “A few differences that stand out are that people that cross borders are generally accounted for by host country authorities either in the process of refugee status determination or as part of national or international assistance programmes,” explained Alexandra Bilak, IDMC’s head of policy and research, in an e-mail.

“Internal displacement is not a legal status, meaning that people need not be accounted for and registered (and this [is] normally not encouraged),” she wrote. “Also, people crossing borders tend to follow somewhat similar routes and crossing points, while people displaced in their own country may go to any place within their own country. Finally, at the end of crises, the return and repatriation of refugees are generally managed processes (either by governments or international agencies or both), which has an impact on keeping track of their numbers.”

The IDMC uses household surveys and extrapolation techniques to come up with its numbers.

The UNHCR and IDMC numbers are the most relied upon by international agencies because they are considered relatively rigorous. But not always. For example, governments provide more than a third of the UNHCR’s refugee numbers. These can vary widely in quality, especially when governments bring a political agenda to bear on the collection of statistics. “Perhaps the most important feature about internal displacement—especially if you factor in disaster-induced displacement—is that it happens worldwide and in countries with very different levels of political openness, data systems, and civil society and academic sectors,” Bilak said. “The accuracy and reliability of data on internal displacement varies accordingly.”

As a result, major international humanitarian organisations do not always trust these numbers. Instead, they employ people like Brian Root, a quantitative analyst for Human Rights Watch, a New York-based pressure group. He is responsible for double and triple-checking all data before they are used in any kind of public communication.

Root considers several items: “how something was measured, what were the definitions used, what social constructs were at play in deciding what/who/how to measure, what was the sampling methodology, what was the survey instrument, what assumptions are made in the statistical analysis, what biases are present, etc.,” he wrote in an e-mail to Africa in Fact. “It’s a long list. And therefore, we often don’t feel comfortable citing numbers—especially those collected in active conflicts—because many, many numbers that are generated simply don’t pass many of the methodological tests.”

The Syrian report published in December 2013 by the American Bar Association (ABA) is an example of suspect methodology, according to Mr Root. The lawyers group went beyond merely counting displaced Syrians. They interviewed nearly 900 refugees. The ABA asked them why they left the country, what types and level of harm they suffered, and who was responsible. The international media widely publicised the report’s headline conclusions.

“The issue with the ABA numbers is that they cannot be generalised to anyone besides the 800+ people that were interviewed,” Root wrote. “So when they put out a number that says that people claimed government responsibility for violations 19 times more than non-government forces, this number is only representative of the claims of 800 people and not a single person more. (This is not to say that the government probably is not responsible for the majority of attacks, because they likely are.) This is because of the difficulty in getting a true randomised sample in such an insane environment.”

The very nature of working in a conflict zone means that it is misleading to use randomised, localised samples to explain national issues, Root emphasised. “Even if [the ABA team] had gotten to everyone they had selected, the original sampling frame was people on the lists of aid agencies in these certain areas,” he wrote. “So those people wouldn’t be representative of all Syrian refugees—only those who had applied for aid from these specific agencies. It would not represent any refugees not on those original lists. For example, they will trend towards lower economic status, as those with money would probably not be receiving aid from these groups. Those who may be Alawite or other pro-government refugees have fled to the Syrian coast. Surely their responses on who was responsible would differ.”

Other recent figures which Root has had to treat with caution are the conflicting reports of casualty counts from South Sudan. The International Crisis Group, a Brussels-based think-tank, claimed in early January that the conflict between government and rebel forces had claimed nearly 10,000 lives. This was ten times the number cited just a few weeks earlier by Hilde Johnson, the UN special representative for South Sudan. Such a huge discrepancy could suggest that at least one of these figures is based more on guesswork than sound statistical methodology; or that new information suddenly became available.

Any quantification of mass refugee flows in Africa will have “some degree of uncertainty”, concluded Root.

The importance of good data in humanitarian work cannot be overestimated. “Reliable and accurate data are central to any effective policy-related decision-making on forced displacement,” explained the UNHCR’s yearbook. “In such situations, poor policy decisions, based on inaccurate or unreliable data, can have catastrophic humanitarian consequences.”

Working to prevent these disasters are teams of fieldworkers, statisticians and analysts, who have developed rigorous procedures to produce sound data. Is the data perfect? Certainly not. But it is the best available. Humanitarian work would be rudderless without it. DM

This article was originally published in Africa in Fact, a monthly magazine published by Good Governance Africa (GGA). GGA is a research and advocacy organisation that works to improve government performance on the continent.

Photo: The World Food Program has for the first time begun to transport food to Sudanese refugees in Chad by crossing the Sahara desert on 29 August 2004. EPA/SABRI ELMHEDWI


Support DAILY MAVERICK & get FREE UBER vouchers every month

An increasingly rare commodity, quality independent journalism costs money, though not nearly as much as its absence can cost global community. No country can live and prosper without truth - that's why it matters.

Every Daily Maverick article and every Scorpio exposé is proof of our dedication to this unshakeable mission. Investing in our news media is by far the most effective investment into South Africa's future.

You can support Independent and Investigative journalism by joining Maverick Insider. If you contribute R150 or more per month you will receive R100 back in UBER vouchers. EVERY MONTH until October 2019.

So, if you'd like to help and do something meaningful for yourself and your country, then sign up to become a Maverick Insider. Together we can Defend Truth.

Days of Zondo

Meet the Watsons: They will make the Gupta’s look like amateurs

By Ferial Haffajee

Elon Musk has specifically written into their spacesuit designers' contract that they "must look badass".