Update on Togo’s Novissi emergency fund
Background: Togo and the COVID-19 crisis
In Togo, a small country in West Africa, over 50% of the population lives in poverty. The COVID-19 pandemic threatens to reverse years of poverty reductions, as lockdown measures designed to stop the spread of the virus limit economic activity and threaten food security. More broadly, the World Bank estimates that the pandemic will force between 119 and 124 million people in low and middle-income countries into extreme poverty in 2020.
In response to this crisis, the Government of Togo launched “Novissi,” an innovative social assistance program designed to provide emergency cash assistance to Togo’s neediest families. Over the course of just a few weeks, the government built and deployed a completely contactless, digital system that provided over half a million individuals with cash. Beneficiaries registered using their mobile phones; after entering basic information into a USSD menu, they were then immediately sent mobile money transfers of approximately $20/month, lasting for three months. This program was the first of its kind, described as an “exemplary case of social protection in response to the coronavirus pandemic in Africa.”
The targeting challenge
A key question at the heart of Novissi — and at the heart of the hundreds of other targeted social protection programs that have been launched in response to COVID-19 — is how to prioritize those people with the greatest need?² The difficulty was that the Togolese Government did not have a comprehensive social registry that would allow them to directly identify and prioritize its poorest people. The last census was conducted in 2011, and even that database did not have any information that could allow the government to determine who should be prioritized for assistance. And in the middle of a pandemic, it was impractical to collect the data required to create a new registry.
When Novissi first launched in April 2020, the government chose to prioritize informal workers living in the regions most impacted by the lockdown restrictions. To determine eligibility, they relied on a voter database that had been recently updated, and in which people had self-declared their home location and their occupation.
Our work: Guiding the expansion of Novissi
The Government of Togo, in partnership with GiveDirectly, is currently expanding Novissi to rural areas of the country, where extreme poverty is most severe. A top priority in this expansion is ensuring that benefits are targeted toward the poorest citizens. Our team of researchers — including Emily Aiken (UC Berkeley), Suzanne Bellue (U. Mannheim), Joshua Blumenstock (UC Berkeley / CEGA / IPA), Dean Karlan (Northwestern / IPA), and Chris Udry (Northwestern / IPA) — are supporting this effort. The novel approach we have developed has two main stages.
Stage 1: Prioritizing the poorest villages and neighborhoods
The Government of Togo was interested in prioritizing support to the poorest cantons (admin-3 regions) in Togo. However, prior to our collaboration, the Government of Togo did not possess a poverty map that would allow for such granular targeting. The government had previously conducted nationally representative household surveys, but those surveys could only produce estimates of poverty representative at the national (admin-0) or regional level (admin-1). To enable more granular geographic targeting for the program, we used the deep learning pipeline described in Chi et al. (2020) to identify the poorest 100 cantons in the country (the number 100 was chosen by our government partners, based on the estimated distribution of wealth of the population living in those cantons).
Specifically, we started with a large, nationally-representative household survey that had reliable estimates of the living conditions of thousands of households. We used those survey-based estimates of poverty as “ground truth” to train a machine learning algorithm to estimate the wealth of very small regions (i.e., 2.4km tiles) based on the geographic characteristics of that region — where we use rich data from satellites and other sources to measure those geographic characteristics. Intuitively, the algorithm learns that certain patterns in the imagery are indicative of wealth (such as places with metal roofs and high quality roads) while others are indicative of poverty (such as places with certain types of terrain and weather). We then overlaid tile-level consumption estimates with high resolution population density estimates from Tiecke et al. (2017) to estimate average per capita household consumption for each of Togo’s 397 cantons. The tile-level and canton-level estimates are shown in Figure 1 below. These maps were used to determine the 100 poorest cantons.
Figure 1: Prioritizing the poorest villages and neighborhoods
We produce micro-estimates of the wealth of each 2.4km grid cell by applying deep learning algorithms to high-resolution satellite imagery (left), combine those estimates with information on the population density of each grid cell (middle), and use this information to determine the 100 poorest cantons in Togo (right).
Stage 2: Prioritizing the poorest individuals in the poorest villages
In expanding Novissi to rural areas, one of the government’s goals was to ensure that benefits flowed to the poorest individuals living in the 100 poorest cantons. Specifically, in collaboration with GiveDirectly, the government secured sufficient funding to provide benefits to roughly 57,000 of the approximately 580,000 citizens living in the poorest 100 cantons. Building on recent advances in the research literature, our team developed methods that use mobile phone metadata to help identify the individuals with the greatest need (see Blumenstock et al. (2015) and Aiken et al. (2020) for details).
Specifically, with financial support from the World Bank, we conducted a large phone survey in September 2020, immediately prior to the expansion of Novissi in rural areas, to provide “ground truth” information on the living conditions of roughly 10,000 individuals. We went to great lengths to ensure that the individuals in this phone survey were representative of the broader population of people living in the 100 poorest cantons, through a combination of adaptive survey design and the use of survey sample weights. This helps ensure that difficult-to-reach populations, such as the extreme poor and those living in remote villages, are represented in the training data. Each survey respondent provided informed consent to participate in this research program.
For each of the 10,000 survey respondents, we then matched information they provided on their daily consumption to their mobile phone metadata, which was obtained from the mobile phone operators in Togo — see below for a discussion of how we protect data privacy. The metadata includes information about the date, time, duration, and cell tower used for calls and texts, as well as information about each subscriber’s volume of mobile data usage and mobile money transactions. We processed these raw data with algorithms that construct aggregate statistics of each subscriber’s pattern of mobile phone use, including information that is correlated with wealth, such as the total volume of international phone calls made, or the average mobile money balance. On an anonymized version of this dataset, we then trained machine learning algorithms to predict consumption from mobile phone metadata. These algorithms were then used to generate a consumption estimate for each of the 5.7 million mobile subscribers in the country.
Prioritising the poorest individuals and subscribers
Using ground truth wealth and poverty collected through a large phone survey of active mobile phone subscribers, we train machine learning algorithms to estimate the wealth of each mobile subscriber (left). In the 100 poorest cantons (red distribution in right figure), those estimated to consume less than $1.25/day are prioritized for Novissi (dashed vertical line). These individuals are substantially poorer than the average resident of Togo (blue distribution).
Preliminary findings
After developing this approach, we carefully and critically evaluated it using several independent datasets. We focused in particular on comparing this new targeting approach to alternative approaches that could have been used by the Government of Togo at the time. In particular, for the expansion of Novissi to rural areas, the Togolese government was considering two alternatives in addition to the phone-based approach: an expansion of the existing occupation-based targeting; and a “geographic blanketing” approach that would provide benefits to all individuals residing in specific villages.
We used data from the September 2020 phone survey to compare these options, and evaluate the effectiveness of each of the different methods in reaching the poor. Our preliminary results indicate that, assuming the goal is to reach the poorest 57,000 people in the 100 poorest cantons, the satellite+phone approach is significantly more accurate than the alternative approaches available to the government. In particular, as shown in the figure below, the satellite+phone approach is expected to provide benefits to nearly 2.5X as many of the poorest citizens as a program that provided benefits based on occupation.
Comparison of the phone and satellite approach to feasible alternatives
Ongoing and Future work
Through a collaboration with the Government of Togo and GiveDirectly, roughly 57,000 new beneficiaries are being enrolled in Novissi using the satellite+phone method described above. We are currently conducting a careful evaluation of this program expansion. In particular, we have planned in-person surveys for Spring 2021 — or as soon as public health restrictions allow — to talk to program beneficiaries, community members, and village leaders. These interviews will make it possible to critically assess the program and compare the new targeting methodology to common alternatives used in the design of anti-poverty programs, such as a proxy means test (PMT) and community-based targeting (CBT). Although the Togolese government was not in a position to use a PMT or CBT for the Novissi program, they represent important benchmarks for comparison.
Key Concerns and Limitations
We are optimistic that this new approach to targeting can provide an effective means to distribute cash to extremely poor people, extremely quickly. At the same time, there are important practical, social, and ethical considerations involved in the real-world implementation of such a program — these considerations were a major focus of our time and energy as we worked with the Government of Togo and GiveDirectly to design a humanitarian aid program based on this technology. We highlight a few key concerns, as well as our approach to dealing with those concerns.
Program exclusions: Of central importance to any targeting mechanism, including the one we develop, are exclusion errors (people who need benefits but don’t receive them) and inclusion errors (people who don’t need benefits but do receive them). There is a large academic literature that documents the targeting errors that are likely to occur with existing targeting methods (cf. Banerjee et al. 2009, Alatas et al. 2012, Alatas et al. 2016). Thus, a major focus of the in-person survey we have planned for early 2021 is to rigorously document the targeting errors that occurred in this initial roll-out — to make sure such errors are transparent and can be addressed in the future.
For this program, one important concern is exclusions that occur for people who don’t have mobile phones. While this type of exclusion is not a product of our new method — it exists by virtue of the fact that the only way the government could quickly distribute cash en masse during the pandemic was by using mobile money — it is a concern that we are tracking very carefully through surveys, audits, and engagement with community leaders (see below for more on this). Based on our analysis of recent nationally-representative household survey data, we estimate that roughly 90% of households in Togo have at least one mobile phone, which might limit the scope of such exclusions. In addition, the current expansion was designed to allow anyone with a SIM card to register for Novissi. In other words, someone without a phone could acquire a SIM card (which is much cheaper than a phone), and use that SIM card in someone else’s phone to register for the program and receive benefits. In future expansions, we are exploring the possibility of making low-cost mobile phones available through pay-as-you-go subsidies.
We have also taken several measures to advertise the program to the public and encourage enrolment. For instance, we have briefed village leaders about the program, sent hundreds of thousands of SMS messages to the intended population of beneficiaries, distributed posters to hundreds of communities, and placed advertisements on local radio stations. Our hope is that these efforts to promote program awareness can help reduce the scope for exclusion.
Data privacy: In designing this program, user privacy was a top priority. We took several steps to minimize the use of private data to the greatest extent possible, and had U.C. Berkeley’s Committee for the Protection of Human Subjects review all of the research procedures involved in this work. Among other protections, we built in organizational safeguards that minimized the data to which each party had access: for instance, neither GiveDirectly nor the Government of Togo ever has access to any data collected by the mobile phone operators. Likewise, neither the government nor GiveDirectly received access to the poverty scores derived from the mobile data. Instead, the research team produced a list of beneficiaries based on the poverty scores, and those beneficiaries were made eligible to receive transfers. The only data the government received was this list of eligible beneficiaries. We also implemented strict anonymization, encryption, and access protocols, for instance to ensure that all data are pseudonymized to remove personally identifying information prior to analysis. Third, we collected consent whenever possible: for instance, phone survey respondents provided informed consent before participating, and program beneficiaries are asked for consent to use their data for program administration during the USSD registration process.
Fairness and bias: We recognize the fact that all algorithms are likely to be biased. As noted earlier, we were careful to design our survey and sampling strategy to ensure that the data used to train the machine learning algorithm were as representative as possible of the broader population of mobile phone subscribers in the 100 poorest cantons. We also used several independent sources of survey data to calibrate and double-check the output of these algorithms, and to conduct algorithmic audits to examine if specific vulnerable subgroups were more likely to be excluded on the basis of the algorithm. We also have planned extensive survey activities for early 2021 that are designed to determine if the algorithm’s predictions were inadvertently biased against any particular type of beneficiary (such as women, illiterate, and marginalized subgroups); these surveys will also help determine if this method is better at reaching some populations that are often excluded through alternative approaches to targeting the poor (e.g., community-based targeting can often neglect socially excluded individuals). Still, we are aware that not all types of bias can be anticipated, and thus are actively engaging with community members through qualitative interviews and other feedback mechanisms to try and surface any unintended biases or exclusions created by this program.
Community outreach and support: Prior to the launch of the Novissi in rural areas, and after extensive consultation with our partners in Togo, GiveDirectly and members of the research team visited several communities to conduct qualitative focus groups and interviews with likely beneficiaries. We also worked with the government to contact more than 100 village and community leaders, to let them know about the program, and to create a relationship to receive feedback on program design and implementation.
As noted, we also took several measures to communicate the program to the public, through segments that were aired on local radio, as well as broadcast SMS messages and public posters. We established a call center that provides real-time support and troubleshooting, and beneficiaries can also directly submit feedback via the phone-based USSD platform. We also have several real-time auditing mechanisms: our own team is constantly monitoring program data to ensure there are no issues with registration or payments; and Ernst & Young were hired as external auditors; and GiveDirectly continually conducts qualitative follow-up calls with program participants and community members to monitor program integrity.
Summary
In partnership with the Government of Togo and GiveDirectly, we have helped build an innovative approach to humanitarian crisis response, designed to help as many people as possible, as quickly as possible. By leveraging new sources of digital data and recent advances in machine learning, we demonstrate how to target humanitarian cash transfer relief programs to people in need, even in settings where no universal social registry exists.
