Targeting poverty with satellites

[This post is co-authored with Matt Davis, co-author and RA extraordinaire...]

About six months ago, our Stanford SustainLab crew had a paper in Science showing that you can make pretty good predictions about local-level economic wellbeing in Africa by combining satellite imagery with fancy tools from machine learning.  To us this was (and is) a promising finding, as it suggests a way to address the fundamental lack of data on economic outcomes in much of the developing world.   As has been widely acknowledged, these data gaps inhibit our ability to both evaluate what interventions reduce poverty and to target assistance to those who need it most.

A natural question that comes up (e.g. here) is: are these satellite-based estimates good enough to actually be useful for either evaluation or targeting?  Our original paper didn't really answer that question.  We're in the process of putting together a follow-up paper that looks at this question on an expanded country set and with an improved machine learning pipeline, but in the meantime we [by which I mean "we", meaning Matt and Neal] wanted to use some of the data from our original paper to more quantitatively explore this question.

Folks have been thinking for decades about how whether using geographic information to inform the targeting of anti-poverty programs could improve their efficiency.  The standard thought experiment goes like this.  Imagine you're a policymaker who has a fixed budget F that she can distribute as cash transfers to anyone in the country (this sort of cash transfer program happens all the time these days, it turns out).  Lets say in particular that the poverty metric that this policymakers cares about is the squared poverty gap (SPG), a common poverty measure that takes into account the distance of individuals from the poverty line.  [If you're having trouble sleeping at night:  For a given poverty line P, an individual with income Y<P has a poverty gap of P-Y and an SPG of (P-Y)^2.  The SPG in a region is the average over all individuals in that region, where anyone with Y>=P has a SPG==0.  So this measure gives a lot of weight to people far below the poverty line].  If you're goal is to reduce the SPG, you do best by giving money to the poorest person you can find until they're equal to the next poorest, giving them both enough money until they're equal to the third poorest, and so-on.

So how should the policymaker distribute the cash?  If she knows nothing about where poor people are, a naive approach would be to distribute money uniformly -- i.e. to just give each of n constituents F/n dollars.  Clearly this could be pretty inefficient, since people already above the poverty line will get money and this won't reduce the SPG.

An alternate approach, now a few decades old in the economics literature, has been to construct "small area estimates" (SAE) of poverty by combining a detailed household survey with a less detailed but more geographically-comprehensive census.  The idea is that while only the household survey measures your outcome of interest (typically consumption expenditure), there are a small set of questions common to both the detailed household survey and the census (call these X).  These are typically questions about respondent age, gender, education, and perhaps a few basic questions on assets.  So using the household survey you can fit a model Y = f(X), which tells you how the Xs map into consumption expenditure (your outcome of interest), and then using the same Xs in the census and your model f(X) to predict consumption expenditure for everyone in the census.  Then you can aggregate these to any level of interest (e.g. village or district), and use them to potentially inform your cash transfers.  This has been explored in a number of papers, e.g. here and here, and apparently has been used to inform policy in a number of settings.

Our purpose here is to compare a targeting approach that uses our satellite-based estimates to either the naive (uniform) transfer or a transfer that's informed by SAE estimates.   To actually evaluate these approaches against each other, we are going to just use the household survey data, aggregated to the cluster (village) level.  In particular, we estimate both the SAE and the satellite-based model on a subset of our household survey data in each country, make predictions for the remainder of the data that the models have not seen, and in this holdout sample, evaluate for a fixed budget the reduction in the SPG you'd get if you allocated using either the naive, SAE, or satellite-based model.  The allocation rule for SAE and satellites is the one described above:   giving money to the poorest village until equal to the next poorest, giving them both enough money until they're equal to the third poorest, and so-on.

Below is what we get, with the figure showing how much reduction in SPG you get from each targeting scheme under increasing program budgets.  The table summarizes the results, showing the cross-validated R2 for the SAE and satellite-features models (the goodness-of-fits from the models that we then use to make predictions that are then used in targeting), and the amount of money each approach saves relative to the uniform transfer to achieve a 50% decline in SPG.

	R-squared		% Reduction in budget to achieve 50% decline in SPG
Country	SAE	Features	SAE	Features
Malawi	0.45	0.44	6.8	8
Nigeria	0.38	0.4	30.7	25.9
Tanzania	0.62	0.54	36	18.3
Uganda	0.64	0.44	39.1	20.7

What do we learn from these simulations?  First, geographical targeting appears to save you money relative to the naive transfer.  You achieve a 50% reduction in SPG for 7-40% less budget than a naive transfer when you use either SAE or satellites to target.  Second, and not surprisingly, when the satellite model and the SAE model fit the data roughly equally well (e.g. Malawi, Nigeria), they deliver similar savings relative to a uniform transfer.  But the amount of budget that you save by using SAE or satellites to target transfers can differ even for similar R2.  Compare Malawi to Nigeria:  the targeted approaches help a lot more in Nigeria than in Malawi, which is consistent with Malawi having poor people all over the place (e.g. see the maps we produced for Malawi and Nigeria)  including in somewhat better-off vilages, which in turn makes targeting on the village mean not as helpful.  Third, SAE leads to more efficient targeting in the two countries where the SAE model has more predictive power -- Tanzania and Uganda.

We're somewhat biased of course, but this to us is fairly promising from a satellite perspective.   First, these SAE estimates are probably and upper bound on actual SAE performance, since it's very rarely the case that you have a household survey and a census in the same year, and we've been generous in the variables we included to calculate the SAE (some of which are not be available in many censuses).  Second, since many countries lack either a census or a household survey, it's not clear whether we can use SAE in these countries, whereas in our Science paper we showed decent out-of-country fits for the satellite-based approach.  Third, we're working on improvements to the satellite-based estimates and anticipate meaningfully higher R2 relative to these benchmarks.  And finally, and perhaps most importantly, the satellite-based approach is going to be incredibly cheap to implement relative to SAE in areas where surveys don't already exist.  So you might be willing to trade off some loss in targeting performance given the low expense of developing the targeting tool. 

So our tentative conclusion is that satellites might have something to offer here.  They're probably going to be even more useful when combined with other approaches and data -- something that we are exploring in ongoing work.