We've got a paper out in Science today that demonstrates a new way to use satellite imagery to predict economic well-being in poor countries (see project website here). The paper is a collaboration between some of us social scientists (or social "scientists", with emphatic air quotes, as my wife puts it) and some computer scientists across campus -- folks who have apparently figured out how to use computers for more than email and Youtube surfing.
We're hoping that this is the first of many projects with these guys, and so have codified our collaboration here, with one of those currently-popular dark-hued website designs where you scroll around a lot.
So why is it sensible to try to use satellite imagery to predict economic livelihoods? The main motivation is the lack of good economic data in many parts of the developing world. As best we can tell, between the years 2000 and 2010, one quarter of African countries did not conduct a survey from which nationally-representative poverty estimates could be constructed, and another 40% conducted only one survey. So this means that in two-thirds of countries on the world's poorest continent, you've got very little sense of what's going on, poverty-wise. And even a lot of the surveys that do get conducted are only partially in the public domain, meaning you've got to employ some trickery to even get the shape of the income distribution in these countries (and survey locations are still unavailable!).
This lack of data makes it hard to track specific targets that we've set, such as the #1 Sustainable Development Goal of eliminating poverty by 2030. It also makes it hard to evaluate whether specific interventions aimed at reducing poverty are actually working. The result is that we currently have little rigorous evidence about the vast majority of anti-poverty interventions undertaken in the developing world, and no real way to track progress towards SDGs or any other target.
While we don't collect a lot of survey data for many locations in the developing world, we collect other sources of information about these places constantly -- satellite information being one obvious source. So our goal in this paper was to see whether we could use recent hi-res recent imagery to predict economic outcomes at a local level, and fill in the gaps between what we know from surveys.
We are certainly not the first people to think of using satellites or other "unconventional" data sources to study economic output in the developing world. For instance, here is a 2012 paper by Adam Storeygard that uses nightlights to improve GDP estimates at the country level, and here is a paper from about 9 months ago by Josh Blumenstock and company where they use call data records from a cell phone company to predict local-level economic outcomes in Rwanda. But what our approach brings to the table is that (unlike Storeygard et al) we can make very local predictions, and that (perhaps unlike Blumenstock et al) our approach is very easy to scale, given that satellite imagery are available free or at very low cost for every corner of the earth and more rolls in each day.
For a quick explanation of what we do in the paper, check out this short video that we made in collaboration with these guys. Sort of an experiment on our end, comments or slander welcome in the comments section.
The main innovation in the paper is in figuring out what information in the hi-res daytime imagery might be useful for predicting poverty or well-being. Standard computer vision approaches to interpreting imagery typically get fed huge training datasets - e.g. millions of "labeled" images (e.g. "dog" vs "cat") that a given model can use to learn to distinguish the two objects in an image. But the whole problem here is that we have very little training data -- i.e. few places where we can tell a computer with certainty that a specific location is rich or poor.
So take a two-step approach to solving this problem. First, we use lower-resolution nightlights images to train a deep learning model to identify features in the higher-resolution daytime imagery are predictive of economic activity. The idea here -- building on the paper cited above -- is that nightlights are a good but imperfect measure of economic activity, and they are available for everywhere on earth. So the nightlights help the model figure out what features in the daytime imagery are predictive of economic activity. Without being told what to look for, the model is able to identify a number of features in the daytime imagery that look like things we recognize and tend to think are important in economic activity (e.g roads, urban areas, farmland, and waterways -- see Fig 2 in our paper).
Then in the last step of the process, we use these features in the daytime imagery to predict village-level wealth, as measured in a few household surveys that were publicly available and geo-referenced. (As our survey measures we use data from the World Bank LSMS for consumption expenditure and from the DHS for assets.) We call this two step approach "transfer learning", in the sense that we've transferred knowledge learned in the nightlights-prediction task to the final task of predicting village poverty. Nightlights are not used in the final poverty prediction; they are only used in the first step to help us figure out what to use in the daytime imagery.
Josh Blumenstock (or some Science art editor) have a really nice depiction of the procedure, in a commentary that Josh wrote on our piece that also appeared today in Science.
The model does surprisingly well. Below are cross-validated model predictions and R-squareds for consumption and assets, where we are comparing model predictions against survey measurements at the village level in five African countries (Uganda, Tanzania, Malawi, Nigeria, Rwanda). The cross-validation part is key here -- basically we split the data in two, train the model on one part of the data, and then predict for the other part of the data that the model hasn't seen. This guards against overfitting.
We can then use these predictions to make poverty maps of these countries. Here is a prototype (something we're still working on), with estimates aggregated to the district level:
[Edit: Tim Varga pointed out in an email that, while beautiful, the below plot is basically meaningless to the 10% of men and 1% of women who are red/green colorblind. Duh - and sorry! (Only silver lining is that this mistake harmed men differentially, subverting the normal gender bias). Nevertheless, we will fix..]
Maybe the most exciting result is that a model trained in one country appears to do pretty well when applied outside that country, at least within our set of 5 countries. For example, a model trained in Uganda does a pretty good job of predicting outcomes in Tanzania, without ever having seen Tanzanian data. Granted, this would likely work a lot worse if we were trying to make predictions for a more dissimilar country (say, Iceland). But it suggests that at least within Africa -- the continent where data gaps remain largest -- our approach could have wide application.
Finally, we don't really view our approach as a substitute for continuing to do households surveys, but rather as a strong compliment -- as a way to dramatically amplify the power of what we learn from these surveys. It's likely that we're going to continue to learn a lot from household surveys that we might never learn from satellite imagery, even with the fanciest machine learning tricks.
We are currently trying to extend this work in multiple directions, including evaluating whether we can make predictions over time using lower-res Landsat data, and in scaling up the existing approach to all of Africa. More results coming soon, hopefully. We also want to work with folks who can use these data, so if that happens to be you, please get in touch!
We're hoping that this is the first of many projects with these guys, and so have codified our collaboration here, with one of those currently-popular dark-hued website designs where you scroll around a lot.
This lack of data makes it hard to track specific targets that we've set, such as the #1 Sustainable Development Goal of eliminating poverty by 2030. It also makes it hard to evaluate whether specific interventions aimed at reducing poverty are actually working. The result is that we currently have little rigorous evidence about the vast majority of anti-poverty interventions undertaken in the developing world, and no real way to track progress towards SDGs or any other target.
While we don't collect a lot of survey data for many locations in the developing world, we collect other sources of information about these places constantly -- satellite information being one obvious source. So our goal in this paper was to see whether we could use recent hi-res recent imagery to predict economic outcomes at a local level, and fill in the gaps between what we know from surveys.
We are certainly not the first people to think of using satellites or other "unconventional" data sources to study economic output in the developing world. For instance, here is a 2012 paper by Adam Storeygard that uses nightlights to improve GDP estimates at the country level, and here is a paper from about 9 months ago by Josh Blumenstock and company where they use call data records from a cell phone company to predict local-level economic outcomes in Rwanda. But what our approach brings to the table is that (unlike Storeygard et al) we can make very local predictions, and that (perhaps unlike Blumenstock et al) our approach is very easy to scale, given that satellite imagery are available free or at very low cost for every corner of the earth and more rolls in each day.
For a quick explanation of what we do in the paper, check out this short video that we made in collaboration with these guys. Sort of an experiment on our end, comments or slander welcome in the comments section.
The main innovation in the paper is in figuring out what information in the hi-res daytime imagery might be useful for predicting poverty or well-being. Standard computer vision approaches to interpreting imagery typically get fed huge training datasets - e.g. millions of "labeled" images (e.g. "dog" vs "cat") that a given model can use to learn to distinguish the two objects in an image. But the whole problem here is that we have very little training data -- i.e. few places where we can tell a computer with certainty that a specific location is rich or poor.
So take a two-step approach to solving this problem. First, we use lower-resolution nightlights images to train a deep learning model to identify features in the higher-resolution daytime imagery are predictive of economic activity. The idea here -- building on the paper cited above -- is that nightlights are a good but imperfect measure of economic activity, and they are available for everywhere on earth. So the nightlights help the model figure out what features in the daytime imagery are predictive of economic activity. Without being told what to look for, the model is able to identify a number of features in the daytime imagery that look like things we recognize and tend to think are important in economic activity (e.g roads, urban areas, farmland, and waterways -- see Fig 2 in our paper).
Then in the last step of the process, we use these features in the daytime imagery to predict village-level wealth, as measured in a few household surveys that were publicly available and geo-referenced. (As our survey measures we use data from the World Bank LSMS for consumption expenditure and from the DHS for assets.) We call this two step approach "transfer learning", in the sense that we've transferred knowledge learned in the nightlights-prediction task to the final task of predicting village poverty. Nightlights are not used in the final poverty prediction; they are only used in the first step to help us figure out what to use in the daytime imagery.
Josh Blumenstock (or some Science art editor) have a really nice depiction of the procedure, in a commentary that Josh wrote on our piece that also appeared today in Science.
The model does surprisingly well. Below are cross-validated model predictions and R-squareds for consumption and assets, where we are comparing model predictions against survey measurements at the village level in five African countries (Uganda, Tanzania, Malawi, Nigeria, Rwanda). The cross-validation part is key here -- basically we split the data in two, train the model on one part of the data, and then predict for the other part of the data that the model hasn't seen. This guards against overfitting.
[Edit: Tim Varga pointed out in an email that, while beautiful, the below plot is basically meaningless to the 10% of men and 1% of women who are red/green colorblind. Duh - and sorry! (Only silver lining is that this mistake harmed men differentially, subverting the normal gender bias). Nevertheless, we will fix..]
Maybe the most exciting result is that a model trained in one country appears to do pretty well when applied outside that country, at least within our set of 5 countries. For example, a model trained in Uganda does a pretty good job of predicting outcomes in Tanzania, without ever having seen Tanzanian data. Granted, this would likely work a lot worse if we were trying to make predictions for a more dissimilar country (say, Iceland). But it suggests that at least within Africa -- the continent where data gaps remain largest -- our approach could have wide application.
Finally, we don't really view our approach as a substitute for continuing to do households surveys, but rather as a strong compliment -- as a way to dramatically amplify the power of what we learn from these surveys. It's likely that we're going to continue to learn a lot from household surveys that we might never learn from satellite imagery, even with the fanciest machine learning tricks.
We are currently trying to extend this work in multiple directions, including evaluating whether we can make predictions over time using lower-res Landsat data, and in scaling up the existing approach to all of Africa. More results coming soon, hopefully. We also want to work with folks who can use these data, so if that happens to be you, please get in touch!
Neat stuff Marshall. Have you tried predicting changes over time in consumption or assets?
ReplyDeleteHey Mike - that's what we're trying to do now, using Landsat imagery. hard part is getting good ground data in the same spots over time..
ReplyDeleteWould ICRISAT data work? India, not Africa, but still pretty relevant. I gather you know about it, but wonder what difficulties it poses. http://vdsa.icrisat.ac.in
DeleteThats a good idea actually. We've probably been a little too African focused...
DeleteOh, this is absolutely brilliant. Great stuff, Marshall!
ReplyDelete-gautam