Fanning the flames, unnecessarily

This post is a convolution of David's post earlier today and Sol's from a few days ago.  Our debate with Buhaug et al that Sol blogged about has dragged on for a while now, and engendered a range of press coverage, the most recent by a news reporter at Science.  "A debate among scientists over climate change and conflict has turned ugly", the article begins, and goes on to catalog the complaints on either side while doing little to engage with the content.

Perhaps it should be no surprise that press outlets prefer to highlight the mudslinging, but this sort of coverage is not really helpful.  And what was lost in this particular coverage were the many things I think we've actually learned in the protracted debate with Buhaug.

We've been having an ongoing email dialog with ur-blogger and statistician Andrew Gelman, who often takes it upon himself to clarify or adjudicate these sorts of public statistical debates, which is a real public service.  Gelman writes in an email:

In short, one might say that you and Buhang are disagreeing on who has the burden of proof.  From your perspective, you did a reasonable analysis which holds up under reasonable perturbations and you feel it should stand, unless a critic can show that any proposed alternative data inclusion or data analytic choices make a real difference.  From their perspective, you did an analysis with a lot of questionable choices and it’s not worth taking your analysis seriously until all these specifics are resolved.

I'm sympathetic with this summary, and am actually quite sympathetic to Buhaug and colleagues' concern about our variable selection in our original Science article.  Researchers have a lot of choice over how and where to focus their analysis, which is a particular issue in our meta-analysis since there are multiple climate variables to choose from and multiple ways to operationalize them.  Therefore it could be that our original effort to bring each researcher's "preferred" specification into our meta-analysis might have doubly amplified any publication bias -- with researchers of the individual studies we reviewed emphasizing the few significant results, and Sol, Ted, and I picking the most significant one out of those.  Or perhaps the other researchers are not to blame and the problem could have just been with Sol, Ted, and my choices about what to focus on.


So I actually thank Halvard and coauthors for pushing us to look more carefully at this.  We have now done so, with results provided in our reply to Buhaug et al's comment (Figures 1E and F) and additional expanded results in the NBER working paper we released last week.  What we do is pull all available results from the papers and data we had on hand, which means both temperature and precipitation if they were both reported, and both contemporaneous and lagged effects if they were reported.  And for all datasets where authors did not report lagged effects but where we had the data, we re-ran additional models with lags included so we could report those estimates too.

The results are best seen in the figure Sol posted on Monday, which is Figure 6 in the new working paper and is reproduced below.  Contemporaneous temperature effects are still large and highly significant, as we found in the original paper.  The lagged effects of temperature, which again NO study emphasized as important as Sol discussed on Monday, are also large and positive but not quite significant (for group conflict).  The combination of the contemporaneous and lagged effects, which we view as the most relevant measure (as it can account for both persistence and displacement), is large and highly significant.

Summary of meta-analysis for studies with distributed lag structure. Estimated precision-weighted mean effects and 95% confidence intervals for intergroup (left panel) and interpersonal conflict (right panel), for both contemporaneous and one period lagged temperature (red, left offset) and precipitation (blue, right offset). "Combined" effects equal the sum of the contemporaneous (0 lag) and one period lagged effects for studies where the calculation was possible.  The number of studies contributing to each estimate is given in parentheses. From Burke, Hsiang, Miguel (2014).

The results for precipitation are somewhat more mixed:  significant combined effects, not quite significant on the contemporaneous effects, but in any case point estimates that are about 1/3rd the size of the temperature effects.  This is useful to know:  results are less consistent for precipitation, and are substantially smaller in magnitude.

So what is the reasonable conclusion from this exercise?  Even when we've swept out all researcher degrees of freedom in terms of choice of climate variable, we continue to find consistent and strong results for the effect of temperature on conflict.  We took on the burden of proof here, whether or not it was ours to take on, and showed that the main result from our original paper -- that there is a strong and surprisingly consistent relationship between temperature and conflict  -- is robust.  We are actually now more confident in our results than before, and so again thank Halvard and colleagues for pushing us to do this.

Halvard et al's second main concern is about the broader meta-analytic exercise, and whether it even makes sense to estimate an "average" response of temperature to conflict across different settings and types of conflict.  Again, I think this is a quite sensible concern and one we are sympathetic to -- it's clearly plausible that riots in India and civil wars in Africa could respond differently to changes in temperature.  As explained in Gelman et al's really nice textbook, however, it turns out you can get some quantitative traction on this question.  In particular, you can let the data tell you whether the variation in estimated effects that you see across settings is consistent with sampling variation alone (and thus that there is likely to be a common underlying response), or whether there is substantial variation in effects that likely cannot be explained by sampling variation (indicating that the "true" effect of temperature on conflict likely differs across settings).

We go through this exercise starting on page 14 of the new working paper, with results reported in Table 2.  In short, the data suggest that some of the variation in effects we observe cannot be explained by sampling variation alone, again implying that the "true" effect of temperature on conflict likely differs across settings.  Again, this is useful, and not something we would have known before gathering the data and actually doing the analysis.  [We implemented a similar exercise in the Science paper; replication code is here].  In this way I think the science has actually benefitted from the back-and-forth with Buhaug et al.

So what's the overall conclusion, then?  Mine would be:  (1) Hotter-than-average temperatures are associated on average with more conflict, and (2) the magnitude of this effect likely differs across settings, perhaps substantially.  Both of these are "new" stylized facts that we hope both Halvard's group and ours can now try to understand.  And perhaps other reporters can help us highlight what we've actually learned in this debate, and what we still need to learn, rather than further fanning the flames.

Go Giants!