Do journal impact factors distort science?

From my inbox:

An ad hoc coalition of unlikely insurgents -- scientists, journal editors and publishers, scholarly societies, and research funders across many scientific disciplines -- today posted an international declaration calling on the world scientific community to eliminate the role of the journal impact factor (JIF) in evaluating research for funding, hiring, promotion, or institutional effectiveness.

Here's the rest of the story at Science Daily.

And a link to DORA, the "ad hoc coalition" in question.

It seems fairly obvious that impact factors do distort science.  But I wonder how much, and I also wonder if there are realistic alternatives that would do a better job of encouraging good science.

There are delicate tradeoffs here: some literatures seem to become mired within their own dark corners, forming small circles of scholars that speak a common language.  They review each others' work, sometimes because no one else can understand it, or sometimes because no one else cares to understand it.  The circle has high regard for itself, but the work is pointless to those residing outside of it.

At the same time, people obviously have very different ideas about what constitutes good science.

So, what does the right model for evaluating science look like?

Consensus Statements on Sea-Level Rise

In my mailbox from the AGU:

After four days of scientific presentations about the state of knowledge on sea-level rise, the participants reached agreement on a number of important key statements. These statements are the reflection of the participants of the conference and not official positions from the sponsoring societies.

 

Earth scientists agree that the global sea level is rising at an accelerated rate overall in response to climate change.

Scientists have a professional responsibility to inform government, the public, and the private sector about the impacts of rising sea levels and extreme events, and the risks they pose.

 

The geological record indicates that the current rates of sea-level rise in many regions are unprecedented relative to rates of the last several thousand years.

Global sea-level rise has changed rapidly in the past and scientific projections show it will continue to rise over the course of this century, altering our coasts.

 

Extreme events and their associated impacts will be more damaging and pose higher risks in the immediate future than sea-level rise.

Increasing human activity, such as land use change and water management practices, adds stress to already fragile ecosystems and can affect coasts just as much as sea-level rise.

 

Sea-level rise will exacerbate the impacts of extreme events, such as hurricanes and storms, over the long-term.

Extreme events have contributed to loss of life, billions of dollars in damage to infrastructure, massive taxpayer funding for recovery, and degradation of our ecosystems.

 

In order to secure a sustainable future, society must learn to anticipate, live with, and adapt to the dynamics of a rapidly evolving coastal system.

Over time feasible choices may change as rising sea level limits certain options. Weighing the best decisions will require the sharing of scientific information, the coordination of policies and actions, and adaptive management approaches.

 

Well-informed policy decisions are imperative and should be based upon the best available science, recognizing the need for involvement of key stakeholders and relevant experts.

As we work to adapt to accelerating sea level rise, deep reductions in emissions remain one of the best ways to limit the magnitude and pace of rising seas and cut the costs of adaptation.

 

Spatial Econometric Peeves (wonkish)

Nearly all observational data show strong spatial patterns.  Location matters, partly due to geophysical attributes, partly because of history, and partly because all the things that follow from these two key factors tend to feedback and exaggerate spatial patterns.  If you're a data monkey you probably like to look at cool maps that illustrate spatial patterns, and spend a lot of time trying to make sense of them.  I know I do.

Most observational empirical studies in economics and other disciplines need to account for this general spatial connectedness of things.  In principal, you can do this two ways:  (1) develop a model of the spatial relationship; (2) account for the spatial connectedness by appropriately adjusting the standard errors of your regression model.

The first option is a truly heroic one, and most all attempts I've seen seem foolhardy.  Spatial geographic patters are extremely complex and follow from deep geophysical and social histories (read Guns, Germs and Steal). One is unlikely to uncover the full mechanism that underlies the spatial pattern.  When one "models" this spatial pattern, assumptions drive the result, and the assumptions are, almost always, a heroic leap of faith.

That leaves (2), which shouldn't be all that difficult using modern statistical techniques, but does take some care and perhaps a little experimentation.  It seems to me many are a little too blithe about it, and perhaps select methods that falsely exaggerate statistical significance.

Essentially, the problem is that there's normally a lot less information in a large data set than you think, because most observations from a region and/or time are correlated with other observations from that region and/or time.  In statistical speak, the errors are clustered.

To illustrate how much this matters, I'll share some preliminary regressions from a current project of mine.  Here I am predicting the natural log of corn yield using field-level data that span about 15 years on most of the corn fields in three major corn-producing U.S. states. I've got several hundred thousand observations. Yes, you read that right--it's a very rich data set.

But corn yields, as you can probably guess, tend to have a lot of spatial correlation. This happens in large part because weather, soils, and farming practices are spatially correlated.  However, there isn't a lot of serial correlation in weather from year to year, so, my data are highly correlated within years, and average outcomes have strong geographic correlation, but errors are mostly independent between years in a fixed location.

Where the amount of information in the data normally scales with the square root of the sample size, when the data are clustered spatially or otherwise, a conservative estimate for the amount of information is the square root of the number of clusters you have.  In this data set, we don't really have fixed clusters.  It's more like smooth overlapping clusters.  But we might proxy the "number" of clusters  around the square root of 45, the number of years X states I have, because most spatial correlation in weather fades out after about 500 miles.  Although these states border each other, so it may be even less than 45.  Now, I do have weather matched to each field depending on the field's individual planting date, which can vary a fair amount.  That adds some statistical power. So, I hope it's a bit better than the square root of 45.  Either way, in the ballpark of 45 is a whole lot less than several hundred thousand.

I regress the natural log of corn yield on

YEAR:             a time trend
log (potential):  (output of a crop model calibrated from daily weather inputs),
gdd:                  growing degree days (a temperature measure),
DD29:              degree days above 29C (a preferred measure of extreme heat),
prec & prec^2:  season precipitation and precipitation squared,
PDay:                number of days since Jan 1 until planting
interaction between DD29 and CO2 exposure.

CO2 exposure varies a little bit spatially, and also temporally, both due to a trend from burning fossil fuels and other emissions, as well as seasonal fluctuations following from tree and leaf growth (earlier planting tends to have higher CO2, and higher CO2 can improve radiation water use efficiency in corn, which can effectively make the plants more drought tolerant).

The standard regression output gives:

Coefficients:
                 Estimate Std. Error t value Pr(>|t|)    
(Intercept)     2.320e+00  3.014e-02   76.98   <2e-16 ***
I(YEAR - 2000)  1.291e-02  4.600e-04   28.06   <2e-16 ***
log(Potential)  5.697e-01  5.470e-03  104.14   <2e-16 ***
gdd             1.931e-04  4.177e-06   46.24   <2e-16 ***
DD29           -2.477e-02  1.149e-03  -21.56   <2e-16 ***
Prec            1.787e-02  9.424e-04   18.96   <2e-16 ***
I(Prec^2)      -4.939e-04  2.038e-05  -24.24   <2e-16 ***
PDay           -6.798e-03  6.269e-05 -108.45   <2e-16 ***
DD29:AvgCO2     6.229e-05  2.953e-06   21.09   <2e-16 ***

Notice the huge t-statistics: all the parameters look precisely identified.  But you should be skeptical.

Most people now use White "robust" standard errors, which uses a variance-covariance matrix constructed from the residuals to account for arbitrary heteroscedasticity.  Here's what that gives you:


                    Estimate   Std. Error   t value      Pr(&gt;|t|)
(Intercept)     2.319894e+00 3.954834e-02  58.65970  0.000000e+00
I(YEAR - 2000)  1.290703e-02 5.362464e-04  24.06922 5.252870e-128
log(Potential)  5.696738e-01 7.161458e-03  79.54718  0.000000e+00
gdd             1.931294e-04 5.058033e-06  38.18271  0.000000e+00
DD29           -2.477002e-02 1.397239e-03 -17.72783  2.557376e-70
Prec            1.786707e-02 1.099087e-03  16.25627  2.016306e-59
I(Prec^2)      -4.938967e-04 2.327153e-05 -21.22321 5.830391e-100
PDay           -6.798270e-03 7.381894e-05 -92.09386  0.000000e+00
DD29:AvgCO2     6.229397e-05 3.616307e-06  17.22585  1.698989e-66

The standard errors are larger and the T-values smaller, but this standard approach still gives us extraordinary confidence in our estimates.

You should remain skeptical.  Here's what happens when I use robust standard errors clustered by year:


                Estimate Std. Error t value Pr(&gt;|t|)    
(Intercept)     2.32e+00  5.57e-01  4.17 3.094e-05 ***
YEAR            1.29e-02  8.57e-03  1.52   0.12920    
log(Potential)  5.70e-01  9.11e-02  6.25 4.000e-10 ***
gdd             1.93e-04  7.89e-05  2.45   0.01443 *  
DD29           -2.48e-02  1.35e-02 -1.83   0.06719 .  
Prec            1.79e-02  1.06e-02  1.68   0.09243 .  
I(Prec^2)      -4.94e-04  2.15e-04 -2.29   0.02178 *  
PDay           -6.80e-03  8.17e-04 -8.32   2.2e-16 ***
DD29:AvgCO2     6.23e-05  3.50e-05  1.78   0.07510 .  


Standard errors are an order of magnitude larger and T-values are more humbling.  Planting date and potential yield come in very strong, but now everything else is just borderline significant.  It seems robust standard errors really aren't so robust.

But even if we cluster by year, we are probably missing some important dependence, since geographic regions may have similar errors across years, and in clustering by year, I assume all errors in one year are independent of all errors in other years.

If I cluster by state, the standard robust/clustering procedure will account for both geographic and time-series dependence within a state.  Since I know from earlier work that one state is about the extent of spatial correlation, this seems reasonable.  Here's what I get:



                  Estimate  Std. Error  t value  Pr(>|t|)    
(Intercept)     2.32e+00  1.1888e+00   1.9514 0.0510065 .  
YEAR            1.29e-02  4.6411e-03   2.7810 0.0054194 ** 
log(Potential)  5.70e-01  1.6938e-01   3.3632 0.0007706 ***
gdd             1.93e-04  2.2126e-04   0.8729 0.3827338    
DD29           -2.48e-02  2.6696e-02  -0.9279 0.3534781    
Prec            1.79e-02  1.2786e-02   1.3974 0.1622882    
I(Prec^2)      -4.94e-04  2.7371e-04  -1.8045 0.0711586 .  
PDay           -6.80e-03  4.9912e-04 -13.6205 < 2.2e-16 ***
DD29:AvgCO2     6.23e-05  6.8565e-05   0.9085 0.3635962    

Oops.  Now most of the weather variables have lost their statistical significance too.  But since I'm explicitly limiting assumed dependence in the cross section within years, now the time trend (YEAR) is significant, and it wasn't when clustering by YEAR.  We probably shouldn't take that significance very seriously, since some kinds of dependence (like technology) probably spans well beyond one state.

Note that this strategy of using large clusters combined with robust SE treatment (canned in STATA, for example) is what's recommended in Angrist and Pischke's Mostly Harmless Econometrics.

There are other ways of dealing with these kinds of problems.  For example, you can use a "block bootstrap" that resamples residuals whole years as a time, which preserves spatial correlation.  This is great in agricultural applications since weather is pretty much IID across years in a fixed locations and we should feel reasonably comfortable that there is little serial correlation.  One can also adapt the method by Conley for panel data.  Soloman Hsiang has graciously provided code here.  In earlier agriculture-related work, Wolfram Schlenker and I generally found that clustering by state gives similar standard errors as these methods.

The overarching lesson is this: try it different ways and err on the side of least significance, because it's very easy to underestimate your standard errors and very hard to overestimate them.

And watch out for data errors: these have a way of screwing up both estimates and standard errors, sometimes quite dramatically.

If you had to patience to follow all of this, you might appreciate the footnotes and appendix in our recent comment on Deschenes and Greenstone.

Is peace needed in the climate/conflict debate?

Yes, argues Andy Solow in a comment in Nature today.  He bemoans the "fierce battle that has broken out within the research community" over whether there is a link between climate and conflict, highlighting the protracted argument we've been having with Halvard Buhaug and colleagues over whether there is a link between temperature and conflict in Africa (see here, here, and here).  Solow's main point seems to be that these sorts of "reduced form" analyses of climate and conflict reveal little about the true underlying processes that drive conflict, and thus that people running these models ("quants") need to engage with folks that are studying individual conflicts in depth ("quals") to make the analyses meaningful.

Overall his comments are quite reasonable, and if I can speak for my co-authors I think we are sympathetic to many of his arguments.  I have a few main points of disagreement, however.  First, I would not characterize the disagreements between Halvard (and colleagues) and us as a disagreement between "quants" and "quals".  We would certainly put ourselves in the former category and I'm guessing Halvard would too.  The disagreements between us have really been about how to do the quantitative analysis correctly, something over which we continue to disagree and will continue to disagree as long as folks are estimating panel models without fixed effects, or estimating models that include outcome variables as covariates.  But contrary to what Solow suggests, we definitely have been engaging with Halvard and his colleagues.  For instance, at Halvard's generous invitation I participated in a couple sessions on climate and conflict at last month's annual meeting of the International Studies Association, sessions organized by Halvard and colleagues. 

More importantly, though, I think Solow misperceives what estimating reduced form models is all about in this setting, and does not allow that the very reason a "quants" might want to engage with a "quals" is because a reduced form effect has been established.  A typical quant approach would be to start with an initial hunch that conflict and (say) temperature might be related, put together some data, and estimate a regression of conflict on temperature.  If there is a relationship in this "reduced form", then this would motivate more careful study of the underlying cases and mechanism - i.e. an engagement with the quals.  On the other hand, if there is no relationship in the reduced form, there is arguably no reason to take a "deeper look behind the statistics", as Solow puts it.  This is because the reduced form effect is going to encompass all possible ways in which climate will affect conflict (even if it doesn't illuminate them), and a null result in the reduced form would tell you that there is no mechanism linking the two phenomena.  

(Some initial hint at the underlying mechanism can certainly motivate how the reduced form is estimated, and this is something people are already attentive to.  For instance, suppose you think that shortfalls in income are what link climate variables to conflict events; in a setting where income is earned in agriculture, then focusing on climate during growing season months might make sense.  This approach for example is pursued in a nice paper by Harari and La Ferrara, in which they show that drought during the growing season matters a lot more for conflict than drought in the non-growing season.)  

The argument with Halvard and colleagues is about whether there is a reduced form effect.  Because my colleagues and I think that the reduced form relationship between temperature and conflict in Africa has been pretty well established by different research groups working at multiple scales (e.g. here, here, here, here, and here), we are in full agreement with Solow that a closer look at mechanism is now warranted, and that we need to learn a lot from quals about what might be going on.  But if you do not believe this reduced form, then it doesn't make sense to engage with quals on how climate might affect conflict.  

But a reduced form relationship is also much more than just a reason to talk to qualitative folks.  For the sake of argument, let's imagine that no manner of further engagement with quals is ever able to fully isolate the mechanism linking climate to conflict - perhaps there are just too many factors that are both affected by climate and that are potential contributors to conflict.  Does this mean that we learn nothing from the reduced form, or that it should be ignored by policy-makers?  

Probably not, for two reasons.  First, the reduced form effect of climate on conflict, when properly estimated, is estimating the causal effect of climatic fluctuations (typically inter-annual) on conflict outcomes.  This is because year-to-year variation in climate is pretty random and unlikely correlated with other unobserved variables that also affect conflict.  We know climate has "caused" conflict, we just don't know why.  

Second, even if we don't yet understand the mechanism linking the two variables, the fact that warmer temperatures and conflict are linked would appear useful to policymakers interested in whether we should take action to mitigate future climate change.  Clearly, as Solow points out, past responses to year-to-year variation and future response to long-run changes in means are not the same thing.  But given that adaptation to long-run temperature changes appears very slow at best, that sensitivity to extreme heat does not appear to be diminishing, and that past relationships are all we really have to go on, then assuming these relationships might carry into the future does not seem like a crazy place to start.  Moreover, important decisions are often made solely on "reduced form" evidence.  For example, I don't need to know how snake venom kills me to figure that, if I find myself in a room full of poisonous snakes, it might make sense to either invest in snake-proof boots or see if I can find a door.  The "adaptation" strategy here - buying boots - does involve knowing a little something about the mechanism: i.e. snake --> bite --> die;  the "mitigation" strategy - leaving the room - does not.  As a less prosaic example, aspirin was discovered before 1900 and was prescribed for decades before the mechanism underlying its efficacy was understood in the 1960s. Similarly, quinine has been used as a malaria drug since the 1600s although we still do not understand why it works.  So failing to understand a mechanism does not typically keep us from taking meaningful action when we've decided a reduced form effect is real.

Nevertheless, I hope (with Solow) that we can stop arguing about a few of the primary reduced forms in question -- e.g. the link between temperature and conflict in Africa -- and do more to understand what's going on and what to do about it.  Doing so will help inform adaptation investments in particular, and Solow is right that detailed case-study knowledge from the "quals" will be critical in building this understanding.  But moving in this direction involves establishing that there is a reduced form relationship between temperature and conflict, and it is for this reason we are arguing with Halvard and co.  

Irrigation in South Africa

Somewhat buried in a recent USDA update on the corn situation in South Africa are some interesting statistics on center pivot irrigation in South Africa. It’s a region where climate change is projected to have very negative effects on rainfed corn, so the trend towards irrigation could represent an important adaptation response, especially if it continues:

Irrigated corn area in South Africa has more than doubled during the past 12 years. It has increased from 100,000 hectares in 2000/01 to 240,500 hectares in 2011/12, as shown in Figure 5. Irrigated corn area in South Africa is also dependent on corn price, with the 2005/06 irrigated corn area being reduced by 73,000 hectares from the previous year when corn prices were low in 2005/06. In contrast, corn prices this year have been high and USDA/FAS forecast the 2012/13 irrigated corn area to be greater than last year’s record irrigated corn area of 240,500 hectares. By assuming average irrigated corn yields at 10-tons/hectare on 250,000 hectares of irrigated corn, USDA/FAS-Washington estimates the 2012/13 irrigated corn output to be nearly 2.5 million tons, or about 20 percent of South Africa’s total corn output of 12.5 million tons.

Mitigation with magic

This week marks a milestone of sorts. The carbon dioxide (CO2) levels measured at Mauna Loa in Hawaii have passed 400ppm for the firsttime. For all of the concerns and talk about global warming over the years, the bottom line is there have been very few successes yet at the scale required to slow down global emissions rates. If anything they are going faster than expected.

There have been tons of studies on potential mitigation strategies, and a clear message that multiple changes are needed to make a real dent in CO2 growth rates. For example, see here for a nice recent update by Steve Davis and colleagues on the concept of mitigation "wedges", where a wedge is enough to avoid 1 Gt of C emissions per year. The short story is that lots (about 9) of wedges are needed just to stabilize emissions as energy consumption grows, and another 10 are needed to reduce emissions enough to stabilize climate (in the figure below, "hidden" wedges refer to progress that is expected based on business as usual improvements in the emission intensity of economic growth).

One thing that has always puzzled me about mitigation discussion is how agriculture is treated. It's usually included, but mainly in the role of sequestering carbon in the soil, or reducing nitrous oxide and methane emissions. Not that these aren't worthwhile, but I think this emphasis tends to miss a more fundamental role -- demand for agricultural land drives much of deforestation, which remains a substantial (~10%) driver of global emissions.

As people search for investments to slow climate change, it seems useful to think about how to reduce demand for agricultural land. This roughly translates to a question of how to keep agricultural prices low enough that the incentives to clear new land don't outweigh the costs. On that, the evidence is pretty clear that investments in research and development for agriculture is a good bet. In an important report in 2001, Phil Pardey called agricultural R&D “slow magic” because it was both powerful but relatively slow to materialize. More recently, Keith Fuglie has provided some further evidence on the benefits of capacity to generate and deploy new technologies – countries with greater capacity consistently have higher growth in total factor productivity (TFP).

So investments in R&D Help, but do they really help enough to matter in terms of lowering land use change and slowing emissions? First, in the context of emissions, it's important that the technologies and practices that emerge from R&D are improving yield, and not just overall productivity of inputs. For example, reducing inputs to achieve the same yield is great for many reasons and will be measured as a rise in TFP, but will actually tend to increase land clearing because it lowers production costs without changing yield. To folks like Fuglie, who are mainly concerned about TFP and associated changes in prices, yields are a secondary issue. But in the context of land use and emission rates, it is yield and not overall TFP that is most critical.

Second, it’s possible that yield improvements lead to counteracting "rebound" effects that reduce their effectiveness in lowering emissions. For example, if yield increases are localized in land rich regions, they will tend to increase profits, raising the incentives to clear. Or if demand is elastic with respect to prices, so that small price reductions lead to higher demand, then total land savings are likely to be small.

Capturing all of these requires a reasonable model of agricultural markets and land use. In recent work with Uris Baldos and Tom Hertel of Purdue, we made an attempt to quantify how much mitigation benefit you get from investment in agriculture. In this case, we were focused specifically on investments related to climate change adaptation. To make a long story short, we estimated that broad-based investments in yield improvements reduce emissions at a cost of roughly $15 per metric ton of CO2. There's lots of assumptions embedded in that number, and we try our best to look at a full range of possible conditions at each link in the chain, coming up with a 95% confidence interval of $11-$22 per ton (see full paper here). Even the high end is pretty competitive with other mitigation actions. So it seems like “slow magic” may be a good mitigation strategy after all.