Seminar talk on Spatial First Differences

Hannah Druckenmiller and I wrote a paper on a new method we developed that allows you to do cross-sectional regressions while eliminating a lot (or all) omitted variables bias. We are really excited about it and it seems to be performing well in a bunch of cases. 

Since people seem to like watching TV more than reading math, here's a recent talk I gave explaining the method to the All New Zealand Economics Series. (Since we don't want to discriminate against the math-reading-folks-who-don't-want-to-read-the-whole-paper, we'll also post a blog post explaining the basics...)

If you want to try it out, code is here.

Daily or monthly weather data?

We’ve had a few really hot days here in California. It won’t surprise readers of this blog to know the heat has made Marshall unusually violent and Sol unusually unproductive. They practice what they preach. Apart from that, it’s gotten me thinking back to a common issue in our line of work - getting “good” measures of heat exposure. It’s become quite popular to be as precise as possible in doing this – using daily or even hourly measures of temperature to construct things like ‘extreme degree days’ or ‘killing degree days’ (I don’t really like the latter term, but that’s beside the point for now).

I’m all for precision when it is possible, but the reality is that in many parts of the world we still don’t have good daily measures of temperature, at least not for many locations. But in many cases there are more reliable measures of monthly than daily temperatures. For example, the CRU has gridded time series of monthly average max and min temperature at 0.5 degree resolution.

It seems a common view is that you can’t expect to do too well with these “coarse” temporal aggregates. But I’m going to go out on a limb and say that sometimes you can. Or at least I think the difference has been overblown, probably because many of the comparisons between monthly and daily weather show the latter working much better. But I think it’s overlooked that most comparisons of regressions using monthly and daily measures of heat have not been a fair fight.

What do I mean? On the one hand, you typically have the daily or hourly measures of heat, such as extreme degree days (EDD) or temperature exposure in individual bins of temperature. Then they enter into some fancy pants model that fits a spline or some other flexible function that capture all sorts of nonlinearities and asymmetries. Then on the other hand, for comparison you have a model with a quadratic response to growing season average temperature. I’m not trying to belittle the fancy approaches (I bin just as much as the next guy), but we should at least give the monthly data a fighting chance. We often restrict it to growing season rather than monthly averages, often using average daily temperatures rather than average maximums and minimums, and, most importantly, we often impose symmetry by using a quadratic. Maybe this is just out of habit, or maybe it’s the soft bigotry of low expectations for those poor monthly data.

As an example, suppose, as we’ve discussed in various other posts, that the best predictor of corn yields in the U.S. is exposure to very high temperatures during July. In particular, suppose that degree days above 30°C (EDD) is the best. Below I show the correlation of this daily measure for a site in Iowa with various growing season and monthly averages. You can see that average season temperature isn’t so good, but July average is a bit better, and July average daily maximum even better. In other words, if a month has a lot of really hot days, then that month's average daily maximum is likely to be pretty high.

You can also see that the relationship isn’t exactly linear. So a model with yields vs. any of these monthly or growing season averages likely wouldn’t do as well as EDD if the monthly data entered in as a linear or quadratic response. But as I described in an old post that I’m pretty sure no one has ever read, one can instead define simple assymetric hinge functions based on monthly temperature and rainfall. In the case of U.S. corn, I suggested these three based on a model fit to simulated data:

This is now what I’d consider more of a fair fight between daily and monthly data. The table below is from what I posted before. It compares the out-of-sample skill of a model using two daily-based measures (GDD and EDD), to a model using the three monthly-based hinge functions above. Both models include county fixed effects and quadratic time trends. In this particular case, the monthly model (3) even works slightly better than the daily model (2). I suspect the fact it’s even better relates less to temperature terms than to the fact that model (2) uses a quadratic in growing season rainfall, which is probably less appropriate than the more assymetric hinge function – which says yields respond up to 450mm of rain and are flat afterwards.

Model	Calibration R2	Average root mean square error for calibration	Average root mean square error for out-of-sample data  (for 500 runs)	% reduction in out-of-sample error
1	0.59	0.270	.285	--
2	0.66	0.241	.259	8.9
3*	0.68	0.235	.254	10.7

Overall, the point is that monthly data may not be so much worse than daily for many applications. I’m sure we can find some examples where it is, but in many important examples it won’t be. I think this is good news given how often we can’t get good daily data. Of course, there’s a chance the heat is making me crazy and I’m wrong about all this. Hopefully at least I've provoked the others to post some counter-examples. There's nothing like a good old fashioned conflict on a hot day.

Massetti et al. - Part 3 of 3: Comparison of Degree Day Measures

Yesterday's blog entry outlined the differences between Massetti et al. derivation of degree days and our own.  To quickly recap: Our measure show much less variation within a county over the years, i.e., the standard deviation of fluctuations around the mean outcome in a county are about a third of theirs. One possibility is that our measure over-smoothes the year-to-year fluctuations, or alternatively, that Massetti et al.'s fluctuations might include measurement error, which would result in attenuation bias (paper).

Below are tests comparing various degree day measures in a panel of log corn and soybean yields. It seems preferable to test the predictive power in a panel setting as one does not have to worry about omitted variable bias (As mentioned before, Massetti et al. did not share their data with us and we hence can't match the same controls in a cross-sectional regression of farmland values). We use the optimal degree days bounds from earlier literature.

The following two tables regress log corn and soybean yields, respectively, for all counties east of the 100 degree meridian (except Florida) in 1979-2011 on four weather variables, state-specific restricted cubic splines with 3 knots, and county fixed effects. Column definitions are the same as in yesterday's post: Columns (1a)-(3b) use the NARR data to derive degree dats, while column (4b) uses our 2008 procedure. Columns (a) use the approach of Massetti et al. and derive the climate in a county as the inverse-distance weighted average of the four NARR grids surrounding a county centroid.  Columns (b) calculate degree days for each 2.5x2.5mile PRISM grid within a county (squared inverse-distance weighted average of all NARR grids over the US) and derives the county aggregate as the weighted average of all grids where the weight is proportional to the cropland area in a county. 

Columns (0a)-(0b) are added as baseline using a quadratic in growing season average temperature. Columns (1a)-(1b) follow Massetti et al. and first derive average daily temperatures and degree days using daily averages, i.e., degree days are only positive if the daily average exceeds the threshold. Columns (2a)-(2b) calculate degree days for each 3-hour reading. Degree days will be positive if part of the temperature distribution is above the threshold, but not the daily average.  Columns (3a)-(3b) approximate the temperature distribution within a day by linearly interpolating between the 3-hour measures.  Column (4b) uses a sinusoidal approximation between the daily minimum and maximum to approximate the temperature distribution within a day.

Explaining log corn yields 1979-2011.

Explaining log soybean yields 1979-2011.

The R-square is lowest for regressions using a quadratic in average temperature (0.37 for corn and 0.33 for soybeans).  It is slightly higher when we use degree days based on the NARR data set in columns (1a)-(3b), ranging from 0.39-0.41 for corn and 0.35-0.36 for soybeans.  It is much higher when our degree days measure is used in columns (4b): 0.51 for corn and 0.48 for soybeans.

The second row in the footer lists the percent reduction in root mean squared error (RMSE) compared to a model with no weather controls (just county fixed effects and state-specific time trends). Weather variables that add nothing would have 0%, while weather measures that explain all remaining variation would reduce the RMSE by 100%.  Column (4b) reduces the RMSE by twice as much as measures derived from NARR. Massetti et al.'s claim that they introduce "accurate measures of degree days" seems very odd given that their measure performs half as well as previously published measures that we shared with them.

The NARR data set likely includes more measurement error than our previous data set. Papers making comparisons between degree days and average temperature should use the best available degree days construction in order not to bias the test against the degree days model.

Correction (January 30th): An earlier version had a mistake in the code by calculating the RMSE both in and out-of-sample. The corrected version only calculates the RMSE out-of-sample.  While the reduction in RMSE increased for all columns, the relative comparison between models is not impacted.

Massetti et al. - Part 2 of 3: Calculation of Degree Days

Following up on yesterday's post, let's look at the differences in how to calculate degree days. Recall that degree days just count the number of degrees above a threshold and sum them over the growing season.  Massetti et al. argue in their abstract that "The paper shows that [...] hypotheses of the degree day literature fail when accurate measures of degree days are used." This claim is attributed to the fact that Massetti et al. supposedly use better data and hence get more accurate readings of degree days, however, no empirical evidence is provided. They use data from the North American Regional Reanalysis (NARR) that provides temperatures at 3-hour intervals. The authors proceed to first calculate average temperatures for each day from the eight readings per day, and then calculate degree days as the difference of the average temperature to the threshold.

Before we compare their method to calculating degree days to ours, a few words on the NARR data. Reanalysis data combine observational data with differential equations from physical models to interpolate data. For example, they utilize mass and energy balance, i.e., a certain amount of moisture can only fall once at precipitation.  If precipitation comes down in one grid, it can't also come down in a neighboring grid.  On the plus side, the physical models construct an entire series of data (solar radiation, dew point, fluxes, etc) that normal weather stations do not measure.  On the downside, the imposed differential equations that relate all weather measures imply that interpolated data do not always match actual observations.

So how do the degree days in Massetti et al. compare to ours? Here's a little detour on degree days - this is a bit technical and dry, so please be patient.  The first statistical study my coauthors and I published using degree days in 2006 used monthly temperature data since we did not have daily temperature data at the time.  Since degree days depend how many times a temperature threshold is passed, monthly averages can be a challenge as a temporal average will hide how many times a threshold is passed.  The literature has gotten around this problem by estimating an empirical link between the standard deviation in daily and monthly temperatures, called Thom's formula. We used this formula was used to derive fluctuations in average daily temperatures to derive degree days.

The interpolation of the temperature distribution when only knowing monthly averages is certainly not ideal, and we hence went through great length to better approximate the temperature distribution. All of my subsequent work with various coauthors hence not only looked at the distribution of daily average temperatures within a month, but went one step further by looking at the temperature distribution within a day.  The rational is that even if average daily temperatures do not cross a threshold, the daily maximum might.  We interpolated daily maximum and minimum temperature, and fit a sinusoidal curve between the two to approximate the distribution within a day (See Snyder). This is again an interpolation and might have its own pitfalls, but one can empirically test whether it improves predictive power, which we did and will do for part 3 of this series.

Here is my beef with Massetti et al: Our subsequent work in 2008 showed that calculating degree days using the within-day distribution of temperatures is much better.  We even emphasize that in a panel setting average temperatures perform better than degree days derived using Thom's formula (but not in the cross-section as the Thom's approximation works much better at getting average number of degree days correct than year-to-year fluctuations around the mean). What I find disingenuous in the Massetti et al. is that it makes a general statement about comparing degree days to average temperature, yet only discusses the inferior approach for calculating degree days using Thom's formula.  What makes things worse is that we shared our "better" degree days data that uses the within day distribution with them (which they acknowledge).

Unfortunately, Massetti et al. decided not to share their data with us, so the analysis below uses our construction of their variables.  We downloaded surface temperature from NARR.  The reanalysis data provides temperature readings at several altitude levels above ground, and in general, the higher the reading above the ground, the lower temperatures, which will result in lower degree day numbers.

The following table constructs degree days for counties east of the 100 degree meridian in various ways.  Columns (1a)-(3b) use the NARR data, while column (4b) uses our 2008 procedure. Columns (a) use the approach of Massetti et al. and derive the climate in a county as the inverse-distance weighted average of the four NARR grids surrounding a county centroid.  Columns (b) calculate degree days for each 2.5x2.5mile PRISM grid within a county (squared inverse-distance weighted average of all NARR grids over the US) and derives the county aggregate as the weighted average of all grids where the weight is proportional to the cropland area in a county. Results don't differ much between (a) and (b).

Columns (1a)-(1b) follow Massetti et al. and first derive average daily temperatures and degree days using daily averages, i.e., degree days are only positive if the daily average exceeds the threshold. Columns (2a)-(2b) calculate degree days for each 3-hour reading. Degree days will be positive if part of the temperature distribution is above the threshold, but not the daily average.  Columns (3a)-(3b) approximate the temperature distribution within a day by linearly interpolating between the 3-hour measures.  Column (4b) uses a sinusoidal approximation between the daily minimum and maximum to approximate the temperature distribution within a day.

Average temperature and average season-total degree days 8-32C in 1979-2011 are fairly consistent between all columns.  We give the mean outcome in a county as well as two standard derivations: the between standard deviation (in round brackets) is the standard deviation in the average outcome between counties, while the within standard deviation [in square brackets] is the average standard deviation of the year-to-year fluctuations around a county mean. The between standard deviation is fairly consistent across columns, but the within-county standard deviation is much lower for our interpolation in column (4b).

As a result of the lower within-county variation, fluctuations are lower and hence the threshold is passed less often in column (4b).  Extreme heat as measured by degree days above 29C or 34C are hence lower when the within-day distribution is use din column (4b) compared to columns (2a)-(3b). There are two possible interpretation: either our data is over-smoothing and hence under-predicting the variance, or NARR has measurement error which will lead to attenuation bias.  We will test both possible theories in part 3 tomorrow.

Massetti et al. - Part 1 of 3: Convergence in the Effect of Warming on US Agriuclture

Emanuele Massetti has posted a new paper (joined with Robert Mendelsohn and Shun Chonabayashi) that takes another look at the best climate predictor of farmland prices in the United States.  He'll present it at the ASSA meetings in Philadelphia - I have seen him present the paper at the 2013 NBER spring EEE meeting and at the 2013 AERE conference, and wanted to provide a few discussion points for people interested in the material.

A short background: several articles of contributors to this blog have found that temperature extremes are crucial at predicting agricultural output. To name a few: Maximilian Auffhammer and coauthors have shown that rice have opposite sensitivities to minimum and maximum temperature, and this relationship can differ over the growing season (paper). David Lobell and coauthors found that there is a highly nonlinear relationship between corn yields and temperature using data from field trials in Africa (paper), which is comparable to what Michael Roberts and I have found in the United States (paper).  The same relationship was observed by Marshal Burke and Kyle Emerick when looking at yield trends and climate trends over the last three decades (paper).

Massetti et al. argue that average temperature are a better predictor of farmland values than nonlinear transformations like degree days.  They exclusively rely on cross-sectional regressions (in contrast to the aforementioned panel regressions), re-examining earlier work Michael Hanemann, Tony Fisher and I have done where we found that degree days are better and more robust predictors of farmland values than average temperature (paper).

Before looking into the differences between the studies, it might be worthwhile to emphasize an important convergence in the sign and magnitude of predicted effect of a rise in temperature on US agriculture.  There has been an active debate whether a warmer climate would be beneficial or detrimental. My coauthors and I have usually been on the more pessimistic side, i.e., arguing that warming would be harmful. For example, a +2C and +4C increase, respectively, predicted a 10.5% and 31.6 percent decrease in farmland values in the cross-section of farmland values (short-term B1 and long-term B2 scenarios in Table 5)  and a 14.9 and 35.3 percent decrease in corn yields in the panel regression (Appendix Table A5).

Robert Mendelsohn and various coauthors have consistently found the opposite, and the effects have gotten progressively more positive over time.  For example, their initial innovative AER paper that pioneered the cross-sectional approach in 1994 argued that "[...] our projections suggest that global warming may be slightly beneficial to American agriculture." Their 1999 book added climate variation as an additional control and argued that "Including climate variation suggests that small amount of warming are beneficial," even in the cropland model.  A follow-up paper in 2003 further controls for irrigation and finds that "The beneficial effect of warmer temperatures increases slightly when water availability is included in the model."

There latest paper finds results that are consistent with our earlier findings, i.e., a +2C warming predicts decreases in farmland values of 20-27 percent (bottom of Table 1), while a +4C warming decreases farmland values by 39-49 percent. These numbers are even more negative than our earlier findings and rather unaffected whether average temperatures or degree days are used in the model.  While the authors go on to argue that average temperatures are better than degree days (more on this in future posts), it does change the predicted negative effect of warming: it is harmful.