Seminar talk on Spatial First Differences

Hannah Druckenmiller and I wrote a paper on a new method we developed that allows you to do cross-sectional regressions while eliminating a lot (or all) omitted variables bias. We are really excited about it and it seems to be performing well in a bunch of cases. 

Since people seem to like watching TV more than reading math, here's a recent talk I gave explaining the method to the All New Zealand Economics Series. (Since we don't want to discriminate against the math-reading-folks-who-don't-want-to-read-the-whole-paper, we'll also post a blog post explaining the basics...)

If you want to try it out, code is here.

Conflict in a changing climate: Adaptation, Projection, and Adaptive Projections (Guest post by Tamma Carleton)

Thanks in large part to the authors of G-FEED, our knowledge of the link between climatic variables and rates of crime and conflict is extensive (e.g. see here and here). The wide-ranging effects of high temperatures on both interpersonal crimes and large-scale intergroup conflict have been carefully documented, and we’ve seen that precipitation shortfalls or surpluses can upend social stability in locations heavily dependent on agriculture. Variation in the El Niño cycle accounts for a significant share of global civil conflict, and typhoons lead to higher risks of property crime.

Much of the research in this area is motivated by the threat of anthropogenic climate change and the desire to incorporate empirically derived social damages of climate into policy tools, like integrated assessment models. Papers either explicitly make projections of conflict under various warming scenarios (e.g. Burke et al., 2009), or simply refer to future climate change as an impetus for study. While it’s valuable from a risk management perspective to understand and predict the effects that short-run climate variation may have on crime and conflict outcomes today, magnitudes of predicted climate changes and the lack of clear progress in climate policy generally place future projections at the heart of this body of work.

However, projections are also where much of the criticism of climate-conflict research lies. For example, see Frequently Heard Criticisms #1 and #6 in this post by Marshall. These criticisms are often reasonable, and important (unlike some other critiques you can read about in that same post). The degree to which impacts identified off of short-run climate variation can effectively predict future effects of long-run gradual climate change involves a lot of inherent (statistical and climatological) uncertainty and depends critically on likely rates of adaptation. Because the literature on adaptation is minimal and relies mostly on cross-sectional comparisons, we are limited in our ability to integrate findings into climate policy, which is often the motive for conducting research in the first place. 

Recently, Sol, Marshall and I published (yet another) review of the climate and conflict literature, this time with an emphasis on adaptation and projection, to try to address the concerns discussed above. I’m going to skip the content you all know from their detailed and impressive previous reviews, and talk about the new points we chose to focus on.

A decade of advances in climate econometrics

I have a new working paper out reviewing various methods, models, and assumptions used in the econometrics literature to quantify the impact of climatic conditions on society.  My colleagues here on G-FEED have all made huge contributions, in terms of methodological innovations, and it was really interesting and satisfying to try and tie them all together.  The process of writing this was much more challenging than I expected, but rereading it makes me feel like we as a community really learned a lot during the last decade of research. Here's the abstract:

Climate Econometrics (forthcoming in the Annual Reviews) 

Abstract: Identifying the effect of climate on societies is central to understanding historical economic development, designing modern policies that react to climatic events, and managing future global climate change. Here, I review, synthesize, and interpret recent advances in methods used to measure effects of climate on social and economic outcomes. Because weather variation plays a large role in recent progress, I formalize the relationship between climate and weather from an econometric perspective and discuss their use as identifying variation, highlighting tradeoffs between key assumptions in different research designs and deriving conditions when weather variation exactly identifies the effects of climate. I then describe advances in recent years, such as parameterization of climate variables from a social perspective, nonlinear models with spatial and temporal displacement, characterizing uncertainty, measurement of adaptation, cross-study comparison, and use of empirical estimates to project the impact of future climate change. I conclude by discussing remaining methodological challenges.

And here are some highlights:


1. There is a fairly precise tradeoff in assumptions that are implied when using cross-sectional vs. panel data methods. When using cross-sectional techniques, one is making strong assumptions about the comparability of different units (the unit homogeneity assumption). When using panel based within-estimators to understand the effects of climate, this unit homogeneity assumption can be substantially relaxed but we require a new assumption (that I term the marginal treatment comparability assumption) in order to say anything substantive about the climate. This is the assumption that small changes in the distribution of weather events can be used to approximate the effects of small changes in the distribution of expected events (i.e. the climate).  In the paper, I discuss this tradeoff in formal terms and talk through techniques for considering when each assumption might be violated, as well as certain cases where one of the assumptions can be weakened.


2. The marginal treatment comparability assumption can be directly tested by filtering data at different frequencies and re-estimating panel models (this is intuition between the long-differences approach that David and Marshall have worked on). Basically, we can think of all panel data as superpositions of high and low-frequency data and we can see if relationships between climate variables and outcomes over time hold at these different frequencies.  If they do, this suggests that slow (e.g. climatic) and fast (e.g. weather) variations have similar marginal effects on outcomes, implying that marginal treatments are comparable.  Here's a figure where I demonstrate this idea by applying to US maize (since we already know what's going on here).  On the left are time series from an example county, Grand Traverse, when the data is filtered at various frequencies. On the right is the effect of temperature on maize using these different data sets.  The basic relationship recovered with annual data holds at all frequencies, changing substantially only in the true cross-section, which is the equivalent of frequency = 0 (this could be because that something happens in terms of adaptation at time scales longer than 33 yrs, or that the unit-homogeneity assumption fails for the cross-section).

3. I derive a set of conditions when weather can exactly identify the effect of climate (these are sufficient, but not necessary).  I'm pretty sure this is the first time this has been done (and I'm also pretty sure I'll be taking heat for this over the next several months).  If it is true that

• the outcome of interest (Y) is the result of some maximization, where we think that Y is the maximized value (e.g. profits or GDP),
• all adaptations to climate can take on continuous values (e.g. I can chose to change how much irrigation to use by continuous quantities),
• the outcome of interest is differentiable in these adaptations, 

then by the Envelope Theorem the marginal effect of weather variation on Y is identical to the marginal effect of identically structured changes in climate (that "identically structured" idea is made precise in the paper, Mike discussed this general idea earlier). This means that even though "weather is not equal to climate," we still can have that "the effect of weather is equal to the effect of climate" (exactly satisfying the marginal treatment comparability assumption above). If we're careful about how we integrate these marginal effects, then we can use this result to construct estimated effects of non-marginal changes in climate (invoking the Gradient Theorem), which is often what we are after when we are thinking about future climate changes.

From the archives: Friends don't let friends add constants before squaring

I was rooting around in my hard-drive for a review article when I tripped over this old comment that Marshall, Ted and I drafted a while back.

While working on our 2013 climate meta-analysis, we ran across an interesting article by Ole Thiesen at PRIO where he coded up all sorts of violence at a highly local level in Kenya to investigate whether local climatic events, like rainfall and temperature anomalies, appeared to be affecting conflict. Thiesen was estimating a model analogous to:

and reported finding no effect of either temperature or rainfall. I was looking through the replication code of the paper to check the structure of the fixed effects being used when I noticed something, the  squared terms for temperature and rainfall were offset by a constant so that the minimum of the squared terms did not occur at zero:



(Thiesen was using standardize temperature and rainfall measures, so they were both centered at zero). This offset was not apparent in the linear terms of these variables, which got us thinking about whether this matters. Often, when working with linear models, we get used to shifting variables around by a constant, usually out of convenience, and it doesn't matter much. But in non-linear models, adding a constant incorrectly can be dangerous.

After some scratching pen on paper, we realized that

for the squared term in temperature (C is a constant), which when squared gives:

because this constant was not added to the linear terms in the model, the actual regression Thiesen was running is:

which can be converted to the earlier intended equation by computing linear combinations of the regression coefficients (as indicated by the underbraces), but directly interpreting the beta-tilde coefficients as the linear and squared effects is not right--except for beta-tilde_2 which is unchanged. Weird, huh? If you add a constant prior squaring for only the measure that is squared, then the coefficient for that term is fine, but it messes up all the other coefficients in the model.  This didn't seem intuitive to us, which is part of why we drafted up the note.

To check this theory, we swapped out the T-tilde-squared measures for the correct T-squared measures and re-estimated the model in Theisen's original analysis. As predicted, the squared coefficients don't change, but the linear effects do:

This matters substantively, since the linear effect of temperature had appeared to be insignificant in the original analysis, leading Thiesen to conclude that Marshall and Ted might have drawn incorrect conclusions in their 2009 paper finding temperature affected conflict in Africa. But just removing the offending constant term revealed a large positive and significant linear effect of temperature in this new high resolution data set, agreeing with the earlier work. It turns out that if you compute the correct linear combination of coefficients from Thiesen's original regression (stuff above the brace for beta_1 above), you actually see the correct marginal effect of temperature (and it is significant).

The error was not at all obvious to us originally, and we guess that lots of folks make similar errors without realizing it. In particular, it's easy to show that a similar effect shows up if you estimate interaction effects incorrectly (after all, temperature-squared is just an interaction with itself).

Thiesen's construction of this new data set is an important contribution, and when we emailed this point to him he was very gracious in acknowledging the mistake. This comment didn't get seen widely because when we submitted it to the original journal that published the article, we received an email back from the editor stating that the "Journal of Peace Research does not publish research notes or commentaries."

This holiday season, don't let your friends drink and drive or add constants the wrong way in nonlinear models.

Adaptation Illusions

For some reason, the topic of adaptation can sometimes turn me into a grumpy old man. Maybe it’s because I’ve sat through hours of presentations about how someone’s pet project should be funded under the name of climate adaptation. Or maybe it’s the hundreds of times people say that adaptation will just fix the climate problem, with no actual evidence to support that other than statements like “corn grows just fine in Alabama.” Or maybe it’s just that I’m really turning into a grumpy old man and I like excuses to act that way.

In any case, I was recently asked to write a piece for Global Food Security, a relatively new journal run by Ken Cassman that I’m on the editorial board for. I used it as a chance to articulate some of the reasons I am unconvinced by most studies trumpeting the benefits of adaptation. That’s not to say I don’t think adaptation can be effective, just that most of the “evidence” for it is currently quite weak. In fact, I think some adaptations could be quite effective, which is why it’s so important to not spend time on dead ends. The paper is here, and abstract below.

"Climate change adaptation in crop production: Beware of illusions"

A primary goal of studying climate change adaptation is to estimate the net impacts of climate change. Many potential changes in agricultural management and technology, including shifts in crop phenology and improved drought and heat tolerance, would help to improve crop productivity but do not necessarily represent true adaptations. Here the importance of retaining a strict definition of adaptation – as an action that reduces negative or enhances positive impacts of climate change – is discussed, as are common ways in which studies misinterpret the adaptation benefits of various changes. These “adaptation illusions” arise from a combination of faulty logic, model errors, and management assumptions that ignore the tendency for farmers to maximize profits for a given technology. More consistent treatment of adaptation is needed to better inform synthetic assessments of climate change impacts, and to more easily identify innovations in agriculture that are truly more effective in future climates than in current or past ones. Of course, some of the best innovations in agriculture in coming decades may have no adaptation benefits, and that makes them no less worthy of attention.

Here a MIP, there a MIP, everywhere a MIP, MIP

A growing number of papers are looking at climate change impacts using multiple models. A few more are out this week in a special issue of PNAS. Mostly I just want to point readers of this blog to them if they are interested. I am generally a big fan of model intercomparisons (MIPs). I talk about them so much in my modeling course that I can usually hear the students’ collective sigh of relief when I move on to another topic.

Most of the strengths of MIPs have been demonstrated clearly with the climate MIPs, now on their 5^th rendition. They are useful for estimating uncertainties, they can point to important weaknesses in some models, and most of all they can create something that is more than the sum of its parts – the magical multi-model mean – where independent model errors cancel and estimates become more reliable. And a positive externality of these activities is usually that experiments and observational data to rigorously test models tends to improve, since each model isn’t in charge of testing their own model (“trust is, it’s great, we validated it years ago!”)

I think two reasons the climate MIPs were so successful is that the results were made available to the entire community, and relatedly that most of the groups performing the comparisons were not simultaneously working on model improvement. I’m not sure yet if AgMIP will follow this example. In conversations with them, I think they’d like to, but are not quite there.

With all of the positives going for it, I am still a little puzzled by a few things in the recent MIP papers. For one, it’s not clear to me why agriculture studies still do so much comparison of “no CO2” and “with CO2” runs, and conclude that the difference represents some indication of how much more work needs to be done on CO2. I’m not saying that I haven’t heard various explanations, but none of them are satisfying. The chance that CO2 has no effect on crops is about the same as the chance that Wolfram will show up to work tomorrow in a dress (that’s a very low chance, in case you were wondering).  If you are looking at uncertainty from CO2, you should look at various plausible responses to CO2, and zero isn’t one of them. I can see reasons to make estimates without CO2, including if your model doesn’t treat it, or if you are focused on effects of heat in order to test adaptations, but if you are trying to look at impacts of different emissions scenarios, why keep running a model without CO2? It reeks of trying to make the problem look worse than it is.

Another niggle is that the experimental design isn’t always set up to provide insight into what causes differences between models. I’m sure that will improve with time. But for now they are drawing some big conclusions from fairly weak comparisons. For example, the figure below shows the huge spread in model results is largely from two models from LPJ and one from GAEZ being very positive. They use this to conclude that models without N limitation have more positive impacts. But there are tons of differences between these and other models, why not conclude that models that start with L or G are more optimistic? The theory is that models with N limits can’t respond as much to CO2, but it should also be the case that they can’t respond as much to temperature, as work Wolfram and I did a while back concluded. (We also saw the GAEZ model was way positive in regions that shouldn't have much N stress). They’d need to have an experimental design to really demonstrate that it's nitrogen and not something else.  

Just to be clear, I really do like the MIPs and the people involved are high quality and have been very generous with their repeated offers to participate. Unfortunately, they have more meetings than Australia has poisonous snakes (which is a lot, in case you were wondering). I am participating in a wheat site-level intercomparison, which hopefully will be out this year.

On another note, I am now fully settled in to live in Brisbane (on sabbatical), and will try to post a little more often. I’m learning lots of interesting stuff, and not all of it about cricket (although there has been a curious uptick in the national team’s performance since I went to their match my first week – see “Australia’s resurgence as a world power in cricket has been swift, ruthless and dangerous”).  Mostly I’m deep into crop physiology, which readers of this blog (if there are any left) may or may not care about, but it may be the only thing I have to talk about for a while. Also, there’s the IPCC approval session coming in a few weeks which should be interesting. I think Max will also be there blogging for one of the other sites he actually writes things for.

An orgy of technical advice for applied economists

Apologies for the tepid pace of G-Feed blogging.  Some of us are on sabbatical, some are looking for post-graduate employment, some of us have a new job and multiple cats, and some are mentally and physically preparing for the upcoming Formula 1 season.

In lieu of actual content, allow me to point folks toward a incredibly useful compendium of methodological advice for folks doing applied economics research. It's here, and is a list of the somewhat more technical blog postings that have shown up on the excellent Development Impact blog.
There you will find advice on a range of important topics from how to account for observer ("Hawthorne") effects in experimental work, to how to design randomized controlled trials to measure spillover effects, to how best to implement various popular strategies for causal inference in non-experimental data.

Also, if you're even remotely interested in development or in these methodological topics, the Development Impact blog is worth a follow.  

More fun with MARS

(But not as much fun as watching Stanford dominate Oregon last night).

In a recent post I discussed the potential of multivariate adaptive regression splines (MARS) for crop analysis, particularly because they offer a simple way of dealing with asymmetric and nonlinear relationships. Here I continue from where I left off, so see previous post first if you haven’t already.

Using the APSIM simulations (for a single site) to train MARS resulted in the selection of four variables. One of them was related to radiation which we don’t have good data on, so here I will just take three of them, which were related to: July Tmax, May-August Tmax, and May-August Precipitation. Now, the key point is we are not using those variables as the predictors themselves, but instead using hinge functions based on them. The below figure shows specifically what thresholds I am using (based on the MARS results from previous post) to define the basis hinge functions.  

I then compute these predictor values for each county-year observation in a panel dataset of US corn yields, then subtract county means from all variables (equivalent to introducing county fixed effects), and fit three different regression models:

Model 1: Just quadratic year trends (log(Yield) ~ year + year^2). This serves as a reference “no-weather” model.

Model 2: log(Yield) ~  year + year^2 + GDD  + EDD + prec + prec^2. This model adds the predictors we normally use based on Wolfram and Mike’s 2009 paper, with GDD and EDD meaning growing degree days between 8 and 29 °C and extreme degree days (above 29 °C). Note these measures rely on daily Tmin and Tmax data to compute the degree days.

Model 3: log(Yield) ~  year + year^2 + the three predictors shown in the figure above. Note these are based only on monthly average Tmax or total precipitation.

The table below shows the calibration error as well as the mean out-of-sample error for each model. What’s interesting here is that the model with the three hinge functions performs just as well as (actually even a little better than) the one based on degree day calculations. This is particularly surprising since the hinge functions (1) use only monthly data and (2) were derived from simulations at a single site in Iowa. Apparently they are representative enough to result in a pretty good model for the entire rainfed Corn Belt.

Model	Calibration R2	Average root mean square error for calibration	Average root mean square error for out-of-sample data  (for 500 runs)	% reduction in out-of-sample error
1	0.59	0.270	.285	--
2	0.66	0.241	.259	8.9
3*	0.68	0.235	.254	10.7

*For those interested, the coefficients on the three hinge terms are -.074, -.0052, and -.061 respectively

The take home here for me is that even a few predictors based on monthly data can tell you a lot about crop yields, BUT it’s important to account for asymmetries. Hinge functions let you do that, and process-based crop models can help identify the right hinge functions (although there are probably other ways to do that too).

So I think this is overall a promising approach – one could use selected crop model simulations from around the world, such as those out of agmip, to identify hinge functions for different cropping systems, and then use these to build robust and simple empirical models for actual yields. Alas I probably won’t have time to develop it much in the foreseeable future, but hopefully this post will inspire something.

Taking crop analysis to MARS

I couldn’t bear to watch the clock wind down on October without a single post this month on G-FEED. So here goes a shot at the buzzer…

A question I get asked or emailed fairly often by students is whether they should use a linear or quadratic model when relating crop yields to monthly or growing season average temperatures. This question comes from all around the world so I guess it’s a good topic for a post, especially since I rarely get the time to email them back quickly.  If you are mainly interested in posts about broad issues and not technical nerdy topics, you can stop reading now.

The short answer is you can get by with a linear model if you are looking over a small range of temperatures, such as year to year swings at one location. But if you are looking across a bigger range, such as across multiple places, you should almost surely use something that allows for nonlinearity (e.g., an optimum temperature somewhere in the middle of the data).

There are issues that arise if using a quadratic model that includes fixed effects for location, a topic which Wolfram wrote about years ago with Craig McIntosh. Essentially this re-introduces the site mean into the estimation of model coefficients, which creates problem of interpretation related to a standard panel model with fixed effects.

A bigger issue that this question points to, though, is the assumption by many that the choice is simply between linear and quadratic. Both are useful simple approaches to use, especially if data is scarce. But most datasets we work with these days allow much more flexibility in functional form. One clear direction that people have gone is to go to sub-monthly or even sub-daily measures and use flexible spline or degree day models to compute aggregate measures of temperature exposure throughout the season, then use those predictors in the regression.  I won’t say much about that here, except that it makes a good deal of sense and people who like those approaches should really blog more often.

Another approach, though, is to use more flexible functions with the monthly or seasonal data itself. This can be useful in cases where we have lots of monthly data, but not much daily data, or where we simply want something that is faster and easier than using daily data. One of my favorite methods of all time are multivariate adaptive regression splines, also called MARS. This was developed by Jerome Friedman at Stanford about 20 years ago (and I took a class from him about 10 years ago). This approach is like a three-for-one, in that it allows for nonlinearities, can capture asymmetries, and is a fairly good approach to variable selection. The latter is helpful in cases where you have more months than you think are really important for crop yields.

The basic building block of MARS is the hinge function, which is essentially a piecewise linear function that is zero on one side of the hinge, and linearly increasing on the other side. Two examples are shown below, taken from the wikipedia entry on MARS.

MARS works by repeatedly trying different hinge functions, and adds whichever one gives the maximum reduction in the sum of squared errors. As it adds hinge functions, you can have it added by itself or have it multiply an existing hinge in the model, which allows for interactions (I guess that makes it a four-for-one method). Despite searching all possible hinge functions (which covers all variables and hinges at all observed values of each variable), it is a fairly fast algorithm. And like most data mining techniques, there is some back-pruning at the end so it isn’t too prone to overfitting.

For a long time I liked MARS but couldn’t figure out how to apply it to data where you want to include spatial fixed effects to account for omitted variables. Unlike models with pre-determined predictors, such as monthly average temperature or squared temperature, MARS has to search for the right predictors. And before you know what the predictors are, you can’t substract out the site-level means as you would in a fixed-effect model. So you can’t know what the predictors are until you search, but you can’t search if you can’t compute the error of the model correctly (because you haven’t included fixed-effects.)

One semi-obvious solution would be to just ignore fixed-effects, find the hinge-function predictors, and then rerun the model with the selected predictors but including fixed effects. That seems ok but all the problems of omitted variables would still be affecting the selection process.

Recently, I settled on a different idea – first use a crop simulation model to develop a pseudo-dataset for a given crop/region, then run MARS on this simulated data (where omitted variables aren’t an issue) to find the predictors, and then use those predictors on an actual dataset, but including fixed effects to account for potential omitted variables.

I haven’t had much time to explore this, but here’s an initial attempt. First, I used some APSIM simulations for a site in Iowa that we ran for a recent paper on U.S. maize. Running MARS on this, allowing either monthly or seasonal average variables to enter the model, results in just four variables that are able to explain nearly 90% of the yield variation across years. Notice the response functions (below) show the steepest sensitivity for July Tmax, which makes sense. Also, rainfall is important but only up to about 450mm over the May-August period. In both cases, you can see how the results are definitely not linear and not symmetric. And it is a little surprising that only four variables can capture so much of the simulated variation, especially since they all contain no information at the sub-monthly time scale.

Of course this approach relies on assuming the crop model is a reasonable representation of reality. But recall we aren’t using the crop model to actually define the coefficients, just to define the predictors we will use. The next step is to then compute these predictors for actual data across the region, and see how well it works at predicting crop yields. I actually did that a few months ago but can’t find the results right now, and am heading off to teach. I’ll save that for a post in the near future (i.e. before 2015).

The good and bad of fixed effects

If you ever want to scare an economist, the two words "omitted variable" will usually do the trick. I was not trained in an economics department, but I can imagine they drill it into you from the first day. It’s an interesting contrast to statistics, where I have much of my training, where the focus is much more on out-of-sample prediction skill. In economics, showing causality is often the name of the game, and it’s very important to make sure a relationship is not driven by a “latent” variable. Omitted variables can still be important for out-of-sample skill, but only if their relationships with the model variables change over space or time.

A common way to deal with omitted variable bias is to introduce dummy variables for space or time units. These “fixed effects” greatly reduce (but do not completely eliminate) the chance that a relationship is driven by an omitted variable. Fixed effects are very popular, and some economists seem to like to introduce them to the maximum extent possible. But as any economist can tell you (another lesson on day one?), there are no free lunches. In this case, the cost of reducing omitted variable problems is that you throw away a lot of the signal in the data.

Consider a bad analogy (bad analogies happen to be my specialty). Let’s say you wanted to know whether being taller caused you to get paid more. You could simply look at everyone’s height and income, and see if there was a significant correlation. But someone could plausibly argue that omitted variables related to height are actually causing the income variation. Maybe very young and old people tend to get paid less, and happen to be shorter. And women get paid less and tend to be shorter. And certain ethnicities might tend to be discriminated against, and also be shorter. And maybe living in a certain state that has good water makes you both taller and smarter, and being smarter is the real reason you earn more. And on and on and on we could go. A reasonable response would be to introduce dummy variables for all of these factors (gender, age, ethnicity, location). Then you’d be looking at whether people who are taller than average given their age, sex, ethnicity, and location get paid more than an average person of that age, sex, ethnicity, and location.

In other words, you end up comparing much smaller changes than if you were to look at the entire range of data. This helps calm the person grumbling about omitted variables (at least until they think of another one), and would probably be ok in the example, since all of these things can be measured very precisely. But think about what would happen if we only could measure age and income with 10% error. Taking out the fixed effects means removing a lot of the signal but not any of the noise, which means in statistical terms that the power of the analysis goes down.

Now to a more relevant example. (Sorry, this is where things may get a little wonkish, as Krugman would say). I was recently looking at some data that colleagues at Stanford and I are analyzing on weather and nutritional outcomes for district level data in India. As in most developing countries, the weather data in India are far from perfect. And as in most regression studies, we are worried about omitted variables. So what is the right level of fixed effects to include? Inspired by a table in a recent paper by some eminent economists (including a couple who have been rumored to blog on G-FEED once in a while), I calculated the standard deviation of residuals from regressions on different levels of fixed effects. The 2^nd and 3^rd columns in the table below show the results for summer (June-September) average temperatures (T) and rainfall (P). Units are not important for the point, so I’ve left them out:

	sd(T)	sd(P)	Cor(T1,T2)	Cor(P1,P2)
No FE	3.89	8.50	0.92	0.28
Year FE	3.89	4.66	0.93	0.45
Year + State FE	2.20	2.18	0.84	0.26
Year + District FE	0.30	1.63	0.33	0.22

The different rows here correspond to the raw data (no fixed effect), after removing year fixed effects (FE), year + state FE, and year + district FE. Note how including year FE reduces P variation but not T, which indicates that most of the T variation comes from spatial differences, whereas a lot of the P variation comes from year-to-year swings that are common to all areas. Both get further reduced when introducing state FE, but there’s still a good amount of variation left. But when going to district FE, the variation in T gets cut by nearly a factor of 10, from 2.2 to 0.30! That means the typical temperature deviation a regression model would be working with is less than a third of a degree Celsius. 

None of this is too interesting, but the 4^th and 5^th columns are where things get more related to the point about signal to noise. There I’m computing the correlation between two different datasets of T or P (details of which ones are not important). When there is a low correlation between two datasets that are supposed to be measuring the same thing, that’s a good indication that measurement error is a problem. So I’m using this correlation here as an indication of where fixed effects may really cause a problem with signal to noise.

Two things to note. First is that precipitation data seems to have a lot of measurement issues even before taking any fixed effects.  Second is that temperature seems ok, at least until state fixed-effects are introduced (a correlation of 0.842 indicates some measurement error, but still more signal than noise). But when district effects are introduced, the correlation plummets by more than half.

The take-home here is that fixed effects may be valuable, even indispensible, for empirical research. But like turkey at thanksgiving, or presents at Christmas, more of a good thing is not always better.

UPDATE: If you made it to the end of this post, you are probably nerdy enough to enjoy this related cartoon in this week's Economist.