Monday, September 26, 2016

Evidence Should be Used in Global Management of Endangered Species: reply to the CITES Technical Advisory Group

In anticipation of the 17th meeting of the Conference of the Parties of the Convention on the International Trade in Endangered Species (CITES), the CITES Technical Advisory Group (TAG) on elephant poaching and ivory smuggling released a document (numbered CoP17 Inf. 42) in which they directly addressed a working paper released by Nitin Sekar and myself.

The document did not introduce new substantive arguments (see end of post for one exception regarding dates)—in fact, the document’s claims were nearly identical to those made in Fiona Underwood’s prior blog posts about our analysis (here and here). In her second post, Dr. Underwood (one of a handful of members of the TAG) openly states that the TAG asked her to evaluate our analysis, and it seems they have simply adopted her view as the official stance. These arguments provide the basis for the TAG's conclusion:
13) The claims in the working paper by Hsiang and Sekar are fundamentally flawed, both in logic and methodology. The MIKE and ETIS TAG is therefore of the view that the study should not be used to inform CITES policy on elephants.
Given that the arguments have not changed, we have previously responded to almost all the document’s claims in our two prior blogposts, here and here. Below we summarize the relevant points and address new points.

As shown below,
  • numerous statements about our analysis (either in the manuscript or follow-up analysis in these prior posts) made by the CITES white paper are factually incorrect;
  • in supporting its narrative, the white paper misreports results and conclusions of other studies;
  • the white paper makes erroneous statistical arguments;
  • when the approach advocated for in the white paper is correctly implemented, it provides results virtually identical to the original findings in our analysis.
Thus, overall, there is no evidence to support the claims in the CITES white paper and therefore no reason for CITES to refuse to consider these results when developing policy.


The white paper reiterates the claim that previous findings reported by CITES using Aggregate PIKE statistics (which do not indicate a 2008 jump in poaching) are correct and reliable. In our previous post we carefully derived in detail why this Aggregate PIKE measure is a meaningless number. In that post we summarized:
Aggregate PIKE is a complex mathematical object with no useful interpretation or application in policy analysis. Our best guess is that it came into common use because the useful properties of PIKE computed locally (i.e. removing confounding influences of elephant population and surveyor effort) seemed appealing and seemed like they ought to apply to Aggregate PIKE as well. They do not.”
We refer interested readers to that post for details on this particular matter.

This critical omission reflects a broader pattern clear in the CITES document, where non-reviewed opinions publicly expressed by Dr. Underwood are treated as if they are facts, while the document simultaneously ignores direct public responses to those views that contain carefully derived results. This occurs in multiple cases where “facts” originating from Dr. Underwood's blog have been publicly overturned by actual empirical results or derivations presented in this blog—results that we personally communicated to Dr. Underwood and other members of the TAG. The CITES white paper repeatedly and directly cites Dr. Underwood’s erroneous claims on her blog but does not reference or acknowledge our direct point-by-point replies (here and here) where replication code and mathematical derivations have been made completely public.

Substantive arguments made in the CITES white paper


The core substantive arguments of the CITES TAG position are found in these three paragraphs:
8) Regardless of whether the increase in poaching observed over the last 10 years was gradual or sudden, there are many other factors that could have caused, or contributed to causing, an increase in poaching levels. For example, the impact of the Global Financial Crisis was also felt worldwide at the time of the ivory sale, and this could have affected trends in the illegal ivory trade. In 2008 there was a drop in international shipping costs, which may have made illegal trade in ivory more profitable, and thereby more attractive to criminal syndicates [17]. There is also evidence of similar trends in the price and volume of trade in other commodities over the same period, including goods that take a similar role to elephant ivory, such as gold, luxury watches and semiprecious stones [18, 19, 20]. As mentioned above, 2008 also marked the start of the still-ongoing 9- year moratorium on further ivory sales. By the authors’ logic, events such as these could have also been causes for the step change they claim. It is also worth noting that there has also been a rapid increase in poaching of other high-value species, such as African rhinos, over the same period but no one-off sales of these species’ products have taken place in that time [21, 22]. Other potential causes of the increase in poaching over the last 10 years cannot be ruled out without considering factors such as these. 
....
10) With regard to the statistical analysis, the TAG’s view is that the authors have not properly modelled the properties or structure of the data; in particular, the linear model they used is inappropriate [23, 24, 25]. The authors illustrate their argument using a plot which appears to show a step change in the average estimate of elephant poaching levels from 2008. This pattern is different to the results reported by the MIKE programme, which, as described above, essentially show a gradual increase in PIKE from 2006 to 2011. The key factor leading to the difference between these results is that the MIKE analyses take account of the variation in the total number of carcasses reported by each site each year, whereas this is not the case in Hsiang and Sekar’s analysis. 
11) Accounting for the variation in total carcass counts is not a matter of choice. The data show that the total number of carcasses encountered at MIKE sites varies greatly between sites and over time, making it a confounding factor that needs to be adjusted for in the analysis. Hsiang and Sekar could have accounted for this variability using a weighted linear regression, with the total carcass counts as the weights. Such weighting is vital to avoid violating a basic assumption in linear regression, namely that of variance homogeneity. Hsiang and Sekar have ignored these facts, and this renders their analysis, and thereby their conclusions, invalid [24]. Had Hsiang and Sekar weighted the data appropriately, or used more suitable methods for modelling proportions, which automatically account for the variability in total carcass counts [26], they would have obtained essentially the same results as those presented in the MIKE analyses, which show no discontinuity in 2008.
We address each statement in these paragraphs below. Overall, they are either (i) false, (ii) misrepresentations of statements made by researchers or statistics generally, or (iii) do not openly convey the depth of analysis we previously presented.

Other possible causes (section 8 in the CITES white paper)


In our original analysis, we examined sixteen variables proximate to causes proposed by earlier analyses to contribute to the recent rise in elephant poaching. Many trend upward over time, but none displayed discontinuities that could potentially explain this observed jump in 2008:


The CITES white paper suggest four additional hypotheses that are testable with data, three of which were directly addressed in our paper or earlier response to Dr. Underwood but have been ignored in the white paper.


First, the CITES document hypothesizes
In 2008 there was a drop in international shipping costs, which may have made illegal trade in ivory more profitable, and thereby more attractive to criminal syndicates [17].
Where the reference [17] supporting the claim is Moyle (2014) in Ecological Economics. However, that paper does not provide any evidence nor claim that changes in shipping costs may have increased global elephant poaching in 2008. Rather, Moyle argues that falling shipping costs in 2009 might have led to increased smuggling from storehouses in elephant range states to storehouses in Asia. In fact, the evidence presented in the paper suggests that abnormally high shipping costs in 2008 should have suppressed smuggling in 2008, rather than elevating poaching as the CITES document suggests. The relevant figure from Moyle is:

BDI is the shipping cost index. From: Moyle (2014)

Notably, the analysis in Moyle provides no findings regarding direct effects on poaching, as the CITES document suggests. Furthermore, Moyle directly and explicitly states: 
“The CITES secretariat (2010, n26) claimed that some of the recent seizures came from elephants poached in the early 2000s. This is consistent with the high shipping costs in the mid-2000s deterring smuggling until these costs fell.”
Indicating that both Moyle and CITES think the change in smuggling might be unrelated from concurrent poaching activity in 2009.

Thus, the purported 2009 change in shipping costs cited by the CITES white paper occurred in the wrong year to trigger a 2008 jump in poaching, the referenced analysis does not argue that this change should have triggered additional poaching, and CITES previously stated that the possible surge in smuggled ivory appeared to be from elephants poached much earlier that decade. 

Thus, inspection of the reference provided by CITES does not provide any support for the hypothesis that shipping cost changes triggered the 2008 discontinuous change in poaching.


Second, the CITES document hypothesizes:
There is also evidence of similar trends in the price and volume of trade in other commodities over the same period, including goods that take a similar role to elephant ivory, such as gold, luxury watches and semiprecious stones [18, 19, 20]. 
However, this claim was rejected in our previous response to Dr. Underwood's similar claim made earlier.  If the discontinuous jump in poaching we observed was caused by a jump in use of ivory as an investment good, then similar patterns should be observed for similar precious materials that are treated as similar investments.  In our previous analysis, we obtained data from on sales of jewelry, gold, and diamond sales in China during this period from Hsu et al (2014), which also is reference [20] in the CITES document. In contrast to this investment hypothesis, there is no discontinuity in sales for any of these precious materials:

click to enlarge, details here
There are kinks in these trends, but no discontinuities similar to the ivory poaching data. Trends in purchasing of these materials may have changed in 2008, but there is no evidence to support the hypothesis that this change was so abrupt it would generate discontinuous supplier behavior in 2008. In all cases, markets in 2009 appear to behave exactly as a linear extrapolation from 2008 would predict. Even though this result was presented in our previous reply to Dr. Underwood, it was ignored in the CITES white paper and the data used to make this figure was cited by CITES as supporting exactly the opposite claim. 

Thus, inspection of sales data for precious materials in China provides no support of the hypothesis that abrupt changes in investment behavior in 2008 could have generated the observed global discontinuity in elephant poaching.


Third, the the CITES document hypothesizes:
As mentioned above, 2008 also marked the start of the still-ongoing 9- year moratorium on further ivory sales. 
However, this argument was addressed in the our original text that the CITES document is responding to. Specifically, in footnote 12 (shown below) of our analysis, we point out that the commitment to the moratorium occurred in 2007, not in 2008 when the abrupt increase in poaching occurred. The original analysis stated in footnote 12:

Footnote 12 from Hsiang & Sekar (2016)

This fact as well as its presentation in the original analysis were omitted from the CITES evaluation.  If the declaration of the (future) moratorium were responsible for a rise in poaching, we would have expected the discontinuous change in poaching to have occurred in 2007.

Furthermore, we note that one cannot just speculate that any event in 2007 may have caused the increase in poaching in 2008 without some plausible theoretical economic framework. Nowhere have we seen a formal theoretical explanation of how or why the 2007 partial moratorium would cause an increased incentive to poach. In contrast, our original analysis provides a formal theoretical explanation of how partial legalization of ivory (e.g., the one-time sale) could lead to more poaching by directly affecting the cost of supplying illegal ivory and the demand for illegal ivory, and we know that the 2008 decision to sell ivory to China provided a clear and definitive signal to ivory markets.

Thus, we assert that the hypothesis that a moratorium caused an increase in poaching is unsubstantiated: it lacks a theoretical foundation, does not have timing that corresponds with the abrupt increase in poaching, and was presented speculatively without either qualitative or quantitative empirical backing. 


Fourth, the CITES document hypothesizes that the response of elephant poaching to conditions in 2008 was not unique, suggesting that non-elephant-specific causes of poaching patterns are plausibly responsible for the 2008 jump in elephant poaching. The document states:
It is also worth noting that there has also been a rapid increase in poaching of other high-value species, such as African rhinos, over the same period but no one-off sales of these species’ products have taken place in that time [21, 22]. 
However, this suggestion was rejected in our previous response to Dr. Underwood's similar statement made earlier.  We obtained what public data was available on poaching of rhino, leopards, and tigers (we also requested from Dr. Underwood links/names of non-specific data sources referenced by her, and received no reply).  Similar analysis of these data (which are admittedly of lower quality and breadth than the MIKE data) show no similar discontinuous jump in 2008:

click to enlarge, details here
Rhino poaching in Kenya jumps discontinuously in 2009, but that is the wrong year and an isolated case (it is not systematic as the elephant poaching patterns are). Other species show kinks in their trends, similar to precious materials above, but none display a discontinuity in 2008. This result was also released in response to prior criticisms by Dr. Underwood but not referenced in the CITES document.

Thus, inspection of poaching data for non-elephant species provides no support of the hypothesis that the abrupt 2008 jump in elephant poaching was representative of broader global patterns of poaching.

Methodological Concerns (section 10-11 in the CITES white paper)


The CITES white paper makes multiple claims about that validity of the statistical results in our analysis (the simplest of which is shown in the first figure above).  This result is an extremely simple annual average of PIKE, after site-level means have been removed. The validity of this approach was explained in detail in our first and second replies to Dr. Underwood, where we created an Excel replication file that can be easily understood by non-statisticians. We respond briefly again to the main criticisms in the CITES document, as it appears to ignore those earlier responses.

First, the CITES white paper first states
With regard to the statistical analysis, the TAG’s view is that the authors have not properly modelled the properties or structure of the data; in particular, the linear model they used is inappropriate [23, 24, 25].
Which is a criticism of the linear probability model we use (this critique is more explicit in Dr. Underwood's blog posts, which are references [23] and [24] in the CITES document). The central concern is whether variations in PIKE appear to be normally distributed, after site-level and annual averages have been removed. This point was directly and explicitly addressed in our second response to Dr. Underwood:
In Dr. Underwoods critique, she correctly describes the most parsimonious statistical model that we use (and describe in our previous post in detail). From Dr. Underwood’s post: 
however she argues that the assumption that the residuals (p_ij in her Table) cannot be correct based on her intuition about the data. This motivates her to utilize more complex GLM approaches that make more and stronger assumptions that are very difficult to defend (such as the assumption that the number of elephant carcasses at each site are predetermined each year, which we think is indefensible), and that lead to her different conclusions. This logic is spelled out explicitly in her replication code where she writes (emphasis added):
# Average to get  mean value for each year 
pred.av <- tapply(pred,exp.gd$year.f,mean)... 
# But I don't like the fact that the data are treated as normal data
#Fit a Binomial model instead 
resp<- with(data.0,(cbind(illegal,totcarc-illegal)))glm1 <- glm(resp~siteid+year.f,family=binomial,data=data.0)
In contrast to Dr. Underwood, in our original analysis we did not simply trust our intuition about what we thought the data should look like. Instead, we looked at what the data looked like. And we included the necessary checks in our original paper. In the original appendix section "Checking assumptions of a linearized approach," we discussed the assumption that residuals of the model were normal and we presented Appendix Figure A8, which compared the CDF of our residuals to the CDF of a normal distribution.  
The distribution of the data do in fact look normally distributed, as Dr. Underwood agreed they would have to look in order for the model to be valid:


Despite this direct check of the condition set forth by Dr. Underwood, the CITES white paper does not acknowledge the appropriateness of the model.

The CITES white paper goes on to state
Hsiang and Sekar could have accounted for this variability using a weighted linear regression, with the total carcass counts as the weights. Such weighting is vital to avoid violating a basic assumption in linear regression, namely that of variance homogeneity. Hsiang and Sekar have ignored these facts, and this renders their analysis, and thereby their conclusions, invalid [24].
A statement that has two components. First, there is the recommendation to use weighted linear regression. Second, there is the logic behind this recommendation. We address the logic first.

The CITES document states that "variance homogeneity" is "a basic assumption in linear regression" that we violate. This is false on two counts. First, variance homogeneity is not a fundamental assumption of linear regression. It is required for the Gauss-Markov theorem to hold (which is why many individuals become confused and think it is required more generally) but that is a different matter entirely. Most analyses in modern econometrics do not assume "variance homogeneity" (known as homoscedasticity) even if they implement linear regression, and the methods to handle this situation are well understood. This brings us to the second erroneous aspect of the statement: in our original analysis we did not make this "variance homogeneity" assumption that the document claims we did. On page 47 of the original paper, we explained that this assumption was clearly inappropriate and explained that our approach accounted very flexibly for various structures in the data: 

Clipping from pg 47 of Hsiang & Sekar (2016)

The details of this particular statistical technique are inessential here.  We raise this point simply to demonstrate that the statistical logic used to support claims in the CITES white paper are erroneous and its portrayal of the statistical analysis we implemented is incorrect.

Now for the recommendation that we should have used weighted linear regression. This was the focus of Dr. Underwood's second critique. In our reply, we noted that the implementation and justification proposed by Dr. Underwood was incorrect, since the number and quantity of total carcasses observed at any site was not the population from which poached elephant carcasses are drawn. We also derived why the implementation that Dr. Underwood was proposing, which is also what is proposed in the CITES white paper, is guaranteed to provide biased results. Furthermore, recent work by Solon et al (2015) explicitly states that the logic employed by Dr. Underwood and the CITES white paper is not correct. 

Despite the absence of a valid logical foundation, we can nonetheless implement the weighting scheme proposed by by the CITES white paper, using total carcass counts as weights.  Importantly, however, to avoid the systematic bias caused by higher poaching years changing the weights of observations within a single site, we use the average total carcass count at each site as the analytical weight. This provides a constant weight for each site that reflects whether many or few total carcasses are reported from that site on average and it assumes that this value is inversely proportional to the variance of the observation (as the CITES white paper and Dr. Underwood's prior post advocate).  When we implement this weighted regression, we recover a discontinuity with a size (+0.128) virtually identical to our originally reported value (+0.129):

Weighted regression results (as prescribed by the CITES TAG white paper) also show a clear 2008 jump in poaching

Code to replicate this figure is here.

Thus our originally reported finding is unchanged if one implements the weighted regression approach prescribed by the CITES white paper.  

This finding is unsurprising, given that in our response to the World Bank blog we show the estimated discontinuity is essentially constant across sites that report different numbers of total carcasses.  This result indicates that the statement in the CITES document
Had Hsiang and Sekar weighted the data appropriately, or used more suitable methods for modelling proportions, which automatically account for the variability in total carcass counts [26], they would have obtained essentially the same results as those presented in the MIKE analyses, which show no discontinuity in 2008.
is not true.  

Overall, the CITES white paper provides no valid methodological ground for dismissing the results of our analysis.  Various methodological criticisms are grounded in erroneous logic or false description of the content of our analysis. In two cases, the document (and Dr. Underwood's public critique) lay out explicit criteria for our results to be valid (normality in residuals, non-homogenous errors, and weighting by total carcasses). In all cases these issues have been addressed such that we "pass" the tests laid out by Dr. Underwood and the CITES white paper. Thus we see no valid reason for CITES to not consider our findings.

Conclusion


What is critical for readers to understand is that our argument is qualified and evidence-driven. We are not saying (and have never said) that the one-time sale of legal ivory in 2008 was the only cause for the recent increase in poaching. We are saying that the abrupt increase in poaching in 2008 and all the evidence detailed above is consistent with the hypothesis that the sale triggered the observable rapid rise in poaching. At this time, we see no other hypothesis that is theoretically substantiated and matches the data. We have seriously considered and tested all alternative hypotheses put forward by our colleagues and the CITES (MIKE ETIS) TAG and none are supported by any available evidence.

Given that all of the substantive criticisms of our analysis presented in the CITES white paper are either factually incorrect, based on logical errors, or have been directly tested and rejected using data, we see no justifiable reason that the results of our analysis should be dismissed when considering future CITES policies. We firmly believe that all of the best available data and evidence should be carefully considered in the construction of global policies that govern the fate of entire species. 


Ancillary note on documentation of facts immediately relevant to the timing of events


As an aside, we note that some of the facts recounted in CITES TAG document Inf. 42 differ from facts described in other official CITES documents. The authors write, 
In 2007, at its 14th meeting, the CITES Conference of the Parties approved, by consensus, the international sale of government-owned raw ivory from the four African elephant populations included in Appendix II (i.e. those of Botswana, Namibia, South Africa, and Zimbabwe) to approved trading partners (namely China and Japan)… the ivory reached its destinations in January 2009. At the same time, the Conference of the Parties established a moratorium of nine years, from the date of the sale, on the submissions of further proposals to the Conference of the Parties to allow trade in elephant ivory from those four populations.
This account differs from that we found in our research utilizing primary CITES documents. Key to our argument are the following differences:
  1.  In 2007, CITES only approved Japan as a trading partner for the sale, which remained unscheduled. In fact, CITES voted against approving China as a trading partner, leaving their involvement in the sale uncertain. 
  2. In March 2008, after a visit to China, the CITES Secretariat wrote favorably of China’s readiness to participate in a legal sale. In July 2008, CITES approved China as a trading partner, ending uncertainty about whether the hitherto largest market for illegal ivory would receive an injection of legal ivory. 
These differences are particularly important because, in our analysis, it was the events in 2008 that established definitively that China would receive legal ivory and acted as a signal that, based on our inference, is likely to have caused the increase in poaching that began that year. We do not know why the current CITES document is the first (to our knowledge) to describe the approval of China as a buyer to have occurred in 2007.

1 comment:

  1. Dear Dr. Hsiang,
    Thank you very much for your thought-provoking piece. Although I am no expert in statistics, I found your rebuttals to the TAG's criticisms very convincing. Also, your hypothesis linking the legal sale of ivory with increased poaching reminds me very much of a study that was recently published on the legal culling (and illegal poaching) of wolves in Wisconsin and Michigan:

    https://news.mongabay.com/2016/05/new-research-contradicts-claim-legal-hunting-reduces-poaching/

    The authors of this study similarly found that “allowing wolf (Canis lupus) culling was substantially more likely to increase poaching than reduce it" and they hypothesize that when a government authorizes the killing of a protected species, "the perceived value of each individual of that species may decline." I am not sure whether the current CITES conference is talking about possibly allowing just the legal trade in ivory, or also the legal "sustainable" culling of elephant populations, but it's important to keep in mind that governments' announcements about the protection or value of certain species *do* affect how people value those species! Good luck to you in your research in this area!

    ReplyDelete