This is a post I’d thought of writing for a while, but given the small furore over the error in the recent Cawley et al. (2015) paper, I thought I’d do so now. Let me first explain my title (which I had thought of calling “the 2+2=4 fallacy”). What I sometimes encounter are those who think that if you are to critique their work, you need to find some kind of actual mistake in one of their calculations. If you can’t, then they conclude that they’re right. The problem is, that it’s quite possible to do a calculation that is correct and then draw conclusions that are not; hence the title.
A research study involves a number of important steps. You need to define the problem you want to solve. You need to set up the problem and define your assumptions. You need to collect your data and carry out your calculations/modelling. You then need to analyse your results, interpret your analysis, and draw your conclusions. However you do need to interpret your results and draw your conclusions in light of the assumptions that were made at the beginning. So, there are many aspects of a study that could be criticised. Just because a study has no explicit errors, doesn’t mean the conclusions are correct. Similarly, just because a study has an error, doesn’t mean the conclusions are wrong; it depends on the significance of the error and how it would influence the conclusions.
Now we come back to the motivation for this post: the error in the recent Cawley et al. paper. Cawley et al. (2015) was mainly a comment on a paper by Craig Loehle called A minimal model for estimating climate sensitivity. Let’s first consider Loehle (2014). It attempts to use a three-component model for the surface temperatures since 1880. The model has two cyclical functions, one with a 20-year period and the other with a 60-year period, and a linear trend that is meant to represent the recovery from the Little Ice Age (LIA). After 1950, these no longer properly fit the temperature data, so another linear trend is introduced – starting in 1942 – which is meant to represent the temperature responding to the increased atmospheric CO2 concentration.
Using this model, Loehle then does a calculation to determine climate sensitivity. The problem is that the model and the assumptions don’t make any physical sense. What does recovery from the LIA even mean? Our climate isn’t a blow-up ball that returns to its original shape after being compressed. Our climate largely responds to changes in external forcings. The LIA was a period with reduced solar insolation and increased volcanic activity. The period after that, warmed in response to increases in external forcings. There aren’t really 20 and 60-year cycles in the external forcing datasets. Anthropogenic forcings didn’t start in 1942, they started when we started increasing atmospheric CO2 concentrations in the 1800s. The model is basically a curve fitting exercise with almost no physical basis whatsoever. I don’t need to really go any further. It doesn’t really matter if the subsequent calculations have errors or not; the study doesn’t really make physical sense.
What about Cawley et al. (2015)? They show that the 20 and 60-year cycles aren’t really supported by the observations. They show that Loehle et al. (2014) underestimated the uncertainty in their climate sensitivity analysis. They then present a minimal model of their own (that actually has fewer parameters than the minimal model presented by Loehle et al.) to show how one might develop one that is physically motivated, rather than one that is simply a curve-fitting exercise. They also discuss how Loehle et al. (2014) only used the forcing due to CO2, rather than due to all anthropogenic influences. Consequently, Loehle et al. underestimated the change in forcing by about 13%.
Here’s where there was a mistake. Rather than pointing out that this would have reduced Loehle’s estimate for climate sensitivity (making it even more unrealistic), they suggested it would have increased it. A mistake. However, this didn’t influence the minimal model that they presented and it didn’t really influence their discussion of Loehle’s model. It was simply a silly mistake. Just because Cawley et al. (2015) made a silly mistake doesn’t make Loehle’s model any more realistic. Just because Cawley et al. (2015) made a silly mistake doesn’t invalidate the rest of their paper.
So, if people were serious about discussing papers like this, they’d focus on more than whether or not they can find some kind of silly arithmetic error (especially as it’s normally easy to establish if such errors are significant or not). They’d focus on the setup of the problem, the assumptions, the analysis of the data, and the conclusions that are drawn. Of course, there are probably reasons why people are focusing on a minimal error in Cawley et al. (2015) and ignoring a completely unrealistic model presented by Loehle (2014); it would be inconvenient to do otherwise.