Watt about the null hypothesis?

There’s a recent post on Watts Up With That (WUWT) called Can we actually even tell if humans are affecting the climate? What if we did nothing at all?. It’s part of an essay by someone called Charlie Martin.

The post has a set of clear points about the practice of science

1. We generate a number of alternate explanations, hypotheses, that might explain the phenomenon.

2. For each hypothesis, we come up with an experiment which will prove the hypothesis wrong. That is, not one that “proves the hypothesis”, but one which, if successful, would disprove or falsify the hypothesis. (Sir Karl Popper argued in his book The Logic of Scientific Discovery that this falsification was the core of scientific knowledge.)

3. We do the experiments. If an experiment falsifies a hypothesis, we discard it ruthlessly. Then we go back to (1) and try again.

I was thinking about this a little yesterday because I came across a post called evidence, absence and the Type II monster on a blog called In The Dark. I found this quite an interesting post, but should acknowledge that I’m not an expert at this type of statistical thinking.

The basic points made in the WUWT post (shown above) are probably, strictly speaking, correct but rather overstate the case. There are known errors associated with testing the null hypothesis. A type I error is one in which you reject the null even though it is correct (false positive). A type II error is when the null hypothesis is not rejected even though it’s false (false negative). So simply testing the null hypothesis and getting a result does not immediately indicate that one has proven or disproven a hypothesis. One needs to check on the chance of false positives or false negatives. Furthermore, I think point 3 above rather oversimplifies the actual process. One could run an experiment that turns out to be marginally consistent with the null hypothesis. Therefore, the null hypothesis cannot be rejected. This doesn’t immediately mean that the original hypothesis is false. It could simply be that more data is needed so as to improve the statistical confidence. So the data being consistent with the null hypothesis may simply mean that one cannot make any strong statements about the original hypothesis. Of course, if the data is highly consistent with the null and the chance of a false positive is very low, then one may well conclude that the original hypothesis is false.

So, this essay by Charlie Martin goes on to consider climate change. To paraphrase, he suggests that the hypothesis is that humans are emitting CO2 into the atmosphere and that this is leading to global warming. Let me simplify that to the planet is currently undergoing global warming. His essay then includes the following figure.

Comparison between model estimates of the temperature anomaly and the observed temperature anomaly.

Comparison between model estimates of the temperature anomaly and the observed temperature anomaly (credit : Ed Hawkins).


He then goes on to say that the observed temperature is outside the 95% confidence interval and hence (I assume) that this is consistent with the null hypothesis (no warming) and therefore it’s time for some new hypotheses.

This is, in my opinion, a great example of why one has to careful about how to apply the null hypothesis. As pointed out in the In The Dark blogpost, the null hypothesis has to be well-defined in terms of the model. The hypothesis that is being tested is not that global surface temperatures will rise. The hypothesis being tested is, the Earth is undergoing global warming. Global warming is about energy, not simply surface temperatures. To test the null hypothesis one has to consider the energy in the climate system, not simply the global surface temperature anomaly. If one were to do this properly one would consider the measured top-of-the atmosphere energy imbalance, the ocean heat content, the various temperature anomaly datasets (land, sea surface, troposphere), the evolution of arctic sea ice. The null hypothesis would be that, on average, the energy in the climate system has not increased. Well, you only need to look at the various datasets for it to be clear that the data is not consistent with the null hypothesis and hence we can conclude that global warming is indeed taking place (I appreciate that this isn’t really a proper statistical test, but it should be obvious. Also, strictly speaking, one would reject the null in favour of the alternative, rather than simply accepting the alternative).

I appreciate that above I haven’t been considering the full hypothesis (global warming is happening and is anthropogenic) but that’s because that just adds an extra level of complexity. It could be tested in the same way but one would need a null hypothesis that considered how the energy in the climate system would evolve if there was no anthropogenic influence and then compare this with the actual data. It must be possible to do this, but simply comparing global surface temperature anomalies with model predictions is certainly not the correct way to do so.

Advertisements
This entry was posted in Anthony Watts, Climate change, Global warming, Watts Up With That and tagged , , , , , , , . Bookmark the permalink.

17 Responses to Watt about the null hypothesis?

  1. Lars Karlsson says:

    Hre is what Ed Hawkins, the source of that graph has to say about the matter:

    “What can be learnt from this comparison? Simply, global temperatures have not warmed as much as the mean of the model projections in the past decade or so and are currently at the lower edge of the ensemble of simulations. However, there are simulations which are consistent with the observations.”

    Charlie Martin seems very confused:

    “In global warming, the null hypothesis would be that the “treatment” has no effect, or in other words that the human-caused increase in CO2 is overwhelmed by other effects. And again, note that this isn’t the same as saying “it has no effect,” just that we can’t tell if there has been an effect.”

  2. Thanks, I shall have to give due credit. That’s exactly as I would see it. Simply because global surface temperatures have not risen as fast as the mean of an ensemble of models does not provide evidence against global warming, especially as the range of many of the models is consistent with the observations.

  3. In fact, the lower of the figures in the link to Ed Hawkins’s blog is quite interesting as it shows how including the uncertainties in the measurements as well as in the model gives a much better indication of the agreement than if you consider only the running mean of the observations.

  4. Roy Morris says:

    Skeptics like to claim that the study of climate is not a science because there can be no experimental proof, as your article describes very well. But this problem exists in other sciences as well. Consider the study of human evolution for example – there is no way to test hypotheses there either. We just have to wait for more evidence to be found. I imagine there are other sciences where this is also true – can anyone come up with others?

  5. I was going to expand a little on this in my post, because I did wonder something similar myself. Sometimes you believe something to be “true” because the evidence supports your hypothesis. So, in some sense, you haven’t explicitly tested the null hypothesis. You’ve simply found evidence to support the original hypothesis. I wondered if this wasn’t an illustration of the difference between frequentist thinking and Bayesian thinking. I had a discussion about this with some colleagues yesterday and discovered that I don’t really know enough about this to really discuss it in any detail. In a sense, there is always a way to test your evidence against the null. You just have to define the null carefully and appropriately. In fact, I think evolution was given as an example by one of my colleagues. In a sense the null would be that there has been no evolution. The data is not consistent with this null and hence one can conclude that the hypothesis that evolution has occurred is correct. Having said that I do find myself getting slightly confused about this at times, so I may need to give it all a little more thought 🙂

  6. Marco says:

    And what experiment does Charlie Martin propose to falsify that hypothesis?

    Ooohh, now that could be fun to ask. Anyone here dare to poke WUWT? I don’t…(I know, I’m weak, I prefer the “safe” websites with rational hosts).

  7. Indeed, the whole modus operandi of my blog is that it allows me to essentially comment on WUWT without actually having to directly engage with the site 🙂

  8. Rachel says:

    The Ed Hawkins model estimate is not the only one. I notice that the readers at WUWT never seem to mention this one from Myles Allen which has been pretty much spot-on – http://www.guardian.co.uk/environment/2013/mar/27/climate-change-model-global-warming

    Myles Allen had a paper published in March this year – Test of a decadal climate forecast (http://www.nature.com/ngeo/journal/v6/n4/full/ngeo1788.html) – which evaluates this particular forecast. In it he finds that even if temperatures for the decade 2007 – 2016 remain no higher than for 2002 – 2001, the forecast will not be falsified at the 10% level. In order to falsify at the 10% level, temperatures will have to remain flat for the 2017 – 2026 decade (in the absence of a volcanic eruption, asteroid strike, nuclear exchange or some other short-term climate forcing).

  9. Pingback: The Desperate and the Damned: “Global Warming” as PTSD?? | Truth60

  10. Thanks, that’s really interesting. I hadn’t seen that. Should really be given a higher profile as it appears to be completely at odds with the standard argument made by those who claim that climate models have failed.

  11. Rachel says:

    I’ve mentioned it a couple of times on my blog in response to arguments made by people who do not endorse AGW. But perhaps we need to be splashing it all over internet blogs on a weekly basis, “Climate models spot-on”, just as blogs like WUWT do with their tired, old argument that the models are wrong.

  12. Yes, we probably should but maybe in a more balanced and less extreme way than would be the norm on WUWT 🙂

  13. Rachel says:

    But of course 🙂

  14. Pingback: Hot weather and climate change – a mountain from a molehill? | The GOLDEN RULE

  15. Orson Olson says:

    Actually, Rachel, WUWT already did a critique of Myles Allen’s Nature Geoscience piece that spring, as expounded in the Guardian by Duncan Clark. Author Paul Homewood makes a variety of points of varying significance. The two most important points being that one hit by computer model does not prove general significance; and that only an anomalously low ECS gets a computer model there.

    Allen’s model equates to about 1.5C degrees per century – awfully far from the object of worry like the IPCCs 3.5C degrees, or AR5s implied 2.3C degrees, or even the 2.1C degrees seen the recent suite of recent papers on ECS.

    Getting lucky is not the same as getting it right. Furthermore, I’m not sure Myes Allen deserves a pass on the question of whether or not his “guess” was bona fide or not. I’d have to read to paper to know if my suspicions are warranted or merely my imagination to know better.

    At this point, in a decade after the defrocking of the Hockey Stick temperature graph, “Trust me, we’re scientists” is no longer a sound policy where climate science is concerned.

    http://wattsupwiththat.com/2013/03/31/global-warming-predictions-prove-accurate-guardian/

  16. Orson,

    At this point, in a decade after the defrocking of the Hockey Stick temperature graph, “Trust me, we’re scientists” is no longer a sound policy where climate science is concerned.

    And climate science deniers are not really welcome here. Paul Homewood’s analysis falls into the “too silly to even criticise” category. People have to say something moderately coherent before it can actually be critiqued.

  17. Marco says:

    Let me get this straight. Orson Olson, who has not read the paper, blindly believes what a blogger says on WUWT….

    Words fail me. Oh wait, no they don’t: FAIL!

    Read the paper and you’ll find that Allen’s model actually isn’t Allen’s model, but HadCM2, which has one of the higher ECS values amongst the various models (well above 3 IIRC). It also looks to me that you took your 1.5C per century from the original forecast paper (reference 2 in Allen et al 2013), which, if true, is a horrible misreading of that figure.

    And no, I won’t go to WUWT to see what Paul Homewood has had to say.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s