Andrew Neil and the smoothed data

In case you don’t already know, last Sunday Andrew Neil interviewed Ed Davey on the Sunday Politics show. Andrew has come under quite a lot of criticism for using what many feel are typical skeptic (denier?) arguments, when conducting this interview. In particular, he showed the following graphic. What Andrew was arguing was that this graphic shows that even though CO2 concentrations have continued to rise, temperatures – since about 2000 – have not and may actually have fallen.

Temperature anomaly and CO2 changes since 1980 (credit : Met Office, BBC).

Temperature anomaly and CO2 changes since 1980 (credit : Met Office, BBC).

The first thing to say is that even if the above graphic is a correct representation of the changes in CO2 and temperature it may still not be particularly relevant. The temperature is influenced both by a long-term rising trend due to increasing atmospheric CO2 concentrations, and by natural variations, that can quite easily mask this rising trend over periods of a decade or so. However, one issue with the above graphic is that noone is quite sure what data was used to produce the temperature line. It’s clearly smoothed, but it’s not quite clear how this smoothing was done.

Andrew Neil claims that this is data from the Met Office, and seems to think that this alone is justification for his interpretation of the data. The only data that I’m aware of that is similar is their decadally smoothed data. The data value for each year is produced by averaging the data over a 21 year period (10 years before the year of interest, the year of interest, and 10 years after the year of interest). For example, the value for 1980 is produced by averaging the values from 1970 to 1990. The problem though is that once you get past 2003, there isn’t 10 years worth of data after the year of interest. What the Met office does is to simply use data up until 2013. This means that as you get closer to 2013, the average will be biased by data from the earlier time interval. This means that even if there was a continually rising trend, the decadally smoothed values for the period 2003 to 2013 would appear to show a slowdown.

I’ve produced a figure below that illustrates exactly this. I’ve produced fake temperature anomaly data, for the period 1970 to 2013, assuming a trend of 0.1oC per decade. My fake data therefore continues to rise right up until 2013. This is shown by the solid line. I then smooth this using the 21 year smoothing used by the Met Office, but obviously (as they do too) truncating the smoothing at 2013 as I have no data past this date. This is dashed line in the figure below. Even though the raw data rises with the same trend throughout, the decadally smoothed data shows a slowdown after 2003, simply due to the averaging procedure.

Illustration showing how decadal smoothing produces an artificial slowdown during the final 10 years.

Illustration showing how decadal smoothing produces an artificial slowdown during the final 10 years.

Now I should make it clear that I’m not claiming that the slowdown during the 2000s is simply due to the averaging procedure. It’s clear that looking at monthly or annually averaged data, there has been a slowdown in the last decade or so. The point I’m trying to make is that if Andrew Neil has used the Met Office’s decadally smoothed data, then this averaging procedure will produce, or enhance, a slowdown in the last decade simply because there is no data beyond 2013. The procedure will over-emphasise any difference between the temperature anomaly and CO2 concentrations and therefore should be interpreted carefully.

I get the impression that Andrew thinks that the only thing that matters is that the data is from the Met Office. No, this is not quite correct. If you’re going to use data from some source, you also need to understand how the data was produced and should put some effort into ensuring that your interpretation of the data is appropriate. You can’t simply plot some data on a graph and then pontificate about what it means. Science is a little more complicated than that.

This entry was posted in Climate change, Global warming and tagged , , , , , . Bookmark the permalink.

34 Responses to Andrew Neil and the smoothed data

  1. uknowispeaksense says:

    Colour me cynical, but given Andrew Neil’s recent twitter comments and complete lack of self awareness suggest to me that he couldn’t care less about statistical procedures or facts. He is a denier through and through and considers climate science propaganda.

  2. Yes, I probably agree but live in hope. Just left a comment on your recent post. I saw and was amazed by Andrew Neil’s comment on Twitter.

  3. uknowispeaksense says:

    As much as he might pretend to be neutral, like any denier, his true position leaks through eventually. He should just embrace his idiocy.

  4. Just use the term so-called sceptic if you are of the opinion someone isn’t a true sceptic. 😉

    I use it to distinguish between people who for whatever reason aren’t actually applying scepticism for whatever reason. That way you give someone the benefit of the doubt that for example it might just be based on a failure to be informed correctly (which can be unintentional).

    I reserve the term climate science denier for those that completely refuse to acknowledge evidence.

  5. Yes, so-called sceptic is probably about right. I’ve being trying to avoid using the term denier simply because some find it objectionable. Having said that, it is a little surprising that those who obviously deny AGW object to being called deniers 🙂

  6. Rachel says:

    I’ve decided to use the phrase climate science critic. I can’t imagine anyone would be offended by that and I think it’s an improvement on skeptic.

  7. uknowispeaksense says:

    I wrote this piece… … but lately I think climate change cynic is a better description.

    cyn·ic (snk)
    1. A person who believes all people are motivated by selfishness.
    2. A person whose outlook is scornfully and often habitually negative.

  8. Well, that’s because they link it with holocaust denier. In part because some have actually made the comparison (as how bad they see climate science denial) and because some see them linked as both use the term denier. Ignoring that a holocaust denier is a history denier, in other words holocaust deniers are revisionists. A completely different type of denial.

    Both deny something, but that is as far as the connection goes. Both terms are completely independent of each other.

  9. Paul says:

    I agree with you all. I’m thinking of writing to Neil and tell him of the experiences I had snorkeling in the Maldives. Crabs with seven eyes and fourteen claws. Turnip fish with no dorsal fin and strange four feet long tails shimmering in the moonlight. Penguins – yes penguins inhabiting the far south of one of the islands because of the rising temperatures in Antarctica. Awful, just awful.
    It makes me hopping mad to see a denier like Neil still occupying a supposedly neutral position on what is laughingly passed off as a balanced news channel. Let him see me dive into one of the pools on the north island and smash my head to smithereens because of environmental pollution caused by carbon dioxide has led to evaporation.
    Let him witness the statements of marine biologists working in fast food outlets because of the deep cuts enacted by tory fascists in the UK.
    Let him hear the trees wail and scream because of the increased wind shear virtually pulling the roots of these majestic animals (yes trees are sentient animals).
    I despair I really do.
    When I’ve finished my day shift at the rig I’m gonna get writing to him real fast.
    That’ll teach him.

  10. dana1981 says:

    I can understand Neil making the mistake. It is similar to a graph presented by the Met Office, and most people aren’t going to read the fine print about the smoothing procedure. Frankly I’d prefer that the Met Office not extend the graph to dates where it is invalid (i.e. if it’s a 21-year smooth, end the graph in 2002, don’t artificially extend it).

    The problem is that I pointed out the problem with the graph in my Guardian post, and Neil still won’t acknowledge the mistake in using it to define the ‘pause’. I don’t blame him for initially saying “it’s Met Office/Phil Jones graph”, but once the problem with it is explained, that’s no longer a valid excuse.

    He’s suggested on Twitter that he’s got a response to my post coming tomorrow. I wouldn’t be surprised if he keeps using the excuse anyway.

  11. Precisely. I have no issue with someone misusing something and then acknowledging the problem. I probably agree that it would be better if the Met Office didn’t extend the data until 2013 and simply stopped at 2003. The real issue is that Andrew Neil seems completely unwilling to listen to those who know more about this than he does.

  12. Ooh, “climate change cynic” – I think l’ll try that.

    I have been using “deniar” as in “denier + liar” – I only deploy it against the rudest and kookiest. False sceptic is too weak. I like CCC.

  13. Rachel says:

    I have another reason for disliking the word denier that is not related to the apparent association with the holocaust which is a very weak association in my view. Using the word denier implies that there’s an unassailable truth that these people are deliberately refusing to accept. I think it’s more accurate to say there’s a compelling mountain of evidence for anthropogenic global warming rather than to call it an unassailable truth. John Cook calls it a consensus denial –

  14. Yes, I do think that’s a subtlety that we should be willing to acknowledge. The scientific evidence for anthropogenic global warming is very strong but we can’t claim that it is an absolute fact. If we do use the term denier it should be clear that we mean “deny the evidence” rather than “deny a fact”.

  15. dana1981 says:

    Except those like Tol who tell him what he wants to hear, of course.

  16. Tom Curtis says:

    Rachel, IMO too much is made of the uncertainty of science. Yes it is true that even the notion that the Earth is not flat is open to revision in the face of contrary evidence. But that contrary evidence, and the theory supplanting the current theory that the Earth is an oblate spheroid would have to explain the totality of observations better than the current theory.

    In particular, it would need to explain all past observations at least as well the current theory. That means the new theory would stand to the old theory in much the way that Special Relativity stands to Newtonian Dynamics. It must, in effect, include the old theory as a limiting condition on observations at least.

    In practice, observations that would in fact force us to revise the theory that the Earth is an oblate spheroid to the theory that the Earth is flat would destroy science as an enterprise. They would demolish the presumption that the universe, and its laws are projectible, ie, the idea that we can have inductive knowledge (even if only the limited version of induction accepted by Popper).

    So, while in principle all scientific knowledge is open to revision in the face of contrary evidence; in practice some scientific knowledge is as certain as empirical truths can be. So certain that for practical purposes we can accept them as unassailable bedrock. An even larger body of scientific knowledge are unassailable bedrock at least in terms of observational consequences. We can well imagine a theory supplanting quantum mechanics, but it is all but inconceivable that the new theory will differ from quantum mechanics in its observational predictions in conditions routinely achieved in high energy physics labs, let alone at the macro scale of refrigerators and trees.

    Climate science, contrary to some claims, is not as well established as quantum mechanics (or evolution). But a very large part of it is now practically certain. No alternative theory other than that humans have caused the CO2 rise over the twentieth century could plausibly replace that theory; and no theory could plausibly replace the basic theory of the greenhouse effect except by including the later as an observational limit. When “climate science cynics” reject, or hold as dubious these claims, they are denying truths that are unassailable in all but the most theoretical sense.

    On the other hand, disputing climate sensitivity, or the harmfulness of increased temperature need not involve denial in any way. It is, however, testament to the weakness of those arguments that so man proponents of them feel it necessary to fall back on full blown denial.

  17. Rachel says:

    I’m not disputing the weight of evidence for anthropogenic global warming. I’m just mindful that if Bertrand Russell felt it was necessary to prove 1+1=2 in the 20th Century, then perhaps harboring some doubt, however minuscule, is a good thing.


    What I think critics of climate science do is apply one-sided skepticism. They indiscriminately accept everything that supports their view while applying skepticism to every shred of evidence that doesn’t. This is unsymmetrical skepticism and not really skepticism at all. Maybe that’s a better word for them: unsymmetrical skeptics.

  18. Tom Curtis says:

    Rachel, you are speaking to a philosopher. Philosopher’s take doubt very seriously. So much so that the question as to whether or not we are “brains in vats“, or whether or not Berkeleyian idealism are true are taken seriously. Unfortunately philosophers of science take this same seriousness about doubt with them, and that that is not recognized by the scientific community or its commentators. So, because we cannot prove empirically that we are not brains in vats, we also cannot prove empirically that the world is not flat.

    IMO, however, that is irrelevant to science. The probability that the world is not flat, given that the world is material and we sense it with reasonable accuracy approaches 1. And that is what science concerns itself with. That the probability of the antecedent of the conditional is undefined is irrelevant to science. Science merely assumes that we can observe the external world, and proceeds to determine what we can learn on that basis.

    You talk about harbouring some doubt, but the doubt regarding the flatness of the Earth is so small, given the basic assumption of science, that it can be entirely disregarded. The doubt about the basic physics of the greenhouse effect is so small as to be incalculable for all practical purposes. Pretending that the probability of such a fundamental error is greater than the probability of winning successive lotteries does not make reasoning more rational, but less so. But that is what we must do to pretend that Gerlich and Tscheuschner, for example, are making rational criticisms.

    Finally, for what it is worth – Russel and Whitehead did not attempt to prove that 1+1=2, but rather that the Peano Arithmetic could be proven within set theory; and indeed that they could prove all of mathematics within set theory. The question they addressed was the unity of mathematics.

  19. Rachel says:

    Hmm, I should have said asymmetrical. Sounds better.

  20. Rachel says:

    Ok sure. I’m out of my depth here. But out of curiosity, how would you describe someone like the physicist Henrik Svensmark who acknowledges the greenhouse effect but who has some sort of hypothesis involving clouds and cosmic rays to explain global warming. Would you call him a denier too?

  21. uknowispeaksense says:


  22. Interesting discussion. I think I’m also out of my depth here a little too but agree, in general, with what I think Tom is trying to say. I tried yesterday, on Twitter, to clarify a statement someone had made about the robustness of climate change. I tried to point out that they probably meant global warming, rather than climate change, and that in fact the evidence for global warming was very strong and that the theory was very robust. I pointed out the ocean heat content, direct satellite measurements of the changes in the outgoing spectrum, direct satellite measurements of the energy imbalance. Could also have pointed out laboratory experiments and detailed modelling. However, it had absolutely no effect. I was told I was hand waving and that I hadn’t presented any real evidence. So, as much as I agree that the uncertainties about global warming are low, this doesn’t mean that we can easily convince others that this is the case. I guess this doesn’t mean that we shouldn’t continue to make the case for the robustness of the theory of global warming, but those who don’t want to know will continue to mis-represent the strength of the evidence.

  23. Also, as far as I can tell Svensmark is ignoring quite a lot of evidence that suggests that his ideas don’t actually work. Some of this is quite recent (the CLOUD results from CERN) but it does seem as though he is basing his ideas on correlations and doesn’t have a physical mechanism for his ideas and all work trying to establish if such a physical mechanism exists, finds that it doesn’t.

  24. uknowispeaksense says:

    and having witnessed that exchange, there are several words I can think of to describe that particular person and his position…. recalcitrant, obnoxious, beligerant, but in the end the only word that adequately described him…..I won’t repeat here.

  25. Rachel, thanks that does seem quite apt. Yesterday’s Twitter discussion was with someone who redefined what the term “evidence” meant. The day before was with someone who redefined what a “model” was. In both cases it wasn’t clear if those who were doing the re-defining had any actual experience with either looking at evidence or using scientific models.

  26. Tom Curtis says:

    I give a fair amount of credence to Svensmark. He does not reject any part of climate science that is very solidly based such as the physics of radiative transfer SFAIK. He does tend to do science by press release. He also tends to assume that the particular area of science he is interested in has all encompassing implications. These are both flaws, IMO, but are hardly rare flaws among scientists. Further, he is obstinate in the pursuit of his hypothesis despite contrary evidence. That is not a flaw, however. It is a characteristic that some scientists must possess if old paradigms are to be challenged and replaced. It is not yet certain that his hypothesis is false, and though it is very unlikely that it is as important as he suggests, it is still plausible that the solar forcing be greater than that from TSI alone through some mechanism such as Svensmark suggests.

    What I do find disturbing about Svensmark is that he shows no recognition that evidence is stacking up against his hypothesis. It is OK to push a hypothesis against recalcitrant experience, but you have to keep proper score. Otherwise the slide into pseudo-science becomes far to easy.

  27. Al Rodger says:

    I think you are being too kind to Neil over the graph. The first crime is to suggest that the graph illustrates a 15 year plateau. While the x-axis labeling is ‘indicative’ rather than ‘precise’, if you measure it up it is impossible to see a plateau more than 9 years long. And that is being as generous as can be.

    The second crime is the smoothing process. I would suggest it is not as Neil told us, an honest smoothing out of the “ups and downs.” To get rid of the ups and downs the smoothing will cut off extremes like 1998 El Nino and, to get it as smooth as Neil’s graph, a lot more as well. By the time the “ups and downs” are smoothed away, it looks something like the graph linked here (usually two clicks to ‘download your attachment’). I do not know how Neil managed to accentuate his ‘plateau’ without leaving wobbles. It’s not using a smoothing method I know of. If it is some obscure smoothing method, why use it? Why not stick to the normal ones? And if it is not some obscure smoothing method, then the graph goes from not being entirely honest to become down-right dishonest.

    Also note the monthly data on the linked graph where the ‘slowdown’ is only evident since 2007, thus its length is exaggerated even by a correctly smoothed graph. There is often confusion between the ‘slowdown’ which is discussed by science and which is still less than a decade long, and the 16 years with no statistically significant temperature rise or the more simplistic “1998 was the hottest year” non-argument. This confusion is standard fare for the Daily Mail but we would expect Neil to research a little more widely than that. Sadly, Neil’s misquoting of Myles Allen & Peirs Forster suggest that he didn’t.

  28. Thanks for the interesting comment. I may well be being too kind to Andrew Neil. One of my many failings. I agree with you about the smoothing problem and also about the confusion between the slowdown and the statistical significance of the warming trend. Been confused or misused by many for a very long. I’ve written a number of earlier posts trying (as best as I can) to address some of this but it’s not easy to get people to recognise the difference between statistical significance and significance. It’s also a convenient argument for those who want to mislead.

  29. Yes, I agree. People who have contrarian views and try to study if alternatives are possible or likely, do play a valuable role in advancing science. As you mention, it is indeed possible that solar forcing is bigger than that from TSI alone. However, it can become damaging if such people continue to push these ideas well past the point at which the evidence is solidly against them. Again, science tends to advance anyway and such people’s views become less and less significant even if they don’t realise it themselves. The problem here, really, is that others can continue to promote these ideas to the public who typically don’t know enough to know that the evidence is firmly against it.

  30. The stance is called Just Asking Questions, or JAQing.

  31. For what it,s worth, I never had any problem with “contrarian”. My experience might be biased. Carrick, the contrarian responsible for Richard’s Twitter avatar, prefers “dissenter”.

    I too may be considered a cynic, and everyone should be critical of science, so please beware.

  32. Al Rodger says:

    It’s always good to finish what was started. So now we know where Neil’s graph came from, it’s possible to see what was and was not done.
    It was not true when Neil said in the interview that “…we (ie Sunday Politics) flattened it out a bit…” because it was already smoothed by the CRU – Neil doesn’t come clean about this error in his defense of his interview. Nor does he properly explain that he pinched a postage-stamp-sized final 33 years of the 162 year record and airbrushed out the annual data which gives a certain understanding of the interannual variation present for such a record. (And it is HadCRUT4. Neil is wrong calling it HadCRUT3 within his defense.)
    The reproduction of the line isn’t so bad given the tiny size of the orignal. I’ve superimposed the two graphs here Where Neil does appear to manipulate the illustration is in using a narrower line and in the position of the 2012 axis (which is positioned to be 2013). The original’s thicker trace made clear visually that it ended above the 2012 bar with the curve of the end overlapping past 2012. Neil extended his narrower line to the end of that overlap and then positioned his axis further out again. So Neil’s graph is actually forecasting that there will be nothing dramatic temperature-wise for the rest of 2013.

    As for the rest of his defense, it is pretty lame to say the least. Relying on just one Nature article? And one that is about a 6-year-long pause, 2 years of which were predicted? And the suppliment box he is probably trying to cite has a real whiff of Daily Mail about it. “…stalled warming … 15th year.” Quoting Ed Hawkins (who felt he had to rebut the Mail’s assertion that models didn’t predict today’s paused temperatures because the Mail’s source data was from Hawkins). And its end which balances a low ECS result from Ring et al 2012 with a quote from Alexander Otto saying there is no reason for complacency – Ring et al 2012 actually conclude with the same point but stated more strongly. “Climate scientists of course know that the large im- balance between current CO2 emissions and natural re- moval processes, and the long resident lifetime of CO2 in the atmosphere, render the “wait-and-see” approach im- possible and dangerous. Mitigation of human-caused climate change requires immediate corrective action.” So by using this Nature article as an argument for having the interview with Davey, Neil is kicking his own stool out from under himself.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.