Apparently Richard Tol has managed (after an impressive number of attempts – you can’t fault his commitment to this cause) to get his paper about the Consensus project published. I’ve discussed his attempts to discredit the consensus project in a number of previous posts (here, here, here, and here). As I understand it, there is virtually no doubt that there is a very strong consensus in the scientific literature about anthropogenic global warming (AGW). Whether it’s 97%, or 95%, or 98%, doesn’t really matter – it’s certainly strong. Anyone who thinks there isn’t is either deluded, lying, woefully misinformed, or really doesn’t understand the physics behind AGW – or some combination of the 4. Just to be clear, the existence of such a consensus doesn’t immediately mean that the science of AGW is settled – it just means there’s a great deal of agreement.
Richard Tol discusses his paper and some other analysis in a recent blog post. In his post he says,
individual raters systematically differed in their assessment of the literature. This is illustrated by this figure; the circles are aligned if the raters are the same.
The figure he mentions is one produced by Brandon Shollenberger, which I won’t include, but which you could find if you really wanted to. What Richard is suggesting is that the different raters initially produced different ratings. Let’s consider what the original paper said
Each abstract was categorized by two independent, anonymized raters. A team of 12 individuals completed 97.4% (23 061) of the ratings; an additional 12 contributed the remaining 2.6% (607). Initially, 27% of category ratings and 33% of endorsement ratings disagreed. Raters were then allowed to compare and justify or update their rating through the web system, while maintaining anonymity. Following this, 11% of category ratings and 16% of endorsement ratings disagreed; these were then resolved by a third party.
Okay, so different raters produced different ratings initially, which were later reconciled or resolved by a third party. So, it seems that Brandon and Richard are simply pointing out something that was both acknowledged by the original paper and, presumably, entirely expected. Why have each abstract rated by two people and then have a reconciliation procedure if you expected the raters’s initial ratings to agree? Remember, the goal was to rate a sample of abstracts using people who read and assessed each abstract. The raw data was the abstracts and the output was the final, reconciled ratings. Finding some – essentially entirely expected – issues with some intermediate data doesn’t tell you that the final ratings are “wrong”.
Richard concludes with
This undermines Cook’s paper. Theirs was not a survey of the literature. Rather, it was a survey of the raters.
No, Richard, Cook’s paper was a survey of the literature. What you appear to have done is indeed a survey of the raters, but that would seem to make your results largely irrelevant, with respect to the overall consensus at least – which is what the original paper was trying to assess. In my opinion, if Richard wants to criticise the consensus paper, he should really do his own analysis of the literature. I wonder why he doesn’t seem to want to do that? Also, is Richard actually suggesting that the result is wrong? That the consensus in the scientific literature with regards to AGW is not nearly as strong as Cook et al. (and others) suggest?
Richard also adds
There were only 12 raters (24 at first, but half dropped out), picked for their believe (sic) in the cause.
Ooooh, a conspiracy was it?
So, Richard (and Brandon I guess) has shown – just as the original paper pointed out – that there was some disagreement in the initial ratings. All I can really say is “So what?”. This doesn’t indicate a problem with the final ratings. Also, as I’ve already said, there is very little doubt that there is very strong agreement about AGW in the scientific literature. The interesting issue is not whether or not it exists, but how this information should be used. Personally, I think that making the existence of this consensus clear is the right thing to do. I’m aware that others think that this is a poor communication strategy as they seem to think that it might come across as an attempt to close down the debate. There may be some merit to this, but it does seem to be a suggestion that we shouldn’t tell people the truth about what’s actually in the scientific literature. Telling them that it exists doesn’t mean that they have to accept the consensus position. Telling them that it doesn’t, however, would seem to make it much easier to regard the non-mainstream ideas as being more credible than they actually are.
Disclosure : Given that I’m commenting on this, I should probably disclose that since I last wrote about this, I’ve started discussing this with some at Skeptical Science and could probably now be regarded as being associated – in some undefined way at this stage – with Skeptical Science. I should add that this is quite recent – I had no association with Skeptical Science when I started this blog or when I wrote about the consensus project before. Of course, some will hate me even more now that this is known but – guess what – I really don’t care :-)