Apparently Richard Tol has managed (after an impressive number of attempts – you can’t fault his commitment to this cause) to get his paper about the Consensus project published. I’ve discussed his attempts to discredit the consensus project in a number of previous posts (here, here, here, and here). As I understand it, there is virtually no doubt that there is a very strong consensus in the scientific literature about anthropogenic global warming (AGW). Whether it’s 97%, or 95%, or 98%, doesn’t really matter – it’s certainly strong. Anyone who thinks there isn’t is either deluded, lying, woefully misinformed, or really doesn’t understand the physics behind AGW – or some combination of the 4. Just to be clear, the existence of such a consensus doesn’t immediately mean that the science of AGW is settled – it just means there’s a great deal of agreement.
Richard Tol discusses his paper and some other analysis in a recent blog post. In his post he says,
individual raters systematically differed in their assessment of the literature. This is illustrated by this figure; the circles are aligned if the raters are the same.
The figure he mentions is one produced by Brandon Shollenberger, which I won’t include, but which you could find if you really wanted to. What Richard is suggesting is that the different raters initially produced different ratings. Let’s consider what the original paper said
Each abstract was categorized by two independent, anonymized raters. A team of 12 individuals completed 97.4% (23 061) of the ratings; an additional 12 contributed the remaining 2.6% (607). Initially, 27% of category ratings and 33% of endorsement ratings disagreed. Raters were then allowed to compare and justify or update their rating through the web system, while maintaining anonymity. Following this, 11% of category ratings and 16% of endorsement ratings disagreed; these were then resolved by a third party.
Okay, so different raters produced different ratings initially, which were later reconciled or resolved by a third party. So, it seems that Brandon and Richard are simply pointing out something that was both acknowledged by the original paper and, presumably, entirely expected. Why have each abstract rated by two people and then have a reconciliation procedure if you expected the raters’s initial ratings to agree? Remember, the goal was to rate a sample of abstracts using people who read and assessed each abstract. The raw data was the abstracts and the output was the final, reconciled ratings. Finding some – essentially entirely expected – issues with some intermediate data doesn’t tell you that the final ratings are “wrong”.
Richard concludes with
This undermines Cook’s paper. Theirs was not a survey of the literature. Rather, it was a survey of the raters.
No, Richard, Cook’s paper was a survey of the literature. What you appear to have done is indeed a survey of the raters, but that would seem to make your results largely irrelevant, with respect to the overall consensus at least – which is what the original paper was trying to assess. In my opinion, if Richard wants to criticise the consensus paper, he should really do his own analysis of the literature. I wonder why he doesn’t seem to want to do that? Also, is Richard actually suggesting that the result is wrong? That the consensus in the scientific literature with regards to AGW is not nearly as strong as Cook et al. (and others) suggest?
Richard also adds
There were only 12 raters (24 at first, but half dropped out), picked for their believe (sic) in the cause.
Ooooh, a conspiracy was it?
So, Richard (and Brandon I guess) has shown – just as the original paper pointed out – that there was some disagreement in the initial ratings. All I can really say is “So what?”. This doesn’t indicate a problem with the final ratings. Also, as I’ve already said, there is very little doubt that there is very strong agreement about AGW in the scientific literature. The interesting issue is not whether or not it exists, but how this information should be used. Personally, I think that making the existence of this consensus clear is the right thing to do. I’m aware that others think that this is a poor communication strategy as they seem to think that it might come across as an attempt to close down the debate. There may be some merit to this, but it does seem to be a suggestion that we shouldn’t tell people the truth about what’s actually in the scientific literature. Telling them that it exists doesn’t mean that they have to accept the consensus position. Telling them that it doesn’t, however, would seem to make it much easier to regard the non-mainstream ideas as being more credible than they actually are.
Disclosure : Given that I’m commenting on this, I should probably disclose that since I last wrote about this, I’ve started discussing this with some at Skeptical Science and could probably now be regarded as being associated – in some undefined way at this stage – with Skeptical Science. I should add that this is quite recent – I had no association with Skeptical Science when I started this blog or when I wrote about the consensus project before. Of course, some will hate me even more now that this is known but – guess what – I really don’t care 🙂
...and Then There's Physics 
The incessant need to nitpick about methodology is the bread and butter of deniers and clearly demonstrates their fear of engaging in rigorous science to make their point. There are only two possible reasons for not replicating the experiment. The first is laziness. The second is dishonesty. They suspect that they wont get the negative result they want so they deliberately avoid it.
My favorite part of this skirmish is that when (some) “skeptics” aren’t spending gobs of time asserting that the existence of a “consensus” is irrelevant (and that the very notion of a consensus is anti-science), they’re spending gobs of time arguing about the precise magnitude of the “consensus.”
The question remains – if it’s so irrelevant (and anti-science) why is it so important?
I love the logic of “skepticism.”
So let’s see, literature reviews cant’t exist since there can only be author reviews? Tol’s Demon is hard at work, it seems.
And then there’s physics.
Regarding the image, Shollenberger says of it that:
Even with perfectly consistent rating between raters, they will only have the same distribution of ratings if they all had an equal proportion of papers with each rating to rate. Given the low number of papers rejecting the consensus, that is very unlikely. Many raters would have had very different proportions of ratings 1, 2, 5, 6 and 7 relative to papers rated 3 and 4 just because they had a different sample. That is particularly true of raters who rated few papers. Thus pink outlier for rating four, the red and the pink outliers for rating three, and the purple outlier for rating five are probably outliers only because of a different distribution of papers rated rather than because of a tendency to rate similar papers differently than other raters.
Tol totally misses this point, misdescribing by saying:
Absent the outliers from raters with a small number of ratings, the difference in range of distribution of the raters is small. There is some evidence of systematic differences by two raters (red and orange large circles), whose number of papers rated was large enough that their distribution of 3 and 4 papers was unlikely to be greatly different from each other and the sum of other raters.
Finally, and fairly obviously, the image does not indicate whether tie breaks tended where towards the numerically higher (more conservative rating) or not. It, therefore, cannot determine whether the slight differences in ratings shown tended to bias the result in favour of, or against a finding of consensus.
Tol, is therefore, jumping to conclusions. Giving that he has already found results “damaging” to Cook et al from data that contained no information relevant to his finding (by his own admission), this does not surprise me.
It’s odd that none of the folks stirring the Cook et al data and grumbling over it seem to acknowledge the deep literature associated with survey methodology, including reconciliation of output from multiple coders, how some surveys would be fairly useless without multiple coders and how to assess confidence in coder output. It’s almost as though they don’t know what they’re talking about.
The problem at hand w/Cook et al is akin to coding results from a survey with open-ended questions. In order to come off as credible, somebody wanting to offer criticism of the work should be able to speak using terminology actually pertinent to the problems they see.
Get a google education with “intercoder agreement open-ended.”
@dbostrom
You can’t compute intercoder agreement without the coder-IDs … Cook refused to perform these tests and does not allow others access to the data. The new development at Shollenberger’s will shed light on this.
Two tests have already been done. Alternative codes for the same object have been compared (by Shub Niggurath and me), and coding patterns have been compared between coders (by Shollenberger). The results are not pretty. I expect that intercoder agreement will be small.
@wotts
These things matter. The assumption in Cook is that the coding (or rating) is done following a strict protocol and that different people would give the same code to the same abstract. If that assumption is true, the raters are incidental to the research.
However, it could be that the coding protocol wasn’t strict enough, or that people ignored the protocol. If so, the data are invalid.
A software analogue may be illuminating. A piece of code should yield the same results (within precision) when run on different machines. If not, something is wrong.
As dbostrom notes (and others have before), there are agreed statistical tests (and disputed tests too) to check whether Cook’s assumption, that coding follows protocol, cannot be rejected.
Cook has refused to publish the results of such tests; and blocked attempts by others to do so.
One thing that I find extremely odd and bothers me to no end is how the ratings by the authors of the papers are ignored. The criticism focusses only on the ratings of the abstracts done by the Cook team. Like I previously stated on this:
Even if you would find a flaw in the abstract rating results it does not undermine the consensus found via the author ratings.
@Collin
No, Cook’s complete data are not available to outsiders. For instance, we cannot compute inter-rater agreement, and we cannot compute how long it took to complete a rating.
The paper ratings indeed provide a validity test for the abstract ratings. Unfortunately, the data do not pass that test.
Richard,
How can this be true. If it were, you’d only need one rater and no reconciliation procedure.
Well, if the goal was to use abstracts to determine precisely what each paper said about AGW, you might have a point. However, the goal was to use abstracts as proxies for papers and to use those abstracts to determine a consensus. Given that there was no expectation that every paper that takes a position wrt AGW would make that position clear in the abstract, there can be no expectation that the paper ratings would directly validate the abstract ratings. However, they do produce essentially the same consensus, which confirms that abstracts can be used as a proxy to determine a consensus.
It seems to me that you don’t really understand the Cook et al. paper – or the assumptions and methodology – particularly well 🙂
Very simple question for you. Do you dispute that most papers in the scientific literature that take a position wrt AGW, endorse AGW?
@Wotts
Published papers that seek to test what caused the climate change over the last century and half, almost unanimously find that humans played a dominant role.
Funny enough, Cook et al. failed to establish the obvious.
ATTP already said most of what I wanted to say.
However, I’d like to add the following: Speed of rating is irrelevant to the results obtained. What matters is if the abstract ratings found the same result as the author ratings, which they did. Meaning that abstract ratings, as done in the Cook paper, can be used as a proxy.
Something I also said months ago, again this is what I quoted:
Tol, you have everything you need to replicate and verify the results. You current attempts at criticisms are coming awfully close what looks to me like a personal crusade against the paper.
Richard,
Then why are you doing what you’re doing? Just a publicity hound? Given that Cook et al. found that 97% endorse AGW, your final statement would appear to be completely incorrect. That, however, is unlikely to stop you from repeating it over and over again.
@Collin
Speed of ratings becomes relevant if a substantial share of ratings were completed in, say, less than a minute — particularly if done in sequence.
It regularly occurs in this kind of surveys that raters rapidly give random answers, just to finish the task. It has also happened that people program a ‘bot to reply in their stead.
Cook has the data to dispel this concern.
Richard,
That’s rather ironic. As far as I can tell, an algorithm that randomly assigned ratings to abstracts would be more likely to pass your tests than one in which people judge abstracts before giving a rating. That there is disagreement between raters would seem to hint that they weren’t assigned randomly.
I know I’ve said this before, but why don’t you come up with a more robust algorithm and redo the analysis yourself? Ohh, I know why. You don’t seem to dispute the consensus, therefore there’s no real point as you know what you’d find. Therefore this can only really be some kind of crusade against this paper that has a result you don’t dispute, but an algorithm that you don’t seem to like. Care to suggest a different – better – algorithm, rather than simply attacking this one?
@Wotts
I’m no fan of literature classification. I prefer literature review and meta-analysis, and there are excellent recent papers on both.
That said, we are investing in classification software for other purposes, so I may run a test case on this literature (although the Nottingham crowd may beat me to it).
Richard,
That seems obvious. Doesn’t seem a particularly good argument against doing so.
If you do run a test case, my guess would be that if your software were sensible, that you’d get somewhere around 97% of papers/abstracts endorsing AGW. Just a guess, mind you.
Tom,
Yes, that is fairly obvious.
Just for clarity, Brandon is pointing out that the figure was actually based on post-reconciliation data. So, maybe it’s not strictly the initial ratings, but some intermediate stage. I don’t see how that changes anything, given that the quote from the paper that I include in the post also includes that there was still disagreement between raters after they were given an opportunity to reconcile their ratings. That would seem to make the figure even more confusing as it would include some who’s ratings agreed initially, others who’ve changed their ratings so as to agree, and another group who’s ratings initially disagreed but who couldn’t then reach agreement. These then went to a third party, but of course this stage is not included in the figure (otherwise they’d all agree).
So, it would seem – in some sense – to therefore be a whole mixture of different stages. It also doesn’t really illustrate anything that wasn’t already known and acknowledged in the paper.
Pingback: Tol goes emeritus – Stoat
@Tol: Yes, it is irrelevant as it wasn’t a survey. They were categorising abstracts and there’s a lot of different variables that goes into rating that which influence duration (level of technical language used, length of abstract, etc). What makes this then even more irrelevant is that at least two people rated an abstract, and then if there’s disagreement a reconciliation stage was triggered.
But again, this seems more like you having an issue with the paper than anything else. As you agree with what they found. It’s also not alone in finding this level of consensus.
Collin,
Indeed, it is largely irrelevant. The raters were simply the tool used to rate the abstracts. Knowing that there was likely to be disagreement, there was an error correction procedure. Ultimately each abstract was given a single rating. All the intermediate stuff is largely irrelevant with respect to the final result. You could envisage this as “abstracts – algorithm – rating”. The process of rating the abstracts is simply an algorithm. That there were disagreements was expected. Showing this to be true is, in some sense, simply illustrating something that has already been mentioned in the original paper.
Actually Tom makes a point in this comment that I hadn’t quite appreciated (H/T to James Westwood on Twitter too). Unless each rater’s sample size is very large, you would expect the distribution of the ratings to differ between the raters; even if they gave the same abstract the same rating. I think one could quantify this by putting error bars onto the different rating fractions. Since the ratings are randomly distributed, I think one could probably assume that they’re Poissonion and just use Square root N.
There is a coma-inducing troll who comments on climate articles at The Conversation website and is obsessed with Cook et al. The next time he copies and pastes his extensive critique, I will point him to this comment.
@Richard Tol
“Published papers that seek to test what caused the climate change over the last century and half, almost unanimously find that humans played a dominant role.”
Said with the authority of “the 20th most-cited climate scholar in the world “. 🙂
MikeH,
Indeed, I had a quick look through the comments on the WUWT post about this topic, just to see if Richard had clarified this there. I can’t seem to find it. He also seems to still think that – for some reason – Cook et al. is a survey of the raters. Strange that, as it would seem very obvious that it isn’t. Of course, it appears that Richard’s paper is indeed a survey of the raters, but I’m unclear as to what he was hoping to achieve by doing that – “sometimes people get tired”, “sometimes different people judge things differently”, “sometimes people make mistakes”?
As ATTP said, we recognized from the outset that the volunteer raters were the instruments of this study and were imperfect. These instruments all came from different manufacturers and were inconsistently calibrated. We did our best to correct for this source of error and, with more time and resources, and with different protocols and statistical analysis tools, we could probably do a better job. My own, rather crude, analysis of the results leads me to conclude that the consensus, as determined by the analysis of abstracts we did, gives a consensus figure in the range 96-98%.
The lower bound is controlled by the scarcity of published papers that reject the consensus; there just aren’t enough of them in any sample of the literature that isn’t cherry-picked. The upper bound is limited by a fact of human diversity that there are always a few contrarians in any field who are motivated and educated enough to go against the grain. This is usually salutary, but it can be damaging when the contrarian outliers are given undue prominence in public discussions on policy alternatives.
No doubt, using different sampling techniques and protocols, perhaps even adopting some of the suggestions made by Richard Tol, new studies could produce a more accurate measure of the scientific consensus on anthropogenic climate change. I would read any such new study with interest, but I’m not about to invest any of my own resources in conducting one. Our results are sufficiently consistent with other measures of consensus (including our own survey of the authors of the original papers) for me to consider the matter settled, for all practical purposes. Most importantly, as John Cook and others have stressed, measures of scientific consensus differ very widely from public perceptions of that consensus, even among those who are the most zealous proponents of action on climate change. That’s a much more interesting problem.
Another side of a barn for Bob Ward to hit, it would seem.
–> “Cook has the data to dispel this concern.”
Has anyone else niticed that Richard is concerned about a great many things? Quite frankly, I am concerned about his level of concern. Why is Richard so concerned?
Joshua,
All we need is for Willard to thank him.
Here you go Richard. Something else for you to grasp. http://www.otc-frederick.com/wp-content/uploads/2012/03/Drinking-straws.jpg
Pingback: Tol goes emeritus [Stoat] | Gaia Gazette
As Eli mentioned in 2013, Brian had organized a similar rating back in 2007, for which Eli has the ratings and the cross-tabs. The one essential difference was that we went through a preliminary training exercise with a small number of abstracts to get everyone on the same page. See here and here
The results showed, that there were only 13 disagreements amongst 650 abstract. Tol is full of it
jsam
A nasty tale of intimidation by a GWPF trustee against Bob Ward is emerging. Alongside the interesting news that the GWPF is going to hive off a non-charitable bit of itself to disseminate misinformation to the public about climate change now the Charities Commission is getting beady about abuse of charitable status by this front group.
Bob Ward has named – on Twitter – the ‘distinguished Oxford scientist’. Anyone not on Twitter want to guess who it apparently is?
LOL! That’s very funny! He’s as much an Oxford scientist as I am i.e. NOT 🙂
Well, yes, especially as he apparently has no actual association with Oxford – or, at least, none that I’m aware of.
Do tell, ATTP. I hate guessing. So undignified.
BBD, it’s Nic Lewis.
ATTP+Rachel, note the following:
“the trustee said the LSE should be aware that a “distinguished Oxford scientist” had told him: “It’s appalling that the LSE employs people like Bob Ward.” ”
The trustee = Nic Lewis, and Nic Lewis thus claimed a distinguished Oxford scientist had told him “It’s appalling that the LSE employs people like Bob Ward.”
Nic Lewis is thus not the distinguished Oxford scientist, he’s ‘just’ the conduit for what this supposed distinguished Oxford scientist had said.
Hmm…now I am confused. Maybe that trustee wasn’t Nic Lewis, and the trustee couldn’t distinguish his Oxford from his Cambridge?
Ah. Thanks Marco. What a bizarre misrepresentation of Nic Lewis’s affiliations. I hope he will insist on a public correction.
Marco
Nic Lewis is *not* a trustee of the GWPF.
Sorry – skipped the link to the GWPF Board of Trustees.
Marco,
That’s what I’d initially thought, but Bob Ward’s tweet seems pretty straightforward. I don’t know if he’s right of course, but I think he claims to have seen the letter in which the ‘distinguished oxford scientist’ is named.
Of course, even if he has confused Oxford and Cambridge, you wouldn’t normally describe someone who’s simply graduated from a university as still being associated with that university. Odd, but nothing surprises me these days 🙂
I wonder who the trustee was? It’s unfortunate that Bob Ward felt that he could not identify the individual publicly, but I am sure he had good reasons for not doing so.
Back on the Cook paper. Simple question.
Why isn’t all of the data available, raw data plus methods of analysis.
If there is no problem then, er, there will be no problem.
Job done.
End of debate.
Genuine question, not trolling.
Morph,
A simple answer would be that it all is. The raw data is the abstracts. The can be easily extracted from the Web of Science database using the same search as in the paper – although put double inverted commas around the terms. Alternatively you can find them on the Consensus Project webpages. The method of analysis is explained in the paper. So, redoing the analysis is trivial but time consuming. In my opinion, this would be the correct way to check the results in Cook et al. – do your own analysis.
I suspect what you mean is why isn’t all of the Cook et al. ratings data available. Again, most of it is. What is missing (I think) is the information about who specifically rated which abstract. I don’t know why this isn’t available but presumably it has something to do with analysing individuals.
Here’s the fundamental point, in my view. Most – maybe all – of what Tol has done is show that there were disagreements between the raters. This is stated in the paper. That’s why there was a reconciliation procedure and a way of resolving tie-breaks. Essentially Tol is illustrating what’s already acknowledged in the paper. If you want to show that this invalidates the whole paper, you’ve got to do more than simply show that these disagreements exist (that’s obvious from reading the paper) you’re going to have to also show that the reconciliation and tie-break procedures till didn’t produce a result that was representative of the actual consensus.
Morph – all the data are available, except confidential bits like author self-ratings. We even created a website where people can attempt to replicate our results. We could not be more transparent.
http://www.skepticalscience.com/tcp.php?t=rate_papers
I have to say I find Tol’s psychology fascinating. He doesn’t dispute our result, and yet has engaged in a year-long crusade to prove that our methods were somehow flawed, to the extent that he’s submitted 5 versions of his paper to 4 different journals until finding someone willing to publish his tosh (or so he claims – I’ll believe it when I see it). The only explanation I can come up with is that he craves the denier worship that will come with having published a paper supposedly finding something wrong with the 97% consensus study. Which is rather pathetically sad.
Brandon and Richard make for strange bedfellows. https://hiizuru.wordpress.com/2014/05/06/richard-tols-many-mistakes-prove-him-right/
If you are a merchant of doubt, creating the appearance of controversy is the only option when actual controversy is absent.
What BBD said above.
Dana:
Why do you keep repeat this falsehood?
You know better. Do you assume everybody else is just dumb?
There is no need for tol to show the paper is wrong. His task is to find out if smth can be criticised about the method, so others can say the Cook paper is totally wrong. Isn’t that clear? The GWPF likes it like that.
Fascinating. I wonder how many “skeptics,” that are so focused on Cook et. al like Richard, accept that:
–> “Published papers that seek to test what caused the climate change over the last century and half, almost unanimously find that humans played a dominant role. ”
The climate war bickering over Cook et. al is psychology textbook material.
Joshua,
We may have seen nothing yet:
https://twitter.com/nevaudit/status/465605932354125824
The conversation is interesting. Here’s how it started:
How Brandon used Rachel to get noticed by AT seems noteworthy.
Carrick –
–> ” Why do you keep repeat this falsehood?”
I know that you might think I’m wacky for beleiving this, but my guess is that he thinks that you are mistaken , and that it isn’t a falsehood. Since you seem so genuinely curious maybe you could put on your thinking cap and try to figure out why he might see the situation differently than you. You’re a smart guy, and i know that you would find such a view concerning, but if you try, maybe you could figure it out?
Willard –
Nobody argues from incredulity quite as well as Brandon.
I might say he’s incredibly incredulous.
Joshua, the data that includes individual ratings along with timestamp information has not been released, and Dana knows this. I’m perfectly happy with calling a version of the dataset that anonymizes the evaluators a “full data set”, but when Dana say “all the data are available” he knows better.
Carrick, when you quote Dana as saying “all the data are available” without including his qualification you misquote him. You are happy for the evaluators to be anonymized – ie, data that could indicate their identity being excluded from the full data set? Well that is just what Cook et al have done, and all data except that which could be used to identify abstract raters, or self rating authors has been made public.
Tom Curtis, the qualification “except confidential bits like author self-ratings” have nothing to do with the information that is missing, that prevents the data set that has been released from being regarded as “all of the data”. Including that qualification does not help him.
Including a timestamp for each individual rating with anonymized ids in no way reveals the identify of the evaluator.
Richard Tol: “When you find yourself in a hole and want to get out, it’s best to stop digging.”
https://twitter.com/nevaudit/status/465621597832101888
I think what Brandon is asking for, actually, is if there are ethical considerations that precludes the release of this data. Were somebody were to point out an ethical reason for not disclosing the portion of the data that Brandon has that has not been released by Cook et al (in spite of their obviously false claims to the otherwise). I am sure that Brandon would withhold the release of this data.
I think it is appropriate to not reveal the identifies of the evaluators, without their explicit authorization. I will make an argument of this sort to Brandon on his blog, but it is his decision to make. However, I do not view Brandon giving Cook a chance to argue the case for not releasing the data that Cook and Nuccitelli claim has been released to be blackmail.
> I think what Brandon is asking for, actually, is if there are ethical considerations that precludes the release of this data.
Brandon only mentioned a reason:
There is no reason to limit ourselves to ethical considerations alone.
***
Besides, that comment was prefaced by an answer to Shub’s question Where did you get the raters’ id data from?:
***
Also note that the title of the post is called A tease.
Carrick –
Your response has no meaning to me because I have not followed this food fight in any detail. I don’t follow it in any detail because I find the particulars of the food fight to be trivial. It seems to me that both sides are arguing about trivialities. I do find it interesting, however, that people become so fanatical that they spend time arguing about trivialities.
I think that Richard’s statement from above, the one I quoted, is what could be of some (limited) meaning amid all the inanity (I say limited because the existence of a consensus is meaningful but not dispositive) – but even that, IMO is of limited meaning because even though what Richard said should be abundantly obvious to anyone that is paying attention – the reality of what he said (that there is a consensus) has no meaningful impact.
Why is the methodology used by Cook et. al of so much interest to so many people? If it is valid, will it change the public’s opinions on climate change mitigation policies? If it is invalid, will it change the public’s opinion on climate change mitigation policies? If it is valid, will it change the reality of an overwhelming consensus, as Richard spoke of? If it is invalid, will it change the reality of an overwhelming consensus, as Richard spoke of?
So the food fight continues. People call each other names. People are concerned. People are incredulous. People hack. People don’t hack, they only find stuff. People don’t know why otters repeat falsehoods. People “know better.” People employ binary logic. People think they’re victims. People argue about time stamps. Practically no one bothers to listen and practically no one tries to fucking communicate.
Carrick:
“Tom Curtis, the qualification “except confidential bits like author self-ratings” have nothing to do with the information that is missing, that prevents the data set that has been released from being regarded as “all of the data”. Including that qualification does not help him.”
True. It probabliy doesn’t contain information on the rater’s choice of browser or operating system, either, the cads.
Richard’s description of what happened when he and the thousand-breasted goat attempted to do a run pretty much illustrates what I was talking about.
> Practically no one bothers to listen and practically no one tries to […] communicate.
I blame Gremlins.
Carrick, you have all the raw data to reproduce the results in the paper. Why exactly do you need individual ratings and time stamps? Think about your reasoning carefully, Carrick, because the argumentation I have seen so far gets dangerously close to MbW. That is, you are looking for anything that can be used to cast doubt on the end result, and you will continue to argue some crucial information is not provided.
@Dana
Results derived from incorrect methods are, by definition, incorrect.
Did you catch Brandon’s logo?
http://www.urbandictionary.com/define.php?term=Stand+Alone+Complex
Ghost In The Shell: The Stand Alone Complex;
http://en.wikipedia.org/wiki/Laughing_Man_%28Ghost_in_the_Shell%29
http://en.wikipedia.org/wiki/Philosophy_of_Ghost_in_the_Shell
Richard,
AFAICT, there is no evidence to suggest that the method is incorrect. All you’ve done is show that there is inter-rater disagreement, just as discussed in the original paper.
Richard and The Goat should broaden their scope and bring down the entire shibboleth of survey research dependent on coding with imperfect agreement.
A bottomless well of twittering tweeting is available simply by repeating the incantation against countless publications!
Hark! A Revolution is at hand, led in part by a tweeting goat!* Who’d have believed the entire edifice of social science could be collapsed by intensively bursts of 180 characters or less?
Truly comedic.
*Also a guy who apparently has somehow obtained data copied without permission, but let’s not share the limelight too broadly.
He’s spreading FUD (Fear Uncertainty Doubt).
While not proving anything or measuring anything, he’s casting baseless aspersions. Its standard technique really. Its most of what the denial community does.
Its used by people who have no substantial argument.
Carrick, anonymized ids would still allow a determination of how many papers each rater rated. That with information from the SkS forum hack would allow positive identification of raters, including those who asked to be anonymous. Frankly, I do not know how much additional information about raters could be gleaned from the combined sources, and neither do you. Your claims, therefore, are completely spurious.
Further, as to what ethical consideration should lead to Shollenberger not releasing the data – the data is stolen. That is a very simple ethical consideration that seems beyond Shollenberger’s, and apparently your comprehension. Once you simply ignore that fact, any further claim Shollenberger might have to be acting ethically is laughable.
“Privacy concerns” were cited as a reason not to publish in the Recursive Fury manufactured controversy – where citing comments made on public forums shouldn’t be used in papers. Now, “privacy concerns” are not held valid by the same incredulous pseudo-sceptics in Cook et al.
Concerns are very odd things.
As Richard points out, “Results derived from incorrect methods are, by definition, incorrect”. Bob Ward helped him reach that epiphany. 🙂
jsam, ask Tol how comparable the impact estimates of x degree warming are and whether they have all used “correct” methods. Then ask him what ‘economical’ mechanism supports the use of a quadratic equation to fit these impact estimates. I think the answers may surprise you.
I don’t know about jsam, but there’s very little that surprises me these days.
Here’s a funny John Oliver clip on scientific consensus and the bogus, balanced debate. There’s even an image of Cook et al 2013.
For the “Every cloud has a silver lining.” file.
Tol.s obsession with Cook et al reduces his ability to screw up elsewhere.
Richard points out, “Results derived from incorrect methods are, by definition, incorrect”
jsam adds: Bob Ward helped him reach that epiphany.
There are, of course, multiple answers to this. First, that of Dickie, incorrect data are not incorrect methods. Second, that of Eli, the confusion between incorrect and imperfect is the tool of those seeking to confuse. Box’s extended rule is that the results of imperfect methods can be useful. Given that all methods describing the real world are imperfect, this is useful.
Of curse, the Tols, McIntyres, Carricks, Brandons and Lucias of the world, amongst others, would never believe this except for their themselves, and thus waste everyone else’s time and patience.
Why so much attention to claims from someone whose credibility has just been completely blown to bits?
Eli Rabett: “Of curse…” ???
Steve Bloon: Because we have low tolerance levels for people who put themselves on pedestals..
> Results derived from incorrect methods are, by definition, incorrect.
Which definition of correctness do you have in mind, Richard?
AnOilMan
Interesting. Our BS clearly fancies himself as a cyberninja. And he’s making blackmail threats involving the release of hacked, stolen data.
Talk about outing yourself. Perhaps he wants to be arrested and charged? Could all this be a misunderstood cry for help?
BBD,
Just to be clear, judging by what I’ve seen of Willard’s Twitter exchanges, Brandon does not regard this as blackmail, but simply as a request that the owners of the data express a view as to whether or not he can release it. Given that, the obvious response would seem to be “No, you can’t release it. It’s not yours to release. Give it back.”
ATTP
The obvious response would be: you are in possession of stolen information by definition acquired illegally. Return it, now, or face the legal consequences.
BBD,
Indeed, but one could at least start politely, even if that isn’t the norm 🙂
No need for politeness with blackmailers and thieves, ATTP. A proportionate response is mandated, at least in business. Can’t speak for academia.
BBD,
To be fair, from what I’ve seen of some academics involved in the climate change debate, politeness is indeed quite rare.
> Return it, now, or face the legal consequences.
That would be incorrect (TM Richard Tol), for then Brandon knows that John would be willing to consider pressing charges. The most accurate response is the Glomar response, i.e. “We neither confirm or deny that we will or won’t press charges or else until you release the private information you illegally hold. If you could tell us what security breach was exploited in our CMS, that would be nice too.”
For a good story about the Glomar response:
http://www.radiolab.org/story/confirm-nor-deny/
***
If you read Brandon’s offer as literalists would, any reason would trigger non-release. John should consider trying “because 2+2 = 4”. Unless Brandon meant a reason that he himself finds good enough, that ought to suffice. Trusting Brandon’s ethical judgement might not be justified.
The most economical response would be to accept that possibility that Brandon releases information. He already did anyway, as a teaser. Paying attention to the fact that Brandon coerces John to discuss seems more interesting than the “conversation” Brandon wishes to have with John.
BBD: BS inhabits a pile of poppycock, not academia.
If Tol receives any public funding to conduct his research at Sussex University, his communciations with BS might be obtainable vi an FOIA request. Perhaps one of you Brits should explore this option.
I’d have thought the obvious was not that methods (or even results) are imperfect – but that whatever way you try and measure the consensus (and it’s been done many times now) it’s always 97% give or take. Always.
Tolls & Sholls and others who can’t see the woods for the trees might discount many independent, corroborating witness accounts from a crime scene because each had imperfect handwriting or grammar or whatever – and rather miss the bigger picture about epistemology and how we know stuff.
Still, if it keeps them busy…
Jason,
Indeed. The 97% does appear to be a rather robust result.
John,
I can’t really see the point of FOI’s, to be honest. I gather that they do get used in circumstances like this (and I have some recollection that Tol may have tried this himself) but it typically comes across as a form of harassment. Plus, it seems that much of the communication about this has been quite public, so it’s not clear that much would really be learned that wasn’t already known.
> Practically no one bothers to listen and practically no one tries to […] communicate.
I blame society.
Actually, the self-assessment research ethics questionnaire that you can access from the School of Business, Management and Economics seems relevant.
ATTP: In my opinion, filing a single targeted FOIA request is not harassment . On the other hand, bombarding someone with a multitude of FOIA requests would be.
Re the information to be obtained – Given the size of BS’s ego, he may have detailed exactly how he accessed the SkS depository of data.
BTW, my professional career spanned more than 30 years toiling in the public sector. I was on the receiving end of FOIA requests. I did not consider them to be “harassment”. If an academic is expending public funds, he/she should be subject to the same laws/rules as a bureaucrat.
To the best of my knowledge, academicians are not a protected species.
John,
Yes, in general I agree. Certainly academics should not be protected from such requests. My point was more related to whether or not the gain would be worth the claims of harassment that could possibly end up appearing in places like the Daily Mail.
Be sure to check out Sou’s HotWhopper Blog Post, “Anthony Watts’ bombshell goes pear-shaped. 82% of WUWT-ers aren’t interested!” It’s a hoot!
http://blog.hotwhopper.com/2014/05/anthony-watts-bombshell-goes-pear.html
ATTP: If we start worrying about what might appear in the Daily Mail, we might as well fold up our tnets and go home. [I’m in a take no prisoners mode today.]
John,
In a sense I agree : it would be nice if you could just do what you felt was right without having to second-guess what sort of games others might play.
Yes, I’d got that impression 🙂
To send an FOIA would “keep busy” (thanks, Jason!) over something there is little point to focus on unless you’re a contrarian auditor. Since Richard has yet to provide constructive criticisms to improve further research, there’s not much else to do than thank him for his concerns. Nits are, by definition, nits.
John Hartz
Oh, I know. I was making slight reference to our host’s background.
Incidentally, do you know the etymology of “poppycock”? If not, do look it up when you get a minute – it might amuse you in the rather fitting context of “BS”. Consciously or unconsciously, you used just the right term 😉
There ought to be ways to provide constructive criticisms to Richard that would help future surveys. For instance, there is this sentence in Richard’s seventh draft:
https://docs.google.com/file/d/0Bz17rNCpfuDNRllTUWlzb0ZJSm8/edit
The concept of “being silent” certainly deserves due diligence. We surely can say that what is silent is, by definition, silent. But that does not help much. What is meant by “being silent” is a bit unclear, although a bit earlier Richard uses the expression “take a position”.
Let’s assume there are 12k papers. To falsify Richard’s claim, it should be enough to take 70 abstracts and show how they take a position on AGW. That’s not the only method to do so, but it’s the most direct.
This method has also the added benefit that there’s no need to randomize the process. All it takes is after all about 70 abstracts. A small site promoting this collection is all it takes to falsify one of Richard’s claim.
There may be a simpler way than that, but this one seems easier and more profitable than writing an FOIA.
2iiard: The primary target of an FOIA would be what BS said, not what Richard Tol said.
BBD: I am very familiar with the entymology of the word “poppycock.” i frequesntly use the word when sparing with climate denier drones on the comment threads of MSM websites. In fact, i may have coined the term, “pseudo-science poppycock.”
Willard,
Well I did that (see the bottom of this post). I rated 133 abstracts in the Cook et al. sample (randomly selected) and found 50 that took a position (endorse), 83 took no position, and after rating 133 had still not found an abstract that implicitly or explicitly rejected AGW.
I have a suspicion, though, that Richard is slightly playing with words. He may well be referring only to those papers that test the hypothesis, rather than to those that endorse the hypothesis. Given that the Cook et al. survey was specifically about the level of endorsement in the literature, if this is what Richard is suggesting, it would – again – seem irrelevant to this particular issue.
Willard: There are multiple ways for people to communicate constructive suggestions to Tol. Recebnt history suggests that he will promptly deep-six all suggestions. Re Cook et al, Tol has embraced the “I’ve made up my mind. Don’t confuse me with the facts” shool of thought. Tol has a visceral hatred of Cook et al and nothing is likely to cuse him to let that hatred go. .
From the University of Sussex website (and undoubtedly written by Tol):
Biography
Richard S.J. Tol is a Professor at the Department of Economics, University of Sussex and the Professor of the Economics of Climate Change, Institute for Environmental Studies and Department of Spatial Economics, Vrije Universiteit, Amsterdam, the Netherlands. Formerly, he was a Research Professor at the Economic and Social Research Institute, Dublin, the Michael Otto Professor of Sustainability and Global Change at Hamburg University and an Adjunct Professor, Department of Engineering and Public Policy, Carnegie Mellon University, Pittsburgh, PA, USA. He has had visiting appointments at the Canadian Centre for Climate Research, University of Victoria, British Colombia, at the Centre for Social and Economic Research on the Global Environment, University College London, and at the Princeton Environmental Institute and the Department of Economics, Princeton University. He received an M.Sc. in econometrics (1992) and a Ph.D. in economics (1997) from the Vrije Universiteit Amsterdam. He is ranked among the top 100 economists in the world, and has over 200 publications in learned journals (with 100+ co-authors), 3 books, 5 major reports, 37 book chapters, and many minor publications. He specialises in the economics of energy, environment, and climate, and is interested in integrated assessment modelling. He is an editor for Energy Economics, and an associate editor of economics the e-journal. He is advisor and referee of national and international policy and research. He is an author (contributing, lead, principal and convening) of Working Groups I, II and III of the Intergovernmental Panel on Climate Change, shared winner of the Nobel Peace Prize for 2007; an author and editor of the UNEP Handbook on Methods for Climate Change Impact Assessment and Adaptation Strategies; a GTAP Research Fellow; and a member of the Academia Europaea. He is actively involved in the European Climate Forum, the European Forum on Integrated Environmental Assessment, and the Energy Modeling Forum.
Role
Professor of Economics; PhD convenor; Director of Research and Knowledge Exchange
badgersouth,
You don’t have access to Richard’s mind. There may be too much hair in the way for you to tell. That or Gremlins. As someone whose name is in Richard’s acknowledgements, I can attest that sometimes he considers criticisms.
The point is not that Richard acknowledges them anyway. Whatever Richard does with constructive criticisms, what obtains is something constructive. Constructive criticisms are, by definition, constructive criticisms.
I don’t consider venting as constructive. Venting is, by definition, venting.
Tol’s nominal boss at the University of Sussex – per the University’s web site:
Biography
Carol Alexander is Professor of Finance and Head of the Business and Management Department. She is also co-editor-in-Chief of the Journal of Banking and Finance, with Ike Mathur. She was Chair of Financial Risk Management at the ICMA Centre in the Henley Business School at Reading (1999 – 2012) and lecturer in Mathematics and Economics at the University of Sussex (1985 – 1998). She holds degrees from the University of Sussex (BSc First Class, Mathematics with Experimental Psychology; PhD Algebraic Number Theory) and the London School of Economics (MSc Econometrics and Mathematical Economics). She also has an Honorary Professorship at the Academy of Economic Studies in Bucharest, Romania.
Carol has held the following positions in financial institutions: Fixed Income Trader at UBS/Phillips and Drew (UK); Academic Director of Algorithmics (Canada); Director of Nikko Global Holdings and Head of Market Risk Modeling (UK); Risk Research Advisor, SAS (USA). She also acts as an expert witness and consultant in financial modelling. From 2010 – 2012 Carol was Chair of the Board of PRMIA (Professional Risk Manager’s International Association).
She publishes widely on a broad range of topics, including: volatility theory; option pricing and hedging; trading volatility; hedging with futures; alternative investments; random orthogonal matrix simulation; game theory and real options. She has written and edited numerous books in mathematics and finance and published extensively in top-ranked international journals. Her four-volume textbook on Market Risk Analysis (Wileys, 2008) is the definitive guide to the subject.
Role
Professor of Finance, Head of Business and Management Department
Community and Business
Carol frequently consults for banks, asset managers, exchanges, pension funds and other financial institutions. She acts as an expert witness, designs software (some models are patented) and undertakes model validation work.
Carol was the founding chair of the Academic Advisory panel for the Professional Risk Manager’s International Association (PRMIA). She also editied, with Elizabeth Sheedy of Macquarie University, the 3-volume Professional Risk Manager’s Handbook which has been adopted by the major financial institutions in over 100 countries for training their risk management staff.
She has a large and growing network of contacts with finance professionals around the globe, most recently founding the Quantitative Finance International Network (QFIN), with Academic colleagues at Universities in Germany, Norway, Spain, South Africa, Australia and Canada.
For further details of Carol’s netwroking and industry-engagement activities see http://www.carolalexander.org
Willard: I am entitled to my opinions about Tol and you are entitled to yours. I’ll bet you a dollar to donut that he never acknowldges that he was wrong about Cook et al.
It’s made Watts? Tol is doomed. Weatherboy’s intellectual compass unfailingly points to intellectual south.
Tom Curtis:
Actually my claims are not spurious and you have the problem exactly backwards. Leaving aside whether there is even an ethical obligation to post facto protect the identifies of the evaluators, there is no way that releasing the anonymized data by itself could reveal the identities of the evaluators.
It is because the evaluators willingly identified themselves and their participation to third parties that their identifies became known. I agree that the SkS hack revealed this to more third party individuals than they anticipated, but they de-anonymized themselves. That is a fact.
As a researcher, I am under an obligation to protect personal data that I have gathered, but that obligation does not extend to carelessness on the part of the participants in the study. This is the balancing that always happens in any research. You understand as a participant that I will be providing certain information that I have collected from you. If I provide anonymous information that cannot be used to identify you, that limits the extent of my obligation to protecting you.
If you make statements to third parties, without a prior signed non-disclosure agreement (I think we can accept that no such agreements were made here) that deanonymizes you…then you need to be more careful next time you participate in a research project.
Actually no, it was publicly available via the tcp interface provided by Cook, as Brandon has explained.
Indeed, badgersouth, you are entitled to vent your unsubstantiated opinion, and I am entitled to substantiate my opinion through evidence and constructive criticisms. For instance, Richard Tol already conceded that he got something wrong about Cook & alii. This was in a previous thread:
Counterfactual thinking about what Richard would do may not be the best way to provide constructive criticism. Nor is thread bombing irrelevant bits about Richard. I would certainly be entitled to some opinion about your behavior were you to insist in using cheap and dirty tricks like these.
Also note that if you are the “John Hartz” above, perhaps you ought to decide which handle you’ll take to identify yourself. You can’t plead ignorance twice.
Carrick,
Maybe you can answer a serious question. What would be the point of analysing this data? The paper already makes clear that there were inter-rater disagreements. Showing this to be true doesn’t – as far as I can tell – tell us anything about whether or not the consensus result is robust or not. The method is clearly explained. The inter-rater disagreement is clearly explained. The reconciliation method is clearly explained. That there was still a requirement for tie breaks is also explained. If you want to check the results, redo some analysis. Use the same method. Come up with your own method. So, what is the point of analysing this data?
As for whether or not the data is stolen, I don’t know how Brandon got it. It’s very obvious that it wasn’t intended to be public. I think one can make a perfectly plausible argument that it is indeed stolen. I don’t know the relevant legal system particularly well, but it’s certainly my opinion that it’s perfectly reasonable to regard this data as stolen.
> [T]here is no way that releasing the anonymized data by itself could reveal the identities of the evaluators.
Carrick’s right: by itself, anonymized data is, by definition, just anonymized data. The problem is when we consider that people may process these data on the light of other information that is already public knowledge. Then dots start to be connected, and anonymity disappears.
Engineer-minded auditors should leave “by itself” to old, cranky, or post-modern philosophers. Even I try to leave it to them.
Marco:
Tol has discussed many of these.
• To see if there is a bias in individual evaluators.
• To measure the reliability of the rating system.
etc.
In other words, it is needed to evaluate the quality of the data in the study.
It is normal to provide raw data in studies like this. It is frankly bizarre that it was not automatically produced.
Andres, what is the purpose of ever releasing raw data?
To test the validity of the analysis of the paper for any paper.
willard:
Due eventually to behavior on the part of the evaluators that “blew their own cover”. Regardless of whether Cook’s site had been hacked, had they chosen to remain truly anonymous, that anonymity could not be broken by any of information that was hacked from Cook’s site.
Willard: I have never suggested that anything i have posted on this thread is meant to be “constructive criticism,” Like most commenters, I’m brainstorming. If that offends you, so be it.
I am both John Hartz and Badgersouth. I intitlaly signed into this thread via Facebook – which is probably why my intital post identified me as “John Hartz” In the Word Press system, my user name is Badgersouth.
PS – Do you ever get off of your high horse of self-proclaimed superiority?
Carrick,
You haven’t really answered my question. The raw data is the abstracts. The raters were simply the algorithm that was used to score the abstracts. The paper acknowledges inter-rater disagreement. I fail to see how showing that there was indeed inter-rater disagreement in any way illustrates that the final ratings (after reconciliation and tie-breaks) is a reasonable representation of the raw data (the abstracts). Remember, every abstract ends up with 1 rating. So, even if some of the raters had biases, that still doesn’t allow one to infer that the final rating is not a reasonable representation of that abstract.
As far as I’m concerned, Tol is trying to turn this into a survey of the raters, when the original paper was a survey of the abstracts. I don’t see how you can criticise the Cook et al. result – which was a survey of abstracts – by performing a survey of the raters. They’re two different things.
Carrick,
Also, re-defining what the raw data is in this context is also somewhat odd.
Eli, of course I realize that all papers contain flaws. It is a point I make often. For example I made this point recently on Lucia’s blog:
One of the tried and true ways of testing for whether warts matter is to allow other people to analyze your raw data.
Here is a comment that I made on Brandon’s blog that I think summarizes my attitude:
To which Richard Tol replied:
It was completely possible to protect the identities of the evaluators and the original authors evaluations while allowing their results to be evaluated by third parties.
If we care whether results are true or not, we should allow them to be tested, even by people who we think are “just trying to find things wrong with them”.
Carrick\Tol\Laughing Man:
The correct solution is not to cast aspersions on the original study.
The correct solution is to redo the study, and preferably include any extra data which is considered desirable.
The results would speak for themselves.
Blogging to the public, clearly has a different purpose than doing real work for a living.
badgersouth,
I have no reason to take offense in anything you say, including what you say in your post scriptum. In fact, I don’t give a damn. As you say, you are entitled to your opinion. If Richard Tol thinks the same way I do about ad homs, I don’t think your venting will affect him much.
Replying to a commentary that was published in the lichurchur with venting only shows that you are prone to venting. If that suits you, fine. I never said or even implied that you considered venting constructive criticism, so I have no idea why you’re saying this.
Brainstorming works when people bring something useful on the table. Venting brings nothing. I’ve suggested an alternative to sending an FOIA to Essex, an alternative that AT already started, to boot.
Thank you for your concerns about my person,
w
Anders (apologizes for the typo):
I’ve answered it with the specificity needed—you can’t test the validity of the analysis without the raw data. Is it really your view that we should just trust that Cook (or any other researcher) has done it correctly, without any examination of their work?
Anyway..I’d suggest reading Tol’s paper with an open mind. I think he covers it pretty well for this particular case.
You arguing that the abstract are the raw data is like claiming that a glass beaker that I was heating was data.
\The data in that case are the temperature measurements (for example), just as with Cook the data are the evaluations of the abstracts.
The abstracts are the objects being studied. The data are the measurements obtained using those objects.
This has to do with the difference between the objects of the study (the abstracts), the instruments (the evaluators) and the measurements (their evaluations).
> Due eventually to behavior on the part of the evaluators that “blew their own cover”. Regardless of whether Cook’s site had been hacked, had they chosen to remain truly anonymous, that anonymity could not be broken by any of information that was hacked from Cook’s site.
The reality is that Cook’s site has been hacked and that evaluators have not behaved like the true Scotsmen Carrick invokes for his counterfactual proposition to hold.
We’re not discussing a thought experiment.
Carrick,
No, it’s my view that good research should be reproducible. Cook et al. very clearly is. If you want to check their result, redo the study. Do you want me to send you all the numbers that my computer generates in going from some input to some output? If you do, I hope you have a large amount of storage space.
Really? When you put some chemicals in a beaker are you testing the beaker or the chemicals? Are you really suggesting that the abstracts aren’t the raw data? That the Cook et al. study was really a study to see how well people could rate abstracts, rather than a survey of the abstracts. If so, maybe you should try actually reading the paper.
> the instruments (the evaluators)
This seems to conflate the instruments with the observers. The instruments may rather be the scoring system, the site that helped implements it, etc. Even the intersubjective training could be part of the instrumentation.
AnOilMan:
Critical review of other people’s publications is a standard part of the self-correcting nature of the scientific process.
It is an approach, but not the only one, an neither a particularly good one nor one that serves the interest of science.
If I care about the validity of my research (which I do), I would always make the data available to third parties for their review (which I also do), even when there is a danger that they will analyze the data incorrectly (which does happen).
This is really silly. The raw data (the list of abstracts to be rated) has been available since it was published, individual raters will have some bias (duh!) which is why Cook et al implemented multiple raters and reconciliation to minimize that bias, and there’s no determinatino of whether the biases were for or against the concensus without independently evaluating the raw data (again, the abstracts). Given the Cook et al paper self-ratings (not to mention the Anderegg, Doran, and Oreskes papers) the abstract ratings appear to have underestimated the consensus.
Despite the ease of independently evaluating the raw data – SkS has a rating page available to try random abstracts from their list if you’re really lazy – Tol has spent the last year or so working on his “destructive comment”. And in the process has proposed and abandoned a number of avenues of attack. That seems like motivated reasoning, or in Recursive Fury terms “Must Be Wrong”, to me. He certainly concluded that the paper must be wrong prior to having any data whatsoever to look at, which argues that he didn’t have a scientific criticism to start with.
Detailed information on the individual ratings would only be raw data if the paper was on the psychology of ratings – and it’s not. This entire focus on niggly bits – when Cook et al stated that there would be some biases, and took steps to minimize them – gives (IMO) the impression of an attempted ad hominem fallacious degnigration of Cook et al 2013. In other words, rhetoric, not science.
It’s just sad.
Yup, it really is.
willard, as always there is a limit to the utility of any conversation between us. Perhaps if we sat down over a beer someday, we could understand each other’s communications styles well enough to allow for something more than talking past each other.
Regardless of your malarky about Scotsmen and their proclivities, there is no doubt that the evaluators deanonymized themselves. Cook did not do this, and the hacker(s) did not do this, the evaluators did it by their own actions.
KR:
Again, this is not data.
These represent the objects of the study.
The instruments are the evaluators.
The measurements (data) are the evaluations.
Willard: Are you done venting at me?
willard:
I would view the scoring system as the instrumental model and the evaluator as the instrument.
In analogy with temperature of a beaker, the thermometer is the instrument, the assumption of linear expansion with temperature is the instrumental model that relates the heat energy stored in the beaker to the desired measurement quantity.
Carrick,
Whatever you want to define the different stages of the process (and, FWIW I disagree with you – the abstracts, IMO, are the raw data) you – and almost anyone else – had all the information needed to reproduce/test the Cook et al. result. Again, I do not see how showing what is already known – that there was inter-rater disagreement – has any significance with respect to the Cook et al. result. Simply pointing this out over and over again, does make it true.
Here’s a question for you. Do you dispute the Cook et al. result. Do you think that the result does not properly represent the level of endorsement of AGW in the literature?
AOM hits the nail on the head. Just do better science. What the spewdo-sceptics might find really embarrassing is that the new number might well be higher than 97%, as per http://www.jamespowell.org. Is that why they daren’t; for fear of a Muller Moment?
Are there teacups small enough to withstand this tempest?
For the record: As long as Richard Tol associates himself with the GWPF, I shall hold him in contempt. His rants and raves about Cook et al are notjhing more than a distraction from the serious damage he is doing though the GWPF. .
And by the way, if we properly view the abstracts as the objects of the study, that reveals why it is important when we take a sample of a population (as unwittingly happened with Cook’s study), why we must take care to take an unbiased sample, or we must have an approach to correct for the sample bias.
What do you mean if? How can it be otherwise?
What do you mean unwittingly? Care to explain how a database search using well-defined search terms produces a bias? Have you actually read the paper? These comments make me think that you really haven’t.
Anders:
How can you not view the abstracts as samples of the population of AGW related papers? If they are samples of the population to be studied, they aren’t the data (the measurements) they are the objects of study. The Cook study was about the classification of the abstracts, not a proof of their existence, so how can the abstracts be data rather than samples of a population?
You’re asking for my speculation here—well I suspect the “weak AGW” consensus is well over 99% of authors of peer reviewed papers. By “well-over” think in terms of log norm or z-score.
Willard: As Confucius was wont to say,
“Man who pats himself on the back all day will sooner or later tear his rotator cuff.”
Carrick,
This truly is tedious. Here’s how I see it. There is a large sample of papers in the literature. I go to a database. I type in a search term. It returns me a sample of papers – data. I now want to determine what fraction of these papers that endorse AGW. I decide to analyse the abstracts. I develop an algorithm that will convert my raw data (abstracts) into a final rating. That procedure could be highly variable. It could be computerised. It could involve people. How I go from my abstracts to my final rating is, in my opinion, an algorithm. That’s why asking for this data is not unlike asking me to provide every single number my computer generates in going from some kind of input to some kind of output. On the other hand, I don’t really care. Arguing about what is or isn’t the raw data is remarkably irritating.
Then why are we having this discussion? Are you playing the “it doesn’t matter if the result it right if the method is wrong” game. As far as I’m concerned, if it wasn’t for people like yourself and Tol (and many others) we wouldn’t need to do these blasted consensus projects in the first place. It’s obvious that a large consensus exists. The only reason these studies take place is because some dispute the existence of such a consensus, and others pedantically criticise the method whenever one is carried out. If you think the result is right, why don’t you simply acknowledge that (Tol too) and we could make a start at no longer needing such studies. On the other hand, if you actually think the result is wrong, do a proper study of your own.
Anders:
I’m glad you agree here. but if they are the objects of the study, then they are not the measurements. Think beaker versus thermometer versus temperature.
Oh, I’ve read the paper, and obviously a lot more carefully than you apparently have. Again, perhaps reading Tol’s paper with an open mind will clear some of this up for you, but they used peculiar search phrases, like “global climate change” which resulted in a biased sample of the population.
You’ll pardon me I hope if I don’t rehash things that have been well covered elsewhere.
Anders:
No, these are samples of a population. Like samples from a glacier that I wish to take CO2 measurements on.,
Because research ethics matter to me.l
It puzzles me that apparently intelligent observers fail to understand that this is a fake controversy that has been deliberately created because no real controversy exists and merchants of doubt require controversy to peddle their wares.
Carrick,
Likewise.
Really? This is about research ethics? Seriously? You want to analyse the Cook et al. study in more detail because you think there might be ethical issues? Do you want to give this a little more thought? You’re seriously suggesting that they may have behaved unethically? I’m actually lost for words (well, not really, but I’m still trying to keep the discussion civil).
Do you maybe want to consider the irony of suggesting that there are ethical issues with the Cook et al. study in the comments on a blog post motivated by someone accessing data that was not public and that they were obviously not meant to have access to. Do you have any idea how unbelievable what you’ve just said actually is.
I have read the most recent available version and have actually redone some of the analysis. Yes, there is inter-rater disagreement. So what? We know. The paper tells us there is.
Threads like this are the cost of keeping the discussion civil. Fortunately we have Eli.
Carrick: I’m not sure what direction you folks wish this to go. Essentially, you don’t want to do work for a living, and you wish to stand around and say you don’t like what you see.
May I suggest a good art gallery to do that in?
With no data, I really don’t care what you think. Its nothing personal, just there’s no facts to back your concerns. The best approach is never to say “no”, its to perform the study again with any and all improvements you claim are a serious concern. Then you can exactly compare your results and see what a difference it made.
If you walked around a business saying ‘no’ like this, you’d be fired, sidelined, or otherwise removed. At best, people would probably ignore what you said unless it became a pretty obvious issue. However, if you wished to press your point of view, you’d need to show data.
This is because people walking around complaining without facts or numbers are considered to be ass hats by all reasonable people who see them.
Carrick, you make an implicit claim in your response to me that you will need to substantiate: that it is normal to provide that type of specific raw data in studies like this.
I also very much doubt that the information requested will provide any answers to the questions asked, in particular because the person who wants the data had already decided the study was wrong and with incorrect methods and all that before asking for that data. MbW all over.
Did anyone ever stop to think that some of these activities by Tol and company might just Trolling on a bigger scale? You know, stir it up in 100 places rather than just one. Whip it up into a frenzy… Toss in Brandon, self style Super Class A Hacker (Ghost in the Shell term) poised to push this monster conspiracy into the lime light.
When Richard Tol says “Results derived from incorrect methods are, by definition, incorrect.” its really just a big button push absent any data to back it up.
He’s yanking chains. I bet he like doing it too.
AOM.
Yes, it’s certainly crossed my mind. I never know whether it’s best to ignore this kind of stuff or not.
Anders, do the math.
By talking about Tol you are inviting his style of behavior, and lending credence to his otherwise insignificant position.
Don’t feed the Tol.
It’s certainly true that the confecters of fake controversy and peddlers of doubt would have a harder time of it were they shut out of the public discourse.
* * *
AOM observes:
Yup.
We should start a pool on how many comments will be posted on this thread before all is said and done.
Well, I plan to go for a run before I say anything more 🙂
Anders:
Why not?
And presumably because the right answer matters, otherwise what’s the point of the research?
Marco:
Well I always do provide the raw data. After all, if I care about the truth, the more eyes that see it the better.
What if we could refine the 97% number and convert it to 99+%? Is that of value?
Carrick,
I really must go for my run, but if you don’t get my incredulity at what you said, further discussion is probably pointless. I now have 3 trigger phrases : “honest broker”, “I discuss policy without advocating”, “it’s because I care about research ethics”.
Anders, I think we’re through where this conversation can productively take us to, so no more from me on this thread.
> [A]s always there is a limit to the utility of any conversation between us.
As far as I’m concerned, Carrick, the term “conversation between us” ceased to be relevant a while ago:
I’m responding to your claim, or rather, in our two actual cases, how you divide to conquer.
In the first case, what the raters did won’t change Brandon’s responsibility in holding, teasing his audience with, and releasing in full data that is not his. In the second case, an instrument can’t in any meaningful way be analyzed outside its conceptual apparatus and its overall implementation. (And that’s notwithstanding the fact that raters are not measuring instruments, that ratings are not measurements but evaluations, and that the object under study can both be the consensus on the endorsement of AGW in the literature or the abstracts and the papers, depending on how you conceive an object of study [1].) The wedge here serves to say that data is only what you analyze to test formal properties, which goes against just about any kind of empiricism.
In other words, what Brandon does has consequences and what AT says makes sense.
[1] There’s also an interesting distinction between technical objects and epistemic things, e.g.
Click to access TestObjects.pdf
It seems very clear to me that the motivation of Toll and others in wanting all the information that they can from Cook et al is to find something, anything, which hints that the study is unethical in some way. Point the finger of failed ethics at the study, points it at the Skeptical Science website, those associated with it and their activities outside the actual published paper. If the interest of Toll and others is a genuine concern that the published literature has been poorly sampled and/or analyzed by Cook et al then performing their own analysis is the skeptical way to proceed. That is never going to happen for obvious reasons. Toll and others have NO interest, absolutely NONE, in the actual approach used in the study or its findings Their only interest is finding some tidbit with which to paint a larger canvass of malfeasance within the scientific community dedicated to climate research. The pseudoskeptics tried it with Climategate, they will keep on trying. Its their only recourse because the science is so convincing.
Carrick – The raw data are the abstracts, the measurements are the statistical evaluation of consensus agreement. Individual ratings are known to be subject to some bias; Cook et al not only acknowledged that but went to some lengths to minimize it (multiple ratings and reconciliations). Inter-rater bias variations would be of interest if and only if Cook et al made topically significant material claims about confidence intervals, which they did not – reasonable given that there are many options for citation search terms, for abstract databases, and for the explicitly discussed inherent differences between raters. Cook et al stated the issue and took steps to minimize it – if you or Tol are so bloody concerned, repeat the study and see if you can improve upon the methods.
This entire tempest in a teapot would only be relevant if potential uncontrolled biases (once past the reconciliation procedures) were sufficiently large to materially change the conclusions of over 97% consensus. Yet despite having all the data to check those conclusions by looking at some abstracts, nobody (including Tol with his “destructive comment”) has bothered to check, and I suspect that’s because they either cannot (in which case those objecting have insufficient training for complaints), or are well aware of the results they would get. Surely you know enough people with “skeptical” concerns to help identify more than 10 papers out of more than 12,000 in the Cook et al study which explicitly reject AGW – if they actually exist?
Just as there aren’t any paleotemperature reconstructions from McIntyre. Or published works from Watts regarding microsite influences. Or for that matter any hypotheses consistent with the evidence that could account for recent climate change without AGW. What I’m seeing from Tol (and others) is IMO a great deal of substanceless Sturm und Drang intended to induce doubt without, you know, pesky accompanying facts.
My apologies, Watts _has_ published on microsite influences. And in Fall et al 2011 found that _they didn’t matter_: “…the overall mean temperature trends are nearly identical across site classifications.” His only work seems to disprove his favorite hypothesis.
Which has never stopped him from repeatedly claiming (against his own research conclusions) that microsite influences _do_ affect mean temperature records, along with conspiracy theories regarding deceptive adjustments. “Do I contradict myself? Very well, then I contradict myself, I am large, I contain multitudes.” – Walt Whitman
Again, a very serious lack, from any “skeptic”, of research or data that actually supports their varied fantastical hypotheses – work that actually survives examination.
–> ” Because research ethics matter to me. ”
Alls I can say is thank god someone around here cares about research ethics. Well, I mean in addition to Richard. And of course, Brandon. And I would imagine a few others that agree with Carrick about the risks associated with ACO2 emissions.
I mean it’s not like this food fight has anything to do with views about the risks of ACO2. It’s all about the ethics.
It’s about concern about the ethics.
Ever notice how only those who have a certain belief about ACO2 emissions care about research ethics?
‘Prolly just coincidence.
I wonder what “ethical” concerns Carrick has in mind that might be revealed by releasing the rater’s data?
I also found this interesting tidbit from Brandon’s blog:
This thread is in danger of devolvling into yet another discussion about the age old question, “How many angels fit on he head of a pin?”
Meanwhile, back on planet Earth…
Scientists Warn of Rising Oceans as Antarctic Ice Melts by Just n Gillis and Kenneth Chang, New York Times, May 12, 2014
Carrick, Tol and Schollenberger are rent seekers. Tol has to deal with his own demons. Schollenberger may have gone too far this time. Carrick with his baseless accusations of unethical conduct is on the verge.
Eli Rabbett: Well said!
All I can do is echo what afeman has already said
> Fortunately we have Eli.
Don’t forget Bill Nye:
The carrot and the bow tie.
Willard,
Yes, that is excellent.
> a different test could have resulted in a different pattern,
STOP ALL THE PRESSES!
http://memegenerator.net/instance/49742920
Dana Nuccitelli writes, “Morph – all the data are available, except confidential bits like author self-ratings. We even created a website where people can attempt to replicate our results. We could not be more transparent.”
I’m curious…your paper combined the first three bins:
“(1) Explicit endorsement with quantification.”
“(2) Explicit endorsement without quantification.”
“(3) Implicit endorsement.”
How many abstracts were in each of those 3 bins?
Also, your paper mentioned bins 4a and 4b:
“4a) No position: Does not address or mention the cause of global warming.”
“4b) Uncertain: Expresses position that human’s role on recent global warming is uncertain/undefined.”
How many abstracts were in 4a versus 4b?
I think talking about this paper linking weather to Climate Change would be far more interesting than feeding the Tol.
http://www.desmogblog.com/2014/05/10/new-study-links-california-drought-climate-change-and-burning-fossil-fuels
Carrick, Tol and Schollenberger are rent seekers.
I prefer to refer to them as violin players. Fiddlers. Really bad ones.
Accompanied by deck furniture rearrangers.
Mark,
If you go to the Consensus Project site you can download all the data. I know I worked out the number of 1, 2, 3 but can’t quite remember the exact number. For 4a and 4b I believe (Dana can correct me if I’m wrong) there was a reanalysis of 1000 category 4 papers, of which – I think – 5 were rated as being uncertain. Since there were almost 8000 category 4 abstracts, this implied around 40 were probably uncertain.
> I’m curious…
Me too, Dana. Now, would you please scratch my itch?
Beforehand, if you could bring me a cup of coffee, that would be nice.
Willard: Despite our differences of opinion, i really do enjoy your dry humor.
Thank you, Badger. Here’s another one, just for you:
[Anonybunny #2] All economists are wrong, but some are useless.
[Me] All economists are wrong, but some are hairless.
http://rabett.blogspot.com/2014/05/richard-tol-stakes-himself-on-hill.html
willard: Speaking of econom,ists, Paul Krugman rocks!
See his oped, Crazy Climate Economics New York Times, May 11, 2014
Willard: Case solved. I’m using two computers to access this comment thread. On my desk top, I accessed through Word Press (Badgersouth). On my lap top, I accessed tnhrough Facebook (John Hartz).
Peace!
Nice nickname, John!
You can thank Rachel, now 😉
Carrick’s claim that a researcher is not responsible for the release of sufficient information to identify an anonymous participant by releasing “anonymized” information which together with information in the public domain reveals the identity of the person is simply nonsense. It makes a mockery of the idea of keeping data anonymous. To take an example, it would indicate that Lewandowski, if he had an obligation to keep the identities of the subjects of Recursive Fury anonymous (which I deny), “anonymizing” the data by merely referring to subject #1, #2, etc, while publishing the quotes in unedited form and including the links in the references would have satisfied the obligation. Such, at best, proforma satisfaction of obligations is the sign of a scoundrel.
Equally ridiculous is claim that discussion in a restricted private forum, participation in which certainly required a confidentiality agreement (even if unsigned) represents deanonymizing of participants. Such discussion can only represent agreement that your identity be known to the legitimate participants in that forum. It in no way represents agreement that forum members are permitted (in contravention to their agreement on joining) to broadcast that information further. It certainly does not represent agreement for random hackers of the forum to place those comments in the public domain.
As it happens, the forum was illegally hacked. The confidential information in the forum was then unethically distributed by so-called “skeptics”. That places that information in the public Domain, and John Cook needs to consider what inferences about identities can be made by release of anonymized data together with the information from the forum before releasing “anonymized” data. In that consideration he needs to err on the side of caution (ie, of not releasing identitifying information).
As it stands, Carrick places himself in the position of claiming that two illegal acts by others deprives the raters of their right to anonymity. At the same time, he has no word of condemnation for the illegal acts, nor the unethical (and probably illegal) unauthorized distribution of information stolen in the acts. That clearly show how distorted his ethical judgement is.
Hi ATTP,
OK, I found the numbers for categories 1 to 3. I couldn’t find the numbers for 4a versus 4b, and I’m still hoping Dana will provide them, since they aren’t in the paper.
Here are the results for categories 1 to 3:
(1) Explicit endorsement with quantification = 64 abstracts.
(2) Explicit endorsement without quantification = 922 abstracts.
(3) Implicit endorsement = 2910 abstracts.
And 4 to 6:
(5) Implicit rejection = 54 abstracts.
(6) Explicit rejection without quantification = 15 abstracts.
(7) Explicit rejection with quantification = 9 abstracts.
So the total number of abstracts in categories 1 to 3 plus 5 to 7 is 3974.
Therefore, an accurate characterization of the results of the paper would be:
“97% of climate papers papers stating a position on global warming agree global warming is happening–and 1.6% agree we are the primary cause.”
Right?
Cripes Bahner, how many times does Eli have to tell you, it’s all the Honest Broker’s fault.
> Right?
No:
That was more than one year ago, Mark.
> That was more than one year ago, Mark.
Hmmm, that was less than a year ago.
Here’s where we may be going:
Another quote from the same thread:
I do hope we don’t get that same discussion over again, Mark. Last time, it went for over a month.
Willard: The denier set does love to regurgitate — over, and over, and over….
Ok, then:
“98% of climate papers papers stating a position on global warming agree global warming is happening–and 1.6% agree we are the primary cause.”
No, Mark:
http://xkcd.com/169/
Thank you for your concerns.
Let’s put it Bart V’s way:
You just can’t infer that “1.6% agree we are the primary cause,” Mark, since you’re projecting a quantity in qualitative categories.
Wouldn’t be surprised if only 1% of astrophysics papers explicitly agree that gravity in the weak field approximation varies as the inverse square of the distance. Would be surprised if more than a few percent disputed this obvious fact.
It also quantifies it; we know that 16% of endorsement ratings were resolved by a tie-break. If we assume that a) all post-reconciliation disagreement was between categories 3 and 4, b) every single tie-break went to category 3, and c) every single one should have gone to category 4, the much-vaunted 97% consensus collapses to a miserable…er, 94%.
Clearly a serious issue well worth investigating.
Suppose that Cook et al correctly classified all of the rejection papers into categories 5-7. (I haven’t heard anyone complaining that they were misclassified as neutral or accepting so that’s a reasonable assumption.) That’s 78 according to Mark’s figures above. We’ll add to that category the estimated 40 papers that were classified as category 4 but which may have actually expressed uncertainty as to the cause rather than taking no position at all.
Suppose further that Cook et al’s raters were so biased that half of the papers they classified as endorsing the consensus actually did not, and they should have been rated as category 4. (Yet, remarkably, nobody seems to have noticed that roughly half the papers they independently check in categories 1-3 were incorrectly classified as endorsing the consensus.) That’s 3,896/2 = 1,948 papers that should have been classified as endorsing the consensus in this hypothetical scenario.
Even with this ludicrous exaggeration of potential bias, the headline figure would still be an overwhelming 94%.
No matter how you slice and dice it, there is no amount of bias in the paper ratings that is both plausible and makes a meaningful difference to the headline. Given the fact that the ratings they arrived at for each and every abstract are publicly available and I’m yet to see anyone show that they are wrong in the first place, this would have to be one of the most pointless exercises in “auditing” that I’ve seen yet.
The fact that they’re so desperate to manufacture controversy over a paper that even Tol agrees is stating the bleeding obvious just goes to show how important it actually is when it comes to influencing public opinion, which is all that really matters for the “skeptics”.
Many people have wondered why I worry about fatigue.
One of Cook’s raters did 283 ratings in a single day. That is 2 minutes per abstract for 10 hours.
JasonB, that would be why they don’t want to actually ‘do’ the analysis themselves. It’s far more important to cast aspersions on anything (?), rather than do any real work.
Act like that on the job, and you’d be fired.
LOL!!!!! Funniest thing I’ve read in ages. Good one.
ATTP: correct about 4a and 4b.
Richard,
So what?
Also, does this mean that you now have access to the data that identifies the raters?
Given the number of forums Richard posts to I am concerned with his fatigue.
MartinM,
I’m not even sure that’s right. If I reduce the endorse fraction by 16%, the consensus collapses to 96.5%. Well worth all this effort.
Stop the presses: some people work hard! If all this time spent whining and hacking into private websites had been spent designing an “acceptable” algorithm and rating abstracts using it, they’d have completed a useful and independent verification. Several times over by now.
But that would be hard work. Maybe even 10 hours a day! And published papers that seek to test what caused the climate change over the last century and half, almost unanimously find that humans played a dominant role. So don’t, under any circumstances, actually bother to measure that percentage. Never.
If I were the kind of person who’d be interested in making destructive comments I’d point out that the unidirectional gremlins also got to the computation Tol’s recent comment 😉
Well said DS.
Richard:
I think many people wonder why this “worry” doesn’t evolve into a desire to actually check the results rather than remain an oft-repeated “worry”.
The obvious conclusion is that it’s because it’s the only way to maintain the “concern”.
If you’d found any actual evidence of mis-ratings then I’m sure you would have produced it by now. “Worry” is the fall-back position.
Richard,
Can you show us your test data showing any mistakes to justify your concern? Or are you just going with your gut feelings on this?
http://en.wikipedia.org/wiki/Words_per_minute
“The average adult reads prose text at 250 to 300 words per minute.”
It seems the first step is to identify any misclassified papers, and then perhaps try to determine why it was misclassified. That is the scientific method. What you’re doing sounds backwards, you’ve identified a concern, and the cause without looking at the data.
Just so you know, my father was a speed reader. He’d read 6 novels a night, and could recall them perfectly the next day. So to put that in perspective at 200 page books, that’s 1200 pages, and an abstract is about half a page, so.. he could polish off 2400 abstracts an evening. (It became an expensive problem so he had to unlearn it, and start reading at normal human speeds.)
The reviewers you mention seem pretty average. Just because a few dedicated souls decided to knuckle down, doesn’t seem out of place to me. The opposite side of that coin are the guys who quit.
I rated 133 abstracts and I can’t quite remember how long it take, but it was certainly well under 10 hours. Of course, I was just doing it to satisfy my curiosity, not for a publication, but even so, a few minutes per abstract is all you really need.
@AnOilMan
Cook reports an error rate of 19% in the abstract ratings. Paper ratings and abstract ratings deviate in 63% of cases.
Richard,
And given that papers and abstracts are not the same things, the relevance of your comment is…….
Misdirection, pure and simple.
@Wotts
You either agree with Cook that abstracts are proxies for papers and that paper ratings can be compared to abstract ratings (in which case Cook failed his validation test) or you argue against Cook that his attempt to validation was flawed.
Richard,
Where does Cook et al. claim that the abstracts are proxies for the papers? The abstracts were used as proxies in order to estimate the consensus in the literature, not in order to determine the endorsement – or not – of an individual paper. I’m surprised you think that the abstracts were intended as proxies for the papers. Have you actually read Cook et al.?
What would you call the combined intellectual power of Watts, LaFramboise and Tol?
Anyone can do physics. Anyone can do economics. No one knows. No one.
The GWPF sullies all who touch it.
Gor the record…
It’s a well known fact that certain members of the all-volunteer SkS author team are cyberborgs. Dana and Tom Curtis are the two most advanced models.
Me? I’m the resident “Old Fart”.
Richard Tol: How are you and the Tolettes coming along with your analysis of how long it took for members of the all-volunteer SkS author team to track down the correct email address of the key authors of the papers selected for review in the TCP process?
John Hartz, I am certainly not a cyborg, nor anymore part of the SkS team for that matter. But I can certainly testify to Dana’s cyborg like productivity which is truly amazing.
Tom Curtis, From where I sit, you will always be a member of the SkS team. The quantitiy and quality of your posts on the comment threads of articles posted on SkS is truly amazing.
Notice how Richard ignores JasonB’s comments.
For the record…
I did not volunteer to rate the Abstracts of papers during the TCP process. My contribution was limited to tracking down the email addresses lead and key authors. The ferreting process was indeed very tedious, especially for papers that had been published a few years prior. Scientists are a rather mobile group of people, especially during the first few years after getting their degree.
Duly noted.
Meanwhile, back on planet Earth…
Sentinel satellite spies ice cap speed-up by John Amos, BBC News, May 8, 2014
Ice-loss moves the Earth 250 miles down Science Daily, May 11, 2014
JasonB –
==> “The fact that they’re so desperate to manufacture controversy over a paper that even Tol agrees is stating the bleeding obvious just goes to show how important it actually is when it comes to influencing public opinion, which is all that really matters for the “skeptics”.”
I happen to disagree. I think that actually, the validity of this paper (and the quantifying in some precise manner the % of consensus) has relatively limited effect on public opinion.
The fact that “skeptics” are alternately obsessed with arguing that the existence of a consensus view is irrelevant to evaluating science and spending gobs of time arguing about the precise quantification of the consensus does not therefore tell us anything about what influences public opinion. Remember, “skeptics” do much that is emotionally based, and not logical.
That does not mean, however, that I disagree that they’re trying to manufacture controversy about what even Tol agrees is bleeding obvious. It’s quite fascinating how self-described “skeptics” would not begin to ask themselves why they are much more focused on trivialities than they are on what’s bleeding obvious. Seems to me, that an honest-to-god skeptic would ask him/herself that question.
Joshua,
Your last paragraph pretty much sums it up, in my opinion.
Pingback: More pear-shaped trouble for John Cook’s ’97% consensus’ | Watts Up With That?
Joshua: The preliminary results of a new research project being conducted by the SkS team indicates that less than 3% of the pseudo-skeptic population can be classified as “honest-to-god skeptics”,
Out of curiosity, have any “honest-to-god skeptics” posted comments on this thread?
Given the most of us are starting to repeat ourselvs on this thread, I suggest that we help ATTP out by brainstorming sub-titles for this OP. To get the ball rolling, I’ll run this one up the flagpole.
For whom the bell Tols.
Are you Donne yet?
“You just can’t infer that ‘1.6% agree we are the primary cause,’ Mark, since you’re projecting a quantity in qualitative categories.”
So you think “98% of climate papers stating a position on global warming agree global warming is happening–and 1.6% agree we are the primary cause” is not an accurate characterization of the paper’s findings.
But you think, “97% of climate papers stating a position on human-caused global warming agree global warming is happening–and we are the cause”…is an accurate characterization of the paper’s findings?
Richard Tol,
What I’m hearing from you is that you have not actually analyzed the data, and that you have no clue as to what introduced any errors if any. (i.e. One guy could be going through a messy divorce, or was using far too much cocaine that day.) You have performed no actual research into the field in question, and have not even run any sort of trials to determine for yourself what may or may not being going on. (Classification of abstracts could be a regular reading comprehension issue.)
You are going with gut feelings that maybe the reviewers were tired.
I’m not seeing a whole lot of skill here.
Can anyone else confirm this, but didn’t the reviewers discuss all papers that had different classifications by different reviewers in order to correct for errors in the initial classification?
Joshua:
I think that remains to be seen, but from what I’ve seen it seems to have had a big impact on the media, where I see the 97% figure now being mentioned over and over again, and the absurdity of “false balance” now being openly ridiculed e.g. in the John Oliver clip above. In fact, a very cynical and jaded me has been somewhat surprised at how quickly it has gained traction and how ineffective “skeptics” attempts to discredit it have been outside of their own little world.
If the surveys suggesting that the general public believes scientists are still evenly divided on the issue are correct, and if the media continues down this path of emphasising the amount of agreement within the scientific community, then I think the “skeptics” are right to be worried about the impact of this paper. As they have stated themselves, they need to convince the public that the issue is not settled in order to prevent action — hence the clutching at straws.
Mark:
Of course — because any papers that tried to suggest something other than man was the cause, or tried to minimise man’s impact, or explicitly stated that man was responsible for less than half, were categorised as rejecting that statement.
You can’t read the endorsement categories in isolation — you have to take into account the effect of the rejection categories as well. The statement is defined as much by what was classified as rejecting it as it was by what was classified as endorsing it.
I’m surprised at the number of papers in the BMJ that do not explicitly support the germ theory. There must be a lot of “germ sceptics” amongst doctors. Who knew?
Richard’s discussion about abstracts being “proxies” for papers explains part of where he’s going wrong, I think. It’s similar to his objection about the papers that were climate-related but that failed to be analysed due to having the wrong keywords.
The point of the exercise was to assess the level of endorsement of a particular statement in “the scientific literature”.
Since examining all of “the scientific literature” would be impractical, and much of it irrelevant, Cook et al decided to examine a sample instead, in the same way that opinion pollsters use a subset of the entire population to estimate what the entire population’s opinion would be were they to ask them all. Provided the sampling technique did not introduce a bias, there is nothing wrong with this, and the larger the sample, the smaller the error margins.
The first step was to define a subset of all possible journals to search. The second was to use keywords to narrow the set down still further. The third was to examine only abstracts rather than obtain and read entire papers. Each of these decisions made the problem more tractable, and the last increased the number of papers that could be assessed with a given amount of effort (and money!). They were also easy to apply. The fourth was to then examine each of the resulting abstracts manually to see whether they were truly relevant or not, and, if so, what category they belonged to. (Anyone objecting to Category 4 papers not being included in the calculation should ask themselves what makes those papers so special? They simply managed to slip through the first two filters designed to remove irrelevant papers; they were still irrelevant. If you want to include them in the denominator then why not go the whole hog and include every paper ever written in the denominator? That will make the percentages really small.)
It doesn’t matter if a large number of abstracts where classified as “4. No Position” when the papers themselves actually did take a position, provided it did not introduce a bias — in other words, unless endorsement papers were more or less likely to make their endorsement clear in the abstract than rejection papers, the estimate should be the same within the error margins.
Likewise, it doesn’t matter if not every paper in every journal that ever took a position on the cause of AGW wasn’t considered for exactly the same reason that it doesn’t matter if opinion pollsters failed to call me to ask me what I thought about some topic — unless, again, rejection papers were more or less likely than endorsement papers to use those particular keywords, or publish in those particular journals.
Has anyone demonstrated any evidence that any of those steps introduced a bias?
Joshua
‘I happen to disagree. I think that actually, the validity of this paper (and the quantifying in some precise manner the % of consensus) has relatively limited effect on public opinion.”
What if judith said
“I happen to disagree. I think that actually, the validity of this paper (and the quantifying in some precise manner the % of consensus) has relatively big effect on public opinion.”
of course she has said similar things about climategate and you typically repsond asking for evidence.
The point of this episode in the climate debate is that Cook’s paper is junk.
Junk data.
Junk methods
Junk transparency.
That said I think the conclusions are largely correct. The vast majority of papers on climate science accept that C02 is a GHG and that it causes warming.
However, if you do content analysis there are certain fundamentals and yes picky details that
you have to get correct. Cook’s paper does not get these details correct. That’s sad.
What is more sad is the lengths folks go to to defend his crap rather than doing it over
correctly. With all the documentation, all the safeguards in place to detect and correct for rater bias, and a effort that can withstand the attacks of nitpickers. Since the planet is at stake shoddy work and defending shoddy work is simply unacceptable.
But, but… all the FUDders have done is insinuate. They haven’t “done it over correctly”.
So why not address your complaints to them? Or do it yourself?
Since so much is at stake, hypocrisy and false claims are simply unacceptable.
So why do it, Steve?
I wrote, “But you think, ‘97% of climate papers stating a position on human-caused global warming agree global warming is happening–and we are the cause’…is an accurate characterization of the paper’s findings?”
JasonB responded, “Of course — because any papers that tried to suggest something other than man was the cause, or tried to minimise man’s impact, or explicitly stated that man was responsible for less than half, were categorised as rejecting that statement.”
Remarkable.
So let’s say they reviewed 100 abstracts, and 97 came out into category (1) Explicit endorsement with quantification. And 3 abstracts came into category (7) Explicit rejection with quantification. And no abstracts came into any of the other categories.
Would an accurate characterization of that result be: “97% of climate papers stating a position on human-caused global warming agree global warming is happening–and we are the cause”?
Oh for ****’s sake. Just FFS.
Enough.
Darn it. Keep forgetting to close the bold: “Would an accurate characterization of that result be: “97% of climate papers stating a position on human-caused global warming agree global warming is happening–and we are the cause”?
j
> What is more sad is the lengths folks go to [inaudible] rather than doing it over
correctly.
Take that, Richard!
> Since the planet is at stake [inaudible] is simply unacceptable.
Take that, ClimateBall ™ players!
Steven Mosher, if you cannot even get the contents of the paper right, why should be value your opinion on the data, method and transparancy?
I’ll give you a hint: the paper did *not* rate the papers on whether they “accept[ed] that C02 is a GHG and that it causes warming”.
Really, it’s the Moon landing paper all over, where people just *know* it is wrong and junk and whatnot, because.
And no, I did not forget to complete that last sentence.
Oil
“You are going with gut feelings that maybe the reviewers were tired.”
Having done content analysis studies ( as far back as the 80s, some of it
pioneering computer based content analysis of text) It is my experience that
raters do in fact tire over time. That is why we schedule regular breaks.
They also regress to the norm. That is why we ‘renorm’ raters. We re norm them
by putting them through another training period where they are asked to rate or rank
known exemplars. During the rating process we asses rater reliability. We report this.
That is why we keep track of who rated, how many they rated, and their rating over time.
That’s why we do the same thing for the ‘tie breakers’. They too drift over time. they too
have biases.
There are a variety of issues and procedures that require attention.
The paper as it stands is not even a freshman effort.
here is a nice place to start to understand some of the issues.
Click to access lombard_reliability.pdf
“It is widely acknowledged that intercoder reliability is a critical component of content analysis, and that although
it does not insure validity, when it is not established properly, the data and interpretations of the data can not be
considered valid. As Neuendorf (2002) notes, “given that a goal of content analysis is to identify and record
relatively objective (or at least intersubjective) characteristics of messages, reliability is paramount. Without the
establishment of reliability, content analysis measures are useless” (p. 141). Kolbe and Burnett (1991) write that
“interjudge reliability is often perceived as the standard measure of research quality. High levels of
disagreement among judges suggest weaknesses in research methods, including the possibility of poor
operational definitions, categories, and judge training” (p. 248).”
“First and most important, calculate and report intercoder reliability. All content analysis projects should be
designed to include multiple coders of the content and the assessment and reporting of intercoder reliability
among them. Reliability is a necessary (although not sufficient) criterion for validity in the study and without it all
results and conclusions in the research project may justifiably be doubted or even considered meaningless.”
@JasonB
The 4s are fine. They do not affect the results. It is terribly inefficient to have to discard the vast majority of your data, but Cook is free to waste his time and that of his mates.
Sampling is fine too, if the sample is representative. Cook’s sample is not. I reconstructed his data (well, almost) and then constructed a supersample of the literature. Cook’s sample is not representative on discipline, author, or influence.
Cook’s results are sample statistics, rather than estimates for the literature.
“I’ll give you a hint: the paper did *not* rate the papers on whether they “accept[ed] that C02 is a GHG and that it causes warming”.
really what did it rate them on?
This is going to be fun because I will ask you for the training data and the rater reliability results
from that training and the samples they used to train the raters and how each rater performed in the training.
You got that?
If not, then you have nothing.
here is what Im looking for.. evidence ( not your words ) of the following
‘4. Assess reliability informally during coder training. Following instrument design and preliminary coder training,
assess reliability informally with a small number of units which ideally are not part of the full sample (or
census) of units to be coded, and refine the instrument and coding instructions until the informal assessment
suggests an adequate level of agreement.
5. Assess reliability formally in a pilot test. Using a random or other justifiable procedure, select a representative
sample of units for a pilot test of intercoder reliability. The size of this sample can vary depending on the
project but a good rule of thumb is 30 units (for more guidance see Lacy and Riffe, 1996). If at all possible,
when selecting the original sample for the study select a separate representative sample for use in coder
training and pilot testing of reliability. Coding must be done independently and without consultation or
guidance; if possible, the researcher should not be a coder. If reliability levels in the pilot test are adequate,
proceed to the full sample; if they are not adequate, conduct additional training, refine the coding instrument
and procedures, and only in extreme cases, replace one or more coders.”
If the paper is “junk” then surely that can just be proved by simply demonstrating that the ratings (or a significant proportion of them) given to the abstracts do not reflect their actual findings rather than banging on about intercoder reliability?
[Mod: a bit inflammatory]
[Mod: a bit inflammatory]
> here is a nice place to start to understand some of the issues […]
Here’s another one:
BBD
“Since so much is at stake, hypocrisy and false claims are simply unacceptable.
So why do it, Steve?”
#################
ask mann, ask Gleick, ask Lewandowsky, ask Cook, ask yourself.
This is really simple. Cooks paper sucks. the method sucks, the transparency sucks.
Lets play what if.
What if somebody used his methods to assess the intelligence of students and determined that 97% of a certain race where incapable of doing college work.
Well, we would want to examine the rater reliability.
Well, we would be shocked if the researcher refused to share all the data.
Well, we would shake our head that so few raters did most of the work.
In short we would not accept this methodology for admiting students and we would reject this kind of work if it purported to establish something about certain races ability to perform college work.
The answer is simple. Just do the work over and keep better records. The answer will probably come out the same or very close, that’s my expectation. but dont ask me to call shit work good work merely because I think global warming is a problem that we need to do something about.
You can sugarcoat a turd if you like, but dont expect me to eat it
@Steve M
Indeed.
In fact, one of Cook’s raters complained about tiredness and worried how it might affect the quality of his work.
The ratings drift over time. There are patterns that cannot be explained by chance. And now Shollenberger tells us one rater did 765 ratings within 72 hours.
More picking at nits by (in this case) Mosher…
As has been stated (repeatedly, by numerous commenters) if there is uncertainty or drift in the raters, the best measure of whether the rating system was accurate regarding AGW consensus is an independent check of the raw data (the scientific literature).
From Neil dGrasse Tyson: “Once science has been established, once a scientific truth emerges from a consensus of experiments and observations, it is the way of the world. What I’m saying is, when different experiments give you the same result, it is no longer subject to your opinion. That’s the good thing about science: It’s true whether or not you believe in it. That’s why it works.” (emphasis added>
For example: The Cook et al author self-ratings, Oreskes 2004, Doran 2009, Anderegg et al 2010. All of which conclude a consensus around 97%, plus or minus a couple of percent.
> we would shake our head that so few raters did most of the work.
We would also shake our head that so much raters did the work.
Let’s play what if.
What if by some miracle there was just the right number of raters?
We’d still be shaking our heads.
Or stir. Or quiver. Or jiggle, juggle, wobble.
The ways we can shake our heads are multifurious.
Steven
You can assert that there is a turd if you like, but don’t expect me to accept your logical fallacies.
See what he did there?
😉
One note regarding inter-rater reliability, which was indeed reported in Cook et al 2013:
“Initially, 27% of category ratings and 33% of endorsement ratings disagreed. Raters were then allowed to compare and justify or update their rating through the web system, while maintaining anonymity. Following this, 11% of category ratings and 16% of endorsement ratings disagreed; these were then resolved by a third party.”
Regardless of whether or not people agree with the methodology (Mosher?) they reported their results for evaluation.
So that’s argument from (false) assertion and false equivalence, all masquerading as something we are supposed to take seriously.
Mosher shows he either hasn’t read the consensus paper or he does not understand any of the climate debate:
Mosher, do you understand what the “A” in AGW means? I suggest you read some elementary texts and papers on climate science before embarrassing yourself any further.
Richard Tol said, “One of Cook’s raters did 283 ratings in a single day. That is 2 minutes per abstract for 10 hours.”
1) Since there are only date stamps, and no time stamps (which would have been completely useless anyway), on the ratings, you’d have to know WHERE the rater was located relative to the how the date stamp was applied (UTC, Queensland time, or local time) in order to even make that determination. That could have been split across two days. You’d have to look at that figure relative to the pervious and following days to see if it was anomalous.
2) You’d also have to evaluate that relative to other rater’s daily rate. How far out of line was that single day with other rater’s single day figures?
3) You’d also have to look at each individual’s personal circumstances. Is this the only work the person needed to do a the time. For instance: Perhaps a spouse was out of town giving the rater a lot of extra time to focus on doing ratings.
4) You’re forgetting word count. Rating speed would be a function of word count of the abstracts that came up. Some were very short and others absurdly long. It’s perfectly conceivable that a rater could get a large number of short ratings on a given day.
5) You are COMPLETELY neglecting a whole other aspect of doing ratings: Proficiency! Once you’ve done 100+ ratings it becomes a much easier task. You get a sense of what words and phrases are going to indicate the position of the paper. After rating a large number of papers the task actually gets much easier to complete and likely more accurate as well.
6) And, you also neglect the fact that Cook et al understood that individual raters were going to have different positions on papers. If each paper had only been rated once and left at that, then we’d have a paper that had merely presented survey of the raters. But we went through and did each rating twice. And where there were disagreements, everyone took a second look at the paper in order to determine which rating was more accurate. And after that, where two raters could not agree, had a tie breaker.
7) And to further check the results, you avoid the blatant fact that Cook et al took the ultimate step of self-skepticism by asking researchers to self-rate their own papers. It was possible that SkS raters were highly biased in their rating of papers, and that why we check ourselves by asking the people who know their work the best to rate their own papers. It shows that, not only were SkS raters conservative in their ratings, Cook et al’s methods to weed out any bias on the part of raters was very successful.
Steven,
Firstly, I’m not actually defending it as such – I’m more pointing out that none of the criticisms are anything that wasn’t already acknowledged in the paper. Also, why should it be done again? We all seem to agree that the result is roughly correct – most papers that take a position on AGW endorse AGW. The only reason these studies are actually done is because there’s one group who claim that there isn’t a consensus, and another who attack such studies whenever they take place. Telling such people to grow up and stop being so pathetic would be much more effective and do as a great big favour. We could also stop doing these consensus studies, which only exist because of a vocal minority who either don’t understand what’s in the literature or do and are willing to knowingly misrepresent it.
At least you seem to be acknowledging that it’s mainly nit-pickers. Personally, I think we’d all benefit if pedants stopped putting themselves on pedestals where they can pontificate about “research ethics” and “it’s wrong to get the right answer for the wrong reasons”. In my opinion, the credence given to someone should be inversely proportional to the height of the pedestal upon which they’re willing to put themselves (or, be put, in some cases).
Let’s also put this into some kind of context. This is a citizen science project that ended up being one of the highest impact papers of last year. In any other area, this would be lauded as remarkable. Instead a bunch of people who don’t like the authors and don’t like the result are pretending that they’re standing for honesty and ethics while doing their damndest to find any kind of error with the paper – and mainly finding issues already addressed in the paper itself.
Well, that has to be in the running for ironic comment of the day award. I gather you don’t like John Cook or Skeptical Science. You may well believe that it’s shoddy. It is almost certainly true that there are other methods that are better. I would suggest, however, that your bias correction process may need a little tweaking.
Richard,
Well, Cook et al. must be in the running for the wrongest paper to get the rightest result. I guess if you keep trying you may actually find a truly valid criticism of the paper. That probably wouldn’t, however, save you from accusations of being petty and vindictive, though.
> really what did it rate them on?
Ask the Auditor, Bishop, Brandon, Judy, Lucia, Poptech, Richard, Shub, Tony.
The list goes on.
Ok, I was the one who did 283 ratings in a single day. I was using far too much cocaine.
> I gather you don’t like John Cook or Skeptical Science.
Perhaps:
But ethics.
[Mod: a bit too OT]
Mosher said… “What if somebody used his methods to assess the intelligence of students and determined that 97% of a certain race where incapable of doing college work.”
Oh puleeeze!
If someone submitted such a paper to a high profile journal it would never make it past the front desk, much less pass peer review.
I’m not hearing any logical progress coming out of the denial community.
Tol himself has never analyzed the data to reach his conclusions. And as he says,
“Results derived from incorrect methods are, by definition, incorrect.”
@AnOilMan
I did analyze Cook’s data.
Well said, ATTP. Mosher did almost get one thing right – here, let me fix it for you:
If you doubt the result or the methods, then just go test it for yourself. We even set up a website to make replication as easy as possible. It’s been a year – nobody has reported significantly different results from ours.
Frankly I think the criticisms of our methods are really stupid (present company very much included). But more importantly, they’re pointless. Go do the analysis yourself however you like; otherwise stop whining.
“We should start a pool on how many comments will be posted on this thread before all is said and done.
Well, I plan to go for a run before I say anything more.”
Me too: time to change the oil in the Jaguar, a *far* more productive activity!
> Oh puleeeze!
I think that’s
I went for a run last night and tonight. If I keep writing posts like this I’ll be fit again in no time.
Richard,
I presume you mean the ratings, rather than the abstracts themselves. So, you’re performing a survey of the raters in order to test a project that was a survey of the abstracts. When you’re finished, we still won’t know if the abstracts were properly rated or not.
“If I keep writing posts like this I’ll be fit again in no time.”
Please don’t! I’m struggling to keep up.
And a request to everyone else: things are not too bad at this stage, but please try to keep this from becoming a food fight.
Rachel,
Yes, good advice, thanks 🙂
I did an experiment.
I read the abstracts of an edition of the journal of the historical metallurgy society. 4 abstracts in 1 minute 15 seconds, and i was able in that time to tell that all of them agree with the hypothesis that humans have in the past used metal, one covering the making of iron and steel in the 18th century, another the medieval European use of copper alloys, and so on.
So exactly how can’t someone check hundreds of abstracts in a few days? Don’t judge others by your own [Mod: a bit inflammatory] standards.
> So exactly how can’t someone check hundreds of abstracts in a few days?
Yes, but you look tired, Guthrie.
You might provide an evaluation that is way too conservative.
Richard… what ATTP said.
You need to compare your example set of abstracts to how the raters performed in order to conclude anything useful.
What Willard said… This may sound surprising, but I’m not against Tol. I just don’t think there is a way to reach any sort of useful conclusion from what he’s done.
Agreed.
Steven –
Heh.
==> “but dont ask me to call shit work good work …”
Ok I won’t, but only if you don’t ask me to tell you that I’ve seen monkeys flying out of your butt. No matter how much you ask, I just won’t do it. It’s a matter of ethics. And integrity™.
And don’t ask me to send you a check for a million dollars.
[Mod: playing the ref.]
You don’t have to dig out the data and do fancy analyses or worry about the tiredeness of the raters to demonstrate that the 97% paper is junk^Wflawed. A quick look at Table 2 suffices.
I won’t bother arguing that the categories overlap (although they do). Instead, assume that the categories make perfect sense and have a look at the examples. The examples given for Levels 1 and 2 actually exemplify Level 3. The guidance given to raters is explicit about this: ‘Mention of increased CO2 leading to higher temperatures without including anthropogenic or reference to human influence/activity relegates to implicit endorsement.’ Worse, the example given for Level 6 – a dread denier level – also belongs among the righteous in Level 3. The example doesn’t minimize human influence on global warming, as it should if it belongs in Level 6. Instead, it (sort of) minimizes the greenhouse effect and, in doing so, implicitly recognizes anthropogenic greenhouse gas emissions.
A study with such a contradictory and confused classification system should never have been taken seriously. Its muddleheaded is mightily obvious. Getting the right answer or being misrepresented by someone as important as President Obama or being voted ERL’s study of the year doesn’t change that.
Dana: ‘Go do the analysis yourself however you like; …’
Right. Just what the world needs: another study showing that almost everyone who studies climate change for a living thinks that what they study exists.
‘… otherwise stop whining.’
OK.
Vinny,
But that’s the whole point. If everyone simply accepted what is self-evidently true, these studies wouldn’t be needed. That doesn’t mean that the existence of a consensus means that the science is right/settled/beyond criticism. It simply means that there is a large level of agreement. Plus, this refers only to AGW, not to the specifics of how AGW will influence our climate. An strong level of agreement about AGW, doesn’t mean a strong level of agreement about all aspects of climate change.
So, as far as I can tell, almost everyone who’s criticised the Cook et al. paper, actually agrees that there is a strong consensus. Fine. That’s really all it was trying to illustrate. Seems to have worked. Now, can we actually move on?
Ah, no. The purpose of this exercise is to create the illusion of controversy in order to peddle doubt.
So like MBH99, the fake controversy will “rage” on and on and on and on. Unless and until the doors are shut on the peddlers of doubt.
Hi ATTP,
You write, “So, as far as I can tell, almost everyone who’s criticised the Cook et al. paper, actually agrees that there is a strong consensus. Fine. That’s really all it was trying to illustrate.”
If that’s all the Cook et al. paper was trying to illustrate, why do they publicize the results as being, “97% of climate papers stating a position on human-caused global warming agree global warming is happening–and we are the cause”?
Do you think that’s an accurate characterization of the results of their paper?
What if somebody used his methods to assess the intelligence of students and determined that 97% of a certain race where incapable of doing college work.
Well, we would want to examine the rater reliability.
Well, we would be shocked if the researcher refused to share all the data.
Well, we would shake our head that so few raters did most of the work.
Well the questions I would ask are –
On what data is that assessment based?
Is that data suitable for the purpose for which it is being used?
Is that data is a sample then is it representative?
Has that data been made available?
Does an independent examination of a reasonable sample of that data indicate that the raters’ judgement is unreliable?
If the answer to the last question was “yes” would I consider the reliability of the raters to be an issue requiring further attention.
Now WRT the Cook paper, as far as I can see the answers to those questions would support the arguments of those defending the paper and not the “skeptics”.
As for whether people here should be spending time defending the paper, well some of them are associated with it so one would expect them to defend it, for the rest of us I agree that there are probable more important things to argue about. The skeptics will try to trash the paper regardless, any confluence of their criticisms and the actual merits of the paper, even if it existed, would be entirely coincidental and has little influence on its wider public perception. Let them obsess about it, the rest of us can move on.
> The examples given for Levels 1 and 2 actually exemplify Level 3.
Then showing that the raters or the authors themselves should have classified as 3 ABSTRACTS or PAPERS that were classified as 1 or 2 should be easy.
Please report when you’re finished, Vinny.
Mark,
Unless you’re playing some kind of word game that I haven’t understood, yes. Most on this thread – including those who are critical of the Cook et al, study – agree that most papers in the literature that address AGW, endorse AGW. That’s what I was getting at. Do you think that this is incorrect? Do you think that the results of Cook et al. (that almost all papers/abstracts published in the last few decades endorse AGW) is wrong?
Tol, above, claims the Cook et al literature survey was not representative. What he does not do is address the potential impact of the “skew” in the the literature sample that he finds, something that I have done. It turns out that had Tol’s “representative” sample been used rather than the sample used by Cook et al, it would only have made a difference of plus or minus 0.6% :
As I said in the conclusion of my blog post:
Mark Bahner said… “Do you think that’s an accurate characterization of the results of their paper?”
Yes, and in fact, accurate by two separate and independent measures.
1) Through ratings by of abstracts by SkS raters.
2) Through self-ratings by authors of the papers themselves.
ATTP: You sure have the knack of attracting a host of “very serious people” to your comment threads.
What is your secret formula?
Of the OPs you have posted to date, which has garnered the most comments? How many comments does that post have?
John, what do you mean by OP?
Steven –
==> “What if judith said
“I happen to disagree. I think that actually, the validity of this paper (and the quantifying in some precise manner the % of consensus) has relatively big effect on public opinion.”
of course she has said similar things about climategate and you typically repsond asking for evidence. “What is more sad is the lengths folks go to to defend his crap rather than doing it over
correctly. …
I don’t find any of this sad. It is what it is. It is what I expect. If anything, I find it amusing. I find it particularly amusing that some “skeptics” argue ’till the cows come home about why “consensus science” is bad, bad, bad, and then turn right around and spend gobs of time arguing about the precise prevalence of the consensus. If they think it is irrelevant to valid science, and bad, bad, bad, then why are they so obsessed with it?
And interesting. I find it interesting that such pitched battles are waged over trivialities, when what might be (IMO, marginally) important about the larger topic is bleeding obvious – as Richard has indicated.
I mean geebus – if there’s anything important related to this food fight, that would be it.
One more thing, steven. I have found that invariably when I take the time to write a longer post in the hope of actually engaging in a good faith exchange with you – you have failed to respond in a way that appears to me to be in good faith. IMO, you have a habit of building straw men, or insulting me personally, or making the discussion about me, or ignoring what I discussed to focus on a non sequitur. But I haven’t tried in a while – and who knows, maybe you’ve change your ways, or maybe engaging in this forum will motivate you to respond to me in a way that seems to me to be more good faith-like?
Here’s your opportunity, my friend. You can still convince me that you’re interested in exchanging views in good faith.
OP = Original Post.
You’re right Willard, I need to get to bed earlier tonight.
Maybe when Tol attempts to replicate the search and rating he can ensure he gets a good nights sleep beforehand?
I think this one may still hold the record.
ATTP: Does your comments policy address the issue of a commenter using the username of a prominant climate scientist as a pathetic way of showing contempt for the scientist?
BTW, Back in the days when I posted comments on climate change articles posted on NPR’s website, I frequently crossed swords with a denier drone who posted as “Phil Jones/” .He was nothing more than a juvenile flaming arsehole in my opinion.
Willard: Do the rules of Climateball address this issue?
ATTP,
I’m not “playing a word game”. I’m concerned with accuracy and precision.
You think, “97% of climate papers stating a position on human-caused global warming agree global warming is happening–and we are the cause” was an accurate description of the results of their paper.
OK, what about if they had examined 100 abstracts, and 97 came out into category (1) Explicit endorsement with quantification. And 3 abstracts came into category (7) Explicit rejection with quantification. And no abstracts came into any of the other categories.
What would be an accurate description of those results? Would “97% of climate papers stating a position on human-caused global warming agree global warming is happening–and we are the cause” be an accurate description of those results?
ATTP: I cannot help but wonder if this thread will wind-up breaking the record for number of comments.
They don’t, but I’m guessing that this may have happened?
Mark,
Yes. I’m not quite sure what you’re getting at.
Mark Bahner: Your concern about “accuracy and precision” suggests to me that you may be an engineer. Are you?
Mark Bahner: What is your working definition of the term “accuracy” and the term “precision” . Please be precise and cite sources. Thanks.
Good grief, I hope not. I suspect Rachel does too 🙂
John Hartz,
“I cannot help but wonder if this thread will wind-up breaking the record for number of comments.”
This thread is only just over 300 comments. I’m sure we’ve had threads with well over 500 comments. They’re usually the ones about how to communicate science. 🙂
JasonB –
==> “I think that remains to be seen, but from what I’ve seen it seems to have had a big impact on the media, where I see the 97% figure now being mentioned over and over again, and the absurdity of “false balance” now being openly ridiculed e.g. in the John Oliver clip above. In fact, a very cynical and jaded me has been somewhat surprised at how quickly it has gained traction and how ineffective “skeptics” attempts to discredit it have been outside of their own little world.” “If the surveys suggesting that the general public believes scientists are still evenly divided on the issue are correct, and if the media continues down this path of emphasising the amount of agreement within the scientific community, then I think the “skeptics” are right to be worried about the impact of this paper. ”
I think that you’re being too general with “the media” there. Fox news is certainly not emphasizing the amount of agreement within the science community, and someone ideologically disinclined to believe that there is a consensus will dismiss media reports from the outlets that talk about a consensus. There is a reason why so much of the public does not have an accurate view of the amount of agreement in the scientific community – and simply more publicity that there is widespread agreement, IMO, does not address the root cause of why so many people don’t know that there is widespread agreement or refuse to believe it or argue that such widespread agreement is irrelevant, or harmful, or the product of librul academics and an agenda-driven cabal.
I did my own little informal survey of abstracts of papers from a few years of the American Association of Petroleum Geologists Bulletin to see how many abstracts endorsed either plate tectonics or the biotic origin of oil. (I admit to bias here, I am a gullible believer in plate tectonics–an unabashed mobilist–and a denier of the abiotic oil ideas put forward by people like Thomas Gold and many Soviet petroleum geologists, so feel free to doubt my results and do your own study.)
What I found was that my petroleum geologist colleagues were even more reticent than climate scientists about expressing endorsement or rejection of their paradigm. (From memory–I don’t have the results handy–the great majority took a “no position” attitude and, to my recollection, very few of them expressed an explicit or quantified endorsement of plate tectonics or biotic sources for oil, and none at all rejected either theory.) No doubt, there are a few stalwart contrarians, aging Russians mostly, who see this as evidence of pathological group think and venality among petroleum geologists.
The observation that scientists don’t clutter up the limited real estate of their abstracts with statements (to them) of the bleeding obvious should not be surprising. One of the interesting observations of Cook et al is that the percentage of no-position abstracts increased over time (Figure 1b), perhaps an indication that an increasing proportion of workers in the field no longer consider the endorsement of AGW as something that needs to be mentioned when addressing their peers. Palaeo-conservatives, repelled by the idea of a strengthening consensus on AGW, will interpret this differently and focus instead on the declining proportion of endorsers.
Maybe I’ll do a blogpost on this sometime.
Oil Guy:”What Willard said… This may sound surprising, but I’m not against Tol. I just don’t think there is a way to reach any sort of useful conclusion from what he’s done.”
Once more mistaking a feature for a bug.
==> “I cannot help but wonder if this thread will wind-up breaking the record for number of comments. ”
Fascinating how much interest is generated about something that is bleeding obvious, as Richard points out.
John Hartz – the point about Climateball is that you make up the rules as you go along in order to confuse and confound your opponent as much as possible. So yes, posting as a real scientists name in order to wind you up is permissible.
ATTP:
“Yes. I’m not quite sure what you’re getting at.”
Fascinating. So a paper that has results of:
(1) Explicit endorsement with quantification = 64 abstracts.
(2) Explicit endorsement without quantification = 922 abstracts.
(3) Implicit endorsement = 2910 abstracts.
(5) Implicit rejection = 54 abstracts.
(6) Explicit rejection without quantification = 15 abstracts.
(7) Explicit rejection with quantification = 9 abstracts.
And a paper that has results of:
(1) Explicit endorsement with quantification = 97 abstracts.
(7) Explicit rejection with quantification = 3 abstracts.
…can both be accurately described with exactly the same words?
Mark,
Okay, so you were playing with words. Good to have that confirmed. ClimateballTM.
==> “I’m not “playing a word game”. I’m concerned with accuracy and precision. ”
Well, it’s good to know that we can add someone to the list. I was getting concerned that it was only Richard and Carrick who were concerned about ethics, accuracy, and the like.
Mark… That is bizarrely random at best, and motivated to extract a predetermined conclusion at worst.
If you question the results, all you have to do is go to the TCP page on SkS and rate papers yourself. See what results you get. http://www.skepticalscience.com/tcp.php?t=home
Mark my words…
Before all the dust settles, this thread will set a new record for number of comments.
I can feel it in my bones.
Indeed, two things which are in certain characteristics the same can be described in the same way. This is, well…sort of how language works. It’s precisely the point of having words for things, in fact.
If you want to take a quick break form this extremely serious discussion by extremely serious people, I recommend that you take a gander at: The Frustrations of Being Scientifically Literate by By Craig Fay, Scientific American, May 13, 2014
Fay’s article actually ties to the OP — even if by a single thread.
Joshua: If i recall correctly, Mark Bahner does not typically respond to questions posed to him by other commenters. Am I correct?
I’m still looking for his working definitions of the words “cccurately” and “precisely”. If we are going to have a serious discussion with him about how these two concepts apply to Cook et al, we all must be singing out of the same hymn book so to speak.
If Mark cannot define these two key terms, he’s merely blowing smoke out of his you-know-what..
guythrie: Are you autjorized by Willard to spek ex-cathedra about Climateball?
It is indeed fascinating, Mark, that language is a social art.
***
> This may sound surprising, but I’m not against Tol.
This may not be true of Richard, since he says in his 7th draft:
https://docs.google.com/file/d/0Bz17rNCpfuDNRllTUWlzb0ZJSm8/edit
Our empĥasis. Now, let’s consider this claim from Richard’s conclusion:
Is Richard’s opinion relevant? Arguably not.
Somehow, Richard failed to disclose that his own opinion on the matter is irrelevant.
Rob Honeycutt writes, “If you question the results, all you have to do is go to the TCP page on SkS and rate papers yourself.”
It’s not that I “question the results.” Your “Skeptical Science” (truly one of the most ironic titles of all time!) is misrepresenting the results. And that is conclusively shown by the fact that ATTP says that the exact same words that your website uses to describe the results of your paper:
“97% of climate papers stating a position on human-caused global warming agree global warming is happening–and we are the cause”
Could be used to describe a paper that had results of:
(1) Explicit endorsement with quantification = 97 abstracts.
(7) Explicit rejection with quantification = 3 abstracts.
If the exact same words are being used to describe two papers with substantially different results, then one of those papers is being misrepresented. (And it isn’t the one with 97 abstracts in Category 1 and 3 abstracts in Category 7.)
John, the first and only rule of ClimateBall ™ is as guthrie says:
http://neverendingaudit.tumblr.com/tagged/climateball
Perhaps we ought to create a CLIMATE CLUB ™, where ClimateBall ™ would be practiced.
The first rule of CLIMATE CLUB ™ would be that you do talk about Climate Club ™.
Willard: What amenities woiuld CLIMATE CLUB ™ have for its members? Sauna? Pool? Excercise machines? Indoor track?
Dear Mark,
I thought I made myself clear yesterday. You’re pushing a peanut that was pushed before you, for months. It does not work.
Categories 1 and 7 are quantitative. If you only had these categories, you could not be able to do what Cook & alii did. You need qualitative categories for that. So your whole line of argument does not work.
If you’re fascinated by this, wait until you discover the concept of function. I mean, isn’t it amazing that 2 + 2 and 1 + 3 equals 4? The only problem would be if 2 + 2 would equal 4 or 5. This was Vinny’s argument, BTW. We’re still waiting for his report on his survey of Cook’s ABSTRACTS classified as 1 or 2.
It’s not that complicated.
[Mod: inflammatory]
Mark stated… “It’s not that I ‘question the results.'”
Excellent. Then let’s move on because saying anything else would be flogging a dead horse.
97% of the scientific research confirms that humans are the main cause of global warming.
Now, let’s get down to something extremely important, and that’s, what are we going to do to avoid making this much worse than it already is?
Willard: ‘Then showing that the raters or the authors themselves should have classified as 3 ABSTRACTS or PAPERS that were classified as 1 or 2 should be easy. / Please report when you’re finished, Vinny.’
If you insist – but I’m only doing one paper. Here it is.
The abstract for the paper that furnished the published example for Level 1 is here:
http://link.springer.com/article/10.1007/BF02915709
That an abstract provided a quote that was used in the published 97% paper as the exemplar for Level 1 is surely proof enough that that particular abstract was rated as Level 1, no? Yet the abstract above doesn’t say explicitly that the increased atmospheric CO2 concentrations studied by the study were due to mankind. They could be due to very little else but that doesn’t matter. The guidance given to raters says that if the the ‘A’ in ‘AGW’ is only implied then such abstracts belong in Level 3.
Dear Vinny,
I’m not asking for a training exemplar, but for an example that your concern has happened. In other words, I want you to show that your concerns about what could happen did happen. In any case, here is the abstract from your training example:
The emphasized bit shows two things. First, that this article was a quantitative analysis. Second, that this quantitative analysis concludes that the global warming during the 20th century is caused mainly by increasing greenhouse gas concentration. I have no idea why you believe this would be a 3.
Please give me an example you would clearly classify as a 1.
The sooner contrarians find it in their hearts to stop spreading misinformation and manufacturing unwarranted doubt, the easier it will be for posterity to forgive them for the delay they exacerbated.
Vinny… The abstract in question also doesn’t explain the radiative properties of CO2. Nor does it explain how the greenhouse effect operates.
Research papers are written for other researchers. They are going to assume you already know a little something about the subject matter at hand, and they’re also going to assume that you’re not a “SkyDragon Slayer” as well.
The only way you can dismiss this paper as not being a #1 would be if you were stretching to find a way to reject it as such.
You can certainly go through and do a similar research project using those sorts of parameters, but when you go and check yourself (as SkS did) against how the actual researchers self-rated their papers, I’m gonna bet you’ll find your results highly skewed from reality.
Indeed. The 97% does appear to be a rather robust result.
Of nearly 12,000 abstracts analyzed, there were only 64 papers in category 1 (which explicitly endorsed man-made global warming). Of those only 41 (0.3%) actually endorsed the quantitative hypothesis as defined by Cook in the introduction. A third of the 64 papers did not belong.
Very robust.
Hi ATTP,
I do have another question. Suppose the paper’s results had been:
(1) Explicit endorsement with quantification = 64 + 922 + 2910 = 3896 abstracts.
(2) Explicit endorsement without quantification = 0 abstracts.
(3) Implicit endorsement = 0 abstracts.
And 5 to 7:
(5) Implicit rejection = 54 abstracts.
(6) Explicit rejection without quantification = 15 abstracts.
(7) Explicit rejection with quantification = 9 abstracts.
In other words, what if all the abstracts that had been rated as (2) or (3) had actually been rated as (1). Would the results be properly described as: “97% of climate papers stating a position on human-caused global warming agree global warming is happening–and we are the cause”?
> I do have another question.
You might also like:
http://rationalwiki.org/wiki/Just_asking_questions
What if I told you that Mark is jaqing off by throwing counterfactuals?
Mark Bahner… Read the paper:
Click to access 1748-9326_8_2_024024.pdf
Specifically, read table 2. You’re missing or avoiding the full explanations for each level endorsement.
“What amenities woiuld CLIMATE CLUB ™ have for its members? Sauna? Pool? Excercise machines? Indoor track?”
Ice rink with goals.
Eli Rabeet: Movable I presume.
Mark Bahner, Cook et al rates abstracts into three basic categories:
A) Those that endorse AGW;
B) Those that neither endorse nor reject AGW;
C) Those that reject AGW.
Category A abstracts are subcategorized by the nature of endorsement, ie, whether it is explicit with quantification; explicit but unquantified, or implicit. No amount of shifting numbers between those sub-categories can change the total included under category A. The same applies also to category C. Category B was sub-categorized based on a randomly chosen sub-sample into abstracts that do not address the issue, and those that do, but are inconclusive.
Your objection is no more sensible than attempting to refute a claim that 30% of brigade where cavalry because it would be equally true whether the 30% were composed of hussars or dragoons.
I have spent the last two hours conducting an independent/objective review of the comments posted on this thread to date. Based on that review, I can state with the highest level (95%) of confidence that:
“The feeble attempts by Tol and the Tolettes to discredit Cook et al do not even amount to a bucket or warm spit.”
You can take that conclusion to the bank!
I also have it on good authority that Cook and Lewandowky intend to mine this comemnt thread for data for their new paper on the relationship between pseudo-science skeptics and wood fairies.
Pingback: Another Week of Climate Disruption News – May 11, 2014 – A Few Things Ill Considered
Vinny is trading on an ambiguity in the distinction between “explicit” and “implicit”. Taking a very literalist reading, if I state that there are exactly three women, and two men in a room, I have not explicitly stated that there are five (adult) people in the room. That is because it requires the additional information that 3+2=5, and an inference to reach that conclusion.
Of course, if I were in a conversation and somebody were to make that “objection” to my comment, I would think them an obnoxious fool who was being deliberately obtuse to score rhetorical points. Every statement no matter how explicit will assume background information and the related inferences to make claims. The only exceptions are to be found in formal logical proofs, and mathematical proofs. Selectively declining to recognize that convention does not make somebody smart – merely obnoxious and foolish.
When rating abstracts, I (for the few I did), John Cook (from his examples) and I believe all abstract raters applied the rule that if an explicit statement was made that together with an inference on a matter of settled science (eg, that the increase of GHG concentration in the twentieth century was anthropogenic in origin) implied the primarily anthropogenic origin of the temperature rise in the later half of the twentieth century, it was taken as explicitly endorsing AGW except when the abstract explicitly called that matter of settled science into question. That is the correct way to proceed, and certainly the way that (in general) the self rating authors proceeded.
However, even if, Vinny disagrees with this, it makes no difference. The key result from Cook et al is the overall level of endorsement – not the sub-categorization into explicit and implicit endorsement. Adopting his ridiculously pedantic approach would not change that result one iota.
Mark Bahner – You are putting an _awful_ lot of effort into building your strawman argument, attempting to construct something that could be (mis)interpreted as an issue with the paper. And that’s frankly quite clear to all reading this thread.
If you have to go to such lengths to come up with a linguistic objection, it’s clear that you don’t _have_ anything supportable to say in this regard. Just pointing that out…
Tom,
So, in your opinion, a paper with the results of:
(1) Explicit endorsement with quantification = 64 + 922 + 2910 = 3896 abstracts.
(2) Explicit endorsement without quantification = 0 abstracts.
(3) Implicit endorsement = 0 abstracts.
And 5 to 7:
(5) Implicit rejection = 54 abstracts.
(6) Explicit rejection without quantification = 15 abstracts.
(7) Explicit rejection with quantification = 9 abstracts.
…and a paper with:
(1) Explicit endorsement with quantification = 64 abstracts.
(2) Explicit endorsement without quantification = 922 abstracts.
(3) Implicit endorsement = 2910 abstracts.
And 5 to 7:
(5) Implicit rejection = 54 abstracts.
(6) Explicit rejection without quantification = 15 abstracts.
(7) Explicit rejection with quantification = 9 abstracts.
…could both be accurately characterized by:
“97% of climate papers stating a position on human-caused global warming agree global warming is happening–and we are the cause”?
Mark Bahner, yes – and obviously yes. In just the same way, a class containing:
1) 5 females 5′ 8″ or over;
2) 7 females under 5′ 8″ and over 5′
3) 4 females 5′ or under,
and a class containing:
1) 16 females 5′ 8″ or over;
2) 0 females under 5′ 8″ and over 5′
3) 0 females 5′ or under;
can both be accurately characterized as a class containing 16 females. The characterization contains less information than the full description. That does not mean the characterization is inaccurate – only that it is incomplete.
I could put that into formal logic if you like, but that would do you know good. This is such a simple point that your problem with it cannot be intellectual. It is also such a simple point that if you do not get it now, there is no point wasting further time on you.
Mark,
You’re asked the same question twice now. Let’s move on please. You don’t have to accept the responses given to you but please don’t repeat the same thing. The thread is long enough already!
Well said!
Rachel,
I asked ATTP, but he never answered. Then I asked Tom, and he did answer.
Mark
Mark: Of course. The paper was on the question of whether the papers endorsed or rejected the proposition. Categories 1-3 and 5-7 merely state the manner in which that endorsement or rejection was expressed.
The fact that you have chosen category 1 in particular seems to suggest that you think it’s the most extreme level of endorsement. It’s not. It’s simply one that explicitly quantified the degree of human influence, meaning it’s likely to be an attribution study (as is level 7).
And it’s legitimate to argue that categories 1-3 all support the statement because any paper that suggested that global warming was not happening or that we were not the cause would end up in categories 5-7, depending on how they expressed that view. I’ve yet to see anyone produce an example of a paper’s abstract that disagrees with that statement and that would not end up in categories 5-7.
Joshua:
I think it’s long past time we gave up caring what the dismissives think. There’s not enough of them to matter if the disengaged become convinced — and that’s who the real target is.
Right now there are mainstream voters who don’t watch Fox News but who think that there’s a genuine debate in the scientific community and that the smart thing to do would be play it “safe” and wait until we’re more certain about what’s going on before doing anything drastic. If they come to understand the truth, and learn to recognise the tricks being played to try to keep them in the dark, then the dismissives won’t have a chance. Right now they have a disproportionate influence on policy because the overwhelming majority are ignorant of what they’re up to. The “skeptics” are right to be concerned.
IMO, of course.
Richard:
I’m not sure there was any way to avoid it. Search terms are well and good, but at the end you’re going to have to roll your sleeves up and sort the wheat from the chaff the hard way.
OK, so what you’re saying is that one or more of the filters they applied did introduce a bias. I look forward to reading what that was, what the evidence for it was, and what impact it had on the results.
Steven Mosher:
Later:
Well, I’m convinced. You should write up that devastating rebuttal and get it published. You’re nearly there.
It’s amazing that the consensus is so strong that despite using junk methods that suck, somehow they still managed to get the right result.
But then: How do you know the conclusions are roughly correct? Is there a survey that was done “right” that you’re comparing it to? Or are you relying on your own survey of the literature to assess what the correct conclusion is? What data did you use? What methods? I don’t see any transparency.
What is more sad is the lengths folks go to to argue that it’s crap rather than doing it over “correctly”. I look forward to seeing your results.
[Mod: Comments about moderation will be deleted. Your suggestion is good though and is in the moderation policy already 😉 ]
Jason B:
Good luck with that. Tol never details the likely impact on the results from any of his criticisms. Were he to do so, people would see how small the impact is likely to by, and recognize the carping nature of his criticisms.
@JasonB
A link is with Willard’s post at May 13, 2014 at 10:30 pm
@TomC
My paper does a number of bias corrections, one at a time. The dissensus rate invariably goes up, from the 2% estimated by Cook to 3, 4, 8, or 12%.
[Mod: defamatory]
Pingback: Another Week of Climate Disruption News – May 11, 2014 [A Few Things Ill Considered] | Gaia Gazette
Thanks Richard. It makes me worry about how carefully you studied the material when your paper claims it is not known whether 40 were found in the sample of 1,000, or 5 were scaled up to 40. Hint: It was 5 scaled up to 40, as communicated by one of the authors at the time and subsequently verified by posting the re-rating data on their website.
You’re welcome.
Earlier you said:
But then, in your paper you complained about them using the search time “global climate change” rather than “climate change”, where the latter significantly increases the number of papers that would need checking by hand. As it seems reasonable to assume that a higher proportion of papers that do not use the word “global” will have to be discarded as not relevant to the question at hand than of papers that do use the word “global”, your suggestion would make the exercise even more inefficient.
I look forward to seeing the results of your review of that larger set.
You also spend a lot of time comparing the characteristics of the set of papers chosen by Cook et al to the set of papers returned by your preferred search terms in your preferred database, but I couldn’t see any proof that the latter is a better representation of “the scientific literature” than the former. It’s bigger, yes, but what analysis have you done to show that repeating the exercise using your preferred set would give a more accurate representation of “the scientific literature” than Cook et al’s? What’s to stop someone using the exact same figures you’ve produced to argue that Cook et al’s sample is a better representation of the literature because of those differences?
Page 3, line 15: “This introduces a bias against endorsement.” How do you know? Did you rate papers in the undersampled disciplines and determine that their endorsement levels were higher? If so, where is the data?
Page 3, lines 19-20 and page 4, lines 2-3 both make unsubstantiated claims, the first that the most active scholars tend to support ACC and were undersampled, the second that the most influential papers tend to support ACC and were oversampled. Data?
Page 3, line 9: “This introduces a bias towards endorsement.” Data?
Page 3, line 17: “Such journals tend to be kinder on heterodox material.” Data? If true, why should we assume that’s better? Perhaps “young and obscure journals” should be filtered out lest we end up reviewing papers in dog astrology journals?
After all that, you conclude that there must be a bias, “but the direction […] is unknown”. You just know it’s wrong, even though you can’t actually say in which way! Isn’t that putting the cart before the horse?
Skimming the rest, I see a lot of concern (e.g. “Fatigue may have been a problem”) and claims like “One could argue that impact papers should be rated as neutral” (alternative, “One could argue” that impact papers taking for granted that the statement is correct are an excellent indication of the existence of a consensus; eventually, if the consensus position is correct, all the papers will be like that) and “The sampled papers are not representative of the population of papers” (where’s the proof that your sample is more representative of “the population of papers”?), but what I was looking for was a demonstration of both the bias their technique introduced and a quantification of the impact it had on the results.
I did not find it.
@JasonB
Re 4a &b, that draft was written before that data were released. Different members of the Cook team had written different things about what was done, hence the uncertainty in my tone.
Mark Bahner
Give the man a chance 😉 ATTP is based in the UK and keeps to sane hours. He was probably asleep.
Richard,
Sometimes I really do wonder if you are who you say you are – a senior academic at a major university who’s published hundreds of papers. It’s almost as if you don’t understand the academic process. It’s bad enough that you’ve been trying to get more and more of the data from the team despite the result being reproducible with the information in the paper, but now you appear to be assessing things on the basis of what some of the team might have said – presumably in blog comments or on a private forum that was hacked. It just seems absurd. If you aren’t Richard Tol, you really should stop this as you’re making a serious academic seem petty, vindictive and not very bright. If you are Richard Tol, you should probably stop for roughly the same reason (although if you are Richard Tol, I doubt you take advice from many people and would never from me). I’m embarassed enough about being involved in what’s becoming (or became a long time ago) a ridiculous discussion, and I have the advantage of still being anonymous.
BBD,
I was asleep but also can’t really be bothered engaging in a discussion with someone who doesn’t seem to get that it is indeed possible to describe two things that are differ in the details, using the same words – If I had 97 dogs and 3 cats, I could say “97% of those animals are dogs” whether or not the dogs were all the same breed, or a whole lot of different breeds.
ATTP
I can well understand that, but didn’t want to speak for you 😉 MB is being extremely tiresome.
@Wotts
It’s very simple. The paper is ambiguous. Different authors made statements that are clear but contradictory. Ergo, I don’t know.
Or rather, I didn’t know until I inspected the data myself.
The reason I raise this has nothing to do with the ambiguity in the text. The real point is that Cook seems to have changed the survey after inspecting the results — turning the whole thing into a farce.
Richard,
And my point is : so what? Either redo the study to show that the results are wrong, or prove that they behaved in a way that wasn’t ethical. Implying the latter is, in my opinion, bordering on unethical itself. And, since we’re discussing ethics, if you have access to the data that Brandon has it would seem to me that you would need to consider ethics approval from your university, given that the data identifies individuals. In fact, that you are inferring things about the paper based on what some have said, I would argue that you’re already doing research that involves indentifiable individuals. I would actually quite like to see the ethics approval process; it might be a study in irony :
Q : Do you have permission to use this data?
RT : No.
Q: How did you get this data?
RT : Someone who may have stolen it from a private forum gave it to me.
Q : Have those who will be identified given you permission to use the data?
RT : No.
Q: Have those who might be identified given you permission to identify them?
RT : No.
Q : Why do you want to do this research?
RT : I want to show that a citizen science project that published possibly the highest impact paper of 2013 is flawed, even though the result is probably right and even though the main reason for the study is because people like me keep criticising these studies when they happen.
Q : Why do you want to do that?
RT : Because I don’t like the authors.
How does the mythical RT’s answer to the last question not involve the Global Warmers Protection Fund? I’m concerned.
jsam,
Are you suggesting that the correct answer to
Q : Why do want to do that?
is
Because an ex-Chancellor of the exchequer is concerned that studies like this make him look like an ill-informed idiot?
Great comment at May 14, 2014 at 8:35 am, ATTP. Bravo.
I think the response to a lot of Tol’s comments will start with “as already pointed out in the original article…”
Might I suggest:
ATTP: You rock!
I would like your permission to write an article for posting on Skeptical Scince about this OP and your excahnges with “Richard Tol.”
Richard Tol: How many graduate students have you assigned the task of finding flaws in Cook et al?
Do bears shit in the woods?
Abstract:
Voulnteers collected 10,000 ursine turds. Analysis of the lignin content of the locality of the turds was conducted and the localities of the specimens were rated as
1) Wooded area
2) Unspecified foliage
3) Non-wooded
We find using a conservative methodology that 97% of ursine turds are located in wooded environments. Further, by asking the bears themselves, 95% of bears also confirmed that of their previous 10 bowel movements, all 10 were conducted in wooded environments.
Tol
Ah, but DNA analysis of the turds showed that of the 3% found in non-wooded environments fully 50% of those were traced back to bears who claimed 100% shitting in the woods rates. Also unspecified foliage is hardly a scientific term, and the use of questioning the bears themselves would never pass a full ethics panel review.
This fatally flawed paper should be withdrawn and no quantitative conclusions on the locality of ursine turds can be taken from it.
John,
Sure, be my guest.
I look forward to Richard’s full response to JasonB’s constructive criticisms on May 14, 2014 at 5:49 am:
Many thanks in advance, Richard!
Willard: Do you know that the real Prof Richard Tol of Sussex University is the person posting on this comment thread as “Richard Tol.”?
I have no reason to doubt it, John, nor do I have reason to believe that AT was disbelieving it. My hypothesis is that it’s just a way to express incredulity. The discussion is not be about Richard, but about the commentary he published.
Consider our actual ClimateBall ™ episode that way. If Richard succeeds in making the discussion about him, concerned contrarians win. If Richard needs to face constructive criticisms such as JasonB’s, they should lose.
In either case, Richard wins. Raising concerns the way Richard does may therefore amount to some kind of Dutch book. The only thing you can do is to seek an outcome that would make you win too.
For instance, I see no reason why paying diligence to the content analysis references hand waved above would not be beneficial in the long run. First, it may show that the framework does not really apply to Cook & alii. Second, it may contain suggestions that may be helpful for future research.
If you only look at what has been played, you will not enjoy the game as much as if you also think about all the moves that lie ahead.
@John Hartz
All 55 of them. We are negotiating with a publisher to start the Journal of Cocology.
Willard: From your perspective, is Climateball anything more than a cute name for “Propaganda War” ?.
Richard ToL: At least you have a sense of humor.
Willard: Your points are well taken. My problem is that I have neither the time nor the inclination to get into the nitty gritty of the “Ethics Crusade” launched by Tol.
Tol’s propensity to piously pontificate about Cook et al does, however, make my blood boil. I expect more professionalism from someone of Tol’s stature.
John Hartz: Actually there was a thread on ClimateBall ™ and it was very very informative.
See, logical technical people are used to arguing to convince others. That is not so much what is happening to us logical folks when discussing Climate Change.
How do you have a discussion when the people you are engaged with are cross checking you with illegal tackles? In many cases those people have no knowledge of how to engage in this kind of discourse. In our case, neither do we apparently… (cross checking, illegal tackles)
http://thechive.com/2011/05/04/next-time-you-listen-to-a-debate-keep-these-words-in-mind-video/
Willard is merely observing the meta-game. And he calls fouls on both sides of the net. (Although I notice that he seems pretty interested in the Tol stuff.)
Are you up on your Gish Gallop Defense? What if you encounter someone JAQing counterfactuals? How about SPAMming FUD?
.
.
.
.
.
At the end of this thread, there will be cake!
AnOilMan: Having been a member of the Skeptical Science, all-volunteer, author team for five years, I am very familiar the techniques you have listed.
Chad stated… “Of nearly 12,000 abstracts analyzed, there were only 64 papers in category 1 (which explicitly endorsed man-made global warming). Of those only 41 (0.3%) actually endorsed the quantitative hypothesis as defined by Cook in the introduction. A third of the 64 papers did not belong.”
And if we apply the same logic and standards to the explicit rejection papers you get a figure of about 3 papers. Or, 0.003% of the abstracts analyzed.
Nice work, Chad.
Richard Tol: What (perhaps who) prompted you to critique Cook et al almost immediately after it was published by ERL?
I ask becasue Cook et al doesn’t seem to be in your wheel-house.
John Hartz: I didn’t learn it that way… By dealing with trolls so much at desmogblog, I became one. They don’t spout their crap anymore, and they leave me alone.
ATTP said… “Either redo the study to show that the results are wrong, or prove that they behaved in a way that wasn’t ethical.”
I was actually calling on Richard to do this over a year ago when he first started complaining about TCP. And I’ll restate it here again…
Richard Tol, if you think there is a problem with Cook et al 2013 then redo the research. Stop blowing smoke up everyone’s wazoo and do the work! You don’t even need to rate 12,000 papers! You know that was over-kill and that you can get a statistically significant result with a much smaller sampling of the available research.
With all the time you’ve applied in your irrational attempts to destroy Cook13 you could have done the research 10 times over. This speaks volumes about what your motivations are. It’s not about getting the research right. It’s about the fact that you don’t like the results. In fact, please just be honest for once and admit that you want to do everything you can to avoid the actual results.
AnOilMan: Prior to hooking up with SkS, i spent a few years patolling the comment threads of MSM articles on climate change.. Back then, I used the name “Badgersouth”. I also enmgaged climate denier drones on DeSmog Blog.when it was infested. My biggest nemesis in the “blog wars” on USA Today was the infamous, netdr.
> Although I notice that he seems pretty interested in the Tol stuff.
Start here:
http://rabett.blogspot.ca/2013/05/tol-erasion.html
For instance:
https://twitter.com/nevaudit/status/339481724499161088
I think this may provide an answer as to how Richard has been pulled in this series of episodes. What if I told you that Richard simply wants more data to fuel his duel against Dana? We already have circumstantial evidence.
Let’s add a second rule of CLIMATE CLUB ™:
CLIMATE CLUB ™ is a knife fight, so the first rule of a knife fight applies: Bring a gun.
Rob –
==> “It’s about the fact that you don’t like the results. ”
I’m struggling with the logic there. If Richards says:
then why would his focus on Cook et al. be “motivated” by not liking the results? It seems that he, essentially, agrees with the results.
I try to avoid conclusions about someone’s motivations. But if we consider biased reasoning, as opposed to motivations, it seems to me that Richard has biases related to the interpersonal and tribal aspects of the climate wars. Perhaps he’s locked into a tribal struggle, and his biases lead him to believing that he is taking on a noble cause and defending science against poor research ethics and other concerns that he has?
John Hartz: Here we are! Say Cheese!
http://desmogblog.com/santorum-calls-global-warming-hoax-suggesting-full-fledged-climate-conspiracy-theory
.
.
.
And remember, there will be cake!
> Although I notice that he seems pretty interested in the Tol stuff.
Start here:
http://rabett.blogspot.ca/2013/05/tol-erasion.html
> It’s about the fact that you don’t like the results.
Circumstantial evidence for an alternative explanation can be seen by the twitterstorm between Dana and Richard, e.g.:
https://twitter.com/nevaudit/status/339501742054047744
Joshua… To me, none of what Tol does makes sense. He’s also stated that his purpose is not to be constructive, but to be destructive. That strikes me as a demonstration of disingenuous motivation.
JasonB –
==> “I think it’s long past time we gave up caring what the dismissives think. There’s not enough of them to matter if the disengaged become convinced — and that’s who the real target is.
Right now there are mainstream voters who don’t watch Fox News but who think that there’s a genuine debate in the scientific community and that the smart thing to do would be play it “safe” and wait until we’re more certain about what’s going on before doing anything drastic. If they come to understand the truth, and learn to recognise the tricks being played to try to keep them in the dark, then the dismissives won’t have a chance. Right now they have a disproportionate influence on policy because the overwhelming majority are ignorant of what they’re up to. The “skeptics” are right to be concerned.
IMO, of course.” <==
How do you define "dismissives?" I think that there are a lot of people who are quite convinced about the level of risk associated with ACO2 but who know little about the underlying science. I would put most Americans into that category. I don't know how you draw your line between "dismissives" and disengaged is unclear.
Certainly, there are many mainstream voters who watch Fox News, read the editorials in the WSJ, listen to Limbaugh, etc. Those audiences are massive. My guess is that there are relatively few people who are confused about the prevalence of view in the scientific community who aren't identified as Fox News watchers or with an ideology that is congruent with Fox News watchers.
Look at this poll finding:
Those data suggest that there are many who are not aware of the prevalence of opinion in the scientific community, but who view themselves as "well informed." Someone who thinks they are well-informed are more likely to dismiss information that contradicts their view than to change their views with the introduction of more information, IMO.
And look at this poll finding:
Which to me, suggests a similar pattern. At that time, quite a few folks identified with the Tea Party. I'd guess that virtually all of them are pretty convinced that there is an AGW cabal. They are not simply "disengaged," and waiting to be persuaded by Cook et al.
And look at this poll finding:
I think those data do suggest that there is some room for your perspective to be correct. There is evidence there that a sizable % of Dems and Indies, who presumably might be open to persuasion, are simply being deceived about the level of agreement in the scientific community. So I agree that the answer "remains to be seen." but I think that most of those who are unaware of the level of agreement in the scientific community are not open to persuasion – and so we're not talking about a group of people that is large enough to swing support for various policies to a large degree. I also think that persuading those who might be more likely persuadable are likely influenced by short-term weather phenomena, the state of the economy, etc., the basic uncertainties of assessing risk that has a long time horizon, than they are by the battles over the accuracy of Cook et al. or by more information about the amount of agreement in the scientific community.
But as you say, IMO, of course.
Willard: That was a good read.
Richard Tol, many of the journals that publish papers that support Climate Change Denial are junk. That’s why much of this crap isn’t included in the Cook study;
http://www.populartechnology.net/2009/10/peer-reviewed-papers-supporting.html
The papers supporting denial are sandwiched in there with Dog Horoscope Statistics, and Analysis of Potential UFO Landing Sites. Its not that I have a problem with that, but those would not exactly be journals that A) have the credentials to understand the material, let alone critically analyze it, B) Don’t contain material salient to climate change, and C) would seriously dilute your sample size.
If you want to open up to that kind of fun, read this;
Click to access BieberFever.pdf
I wonder if someone could model Tol Fever?
AOM –
Talk about a good read… this is beautiful:
==> “The inclusion of a paper in this list does not imply a specific personal position to any of the authors. While certain authors on the list cannot be labeled skeptics (e.g. Harold Brooks, Roger Pielke Jr., Roger Pielke Sr.) their paper(s) or results from their paper(s) can still support skeptic’s arguments against ACC/AGW Alarm.
Indeed, they can. That’s the beauty of “skepticism.” Anything and everything can still support their views*. No wonder Bob, Luboš, Joanne, and Willie are so impressed.
*If you squint just right.
AOM you have invoked Poptech. I can only assume you bet the over on this threads comments because if he arrives there will be no end to it.
Joshua
However, we may have some skill in modeling Tol Fever…
Sheece – another 164 comments at Rabbet’s warren.
I wonder what number you’d get if you added up all the time spent writing comments devoted to discussing Cook et al. – the findings of which are bleeding obvious?
Think of the human capital represented by all those people-hours, and the opportunity cost. What might have been accomplished had that time been spent on something constructive (not that it would have – it would just have likely been spent throwing another flavor of Jello-O).
It’s staggering.
pbjamm: There comes a point where you say to yourself Whiskey Tango Foxtrot. How can it go this long with so little being offered from these guys? Is this really the best they got?
Why are we even considering this guy as seriously? This is much ado about nothing. A real scientist would have just published the article rather than trying to get some media time with a blog post. Given his patronage with GWPF, I think we have a better understanding of his motivations.
Better yet, don’t feed the Tol.
Besides afeman already said he did the analysis of 283 papers and was using way too much cocaine at the time to be accurate; 🙂
.
.
.
On the plus side there will be cake at the end of the thread.
> Is this really the best they got?
I’m sure they ask themselves the same question.
Let’s add a second rule of CLIMATE CLUB ™:
CLIMATE CLUB ™ is a knife fight, so the first rule of a knife fight applies: bring a gun.
Willard: Re this thread, i’d say that ICBMs have already been l;aunched by both sides — although ATTP and Rachel to do a good job of keeping the conversation civil.
In many respects, Tols’ attempts to bring down John Cook via Cook et al are similar to the Republcans’ attempts to bring down Hillary Clinton via Benghazi,
Since Tol has been allowed to voice his concerns about the ethics of the authors of Cook et al on this thread,, I am am compelled to voice my concern about the ethis of Tol’s cozy relationship with the GWPF.
One of Shollenberger and Tol’s big concerns seems to be with the sheer number of ratings done by the most active 12 raters. They make the classic mistake of not attempting to put the numbers into some kind of perspective.
I did a quick time study on the SkS rating system and found that it took me about 3 1/2 mins to rate each set of 5 abstracts. Giving a very generous allowance, you could say that each abstract takes about 1 minute to rate. That means the group of 12 raters, who did ~26k ratings, put in something like 36 hours of total work each (on average).
No doubt, there was a lot of time put into rating the papers. But the work did take place over, if I remember correctly, something like 6 to 8 weeks. It was hardly a Herculean task. Far more time was spent compiling and evaluating the resulting data, not to mention writing the paper.
AOM you have invoked Poptech.
Indeed. Whenever someone has linked to Poptech previously, he has turned up here to comment and things don’t always go so well. We might be lucky this time but it would be good if in future people could link to an archived page instead. Don’t invoke the Poptech! We should put this in the moderation policy 🙂
The much crowed joining of Bengtsson (bringing down the GWPF’s average age) has been reversed, http://www.staatvanhetklimaat.nl/2014/05/14/bengtsson-resigns-from-the-gwpf//
McCarthy has been invoked. The man has obviously never had his email address published by Watts or Morano.
Can anyone pinpoint the exact time that we crossed over into the Twilight Zone on this thread?
When do we get our next meal?
Pass the coffee please.
To be fair, I invoked a food fight and Justin Bieber too… I think 10,000 screaming girls flinging Bieber Jello would be preferable to this thread.
> Another 164 comments […]
Not so fast:
http://rabett.blogspot.ca/2013/05/memorial-day-puzzler.html
There are many other threads like that. Dozens. Too bad I deleted all my research on this.
Does that mean we can have cake?
May I suggest: Don’t invoke the Poptol. For similar reasons.
Now, will someone put this thread out of its misery so Tol and his minions can refocus on traditional pastimes like the deletion of minus signs?
Bang!
[Reloads]
Where did everyone go? It’s awfuly dark in here.
I dunno… cliff? Lemmings?
Hey, how would Richard Tol see the movie Frozen?
I don’t see why the thread needs to end, but threads do end naturally if you run out of things to talk about.
The audit never ends.
I’m not going to close the thread. It’s not really up to me anyway. We can listen to music instead:
That’s not the right album. I meant to post this one:
This is what I’m listening to right now. 🙂
The sun never sets on an ATTP comment thread!
Pip. Pip. Cheerio.
This neverending torrent of baseless accusations might be frustrating for scientists, but let’s take a moment of silence to remember the true victims here:
Listening to music certainly would be more productive than the person-hours wasted on all these “concerns” stuck in a quantum superposition: simultaneously important enough to whine about for months and hack into private websites, and at the same time not important enough to motivate the “concerned” to attempt even a token independent verification.
“I will not be misdirected by denial. I will give my attention to the people hanging on for their survival.”
DS: Well said!. Bravo!
And everybody knows…
That’s great, OilMan.
Here’s another fave of mine:
Here’s an ode to Anders… Since no one knows what he looks like, and… He got hair down to his knees…
Come Together
http://www.albinoblacksheep.com/flash/cometogether
And somehow this one fits…
Red Rider – Lunatic Fringe
Ah, um, well, that’s terribly kind of you Eli but the thing is you see, ah, er, anyway, what absolutely splendid whiskers you have. You really must tell me your secret. And the weather today: simply glorious. Goodness me – look! A raven!
Joshua:
Sorry, I thought everyone was familiar with Yale’s Six Americas surveys. The results of the most recent one are in this book chapter.
I mis-remembered the significance of the Disengaged, though; they’re actually only 5%, whereas the Alarmed (16%), Concerned (26%), and Cautious (25%) collectively form a strong majority of 2/3 of the American population. “Dismissives”, by comparison, represent just 13%, despite appearing to account for a much larger proportion of the elected conservative party members in both Australia and the US.
But my point stands — there’s no point wasting time trying to convince those who have proven themselves impervious to reason; we just need to convince the Concerned and Cautious, who have no a-priori reason to disbelieve the science but who have perhaps been mislead about the certainty of the science due to the media portrayal.
Figure 5 of the 2012 report summarises nicely the point I was getting at; 37% of the Concerned and 46% of the Cautious think there is a lot of disagreement among scientists about whether global warming is even happening! Given that they are already Concerned/Cautious even with that misunderstanding, it seems that papers like Cook et al have a good chance of making those two groups in particular — representing 55% of the population — more engaged and maybe, just maybe, reduce the discrepancy in representation between conservative politicians and the wider population.
In other words, I think the only reason that Dismissives have been so successful in gaining parliamentary representation way beyond their relative proportion is because they are very politically engaged and the silent majority hasn’t noticed or cared enough to do anything about it.
This is why I think the paper is important.
Footnote: Comparing the 2012 figures with the 2013 figures, it seems that Alarmed has gained three percentage points (from 13% to 16%), Cautious has dropped four (29% to 25%), Disengaged has dropped one (6% to 5%), and Dismissive has gained three (10% to 13%). (Some rounding error there!) Concerned and Doubtful remained the same. But the margin of error is +/- 3 percentage points (95%) so it may mean nothing.
JasonB: Excellent comment. Perhaps ATTP will let you transform it into a guest OP.
Out of curiosity, has a survey similar to the Yale survey cited by JasonB been conducted in the UK? OZ? Canada?
For the record…
Although I’ve done a lot of horsing around on this thread, I’ve also benefited from particpating in the serious side of the dialogue, Kudos to ATTP for creating a forum where the discussion is civil – at least for the most part,
Beam me up Scottie!
If Jason is willing, it would make an interesting post.
JasonB,
puzzling isn’t it that the import of the study isn’t really addressed by the pseudoskeptics? OTOH imagine how different the world would be if the ‘low information voter’ was treated with the same scorn as Clive Bundy…
> puzzling isn’t it that the import of the study isn’t really addressed by the pseudoskeptics?
Auditors might think of cleansing, fatwa, or even better:
The “other side” may or may not be conflated with the dark side.
***
The distanciation in that sentence is noteworthy:
It is a fact.
The fact that one.
One has to reach.
To figures for analogies.
The fact one has to reach to analogies tells you.
It tells you about the other side.
The other side that is turning itself into.
Now, where’s Ross in all this?
Interesting WIllard…
http://klimazwiebel.blogspot.nl/2014/05/lennart-bengtsson-leaves-advisory-board.html
The fact is that Lennart Bengtsson’s first complaint is that he couldn’t collaborate with other scientists. Apparently that means this is the majority of people he works with in meteorology.
This also speaks on several levels…
First is that a lot of scientific work is collaboration with others.
Second is that not all scientists do collaborate or work with teams.
Third if you don’t work with others, you are vulnerable to limiting your thinking.
Fourth its hard to work with others if you say its all bullshit.
How many people does Tol collaborate with?
Click to access Tol_impacts_JEP_2009.pdf
Hmmm…
AoM,
What confuses me about that statement is that if he feels that he has to resign or else he won’t be able to work with his collaborator’s anymore, why did he imply that they were McCarthyites? If some people sent him emails suggesting that he might be making a mistake joining the GWPF Academic Advisory Board, they might be rightly ticked off by his characterisation. Of course, I have no actual idea of what sort of exchanges took place, but it’s hard to see how collaborations could continue whatever happened – either some of his colleagues behaved very badly, or he’s just insulted them very badly.
Judith’s crib has turned into one massive Godwin-a-palooza.
We can see analogies (not an exhaustive examination) to McCarthy (the most favored, and we are told was a “lightweight” in comparison), Goebbels, Sandinistas, Nazis, terrorism, Stalinism, Russian and Chines communists, Lysenkoism.
Thank god they got to Lysenkoism. The climate wars just wouldn’t be the same w/o Lysenkoism.
Joshua:Tell them they need to also include Calligula, Gengis Khan, Attila the Hun, etc.
Anders…
I bet he got all kinds of grief.
From nut job eco green freak hippies (these would be the death threats), to shunning by distant colleagues, to a genuine unwillingness to work with someone who’s politics run counter to his work. Its all conflated in that resignation.
A genuine unwillingness to work with him, may very well appear to be McCarthyism. I can’t say whether it really is. However, I can assure you that no one wants to work with the guy walking around saying all of their work is bullshit and lies. No one. I mean NO ONE.
John –
Those are good suggestions. I have passed them on. If anyone else has any more suggestions, I’ll pass them on a well.
Willard – the audit is above and beyond any one thread.
As above a thread, guthrie, so below.
> Of course, I have no actual idea of what sort of exchanges took place […]
As I suggested at Judy’s, AT, let’s get all FOIA on this.
Release the clowns.
Joshua says: May 10, 2014 at 11:17 pm “The question remains – if it’s so irrelevant (and anti-science) why is it so important?”
Why indeed. Any person’s reasons will differ from the next.
The 97 percent is irrelevant to me. It is a statistic — is there anyone here not familiar with the famous quote about lying and statistics? It’s a factoid. It could be ANY number and be just as relevant. That’s not to say I discount it or ignore it, I just recognize that it is a calculation based on a filtered set of papers. It surely means SOMETHING.
The issue therefore (as I see it) is not the actual number, 97, but the METHOD of its calculation and the secrecy surrounding it. It is refusing to disclose data and methods that creates intrigue and drama.
Way back in 2008 or 2009, had UEA simply coughed up some data, this whole debate would be limited to a few passionate individuals with burning interest in such esoteric things. But, no, as Will Smith describes it in “I, Robot”: “stupid smart people” made unwise decisions and in so doing focused the huge lens of popular interest on themselves.
But I’ll give Lewandowsky some credit. There’s no such thing as bad press. It is a risky game focusing the whole world on one Australian university after another. It puts the topic on the minds of millions of people that otherwise would not have cared.
What harm is there in University of Queensland in releasing all that data? Let the game begin and tie up a few advocates for a few years.
[Mod: let’s keep religion out of this, thanks]
Above, Tol claims that he now identifies the impacts of his criticisms of Cook et al, saying:
My, which indicated that Tol’s calculations would not be credible, was considered defamatory.
It turns out that Tol has a blog post discussing errors in Cook et al in which he finds an adjustment to the data of 7%, bringing the “dissensus rate” up from 2 to 9%. I assume this to have been corrected slightly in the final paper to be the 8% dissensus rating.
HIs method is fairly simple. First, from the information that initially 33% of category ratings disagreed (published in Cook et al), he determines that:
That, of course, is the average across all classifications, and treats all differences of classification, whether between two endorsing categories, or between an endorsing category and a rejection category, as equally errors.
Tol then takes the data on initial and final ratings in Cook et al and determines the average change for ratings that differ between initial and final rating. From this, together with his estimated error rate, he concludes that approximately 7% of abstracts rated as neutral should have been rated as rejecting the consensus, hence the large change in dissensus rate.
The problem is that there is no reason to think that each category has the same error rate, or that errors are equally likely to be toward or away from the consensus for each category, Indeed, it is impossible that they be equal. Errors for category 1 most be towards a numerically higher category, while those for category 7 must be towards a numerically lower category. Tol’s method, in other words assumes an impossible condition. It is known to be wrong from the get-go.
The correct approach is to determine the error rate individually for each initial classification; and the ratio of changes in classification individually for each initial classification. Doing the latter, we find that all endorsing or neutral categories favour adjustment to (other) endorsing rather than neutral categories by 98.32% or more. As that is a larger percentage than the final endorsement rate reported in Cook et al, it is impossible that the adjustments result in a large change in the dissensus rate.
Overall, correctly carried through it is likely that such corrections would result in a very small increase in the dissensus rate, primarily because neutral ratings had the lowest error rate so that there is a slight drain of endorsing to neutral ratings. This is unlikely to effect the dissensus rate by more than 1, or at most 2%. In his eagerness to find fault, Tol has engaged in shoddy maths of astonishingly poor quality. He has used simplifying conditions that were easily checked, and falsified, and indeed lead to impossible conclusions if applied rigorously.
Given the frequency with which he has resorted to such shoddy methods to try and find fault with Cook et al, at least twice having relied on impossibilities to carry through his critique, and given his refusal to significantly correct errors when advised of them by others, I think I can strongly defend my prior comment as not defamatory because true.
Michael:
1) John Cook has released “all that data” except for that which risks breaching confidentiality.
2) They have very carefully detailed their methodology. They only failed to mention one part of the methodology. That is that if original raters were unable to rate the abstract, either because it had no abstract, or they did not understand it, etc, it was given a rating of 0. That rating was then treated like all other ratings as per their described methodology except that if the rating could not be resolved the data was excluded (which was mentioned in the paper).
They did make some minor mistakes. For example they over stated the initial disagreement rate between ratings as 33% (it was actually 30.3%), but the methodology is sound and the results easily checkable from data they publicly released.
These facts will not, of course, interest you given your clear projections about “religion”.
Sorry, that should have been for 23.3% of abstracts, initial ratings disagreed.
[Mod: inflammatory]
“Way back in 2008 or 2009, had UEA simply coughed up some data, this whole debate would be limited to a few passionate individuals with burning interest in such esoteric things.”
That’s a very imaginative fantasy you have staked your recommendations on there.
Michael: Why does social change scare the bejesus out of you?
Michael is selling Happy Meals to all who would listen;
[Mod: name calling]
Michael: Does croporate takeover of the governments of the major countries of the world give you pause?
[Corrected version.]
Michael –
–> “The 97 percent is irrelevant to me. It is a statistic ”
Actually, I pretty much agree. What is of (somewhat limited, IMO) relevance is as Richard said:
==> “Published papers that seek to test what caused the climate change over the last century and half, almost unanimously find that humans played a dominant role. ”
You say:
–> ““stupid smart people” made unwise decisions and in so doing focused the huge lens of popular interest on themselves.”
What do you think about the wisdom of “skeptics” spending so much time arguing about the precise quantification of the prevalence of view among climate scientists when what Richard says is bleeding obvious? Are they stupid smart or just smart?
How does one focus someone else’s interest on themselves? Is it the focuser who is responsible for their own actions, or is the focusee the one responsible? Did the devil make them do it?
So in the end, you haven’t answered my question. You only say that the reasons might differ. But I’m sure that we’ve both read many “skeptics” arguing that “consensus” is irrelevant to good science. How many times have you read “skeptics” quoting MIchael Crichton about consensus? How many times have you read “But the ‘consensus’ said that plate tectonics was crap?” Judith has devoted many a post to the topic of how wrong, wrong, wrong is a focus on consensus.
So then, since there is so much widespread argumentation from “skeptics” about how irrelevant and anti-science they think the very notion of consensus is, why do so many of them spend so much time arguing about the precise quantification of the consensus? I don’t need all the possible answers. Just one logical or two logical answers would suffice. Or do you agree with me, that many self-described “skeptics” are actually not very skeptical?
[Mod: I think this is enough questions for Michael now]
Hey Richard complained that Scopus returned 20,772 papers that are relevant!
https://docs.google.com/file/d/0Bz17rNCpfuDNRllTUWlzb0ZJSm8/edit?pli=1
And Poptech claims that there are 1350 papers endorsing natural causes… or 6.5%! (NOTE: I have no clue how many of the 1350 are in scopus.)
So… Richard is claiming 94% endorsement for AGW!
Good man Richard Tol, good man. Thanks for backing us up 100%.
“Scopus”?
AnOilMan, are you claiming Tol agrees with Poptech about that 6.5%? Do you really want to damn him that much?
John Hartz, scopus.
I’m goofing around…
I don’t know which is worse, that Richard Tol employs the fallacy fallacy or that he has teamed with arch denier ideologue “Shub Niggurath”.
“The 97 percent is irrelevant to me. It is a statistic — is there anyone here not familiar with the famous quote about lying and statistics? ”
Yes, we’re certainly familiar with this popular anti-science, anti-intellectual meme. It’s strange to see someone offering it as a serious argument on a science blog.
@Tom Curtis
You don’t need to assume that every rating has the same error rate, the same bias and the same bias correction, and indeed I didn’t. You can check my computations at my website.
Richard,
I got the impression from Tom’s comment that this is essentially what he’s done.
“A genuine unwillingness to work with him, may very well appear to be McCarthyism. I can’t say whether it really is. ”
Really? Who would the McCarthy be here? That is, the powerful political figure who was forcing people to testify against their friends and colleagues lest their own lives be destroyed? Unwillingness to work with those tagged by McCarthy was not “genuine” but rather coerced. No, the reaction that Bengtsson is reporting reflects heartfelt belief. Ironically, it’s yet another demonstration of consensus.
“You don’t need to assume that every rating has the same error rate, the same bias and the same bias correction, and indeed I didn’t. ”
Tom wrote about *categories*, not ratings. And the assumption appears to be implicit. Of course one need not explicitly assume things, one can simply fail to correct for impossible conditions.
The finding that as much as 12% of peer-reviewed papers dissent from AGW, given the scientific evidence, would be astounding and alarming if valid. What could be the cause of such a high level of erroneous claims in scientific journals?
Richard Tol, I did check your web site. Further, I linked to the relevant page, and quoted it extensively. If you have some other post in which you perform the calculation more accurately, please provide the link so I can update my criticism, or retract it if appropriate. If you have discussed the 12% dissensus, I would appreciate a link to that as well (or is it just the compounded dissensus from all other factors)?
Jibalt, I was talking about the different categories for rating consensus (ie, 1) Explicit endorsement with quantification, 2) explicit endorsement without quantification, etc).
ATTP: I have a gut feeling that “micahel” is a drive-by and will not return to respond to questions about his/her tome.
I also wonder if “michael” might not be a sockpuppet. His/her rhetoric is vaguely familiar to me. I do not, however, associate it with the name “michael.”
John,
You may be right that Michael won’t return. It doesn’t appear to be a sockpuppet, but I obviously don’t know if the same person comments under a different name elsewhere.
ATTP: Thanks for checking.
Sorry about posting a long string of separate comment/questions addressed to michael. I’d post one and another one would immediately pop up. In retrospect, I should have combined them into a separate post.
BTW, does Word Press offer an option whereby a commenter can edit his/her comment after it has been posted? Wht about spell check.
John,
What I have doesn’t allow that. It might be possible if I paid for an upgraded account, but I haven’t done that …. yet.
Willard – that leads to further pointless specualtion; which translation of the Emerald Tablet do you prefer? One is as simple as “As above, so below”, which clearly identifies the exalted and the lowest as being the same or very parallel, whereas another goes “What which is above is like to that which is below”, and like simply isn’t as strong a term.
Or maybe the audit is all encompassing, beneath and between all reality. Or rather it is outwith reality altogether…
John Hartz (May 16, 2014 at 4:11 am) “Michael: Why does social change scare the bejesus out of you?”
THANK YOU for a worthy question!
Short version:
If you were three steps from the summit of Mount Everest, would you take a step in a RANDOM DIRECTION?
Not me! The closer I get to the top, the more carefully I choose my steps and *I* will do the choosing, thank you very much — although I’ll listen carefully to your wise suggestion that might help me avoid a fatal misstep.
Longer version:
LOGIC:
If your worldview is binary consisting of only two states, good and bad and…
1. If you consider that you are now in a bad state, ANY change can therefore only be “good” toward a good state OR
2. If you consider that you are now in a good state, any change whatsoever must necessarily be to a “bad” state.
PRACTICAL CONCLUSIONS:
“1” describes people that fear the consequences of global warming; for them, change cannot be worse than failing to act and failing to act is itself a culpable act. Therefore change is GOOD.
“2” describes people with a doubt either about the global warming itself, or the dire consequences thereof or the claims of certainty. For this group, change is BAD.
Nuanced people will of course add nuances and probabilities making things complicated but ultimately it boils down to the same calculation. People that like to argue (me for one) will sometimes quibble over details largely for its entertainment value since really it has nothing to do with the truth of any claim.
SCENARIO
The western hemisphere is populated by the result of of millions of years of human evolution combined with its isolation from most of that development still contained most of its natural resources.
That’s the baseline — starting North American societies where Europe ended. Starting fresh with a huge natural barrier to allow liberty, science, manufacturing to develop without imperial adventures from the rest of Europe.
Taking the best and generally avoiding the worst of the rest of the world’s social experiments, the United States became, and remains, a beacon of excellence in most areas of human experience. Your mileage may vary of course, but with a few exceptions and then only in specific categories of life that you happen to prefer, it is so.
So this grand, unspecified “change” that you have in mind — why do you think you will succeed where so many have failed? What is so much better if it were done your way?
WARNING, WILL ROBINSON!
In this seemingly pointless discussion of the exact magnitude of consensus, some seem to forget that what is CERTAIN is exhaustion of fossil fuel. Really, I worry a lot more about this than global warming.
John Hartz (May 16, 2014 at 11:39 am) “His/her rhetoric is vaguely familiar to me.”
I am uncommon so it probably is me that you remember although it would have to have been on different topics, different places since I seldom — maybe never — debate the actual science. I am really not equipped for that. I am still in a learning mode but many claims are either incomplete or just plain wrong. What an ordeal these past 6 years have been trying to “smarten up” as we’d say in the Navy before blurting out anything just to appear foolish.
I love a good argument. Knowledge is defined by content but refined in its EDGES. Through arguing or discussion, I discard the fuzzy bits of my own thinking — or at least mark them “fuzzy” and needs further analysis.
You are my sharpener, the hammer that knocks off the slag from the metal of my knowledge. I could wish for more civility from some people but it is what it is; those that cannot stand the heat should not be in the kitchen. Your expectation of what you think is slag and mine probably are not the same — so let the hammer fall and we will see what survives it.
Michael –
Did you just miss my questions? Haven’t gotten around to answering them? Don’t think they’re worthy of your attention?
Joshua (May 16, 2014 at 4:18 am) questions will be in quotes.
Quoting Richard: ==> “Published papers that seek to test what caused the climate change over the last century and half, almost unanimously find that humans played a dominant role. ”
Agreed; but maybe we place slightly different emphasis on what this MEANS. If a thousand scientists are hired to do exactly that, unanimity is to be expected. I cannot distinguish from that possibility versus a thousand scientists free to study and report whatever they like, individually and independently arriving at the same conclusion, which would of course then be taken much more seriously.
Consider three astronomers each discovering an asteroid on collision course with earth. Their independence from each other is crucial to acceptance of the risk. If they are not actually independent then you don’t really have three samples (in my somewhat limited understanding of that sort of thing).
Linguists operating independently of each other arrive at the same or similar translation of the “Rosetta Stone” can be taken to validate not only the translation, but the capability of each translator. But if they are related in any way, including just by having had the same professor of Egyptology, they are not independent and they could ALL be wrong (but far from assuredly so).
Since the topic of discussion is ENORMOUS, great claims require great proof.
MORE to come on your questions rather than one big huge post burdening the moderator.
More in reply to Joshua
“You say: –> ‘stupid smart people’ made unwise decisions and in so doing focused the huge lens of popular interest on themselves.’ What do you think about the wisdom of “skeptics” spending so much time arguing about the precise quantification of the prevalence of view among climate scientists when what Richard says is bleeding obvious? Are they stupid smart or just smart?”
Hmm — it is bleeding obvious if you read the actual Cook report (which I finally did) but I strongly suspect the intention of this whole thing was that the entire report would NOT be read by millions of people and only the seemingly ultra-precise and scientifically accurate 97.2 percent (or whatever it is) made known via the abstract and further condensed in various news media outlets, gradually morphing into “97 percent of ALL SCIENTISTS” say…
Let us imagine a huge court of law, with the Earth’s population as both jury and also the accused. The accused (human beings) have been accused of poisoning the Earth. The defense needs only create reasonable doubt to acquit. That is ALL it needs to do.
It doesn’t even matter what detail is used — the doubt must be in the ACCUSER.
Doubt now exists. Certainty is not certain. But I am not a representative sample of humanity. I understand that it is actually quite certain among the rather select and small group of published scientists on the topic. That’s okay — but is not the “accusation” that was presented to the world.
In that court of law, the prosecutor must be precise in the accusation, and if THAT accusation is not proven, the defendant goes free — even if he is actually guilty of something ALMOST what was declared.
So it is here. The accusation is specific and precise. The details admit to some fuzziness but the jury doesn’t see that part. They see, and must see, a specific accusation so they can decide whether it is “true” and if so spend billions of dollars and roll back evolution (or some such thing).
Well, guess what — it might not be “true”. It might be 96 percent. But that wasn’t the accusation. The defendant might “walk” because of that tiny little detail.
I don’t know who is the prosecutor, and I don’t know who is the defender. Behind warmism and skepticism are mighty powers and big money. But nobody can beat 20 billion dollars a year just in the US for global warming and the welfare state it creates and THAT is why it has become aligned with a certain political party. Why should Koch brothers fight it? Exxon can make and sell windmills if they wish, and they would, if there was money to be made in it.
Whups, just erased part of your question so I’ll post this and go find the rest of it…
Joshua says: May 16, 2014 at 9:46 pm Michael -Did you just miss my questions? Haven’t gotten around to answering them? Don’t think they’re worthy of your attention?
Sorry! I was reading this column from the BOTTOM UP and the blizzard of responses, while thrilling to me, took me by surprise. It takes me a long time to reply because I brainstorm thoughts to the text entry window, then prune it, then add more, then make a total mess of it, then forget the point I was trying to make, then eat a snack and finally whittle it down to a few paragraphs that might answer the question, be worth a few minutes of your life to read, and not come back to haunt me much.
“Agreed; but maybe we place slightly different emphasis on what this MEANS. If a thousand scientists are hired to do exactly that, unanimity is to be expected…”
Conspiracy ideation. A well-worn and accurate phrase.
jibalt says: May 16, 2014 at 5:56 am (quoting me) ‘The 97 percent is irrelevant to me. It is a statistic — is there anyone here not familiar with the famous quote about lying and statistics? ‘
Yes, we’re certainly familiar with this popular anti-science, anti-intellectual meme. It’s strange to see someone offering it as a serious argument on a science blog.”
There is no WE. You do not know who else is familiar with this, but I am glad you understand it and it saves us (you and me) a bit of time.
What I find a bit odd is that you consider statistics to have anything to do with science — not directly. It has to do with the massaging of results. Science is in the data, and the methods used to produce that data. Science is in kilometers of ice cores and the chemical assays that result from all that.
97 percent — shucks, that’s just a number, with a tiny bit of filter tweaking suddenly it is 99 percent, but that number is “taken” and has a memetic collision with the Occupy Wall Street Movement. It is also a nice prime number and ends with the lucky number “7”. It is a very auspicious number.
But not very scientific. Not “anti-scientific” just not scientific.
“Since the topic of discussion is ENORMOUS, great claims require great proof.”
The topic is only enormous due to the political and societal consequences. As a matter of physics, the claims are relatively minor and are uncontroversial: CO2 is a GHG, the water vapor feedback is a basic consequence of well-understood physics (look at a lenticular cloud, sometime), etc etc.
The great claim that requires great proof is that for some reason, what’s known of atmospheric physics magically goes poof! into the dustbin once physics is applied to CO2 released by the burning of fossil fuels.
Why should that be true?
“But nobody can beat 20 billion dollars a year just in the US for global warming…”
I think we have a cut-and-paste artiste, here.
Michael, in your own words, would you care to explain how that 20 billion dollars is actually being spent “for global warming”? Would you mind doing a teensy-weensy bit of research to break that down for us?
Would it surprise you to learn that the number you have cited is bogus?
ON THE IMPORTANT OF INDEPENDENCE
Note to all — I occasionally challenge the use of “we” by persons assuming to the existence of a group for which such person has suddenly become an authoritative spokesperson. I might gently question how many of you are in there.
“WE” is BAD. It shows a lack of independence. If 500 researchers use the same model and data, well they’d better all get the same result, but it is still just one result — one SAMPLE. I don’t know who is persuaded by mere consensus (well, my brother for one…).
Quite frankly; one good scientist somewhere is ALL IT TAKES. That’s all you had for a hundred years on Plate Tectonics. Of course he wasn’t successfully persuasive in his lifetime and neither was he asking for hundreds of billions of dollars.
Anyway, on with answer more of the excellent and challenging questions put to me.
Michael: Are you a speech writer by trade?
Michael:
“Quite frankly; one good scientist somewhere is ALL IT TAKES. That’s all you had for a hundred years on Plate Tectonics. Of course he wasn’t successfully persuasive in his lifetime and neither was he asking for hundreds of billions of dollars.”
Laughable. The proposed mechanism for continental drift – the continents plowing through the ocean floor like icebergs through the sea – was dead wrong, and the physical impossibility of this mechanism being responsible for the jigsaw puzzle-like fit of the continents was a big reason for the hypothesis being met with a great deal of skepticism.
Plate techtonics is an extremely different mechanism. Not even close to the proposed mechanism of continental drift. Plate tectonics would’ve been discovered without Wegener, and the implications immediately understood without Wegener.
“neither was he asking for hundreds of billions of dollars.” – extraordinary claims demand extraordinary evidence. Please, in your own words, just where who is asking for “hundreds of billions of dollars” and where are they going?
Also, the work that led to the discovery of the mechanism of plate tectonics (i.e. the discovery of sea floor spreading) required very sophisticated remote sensing equipment lowered thousands of fleet to the ocean floor in an era where electronics technology was much more primitive and expensive than today.
You don’t really know much, do you?
Michael’s next move will be either the Galileo Gambit or the Feynman Finesse.
Michael:
‘“WE” is BAD. It shows a lack of independence. If 500 researchers use the same model and data, well they’d better all get the same result, but it is still just one result — one SAMPLE.’
Thank God this strawman argument isn’t even close to an accurate description of the state of atmospheric physics or climate science.
Is there any particular reason why you waste our time with hypotheticals that bear no resemblance to reality?
Micheal –
–> “Agreed; but maybe we place slightly different emphasis on what this MEANS.”
I doubt it. It means what it mean. It says what it says. There is a vast prevalence of view in the peer reviewed literature. That isn’t dispositive. But IMO, it is evidence. It speaks to probabilities.
I appreciate the lengthy response, but I’m afraid that I got quite lost shortly into it, and a quick reading of the rest left me thinking that you didn’t really answer my questions.
I’ll try reading again at a later time, and if I can decipher any answers I’ll speak to them.
Michael, you write:
“So it is here.”
No, it’s not on several different levels, and I suspect that’s contributing to some of your muddled thinking.
Firstly you might ponder that criminal courts in most jurisdictions have the standard of “beyond a reasonable doubt” rather than “beyond all doubt”. Criminal convictions occur in the presence of unreasonable doubt – as they should because unreasonable doubt can never be eliminated in matters requiring inference from evidence. Then there are civil proceedings don’t even require that level of confidence when they insist on “the balance of probabilities” to make a judgement, i.e. “more likely than not”. Since science and economics both rest on inference from evidence, this suggests that your application of the analogy is flawed.
In addition, the way that you use the analogy to advocate certain risk management responses is foolish. If my doctor tells me that it is merely likely that my cigarette smoking is the dominant cause of my serious heart disease or merely likely that if I continue I’ll suffer very serious additional medical consequences, I don’t wait until that claim is proven beyond all doubt (or even considered “highly likely”) to stop smoking unless I’m an idiot in denial. And I don’t even need that much non-doubt! The same response applies if the doctor merely tells me that I’m at risk of developing serious disease from smoking. (And if I weigh the “costs” of giving up smoking in deciding how to respond to that information, which I might feel bring me relaxation or feeds some social or business networking or put me through a lot of unpleasant experiences when I kick the addiction, then I’d be an idiot for doing that as well.)
In the real world we face risk management decisions every day, and for just about all of those decisions the information available is insufficient to eliminate all reasonable doubt, let alone all doubt. Despite that, in the face of plausible risks whose impacts are considered undesirable one must still choose a response. (And there’s no “I’m not going to choose yet” option! Choosing to “delay deciding” is choosing to continue on the current course which is choosing a response. It’s just that those are almost always suboptimal choices, sometimes very profoundly so.) And when the potential impacts are unimaginably severe prudent risk management requires moving heaven and earth to mitigate the possibility. You appear to be advocating a suboptimal risk management strategy for this issue that wouldn’t pass muster for just about any other issue.
And the analogy is foolish in another way. You’ve placed the burden of proof on the wrong side! The burden of eliminating reasonable doubt here is on those who wish humanity to continue conducting an experiment on the climate system that is unprecedented in human history. We have no experience of humanity and the ecosystem it depends on for economic and physical wellbeing under the climate that we’re driving hard to create. There’s entirely reasonable doubt that this will turn out well for us, especially given the population levels that are anticipated in the next century or so. If you applied your doubt-based reasoning to that concept you’d conclude that we should do just about everything possible to keep the climate within the envelope that most of modern agriculture evolved within, since we’re so utterly dependent on that industrial scale capability.
It would be prudent to ponder the factors that lead you to approach risk management so suboptimally on this issue, especially if you approach it much better on most other issues.
Continuing to answer Joshua. This one is difficult, a complex question that needs some context so please forgive quoting rather a lot of it so readers don’t have to scroll back to find it.
Joshua: “How does one focus someone else’s interest on themselves? Is it the focuser who is responsible for their own actions, or is the focusee the one responsible? Did the devil make them do it?”
At the outset I will assert that there is no such thing as a cohesive group sharing common ideas called “Skeptics”. The word simply describes anyone NOT in the “consensus” by reason of DOUBT rather than CERTAINTY (the latter I accept can be called “deniers”).
In a Venn Diagram, it is the AGW proponents that occupy a defined circle or area. It is impossible for me, occupying the upper left hand corner of the diagram, to understand or speak for someone down in the bottom right even though you label us both “skeptic” and really seem to think we have the same interests, thought processes, educational levels and commitment to the debate.
1. “How does one focus someone else’s interest on themselves?”
How indeed. The usual method is drama or conflict. Beginning writers are told there must be a conflict in the story. This aspect is revealed in an entertaining way in the movie “Rango”.
I would not be a bit surprised but what Lewandowsky himself leaks juicy tidbits to stir up some drama rather than let science return to its dusty closets where it usually is found.
2. “Is it the focuser who is responsible for their own actions, or is the focusee the one responsible?”
I am a libertarian and believe all persons make their own CHOICES. I do not know if that meets your definition of responsible. A child makes choices, but is she responsible for the consequences? A 3 year old sets her house on fire, is she RESPONSIBLE? No, she is the cause. The responsible person is the parent that allowed it to happen, but only up to a point.
During my Navy career I discovered I lack what others seem to possess, a sense of when other people have a hidden motive, or are lying. Consequently I judge by what you DO. WHY you did it might matter to a jury or to he-who-cannot-here-be-named but you might meet him in the next life if there is one.
3. “Did the devil make them do it?” Maybe, but if so then the devil is responsible for both the choice and any resulting consequence. Theologically, and I’m taking a chance here with the moderator, if this was even possible human beings would get off “scot free”. It must be possible to make choices and it must be possible to be “responsible”. That is my philosophy the “possibilty principle” that makes ME responsible for my choices.
4. “You only say that the reasons might differ.”
Exactly. My sample size is less than the Universe of possibilities. It depends on many factors why one might doubt the Dogma or just the nit of the magnitude of certainty.
4.a “But I’m sure that we’ve both read many ‘skeptics’ arguing that ‘consensus’ is irrelevant to good science.”
Yes, those are “smart skeptics” (includes me). Being true is not the same as being effective. In my Navy career I was judged, and judged others, on the *correct* use of the English language, but also *effective* use — which at times required considerable crudity.
It may well be that among themselves, the elite of academia know with precision exactly how fuzzy any particular item is. Statistical methods exist to define bounds of certainty.
Consider Newtonian physics — accelleration of gravity 9.81 meters/sec^2 if I remember right. Is that correct? No, it is an approximation but works really well to estimate the height of a cliff by throwing a rock over and timing the sound of it hitting the bottom. To be precise you’d also throw in the Lorentz Transformation invoking the speed of light and relativisitic phenomenon.
But do I tell all that to a student? No. It would be a disaster. Still, it would be honorable to mention that “Oh, by the way, when you get into college you’ll learn that this value can be bent slightly in some circumstances” leading him perhaps to write a science fiction story about bending it a LOT.
4.b “How many times have you read ‘skeptics’ quoting MIchael Crichton about consensus?”
Never. That is to say, I wouldn’t know that it was an MC quote. Andromeda Strain comes to mind — yep, good memory. I didn’t know that Jurassic Park was one of his. The movie incarnation was at least was a bit “Luddite” and the movie incarnation of Andromeda Strain showed the tedium of the scientific method and how easy it could be to miss an extremely important detail that changes everything. But with many eyes on the problem this is less likely.
4.c “How many times have you read “But the ‘consensus’ said that plate tectonics was crap?”
Never, but I have *written* it quite a few times. Maybe you are thinking of me? But there’s more to the story — I *accept* that it was reasonable for the consensus to be wrong. But it WAS wrong. Why do I engage in semantic tactics that I know to be incomplete and misleading? Because I am not the only one doing so.
Reminding people that *scientists* were wrong for nearly 100 years about that topic is just a reminder that human beings should not be assumed to be perfect, not even scientists, although culturally they do and are required to try harder to be correct, perfect, accountable, reproducible and so on.
The desired EFFECT of all that is BALANCE. Yes, it is balanced. Anyone can claim to be a scientist and the incredible “SciGen” scientific paper generating program shows that the process is not perfect.
For 100 billion dollars a year, y’all need to step just a wee bit *closer* to unobtainable perfection.
5. “So then, since there is so much widespread argumentation from “skeptics” about how irrelevant and anti-science they think the very notion of consensus is, why do so many of them spend so much time arguing about the precise quantification of the consensus?”
Speaking only for me, I challenge the “holiness” (purity, assumed absolute correctness, as if handed down from “on high”) of that number and not its value. Others obviously approach this with a different passion, but even Richard Tol is not significantly challenging the actual number (seems to me anyway), only the *possibility* that it might be different if “the secret” were known. That seems a bit unlikely but it might change by a few percentage points. With a bit of effort anyone can replicate the entire study (now THAT is scientific — repeatability!).
I do not and cannot dispute that approximately 97 percent of all papers that meet certain criteria agree that humans are at least half responsible for global warming. I could do that study myself. So can you. I have little doubt my result would be similar to yours. Ought to be identical actually. Change the criteria and of course the result changes. Change what it takes to get a government grant and then publish and the result will change. By how much? I have no idea; I could easier predict next week’s weather.
As I have mentioned, research can only show “what is”, it cannot show “what is not”. The problem is one of dissymmetry. You will ALWAYS get more proof papers than disproof papers — that’s my own theory anyway which I could demonstrate but have no time — just repeat Cook’s study but change the words “Climate change” to pretty much anything else. Who gets a government grant for disproving anything?
“do you agree with me, that many self-described ‘skeptics’ are actually not very skeptical?”
As to that, yes, it can hardly be otherwise. After all, what exactly is a “skeptic”? It is not a thing, it has no properties — the word describes a LACK of a thing while also not imputing resistance to the thing. Just doubt, uncertainty, question; traits I value in science and so should you.
I was so pleased with that final sentence I’m going to hang it out here all by itself:
My Theory on Proof Papers
You will ALWAYS get more proof papers than disproof papers. Who gets a government grant for disproving anything? I could demonstrate but have no time — just repeat Cook’s study but change the words “Climate change” to pretty much anything else.
Michael:
“At the outset I will assert that there is no such thing as a cohesive group sharing common ideas called “Skeptics”. The word simply describes anyone NOT in the “consensus” by reason of DOUBT rather than CERTAINTY (the latter I accept can be called “deniers”).”
No, so-called skeptics are not doubters, they reject climate science with absolute certainty.
dhogaza says: May 17, 2014 at 1:08 am “Thank God this strawman argument isn’t even close to an accurate description of the state of atmospheric physics or climate science.”
It is very relevant. The stated number of climate models (IIRR) is 32. However as each is itself derived from an ancestor, the shared code is substantial and so they are not truly independent. They are also not “clones”. If I remember right the effective number of independent models is around 6.
In the case of scientists, what exactly is a climatologist? Is there a degree in it? Who got the first such thing and who gave it to him? Briffa is a Dendrochronologist. He at least is expert in his field and it would be absurd for me to challenge his research. But he did not make his own conclusions– his research was used by others, one in particular, and the independence of that research collapses into a singularity: that of the EDITOR. (or, right here, blog owner).
“Is there any particular reason why you waste our time with hypotheticals that bear no resemblance to reality?”
How about “I was asked?”
You can always ignore me. I cannot write Twitter-sized responses (I try, really I do).
Michael:
“You will ALWAYS get more proof papers than disproof papers. Who gets a government grant for disproving anything? ”
Cold fusion.
The supposed link between vaccines and autism.
Next?
“Lotharsson says: May 17, 2014 at 1:38 am . Firstly you might ponder that criminal courts in most jurisdictions have the standard of ‘beyond a reasonable doubt’ ”
I’m pretty sure that is exactly what I wrote * (above, timestamp 1215 a.m.) and I suppose I should have specified criminal court. You are right that civil court requires usually only a “preponderance” of evidence. Either way, 97 percent is way beyond the requirement of preponderance OR reasonable — it is dang close to “certain” which is the entire purpose of the Cook paper.
* “The defense needs only create reasonable doubt to acquit. That is ALL it needs to do.”
“You don’t really know much, do you?”
Don’t you / do you questions cannot be unambigously answered.
I have stated often enough that I come here to gain knowledge. Whether what I possess is “much” I leave to your superior judgement.
Sometimes I can just ask for knowledge, but this outpouring of knowledge is as a result of my statements sometimes coming from ignorance, so I thank you for what bits of knowledge I may yet find in your comments.
Michael:
“It is very relevant. The stated number of climate models (IIRR) is 32. However as each is itself derived from an ancestor, the shared code is substantial and so they are not truly independent. They are also not “clones”. If I remember right the effective number of independent models is around 6.”
Bollocks.
But even if you were right, six is still six times larger than one, in other words your claim of a lack of independence was false, and you knew it was false when you made it.
This doesn’t bode well for an honest discussion, does it?
And, of course, models aren’t the foundation of climate science anyway. There are many independent lines of evidence that aren’t dependent on models.
Your claim was bogus, pure and simple.
“But he did not make his own conclusions– his research was used by others, one in particular, and the independence of that research collapses into a singularity”
Likewise, the “hockey stick”, while first uncovered by MBH 98, doesn’t rest on that 16 year old paper. Many independent researchers using many proxies have reached the same conclusion.
You are a cut-and-paste artiste, that is clear. It is also very, very boring.
Thus far you haven’t raised an argument that wasn’t debunked at least a decade ago.
How about trying some independent *skeptical* thought for a change?
Michael:
“I have stated often enough that I come here to gain knowledge.”
Your posts demonstrate that you aren’t.
Michael:
“As I have mentioned, research can only show “what is”, it cannot show “what is not””
This, too, is not true.
I might have confused readers seeming to waffle on the issue of certainty.
If the claim is of “certainty” then the issue is binary and doesn’t involve the 97 percent figure. It need only be proven true or not true. In a *criminal* court of law, particularly where life and liberty is at stake (which it sort of is in this context), all it takes is to knock “true” off its perch is a LITTLE BIT OF DOUBT.
If the prosecutor goes for “Murder 1”, but really the defendant is guilty only of Murder 2, he might be acquitted (depending somewhat on jurisdiction whether it is permitted to accuse someone of two different but similar crimes simultaneously as in the Martin Zimmerman trial).
In the “public mind” IF by “consensus” you mean essentially 100 percent agreement, it doesn’t take much to derail that strategy. Hence the circling of wagons, threats to delete data, lawsuits and so on.
It IS important. It DOES matter, that last little percent matters a LOT.
Michael, I would prefer if you stop misrepresenting the legal process. It does not take only “a little bit of doubt” to warrant a verdict of not guilty. Rather, it takes reasonable doubt. Any body who, as a juror, returns a verdict of not guilty because they personally believe the death may have been caused by a dopelganger from Mars framing the accused has not fulfilled their role as a juror. They have breached their oath.
Even accepting your flawed analogy, the endless trumping up of absurd claims by pseudo-skeptics (including the claim that there is not a concensus of climate scientists accepting the theory of AGW) are not generating reasonable doubt and so are irrelevant to your analogy.
More generally, I would be quite happy for you to apply your legal analogy provided you applied it consistently. Thus applied, the US would reject permission to construct a pipe line for oil for Canadian oil if there is a reasonable doubt that it was beneficial. They would reject permission for new coal, oil, or gas fields on the same basis. Indeed, all construction projects would be prohibited, all government policies forbidden, if there was a reasonable doubt of their benefit. Such a world would be have a complete freeze on both policy and investment.
You can only think your analogy is applied reasonably if you have in fact applied it inconsistently in support of opinions you hold independently of the purported reason, and for ideological reasons.
“However as each is itself derived from an ancestor, the shared code is substantial and so they are not truly independent. “
To what extent should the models express “shared physics” in order to achieve the best scientific outcomes?
And given your (no doubt) extensive software development experience to what extent would you expect that “shared physics” leads to “shared code”?
Michael:
A piece of free advice
When you find yourself i n ahole and want to get out, it’s best to stop digging. ;
Michael:
You have not confused anyone with your posts. It is crystal clear that they make no sense.
“If the claim is of “certainty” then the issue is binary and doesn’t involve the 97 percent figure. It need only be proven true or not true.
No scientific claim is of “certainty”, so that sentence and the line of argument you try to develop from it are moot. Never mind that many pseudo-skeptics routinely violate this principle on one side when they implicitly or explicitly claim that they are certain that the climate science consensus is wrong, and on the other side when they insist on perfect certainty before we act to mitigate!
To put it another way, reasonable doubt is a sword that cuts both ways but you only seem to be looking at one edge. Given less-than-perfect-certainty in our knowledge it’s not very useful to either insist on certainty before making a decision. (The Navy would grind to a halt in a million different ways if that were the standard.)
A much more useful question to ask is “given what we know and how confident we are of it, what’s the best course of action now?” Note that this question does not embed the fallacious presumption that “doing what we were already doing is the optimal decision given our current state of knowledge”. Thinking in terms of that is question commonly considered good practice in the in business, in medicine, in economics, in the military, in government, in any other endeavour where there are different courses of action along with imperfect knowledge. Try using that framework with respect to climate science and see where it leads.
Michael:
“It IS important. It DOES matter, that last little percent matters a LOT.”
Do you feel the same way about whether or not the earth is flat?
Or that the universe is only 6000-8000 years old?
Every argument you make against climate science is equally valid if applied to those who hold that the earth is neither flat nor a few thousand years old.
Pointless.
dhogaza says: May 17, 2014 at 1:58 am. “No, so-called skeptics are not doubters, they reject climate science with absolute certainty.”
Thank you for that illustration of absolute certainty 😉
I am not certain of anything save my own existence and one other thing off topic here. You could be a computer program for instance but if so someone is a VERY good programmer.
Thank you Descartes for making me uncertain.
But, provisionally, I must go through life, complete a military career, raise a family and I *seem* to enjoy some hobbies and of course these conversations. So I’ll take them at face value.
[Mod: Sorry OilMan, this can’t stay]
John Hartz says: May 17, 2014 at 3:25 am. “Michael: A piece of free advice When you find yourself i n ahole and want to get out, it’s best to stop digging.”
Thanks! So far it appears I am in no hole but you are right I have spent too much time on this blog. I was accused of being a “drive-by” and now that y’all are reasonably assured I am a real person it would be better to just say “thanks for stopping by” rather than pretending I am in any kind of hole.
I’ll go back to just reading what people have to say and glean what bits of science I can from it.
So y’all have some comfort, I’ll try to answer the most recent round of questions while waiting for my teenager to get off work then I might not answer the rest unless unique and interesting.
Lotharsson says: May 17, 2014 at 3:33 am quoting me: ‘If the claim is of certainty then the issue is binary and doesn’t involve the 97 percent figure. It need only be proven true or not true.’
“No scientific claim is of “certainty”, so that sentence and the line of argument you try to develop from it are moot.”
I am not speaking of the scientific claims, rather the political claims presented to governments and taxpayers around the world. I have been careful in my presentation to acknowledge Cook’s description of possible sources of uncertainty and bias although of course believed to be small and possibly even over-corrected. This is not made clear in the abstract and certainly not in the public pablum.
Binary is an artificial creation. Your computer isn’t binary for instance… It operates on statistical groupings of high and low voltages. It’s an issue near and dear to me. Down hole equipment fail because the groupings merge.
Thanks for stopping by, and for your concern.
dhogaza says: (May 17, 2014 at 4:34 am)
“Do you feel the same way about whether or not the earth is flat?”
Ah so, inspired by Lewandowsky’s “Fury” paper. I wish he would have surveyed ME. The two are unrelated (even if mysteriously correlated — there’s an interesting website of non-sequitur correlations that is quite amusing). A flat, young earther could still fear flat-earth warming while a highly literate, old, round earther might have doubts.
It depends on the circumstances. As I drive across Ohio or Kansas, I can understand why children there easily believe the Earth is flat. As children grow older they develop their own proofs if one is still needed.
But whether it is important to argue about, probably not, unless you are building me a bridge, airplane or rocket in which case your physics need to be up to the task (as I suspect yours are).
For the record, I’ve never believed the earth was a disk, not even as to my earliest memories. My parents gave us good educations, National Geographic from before I could read (and continuously ever since), books galore.
But my own child did not believe in “air” and considered me silly for claiming such a thing existed — it is invisible and cannot be touched. Then she took swimming lessons and discovered “air”.
“Or that the universe is only 6000-8000 years old?”
That is probably a little more important to be approximately correct although I would not want to be the one to try to persuade a young-earther exactly WHY it is important. I suspect we are dancing around a prohibited topic — what matters is the existence or not of the agent of how the young earth came into existence. If there’s an agent that can change physics at the blink of an eye then all human effort at anything is pointless. My counterargument is that “while that may be so, he’s not GOING to do that and destroy an obviously complex and carefully laid out plan.”
It avoids unnecessary detours into the unproveable.
I accept current estimates of about 4.7 billion years for the age of the Earth and about 13 billion years for the age of the universe, recognizing that “age” itself is subject to the Lorentz Transformation (relativity) so the measuring rod itself might have changed in all that time. But, so did everything else so it kinda works out the same as just calling it 13 billion years old.
“Every argument you make against climate science is equally valid if applied to those who hold that the earth is neither flat nor a few thousand years old.”
As I have not argued against climate science (or for it, per se) I also have not argued against those that believe the earth to be round and old (or young and flat).
Usually it doesn’t really matter. For me, it DOES matter for some mundane reasons — I’m an amateur radio operator and knowing the earth is “round” is important to aim the antenna northeast when on the map my destination is just east. Great Circle and all that. I was also involved in aviation in the US Navy — more great circles, ice edge reconnaissance. It is crucial for the great navigational invention of the 20th century: GPS. I continue to be amazed that in my hand I hold the world’s most powerful navigational instrument EVER.
“Pointless.”
NOT pointless. Hundreds, possibly thousands of people are following this conversation and judging the knowledge, relevance and character of the participants. They will jump in when they possess some knowledge that I lack if the opportunity is presented, relevant and on-topic.
I will say again that I do not blame others, or you, for your manner of thinking. Many people, maybe most, are exactly as you believe. But you cannot *prove* it neither, I think, does it matter much.
AnOilMan says: (May 17, 2014 at 5:23 am) “Binary is an artificial creation. Your computer isn’t binary for instance… It operates on statistical groupings of high and low voltages.”
I salute you sir! All computer chip transistors are analog, merely driven to saturation. Back in the day a cheap amplifier was to take an ordinary NAND gate and bias the input to 2 volts. Instant (but unstable) amplifier.
PEOPLE, however, can be VERY binary, and for the same reason — DRIVEN TO SATURATION. If you believe just a little about global warming, you start to “see” it everywhere and strengthens your belief. If you doubt it, then you discard every evidence of it. Soon, everyone has taken a stand either for or against. It is “bistable” and probably impossible to NOT take a stand.
However, bistable can be UNstable and oscillate. This is seen in politics as vast swings from left to right to left to right…
dhogaza says: May 17, 2014 at 2:10 am Michael: “As I have mentioned, research can only show “what is”, it cannot show “what is not”” This, too, is not true.
An example would help at this point. I believe this as a general principle of logic. A specific contrary example would be interesting, many such examples challenges my believe in the generality of this principle.
People go out to find things. They do not go out to NOT find things.
Lotharsson says: May 17, 2014 at 3:19 am. “And given your (no doubt) extensive software development experience to what extent would you expect that “shared physics” leads to “shared code”?”
Good question — this appears to be behind the 6 billion dollar lawsuit of SCO vs IBM.
Just how many ways exist to implement a Heap Sort? Not very many. Nearly all such implementations will be identical.
In this instance I am choosing to *believe* claims that GCM’s evolve and adapt from previous generations of the same code, which is true of nearly all software. As I understand it such programs are extremely complex and you don’t WANT to start from scratch! Refine it, correct it.
And yet, some value exists in starting from scratch. So, you have a few *families* of GCM; between families, no sharing, within a family, some degree of inheritance of previous versions.
Consider virtualization — VMWare and XenServer. It looks to me like both evolved from a common ancestor, QEMU and in fact is still named such in XenServer. But VMWare and XenServer refined the emulator, it is no longer your father’s QEMU.
Michael,
I’m afraid that I haven’t read much of this exchange, but you are starting (have already?) to bomb this thread. Could I ask that you show some restraint in your commenting. It’s hard to have a sensible discussion when on person says so much that it’s hard to know where to start with a response.
This will be my last for now on this topic. I’ve had a good round and appreciate the participation. I’ll finish up with a few words on this one:
dhogaza says: May 17, 2014 at 2:07 am “But even if you were right, six is still six times larger than one, in other words your claim of a lack of independence was false, and you knew it was false when you made it.”
On the contary. My claim is that the 32 are not each independant. 6 families of code appear to be independent and yes, 6 is plenty. Publishing 32 climate outputs looks like spaghetti. As I have not evaluated GCM’s this is my opinion and is not authoritative. Your mileage may vary. Knowing what my opinion is may help you understand the world a bit better.
“This doesn’t bode well for an honest discussion, does it?”
We agree on this point.
“And, of course, models aren’t the foundation of climate science anyway. There are many independent lines of evidence that aren’t dependent on models.”
Indeed. We agree on this as well. Models are for forecasting. Research is for discovery of the essential data and methods that go into a model. Eventually someone will get it right although knowing it is right will take a long time. For now, they are not right.
“You are a cut-and-paste artiste, that is clear. It is also very, very boring.”
And yet here you are. The cut-and-paste is necessary where context is complex and a long way up the thread. I believe it is important for readers to know what exactly I am commenting on to reduce accusations of what I did or did not say and that benefits you as well.
“Thus far you haven’t raised an argument that wasn’t debunked at least a decade ago.”
I am almost tempted to get into a discussion of what constitutes a successful debunking. Gotta try harder! Don’t do it! Good night, all.
Thanks for the compliment John & Anders but I’m afraid I don’t have time to add much to what was already said. The question is whether we should be wasting our time on “Dismissives” and those who are “Doubtful” (like Michael here — who seems to think he’s the latter but so far has shown no signs of wishing to have his doubts addressed, but rather seems intent on pontificating widely on a range of subjects that he appears to know considerably less well than he believes. Here’s a tip: when you’ve started trying to impress people by finding a way to shoe-horn the names of a couple of virtualisation products into a completely unrelated discussion, you’ve actually started to look like someone desperately trying to impress people…) or we should instead simply point out that what they are saying contradicts current mainstream understanding (which is where Cook et al’s results come in) and point to more reliable sources of information for the “Concerned” and “Cautious” who may not realise just how strong the consensus is. It probably doesn’t hurt to poke fun at the crazies swamping every thread as well, which would be why they hate Lewandowsky’s work so much.
As for Michael: I have to agree with him. We should not change our behaviour until someone proves beyond a reasonable doubt that it will have negative consequences, in exactly the same way that we never insure our homes until someone can prove beyond a reasonable doubt that it will burn down in the next year, or insure our cars until someone can prove beyond a reasonable doubt it will be stolen. We should also continue to smoke until someone can prove beyond a reasonable doubt that it will cause lung cancer (and, if we really want to get picky, prove that the next cigarette we are considering smoking will cause lung cancer) because that’s really how grown-ups make decisions in the face of uncertainties.
For the humour impaired: Of course I don’t really think that. I’m not a complete idiot who knows nothing about risk management.
I’ll finish with an interesting thought experiment:
Suppose we lived in a world where the electric car had prevailed, nuclear was cheap and safe, nobody was burning fossil fuels, and the climate had not changed. Then Russia, based on the research of its scientists about the link between CO2 emissions and global warming, decides that it’s had enough of the cold weather and wants to live in the tropics, so starts burning coal and oil like crazy to raise the level of CO2.
What would the reaction of the American government be?
a) Refuse to make a statement about Russia’s actions until someone can prove beyond a reasonable doubt that it will have the stated effect and that will be bad (for America at least)? And then, when every scientist in the world outside of Russia releases a report explaining just what is known, refuse to believe it because some of the Russian scientists say there is too much uncertainty to be sure exactly what the outcome will be?
b) Declare Russia’s attempts to alter global climate for their own benefit without regard to the potential harm it may cause others a dangerous experiment that’s tantamount to an act of war and, quite likely, a despicable attempt to attack America indirectly through increased natural disasters?
There are others (e.g. offer to sell coal to the Russians to make a quick buck, which can’t be ruled out 🙂 ) but I’ll leave those to your imagination.
Rather than waste too much time with Michael, you should tag team him, that way the gish gallop has less to work with and you’ve a good chance of wasting more of his time than yours.
Given:
“As I have not argued against climate science (or for it, per se)”
we have here a good study for Willard, but you’ll note how what seems at one point to be a discussion of the philosophy of science and how we know about things, veers back towards throwing doubt, carefully phrased as just his opinion and not even backed up by anything.
“…you should tag team him…”
I suspect that only really works if he cared about answering the serious critiques provided to him.
So far it appears that he’s quite happy to ignore many of them. On the other hand, some might suggest that means the tactic is already working 😉
“So far it appears that he’s quite happy to ignore many of them.”
While claiming he is here to learn, nonetheless.
I’ve posted a comment on my blog:
Dream all you want, the emperor’s still naked
Carrick,
[re ‘the data is stolen’]
“Actually no, it was publicly available via the tcp interface provided by Cook, as Brandon has explained.”
Non sequitur. If someone steals a car then it is no defence to say the car was unlocked and the keys were in the ignition.
Similarly accessing publicly available URLs CAN be an offence and people have been successfully prosecuted for doing so. The legal test is usually along the lines of knowingly accessing something without authorisation, and has absolutely nothing to do with the adequacy of any measures taken to protect it.
I appreciate that the topic that jsam (now edited, thanks) and Frank mention is possibly of interest, given that aspects of it could be potentially serious, can I ask that people show a great deal of restraint in their comments – possibly to the point of not mentioning it at all unless it happens to be for a very good reason.
JasonB –
I just saw a post at Skeptica Science that has poll data w/r/t public confidence in their knowlegdge about climate change that are in strong contrast to the data I Excerpted above in previous exchange i had with you. Interesting.
Frank O’Dwyer:
“on sequitur. If someone steals a car then it is no defence to say the car was unlocked and the keys were in the ignition.”
Indeed, I had the great pleasure of helping treat someone to a free, three week stay at our county jail, followed by a couple of years probation, for stealing a book from my unlocked car inside my unlocked garage.
Michael number 2,
There’s already a ClimateBall ™ player with the “Michael” nickname:
Somehow, I have the impression that you’re not that other Michael.
Would you please be so kind as to take another nickname?
Thanks!
Michael:“I have stated often enough that I come here to gain knowledge.”
Me: ROFL
Joshua: Indeed. And most importantly, look at who most people trust to give them information about global warming: NOAA; science programs; NSF; scientists. Who don’t they trust? Mainstream media. I’m pretty sure I’ve seen another such survey where blogs were placed even lower.
I’ve pointed this out here before in response to someone’s claims that scientists had a credibility problem. They don’t. They only have a “credibility problem” among a small segment of the population who already have a strong bias against believing them in the first place, and who probably won’t believe them unless they say that AGW isn’t a problem, in which case they’ll suddenly be completely credible. “Credibility” among those people, in other words, is worthless.
This is also why “skeptic” organisations like being able to trot out scientists to proclaim their message, and therefore why it’s so useful to be able to point to Cook et al and tell the overwhelming majority of people who trust scientists and scientific organisations that those scientists are part of the 3%. (Which is why Roy Spencer, for example, was so keen to claim before Congress that he was part of the 97%.)
(I’m conflating scientists with papers in this instance, but it’s a reasonable proxy; Spencer had a paper that was part of the 3% and none that were part of the 97%.)
I think to top it off it’s also worth pointing out that those 3% who don’t agree with the mainstream view also don’t agree with each other. The only thing they all have in common is a belief that the mainstream is wrong. I believe most people accept that every field has a few cranks, or, at the very least, contrarians, and that a lot more turned out to be like Bozo the Clown than like Galileo.
For people who are already interested in the science and willing to put a lot of time and effort in to informing themselves, then Cook et al’s results are obvious and of little value. But that’s not the purpose. Rather, it’s a tool for letting mainstream people know what the actual state of the scientific literature is, because they are being seriously mislead and that’s dangerous in a society where we need enough people onboard to drive significant change.
I’ll be “Michael 2”.
willard (@nevaudit) says: (May 12, 2014 at 3:30 pm) nit: “this sentence in Richard’s seventh draft: … 95% of the surveyed paper are silent on the hypothesis of anthropogenic climate change. … Let’s assume there are 12k papers. To falsify Richard’s claim, it should be enough to take 70 abstracts and show how they take a position on AGW.”
I suggest you need 600. (5 percent of 12,000). Your idea is worthy but there’s a big difference between needing 70 vs needing 600.
JasonB says: (May 19, 2014 at 3:37 am) “those 3% who don’t agree with the mainstream view also don’t agree with each other.”
THANK YOU for reinforcing my passionate testimony that there’s no such thing as “skeptics” that can be lumped together into any kind of “group”. Some of the “skeptics” aren’t skeptics at all, but long-shot investors. That’s just my guess but the wealthiest investors make money no matter which way the wind blows — provided of course the wind blows. If it doesn’t, they make it blow.
JasonB says: (May 17, 2014 at 7:29 am)
As for Michael: “we never insure our homes until someone can prove beyond a reasonable doubt that it will burn down in the next year”
If in all the history of mankind no house had EVER burned down, you’d have to be an idiot to buy fire insurance — but there be plenty willing to sell it to you along with magnetic bracelets and ionized water.
“I’m not a complete idiot who knows nothing about risk management.”
That’s not quite declaring what you DO know…
Risk management is a balancing act. Your budget is usually fixed. You weigh your choices based on frequency of occurrence and impact, and assign (although it is a WAG) a cost so that you can subsequently make rational decisions.
“I’ll finish with an interesting thought experiment:” (it’s big so see above for the details).
a) Appeasement and study. Russia is a formidible foe (militarily). China even more so (economically) that is why this option appears to be the most accurate description of current policy.
b) Act of war. This is more or less exactly Japan’s reason for entering WW2. The cost to all participants is enormous.
Unlike America’s embargo of oil to Japan, a really serious matter, no nation has actually been damaged by my automobile in a way that is satisfactorily proven to me. What is more likely to start a war is demanding reparations where I believe none are justified.
> I suggest you need 600.
Oups. I stand corrected. Thank you, Michael 2!
Not sure, though, that Richard would like me to tweet him the 700 abstracts…
“Your budget is usually fixed.”
Not if the consequences are severe enough, unless you intend to impose a constraint that will force you into a suboptimal risk management strategy. For the most severe classes of consequences, if your budget is less than “everything we can possibly deploy” then you’re doing it wrong.
And you’re also doing it wrong if your risk management experts say “it’s a serious risk, but we can sort it out within a comparatively small budget if we start now” and you um and ahh and cause delay for dubious reasons, including misapplication of the principles of risk management up to and including complaints that the costs are too high or that we need to be more certain that we’ll suffer impacts before acting.
“You weigh your choices based on frequency of occurrence and impact…”
Not if there are identifiable factors causing non-trivial changes to frequency and impact. If that’s the case and you do that, then you’re doing it wrong.
And you would also doing it wrong if you were to insist that the likely frequency and probable range of impacts assessed by the experts had not been satisfactorily proven to you, when you don’t have the skills to make that assessment yourself.
But all this has been pointed out before, and will be pointed out again…
ATTP: What is your best guesstimate of the number of people who have read this comment thread from beginning to end? Of that number, how many are not people who have posted a comment(s) on th the thread? My workjing hypothesis is that the number of interested, non-particpating readers is vey low — less that 50 perhaps. With re to comment threads, ClimateBall is not a spectator sport.
John,
It’s quite a lot. Appears to be my second most read post, so in excess of 5000 views. No idea, though, how many independent views.
It would also be informative to know how many of the people who posted on this thread actually read all of Michael’s posts from beginning to end. I, for ecample, rarely bothered to read more that the first couple of paragraphs of his lengthier tomes. The best comments are, in my opinion, brief and to the posint.
ATTP: Your OP may have received in escess of 5,00 views, i suspect that the overwhelming majority of those readrs,did not, hweever, read the comment thread in its entirety. I wonder if anyone has ever tracked viewership along these lines in a rigorous mannerr.
Hi John, I rarely comment but revisit old posts to read the complete comment thread once activity quietens down. I doubt that I’m unique.
It’s obvious that a large number of comments increases the number of page loads (with 500 comments 5000 page loads could be reached by 10 really active readers who check every new comment immediately). It likely, but less certain that the average number of people reading each individual comment goes simultaneously down.
I was traveling when this thread was most active, and have no plan to go through all the numerous comments.
Pekka,
Judging from my rather simplistic stats page, the ratio between visitors (which I think means a unique IP) and views is about 1:3. If so, 5000 pages views could mean around 1500 unique visitors. However, as you point out, a more active comment thread could mean a smaller number are checking more often.
I don’t blame you. I wouldn’t do it either 🙂
Tol has now revealed the title, and publisher of his paper. It is to be “Quantifying the consensus on anthropogenic global warming in the scientific literature: A reanalysis, and published by Energy Policy. He reveals this on an audioslide he is preparing for the release of the article. Given that Tol thinks scientific criticism and rebutal is done by legal teams (as least as regard to criticisms of his papers), Energy Policy should look long and hard at that audioslide prior to release. Truth will not be a defense if Cook decides that sauce for the goose is sauce for the ganda.
Graham Readfearn has a good article on Tol’s paper and its problems at DeSmogBlog.
I suspect we’ll hear much more about it next week too. The poop is about to hit the fan.
Tol’s ‘audioslide’ discussed a lot of the material in his paper prior to its publication, which I believe violates the journal’s publication guidelines. I also have reason to doubt that they’ll care.
[Mod: defamatory] I also wonder who picked uo the tab for his trip to Wahsington DC to testify at the House Science Committee on May 29,
Moderator: I thought i might have been able to sip that one by your eagle eye, but alas…
Nothing slips unseen past my squirrel eye. Squirrels have excellent vision didn’t you know?
Thanks for the heads-up, Dana. I’m a big fan of that sort of poop. 🙂
I’m quite sure Ethon would have allowed it. 🙂
Someone put this on the internet (NOTE! It will download a pdf). I’m not an expert at all, but if this is correct, Tol’s credibility will go down yet another few notches.
Marco,
Thanks, I saw that on Eli’s blog. Tom, I think, essentially pointed something similar out in an earlier comment.
Having had a chance to look at the PDF, I think it’s quite an interesting analysis. If I understand it properly, it does the reverse calculation at an intermediate stage and shows that if Tol’s method is correct, it would imply a stage where the consensus was 116%, which is clearly nonsensical as it would imply a negative number of abstracts rated as rejecting AGW.
I also thought the author’s note was telling. Whether justified or not, someone choosing to remain anonymous because of the reputation of the person who’s work they’re discussing, does not reflect well on the person who’s work is being discussed
Well, I am aware of the Tol – Ackermann conflict, and there Tol’s behaviour indeed does not reflect well on Tol. It’s one thing to consider the criticism invalid, and forcefully argue that it is invalid. It’s another thing to then start writing to Ackermann’s employers and publisher with implicit threats of legal action.
But nothing surprises me anymore with Tol. Have you seen the photoshop of Bob Ward he posted on Twitter? That was just beyond childish, but perhaps to be expected when he just could not defend his errors through factual discourse (see also discussion with Andrew Gelman).
And yes, Tom already pointed out the same problem. I wonder if this is going to be in the response of Cook et al to Tol’s paper.
Marco,
But according to Richard, he’s bound by the laws and customs of academics’ behaviour (I’m somewhat unclear as to what those actually are, never having really seen them – which may explain Richard’s behaviour).
If we’d take Tol’s public behaviour as indicative of the laws and customs of academics’ behaviour in public, I might have to reconsider being an academic.
Marco, if there is any relationship between Tol’s behaviour and the laws and customs of academics’ behaviour in public, it is simply that you can largely behave any way you like. So, you can simply choose to behave differently 🙂
If you have not alrady done so, you will wnat to check out Dana Nuccitelli’s Guardian blog post article of June 2, Republican witness admits the expert consensus on human-caused global warming is real..
Needless to say, Richard Tol is not a happy camper about Nuccitelli’s analysis of his testimony beofre the Congresional Committee.
ATTP: I thought I had embedded the link to Dana’s article in m,y prvious post but.alas it does not work. Please emed it or me. Thanks.
http://www.theguardian.com/environment/climate-consensus-97-per-cent/2014/jun/02/republican-witness-global-warming-consensus-real
John, that article was the cause of the Tol-tweet that ATTP quoted above.
Will Tol be cashiered from the GWPF for his unintentional truth? What is their equivalent of the ripping off of epaulettes from that secretly funded PR fluff vehicle? Will he have to hand back his scientific advisor badge along with the cereal box it came in?
Pingback: Richard Tol accidentally confirms the 97% global warming consensus | Gaia Gazette
Andy Skuce has just posted a detailed anaysis of Richard Tol’s critques of Cook et al (2013) publshed in the journal Energy Policy on his website,Critical Angle. The title of Skuce’s post is Consensus Matters..
SKS has its reply up…
http://www.skepticalscience.com/climate-contrarians-accidentally-confirm-97-percent-consensus.html
Pingback: Bravo, Richard Tol, Bravo! | And Then There's Physics
Pingback: Same ol’ same ol’ | …and Then There's Physics