Science and Skepticism

oarobin posted a comment highlighting a video of a talk about Science and Skepticism by Steven Goodman. It essentially disusses the issue of reproducibility in science, and mentions some issues that I have myself. Ultimately, science is about unconvering “truth”. Of course, I don’t mean absolute truth, but that we want to tend towards the best possible understanding of whatever it is that is being studied.

One way to do this is to test hypothesis and to then try to reproduce these results. Doing so requires having access to what others have done before. However, this is sometimes interpreted as completely redoing what was done before, often using exactly what was used before. There’s nothing fundamentally wrong with this, but if you get exactly the same result, all you’ve really shown is that they didn’t make a mistake. On the other hand, if you find a mistake, or are able to question something they’ve done, then you potentially have ammunition if your goal is to undermine their results. This could be true, even if the consequences of this issue is negligible.

What I think is more important is to see what happens if you try to test the same hypothesis. Do you get a consistent result, or not. If not, why not? Ultimately, we’re looking for consilience; the idea that multiple lines of evidence, converge towards a strong, consistent, conclusion. This doesn’t mean that every line of evidence has to independently point at exactly this conclusion, or that every line of evidence has to be significant. It simply means that the weight of all the evidence points towards this conclusion.

As usual, I’ve said too much and you really should watch the talk. However, I’ll add one more thing that I found interesting. The speaker did make the point that a number can’t be true or false. What can be true or false is what we infer from an analysis, not the numerical result of the analysis itself. If we get the same results, do we draw the same conclusions? If we don’t get the same numerical results, do we draw inconsistent conclusions, or could what we infer be consistent, even though the numerical results appear not to be?

There’s more that could be said, but I’ll stop there.

Advertisements
This entry was posted in Research, Science, The philosophy of science, The scientific method and tagged , , , . Bookmark the permalink.

204 Responses to Science and Skepticism

  1. As usual, I’m going to end up adding a comment to add something I didn’t get into the post. Although I think there are issues with how we do research and we could do many things in ways that would be better, or more efficient. However, I think many of the criticisms are hyperbolic. I don’t really think there’s a crisis, although some fields may have more severe problems than others. I also think that open data, and transparency, are important and we should all strive for this. However, how we do so probably isn’t the same in all fields and we should be careful of trying to find a one size fits all solution. People should be able to work out what others have done and, if necessary, have access to data, or codes, that they can’t easily produce themselves. This doesn’t necessarily mean that researchers must provide every single thing they’ve done.

    For example, today I was trying to reproduce some numbers in a table from another paper. They cited another paper, where I found the equations that they’d used. I then realised that they hadn’t fully defined all their terms, but I sort of knew what the likely values were and eventually managed to work out how they’d got the numbers. Didn’t take me particularly long, and working through this helped me to better understand what was being done.

  2. Advanced tech research is often coy about presenting exact recipes to enable reproducibility. As they say, the proof is in the pudding.

  3. angech says:

    ” Ultimately, we’re looking for consilience; the idea that multiple lines of evidence, converge towards a strong, consistent, conclusion.”
    Skepticism is science.
    Skepticism is not denial [sorry Eli].
    Consilience is what we all aim for.
    Along the way the multiple lines of evidence [all] have to be reproducible and consistent.
    ” On the other hand, if you find a mistake, or are able to question something they’ve done, then you potentially have ammunition if your goal is to undermine their results. This could be true, even if the consequences of this issue is negligible.”
    Surely this is a good thing?
    If a line of evidence is proven wrong the consequences are not negligible.
    The consilience is partly lost and worst the other lines of evidence may have drawn [or not] on the same mechanism in which case they would be less admissible as well.

  4. angech “Along the way the multiple lines of evidence [all] have to be reproducible and consistent.”

    no, they only need to all be consistent if they form a sequential chain, if they are parallel, you only need one link to be valid. For instance there are many lines of evidence to show that the rise in atmospheric CO2 is not a natural phenomenon, however we only need the fact that atmospheric levels are rising more slowly than anthropogenic emissions to know that. That would still be the case, even if a fundamental flaw were found in each of the remaining lines of evidence. ISTR we have discussed that before. Also science doesn’t need to be reproducible to be correct, reproducibility just gives us confidence in its correctness, that isn’t the same thing.

  5. “On the other hand, if you find a mistake, or are able to question something they’ve done, then you potentially have ammunition if your goal is to undermine their results.”

    Showing somebody is wrong is what drives science (at least according to Kuhn). Good scientists do their best to undermine their own work before they submit it for publication. The best sort of skepticism is self-skepticism. Of course we all fail at that from time to time.

    I agree about independent testing of hypotheses rather than simply repeating the experiment verbatim. While I think the second type of reproducibility is a good thing (but not always easy to achieve), often complaints that code and material are not made available is often just a rhetorical device to undermine work that someone doesn’t like. For example, the code for many GCMs have been available for years, but have climate skeptics done anything with it? No, not AFAICS.

  6. angech says:

    DM
    Agree with your second posting.
    In regard to the first “they only need to all be consistent if they form a sequential chain, if they are parallel, you only need one link to be valid.” I do not get it completely though I think that there are different tacks here . Your example eludes me.

    Basically there are [multiple] good lines of evidence [reasons] for some AGW.
    There is a proven rise in CO2 levels.
    There is a scientific expectation [good theory] that such levels would result in warming.
    There is a rise in global temperature [ time duration significance is an issue].
    There are some lines of evidence that this may cause problems.
    Sea level rise.
    An expectation [scientific?] that weather events become more extreme with a temperature rise.
    Agriculture problems.
    Infection [malaria] problems.
    Migration problems.
    Over an indeterminate and almost inevitable but sooner than later date possibly this century.

    I can agree with 1,2,3. 4 and 5 may have arguable problems which is why we discuss and defend them. 5,6 and 7 are lines of conjecture rather than lines of evidence, in my opinion, at this stage and others are free to have exactly the opposite opinion.
    The chain from CO2 to disaster in the next 1-2 centuries is a definite possibility but each of these arguments can be approached skeptically and answered scientifically and some flaws are apparent and need answering with we do just not know yet, not the science is proven in all cases.

  7. Marco says:

    “Surely this is a good thing?”

    Since when is motivational reasoning “a good thing”?

    Going into something with the explicit starting point something is wrong means that you almost *have to* find something to be wrong. And then you *will* find something that is wrong (even if it isn’t really wrong), and need to make a big deal out of it, otherwise you must admit to yourself that it wasn’t really wrong – worse even: you may have to admit *you* were wrong.

  8. “not the science is proven in all cases.”

    angech, how many times do you need to be told that science can never be “proven” and that request for that are unreasonable and are not skepticism, but something else.

  9. angech wrote “4 [sea level rise] and 5 may have arguable problems which is why we discuss and defend them..” rubbish, if you think that there are arguable problems whether with an anthropogenic increase in sea level then your grasp of the subject is deficient.

  10. angech says:

    Marco says: June 5, 2018 at 9:44 am
    “Since when is motivational reasoning “a good thing”?
    Take for example.
    Going into something with the explicit starting point something is right means that you almost *have to* find something to be right.
    It is called Confirmation bias for a reason, bias perhaps. Not always a skeptic problem.
    Surely this is not a good thing?
    On the other hand being scientific ie skeptical means having an open mind, not a bias to finding something negative. Could I adjust the sentence to start with an if to ease your concern and make the meaning clear, though “on the other hand” would seem to imply this anyway.
    If ” On the other hand, if you find a mistake, or are able to question something they’ve done, then you potentially have ammunition if your goal is to undermine their results. This could be true, even if the consequences of this issue is negligible.”
    Surely this is a good thing?
    I did post the pro reasons above.

  11. angech wrote “Going into something with the explicit starting point something is right means that you almost *have to* find something to be right.”

    utter rubbish. Bayes’ rule provides one reasonable basis for updating your beliefs given evidence:

    P(hypothesis|evidence) = P(evidence|hypothesis)P(hypothesis)/P(evidence)

    where (for a dichotomous question)

    P(evidence) = P(evidence|hypothesis)P(hypothesis) + P(evidence|~hypothesis)|P(~hypothesis)

    P(hypothesis) is your prior belief that the hypothesis is correct and ~ means “not”.

    So unless you start out with P(hypothesis) = 1 (i.e. P(~hypothesis) = 0) then the evidence will always change your belief in the truth of the hypothesis. If P(hypothesis) is large, then there will need to be a big difference in P(evidence|hypothesis) and P(evidence|~hypothesis), but that is just saying “big claims require big evidence”, which is just common sense.

    You don’t *have* to find something right, rational evaluation of the evidence is possible, and it doesn’t necessarily depend on the strength of your prior beliefs.

  12. “On the other hand being scientific ie skeptical means having an open mind, not a bias to finding something negative.”

    No, being scientific means rationally evaluating all of the evidence, which includes your prior knowledge (c.f. the problem with Nic Lewis’ objective TCS estimate). Having an open mind does not mean having no opinion, it just means allowing your opinions to be altered by evidence.

  13. is the “reproducibility crisis” just the latest tactic in the climate wars? I can see that after a particular test sets forth a theory, it is important that subsequent studies test that the theory is reliably correct, in the absence of indications that the theory is correct, a second group might run the original study in very similar ways to check for errors in setup, data collection and analysis and in those unusual cases, errors might be found, this would show the original study and results cannot be reproduced, but the universe of studies subject to this test of reproducibility would normally be a universe of studies where the original results are more likely to be flawed than in the larger universe of all studies. I haven’t thought much about the reproducibility crisis, so sorry if this question has an obvious answer and this flap is just the latest ideological attack on science. I block angech for reasons that are obvious in this comments thread. I believe Angech does not post in good faith, just wants to play with smoke and mirrors. I think bad faith posters should be tossed out, but hey, that’s just my take on this kind of thing.

  14. Marco says:

    Angech, you keep on ignoring the “This could be true, even if the consequences of this issue is negligible” referring to “you potentially have ammunition if your goal is to undermine their results”.

    Confirmation bias is just a sister to motivated reasoning, but you are defending (the “if” addition doesn’t solve that) people attacking scientific results if there are some perceived errors, even if those errors are inconsequential.

    Scientific skepticism is more like conditional acceptance, and very much unlike unconditional rejection.

  15. tedpress says:

    Smallbluemike, reproducibility issues were highlighted in several other fields before being brought up with regards to climate science.

  16. izen says:

    A key point in the video is that not ALL Science is under attack. Much of R&D is accepted without demands for transparency.
    I doubt those involved in discovering exoplanets have faced such intense pressure to ‘reform’ their scientific behaviour to ensure all data, code, methods, and documentation are easily available to ensure reproducibility.
    Perhaps I am wrong.

    The Heliocentric solar system and Evolution are early examples of targets for skepticism.
    Epidemiological studies of harmful chemicals (Lead, Asbestos, SOx…) have also been disputed, the particulate pollution example in the video is another. As that shows, even when all the hoops of ‘Open Science’ are passed, if the truth is inconvenient, the Scott Pruitt will still exclude it from any consideration in making policy.

    The historical pattern is that the scientific targets of skepticism are those aspects of discovery or invention that threaten the authority, power, and profit, of Religion, Governements or Commerce.
    (In some regions the last two may be difficult to distinguish.)

  17. ted press says: “reproducibility issues were highlighted in several other fields before being brought up with regards to climate science.”

    Mike says: examples, citations, links please. Are we talking largely about culture war science topics?

  18. izen says:

    @-Mike
    “Mike says: examples, citations, links please. Are we talking largely about culture war science topics?”

    Largely culture war topics.
    Evolution has been questioned as a valid theory because it cannot repeat experiments.

    https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5384162/
    Metaresearch for Evaluating Reproducibility in Ecology and Evolution

    (just to poison the well, some funding for this research was from a religious/spiritual philanthropic foundation.)

  19. Michael 2 says:

    ATTP writes: “completely redoing what was done before, often using exactly what was used before. … if you get exactly the same result, all you’ve really shown is that they didn’t make a mistake.”

    That is what got me banned at Scientific American! Dr Mann offered a download of data and a program so that you too could get what he got. I replied that the offer was a fine thing for people wanting to see how a small climate model works, but everyone is going to get exactly what he got unless their computers are defective. It is deterministic.

  20. Steven Mosher says:

    “Smallbluemike, reproducibility issues were highlighted in several other fields before being brought up with regards to climate science.”

    people really misunderstand reproduceibility and the speaker in the video is no exception.

    The movement began with Jon Clarebout at Stanford. He worked in geoscience, signal procesing to be specific. What he and others noted was the following.

    1. When asked researchers could not reproduce their own work with the code and data they had.
    I will have to search for the reference, but what they did was ask people to reproduce the very
    charts and tables that they had published. In 50% of the cases the researcher could not
    reproduce HIS OWN DAMN WORK. There are many ways this can happen. You are working
    on your paper, putting out charts and graphs, lets say natively in some programming language
    but for publication you move your data to a different system, say a professional plotting
    package. And in that process you make a few tweaks and changes. You don’t keep track
    of all the changes. You dont keep a change log. You dont use SVN or Git. You just hack away.
    When the time comes to reproduce your own work, you cant.
    2. Since what they were doing was signal processing it was very difficult to explain in words
    EXACTLY what they were doing in code so that others could re write an algorithm from
    a verbal description. So readers had to guess at what to do. A mundane example,
    If you were trying to check what CRU was doing before climategate, you had no way of
    knowing What stations they selected out of GHCN.
    3. As an intensive methods based science, they found that sharing their work was the most efficient way to press push the science forward. I invent a method, I share the method. You use it, modify it, build on it.

    ideally reprodicibility is just a checkbox. ideally, and quite a few of us work this way, you use either markdown or a notebook. basically your documents are a mixture of code and data that
    is executable. the code is embedded in the very document you publish. Clarebout had a saying that papers were advertisements for science. In reproducible science, the paper IS THE SCIENCE.
    When you have a reprodicicle paper there is no question about what code was run and what data was used to produce the graphs and tables AS PUBLISHED. Further, there is no doubt about what the algorithm was actually doing. because its there! part of the paper.

    Once folks set this up as an ideal there will be exceptional cases. What about GCMs, what about medical data? what about IP?

    the people making these objections, typically dont have any IP, are not working with GCMs, and are not working with private health data. The people making these objections typically

    A) dont want to share their code/data/workflow because its a mess
    B) want to keep some things secret so that they can establish professional dominance
    they want to advance their career more than knowledge
    C) are dinosaurs or emeritus.

  21. izen says:

    @-SM
    “nice” {LINK}

    Indeed! A quote;-
    “EPA must produce the opposing body of science Administrator Scott Pruitt has relied upon to claim that humans are not the primary drivers of global warming, a federal judge has ruled.”

    But It would be interesting to see the EPA argument in full, this comment by the Judge;-

    ““Particularly troubling is the apparent premise of this agency challenge to the FOIA request, namely: that the evidentiary basis for a policy or factual statement by an agency head, including about the scientific factors contributing to climate change, is inherently unknowable.”

    indicates the type of approach they may be taking. And why they are keen to recruit JC to the cause as ‘scientific’ validation that the climate is just all far too inherently uncertain to be able to make any part of it knowable. dpy has been strenuously pursuing that trope on the ‘initial conditions’ thread.

  22. Steven Mosher says:

    “indicates the type of approach they may be taking. And why they are keen to recruit JC to the cause as ‘scientific’ validation that the climate is just all far too inherently uncertain to be able to make any part of it knowable. dpy has been strenuously pursuing that trope on the ‘initial conditions’ thread.’

    ya looks like she has been called on to produce things.

    Also interesting in light of the present thread over there.

    outsourced to the blogesphere

  23. izen says:

    @-SM
    “people really misunderstand reproduceibility and the speaker in the video is no exception.”

    Are you really claiming that Steve Goodman, one of the most prolific authors and leading expert on the subject has got it wrong ?
    https://scholar.google.co.uk/citations?user=jlBqYAQAAAAJ&hl=en

    @-“The people making these objections typically
    A) dont want to share their code/data/workflow because its a mess
    B) want to keep some things secret so that they can establish professional dominance
    they want to advance their career more than knowledge
    C) are dinosaurs or emeritus.”

    D)have spotted that it is unnecessary, arbitrary and inappropriate to apply guidelines for the analysis of seismic data outside of the narrow domain that Clarebout identified.
    E)know that the demand for ‘reproducibility’ is often sciency window-dressing for the demands that certain scientific knowledge is excluded from public policy consideration.

    That is certainly the way it has been used in practise of a number of occasions.

    Of your 3 arguments in favour of ‘Transparency’, only the 3rd has any real weight.
    Even there the amount of information that is required for efficiency would vary from field to field.
    Perhaps Dave_G has some insight into how much sharing and open source goes on in the commercial side of seismic analysis compared to proprietary, commercially confidential data and code.

    @-“In 50% of the cases the researcher could not reproduce HIS OWN DAMN WORK.”

    The researcher could not reproduce an exact CLONE of the original published text and images. But it is not credible to conclude from that they were unable to use some form of the original data and similar methods to compose a paper which by following the same lines of evidence reached consilient conclusions.
    That the new graphs were not plotted or labelled with the Stamford Graphics software is an irrelevance.

    @-“A mundane example, If you were trying to check what CRU was doing before climategate, you had no way of knowing What stations they selected out of GHCN.”

    Very mundane. If you wished to carry out research that would either be consilient with the CRU work, or contradict it, it would be better to make an arbitrary selection of stations and develop independent methods of kridging or analysis. More efficient at advancing the science, the only reason to want all the original data and code in that case would be if you had a strong suspicion (or wanted to imply) that they had cheated, or at least cherry-picked their data and methods.
    Even then performing an independent but similar analysis would be stronger evidence than repeating their exact procedure if you are attempting to identify fraud.

    In the video Goodman mentions the six cities particulate study and the reproducibility push-back against it. In another case the EPA was going to use research that had exposed individuals to varied doses of typical city vehicle pollution and measured lung function before and after. It had found (unsurprisingly) that 12 hours breathing car exhaust in increasing amounts was bad for you.

    Industry lobbyists went into high gear to have this rejected as evidence used in policy making. They demanded that all the data, code and methods be made available or it must be rejected.
    When that demand was met, they focused on the fact that in the exposure experiments, high levels for a short time were used to extrapolate to longer exposure at lower concerntrations and claimed the research must be rejected because it was not possible to compare short/high with long/low exposure.

    When the abundant scientific evidence was provided showing there is a strong correlation with cumulative exposure in epidemiological studies and animal experiments that validated the correlations used, the lobbyists tried another tactic targeting reproducibility.

    They demanded the name, age, gender and current medical records of all the experimental participants on the grounds that any unreported, or un-diagnosed cardio-respiratory disease would totally invalidate the results and so without this information the research must be rejected.

    They also claimed the research must be rejected because the participants were unable to provide informed consent as the actual harm that might be caused was uncertain. Therefore the experiment contravened ethical guidelines and must be rejected as input to policy considerations.

    This is the pattern of how ‘Reproducibility’ as a requirement of research is deployed when science and commerce clash. Improving the efficiency of scientific research has nothing to do with it.

  24. “If you were trying to check what CRU was doing before climategate, you had no way of knowing What stations they selected out of GHCN.”

    IIRC they were also using data that wasn’t publicly available so that they had as much data as they could get their hands on, so even if you knew which GHCN stations they used, you wouldn’t be able to reproduce the results anyway. So why didn’t they just use a subset of the data that was available? Well how do you know that the additional data don’t make a difference unless you try and see? Of course it is possible that I don’t recall correctly, it isn’t my area.

    Exact reproducibility is not very interesting. There is a problem called “software rot”, which means even if I archive my (MATLAB) research code today, that doesn’t mean it will necessarily give exactly the same results in ten years time as it does if I run it tomorrow. I know this from experience (as I needed to change the code anyway, I improved the experimental methodology slightly as well, but at least now there is some code that currently works). Being able to independently re-implement the work and test it in a slightly different way is much more useful.

    A more difficult issue is that I now have to run my code on a high performance computing facility because it is computationally demanding. This raises at least two issues: Firstly will it give the same results if run on a different HPC with a different queuing system (and possibly different versions of MATLAB, and possibly different linear algebra packages on which it runs) and secondly, how can I tell if it gives exactly the same results if I run the simulations again, given that I don’t have the computational budget to re-run the experiments (which take months->years of wall-clock time, even when running hundreds of processes in parallel).

    Do I care if a researcher can exactly reproduce his/her results? No, not really, what I care about is whether their results are solid and they can tell me how to reproduce results that are qualitatively and quantitatively similar. I am much more concerned about papers where there are apparent incongruities, for instance how can a 20 year sinusiodal component of a model fit to the period 1850-1950 end up with a phase offset of 1998.58 (did anything interesting happen in 1998?)? ;o)

  25. Steven Mosher says:

    “Are you really claiming that Steve Goodman, one of the most prolific authors and leading expert on the subject has got it wrong ?
    https://scholar.google.co.uk/citations?user=jlBqYAQAAAAJ&hl=en

    On the difference between reproducibility and replication? yes.
    Note for example that one of the arguments for reproducibility is EFFICIENCY which is the thing
    he argues for.

    In his publication list I see NOTHING on open science and the difference between reproducibility and replication. Zilch, zero.

    Now contrast that with Roger Peng, whose paper he cited as an example, I would consider Roger
    and expert.

  26. Steven Mosher says:

    “IIRC they were also using data that wasn’t publicly available so that they had as much data as they could get their hands on, so even if you knew which GHCN stations they used, you wouldn’t be able to reproduce the results anyway. So why didn’t they just use a subset of the data that was available? Well how do you know that the additional data don’t make a difference unless you try and see? Of course it is possible that I don’t recall correctly, it isn’t my area.”

    yes isnt your area. and you got it wrong.
    You can always remember my guide to “try to have no opinion”

  27. Steven Mosher says:

    “D)have spotted that it is unnecessary, arbitrary and inappropriate to apply guidelines for the analysis of seismic data outside of the narrow domain that Clarebout identified.
    E)know that the demand for ‘reproducibility’ is often sciency window-dressing for the demands that certain scientific knowledge is excluded from public policy consideration.”

    D.. Except he didnt identify that as the ONLY DOMAIN, my bet is you have read nothing in this
    field.

    E? Of course people can misuse it. Same with cars, guns, knives, peer review… everything can be used for the wrong purpose. Welcome to earth

  28. SM “yes isnt your area. and you got it wrong.
    You can always remember my guide to “try to have no opinion””

    How helpful. Providing some information to allow me to correct my misunderstanding.

    You can always remember my guide to “try not to be a jerk”. I’d rather not have to fight cognitive biases in order to accept what you say when you are right.

  29. This CRU webpage discusses the availability of data used by CRU in making their gridded datasets. It seems to make it pretty clear that it uses data not included in the GHCN, and not publicly available:

    The Climatic Research Unit (CRU) at the University of East Anglia (UEA) has, since 1982, made available gridded datasets of surface temperature data over land areas and averages for the Northern and Southern Hemispheres and the Globe. Until the development of the internet these were made available via various media. These datasets (the latest being CRUTEM4, http://www.cru.uea.ac.uk/cru/data/temperature/) have been developed from data acquired from weather stations around the world. Almost all these weather stations are run by National Meteorological Services (NMSs) who exchange these data over the CLIMAT network, which is part of the World Meteorological Organization’s (WMO) Global Telecommunications System (GTS). Much of the original data in the early 1980s came from publications entitled ‘World Weather Records, WWR’. We also make use of data available from the National Climatic Data Center in Asheville, North Carolina (their Global Historical Climatology Network, GHCN). We are also constantly striving to find additional, and homogenized, data from a wide range of sources (see details of earlier work in the publications below and the references in Jones et al. 2012). Both the gridded datasets and the station data archive have evolved over the years and we introduced dataset version numbers in the early 1990s. The methodology we have used in developing the gridded datasets has been described in numerous publications in the climate literature (see list at the end of this document and also http://www.cru.uea.ac.uk/cru/data/temperature/ and the linked FAQ

    ….

    Since the early 1980s, some NMSs, other organizations and individual scientists have given or sold us (see Hulme, 1994, for a summary of European data collection efforts) additional data for inclusion in the gridded datasets, often on the understanding that the data are only used for academic purposes with the full permission of the NMSs, organizations and scientists and that the original station data are not passed onto third parties. This is not just temperature data, but station data for some other variables, particularly precipitation. Below, we list the agreements that we still hold. Additional agreements are unwritten and relate to partnerships we’ve made with scientists around the world and visitors to the CRU over this period. In some of the examples given, it can be clearly seen that our requests for data from NMSs have always stated that we would not make the data available to third parties. We included such statements as standard from the 1980s, as that is what many NMSs requested.

    ….

  30. izen says:

    -[Staircase wit]-

    The (ahem) BEST method of testing the CRU results would be to make an informed selection of the available data, process it with independent software that you could justify as reliable and produce results that either supported or contradicted the conclusions reached by CRU.
    If only someone had bothered to do that even the WUWT crowds doubt about the validity of the historical GMST trend would have long been put to rest…

  31. “[emphasis mine]” in my previous comment, as usual

    izen, LOL (especially the last bit)! ;o)

  32. Steven Mosher says:

    A couple of points

    “The researcher could not reproduce an exact CLONE ”

    Actually worse than that. But it amazes me that people are so uncaring about QA.

    Wrt to do code rot argument. All the more reason!!!
    To archive and supply your code.

  33. Steven Mosher says:

    If only someone had bothered to do that even the WUWT crowds doubt about the validity of the historical GMST trend would have long been put to rest…

    I did exactly that years ago.

    1. They posted their code.
    2. I did my own from scratch without looking at their code.
    3. Answers did not match.

    Want to know why

  34. Steven Mosher says:

    Another example comes from climategate.
    McIntyre could not reproduce Jones. Asked for code.
    Jones refused and he knew why the results could not be replicated.. his paper left out a crucial step only revealed in the code.

    So replication failed..

  35. “Wrt to do code rot argument. All the more reason!!!
    To archive and supply your code.”

    which will rot and therefore not reproduce the results when you run it ten years later.

  36. paulski0 says:

    izen,

    indicates the type of approach they may be taking. And why they are keen to recruit JC to the cause as ‘scientific’ validation that the climate is just all far too inherently uncertain to be able to make any part of it knowable.

    Problem with that would be JC putting her name to energy balance sensitivity papers, and subsequently promoting the results, which inherently assume a very high degree of linearity and predictability of climate.

  37. Steven Mosher says:

    And… reproducibility does not preclude doing the more valuable replication. It makes replication more efficient… as with my cru example. Because I had their code…I could do my own. And when the answer didn’t match I did not bother them with questions…I just figured out why…

    When replication fails you know nothing.

    The issue is when it fails authors are under no obligation to work with you. See scaffetta and gavin.

    Another example..mckittrick on uhi…

    For years replicators never found the simple error.
    Sad.. because Jones made up a story in ar4. That later Peter Thorne had to recant.

    Error was in the metadata.

    But hey.. wanting reproducibility doesn’t mean you don’t think replication is more important..

    Seen that strawman before.

  38. “The issue is when it fails authors are under no obligation to work with you. See scaffetta and gavin.”

    Actually Craig Loehle did answer some of my questions (I didn’t ask that particular question as it would be likely to sour the discussion and I had more important questions to ask. If you are a jerk, people are less helpful, that is not all their fault. If you want people to be helpful, being [genuinely] polite and respectful is a good idea. Scafetta wasn’t on the paper that I actually responded to, although the new paper actually had negligible novel content, so I didn’t ask him.

    “And when the answer didn’t match I did not bother them with questions…I just figured out why…”

    Bad approach, you don’t know whether the answers you found are the right ones (as is actually correct, rather than just the “right” answers for your argument) as you might be missing something.

  39. Dave_Geologist says:

    If a line of evidence is proven wrong the consequences are not negligible.

    angech, gravity tells us things fall to the ground. But birds can fly. Therefore gravity is falsified.

    No it’s not. Why not? Because consilience.

  40. izen says:

    @-SM
    “Want to know why”

    Not really, I am more interested in whether your results were consilient or were significantly at odds with the GMST warming trend reported by CRU.

    If you got the ‘same’ answer then the error of theirs you found and corrected(?) was not a significant factor in the result.

  41. Steven Mosher says:

    https://academic.oup.com/aje/article/163/9/783/108733

    “The replication of important findings by multiple independent investigators is fundamental to the accumulation of scientific evidence. Researchers in the biologic and physical sciences expect results to be replicated by independent data, analytical methods, laboratories, and instruments. Epidemiologic studies are commonly used to quantify small health effects of important, but subtle, risk factors, and replication is of critical importance where results can inform substantial policy decisions. However, because of the time, expense, and opportunism of many current epidemiologic studies, it is often impossible to fully replicate their findings. An attainable minimum standard is “reproducibility,” which calls for data sets and software to be made available for verifying published findings and conducting alternative analyses. The authors outline a standard for reproducibility and evaluate the reproducibility of current epidemiologic research. They also propose methods for reproducible research and implement them by use of a case study in air pollution and health.”

    Instead of Goodman who has no expertise in reproducibility Or Polar bears ( thanks for his google scholar cites)
    I would suggest Roger Peng. ( hat tip to one of the R gods)

    http://www.biostat.jhsph.edu/~rpeng/

    http://www.biostat.jhsph.edu/~rpeng/research.html

  42. Steven Mosher says:

    Now what is hilarious is the notion that somehow Open science is at odds with things like epidemiology and air pollution studies.. 6 cities and all that.

    of course Goodman and izen know little about the guys who publish in the field

    Here
    http://www.biostat.jhsph.edu/~rpeng/RR/index.html

    and what the motivation for reproducibility is in Biostats

    https://academic.oup.com/biostatistics/article/10/3/405/293660

  43. Dave_Geologist says:

    Mosh, I did read the Clarebout stuff you pointed me to a month or so back. Not the same thing. Although often labelled computer science, his team were mostly engaged in applied mathematics and software engineering. Reading the stuff he’s written himself, it’s hard to see how it applies to an observational science (and yes, I know the signals he was processing are observational, but they’re the results of one-off experiments). Yes when it comes to keeping your R code straight, but not when it comes to evaluating new science and advancing science. The best you can hope for is to avoid or pick up a computational mistake. They are of course Bad Things. But you can do the computation perfectly and still get it wrong because you used flawed data (shipboard SSTs during WWII), ignored prior knowledge (Lewis’s “objective” Bayes) or oversimplified (the paper that “proved” 100 years ago that CO2 was not an effective GHG by collapsing it to a one-layer model at surface, so the the radiating temperature was the same as the absorbing temperature).

    You advance science by running a fresh experiment that addresses the same problem, sometimes from the same angle; other times, more powerfully, from a different angle. And reaching mutually consistent conclusions.

  44. Steven,
    Your interpretation of this seems quite different to mine. I think the point is mainly that there is not some simply way to do this and that ultimately the goal is to uncover “truth”. Also, we should be thinking about how to do so efficiently.

  45. angech says:

    Judith Curry article “Best Schools has put together another very interesting list: Top 15 Climate Change Scientists – Consensus and Skeptics. “
    5 skeptics and 10 consensus, not quite 97% is it?

  46. angech,
    I don’t think it should be interpreted as those 5 skeptics being in the top 15. I think it is the top 10 consensus scientists *and* the top 5 “skeptic” scientists.

  47. verytallguy says:

    it’s kinda the top ten climate scientists plus all the skeptics we could find without scraping the bottom of the barrel so hard we drop through it.

  48. BBD says:

    And JC is not even a skeptic!

    Like all strong promoters of low sensitivity, she’s a scientific evidence (aka consilience) denier.

  49. izen says:

    @-SM
    “of course Goodman and izen know little about the guys who publish in the field”

    Peng is certainly a leading figure in pushing for computational reproducibility, specifically in R code. The application of his approach in the methods and inferences that do not rely on R computation is less clear.

    While you may be accurate about my level of ignorance, I think you may be under-estimating Goodman.

    It is notable that many of the Peng papers on computational reproducibility cite papers by Goodman.
    And several of the Goodman papers on the different forms of reproducibility and replication cite Peng.

    What does research reproducibility mean?
    http://stm.sciencemag.org/content/8/341/341ps12.full
    “CONCLUSIONS
    The lexicon of reproducibility to date has been multifarious and ill-defined. The causes of and remedies for what is called poor reproducibility, in any scientific field, require a clear specification of the kind of reproducibility being discussed (methods, results, or inferences), a proper understanding of how it affects knowledge claims, scientific investigation of its causes, and an improved understanding of the limitations of statistical significance as a criterion for claims. Many aspects of the new interest in research reproducibility have been salutary, but we need to move toward a better understanding of the relationship between reproducibility, cumulative evidence, and the truth of scientific claims.”

  50. angech says:

    I know there are a lot more consensus scientists than skeptic ones and your interpretation is right As the subject was skepticism and science and concilience I was trying to get the message out that the lines of evidence are not convincing some scientists who have good and proven track records.
    There will be a debate soon between MM and JC.
    There should have been more debates by the leaders of these ideas to get them out there.
    Not saying the science is settled.
    That is one of those things that gets to me like a red cape.
    Science is a journey of discovery, we are at one stop on the journey.
    In medicine alone I have seen many beliefs come and go in 40 years.
    Rock solid truths with more lines of evidence than global warming ever had and yet years later they lie in ruins with new truths in their place.
    There are good lines of evidence, I listed some of them above.
    Global warming due to CO2 is unarguable on our current understanding.
    But inevitable, serious and all or more due to man?
    One thing that never changed in medicine is everyone had to find something to blame.
    (Particularly relevant to Marco and my discussions above)

  51. angech says:

    And JC is not even a skeptic!
    Though I have hopes for her.

  52. JCH says:

    5 skeptics and 10 consensus, not quite 97% is it?

    Ever more abject ridiculousness. You’re like fountain.

    Two of them clearly are not skeptics: JC and LB.

    Even Lindzen, if you read the transcript of the physics society roundtable, is a complete dud as a skeptic.

  53. Willard says:

    > In his publication list I see NOTHING on open science and the difference between reproducibility and replication. Zilch, zero.

    Again arguing by negative existential. How fascinating.

    Jump at 16:00.

    See also his criticism of Nozek’s crap at 21:00.

  54. Dave_Geologist says:

    Perhaps Dave_G has some insight into how much sharing and open source goes on in the commercial side of seismic analysis compared to proprietary, commercially confidential data and code.

    A lot of the basics are open source in the sense that it’s just textbook mathematics. I was never involve at the sharp end, just in post-stack processing on a workstation or as an informed buyer of pre-stack processing performed in-house or externally (but with a hand-holder in that case). Although I knew enough about the basics to teach on a cross-discipline training course. Amoco caused a big brouhaha when they patented coherency. Since it was just maths (which IIRC is forbidden by the Constitution from being patented in the US), it was regarded as particularly egregious. I understand that the CS nerds argue (after Turing) that all general-purpose computer programs can be reduced to a mathematical algorithm, and are therefore unpatentable. But this was so obviously “just maths” everyone wondered how they got away with it. After a couple of years everyone just called it semblance instead of coherency, confirmed they hadn’t accidentally used the same algorithm as Amoco, and carried on as before.

    There is a lot of sharing. Most oil and gas fields have a number of partners, so it’s in the industry’s interest that all the major operators are pretty much on the same page. If your Operator can’t do something efficiently, you get billed for what it cost him, not what it would have cost you.

    My employer released some of our independently developed software (and at one stage an entire seismic workstation package) to our major suppliers to encourage competition and facilitate outsourcing. Presumably there was a behind-the-scenes deal about discounted rates, a share of third-party rates, no technology pass-through to third parties, etc. They also have a venture-capital arm which invests in small companies which might be future technology or service.suppliers or product customers. Usually cash for shares, but it could include IP or technical advice. They turned a tidy profit on a workstation software supplier when it was bought by one of the major service companies.

    None of this (nor the stuff I’ve read from Claerbout) bears any relationship to the sort of research the rest of us are talking about here. It’s (software) engineering R&D, not pure science. They generally don’t invent a new mathematical transform, they take an existing one and implement it efficiently. Even if you invent a new one, you’re in a bind. We wouldn’t buy from any supplier of “secret sauce” processing (there are lots, including some whose claims appear to violate the laws of physics 🙂 ). You had to tell us the maths and persuade us it was valid. Run your own algorithms of course, and keep them a trade secret. But we reserved the right to replicate standard test suites with our own algorithms and your result better stack up with ours. Nowadays with HPCCs a lot of the real efficiency-saving stuff is not transferable. It’s written to perform on a particular cluster and may not on another.

    And of course even in other areas where I was much more involved with the development and first-line Help, you can’t exactly replicate. If you run the same stochastic model on a different version of the software on the same hardware, you’ll get a slightly different answer (and on some packages you can’t even specify the seed value, others let you change convergence limits or number of iterations). If you get an OS upgrade with new library functions, you’ll get a slightly different answer on the same software version. If you access some hardware-embedded functions, you’ll get a slightly different answer on different hardware. Does any of that matter? Not in the slightest. While Steven describes Claerbout as the father of reproducibility, from his website you very much get the impression of a voice crying in the wilderness. Which is not to say he’s a crank. He really is up there among the fathers of the seismic processing field. But reproducibility is not what he’s famous for. Not even close.

  55. izen says:

    @-SM
    Peng and Goodman seem to be on the same page, literally.

    Reproducible Research in Computational Science
    Roger D. Peng
    http://science.sciencemag.org/content/334/6060/1226.full
    |
    C. Laine, S. N. Goodman, M. E. Griswold, H. C. Sox, Reproducible research: Moving toward research the public can really trust.

    What Should Researchers Expect When They Replicate Studies? A Statistical View of Replicability in Psychological Science
    Prasad Patil, Roger D. Peng, Jeffrey T. Leek First Published July 29, 2016
    http://journals.sagepub.com/doi/abs/10.1177/1745691616646366
    |
    Goodman, S. N. (1992). A comment on replication, P-values and evidence. Statistics in Medicine,

  56. Dave_Geologist says:

    Actually worse than that. But it amazes me that people are so uncaring about QA.

    Do you really think the are SM? Have you considered the possibly that they are not, ahem, penny-wise-pound-foolish? That they know what matters and focus on that, rather than what has, historically, been a tool used to harass them in bad faith. Maybe there are some good-faith actors out there. Shame the other guys spoiled it for them.

    BEST and Cowtan & Way have proved CRU were right and their detractors wrong 1000 times more clearly than 1000 replications of released data and software would have done. Methinks Richard Muller has a better handle on what matters than you do.

  57. Dave said:

    “You advance science by running a fresh experiment that addresses the same problem, sometimes from the same angle; other times, more powerfully, from a different angle. And reaching mutually consistent conclusions.”

    This is exactly right and in IMO there are likely too many snowflakes that want things served up on a silver platter. Like I said in the 2nd comment above, in advanced tech, research recipes aren’t an absolute requirement because if you have something cool, your scientific colleagues will quickly figure it out as long as some basic hints were provided.

    I remember one case when the scanning tunneling microscope came out (which was the first apparatus to be able to “see” atoms directly) that there were soon stories of industrious students and scientists that were able to build these things in their basement. No one had the exact schematics but they had the general idea and just went with it. Curiosity drove the research more than anything else, and there was always an implied trust that the results were solid and you wouldn’t be sent off on a wild-goose chase.

  58. Dave_Geologist says:

    Please, please Steven, don’t bring in McIntyre.

    Mr. “I published my code long after I’d made hay with my flawed results”, “I forgot to mention I used a different noise model than Mann, one which, unlike his, had decadal autocorrelation and was capable of producing a hockey-stick blade”, “what, you mean Mann’s model matches the noise in the real data and mine is a horrible mismatch – who’d’a thunk it?”, “oh yes, and there was that 1-in-100 cherry pick to get the most hockey-sticky simulations”, “still, at least I picked randomly among the choice 1% for my ‘random’ selection”. One out of five ain’t bad.

    Reproducibility is important, but reliability is more important. Accuracy is more important than precision, unless you’re an accountant or a metrologist rather than a scientist.

  59. Dave_Geologist says:

    Researchers in the biologic and physical sciences expect results to be replicated by independent data, analytical methods, laboratories, and instruments.

    Err. Steven: you do realise this quote (my bold) shoots down your FOI-and-rerun-the-code-with-the-same-inputs-argument? Don’t you? It’s replication not reproduction, replicability not reproducibility, science not auditing and consilience not rhetorical games or nit-picking. And useful, not a distraction.

  60. Dave said:

    ” Amoco caused a big brouhaha when they patented coherency. Since it was just maths (which IIRC is forbidden by the Constitution from being patented in the US), it was regarded as particularly egregious.”

    True that geophysicists were responsible for many cool signal processing algorithms. There is one case of a geophysicist named J.P.Burg who came up with a algorithm that lead to Maximum Entropy Spectral Analysis and tested it out on his first prospect — the story is that the producing well was called “Rock Entropy #1”.

    I looked at the Amoco patent https://patents.google.com/patent/US5563949A/en, and yes you are right that there isn’t much to that. It appears to be just an application of cross-correlation to establish continuity of fault directions in 3D data.

  61. Dave_Geologist says:

    Ah, nothing like the old basement experiment. But in an official lab which nevertheless resembled a very untidy basement.

    True but sad story. “I could’a been a contender!”. I’m convinced I made carbon nanotubes in 1976. I was trying to make the Metallurgy Department’s electron microprobe work for geological specimens. Metals are easy. They conduct electricity and are observed under a reflected-light microscope. Charge the sample-holder, make sure the optical focus and electron beam are coincident at the surface of the specimen (basically, observe where the electron beam is burning a hole in the specimen and adjust the optics), and job’s a good’un.

    Unfortunately, rocks are observed in thin sections in transmitted light. Once you’ve got round the problem of the standard metallurgy sample-holder breaking the flimsy glass plate, you need to charge the sample to get keep the beam stably aligned. Problem is, rocks are poor electrical conductors so even if you build up a static positive charge, the target location quickly acquires a negative charge from the electrons, and the beam wanders. So you coat the sample surface with a thin layer of graphite, thin enough to be optically transparent but thick enough to maintain the positive charge (more sample-holder messing-around required to attach the positive lead to the top rather than the bottom of the sample). How did we make the thin layer? Well, if you buy time on a commercial instrument, you send the naked slides to them and they stick it in a machine which runs 24/7 and churns them out the other end in bulk. If you’re name is Heath Robinson, you fill a bell-jar with an inert gas (nitrogen if you’re operating on the cheap), mostly evacuate it (no need to go all the way to a diffusion pump) and sputter carbon from an arc between two graphite electrodes. Sound familiar? I must have spent hours, days and weeks using trial and error to get the carbon layer not too thick, not too thin, not too lumpy, all the way to the edge, stable enough not to peel off in storage, ditto in transit, ditto in mounting, ditto when the beam hits and the tiny target spot gets rather hot (the graphite directly under the beam is probably gone in a nanosecond, but that’s OK).

    I must have made lots of nanotubes, and even the peeling-off layers maybe had some. If only I’d thought to insert some sticky tape! Instead I cleaned out all the dirty-looking crud and dumped it. Ah well, c’est la vie 😦 .

  62. izen says:

    @-SM
    “In his publication list I see NOTHING on open science and the difference between reproducibility and replication. Zilch, zero.”

    You are right that I only have little knowledge in this area.
    I had encountered references to Goodman as the scourge of P and H -hacking in biomedical research back a couple of decades. I was pleasantly surprised to find he has expanded his view on the subject to include the application of reproducibility/replicability in other fields, and called out its misapplication (Nozek) and the ideological and political misuse.

    I think Goodman deserves defence from you attempt to portray him as a peripheral figure in the debate on this subject. Peng by contrast seems to be concerned with the specific issue of R-code and data transportability, while Goodman has a much more comprehensive take on the subject.
    In the link I gave above he discusses in detail the difference between reproducibility and replication –

    “the modern use of “reproducible research” was originally applied not to corroboration, but to transparency, with application in the computational sciences. Computer scientist Jon Claerbout coined the term and associated it with a software platform and set of procedures that permit the reader of a paper to see the entire processing trail from the raw data and code to figures and tables (4). This concept has been carried forward into many data-intensive domains, including epidemiology (Peng), computational biology (6), economics (7), and clinical trials (8).
    .
    (NSF definitions of reproducibility and replicability)
    .
    Although the preceding conceptual distinctions might seem clear, the definitions do not provide clear operational criteria for what constitutes successful replication or reproduction. Furthermore, the terminology is not universally used, and sometimes the meanings above are reversed.”

    And goes on to suggest a solution that goes beyond a cloud cache of data and code as a guarantee of credible and efficient science.
    Read harder ?

  63. dm said:

    “Exact reproducibility is not very interesting. There is a problem called “software rot”, which means even if I archive my (MATLAB) research code today, that doesn’t mean it will necessarily give exactly the same results in ten years time as it does if I run it tomorrow.”

    This is so true. That’s why many algorithms are given in pseudocode, and there is not even a standard for pseudocode and it can’t even be executed. We used a concept called executable specifications often.

    izen said

    ““the modern use of “reproducible research” was originally applied not to corroboration, but to transparency, with application in the computational sciences. Computer scientist Jon Claerbout coined the term and associated it with a software platform and set of procedures that permit the reader of a paper to see the entire processing trail from the raw data and code to figures and tables (4). This concept has been carried forward into many data-intensive domains, including epidemiology (Peng), computational biology (6), economics (7), and clinical trials (8).”

    To combat rot and promote reproducibility in the software age, I think the idea of scientific workflow and provenance was around prior to Claerbout and it has evolved since, with the concept of notebooks pioneered by Mathematica. This gives some of the history
    http://web.cs.ucdavis.edu/~ludaesch/pubs/scientific-workflows-encyclopedia-2009.pdf

  64. BBD says:

    Dave_G

    Steven was involved with the BEST project, just a heads-up.

    But despite the fact that that Koch-funded effort to discredit the temperature reconstructions didn’t pan out, Steven keeps up with the ‘Climategate’ this and that, always implying that there’s less-than-sound science we should be concerned about.

    Pay it no nevermind.

  65. angech says:

    “Best Schools has put together another very interesting list: Top 15 Climate Change Scientists – Consensus and Skeptics. the mainstream position will be represented by 10 scientists; the skeptical position by five.5 skeptical scientists: Lennaert Bengtsson John Christy Judith Curry Richard Lindzen Nir Shaviv”
    JCH says: June 6, 2018 at 3:08 pm “5 skeptics and 10 consensus, not quite 97% is it? Ever more abject ridiculousness. You’re like fountain.Two of them clearly are not skeptics: JC and LB. Even Lindzen, if you read the transcript of the physics society roundtable, is a complete dud as a skeptic.”
    I did not say it, JC, I found it. Argue your point with Best Schools, not me. And I acknowledged, did you miss it that JC was not a skeptic, yet.

  66. angech says:

    Dave_Geologist says: June 6, 2018 at 5:04 pm
    ” Methinks Richard Muller has a better handle on what matters than you do.”
    Is that the guy investigating Trump [joke].

  67. angech wrote “I know there are a lot more consensus scientists than skeptic ones and your interpretation is right ”

    In which case why mention the 97% unless it was just your usual trolling? Using arguments without caring whether they are true or not is pretty much the definition of bullshit.

  68. SM wrote “SM “yes isnt your area. and you got it wrong.
    You can always remember my guide to “try to have no opinion”

    I’m glad I took the trouble to check my recollection was correct, SM’s retraction made it all worth it. ;o)

  69. angech wrote “I did not say it, JC, I found it. Argue your point with Best Schools, not me. And I acknowledged, did you miss it that JC was not a skeptic, yet.”

    How many times has she published articles on her blog questioning whether the rise in atmospheric CO2 is anthropogenic? How many times has she commented to agree that we know it is predominantly anthropogenic in origin? How many times has she rejected the mass balance argument, which shows that the natural carbon cycle is a net carbon sink and hence is opposing the rise and not causing it.

  70. Dave_Geologist says:

    Indeed Paul, re coherency, it wasn’t new. People had been doing it for years using their own algorithms. Before that we used illumination and edge detection algorithms from the satellite imagery business. Indeed for some purposes they’re better, I started with ER Mapper when it was a baby back in the early 90’s (on satellite images when I was in frontier exploration) and used similar techniques in other software practically until I retired. Horses for courses.

    The problem wasn’t that they wanted to protect their algorithm. That’s fine. Although Trade Secrets would have been the right way to go about it. It was their claim that everyone else’s independently developed algorithms belonged to them as well. Which had been developed previously so there was lots of Prior Art. One reason they got away with it IMHO (apart from the usual excuse of overworked Patent Inspectors deciding on patents outside their field of expertise) was that the prior art was not published. Because everyone else decided Trade Secrets was the way to go. I realised that when the dust settled and various academics got access to 3D seismic workstations and commercial software and began publishing papers using coherency, written as if they’d just invented it (the application, not the software). The industry had been at it for at least a decade by then, but just not published. The exception to the sharing I mentioned is in front-end exploration, before you’ve got the contract tied up. There companies do jealously guard anything they see as a competitive edge. Once the contract is signed, it’s all about being efficient, not about being especially clever. Yes clever ideas can add value there, but it’s incremental value, not the all-or-nothing of a bidding round.

  71. angech says:

    “In which case why mention the 97% unless it was just your usual trolling?”
    So, in expert opinion there exists a 3% chance of being wrong.
    No certainty then.
    It is hard enough to put up arguments, sensibly, without anything that disagrees with your world view point being accused of being trolling. Why don’t you give me a break and try to work on convincing me with sensible arguments instead. I am trying to engage, admittedly it annoys you and I am sorry about that but discuss the arguments, not diss the person.
    Thanks.
    “angech wrote “I did not say it, JC, I found it. Argue your point with Best Schools, not me. And I acknowledged, did you miss it that JC was not a skeptic, yet.
    How many times has she published articles on her blog questioning whether the rise in atmospheric CO2 is anthropogenic? How many times has she commented to agree that we know it is predominantly anthropogenic in origin? How many times has she rejected the mass balance argument, which shows that the natural carbon cycle is a net carbon sink and hence is opposing the rise and not causing it.”
    JCH said June 6, 2018 at 3:08 pm
    “Two of them clearly are not skeptics: JC and LB. Even Lindzen, if you read the transcript of the physics society roundtable, is a complete dud as a skeptic.”
    So your view on JC as a skeptic is not shared by JCH, JC herself, WUWT [she was listed as a lukewarmer on the old site] or the skeptic community as a whole.
    Like me, she has acknowledged the warming effect of CO2. Did you miss that? The fact that she has raised and raises questions is Skepticism, not denial.
    A perfectly legal, temperate approach to science which you should embrace and answer.

  72. Dave_Geologist says:

    Steven was involved with the BEST project, just a heads-up.

    I was unaware of that BBD, thanks. BEST is a perfect example of what I consider good replication. Not FOIing the data and code, and nitpicking it for talking points which can be used on the Web, in the Press and with politicians. Instead doing the hard graft of compiling your own data and performing your own analysis, reporting the result even though it wasn’t what you expected to find, and acknowledging (very publicly in Muller’s case) that your preconceptions were wrong. At least in the case of all but one or two participants.

    BEST and Cowtan & Way are perfect examples of consilience and have moved the science forward in a meaningful way as well as firming up the previous consensus. A whole army of Auditors could never do that. Indeed the Auditors probably held the science back. It astonished me, coming from a place where I always run kriging as a benchmark, even if I decide in the end to go for something else, that it hadn’t been done before. It made me think that the main players had been so bruised by the organised campaigns of harassment and false accusations, that they were afraid to do anything which stepped beyond the simplest forms of data processing.

    Consilience has also settled the Mann/McIntyre dispute solidly in Mann’s favour. Even if McIntyre had done exemplary work, the National Academies report and the subsequent locker-room’s worth of hockey sticks have demonstrated who had the right of it.

  73. BBD says:

    And yet Steven still has concerns.

  74. On exact reproducibility, one of the projects I am working on at the moment is so computationally intensive that some of the individual processes comprising the “experiment” take so long that they can no longer be accommodated by the “long” queue (five CPU days) on the high performance computing facility. Today I have re-factored the processes, so that they now checkpoint, so I can just re-sumbit the unfinished jobs to the long queue and they will pick up again from (more or less) where they left off. Unfortunately the no longer give exactly the same result. This appears to be due to the third party library that does most of the numeric computation. Should I really be required to find out exactly what the issue with the third part library is, so the program is completely deterministic (even with the external interrupt), or is just giving statistically consistent results every time good enough?

    I’ve done checkpointing before with similar processes using my own machine learning libraries, and they do seem to be exactly reproducible, so I suspect the problem isn’t with the checkpointing code.

    Of course, I suspect that someone who wanted to cast doubt on my research could do so to the satisfaction of an audience that would also like to see doubt cast on my work on the basis of a lack of exact reproducibility.

  75. Jeffh says:

    You will notice that three or four of the five luminaries on Angech’s denier list are old. Over 60. Where is the fresh denier blood? The truth is that there isn’t much of it. The vast majority of the tiny minority of climate scientists that deny or downplay the human fingerprint on the recent warming are near or past the retirement age. Moreover, the list isn’t actually much bigger than these five. Throw in Spencer, Pielke Sr., Michaels and a couple of others and that’s it. Finito. This is colloquially called scraping the bottom of the barrel.

  76. tedpress says:

    Yes, Jeffh, younger climate scientists may well have learned to avoid the difficulties associated with disagreement on this issue. They’ve seen the targets painted on their back by Anderegg, Prall, et al (PNAS 2010), watched the absurdity of people with no history of expertise in SNP attacking Susan Crockford for unknowingly contributing to SNP issues amongst skeptics while showing no history of expertise on polar bears, etc.

    They know what happens to critics. They lose funding–just ask Roger Pielke Sr.

    You should consider the possibility that they have not disappeared, but are keeping a lower profile.

  77. Dave said:

    ” I’m convinced I made carbon nanotubes in 1976.”

    The resolution of the microscopy likely wasn’t high enough at the time to see the detail. Yet, electron diffraction might have picked the geometry out due to the asymmetry in the structure. Would have seen lots of streaking in diffraction spots. And the cylindrical nature would have given a Bessel function intensity profile.

    Historically, most structures were determined by diffraction before they were visible by direct visual instrument.

  78. Tom,
    I think you misunderstand how science works. Young scientists are often keen to develop their own ideas and to overthrow a paradigm. Good work is almost always recognised.

    They know what happens to critics. They lose funding–just ask Roger Pielke Sr.

    Almost everyone loses funding at some point. Most of us don’t assume some kind of conspiracy.

  79. “Almost everyone loses funding at some point. Most of us don’t assume some kind of conspiracy.”

    Svensmark and Calder’s book “The Chilling Stars” is an excellent example of that. I’d love to have been as well funded as Svensmark! ;o)

  80. Tedpress? Is that you, Tom? If yes, I want to look back in the pdc records to confirm your donations to Al Gore in the past. Please provide years, races, states, so I can confirm. This is just data checking. I would like to be sure that you are a reliable source when you post here.

    In the alternative, you can say, oops, I misspoke, I never donated to Al Gore’s election efforts. That really seems like the more likely scenario.

    Thanks

    Mike

  81. dpy6629 says:

    Goodman talks about an example where consilience gives a different result than each result individually. The problem here is that if there is a strong positive bias in the literature, consilience will not help much. The consilience will be wrong too.

    The talk seems to me to be more about “framing” than about the substance of the issues. Dealing with substance is always more productive. On substance, he is probably wrong about publication bias being to blame for the “crisis.” The root cause is it seems to me a cultural issue. Science is now almost completely funded by soft money and Universities highly reward those who excel at getting soft money. Top University “entrepreneurial” scientists can make quite impressive salaries. That leads to the need to produce a stream of positive and exciting results. The supply of scientists probably exceeds the demand for their services.

    There is also I think a reluctance to acknowledge some of the big failures of the science/government/industry complex in the past. Dietary fat “research” and the resulting guidelines are a good example where the public’s health was harmed over at least 50 years. All where complicit, the food industry, government regulators, and scientists. Carbohydrates were substituted for fat in “low fat” food products that could be priced higher than the “high fat” older product. That almost certainly contributed to the obesity epidemic and an increase in health problems like high blood pressure and diabetes. ironically, a high fat and protein diet actually can be quite useful in controlling weight.

    Replication is always good, but I’m not sure it will have much effect in terms of “fixing” the problems.

  82. Svensmark is an excellent example. He personally may have had some problems, yet his actions may have set the folks at CERN off on a wild goose chase. Doing fast-particle accelerator experiments at CERN certainly ain’t cheap!

  83. Dave_Geologist says:

    The problem here is that if there is a strong positive bias in the literature

    There is a strong positive bias in the literature for gravity dpy. Lets go of chair arms. Nope, I didn’t float away. Funny that.

  84. Jeffh says:

    Tedpress (Tom) your arguments are absurd. Being a climate change denier is a sure fire way to become well known even when your credentials are wafer thin or non-existent. Look at some of the bottom-feeders that are often promoted on denier blogs and by think tanks. Back up your argument with a citation, or proof of some kind. Otherwise please desist.

  85. dpy6629 said

    “There is also I think a reluctance to acknowledge some of the big failures of the science/government/industry complex in the past. “

    Right, controlled experiments trying to replicate the earth’s climate are expensive. How much of the CLOUD experiment in CERN was motivated by Svensmark?

    This is a PPT presentation by the CLOUD team leader, who mentions Svensmark for his correlation findings:
    http://slideplayer.com/slide/12963073/

    Yet, more recently

    ““The authors need to quantify the effects in an atmospheric model rather than just speculating,” says Ken Carslow, of the University of Leeds, UK, who has also studied potential links between cosmic rays and aerosol formation as part of CERN’s Cosmics Leaving Outdoor Droplets (CLOUD) experiment. “It’s a tiny effect and previous studies suggest it will not be important,” he states.”

  86. Joshua says:

    Tom –

    Yes, Jeffh, younger climate scientists may well have learned to avoid the difficulties associated with disagreement on this issue.

    They also “may well” believe that they’ve seen monkeys flying out of your butt. Do you have some evidence to back up your speculation? Or are you just inventing hypotheticals to reinforce your preexisting beliefs?

  87. Joshua says:

    dpy –

    “There is also I think a reluctance to acknowledge some of the big failures of the science/government/industry complex in the past. “

    I agree that such a reluctance exists to some extent. It only makes sense that it would.

    On the other hand, I also think that there is a fairly common tendency, among some who have any of a variety of “motivations, ” to extrapolate from relatively few “big failures of the science/government/industry complex in the past” to confirm their biases and as a result over-estimate the prevalence of those failures.

    And then they display a reluctance to face up to the outcomes of their motivated reasoning. For example, I have found when asked to provide evidence that they aren’t over-estimating the (relative) prevalence of such “failures,” they seem quite reluctant to do so.

  88. dpy6629 says:

    Joshua, Every failure is important and impacts real people. A huge failure impacts the health of hundreds of people. Improvement is always good, is it not?

  89. > Goodman talks about an example where consilience gives a different result than each result individually.

    No, he’s not. Consilience isn’t p-hacking.

  90. > The talk seems to me to be more about “framing” than about the substance of the issues.

    Follows DavidY’s usual peddling.

  91. BBD says:

    Top University “entrepreneurial” scientists can make quite impressive salaries. That leads to the need to produce a stream of positive and exciting results.

    I can think of nothing more positive and exciting that demonstrating that AGW is not going to be a significant problem. But, ivory crickets…

  92. Joshua says:

    dpy –

    Every failure is important and impacts real people. A huge failure impacts the health of hundreds of people. Improvement is always good, is it not?

    Sure. I don’t wish to minimize the harm from failures. The question is how can we maximize the rate of improvement.

    I think that exploiting failures to score ideological points, while entirely understandable and easily predictable, leads to sub-optimal progress.

    It’s a balancing act. Reluctance to address failures retards progress. Exploiting failures retards progress. We see quite a bit of evidence of both, IMO, in the public discourse recently. I don’t know if either one is more counterproductive than the other, but neither is inevitable, IMO, if people commit to diligence.

    It looks to me like Goodman adds perspective towards creating a balance. His talk focuses on an important, but largely neglected aspect. It’s complicated, less accessible, and less sexy than the more commonly found material – but should be interrogated before making overarching statements about a “crises” in science (based on replication studies).

    The existence of a positive bias in the literature is important but it is mostly orthogonal to Goodman’s talk. The existence of that positive bias is non-informative w/r/t the issues he discussed.

    The talk seems to me to be more about “framing” than about the substance of the issues

    I don’t fully understand the more technical components, but from what I could understand, I don’t agree that his talk was about “framing.” Developing a deeper understanding of replicibility and reproducibility is important, similar to the importance of understanding p-hacking. I think that there is much being said about a “crises” in science that lacks nuance, such as the importance of grounding claims about a “crisis” in the larger context. A nuanced approach is necessary, IMO, to avoid counterproductive alarmism and the potential for an associated, and counterproductive “backfire” effect.

  93. dpy6629 says:

    I don’t disagree Joshua. What has shocked me is how little the dietary fat failure has excited any response at all. It is important to learn from past failures especially large ones and there has been none of that at all. People are more focused on replication or on just generally admitting there is a problem. Nothing really has happened to make improvements, except perhaps in areas like particle physics. The suggestions for improvement mostly miss in my view the root causes which are cultural.

  94. JCH says:

    The food industry will do exactly what it did last time: buy congress; sell sugar.

  95. izen says:

    @-dyp
    “What has shocked me is how little the dietary fat failure has excited any response at all. It is important to learn from past failures especially large ones and there has been none of that at all.”

    That is because you are seeing it as a dietary fat failure.

    It is actually a sugar industry success. For a surprising number of decades they manipulated the scientific research to minimise any harm attributed to refined sugar and transfer a lot of blame for obesity and cardio-vascular morbidity to fats.

    It is a failure of the science, in that was inacapable of altering a false scientific narrative that had strong commercial support.
    But an outstanding sucess on the part of the food industry.
    As long as your culture values profits before scientific truth and the health of the population.

    https://jamanetwork.com/journals/jamainternalmedicine/article-abstract/2548255

    Only now are some governments trying to correct this ‘failure’ with a sugar tax. It has faced about the same opposition as a carbon tax…

  96. izen says:

    @-Joshua
    “For example, I have found when asked to provide evidence that they aren’t over-estimating the (relative) prevalence of such “failures,” they seem quite reluctant to do so.”

    I have no such aversion.
    There is a list of substances that garnered a significant amount of scientific evidence against their safety that was disputed, and regulation was at least delayed, or weakened.

    Lead, Asbestos, Arsenic, SOx NOx Ozone, CFCs, Organo-Phosphates, particulates Mercury, Neonicotinides, phalates, Tobacco… I have probably missed some out, I will leave it to otters to supply more examples with the obvious common feature.

  97. izen says:

    @-JCH
    “buy congress; sell sugar.”

    Damn that’s good!
    Regulatory capture in a nutshell.

  98. Joshua says:

    izen –

    There is a list of substances that garnered a significant amount of scientific evidence against their safety that was disputed, and regulation was at least delayed, or weakened.

    First, just straight up, I don’t see how that’s exactly evidence of a “crises” in science, or a: “replication crisis.” I also don’t see how that’s evidence that places the “failures” of science into the full range of context that includes the “successes” of science.so as to make some kind of an evidence-based argument about the relative magnitude. What you’re doing there, it seems to me, is rather like when people blame climate scientists for the lack of ACO2 mitigation policies, through some kind of faith-based reverse engineering process.

    Second, the inability of “science” to negate or overcome countervailing political or economic forces is rather an indirect way of discussing the “failures” of science, IMO – except at the point where \scientists are working directly in coordination with politicians or outcome-focused funders. Where such coordination exists, IMO, it is important to view it within the larger context which includes quantification of the extend to which scientists reject pressures for that kind of coordination, or simply perform science in environments where there is no such pressure.

  99. angech says:

    Michael Mann 52 means one under 60. Proofread proofread proofread angech.

  100. dpy6629 says:

    Geez Izen, Academic researchers produced the evidence the government relied on. Aren’t regulators supposed to use the best science and keep special interests honest? What this shows I think is that no one is blameless in this case.

    And also why didn’t anyone in the scientific community do a contrary analysis of the data Keyes used? It is all in the data Keyes’ carefully selected to make his flawed case. Could it be that even in the 1950’s the culture of science was flawed?

  101. I’m shocked to see how little this has excited any response at all from the Contrarian Matrix:

    A former Cambridge Analytica engineer says he’s received more than 30 job inquiries from recruiters and executives at big data companies in the weeks since the scandal-plagued data firm announced it was shutting down and filing for bankruptcy.

    The employee, who requested anonymity because legal proceedings surrounding the company are currently ongoing, told Business Insider that despite Cambridge Analytica’s litany of scandals, he has fielded numerous offers from recruiters and CTOs working with big data. And he said that some of his colleagues at CA have received similar levels of interest from recruiters as well

    http://www.businessinsider.com/cambridge-analytica-engineers-getting-flooded-with-job-inquiries-2018-5

  102. izen says:

    @-dpy
    “Academic researchers produced the evidence the government relied on. ”

    And the Sugar industry, directly or indirectly produced much of the funding for the academic researchers.
    In the field of dental research they encouraged and funded treatments that would reduce the harm of consumption, not the effect of reduced consumption.
    As the affected industry they demanded representation on the scientific bodies that decided what research should be done. Just as the Tobacco companies had done in the field of cancer research.

    @-“Aren’t regulators supposed to use the best science and keep special interests honest?”

    Scott Pruitt.

    @-“And also why didn’t anyone in the scientific community do a contrary analysis of the data Keyes used?”

    They did. And other scientists (Yudkin) and research emerged with contrary results implicating sugar.
    Guess what arguments were used by the dominant (industry supported) voices like Keyes, within the dietary science establishment to discredit any such work.
    (Hint, reproducibility, confounding factors, statistically weak…)

    @-“Could it be that even in the 1950’s the culture of science was flawed?”

    No, just prone to attack as it always is, if it produces results inconvenient to political power or commercial profit.

  103. izen says:

    @-Joshua
    “First, just straight up, I don’t see how that’s exactly evidence of a “crises” in science, or a: “replication crisis.””

    It isn’t. I’m with Goodman, most claims there is a crisis in science, especially a ‘reproducibility crisis’ are fake claims intended as a means of doubting, disputing, disparaging and denying an inconvenient scientific truth. The science stuff that is profitable without a threat to power and commerce does not get the skeptical treatment.
    (except from those that distrust power and profit; antivaxers?)

    @-“Second, the inability of “science” to negate or overcome countervailing political or economic forces is rather an indirect way of discussing the “failures” of science, ”

    Failures of science are rather indirectly caused by outside influences. I think it is hard to find major failures that are purely intrinsic. perhaps a case could be made for the CO2 saturation error, or the reluctance to accept the atomic/quantum version of physics in the 1900s.

    @-“Where such coordination exists, IMO, it is important to view it within the larger context which includes quantification of the extend to which scientists reject pressures for that kind of coordination, or simply perform science in environments where there is no such pressure.”

    Where a conflict exits between the science and commercial or political interests I don’t think there is an environment without pressure, even if it can be reduced to small and indirect.
    The extent to which individual scientists might reject pressures can be the degree to which they become public targets for other ‘interested parties’.
    It could look like….Advocacy!

  104. JCH says:

    Blame McGovern. Beats thinking.

  105. Not much of a response from the Contrarian Matrix:

    A Volkswagen engineer pleaded guilty to U.S. federal charges for his role in the diesel emission cheating scandal.

    James Robert Liang, 62, who worked at Volkswagen for more than 30 years, faces up to five years in federal prison. They were the first U.S. criminal charges in the Volkswagen (VLKAF) case, where the company sold about 500,000 cars loaded with software that cut back on pollution during emissions testing. The same cars emitted up to 40 times the allowed level of various pollutants when actually driven.

    http://money.cnn.com/2016/09/09/news/companies/volkswagen-engineer-emissions-scandal-guilty-plea/index.html

  106. dpy6629 says:

    Yes Willard, Those are all failures of regulatory authorities to carefully monitor those they are supposed to regulate. People haven’t changed since the Gilded Age. Teddy Roosevelt set up a system to keep corruption out. Perhaps we have lost our mojo and are returning to the Gilded Age.

  107. dpy6629 says:

    The FAA shows how this is supposed to work. The track record is phenomenal with commercial aviation having improved dramatically.

  108. dpy6629 says:

    Well Izen, There are always witches to be burned when things don’t go the way we want. The failures you point to are failures of the scientific/government/industrial system we have built up. Who is more to blame is not a useful exercise in my opinion. We simply need to do better.

    There were plenty of opportunities for Journals, editors, and academic scientists to publish skeptical results on the issue.

  109. > The track record is phenomenal with commercial aviation having improved dramatically.

    Indeed, DavidP:

    Boeing Co. has reached a tentative agreement to pay $615 million to end a three-year federal probe into two high-profile Pentagon contracting scandals, the Justice Department said Monday.
    It would be the largest financial penalty ever levied on a military contractor.

    The settlement would end one of the more tumultuous periods for the aerospace company, whose reputation was tarnished by the cases.

    The scandals involved allegations that Boeing improperly acquired thousands of pages of proprietary documents from rival Lockheed Martin Corp. to help it win rocket contracts, and that Boeing illegally recruited a senior Air Force procurement official while she was awarding contracts worth billions of dollars to the company.

    Under terms of the agreement Boeing will not face criminal charges, but it will accept responsibility for the conduct of its employees that led to the ethical lapses, according to a senior Justice Department official. The deal is expected to be signed in a few weeks.

    http://articles.latimes.com/2006/may/16/business/fi-boeing16

    Either we continue to play squirrels, or we return to SteveG’s points.

    Showing you have listened to the talk would also be nice.

  110. izen says:

    @-dyp
    “There were plenty of opportunities for Journals, editors, and academic scientists to publish skeptical results on the issue.”

    Plenty might be overstating it.
    But Journals, and editors certainly missed opportunities to be as skeptical as was warranted. They seem to have been all too easily infiltrated with an attitude of caution. This was not entirely unprompted, the most well known example, because all the documentation has been made public, is the strategy of suppression, distraction, concealment, and manipulation that was used to minimise the publication of ‘skeptical’ results in the field of tobacco use.

    https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1470431/
    “Manipulation.
    Some articles and presentations were vetted by the industry before publication or presentation. A Philip Morris document reports, “The VdC has influenced Dr Schmähl and his group to speak out against a poor publication which is hurting the industry. . . . The VdC is also influencing publications which will be presented at the Fourth World Health Conference that deals with the cost to the economy due to smoking.”

    Although some individual scientists did push back, often their work was marginalised and some resorted to dumbing, down, or hyping up their position in ‘popular media’ articles or books published outside the mainstream science press. Yudkin – Pure, White, and Deadly on sugar would be an example.

    @-“Who is more to blame is not a useful exercise in my opinion. We simply need to do better.”

    Agreed. Examining the processes that distorted science in some of these examples is pointless if it is to apportion blame. The reason to do it is to find out what to avoid and what to look out for to make things better.

    Follow the money would seem to be a good basic rule.

    Claims that the science is invalid or unreliable because it fails reproducibility or replication, (perhaps because of a chaotic process) is most often a sign of attack rather than support for a new scientific truth.
    Especially if that measurement or modelling difficulty occurs in other scientific fields without being a major inhibitant to utile outcomes.
    Like wings.

  111. BBD says:

    I think it’s very helpful that David articulates his concerns in such a revealing manner. Sound science, replete with deliverables, is a vital tool when it comes to ensuring that regulators are guided appropriately.

  112. Dave_Geologist says:

    Evolution has been questioned as a valid theory because it cannot repeat experiments.
    Only skimmed the paper but strangely, I couldn’t find one reference to evolution research. Not one. Nada. It appears to be all about ecology, and evolution gets dragged along because there’s a Thing called “tools for transparency in ecology and evolution”.

  113. Dave_Geologist says:

    (just to poison the well, some funding for this research was from a religious/spiritual philanthropic foundation.)
    A cynic might think that “tools for transparency in ecology and evolution” was a convenient Trojan Horse to get the paper quoted as “repeatability crisis in evolution”, even though they provide no evidence for same.

  114. Is being over 60 an excuse for trolling, for instance making statements like

    Judith Curry article “Best Schools has put together another very interesting list: Top 15 Climate Change Scientists – Consensus and Skeptics. “
    5 skeptics and 10 consensus, not quite 97% is it?

    and then disavowing responsibility for the validity of the evidence when it is questioned

    “I did not say it, JC, I found it. Argue your point with Best Schools, not me. And I acknowledged, did you miss it that JC was not a skeptic, yet.”

    This is one of the most egregious examples of BS I have seen from angech. He is questioning a very well researched finding on the consensus (multiple peer-reviewed journal papers) using a blog post reporting a list that doesn’t purport to be a sample of the best 15 climate scientists (and hence doesn’t tell you anything about the consensus level. And then angech isn’t even willing to defend his source.
    I’m sorry angech, but making arguments without caring whether they are true or not is BSing.

  115. Dave_Geologist says:

    What has shocked me is how little the dietary fat failure has excited any response at all. It is important to learn from past failures especially large ones and there has been none of that at all.

    Indeed it is dpy. The lesson to learn is that when the profits of billion-dollar industries are put at risk, or those of their investors, they can’t be trusted.

    The lesson for AGW is don’t trust ExxonMobil (pre-Tillerson), Chevron (probably), the Heritage foundation, Peabody Coal, the Cato Institute, anything funded overtly or covertly by the Kochs. Or any of the individuals identified in Merchants of Doubt.

    Do trust the IPCC. Because they don’t have billions in profits at risk, and their stakeholders, who know the truth will expose them to political and economic costs, have an effective veto on content in the Advice to Policymakers under the UN consensus procedure. Also trust BP, Shell Total etc., because they’ve publicly accepted the IPCC findings for decades, despite it being against their financial interest to do so. But watch closely to see whether they practise what they preach.

    Who do you trust dpy? Why?

  116. angech says:

    I forgot here everybody is under 60, except Victor and me [sorry Victor if that is wrong]

  117. izen says:

    @-angtech
    “I forgot here everybody is under 60, except Victor and me”

    Not quite.

  118. izen says:

    @-Dave_G
    ” a convenient Trojan Horse to get the paper quoted as “repeatability crisis in evolution””

    By the time they reach the Conclusions they seem to equate a lack of reproducibility, the computational Clarebout/Peng version, with the research being potentially damaging and a waste of funding.

    “Although the reproducibility rates in ecology are not currently known, if they were to approximate the rates in biomedicine or psychology, then we might expect that up to half of the current research expenditure in our own field has funded irreproducible research. The financial cost of these avoidable errors may be staggering, let alone the environmental costs.”

  119. dpy,

    What has shocked me is how little the dietary fat failure has excited any response at all.

    What do you think should have happened, and why? Also, what is the broader relevance? Can you try to be quite specific.

  120. Dave_Geologist says:

    Methinks I smell a rat izen. How do you get from “However, as we also mentioned above, less than 10% published articles sampled in conservation biology and ecology journals report the statistical power of their own research” to “Measuring questionable research practices in ecology and ­evolution”. As a section heading, just in case you missed it. Oh, but they don’t actually measure any. Just say it would be a nice thing to do. And in the conclusions “Although the reproducibility rates in ecology) are not currently known, if they were to approximate the rates in biomedicine or psychology, ..”

    So basically, a paper that found no replication crisis in ecology, but suggests that there might be one because there is one in biomedicine and psychology, didn’t look at evolution research at all but pairs ecology and evolution at every turn, is a challenge to evolution because evolution research is irreproducible? I feel like I’m a guest at the Mad Hatter’s tea party. If that’s the best Creationism has got, there’s no there, there.

    And I’m not even sure I agree with the premise of their statistical power argument. It’s a bit like the argument that papers which don’t attribute AGW to humans, even if they’re by people who have published parallel attribution research and are investigating the consequences of AGW, are neutral and not pro-AGW.

    In an unbiased literature, the proportion of significant studies should roughly match the average statistical power of the published research. When the proportion of significant studies in the literature exceeds the average power, bias is probably in play. Publication bias can result in a false positive error rate for the literature well beyond what is expected from the disclosed, accepted false positive rate.

    Surely the most likely source of publication bias is not inflation of positive results through p-hacking, cherry-picking or undisclosed researcher-degrees-of-freedom. It’s the well-known difficulty of getting null results published. Strong results don’t mention statistical power because from their viewpoint, it’s irrelevant (but it does make meta-analysis of the field more difficult so stating it should probably be encouraged). Weak but statistically significant results may quote statistical power as a reassurance that they’ve used a big enough sample and you probably are seeing a real, weak effect and not a chance false positive. Statistically insignificant results generally don’t get published unless it’s a burning issue or something that everyone expected to be true but maybe isn’t. Then you might use it as a caveat: “the results were not statistically significant, but the statistical power of the experiment was poor and if you really want to confirm that there is no correlation, you need to use a much bigger sample”.

    However, failure to publish non-significant studies is really only a problem if they replicate published, significant studies. Even there you need (say) 19 non-significant ones to cancel out the significant one. Or you need to know that the researchers tested 19 other explanations which were non-significant and published the one “significant” one. Failure to reach 95% significance doesn’t prove that the null hypothesis is right. Only that the hypothesis fails to reached the required significance threshold. The correct interpretation is not that the hypothesis is wrong. It’s that we don’t know. Understandably. “we don’t know” is a hard sell to editors. 😉 .

    * Missed a chance to tar evolution research there. Or maybe a peer reviewer had had enough.

  121. “However, as we also mentioned above, less than 10% published articles sampled in conservation biology and ecology journals report the statistical power of their own research”

    I may be being a bit dim here, but you only really need to report the statistical power if you are claiming that the lack of a significant effect is an interesting finding. If you have found an statistically significant effect (which is what we mostly are looking for) then power isn’t too important.

  122. > what is the broader relevance?

    The relevance, AT, is that according to our visiting engineer the talk seems to me to be more about “framing” than about the substance of the issues. From there you can peddle in just about anything that one can find of substance. It’s one way to frame the “but ze issue is” switch and bait.

    A form of “substance” abuse, rhetorically speaking.

    It’s all your fault, BTW. You decided to post something that doesn’t get to the substance. ClimateBall players need their daily dose of substance. If a thread isn’t dedicated to it, it needs to be corrected.

  123. Dave_Geologist says:

    You phrased it more concisely than me dikran 🙂 .

    The other area I was alluding to is where the paper is making the case for a larger trial. For example, using data-of-opportunity to suggest an effect might be present, not quite finding it but noting that the power was low for the effect size of interest, and recommending a larger, controlled trial. In medicine, small effect sizes may still be worth pursuing, e.g. successfully treating 10% or 20% of patients is not the best thing since sliced bread, but still useful.

    Similarly in oil exploration. In a stratigraphic play I was involved in, using a particular processed seismic amplitude anomaly to target oil sands only gave you about a 30% chance of success. But random drilling only gave 5%-10%. 30% meant it was worth pursuing because the successes delivered enough value to pay for a 70% failure rate. They didn’t deliver enough to pay for a 90%-95% failure rate.

  124. dpy6629 says:

    I don’t know ATTP what should happen. The question to ask is
    Why was there not a broader and more skeptical scientific debate surrounding the original findings? Peer review obviously failed here. It baffles me that some ambitious young scientist wouldn’t have built a career by pointing out the obvious flaws. Looking back on it, it points to some real issues.

  125. Dave_Geologist says:

    dpy, are you claiming that dietary fat does not contribute to CHD and only sugar does, or that dietary fat and sugar both contribute?

    If the latter, science got it partly right (eat less saturated fat), but not completely right (replace it with polyunsaturated fats not carbohydrates, especially not refined ones). Science works like that. The edifice is built brick by brick. See The Relativity of Wrong if you’re unable to grasp the difference between nonsense beliefs and the gradual improvement in scientific understanding.

    If the former, I want to see some evidence (but I’m not holding my breath).

    Why was there not a broader and more skeptical scientific debate surrounding the original findings? Peer review obviously failed here.

    1. Because the original findings regarding the CHD risk from saturated fat were correct. Just not the whole story.
    2. No it did not. Nothing sent for peer review is ever the whole story. Science Marches On. But one step at a time.

    some ambitious young scientist wouldn’t have built a career by pointing out the obvious flaws

    Please point out the obvious flaws. Note 1: blogs and newspaper articles don’t count. Note 2: remember that absence of evidence is not evidence of absence.

  126. dm said:

    “I may be being a bit dim here, but you only really need to report the statistical power if you are claiming that the lack of a significant effect is an interesting finding. If you have found an statistically significant effect (which is what we mostly are looking for) then power isn’t too important.”

    This is an interesting observation and one that is not discussed often. For example in tidal prediction analysis, the significance is never quoted even though many factors are included — so many factors that it appears to border on over-fitting. The initial fit is typically 4 factors, and then up to 7 for more accuracy, yet there can be more than 100 to 200 involved in a detailed model. That would look like extreme over-fitting according to a multiple regression statistical view and the penalty for number of factors would be severe, yet it is a real physical effect and so the statistical power or significance is less relevant. The value of the model is then based on how well the model performs during cross-validation experiments.

    So at some point the hypothesis is accepted, and from then on the statistics are not as important as during the initial hypothesis testing, from my bunny rabbit view.

  127. izen says:

    @-dpy
    “Why was there not a broader and more skeptical scientific debate surrounding the original findings? Peer review obviously failed here.”

    There was, for a few years as the evidence that while reducing fats worked, replacing them with sugar just made things worse accumulated the scientific debate progressed. Much to the alarm of the sugar industry.

    The failure was engineered.
    http://math.colorado.edu/~walter/MATH2380/Resources/sugar.heart.2016.pdf
    In 1954 the sugar industry identified an opportunity to increase consumption of its product by 30% by replacing half the fat in the American diet with sugar, on the grounds that early research showed reducing fat in the diet reduced cholesterol which was associated with heart disease.
    By 1962 there was research emerging that found that a low fat, high sugar diet increased serum cholesterol. Just as Yudkin had suggested in 1957.
    By 1965 the evidence in the scientific literature was becomng so strong that it made it into the popular press.
    The Sugar industry, in the form of the Sugar Research Foundation, SRF, which had been using its carefully nurtured contacts in the field to minimise the impact of these new findings, went on the offensive. –
    “On July 13, 1965, 2 days after the Tribune article, the SRF’s executive
    committee approved Project 226,31 a literature review on “Carbohydrates and Cholesterol Metabolism”byHegsted and Robert McGandy,
    overseen by Stare.” –
    “Nine months into the project, in April 1966, Hegsted told the
    SRF that the review had been delayed because of new evidence linking
    sugar to CHD: “Every time the Iowa group publishes a paper we
    have to rework a section in rebuttal.” The “Iowa
    group” included Alfredo Lopez, Robert Hodges, and Willard Krehl,
    who had reported a positive association between sugar consumption
    and elevated serum cholesterol level.
    |
    These internal documents show that the SRF initiated CHD research
    in 1965 to protect market share and that its first project, a literature
    review,was published inNEJMin 1967without disclosure of the sugar
    industry’s funding or role. The NEJM review served the sugar industry’s
    interests by arguing that epidemiologic, animal, and mechanistic
    studies associating sucrose with CHD were limited, implying they
    should not be included in an evidentiary assessment of the CHD risks
    of sucrose.
    —————-
    Lessons have been learnt. Industry targeted review papers and reports because such meta-analysis of the issue had greater weight and shaped the subject more than an individual research result.
    It is because of such past failures that the authors of review papers, and reports for governments or the IPCC are now required to fully disclose their funding and possible conflict of interests. And has strengthen suspicions of any industry funding of science ‘product’ as with W Soon.

  128. PP there is a difference between over-parameterised (more parameters than necessary) and over-fit (error reduced on the calibration set to the point beyond which the error on unseen/test data starts to gets worse again). You can have a model with too many parameters (even notionally an infinite number of parameters) that generalise well because they have not been over-fitted to the data (typically because you have a penalty/prior over the parameters). Traditional statistics doesn’t always make that distinction as the number of parameters is often the only control over model complexity.

    Note that you can over-fit cross-validation by testing too many hypotheses on it (or equivalently tuning hyper-parameters to minimise the cross-validation error, see this paper [of mine and Mrs Marsupial’s] for example).

  129. > The failure was engineered.

    I like that wording.

  130. dpy,

    I don’t know ATTP what should happen.

    Then why do you keep bringing this kind of thing up? I’m trying to get you to explain the relevance. If there isn’t any, then what’s the point of highlighting these kind of things?

  131. angech says:

    Dave_Geologist says: June 8, 2018 at 5:33 pm
    “dpy, are you claiming that dietary fat does not contribute to CHD and only sugar does, or that dietary fat and sugar both contribute?”

    Em.
    Having a heart and blood vessels contributes to CHD.
    The cause of CHD or the one most of us are interested in is acute or slow blockage of said heart arteries by atheroma, there are other causes by the way, admittedly rare.
    Atheroma, the buildup of cholesterol and fat laden plaques, as a disease, is of itself, only an outcome of the genetics of some humans and some groups of some other species.
    Hence some 20-30 % of us are genetically designed to develop atheroma and die from heart attacks or, elsewhere in the body of strokes, kidney failure or leg ischaemia at an earlier age than the rest of us.
    This will happen regardless of how much fat we consume in our diet. I.e. in a general sense dietary fat [and/or sugar] are not important. We all know of people fit and healthy and thin who have died unexpectedly in this way.
    As an ambit claim, not being a veterinarian, I am fairly sure that most animals, birds and fishes do not die of CHD, despite having coronaries. I think that there are only some certain breeds of mice and rats and other animals which have a propensity for CHD.
    In humans there are groups that have high cholesterol naturally, without dietary intake excess who have quite low rates of CHD and others who have extremely high rates.
    My view is that genetically you either have good or bad blood vessels. Other genetic factors, diabetes, and external factors, smoking can cause atheroma to develop. You only get epidemics of obesity in situations of increased life expectancy.
    So enjoy the sugar, enjoy the sausage, don’t eat too much and you will have done no harm to your life expectancy.
    Don’t have parents/family who died of CHD at an early age and do not smoke, unless you want to, of course

  132. angech says:

    “An Evidence-Based Guide for Coronary Calcium Scoring in Asymptomatic Patients without Coronary Heart Disease”
    and “The CAC score is fundamentally not normally distributed because of the large percentage of zero measurements, and hence is not amenable to a normalizing transformation, as noted by others”
    Basically some 20-30% of us have arteries that turn into stone. Nothing you can do to prevent it yet. You can pay for an expensive test [not recommended] to see if you are one of the lucky ones who will die of something else nastier or live in ignorance. One of the few tests without a Bayesian curve for the whole population.

  133. “Note that you can over-fit cross-validation by testing too many hypotheses on it (or equivalently tuning hyper-parameters to minimise the cross-validation error”

    If the cross-validation was performed on out-of-band data, and the error was minimized by tuning, this is not necessarily misguided. As an example, consider predicting an eclipse. If the model was trained on an interval of in-band data and then tested on the out-of-band data, it would be OK to recalibrate with the cross-validation error. But again this is for a model that is definitely statistically significant in the first place. The cross-validation error is serving more to tell you how well the training calibration works.

  134. dikranmarsupial says:

    PP “If the cross-validation was performed on out-of-band data, and the error was minimized by tuning, this is not necessarily misguided. ”

    No Paul, read the paper. The cross-validation error is a statistic evaluated on a finite sample of data. Thus there are two ways to minimise it, you can minimise it in ways that genuinely improve generalisation and in ways that exploit the noise (e.g. sampling variation) in the particular sample you have. The second way is over-fitting the criterion. It makes no difference whatsoever whether the it is out-of-band and it is minimising the error that causes the over-fitting in model selection. This is an issue that isn’t widely appreciated, but that doesn’t mean it isn’t a pitfall.

  135. I am reading your paper and trying to get the essence of it. My goal is to explore fitting a simplified model in the context of a single time series, which means a sample drawn from a single set of data points. The over-fitting danger is to take too short an interval or apply too many parameters. My approach is to use an independent validation set which is essentially an uncontaminated out-of-band part of the time series for cross-validation.

    The uncontaminated part of the time series is almost twice as long as the in-band time series data set that I have been analyzing. I have been doing cross-validation on the in-band set but having an uncontaminated set is preferred, according to what I have read. Of course the preferred out-of-band set is from the future, because that is the only one that is guaranteed to be uncontaminated due to the implied untrustworthiness of scientists.

  136. Basically some 20-30% of us have arteries that turn into stone. Nothing you can do to prevent it yet.

    I don’t believe that’s true.

    Calcification ( and cholesterol profiles ) occur because of inflammatory response.
    There’s evidence that the inflammatory response is due to insulin and glucose and also obesity.

    Cholesterol particles have many different types far beyond the ‘good’ and ‘bad’ designations.
    Some of the HDL particles transport fats from the body to the liver.
    Some of the LDL particles transport fats from the liver to the body.
    Other cholesterol particles are part of immune response and repair and probably yet to be discovered roles.

    When one is obese, or even normal weight but with fatty liver, the liver attempts to clear the harmful harmful fats ( necessarily created from alcohol and fructose metabolism ) by increasing LDL particles which transport fats away to the safe fat cells. But the fat cells in the obese are full, forcing fats where they can, unfortunately including the arteries.

    Limit carbs to about 100g a day, which also limits insulin.
    Limit fructose to desert once a week and perhaps fruit once a day.
    Limit alcohol.

    Do this and you will lose body fat and increase transport of fats away from your arteries to your liver for conversion.

  137. The basic rule is that the data used for performance evaluation should not have been used to make any choices about the model. Cross-validation is often used for tuning hyper-parameters, but of you do that then (i) it gives an optimistically biased performance estimate and (ii) you can end up with sub-optimal hyper-parameter estimates as a result of over-fitting the cross-validation estimate. The degree to which this is a problem varies from model to model, but in general the more hyper-parameters to tune the worse the problem is likely to be, the larger the amount of data the better.

    This includes “research degrees of freedom” where the researcher becomes part of the model selection algorithm by redesigning the model in the light of observed performance on the held-out data.

  138. “This includes “research degrees of freedom” where the researcher becomes part of the model selection algorithm by redesigning the model in the light of observed performance on the held-out data.”

    Based on holding-out data during the training process, I discovered that there were several parameters that could be dropped when cross-validated with the uncontaminated data. So in this case, redesigning the model to be simpler is telling me that it is heading in the right direction. As always, without the benefit of a controlled experiment, all you can do is be creative in evaluating the data.

  139. Dave_Geologist says:

    Nothing yet dpy? Try Google Scholar. Looks like the scientific community was not sleeping on the job and was well on the case, even back in the 1960s. Policymakers, not so much. Wonder why? Naomi should add a chapter to the next edition of Merchants of Doubt.

    And according to a recent review, the situation is more nuanced. Fructose may be no worse than sucrose, and sucrose may be no worse than other calorie sources, at least where it provides <25% of the daily calorie intake. At lower levels the primary health-impacting mechanism may just be weight gain. Which still leaves sugar in a bad place, but through the "empty calories" argument (the only one I was familiar with before researching the topic). A litre of your favourite carbonated drink contains as many calories as five bananas or five slices of bread, but is a lot easier to down and a lot less satiating. However, others disagree, including about fructose The Evidence for Saturated Fat and for Sugar Related to Coronary Heart Disease, and from 1998 Reassessing the Role of Sugar in the Etiology of Heart Disease, and in finding a synergistic response to saturated fat and sucrose intake (i.e. the whole is greater than the sum of the parts, making confounding effects even harder to untangle)

    Substitution of 40 % of calories as sucrose for starch resulted in higher serum cholesterol, phospholipid and triglyceride levels in all patients. The mean increases were highly significant (P < 0.001, < 0.001 and < 0.01, respectively). The differences in serum lipids, especially triglyceride, were more marked for type III, IV or V than for type II patients. In view of our previous results (Parts 1 and 2 of this series) with diets high in polyunsaturated fat and low in cholesterol which showed no uniform hyperlipidemic effect for sucrose, the present findings suggest that there is a synergistic effect between dietary sucrose and animal fat.

    So part of the science “failure” is because (a) there is still ongoing research and (b) the situation is far more nuanced than tobacco/lung cancer or anthropogenic CO2/climate change. With multiple, real confounding factors and synergistic effects, as opposed to unicorns and wishful thinking. People don’t eat saturated fat or sugar on their own, but mixed into foodstuffs. People who eat a lot of sugar are often obese, which also increases CHD risk. Experiments which show negative effects at high doses may be over-loading the body’s ability to cope, and perhaps can’t be extrapolated to more realistic, medium doses (but since a litre of fizzy drink supplies about 20% of a teenager’s daily energy requirements, it won’t take much more sugar to go above the first paper’s 25% threshold). Partially substituting sugar for saturated fats may be worse than sticking to fats, but increased sugar intake may be OK if you replace all your saturated fat intake with polyunsaturated (not hydrogenated vegetable oil: hydrogenation = increasing saturation).

    Added sugars drive coronary heart disease via insulin resistance and hyperinsulinaemia: a new paradigm (2017, with a quote from 1966 (the new paradigm is of course the biochemical mechanism, not the epidemiological evidence): ‘I know of no single acceptable study that shows a high intake of sugar in a population that is almost entirely free from heart disease.’1—John Yudkin.)

    Fructose and Cardiometabolic Health: What the Evidence From Sugar-Sweetened Beverages Tells Us (from 2015, reviewing all the mechanisms)

    Sugar-Sweetened Beverages and Cardiovascular Disease (2012)

    Sugars, hypertriglyceridemia, and cardiovascular disease (from 2003, on the biochemical mechanism

    Dietetic aspects of atherosclerosis. (Rudkin 1966; in the peer-reviewed journal Angiology)

    In this paper, I do not intend to review the whole field of the work that has been done on the relationship between diet and ischaemic heart disease (IHD), partly because the field is now so enormous, and partly because there already exist many excellent reviews. These reviews are almost exclusively concerned with assessing the evidence connecting the disease with dietary fat. Here I shall briefly survey the evidence that suggests that the dietary factor involved is not fat but sugar (sucrose). This evidence is threefold; evolutionary, epidemiological, and experimental.

    Sugar intake and coronary heart disease (a contrarian view from 1971, challenging Rudkin and saying it will always be difficult to prove due to confounding factors). I see he also criticises its association with dental caries. But. Hey!, maybe you can build immunity. A Scottish cohort with no CHD ate twice as much sugar as another cohort with CHD. Must replace that diet Irn Bru with the real thing 🙂 .

    Interesting that the search turned up only one study saying sugar was (probably?) OK (the other said possibly OK if it makes up less than 25% of your calories and it doesn’t cause you to put on weight). And it was from almost half a century ago, from a contrarian who also argued against it causing dental caries.

    Hmmm, I think I see a pattern here. ABC, Anything But Carbon; ABS, Anything But Sugar. Tantalisingly familiar eh? Lone scientist speaks out for industry against the community. And it didn’t work. Why? Because Science, because Scientists and because Peer Review.

    Next time you throw in a gotcha, dpy, do some research first. Maybe then you won’t shoot yourself in the foot so badly. The sugar/CHD story paints a very clear picture when it comes to assessing the reliability of scientific claims. Trust the bulk of the peer-reviewed literature. Ignore the handful of contrarians (in this case, AFAICS, one) coming up with outlier results which just happen to suit the purposes of vested industrial interests (also, in the case of AGW, political or religious beliefs which are incompatible with the solutions, or with the idea that God would break His promise to Noah).

  140. Joshua says:

    TE –

    From reading many comments in the “climate-o-sphere” on the topic of the relationship between diet and human health, I have concluded thst one of two things is true: either, (1) some people really are smart enough master the vastly complicated network of evidence related to climate change to reach highly certain conclusions AND still find time to master the vastly complicated network of evidence related to the relationship between diet and human health to reach highly certain conclusions, or (2) people who tend to overestimate their ability to master vastly complicated networks of evidence to reach highly certain conclusions on one topic tend to overestimate their ability to master vastly complicated networks of I nformation to reach highly certain conclusions, more generally.

    My general impression is that some smart people tend to overestimate their ability to master vastlly complicated networks of information, but in the other hand, perhaps if I were smarter, I would have the ability to master all the evidence on the vastly complicated topic of how people reach such highly certain conclusions from synthesizing vastly complicated networks of information.

  141. Curry is at it again with a blog post called “Beyond ENSO”. Just watch how she works her agenda. In political and policy circles, she stresses uncertainty and the difficulty in being able to make predictions with respect to climate. Yet, in her capacity to make money, she positions her company as being able to predict ENSO and climate.

    Watch the competing hypotheses bounce back and forth. This hucksterism is second only to Trump.

  142. Dave_Geologist says:

    if I were smarter, I would have the ability to master all the evidence on the vastly complicated topic of how people reach such highly certain conclusions from synthesizing vastly complicated networks of information.

    There’s an easier approach Joshua, which also comes in handy if you can master it but are just lazy or don’t have time. Go with the consensus of peer-reviewed literature. If it’s an important enough issue, there will be the equivalent of a National Academies report, IPCC report, Royal Society report etc., which summarises the consensus for you. failing that, Find a top review journal (e.g. Earth Science Reviews in the case of geology) and search within it. If you’re feeling diligent, check out the lead author on Google Scholar. You should find lots of papers with lots of cites, mostly positive.

  143. Dave_Geologist says:

    For clarity Joshua, I realise the quoted text was referring to TE. The advice should have bee directed to him.

  144. “So in this case, redesigning the model to be simpler is telling me that it is heading in the right direction”

    No, that is a degree of researcher freedom every time you do it. Everytime you do it, you bias the performance estimate, by how much depends on the nature of the problem, but it is not something to ignore.

  145. “all you can do is be creative in evaluating the data.”

    the laws of statistics apply, even when you don’t know about them. Being “creative” can be a bad thing as well.

  146. dm said:

    ” you bias the performance estimate, by how much depends on the nature of the problem”

    Yes, and I know the problem that I am working on and so know the characteristics of the physics and constraints, which goes into the simplification I apply. So if I had included what I thought were second-order effects in the in-band fit, but find they were not important in the out-of-band fit, then they may have been over-fitted parameters and therefore misidentified as being crucial.

    “the laws of statistics apply, even when you don’t know about them. Being “creative” can be a bad thing as well.”

    I first discussed this in the context of a tidal analysis problem. There are no statistics to speak of here, as the behavior is completely deterministic and the noise level is typically very low.

    This is going full circle because the discussion started with cases in which the statistical power is irrelevant.

  147. PP In my paper I include an example where over-fitting in model tuning results in a model that under-fits the data. Ending up with a more simple model does not imply that you are not over-fitting the tuning criterion.

    Now sometimes this sort of “researcher degrees of freedom” issue is unavoidable. In that case, the best thing to do is to acknowledge it and give an explicit caveat. Taking the “its not a problem in my research” is recipe for over-fitting and is a “bad smell” in analysis (looks like an interesting paper).

  148. Joshua,

    Climate and Diet are similar in that there aren’t readily available controlled studies.

    Diet studies, that span a human life with strictly controlled eating are not feasible.
    However, I have performed the study on myself.

    Using the principles outlined above, I lost forty pounds within a year and twenty more within the following year and have kept it off for five years now. I also nearly doubled my HDL number.

    Both things (losing weight and raising HDL ) were said to be extremely rare but adhering to known scientific principles make them straightforward to achieve.

    Of course things are multi-factoral.

    But insulin is clearly the fat storing hormone and carbohydrates stimulate insulin more than other foods do. Also, obesity and fatty liver coincide with atherosclerosis by known transport and filled fat storage.

    Climate and diet also have a connection.

    Ancel Keys had a plausible but simplistic and incorrect theorem that dietary fat caused heart disease. Like the climate movement, he steamrolled opposition and climbed to political power ( American Heart Association ) to promulgate his ideas ( like the IPCC ). He is now in disgrace.
    Climate hysterics promulgate the idea that thermal energy imbalance will lead to great harms. I believe the balance of evidence is more and more proving this incorrect and like the lipid hypothesis, will be later viewed with derision.

  149. An example of being creative when it comes to tuning a model is to allow a period to fluctuate slightly if it is using an iterative optimization procedure. This might help the search avoid getting stuck in a local minimum. The fluctuating period is not used in the end because that would be an over-fitted tuning, but is simply a means to an end.

  150. Joshua says:

    TE –

    However, I have performed the study on myself..

    Another aspect of the patterns I’ve seen from reading comments from many people who have great certainty about their interpretation of the complicated evidence on climate change, is that many of those folks also, quite frequently, extrapolate from their own experiences – apparently thinking that in doing so they’re drawing conclusions in a scientific manner.

    Perhaps the best example of this that I’ve seen,is when “skeptics” skept-a-explain how the public formulates opinions on climate change by describing their own trajectory of opinion formation on the subject. It is a striking pattern, because it is extremely unscientific behavior (IMO), being displayed by very intelligent and knowledgeable people who seem to have a great deal of experience with scientific analysis. What I always find interesting is when “skeptics” extrapolate from the example of their own observations of their own process w/o even attempting to reconcile their own observations about their own process with the scientifically collected evidence that is readily available. That is particularly fascinating since they don’t even seem (usually, that I can see) to account for the fact that merely by focusing so much on their own process of opinion formation about climate science, they are necessarily an outlier in many important respects.

    I would think that with along your appreciation of the pitfalls of trying to draw conclusions about the interaction between diet and human health outcomes w/o the ability to perform RCTs, you should also have a solid understanding that convenience samples of incredibly small size, which have no hope of being anywhere near representative across a diverse population, might be meaningfully instructive as to the range of possibilities but are next to useless for drawing conclusions about cause-and-effect.

    As I’m sure that you know, there are problems with an epidemiological approach to disease science broadly, but what you’re describing isn’t even close to the kind of analytical process epidemiologists undertake (e.g., diagnosis, description, investigation, intervention, analysis). But even more problematic is your failure to account for the most components of a scientific approach which build in a process for controlling for biases and confounds.

    I would argue that you have not actually performed a “study” on yourself; instead, I would say that you have performed an uncontrolled experiment on yourself. I would reserve the term “study” for something at least somewhat scientific. You haven’t done any of many steps that people usually take when the perform a “study,” such as to trying to control for any variety of confounds (e.g., characteristics of your specific biological profile, a Hawthorne effect or any other of a variety of psychological confounds, etc.)

  151. izen says:

    @-TE
    “Ancel Keys had a plausible but simplistic and incorrect theorem that dietary fat caused heart disease. Like the climate movement, he steamrolled opposition and climbed to political power …”

    That’s the wrong way round. It is because he already had some status within the field that he was targeted by the industry, same with Stare mentioned in the paper above, and there were others. The methods that industry uses to co-opt scientists and capture the narrative are most fully exposed in the Tobacco papers.
    It is because of those past failures of science to resist undue influence that any reputable research and researcher declares the funding and any conflict of interest. Why it would not be acceptable for any contributor to the IPCC reports to have accepted payment from exxon for writing their review or report.

    Ancel pushed the industry line because he was part of a engineered mechanism that had ensured regulatory capture of the science as well as the political governance and media narrative by industry. A pattern repeated in other fields, tobacco, Seitz and Singer, Climate…

    His disgrace is because he fell prey to a well documented method that commercial interests use to deal with inconvenient science. It is unclear what commercial interest would be manipulating scientists and science to PROMOTE an inconvenient truth. You have to invent NWO conspiracy myths.

    @-“Climate hysterics promulgate the idea that thermal energy imbalance will lead to great harms.”

    Climate realists communicate that agricultural ecologies, the probability of extreme weather events and sea level rise have already happened and will continue with a similar trend if cumulative emissions further shift the energy balance.

    Climate lukewarmers promulgate the idea that further changes in the thermal energy balance will cause negligible harm.

    Both hysterics and luckwarmers separate ‘Harm’ for the material events that cause it. The concept gets discussed as more or less without reference to the physical changes and ecological shifts. Which physical effects of a thermal imbalance do you think will NOT happen?

    @-“I believe the balance of evidence is more and more proving this incorrect and like the lipid hypothesis, will be later viewed with derision.”

    Your belief about the balance of evidence is not widely shared. The consilience of evidence, as more and more has accumulated, is indicating greater certainty that the realists are correct.

    And the lipid hypothesis is not regarded with derision, just as incomplete without the consideration of other dietary factors.

  152. angech says:

    Turbulent Eddie says: June 9, 2018 at 1:27 pm
    “Basically some 20-30% of us have arteries that turn into stone. Nothing you can do to prevent it yet”. I don’t believe that’s true. Calcification ( and cholesterol profiles ) occur because of inflammatory response.There’s evidence that the inflammatory response is due to insulin and glucose and also obesity.”

    Belief is only a substitute for knowledge, TE.
    You know that.
    Calcification certainly can occur in response to and as part of an inflammatory response. But that is not the only reason. Note that we have bones and teeth in our bodies made out of major calcium depositions without a hint of inflammation. Look at sharks, teeth yes, bones no.
    The body genetics also determine which type of inflammatory response can occur, where, and if it will produce calcium deposits or not.
    There is a theory that all atheroma is due to injury and inflammation in response to circulating bacteria. I do not hold much sway in it yet if true it would mean it was bacteria , not inflammation that is the cause, correct? In which case people would advocate taking antibiotics to prevent heart attack.
    Wait they do. Aspirin reduces inflammation and drops the risk of heart attack [and most cancers] by 50%. But food does not cause the inflammation, our rogue [genetically misprogrammed] cells do.
    Cholesterol profiles might change in the presence of an inflammatory response. I do not know. But cholesterol profiles are programmed in. I have a cholesterol at times usually under 4.0 [Australian levels] Wife has a Chol around 7.8. This has been the case for the last 40 years. We eat and drink the same food and take no medications. Our true risks are unknown as we have not had the CAC.
    Look it up very interesting.

  153. angech says:

    Joshua says: June 9, 2018 at 8:41 pm
    “Another aspect of the patterns I’ve seen from reading comments from many people who have great certainty about their interpretation of the complicated evidence on climate change, is that many of those folks also, quite frequently, extrapolate from their own experiences – apparently thinking that in doing so they’re drawing conclusions in a scientific manner.

    convenience samples of incredibly small size, which have no hope of being anywhere near representative across a diverse population, might be meaningfully instructive as to the range of possibilities but are next to useless for drawing conclusions about cause-and-effect.”

    Convergence , where two completely different species end up looking the same or two sinusoids crossing in opposite directions.
    Spot on Joshua. Just do not limit it to skeptics. It happens in the best of circles.

  154. Dave_Geologist says:

    It happens in the best of circles.

    Very true angech. But in the best of circles, we have control mechanisms. In the case of climate science, consilience of everything from quantum mechanics, through lab and field measurements, to good old fashioned thermometers. That’s how we know Mann, Hansen, Hausfather and Muller are right, and Curry, M&M, S&C, Lindzen and Soon are wrong.

  155. dpy6629 says:

    ATTP, I’ll get to the relevance.

    If you cut through the ideological slant of Izen and Dave’s comments, they paint a picture of 50 years of work and hundreds of papers not giving a correctly balanced picture and being unable to influence regulators. There was a money influence, but there always are such influences (particularly with the recent academic demand for soft money). If the system was working correctly, journal editors and reviewers and fellow scientists would have given a more balanced picture. There would have been a little bit more zeal to point out the glaring statistical flaws in Keyes’ original analysis.

    There were lots of consilience studies and this directly goes to Goodman’s point about conciliance can give a more correct picture. If the studies themselves are biased and suffer from in particular positive bias, the conciliance will have that same problem.

    I don’t know what the best way to address this is, but consilience won’t help much just as replication won’t help much (as Goodman points out correctly I think) and I said above. It’s clear that one thing medicine has done that helps is having outside statisticians help design every study and analyze the results. That’s a good thing journals can do.

  156. dpy,

    I’ll get to the relevance.

    When?

  157. izen says:

    @-dpy
    ” There was a money influence, but there always are such influences (particularly with the recent academic demand for soft money). ”

    Just because money always tries to influence inconvinient science does not mean it is pointless to try and exclude, or at least reduce the capture of a field of science.
    The demand for ‘soft’ money is because goverments have been abdicating their responsibility to fund R&D. It is as much a civic duty as establishing a police force.

    @-“If the system was working correctly, journal editors and reviewers and fellow scientists would have given a more balanced picture. ”

    Why the system was NOT working correctly can be traced to a specific cause. That is true whatever inclination, or slant you look at it from.

    @-“There would have been a little bit more zeal to point out the glaring statistical flaws in Keyes’ original analysis.”

    AFAIK, there were no glaring flaws in Keyes work by the standards and methods of the time. The problems arose from industry influence and funding acting to shape any further development in the field. Often by directly funding some scientists to generate product that reproduced the results they liked, and avoided new lines of evidence that might produce results they didn’t.

    @-“It’s clear that one thing medicine has done that helps is having outside statisticians help design every study and analyze the results.”

    That is rare. Partly because there is a shortage of outside statisticians to go round. Especially ones capable of dealing with the sort of data involved in different fields of biomedical research. They tend to already be working in the field rather than being outsiders. Nozek shows what outsiders can get wrong if they try to analyze without understanding.

    Goodman mentions, and is a source for some of the improvements that have been made. Pre-registration of aims and methods to avoid p-hacking and post-hoc hypothesis. Less tolerance of papers that fail to meet sample size and experimental design guidelines.

    The idea that big Pharma would allow outsiders, even statisticians, to design their research and analyze the results when it is all so commercially sensitive is endearingly naive.

  158. izen says:

    A short additional point.
    The biggest improvement made to the integrity of the science research in the contested fields has not been making data openly available.
    It is requiring transparency of the source of funding.

  159. Reproducibility has never stood in the way of the Global Warming Policy Forum.

    Benny Peisner running a deadpan spoof of Cli-Psy
    “https://www.thegwpf.com/the-neurobiology-of-climate-change-denial/

    that runs afoul of the GWPF’s failure to recall its own decade-long track record of mistaking comedy for fact:

    https://vvattsupwiththat.blogspot.com/2018/06/this-is-your-brain-on-contrarian.html

  160. dpy6629 says:

    ATTP, Did you read the last paragraph of my comment? It is quite relevant to one of the substantive point Goodman made on consilience. I also agree with his other main point.

    In all honesty those two points were all I got out of Goodman’s talk perhaps because I missed some other things.

  161. dpy6629 says:

    Izen, Just a question about funding disclosures. Normally those I would think don’t make a difference as to whether a study is considered in a meta-study. That would be based on things like sample size, predictive power, etc. Funding disclosures are normally required by those providing the funding so they can get credit for the work.

    I think you are wrong about outside statisticians in medical studies. My former boss, a professional statistician who had done this many times pointed out that to be taken seriously in medicine you should employ an outside statistician.

  162. dpy6629 says:

    There was glaring selection bias in Keyes’ original study. He selected a few European countries from a much larger set of data.

  163. dpy6629 says:

    Of course Big Pharma would accept outside statisticians if it was an FDA requirement. They might not like it but they would do it. The FAA is much more intrusive and aircraft makers accept it as a cost of doing business.

  164. Joshua says:

    angech –

    Just do not limit it to skeptics.

    Of course not. It’s just that some “skeptics” make nice tidy examples with which we are all quite familiar!

  165. dpy6629 says:

    Izen, With respect I think you are wrong on the soft money issue. Universities have implemented getting soft money as a prime criterion for salary and advancement. It enables them to fund graduate students who then pay tuition, hire post docs, and pay for large laboratories and incidentally pay high University overhead rates to support often very high administrator salaries and benefits. At places like MIT, the number of funded research professors has not changed much. But to get paid megabucks you need to find lots of soft money. Even at MIT, there is a not insignificant cadre of “research engineers” paid exclusively by soft money.

    Part of this is because the scientific enterprise has grown so fast and has produced quite an excess of trained scientists. But that’s another story.

  166. izen says:

    @-dpy
    “Of course Big Pharma would accept outside statisticians if it was an FDA requirement. They might not like it but they would do it.”

    At present they are fighting hard against any such regulatory requirement.
    Some have voluntarily agreed to some of the provisions of pre-registration and data transparency, but with loopholes.
    The regulatory capture of the FDA is most clearly evidenced by its approval of prescribing opiods without any research into their long-term effectiveness in pain relief.
    That is not just manipulating the science, but avoiding the research that might, and does, destroy their argument for approval.

  167. dpy,

    With respect I think you are wrong on the soft money issue. Universities have implemented getting soft money as a prime criterion for salary and advancement. It enables them to fund graduate students who then pay tuition, hire post docs, and pay for large laboratories and incidentally pay high University overhead rates to support often very high administrator salaries and benefits.

    A very US-centric view of things.

  168. dpy,
    A question you’ve never answered (as far as I can remember) is how you keep these “outside statistician” independent. Who pays them? How do they advance in their careers? Do they take any pride/ownership of their research? Is there a limit to their involvement (i.e., if they do too much work in this research area then they won’t really be outside anymore)?

  169. Dave_Geologist says:

    If you cut through the ideological slant of Izen and Dave’s comments

    Really dpy? If you do the search yourself you’ll see I selected indiscriminately from the top hits. There only was one absolving sugar, and out of interest I searched for that author, knowing the track record of certain physicists on multiple industry-supporting topics. And lo and behold he defends sugar against dental caries. On the first page.

    Ideological slant? Projection, much.

  170. Dave_Geologist says:

    If the system was working correctly, journal editors and reviewers and fellow scientists would have given a more balanced picture.

    The scientific system was working correctly dpy. It painted a clear picture with sat fat, sugar, obesity and various other factors contributing.

    Just add it to the list of climate change, ozone hole, tobacco, second-hand smoke, acid rain, and falsely blaming Rachel Carson for malaria deaths when (a) DDT was not banned for malaria use but (b) in many parts of the world the mosquitoes are immune – which would have happened faster with indiscriminate agricultural use. One or a few contrarians buck the consensus and get massive political and PR support. And in many cases, direct (but secret) industry funding. Politicians are bought or persuaded to toe the industry line, and use the contrarian scientists as cover. Nothing changes. And nothing will change until politics and corporate morals change. Scientists don’t have the power to do that. The public, including people like you if you’d only lift the scales from your eyes, do.

  171. Dave_Geologist says:

    If the studies themselves are biased and suffer from in particular positive bias, the conciliance will have that same problem.

    Which of course is also true of gravity, non-geocentric and non-heliocentric astronomy and every other aspect of modern science. Very convenient for deniers to say that because everyone agrees with something, it must be wrong. Add Motivated Reasoning to the (intellectual) crime of Projection.

    And of course a mandatory dose of Conspiracy Theory. Out of interest, are you one of those who believe Trump did have a bigger crowd than Obama? That Clinton brushed off the incompetent congressional witch-hunts because she’s a superb liar, not because she just kept on telling the truth end there was no there, there? That NASA faked the moon landings? Were you suckered by Pizzagate?

  172. Dave_Geologist says:

    In all honesty those two points were all I got out of Goodman’s talk perhaps because I missed some other things.

    Ah, our old friend Motivated Reasoning rears its ugly head again 😦 .

  173. Dave_Geologist says:

    Funding disclosures are normally required by those providing the funding so they can get credit for the work.

    Of course. So why are the funders of the industry-supporting contrarians so shy about taking credit for their public service? Just askin’.

  174. izen says:

    @-dpy
    ” Funding disclosures are normally required by those providing the funding so they can get credit for the work.”

    I suggest you look at the case of W Soon who was funded over a million by exxon, the Kochs and the ‘Donor Trust’ to ‘generate product’ in the form of review papers that attributed most global warming to solar activity.
    It was a condition of the funding that he did NOT disclose the source.

    He was a collaborator on the Monckton paper that claimed NONE of the participants needed to disclose any outside funding, because they had done all the work in their ‘own time’.

  175. izen says:

    @-dpy
    “If the studies themselves are biased and suffer from in particular positive bias, the conciliance will have that same problem.”

    I think this underestimates the range and role of consilience.
    The assessment is not confined to ‘the studies’.
    It is the match with other areas of the subject, and other fields of science.
    So the conclusions that could be drawn from the Six Cities air pollution study were not only consilient with other air pollution studies, but matched the pattern of results in epidemiological studies of other factors impacting illness. Like smoking.
    And were consistent with the mechanistic understanding of the biochemistry and phisiology that underlay the disease process.

    Consilience in climate science is in part the match between the thermodynamics of the system over different timescales and volume. So the seasonal and volcanic responses match the decadel and millennia forcings of CO2 and orbital precession/libration(?).

    It is this consistency with aspects of the physics, chemistry and biology that are well established independently of climate science that strongly indicates (!) that there is not a positive bias because of that consilience.

    @-“just as replication won’t help much (as Goodman points out correctly I think) ”

    He pointed that out in the context of the Six Cities pollution study. Which even after it was reproduced, (exactly copied with the same data and methods) and replicated, (same results with different data and methods) was STILL resisted because of the ‘inconvenient’ conclusions.
    And can be tossed out on Scott Pruitt’s whim.

    Is that one of the two points you got out of the Goodman lecture ?
    (it is very good, but dense. It can repay re-watching, you may miss less )

  176. Steven Mosher says:

    ATTPs thoughts

    View story at Medium.com

  177. Steven,
    I did see that. I think I saw something ages ago suggesting that the critic had made a mistake. They’re now claiming that they haven’t, but I’m not sure.

  178. Here’s an article I read a couple of years ago. I don’t know if anything has changed.

  179. Dave_Geologist says:

    A bit young to have gone Emeritus but it can happen. Hmm… businessman with a decades-old relevant degree but no subsequent experience in the area comes up with a crackpot theory, self-publishes error-strewn work, and accuses NASA and others of data fraud. Now where have I heard that one before?

    Let’s just say that my mind is not so open that my brain has fallen out.

  180. I was scanning through the more recent article and paper and it seems to have changed from they’re making huge mistakes, to they’re over-confident and the uncertainties are larger than they claim.

  181. Dave_Geologist says:

    Ah, the uncertainty monster. Did he own up to his mistakes and withdraw the allegations of fraud, I wonder?

    I see he also weighed in on dinosaur size and growth rates. So out of interest I looked at the cites (the paper was in PLOS ONE, which, while it is peer-reviewed, has a low threshold for novelty and significance and is prepared to publish basic data which may be of value later as part of the general data pile to be integrated by someone else). The first hit was a review which included him among the references but didn’t cite the paper in the text. So probably there for completeness, but anything controversial he claimed was deemed not important enough to mention. The second hit is to supplementary material which discusses his recommendations in the context of a Science paper (section VII). TL;DR version: The bit that’s correct is not novel, and makes no difference to the result. The bit that’s novel is wrong when applied to real live animals. Using Myhrvold’s method, the sperm whale “is predicted to reach a final size of over 500,000 meters, but a maximum growth rate of only 20 cm per year”.

    So, angels-on-a-pinhead and rookie errors. Including conflation of measurement errors with natural variability. Hmmm, seems like I’ve heard that one before too 😦 .

    My Bayesian prior for Myhrvold being right about the asteroids and NASA being wrong? Do you need to ask 🙂 ?

  182. Magma says:

    Seems that the Dunning-Krugerites at WUWT have just adopted Myrhvold as their latest champion. I haven’t been so shocked and surprised since the sun rose AGAIN. (Seriously, basic statistics show it’s WAY overdue to not rise.)

    As for Myrhvold’s claims, reading between the lines of Phil Plait’s 2016 column on the matter, he made an obstinate pest of himself with the NASA PIs and they eventually blocked him.

  183. It would have been more intriguing Myhrvold had uncovered something foundational. Too many degrees of freedom in the arguments back and forth as it stands.

  184. “I was scanning through the more recent article and paper and it seems to have changed from they’re making huge mistakes, to they’re over-confident and the uncertainties are larger than they claim.”

    I used to wonder what the deniers would say when the facts/heat on the ground become so great that their position looked to ridiculous to maintain. The position and stance of lukewarmer came to mind and has been adopted by many of the deniers. The change in the positions that you describe above is similar.

  185. Marco says:

    “WUWT have just adopted Myrhvold as their latest champion.”

    Maybe someone should post a link there to the Myhrvold and Caldeira papers from 2012 and 2013 in ERL…

  186. Maybe the Myrhvold episode establishes that peddling of ideas is acceptable. Journal editors could have easily told him to get lost, yet they didn’t.

  187. Dave_Geologist says:

    Interesting. I had previously downloaded CM13 but not made the connection. So with someone knowledgeable in the field holding his hand, he can avoid rookie errors and conspiracy theorising.

    You’re right, not much comfort in either for wattsuppians. And they’re not inconsistent. Getting a benefit ten years after instantaneously stopping carbon emissions is a useful thought experiment, and IIRC there are other papers that say temperatures would plateau almost immediately. CM13 have a short peak and a very slow decline, but are qualitatively no different. Which is an important message, not to oversell the benefits of climate action. When the public hears “we’d see the benefits within ten years”, most will translate that into “we’ll see a decline in temperatures and remission from cyclones, wildfires, droughts etc. within ten years”. Whereas it actually means “we’ll freeze things where they are and stop even more damaging things happening in the future”. MC12 posits a multi-decade build-out of low carbon power, so of course it takes multi-decades to see the benefit.

    The take-home message from both should not be “it’s all too hard”. It should be “the sooner we make a start the better”.

  188. Dave,
    I think the important result from those papers is that they show that we can have an impact on short timescales. There tends to be a sense that emission reductions now won’t change anything for ~100 years, and that is simply not true – the peak warming from a pulse of emission occurs within about 10 years. So – as you suggest – the sooner we start, the sooner we’ll see the benefits of having done so.

  189. Dave_Geologist says:

    And here is the Reply to Myhrvold’s dinosaur criticism. Basically a storm in a teacup. There were a couple of typos in the original paper which didn’t help, and the methodology was not explained as fully as it might have been. But Myhrvold made rookie errors and in his “replication”, did silly things like using all the samples, not just those which had been histologically aged, to generate his curves. Even though the former was clearly stated in the text and obvious from the figure. And he digitised stuff from the paper instead of asking for the data. Or maybe, based on the NASA episode, he asked but they declined to respond because he was rude and/or made false allegations.

    The typos/transcription errors may not even have been the authors’ fault. I have a paper where the first equation contains an error. There was an error in the proof copy which I corrected. But that meant the equation had to be split across two lines and they introduced another error which I had no opportunity to fix. The two-lines choice was a stupid one by the publisher because it had a knock-on effect on the layout of subsequent pages, which IIRC they fixed after a couple of pages by resizing a figure. They should just have used a smaller font in the equation.

    Ironically, I’d produced it by rearranging an equation from the seminal paper in the field – which also contains a typo! (But I also referred to a subsequent paper by the same author which corrected the typo – I quoted the original to correctly attribute priority, since I also quoted other papers which had used the same equation in the intervening years.) The seminal paper was published in a journal which shortened turnaround time by having editorial staff rather than the author perform proof-reading.

    That sort of typo didn’t matter before the days of citizen science, because any professional working in the field would not be misled, but would say “that’s silly – must be a typo”.

  190. Steven Mosher says:

    “That sort of typo didn’t matter before the days of citizen science, because any professional working in the field would not be misled, but would say “that’s silly – must be a typo”.

    The cool think about markup is that your paper wont compile with an error in the equation

  191. Steven Mosher says:

    “Myhrvold’s failure to replicate our growth
    curve findings stems partly from our not disseminating
    sufficient data to replicate the many steps that go into a
    growth curve analysis for a fossil vertebrate”

    well duh.

    here is the thing. Since this study deals with a unique dataset Replication is strictly speaking not possible. REPRODUCEABILITY is the only thing you can do unless you get additional data
    or unless there is other data that could be used to estimate growth.

  192. Steven Mosher says:

    “As for Myrhvold’s claims, reading between the lines of Phil Plait’s 2016 column on the matter, he made an obstinate pest of himself with the NASA PIs and they eventually blocked him.”

    yes one of the problems with having scientists being the custodians of their own data is that they may have to make personality driven decisions about data access. Post your data or turn data over to a data custodian/archivist and move on to do more science rather than babysitting data
    and trrying to deiscern who is a pest and who is not.

  193. Dave_Geologist says:

    Steven

    The English language is more fault-tolerant than numerical algorithms. And since you’ve read the paper, you know that it was primarily transcription/typo issues. On data archive/access, doing it right is a LOT of effort. Effort that is generally not funded by a grant or allocated time by an employer. And as you seem to acknowledge, whether dinosaur growth rates plateau in adulthood or not is not demonstrated by endlessly reworking one dataset. It’s demonstrated by consilience. What about other dinosaurs? What about these dinosaurs using methods other than bone histology? Do other histologists agree with my interpretation of the bones? What about their nearest living relatives? Their ancestors? What if I use a different in-sample and out-of-sample set? Oh dear, I don’t have the microscope time because I’m too busy dealing with vexatious requests. How come it’s only us evolutionary biologists and climate scientists who get swamped by vexatious requests? What have we done wrong?

    Do you really think scientists can’t recognise pests (even good-faith ones) and bad-faith actors when they come across them? If someone asks a stupid question, (s)he’s probably not competent to redo the analysis so why bother? If someone makes a loaded request, motives can quite rightly be questioned. If someone asks for (more often demands) material additional to that required for scientific work, ditto. If the request contains a direct or plausibly-deniable-but-clearly-dog-whistling accusation of fraud, ditto. Especially when you can see their output and the company they keep on the blogosphere. If I were a climate scientist and I got a request from the usual suspects, or from a Republican pol., I’d be quite entitled to regard it as vexatious, based on demonstrated track record. They’ve poisoned their own wells, by their own actions. It’s not a matter of personality, it’s a matter of standards of behaviour. A good neighbour suppresses his baser personality traits in the interests of civility.

  194. Dave_Geologist says:

    The cool think about markup is that your paper wont compile with an error in the equation
    The two examples I quoted would have compiled and run just fine. Just given the wrong answer. Neither was a syntax error.

  195. Dave_Geologist says:

    More Myhrvold…

    The alternative that Myhrvold proposes is to regress the rate parameter k/e against M, instead of Gmax. However, this approach simply rescales the analysis, leaving the error structure unchanged. While our regression of Gmax versus M yields a slope of α, regressing k/e versus M gives a slope of α – 1, with an identical standard error (Table 1). … However, a notable contrast between our approaches is that maximum growth rate has a clear biological meaning, unlike the model parameter k divided by natural base e (units: time–1), a mathematical abstraction that cannot be readily interpreted. … For its conceptual clarity, analytical utility, and empirical justification, we prefer analysis of Gmax rather than k/e. In either case, though, the conclusions are the same.

    Angels, pinheads. And while not wrong, certainly not useful.

    The peril of misapplying statistical techniques is perhaps best illustrated by Myhrvold’s choice of range polygons to present the growth data [figure 1, in (3)]. Range polygons enclose the entire space occupied by a data set and will generally increase with sample size. Although large samples with large ranges may suggest increased uncertainty, in fact, the converse is true. Enhanced sampling reduces the standard error and increases the statistical power for detecting real diferences between taxa. Myhrvold’s use of range polygons is misleading in that it implies considerable overlap between taxa that are statistically quite distinct. In contrast, we computed 95% confidence bands that provide a quantitative illustration of the statistical diferences. As the confidence bands reveal, regardless of whether k/e or Gmax versus M is plotted, dinosaurs grew significantly faster than ectotherms throughout most of their body range, intersecting with endotherms at the largest sizes (Fig. 1). They are statistically indistinguishable from mesothermic tuna.

    Someone call the cops! They mis-spelled “diferences”. Twice!

    Myhrvold states that he discovered 11 errors in our data set and critiques our handling of Archaeopteryx. We disagree that including Archaeopteryx with other Mesozoic dinosaurs was inappropriate, as we discussed above. It is unfortunate, but not surprising, that some errors may be found among our ~29,000 growth values. A larger data set, however, makes it unlikely that any particular error will influence our results, and that is the case here. … The phylogenetically informed slope of Gmax versus M changes from 0.76 to 0.77 and the r2 from 0.92 to 0.91.

    Quick! Call for an Auditor!

  196. Dave_Geologist says:

    Well duh. You missed out the best bit Steven. The next sentence.

    The transcription mistakes we made in our presentation, which we correct here, were not implemented in the growth curve analysis and did not compromise the conclusions we made about P. lujiatunensis biology.

  197. Dave said:

    “And here is the Reply to Myhrvold’s dinosaur criticism. Basically a storm in a teacup. “

    I do find it fascinating how one guy can yield such control over the work of these independent scientists. They should be able to just blow him off, yet they don’t and spend all that effort with rebuttals. Myrhvold probably has some valid criticisms but he does understand how to push the right buttons.

    “Do you really think scientists can’t recognise pests (even good-faith ones) and bad-faith actors when they come across them? “

    Good question. Myrhvold is definitely scientifically curious.

  198. Dave_Geologist says:

    They chose to blow him away rather than blow him off Paul. Once Myhrvold got his Discussion letters accepted, they had to respond. They’re now part of the formal scientific literature, not just blog posts. They chose to bury him rather than just say it doesn’t matter. You’ll have noticed I have a tendency to do that 🙂 .

    Compare that, for example, to certain contrarian economists, scientists and statisticians, who tend to just give a dismissive “doesn’t matter”, without demonstrating why it doesn’t. Or to ignore the criticism entirely. The consequence is that their work has no standing in the community and no influence outside politics and the blogosphere/press. For it to respected, you have to defend your work when it’s criticised, with meaningful arguments. Of course if the objective was just to get the headline “peer-reviewed paper finds (insert contrarian meme here)”, and not have any lasting influence on the science, there’s no need to defend the publication. But maybe that’s just me being cynical.

  199. “There tends to be a sense that emission reductions now won’t change anything for ~100 years, and that is simply not true – the peak warming from a pulse of emission occurs within about 10 years. So – as you suggest – the sooner we start, the sooner we’ll see the benefits of having done so.”

    I think this is exactly right and yet, despite knowing this situation for many decades, so far, our actions look like deck chair management on cruise ships to me. If you are considering a cruise, definitely pick a cruise line with solar panels and a slideshow about AGW. That’s the least that we (our species) should do and we are pretty darn good at doing the least that we can/should do.
    https://www.google.com/search?q=green+cruise+ships&ie=utf-8&oe=utf-8&client=firefox-b-1-ab

  200. Steve said:

    ” Since this study deals with a unique dataset “

    Dave said:

    “What if I use a different in-sample and out-of-sample set? “

    An out-of-band cross validation is the best bet for resolving these kinds of issues.

    “They’re now part of the formal scientific literature, not just blog posts. “

    I imagine that Myhrvold treats the formal scientific literature the same as a blog post. He has a huge administrative staff working for him and so can submit a quality paper (i.e. not easily dismissed) as easily as we can write a blog comment.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.