Bias in science

There are quite often claims that there are significant biases in science and that this is strongly influencing research results. Typically this is based on known problems in certain fields; the replication crisis in psychology, or the failure to publish negative results in medicine. My problem with this is that how research is conducted can vary greatly across different disciplines, and so using isolated examples to infer a major problem across all research areas may not be justified.

Joshua, however, has made me aware of a paper that does a [m]eta-assessment of bias in science, by Fanelli, Costas, and Ioannidis (also discussed in this article). They looked at a large sample of meta-analyses that considered a number of different bias-related patterns, and also considered various risk factors. The basic results was essentially that

The magnitude of these biases varied widely across fields and was overall relatively small. However, we consistently observed a significant risk of small, early, and highly cited studies to overestimate effects and of studies not published in peer-reviewed journals to underestimate them.

So, the biggest biases were associated with small studies that reported effects of large magnitude, studies published early because of an extreme results, studies that ended up being highly cited (although I’m not sure how this can be a bias, given that this can’t be known in advance), and studies published in the non-peer-reviewed literature, but the effect was relatively small and varied widely across fields. In fact, the paper explicitly says that, when testing the various bias-related patterns,

[t]he ratio of studies concluding in favor vs. against a tested hypothesis increases, moving from the physical, to the biological and to the social sciences, suggesting that research fields with higher noise-to-signal ratio and lower methodological consensus might be more exposed to positive-outcome bias.

So, the magnitude of the bias is lower in the physical sciences, compared to the social sciences.

The paper also considered various risk factors (such as pressure to publish, or career level) and mostly found that there was no relationship between these and the presence of bias. The size of the team and the distance between collaborators were two that did have some influence, but most of the others had little effect. I found this quite interesting, because my own view was that part of the problem was the system in which the researchers operate, and this suggests that this plays little role. If anything, it seems as though most of the bias comes from researchers getting excited by what appear to be interesting results, rather than them seeing a way to advance their careers through publishing results that might be biased.

Overall, the paper concludes with

Our results should reassure scientists that the scientific enterprise is not in jeopardy, that our understanding of bias in science is improving and that efforts to improve scientific reliability are addressing the right priorities.

When it comes to dealing with bias, the paper made – in my view – some interesting points

However, our results also suggest that feasibility and costs of interventions to attenuate distortions in the literature might need to be discussed on a discipline and topic-specific basis and adapted to the specific conditions of individual fields. Besides a general recommendation to interpret with caution results of small, highly cited, and early studies, there may be no one-size fits-all solution that can rid science efficiently of even the most common forms of bias.

I think the latter is an important point. Science is a human endeavour and so will always be influenced by human flaws. Even though we should be aiming to reduce bias as much as possible, we can’t expect perfection and if the magnitude of bias is small (as this paper suggests) then we should be careful of introducing all sorts of possible solutions that might do little to actually solve the problem, are only relevant in certain circumstances, and might end up doing more harm than good. Being aware of where bias is most likely to exist (small studies, for example) is probably a good place to start.

Posted in ethics, physicists, Research, Science, The philosophy of science, The scientific method | Tagged , , , , , | 89 Comments

Based on Observations Only!

The Global Warming Policy Foundation (GWPF), who I’ve written about many times before, have released a report which they’ve described as [t]he World’s first state of the climate survey based on observations only. I think it’s meant to the a response to the WMO’s State of the Global Climate Report, which I would recommend reading, rather than reading the report produced by the GWPF.

The GWPF’s report is full of the standard talking points, so I won’t bother rebutting them again. I was going, instead, to make a more general point about their suggestion that it is based on observations only. It’s clearly intended to suggest that their report is somehow pure and uncontaminated by nasty things like models. However, this doesn’t make any real sense. You can’t really do any kind of research without some kind of model.

Even making observations requires models; you need some way to convert whatever happens in the measuring device into a quantity that represents some property of the system that is being observed. It could be as simple as converting a change in height of Mercury in a thermometer into a temperature, or as complex as trying to determine the temperature in the troposphere from satellite radiance measurements. Either way, you still need some kind of model in order to make meaningful observations of the system being studied.

However, even if we accept that we can make observations that are not significantly influenced by the underlying models, observations – by themselves – tell us very little. They might be able to tell us if some property of a system is changing, but without some kind of model, we won’t know why it is changing, or what such changes might tell us about the system being observed. Models are a key part of doing research and suggesting that their report is based on observations only is not only wrong, but also indicates that they don’t even have a basic understanding of how research is undertaken (okay, there is an alternative interpretation, but I’m trying to be polite).

Posted in Climate change, ClimateBall, Research, Science, The scientific method | Tagged , , , , , | 39 Comments

A reduced climate sensitivity!

Now that I have your attention, I should probably make clear that this post is not about the Earth. I’m just back from a meeting where one of the speakers was Ian Boutle, lead author of a paper in which they Explor[ed] the climate of Proxima B with the Met Office Unified Model (pre-print available here).

Proxima Centauri B is a recently discovered Earth-sized planet in an 11-day orbit around Proxima Centauri, the closest star to the Sun. There are a couple of aspects of this system that may influence the planet’s climate sensitivity. One is that the star is much cooler than the Sun, and so emits most of its radiation at longer wavelengths. The other is that the planet is probably tidally locked – its rotation period will match its orbital period so that one side always faces its host star.

What Boutle et als. model indicates is that the above factors appear to result in a climate sensitivity that is quite a bit lower than that of the Earth (about two-thirds). One reason is that the albedo of ice decreases with increasing wavelength. Since the host star to Proxima Centauri B emits mainly at longer wavelengths (compared to the Sun) the ice albedo feedback is significantly reduced. Also (and this is the bit I wasn’t quite clear on) the changes in cloud cover appear to mainly occur on the night side, and so have little impact on climate sensitivity. There also appears to be global-scale circulations that also suppress the temperature on the day side, due to the efficient cooling of the night side of the planet.

The above has some potentially interesting implications for habitability. To be clear, we don’t really know what is required for a planet to be habitable, or not, so – in this context – it simply refers to the possibility of there being liquid water on the surface. However, if Proxima Centauri does have a smaller climate sensitivity than the Earth, then this implies that it is less sensitive to changes in stellar flux and, hence, that there is a greater range of parameter space over which it could support liquid water on its surface.

Of course, this is all based on models, so we don’t know even if Proxima Centauri B actually has an atmosphere and, if it does, if it can actually support liquid water on its surface. However, future space missions (such as the James Webb Space Telescope) and future ground-based telescopes (such as the European Extremely Large Telescope) might be able to make observations that could tell us something about Proxima Centauri B’s atmosphere, so we may have some idea about this in the not too distant future.

Posted in Climate sensitivity, Research, Science, The scientific method | Tagged , , , , , | 14 Comments

The feedback paradox

Realclimate has a new post, by Rasmus Benestad, that discusses predcitable and unpredictable behaviour. It focuses a little on Judith Curry’s recent report about climate models, that I discussed here. The Realclimate post is well worth reading, and I encourage you to do so, but there was one thing that I really liked and that I thought I would repeat here.

What’s quite often been discussed/mentioned here is that if one argues for a significant natural contribution to our long-term warming, then that’s potentially arguing for a high climate sensitivity. Any internally-driven long-term warming will require some kind of feedback in order for it to be sustained. However, such a physical process should respond to both internally-driven and externally-driven perturbations. Therefore arguing for a significant natural contribution to our observed warming AND arguing for a low climate sensitivity is potentially paradoxical.

I won’t say more and will simply repeat, below, the section from the Realclimate post, which explains it better than I can (credit: Realclimate/Rasmus Benestad).

A potential feedback paradox

Curry also introduces a potential paradox in her report when she emphasises natural variations. The magnitude of natural temperature variation are regulated by feedback processes and have physical causes. The climate sensitivity also involve such feedback processes.

Any feedback process based on temperature will act on both natural and forced changes in the temperature. If such feedbacks result in pronounced natural temperature variations, they also imply that the climate sensitivity is high.

Examples of such feedbacks include increased atmospheric humidity and reduced snow/ice cover. Processes involving clouds are more uncertain, but they too are likely to be affected by temperature (convection) and act to modify the climatic response.


It is possible to get enhanced variability on those timescales as a result of dynamical mechanisms without needing to appeal to higher climate sensitivity.

Nevertheless, the bottom line is that Curry must prove that the feedbacks involved in the natural variations are different to those affecting the climate sensitivity before she can conclude that natural variability dominates over a warming due to increasing greenhouse gases.

Posted in Climate sensitivity, Judith Curry, Science | Tagged , , , , | 149 Comments

Matt Ridley responds to Tim Palmer

I came across a response, by Matt Ridley, to Tim Palmer’s talk. I’ve posted Matt Ridley’s response below. One interesting aspect of his response is that it is written as if he is someone with the expertise to actually debate the science. Of course, it’s a free world, so anyone can choose to do so, and Matt Ridley does have a science PhD (DPhil actually), but his research work was in biology (which he himself points out) and he hasn’t – as far as I’m aware – been actively involved in research for over 30 years. So, his science background is not really relevant to climate, his career has mainly been in journalism, banking and politics, and yet his response does not make any of this clear. It’s not necessarily required, but I do think most would acknowledge this type of thing.

Anyway, his response is below and I’ll make some specific comments below it.

Credit: Matt Ridley

He seems to dismiss paleo estimates in a manner that does not make much sense. I don’t think the higher end of climate sensitivity, or the dependence on temperature, somehow implies runaway. Mostly, the paleo estimates are consistent with the climate sensitivity range presented by the IPCC.

  1. This point is almost saying that he agrees with Tim Palmer followed by a comment that suggests he missed the point. The outcome is, of course, not certain, but depends on a number of factors, such as climate sensitivity (which we don’t know, but we can at least produce a likely range) and how much we will emit (which we also don’t know, but can at least influence). In a sense, the more we dismiss the possibility of it being dangerous, the more we are likely to emit, and the greater the possibility of it then being dangerous.
  2. Maybe the reason Tim Palmer presented only one dataset is because all the surface datasets are very similar? Maybe the reason he chose to present GISSTemp is because it suffers from less coverage bias than HadCRUT4? Maybe the reason he didn’t show the satellite datasets is because he was talking about surface temperatures, which they don’t measure? Maybe the pause isn’t quite as big a deal as Matt Ridley would appear to think that it is?
  3. Matt Ridley is, of course, free to be unconvinced; there isn’t some requirement that he be convinced. Of course models are tuned in some respects, but this doesn’t suddenly mean that climate sensitivity is not emergent. In fact, Tim Palmer explains this all quite clearly in his talk. Also, basing his concern about models in general on possible problems with economic models, suggests he doesn’t understand the concept of structural constancy.
  4. I think he’s wrong about biology being left out of the story. As far as I’m aware, biology is considered when studying the carbon cycle.
  5. What mismatch between models and observations?
  6. As far as I’m aware, his PDF did take Nic Lewis’s work into account; the lower bound was 1.5K which – I think – was reduced from 2K mostly because of recent energy balance estimates which we should treat with some caution (to be fair, we should treat all estimates with some caution). Also, discussing these energy balance models and why we should be cautious about accepting their results was a pretty key part of Tim Palmer’s talk, so it’s odd that Matt Ridley would ask this question.
  7. I’m guessing Matt Ridley doesn’t get the irony of this comment?

It’s clear that Matt Ridley does not like Bob Ward.

Posted in Climate change, Climate sensitivity, ClimateBall, Global warming, Research, Science, The scientific method | Tagged , , , , , | 52 Comments

Informing versus convincing

I want to clarify something about yesterday’s post that seems to have at least got one person up in arms. The key point that I was trying to get across (and that I think is the same as Michael Tobis’s point) is that, formally, the role of scientists/researchers is to try and understand whatever system it is that they are studying. They also have a role in informing the public and policy makers about their research. However, they are not responsible for whether or not what they present is accepted; they’re not salespeople trying to sell a product.

However, this does not mean that they’re absolved of all responsibility. I do think that scientists/researchers should (mostly) be obliged to speak out when they’re aware that our best understanding is being misrepresented publicly. This, however, does not mean that they should be responsible if the public remains unconvinced. It’s neither their remit, nor something for which we’d expect them to typically have the necessary skills. To be clear, if some scientists do want to try and convince the public, I think that’s fine, as long as they’re honest about what they’re doing. There’s nothing wrong with scientists becoming activists as long as they make their role clear.

I think there is also a few other things to bear in mind. Many scientists who do speak out, do so in a largely personal capacity; they don’t get supported, or rewarded, for doing so. It can therefore be very difficult. It’s time consuming and – certainly in my case – can be very stressful at times. I’ve learned – the hard way mostly – what I can do without negatively impacting my family life, my job, or my health. Even then I don’t get it right all the time. I’ve spent the last few days being verbally abused on another blog because – I think – I didn’t treat someone with the kind of respect they expected. Admittedly, it was my own fault for expecting anything different.

In my view we need to recognise some of this. Some people are doing the best they can and – in my case – don’t always get it right. It is a difficult topic and I think we need to spend more time supporting those who are trying to make a positive contribution, rather than criticising them for not doing enough, or for not getting it completely right all the time. I even accept that I’ve done some of this myself, and so certainly regret some of my own interventions.

A key reason why I think it’s important to distinguish between scientists’ role in informing (it is one of their roles) and their role in convincing/persuading (it isn’t formally one of their roles) is that I fully expect us to recognise at some point in the future that we haven’t taken this issue seriously enough. I also fully expect some to blame scientists for not having done enough. I think this would be wrong and I think we should be careful of laying the groundwork for this.

Posted in ClimateBall, ethics, Global warming, Research, Science, The scientific method | Tagged , , , , , , | 115 Comments

Scientists are not salespeople!

Gavin Schmidt posted a bunch of tweets in response to a post by Scott Adams (of Dilbert fame) in which he claims to illustrate how climate scientists can persuade skeptics. If you want to read Gavin’s tweets, Greg Laden has a post as does Mark Brandon. I think Gavin’s tweets present an excellent explanation of our current understanding. However, I would like to briefly discuss a different aspect of this issue.

Scott Adams’s argument seems to be that it should be easy for scientists to present some kind of persuasive/convincing argument and that they can’t is, therefore, indicative of some kind of problem. The issue with this is that this is not what scientists/researchers should be doing. The role of a scientist/researcher is to understand whatever systems it is that they’re studying. They then present their results to colleagues and others in the field, and they should also aim to engage with the public/policymakers. However, their role is not to convince the public/policymakers, it is simply to present information. It’s for others to decide if the public should be convinced and it is the role of others to do the persuading/convincing.

What motivated this was a series of tweets by Michael Tobis which encapsulates the issues. So I’ll leave it there and you can read Michael’s tweets, which are below.

Posted in Climate change, Research, Science, The philosophy of science, The scientific method | Tagged , , , , | 51 Comments