This whole furore about Karl et al. has got me thinking more about how we actually conduct science/research. There is, I think, a perception that doing science/research involves following a fairly well defined set of procedures about which there should be little disagreement. The reality is that it is much more complex, with there being quite large differences between disciplines and sometimes even within disciplines. It often seems that much of the criticism about climate science comes from those who have some kind of relevant experience, and who then seem to think that everything should happen as they think it should; without considering that what works in their area, may not in another.
For example, I was at a meeting a few weeks ago at which one of the speakers pointed out that cosmology/astronomy was one of the few research areas that is primarily observational; climate science being one of the others. You can’t really do experiments. There is no control (we’re studying a single universe, or a single planet). We can’t go back and redo observations if they aren’t as good as we would like. Observations are often beset by problems that were not anticipated and that you can do little about. Understanding and interpreting observations requires models of various levels of complexity. All these factors mean that the details of how research is undertaken in these areas might be quite different to how it would take place in another. This doesn’t mean that the underlying scientific philosophy is different, just that the details of how it is undertaken might differ from what would happen in other areas.
In some cases, it is possible to develop a well-defined observational and analysis strategy, but in many cases it is not. Either you’re trying to use some data to do something that was not anticipated when the data was collected, or something unanticipated happens when the data is being collected that then requires some kind of correction. You might argue that in such circumstances there should be a process that is checked and authorised, but who should do this? Also, scientists ultimately want to do research, publish their findings, and let it be scrutinised by the scientific community. Following a well-defined procedure to the letter doesn’t somehow validate research results, and not doing so doesn’t somehow invalidate them. Our understanding increases when more and more studies (ideally by different groups of researchers) return results that are largely consistent; it isn’t really based on a sense that the research obeyed some well-defined procedure.
Something else to bear in mind, is that research is carried out by humans, and not by robots. Not only are they typically trying to solve problems that are perceived as of interest, they would also like others to be interested in what they have done. They try to write their papers in a way that highlights what might be of interest. There’s nothing wrong with this at all; we’re not funding research so that people can do things that are boring, and there’s no point in doing something interesting if people don’t notice.
However, there are certainly cases where researchers are regarded as having hyped their work too much (and some where they may not have hyped it enough). There are – in my view – even valid criticisms of the manner in which Karl et al. framed their results. However, precisely defining the correct framing is probably not possible, and that some might object does not necessarily mean that it was wrong. I’m, of course, not suggesting that everything that is done does not deserve criticism, or that there aren’t cases where it’s obviously deserved. However, there are many where it’s not clear, and where the critic may simply not have sufficient understanding to make the claims that they’re making.
At the end of the day, research is never easy and rarely works as expected; if it did, the answer would probably be obvious. It can, of course, be perfectly reasonable to criticise how research is done, and how it’s presented. However, this would ideally be in the interests of improving our overall understanding, not undermining it.