There’s been quite a lot in the media, and elsewhere, about problems with science. A common theme at the moment is the replication crisis, but you regularly see claims in the blogosphere that papers should be retracted because they supposedly have errors, and I’ve even seen some argue that we should find ways to penalise scientists who misrepresent our current understanding.
I’m a bit tired, and can’t really face writing anything too lengthy (I may fail) so thought I’d make some basics points and let others express their views in the comments. It’s clear that some of the criticisms have merit. We should promote good practice and should encourage people to publish research that is careful and thoroughly checked. What we value (lots of grant funding and papers in Nature, for example) does not always reflect actual quality. We should publish negative and null results and should aim to also publish replication studies. We could improve how we undertake peer-review.
However, science (by which I really mean fundamental research) is about understanding things, typically things we don’t yet understand fully. People should take risks; that’s how we solve interesting problems. People will get things wrong; it’s part of the process. People will try methods that turn out not to be very good; again it’s part of the method. We should encourage people to challenge existing paradigms, even if what they do ends up being horribly wrong – the quality of the challenges can tell us something of how hard it is to overthrow well-established ideas. We should base our understanding – or lack thereof – on an assessment of all the available evidence, not simply on one, or two, studies.
In some sense, that seems to be where some of the problems lie. In normal science, we trust our overall understanding when there is a large collection of consistent evidence. In some cases, however, important decisions are based on one, or a few, studies. When these turn out to be wrong, the impact can be substantial. In my view, however, this is not normal science; we shouldn’t be basing our understanding on only a few studies. Sometimes, it may be unavoidable, but we should still be careful of judging science overall, on the basis of a few examples like this.
Maybe in cases like this (when we do need to make decisions with limited evidence) we should scrutinise the evidence more carefully. In normal science, however, I would still favour trusting the method, rather than encouraging detailed scrutiny of individual studies (nothing wrong if people want to do this, but I don’t see the overall value). That doesn’t mean that there aren’t things that we could do to improve the overall quality of research, just that the problems are not – in my opinion – nearly as great as might be indicated by some of the examples that are often given.
Anyway, that’s a summary of my general views. If anyone else would like to add anything, or make a different argument, feel free to do so through the comments.