I noticed, via a tweet from Judith Curry, that the House of Commons Science and Technology Committee is holding an inquiry into Research Integrity. I also encountered this written evidence by Michael J Kelly, Emeritus Prince Philip Professor of Technology, University of Cambridge.
In my view, there are many aspects of how we undertake research that could be improved, and many of the critiques have elements of truth. However, how we conduct research can vary greatly between different disciplines, and even within disciplines. Doing applied research, where one might be trying to develop some technology, is quite different to doing more fundamental research, where the goal might be to understand some aspect of a system that is not yet fully understood. Research that relies on observations, which can often not be easily repeated, can be quite different to research that relies more on experiments, which can often be repeated many times. Research in areas with conservation laws that provide structural constancy, such as physics, can be different to that in areas without such conservation laws (such as the social sciences). Many critiques seem to assume that research is somehow homogeneous and that a problem in one area immediately applies to all areas.
For example, the written evidence by Michael J Kelly appears to be arguing that all research should be conducted like engineering. I’m sure there are many engineers who are very good researchers, but I don’t think that engineering is necessarily an examplar of how research should be conducted, and nor should we necessarily impose the same constraints, that might apply in engineering, to other disciplines. There may be some circumstances where we would expect researchers to be risk averse, and others where we should encourage risk. If the foundations of a research areas are extremely well understood, there may be well defined procedures that we would expect researchers to follow. In areas where the foundations are less well understood, we may not be able to impose strict rules as to how researchers should carry out their analysis.
Some of what is presented in Michael Kelly’s evidence also appears to illustrate a misunderstanding of what is actually possible in other fields. For example
In cosmology, a new theory is generally not subject to such a clear and discriminatory experiment, and the claims are not testable empirically. So cosmology remains a plausible narrative of the origins of the universe, and nothing more.
Well, cosmology is simply the study of the origin and evolution of the universe, so the last sentence doesn’t even really make sense. However, the claim that the theories cannot be tested is simply wrong. The figure on the right is the Cosmic Microwave Background (CMB) power spectrum. This is the power spectrum of the tiny temperature perturbations in the microwave radiation from a time about 400000 years after the Big Bang. The data points are from observations using the Planck satellite, while the curve is the best-fit lambda-CDM model (CDM – Cold Dark Matter).
This indicates that, to explain the observations, we need some kind of cold matter that interacts only via gravity (Dark Matter), and also some kind of extra energy, known as Dark Energy. Admittedly, we have not yet directly detected Dark Matter, and do not yet know the form of Dark Energy. There are also still people working on alternatives, such as modified forms of gravity. However, the claim that we cannot test these cosmological models, is simply wrong.
Similar, the submitted evidence says
In climate science, the models struggle to faithfully represent what has happened in the last 100 years and there is no convergence theorem that says that the models are capable of predicting what will happen in the next 10-100 years. No amount of simulation is an alternative to empirical data to make a point.
Firstly, as Tom Knutson and Robert Tuleya said, if we had observations of the future, we obviously would trust them more than models, but unfortunately…..observations of the future are not available at this time. Secondly, we can’t go back in time to make extra observations of the past. We also cannot rerun our climate with slightly different initial conditions. Models provide a way of understanding how our climate responds to various changes, and provide information as to how it might change along various different future emission pathways. Models are, however, not the only source of information; there is also a lot of empirical data. Even though it is certainly true that it is important to compare simulations to empirical data, it’s also the case that data without some kind of model is also pretty useless; you can’t interpret observations without some kind of model of the system being observed.
I’d actually been tempted to not write this post as this is all getting rather tedious. However, Michael Kelly’s submitted evidence includes a discussion of the scientific literature and how it is difficult to publish a paper with a different view, and how it’s also difficult to publish a correction. Well, Michael Kelly recently published a paper on extreme events, that I discussed in this post. The paper was pretty poor and I emailed Michael Kelly to point out a very obvious error. To his credit he admitted the error (I wasn’t the only one to point it out) and claimed that he would try to publish a correction. I’ve just checked his paper on Google Scholar, and it appears to have one citation which is not a correction. I think I’ll leave it at that.