I noticed, via a tweet from Judith Curry, that the House of Commons Science and Technology Committee is holding an inquiry into Research Integrity. I also encountered this written evidence by Michael J Kelly, Emeritus Prince Philip Professor of Technology, University of Cambridge.
In my view, there are many aspects of how we undertake research that could be improved, and many of the critiques have elements of truth. However, how we conduct research can vary greatly between different disciplines, and even within disciplines. Doing applied research, where one might be trying to develop some technology, is quite different to doing more fundamental research, where the goal might be to understand some aspect of a system that is not yet fully understood. Research that relies on observations, which can often not be easily repeated, can be quite different to research that relies more on experiments, which can often be repeated many times. Research in areas with conservation laws that provide structural constancy, such as physics, can be different to that in areas without such conservation laws (such as the social sciences). Many critiques seem to assume that research is somehow homogeneous and that a problem in one area immediately applies to all areas.
For example, the written evidence by Michael J Kelly appears to be arguing that all research should be conducted like engineering. I’m sure there are many engineers who are very good researchers, but I don’t think that engineering is necessarily an examplar of how research should be conducted, and nor should we necessarily impose the same constraints, that might apply in engineering, to other disciplines. There may be some circumstances where we would expect researchers to be risk averse, and others where we should encourage risk. If the foundations of a research areas are extremely well understood, there may be well defined procedures that we would expect researchers to follow. In areas where the foundations are less well understood, we may not be able to impose strict rules as to how researchers should carry out their analysis.
Credit: ESA and the Planck collaboration
Some of what is presented in Michael Kelly’s evidence also appears to illustrate a misunderstanding of what is actually possible in other fields. For example
In cosmology, a new theory is generally not subject to such a clear and discriminatory experiment, and the claims are not testable empirically. So cosmology remains a plausible narrative of the origins of the universe, and nothing more.
Well, cosmology is simply the study of the origin and evolution of the universe, so the last sentence doesn’t even really make sense. However, the claim that the theories cannot be tested is simply wrong. The figure on the right is the Cosmic Microwave Background (CMB) power spectrum. This is the power spectrum of the tiny temperature perturbations in the microwave radiation from a time about 400000 years after the Big Bang. The data points are from observations using the Planck satellite, while the curve is the best-fit lambda-CDM model (CDM – Cold Dark Matter).
This indicates that, to explain the observations, we need some kind of cold matter that interacts only via gravity (Dark Matter), and also some kind of extra energy, known as Dark Energy. Admittedly, we have not yet directly detected Dark Matter, and do not yet know the form of Dark Energy. There are also still people working on alternatives, such as modified forms of gravity. However, the claim that we cannot test these cosmological models, is simply wrong.
Similar, the submitted evidence says
In climate science, the models struggle to faithfully represent what has happened in the last 100 years and there is no convergence theorem that says that the models are capable of predicting what will happen in the next 10-100 years. No amount of simulation is an alternative to empirical data to make a point.
Firstly, as Tom Knutson and Robert Tuleya said, if we had observations of the future, we obviously would trust them more than models, but unfortunately…..observations of the future are not available at this time. Secondly, we can’t go back in time to make extra observations of the past. We also cannot rerun our climate with slightly different initial conditions. Models provide a way of understanding how our climate responds to various changes, and provide information as to how it might change along various different future emission pathways. Models are, however, not the only source of information; there is also a lot of empirical data. Even though it is certainly true that it is important to compare simulations to empirical data, it’s also the case that data without some kind of model is also pretty useless; you can’t interpret observations without some kind of model of the system being observed.
I’d actually been tempted to not write this post as this is all getting rather tedious. However, Michael Kelly’s submitted evidence includes a discussion of the scientific literature and how it is difficult to publish a paper with a different view, and how it’s also difficult to publish a correction. Well, Michael Kelly recently published a paper on extreme events, that I discussed in this post. The paper was pretty poor and I emailed Michael Kelly to point out a very obvious error. To his credit he admitted the error (I wasn’t the only one to point it out) and claimed that he would try to publish a correction. I’ve just checked his paper on Google Scholar, and it appears to have one citation which is not a correction. I think I’ll leave it at that.
...and Then There's Physics 
A couple of things that I couldn’t get into the post. One thing that some might notice is that one of the sources that Michael Kelly cites is WUWT. Another is that he says
A key aspect of science is to present results that quantify our confidence in a particular analysis (or, quantify our uncertainty). Climate science does exactly this. If someone thinks that the uncertainty is much greater than is presented in the scientific literature (as I assume is being implied by using the term deep uncertainty) then one should really do an analysis that shows this, not just hand-wavingly claim it. Promoting the idea that we are much less certain than the scientific analysis suggests is – as far as I’m concerned – equivalent to promoting the idea that we are much more certain than the scientific analysis suggests.
To be quite direct, Kelly’s paper was a prime example of how NOT to do research, and a prime example of questionable research integrity: apart from the bad science evident in the completely weird references (and I include the difficulties anyone would have to even *find* the source in many cases), he knew he submitted to a journal that publishes just about anything, as long as you pay.
And that latter issue is considered scientific misconduct in the revised guidelines for research integrity:
Click to access ALLEA-European-Code-of-Conduct-for-Research-Integrity-2017-1.pdf
page 9, “Establishing or supporting journals that undermine the quality control of research (‘predatory journals’)”
Marco,
Indeed. I had thought of saying more about his paper, but was confident someone would say what I would have said in the comments 🙂
And there is a big difference between scientific certainty and policy certainty.
40 years ago, it was still possible to argue that though the basics of the science linking CO2 to climate change was plausible, global temperatures had in fact hardly risen. Meanwhile, the costs of alternatives to fossil fuels were very high. The ratio of “risk” to cost was very low. Postponement of aggressive action was in those circumstances rational, and postponement duly occurred. (Note however that fossil fuel companies which have long investment horizons knew the risks very well, and started funding their programs of lies and dissimulation about then)
Today, however, the likelihood that the theory of global warming/climate change is correct in its essentials is extremely high, even if the precise parameters haven’t been settled. We have had 40 years plus of inexorably rising temperatures. The costs of renewables have on the other hand collapsed. Wind and solar are cheaper than coal, and are still falling. Storage (via batteries or molten salts) gets cheaper almost every day. Electric cars are already cheaper to run than petrol/gasoline/diesel cars and will very soon cost the same as or less than their fossil fuel counterparts. The risks of doing nothing about global warming is high; the costs of doing something are low. So the ratio has inverted. Now it is very high.
A scientist/climatologist/statistician would not be able to say with certainty that the decadal rate of increase in temperatures has risen from 0.2 degrees C to 0.4 or more, because there aren’t enough observations (even though 2017 looks as if it could, terrifyingly, be hotter than 2016). Nor could RobertScribbler’s “Armada of Icebergs” with certainty be described as the beginning of a period when sea levels will rise by a meter every 20 years.
But politicians (and societies) face a different calculus, one that considers risks, and worst cases. Of course we have Trump and other demented right-wing governments (Australia’s, for example) in the pockets of fossil fuel producers doing their best to deny and lie and prevaricate. But as the trade-off continues to deteriorate, angst and even panic will start to build in chancelleries round the world, and decisive action will start to be taken. We may still of course be too late.
My thoughts, in more detail:
http://volewica.blogspot.com.au/2017/03/policy-vs-statistics.html
http://volewica.blogspot.com.au/2017/03/hope-and-despair.html
Indeed, and I think this is a key point. Research can allow us to quantify something and to become quite confident – in some cases – about our understanding of some system. However, that doesn’t tell us what we should do, given that information. It’s perfectly reasonable for different people to have different views as to what we should do, given the same information.
Since the CCC advises on the government on policy, I can’t help wonder what use “drawing attention to the deep uncertainties” would have. The most likely outcome still remains the most likely outcome, however wide the uncertainty is and there seems to be the familiar incorrect assumption built into this statement that because the future could be less worse that somehow makes the most likely outcome skewed towards an “alarmist” conclusion, whilst ignoring the fact that the uncertainty could also mean that the future was much worse.
Phil,
Yes, I agree. The possibility that everything could be fine is not normally a particularly good argument for not doing something to reduce the risks of things not being fine.
Don’t forget that when people like Kelly refer to uncertainty, it generally is in the most benign direction only, which in terms of risk analysis is a very unengineer-like thing to do.
Prof Kelly’s written submission is a little bombastic.A couple of things:
That’s surely incorrect. A recent summary that addresses this for example is:
F. Estrada et al. (2013) “A Time-Series Analysis of the 20th Century Climate Simulations Produced for the IPCC’s Fourth Assessment Report” PLoS One http://dx.doi.org/10.1371/journal.pone.0060017
Lots of instances of “making an example of a few miscreants”. A brief list can be found here: ( https://en.wikipedia.org/wiki/Scientific_misconduct ).
One might note that the Italian geologists are not on this list. That’s because they were acquitted of misconduct and in fact didn’t “giv[e] professionally unconscionable reassurances about the absence of future earthquake”, although a government public official was found to remain culpable of inappropriate reassurance IIRC.
“liberal elites” eh!
Yes, I was also going to say something about his liberal elites, but – like you – I didn’t really know what to say.
Speaking as an engineer with 40 years practice in industry, I think it hilarious that Kelly holds profit-motivated engineering up as an exemplar for the Sciences. I know of countless examples, even by companies which are generally ethical, where shortcomings of products were deemphasized, and attempts by knowledgeable and responsible engineers to highlight them countered and sometimes punished by retribution. Once, as a test engineer, in fact, a violation of ethics on a product for government was so egregious that I found myself having to resign from the company and turn them into government. Why did they do this? Admitting to the violation was keeping a hundred or so million dollar progress payment from being paid.
And, yes, I’m employed again, in industry, for a company that admires ethics.
Engineering can be very good, but it is hardly, in practice to what the Sciences should aspire.
Nick.
Please chill with the adjectives. It doesn’t help.
“I’m sure there are many engineers who are very good researchers”
…and I know (and have to work with) some who have no idea what is involved in proper research and are terrible at it. Unfortunately, they also tend to share that oft-complained-about characteristic that they think they are the only ones that know how to do anything, and are completely close-minded about learning anything from anyone outside their silo.
With the history of that committee and it’s members, I have the feeling we will see a lot of questions of the sort “Are you now, or have you ever been, a member of the climate consensus?”
Steve McIntyre used to go on about how climate science papers and reports would be so much better if only they were prepared in the rigorous way engineers would write them. Engineering practices are based on well-established—dare I say, consensus—standards. These tend to err on the side of caution, as they should. Standards get revised, but slowly and conservatively. Engineering reports generally apply established principles and templates to repetitive topics: eg, construction design and safety; oil/mineral reserve estimation.
Scientific research, on the other hand, almost by definition, goes beyond the bounds of established knowledge. By its nature it is provisional, uncertain and, more often than not, wrong. Of course, scientists have their own standards, but the varied nature of science from palaeontology to astrophysics defies applying one set of standards across the board. Every scientific research problem has its own mix of issues and there is no inflexible template that can applied to how the results should be presented.
I worked for many years in industry in teams comprising engineers and geoscientists. There is indeed a big difference in culture, a bit like that between roundheads and cavaliers. The geoscientists would sometimes stereotype the engineers as plodding, unimaginative Excel jockies, while the engineers tended to see the geoscientists as flighty and unprofessional. However, being obliged to work together—and it was a deliberate management policy to enforce multi-disciplinary teams—was nonetheless salutary, resulting in grudging mutual acknowledgement of the qualities of the two technical cultures and of the differing challenges that they face.
There probably is a limited role for more involvement by engineers in writing scientific reports intended for the general public. (I’m sure that engineers are already involved in IPCC WG 2&3.) Engineers do have more training and experience than most scientists in preparing such reports. However, the kind of professional arrogance exhibited by Professor Kelly is not likely to encourage future inter-disciplinary cooperation.
That sounds awful 😦
Happily if you peruse the references that Dr. Kelly cites (e.g. his ref [3]) you find rather the opposite:
The Editors: “In science we trust: Poll results on how you feel about Science” Sci. Am. Sept 22, 2010.
“We asked respondents to rank how much they trusted various groups of people on a scale of 1 (strongly distrust) to 5 (strongly trust). Scientists came out on top by a healthy margin”:
scientists: 3.98 average
friends/family: 3.09
nongovmntl org: 3.09
citizens’s groups: 2.69
journalists: 2.57
companies: 1.78
elected officials: 1.76
religious authorities: 1.55
Is science worth investigating? Respondents who agreed with the following statements:
Investment in basic science may not have immediate payoffs for the economy, but it lays the foundation for future growth: 89% agree
Investment in basic science is one of the best ways to stimulate the economy and create jobs: 72% agree
Science doesn’t necessarily lead to economic growth but should be supported for other reasons: 47%”
That doesn’t sound at all like “the public no longer automatically expects new and positive research from science” (a question that wasn’t asked in the poll, btw) nor that “the public no longer ….. trusts their (scientists) statements”.
If that (i.e. citing publications that don’t in fact support one’s assertions at all) is an example of “engineering-type rigour” I’m not too impressed…
> [The Auditor] used to go on about how climate science papers and reports would be so much better if only they were prepared in the rigorous way engineers would write them.
At least until Robert Grumbine visited the Auditor’s, a guest appearance establishing that an example of the engineer-level formal derivation such as one envisioned by the Auditor wasn’t forthcoming:
Kelly goes on:
“By contrast many claims of condensed matter physics, technology and engineering are there for everyone to see, touch feel and use” … “engineering-type rigour”.
The Millennium Bridge, London comes to mind.
In condensed matter physics are there many exact solutions from fundamental principles (vs. computational simulations of toy models)?
Would he consider black holes a “a plausible narrative … and nothing more”?
I think that people expecting an engineering style report—whatever that means and Grumbine’s comments at The Auditor’s blog were spot-on—would somehow recommend reduced efforts, because of the uncertainty, would be disappointed.
For example. If you were designing a bridge, you would make it strong enough to support the highest imaginable load. For example, foreseeing congested traffic, going both ways, of just the heaviest trucks allowed on the road. This load would be very unlikely to occur, but you had better build to withstand it.
Or, if you were charged with building a sea wall or a dock that client wants to last for eighty years, an engineer would design it to cope with the high estimate of sea level rise in 2100. Or, if designing storm drainage system, would you build it to cope only for the historical rainfall record or for the (uncertain) projections that predict that precipitation could be much worse?
Conservatism means preparing for—or attempting to prevent—the worst-case scenario. Professionalism requires you not ignoring the opinions of experts outside your own field. If you want engineers to influence how science is reported, you may well end up getting more, not less, focus on the worst-case outcomes than the IPCC currently provides.
@AndySkuce, yes, definitely. I have been to several policy-oriented meetings, predominantly at local and state levels, and I always thought it quite odd that “resilience planning” was done based upon mean or median projections of, say, sea level rise rather than some extreme quantile. Worse, the projections don’t even acknowledge that the true density is likely to be skewed high, making even classical 1-sigma bounds misleading. I can only imagine that this is an attempt to bake in a limitations on costs to counter early in the analysis. I see no other reason for being so disingenuous.
The Auditor® has talked himself up for a dozen-plus years now as a highly successful consultant who gave up lucrative opportunities* in the hard-nosed practical field of mining to devote time to studying climate alarmism. As per standard operating procedures, this claim has been taken at face value and credulously repeated by our skeptical friends.
And yet, whenever I’ve tried to find much about that (industry awards & citations, company directorships, news releases, trade and technical journals), those claimed achievements recede off into the distance like a mirage.
Maybe I’m just too skeptical.
*Cheyenne: Hundreds of thousands of dollars. Hey, more than that… thousands of thousands.
Harmonica: They call them “millions”.
On topic I think….
https://www.insidehighered.com/news/2017/01/18/librarians-list-predatory-journals-reportedly-removed-due-threats-and-politics
An academic librarian’s lists of “predatory” journals and publishers on Sunday vanished from the internet without explanation. His business partners now say he was forced to shut down the website.
izen,
I knew that it had gone, but it wasn’t clear quite why Beall had decided to take it down.
Andy Skuce: Re designing bridges, engineers always include a safety (fudge) factor as well — just in case something isn’t accounted for.
I’m sure Kelly would be happy to pay the extra taxes needed to pay more scientists and equip them in order to achieve the dizzy heights of being merely as good as engineers.
Recommended supplemental reading…
Increasingly in the current U.S. administration and Congress, questions have been raised about the use of proper scientific methods and accusations have been made about using flawed approaches.
This is especially the case with regard to climate science, as evidenced by the hearing of the House Committee on Science, Space and Technology, chaired by Lamar Smith, on March 29, 2017.
As reported by William Thomas, a senior policy analyst at the American Institute of Physics, the hearing “represented the latest in a string of committee activities revolving around methodological legitimacy in scientific research. Notably, Committee Chair Lamar Smith (R-TX) used the occasion to further articulate his conception of what constitutes ‘sound science.’ The hearing also explored in some detail the question of whether entire fields of research can become corrupted, thus necessitating congressional attention.”
Yes, we can do ‘sound’ climate science even though it’s projecting the future by Kevin Trenberth & Reto Knutti, Conversation US, Apr 5, 2017
Lamar Smith and the ‘sort of fringe’scientists who enable his claim that entire fields of research can become corrupted, thus necessitating congressional attention, have a hard task. The assertion is that Climate science has by some process of motivated group think and confirmation bias promoted Alarmist climate predictions unjustified by sound science. A persistent global ‘thumb on the scales’ in the scientific community that has promulgated a false AGW narrative. Even co-opting the IPCC into legitimising the ‘fake science’ Although some see the IPCC as a part of the UN which they think is a source of radical politics. Rather than a sink.
The problem is going to be finding some criteria of methodological legitimacy that does includes climate science without including a lot of science I doubt they want subject to congressional attention. Reproducability, data access and reliance of computer models is a bigger methodological ‘problem’ in biology and medicine than it is for the physics of radiative transfer. It is a demarcation impossible to make.
Perhaps Lamar Smith should consider whether it is possible for a scientific subject to have mainstream support for 40 years and then be exposed as a mistake, myth or hoax. There are possible examples, but they do not support the idea that some integral flaw in the legitimacy of climate science is credible. Not least because of its consilience with general scientific knowledge.
An inquisition into scientific integrity based on strict adherence to rules to ensure legitimacy takes on a Kafka-resque air when carried out by politicians who apparently regard their own rules to ensure the integrity of the process subject to change by vote…
“This indicates that, to explain the observations, we need some kind of cold matter that interacts only via gravity (Dark Matter), and also some kind of extra energy, known as Dark Energy. Admittedly, we have not yet directly detected Dark Matter, and do not yet know the form of Dark Energy. ”
Confused in a good way. By cold matter I take it that you mean matter that does not officially exist. In that it cannot physically interact with other matter, bounce of it or crush it etc. This though raises the question of the meaning of mass and gravity, both previously inextricably entwined. If, for instance, we put a clump of dark matter at the heart of a galaxy to explain it’s overtight spin and speed but say it does not otherwise interact with matter it is hard to see why it would stay at the center of said galaxy. The alternate explanation is that the mass that is there is producing the gravity which could mean that they have more mass there than they do here. Strange concepts. Even if we tag a component of dark matter to a component of real matter giving double or more the gravity say, the net effect would be that that pound of matter is now behaving as if it were 2 or more pounds of matter. Ie we should just say that it weighs more where it is associated with dark matter?
angech,
In this context, cold means dynamically cold (i.e., the particles are moving slowly relative to the speed of light). This influences how this matter clumps and consequently influences structure formation in the universe. This matter does interact with other matter through its mass (gravity) however it doesn’t radiate and so the only way to infer it’s presence (at the moment) is to detect how it influences the matter around it.
There are people considering alternative forms of gravity, however – as I understand it – these models have trouble explaining some of the observations, such as the bullet cluster.
Cosmology may not be a good subject to raise in the context of political oversight of scientific research. Bad enough that astronomers indulge in the hyperbolic overstatement about their ability to detect and even describe the conditions on planets they claim to be orbiting stars many light-years away. Information with no policy utility and yet it is funded by governments enabling this self-promotion by astronomers speculating about planets. Clearly a suitable target for politicians seeking to curb government subsidy of stuff the free market would reject.!
(grin)
Then there is Cosmology. Where un-testable speculation holds sway. Cold Dark matter to explain observations invokes ‘stuff’ that interacts with mass, it changes the shape of Space-Time, but does not interact with photons (or the weak nuclear force?) so only its effect on gravitation is detectable.
This is problematic for even the extended Standard particle model.
Should Congressional attention be directed at the integrity of a field of science where it is openly admitted that they are completely in the dark about over 80% of what matters?
Off topic (even more).
My two personal favourite cosmological theories come from fiction.
1) Stanislaw Lem in a fictional book review puts forward an idea about the contradiction between the laws of physics we seem to measure here and now that are inadequate to explain the events we see in deep space, and therefore in deep time. His hypothesis is that the laws of the universe really were different in the past, and what we see is the transitions and conflict of the first intelligence to emerge altering the laws of the universe to suit them better after first gaining an understanding of how.
The extreme events and inconsistent gravitational appearance of the far universe is a view of the early changes wrought by the first intelligence that emerged. The point of sentience is not to understand reality, but to change it.
2) The second speculates that the universe is an informational matrix. Each quantum event and interaction between particles is determined by the data about that particle. Its energy, vector, quantum state etc.
All that data has to be stored and accessed. Information has mass, or at least it has a positive energy [Landauer’s principle]. The missing mass in the universe is the bookkeeping. The accumulated data that is required to ‘run’ the universe and ensure that all quantum interactions follow the rules is the hidden (bureaucratic?) weight of the informational matrix that underpins (or is perhaps rolled up into String dimensions) the observable universe. Unfortunately the maths does not work out. Unless you postulate a MASSIVELY inefficient data management system to generate the 80%+ extra mass.
That one comes from the preface of a Discworld story. Can anyone remind me which one?
> There are people considering alternative forms of gravity
Paging Lamar.
ATTP thanks,
Izen 2) The second speculates that the universe is an informational matrix.
Explains things like those small particles moving apart at the speed of light responding at the same time to a stimulus applied to one of them.
“Models provide a way of understanding how our climate responds to various changes”
I almost agree but I insert “might” or “probably” before “responds”. Your mileage varies.
Dark matter research I find worthy. It seems reasonable to answer something that fundamental if the human species is to avoid eventual extinction because it failed to leave Earth.
Michael2: Once climate scientists have perfected the Time Machine they’ve been secretly working on for the past three decades, they’ll be able to zap into year 2100 and tell us which of the GCM simulations best match what they find — assuming the Time Machine works both ways.