Why do we do research?

I thought I would briefly comment a little more about the claims made in John McLean’s thesis. Something to bear in mind is why we do research. Essentially, research is very simply about trying to understand something; to answer some question, or test a hypothesis. If you’re lucky, you can undertake a carefully designed, controlled experiment that produces data that can be easily use to answer the question that was posed. In many cases, however, this isn’t possible, and you need to undertake some kind of complex data analysis in order to get an answer.

Consider global surface temperatures. We’d like to understand if we’ve warmed over the last 100 years or so and, if we have, by how much. The problem is that we didn’t set up monitoring stations in the mid-1800s with this in mind. We do, however, have temperature measurements that go back to the 1800s, so we can work with these. However, instruments have changed, measuring stations have moved, the number of measurements has changed, the environment in which the measurements are made can have been artificially altered, the time at which measurements were made can have changed, and – in some cases – even the way in which measurements were made has changed. Therefore, if you want to construct some kind of global surface temperature record, you need to try and correct for these various non-climatic factors.

However, doing so requires developing some kind of data analysis technique and also using some judgement as to how to process this data. It can’t be perfect, but you can test to see how various methodological choices influence the results. People might even disagree with some of these choices; this doesn’t make them wrong. Some of the data might even be processed in a way that is clearly wrong, but if there is a lot of data, you might not be able to check how every data point is influenced by the analysis method. Again, you can check how this kind of thing would influence the results.

So, what about John McLean’s thesis? There’s nothing wrong with checking the data used to generate global temperature datasets. However, if this is to be a serious research project, then a key aspect is to understand how potential errors might influence the results. Simply pointing out possible errors tells us little if we don’t also understand the significance of these errors. By itself, it doesn’t really advance our understanding at all. If you really want to advance our understanding, you need to do more than simply highlight possible data errors (I’m ignoring for now that Berkeley Earth appears to have already flagged most of the issues highlighted in McLean’s thesis).

All I’m really trying to point out is that research is fundamentally about improving our understanding. Auditing a data set so as to point out possible errors can certainly play a role, but by itself does little. This is especially true if another analysis has already identified most of these issues and shown that the impact on the result is negligible.

Advertisements
This entry was posted in Climate change, Global warming, Research, The philosophy of science, The scientific method, Uncategorized and tagged , , , , . Bookmark the permalink.

42 Responses to Why do we do research?

  1. Everett F Sargent says:

    I think I’ve already said as much, although I’m using the WMC “so called” rude approach.

    But over at WTFUWT? they have just now posted …
    The DIY Nonsense Detector Kit: How to differentiate science from psychobabble.
    (sorry no direct link)

    BEST passes Number 1

    “Number 2: Nonsense Indicator: Claimant Has Only Searched for Confirmatory Evidence”

    McLean fails Number 2

    “Number 3: Nonsense Indicator: Personal Beliefs and Biases Drive the Conclusions”

    McLean fails Number 3

    Numbers 4, 5 and 6 are not applicable to the specifics of this “so called” issue IMHO. 🙂

    Have a nice day.

  2. Everett F Sargent says:

    On another note …

    Did McLean even do any “so called” formal hypothesis testing? Meaning p-values? AFAIK there was no formal hypothesis testing, just floating a conjecture because “look see I found some issues” therefore the entire data set is suspect, and guilt by association all such data sets are suspect.

  3. dikranmarsupial says:

    I’m glad I safely stowed my irony-meter when I had a look at the toolkit:

    #1 “Claims Haven’t Been Verified by an Independent Source”

    So how long were Einstein’s theories of special and general relativity “nonsense”? ;o)

    #6 “Use of Excessive ‘Science Sounding’ Words or Concepts”

    Right, so the use of technical terms (“space-time”, “accretion disk”, “quantum superposition”…) is indicative of nonsense rather than science?

    Thanks that humour post at WUWT made my day, but I think not in the manner intended! ;o)

  4. izen says:

    Nick quotes jcmdl

    “This thesis makes little attempt to quantify the uncertainties exposed by this investigation, … It has been left to others to quantify the impact of incomplete data, inconsistencies, questionable assumptions, very likely data errors and questionable adjustments of the recorded data.”

    The methods required to quantify the uncertainties are beyond my understanding, but SM and VV have demonstrated that the methods are known, and can be applied to measure the impacts of quality control methods.
    It seems … odd, that jcmdl made no attempt to quantify the uncertainties exposed or at least get some assessment of the impact of the errors he detected. Or is it too Machiavellian to guess he did, and decided to omit that.

    I have no direct experience of PhD thesis work, but I know people who have. One example from long in the past was a thesis that would have been quite impressive it it had stopped at the hypothesis that a particular biochemical could be a significant predictive marker for a serious deterioration in a life threatening disease state.
    The inference of how the biochemical could be a precursor and would give several days warning of catastrophic organ failure was credible. The method by which reliable samples could be taken and processed rapidly to provide a timely warning were developed and described.

    Unfortunately they then went on to actually take enough samples from test subjects and real patients to test the hypo-thesis.
    Which resulted in the conclusion of the thesis being that with a higher than 95% confidence, the bio-marker had absolutely NO diagnostic utility.

    Negative results have their own value, but I suspect the thesis might have gained more attention if it had stopped at describing the potential utility of the hypothetical link between the detectable bio-marker, the means of measuring it, and suggested that, “It has been left to others to quantify the impact …”

  5. There are variations of PhD dissertations that are essentially composed of a candidate’s peer-reviewed papers, stitched together by a minimal narrative. That often happens if the research area is target-rich for findings. McLean’s thesis looks like a negative findings work, which has got to be difficult to defend.

  6. dikranmarsupial says:

    “McLean’s thesis looks like a negative findings work, which has got to be difficult to defend.”

    It shouldn’t be. If the candidate can show that they followed a research programme that would have produced a positive finding if there was one to be found, then the examiners should be satisfied with that. The requirement for a PhD is that you demonstrate the ability to perform research competently in a particular field. There is a requirement to make a contribution to the field, but that can be positive or negative. A positive finding from a questionable research programme is not as good as a negative finding from a thorough, careful, well designed research program.

  7. Mitch says:

    I am amused that a mediocre-to-poor thesis is generating so much buzz. Denialati that publicize this piece of work make themselves look foolish.

  8. The Very Reverend Jebediah Hypotenuse says:

    Why do we do research?

    A few do it for the money – Ha!
    Some do it for the fame,
    Some do it to lob cocktails,
    ‘Tis all part of The Game.

    There are many roads to ‘Doctor’
    To a new-minted PhD,
    Some paths so dark and twisty
    that they defy others to see.

    One can search for, and hope to
    uncover, astonishing new facts,
    or one can stand and deliver thinly-
    veiled ad hominem attacks.

    The construction of science paradigms
    are such labours of Hercules
    that some will quit, and heed advice:
    “get a real job”, if you please.

    Finding nits is a thankless, boring task.
    So few do part the hairy data actually to look,
    (Tho’ the wiggy folks at GISS were kind enough to ask)
    But – hey – if I evoke sufficient FUD, I can sell a book.

    Through posts on blogs around the globe
    Doc McLean now joins the frenzied spate
    To add to scandals “Climate-” and “Glacier-”
    A devastating dissertation, hashtag “Data-Gate!”

    The integrity(tm) of science cannot be too good
    ‘Cause Nova, Watts, and Delingpole are working on the case.
    Hadley-data Sherlocks are wanderin’ ’round the ‘hood,
    Alarmist climate scam-artists are being shown their place.

    After everything they’ve exposed, it’s impossible to say
    that climate science is getting any nearer to the Truth
    concerning expected average weather to be encountered by our youth.
    No – really – it’s a just a money-grab – by those grant-funded uncouth.

    But as the naturalists so long observed, a flea
    hath even smaller fleas that on him prey;
    And these have smaller still to bite ’em,
    And so on, ad infinitum.

    Thus it follows that Doctor John McLean,
    like every poet in his kind,
    Will be bit upside his thesis,
    by them that comes behind.

  9. What I would consider a typical negative finding in climate science is to assert some behavior as chaotic or impossible to solve. I think it would be hard to defend because you would have to rule out all the possible non-chaotic or non-linear behaviors that could produce the same result.
    PhD committees are sensitive to this class of theses because they know better than others that something deemed impossible often becomes solvable over time. So they find it’s better to take a risk on a novel finding rather than accept a pedantic negative finding that may not age well.

    Moreover, people do research because of the motivation of discovering a breakthrough, rather than to assert a dead-end. That spirit is passed on generation to generation in research academia.

  10. hi strangers, dropped in in hopes of finding a place to recommend long-read article about Bruno Latour. While sideways to this topic, it is not entirely irrelevant. I come to Latour unprejudiced by prior knowledge, which may be why I liked what I found, particularly in the latter part (do scroll down). https://www.nytimes.com/2018/10/25/magazine/bruno-latour-post-truth-philosopher-science.html

    Part of the trouble with climate change has been that its breadth and complexity defy disciplinary boundaries, making it difficult for specialists to convey the implications of atmospheric patterns from their data alone. What the critical-zone observatories had done, Gaillardet said, was to draw together scientists working in Balkanized disciplines to describe minute environmental changes that more general models of earth-systems science could not detect. But even though human beings were the cause of these changes, earth-systems science had until recently focused on the natural world to the exclusion of the social.

    With Latour’s appearance on the scene, labs like Gaillardet’s have started to study environmental changes with a thorough recognition that humans and nonhumans, society and nature, are inseparable, bound together in a web of reciprocal influence. This is not simply philosophical conjecture. As Latour has long maintained, critical-zone scientists themselves — like many environmental researchers — play a part in the cyclical processes they study: Others use their research to make changes to the very environment they are measuring, in turn challenging the traditional image of scientists as disinterested observers of a passive natural world.
    ….
    … Latour’s description of the earth in the Anthropocene as “an active, local, limited, sensitive, fragile, trembling and easily irritated envelope.” He stood before the small monitor, rapt. “It’s beautiful that ocean waves can actually be heard in the middle of the Vosges,” he said. “The whole earth is made sensitive here. It’s very moving.”

    … what they have always missed — was that Latour never sought to deny the existence of gravity. He has been doing something much more unusual: trying to redescribe the conditions by which this knowledge comes to be known.

    Crowded into the little concrete room, we were seeing gravity as Latour had always seen it — not as the thing in itself, nor as a mental representation, but as scientific technology allowed us to see it. This, in Latour’s view, was the only way it could be seen. Gravity, he has argued time and again, was created and made visible by the labor and expertise of scientists, the government funding that paid for their education, the electricity that powered up the sluggish computer, the truck that transported the gravimeter to the mountaintop, the geophysicists who translated its readings into calculations and legible diagrams, and so on. Without this network, the invisible waves would remain lost to our senses. For a few moments, Latour stood reverently before the rolling waves on the screen. Then he said to the assembled scientists, as though he were admiring a newborn child, “Beautiful — you must be really proud.”

    (apologies for the length! I was moved by this.)

  11. Hi Susan,
    Hope all is well. I had seen that article and was thinking of writing about it, but wasn’t sure what to make of it. I’ll try reading it again.

  12. aTTP: Glad it was not inappropriate. We muddle along, Dad nearly 95. Things a bit wild this side of the “pond” as you might imagine (Virgil, Milton, Goethe speak to the losses inherent in true evil …).

    I was irritated in the quote above about gravity. which was not “created” by scientists though the rest seems accurate to me. That’s all for now.

  13. David B. Benson says:

    dikran the early mammal — The special theory of relativity was immediately seen as correct. No one doubted the general theory of relativity after the measurement of light bending during an eclipse in about 1919. The main difficulty was that few understood it. Even Einstein had his difficulties witnessed by the so-called cosmological constant.

  14. Everett F Sargent says:

    SA,

    The part about ocean waves is very interesting (infragravity waves (30-300 seconds) and ocean waves (3-30 seconds)) in relationship to the Earth’s Hum (microseisms). I actually did a lot of work in that area starting in 1983 (FRF for 1st hand knowledge of infragravity waves and later on harbor resonance and moored ship motion in harbors (primarily Los Angeles and Long Beach Harbors)). Others where I worked discussed the Hum (bit foggy about the acoustics and whatnot, but it was always discussed in relationship to speeding up the recording into the audible spectrum (above 20 Hz)).

    I chased down a bunch of relatively recent and very interesting papers. Thanks.

    Note to self: Watch out for less then and greater then keys, it sort of disappears stuff.

  15. angech says:

    Everett F Sargent says:
    “The part about ocean waves is very interesting (infragravity waves (30-300 seconds).”
    Thanks Everett, gravity waves and gravitational waves. Getting their slowly. Always good to find something new to make it more complicated.

  16. Pingback: A defense of science? | …and Then There's Physics

  17. Dave_Geologist says:

    What I would consider a typical negative finding in climate science is to assert some behavior as chaotic or impossible to solve.

    Paul, I rather suspect that the set of such theses is the empty set.

    As dikran has said, one of the things an examiner is looking for is a demonstration that the candidate has learned how to do research. To fulfil the “apprenticeship” role of a PhD. Very few come up with a ground-breaking breakthrough. If they did, we’d be reading about tens or hundreds of thousands such every year. Part of the art of being a supervisor lies in recognising where the student is heading up a blind alley and advising on alternative directions for the research. Part of the apprenticeship lies in butting your nose against the end of a blind alley and realising that’s what you’re in. In my field (not just the systematic palaeontology part) there’s still a lot of ground to cover and a lot of basic description, measurement and analysis to be done, which may be integrated by you or by someone else five or ten years later. The same must be true of biology and astronomy. You’ll generally turn up something interesting anyway, and if not, there are publication outlets such as PLOSOne and museum monographs. Science is not just built on the shoulders of giants, but is an edifice built brick-by-brick.

  18. dikranmarsupial says:

    “What I would consider a typical negative finding in climate science is to assert some behavior as chaotic or impossible to solve. ”

    You don’t get a PhD by merely asserting anything.

  19. Dave_Geologist says:

    You don’t get a PhD by merely asserting anything.

    Another reason for my empty-set comment dikran 😉 . As in geology, there’s a wealth of information which can be researched about a climate phenomenon without coming up with some “ultimate cause” or even “predictability”. For example “drought here, floods there during an El Niño” can be useful to science and to society, even if it doesn’t advance the cause of predicting which year each of the next six El Niños will arrive in. We generally get three to six months notice anyway, once it’s decided to manifest.

  20. dikranmarsupial says:

    Would Godel get a PhD for his incompleteness theorems (which are a negative result, although (AIUI) not absolutely conclusively so)?

  21. Dave_Geologist says:

    And with a nod to our host, astronomers didn’t stop searching for exoplanets once the first discovery had proven that exoplanets exist (RIP Kepler 😦 ).

  22. Dave,
    Indeed, although we do keep improving the methods so that we can find exoplanets that we couldn’t find before (i.e., mass/radius, or orbital properties).

  23. “Paul, I rather suspect that the set of such theses is the empty set.”

    Or some behavior that is asserted to be statistical, when other potential deterministic factors have not been ruled out. I see many research papers, if not theses, that make this claim.

  24. dikranmarsupial says:

    PP I think you ought to define what you mean by “statistical”. Above the quantum level, there is very little in the physical world that is genuinely random – it is just used to represent factors of which we are ignorant or which it would be unduly expensive to include explicitly. Note chaotic processes are by definition deterministic.

  25. Willard says:

    > No one doubted the general theory of relativity after the measurement of light bending during an eclipse in about 1919.

    Never misunderestimate contrarians:

    A collection of various criticisms can be found in the book Hundert Autoren gegen Einstein (A Hundred Authors Against Einstein), published in 1931. It contains very short texts from 28 authors, and excerpts from the publications of another 19 authors. The rest consists of a list that also includes people who only for some time were opposed to relativity. Besides philosophic objections (mostly based on Kantianism), also some alleged elementary failures of the theory were included; however, as some commented, those failures were due to the authors’ misunderstanding of relativity. For example, Hans Reichenbach described the book as an “accumulation of naive errors”, and as “unintentionally funny”. Albert von Brunn interpreted the book as a backward step to the 16th and 17th century, and Einstein said, in response to the book, that if he were wrong, then one author would have been enough.

    According to Goenner, the contributions to the book are a mixture of mathematical–physical incompetence, hubris, and the feelings of the critics of being suppressed by contemporary physicists advocating for the new theory. The compilation of the authors show, Goenner continues, that this was not a reaction within the physics community—only one physicist (Karl Strehl) and three mathematicians (Jean-Marie Le Roux, Emanuel Lasker and Hjalmar Mellin) were present—but a reaction of an inadequately educated academic citizenship, which didn’t know what to do with relativity. As regards the average age of the authors: 57% were substantially older than Einstein, one third was around the same age, and only two persons were substantially younger. Two authors (Reuterdahl, von Mitis) were antisemitic and four others were possibly connected to the Nazi movement. On the other hand, no antisemitic expression can be found in the book, and it also included contributions of some authors of Jewish ancestry (Salomo Friedländer, Ludwig Goldschmidt, Hans Israel, Emanuel Lasker, Oskar Kraus, Menyhért Palágyi).

    https://en.wikipedia.org/wiki/Criticism_of_the_theory_of_relativity#A_Hundred_Authors_Against_Einstein

    Emanuel Lasker was World Chess Champion for 27 years, a record that may not be broken. He was also a mathematician (we owe him the Lasker-Noether theorem), and a good friend of Albert.

  26. Dave_Geologist says:

    Einstein said, in response to the book, that if he were wrong, then one author would have been enough

    Haha. Popper, decades before Popper. Assuming he actually said it. Why are all the best quotes apocryphal? Life isn’t fair 😦 .

    A Hundred Authors Against Einstein. Nowadays, as per evolution and AGW denial, they’d have had no qualms about calling it A Hundred Scientists.

  27. Willard says:

    That ad hominem is getting old, Dave. Once upon a time, scientists called themselves philosophers. Scientists borrowing Albert’s fame and fortune is not very different from when contrarians claim to be of Galileo’s descent.

    Perhaps you’d prefer a hall of fame:

    Dayton Miller performed thousands of tests with his interferometer equipment to quantify the ether drift on Earth’s surface. His prediction of the ether drift in free space was confirmed by the determination much later of the velocity of the sun through space. It was further confirmed by Yu. M. Galaev in 2002.

    His tireless work spanning 30 years was buried by mainstream astrophysics in favour of Einstein’s theory of General Relativity.

    Einstein himself knew Miller’s work, if successful, would falsify his hypothesis.

    The dirty deed was done by Robert Shankland, Miller’s former assistant.

    https://tallbloke.wordpress.com/hall-of-fame/

  28. DM, I am thinking in terms of all the major climate behaviors that are under intense scrutiny. Need some terminology here to distinguish how how predictable we can make our models.

    For example, the mechanism of tides is one of countless statistically interacting particles with possibly chaotic trajectories, yet the end result is one of useful predictability.

  29. dikranmarsupial says:

    sort of relevant to falsificationism (from this excellent paper)

    These tests posed for us an unusual problem in experimental work. Normally, one does experiments in which there is some uncertainty in the expected outcome. In these experiments, however, our confidence in the idea that the Earth rotates, and in the applicability of conservation of angular momentum to masses of fluid, was probably so strong that experimental denial would have been almost inadmissible. We should have gone to unusual lengths to get the apparatus to work as expected. Realizing this, we found ourselves reluctant to accept as conclusive the results we were getting, results which apparently confirmed our ideas. Once can never prove, for example, that it was not some small air current which persistently maintained a circulation that gave the results we observed, and that a quantitatively comparable, but oppositely directed air current caused Shapiro’s results. There is, in principle, an infinite number of hypotheses that can explain any set of observations. This difficulty in validation of scientific theories is not a new one and, in this instance, as in all instances, it cannot be proved that any one hypotheses is correct. Nevertheless we have acquired confidence in the hypothesis that carefully performed experiments on liquid drainage from a tank will show clockwise rotation, if done in the southern hemisphere.

  30. dikranmarsupial says:

    “Need some terminology here to distinguish how how predictable we can make our models.”

    Then statistical, as opposed to deterministic is an extremely poor choice of words as they are by no means antonyms. Chaotic processes are completely deterministic but may not be predictable (although they may have statistical properties that are predictable).

    “For example, the mechanism of tides is one of countless statistically interacting particles with possibly chaotic trajectories, yet the end result is one of useful predictability.”

    Again, chaotic processes are deterministic by definition, so it is unclear what you mean by “statistically interacting particles”.

  31. A statistical interaction is to interactions as statistical mechanics is to mechanics, simply a useful way to understand properties of matter that exist as an ensemble of particles.

    Is the concern that scientists have hijacked the term statistical?

  32. dikranmarsupial says:

    Paul, you are still being opaque, and expressing yourself in cryptic manner really doesn’t help.

    The “statistical” in “statistical mechanics” has nothing to do with predictability.

  33. It certainly does have to do with predictability. A mean flow of particles as defined by the Fokker-Planck equations (which is derived from statistical mechanics) can show a high degree of predictability. This simply results from the law of large numbers applied to the particle population.

    A predictable process is a stochastic process whose value is knowable at a prior time:
    https://en.wikipedia.org/wiki/Predictable_process

    Lots of these examples in everyday life.

  34. dikranmarsupial says:

    PP you are missing the point. Some “statistical” interactions are predictable and some are not, hence it doesn’t fit your stated aim ” Need some terminology here to distinguish how how predictable we can make our models.”. However I suspect this discussion is going nowhere, so I’ll leave it there.

  35. I would think that a discussion of what can be predicted or not would be at the heart of any research concerning climate or meteorological behaviors. This discussion may not go anywhere here, yet it’s certainly a valid thesis topic for many a PhD dissertation to come.

  36. dikranmarsupial says:

    “I would think that a discussion of what can be predicted or not would be at the heart of any research concerning climate or meteorological behaviors.”

    This is just straightforward evasion. Nobody was arguing that predictability was not important, just that your statistical-versus-deterministic terminology doesn’t make that distinction. It is a pity that discussions of science end up in this sort of rhetorical evasive world-play.

  37. angech says:

    My wife asked what it would take to make me admit AGW was real and a concern.
    In trying to explain it I said a rate of warming if 0.3C a decade for decades.
    Why she said.
    Which brought me to
    “All I’m really trying to point out is that research is fundamentally about improving our understanding.”
    So, I looked that rate up. Turns out it comes from IPCC 1990.
    But later IPCC’s seem to dismiss it.
    So there is the rub. Is that the rate expected and why is it not happening?

  38. angech,
    I think the answer to your question is that it isn’t the rate expected. As I understand it, the observed rate of warming is somewhat below the multi-mode mean, but the multi-model mean is more like 0.2K/decade, than 0.3K/decade.

  39. dikranmarsupial says:

    “So, I looked that rate up. Turns out it comes from IPCC 1990.”

    Can I have a page number for that, so I can check?

  40. angech says:

    In executive summary near start page 19 out of 414
    Business as usual scenario A.
    Based on current models we predict …. FAR 1990.

  41. A link would help. I can’t even work out which document you mean (there isn’t one called “Executive Summary”).

  42. dikranmarsupial says:

    Angech is referring to the first IPCC report however it is a case of cherry picking, and a lack of self skepticism. The 0.3C per decade is an average warming rate over the 21st century under a business as usual scenario. The first thing angech is not letting on about is that the error bars on that are 0.2-0.5C per decade. So requiring 0.3 as the threshold where 0.2 is possible is clearly not being very self-skeptical. Also the rate of warming under a BAU scenario is projected to increase as the 21st century progresses, so even if 0.3C is the average rate, we would expect to see a lower rate at the start. So again angech’s test is strongly biased in favour of protecting his current position. Have we followed the 1990 BAU scenario? I don’t have the figures to hand, but I don’t think we have; many nations have taken action to limit FF use, rather than exploit them as fast as possible. so the scenario choice is also a source of bias. Scenario B (which IIRC is more like what we have actually followed) gives an average rate of 0.2C.

    Interestingly the BAU scenario suggests a rise of 1C by 2025, and we are not far off that now (IIRC), even thought that scenarios is a bit pessimistic in terms of emissions reductions. £C by the end of the century, which reinforces my point about the accelleration in warmimg making the 0.3C/decade criterion an unreasonable test.

    So in short, all angech has done is illustrated that his self-skepticism is rather limited.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.