I listened to an interesting podcast that some of the regulars may find interesting. It was on the Volts podcast and was on the abuse (and proper use) of climate models. It is an interview with Erica Thompson, who has just published a book on Escape from Model Land: How mathematical models can lead us astray and what we can do about it. Erica is also the originator of the Hawkmoth Effect, which I’ve written about before.
I don’t agree with everything in the podcast, but since I haven’t yet read the book, I thought I would focus on the things I did think were interesting, and did agree with. Something I’ve pondered myself, is the issue of the self-consistency of scenario modelling. Strictly speaking, scenario modelling is typically not self-consistent in the sense that the models rarely, if ever, consider how the impacts will feedback onto the scenario itself. This may not really matter if it is still possible to actually follow that scenario, but may be a problem for extreme scenarios in which the impact may make the scenario essentially impossible.
For example, if the impact of climate change is severe enough, it may have socio-economic impacts that essentially prevent us from actually following a worst-case, high-emission pathway. Similarly for “do nothing” scenarios in pandemic modelling. I don’t think this means we shouldn’t do this. It’s important to explore extreme scenarios, and trying to make these type of models truly self-consistent is probably beyond the scope of most of these modelling efforts. However, it is still worth thinking about this particular issue and how to present these results publicly.
Another point that I found interesting was the idea that we should avoid trying to develop models that converge on some kind of “truth”. There may be cases where models do converge, but we should consider the widest range of outcomes that we can get from plausible models. This is related to a common theme in the climate debate; if people really think climate sensitivity is low, why has someone not funded a group to develop a plausible model that will give such a result?
The final bit of the podcast was mostly about IAMs, which I think my regulars will probably enjoy. It wasn’t complimentary.
A couple of general criticisms. I don’t think it really distinguished well between models of systems that have some structural constancy (i.e., physical systems that obey conservation laws) and those that don’t (i.e., socio-economic systems) and I do think the criticism of the “follow the science” narrative was somewhat overblown. It’s true that science doesn’t tell us what to do, but I think that those who use “follow the science” have made their value judgements and it’s mostly an overly-simplistic slogan, as most slogans are.
Links:
The abuse (and proper use) of climate models – Volts podcast.
Escape from model land: How mathematical models can lead us astray, and what we can do about it – Erica Thompson’s book.
Hawkmoth Effect – my post about the Hawkmoth effect.
...and Then There's Physics 
I purchased this book – both the Kindle and audible versions – as soon as it was available.
But there was so much buzz and enthusiasm on twitter for the podcast interviews, I’ve listened first to three of them – Volts, Challenging Climate, and The Great Simplification.
I still plan on reading/listening to the book itself, but I gotta say that I really didn’t get what all the fuss was about. I definitely got some general vibe from hosts of “I just generally don’t like models, and especially what models seem to be (sometime) telling me, but I have never quite been able to put my finger on it, but in reading your book, it really spoke to me because you also seem to not like models – but you, unlike me, are a *modeller*! Although, to be fair, I still can’t actually put my finger on any of it enough to say exactly why on my own.”
That’s not entirely fair – and especially in the case of Challenging Climate, where, probably not incidentally, co-host Peter Irvine does extensive modelling.🤷 There seemed to be more pushback, interrogation there.
As I said, I am still going to look into the book itself. Podcasts can only touch on so much (and we tend to forget they serve as modern book-tours as well.🤷)
Maybe I didn’t pay enough attention. I going to walk the dog(s) and re-listen to the Volts episode and maybe report back on a second listen later.
Rust,
I do think the points about models typically involving some amount of value judgements are well made, and that modellers should be more conscious, and open, about this. However, I do think there is a difference between judgements being made about parameters in climate models that may not be well defined, but that are often still constrained in some way, and decisions about how to value things in economic models.
I’ve only listened to the one podcast, but I didn’t get the sense that this distinction was made very clear. There is a tendency in the climate debate to be critical of models because “all models are wrong” but there is a big difference between models that are fundamentally based on conservation laws, even if they do have some parameters, and socio-economic models that, as far as I can tell, can vary wildly depending on the judgements of those who use/develop them.
Whatever the problems with models, they do have a key benefit, which is that you can’t make a prediction without one (even if it is a mental model). The further advantage of mathematical and computational models is that they make your assumptions explicit and they can be tested.
Important to know what kind of truth it is. I still don’t understand how the IPCC could think that a multi-model ensemble projection could be considered as “truth centred” rather than being based on “statistical exchangeability”.
The main problem with climate models in the public debate seems to be that the multi-model mean is a conditional prediction of the actual trajectory of the climate on the real Earth, but it is very easy to show that is not the case.
dikran,
Indeed, but I think there is more awareness of this now, compared to what it was like a few years ago.
An ineffective antidote for hawkmoths
https://link.springer.com/article/10.1007/s13194-022-00459-9
I’d like to see some actual worked numerical examples in fluid dynamics showing structurally unstable solutions. Something a little bit more concrete than philosopher A arguing against philosopher B.
All models are wrong and none are useful, according to deniers.
DK, ATTP.
Don’t understand. What has the IPCC stated and/or can you expound/expand your thoughts to those of us that are more simple minded? Thanks in advance.
EFS,
I think this post explains it well. As I understand it, the general IPCC paradigm had been one in which the assumption was that the “truth” would be near the ensemble mean. I think when it comes to weather forecasting, this can sometimes be the case. However, for climate modelling this seems unlikely and it is better to assume that the “truth” (for want of a better term) will probably lie somewhere in the likely range, rather than assuming that if you have a large enough ensemble that it will lie near the ensemble mean.
EFS,
Yes, I agree. The suggestion seems to be that the Hawkmoth Effect is, in some sense, similar to the Butterfly Effect; a structural instability of complex systems that can’t be avoided. However, the senstivity of non-linear, chaotic systems to their initial conditions is well defined. We can actually run non-linear, chaotic models and demonstrate this sensitivity. I don’t the same is true for the Hawkmoth Effect. It seems to be an effect that they claim exists but that hasn’t actually been demonstrated.
If it’s simply a complex way to say “all models are wrong, but some are useful” then it’s not obviously illustrating anything surprising. If it’s more than that, then I’m not entirely sure what it is.
The climate models are sort of “truth centred”, it is just that the modellers hope that they will be centred on the true forced response of the climate system rather than the observations (which are a mixture of the forced response and “weather noise”/unforced response). However, that the approach is “truth centred” even in that sense requires some assumptions about the way in which the models are generated and evolve (basically that they do so in an unbiased manner, in a statistical sense). However I don’t see any reason to think that those assumptions will be exactly met, especially as the modellers are aware of what other groups are doing and broadly what results they get.
Rather than “truth centred”, I think it would be better to describe the models as “our best estimate of the truth, given what we know”. If we are lucky this will converge to the truth, but we need to bear in mind that we may not be lucky.
My main point was though that you need to understand the models well enough to know what they are actually expected to predict/project. While I don’t think the scientific community still views the multi-model ensemble as “truth centred”, I suspect the denizens of climate blogs still do, because it gives a specious way of criticising the models and have their fingers in their collective ears when anybody tries to explain why that is not what the multi-model mean actually tells us.
Basically “truth centred” involves assumptions about the modelling procedure and the processes being modelled, and is my no means an automatic consequence of modelling. As I think ATTP has often said, models are primarily made to help us understand the observations – which is a much better way of looking at them.
I’m glad to see Dikran mentioning “mental models”. Another term would be “descriptive models”. Even in the absence of mathematics, describing what you think a system is doing is an act of creating a model. We use words to communicate all the time, and those words are a form of model – an abstract representation of what we are thinking about the world around us.
“Models” that make no predictions are not of much use – but even a descriptive model typically leads to us saying “..and if my mental model is correct, then this is what we should see in {some other circumstance}…”.
The output of a mathematical model is typically used the same way. If our understandings and assumptions going into the model (data, theoretical constructs, relationships) are reasonably correct, then the outputs are the logical consequences of that understanding. We did not assume the output – we gave our best estimate of the relationships within the data, and the output flows from that.
If we’re pretty sure that A is about 27, and B is about 4, and our theory is that C = A + B, then even though C = 31 was not measured or assumed, C = 31 is a logical consequence of our model that uses values of A and B and a relationship between A, B, and C.
Models are “consequence generators”. If (A,B) then (C).
Known that non-linear otherwise chaotic systems can also deterministically follow the forcing applied. This is where the forced response overrides the natural response. Doesn’t mean that it’s easy to figure out what the response is (based partly on “hawkmoth” structural uncertainty as described by Leonard Smith), but like other forced responses, the dependence on initial conditions becomes irrelevant once it synchronizes with the forcing applied.
The excerpt above from “Synchronization in Oscillatory Networks”, Osipov et al (Springer, 2007)
Having grown up with models of reaction evolution on potential energy surfaces Eli has often thought that the most useful thing about models was observing where they do not go
All bets are off when tipping points are reached. Some, like ice sheets, aren’t even in climate models. Our “model land” view of the future is quite limited. We tend to shine the light on a future that doesn’t differ too much from the present. We don’t have the data or theory to stray too far.
Same goes for economics or politics. A game changer like solar panels is missed.
Something I’ve pondered myself, is the issue of the self-consistency of scenario modelling. Strictly speaking, scenario modelling is typically not self-consistent in the sense that the models rarely, if ever, consider how the impacts will feedback onto the scenario itself.
yes, gosh i remember trying to have this conversation back when SRES approach was used.
basically high levels of economic activity lead to more emissions and more emissions to more damage, but that damage never dampens the economic activity
however, asking this kind of question led to interesting attacks.
in my experience running expensive models for big ticket processes
adding complications never gave us more confidence.
if paper and pencil told you you were fucked, then any mere wrinkle you added that unfucked the system was highly suspect.
pen and paper tell you you cant emit forever. adding knobs to calculate the number of tons, pounds and ounces, better not change that picture
Eli has often thought that the most useful thing about models was observing where they do not go
It is interesting to compare the bulk of IPCC reports to the operating manuals of other Really Big Complex Things.
I was once obliged to study the seven pound thousand page looseleaf manual for the then-new Boeing 757 , prefatory to an inside-out tour of the various electronics bays on the plane, guided by its pilots and engineers.
What I learned was that 900 of the pages and most of the instrumentation on boardpertained to the limits of its flight envelope in terms of speed, pressure , temperature and Mach number, within which it could fly til the tanks ran dry, but outside of which it could be relied upon to crash and burn.
’m glad to see Dikran mentioning “mental models”. Another term would be “descriptive models”. Even in the absence of mathematics, describing what you think a system is doing is an act of creating a model. We use words to communicate all the time, and those words are a form of model – an abstract representation of what we are thinking about the world around us.
yes, one fruitful thing to notice is that models can be seen as a form of data compression.
instead of writing down every force we record for masses under acceleration we
write F=MA
instead of detailing every feline with 4 legs, we say cat.
you cant think without abstractions just ask
Funes the Memorious
“To think is to forget a difference, to generalize, to abstract.
Just thought I’d plant a flag pointing to this online discussion👇*tomorrow* (Friday, March 3) organized around Erica Thompson’s book “Escape from Model Land”.
It’s hosted by Andrew Revkin, but he has two other experts/interested parties on the topic joining them.
I have only listened to a few of these Revkin sessions, but as a general observation, I’ve found the roundtable discussion format amongst various experts much more satisfying for certain topics than the more prevalent “expert interviewed by the non-expert podcast host”.
I *really* think this would be the case for this topic.
I am going to leave this pointer as standalone comment and follow with another separate comment on the book as I have worked my way through parts of it.
As mentioned above in the first comment (mine) to the OP, and prompted by returning to link the pointer to tomorrow’s discussion with the author of “Escape from Model Land”, I thought I would add some comments having delved a bit more into the book itself (rather than just the earlier podcasts).
I have to say, I still don’t quite see what the fuss is about this book.🤷
There’s a lot of retrospective critiquing about how various modeling efforts and groups evolved organically and may have – inadvertently or not – incorporated blindspots or rigidities that might not have been the dependent path had other areas of expertise been involved earlier. And how the “establishment view” might now be biased – by demographics and training of the modelers, model organization, “inside knowledge”/familiarity of the model design and assumptions, etc. – against “outside the box” inputs/contributors.
I was struck by how familiar this sounded to the c. mid-2000’s whining about climate science by outside, er, “auditors” and fellow travellers.
• The surface temperature record had a number of accumulated, bottom-up biases which would not be how a study started fresh today would conduct such an effort.
• The weather station data itself was hopelessly biased and corrupt, and could only be salvaged by a massive citizen science effort to try to identify and fix all the errors.
• An “engineering quality” audit of the climate models and data needed to be made in order to have any confidence whatsoever in the outputs.
• The latter of which couldn’t really be undertaken because the code and data wasn’t readily open.
• etc., etc.
All of which could sound pretty devastating charges at the time, so long as they were in the abstract and just sort of free-floating out there without having to positively identify any specific, concrete, material issues.
But then some funny things happened.
• Partly in response, models and code *were* made much more available and transparent.
• Some intrepid groups actually tried to follow through on their hunches as to where deficiencies (beyond those the modellers themselves were aware of and working to improve) might exist.
And… what came of it all exactly?
For the most part, faced with having to now actually do the work and show how a different approach would have reached a significantly different outcome… for the most part… *CRICKETS*.
Of the few that actually did seem to attempt to do original work, the WUWT surface station projects and the GWPF whatever-it-was just seemed to quietly disappear, never to be heard from again. Presumably because their independent, first-principles projects had not delivered them (a) the results they wanted/expected, nor (b) results materially different to the extant science/datasets.
To their considerable credit, the Berkeley Earth BEST project *did* publish their findings. Which overwhelmingly confirmed the already existing analyses. Much to the apparently slack-jawed bewilderment of some of the most vocal prior critics of the methodologies and professionalism of the earlier modelers.
And I get a real sense of deja vu with Thompson’s book.
Easy to sit on the edges and say “What about this? What about that? I would have done this differently, and that differently. Does this physical climate model do it like e̶n̶g̶i̶n̶e̶e̶r̶s̶ ecologists would have done it? That might overturn everything we know about the planet warming. Why do these economic models assume future individuals will try to invest expecting a positive return – isn’t that a bit narrow-minded?”
I am reminded of Ajay Gambhir’s (et al.) 2019 paper “A review of criticisms of integrated assessment models and proposed approaches to address these, through the lens of BECCS, which includes as part of the conclusion this paragraph:
There’s much more to that paper – and it’s not a love letter to IAM’s by any means.
But it’s much easier to carp at things from the outside and assume and speculate the modelers have somehow completely overlooked something that is just so blindingly obvious to you and/or experts from other domains.
The proof is in actually showing how your novel approach would work, and whether it is materially different.
I have probably overstayed my ability to hold anyone’s attention on this, so I will leave my commentary here…
… except to close by saying that I was *particularly* underwhelmed by Thompson’s chapter 8: “The Atmosphere is Complicated”, where she brings her general critiques of modeling to bear on specific alleged problems with physical climate models (AOGCM’s, ESM”s, ESS’s) and climate-economics models (IAM’ s).
In discussions elsewhere in the book about areas (epidemiology, etc.) where I have effectively no familiarity, I took her examples of problems at her word.
But in the case of some of her critiques in the climate chapter, that in turn seems to be what she herself has done! She’s read someone else’s critiques and just assumed that they were valid. And in some cases where I am familiar, it was clear to me that the critique itself was where the deficiency was. Or, it was unclear to me that the proposed alternative approach could actually be productive or significantly change things (e.g., at one point she suggests that rather than having built up climate models from energy balance models, fluid dynamics, coupling in atmospheric and ocean models, etc., we start from a blank page and model ecosystems and work through their interactions to see what climate it yields… Like, I guess, how does the Serengeti currently determine the climate of the Arctic tundra and vice versa and how would a climate model work if we removed lions or something?🤔)
Like, interesting as thought experiments or daydreams, but I was just sort of underwhelmed thinking such efforts were going to get us to substantially better understandings.
It may come down to how much relative confidence/respect an individual has in (presumably diligent!) professionals in other domains.🤷
Rust,
Indeed, you highlight a number of issues that I have. I think it’s good to challenge people, but there is an element of academic criticism that seems to involve people who spend most of their time criticising and not much time presenting constructive solutions. I don’t think this book quite falls into that category, but I do worry that it will be mostly used by those who do.
RNS,
I’ll wait for the transcript and maybe even read the book or read the reviews from actual climate science modelers. Pop science is only interesting if it makes it to something like PBS Nova. Oh and Revkin is a toad. 😀
Tamsin Edwards gives a high-level modeler’s perspective in this recent podcast. Jives with my experience – no one understands model limitations as well as modelers.
https://www.ukaht.org/antarctica-in-sight/podcasts/all-models-are-wrong/
Chubbs,
Thanks, that is good.
All models that cite G.E.P. Box are wrong.
All models that cite George Cox are also wrong
LOL