I thought I would highlight a recent video presentation by Eric Winsberg, called Moral Models, Crucial Decisions in the Age of Computer Simulations. Some may remember that Eric co-wrote a post here about extreme weather event attribution.
The theme of Eric’s presentation is the moral significance of models and their influence on society. Eric makes a number of points that I largely agree with. A key point is that science doesn’t make decisions, people do. Models can inform decision making, but can’t define it.
Eric focuses particularly on the models used to understand the spread of the COVID-19 virus. Eric is rather critical of some of the modelling, in particular the influential Imperial College model. Eric highlights that some of the early models didn’t do much in the way of sensitivity tests and didn’t really consider the broader implications of their assumed interventions. I’m aware of an attempt to do a sensitivity analysis of the Imperial College model, but I didn’t think it was particularly useful.
What Eric stresses is that the assumptions that go into these models are not value-free. For example, in the case of COVID modelling, modellers will need to decide which potential interventions to consider, and they can almost certainly not do this in an entirely value-free way. What Eric suggests is that the modellers should have given more thought to how the interventions they chose to consider might influence other sectors of society, in particular those sectors with which the modellers probably have no association.
Eric also criticised the models for not considering the broader implications of the potential interventions. How would closing schools influence school children, and their parents? How would closing sectors of the economy influence those who might not be able to easily work from home? etc. Although these are perfectly valid concerns, this is where I somewhat disagree with Eric.
I think it’s very challenging to self-consistently include these impacts in the models and this may well go beyond what these models are designed to do, or should even try to do. Also, I think these are issues that policy makers should be aware of. They should be getting advice from other experts about the economic, and social, impacts of the various possible strategies. I do think we should be careful of suggesting that it was the responsibility of the modellers to provide this broader perspective.
However, I do think that modellers should be as clear as possible about the limitations of their models. This is partly simply because it’s the right thing to do, but also because it’s key to stress that models only inform decision making, not define it. If I was ever in a position to provide advice to policy makers, I would be pretty worried if I thought their decisions were based only on the information I presented. I would want policy makers to be informed by a broad range of experts. It’s key, in my view, to stress that the responsibility for making decisions lies with them, not with modellers/scientists.
Anyway, as usual I’ve written more than I had intended. Eric’s video presentation is below. Even if you don’t agree with it all, it certainly presents some ideas that are worth thinking about.
...and Then There's Physics 
Interesting video, but I don’t think Putin got the message:
https://vvattsupwiththat.blogspot.com/2022/02/a-brief-history-of-russian-climate.html
Not one mention of game theory by Winsberg. It really does make the discussion moot.
ATTP: Could you explain why you thought the sensitivity analysis of Covidsim wasn’t very useful?
(from my point of view the usual problem with systematic uncertainty quantification stuff is that the uncertainty of the parameters is not known, and there are a whole bunch of unknown unknowns anyway)
Ben,
My problem with it wasn’t that it didn’t (IIRC) distinguish between biological parameters and scenario (or intervention) parameters. We already knew from the original report 9 that there were a wide range of possible outcomes depending on the scenario that was modelled. I think their senstivity analysis simply selected a sample of parameters and varied them, producing a wide range of results. This wasn’t really a surprise.
What I think would have been more interesting would have been to select either biological parameters and varied those, or considered various specific scenarios and varied the parameters associated with them. I think that would have been more informative than simply selecting a small sample of the parameters and varying them.
ATTP: Thanks, I think I went in with very low expectations of them doing much except varying a bunch of parameters and then finding an output distribution (instead of just a point prediction): my experience with the VVUQ crowd is they tend to mostly talk about how useful VVUQ is, and how everyone who isn’t doing it is getting useless answers.
So I hadn’t even noticed that they were varying intervention parameters as well as biological ones. It is indeed a bit pointless to regard differing responses to differing interventions as an “uncertainty” when the main reason for doing the modelling is to try to choose a strategy.
Ben,
I’ve just had a quick look through the paper again, and maybe I’m being a bit unfair. They did at least identify the different types of parameters. I just didn’t think their analysis really told us anything particularly useful, but maybe I’m missing something.
Ben McMillan says
“(from my point of view the usual problem with systematic uncertainty quantification stuff is that the uncertainty of the parameters is not known, and there are a whole bunch of unknown unknowns anyway)”
–
Thanks, Ben.
A moral answer to a moral model question.
But more pointedly highlighting the aspect of how much we need to know in the other fields of claimed certainty.
–
Skepticism has a part to play after all.
Since UAH and RSS are peas in a pod as models apart from the model assumptions.
Basically the data should correlate extremely well.
Will RSS follow suit again or deviate up?
” Skepticism has a part to play after all.”
news at 11! (not)
” Basically the data should correlate extremely well.”
and indeed they do
https://woodfortrees.org/graph/uah6/normalise/plot/rss/normalise
IIRC the original March IC model was tuned to not letting ICUs be overwhelmed like in northern Italy. Hence the successive lockdowns being introduced just in time to stop ICUs being overwhelmed. You could debate whether having double or triple the IFR due to people dying untreated on trolleys is a more or less harmful thing than kids missing school and their parents having to make childcare arrangements, but I know which side I’d be on.
They might not have been as explicit as they could have been in the text, although it should have been obvious from the graphs, and from the statement that the update which triggered lockdown was driven by new information from Italy on what percentage of patients needed ventilators, and an outbreak of realism in the NHS over just how many ventilators they could staff, however many they bought.
There was an interview with Nature which went into how they handled the uncertainties. IIRC they weren’t set up for rapid re-runs given their limited computer hardware, and were used to running jobs for weeks to feed an academic paper six months or a year later. For rapid updates they resorted to running a few sensitivities, fitting a system of coupled differential equations to those runs, and simulating sensitivities in that closed-form world. Obviously, someone should have bought them a bigger computer, or lent them time on one.
I guess one useful feature of producing a probability distribution of outputs (for covidsim) is that it shows that the distribution is skewed to bad outcomes with a long tail: there is a substantial risk of outcomes much worse than expected (just like climate change).
As usual, uncertainty is not our friend; when things are uncertain, that tends to strengthen the case for early action.
Sorry angech, but the only viable response is: Hahahahahahaha.
Or perhaps “special military operation”.
@Dave_Geologist Indeed, the question is why we would assume that it is UAH that has the correct set of modelling assumptions? It is almost as if it is the one that shows least warming at the current time that is the reliable one (I’ve noticed that WUWT seem to alternate between RSS and UAH apparently on that basis).
Anders –
Maybe I have a bit of an overreaction because of my impression previously from Eric’s input on the pandemic…but…
I do agree with what I see as one of Eric’s main points – in that I think there is a problematic aspect in how people seek to use models – what I would describe as providing answers rather than helping to circumscribe the uncertainties.
But…
In the end this video feels a bit like a “motivated” piece to me.
I think he somewhat overestimates the level of impact of the modeling on policy development (at least in the US)..- in the sense that he doesn’t sufficiently recognize the full range of influencing factors. comparison to many other influence
I think he somewhat mischaracterizes the “fine-grained and precise terms..” of the “projections – in that the projections suggested fairly wide ranges…
Most significantly, IMO, when he talks about the models not considering “costs” For example, “there’s nothing in the model about what will happen to hospital systems if the healthcare workers’ children are forced to stay home.” Well, ok. But there’s also nothing in the model about what will happen if people are forced to choose between sending their children to school with the risk of them bringing COVID back home, or staying home with their children and getting fired from their job without having an opportunity to collect unemployment.
That’s not to say that I think he’s wrong that those questions don’t matter – just that his considering about what “there’s nothing in the model” to estimate is entirely one-sided. I think he is dealing poorly with the whole counterfactual assumption problem – as if “lockdowns” are associated with associated problems and so are causal to them.
When he discusses that the people who built the models are in the class of people most likely to benefit. Well, ok, but he completely ignores that the people most likely to have suffered absent interventions were the same people who did most poorly with the interventions. As if it’s the interventions that create that disparity rather the other societal factors. And he ignores that generally, the interventions were supported by those people most likely to suffer (either way).
I dunno. I have to say that this video feels pretty standard libertarian/contrarian in that while it raises reasonable issues, it presents a very slanted take, IMO.
Joshua,
Eric and I did have a brief chat about this on Twitter. I think he acknowledges that the modellers couldn’t put everything into their models. I think he’s also pretty openly critical of how Report 9 was presented and the influence of Neil Ferguson.
I agree that some of these type of criticisms don’t themselves necessarily consider the impact of not imposing strong intervention. One potential difference is that the UK did try to support people, even if it wasn’t perfect. This wasn’t the case everywhere. So, I have some sympathy with Eric’s criticisms, but I do think that it’s not as simple as just the modellers not considering these other issues (which I think Eric also acknowledges).
I liked the video, and found Eric’s message powerful. That being said, systematic traders would suggest that we simplify modulz, not complexify them. Speaking of which, I listened to that old episode of Better System Trader. It features Kevin Saunders, whom at one point mentions that his wife went to London to study mathematical modelling for… epidemiologists.
He told that every researcher had a unicorn on their desktop to remind them of the fact that models were abstractions. It’s not a weapon or anything. I just found it funny that an old podcast featured an Aussie systematic trader mentioning models that have become icons years later.
I do not think that models had much influence on the responses to Covid. Different nations followed all possible responses from harsh lock-downs and exclusion (New Zealand, Vietnam) to no response beyond suggesting hydroxyquinone would prevent it (Brasil). Some chose an intermediate path, initiating limits when the healthcare system looked like being overwhelmed.
The death rate reflects this.
Whether it was moral to allow more deaths and greater ‘freedom’ or less deaths but greater societal conformity is largely a matter of political viewpoint.
Unless you think that any death diminishes you.
I foresee the same response to climate change. Nations with a strong social zeitgeist will take effective collective action, those Nations with a more individualistic outlook will do little.
@Dave_Geologist, do you have a link for this interview you mentioned, quote “There was an interview with Nature which went into how they handled the uncertainties. IIRC they weren’t set up for rapid re-runs given their limited computer hardware, and were used to running jobs for weeks to feed an academic paper six months or a year later. For rapid updates they resorted to running a few sensitivities, fitting a system of coupled differential equations to those runs, and simulating sensitivities in that closed-form world. Obviously, someone should have bought them a bigger computer, or lent them time on one.”. I wasn’t able to find it.