There’s a recent Nature comment lead by Andrea Saltelli called Five ways to ensure that models serve society: a manifesto. Gavin Schmnidt has already posted a Twitter thread about it. I largerly agree with Gavin’s points and thought I would expand on this a bit here.
The manifesto makes some perfectly reasonable suggestions. We should be honest about model assumptions. We should acknowledge that there are almost certainly some unknown factors that models might not capture. We should be careful of suggesting that model results are more accurate, and precise, than is actually warranted. We should be careful of thinking that a complex model is somehow better than a simple model. Essentially, we should be completely open and honest about a model’s strengths and weaknesses.
However, the manifesto has some rather odd suggestions and comes across as being written by people who’ve never really done any modelling. For example, it says
Modellers must not be permitted to project more certainty than their models deserve; and politicians must not be allowed to offload accountability to models of their choosing.
How can the above possibly be implemented? Who would get to decide if a modeller projected more certainty than their model deserved and what would happen if they were deemed to have done so? Similarly, how would we prevent politicians from offloading accountability to models of their choosing? It’s not that I disagree with the basic idea; I just don’t see how it’s possible to realistically enforce it.
The manifesto also discusses global uncertainty and sensitivity analyses, and says
Anyone turning to a model for insight should demand that such analyses be conducted, and their results be described adequately and made accessible.
Certainly a worthwhile aspiration, but it can be completely unrealistic in practice. If researchers get better resources, they often use this to improve the model. A consequence of this is typically that there is then a limit to how fully one can explore the parameter space. A researcher can, of course, choose to make a model simpler so that it is possible to do a global uncertainty and sensitivity analysis, but this may require leaving out things that might be regarded as imortant, or reducing the model resolution. This is a judgement that modellers need to make; do they focus on updating the model now that the available resources allow for this, or do they focus on doing global uncertainty and sensitivity analyses? There isn’t always a simple answer to this.
We could, of course, insist that policy makers only consider results from models that have undergone a full uncertainty and sensitivity analysis. The problem I can see here is that if policy makers ignore a model for this reason, and it turns out that maybe they should have considered it, I don’t think the public will be particularly satisfied with “but it hadn’t undergone a full uncertainty and sensitivity analysis” as a justification for this decision.
I don’t disagree with the basic suggestions in the manifesto, but I do think that some of what they propose just doesn’t really make sense. Also, the bottom line seems to be that modellers should be completely open and honest about their models and should be upfront about their model’s strengths and weaknesses. Absolutely. However, this shouldn’t just apply to modellers, it should really apply to anyone who is in a position where they’re providing information that may be used to make societally relevant decisions. I don’t think hubris is something that only afflicts modellers.