There’s a short comment by Andrea Saltelli in Nature Communications on Statistical versus Mathematical Modelling. The general premise is that, like statistics, there is also a crisis in mathematical modelling. However, there isn’t the same sense of crisis about mathematical modelling as there is about statistical modelling. He thinks something can be learned by comparing the two and that [s]ociology of quantification and post-normal science can help.
I do, however, have a number of issue with the article. Firstly, I think it over-simplifies, or possibly even misrepresents, the crisis in statistics. It’s not really a crisis in statistics, it’s a problem with how some use statistics. It’s also not a problem that exists in all research areas; it’s predominantly in areas that have relied on null-hypothesis testing. There are – as far as I’m aware – many research areas where this really hasn’t been a major problem.
The other problem I have is that it seems to treat mathematical modelling as if it’s homogeneous, both in terms of how the models are developed and in terms of the model relevance. However, there is a vast difference between physical models and economic models. Mathematical models are also used for many different reasons. It can vary from theoretical studies aimed at understanding some physical system, to models used to interpret some dataset, through to models used to explicitly inform decision making. There may well be cases where the limitations of a model is not made clear, but this doesn’t imply some kind of general crisis in mathematical modelling.
This is not to say that I don’t think there aren’t any problems. There are certainly cases where people use models in ways that aren’t suitable, or use the results of a model without understanding the limitations of that model, or the significance of the assumptions that were used. I think it would be useful if there were a better understanding of the different types of models, the strengths and limitations of the various types of models, and how we should probably be utilising model results when informing decision making. I don’t, however, think that generalising about mathematical models is particularly helpful.
I also find these kind of commentaries somewhat ironic. In many cases, the problem isn’t really with the statistical method, or with the mathematical model, but with how it’s being used, or how it’s being presented. In my view, it’s important to understand when one can use a satistical method/mathematical model, be clear about the assumptions used, and be clear about the limitations and strengths of the model/method. Yet, this seems to be essentially what these commentaries lack; they present simplistic generalisations, aren’t particularly careful about their terminology, and don’t seem to be clear about the strengths and limitations of what they’re suggesting.
If there is a group of researchers who think that they’re in a position to critique the research conduct of others, you would hope that their own research satisfied this ideal. My impression, however, is that it doesn’t. If anything, I’m not even sure that some of this even really qualifies as research; it just seems to be someone’s opinion about a topic that they don’t even seem to understand all that well. I don’t think there is anything wrong with critiquing the research done by others, but doing so doesn’t mean that one is somehow immune from criticism.