oarobin posted a comment highlighting a video of a talk about Science and Skepticism by Steven Goodman. It essentially disusses the issue of reproducibility in science, and mentions some issues that I have myself. Ultimately, science is about unconvering “truth”. Of course, I don’t mean absolute truth, but that we want to tend towards the best possible understanding of whatever it is that is being studied.
One way to do this is to test hypothesis and to then try to reproduce these results. Doing so requires having access to what others have done before. However, this is sometimes interpreted as completely redoing what was done before, often using exactly what was used before. There’s nothing fundamentally wrong with this, but if you get exactly the same result, all you’ve really shown is that they didn’t make a mistake. On the other hand, if you find a mistake, or are able to question something they’ve done, then you potentially have ammunition if your goal is to undermine their results. This could be true, even if the consequences of this issue is negligible.
What I think is more important is to see what happens if you try to test the same hypothesis. Do you get a consistent result, or not. If not, why not? Ultimately, we’re looking for consilience; the idea that multiple lines of evidence, converge towards a strong, consistent, conclusion. This doesn’t mean that every line of evidence has to independently point at exactly this conclusion, or that every line of evidence has to be significant. It simply means that the weight of all the evidence points towards this conclusion.
As usual, I’ve said too much and you really should watch the talk. However, I’ll add one more thing that I found interesting. The speaker did make the point that a number can’t be true or false. What can be true or false is what we infer from an analysis, not the numerical result of the analysis itself. If we get the same results, do we draw the same conclusions? If we don’t get the same numerical results, do we draw inconsistent conclusions, or could what we infer be consistent, even though the numerical results appear not to be?
There’s more that could be said, but I’ll stop there.