Unless you’re a UK-based academic, you may not know that yesterday was an important day in UK academia; it was the release of the results from the 2014 Research Excellence Framework (REF) exercise. This is an assessment of most of the academic departments at all UK universities, that takes place about every 7 years. It involves around 300 senior academics spending about a year reading hundreds of papers each, so as to produce an overall score for each department; a result that could probably be roughly reproduced in a few hours using any of the available research databases. Amongst many others reasons, that one’s of the issues that I have with the whole exercise. As pointed out in this Guardian article
‘Excellence’ tells us nothing about how important the science is and everything about who decides.
Essentially we end up quantifying excellence using metrics that the system itself has decided are some kind of measure of quality, but may be no such thing.
The problem as I see it is that the desire is to quantify something that is probably unquantifiable, and so what happens is that it’s done using simplistic metrics that may not be a particularly good indicator of excellence, but are more an indicator of what is regarded as interesting – by the system itself – at that time. There are certainly people who publish lots of papers and get lots of citations, but don’t do particularly excellent work. There are others who publish less and don’t get lots of citations, but do work of a very high quality. You may think that the latter may not be worthwhile if it isn’t noticed, but sometimes you need to do high-quality work to make that crucial incremental step that allows others to solve the next big problem.
Also, there are impacts that are even harder to quantify. Did some work capture the public’s imagination and encourage an interest in science and the humanities? Did some research lead to the development of some new technology? Did some research simply satisfy our desire to better understand the world around us? Will something we do today play a small, but important role, in something amazing that will happen in 50 years time? Some things just aren’t really quantifiable and I think sometimes we should be careful of trying to do so.
Another problem with such an assessment exercise is that it involves so much money, and doing well makes such a large difference to a university’s finances, that you start being encouraged to think about how your work could score well in such an exercise, rather than thinking about what interesting problem you could be trying to solve. Not only does it reward those who just happen to score well in such an exercise, it encourages everyone else to try and do the same. In some sense, it risks reducing diversity. Of course, some might argue that that’s okay because it will all be excellent, but that’s only because we’ve defined excellence in that way, not because it is excellent in some intrinsic sense. We also run the risk of missing out on potentially excellent research that isn’t done because it might not score well on an assessment exercise.
Additionally, it runs the risk of focusing resources in an ever decreasing number of institutions. If you do very well you get a lot more money per person than if you do poorly. Hence those that do well in one exercise could do even better the next time, and could collect an even bigger fraction of the available funding. If it was truly a good measure of excellence, maybe this would be fine, but it probably isn’t. We risk damaging pockets of excellence in small universities if we make the difference between doing well and doing poorly too great.
There are even other issues with the whole process. It’s research only. Universities are also places where we teach and it’s certainly my view that the biggest impact I’ll have in the short (and maybe medium) term is that I will be involved in teaching people who will hopefully benefit from their education, and who will hopefully contribute to society as a result of their education. There’s no incentive to reward good teaching because doing so doesn’t actually make any immediate difference to a university’s finances. Someone who can publish papers that will score well on an assessment exercise is, however, worth rewarding even if their teaching is poor.
Anyway, that’s my little rant over. We actually did quite well in REF2014, so this isn’t some kind of sour grapes. I just think that universities are important and complex organisations, and that trying to quantify their value in some simplistic way could end up doing more harm than good. I don’t think that we should be funding things that don’t have any value, but I also don’t think there’s much point in justifying our funding decisions using a complicated and time-consuming process that probably doesn’t really have the ability to judge intrinsic value. I don’t really have a solution though, but – fortunately – it’s not my job to find one.