REF pushes academics to churn out lower quality research, new study shows

The research team analysed 3,597,272 publications made by UK researchers before and after the deadlines for the Research Excellence Framework (REF), the UK government’s evaluation system for universities, and found worrying data trends that suggest REF timings drive academics to publish more in quantity at the cost of quality.

Many academics face evaluation pressure from their institutions and grant bodies, so regular assessments like REF are used to encourage research activity and allocate funding, which can have major financial and career consequences for universities and researchers. 

Before each REF deadline in the new study (1996, 2001, 2008 and 2014), UK researchers published significantly more papers – and these tended to be in journals with lower impact rankings, where they then generally receive fewer citations and stop receiving them sooner, as well as being more likely to get retracted.

Researchers noticed this trend reversed abruptly after the REF assessments, when UK researchers started producing higher-quality papers in lower numbers. The study also found that publications made after REF showed more variation in quality, indicating that researchers experiment in novel areas more when they’re not as focused on REF deadlines.

Lead author Dr Moqi Groen-Xu, senior lecturer in Queen Mary’s School of Economics and Finance, said: “These patterns are especially worrisome because they do not seem to just set the time for natural research cycles with their ups and downs. The same researchers produce a steadier flow of papers in years that they spend outside the UK.”

While the paper concludes that REF’s shifting incentives are unintentional, the authors also call for change to rebalance the scales and better support long-term exploratory research. Dr Groen-Xu explained: “If you work in a fast-paced field such as computer science, an evaluation every five years may not matter so much in terms of which projects you pursue, or which journals you publish in – but if your projects can take more than five years, the REF can be really disruptive.

“If you give researchers too much time, they operate under less pressure and may slack off, or are reluctant to cut ambitious projects that have not taken off despite investment. If you give them too little time, they may stick to low-hanging fruit: more established research streams, easier journals. It’s unfortunate that designers of cross-field evaluations often forget that research areas differ in where the sweet spot is.”

The full research paper was published online this week in Research Policy, ahead of printing in the journal’s July issue.