Is groundbreaking research disruptive? The covariation of disruption indices with researchers’ self-assessments

Is modern science producing fewer and fewer major breakthroughs? This much-discussed thesis was put forward in a widely noted article in Nature (see https://doi.org/10.1038/s41586-022-05543-x). It was based on a bibliometric measure derived from citation patterns, intended to capture whether a publication displaces older research and opens up new scientific pathways. Christian Leibel (LMU Munich, Max Planck Society), Alexander Tekles (University of Passau), Dag W. Aksnes (NIFU), and Lutz Bornmann (Max Planck Society) examine in a new publication in Quantitative Science Studies whether this measure actually corresponds to what researchers themselves consider to be groundbreaking research.

The study links bibliometric data with a survey of scientists. Respondents assessed their own publications, among other things, according to how important and original they judged them to be. This makes it possible to investigate whether the citation measure identifies those works that researchers themselves regard as particularly significant contributions to science.

The results show that researchers understand groundbreaking research primarily as a combination of high scientific significance and originality. However, the bibliometric measure examined shows little agreement with these assessments. Publications that researchers themselves classify as groundbreaking do not, on average, receive higher scores on the citation measure under investigation. In some cases, they even perform worse than other publications.

The authors conclude that great caution is warranted when such metrics are used to assess scientific breakthroughs. The thesis that modern science is becoming less and less groundbreaking should therefore not be based solely on citation measures whose validity remains unclear. For research evaluation, this means that before bibliometric indicators are used to assess scientific quality or innovative capacity, it must be carefully examined whether they actually measure what they claim to measure.