Introducing multiverse analysis to bibliometrics: The case of team size effects on disruptive research

Do small research teams produce more original scientific contributions than large teams? This thesis was put forward in a widely cited article in Nature (see https://doi.org/10.1038/s41586-019-0941-9) and has important implications for science policy: if small teams do indeed contribute systematically more to the development of new ideas, funding organizations might conclude that smaller research groups should be specifically supported. However, within the bibliometric research community there has been intense debate about whether this finding is robust or whether it depends on which data, metrics, and statistical models are used. Christian Leibel (LMU Munich, Max Planck Society) and Lutz Bornmann (Max Planck Society) take up this debate in a new publication in Quantitative Science Studies.

The study uses the case described above as an occasion to introduce multiverse analysis into bibliometrics. Rather than calculating just a single statistical model, many plausible variants of the same analysis are systematically compared with one another. This makes it possible to check whether a result remains stable or only appears under certain methodological assumptions.

The results show that the central finding of the Nature article is fundamentally robust: small teams score better on average than large teams on the metric used there. This metric attempts to infer from citation patterns whether a publication tends to continue existing lines of research or opens up new ones. Put simply: if later works cite a new publication but hardly mention the older literature that this publication builds on, this is taken as an indication that the new publication has initiated a more independent line of research. According to this logic, higher scores are considered a sign of particularly original or trailblazing research.

The analysis shows that small teams do indeed achieve higher scores on this metric. At the same time, the size of this effect depends heavily on methodological choices. In some models the difference between small and large teams is clearly visible, while in others it is very small. Overall, the effects are too small to derive far-reaching science-policy recommendations from them.

The authors therefore argue that bibliometric findings must be not only statistically significant but also practically relevant and methodologically robust before they can serve as a basis for research policy. The study also shows that multiverse analyses can help make the uncertainty behind bibliometric results more transparent. The paper thus provides not only new insights into the role of team size in science, but also a methodological proposal for a more robust and more comprehensible research evaluation.