This past October, Johan Chu of Northwestern University and James Evans of the University of Chicago published their analysis of bibliometric data drawn from 10 broad fields in science, medicine, engineering, and mathematics.1 Their main conclusion is sobering: As the number of papers in a field increases, researchers find it harder to recognize innovative work. Progress seems to be slowing.
Chu and Evans present a plausible hypothesis. Rather than evaluate new papers on their individual merits, researchers increasingly resort to comparing them with existing paradigms. What’s more, when new papers are published at a high rate, truly novel ideas struggle to prevail—let alone be noticed—over the flood of competitors.
A strength of Chu and Evans’s paper is that they used their hypothesis to make six predictions, all of which they could test by looking for correlations. The predictions are, to quote the paper,
1) new citations will be more likely to cite the most-cited papers rather than less-cited papers; 2) the list of most-cited papers will change little year to year—the canon ossifies; 3) the probability a new paper eventually becomes canon will drop; 4) new papers that do rise into the ranks of those most cited will not do so through gradual, cumulative processes of diffusion; 5) the proportion of newly published papers developing existing scientific ideas will increase and the proportion disrupting existing ideas will decrease; and 6) the probability of a new paper becoming highly disruptive will decline.
Citations are straightforward to count. To characterize a paper’s canonicity, disruptiveness, and diffusibility, Chu and Evans developed statistical measures. In all, they examined 1.8 billion citations of 90 million papers published from 1960 to 2014. Each of their six predictions was affirmed by a significant correlation.
Physics was represented in Chu and Evans’s study by applied physics. Could averaging progress across such a broad field yield spurious correlations? I wondered. At the start of their range, 1960, the laser had just been invented, and transistors had yet to supplant vacuum tubes in electronic devices. At the middle of their range, 1987, high-Tc superconductivity had recently been discovered, and magnetic tape remained a popular storage medium. Research activity in those and other areas of applied physics waxed and waned with fashion, but I couldn’t see how those fluctuations might mimic a secular rise in, say, the canon’s rigidity. It’s also hard to see how correlations that began to be noticeable in the 1970s could be attributed to China’s impressive and more recent growth in scientific output.
One alternative explanation for the correlations could be that the most-cited papers are not the revolutionary paradigm shifters that Chu and Evans presume. In 2014 Richard Van Noorden, Brendan Maher, and Regina Nuzzo published their analysis of the 100 most cited papers in science.2 Top of their list was “Protein measurement with the Folin phenol reagent,” which in 1951 introduced what became known as the Lowry protein assay.3 Like other top papers, it describes a useful, widely applicable method. When the number of scientists increases, so does the number of citations to papers that describe such methods.
But that explanation doesn’t hold up, at least not in physics. The most cited papers in Applied Physics Letters are not method papers. Indeed, one of them helped Shuji Nakamura win a share of the 2014 Nobel Prize in Physics for developing the blue LED.4
If Chu and Evans’s conclusions are valid—and I think they are—then what is to be done? Telling physicists to publish less, publishers to stop launching new journals, and editors to reject more papers are all illiberal restrictions on freedom of expression. But scientists need a better way to evaluate papers’ novelty. Whoever develops one would earn scientists’ gratitude. They might even become rich.
References
