A short web page at the Journal of Mathematical Physics highlights that publication’s journal impact factor (JIF). But at the blog Nanoscale Views last month, physicist Douglas Natelson found a vivid way to illustrate JIFs’ inherently limited usefulness in assessing research. Invoking the “Moneyball” approach, he contrasted JIFs with the well-demonstrated usefulness of statistics in assessing performance in data-rich baseball. In the scientific press this summer, skepticism about JIFs has been rising, with news and commentary engaging the challenging search for better assessment methods.
The discussion catalyst is the bioRxiv paper “A simple proposal for the publication of journal citation distributions.” Its authors include geophysicist and National Academy of Sciences president Marcia McNutt, formerly editor-in-chief of Science. Other authors represent Science, eLife, the EMBO Journal, the Journal of Informetrics, the Proceedings of the Royal Society B, three Public Library of Science journals, Nature, and two other Nature publications.
Early on, the paper lists a JIF’s “main deficiencies”:
[It] is calculated inappropriately as the arithmetic mean of a highly skewed distribution of citations; it contains no measure of the spread of the distribution; it obscures the high degree of overlap between the citation distributions of most journals; it is not reproducible and the data that support it are not publicly available; it is quoted to a higher level of precision (three decimal places) than is warranted by the underlying data; it is based on a narrow two-year time window that is inappropriate for many disciplines and takes no account of the large variation in citation levels across disciplines; it includes citations to “non-citable” items and citations to primary research paper are conflated with citations to reviews—it is therefore open to gaming and subject to negotiation with Thomson Reuters [the organization that calculates and publishes JIFs]; its relationship with citations received by individual papers is questionable and weakening.
The paper presents what it calls “a simple method for generating the citation distributions that underlie JIFs.” It declares, “Application of this straightforward protocol reveals the full extent of the skew of distributions and variation in citations received by published papers that is characteristic of all scientific journals.” It proposes “that this methodology be adopted by all journals as a move to greater transparency” in order to help “refocus attention on individual pieces of work and counter the inappropriate usage of JIFs.”
A 28 July Nature editorial exemplified the harshness of some of the criticism of JIFs. The editors opened by charging that reliance on such an “intrinsically reductive” metric “as a yardstick of performance, rather than as a pointer to underlying achievements and challenges, usually leads to pathological behaviour.” Calling the JIF “crude and also misleading,” they added that it “effectively undervalues papers in disciplines that are slow-burning or have lower characteristic citation rates” and that as “an arithmetic mean, it gives disproportionate significance to a few very highly cited papers, and it falsely implies that papers with only a few citations are relatively unimportant.” The editors described what they see as the “pernicious” practices of “a metrics-obsessed” scientific research culture where mistakenly overvalued journal impact factors can cause a good researcher to be unfairly—and unwisely—overlooked, underappreciated, or ruled out.
Something like harshness showed in the very headlines of July news reports at Science, Nature, and The Scientist, respectively:
- “Hate journal impact factors? New study gives you one more reason.”
- “Beat it, impact factor! Publishing elite turns against controversial metric: Senior staff at leading journals want to end inappropriate use of the measure.”
- “Ditching impact factors for deeper data.”
In a Public Library of Science blog post, three of the bioRxiv paper’s authors expressed the “hope to strengthen a call for action” originally voiced 14 months ago by another author, structural biologist Stephen Curry of Imperial College London. That call lamented the widely known “invidious effects of the impact factor,” which Curry charged “has burrowed deep into the soul of science and is proving hard to shift.” He reported that a Royal Society meeting had “repeatedly circled back to the mis-application of impact factors as the perverse incentive that is at the root of problems with the evaluation of science and scientists, with reproducibility, with scientific fraud, and with the speed and cost of publishing research results.” He also noted that the meeting “discovered once again—unfortunately—that there are no revolutionary solutions to be had.”
Curry ended with a mildly impassioned litany of actions concerned citizens of science can take upon encountering “bragging about this or that journal’s impact factor”:
Ask them why they think it is appropriate to reduce their journal to a single number, when they could be transparent about the full range of citations that their papers attract. Ask them why they are not showing the data that they rightly insist authors provide to back up the scientific claims in the papers they publish. Ask them why they won’t show the broader picture of journal performance. Ask them to help address the problem of perverse incentives in scientific publishing.
Still, not every voice opposes JIFs. Nature’s news article quoted Ludo Waltman, a bibliometrics researcher at Leiden University in the Netherlands, concerning the task of selecting research papers to read: “Denying the value of impact factors in this situation essentially means that we deny the value of the entire journal publishing system and of all the work done by journal editors and peer reviewers to carry out quality control. To me, this doesn’t make sense.”
In a 21 July posting at the Scholarly Kitchen, blog founder Kent Anderson warned that the approach envisioned by the bioRxiv paper could yield, in “a hundred or thousand different editorial offices,” a “baffling array of charts—different and irreconcilable colors, labels, axes, and legends.” He emphasized that “a distribution is unlikely to shift the psychology of authors or tenure committees.” He predicted, “Getting the data for these distributions would add work and expense to a system already dealing with the workload and complexity of supporting a lot of steps and processes.”
In Nature’s 28 July editorial, under the headline “Time to remodel the journal impact factor,” editors stipulated that they could offer no overall solution. But they did propose two countermeasures:
- To keep the focus “properly” on candidates rather than on journals, deciders about jobs, promotions, or funding should ask applicants for short summaries of their achievements.
- Journals should “be more diverse in how they display their performance,” using wider arrays of bibliometric data—as Nature itself is now doing, the editors noted.
The editorial ended with this: “However, whether you are assessing journals or researchers, nothing beats reading the papers and forming your own opinion.”
That thought recurs in the coverage. Retraction Watch founders Ivan Oransky and Adam Marcus, for instance, declared that the “best way to judge quality ... is quite old-fashioned. Throw away the CliffsNotes and read the paper.”
---
Steven T. Corneliussen, a media analyst for the American Institute of Physics, monitors three national newspapers, the weeklies Nature and Science, and occasionally other publications. He has published op-eds in the Washington Post and other newspapers, has written for NASA's history program, and was a science writer at a particle-accelerator laboratory.
