Skip to Main Content
Skip Nav Destination

New metrics rank physicists and their work

7 June 2018

Inspired by a Google algorithm, the indices factor in multiple generations of citations for published research.

Two CERN theoretical physicists have taken a page out of Google’s playbook in an attempt to better measure the impact of scientists and their published research. In a recent study published on arXiv, Alessandro Strumia and Riccardo Torre propose new metrics that use principles of Google’s PageRank algorithm, which determines the order in which websites appear in search results.

Witten and Dirac
Paul Dirac (right) and Edward Witten have a discussion in an undated photo. The two physicists are rated the most impactful by a new productivity metric. Credit: Emilio Segrè Visual Archives

Productivity metrics, such as the h-index, are increasingly becoming part of research and grant evaluation processes and academic job applications. The physicists say their comprehensive indices improve on these measurements. Outside researchers are intrigued but warn that the new metrics still have transparency issues.

Strumia and Torre are just two of the many academics working to find ways to translate the vast amounts of published work into reliable indicators of researcher impact. “Bibliometric evaluation is important because the number of papers and authors is becoming very large,” Torre says. Critics say that existing metrics are flawed due to their simplicity, and they fear that academics will attempt to game the system to boost their own scores. One way to raise your h-index, for instance, is to cite your own work excessively.

The CERN physicists chose an approach that relies on expanding the definition of a citation. Traditional metrics take into account only direct citations of the study whose impact is being measured; they overlook what Strumia and Torre call second-generation citations—studies that cite studies that in turn cite the original work—and other references in later generations.

Believing that these later citations should be counted, the authors devised PaperRank, which calculates the impact of studies on the basis of all generations of citations. It works similarly to Google’s PageRank, which ranks websites by calculating the number and quality of webpages that link to a particular site. Granted, it takes a while for PaperRank results to become meaningful as those second-, third-, and future-generation papers surface over time, Torre says. On the other hand, because PaperRank depends on all other studies in the literature and not just on one or a few authors’ behavior, it is hard to game. “If someone does something that’s much more important than what you do, your rank will remain small, even if you get many citations,” Torre says.

Of course, not all citations are equal. Researchers want their work to be recognized by respected and influential scientists in their field. To address this, Strumia and Torre created AuthorRank, which rewards researchers who are cited by highly cited authors.

Torre says that in a perfect world, evaluators would read the papers of a job candidate or grant applicant to gauge that person’s impact, but that’s difficult when reviewing hundreds of applications. If any metrics are to be used in such evaluations, Torre says, he hopes they will be his. AuthorRank is especially effective at recognizing the influence of older work, including that of scientists whose major papers came out in the pre-internet era and thus are underrepresented in online databases.

To test their metrics, Strumia and Torre ran AuthorRank and PaperRank on more than a million papers by over 70 000 authors in the online physics database INSPIRE. Paul Dirac is the all-time most impactful physicist, according to AuthorRank. The next three physicists on the list are still alive and working: Edward Witten of the Institute for Advanced Study in Princeton, New Jersey; Nobel laureate Steven Weinberg of the University of Texas at Austin; and Nobel laureate Gerard ’t Hooft of Utrecht University in the Netherlands. By PaperRank’s calculation, Weinberg’s 1967 paper on unifying the weak and electromagnetic forces, “A model of leptons,” is the most impactful work on INSPIRE.

The authors also used PaperRank to rate relatively new papers. Among research published since 2010, a 2012 paper on black hole firewalls, “Black holes: Complementarity or firewalls?” topped the list.

The most impactful physicists of all time, per AuthorRank:

  1. Paul Dirac
  2. Edward Witten
  3. Steven Weinberg
  4. Gerard ’t Hooft
  5. Albert Einstein
  6. Julian Schwinger
  7. Stephen Hawking
  8. Alexander Polyakov
  9. Richard Feynman
  10. Murray Gell-Mann

Researchers familiar with the recent flood of impact-measuring indices have mixed reactions about how PaperRank and AuthorRank stack up. “These metrics would be useful in certain instances like when honoring researchers for lifetime achievement in specific fields like fundamental physics,” says Justin Flatt, a biologist at the University of Helsinki in Finland. “But I would avoid using them when making decisions about hiring, promoting, and funding, especially for early-career researchers.”

Ludo Waltman, deputy director of the Centre for Science and Technology Studies at Leiden University in the Netherlands, finds the idea of using a PageRank-based approach interesting and worth exploring. But due to their complexity, PaperRank and AuthorRank are less transparent than simple citation-based metrics, he says, which means people will be more likely to misunderstand them. And though the indices are difficult to game, they can be skewed by a single highly cited paper, Waltman warns.

A third metric Strumia and Torre suggest, CitationCoin, is calculated by subtracting the number of citations an author gives to another researcher from the number they receive from the same person. The index is designed to weed out the problem of citation cartels—groups of researchers who cite each other superfluously to inflate one another’s citation counts—and excessive self-citations.

But such a metric could turn citations simply into a currency of recognition rather than a vehicle for “linking together concepts, technologies, and progress,” says Flatt, who has proposed another metric to help highlight self-citation practices. “Both self and nonself citations are important to include as long as they result from coordinated, sustained, leading-edge efforts,” he says.

Close Modal

or Create an Account

Close Modal
Close Modal