Following the rollout of a controversial policy that makes the promotion of academics contingent on bibliometric success, scholars in Italy increasingly cited their own work and that of other researchers based in the country, according to a new study. The paper, published in PLoS ONE on 11 September, also shows that the country’s citation practices stand out when compared with those of other nations. Although the data cannot be used to determine what caused the increase, the study authors suspect the use of domestic citations is an attempt by Italy-based scholars to game the system.
The 2010 law passed by the Italian parliament requires academics in the country to meet productivity thresholds—calculated using bibliometric measures that include the number of papers and citations—to be eligible for promotions. The policy was designed to advance meritocracy and reduce localism, cronyism, and nepotism.
But some observers have warned that using metric-based approaches in isolation to judge researchers can lead to a spike in questionable behavior. For example, last year academics in Indonesia voiced concerns about manipulation of the country’s researcher ranking system via methods that include excessive self-citations. The new study also comes as the UK finalizes its latest Research Excellence Framework, due to take place in 2021, which uses citation counts and other indicators to assess the quality of academics’ research.
Using the database Scopus, Alberto Baccini, an economist at the University of Siena in Italy, and two colleagues evaluated the citation habits of the authors of more than 32 million papers that were published by academics from 11 nations between 2000 and 2016. To quantify how frequently authors cited their compatriots, the researchers developed a metric called inwardness. They divided the sum of the citations from researchers to their own work or that of other academics in the same country by the total number of citations of research in that country.
In 2008 Italy had an inwardness of 22.4%, a figure that lagged behind that of the US, Japan, and Germany. That year researchers in Italy were just about as likely to cite other researchers in Italy as UK scholars were to cite their compatriots. By 2016 Italy had climbed to the second spot, behind only the US, with 30.7% inwardness. Although all the examined countries saw an increase in inwardness, Italy’s jump was particularly pronounced. And the trend seemed to accelerate around 2010, Baccini says.
The fact that inwardness rose in all the countries isn’t surprising, Baccini says, because researchers are increasingly sharing authorship with collaborators in different countries. When a researcher in France, for example, cites a paper from an international collaboration that includes a French member, it’s considered an inward citation. Yet during the examined 17 years, Italy’s rate of international collaboration was relatively low among the 11 nations in the study; it ranked ahead of only the US and Japan, two countries with far greater research output.
It’s possible that academics in Italy are producing better research and therefore citing themselves more, Baccini says, but he and his colleagues did not find solid evidence to back that claim. He believes it’s more likely that the country’s researchers have changed their citing habits, perhaps to the degree of deliberately gaming the system to help themselves and their friends and forming so-called citation clubs to meet required thresholds. On the basis of the study, Baccini thinks the Italian government’s policy should be scrapped.
The new work follows a 2017 paper that found a rise in self-citations among nearly 900 Italy-based researchers in a handful of fields. Marco Seeber, a sociologist at Ghent University in Belgium who coauthored that analysis, says he agrees that the 2010 policy is likely leading to opportunistic behavior.
Ludo Waltman, an information scientist at Leiden University in the Netherlands who was not involved with either study, agrees. He reproduced the findings of Baccini and colleagues using roughly a million papers from the Web of Science database that were published between 2000 and 2018. “I have obtained essentially the same results,” he says. “I find this disturbing.”
Physics and astronomy, materials science, and Earth and planetary sciences were the only fields in which Italy’s inwardness rise, as measured by percentage points, didn’t outpace other countries’. “This is probably due to the giant international collaborations” in those fields, Baccini says. But related disciplines such as chemical engineering, mathematics, and computer science did see notable jumps in inwardness.
Giorgio Chiarelli, research director of the National Institute of Nuclear Physics in Pisa, Italy, says his institution “does not use bibliometric indices for recruitment or for promotion. We believe that researchers should be judged by their outputs, not by generic figures like number of papers or citations.” He says he is not surprised that the 2010 policy may have changed researcher behavior. “Every measurement changes the measured,” he says.
But Chiarelli notes that the study authors don’t propose an appropriate level of inwardness. And he adds that it’s possible the law has spurred Italy’s academics to pursue better citation practices.
In a statement to Physics Today, representatives from Italy’s National Agency for the Evaluation of Universities and Research Institutes, the governmental body that defines and enforces the bibliometric guidelines, challenge the interpretation of the publication data. Given that inwardness started rising before 2010, one cannot say the trend is definitely a result of the regulations, write Paolo Miccoli and Daniele Livon, the president and the director of the agency, respectively. “At the moment there are not enough elements to support the view of an Italian paradox in terms of self-citations, nor of a perverse effect of new regulation on this issue,” they conclude.
