Shock and anger have pervaded Portugal’s scientific community since early this summer, when the results of an evaluation organized by the Science and Technology Foundation (FCT) threatened nearly half of the country’s research centers with death by financial starvation. The evaluation was the first of a two-part look at more than 320 research centers across all areas of science and humanities. A second phase is under way, with final grades expected at the end of the year; the grades determine core lab funding for the six-year period 2015–20.
Although it’s widely accepted that funding is tied to such evaluations, researchers say that the current process is unfair. Among other complaints, they say that not all sites were visited, the process didn’t permit fair comparisons across research centers, the expertise of the more than 650 outside evaluators represented an inadequate range of subdisciplines, there were irregularities in scoring, and the results are skewed geographically. “This evaluation is flawed,” says Carlos Fiolhais, a physicist at the University of Coimbra. “We are asking for a suspension of this process and a reanalysis.”
Research centers in Portugal, including at the 700-year-old University of Coimbra, could wither and die if the results of an evaluation are not reversed. Most of the physics labs that scored well in the first part of the evaluation are in Lisbon.
Research centers in Portugal, including at the 700-year-old University of Coimbra, could wither and die if the results of an evaluation are not reversed. Most of the physics labs that scored well in the first part of the evaluation are in Lisbon.
The uproar is making national headlines as university rectors, the Portuguese Physical Society and other scholarly societies, and major laboratories protest the process. The complaints reached fever pitch in mid-July, when reports emerged that the FCT had specified from the start that only half the research groups should score well enough to proceed to the second phase of the evaluation. The FCT, the main government funding agency, denies that it made any such specification.
Scoring the evaluation
The research centers, or “units,” being evaluated can be based at a single university or research institution, or have members from multiple institutions. The units range in size from a handful of researchers to hundreds. The first phase of the evaluation was outsourced to the European Science Foundation (ESF) and used bibliometrics, performance reports, and future plans.
The 52% of all research units that received a score of “very good,” “excellent,” or “outstanding” move on to the second phase of the evaluation, which involves site visits. A score of “poor” or “fair” (22% of all units) zeroes out the base funding from the FCT; a unit with a “good” rating (26%) is eligible for a maximum of €40 000 ($54 000) per year and may receive as little as 10% of what it was getting previously. Over the past few years, the FCT’s annual budget for core laboratory costs has been €50 million. A foundation spokesperson says it will remain at that level.
As Physics Today went to press, appeals filed by some 80% of research units that did not make the first cut were being considered.
Not everyone is upset. Antonio Heitor Reis, director of the University of Évora’s geophysics center, which did well in the evaluation, says that despite the criticism, “there is a positive aspect. In previous evaluations, the FCT invited external panels. This is the first time that Portuguese units have been evaluated by an independent external entity. In the abstract, this is a good idea.” And not surprisingly, Nicolas Walter, who supervised the process for the ESF, says the ESF “strongly refutes” the allegations of unfairness.
Still, even people from the more successful research units question the evaluation’s fairness and its implications; they fear that in the second phase their own labs could still drop below the threshold score for receiving funding. Says José Mendonça, president of the 800-member information technology lab INESC-TEC, which did well in the first phase of the evaluation, “People worry that the same problems will happen.” The site visits planned for the second phase are limited to three hours, he says, and “people fear it will be impossible to show reviewers the labs.”
Adding to the doubts about the fairness of the evaluation is that the physics units that made it to the second phase are overwhelmingly in the capital city of Lisbon. “Other important cities like Oporto, Coimbra, and Braga will be affected. They have only 3% of the researchers belonging to successful physics research units,” says Nuno Peres, director of the center for physics at the University of Minho in Braga. “There is a sense that the grades are like the flip of a coin.”
Longtime efforts at risk?
Since the collapse of its dictatorship 40 years ago, and particularly over the past 25 to 30 years, Portugal has invested steadily in science and education. The country doubled the percentage of its gross domestic product that it puts into research and development, from 0.7% in the early 1990s to 1.5% today; the average for Europe is 2.1%. The country attracts foreign students, postdocs, and more-senior researchers, and sends scientists abroad. It also participates in international research organizations such as CERN, the European Southern Observatory, and the European Space Agency.
“Portugal was joining the international community. There was brain circulation,” says Alexandre Quintanilha, secretary of the Council of Associated Laboratories and a biophysicist at the University of Oporto. “Now there is a general feeling of frustration in physics and other areas in which we have worked hard to build the necessary critical mass.” After decades of investment, he says, “this is a dramatic alteration to a policy that was working, producing results, and now half of the research centers of Portugal will just have no possibility of surviving.”
Adding to the concerns are other recent moves by the FCT, including a significant reduction earlier this year in the number of PhD and postdoctoral fellowships that it administers; some were cut and some were moved to specific universities or laboratories. And although the current evaluation process doesn’t directly affect universities, if faculty cannot do research then the universities will become less attractive to faculty and to students. “Young Portuguese students who are finishing their degrees are now wondering if there is a future for them,” says Quintanilha.
Peres, one of Portugal’s most cited scientists, was stunned when his center received a grade of “good.” In previous evaluations, and even just weeks earlier, the FCT had rated the center excellent. Unless the center’s appeal is successful, its annual base funding stands to plunge nearly 90%, from €380 000 to €40 000. “We will need to be more imaginative and increase our sources of funding,” says Peres. “We already have projects with national and European funding, so it’s difficult to see how to improve our status.”
The FCT funding covers laboratory running costs—electricity, chemicals and other consumables, travel, occasionally a student or technician—that are not typically covered by project grants. The grades from the evaluation, says Mendonça, “will be very important, not just for financing from the FCT, but also for the image and credibility of an institute.” Research units that are classified as excellent or very good “will have more chances to get postdocs, students, grants, et cetera, than ones with a more modest classification. Those won’t have much opportunity. Their destiny will be determined,” he says.
“If this does not finish well, the efforts of 20 years may be at stake,” says Mendonça. Moreover, he adds, the country will feel the economic impact in the coming years, “since a number of research laboratories have sustainable, fruitful partnerships with the leading Portuguese industrial companies.”
Sustaining competitiveness
So why is the FCT targeting half the research units? The simple explanation, says José Paixão, vice president of the Portuguese Physical Society and a solid-state physicist at the University of Coimbra, is a lack of money and a reduction in investment in response to the country’s financial crisis. But the FCT says that despite the country’s financial straits, funding for science is not decreasing. “If the money hasn’t gone down, then where is it?” asks Quintanilha. “That is a mystery.”
The official aim of the evaluation is to promote excellence, says Paixão. The thinking, he says, is that science has been boosted across the board, “and now it is time to promote excellence above all.” But, he says, “I am not sure we are at the stage of promoting only excellence. We have to look at good science, and to give it a chance to blossom.”
For its part, the FCT calls the evaluation procedures “robust” and “thorough.” A communications officer at the foundation says the process aims at “strengthening the role of research units in sustaining a modern and internationally competitive R&D system.” And on 26 July the ministry of science and education announced the creation of a €6 million fund to help research units with poor ratings improve their chances in the next review. An interim evaluation is slated for 2017.
