Several years ago, at a symposium for prize postdoctoral fellows in astronomy, a group of us started reminiscing about graduate school. The conversation inevitably gravitated to the Graduate Record Exam in Physics, or PGRE. Despite most of us having taken the test a good 5–10 years earlier, the whole group collectively shuddered at the mere mention of the test. A round-robin of comments commenced:
“Ugh, my score on that exam was awful.”
“There’s no way you did worse than I did. I’m lucky I got into grad school at all.”
“Please, I’m seriously willing to bet that I have the lowest score of anyone in the room.”
The discussion then died off. Although there is an obvious way to settle that kind of debate, we certainly weren’t about to volunteer our actual scores out loud. Never mind that we had all earned our PhDs and had just spent two days presenting research deemed worthy of a competitive fellowship. A small part of each of us still worried that our peers would judge us for a years-old standardized test score.
Clearly the PGRE leaves an impression on those who take it, even years later. But it’s also become increasingly clear, through research by us and others, that the test is not an effective gauge of future success in astronomy.
That realization is finally starting to lead to changes in graduate admissions. In 2015 every one of the 20 highest-ranked astronomy graduate programs required a PGRE score report as part of a complete application. Over the past year, several schools have dropped that requirement. Just last month Harvard University announced that its astronomy department would no longer require the PGRE. The challenge now is to find alternative evaluation methods that do not suffer from the same weaknesses as the much-maligned test.
Strikes against the test
Test takers sitting down to the PGRE are given 170 minutes to solve 100 multiple-choice questions that are meant to span the full breadth of knowledge covered in an undergraduate physics curriculum. Those 100 questions can dramatically impact students’ careers. Graduate school and fellowship applications are often subject to cutoffs based on the PGRE percentile score (a perfect score earns a 99th percentile). Lower-scoring applicants are either immediately eliminated from the pool or required to compensate with exceptional credentials in other areas. Promising students with low PGRE scores often decide not to apply to top programs—or to graduate school altogether. They don’t feel it’s worth investing time and money in applications that may be tossed straight into the trash on the basis of a single number.
The continued use of the PGRE in graduate admissions is based on the assumption that the test is a useful gauge of applicants’ future success in and beyond graduate school. But there is scant quantitative evidence to support that claim. One problem is that “success” is a nebulous concept. David Payne, who runs the GRE program at the Educational Testing Service, points to a meta-analysis that found that GRE subject tests are good predictors of graduate grade-point average. But a 1996 study by David Morin of the Harvard physics department found that, although PGRE scores did correlate weakly with average grades in graduate coursework, neither metric correlated with other indicators of success in graduate school, such as research productivity.
A far more serious problem with the PGRE is that, like other standardized tests, it is inherently biased. More than two decades’ worth of studies demonstrate strong correlations between test scores and the gender, race, and socioeconomic status of test takers. Documents from the Educational Testing Service confirm those findings. (Payne says that the differences stem from the varying educational experiences of the test takers, and that those with more privileged experiences have a better chance of succeeding in graduate school.)
The exams are also expensive. Last year a student applying to 10 astronomy graduate programs had to pay $312 in registration and score report fees each time that student took the PGRE. And students incurred additional $50 fees each time they rescheduled a test date or opted for a different test center than the one they were automatically assigned. For rural and international test takers, simply getting to a test center introduces significant additional expenses.
A survey and a shift
Our own work on the PGRE began when the three of us met as Hubble and Einstein Fellows. The fellowships award three years of salary and research funding to “outstanding postdoctoral scientists” in astronomy, and fellows present their research at annual symposia.
After seeing so many people at the symposia bemoaning their low scores—people who could hardly be called unsuccessful—we decided to investigate further. We circulated a short, informal, and anonymous questionnaire to 271 people who had held one of five national prize postdoctoral fellowships in astronomy from 2010 to 2015 and asked the poll respondents to report their PGRE scores. The results of the survey are available here.
We found no statistically sound evidence that the PGRE could be used as a reliable predictor of future “success” or as an effective filter in eliminating weak candidates from an applicant pool. Of the 149 prize fellows who responded to our questionnaire and reported scores, 65 (43.6%) would have been rejected from a graduate school applying a strict 60th-percentile cutoff (a threshold reportedly used by some schools) to its applicants.
The gender impact is also disproportionate: The same cutoff would reject 60.4% of all female fellows, and our results showed a clear disparity in gender performance on the PGRE among prize fellows. Even a 10th-percentile cutoff would eliminate multiple prize fellows of all genders. The PGRE score also showed no correlation with the number of first-author papers written during graduate school.
After considering our work along with other studies of standardized tests and their use in admissions, the American Astronomical Society (AAS) released a statement in January 2016 recommending that all graduate programs in astronomy eliminate the PGRE as an admissions requirement. In the first year after the AAS recommendation, the astronomy departments at the University of Washington and the University of Arizona dropped the PGRE requirement; they now list the test as optional. Harvard joined that group last month. The University of Texas at Austin and New Mexico State University went a step further, prohibiting applicants from submitting PGRE scores. [Update, 2 October 2019: Many more schools have moved toward minimizing the role of the PGRE in the application process; see this document for a complete list of schools’ policies.]
Replacing the test
Additional astronomy programs are also considering dropping the PGRE, but questions persist about what this will mean for graduate admissions going forward. Although the PGRE is a demonstrably ineffectual and biased means of evaluating applicants, removing it puts more weight on other application components that are also far from perfect. Recommendation letters display a known gender and racial bias, written essays put nonnative English speakers at a disadvantage, and even resumés and transcripts can be more reflective of an applicant’s undergraduate institution and socioeconomic status than of the candidate’s skills and potential.
There’s also the question of whether astronomy departments should refuse to consider the PGRE. The astronomy department website at UT Austin explicitly states that PGRE scores will not be considered even if students voluntarily submit them. The school concluded that “optional reporting would send mixed messages to the admissions committee on the value of the PGRE as a success metric, making the admission procedure more dependent on individual bias,” says Caitlin Casey, a professor in the department. But some professors who are familiar with admissions argue that a high PGRE score can still, in some cases, help strengthen an application. They point to the hypothetical example of students at small liberal arts colleges who may use exam scores to demonstrate the extent of their physics ability.
The ultimate aim is to develop a more effective and equitable graduate admissions process. Some programs employ Skype interviews for a “long list” of applicants, giving prospective students a different means of demonstrating excellence and potential. The Fisk–Vanderbilt Master’s-to-PhD Bridge Program has assembled a repository of interview protocols and assessment rubrics that committees can use to evaluate “noncognitive attributes” such as perseverance and leadership. Finally, simply educating graduate admissions committees about common sources of inequity and bias in application materials can help.
Although the future of graduate admissions in astronomy is still in flux, the current situation is clear: Reliance on the PGRE is on its way out.
Editor’s note, 2 October 2019: This article has been updated to account for recent changes in astronomy department policies. A sentence about the University of California, Santa Cruz’s policy has been removed—submitting the PGRE is optional for admission to the school’s astronomy and astrophysics department, and there is no score threshold for consideration. See here for a complete list of admissions fees and physics GRE requirements for astronomy and physics programs in the US and Canada.
Emily M. Levesque is an assistant professor of astronomy at the University of Washington who studies massive star populations, their evolution, and their explosive deaths as supernovae. Rachel Bezanson is an observational astronomer at Princeton University whose expertise is in massive galaxy formation and evolution; she will join the faculty at the University of Pittsburgh as an assistant professor of physics and astronomy later this year. Grant R. Tremblay is an Einstein Fellow at Yale University who studies supermassive black holes and the evolution of galaxies in which they reside; he will become an astrophysicist at the Harvard–Smithsonian Center for Astrophysics later this year.
