Fixing the PhD qualifying exam

Figure 1 provides a visual summary of the new process and its place within the overall PhD track. It centers on a semester-long course typically taken in the spring of a student’s second year. In the course, the student works with their adviser to formulate a project that will lead to a research paper.

The student presents their paper idea in two meetings. In the first, the student delivers their proposal during the first 15 minutes, followed by a 75-minute period during which panel members pose critical questions about it, as shown in figure 2. Half the department’s faculty members are present in that meeting. About four weeks later, the student presents to the other half of the faculty. The two faculty panels function independently without communicating with the other. The autonomy provides the student with two independent opportunities to present their work at their best, free from biases influenced by the previous performance.

After each meeting, the student receives written feedback from each panel member on various aspects of their paper idea. That feedback includes an assessment of the student’s grasp of the relevant literature, their physical understanding of the scientific problem, their ability to perform quantitative analysis, and their effectiveness as a communicator. Panel members evaluate each of the categories, but they don’t assign a grade. The feedback is intended to be constructive—to help the student identify areas in which they need improvement.

By the end of the semester, the student submits either a manuscript that is nearly ready for submission to a peer-reviewed journal or, if the research is not yet completed, a proposal for a scientific paper that incorporates their original research. All panel members read the student’s document. On the final day of the course, the student gives a longer oral presentation to the entire faculty. The faculty members then discuss and make recommendations to the program director.

The new qualifying process has several advantages over the traditional format. First, instead of assessing a student’s knowledge, faculty members evaluate the student’s ability to perform the activities critical to scientific inquiry: identifying a scientific problem, devising solutions, and engaging in discourse. Second, the process spans an entire semester, so decisions on a student’s performance are not based on a singular moment. Third, each student chooses their own research topic, affording them the opportunity to showcase their creativity.

Furthermore, a student receives questions tailored to their chosen topic. That focus avoids the pitfall of evaluating them on their response to questions far removed from their prepared area. Although confronting a student with unanticipated questions can have merits, using that approach as a sole determinant of the student’s future is risky. A semester-long engagement provides a more reliable assessment of potential.

The new qualifying process also offers each student multiple opportunities to succeed. Because the two faculty panels are independent, a student who struggles in the first meeting can present a revised version of their work to a fresh audience. Some students might face special challenges in the oral presentation, such as stage fright or language deficiencies. To address that issue, the new process includes a written submission as an additional means to demonstrate the student’s abilities.

Grading in the qualifying process does not rely on averaging individual scores. The primary goal is to identify evidence of the student’s research capabilities. The evidence may not be uniformly apparent across all components, but outstanding performance in a single aspect can eclipse weaker performance in other areas. Additionally, we consider the student’s progress throughout the semester because improvement is often a key indicator of potential success in research.

Another advantage is that a student has ample opportunity to revise their work. Even the best student may not fully explore their ideas initially. Our semester-long process mimics the peer-review process and typically exposes any serious shortcoming that may exist in a student’s proposal. Observing how students adapt to constructive feedback often provides more insight into their potential than does reviewing their initial proposals.

Unavoidably, subjective judgments affect the final decision, and they may be influenced by biases tied to race, gender, sexual orientation, or disability. Even the traditional format, with its fact-based questions, involves subjective judgments in deciding the acceptability of a student’s response. One way to counter biases is to involve a diverse panel of judges whenever subjective judgments come into play. In the traditional format, the decision often rests with a select few faculty members. In the new format, the entire faculty openly participate in the decision-making process, which brings a wider range of perspectives into the discussion.

By distributing responsibility across all faculty members, the new process also lightens the burden on individual advisers, who often hesitate to single out their own struggling students. When a student is redirected, their adviser usually appreciates the collective intervention.

Although the new format requires a greater investment of time from faculty, productive scientists are accustomed to allocating time for conferences and peer-review duties. And the new qualifying process calls for minimal preparation by faculty, with only modest tasks required post-meeting, such as filling out evaluation forms. When it comes to peer-review services, the question is how to best manage one’s time reviewing others’ work. Allocating a portion of that time to assisting students in one’s own department proves to be a sound investment in upholding the quality and integrity of the qualifying process. Ultimately, the efforts produce better student outcomes, which, in turn, cast a positive light on the faculty and the department.

Fellow scientists who hear about our qualifying process are often doubtful about its feasibility in their own departments. They cite factors such as a large student population. We are confident, however, that the new process can be tailored to any department. Our PhD program at George Mason has a dozen faculty members and admits three to six candidates per year. For larger departments, splitting students and faculty into smaller cohorts operating in parallel is a feasible solution.

Another concern has been the perceived inefficiency of involving faculty who lack expertise in a student’s chosen topic. But we have found the opposite to be true: Observing how the student articulates their research to nonspecialists, who nonetheless possess broad scientific knowledge, has several advantages. Incorporating diverse expertise in faculty panels, for instance, ensures that a mix of technical and foundational questions will be addressed, which makes the evaluation more thorough.

The new process also encourages faculty to engage with each student out of genuine interest, thus fostering a less adversarial interaction than the traditional approach. The reversal of the conventional roles of teacher and student mirrors what a student will encounter in advanced doctoral research. Furthermore, the shift in dynamic creates opportunities for a student to demonstrate creativity in handling conflicting criticisms that arise from reviewers with different knowledge backgrounds.

One issue that has generated considerable debate among our faculty is the grading policy. Currently, a student who passes the qualifying course receives either an A or a B. The A grade, however, is reserved for students who submit a manuscript that the faculty believes can be refined into a publishable paper after a few months of revision. That’s a high standard, and not all exceptional students meet it.

We believe that a significant distinction exists between a student who develops a nearly publishable paper in their second year and one who does not, and the grade assigned to each one is intended to reflect and reward that difference. Moreover, the standard is attainable: One or more students achieve it each year.

Most second-year graduate students find the prospect of formulating and defending a publishable scientific analysis in a single semester daunting. Indeed, many students have never presented their own research in front of a group of scientists. To address the issue, we have implemented support mechanisms to assist each student throughout the qualifying process.

First, the student works with their adviser to formulate an idea that will be integrated into their dissertation. If the student is supported by a grant, they are encouraged to select a topic related to that grant, but their contributions must be independent. The new format provides opportunities for the student to innovate while still benefiting from their adviser’s guidance.

Advisers must avoid overdoing their guidance; otherwise, the process becomes an evaluation of the adviser instead of the student. Our tenet is that the process should not disrupt the natural interaction between student and adviser. Reasonable guidance includes suggesting research topics, offering feedback on presentations and written materials, assisting in problem diagnosis, and helping the student devise strategies for solutions.

Beyond that, it is left to the student to use the information they receive. The adviser should avoid writing code on behalf of the student or producing text that could be copied into the student’s written submission. The student is expected to defend their ideas without assistance from their adviser. In addition, we advise each student to reduce their course load or take a reading course during the qualifying process to allow more time for conducting independent research.

The two of us currently lead the qualifying course, guiding students through the process. We listen to practice talks prior to panel meetings and offer guidance on delivering effective presentations. Students consistently underestimate the level of detail necessary to communicate their research plan effectively, and some are unwittingly too dependent on their advisers to address basic questions related to their project. We strive to inspire students to take ownership of their work and to thoroughly understand the models and data that they use. Practice talks can expose potential research flaws early enough for students to make adjustments before their first panel meeting. For a point-by-point comparison of the traditional exam’s shortcomings and the advantages of the new process, see figure 3.

We also meet with each student after their panel meeting to discuss written feedback from the faculty. That feedback resembles the kind that might be encountered during a genuine peer-review process, encompassing not only constructive feedback but also potential contradictions. As educators, we believe that it is crucial not to shield students from that reality. Instead, we strive to expose students to diverse perspectives and help them interpret the resulting feedback constructively.

Sixteen weeks is hardly enough time to complete a serious research project. Accordingly, we encourage every PhD student to begin preparing for the qualifying process as soon as they enter our program. Any research conducted by a student during their time in the graduate program is permitted for use in the qualifying process. And the qualifying process is explained in detail to PhD students at the end of their first spring semester. The early introduction helps instill a productive mindset in each student as they approach their first summer in the program—a period devoid of course distractions that allows them to focus wholeheartedly on their research goals.

As advisers, we try to strike a balance between providing assistance and giving students space to develop their own thinking. To do that effectively, we’ve adopted a strategy inspired by what’s known as the Heilmeier catechism. George Heilmeier, who led the Defense Advanced Research Projects Agency in the mid 1970s, crafted a series of questions that every good proposal should answer.¹ The list distills years of wisdom into a concise question set. We have adapted it to create the following questions that every research project should address:

• What are you trying to do? State your objectives without jargon.

• Who should care? If you are successful, what difference will it make?

• What research has been done about the topic in the past?

• What is the precise gap that you are trying to fill?

• What is new in your approach?

• Why do you think your approach will be successful, and how will you measure success?

The simplicity of the questions can deceive students into underestimating the effort needed to answer them effectively. To ensure the development of thorough answers, students write an abstract for their projects early in the semester. Those abstracts are then shared with the class, sparking discussions about best practices and common pitfalls when communicating scientific ideas. Invariably, some abstracts fail to address one or more key questions. Some students are convinced that they have responded adequately to a question until further discussion reveals gaps in their explanation. The value of carefully addressing the questions often hits home during the discussions.

Many advanced students begin the semester with a well-defined paper plan but are surprised by the challenge of communicating their plan to others. The situation reflects a fundamental reality: Success as a scientist relies on communication as well as critical thinking. Proficient communication skills are vital for today’s PhD graduates, whether it’s for securing funds, responding to peer review, teaching, or mentoring. Although research and communication skills are often regarded as distinct, we find much truth in the adage that poor communication may reflect poor thinking.

The final decision of whether a student passes the qualifying process is a collective one. Although faculty opinions may vary initially, discussions usually lead to a consensus. Students who do not pass typically display one or more common traits: an inability to demonstrate quantitative analysis; a failure to understand the relevant literature; insufficient familiarity with the data or model chosen for study; an inability to answer questions related to calculations, models, or assumptions; an inability to articulate how the proposed research addresses key questions; difficulty in communicating the research plan; and an inability to understand or convey the relevant concepts.

A student at risk of failing is typically alerted after the panel meetings. That early notification gives them time to make improvements and address the concerns. Consequently, a negative outcome is rarely a surprise to the student. Some students have voluntarily withdrawn from the qualifying process during the semester after recognizing that they were unlikely to meet the necessary requirements. The self-selection process allows students to make informed choices about their academic path and potentially explore alternative options that better align with their capabilities and interests.

A student who fails the qualifying process has the opportunity to reframe their work into a master’s thesis and complete that degree instead. Some students have retaken the PhD qualifying process and progressed to candidacy, having benefited from the early identification of areas for improvement.

A fortunate byproduct of the new qualifying process is that it energizes the student for their dissertation research. Throughout the semester, the student has multiple opportunities to present and refine their research ideas. When the student successfully passes the qualifying process, they do so with confidence in their ideas and typically reduce the time required to complete their first paper. Compare that with the traditional qualifying exam, which a student might pass without gaining any clearer direction for their research. Another byproduct is that faculty members learn of the student’s research topic early and may develop productive dialogues with them. Likewise, a student becomes familiar with faculty interests early on. That helps them identify suitable candidates for members of their dissertation committee.

Each year we watch students rise to the challenge of crafting original ideas. Witnessing students’ growth and maturation is inspirational. Indeed, the faculty also learn from the process. Many of us feel that our skills as research advisers improve as a result of it. The new format elevates the qualifying process from a routine student assessment to a shared journey of scientific discovery.

We thank our fellow faculty members for their patience and for helping to refine the qualifying process, testing it (with two of our own students), and ultimately integrating it into the department’s curriculum. We also thank faculty and students for their feedback on the manuscript.

Tim DelSole (tdelsole@gmu.edu) and Paul Dirmeyer are professors of climate dynamics at George Mason University and senior research scientists at its Center for Ocean-Land-Atmosphere Studies in Fairfax, Virginia. Together they have overseen the PhD qualifying exam in their department for more than two decades.