Decades of physics education research (PER) have produced valuable insights into how students think and learn. (See the article by Carl Wieman and Katherine Perkins, Physics Today, November 2005, page 36.) The work has also helped teachers create better curricula and more effective assessments and has improved the quality of physics education. But according to a new study from Stephen Kanim (New Mexico State University) and Ximena Cid (California State University, Dominguez Hills), data used in PER studies in the US may not reflect the average student experience. Certain types of students are overrepresented, and when studies rely on the experiences of atypical students, their findings may apply to only a subset of the physics student population.
Kanim and Cid considered the PER studies published from 1970 to 2015 in three prominent journals—the American Journal of Physics, Physical Review Physics Education Research, and the Physics Teacher. Of the 1031 papers with student data, 417 included American students and contained the necessary information—namely, the students’ course level and institution—to be considered in the researchers’ analysis. That subset of papers included data on more than 250 000 physics students. When the researchers compared the demographics of the students in those PER papers with the nationwide student population, they found significant differences regarding both the students and the institutions being represented.
According to Kanim and Cid, one source of bias is the choice of university students, who, for convenience, are often used as study subjects. Based on standardized test scores, it is likely that nearly all the college students included in the PER studies had above-average math skills. The first figure compares the SAT math scores of all incoming college students (not just physics students) with those of students at the 39 universities that provided 95% of the student subjects in the PER studies. It highlights how skewed the sampling is: At 15 of the universities, the range of SAT math scores for the middle half of students doesn’t even overlap with the range for all students.
Relying on that small set of universities also leads to white and Asian students being overrepresented in PER studies, and that overrepresentation comes at the expense of Black, Hispanic, and Indigenous American students (second figure). Students in PER studies also come disproportionately from households in the top income quintile, 48.7% in the study population compared with 29.9% of physics students overall.
The reliance on college students is also problematic because most physics teaching happens in high schools. Even though about 75% of physics students are in high school, they make up only about 10% of study subjects. Kanim and Cid note that the discrepancy may be in part because high school students are minors, so the study approval process is more burdensome. But the plentiful studies on high school math students suggests that approval isn’t an insurmountable roadblock. “More people may be willing to go through the approval process for a math class because every secondary student takes several math classes, so math touches every student multiple times,” says Susan White, interim director of the Statistical Research Center at the American Institute of Physics (which publishes Physics Today). “That’s just not true for physics. Less than half of high school graduates in the US have taken a physics class.”
Another possible explanation for the relatively few studies on high school physics students is that university researchers prioritize understanding their own institution’s students, according to Paula Heron, a physics education researcher at the University of Washington. The researchers may feel that they have less to contribute in a high school setting.
Students at two-year colleges have even lower representation in the literature. Although 26% of postsecondary introductory physics students attend two-year colleges, those students make up only 0.3% of PER subjects (third figure). And at four-year colleges, PER is biased toward more advanced students, with more than 80% of subjects coming from calculus-based courses.
Given that past PER studies have relied disproportionately on privileged, high-achieving students, their conclusions may not apply to all students. After all, high school students taking their first conceptual physics course have needs and motivations different from those of physics majors taking a calculus-based course at a four-year university. “This study is a powerful and timely reminder that even the most carefully developed and rigorously tested instructional approaches may not translate perfectly to new environments, especially if the student populations differ significantly from those at the originating institution,” says Heron.
The news isn’t all bad. Kanim and Cid note that studying a homogeneous population has likely helped PER studies generate clearer results and recommendations by minimizing demographic variations as a confounding variable. Those studies have led to improvements in curriculum development and instructional techniques. The researchers liken the situation to learning physics: Students begin by studying idealized systems, from which they glean important information and intuition. But to understand the real world, they need to move beyond that simplification. It may be time, the authors say, for PER to do the pedagogical research equivalent of including friction and deformable objects of finite size.
Beyond identifying skewed demographics, the researchers’ analysis uncovered that many PER papers don’t adequately characterize their subjects. “I was struck by how difficult it was for the authors to figure out some very basic details about the subjects involved in the studies,” says Heron. Simply including information about students and their learning environments in publications will make it easier for researchers to understand, contextualize, compare, and replicate future studies.
Kanim and Cid recommend including more diverse populations in new PER studies and replications of existing studies; they also encourage professional societies to advocate for such increased diversity and urge funding agencies to incentivize it. The information from those studies would help researchers identify what makes a student well prepared to study physics and give educators a clearer understanding of incoming students’ abilities and needs.
