The article by Edward Prather, Alexander Rudolph, and Gina Brissenden (Physics Today, October 2009, page 41) makes several interesting points regarding the effectiveness of interactive learning environments in introductory astronomy courses. Yet two serious points regarding the instrumentation and methodology used in the study they discuss appear to cast doubt on the results and subsequent conclusions.
The first point concerns the use of the Light and Spectroscopy Concept Inventory. The authors state that “the LSCI is capable of measuring changes in student understanding and, by extension, the effectiveness of teaching about light and spectroscopy in Astro 101” (page 45). Underpinning the pertinent discussion in their article is the assumption that increasing students’ content knowledge and understanding is the goal of the Astro 101 course. However, instructors can have different educational objectives—for example, general science literacy, development of critical thinking skills, a reduction of science anxiety, or the linking of science to society and everyday life. 1 Astronomy content, rather than being the primary emphasis of the course, may be used as a vehicle to discuss science more broadly.
In addition, light and spectra make up one small portion of the Astro 101 course, and not all instructors may allot equal classroom time or cover the topic in equal detail, so students’ opportunities to learn the concepts assessed by LSCI may vary greatly. The authors initially acknowledge some limitations of the LSCI, but the discussion quickly moves to teaching effectiveness in general.
The second point concerns the survey that is used to determine the Interactive Assessment Score (IAS), a measure of the “nominal percentage of time … spent on interactive learning strategies during the term” (page 45). The survey provided by the authors 2 relies on instructor-reported data on typical classes and term averages. As a result, it is not clear whether the split the authors make in figure 4 of their Physics Today article between an IAS less than 25% and an IAS greater than or equal to 25% indicates a substantially different level of interaction in the classroom and whether the split actually occurs when light and spectra are discussed.
Simply moving the split point to around 30% creates more even sample sizes and still yields a statistically significant result (according to the data in figure 4) but dramatically reduces the strength of the relationship, as indicated by Cohen’s d, for example, a measure of effect size. One is left to wonder how much of the authors’ conclusion is thus due to artifacts in the data rather than to real effects.
In addition, the authors mention that they removed from the sample all classes with fewer than 25 students because they “believe that the teaching and learning in classes with a very small number of students can be a special case, bordering on personalized instruction” (page 45). They offer no research evidence for that belief, nor do they explain why the limit was set at 25 students; it is not clear whether the removal of those data is warranted. Arguably, smaller classes may have more opportunities for interactive elements—for example, class discussions—both among the students and between students and instructor. Supposedly, such classes would be in the higher end of the self-reported IAS range. Inclusion of the smaller classes thus has the potential to strengthen or weaken the authors’ result.
Although the educational literature leaves little doubt about the benefits of interactive elements in class and the benefits faculty receive from having access to professional development opportunities, this study does not present a strong enough case, given the uncertainties in and assumptions of the instrumentation used.
