Arthur Compton and the mysteries of light

The mysterious rays discovered by Wilhelm Röntgen in 1895 were a puzzle from the start. Their ontological status was continually under discussion.⁵ The mechanism used to produce them—bombarding an anode with an electron beam—was highly suggestive of bremsstrahlung emission, so x rays were viewed as an unusual cousin of visible light. The rays propagated in straight lines, were not deflected by electric and magnetic fields, and could expose photographic plates: clear visual evidence of familiar light-like behavior. But emission from charged-particle collisions should occur in short pulses, so a picture of x rays as innumerable, aperiodic spherically propagating impulses held sway in the prewar years. That view gained credence when J. J. Thomson calculated the distribution of energy in such pulses and found that it was in accord with early x-ray experiments.

But for a decade following the discovery of x rays, further evidence of well-known wave phenomena was hard to come by: X rays did not obviously diffract or interfere and were only first seen to be polarizable in 1905 by Charles Barkla.⁶ Their seemingly contradictory properties led to some intriguing alternative ideas: William Henry Bragg, for example, argued for years that the rays were actually electrons paired with a putative positive charge.⁷ But by 1912 Bragg’s position had evolved, and he began searching for an x-ray theory that included the characteristics of both a particle and a wave.⁵ 

That hypothesis was driven in part by a growing recognition that x-ray scattering in gases posed a serious challenge to classical electromagnetism. If x rays emanate from decelerating charges, the expanding sphere of influence will rapidly attain a size far exceeding the distances between atoms in a rarefied gas. Yet the electrical currents through ionized gases pointed to a very small ionization rate, on the order of one in 10¹² atoms.⁸ Why should so few atoms be affected by a wave passing equally over all? Moreover, the released electrons contained a significant part of the incident energy, as if the expanding pulse suddenly concentrated all its energy on a vanishingly tiny portion of its surface.

Despite their best efforts, researchers failed to resolve those anomalies in the first two decades of the 20th century. Arnold Sommerfeld suggested that a focused relativistic beam of “needle radiation” might explain the phenomenon, but the beam would still illuminate broad regions of the gas. A version of the same problem arose in the photoelectric effect: Even at vanishingly small light intensities, electrons were emitted the instant a metal surface was illuminated. That meant it was impossible for energy to slowly accumulate from successive spherical disturbances. Worse, as those experiments pushed into the x-ray regime, the seemingly problematic behavior of x rays began to infect visible light, which had heretofore been safely in classical territory.

Still, evidence that x rays were simply unusual electromagnetic impulses began to pile up. In the spring of 1912, at the suggestion of Max Laue—whose name would not carry the aristocratic von until his family was ennobled the next year—Paul Knipping and Walter Friedrich demonstrated that diffraction patterns arose from x rays passing through copper sulfate crystals en route to a photographic plate. Although those patterns evince periodic wave behavior, many prominent individuals who supported the wave picture of x rays, including Barkla and Sommerfeld, were either highly resistant to that interpretation or skeptical that such behavior could be observed.⁹ 

At any rate, Hendrik Lorentz soon demonstrated how short impulses—namely, finite trains of oscillating waves representing x rays—could show interference just like monochromatic waves. And within a year, the father–son team of William Henry Bragg and William Lawrence Bragg, who were already comfortable with a more corpuscular picture of x-ray motion, also derived the equation governing coherent reflection from successive separated crystalline layers. In a prescient letter to Ernest Rutherford that anticipated the eventual discovery of wave–particle duality, William Henry noted that “the ray travels from point to point like a corpuscule [yet] the disposition of the lines of travel is governed by a wave theory. Seems pretty hard to explain, but that surely is how it stands at present” (reference 5, page 210; brackets in the original).

Researchers also discovered that scattered x rays displayed a phenomenon akin to fluorescence. In the classical theory, electrons accelerated by passing radiation can reradiate only at the incident frequency, so the observed wavelength increase was ascribed either to inhomogeneous secondary bremsstrahlung emission from electrons liberated by the primary beam or to a material-specific homogeneous emission. In many pre–World War I experiments, the inhomogeneity of primary x-ray beams made the resolution of secondary emission sources difficult. But even after switching to use the more homogeneous characteristic rays as a source, researchers continued to detect a mysterious spectrum of scattered rays at lower energies than the incident beam.

The advent of crystal diffraction delivered even more superior beams, sourced from Bragg spectrometers, that were bright, monochromatic, and tunable. But the roaring success of Niels Bohr’s atomic model, which he presented in 1913, proved to be far more attractive for experiments than the muddle of contradictory facts surrounding free radiation. For several years afterward, x-ray spectroscopy was almost exclusively applied to exploring atomic energy levels. With the onset of World War I, researchers shelved interest in the fundamental nature of x rays.

With hindsight, Einstein’s quantum theory of light would resolve those difficulties, but for years the idea was radioactive. Modern textbooks list the photon among the revolutionary ideas in Einstein’s 1905 annus mirabilis, but at the time virtually no one aside from Einstein felt the idea had credence. Johannes Stark argued as early as 1909 that light quanta would scatter from electrons in a particle-like fashion, and he highlighted momentum conservation even before Einstein. But Stark’s enthusiastic support was not widely shared. In 1907 Wilhelm Wien employed Planck’s quantum theory to relate the kinetic energy of an electron to the width of an x-ray impulse, but he avoided any interpretation in terms of a spatially localized quantum like Einstein’s Lichtquant.

The idea of a localized quantum of light was not unknown, but it was just a leap too far. Even William Henry Bragg’s theory that x rays were a corpuscular neutral pair was applied only to high-energy rays and carried no notion of a connection to visible light. Although the 1921 Nobel Prize in Physics awarded to Einstein lauded “his discovery of the law of the photoelectric effect,” the actual linchpin of his argument—that light is composed of quantized packets—was not widely accepted.

In his own Nobel lecture a year later, Bohr sardonically punned away the quantum theory as a model of merely “heuristic value” that was “not able to throw light on the nature of radiation.” In 1916 Robert Millikan, whose own precision measurements left no doubt as to the total validity of Einstein’s photoelectric equation, dismissed the quantum theory of light as “so untenable that Einstein himself, I believe, no longer holds to it.”¹⁰ Even Stark, the quantum theory’s early and outspoken supporter, gave up on the Lichtquant following the discovery of x-ray diffraction. As for Compton, he was firmly on the side of classical electromagnetism.

Born in 1892 Compton showed an aptitude for experimental science from an early age. He built a camera that attached to a telescope so that he could photograph Halley’s comet in 1910. He reproduced the Wright brothers’ flight experiments by building his own triplane glider and flying it an exhilarating distance of 185 feet. But at the alarmed urging of his parents, he abandoned further flying and gave his craft a bonfire send-off. At 21 he published a method for measuring Earth’s rotation based on the momentum of water flowing in a circular tube.¹¹ 

Compton and his brother Karl first encountered x rays in the physics laboratory at the College of Wooster, where their father was a professor, and continued investigating them during their graduate studies at Princeton University. After Karl became an assistant professor at the Ivy League institution, the brothers collaborated on several projects, such as improving electrometer precision. Their model was the most precise electrometer of its day: It was able to resolve currents of 10 fA in a minute.¹² Those experiences informed Compton’s lifelong preference for building his own apparatus. An accomplished glassblower, he fabricated his own x-ray tubes, and he greatly improved the precision of x-ray spectroscopy by using his own electrometer in place of a traditional electroscope to measure the x-ray-induced ionization charge.

After finishing his PhD and teaching at the University of Minnesota for a year, Compton worked at the Westinghouse Lamp Company for two years. Although he was promised resources for his experimental x-ray research, the needs of the company quickly took precedence: He was tasked with improving light bulbs. So he began pursuing theoretical work on x rays in his spare time, and he started analyzing two discrepancies between recent experimental work and the classical picture of x-ray scattering from point-like electrons. The first of those anomalies had to do with the radiation from electrons excited by transverse fields. Although Thomson’s classical theory predicted it should be symmetric along the incident axis, gamma-ray-scattering experiments clearly showed an excess of scattering in the forward direction. The second discrepancy came from x-ray absorption coefficients, which Barkla and others had reported to be unexpectedly low for beams of increasingly shorter wavelength.

Both observations violated Thomson’s scattering theory of radiation. Compton pursued those issues to surprising ends. Regarding the first anomaly, he showed complete confidence in classical electrodynamics by proposing that x rays must not only reradiate but also diffract from electrons in materials. Because diffraction occurs for light scattering from objects whose size is of the order of the wavelength, Compton sought to determine the shape and size required to generate the observed scattering distribution. For nearly three years, he would argue for a ring-shaped electron with an enormous radius of approximately 2 pm that reproduced the excess forward scattering.¹³ 

To pursue the second issue of anomalously low absorption, Compton left Westinghouse in 1919 to spend his National Research Council fellowship with Rutherford at Cambridge University’s Cavendish Laboratory. In the UK, he absorbed the latest gamma-ray experiments and saw his ring-electron theory publicly rebuffed by Rutherford. Introducing him before a talk, the famous physicist once pronounced, “This is Dr. Compton…. I hope you will listen to him attentively. But you don’t have to believe him!” (reference 14, page 29).

Replacing Rutherford’s venerable gold-leaf electroscope with his own four-quadrant design, Compton proceeded to study gamma-ray scattering. He soon found that the energy of secondary rays decreased unexpectedly when scattered to higher angles. Compton deemed that effect a new type of fluorescence: Guided by a strong classical intuition, he had begun to delineate the secondary rays into “truly scattered” and fluorescent radiation. As required by classical electromagnetism, truly scattered rays had the same wavelength as the primary beam. In contrast, Compton ascribed the longer-wavelength fluorescent radiation to Doppler-shifted emission from high-velocity electrons set in motion by the primary beam. Interestingly, Compton still relied on diffraction to account for the angular distribution, although his results forced him to abandon the ring-shaped electron in favor of a solid, spherical model.⁴ But with a 5 pm radius, his proposed spherical electron was still enormous!

Buoyed by those experiences, Compton returned to the US to take up a position at Washington University in St Louis. It may be surprising that he chose a university that at the time, in his own words, “was a small kind of place,” but Compton was intentionally seeking to avoid the centers of x-ray science, where he worried he might be “led away by the thinking of the time” (reference 14, page 31). En route to his post, he penned on ocean-liner stationery a plan of attack several pages long that he labeled “Problems to be Tackled At Saint Louis,” shown in figure 2. Intriguingly, that plan shows he was aware of the quantum relation, $E=hν$⁠, and the implication of such an interaction with single electrons. It also contains a sketch of a direct beam from an x-ray tube, which Compton ultimately abandoned. Instead, from the outset he used a Bragg spectrometer as a wavelength selector to deliver precise monoenergetic beams (see figure 3). He was soon making rapid progress.

Now Compton was able to show for certain that, contrary to two other contemporaneous findings,⁴ the longer wavelength fluorescent radiation he had first seen in gamma rays persisted in x-ray scattering. Then, in collaboration with Charles Hagenow, Compton found that the fluorescent radiation was completely polarized: a result that could be naturally explained as scattered light. But truly scattered light should not change its energy. Perhaps, he hypothesized, the polarization could be maintained if the secondary emission were to occur at the same instant an electron scattered the primary emission. That was the first time Compton considered the possibility of simultaneous scattering and emission events. Around the same time, he was tasked with writing a review on the status of secondary radiation for the National Research Council, which obligated him to revisit older literature and confront the possibility of a quantum nature of light. He tacked a note to that effect at the review’s end that signaled his first willingness to break with total adherence to classical electromagnetism.

In October 1921 Compton made the crucial choice to explore the spectrum—rather than just the intensity—of the secondary radiation. Employing his Bragg spectrometer for its original purpose, he measured the spectrum of a molybdenum K-alpha line scattered to 90 degrees from a block of graphite and produced the data seen in figure 4a. But in his early analyses, Compton apparently referred only to tables of data rather than plots and so mistook the small peaks at right as the Doppler-shifted emission from rapidly recoiling secondary electrons that absorbed all the energy of the strong peak in the primary beam.

In the ensuing months he realized the error and understood that the wavelength shift was far smaller. Now believing the second tall peak—the solid line in figure 4a—to be the Doppler-shifted K-alpha line, he erred again when determining, by solely conserving momentum, the velocity of the recoiling electron. Just weeks later he got it right: Drawing on the same data, he entirely abandoned the Doppler shift in favor of a pure scattering picture and drew the diagram reproduced in figure 4b. From that he immediately derived his quantum theory of scattering by conserving both energy and momentum.

Further measurements of the intensity of scattered radiation and the absorption coefficient agreed with the quantum theory and explained the low absorption that had set Compton on that path. As he noted, there was now “little doubt that the scattering of X-rays is a quantum phenomenon” (reference 2, page 501). He presented those results outside his classroom for the first time at an American Physical Society meeting in Chicago in early December 1922, a little over a week before Bohr delivered his Nobel lecture in which he expressed skepticism of Einstein's Lichtquant. Compton submitted his paper to the Physical Review on 13 December, just two days after Bohr’s lecture. Interestingly, his paper does not cite Einstein. Instead, it begins by discussing the failure of Thomson’s scattering theory, as Compton bid goodbye to a purely classical world. By the end of 1923, Charles Wilson had observed the recoiling electrons in his cloud chamber,¹⁵ as depicted in figure 5.

Of course, apart from his x-ray tubes, Compton was not working in a vacuum. The speed at which his discovery was accepted—indeed, the seeming about-face regarding the reality of photons that was quickly performed by most physicists—was possible because those ideas had been in the air. With World War I over, European scientists had returned to their labs, where old ideas, including the paradoxes of x rays, were being reconsidered.

Notably, the French noble brothers de Broglie—Maurice, an accomplished amateur x-ray scientist, and Louis, a budding theorist—were intrigued by the quantum theory from early on. Maurice took advantage of the developing field of beta-ray spectroscopy to explore the transfer of energy during x-ray absorption in the photoelectric effect. He soon became convinced that electrons were emerging with the entire energy of the incident x rays and argued that it “must be corpuscular, or, if it is undulatory, its energy must be concentrated in points” (reference 5, page xi). Hendrik Kramers, too, had apparently sketched the basic picture of the quantum scattering result in 1921 but had been harried by Bohr into not publishing it.¹⁶ 

Motivated by the same issue of low absorption that had spurred Compton’s investigations, Peter Debye derived a more general scattering theory than Compton and prodded his colleague Paul Scherrer to pursue experiments on the secondary rays. Unfortunately for Debye, Scherrer did not take up that invitation. Although Debye submitted his article after Compton, it appeared in print two months earlier. But he always acknowledged Compton’s priority. In any case, that the same ideas were being discussed simultaneously on two continents by established but independent researchers no doubt hastened their acceptance.

In 1922–23 Sommerfeld was a visiting professor at the University of Wisconsin, and while in the US, he closely followed Compton’s advances. Upon his return to Europe, Sommerfeld became a key proselytizer of the new paradigm. Writing to Compton in October 1923, he reported on how the discovery “keeps the scientific world in Germany extremely busy” (reference 4, page 247), and he noted that he was already including a section on the Comptoneffekt in the next edition of his textbook Atombau und Spektrallinien (Atomic Structure and Spectral Lines). With some hyperbole, he claimed it “sounded the death knell” of the wave theory.

On the contrary, Compton’s discovery forced the physics community to reckon with the persistence of obvious wavelike aspects of light. Light was not suddenly and only a particle. It couldn’t be, given the established observations of polarization and diffraction. Compton himself made that point in a remarkable way: A mere four days before submitting his quantum scattering results, he submitted a separate paper announcing the discovery of total reflection of x rays,¹⁷ a wavelike effect if ever there was one. There could be no objection on the basis that light was either a particle or a wave; physicists would now have to contend with the fact that it was both.

Not everyone was on board. For some time Bohr remained incapable of relinquishing his devotion to a purely wave-based conception of the nature of light. The depth of his opposition was sufficient to motivate him to jettison an absolute notion of the conservation of energy and momenta in favor of a weaker statistical conservation. Working with Kramers—who himself was now fighting a rearguard action against a theory he had previously argued for—Bohr adapted John Slater’s idea of a virtual radiation field and developed a theory that had discontinuous quantum jumps between atomic states in a classical electromagnetic field.¹⁸ But the nature of those novel virtual oscillators was far from clear.

For a paper with only a single equation, the Bohr-Kramers-Slater article, as it became known, is notable for proposing a probabilistic formulation of conservation laws in a manner robust enough to be disproved. Indeed, Walther Bothe and Hans Geiger soon demonstrated that the arrival times of recoiling electrons and scattered x rays were sufficiently close to rule out a statistical interpretation. Compton weighed in with similar findings. Thus, as a direct consequence of the quantum picture of scattering, an experimental basis for the event-wise conservation of energy and momentum was also found.

Compton’s repeated and sometimes outlandish efforts to couch the outcomes of x- and gamma-ray-scattering experiments in the language of classical electrodynamics vividly illustrate how he had not initially set out to spark the shift in viewpoint from a wave–particle dichotomy to a wave–particle duality. But once he accepted the new quantum scattering paradigm revealed by his experiment, he elegantly argued in the October 1925 issue of Scientific American that “most physicists look forward” to a solution that would be found in “some combination of the wave and quantum theories,” just as William Henry Bragg had anticipated in 1912.

Erik Henriksen is an associate professor of physics at Washington University in St Louis in Missouri. His research focuses on fundamental electronic structures and emergent quantum phenomena of low-dimensional materials.