The human visual system can perceive only about 10 distinct images per second. So to record everyday life as we experience it, a standard video capture rate of 24 frames per second (fps) more than suffices. But many physical phenomena unfold faster than the eye can see. To record them, researchers seek ever-faster cameras.
Now Lihong Wang and coworkers at Washington University in St. Louis have developed a camera that can record at 1011 fps, good enough to capture the movement of light at millimeter length scales.1 The camera marries streak photography, an ultrafast imaging technique previously limited to filming in one spatial dimension, with compressive sensing, a mathematical strategy for reconstructing an entire scene from an incomplete set of measurements. “The camera has countless potential applications,” comments Mário Figueiredo of Instituto Superior Técnico in Lisbon, Portugal. “It’s sort of like a microscope for time.”
Streak photography
Unlike traditional cameras, which capture a sequence of frames by repeatedly opening and closing a shutter, a streak camera captures an evolving scene by deflecting incoming light by an amount proportional to its arrival time. In early streak cameras developed in the mid 1900s, that time-dependent deflection, or shearing, was performed by bouncing incoming light off a rotating mirror. In newer models like the one used by Wang and company, light impinging on a photocathode creates photoelectrons, which are deflected by a transverse voltage ramp and then converted back to photons by a phosphor screen.
Shearing maps the time dimension of a scene onto a spatial dimension of the camera’s pixel array detector. In the example illustrated in figure 1, for instance, an object moves from left to right across a narrow aperture, and the shearing operation translates the scene’s temporal domain to the detector’s y-domain. To reconstruct a movie from the detected signal, one simply takes each row of pixels as a frame.
Figure 1. In a streak camera, incoming light is deflected by an amount proportional to the time of its arrival, a procedure known as shearing. When a one-dimensional scene like the one shown here is sheared vertically, the scene’s time domain maps to the detector’s y-domain, and a movie can be reconstructed by taking each row of pixels as a frame. When the same shearing is applied to a 2D scene, the would-be frames overlap at the detector, which complicates the task of inferring a scene from the detected image. By masking the 2D scene with a known pixel pattern, one can encode the detected image with spatial information that facilitates the faithful reconstruction of a frame sequence.
Figure 1. In a streak camera, incoming light is deflected by an amount proportional to the time of its arrival, a procedure known as shearing. When a one-dimensional scene like the one shown here is sheared vertically, the scene’s time domain maps to the detector’s y-domain, and a movie can be reconstructed by taking each row of pixels as a frame. When the same shearing is applied to a 2D scene, the would-be frames overlap at the detector, which complicates the task of inferring a scene from the detected image. By masking the 2D scene with a known pixel pattern, one can encode the detected image with spatial information that facilitates the faithful reconstruction of a frame sequence.
Because one of the pixel array’s dimensions is reserved for mapping time, conventional streak cameras can film in only one spatial dimension. For a 2D scene, such as in the middle example in figure 1, one encounters the complication of mapping three of a scene’s dimensions—two spatial dimensions and time—onto two dimensions in the pixel array. The would-be frames overlap on the detector in a way that can’t easily be disentangled.
Mathematically speaking, the problem of deducing the original scene from the detected image is said to be underdetermined. To reconstruct a scene that’s m pixels wide, n pixels tall, and p frames in duration, one must solve for m × n × p unknowns. Each pixel in the detector provides a constraint on that solution, but the detector comprises only around m × (n + p) pixels. Because many possible solutions exist, there’s little guarantee that any particular one will bear much resemblance to the true scene.
In 2011 researchers at MIT found a way to circumvent that problem under special circumstances: They filmed their 2D scene one row of pixels at a time.2 Although they could capture video at an effective frame rate of nearly 1012 fps, the approach only works for scenes that can be replayed at will, with near-perfect reproducibility. By contrast, Wang and his coworkers sought to image 2D scenes in a single shot. To do that, they took a cue from the field of signal processing.
Compressed sensing
Image compression is based on the concept that most images contain relatively little crucial information. The essence of a person’s face, for instance, can often be captured with just a few lines tracing the important contours. By strategically discarding unneeded data, one can reduce an image file’s size, which facilitates storage and transmission, while retaining enough information to reconstruct a version that’s faithful to the original.
In compressed sensing, developed about a decade ago, the idea is to avoid acquiring the unneeded data in the first place.3 Theoretical work by David Donoho (Stanford University), Terence Tao (UCLA), and Emmanuel Candès (then at Caltech) showed that if a signal is sparse—that is, if it can be represented as a matrix in which most of the entries are zeros—then a random, partial sampling of that signal can yield sufficient data to reconstruct the signal in its entirety.
For Wang and his coworkers, compressed sensing pointed to a clever path around the streak camera’s determinability problem. The researchers approximated random sampling of a scene by masking their camera’s 5-mm-wide aperture slit with a programmable spatial filter. The filter encodes the incoming scene with a pixel pattern that is then translated to the camera’s CCD detector.
The mask introduces additional constraints on the computed reconstructions of the original scene: Any solution must be consistent not only with the effects of shearing but with the predictable effects of the complicated mask. (Roughly 150 × 150 pixels, the actual masks used in Wang and company’s camera are much more intricate than the example mask shown at right in figure 1.) As long as the image is sparse—say, if intensity gradients are concentrated along just a few lines—it doesn’t matter that only a portion of the actual scene was seen by the detector. The missing pixels can be filled in with standard reconstruction algorithms that seek out maximally sparse solutions.
“There have been several applications of compressed sensing that allow you to acquire data more quickly or more cheaply than you could with conventional techniques,” says Figueiredo. “But here it’s being used to solve a problem that couldn’t be solved any other way.”
Light on the move
Wang’s group isn’t the first to apply compressed sensing to perform single-shot imaging in more than two dimensions. In the 2000s, David Brady and coworkers at Duke University developed a similar technique called coded-aperture snapshot spectral imaging.4 Whereas Wang and company’s streak camera captures image sequences in the time domain, the Duke device captures them in the spectral domain. Many systems of interest for spectral imaging, however, turn out to be too complicated to accurately reconstruct using compressed-sensing algorithms.
The advantage of recording in the time domain is that even visually simple systems can be scientifically interesting when they’re captured at billions of frames per second. The ability to film the propagation of light through optical media, for example, could inform the design of invisibility cloaks and other optical metamaterials. (See the article by Martin Wegener and Stefan Linden, Physics Today, October 2010, page 32.)
To demonstrate their camera, Wang and his coworkers filmed 7-ps pulses of visible light reflecting off a mirror, refracting at an air–resin interface, illuminating striped walls, and exciting fluorescence in rhodamine dye. As shown by the still frames in figure 2, the reconstructed frame sequences display light trajectories that closely match paths predicted by ray optics. (The splotch of light that appears behind the mirror in figure 2a is due to the imperfection of the mirror.)
Figure 2. Still frames from movies captured with compressed ultrafast photography show a light pulse (a) bouncing off a mirror and (b) refracting at an air–resin interface. Dashed lines indicate trajectories calculated from ray optics. In both movies, frames were captured every 20 ps. (Adapted from ref. 1.)
Figure 2. Still frames from movies captured with compressed ultrafast photography show a light pulse (a) bouncing off a mirror and (b) refracting at an air–resin interface. Dashed lines indicate trajectories calculated from ray optics. In both movies, frames were captured every 20 ps. (Adapted from ref. 1.)
Wang envisions using the camera to film biochemical reactions, nuclear explosions, and other quickly unfolding events. He says, “The camera itself is generic—it can be coupled with all sorts of scopes, from microscopes to telescopes. I’m enamored with the idea of coupling it to the Hubble telescope, where it might be able to image collapsing supernovae.”
