Three-dimensional displays, past and present

Ideally, a 3D display should render objects as they naturally appear. To do so, it has to provide visual cues both psychological and physiological.² Vision is a learned art, and psychological cues embedded in a 2D image enable us to perceive a sense of depth through our experience of the world or through the relationship between objects in a scene. Normally, depth and order in a 2D image come from our sense of linear perspective, in which parallel lines seem to converge at vanishing points, and from overlapping objects, shading, shadows, and texture gradients. It’s no surprise that such psychological cues are ubiquitous in paintings, frescoes, and computer graphics.

Physiological cues, in contrast, originate either from binocular disparity or from the physical changes that occur in each eye as it tracks or focuses on an object. As outlined in figure 1, each eye also rotates—by a larger or smaller degree depending on how close an object comes—so that the eyes’ lines of sight converge with it, an effect known (appropriately) as convergence; the curvature of each eye also changes, depending on the focal length needed for the eye to clearly image the object, an effect known as accommodation. And if the object or observer moves, motion parallax may give a sense of depth; closer objects move faster across the field of view than those further away. For that cue to be implemented ideally, different sides of an object become apparent with the change in position.

Most 3D display technologies, however, can induce only a fraction of the physiological depth cues. In the 3D televisions recently commercialized, binocular disparity and convergence are induced by the use of a pair of glasses. Autostereoscopic displays, which don’t require glasses, can also induce those cues and a sense of motion parallax as well—when, for instance, the observer’s perspective changes.

Moreover, neither stereoscopic nor autostereoscopic displays induce an accommodation cue that prompts a viewer’s eyes to change focus. Both technologies show interwoven or fused stereo images using a single, fixed screen. In that situation, each eye tends to focus on the screen to see clear images, rather than on the location where the image is intended to float. The result is that accommodation and convergence cues conflict with each other, which confuses the brain and produces headaches.

Although stereoscopic displays are not ideal, at the moment they are the only 3D technology to have successfully penetrated the consumer TV market and 3D box office.³ Commercial displays can be categorized into those that require polarized glasses and those that require liquid-crystal glasses whose right and left sides act like shutters that alternately transmit and block screen images.

Stereoscopy using polarized glasses has found wide usage in 3D cinemas since 1952. Two orthogonal states, such as left and right circular polarization, are used to distinguish the left and right perspectives. The two views are projected onto a highly reflective screen through orthogonal polarization filters; an observer’s polarized glasses then separate the left and right views.

In one way to adapt that cinematic technique for flat-panel TV screens, wave-retarder plates with a phase difference of π between them are attached alternately to even and odd rows of the screen to spatially separate and interlace the left- and right-eye views, as shown in figure 2a. Although the vertical resolution of the image presented to each eye is cut in half, the overlap, or crosstalk, between left- and right-hand images is relatively low, and the polarized glasses are easy and economical to fabricate. An improved version of the technology uses an active retarder to spatially modulate the polarization of the entire screen with a high refresh rate without sacrificing resolution.

In contrast, stereoscopic displays designed to work with liquid-crystal shutter glasses are based on a time-multiplexing technique in which the left and right perspective views are kept in sync with the glasses’ transmission of images to each eye. To repeatedly block and pass each image without flickering, the display panel and shutter glasses must operate at more than twice the conventional refresh rate of 60 Hz. And to remove crosstalk during the line scanning of images, the rate is generally kept four times higher, at 240 Hz, with black frames inserted between the left and right eye views.

Current LCD technology enables a high enough refresh rate, and plasma display panels and recently proposed active-matrix organic LED panels can refresh the screen at rates that are higher still. Although shutter glasses provide high resolution of a 2D screen, the displayed images lose some of their brightness from the periodic insertion of black frames.

Since it requires no special glasses, autostereoscopy is currently receiving attention as the next-generation 3D display paradigm.⁴ Autostereoscopic displays come in three types: ones made from parallax barriers, from so-called lenticular lenses, and from a more general lens system known as integral imaging. What distinguishes them is the way in which the pixels are spatially grouped to provide images at different observation points. Let’s take each of the technologies in turn.

In the parallax barrier method, a sense of depth arises from the fact that narrow slits separated by opaque barriers in front of a screen enable each eye to see different sets of pixels, as illustrated in figure 2b. The simplicity of using barriers is the method’s primary advantage, but because they reduce the number of pixels each eye sees, they also reduce the brightness and resolution of a 3D image. Parallax barriers are therefore generally used in mobile devices or monitors where high image quality is not needed. A few manufacturers have commercialized 2D-to-3D convertible smartphones, for instance, that can display 3D images using electrically switchable parallax barriers made of liquid crystals. Because the positions of the barriers (and thus the pixels viewable by each eye) can shift with time, such devices can provide high resolution. It is necessary, however, to double the refresh rate of the display panel to prevent observers from sensing the presence and shifting of the barriers.

A lenticular display uses an array of 1D cylindrical lenses instead of barriers to spatially separate pixels on the screen. When the lenses are viewed from different angles, they magnify and project different pixels and thus different perspectives. Compared with parallax barriers, lenticular lenses offer higher brightness and sharpness, although they can suffer from manufacturing and alignment issues.

Like parallax barriers, lenticular lenses have also recently been implemented with liquid crystals, which enable the electrical conversion between 2D and 3D images. And numerous companies have patented a wide range of technologies for implementing such switchable lenses. One approach in the lenticular system, for instance, slants the cylindrical lenses at an angle to ameliorate the loss of horizontal resolution by redistributing it in both horizontal and vertical directions. Another approach preserves the lenses’ vertical orientation, but rearranges the subpixel structure in a way that also redistributes the resolution.

Integral imaging, known originally as integral photography,⁵ was invented by Gabriel Lippmann in 1908, the same year he was awarded the Nobel Prize in Physics for work reproducing colors photographically based on wave interference. Integral imaging uses a 2D array of small spherical, square, or hexagonal lenses—sometimes called fly’s-eye lenses—to produce a 3D perspective. It thus provides both horizontal and vertical parallax with quasi-continuous viewpoints within the viewing angle.⁶ That convenience makes it a candidate for next-generation 3D TVs that viewers can enjoy watching from any position they like. The method’s resolution and expressible depth range, however, are inferior to what lenticular-lens displays can offer. Indeed, although integral imaging may appear at first glance to be a 2D expansion of the lenticular lens multiview technology, the detailed designs differ in how light rays are directed to viewers’ eyes.

Resolution, viewing angle, the number of viewing locations, and expressible depth range are all limited by the specifications of an autostereoscopic display panel and by how pixels are spatially grouped and projected into different perspectives. In addition, eye fatigue and the challenge of manufacturing thin displays that can convert between 2D and 3D content continue to be obstacles to commercialization. Nonetheless, most manufacturers and researchers agree that users of future 3D displays will be taking off the special glasses.

Holograms, the product of laser-light interference, and volumetric displays, devices that form the representation of an object in three physical dimensions rather than on a 2D screen, are important technologies. Both provide for natural depth perception, and they prompt viewers to focus on an image itself rather than on a screen.⁷ Indeed, both provide all the physiological depth cues without any conflict among them.

Holography, proposed by Dennis Gabor in 1948, more than a decade before the first lasers appeared, stores information about both the intensity and phase of light scattered from an object in a scene.⁸ Because holograms reconstruct the wavefronts of that light, viewers perceive the light as if it were scattered from the object itself. Historically, holograms have been created on photosensitive materials that record permanent interference patterns between different waves. But researchers are actively exploring holographic materials, such as photorefractive polymers, to generate high-resolution images that may also be updated—in milliseconds to minutes in recent implementations.⁹ 

Electronic devices such as cameras and spatial light modulators, which impose some kind of spatial variation in the light’s intensity or phase, have also been developed recently to record and display dynamic 3D contents as digital holograms.¹⁰ Originally, the terminology implied that the holograms were reconstructed computationally rather than optically. Today many people use it simply to indicate holography that is generated, stored, and retrieved with electronic devices. Unfortunately, the amount of information required to generate holograms that continually refresh at video rates exceeds the power of most computers to generate them, at least at high resolution.

Different strategies have been proposed to resolve the problem. In a holographic stereogram technique, narrow holograms are separately created and synthesized to fit seamlessly together; in essence, parallax is reproduced by the diffraction of light from the holograms confined in multiple windows. Unlike typical stereograms, however, no special glasses are needed to perceive the effect. Another way to reduce the amount of information the computer must handle is to use an eye-tracking method that detects a viewer’s position and generates partial holographic images viewable only from a narrow region around the eyes.

Computing electronics keep improving as well. Parallel computing and graphics processing units adopted in computer-generated holography schemes effectively increase the computational power and thus reduce the lag time between updated images. One could also convert incoherently captured depth into holographic data. That kind of data can be easily acquired by multiple cameras or a lens array and enables engineers to better capture background scenes behind an object.

Unlike holograms, which use 2D screens, volumetric displays form voxels—pixels in a 3D grid—to represent objects in space. Because they emit or scatter light at specified points in space to generate images, the displays support natural depth perception without the accommodation–convergence conflict. In many cases, however, they suffer from an inability to represent shadows properly because light is emitted in all directions. And the viewable images in a volumetric display are often small, limited by the size of the display itself.

Volumetric displays use several methods to simulate depth through a number of different visual effects.¹¹ In one of the most researched methods, the voxels are formed on a rotating light-scattering screen, known as a diffuser, that sweeps an image projected onto it 360°. The projected image is synchronized with the rotation, so viewers watching the image see generated images of an object or scene that appear coherent from anywhere around it,¹² as outlined in figure 3a. In addition to mechanical methods, a 3D image may be produced electrically by a change in the polarization or transmittance of any of several light-scattering screens. One example is based on the depth-fusing effect shown in figure 3b. In that case, a projected image’s apparent depth depends on the locations of a series of liquid-crystal screens spread over a certain volume in front of a viewer. Electric signals determine which screens scatter light and thus where objects appear in space.

Other types of volumetric display don’t rely on a screen at all. One example is a color display whose images emerge from the frequency conversion of light in a nonlinear crystal. Visible light, for instance, is emitted at the intersection of two IR laser beams in rare-earth-doped heavy-metal fluoride glass.¹³ By manipulating the crossed lasers, one manipulates the image. In another approach, a laser plasma display generates light emission in midair,¹⁴ as shown in figure 3c.

There’s no shortage of technologies that have been proposed to achieve the ultimate 3D display. Some are newly invented methods; others are modified versions of existing methods. An example of the latter, the frontal-projection 3D display,¹⁵ is based on the parallax-barrier concept. Whereas typical parallax-barrier systems present images from behind the barrier (see figure 2b), frontal-projection displays do so from the front, as the name suggests. The method adapts parallax barriers to theater screens so that moviegoers can eventually experience 3D images from a standard projector, no polarized glasses required.

The method works by manipulating the polarization of projected images, as outlined in figure 4. The parallax barriers are made of material that blocks horizontally polarized light but passes vertically polarized light. The vertically polarized components of the projected image thus pass through the barriers and the slits between them. The light then passes twice through a quarter-wave film, before and after reflection from the back wall, after which the light becomes horizontally polarized. Viewers therefore see only the reflected light that passes through the slits.

Another promising technology is the so-called see-through 3D display—essentially a transparent screen within which an array of concave half mirrors is embedded.¹⁶ The mirror array works like an integral imaging device, reflecting different light rays of front-projected images from the screen to each eye in order to convey three dimensionality. But objects located behind the screen—or equivalently, objects projected from behind it—can also be seen. That offers viewers the ability to see 3D images overlaid on a 2D background.

One potential application of the technology is a head-mounted display with which viewers look on their surroundings with a sense of augmented reality. Google, for instance, has recently announced its intention to introduce glasses that can superimpose internet data on part of a transparent medium. Virtual 3D images could thus be superimposed if the glasses are modified to accommodate a 3D display.

Display technology, of course, goes beyond television, movies, and the internet. In the near future, 3D displays are expected to influence a wide range of applications, such as medical imaging, education, and virtual exhibitions. The use of 3D displays in surgeries, for example, can improve the precise structural perception often needed to operate on a body crowded with organs and tissues. Imaging and 3D display techniques are already used during laparoscopic surgeries, and efforts are being made to apply them to surgical training, planning, and evaluation.

In a 2010–11 European project known as Learning in Future Education 1, or LiFE 1, researchers examined the effectiveness of 3D content as a teaching tool in classrooms across seven countries. Two groups of students, one learning with 2D content and the other learning the same concepts using 3D content, were tested before and after the lessons. The results of the study indicated that some 86% of the students in the 3D classes improved from pretest to posttest, compared with 52% who improved in the 2D classes.

Display applications are also finding their way into museums. The Louvre, the most visited art museum in the world, formed a partnership with Nintendo in 2012 and adapted an electronic guide, originally designed for handheld entertainment systems, to generate 3D images and animations of the artworks and an interactive map of the museum.¹⁷ 

Nonetheless, 3D display technologies that provide all of nature’s depth cues with high enough image quality to merit widespread commercialization remain out of reach. What’s required is more research, accompanied by breakthroughs in materials science, optics, electronics, computer engineering, and vision science.

Byoungho Lee is a professor of electrical engineering at Seoul National University in Seoul, South Korea.