Live imaging of cells is crucial for understanding microscopic cellular processes and dynamics. Frustratingly for biologists, taking high-resolution images can be challenging. Many cells have low contrast, and the boundaries of different cellular regions and the edges of cells themselves are often difficult or even impossible to discern.

Fluorescence imaging allows biologists to artificially add contrast to specific cellular components and observe them in more detail than with regular optical microscopy. Cell nuclei, microtubules, and other structures can be labeled with fluorescent dye molecules and then observed during dynamic processes such as cell division and organization.1 (See, for example, the Quick Study by Abhishek Kumar, Daniel Colón-Ramos, and Hari Shroff, Physics Today, July 2015, page 58.) Despite its strengths, fluorescence imaging is not a magic bullet: The laser light needed to excite the dye molecules can harm the cells, and the dye can absorb only a certain amount of light before it stops fluorescing. Imaging fluorescent samples always involves a tradeoff between how much light is used, the speed at which pictures are taken, and the spatial resolution of the resulting image.

Sometimes image quality has to be sacrificed for sample health or imaging speed. However, the images can still carry information that is not visible to the naked eye. Researchers then turn to postprocessing techniques—typically deconvolution and denoising algorithms that try to model and undo distortion from imaging—to improve the image quality.

In an effort to extend the limits of postprocessing techniques, Martin Weigert, Florian Jug, Loïc Royer, Eugene Myers, and coworkers at the Max Planck Institute of Molecular Cell Biology and Genetics in Dresden, Germany, have successfully demonstrated a new method based on an artificial neural network.2 They trained their content-aware image restoration (CARE) networks to decipher low-resolution fluorescent images and create accurate, high-resolution reconstructions. To achieve the best possible results, the networks are trained to incorporate information from high-resolution images of the same biological system.

The researchers found that information could be gleaned from previously unusable images, which were taken with 1/60 of the light required for a high-resolution image, after they were restored using CARE. The networks also fixed apparent stretching in three-dimensional data and identified features below the diffraction limit as effectively as a state-of-the-art superresolution imaging technique, but in a fraction of the time.

A CARE network uses deep learning to restore fluorescent images. Its input nodes are the pixels of the raw image; its output nodes are the pixels of the restored image. Unlike conventional machine learning, which requires the user to specify how a network should classify features, a deep-learning network identifies patterns itself directly from raw images.3 The trained network performs a series of operations on the input pixel values to highlight such important features as edge locations while suppressing unimportant variations in intensity.

Each operation a network performs has adjustable parameters that have to be optimized through training before the network can be used. That entails giving the network pairs of images—one low resolution and one high resolution—of the same area in a sample. The network processes the low-resolution image and outputs a restoration that is then compared with the high-resolution image. The network’s goal is for the restoration and high-resolution images to be the same. It adjusts its parameters to minimize their differences, which encodes into the network specific information about what the sample looks like. That information can then be used to better restore images.

The lack of training data has previously been a roadblock for applying machine learning to image-restoration tasks. Weigert and coworkers used two strategies to solve the problem. First, they chose a network structure that was developed for analyzing biomedical images and is known to require less training data than other networks.4 They then generated training data by imaging fixed, or nonliving, samples of biological systems. Fixed samples can be imaged at higher light intensities and lower frame rates than living samples can, so the researchers recorded pairs of high-resolution “ground-truth” images and low-resolution images of the same areas taken at conditions suitable for living organisms.

Once a CARE network has been trained on pairs of images, it can be applied to images of live samples for which there is no ground truth. Despite the name, restoration does not mean an image is returned to its previous condition. “There is an ideal image that is unfortunately not observable with the technology that we have, and we can only perceive something worse than that,” says Royer. “But that image exists in theory,” and that is what a CARE network aims to recover.

Figure 1 shows the result of applying CARE to images of the flatworm Schmidtea mediterranea, which is a model organism for studying tissue regeneration. Under even moderate amounts of laser light the worm flinches its muscles, and raw images taken at tolerable intensities have such a low signal-to-noise ratio that they could not previously be interpreted. The undetectable fluorescently labeled cell nuclei became easily discernible after the raw image was restored with CARE, and the improved image quality rivals that of the ground-truth image. The benefits of CARE can also be quantitative: When a CARE network was used to restore images of a red flour beetle embryo, the accuracy with which individual nuclei could be identified increased from 47% to 65%.

Figure 1.

Image restoration. (a) A trained deep-learning network can turn a raw, low-signal-to-noise image of flatworm cells into a restoration that has a high signal-to-noise ratio and is of the same quality as a real high-resolution, or ground-truth, image. (b) Zooming in on the inset in panel a shows that nuclei cannot be distinguished in the raw image, whereas they are clearly identifiable in the restored image. (Adapted from ref. 2.)

Figure 1.

Image restoration. (a) A trained deep-learning network can turn a raw, low-signal-to-noise image of flatworm cells into a restoration that has a high signal-to-noise ratio and is of the same quality as a real high-resolution, or ground-truth, image. (b) Zooming in on the inset in panel a shows that nuclei cannot be distinguished in the raw image, whereas they are clearly identifiable in the restored image. (Adapted from ref. 2.)

Close modal

CARE networks can be applied to a range of biological systems because the information they use to restore images comes from training data. The researchers demonstrated that versatility using images from eight systems, including fruit fly wings, zebrafish eyes, and rat secretory granules. Once the researchers realized how well the networks addressed the problem of low illumination, they wondered whether the networks could also solve other problems in fluorescence microscopy.

It is often useful for biologists to capture 3D pictures of their samples by taking 2D images at different depths and then stacking them to re-create the entire volume. However, a microscope’s resolution is always worse along the optical axis than in the imaging plane, so the axial images—2D images in which one dimension is in the imaging plane and the other is along the optical axis—appear elongated in that direction. “This is a fundamental problem in microscopy,” notes Royer. “You get beautiful 3D data sets, but when you rotate them, you realize that one dimension is very poor.”

Weigert and coworkers addressed that problem by applying CARE networks to volumetric data with poor axial resolution. Unlike with 2D image restoration, they couldn’t directly acquire high-resolution ground-truth axial images to train the networks because the poor resolution is inherent to the optical system. Instead, they generated training data by computationally modifying well-resolved lateral images to resemble the poorly resolved axial images. A CARE network trained on the modified data was able to restore nearly isotropic resolution and remove the apparent stretching in the axial slices, as shown in figure 2.

Figure 2.

Axial elongation. Three-dimensional images appear elongated along the optical axis because microscopes have lower resolution in that direction. A deep-learning network can be trained to turn a stretched axial slice of a developing zebrafish eye (top row) into a restored image with isotropic resolution (bottom row). (Adapted from ref. 2.)

Figure 2.

Axial elongation. Three-dimensional images appear elongated along the optical axis because microscopes have lower resolution in that direction. A deep-learning network can be trained to turn a stretched axial slice of a developing zebrafish eye (top row) into a restored image with isotropic resolution (bottom row). (Adapted from ref. 2.)

Close modal

Imaging nanometer-scale features is another challenge in microscopy. Superresolution techniques are needed to resolve objects that are smaller than the diffraction limit (see Physics Today, May 2015, page 14), but they usually entail low image-acquisition rates as many images are needed to generate one superresolution image. As a result, superresolution techniques cannot image fast-moving live samples. Weigert and coworkers saw the potential of applying CARE networks to the task: If the networks could restore structures below the diffraction limit, they could greatly increase the speed of superresolution imaging.

Such small objects, though, cannot be imaged directly to generate training data, so the researchers used simulation-generated images. That introduces a complication because it does not guarantee that the training data accurately represent the physics of the system: “If you get the training data wrong, you’re going to have some artifacts,” cautions Royer.

The researchers applied their superresolution technique to two types of structures: rat secretory granules, which are more or less spherical, and meshes of microtubules. In both cases the CARE network revealed substructures that had remained imperceptible after they were enhanced with a traditional deconvolution method of image restoration. For the microtubule sample, the CARE network resolved structures as effectively as a state-of-the-art superresolution technique but did the job 20 times as fast because it required fewer images.

Although their performance is impressive, CARE networks are only useful to the extent that their output is reliable. “For the scientific utility of the network, it is very important to know not only what is predicted but how accurate it is,” says Royer. To that end, the researchers changed the last step of the CARE network so that instead of just reporting a pixel value, it gave a probability distribution for each pixel whose mean was the predicted pixel value and whose width indicated the uncertainty in that prediction.

The consistency of the CARE networks also factored into their reliability measure. Instead of relying on a single network, the researchers trained an ensemble of networks; based on the restorations produced by each network, they calculated an ensemble disagreement value that quantified the networks’ confidence in the predicted pixel value, as shown in figure 3. The networks often—but not always—agreed on pixel values and had a low ensemble disagreement. The ability to identify areas of disagreement is crucial to CARE’s utility because those areas alert researchers to places where an image restoration may not be reliable.

Figure 3.

Assessing accuracy. The faithfulness of a restored image can be assessed by comparing restorations from an ensemble of networks. In each row, a raw image is used as the input for an ensemble of four trained CARE networks. The ensemble disagreement indicates how reliable the restoration is in a particular area, with brighter blue indicating a higher level of disagreement. The top row shows an area where the networks largely agreed on the restoration; the bottom row reveals a region with higher disagreement, which indicates that the restoration may not be accurate. (Adapted from ref. 2.)

Figure 3.

Assessing accuracy. The faithfulness of a restored image can be assessed by comparing restorations from an ensemble of networks. In each row, a raw image is used as the input for an ensemble of four trained CARE networks. The ensemble disagreement indicates how reliable the restoration is in a particular area, with brighter blue indicating a higher level of disagreement. The top row shows an area where the networks largely agreed on the restoration; the bottom row reveals a region with higher disagreement, which indicates that the restoration may not be accurate. (Adapted from ref. 2.)

Close modal

CARE networks have thus far outperformed other currently available image restoration methods for all tasks to which they have been applied. But Royer acknowledges that there is still room for progress: “There are scenarios where more research is needed to get a really secure, really robust estimate of where and when to trust the networks.”

1.
G.
Cox
,
Optical Imaging Techniques in Cell Biology
,
CRC Press
(
2017
).
3.
Y.
LeCun
,
Y.
Bengio
,
G.
Hinton
,
Nature
521
,
436
(
2015
).
4.
O.
Ronneberger
,
P.
Fischer
,
T.
Brox
, in
Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015: 18th International Conference, Munich, Germany, October 5–9, 2015, Proceedings, Part III
,
N.
Navab
et al., eds.,
Springer
(
2015
), p.
234
.
5.
A.
Kumar
,
D. A.
Colón-Ramos
,
H.
Shroff
,
Physics Today
68
(
7
),
58
(
2015
).
6.
J. L.
Miller
,
Physics Today
68
(
5
),
14
(
2015
).