Particle image velocimetry and particle tracking velocimetry have played pivotal roles in flow and particle characterization, owing to their non-invasive and accurate data collection methods. However, their broader application in the biomicrofluidics field is constrained by challenges, such as intensive calibration, high post-processing costs, and optical compatibility issues, especially in settings where space is a bottleneck. This article describes recent advancements in non-iterative ray tracing that promise more streamlined post-capture calibration and highlights examples of applications and areas that merit further technological investigation. The development and adoption of these techniques may pave the way for new innovations.

In the realm of fluid mechanics, particularly within the biomicrofluidics area, progress may be significantly accelerated with advancements in image-based 3D reconstruction methodologies. Particle Image Velocimetry (PIV) and Particle Tracking Velocimetry (PTV) and variants offer solid foundations. Their prominence within the scientific community is given by their non-invasive capabilities, accuracy, and adaptability. However, inherent challenges associated with calibration complexity, extensive post-processing, and optical compatibility underscore the need for further innovative solutions to optimize these methodologies.

Advancements in microlens technology and high-resolution camera sensors have led to the emergence of innovative techniques for flow characterization and particle tracking using a singular light field (LF) camera.1–6 This approach is especially suitable for biomicrofluidics, presenting numerous avenues for technological refinement. Enhancements in optics, computational vision, mathematical modeling, parallel computing, and artificial intelligence are paving the way for advanced real-time diagnostic systems tailored for general configurations.

In particular, integrating deep learning and artificial intelligence is expected to particularly contribute in the context of 3D tracking. Leveraging datasets PIV and PTV, neural networks can optimize and refine the 3D particle tracking. Traditional calibration methods, often time-consuming and prone to error from various sources, can be streamlined with the assistance of these networks. With the predictive power of artificial intelligence, it becomes feasible to foresee and improve calibration challenges in real time, enabling a more efficient and accurate tracking process.

Possibilities are vast and may reach new horizons. For instance, as our understanding of molecular dynamics grows, it is possible to have real-time 3D tracking of DNA transcription using fluorescent fusions, elevating our grasp on processes at the core of life.7,8 Such advancements would uncover DNA mechanisms and could speed up progress in drug development, genetic modification, and our knowledge of diseases. The potential for these 3D reconstructions will extend to multi-cellular interactions during crucial biological events. Visualizing, characterizing, and quantifying in 3D how cells behave during tissue regeneration or in the onset of diseases can be a game-changer, furthering our knowledge and refining therapeutic strategies.9,10 As the convergence of technology continues, 3D bioprinting has a special place. With real-time feedback from advanced image-based 3D reconstructions, the production of functional tissues, organs, or complex cellular formations could be expedited.11 

Although existing biomicrofluidics tools have led to significant progress, integrating new technologies hints at a future filled with untapped potential and increased accuracy.

Illumination constraints and space limitations frequently hinder traditional 3D photogrammetry methods, especially in small-scale biological experiments. To address these challenges, alternative methodologies have surfaced in recent years. These encompass holographic,12–15 confocal,16,17 stereoscopic,18,19 defocus-based,20,21 active scanning,22,23 and structured volume illumination techniques.24 The evolution in microlens production, paired with innovations in camera sensor technology, has fostered the emergence of light field (LF) technology in 3D photogrammetry.25 Also, LF photogrammetry has seen integration with other methods, such as PIV1 for three-dimensional flow measurements. Unlike other techniques that require customized illumination setups, complex components, or synchronized operations between cameras and specific illumination sources, LF 3D photogrammetry has unique advantages. It is pre-calibrated, exhibits versatility, and is cost-efficient, aligning with a wide spectrum of optical configurations.

Despite its potential, the LF approach has not been widely adopted for 3D reconstruction in the micro-biological realm, due to challenges in implementation and operation. Traditional post-processing techniques associated with this method can be computationally intensive. Also, an absence of a clear, standardized protocol, may make it not advantageous. This complexity arises as these methods require adjustments tailored to the specific computational capacities at hand.

At its core, the 3D reconstruction process aims to recover the 3D intensity distribution E ( x , y , z ),
j N i w i , j E ( x j , y j , z j ) = I ( x i , y i ) .
(1)
Here, N i represents the count of voxels that fall within the line-of-sight of the ith pixel. The coefficient w i , j is a weighting factor, describing the relationship between the 3D voxels and their 2D pixel counterparts and I ( x , y ) denotes a 2D image. One of the prevalent methods to tackle this inherently complex problem is the Multiplicative Algebraic Reconstruction Technique (MART). Belden et al.26 pioneered this approach, which was initially used in computed tomography (CT) to reconstruct 3D surfaces from a series of angular projections due to its strong performance in background reconstruction and boundary handling. MART uses these 2D angular projections and correspondence information, iteratively decoding them to obtain 3D structural information as follows:
E ( x j , y j , z j ) k + 1 = E ( x j , y j , z j ) k ( I ( x i , y i ) j N j w i , j E ( x j , y j , z j ) k ) μ w i , j .
(2)

Owing to the considerable data size of w i , j (may exceed 100 GB), the process of iterative solving proves to be laborious. This remains true even when leveraging optimized tile loading via a RAID 0 solid-state disk configuration. Also, precise estimation of the parameter w, requiring pixel-level accuracy in the calibration phase, introduces an additional challenge. This complexity is a barrier to the broader application of LF photogrammetry in practical scenarios.

To expedite computation, the MART method has been enhanced through the incorporation of dense ray tracing-based MART2 and the pre-recognition simultaneous algebraic reconstruction technique (PR-SART),27 which exhibits greater resilience to noise and quicker convergence. Levoy et al.25 and Truscott et al.4 used a 3D deconvolution algorithm based on an estimated point-spread-function (PSF) approach. Compared to algebraic techniques, this 3D deconvolution is more efficient, as it avoids the need to store and manipulate extensive weighting matrices.

Within this framework, we introduced a non-iterative ray tracing technique, complemented by a robust post-capture calibration process; see specific details described in Hong and Chamorro.6 Unlike traditional methods of 3D intensity retrieval, our proposed approach streamlines the process, achieving near-instantaneous reconstruction. This is made possible by efficiently managing sparse cloud points generated via the ray counter matrix, as illustrated in Fig. 1. Consequently, the exhaustive computations conventionally associated with the weight matrix w and iterative procedures are no longer necessary. In our approach, ray tracing hinges on the two-point method, which uses pixels and their corresponding lenslet centers. Remarkably, this entire process is amenable to parallelization on a GPU, allowing for a fast reconstruction speed of roughly 100 ms for each frame.

FIG. 1.

Basic schematics of a 3D light ray diagram of an image volume around the microlens array (MLA).

FIG. 1.

Basic schematics of a 3D light ray diagram of an image volume around the microlens array (MLA).

Close modal

The discussed framework holds particular promise for probing the motion and dynamics of micro-organisms,28 an area of paramount importance in biotechnological and environmental areas. The capability of real-time reconstruction and tracking allows for visualizing, and analyzing complex swimming patterns, trajectory shifts, collective dispersion, and accelerations in reaction to various triggers or environmental changes. Examples include behavior exhibited by, e.g., phototrophs and chemotrophs.29 

Optoelectronic tweezers (OET)30,31 is a valuable non-contact micromanipulation technology, adept at governing the behavior of microparticles and cells.32 The main advantage of OET is its accuracy, allowing the efficient handling of particles, cells, and other small entities without direct physical interaction or the intervention of mechanical apparatuses. Incorporating real-time 3D tracking further augments their efficacy.

Microrobot swarms may offer precise delivery mechanisms. A micro-scale robot, operating within a group, can be directed to specific tasks or locations, making it a critical tool for regulated operations. The ability to track three-dimensional positions in real-time amplifies the potential of microrobots in biological microfluid environments. Such real-time tracking is pivotal when these microrobots are exposed to external forces; they include magnetic pulls, minor acoustic wave pressure, and electric fields. Knowing microrobot trajectories may also inform instantaneous adjustments, refining their motion patterns or strategies in real time. Real-time 3D tracking can provide immediate feedback in a robot that deviates from its intended course, allowing timely corrective measures. Wang and Zhang33 explored the dynamics of such systems, highlighting the benefits and the challenges that remain in harnessing the full potential of microrobot swarms.

Microfluidic devices have emerged at the forefront of innovative biomedical applications, pushing the boundaries of precision medicine and personalized therapies. When considering their potential in fields, such as the production of therapeutic proteins,34 cultivating cells in a 3D environment,35 and developing advanced systems for drug delivery,36 the breadth and depth of their utility are clear. At the heart of maximizing the potential of these devices is the understanding and control of fluid dynamics within their networks. This allows for understanding flow patterns and how drug particles, RNA, DNA, and other key players behave within the confined, controlled spaces of microchannels.

Employing microfluidic technologies in tandem with real-time tracking and management augments the potential for refined precision in both biomedical research and therapeutic pursuits, among others. For example, real-time insight into mixing efficiency can facilitate optimal reactions and interactions.

The integration of ray tracing techniques with the bio-microfluidics area signals a paradigm shift in biomechanics and fluid research. By addressing the inherent challenges faced in traditional Particle Image Velocimetry (PIV) and Particle Tracking Velocimetry (PTV) approaches, the approach discussed streamlines research processes and opens avenues for a deeper, and broad range of investigations.

The ability to provide real-time, precise 3D data expedites research and potentially enables new applications. The applications are multifaceted and transformative, from understanding microorganism behavior to refining the precision and efficiency of optoelectronic tweezers. In the realm of microrobotics, real-time 3D positioning paves the way for enhancing control and maneuverability. Such advancements can greatly influence fields from medicine to environmental research, showcasing the vast potential of synergies. The implications for microfluidic device design are vast. As real-time particle tracking unveils in-depth insights into fluid dynamics, it allows for more efficient, adaptive, and versatile microfluidic systems. Such enhancements have far-reaching implications in drug delivery, therapeutic protein production, and even 3D cell cultures with the possibility of active control and adaptation when needed.

Finally, it is worth stressing that ray tracing’s integration into bio-microfluidic systems may shape the trajectory of biomicrofluidic research.

The authors have no conflicts to disclose.

Liu Hong: Conceptualization (equal); Formal analysis (lead); Methodology (equal); Software (lead); Writing – original draft (equal); Writing – review & editing (supporting). Leonardo P. Chamorro: Conceptualization (equal); Funding acquisition (lead); Methodology (equal); Supervision (lead); Writing – original draft (equal); Writing – review & editing (lead).

Data sharing is not applicable to this article as no new data were created or analyzed in this study.

1.
K.
Lynch
,
T.
Fahringer
, and
B.
Thurow
, “Three-dimensional particle image velocimetry using a plenoptic camera,” in 50th AIAA Aerospace Sciences Meeting Including the New Horizons Forum and Aerospace Exposition (AIAA, 2012), p. 1056.
2.
S.
Shi
,
J.
Ding
,
T. H.
New
, and
J.
Soria
, “
Light-field camera-based 3D volumetric particle image velocimetry with dense ray tracing reconstruction technique
,”
Exp. Fluids
58
,
78
(
2017
).
3.
T. W.
Fahringer
,
K. P.
Lynch
, and
B. S.
Thurow
, “
Volumetric particle image velocimetry with a single plenoptic camera
,”
Meas. Sci. Technol.
26
,
115201
(
2015
).
4.
T. T.
Truscott
,
J.
Belden
,
R.
Ni
,
J.
Pendlebury
, and
B.
McEwen
, “
Three-dimensional microscopic light field particle image velocimetry
,”
Exp. Fluids
58
,
16
(
2017
).
5.
L.
Cao
,
B.
Zhang
,
M. M.
Hossain
,
J.
Li
, and
C.
Xu
, “
Tomographic reconstruction of light field PIV based on a backward ray-tracing technique
,”
Meas. Sci. Technol.
32
,
044007
(
2021
).
6.
L.
Hong
and
L. P.
Chamorro
, “
A fast, non-iterative ray-intersection approach for three-dimensional microscale particle tracking
,”
Lab Chip
22
,
964
971
(
2022
).
7.
M.
Stracy
,
S.
Uphoff
,
F. G.
de Leon
, and
A. N.
Kapanidis
, “
In vivo single-molecule imaging of bacterial DNA replication, transcription, and repair
,”
FEBS Lett.
588
,
3585
3594
(
2014
).
8.
A.
Robinson
and
A. M.
Van Oijen
, “
Bacterial replication, transcription and translation: Mechanistic insights from single-molecule biochemical studies
,”
Nat. Rev. Microbiol.
11
,
303
315
(
2013
).
9.
P.
Friedl
and
D.
Gilmour
, “
Collective cell migration in morphogenesis, regeneration and cancer
,”
Nat. Rev. Mol. Cell Biol.
10
,
445
457
(
2009
).
10.
S.
Even-Ram
and
K. M.
Yamada
, “
Cell migration in 3D matrix
,”
Curr. Opin. Cell Biol.
17
,
524
532
(
2005
).
11.
S. V.
Murphy
and
A.
Atala
, “
3D bioprinting of tissues and organs
,”
Nat. Biotechnol.
32
,
773
785
(
2014
).
12.
J.
Sheng
,
E.
Malkiel
, and
J.
Katz
, “
Digital holographic microscope for measuring three-dimensional particle distributions and motions
,”
Appl. Opt.
45
,
3893
3901
(
2006
).
13.
J.
Soria
and
C.
Atkinson
, “
Towards 3C-3D digital holographic fluid velocity vector field measurement’ tomographic digital holographic PIV (Tomo-HPIV)
,”
Meas. Sci. Technol.
19
,
074002
(
2008
).
14.
P.
Memmolo
,
A.
Finizio
,
M.
Paturzo
,
L.
Miccio
, and
P.
Ferraro
, “
Twin-beams digital holography for 3D tracking and quantitative phase-contrast microscopy in microfluidics
,”
Opt. Express
19
,
25833
25842
(
2011
).
15.
M. U.
Daloglu
,
W.
Luo
,
F.
Shabbir
,
F.
Lin
,
K.
Kim
,
I.
Lee
,
J.-Q.
Jiang
,
W.-J.
Cai
,
V.
Ramesh
,
M.-Y.
Yu
et al., “
Label-free 3D computational imaging of spermatozoon locomotion, head spin and flagellum beating over a large volume
,”
Light: Sci. Appl.
7
,
17121
(
2018
).
16.
H.
Kinoshita
,
S.
Kaneda
,
T.
Fujii
, and
M.
Oshima
, “
Three-dimensional measurement and visualization of internal flow of a moving droplet using confocal micro-PIV
,”
Lab Chip
7
,
338
346
(
2007
).
17.
R.
Lima
,
S.
Wada
,
S.
Tanaka
,
M.
Takeda
,
T.
Ishikawa
,
K.-I.
Tsubota
,
Y.
Imai
, and
T.
Yamaguchi
, “
In vitro blood flow in a rectangular PDMS microchannel: Experimental observations using a confocal micro-PIV system
,”
Biomed. Microdevices
10
,
153
167
(
2008
).
18.
R.
Lindken
,
J.
Westerweel
, and
B.
Wieneke
, “
Stereoscopic micro particle image velocimetry
,”
Exp. Fluids
41
,
161
171
(
2006
).
19.
M.
Bown
,
J.
MacInnes
,
R.
Allen
, and
W.
Zimmerman
, “
Three-dimensional, three-component velocity measurements using stereoscopic micro-PIV and PTV
,”
Meas. Sci. Technol.
17
,
2175
(
2006
).
20.
S. Y.
Yoon
and
K. C.
Kim
, “
3D particle position and 3D velocity field measurement in a microvolume via the defocusing concept
,”
Meas. Sci. Technol.
17
,
2897
(
2006
).
21.
F.
Pereira
,
J.
Lu
,
E.
Castano-Graff
, and
M.
Gharib
, “
Microscale 3D flow mapping with μDDPIV
,”
Exp. Fluids
42
,
589
599
(
2007
).
22.
C.
Brücker
, “
3D scanning PIV applied to an air flow in a motored engine using digital high-speed video
,”
Meas. Sci. Technol.
8
,
1480
(
1997
).
23.
C.
Brücker
,
D.
Hess
, and
J.
Kitzhofer
, “
Single-view volumetric PIV via high-resolution scanning, isotropic voxel restructuring and 3D least-squares matching (3D-LSM)
,”
Meas. Sci. Technol.
24
,
024001
(
2013
).
24.
A.
Aguirre-Pablo
,
A. B.
Aljedaani
,
J.
Xiong
,
R.
Idoughi
,
W.
Heidrich
, and
S. T.
Thoroddsen
, “
Single-camera 3D PTV using particle intensities and structured light
,”
Exp. Fluids
60
,
25
(
2019
).
25.
M.
Levoy
,
R.
Ng
,
A.
Adams
,
M.
Footer
, and
M.
Horowitz
, “Light field microscopy,” in ACM SIGGRAPH 2006 Papers (ACM, 2006), pp. 924–934.
26.
J.
Belden
,
T. T.
Truscott
,
M. C.
Axiak
, and
A. H.
Techet
, “
Three-dimensional synthetic aperture particle image velocimetry
,”
Meas. Sci. Technol.
21
,
125403
(
2010
).
27.
X.
Zhu
,
Z.
Wu
,
J.
Li
,
B.
Zhang
, and
C.
Xu
, “
A pre-recognition SART algorithm for the volumetric reconstruction of the light field PIV
,”
Opt. Laser Eng.
143
,
106625
(
2021
).
28.
S. J.
Lee
,
T.
Go
, and
H.
Byeon
, “
Three-dimensional swimming motility of microorganism in the near-wall region
,”
Exp. Fluids
57
,
26
(
2016
).
29.
R.
Clarke
, “Microorganisms and their response to stimuli,” in Modeling of Microscale Transport in Biological Processes (Elsevier, 2017), pp. 171–206.
30.
P. Y.
Chiou
,
A. T.
Ohta
, and
M. C.
Wu
, “
Massively parallel manipulation of single cells and microparticles using optical images
,”
Nature
436
,
370
372
(
2005
).
31.
S.-M.
Yang
,
T.-M.
Yu
,
H.-P.
Huang
,
M.-Y.
Ku
,
L.
Hsu
, and
C.-H.
Liu
, “
Dynamic manipulation and patterning of microparticles and cells by using TiOPc-based optoelectronic dielectrophoresis
,”
Opt. Lett.
35
,
1959
1961
(
2010
).
32.
S.
Liang
,
C.
Gan
,
Y.
Dai
,
C.
Zhang
,
X.
Bai
,
S.
Zhang
,
A. R.
Wheeler
,
H.
Chen
, and
L.
Feng
, “
Interaction between positive and negative dielectric microparticles/microorganism in optoelectronic tweezers
,”
Lab Chip
21
,
4379
4389
(
2021
).
33.
Q.
Wang
and
L.
Zhang
, “
External power-driven microrobotic swarm: From fundamental understanding to imaging-guided delivery
,”
ACS Nano
15
,
149
174
(
2021
).
34.
V.
Georgi
,
L.
Georgi
,
M.
Blechert
,
M.
Bergmeister
,
M.
Zwanzig
,
D. A.
Wüstenhagen
,
F. F.
Bier
,
E.
Jung
, and
S.
Kubick
, “
On-chip automation of cell-free protein synthesis: New opportunities due to a novel reaction mode
,”
Lab Chip
16
,
269
281
(
2016
).
35.
S.
Sarkar
,
N.
Cohen
,
P.
Sabhachandani
, and
T.
Konry
, “
Phenotypic drug profiling in droplet microfluidics for better targeting of drug-resistant tumors
,”
Lab Chip
15
,
4441
4450
(
2015
).
36.
D.
Liu
,
H.
Zhang
,
F.
Fontana
,
J. T.
Hirvonen
, and
H. A.
Santos
, “
Microfluidic-assisted fabrication of carriers for controlled drug delivery
,”
Lab Chip
17
,
1856
1883
(
2017
).