Deep neural networks are rapidly emerging as data analysis tools, often outperforming the conventional techniques used in complex microfluidic systems. One fundamental analysis frequently desired in microfluidic experiments is counting and tracking the droplets. Specifically, droplet tracking in dense emulsions is challenging due to inherently small droplets moving in tightly packed configurations. Sometimes, the individual droplets in these dense clusters are hard to resolve, even for a human observer. Here, two deep learning-based cutting-edge algorithms for object detection [you only look once (YOLO)] and object tracking (DeepSORT) are combined into a single image analysis tool, DropTrack, to track droplets in the microfluidic experiments. DropTrack analyzes input microfluidic experimental videos, extracts droplets' trajectories, and infers other observables of interest, such as droplet numbers. Training an object detector network for droplet recognition with manually annotated images is a labor-intensive task and a persistent bottleneck. In this work, this problem is partly resolved by training many object detector networks (YOLOv5) with several hybrid datasets containing real and synthetic images. We present an analysis of a double emulsion experiment as a case study to measure DropTrack's performance. For our test case, the YOLO network trained by combining 40% real images and 60% synthetic images yields the best accuracy in droplet detection and droplet counting in real experimental videos. Also, this strategy reduces labor-intensive image annotation work by 60%. DropTrack's performance is measured in terms of mean average precision of droplet detection, mean squared error in counting the droplets, and image analysis speed for inferring droplets' trajectories. The fastest configuration of DropTrack can detect and track the droplets at approximately 30 frames per second, well within the standards for a real-time image analysis.
References
1.
S.
Succi
and P. V.
Coveney
, “Big data: The end of the scientific method?
,” Philos. Trans. R. Soc. A: Math. Phys. Eng. Sci.
377
, 20180145
(2019
).2.
G.
Riva
, B. K.
Wiederhold
, and S.
Succi
, “Zero sales resistance: The dark side of big data and artificial intelligence
,” Cyberpsychol. Behav. Social Networking
25
, 169
–173
(2022
).3.
Y.
LeCun
, Y.
Bengio
, and G.
Hinton
, “Deep learning
,” Nature
521
, 436
–444
(2015
).4.
Darmatasia
and M. I.
Fanany
, “Handwriting recognition on form document using convolutional neural network and support vector machines (CNN-SVM)
,” in Proceedings of the 5th International Conference on Information and Communication Technology (ICoIC7)
(IEEE
, 2017
), pp. 1
–6
.5.
S.
Ahlawat
, A.
Choudhary
, A.
Nayyar
, S.
Singh
, and B.
Yoon
, “Improved handwritten digit recognition using convolutional neural networks (CNN)
,” Sensors
20
, 3344
(2020
).6.
N. H.
Tandel
, H. B.
Prajapati
, and V. K.
Dabhi
, “Voice recognition and voice comparison using machine learning techniques: A survey
,” in Proceedings of the 6th International Conference on Advanced Computing and Communication Systems (ICACCS)
(IEEE
, 2020
), pp. 459
–465
.7.
K.
Han
, D.
Yu
, and I.
Tashev
, “Speech emotion recognition using deep neural network and extreme learning machine
,” in Proceedings of the Interspeech
(2014
).8.
A.
Severyn
and A.
Moschitti
, “UNITN: Training deep convolutional neural network for twitter sentiment classification
,” in Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval)
(Association for Computational Linguistics
, 2015
), pp. 464
–469
.9.
A. M.
Ramadhani
and H. S.
Goo
, “Twitter sentiment analysis using deep learning methods
,” in Proceedings of the 7th International Annual Engineering Seminar (InAES)
(IEEE
, 2017
), pp. 1
–4
.10.
L.
Zhang
, S.
Wang
, and B.
Liu
, “Deep learning for sentiment analysis: A survey
,” WIREs Data Min. Knowl. Discovery
8
, e1253
(2018
).11.
T.
Osman
, S. S.
Psyche
, J. M.
Shafi Ferdous
, and H. U.
Zaman
, “Intelligent traffic management system for cross section of roads using computer vision
,” in Proceedings of the IEEE 7th Annual Computing and Communication Workshop and Conference (CCWC)
(IEEE
, 2017
), pp. 1
–7
.12.
G. T. S.
Ho
, Y. P.
Tsang
, C. H.
Wu
, W. H.
Wong
, and K. L.
Choy
, “A computer vision-based roadside occupation surveillance system for intelligent transport in smart cities
,” Sensors
19
, 1796
(2019
).13.
B.
Yogameena
and C.
Nagananthini
, “Computer vision based crowd disaster avoidance system: A survey
,” Int. J. Disaster Risk Reduct.
22
, 95
–129
(2017
).14.
N.
Ragesh
, B.
Giridhar
, D.
Lingeshwaran
, P.
Siddharth
, and K. P.
Peeyush
, “Deep learning based automated billing cart
,” in Proceedings of the International Conference on Communication and Signal Processing (ICCSP)
(IEEE
, 2019
), pp. 0779
–0782
.15.
N.
Kühl
, M.
Goutier
, L.
Baier
, C.
Wolff
, and D.
Martin
, “Human vs. supervised machine learning: Who learns patterns faster?
,” arXiv:2012.03661 (2020
).16.
A. W.
Senior
, R.
Evans
, J.
Jumper
, J.
Kirkpatrick
, L.
Sifre
, T.
Green
, C.
Qin
, A.
Žídek
, A. W. R.
Nelson
, A.
Bridgland
, H.
Penedones
, S.
Petersen
, K.
Simonyan
, S.
Crossan
, P.
Kohli
, D. T.
Jones
, D.
Silver
, K.
Kavukcuoglu
, and D.
Hassabis
, “Improved protein structure prediction using potentials from deep learning
,” Nature
577
, 706
–710
(2020
).17.
G.
Carleo
, I.
Cirac
, K.
Cranmer
, L.
Daudet
, M.
Schuld
, N.
Tishby
, L.
Vogt-Maranto
, and L.
Zdeborová
, “Machine learning and the physical sciences
,” Rev. Mod. Phys.
91
, 045002
(2019
).18.
L.
Yu
, B.
Liu
, Y.
Zhu
, R.-R.
Chen
, H.-W.
Xi
, C.-J.
Jin
, and B.
Peng
, “A deep learning framework for square kilometre array science data challenge 1
,” Mon. Not. R. Astron. Soc.
511
, 4305
–4315
(2022
).19.
C. N.
Coelho
, A.
Kuusela
, S.
Li
, H.
Zhuang
, J.
Ngadiuba
, T. K.
Aarrestad
, V.
Loncar
, M.
Pierini
, A. A.
Pol
, and S.
Summers
, “Automatic heterogeneous quantization of deep neural networks for low-latency inference on the edge for particle detectors
,” Nat. Mach. Intell.
3
, 675
–686
(2021
).20.
J.
Thiyagalingam
, M.
Shankar
, G.
Fox
, and T.
Hey
, “Scientific machine learning benchmarks
,” Nat. Rev. Phys.
4
, 413
(2022
).21.
B.
Liu
, J.
Tang
, H.
Huang
, and X.-Y.
Lu
, “Deep learning methods for super-resolution reconstruction of turbulent flows
,” Phys. Fluids
32
, 025105
(2020
).22.
C.
Rao
, H.
Sun
, and Y.
Liu
, “Physics-informed deep learning for incompressible laminar flows
,” Theor. Appl. Mech. Lett.
10
, 207
–212
(2020
).23.
X.
Jin
, S.
Cai
, H.
Li
, and G. E.
Karniadakis
, “NSFnets (Navier-Stokes flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations
,” J. Comput. Phys.
426
, 109951
(2021
).24.
L.
Sun
, H.
Gao
, S.
Pan
, and J.-X.
Wang
, “Surrogate modeling for fluid flows based on physics-constrained deep learning without simulation data
,” Comput. Methods Appl. Mech. Eng.
361
, 112732
(2020
).25.
D.
Kochkov
, J. A.
Smith
, A.
Alieva
, Q.
Wang
, M. P.
Brenner
, and S.
Hoyer
, “Machine learning-accelerated computational fluid dynamics
,” Proc. Natl. Acad. Sci.
118
, e2101784118
(2021
).26.
S. L.
Brunton
, B. R.
Noack
, and P.
Koumoutsakos
, “Machine learning for fluid mechanics
,” Annu. Rev. Fluid Mech.
52
, 477
–508
(2020
).27.
P.
Hadikhani
, N.
Borhani
, S.
Hashemi
, and D.
Psaltis
, “Learning from droplet flows in microfluidic channels using deep neural networks
,” Sci. Rep.
9
, 8114
(2019
).28.
Y.
Mahdi
and K.
Daoud
, “Microdroplet size prediction in microfluidic systems via artificial neural network modeling for water-in-oil emulsion formulation
,” J. Dispersion Sci. Technol.
38
, 1501
–1508
(2017
).29.
J. W.
Khor
, N.
Jean
, E. S.
Luxenberg
, S.
Ermon
, and S. K. Y.
Tang
, “Using machine learning to discover shape descriptors for predicting emulsion stability in a microfluidic channel
,” Soft Matter
15
, 1361
–1372
(2019
).30.
A. D.
Bick
, J. W.
Khor
, Y.
Gai
, and S. K. Y.
Tang
, “Strategic placement of an obstacle suppresses droplet break up in the hopper flow of a microfluidic soft crystal
,” Proc. Natl. Acad. Sci. U.S.A.
118
, e2017822118
(2021
).31.
P.
Marmottant
and J. P.
Raven
, “Microfluidics with foam
,” Soft Matter
5
, 3385
–3388
(2009
).32.
D. R.
Link
, S. L.
Anna
, D. A.
Weitz
, and H. A.
Stone
, “Geometrically mediated breakup of drops in microfluidic devices
,” Phys. Rev. Lett.
92
, 054503
(2004
).33.
A.
Montessori
, A.
Tiribocchi
, M.
Lauricella
, F.
Bonaccorso
, and S.
Succi
, “Mesoscale modelling of droplets' self-assembly in microfluidic channels
,” Soft Matter
17
, 2374
–2383
(2021
).34.
P.
Zhu
and L.
Wang
, “Passive and active droplet generation with microfluidics: A review
,” Lab Chip
17
, 34
–75
(2017
).35.
R.
Zamboni
, A.
Zaltron
, M.
Chauvet
, and C.
Sada
, “Real-time precise microfluidic droplets label-sequencing combined in a velocity detection sensor
,” Sci. Rep.
11
, 17987
(2021
).36.
J.
de Novais Schianti
, A. L. C.
Serrano
, D.
Orquiza de Carvalho
, R. A.
Penchel
, J.
Mota Pinheiro
, M. R.
Gongora-Rubio
, and G.
Pamplona Rehder
, “Novel platform for droplet detection and size measurement using microstrip transmission lines
,” Sensors
19
, 5216
(2019
).37.
J.
Chen
, K.
Venkataraman
, D.
Bakin
, B.
Rodricks
, R.
Gravelle
, P.
Rao
, and Y.
Ni
, “Digital camera imaging system simulation
,” IEEE Trans. Electron Devices
56
, 2496
–2505
(2009
).38.
A.
Montessori
, A.
Tiribocchi
, M.
Bogdan
, F.
Bonaccorso
, M.
Lauricella
, J.
Guzowski
, and S.
Succi
, “Translocation dynamics of high-internal phase double emulsions in narrow channels
,” Langmuir
37
, 9026
–9033
(2021
).39.
A.
Montessori
, M. L.
Rocca
, P.
Prestininzi
, A.
Tiribocchi
, and S.
Succi
, “Deformation and breakup dynamics of droplets within a tapered channel
,” Phys. Fluids
33
, 082008
(2021
).40.
J.
Tao
, H.
Wang
, X.
Zhang
, X.
Li
, and H.
Yang
, “An object detection system based on YOLO in traffic scene
,” in Proceedings of the 6th International Conference on Computer Science and Network Technology (ICCSNT)
(IEEE
, 2017
), pp. 315
–319
.41.
J.-P.
Lin
and M.-T.
Sun
, “A YOLO-based traffic counting system
,” in Proceedings of the Conference on Technologies and Applications of Artificial Intelligence (TAAI)
(IEEE
, 2018
), pp. 82
–85
.42.
M.
Durve
, F.
Bonaccorso
, A.
Montessori
, M.
Lauricella
, A.
Tiribocchi
, and S.
Succi
, “Tracking droplets in soft granular flows with deep learning techniques
,” Eur. Phys. J. Plus
136
, 864
(2021
).43.
M.
Durve
, F.
Bonaccorso
, A.
Montessori
, M.
Lauricella
, A.
Tiribocchi
, and S.
Succi
, “A fast and efficient deep learning procedure for tracking droplet motion in dense microfluidic emulsions
,” Philos. Trans. R. Soc. A: Math. Phys. Eng. Sci.
379
, 20200400
(2021
).44.
R.
Benzi
, S.
Succi
, and M.
Vergassola
, “The lattice Boltzmann equation: Theory and applications
,” Phys. Rep.
222
, 145
–197
(1992
).45.
S.
Succi
, The Lattice Boltzmann Equation: For Fluid Dynamics and Beyond
(Oxford University Press
, 2001
).46.
S.
Succi
, The Lattice Boltzmann Equation: For Complex States of Flowing Matter
(Oxford University Press
, 2018
).47.
M.
Bogdan
, A.
Montessori
, A.
Tiribocchi
, F.
Bonaccorso
, M.
Lauricella
, L.
Jurkiewicz
, S.
Succi
, and J.
Guzowski
, “Stochastic jetting and dripping in confined soft granular flows
,” Phys. Rev. Lett.
128
, 128001
(2022
).48.
See http://host.robots.ox.ac.uk/pascal/VOC/ for The pascal visual object classes homepage.
49.
See http://cocodataset.org. for Coco dataset homepage.
50.
F.
Zhou
, H.
Zhao
, and Z.
Nie
, “Safety helmet detection based on YOLOv5
,” in Proceedings of the IEEE International Conference on Power Electronics, Computer Applications (ICPECA)
(IEEE
, 2021
), pp. 6
–11
.51.
L. C. M.
Junior
and J. A. C.
Ulson
, “Real time weed detection using computer vision and deep learning
,” in Proceedings of the 14th IEEE International Conference on Industry Applications (INDUSCON)
(IEEE
, 2021
), pp. 1131
–1137
.52.
R.
Xu
, H.
Lin
, K.
Lu
, L.
Cao
, and Y.
Liu
, “A forest fire detection system based on ensemble learning
,” Forests
12
, 217
(2021
).53.
J.
Yao
, J.
Qi
, J.
Zhang
, H.
Shao
, J.
Yang
, and X.
Li
, “A real-time detection algorithm for kiwifruit defects based on YOLOv5
,” Electronics
10
, 1711
(2021
).54.
I. S.
Walia
, D.
Kumar
, K.
Sharma
, J. D.
Hemanth
, and D. E.
Popescu
, “An integrated approach for monitoring social distancing and face mask detection using stacked resnet-50 and YOLOv5
,” Electronics
10
, 2996
(2021
).55.
J.
Redmon
, S.
Divvala
, R.
Girshick
, and A.
Farhadi
, “You only look once: Unified, real-time object detection
,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(IEEE
, 2016
), pp. 779
–788
.56.
57.
N.
Wojke
, A.
Bewley
, and D.
Paulus
, “Simple online and realtime tracking with a deep association metric
,” in proceedings of the IEEE International Conference on Image Processing (ICIP)
(IEEE
, 2017
), pp. 3645
–3649
.58.
A.
Bewley
, Z.
Ge
, L.
Ott
, F.
Ramos
, and B.
Upcroft
, “Simple online and realtime tracking
,” in Proceedings of the IEEE International Conference on Image Processing (ICIP)
(IEEE
, 2016
), pp. 3464
–3468
.59.
H. W.
Kuhn
, “The Hungarian method for the assignment problem
,” Naval Res. Logist. Q.
2
, 83
–97
(1955
).60.
R. E.
Kalman
, “A new approach to linear filtering and prediction problems
,” J. Basic Eng.
82
, 35
–45
(1960
).61.
F. S.
Saleh
, M. S.
Aliakbarian
, M.
Salzmann
, L.
Petersson
, and J. M.
Alvarez
, “Effective use of synthetic data for urban scene semantic segmentation
,” in Proceedings of the European Conference on Computer Vision (ECCV)
(2018
).62.
E.
Wood
, T.
Baltrušaitis
, C.
Hewitt
, S.
Dziadzio
, T. J.
Cashman
, and J.
Shotton
, “Fake it till you make it: Face analysis in the wild using synthetic data alone
,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)
(IEEE
, 2021
), pp. 3681
–3691
.63.
See https://roboflow.com/ for Roboflow homepage.
64.
M.
Choiński
, M.
Rogowski
, P.
Tynecki
, D. P. J.
Kuijper
, M.
Churski
, and J. W.
Bubnicki
, “A first step towards automated species recognition from camera trap images of mammals using AI in a European temperate forest
,” in Computer Information Systems and Industrial Management
, edited by K.
Saeed
and J.
Dvorský
(Springer International Publishing
, Cham
, 2021
), pp. 299
–310
.65.
P.
Henderson
and V.
Ferrari
, “End-to-end training of object class detectors for mean average precision
,” in Proceedings of the Computer Vision—ACCV 2016
(Springer International Publishing
, Cham
, 2017
), pp. 198
–213
.66.
H.
Rezatofighi
, N.
Tsoi
, J.
Gwak
, A.
Sadeghian
, I.
Reid
, and S.
Savarese
, “Generalized intersection over union: A metric and a loss for bounding box regression
,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
(2019
).67.
See https://github.com/ultralytics/yolov5 for YOLOv5 git repository.
68.
epochs = 500, batch size = 32, number of classes = 1, learning rate = 0.001, momentum = 0.95, decay = 0.001.
69.
M.
Broström
, see https://github.com/mikel-brostrom/Yolov5_DeepSort_Pytorch for “Real-Time Multi-Object Tracker using YOLOv5 and Deep Sort, 2020
.”70.
C.
Zeng
, H.
Bissig
, and A.
Dinsmore
, “Particles on droplets: From fundamental physics to novel materials
,” Solid State Commun.
139
, 547
–556
(2006
).© 2022 Author(s). Published under an exclusive license by AIP Publishing.
2022
Author(s)
You do not currently have access to this content.