It is very difficult for visually impaired people to perceive and avoid obstacles at a distance. To address this problem, the unified framework of multiple target detection, recognition, and fusion is proposed based on the sensor fusion system comprising a low-power millimeter wave (MMW) radar and an RGB-Depth (RGB-D) sensor. In this paper, the Mask R-CNN and the single shot multibox detector network are utilized to detect and recognize the objects from color images. The obstacles’ depth information is obtained from the depth images using the MeanShift algorithm. The position and velocity information on the multiple target is detected by the MMW radar based on the principle of a frequency modulated continuous wave. The data fusion based on the particle filter obtains more accurate state estimation and richer information by fusing the detection results from the color images, depth images, and radar data compared with using only one sensor. The experimental results show that the data fusion enriches the detection results. Meanwhile, the effective detection range is expanded compared to using only the RGB-D sensor. Moreover, the data fusion results keep high accuracy and stability under diverse range and illumination conditions. As a wearable system, the sensor fusion system has the characteristics of versatility, portability, and cost-effectiveness.

1.
R. R.
Bourne
,
S. R.
Flaxman
,
T.
Braithwaite
,
M. V.
Cicinelli
,
A.
Das
,
J. B.
Jonas
,
J.
Keeffe
,
J. H.
Kempen
,
J.
Leasher
,
H.
Limburg
, and
K.
Naidoo
,
Lancet Global Health
5
,
e888
(
2017
).
2.
K.
Yang
,
R.
Cheng
,
L. M.
Bergasa
,
E.
Romera
,
K.
Wang
, and
N.
Long
, in
2018 IEEE International Conference on Robotics and Biomimetics (ROBIO)
(
IEEE
,
2018
), pp.
1034
.
3.
A.
Bhowmick
and
S. M.
Hazarika
,
J. Multimodal User Interfaces
11
,
149
(
2017
).
4.
L.
Shao
,
J.
Han
,
P.
Kohli
, and
Z.
Zhang
,
Computer Vision and Machine Learning with RGB-D Sensors
(
Springer International Publishing
,
2014
).
5.
K.
Yang
,
K.
Wang
,
W.
Hu
, and
J.
Bai
,
Sensors
16
,
1954
(
2016
).
6.
K.
Yang
,
K.
Wang
,
R.
Cheng
,
W.
Hu
,
X.
Huang
, and
J.
Bai
,
Sensors
17
,
1890
(
2017
).
7.
K.
Yang
,
K.
Wang
,
X.
Zhao
,
R.
Cheng
,
J.
Bai
,
Y.
Yang
, and
D.
Liu
,
J. Ambient Intell. Smart Environ.
9
,
743
(
2017
).
8.
R.
Cheng
,
K.
Wang
,
K.
Yang
,
N.
Long
,
J.
Bai
, and
D.
Liu
,
Multimedia Tools and Applications
, (
Springer US
,
2017
), p.
1
.
9.
R.
Cheng
,
K.
Wang
,
K.
Yang
,
N.
Long
,
W.
Hu
,
H.
Chen
,
J.
Bai
, and
D.
Liu
,
J. Electron. Imaging
26
,
053025
(
2017
).
10.
K. W.
Lin
,
T. K.
Lau
,
C. M.
Cheuk
, and
Y.
Liu
, in
IEEE International Conference on Mechatronics and Automation
(
IEEE
,
2012
), p.
1423
.
11.
X.
Zhao
,
K.
Wang
,
K.
Yang
, and
W.
Hu
, in
International Conference on Graphics and Image Processing (ICGIP2016)
(
SPIE
,
2017
), p.
1022509
.
12.
A.
Tamjidi
,
C.
Ye
, and
S.
Hong
, in
2013 IEEE International Symposium on Robotic and Sensors Environments (ROSE)
(
IEEE
,
2013
), p.
178
.
13.
G. L.
Charvat
,
Small and Short-Range Radar Systems
(
CRC Press
,
2014
).
14.
N.
Long
,
K.
Wang
,
R.
Cheng
,
K.
Yang
, and
J.
Bai
,
Proc. SPIE
10800
,
108006
(
2018
).
15.
M.
Bertozzi
,
L.
Bombini
,
P.
Cerri
,
P.
Medici
,
P. C.
Antonello
, and
M.
Miglietta
, in
Proceedings of 2008 IEEE Intelligent Vehicles Symposium
(
IEEE
,
2008
), p.
608
.
16.
J.
Ćesić
,
I.
Marković
,
I.
Cvišić
, and
I.
Petrović
,
Robo. Auton. Syst.
83
,
338
(
2016
).
17.
E.
Cippitelli
,
F.
Fioranelli
,
E.
Gambi
, and
S.
Spinsante
,
IEEE Sens. J.
17
,
3585
(
2017
).
18.
J. H.
Kim
,
J. W.
Starr
, and
B. Y.
Lattimer
,
Fire Technol.
51
,
82
(
2015
).
19.
W.
Liu
,
D.
Anguelov
,
D.
Erhan
,
C.
Szegedy
,
S.
Reed
,
C. Y.
Fu
, and
A. C.
Berg
, in
European Conference on Computer Vision
(
IEEE
,
2016
), p.
21
.
20.
K.
He
,
G.
Gkioxari
,
P.
Dollár
, and
R.
Girshick
, in
Proceedings of the IEEE International Conference on Computer Vision
(
IEEE
,
2017
), p.
2961
.
21.
A. G.
Stove
,
IEEE Proc. F Radar Signal Process.
139
,
343
(
1992
).
22.
D.
Comaniciu
and
P.
Meer
,
IEEE Trans. Pattern Anal. Mach. Intell.
24
,
603
(
2002
).
23.
F.
Gustafsson
,
IEEE Aerosp. Electron. Syst. Mag.
25
,
53
(
2010
).
24.
A.
Jindal
,
N.
Aggarwal
, and
S.
Gupta
,
Pattern Recognit. Image Anal.
28
,
288
(
2018
).
25.
B.
Kaur
and
J.
Bhattacharya
,
J. Electron. Imaging
28
,
013031
(
2019
).
26.
T.
Kiuru
,
M.
Metso
,
M.
Utriainen
,
K.
Metsävainio
,
H. H.
Jauhonen
,
R.
Rajala
,
R.
Savenius
,
M.
Ström
,
T. N.
Jylhä
,
R.
Juntunen
, and
J.
Sylberg
,
Cogent Eng.
5
,
1450322
(
2018
).
27.
K.
Yang
,
K.
Wang
,
L. M.
Bergasa
,
E.
Romera
,
W.
Hu
,
D.
Sun
,
J.
Sun
,
R.
Cheng
,
T.
Chen
, and
E.
Lopez
,
Sensors
18
,
1506
(
2018
).
28.
A.
Aladren
,
G.
López-Nicolás
,
L.
Puig
,
L.
Puig
, and
J. J.
Guerrero
,
IEEE Syst. J.
10
,
922
(
2016
).
29.
S.
Sugimoto
,
H.
Tateda
,
H.
Takahashi
, and
M.
Okutomi
, in
Proceedings of the 17th International Conference on Pattern Recognition
(
IEEE
,
2004
), Vol. 3, p.
342
.
30.
T.
Wang
,
N.
Zheng
,
J.
Xin
, and
Z.
Ma
,
Sensors
11
,
8992
(
2011
).
31.
D. Y.
Kim
and
M.
Jeon
,
Inf. Sci.
278
,
641
(
2014
).
32.
Y.
Fang
,
I.
Masaki
, and
B.
Horn
,
IEEE Trans. Intell. Transp. Syst.
3
,
196
(
2002
).
33.
S.
Wu
,
S.
Decker
,
P.
Chang
,
T.
Camus
, and
J.
Eledath
,
IEEE Trans. Intell. Transp. Syst.
10
,
606
(
2009
).
34.
P.
Molchanov
,
S.
Gupta
,
K.
Kim
, and
K.
Pulli
, in
Proceedings of 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG)
(
IEEE
,
2015
), Vol. 1, p.
1
.
35.
M.
Obrvan
,
J.
Ćesić
, and
I.
Petrović
, in
Robot 2015: Second Iberian Robotics Conference
(
Springer, Cham
,
2015
), p.
437
.
36.
R. O.
Chavez-Garcia
,
J.
Burlet
,
T. D.
Vu
, and
O.
Aycard
, in
2012 IEEE Intelligent Vehicles Symposium
(
IEEE
,
2012
), p.
159
.
37.
P.
Molchanov
,
S.
Gupta
,
K.
Kim
, and
K.
Pulli
, in
2015 IEEE Radar Conference
(
IEEE
,
2015
), p.
1491
.
38.
R.
Rouveure
,
P.
Faure
, and
M.-O.
Monod
,
J. Field Rob.
35
,
678
(
2018
).
39.
H.
Rohling
, in
2011 12th International Radar Symposium (IRS)
(
IEEE
,
2011
), p.
631
.
40.
T.-Y.
Lin
,
M.
Maire
,
S.
Belongie
,
J.
Hays
,
P.
Perona
,
D.
Ramanan
,
P.
Dollár
, and
C. L.
Zitnick
, in
European Conference on Computer Vision
(
Springer, Cham
,
2014
), p.
740
.
41.
S.
Ren
,
K.
He
,
R.
Girshick
, and
J.
Sun
,
IEEE Trans. Pattern Anal. Mach. Intell.
39
,
1137
(
2016
).
42.
M.
Jaward
,
L.
Mihaylova
,
N.
Canagarajah
, and
D.
Bull
, in
2006 IEEE Aerospace Conference
(
IEEE
,
2006
), p.
8
.
43.
R.
Jinan
and
T.
Raveendran
,
Procedia Technol.
24
,
980
(
2016
).
You do not currently have access to this content.