It is very difficult for visually impaired people to perceive and avoid obstacles at a distance. To address this problem, the unified framework of multiple target detection, recognition, and fusion is proposed based on the sensor fusion system comprising a low-power millimeter wave (MMW) radar and an RGB-Depth (RGB-D) sensor. In this paper, the Mask R-CNN and the single shot multibox detector network are utilized to detect and recognize the objects from color images. The obstacles’ depth information is obtained from the depth images using the MeanShift algorithm. The position and velocity information on the multiple target is detected by the MMW radar based on the principle of a frequency modulated continuous wave. The data fusion based on the particle filter obtains more accurate state estimation and richer information by fusing the detection results from the color images, depth images, and radar data compared with using only one sensor. The experimental results show that the data fusion enriches the detection results. Meanwhile, the effective detection range is expanded compared to using only the RGB-D sensor. Moreover, the data fusion results keep high accuracy and stability under diverse range and illumination conditions. As a wearable system, the sensor fusion system has the characteristics of versatility, portability, and cost-effectiveness.
Skip Nav Destination
Article navigation
April 2019
Research Article|
April 22 2019
Unifying obstacle detection, recognition, and fusion based on millimeter wave radar and RGB-depth sensors for the visually impaired
Ningbo Long
;
Ningbo Long
State Key Laboratory of Modern Optical Instrumentation, Zhejiang University
, Hangzhou, China
Search for other works by this author on:
Kaiwei Wang
;
Kaiwei Wang
a)
State Key Laboratory of Modern Optical Instrumentation, Zhejiang University
, Hangzhou, China
a)Author to whom correspondence should be addressed: wangkaiwei@zju.edu.cn
Search for other works by this author on:
Ruiqi Cheng
;
Ruiqi Cheng
State Key Laboratory of Modern Optical Instrumentation, Zhejiang University
, Hangzhou, China
Search for other works by this author on:
Weijian Hu;
Weijian Hu
State Key Laboratory of Modern Optical Instrumentation, Zhejiang University
, Hangzhou, China
Search for other works by this author on:
Kailun Yang
Kailun Yang
State Key Laboratory of Modern Optical Instrumentation, Zhejiang University
, Hangzhou, China
Search for other works by this author on:
a)Author to whom correspondence should be addressed: wangkaiwei@zju.edu.cn
Rev. Sci. Instrum. 90, 044102 (2019)
Article history
Received:
February 19 2019
Accepted:
March 30 2019
Citation
Ningbo Long, Kaiwei Wang, Ruiqi Cheng, Weijian Hu, Kailun Yang; Unifying obstacle detection, recognition, and fusion based on millimeter wave radar and RGB-depth sensors for the visually impaired. Rev. Sci. Instrum. 1 April 2019; 90 (4): 044102. https://doi.org/10.1063/1.5093279
Download citation file:
Sign in
Don't already have an account? Register
Sign In
You could not be signed in. Please check your credentials and make sure you have an active account and try again.
Pay-Per-View Access
$40.00
Citing articles via
Overview of the early campaign diagnostics for the SPARC tokamak (invited)
M. L. Reinke, I. Abramovic, et al.
An instrumentation guide to measuring thermal conductivity using frequency domain thermoreflectance (FDTR)
Dylan J. Kirsch, Joshua Martin, et al.
Uniform microwave field formation for control of ensembles of negatively charged nitrogen vacancy in diamond
Oleg Rezinkin, Marina Rezinkina, et al.
Related Content
A comparative study of real-time object detection using tensorflow’s single shot multibox detector (SSD) and histogram of oriented gradient (HOG)
AIP Conf. Proc. (August 2024)
A detection method for impact point water columns based on improved YOLO X
AIP Advances (June 2022)
Recognition of the internal situation of aircraft skin based on deep learning
AIP Advances (October 2021)
Applying edge artificial intelligence to closed-loop real time control and monitoring of laser based battery pack welding
J. Laser Appl. (July 2022)
Brain tumor detection with high accuracy using K-nearest neighbor comparing with thresholding method
AIP Conf. Proc. (November 2023)