It is very difficult for visually impaired people to perceive and avoid obstacles at a distance. To address this problem, the unified framework of multiple target detection, recognition, and fusion is proposed based on the sensor fusion system comprising a low-power millimeter wave (MMW) radar and an RGB-Depth (RGB-D) sensor. In this paper, the Mask R-CNN and the single shot multibox detector network are utilized to detect and recognize the objects from color images. The obstacles’ depth information is obtained from the depth images using the MeanShift algorithm. The position and velocity information on the multiple target is detected by the MMW radar based on the principle of a frequency modulated continuous wave. The data fusion based on the particle filter obtains more accurate state estimation and richer information by fusing the detection results from the color images, depth images, and radar data compared with using only one sensor. The experimental results show that the data fusion enriches the detection results. Meanwhile, the effective detection range is expanded compared to using only the RGB-D sensor. Moreover, the data fusion results keep high accuracy and stability under diverse range and illumination conditions. As a wearable system, the sensor fusion system has the characteristics of versatility, portability, and cost-effectiveness.
Skip Nav Destination
Article navigation
April 2019
Research Article|
April 22 2019
Unifying obstacle detection, recognition, and fusion based on millimeter wave radar and RGB-depth sensors for the visually impaired
Ningbo Long
;
Ningbo Long
State Key Laboratory of Modern Optical Instrumentation, Zhejiang University
, Hangzhou, China
Search for other works by this author on:
Kaiwei Wang
;
Kaiwei Wang
a)
State Key Laboratory of Modern Optical Instrumentation, Zhejiang University
, Hangzhou, China
a)Author to whom correspondence should be addressed: wangkaiwei@zju.edu.cn
Search for other works by this author on:
Ruiqi Cheng
;
Ruiqi Cheng
State Key Laboratory of Modern Optical Instrumentation, Zhejiang University
, Hangzhou, China
Search for other works by this author on:
Weijian Hu;
Weijian Hu
State Key Laboratory of Modern Optical Instrumentation, Zhejiang University
, Hangzhou, China
Search for other works by this author on:
Kailun Yang
Kailun Yang
State Key Laboratory of Modern Optical Instrumentation, Zhejiang University
, Hangzhou, China
Search for other works by this author on:
a)Author to whom correspondence should be addressed: wangkaiwei@zju.edu.cn
Rev. Sci. Instrum. 90, 044102 (2019)
Article history
Received:
February 19 2019
Accepted:
March 30 2019
Citation
Ningbo Long, Kaiwei Wang, Ruiqi Cheng, Weijian Hu, Kailun Yang; Unifying obstacle detection, recognition, and fusion based on millimeter wave radar and RGB-depth sensors for the visually impaired. Rev. Sci. Instrum. 1 April 2019; 90 (4): 044102. https://doi.org/10.1063/1.5093279
Download citation file:
Sign in
Don't already have an account? Register
Sign In
You could not be signed in. Please check your credentials and make sure you have an active account and try again.
Sign in via your Institution
Sign in via your InstitutionPay-Per-View Access
$40.00
Citing articles via
Related Content
Pedestrian detection and tracking using modified SURF algorithm
AIP Conference Proceedings (March 2015)
MMW/IR beam combiner with graphene IR window for MMW/IR compact range compound test
Appl. Phys. Lett. (June 2017)
RGB laser speckles based 3D profilometry
Appl. Phys. Lett. (May 2019)
Metallicity Discrepancy between RGB and RHB Stars of the Globular Cluster M15
AIP Conference Proceedings (March 2008)
A review of traditional and deep learning approaches for RGB-D face recognition
AIP Conference Proceedings (July 2021)