In this paper, an image-based visual servoing (IBVS) controller with a 6 degree-of-freedom robotic manipulator that tracks moving objects is investigated using the proposed Deep Q-Networks and proportional–integral–derivative (DQN-PID) controller. First, the classical IBVS controller and the problem of feature loss and large steady-state error for tracking moving targets are introduced. Then, a DQN-PID based IBVS method is proposed to solve the problem of feature loss and large steady-state error and improve the servo precision, as the existing methods are hard to use for solve the problems. Specifically, the IBVS method is inherited by our controller to build the tracking model, and a value-based reinforcement learning method is proposed as an adaptive law for dynamically tuning the PID parameters in the discrete space, which can track the moving target and keep the servo feature in the field of the camera. Finally, compared with the different existing methods, the DQN-PID based IBVS method has merits of higher accuracy and more stable tracking, or generalization.

1.
C.
Ru
,
F.
Wang
,
T.
Li
,
B.
Ren
, and
X.
Yan
, “
Outline viewpoint feature histogram: An improved point cloud descriptor for recognition and grasping of workpieces
,”
Rev. Sci. Instrum.
92
,
025010
(
2021
).
2.
H.
Zhang
,
F.
Wang
,
J.
Wang
, and
B.
Cui
, “
Robot grasping method optimization using improved deep deterministic policy gradient algorithm of deep reinforcement learning
,”
Rev. Sci. Instrum.
92
,
025114
(
2021
).
3.
H.
Bettahar
,
O.
Lehmann
,
C.
Clévy
,
N.
Courjal
, and
P.
Lutz
, “
Photo-robotic extrinsic parameters calibration of 6-DOF robot for high positioning accuracy
,”
IEEE/ASME Trans. Mechatron.
25
,
616
626
(
2020
).
4.
B.
Espiau
,
F.
Chaumette
, and
P.
Rives
, “
A new approach to visual servoing in robotics
,”
IEEE Trans. Rob. Autom.
8
,
313
326
(
1992
).
5.
H. J.
Asl
and
H.
Bolandi
, “
Robust vision-based control of an underactuated flying robot tracking a moving target
,”
Trans. Inst. Meas. Control
36
,
411
424
(
2014
).
6.
D.
Lee
,
T.
Ryan
, and
H. J.
Kim
, “
Autonomous landing of a VTOL UAV on a moving platform using image-based visual servoing
,” in
2012 IEEE international conference on robotics and automation
(
IEEE
,
2012
), pp.
971
976
.
7.
D.-H.
Park
,
J.-H.
Kwon
, and
I.-J.
Ha
, “
Novel position-based visual servoing approach to robust global stability under field-of-view constraint
,”
IEEE Trans. Ind. Electron.
59
,
4735
4752
(
2011
).
8.
F.
Chaumette
and
S.
Hutchinson
, “
Visual servo control. I. Basic approaches
,”
IEEE Rob. Autom. Mag.
13
,
82
90
(
2006
).
9.
E.
Malis
,
F.
Chaumette
, and
S.
Boudet
, “
2 1/2 D visual servoing
,”
IEEE Trans. Rob. Autom.
15
,
238
250
(
1999
).
10.
H.
Shi
,
L.
Shi
,
G.
Sun
, and
K.-S.
Hwang
, “
Adaptive image-based visual servoing for hovering control of quad-rotor
,”
IEEE Trans. Cognit. Dev. Syst.
12
,
417
426
(
2019
).
11.
H.
Shi
,
K.-S.
Hwang
,
X.
Li
, and
J.
Chen
, “
A learning approach to image-based visual servoing with a bagging method of velocity calculations
,”
Inf. Sci.
481
,
244
257
(
2019
).
12.
M.
Kang
,
H.
Chen
, and
J.
Dong
, “
Adaptive visual servoing with an uncalibrated camera using extreme learning machine and Q-leaning
,”
Neurocomputing
402
,
384
394
(
2020
).
13.
Z.
Jin
,
J.
Wu
,
A.
Liu
,
W.-A.
Zhang
, and
L.
Yu
, “
Policy-based deep reinforcement learning for visual servoing control of mobile robots with visibility constraints
,”
IEEE Trans. Ind. Electron.
69
,
1898
(
2021
).
14.
T.
Kato
,
K.
Watanabe
, and
S.
Maeyama
, “
Arbitrary trajectory tracking of a mobile robot by an image-based fuzzy controller
,” in
2012 Proceedings of SICE Annual Conference (SICE)
(
IEEE
,
2012
), pp.
160
165
.
15.
M.
Bonkovic
,
D.
Stipanicev
, and
M.
Stula
, “
The efficiency of supplement schema for IBVS algorithms
,” in
Proceedings of the IEEE International Symposium on Industrial Electronics, 2005
(
IEEE
,
2005
), Vol. 1, pp.
189
194
.
16.
A.
Amirkhani
,
M.
Shirzadeh
,
E. I.
Papageorgiou
, and
M. R.
Mosavi
, “
Fuzzy cognitive map for visual servoing of flying robot
,” in
2016 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)
(
IEEE
,
2016
), pp.
128
1376
.
17.
Z.
Liu
,
F.
Wang
, and
Y.
Zhang
, “
Adaptive visual tracking control for manipulator with actuator fuzzy dead-zone constraint and unmodeled dynamic
,”
IEEE Trans. Syst., Man, Cybern.: Syst.
45
,
1301
1312
(
2015
).
18.
T.
Yüksel
, “
Intelligent visual servoing with extreme learning machine and fuzzy logic
,”
Expert Syst. Appl.
72
,
344
356
(
2017
).
19.
K.
Watanabe
,
Y.
Yoshihata
,
Y.
Iwatani
, and
K.
Hashimoto
, “
Image-based visual PID control of a micro helicopter using a stationary camera
,”
Adv. Rob.
22
,
381
393
(
2008
).
20.
H.
Wang
,
Z.
Li
,
X.
Jin
,
Y.
Huang
,
H.
Kong
,
M.
Yu
,
Z.
Ping
, and
Z.
Sun
, “
Adaptive integral terminal sliding mode control for automobile electronic throttle via an uncertainty observer and experimental validation
,”
IEEE Trans. Veh. Technol.
67
,
8129
8143
(
2018
).
21.
I.
Siradjuddin
,
L.
Behera
,
T. M.
McGinnity
, and
S.
Coleman
, “
Image-based visual servoing of a 7-DOF robot manipulator using an adaptive distributed fuzzy PD controller
,”
IEEE/ASME Trans. Mechatron.
19
,
512
523
(
2013
).
22.
X.
Yu
,
Y.
Fan
,
S.
Xu
, and
L.
Ou
, “
A self-adaptive SAC-PID control approach based on reinforcement learning for mobile robots
,”
Int. J. Robust Nonlinear Control
31
,
1
19
(
2021
).
23.
J.
Dong
,
Y.
Hu
, and
K.
Peng
, “
Robot visual servo control based on fuzzy adaptive PID
,” in
2012 International Conference on Systems and Informatics (ICSAI2012)
(
IEEE
,
2012
), pp.
1337
1341
.
24.
C. J. C. H.
Watkins
, “
Learning from delayed rewards
,” Ph.D. thesis,
University of Cambridge
,
1989
.
25.
V.
Mnih
,
K.
Kavukcuoglu
,
D.
Silver
,
A.
Graves
,
I.
Antonoglou
,
D.
Wierstra
, and
M.
Riedmiller
, “
Playing Atari with deep reinforcement learning
,”
Computer Sci.
56
(
1
),
201
220
(
2013
).
26.
M. W.
Spong
,
S.
Hutchinson
,
M.
Vidyasagar
 et al.,
Robot Modeling and Control
(
Wiley
,
New York
,
2006
), Vol. 3.
27.
F.
Chaumette
and
S.
Hutchinson
, “
Visual servo control. II. Advanced approaches [tutorial]
,”
IEEE Rob. Autom. Mag.
14
,
109
118
(
2007
).
28.
I.
Siradjuddin
,
L.
Behera
,
T. M.
McGinnity
, and
S.
Coleman
, “
Image based visual servoing of a 7 DOF robot manipulator using a distributed fuzzy proportional controller
,” in
International Conference on Fuzzy Systems
(
IEEE
,
2010
), pp.
1
8
.
29.
Y.
Zhang
and
Z.
Jiao
, “
Study of grey PID control for discrete system
,” in
2009 International Workshop on Intelligent Systems and Applications
(
IEEE
,
2009
), pp.
1
4
.
30.
S.
Tzafestas
and
N. P.
Papanikolopoulos
, “
Incremental fuzzy expert PID control
,”
IEEE Trans. Ind. Electron.
37
,
365
371
(
1990
).
You do not currently have access to this content.