To accurately reflect the movement of humans or robots, multi-sensor integration strategy is necessary to decouple complex deformations caused by motion in the wearable artificial kinesthetic perception system. The multi-sensor integration strategy could collect multi-dimension information, making up for the deficiency of robustness and accuracy of single sensor in complex motion scenes and extending the sensing capability of the system. However, the inevitable inconsistency between multiple sensors causes difficulties in fabrication, integration, and perception, limiting the development of artificial kinesthetic perception systems toward the direction of compact integration, large-area sensing, and high-resolution decoupling. Here, we propose a strategy to build an artificial kinesthetic perception system based on the differential design of homogeneous sensors. The strategy aims at guiding system design to avoid the inconsistency in multi-sensor integration by revealing the explicit relationship among structure, signal, and movements from the perspective of the physics model. According to the strategy, we presented a simple fabrication method of the kinesthetic perception prototype. The prototype with two homogenous sensors (0.019 56 residual on average) shows three differential signal modes to three deformations. With the help of machine learning, it realized the decoupling task to 25 kinds of complex deformations. The accuracy remains at 95% even though the decoupling resolution is up to 0.2 mm. With more than one prototype arrayed, complex deformation composed with more kinds of basic deformation (shear and twist) could be further decoupled. We believe that the strategy described in this paper will contribute to the development of a compact and programmable kinesthetic perception system.
REFERENCES
1.
I. V.
Grinyagin
,
E. V.
Biryukova
, and
M. A.
Maier
, “
Kinematic and dynamic synergies of human precision-grip movements
,” J. Neurophysiol.
94
, 2284
–2294
(2005
).2.
A.
Kaiser
and
A.
Bullinger-Hoffmann
, “
Precision of human movement in the virtual reach envelope—Measurement of static and dynamic precision depending on different movement Speeds
,” in Advances in Transdisciplinary Engineering
(
IOS Press
, 2020
), Vol. 11, pp. 145
–155
.3.
X.
Chen
,
R.
Li
,
G.
Niu
,
M.
Xin
,
G.
Xu
,
H.
Cheng
, and
L.
Yang
, “
Porous graphene foam composite-based dual-mode sensors for underwater temperature and subtle motion detection
,” Chem. Eng. J.
444
, 136631
(2022
).4.
L.
Yang
,
H.
Wang
,
W.
Yuan
,
Y.
Li
,
P.
Gao
,
N.
Tiwari
,
X.
Chen
,
Z.
Wang
,
G.
Niu
, and
H.
Cheng
, “
Wearable pressure sensors based on MXene/tissue papers for wireless human health monitoring
,” ACS Appl. Mater. Interfaces
13
, 60531
–60543
(2021
).5.
C.
Zhang
,
H.
Chen
,
X.
Ding
,
F.
Lorestani
,
C.
Huang
,
B.
Zhang
,
B.
Zheng
,
J.
Wang
,
H.
Cheng
, and
Y.
Xu
, “
Human motion-driven self-powered stretchable sensing platform based on laser-induced graphene foams
,” Appl. Phys. Rev.
9
, 011413
(2022
).6.
S.
Liu
,
J.
Zhang
,
Y.
Zhang
, and
R.
Zhu
, “
A wearable motion capture device able to detect dynamic motion of human limbs
,” Nat Commun.
11
, 5615
(2020
).7.
S. L.
Colyer
,
M.
Evans
,
D. P.
Cosker
, and
A. I. T.
Salo
, “
A review of the evolution of vision-based motion analysis and the integration of advanced computer vision methods towards developing a markerless system
,” Sports Med.–Open
4
, 24
(2018
).8.
D.
Farina
,
I.
Vujaklija
,
R.
Brånemark
,
A. M. J.
Bull
,
H.
Dietl
,
B.
Graimann
,
L. J.
Hargrove
,
K.-P.
Hoffmann
,
H.
Huang
,
T.
Ingvarsson
,
H. B.
Janusson
,
K.
Kristjánsson
,
T.
Kuiken
,
S.
Micera
,
T.
Stieglitz
,
A.
Sturma
,
D.
Tyler
,
R. F. F.
Weir
, and
O. C.
Aszmann
, “
Toward higher-performance bionic limbs for wider clinical use
,” Nat. Biomed. Eng.
7
, 473
–485
(2021
).9.
G.
Gu
,
N.
Zhang
,
H.
Xu
,
S.
Lin
,
Y.
Yu
,
G.
Chai
,
L.
Ge
,
H.
Yang
,
Q.
Shao
,
X.
Sheng
,
X.
Zhu
, and
X.
Zhao
, “
A soft neuroprosthetic hand providing simultaneous myoelectric control and tactile feedback
,” Nat. Biomed. Eng.
7
, 589
–598
(2021
).10.
J. A.
George
,
D. T.
Kluger
,
T. S.
Davis
,
S. M.
Wendelken
,
E. V.
Okorokova
,
Q.
He
,
C. C.
Duncan
,
D. T.
Hutchinson
,
Z. C.
Thumser
,
D. T.
Beckler
,
P. D.
Marasco
,
S. J.
Bensmaia
, and
G. A.
Clark
, “
Biomimetic sensory feedback through peripheral nerve stimulation improves dexterous use of a bionic hand
,” Sci. Rob.
4
, eaax2352
(2019
).11.
F.
Wen
,
Z.
Sun
,
T.
He
,
Q.
Shi
,
M.
Zhu
,
Z.
Zhang
,
L.
Li
,
T.
Zhang
, and
C.
Lee
, “
Machine learning glove using self‐powered conductive superhydrophobic triboelectric textile for gesture recognition in VR/AR applications
,” Adv. Sci.
7
, 2000261
(2020
).12.
T.
Delrieu
,
V.
Weistroffer
, and
J. P.
Gazeau
, “
Precise and realistic grasping and manipulation in Virtual Reality without force feedback
,” in IEEE Conference on Virtual Reality and 3D User Interfaces (VR)
(
IEEE
,
Atlanta, GA
, 2020
), pp. 266
–274
.13.
S. H.
Nasir
,
O.
Troynikov
, and
C.
Watson
, “
Skin deformation behavior during hand movements and their impact on functional sports glove design
,” Procedia Eng.
112
, 92
–97
(2015
).14.
P. R.
Weeren
,
A. J.
Van Den Bogert
, and
A.
Barneveld
, “
Quantification of skin displacement near the carpal, tarsal and fetlock joints of the walking horse
,” Equine Vet. J.
20
, 203
–208
(1988
).15.
Y.
Jiang
,
Z.
Liu
,
C.
Wang
, and
X.
Chen
, “
Heterogeneous strain distribution of elastomer substrates to enhance the sensitivity of stretchable strain sensors
,” Acc. Chem. Res.
52
, 82
–90
(2019
).16.
Y.
Cheng
,
R.
Wang
,
J.
Sun
, and
L.
Gao
, “
A stretchable and highly sensitive graphene-based fiber for sensing tensile strain, bending, and torsion
,” Adv. Mater.
27
, 7365
–7371
(2015
).17.
C.
Wang
,
G.
Wang
,
J.
Zhao
, and
S.
Sun
, “
A highly accurate, stretchable touchpad for robust, linear, and stable tactile feedback
,” Adv. Mater. Technol.
5
, 1900864
(2020
).18.
Y.
Wu
,
Y.
Liu
,
Y.
Zhou
,
Q.
Man
,
C.
Hu
,
W.
Asghar
,
F.
Li
,
Z.
Yu
,
J.
Shang
,
G.
Liu
,
M.
Liao
, and
R.-W.
Li
, “
A skin-inspired tactile sensor for smart prosthetics
,” Sci. Rob.
9
, eaat0429
(2018
).19.
S.
Gong
,
L. W.
Yap
,
B.
Zhu
,
Q.
Zhai
,
Y.
Liu
,
Q.
Lyu
,
K.
Wang
,
M.
Yang
,
Y.
Ling
,
D. T. H.
Lai
,
F.
Marzbanrad
, and
W.
Cheng
, “
Local crack‐programmed gold nanowire electronic skin tattoos for in‐plane multisensor integration
,” Adv. Mater.
31
, 1903789
(2019
).20.
Z.
Liu
,
Y.
Zheng
,
L.
Jin
,
K.
Chen
,
H.
Zhai
,
Q.
Huang
,
Z.
Chen
,
Y.
Yi
,
M.
Umar
,
L.
Xu
,
G.
Li
,
Q.
Song
,
P.
Yue
,
Y.
Li
, and
Z.
Zheng
, “
Highly breathable and stretchable strain sensors with insensitive response to pressure and bending
,” Adv. Funct. Mater.
31
, 2007622
(2021
).21.
S.
Lee
,
A.
Reuveny
,
J.
Reeder
,
S.
Lee
,
H.
Jin
,
Q.
Liu
,
T.
Yokota
,
T.
Sekitani
,
T.
Isoyama
,
Y.
Abe
,
Z.
Suo
, and
T.
Someya
, “
A transparent bending-insensitive pressure sensor
,” Nat. Nanotechnol.
11
, 472
–478
(2016
).22.
X.
Wu
,
M.
Ahmed
,
Y.
Khan
,
M. E.
Payne
,
J.
Zhu
,
C.
Lu
,
J. W.
Evans
, and
A. C.
Arias
, “
A potentiometric mechanotransduction mechanism for novel electronic skins
,” Sci. Adv.
6
, eaba1062
(2020
).23.
P. R.
Burgess
,
J. Y.
Wei
,
F. J.
Clark
, and
J.
Simon
, “
Signaling of kinesthetic information by peripheral sensory receptors
,” Annu. Rev. Neurosci.
5
, 171
–188
(1982
).24.
D. I.
McCloskey
, “
Kinesthetic sensibility
,” Physiol. Rev.
58
, 763
–820
(1978
).25.
U.
Proske
, “
What is the role of muscle receptors in proprioception?
,” Muscle Nerve
31
, 780
–787
(2005
).26.
R.
Yang
,
W.
Zhang
,
N.
Tiwari
,
H.
Yan
,
T.
Li
, and
H.
Cheng
, “
Multimodal sensors with decoupled sensing mechanisms
,” Adv. Sci.
9
, 2202470
(2022
).27.
See https://www.vicon.com/ for “
Vicon | Award Winning Motion Capture Systems. Vicon
”28.
See https://en.nokov.com/direct for “
Motion Capture and Motion Tracking Systems | NOKOV Mocap
.”29.
See http://www.vive.com/cn/ for “
HTCVive
”30.
M.
Wang
,
Z.
Yan
,
T.
Wang
,
P.
Cai
,
S.
Gao
,
Y.
Zeng
,
C.
Wan
,
H.
Wang
,
L.
Pan
,
J.
Yu
,
S.
Pan
,
K.
He
,
J.
Lu
, and
X.
Chen
, “
Gesture recognition using a bioinspired learning architecture that integrates visual data with somatosensory data from stretchable sensors
,” Nat. Electron.
3
, 563
–570
(2020
).31.
Y.
Lee
,
W.
Do
,
H.
Yoon
,
J.
Heo
,
W.
Lee
, and
D.
Lee
, “
Visual-inertial hand motion tracking with robustness against occlusion, interference, and contact
,” Sci. Robot.
6
, eabe1315
(2021
).32.
C.
Li
,
D.
Liu
,
C.
Xu
,
Z.
Wang
,
S.
Shu
,
Z.
Sun
,
W.
Tang
, and
Z. L.
Wang
, “
Sensing of joint and spinal bending or stretching via a retractable and wearable badge reel
,” Nat. Commun.
12
, 2950
(2021
).33.
S.
Sundaram
,
P.
Kellnhofer
,
Y.
Li
,
J.-Y.
Zhu
,
A.
Torralba
, and
W.
Matusik
, “
Learning the signatures of the human grasp using a scalable tactile glove
,” Nature
569
, 698
–702
(2019
).34.
J.
Lin
,
R.
Zhang
,
J.
Chen
,
L.
Zeng
,
L.
Jiang
,
S.
McGrath
, and
J.
Yang
, “
Using Hybrid Sensoring Method for Motion Capture in Volleyball Techniques Training
,” in 12th International Conference on Sensing Technology (ICST)
(
IEEE
,
Limerick
, 2018
), pp. 217
–222
.35.
J.
Camargo
,
W.
Flanagan
,
N.
Csomay-Shanklin
,
B.
Kanwar
, and
A.
Young
, “
A machine learning strategy for locomotion classification and parameter estimation using fusion of wearable sensors
,” IEEE Trans. Biomed. Eng.
68
, 1569
–1578
(2021
).36.
Y.
Li
,
L.
Yang
,
Z.
He
,
Y.
Liu
,
H.
Wang
,
W.
Zhang
,
L.
Teng
,
D.
Chen
, and
G.
Song
, “
Low‐cost data glove based on deep‐learning‐enhanced flexible multiwalled carbon nanotube sensors for real‐time gesture recognition
,” Adv. Intell. Syst.
4
, 2200128
(2022
).37.
M.
Zhu
,
Z.
Sun
, and
C.
Lee
, “
Soft modular glove with multimodal sensing and augmented haptic feedback enabled by materials' multifunctionalities
,” ACS Nano
16
, 14097
–14110
(2022
).38.
H.
Kim
,
S. H.
Tawfick
, and
W. P.
King
, “
Modeling and design of zero‐stiffness elastomer springs using machine learning
,” Adv. Intell. Syst.
4
, 2200225
(2022
).39.
M. P.
Wilk
,
M.
Walsh
, and
B.
O'Flynn
, “
Multimodal sensor fusion for low-power wearable human motion tracking systems in sports applications
,” IEEE Sens. J.
21
, 5195
–5212
(2021
).40.
Z.
Chen
,
R.
Yu
,
X.
Yu
,
E.
Li
,
C.
Wang
,
Y.
Liu
,
T.
Guo
, and
H.
Chen
, “
Bioinspired artificial motion sensory system for rotation recognition and rapid self-protection
,” ACS Nano
16
, 19155
–19164
(2022
).41.
M.
Zhu
,
Z.
Sun
,
Z.
Zhang
,
Q.
Shi
,
T.
He
,
H.
Liu
,
T.
Chen
, and
C.
Lee
, “
Haptic-feedback smart glove as a creative human-machine interface (HMI) for virtual/augmented reality applications
,” Sci. Adv.
6
, eaaz8693
(2020
).42.
H.
Bredin
and
G.
Chollet
, “
Audiovisual speech synchrony measure: Application to biometrics
,” EURASIP J. Adv. Signal Process.
2007
, 070186
.43.
D.
Hu
,
F.
Nie
, and
X.
Li
, “
Deep Multimodal Clustering for Unsupervised Audiovisual Learning
,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
(
IEEE
,
Long Beach, CA
, 2019
), pp. 9240
–9249
.44.
T.
Kim
,
S.
Lee
,
T.
Hong
,
G.
Shin
,
T.
Kim
, and
Y.-L.
Park
, “
Heterogeneous sensing in a multifunctional soft sensor for human-robot interfaces
,” Sci. Rob.
5
, eabc6878
(2020
).45.
J.
Hughes
,
A.
Spielberg
,
M.
Chounlakone
,
G.
Chang
,
W.
Matusik
, and
D.
Rus
, “
A simple, inexpensive, wearable glove with hybrid resistive‐pressure sensors for computational sensing, proprioception, and task identification
,” Adv. Intell. Syst.
2
, 2000002
(2020
).46.
M.
Totaro
,
A.
Mondini
,
A.
Bellacicca
,
P.
Milani
, and
L.
Beccai
, “
Integrated simultaneous detection of tactile and bending cues for soft robotics
,” Soft Rob.
4
, 400
–410
(2017
).47.
Z.
Cui
,
W.
Wang
,
H.
Xia
,
C.
Wang
,
J.
Tu
,
S.
Ji
,
J. M. R.
Tan
,
Z.
Liu
,
F.
Zhang
,
W.
Li
,
Z.
Lv
,
Z.
Li
,
W.
Guo
,
N. Y.
Koh
,
K. B.
Ng
,
X.
Feng
,
Y.
Zheng
, and
X.
Chen
, “
Freestanding and scalable force‐softness bimodal sensor arrays for haptic body‐feature identification
,” Adv. Mater.
34
, 2207016
(2022
).48.
Z.
Huang
,
Y.
Hao
,
Y.
Li
,
H.
Hu
,
C.
Wang
,
A.
Nomoto
,
T.
Pan
,
Y.
Gu
,
Y.
Chen
,
T.
Zhang
,
W.
Li
,
Y.
Lei
,
N.
Kim
,
C.
Wang
,
L.
Zhang
,
J. W.
Ward
,
A.
Maralani
,
X.
Li
,
M. F.
Durstock
,
A.
Pisano
,
Y.
Lin
, and
S.
Xu
, “
Three-dimensional integrated stretchable electronics
,” Nat. Electron.
1
, 473
–480
(2018
).49.
B.
Hou
,
L.
Yi
,
C.
Li
,
H.
Zhao
,
R.
Zhang
,
B.
Zhou
, and
X.
Liu
, “
An interactive mouthguard based on mechanoluminescence-powered optical fibre sensors for bite-controlled device operation
,” Nat. Electron.
5
, 682
–693
(2022
).50.
H.
Bai
,
S.
Li
,
J.
Barreiros
,
Y.
Tu
,
C. R.
Pollock
, and
R. F.
Shepherd
, “
Stretchable distributed fiber-optic sensors
,” Science
370
, 848
–852
(2020
).51.
H.
Zhao
,
J.
Jalving
,
R.
Huang
,
R.
Knepper
,
A.
Ruina
, and
R.
Shepherd
, “
A helping hand: Soft orthosis with integrated optical strain sensors and EMG control
,” IEEE Rob. Autom. Mag.
23
, 55
–64
(2016
).52.
Z.
Liu
,
D.
Qi
,
W. R.
Leow
,
J.
Yu
,
M.
Xiloyannnis
,
L.
Cappello
,
Y.
Liu
,
B.
Zhu
,
Y.
Jiang
,
G.
Chen
,
L.
Masia
,
B.
Liedberg
, and
X.
Chen
, “
3D-structured stretchable strain sensors for out-of-plane force detection
,” Adv. Mater.
30
, 1707285
(2018
).53.
Y.
Luo
,
X.
Xiao
,
J.
Chen
,
Q.
Li
, and
H.
Fu
, “
Machine-learning-assisted recognition on bioinspired soft sensor arrays
,” ACS Nano
16
, 6734
–6743
(2022
).54.
Q.
Gao
,
S.
Jiang
, and
P. B.
Shull
, “
Simultaneous hand gesture classification and finger angle estimation via a novel dual-output deep learning model
,” Sensors
20
, 2972
(2020
).55.
B.
Xie
,
B.
Li
, and
A.
Harland
, “
Movement and Gesture Recognition Using Deep Learning and Wearable-sensor Technology
,” in Proceedings of the International Conference on Artificial Intelligence and Pattern Recognition (AIPR 2018)
(
ACM Press
,
Beijing, China
, 2018
), pp. 26
–31
.56.
57.
Y.
Luo
,
Y.
Li
,
P.
Sharma
,
W.
Shou
,
K.
Wu
,
M.
Foshey
,
B.
Li
,
T.
Palacios
,
A.
Torralba
, and
W.
Matusik
, “
Learning human–environment interactions using conformal tactile textiles
,” Nat. Electron.
4
, 193
–201
(2021
).58.
P.
Mostafalu
,
G.
Kiaee
,
G.
Giatsidis
,
A.
Khalilpour
,
M.
Nabavinia
,
M. R.
Dokmeci
,
S.
Sonkusale
,
D. P.
Orgill
,
A.
Tamayol
, and
A.
Khademhosseini
, “
A textile dressing for temporal and dosage controlled drug delivery
,” Adv. Funct. Mater.
27
, 1702399
(2017
).59.
Z.
Zhou
,
S.
Padgett
,
Z.
Cai
,
G.
Conta
,
Y.
Wu
,
Q.
He
,
S.
Zhang
,
C.
Sun
,
J.
Liu
,
E.
Fan
,
K.
Meng
,
Z.
Lin
,
C.
Uy
,
J.
Yang
, and
J.
Chen
, “
Single-layered ultra-soft washable smart textiles for all-around ballistocardiograph, respiration, and posture monitoring during sleep
,” Biosens. Bioelectron.
155
, 112064
(2020
).60.
N.
Yi
,
Y.
Gao
,
A.
Lo Verso
,
J.
Zhu
,
D.
Erdely
,
C.
Xue
,
R.
Lavelle
, and
H.
Cheng
, “
Fabricating functional circuits on 3D freeform surfaces via intense pulsed light-induced zinc mass transfer
,” Mater. Today
50
, 24
–34
(2021
).61.
R. G.
Andersen
,
J. G.
Londono
,
P. B.
Woelke
, and
K. L.
Nielsen
, “
Fundamental differences between plane strain bending and far-field plane strain tension in ductile plate failure
,” J. Mech. Phys. Solids
141
, 103960
(2020
).62.
V. V.
Soman
,
D.
Han
,
A. C.
Arias
,
K.
Ghose
,
M. D.
Poliks
,
J. N.
Turner
,
Y.
Khan
,
M.
Zabran
,
M.
Schadt
,
P.
Hart
,
M.
Shay
,
F. D.
Egitto
,
K. I.
Papathomas
, and
N. A. D.
Yamamoto
, “
Reliability Challenges in fabrication of flexible hybrid electronics for human performance monitors: A system-level study
,” IEEE Trans. Compon., Packag., Manuf. Technol.
9
, 1872
–1887
(2019
).63.
D. N.
Wright
,
A.-S. B.
Vardoy
,
B. D.
Belle
,
M. M.
Visser Taklo
,
O.
Hagel
,
L.
Xie
,
M.
Danestig
, and
T.
Eriksson
, “
Bending machine for testing reliability of flexible electronics
,” in IMAPS Nordic Conference on Microelectronics Packaging (NordPac)
(
IEEE
,
Gothenburg
, 2017
), pp. 47
–52
.64.
J.
Haladjian
, see (available at https://github.com/avenix/WDK for “
Wearables Development Toolkit (WDK)
(2023
).”65.
M.
Löning
,
A.
Bagnall
,
S.
Ganesh
,
V.
Kazakov
,
J.
Lines
, and
F. J.
Király
, “
sktime: A Unified Interface for Machine Learning with Time Series
,” arXiv:1909.07872.66.
A.
Dempster
,
F.
Petitjean
, and
G. I.
Webb
, “
ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels
,” Data Min. Knowl. Discovery
34
, 1454
–1495
(2020
).67.
J.
Hills
,
J.
Lines
,
E.
Baranauskas
,
J.
Mapp
, and
A.
Bagnall
, “
Classification of time series by shapelet transformation
,” Data Min. Knowl. Discovery
28
, 851
–881
(2014
).68.
A. P.
Ruiz
,
M.
Flynn
,
J.
Large
,
M.
Middlehurst
, and
A.
Bagnall
, “
The great multivariate time series classification bake off: A review and experimental evaluation of recent algorithmic advances
,” Data Min. Knowl. Discovery
35
, 401
–449
(2021
).69.
E.
Hagert
, “
Proprioception of the wrist joint: A review of current concepts and possible implications on the rehabilitation of the wrist
,” J. Hand Ther.
23
, 2
–17
(2010
).70.
P.
Haggard
,
K.
Hutchinson
, and
J.
Stein
, “
Patterns of coordinated multi-joint movement
,” Exp. Brain Res.
107
, 108
–112
(1995
).71.
M. L.
Wright
,
D. E.
Adamo
, and
S. H.
Brown
, “
Age-related declines in the detection of passive wrist movement
,” Neurosci. Lett.
500
, 108
–112
(2011
).© 2023 Author(s). Published under an exclusive license by AIP Publishing.
2023
Author(s)
You do not currently have access to this content.