Research and Implementation of Key Technologies of Multimodal Gesture Recognition Armband

doi:10.3901/JME.260220

Abstract

Abstract: Against the backdrop of the rapid development of smart wearable technology and the rise of multimodal gesture recognition as a research hotspot, limitations of traditional electromyography armbands in complex scenarios are addressed, and a multimodal gesture recognition armband system integrating triboelectric sensors and gyroscopes is proposed. A sensing unit with a composite structure of copper foil and Polytetrafluoroethylene(PTFE)-Ecoflex materials is developed; under a pressure of 2 N, its triboelectric voltage signal reaches 4.6 V. The sensing unit is expanded into an array and combined with a gyroscope to enable multimodal data collection. Based on the muscle distribution and movement characteristics of the human forearm, a sensing array is designed, and the optimal positions of six channels are calibrated. The armband body and sensing array are fabricated via 3D printing using thermoplastic polyurethane(TPU) material, which enhances both performance and comfort. An enhanced Transformer deep learning model tailored for the armband is innovatively proposed: a temporal enhancement module is incorporated to increase dataset diversity, and methods such as multi-scale feature extraction and position encoding are applied to fuse multimodal information, capturing data features and positional relationships. As a result, gesture recognition accuracy is raised to 95%, while resource consumption is reduced.

Key words: wearable sensor, triboelectric sensor, armband system, multimodal gesture recognition, deep learning

CLC Number:

TP216

ZHAO Zijing, GAO Yuze, WAN Yongtao, ZHU Jianxiong. Research and Implementation of Key Technologies of Multimodal Gesture Recognition Armband[J]. Journal of Mechanical Engineering, 2026, 62(10): 50-61.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

URL: http://www.cjmenet.com.cn/EN/10.3901/JME.260220

http://www.cjmenet.com.cn/EN/Y2026/V62/I10/50

References

[1] YAO K，ZHOU J，HUANG Q，et al. Encoding of tactile information in hand via skin-integrated wireless haptic interface[J]. Nature Machine Intelligence，2022，4(10)：893-903.
[2] LIU L，ZHANG X. A focused review on the flexible wearable sensors for sports：From kinematics to physiologies[J]. Micromachines，2022，13(8)：1356.
[3] PYO S，LEE J，BAE K，et al. Recent progress in flexible tactile sensors for human‐interactive systems：From sensors to advanced applications[J]. Advanced Materials，2021，33(47)：2005902.
[4] MONGARDI A，ROSSI F，PRESTIA A，et al. Hand gestures recognition for human-machine interfaces：A low-power bio-inspired armband[J]. IEEE Transactions on Biomedical Circuits and Systems，2022，16(6)：1348-1365.
[5] GAO W，OTA H，KIRIYA D，et al. Flexible electronics toward wearable sensing[J]. Accounts of Chemical Research，2019，52(3)：523-533.
[6] SHAO Q，SNIFFEN A，BLANCHET J，et al. Teaching american sign language in mixed reality[J]. Proceedings of the ACM on Interactive，Mobile，Wearable and Ubiquitous Technologies，2020，4(4)：1-27.
[7] BASIRI S，TAHERI A，MEGHDARI A，et al. Design and implementation of a robotic architecture for adaptive teaching：A case study on iranian sign language[J]. Journal of Intelligent & Robotic Systems，2021，102(2)：48.
[8] LEE K T，CHEE P S，LIM E H，et al. Data glove with integrated polyethylene-carbon composite-based strain sensor for virtual reality applications[J]. Chemical Engineering & Technology，2023，46(12)：2480-2486.
[9] WONG W K，JUWONO F H，KHOO B T T. Multi-features capacitive hand gesture recognition sensor：A machine learning approach[J]. IEEE Sensors Journal，2021，21(6)：8441-8450.
[10] RAO H C，LUO B B，WU D C，et al. Study on the design and performance of a glove based on the fbg array for hand posture sensing[J]. Sensors，2023，23(20)：8495.
[11] ZHU J，YANG Y，ZHANG H，et al. More than energy harvesting in electret electronics‐moving toward next‐generation functional system[J]. Advanced Functional Materials，2023，33(17)：2214859.
[12] ZHU J，WANG H，ZHANG Z，et al. Continuous direct current by charge transportation for next-generation IoT and real-time virtual reality applications[J]. Nano Energy，2020，73：104760.
[13] CHENG A J，WU L，SHA Z，et al. Recent advances of capacitive sensors：Materials，microstructure designs，applications，and opportunities[J]. Advanced Materials Technologies，2023，8(11)：2201959.
[14] FIORILLO A S，CRITELLO C D，PULLANO S A. Theory，technology and applications of piezoresistive sensors：A review[J]. Sensors and Actuators A：Physical，2018，281：156-175.
[15] LV C，TIAN C，JIANG J，et al. Ultrasensitive linear capacitive pressure sensor with wrinkled microstructures for tactile perception[J]. Advanced Science，2023，10(14)：2206807.
[16] JI S，YU J，SHAO H，et al. Machine learning-augmented wearable triboelectric human-machine interface in motion identification and virtual reality[J]. Nano Energy，2022，103：107766.
[17] CHIRILA R，DAHIYA A S，SCHYNS P，et al. Self-powered multimodal sensing using energy-generating solar skin for robotics and smart wearables[J]. Advanced Intelligent Systems，2024，6：2300824.
[18] ZHANG Z，WANG L，LEE C. Recent advances in artificial intelligence sensors[J]. Advanced Sensor Research，2023，2：2200072.
[19] MA M，ZHANG Z，ZHAO Z，et al. Self-powered flexible antibacterial tactile sensor based on triboelectric- piezoelectric-pyroelectric multi-effect coupling mechanism[J]. Nano Energy，2019，66：104105.
[20] MO Y，HAN H，LIU Y，et al. A tactile sensor based on piezoresistive effect and electromagnetic induction[J]. Sensors and Actuators A：Physical，2022，344：113716.
[21] CAMPO Y A S D，MOREIRA K S，LORENZETT E，et al. Electrostatic charging at the solid-liquid interface：Strategies for liquid flow sensing[J]. Friction，2025，13(2)：9440892.
[22] LIN S，ZHANG C，SHAO T. Techniques for surface charge measurements and exploring contact electrification[J]. Friction，2025，13(2)：9440968.
[23] NIU S，WANG S，LIN L，et al. Theoretical study of contact-mode triboelectric nanogenerators as an effective power source[J]. Energy & Environmental Science，2013，6(12)：3576.
[24] NIU S，WANG Z L. Theoretical systems of triboelectric nanogenerators[J]. Nano Energy，2015，14：161-192.