多模态手势辨识臂环的关键技术研究与实现

doi:10.3901/JME.260220

机械工程学报 ›› 2026, Vol. 62 ›› Issue (10): 50-61.doi: 10.3901/JME.260220

• 仪器科学与技术 • 上一篇

扫码分享

多模态手势辨识臂环的关键技术研究与实现

赵子敬, 高玉泽, 万永涛, 朱建雄

东南大学机械工程学院南京 210096

收稿日期:2025-05-07 修回日期:2025-12-04 发布日期:2026-07-29
作者简介:赵子敬,男,1998年出生,硕士研究生。主要研究方向为可穿戴传感器。E-mail:220220388@seu.edu.cn;高玉泽,男,2002年出生,硕士研究生。主要研究方向为机器视觉与深度学习。E-mail:220240467@seu.edu.cn;万永涛,男,2001年出生,硕士研究生。主要研究方向为深度学习。E-mail:220240418@seu.edu.cn;朱建雄(通信作者),男,1983年出生,博士,副教授,博士研究生导师。主要研究方向为可穿戴柔性感知、气体嗅觉感知、机器学习与人工智能、数字孪生与物联网、光电感知应用。E-mail:mezhujx@seu.edu.cn
基金资助:
长三角联合攻关(2024CSJGG01401)和国家自然科学基金(62471127) 资助项目。

Research and Implementation of Key Technologies of Multimodal Gesture Recognition Armband

ZHAO Zijing, GAO Yuze, WAN Yongtao, ZHU Jianxiong

School of Mechanical and Engineering, Southeast University, Nanjing 210096

Received:2025-05-07 Revised:2025-12-04 Published:2026-07-29

摘要/Abstract

摘要： 在智能穿戴技术快速发展、多模态手势辨识成为研究热点的背景下，针对传统肌电臂环在复杂场景的局限，提出融合摩擦电传感器与陀螺仪的多模态手势辨识臂环系统。研发过程中，研制基于铜箔与聚四氟乙烯(Polytetrafluoroethylene, PTFE)-Ecoflex材料复合结构的传感单元，其在2 N压力内摩擦电压信号达4.6 V，拓展为阵列并结合陀螺仪实现多模态数据采集。依据人体前臂肌肉分布和运动特点设计传感阵列，标定6通道最佳位置，以3D打印热塑性聚氨酯(Thermoplastic Polyurethane, TPU)材料制作臂环主体及传感阵列，提升性能与舒适性。同时，创新性提出应用于该臂环的增强型Transformer深度学习模型，通过结合时序增强模块增加数据集多样性，运用多尺度特征提取、位置编码等方法融合多模态信息，捕捉数据特征和位置关系，将手势识别准确率提升至95%。

关键词: 穿戴式传感器, 摩擦电传感器, 臂环系统, 多模态手势辨识, 深度学习

Abstract: Against the backdrop of the rapid development of smart wearable technology and the rise of multimodal gesture recognition as a research hotspot, limitations of traditional electromyography armbands in complex scenarios are addressed, and a multimodal gesture recognition armband system integrating triboelectric sensors and gyroscopes is proposed. A sensing unit with a composite structure of copper foil and Polytetrafluoroethylene(PTFE)-Ecoflex materials is developed; under a pressure of 2 N, its triboelectric voltage signal reaches 4.6 V. The sensing unit is expanded into an array and combined with a gyroscope to enable multimodal data collection. Based on the muscle distribution and movement characteristics of the human forearm, a sensing array is designed, and the optimal positions of six channels are calibrated. The armband body and sensing array are fabricated via 3D printing using thermoplastic polyurethane(TPU) material, which enhances both performance and comfort. An enhanced Transformer deep learning model tailored for the armband is innovatively proposed: a temporal enhancement module is incorporated to increase dataset diversity, and methods such as multi-scale feature extraction and position encoding are applied to fuse multimodal information, capturing data features and positional relationships. As a result, gesture recognition accuracy is raised to 95%, while resource consumption is reduced.

Key words: wearable sensor, triboelectric sensor, armband system, multimodal gesture recognition, deep learning

中图分类号:

TP216

赵子敬, 高玉泽, 万永涛, 朱建雄. 多模态手势辨识臂环的关键技术研究与实现[J]. 机械工程学报, 2026, 62(10): 50-61.

ZHAO Zijing, GAO Yuze, WAN Yongtao, ZHU Jianxiong. Research and Implementation of Key Technologies of Multimodal Gesture Recognition Armband[J]. Journal of Mechanical Engineering, 2026, 62(10): 50-61.

导出引用管理器 EndNote|Reference Manager|ProCite|BibTeX|RefWorks

链接本文: http://www.cjmenet.com.cn/CN/10.3901/JME.260220

http://www.cjmenet.com.cn/CN/Y2026/V62/I10/50

参考文献

[1] YAO K，ZHOU J，HUANG Q，et al. Encoding of tactile information in hand via skin-integrated wireless haptic interface[J]. Nature Machine Intelligence，2022，4(10)：893-903.
[2] LIU L，ZHANG X. A focused review on the flexible wearable sensors for sports：From kinematics to physiologies[J]. Micromachines，2022，13(8)：1356.
[3] PYO S，LEE J，BAE K，et al. Recent progress in flexible tactile sensors for human‐interactive systems：From sensors to advanced applications[J]. Advanced Materials，2021，33(47)：2005902.
[4] MONGARDI A，ROSSI F，PRESTIA A，et al. Hand gestures recognition for human-machine interfaces：A low-power bio-inspired armband[J]. IEEE Transactions on Biomedical Circuits and Systems，2022，16(6)：1348-1365.
[5] GAO W，OTA H，KIRIYA D，et al. Flexible electronics toward wearable sensing[J]. Accounts of Chemical Research，2019，52(3)：523-533.
[6] SHAO Q，SNIFFEN A，BLANCHET J，et al. Teaching american sign language in mixed reality[J]. Proceedings of the ACM on Interactive，Mobile，Wearable and Ubiquitous Technologies，2020，4(4)：1-27.
[7] BASIRI S，TAHERI A，MEGHDARI A，et al. Design and implementation of a robotic architecture for adaptive teaching：A case study on iranian sign language[J]. Journal of Intelligent & Robotic Systems，2021，102(2)：48.
[8] LEE K T，CHEE P S，LIM E H，et al. Data glove with integrated polyethylene-carbon composite-based strain sensor for virtual reality applications[J]. Chemical Engineering & Technology，2023，46(12)：2480-2486.
[9] WONG W K，JUWONO F H，KHOO B T T. Multi-features capacitive hand gesture recognition sensor：A machine learning approach[J]. IEEE Sensors Journal，2021，21(6)：8441-8450.
[10] RAO H C，LUO B B，WU D C，et al. Study on the design and performance of a glove based on the fbg array for hand posture sensing[J]. Sensors，2023，23(20)：8495.
[11] ZHU J，YANG Y，ZHANG H，et al. More than energy harvesting in electret electronics‐moving toward next‐generation functional system[J]. Advanced Functional Materials，2023，33(17)：2214859.
[12] ZHU J，WANG H，ZHANG Z，et al. Continuous direct current by charge transportation for next-generation IoT and real-time virtual reality applications[J]. Nano Energy，2020，73：104760.
[13] CHENG A J，WU L，SHA Z，et al. Recent advances of capacitive sensors：Materials，microstructure designs，applications，and opportunities[J]. Advanced Materials Technologies，2023，8(11)：2201959.
[14] FIORILLO A S，CRITELLO C D，PULLANO S A. Theory，technology and applications of piezoresistive sensors：A review[J]. Sensors and Actuators A：Physical，2018，281：156-175.
[15] LV C，TIAN C，JIANG J，et al. Ultrasensitive linear capacitive pressure sensor with wrinkled microstructures for tactile perception[J]. Advanced Science，2023，10(14)：2206807.
[16] JI S，YU J，SHAO H，et al. Machine learning-augmented wearable triboelectric human-machine interface in motion identification and virtual reality[J]. Nano Energy，2022，103：107766.
[17] CHIRILA R，DAHIYA A S，SCHYNS P，et al. Self-powered multimodal sensing using energy-generating solar skin for robotics and smart wearables[J]. Advanced Intelligent Systems，2024，6：2300824.
[18] ZHANG Z，WANG L，LEE C. Recent advances in artificial intelligence sensors[J]. Advanced Sensor Research，2023，2：2200072.
[19] MA M，ZHANG Z，ZHAO Z，et al. Self-powered flexible antibacterial tactile sensor based on triboelectric- piezoelectric-pyroelectric multi-effect coupling mechanism[J]. Nano Energy，2019，66：104105.
[20] MO Y，HAN H，LIU Y，et al. A tactile sensor based on piezoresistive effect and electromagnetic induction[J]. Sensors and Actuators A：Physical，2022，344：113716.
[21] CAMPO Y A S D，MOREIRA K S，LORENZETT E，et al. Electrostatic charging at the solid-liquid interface：Strategies for liquid flow sensing[J]. Friction，2025，13(2)：9440892.
[22] LIN S，ZHANG C，SHAO T. Techniques for surface charge measurements and exploring contact electrification[J]. Friction，2025，13(2)：9440968.
[23] NIU S，WANG S，LIN L，et al. Theoretical study of contact-mode triboelectric nanogenerators as an effective power source[J]. Energy & Environmental Science，2013，6(12)：3576.
[24] NIU S，WANG Z L. Theoretical systems of triboelectric nanogenerators[J]. Nano Energy，2015，14：161-192.

多模态手势辨识臂环的关键技术研究与实现

Research and Implementation of Key Technologies of Multimodal Gesture Recognition Armband

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	雷佳为, 王姝婷, 岳重祥, 林萍, 彭文, 孙杰, 张殿华. 基于深度学习的热轧厚度-活套动态建模[J]. 机械工程学报, 2026, 62(6): 142-153.
[2]	徐扬欢, 王东城, 刘宏民, 党丽莹, 刘云飞, 杨爱民. 基于深度学习的图像板形检测方法研究[J]. 机械工程学报, 2026, 62(6): 185-196.
[3]	李雨露, 李俊峰, 万章艺, 魏正英. 基于YOLO算法的激光粉末床熔融成形层形貌分类识别与成形质量预测研究[J]. 机械工程学报, 2026, 62(3): 218-234.
[4]	王祺, 叶仁传, 马国杰, 马佩珏, 范杰, 杨文龙. 钢材表面缺陷轻量化识别算法研究[J]. 机械工程学报, 2025, 61(8): 18-31.
[5]	秦飞红, 袁懋诞, 祝端阳, 刘晓睿, 李明, 纪轩荣. 基于多层特征融合目标检测网络的超声全聚焦钢板焊缝缺陷自动检测研究[J]. 机械工程学报, 2025, 61(4): 55-66.
[6]	陈其怀, 林添良, 马荣华, 温建和, 缪骋, 任好玲. 基于多模态的履带式挖掘机自动驾驶决策网络[J]. 机械工程学报, 2025, 61(4): 380-388.
[7]	黄思翰, 陈建鹏, 徐哲, 阎艳, 王国新. 基于大语言模型和机器视觉的智能制造系统人机自主协同作业方法[J]. 机械工程学报, 2025, 61(3): 130-141.
[8]	刘翰儒, 陈桂映, 许泽林, 彭世通, 郭嘉楠, 刘伟嵬, 王奉涛. 基于SERepVGG-A2的轻量化卷积神经网络在熔池状态识别中的应用[J]. 机械工程学报, 2025, 61(3): 440-448.
[9]	赵阳, 王中任, 史铁林, 谭孟, 张文, 李文喜. 基于Swin-IMSA的外圆磨削过程中粗糙度在机预测方法[J]. 机械工程学报, 2025, 61(23): 373-380.
[10]	战庆亮, 刘鑫, 白春锦, 葛耀君. 三维高分辨率流场重构的深度学习方法[J]. 机械工程学报, 2025, 61(16): 338-346.
[11]	肖宏, 王阳, 刘秀波, 崔旭浩, 张智海, 金锋. 钢轨波磨检测原理、方法及装置研究进展[J]. 机械工程学报, 2025, 61(10): 191-214.
[12]	余智祺, 徐洋, 王元飞, 盛晓伟. 基于多尺度标准化流模型的布匹缺陷检测[J]. 机械工程学报, 2025, 61(10): 241-249.
[13]	李路兴, 魏超, 胡乐云, 随淑鑫, 徐扬, 钱歆昊. 多模态多序列高效越野可行驶区域检测[J]. 机械工程学报, 2025, 61(10): 276-287.
[14]	李春凯, 王嘉昕, 石玗, 代悦. GTAW熔池激光条纹动态行为与熔透预测方法研究[J]. 机械工程学报, 2024, 60(6): 236-244.
[15]	史丽晨, 史炜椿, 王海涛, 李金阳, 刘亚雄. 基于DRSN-BiLSTM模型的刀具磨损预测方法研究[J]. 机械工程学报, 2024, 60(24): 66-74.