面向飞机驾驶舱面板操纵引导的多指动态位姿追踪与操作空间映射方法

doi:10.3901/JME.2025.15.441

机械工程学报 ›› 2025, Vol. 61 ›› Issue (15): 441-452.doi: 10.3901/JME.2025.15.441

• 人机协作装配与调度 • 上一篇

扫码分享

面向飞机驾驶舱面板操纵引导的多指动态位姿追踪与操作空间映射方法

申慧敏¹, 丁金涛¹, 沈梦若², 雷勇²

1. 上海理工大学机械工程学院上海 200093;
2. 浙江大学流体动力基础件与机电系统全国重点实验室杭州 310027

收稿日期:2024-10-08 修回日期:2025-03-03 出版日期:2025-08-05 发布日期:2025-09-28
作者简介:申慧敏(通信作者)，女，1987年出生，博士，副教授。主要研究方向为人-机交互技术及电磁检测技术。E-mail：hmshen@usst.edu.cn;丁金涛，男，2000年出生，硕士研究生。主要研究方向为人-机交互技术及电磁检测技术。E-mail：233381667@st.usst.edu.cn;沈梦若，女，2000年出生，博士研究生。主要研究方向为软组织交互形变建模与计算、手术导航系统。E-mail：shenmr@zju.edu.cn;雷勇，男，1976年出生，博士，教授。主要研究方向为软组织交互形变建模与计算、手术导航系统。E-mail：ylei@zju.edu.cn
基金资助:
国家自然科学基金资助项目(52175055)。

Multi-finger Dynamic Position Tracking and Operation Space Mapping Method for Aircraft Cockpit Panel Maneuver Guidance

SHEN Huimin¹, DING Jintao¹, SHEN Mengruo², LEI Yong²

1. School of Mechanical Engineering, Shanghai University of Technology, Shanghai 200093;
2. State Key Laboratory of Fluid Power and Mechatronic Systems, Zhejiang University, Hangzhou 310027

Received:2024-10-08 Revised:2025-03-03 Online:2025-08-05 Published:2025-09-28

摘要/Abstract

摘要： 随着工业5.0时代的到来，智能制造对人机协同提出了更高的要求。特别是在飞机总装测试过程中，驾驶舱面板的操作极为复杂，操作员需根据测试要求准确操作数百个按键和旋钮，极易因误操作而影响测试效率和飞机安全性。为解决该问题，提出一种基于磁-惯性传感信息融合的多指动态位姿追踪与操作空间映射方法。基于手部指骨生理约束，建立工作手指的D-H运动学模型；利用结构化磁场标记技术获取手部的基坐标空间位姿信息；通过磁-惯性传感信息融合，结合手指运动学模型和空间映射方法，实现作业指尖运动向操纵面板空间映射，可用于操作识别和判定。面向模拟操作面板开展了应用试验，包括按键顺序操作和旋钮操作识别，结果显示按键操作指尖平均定位误差±方差分布在[1.64, 3.73]±[0.44, 0.82] mm，旋钮旋转操作转角识别平均误差±方差分布在[2.11, 2.42]±[0.71, 1.00]°。验证所提出方法应用于驾驶舱面板精细操纵的有效性和适用性。

关键词: 磁-惯性传感信息融合, 手部运动学模型, 操作空间映射, 飞机驾驶舱面板

Abstract: The emergence of the Industry 5.0 era has raised the demands for enhanced human-machine collaboration, particularly in the aircraft final assembly and testing process. Operating cockpit panels is highly intricate, requiring operators to accurately manipulate hundreds of buttons and knobs based on test protocols. These complex tasks increase the risk of errors, potentially compromising testing efficiency and flight safety. To address this challenge, a novel method integrating magnetic-inertial sensor data fusion is proposed for multi-finger dynamic posture tracking and operational space mapping. This approach establishes a D-H kinematic model for working fingers based on physiological constraints of finger bones. Structured magnetic field markers are utilized to capture the spatial pose of the hand’s base coordinates. By combining magnetic-inertial sensor data with the kinematic model and spatial mapping techniques, the method enables precise mapping of fingertip movements to the operational panel space, facilitating accurate operation recognition and assessment. Validation tests are conducted on an emulated panel, covering sequential button operations and knob rotation recognition. The results indicate that the average positioning error ±variance for button operations ranges from [1.64, 3.73] ±[0.44, 0.82] mm, while the average angular error ±variance for knob operations falls within [2.11, 2.42] ±[0.71, 1.00] °. These outcomes confirm the method’s effectiveness and suitability for fine-grained cockpit panel manipulations.

Key words: magneto-inertial sensing information fusion, hand kinematics modeling, operational space mapping, aircraft cockpit panels

中图分类号:

TG156

申慧敏, 丁金涛, 沈梦若, 雷勇. 面向飞机驾驶舱面板操纵引导的多指动态位姿追踪与操作空间映射方法[J]. 机械工程学报, 2025, 61(15): 441-452.

SHEN Huimin, DING Jintao, SHEN Mengruo, LEI Yong. Multi-finger Dynamic Position Tracking and Operation Space Mapping Method for Aircraft Cockpit Panel Maneuver Guidance[J]. Journal of Mechanical Engineering, 2025, 61(15): 441-452.

导出引用管理器 EndNote|Reference Manager|ProCite|BibTeX|RefWorks

链接本文: http://www.cjmenet.com.cn/CN/10.3901/JME.2025.15.441

http://www.cjmenet.com.cn/CN/Y2025/V61/I15/441

参考文献

[1] 马南峰,姚锡凡,陈飞翔,等.面向工业5.0的人本智造[J].机械工程学报, 2022, 58(18):88-102. MA Nanfeng, YAO Xifan, CHEN Feixiang, et al. Human-centric smart manufacturing for Industry 5.0[J]. Journal of Mechanical Engineering, 2022, 58(18):88-102.
[2] 王柏村,易兵,刘振宇,等. HCPS视角下智能制造的发展与研究[J].计算机集成制造系统, 2021, 27(10):2749-2761. WANG Baicun, YI Bing, LIU Zhenyu, et al. Evolution and state-of-the-art of intelligent manufacturing from HCPS perspective[J]. Computer Integrated Manufacturing Systems, 2021, 27(10):2749-2761.
[3] JIANG C, ZHANG J, LONG T, et al. An optimization framework for worker allocation in aircraft final assembly lines based on simulation alternative modelling and historical data[J]. Engineering Optimization, 2023, 55(8):1387-1402.
[4] OTTOGALLI K, ROSQUETE D, ROJO J, et al. Virtual reality simulation of human-robot coexistence for an aircraft final assembly line:Process evaluation and ergonomics assessment[J]. International Journal of Computer Integrated Manufacturing, 2021, 34(9):975-995.
[5] WANG S, WANG X, WANG S, et al. Optical-nanofiber-enabled gesture-recognition wristband for human-machine interaction with the assistance of machine learning[J]. Advanced Intelligent Systems, 2023, 5(7):2200412.
[6] 王瑞平,吴士泓,张美航,等.基于视觉的神经网络三维动态手势识别方法综述[J].计算机科学, 2024, 51(4):193-208. WANG Ruiping, WU Shihong, ZHANG Meihang, et al. Review of vision-based neural network 3D dynamic gesture recognition methods[J]. Computer Science, 2024, 51(4):193-208.
[7] WANG F K, TANG M C, CHIU Y C, et al. Gesture Sensing using retransmitted wireless communication signals based on doppler radar technology[J]. IEEE Transactions on Microwave Theory&Techniques, 2015, 63(12):4592-4602.
[8] AMPRIMO G, MASI G, PETTITI G, et al. Hand tracking for clinical applications:Validation of the Google MediaPipe Hand (GMH) and the depth-enhanced GMH-D frameworks[J]. Biomedical Signal Processing and Control, 2024, 96:106508.
[9] WORRALLO AG, HARTLEY T. Robust optical based hand interaction for virtual reality[J]. IEEE Transactions on Visualization and Computer Graphics, 2021, 28(12):4186-4197.
[10] NI S, AL-QANESS M A A, HAWBANI A, et al. A survey on hand gesture recognition based on surface electromyography:Fundamentals, methods, applications, challenges and future trends[J]. Applied Soft Computing, 2024:112235.
[11] JIANG Y, REIMER V, SCHOSSIG T, et al. Fiber optical multifunctional human-machine interface for motion capture, temperature, and contact force monitoring[J]. Optics and Lasers in Engineering, 2020, 128:106018.
[12] 申慧敏,葛瑞康,顾晓伟,等.基于磁-惯性传感融合的灵巧手指单元全姿态信息反演方法[J].机械工程学报, 2022, 58(18):133-140. SHEN Huimin, GE Ruikang, GU Xiaowei, et al. Method of dexterous finger unit full-pose modeling based on magnetic-inertial sensor fusion[J]. Journal of Mechanical Engineering, 2022, 58(18):133-140.
[13] WANG Z, XIONG C, ZHANG Q. Enhancing the online estimation of finger kinematics from sEMG using LSTM with attention mechanisms[J]. Biomedical Signal Processing and Control, 2024, 92:105971.
[14] LEE Y, KIM M. Wearable finger tracking and cutaneous haptic interface with soft sensors for multi-fingered virtual manipulation[J]. IEEE/ASME Transactions on Mechatronics, 2018, 24(1):67-77.
[15] LEE Y, DO W. Visual-inertial hand motion tracking with robustness against occlusion, interference, and contact[J]. Science Robotics, 2021, 6(58):1315.
[16] SIMAO M A, GIBARU O, NETO P. Online recognition of incomplete gesture data to interface collaborative robots[J]. IEEE Transactions on Industrial Electronics, 2019, 66(12):9372-9382.
[17] LIU Y, TSAI YY. Virtual reality based tactile sensing enhancements for bilateral teleoperation system with in-hand manipulation[J]. IEEE Robotics and Automation Letters, 2022, 7(3):6998-7005.
[18] JIANG J, LEE K M. Digital magnetic tensor sensor with ANN measurement model for human joint motion sensing in sagittal plane[J]. IEEE/ASME Transactions on Mechatronics, 2022, 27(4):2062-2070.
[19] GRINYAGIN I V, BIRYUKOVA E V, MAIER M A. Kinematic and dynamic synergies of human precision-grip movements[J]. Journal of Neurophysiology, 2005, 94(4):2284-2294.
[20] MADGWICK S O H, HARRISON A J L, VAIDYANATHAN R. Estimation of IMU and MARG orientation using a gradient descent algorithm[C]//2011 IEEE International Conference on Rehabilitation Robotics, 2011, 1-7.
[21] ROGER B, GUERMAZI A, SKAF A, et al. Muscle injuries in sport athletes:Clinical essentials and imaging findings[M]. Cham:Springer International Publishing, 2017:452-475.
[22] BRAND P W, HOLLISTER A M. Clinical mechanics of the hand (3rd ed)[M]. Amsterdam:Mosby Publishing:Routledge, 1999.
[23] 王敏,孙景健,丁基恒,等.基于D-H参数与拉格朗日联立方程的仿生水蛇机器人运动学分析及动力学建模[J].机械工程学报, 2024, 60(15):134-148. WANG Min, SUN Jingjian, DING Jiheng, et al. Kinematics analysis and dynamics modeling of bionic water snake robot based on D-H parameters and lagrange equations[J]. Journal of Mechanical Engineering, 2024, 60(15):134-148.
[24] 卡潘德吉.骨关节功能解剖学[M].顾冬云,译.北京:人民军医出版社, 2011. KAPANDJI A. The physiology of the joints[M]. Translated by GU Dongyun. Beijing:People's Army Medical Press, 2011.
[25] AGARWAL P, YUN Y, FOX J, et al. Design, control, and testing of a thumb exoskeleton with series elastic actuation[J]. The International Journal of Robotics Research, 2017, 36(3):355-375.
[26] BARRELL Y, NAUS H W L. Detection and localisation of magnetic objects[J]. IET Science, Measurement&Technology, 2007, 1(5):245-254.
[27] WANG M, SONG S, LIU J, et al. Multipoint simultaneous tracking of wireless capsule endoscope using magnetic sensor array[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70:1-10.
[28] PYLVÄNÄINEN T. Automatic and adaptive calibration of 3D field sensors[J]. Applied Mathematical Modelling, 2008, 32(4):575-587.
[29] HU C, REN Y. Locating intra-body capsule object by three-magnet sensing system[J]. IEEE Sensors Journal, 2016, 16(13):5167-5176.

面向飞机驾驶舱面板操纵引导的多指动态位姿追踪与操作空间映射方法

Multi-finger Dynamic Position Tracking and Operation Space Mapping Method for Aircraft Cockpit Panel Maneuver Guidance

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	章新杰, 汪路航, 郭孔辉, 刘阳, 金耿瑞, 姚全. 主动外倾前束悬架系统协同优化设计方法[J]. 机械工程学报, 2026, 62(8): 21-32.
[2]	陈俊杰, 唐禹洲, 杨树军, 邬明宇. 双层帘线增强空气弹簧橡胶气囊各向异性超-黏弹性本构模型[J]. 机械工程学报, 2026, 62(8): 49-59.
[3]	彭闪闪, 贺林, 徐子昂, 王铭哲, 魏宇江, 石琴. 无感线控转向阻力矩比积观测器与转角反步控制[J]. 机械工程学报, 2026, 62(8): 114-124.
[4]	熊璐, 胡旭歌, 吕浩然, 唐辰. 面向车辆运动控制的路面不平度识别综述[J]. 机械工程学报, 2026, 62(8): 139-156.
[5]	吴建洋, 王俊业, 杨波, 丁晓林, 刘欣, 张雷. 多轴重载分布式电驱动车辆全轮转向控制研究[J]. 机械工程学报, 2026, 62(8): 317-331.
[6]	肖跃, 贺宜, 张鸣. 基于分层式MPC的分布式四轮驱动车辆轨迹跟踪控制方法[J]. 机械工程学报, 2026, 62(8): 475-488.
[7]	王裕祥, 高阳, 蒋恩超, 董娜, 陈旭东, 彭凡, 王立闻. 基于GAF-MACNN的风机叶片健康状态监测方法[J]. 机械工程学报, 2026, 62(7): 139-149.
[8]	雷卓, 乔百杰, 王亚南, 杜军, 梁俊, 刘元是, 伏宇, 陈雪峰. 基于SPOQ正则项的叶端定时信号稀疏重构[J]. 机械工程学报, 2026, 62(7): 208-220.
[9]	郭文华, 汪科, 郑稳, 王玉忠, 徐晨, 冀千瑜, 冯祖仁. 电子枪偏转电压均值估计标定方法[J]. 机械工程学报, 2026, 62(7): 385-395.
[10]	刘俊杰, 张友超, 张厚超, 张兵, 许权, 朱晓阳, 兰红波. 基于电场驱动微3D打印增减材复合制造高精度铜基柔性透明电路[J]. 机械工程学报, 2026, 62(7): 427-438.
[11]	葛正辉, 王睿, 胡启凡, 朱永伟. 铝合金表面电化学毛化处理及其粘接性能研究[J]. 机械工程学报, 2026, 62(7): 462-477.
[12]	邵健, 何安瑞, 杨荃. 宽幅热轧带钢高效集约生产与质量精益管控研究进展[J]. 机械工程学报, 2026, 62(6): 1-28.
[13]	周强, 陆红红, 郭登刚, 陈鹏万, 王宝雨. 爆炸焊接+轧制制备TA2/TC4钛合金复合材料界面微结构及力学行为研究[J]. 机械工程学报, 2026, 62(6): 100-110.
[14]	束学道, 陈宇森, 王英, 代月晨, 左锦荣, 张庆东. 铝合金空心轴形孔同步成形协调机制与智能调控[J]. 机械工程学报, 2026, 62(6): 121-131.
[15]	原源, 刘秀成, 祁攀, 曹文博, 王峥鸿, 吴斌, 高翔. 基于磁致伸缩SH₀导波传感器换能曲线的表面硬度无损检测方法[J]. 机械工程学报, 2026, 62(6): 228-236.