面向飞机驾驶舱面板操纵引导的多指动态位姿追踪与操作空间映射方法

doi:10.3901/JME.2025.15.441

机械工程学报 ›› 2025, Vol. 61 ›› Issue (15): 441-452.doi: 10.3901/JME.2025.15.441

• 人机协作装配与调度 • 上一篇

扫码分享

面向飞机驾驶舱面板操纵引导的多指动态位姿追踪与操作空间映射方法

申慧敏¹, 丁金涛¹, 沈梦若², 雷勇²

1. 上海理工大学机械工程学院上海 200093;
2. 浙江大学流体动力基础件与机电系统全国重点实验室杭州 310027

收稿日期:2024-10-08 修回日期:2025-03-03 发布日期:2025-09-28
作者简介:申慧敏(通信作者)，女，1987年出生，博士，副教授。主要研究方向为人-机交互技术及电磁检测技术。E-mail：hmshen@usst.edu.cn;丁金涛，男，2000年出生，硕士研究生。主要研究方向为人-机交互技术及电磁检测技术。E-mail：233381667@st.usst.edu.cn;沈梦若，女，2000年出生，博士研究生。主要研究方向为软组织交互形变建模与计算、手术导航系统。E-mail：shenmr@zju.edu.cn;雷勇，男，1976年出生，博士，教授。主要研究方向为软组织交互形变建模与计算、手术导航系统。E-mail：ylei@zju.edu.cn
基金资助:
国家自然科学基金资助项目(52175055)。

Multi-finger Dynamic Position Tracking and Operation Space Mapping Method for Aircraft Cockpit Panel Maneuver Guidance

SHEN Huimin¹, DING Jintao¹, SHEN Mengruo², LEI Yong²

1. School of Mechanical Engineering, Shanghai University of Technology, Shanghai 200093;
2. State Key Laboratory of Fluid Power and Mechatronic Systems, Zhejiang University, Hangzhou 310027

Received:2024-10-08 Revised:2025-03-03 Published:2025-09-28

摘要/Abstract

摘要： 随着工业5.0时代的到来，智能制造对人机协同提出了更高的要求。特别是在飞机总装测试过程中，驾驶舱面板的操作极为复杂，操作员需根据测试要求准确操作数百个按键和旋钮，极易因误操作而影响测试效率和飞机安全性。为解决该问题，提出一种基于磁-惯性传感信息融合的多指动态位姿追踪与操作空间映射方法。基于手部指骨生理约束，建立工作手指的D-H运动学模型；利用结构化磁场标记技术获取手部的基坐标空间位姿信息；通过磁-惯性传感信息融合，结合手指运动学模型和空间映射方法，实现作业指尖运动向操纵面板空间映射，可用于操作识别和判定。面向模拟操作面板开展了应用试验，包括按键顺序操作和旋钮操作识别，结果显示按键操作指尖平均定位误差±方差分布在[1.64, 3.73]±[0.44, 0.82] mm，旋钮旋转操作转角识别平均误差±方差分布在[2.11, 2.42]±[0.71, 1.00]°。验证所提出方法应用于驾驶舱面板精细操纵的有效性和适用性。

关键词: 磁-惯性传感信息融合, 手部运动学模型, 操作空间映射, 飞机驾驶舱面板

Abstract: The emergence of the Industry 5.0 era has raised the demands for enhanced human-machine collaboration, particularly in the aircraft final assembly and testing process. Operating cockpit panels is highly intricate, requiring operators to accurately manipulate hundreds of buttons and knobs based on test protocols. These complex tasks increase the risk of errors, potentially compromising testing efficiency and flight safety. To address this challenge, a novel method integrating magnetic-inertial sensor data fusion is proposed for multi-finger dynamic posture tracking and operational space mapping. This approach establishes a D-H kinematic model for working fingers based on physiological constraints of finger bones. Structured magnetic field markers are utilized to capture the spatial pose of the hand’s base coordinates. By combining magnetic-inertial sensor data with the kinematic model and spatial mapping techniques, the method enables precise mapping of fingertip movements to the operational panel space, facilitating accurate operation recognition and assessment. Validation tests are conducted on an emulated panel, covering sequential button operations and knob rotation recognition. The results indicate that the average positioning error ±variance for button operations ranges from [1.64, 3.73] ±[0.44, 0.82] mm, while the average angular error ±variance for knob operations falls within [2.11, 2.42] ±[0.71, 1.00] °. These outcomes confirm the method’s effectiveness and suitability for fine-grained cockpit panel manipulations.

Key words: magneto-inertial sensing information fusion, hand kinematics modeling, operational space mapping, aircraft cockpit panels

中图分类号:

TG156

申慧敏, 丁金涛, 沈梦若, 雷勇. 面向飞机驾驶舱面板操纵引导的多指动态位姿追踪与操作空间映射方法[J]. 机械工程学报, 2025, 61(15): 441-452.

SHEN Huimin, DING Jintao, SHEN Mengruo, LEI Yong. Multi-finger Dynamic Position Tracking and Operation Space Mapping Method for Aircraft Cockpit Panel Maneuver Guidance[J]. Journal of Mechanical Engineering, 2025, 61(15): 441-452.

参考文献

[1] 马南峰,姚锡凡,陈飞翔,等.面向工业5.0的人本智造[J].机械工程学报, 2022, 58(18):88-102. MA Nanfeng, YAO Xifan, CHEN Feixiang, et al. Human-centric smart manufacturing for Industry 5.0[J]. Journal of Mechanical Engineering, 2022, 58(18):88-102.
[2] 王柏村,易兵,刘振宇,等. HCPS视角下智能制造的发展与研究[J].计算机集成制造系统, 2021, 27(10):2749-2761. WANG Baicun, YI Bing, LIU Zhenyu, et al. Evolution and state-of-the-art of intelligent manufacturing from HCPS perspective[J]. Computer Integrated Manufacturing Systems, 2021, 27(10):2749-2761.
[3] JIANG C, ZHANG J, LONG T, et al. An optimization framework for worker allocation in aircraft final assembly lines based on simulation alternative modelling and historical data[J]. Engineering Optimization, 2023, 55(8):1387-1402.
[4] OTTOGALLI K, ROSQUETE D, ROJO J, et al. Virtual reality simulation of human-robot coexistence for an aircraft final assembly line:Process evaluation and ergonomics assessment[J]. International Journal of Computer Integrated Manufacturing, 2021, 34(9):975-995.
[5] WANG S, WANG X, WANG S, et al. Optical-nanofiber-enabled gesture-recognition wristband for human-machine interaction with the assistance of machine learning[J]. Advanced Intelligent Systems, 2023, 5(7):2200412.
[6] 王瑞平,吴士泓,张美航,等.基于视觉的神经网络三维动态手势识别方法综述[J].计算机科学, 2024, 51(4):193-208. WANG Ruiping, WU Shihong, ZHANG Meihang, et al. Review of vision-based neural network 3D dynamic gesture recognition methods[J]. Computer Science, 2024, 51(4):193-208.
[7] WANG F K, TANG M C, CHIU Y C, et al. Gesture Sensing using retransmitted wireless communication signals based on doppler radar technology[J]. IEEE Transactions on Microwave Theory&Techniques, 2015, 63(12):4592-4602.
[8] AMPRIMO G, MASI G, PETTITI G, et al. Hand tracking for clinical applications:Validation of the Google MediaPipe Hand (GMH) and the depth-enhanced GMH-D frameworks[J]. Biomedical Signal Processing and Control, 2024, 96:106508.
[9] WORRALLO AG, HARTLEY T. Robust optical based hand interaction for virtual reality[J]. IEEE Transactions on Visualization and Computer Graphics, 2021, 28(12):4186-4197.
[10] NI S, AL-QANESS M A A, HAWBANI A, et al. A survey on hand gesture recognition based on surface electromyography:Fundamentals, methods, applications, challenges and future trends[J]. Applied Soft Computing, 2024:112235.
[11] JIANG Y, REIMER V, SCHOSSIG T, et al. Fiber optical multifunctional human-machine interface for motion capture, temperature, and contact force monitoring[J]. Optics and Lasers in Engineering, 2020, 128:106018.
[12] 申慧敏,葛瑞康,顾晓伟,等.基于磁-惯性传感融合的灵巧手指单元全姿态信息反演方法[J].机械工程学报, 2022, 58(18):133-140. SHEN Huimin, GE Ruikang, GU Xiaowei, et al. Method of dexterous finger unit full-pose modeling based on magnetic-inertial sensor fusion[J]. Journal of Mechanical Engineering, 2022, 58(18):133-140.
[13] WANG Z, XIONG C, ZHANG Q. Enhancing the online estimation of finger kinematics from sEMG using LSTM with attention mechanisms[J]. Biomedical Signal Processing and Control, 2024, 92:105971.
[14] LEE Y, KIM M. Wearable finger tracking and cutaneous haptic interface with soft sensors for multi-fingered virtual manipulation[J]. IEEE/ASME Transactions on Mechatronics, 2018, 24(1):67-77.
[15] LEE Y, DO W. Visual-inertial hand motion tracking with robustness against occlusion, interference, and contact[J]. Science Robotics, 2021, 6(58):1315.
[16] SIMAO M A, GIBARU O, NETO P. Online recognition of incomplete gesture data to interface collaborative robots[J]. IEEE Transactions on Industrial Electronics, 2019, 66(12):9372-9382.
[17] LIU Y, TSAI YY. Virtual reality based tactile sensing enhancements for bilateral teleoperation system with in-hand manipulation[J]. IEEE Robotics and Automation Letters, 2022, 7(3):6998-7005.
[18] JIANG J, LEE K M. Digital magnetic tensor sensor with ANN measurement model for human joint motion sensing in sagittal plane[J]. IEEE/ASME Transactions on Mechatronics, 2022, 27(4):2062-2070.
[19] GRINYAGIN I V, BIRYUKOVA E V, MAIER M A. Kinematic and dynamic synergies of human precision-grip movements[J]. Journal of Neurophysiology, 2005, 94(4):2284-2294.
[20] MADGWICK S O H, HARRISON A J L, VAIDYANATHAN R. Estimation of IMU and MARG orientation using a gradient descent algorithm[C]//2011 IEEE International Conference on Rehabilitation Robotics, 2011, 1-7.
[21] ROGER B, GUERMAZI A, SKAF A, et al. Muscle injuries in sport athletes:Clinical essentials and imaging findings[M]. Cham:Springer International Publishing, 2017:452-475.
[22] BRAND P W, HOLLISTER A M. Clinical mechanics of the hand (3rd ed)[M]. Amsterdam:Mosby Publishing:Routledge, 1999.
[23] 王敏,孙景健,丁基恒,等.基于D-H参数与拉格朗日联立方程的仿生水蛇机器人运动学分析及动力学建模[J].机械工程学报, 2024, 60(15):134-148. WANG Min, SUN Jingjian, DING Jiheng, et al. Kinematics analysis and dynamics modeling of bionic water snake robot based on D-H parameters and lagrange equations[J]. Journal of Mechanical Engineering, 2024, 60(15):134-148.
[24] 卡潘德吉.骨关节功能解剖学[M].顾冬云,译.北京:人民军医出版社, 2011. KAPANDJI A. The physiology of the joints[M]. Translated by GU Dongyun. Beijing:People's Army Medical Press, 2011.
[25] AGARWAL P, YUN Y, FOX J, et al. Design, control, and testing of a thumb exoskeleton with series elastic actuation[J]. The International Journal of Robotics Research, 2017, 36(3):355-375.
[26] BARRELL Y, NAUS H W L. Detection and localisation of magnetic objects[J]. IET Science, Measurement&Technology, 2007, 1(5):245-254.
[27] WANG M, SONG S, LIU J, et al. Multipoint simultaneous tracking of wireless capsule endoscope using magnetic sensor array[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70:1-10.
[28] PYLVÄNÄINEN T. Automatic and adaptive calibration of 3D field sensors[J]. Applied Mathematical Modelling, 2008, 32(4):575-587.
[29] HU C, REN Y. Locating intra-body capsule object by three-magnet sensing system[J]. IEEE Sensors Journal, 2016, 16(13):5167-5176.

面向飞机驾驶舱面板操纵引导的多指动态位姿追踪与操作空间映射方法

Multi-finger Dynamic Position Tracking and Operation Space Mapping Method for Aircraft Cockpit Panel Maneuver Guidance

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	董非, 易幼平, 黄始全, 何海林. 2060铝锂合金超低温变形行为与复杂薄壁件拉深成形工艺研究[J]. 机械工程学报, 2025, 61(14): 20-28.
[2]	亢鑫, 杨光, 焦娇, 赵子皓, 凡晓波. 超低温变形与时效交互作用下的2219铝合金组织性能演变规律[J]. 机械工程学报, 2025, 61(14): 63-72.
[3]	姚梦佳, 王睿乾, 刘伟. 2195铝锂合金超低温变形各向异性行为及组织演变规律[J]. 机械工程学报, 2025, 61(14): 73-80.
[4]	白鑫, 沈童, 王凡勋, 殷国栋, 王金湘, 方睿祺, 李新修, 梁晋豪. 基于驾驶人操纵稳定域的分布式驱动电动汽车多模态扭矩分配策略研究[J]. 机械工程学报, 2025, 61(14): 166-183.
[5]	杨云帆, 王瑞晨, 常宇健, 陈美, 刘鹏飞. 车轮镟修对重载机车车轮非圆化磨耗消除及演化发展影响研究[J]. 机械工程学报, 2025, 61(14): 184-196.
[6]	武秀恒, 孙亮, 赵晋海, 李亮, 王翔宇, 宋正河. 基于轮组制动力异相位波动重构的智能多轴半挂汽车制动防抱死控制研究[J]. 机械工程学报, 2025, 61(14): 243-260.
[7]	康祥, 李鹏飞, 白雪儿, 李云. 非设计工况下活塞式压缩机热力性能预测算法探究[J]. 机械工程学报, 2025, 61(14): 297-305.
[8]	冯毅雄, 金柯兵, 洪兆溪, 张安思, 李少波, 谭建荣. 面向算网资源智能适配与融合调度的价值链协同模式研究[J]. 机械工程学报, 2025, 61(13): 20-28.
[9]	赵泽田, 胡炳涛, 冯毅雄, 宋秀菊, 谭建荣. 复杂产品全生命周期价值链协同：理论、进展与趋势[J]. 机械工程学报, 2025, 61(13): 96-119.
[10]	罗子超, 刘秀波, 程巍, 李新功, 夏杰, 郑军. 碳化硅陶瓷材料的摩擦与润滑研究进展[J]. 机械工程学报, 2025, 61(13): 192-212.
[11]	陈诺, 李德才. 磁性液体密封自愈合特性的影响因素实验研究[J]. 机械工程学报, 2025, 61(13): 213-221.
[12]	靳茂鹏, 张厚超, 王飞, 李言胜, 陈硕, 李义睿, 白彦军, 李红珂, 许权, 赵佳伟, 张广明, 兰红波. 基于电场驱动微3D打印复合增材制造银/铜/镍基金属网格透明天线[J]. 机械工程学报, 2025, 61(13): 315-326.
[13]	张俊, 李怡, 汤伟民, 葛姝翌, 郭德敏, 落海伟. 基于边界几何方法的铣削加工仿真[J]. 机械工程学报, 2025, 61(13): 418-428.
[14]	刘立飞, 梁奉爽, 吴明阳, 张明德. 基于动态压痕的RB-SiC陶瓷裂纹生成-扩展及材料去除机理研究[J]. 机械工程学报, 2025, 61(13): 449-460.
[15]	李超, 范孟豹, 曹丙花, 孙凤山, 叶波, 严佳航. 热障涂层电涡流检测新变压器模型与信号解耦方法研究[J]. 机械工程学报, 2025, 61(12): 12-25.