Abstract: The emergence of the Industry 5.0 era has raised the demands for enhanced human-machine collaboration, particularly in the aircraft final assembly and testing process. Operating cockpit panels is highly intricate, requiring operators to accurately manipulate hundreds of buttons and knobs based on test protocols. These complex tasks increase the risk of errors, potentially compromising testing efficiency and flight safety. To address this challenge, a novel method integrating magnetic-inertial sensor data fusion is proposed for multi-finger dynamic posture tracking and operational space mapping. This approach establishes a D-H kinematic model for working fingers based on physiological constraints of finger bones. Structured magnetic field markers are utilized to capture the spatial pose of the hand’s base coordinates. By combining magnetic-inertial sensor data with the kinematic model and spatial mapping techniques, the method enables precise mapping of fingertip movements to the operational panel space, facilitating accurate operation recognition and assessment. Validation tests are conducted on an emulated panel, covering sequential button operations and knob rotation recognition. The results indicate that the average positioning error ±variance for button operations ranges from [1.64, 3.73] ±[0.44, 0.82] mm, while the average angular error ±variance for knob operations falls within [2.11, 2.42] ±[0.71, 1.00] °. These outcomes confirm the method’s effectiveness and suitability for fine-grained cockpit panel manipulations.

Key words: magneto-inertial sensing information fusion, hand kinematics modeling, operational space mapping, aircraft cockpit panels