Multimodal Interactive Evaluation and Optimization Design of Intelligent Helmet

doi:10.3901/JME.2023.11.117

Abstract

Abstract: With the development of intelligent wearable equipment, an increasing number of interaction modalities can be selected. Simple interaction modal superposition not only fails to meet interaction needs, but also causes interaction cost issues such as user information overload and increased cognitive load. Therefore, the importance of multi-modal interaction evaluation and optimal design of smart wearable devices has become increasingly prominent. Taking the intelligent helmet as an example, a multi-modal interactive evaluation method (EUC method) based on efficiency, user experience, and cost is proposed. And optimize the design for the modal selection of military and police intelligent helmets based on EUC method. Among them, the interactive evaluation test is based on task performance measures (i.e., task completion time, error rate and number of operations) and user perception measures (i.e., perceived usability and workload), as well as using ANOVA and other methods to study the correlation and influence between interaction indicators and variables to complete the effectiveness of multiple modal combinations under boundary conditions and multiple postures. Finally, the optimal design suggestions and solutions for the multi-modal flexible configuration are proposed for the military and police intelligent helmets based on evaluation results.

Key words: wearable equipment, multi-modal fusion, interaction cost, interaction evaluation, optimal design

CLC Number:

TP302

PENG Jian, WANG Xuepeng, JI Tie, CHEN Junchun. Multimodal Interactive Evaluation and Optimization Design of Intelligent Helmet[J]. Journal of Mechanical Engineering, 2023, 59(11): 117-128.

References

[1] JÖST M. Multimodal interaction for pedestrians:an evaluation study[C]// Proceedings of the 10th International Conference on Intelligent User Interfaces，January 10-13，2005，San Diego，California. New York:Association for Computing Machinery，2005:59-66.
[2] 孙瀚. 基于多模态生物电信号人机交互技术研究[D]. 南京:东南大学，2019. SUN Han. Research of human-computer interaction technology based on multi-modal biopotentials[D]. Nanjing:Southeast University，2019.
[3] COHEN P. Synergistic use of direct manipulation and natural language[C]// Proceedings of the SIGCHI Conference on Human Factors in Computing Systems，April 30- June 4，1989，Austin，Texas. New York:Association for Computing Machinery，1989:227-233.
[4] OVIATT S. Advances in robust multimodal interface design[J]. IEEE Computer Graphics and Applications，2003，23(05):62-68.
[5] OVIATT S. When do we interact multimodally? Cognitive load and multimodal communication patterns[C]// Proceedings of the 6th International Conference on Multimodal Interfaces，October 13-15，2004，State College PA. New York:Association for Computing Machinery，2004:129-136.
[6] 袁晓东. 模块化智能头盔的应用设计研究[J]. 包装工程，2015，36(22):88-91，115. YUAN Xiaodong. The application design of modular intelligent helmet[J]. Packaging Engineering，2015，36(22):88-91，115.
[7] TRIANTAFYLLIDIS E，MCGREAVY C，GU Jiacheng，et al. Study of multimodal interfaces and the improvements on teleoperation[J]. IEEE Access，2020，8:78213-78227.
[8] CHO E，MOLINA M，WANG Jinping. The effects of modality，device，and task differences on perceived human likeness of voice-activated virtual assistants[J]. Cyberpsychology，Behavior，and Social Networking，2019，22(8):515-520.
[9] SCHWARZ F，FASTENMEIER W. Augmented reality warnings in vehicles:Effects of modality and specificity on effectiveness[J]. Accident Analysis & Prevention，2017，101:55-66.
[10] NUOVO A，BROZ F，WANG Ning，et al. The multi-modal interface of robot-era multi-robot services tailored for the elderly[J]. Intelligent Service Robotics，2018，11(1):109-126.
[11] CAROLIS B，GEMMIS M，LOPS P，et al. Recognizing users feedback from non-verbal communicative acts in conversational recommender systems[J]. Pattern Recognition Letters，2017，99:87-95.
[12] REN Gang，WEI Side，O’NEILL E，et al. Towards the design of effective haptic and audio displays for augmented reality and mixed reality applications[J]. Advances in Multimedia，2018，2018:1-11.
[13] MERKOURIS A，CHORIANOPOULOU B，CHORIA-NOPOULOS K，et al. Understanding the notion of friction through gestural interaction with a remotely controlled robot[J]. Journal of Science Education and Technology，2019，28(3):209-221.
[14] WANG Xiangyu，ONG S，NEE A. Multi-modal augmented-reality assembly guidance based on bare-hand interface[J]. Advanced Engineering Informatics，2016，30(3):406-421.
[15] NUAMAH J，MANTOOTH W，KARTHIKEYAN R，et al. Neural efficiency of human-robotic feedback modalities under stress differs with gender[J]. Frontiers in Human Neuroscience，2019，13:287.
[16] HEPPERLE D，WEIS Y，SIESS A，et al. 2D，3D or speech? A case study on which user interface is preferable for what kind of object interaction in immersive virtual reality[J]. Computers & Graphics，2019，82:321-331.
[17] RAU P，ZHENG Jian，WEI Yi. Distractive effect of multimodal information in multisensory learning[J]. Computers & Education，2020，144:103699.
[18] MONTUWY A，DOMMES A，CAHOUR B. Helping older pedestrians navigate in the city: comparisons of visual，auditory and haptic guidance instructions in a virtual environment[J]. Behaviour & Information Technology，2019，38(2):150-171.
[19] WANG Jinling，CHELLALI A，CAO C. Haptic communication in collaborative virtual environments[J]. Human Factors，2016，58(3):496-508.
[20] ESTEVES A，SHIN Y，OAKLEY I. Comparing selection mechanisms for gaze input techniques in head-mounted displays[J]. International Journal of Human-Computer Studies，2020，139:102414.
[21] LUZIO F，LAURETTI C，CORDELLA F，et al. Visual vs vibrotactile feedback for posture assessment during upper-limb robot-aided rehabilitation[J]. Applied Ergonomics，2020，82:102950.
[22] KIM Y，HONG S，KIM G J. Augmented reality-based remote coaching for fast-paced physical task[J]. Virtual Reality，2018，22(1):25-36.
[23] DOUKAKIS E，DEBATTISTA K，BASHFORD- ROGERS T，et al. Audio-visual-olfactory resource allocation for tri-modal virtual environments[J]. IEEE Transactions on Visualization and Computer Graphics，2019，25(5):1865-1875.
[24] SCHÜSSEL F，HONOLD F，WEBER M. Influencing factors on multimodal interaction during selection tasks[J]. Journal on Multimodal User Interfaces，2013，7(4):299-310.
[25] HELLER J，CHYLINSKI M，RUYTER K，et al. Touching the untouchable:Exploring multi-sensory augmented reality in the context of online retailing[J]. Journal of Retailing，2019，95(4):219-234.
[26] XIAO R，SCHWARZ J，THROM N，et al. MRTouch:Adding touch input to head-mounted mixed reality[J]. IEEE Transactions on Visualization and Computer Graphics，2018，24(4):1653-1660.
[27] BAJPAI A，POWELL J，YOUNG A，et al. Enhancing physical human evasion of moving threats using tactile cues[J]. IEEE Transactions on Haptics，2019，13(1):32-37.
[28] RAU P，ZHENG Jian. Modality capacity and appropriateness in multimodal display of complex non-semantic information stream[J]. International Journal of Human-Computer Studies，2019，130:166-178.
[29] HACIOMEROGLU O. C-Thru; Smoke Diving Helmet [EB/OL]. [2022-03-16]. https://www.qwake.tech/.
[30] HELD B，ALJUNEIDI S，PHAM V，et al. Helon 360: A smart firefighters' helmet Integrated augmented reality and 360° thermal image data streaming[EB/OL]. (2019-09) [2022-06-12]. https://www.researchgate.net/publication/ 341764419_Helon_360_A_smart_firefighters'_helmet_Integrated_Augmented_Reality_and_360_Thermal_Image_Data_Streaming.
[31] CAMERON A，HOBSON R. Head-up，eyes-out in day and at night:striker HMD，evolution or revolution?[C]// Degraded Visual Environments:Enhanced，Synthetic，and External Vision Solutions 2016，17-21 April，2016，Baltimore，Maryland. Bellingham:SPIE Press，2016:246-256.
[32] SCHRADIN R. The F-35 Helmet - a modern marvel that creates a unique maintenance challenge[EB/OL]. (2022-06-08) [2022-06-12]. https://govdesignhub.com/2022/06/08/the-f-35-helmet-a-modern-marvel-that-creates-a-unique-maintenance-challenge/.
[33] U.S. Army Program Executive Office. Project manager integrated visual augmentation system (IVAS)[EB/ OL]. (2022-02-19) [2022-06-13]. https://www.peosoldier.amy.mil/Program-Offices/Project-Manager-Integrated-Visual-Augmentation-System/.
[34] CHENG Qiang，SUN Jiping，ZHANG Zhe，et al. ZigBee based intelligent helmet for coal miners[C]// 2009 WRI World Congress on Computer Science and Information Engineering，31 March - 2 April，2009，Los Angeles，California. Washington:IEEE Computer Society，2009:433-435.
[35] NITHYA T，EZAK M，KUMAR K，et al. Rescue and protection system for underground mine workers based on Zigbee[J]. Int. J. Recent Res. Asp，2018，4(4):194-197.
[36] KIM S，WANG C，MIN S，et al. Safety helmet wearing management system for construction workers using three-axis accelerometer sensor[J]. Applied Sciences，2018，8(12):2400.
[37] LI Ping，MEZIANE R，OTIS M，et al. A Smart Safety Helmet using IMU and EEG sensors for worker fatigue detection[C]// 2014 IEEE International Symposium on Robotic and Sensors Environments (ROSE) Proceedings，October 16-18，2014，Timisoara，Romania. New York:IEEE Press，2014:55-60.
[38] WANG C，KIM Y，KIM D，et al. Smart helmet and insole sensors for near fall incidence recognition during descent of stairs[J]. Applied Sciences，2020，10(7):2262.
[39] PIRKL G，HEVESI P，AMARISLANOV O，et al. Smart helmet for construction site documentation and work support[C]// Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing:Adjunct，September 12-16，2016，Heidelberg，Germany. New York:ACM，2016:349-352.
[40] LEE A，MOON J，MIN S，et al. Safety Analysis System using Smart Helmet[C]// Proceedings on the International Conference on Internet Computing (ICOMP). July 29- August 1，2019，Las Vegas，Nevada. Athens:CSREA Press，2019:102-107.
[41] MOHAMMED M，SYAMSUDIN H，AL-ZUBAIDI S，et al. Novel COVID-19 detection and diagnosis system using IOT based smart helmet[J]. International Journal of Psychosocial Rehabilitation，2020，24(7):2296-2303.
[42] BISIO I，FEDELI A，LAVAGETTO F，et al. Mobile smart helmet for brain stroke early detection through neural network-based signals analysis[C]// 2017 IEEE Global Communications Conference，December 4-8，2017，Singapore. New York:IEEE，2017:1-6.
[43] SHAHIDUZZAMAN K，HEI Xiaojun，GUO C，et al. Enhancing fall detection for elderly with smart helmet in a cloud-network-edge architecture[C]// 2019 IEEE Inter-national Conference on Consumer Electronics-Taiwan (ICCE-TW)，20-22 May，2019，YiLan，Taiwan，China. New York:IEEE，2019:1-2.
[44] RASLI M，MADZHI N，JOHARI J. Smart helmet with sensors for accident prevention[C]// 2013 International Conference on Electrical，Electronics and System Engineering (ICEESE)，4-5 December，2013，Selangor，Malaysia. New York:IEEE，2013:21-26.
[45] MELCHER V，DIEDERICHS F，MAESTRE R，et al. Smart vital signs and accident monitoring system for motorcyclists embedded in helmets and garments for advanced eCall emergency assistance and health analysis monitoring[J]. Procedia Manufacturing，2015，3:3208-3213.
[46] MOHANAROOPA D，SOUJANYA N，VAISHNAVI V，et al. An IOT based smart helmet for accident detection and notification[J]. Journal of Interdisciplinary Cycle Research，2020，7:1-7.
[47] RAO P，SAI P，KUMAR N，et al. Design and implementation of smart helmet using IoT[C]// Inter-national Conference of Advance Research & Innovation (ICARI)，19 January，2020，New Delhi，India. Delhi:International Journal of Advance Research and Innovation，2020:323-325.
[48] GABBARD J，SWAN J. Usability engineering for augmented reality:Employing user-based studies to inform design[J]. IEEE Transactions on Visualization and Computer Graphics，2008，14(3):13-525.
[49] 孙启玉，蒋晓. 交互成本在移动互联网产品设计中的应用研究[J]. 大众文艺，2015(2):102. SUN Qiyu，JIANG Xiao. Application of interaction cost in mobile Internet product design[J]. Popular Literature，2015(2):102.
[50] WANG Zhimin，WANG Haofei，YU Huangyue，et al. Interaction with gaze，gesture，and speech in a flexibly configurable augmented reality system[J]. IEEE Transactions on Human-Machine Systems，2021，51(5):524-534.
[51] RAZZAK M，ISLAM M. Exploring and evaluating the usability factors for military application:A road map for HCI in military applications[J]. Human Factors and Mechanical Engineering for Defense and Safety，2020，4(1):1-18.
[52] WEISS A. The USUS evaluation framework for human-robot interaction[C]// AISB 2009，Proceedings of the Symposium on New Frontiers in Human-robot Interaction. April 8-9，2009，Edinburgh，UK. Brighton:SSAISB，2009:11-26.
[53] LEWIS J. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use[J]. International Journal of Human‐Computer Interaction，1995，7:57-78.
[54] SEVINC V，ILKER B. Psychometric evaluation of Simulator Sickness Questionnaire and its variants as a measure of cybersickness in consumer virtual environment[J]. Applied Ergonomics，2020，82:102958.
[55] LOU Xiaolong，PENG Ren，HANSEN P，et al. Effects of user’s hand orientation and spatial movements on free hand interactions with large displays[J]. International Journal of Human-Computer Interaction，2018，34(6): 519-532.
[56] HART S，STAVELAND L. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research[J]. Advances in Psychology，1988，52:139-183.
[57] FAUL F，ERDFELDER E，LANG A，et al. G* Power 3:A flexible statistical power analysis program for the social，behavioral，and biomedical sciences[J]. Behavior Research Methods，2007，39(2):175-191.
[58] VEALE J. Edinburgh handedness inventory-short form:a revised version based on confirmatory factor analysis[J]. Laterality:Asymmetries of Body，Brain and Cognition，2014，19(2):164-177.
[59] PIUMSOMBOON T. User-defined gestures for augmented reality[C]// Proceedings of the IFIP Conference on Human-Computer Interaction. September 2-6，2013，Cape Town，South Africa. New York:Springer，2013:282-299.
[60] 张在房，徐冯，孙习武. 火箭贮箱箱底充液拉深成形工艺的多目标优化[J]. 机械工程学报，2022，58(5):78-86. ZHANG Zaifang，XU Feng，SUN Xiwu. Multi-objective optimization of hydroforming process of rocket tank bottom[J]. Journal of Mechanical Engineering，2022，58(5):282-299.
[61] MARTOWIBOWO S，KASWADI A. Optimization and simulation of plastic injection process using genetic algorithm and moldflow[J]. Chinese Journal of Mechanical Engineering，2017，30(2):398-406.
[62] 万良琪，陈洪转，欧阳林寒，等. 基于灰色模糊田口方法的复杂装备精密产品多质量特性稳健优化设计[J]. 计算机集成制造系统，2018，24(6):1427-1437. WAN Liangqi，CHEN Hongzhuan，OUYANG Linhan，et al. Multi-quality characteristics robust optimization design of complex equipment compliant mechanism based on hybrid gray-fuzzy-Taguchi method[J]. Computer Integrated Manufacturing Systems，2018，24(6):1427-1437.
[63] 王田田. 基于多域均衡的复杂装备能效优化设计方法及其应用研究[D]. 杭州:浙江大学，2020. WANG Tiantian. Research on energy efficiency optimization design method and application of complex equipment based on multi-domain equilibrium[D]. Hangzhou:Zhejiang University，2020.
[64] 谢友柏. 现代设计理论中的若干基本概念[J]. 机械工程学报，2007，43(11):7-16. XIE Youbai. Some basic concepts in modern design theory[J]. Journal of Mechanical Engineering，2007，43(11):7-16.
[65] 闻邦椿. 产品全功能与全性能的综合设计[M]. 北京:机械工业出版社，2008. WEN Bangchun. Comprehensive design of product full function performance[M]. Beijing:China Machine Press，2008.
[66] SUH N，DO S. Axiomatic design of software systems[J]. CIRP Annals - Manufacturing Technology，2000，49(1):95-100.