Environmental Worthiness Testing and Evaluation of Electromechanical System Containing AI Model

doi:10.3901/JME.2025.16.358

Abstract

Abstract: The profound integration of mechanization, informatization, and intelligentization represents the development trend of future equipment. An increasing number of electromechanical systems embed artificial intelligence(AI) models, necessitating the evaluation of environmental worthiness. This evaluation must not only consider hardware failures but also take into account the impact of the environment on the functionality of AI models. Aiming at the requirements of environmental worthiness evaluation for electromechanical systems containing AI models(ESAM), it takes the data distribution measurement function as a starting point, discusses the connotation of environmental worthiness in ESAM, and constructs an environmental worthiness index based on data distribution measurement. A quantitative evaluation method of environmental worthiness based on performance trajectories and distribution is proposed, considering that modeling methods using performance data can provide more quantitative insights by integrating model performance with environmental worthiness functions. Finally, using an AI-based visual perception system as an example, the study examines performance data under varying fog and vibration conditions in testing environments. It derives environmental worthiness curves and average failure discrepancy values, confirming the validity and effectiveness of the proposed method.

Key words: artificial intelligence model(AI), environmental worthiness, data distribution measurement, performance data modeling, meta-dataset

CLC Number:

TB114

ZHANG Shufeng, SONG Guofeng, LI Xingge, CHEN Xun. Environmental Worthiness Testing and Evaluation of Electromechanical System Containing AI Model[J]. Journal of Mechanical Engineering, 2025, 61(16): 358-373.

References

[1] 中国人民解放军总装备部电子信息基础部. 装备环境工程通用要求: GJB 4239—2001[S]. 北京:总装备部军标出版发行部, 2001. The Electronic Information Foundation Department of the General Armament Department of PLA. General requirements for materiel environmental engineering: GJB 4239—2001[S]. Beijing: Military Standard Publishing and Distribution Department of the General Armament Department, 2001.
[2] 中国人民解放军总装备部电子信息基础部. 装备环境工程术语: GJB 6117—2007[S]. 北京:总装备部军标出版发行部, 2007. The Electronic Information Foundation Department of the General Armament Department of PLA. Materiel environmental engineering terms: GJB 6117—2007[S]. Beijing: Military Standard Publishing and Distribution Department of the General Armament Department, 2007.
[3] 韦正现. 智能装备试验与测试的挑战与对策思考[J]. 测控技术, 2021, 40(2): 1-5. WEI Zhengxian. Challenge and countermeasure of intelligent equipment experiment and test[J]. Measurement & Control Technology, 2021, 40(2): 1-5.
[4] 李颖,王荣煊,万成麟,等. 分体式飞行汽车立体环境感知系统设计及试验研究[J]. 机械工程学报, 2024, 60(10): 102-111. LI Ying, WANG Rongxuan, WAN Chenglin, et al. Design and experiment research of 3D environmental perception system for split-type flying vehicle[J]. Journal of Mechanical Engineering, 2024, 60(10): 102-111.
[5] 蒋瑜. 频谱可控的超高斯随机振动环境模拟技术及其应用研究[D]. 长沙:国防科学技术大学, 2005. JIANG Yu. Research on the simulation of super-gaussian random vibration environment with controllable frequency spectrum and its applications[D]. Changsha: National University of Defense Technology, 2005.
[6] LI Li, WANG Xiao, WANG Kunfeng, et al. Parallel testing of vehicle intelligence via virtual-real interaction[J]. Science Robotics, 2019, 4(28): eaaw4106.
[7] 朱冰,张培兴,赵健,等. 基于场景的自动驾驶汽车虚拟测试研究进展[J]. 中国公路学报, 2019, 32(6): 1-19. ZHU Bing, ZHANG Peixin, ZHAO Jian, et al. Review of scenario-based virtual validation methods for automated vehicles[J]. China Journal of Highway and Transport, 2019, 32(6): 1-19.
[8] CARLSON J, MURPHY R R. How UGVs physically fail in the field[J]. IEEE Transactions on Robotics, 2005, 21(3): 423-437.
[9] 邵禹铭. 基于虚拟现实的无人车测试环境构建研究[D]. 西安:长安大学, 2019. SHAO Yuming. Research on the construction of unmanned vehicle test environment based on virtual reality[D]. Xi’an: Chang’an University, 2019.
[10] 王田苗,张韬懿,梁建宏,等. 基于再生能源的极地漫游机器人研究及现场试验[J]. 机械工程学报, 2013, 49(19): 21-30. WANG Tianmiao, ZHANG Taoyi, LIANG Jianhong, et al. Design and field test of a rover robot for antarctic based on renewable energy[J]. Journal of Mechanical Engineering, 2013, 49(19): 21-30.
[11] LI Nianshi, LIU Xiaoyong, YU Bendong, et al. Study on the environmental adaptability of lithium-ion battery powered UAV under extreme temperature conditions[J]. Energy, 2021, 219: 119481.
[12] OMIDSHAFIEI S, AGHA-MOHAMMADI A A, CHEN Y F, et al. Measurable augmented reality for prototyping cyberphysical systems: A robotics platform to aid the hardware prototyping and performance testing of algorithms[J]. IEEE Control Systems Magazine, 2016, 36(6): 65-87.
[13] 武月琴,傅耘. 装备环境适应性表征方法研究[J]. 装备环境工程, 2008, 5(6): 52-55. WU Yueqin, FU Yun. Research on characterization of equipment environmental worthiness[J]. Equipment Environmental Engineering, 2008, 5(6): 52-55.
[14] 王群,陈之光,汪文. 基于灰色理论的电子装备环境适应性评判模型[J]. 装备环境工程, 2010, 7(6): 94-98. WANG Qun, CHEN Zhiguang, WANG Wen, et al. Environmental worthiness model of electronic equipment based on grey theory[J]. Equipment Environmental Engineering, 2010, 7(6): 94-98.
[15] 张礼学,王中伟. 平流层飞艇环境适应性评价模型[J]. 航空学报, 2013, 34(4): 719-726. ZHANG Lixue , WANG Zhongwei. Environmental adaptability evaluation model for stratospheric airships[J]. Acta Aeronautica et Astronautica Sinica, 2013, 34(4): 719-726.
[16] 孟光磊,李树发,刘彬斌,等. 防空预警雷达高原环境适应性评估的自学习模糊灰度方法[J]. 兵工学报, 2022, 43(1): 98-110. MENG Guanglei , LI Shufa , LIU Binbin , et al. Self-learning fuzzy grey method for plateau environmental adaptability assessment of air defense early-warning radar[J]. Acta Armamentarii, 2022, 43(1): 98-110.
[17] HONG Y, LIAN J, XU L, et al. Statistical perspectives on reliability of artificial intelligence systems[J]. Quality Engineering, 2023, 35 (1): 56-78.
[18] DENG W, GOULD S, ZHENG L. What does rotation prediction tell us about classifier accuracy under varying testing environments?[C]//Proceedings of the 38th International Conference on Machine Learning. New York: PMLR, 2021: 2579-2589.
[19] MIN J, HONG Y, KING C B, et al. Reliability analysis of artificial intelligence systems using recurrent events data from autonomous vehicles[J]. Journal of the Royal Statistical Society Series C: Applied Statistics. 2022, 71 (4): 987-1013.
[20] JAFARZADEH M, AHMAD T, DHAMIJA A R, et al. Automatic open-world reliability assessment[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA. NewYork: IEEE, 2021: 1984-1993.
[21] 彭湃,耿可可,王子威,等. 智能汽车环境感知方法综述[J]. 机械工程学报, 2023, 59(20): 281-303. PENG Pai, GENG Keke, WANG Ziwei, et al. Review on environmental perception methods of autonomous vehicles[J]. Journal of Mechanical Engineering, 2023, 59(20): 281-303.
[22] 秦逸. 面向环境不确定性的 CPS 测试技术研究[D]. 南京:南京大学, 2018. QIN Yi. Software testing for cyber-physical systems suffering uncertainty[D]. Nanjing: Nanjing University, 2018.
[23] OBUKHOV A, KRASNYANSKIY M. Quality assessment method for GAN based on modified metrics inception score and Fréchet inception distance[J]. Proceedings of 4th Computational Methods in Systems and Software, 2020, 1: 102-114.
[24] HEUSEL M, RAMSAUER H, UNTERTHINER T, et al. GANs trained by a two time-scale update rule converge to a local nash equilibrium[J]. Asian Journal of Applied Science and Engineering, 2019, 8: 25-34.
[25] 贾学志,张雷,安源,等. 空间光学遥感器精密调焦机构设计与试验[J]. 机械工程学报, 2016, 52(13): 25-30. JIA Xuezhi, ZHANG Lei, AN Yuan, et al. Design and experiment research on precision focusing mechanism of space remote sensor[J]. Journal of Mechanical Engineering, 2016, 52(13): 25-30.
[26] 向红标,杨大虎,杨璐,等. 复杂环境下磁弹性微型游泳机器人的路径规划与识别跟踪[J]. 机械工程学报,2023, 59(5): 89-99. XIANG Hongbiao, YANG Dahu, YANG Lu, et al. Path planning and recognition tracking of a magnetoelastic miniature swimmer in complex environment[J]. Journal of Mechanical Engineering, 2023, 59(5): 89-99.
[27] LI Xingge, ZHANG Shufeng , CHEN Xun , et al. Robustness of visual perception system in progressive challenging weather scenarios[J]. Engineering Applications of Artificial Intelligence. 2023, 119: 105740.