思维链范式智能运维文本基多模态智能体

doi:10.3901/JME.2025.23.058

机械工程学报 ›› 2025, Vol. 61 ›› Issue (23): 58-74.doi: 10.3901/JME.2025.23.058

扫码分享

思维链范式智能运维文本基多模态智能体

黄金凤¹, 王成城², 何宏亮^1,3, 王旭⁴, 李奇¹, 杨康定³, 王凯², 张飞斌¹, 秦朝烨¹, 褚福磊¹

1. 清华大学高端装备界面科学与技术全国重点实验室北京 100084;
2. 机械工业仪器仪表综合技术经济研究所北京 100055;
3. 频率探索智能科技江苏有限公司常州 213162;
4. 智能博弈与决策实验室北京 100091

收稿日期:2025-01-09 修回日期:2025-09-06 出版日期:2025-12-05 发布日期:2026-01-22
作者简介:黄金凤,女,1991年出生,博士。主要研究方向为机械故障诊断及智能运维。E-mail:hjinfeng1991@163.com
张飞斌(通信作者),男,1986年出生,博士,助理研究员。主要研究方向为机械动力学、信号处理和智能运维。E-mail:zfeibin@mail.tsinghua.edu.cn
褚福磊,男,1959 年出生,博士,教授,博士研究生导师。主要研究方向为大型旋转机械的动力学分析和故障诊断。E-mail:chufl@mail.tsinghua.edu.cn;E-mail:chufl@mail.tsinghua.edu.cn
基金资助:
国家自然科学基金(52305115, 52105109, 52161135101);中国博士后科学基金(2023M741938)资助项目

Chain-of-thought Paradigm Text-based Multimodal Intelligent Agent for Equipment Operations and Maintenance

HUANG Jinfeng¹, WANG Chengcheng², HE Hongliang^1,3, WANG Xu⁴, LI Qi¹, YANG Kangding³, WANG Kai², ZHANG Feibin¹, QIN Zhaoye¹, CHU Fulei¹

1. Department of Mechanical Engineering, Tsinghua University, Beijing 100084;
2. Standards and Testing Center, Instrumentation Technology and Economy Institute, Beijing 100055;
3. FreqX Intelligence Technology Co., Ltd., Changzhou 213162;
4. Intelligent Game and Decision Lab, Beijing 100091

Received:2025-01-09 Revised:2025-09-06 Online:2025-12-05 Published:2026-01-22

摘要/Abstract

摘要： 提出了一种面向机械设备运维的人工智能架构——思维链范式文本基多模态智能体。首先，针对实际工程应用场景中，难以直接构建“监测数据-故障模式”高质量大规模数据集的问题，建立了“监测信号-数学特征-文本描述-故障模式”思维链式数据集构建思想，进而提出了信号-文本数据生成器驱动的信号转文本(Sig2Txt)模型。其次，制作了机械设备运维领域的高质量专业化文本数据集，并基于指令微调技术和通用大语言模型建立了智能运维专业文本大模型。最后，基于大模型智能体技术有机融合上述模型，以人类专家开展设备运维的思维模式为指引，创建了思维链范式智能运维文本基多模态智能体。测试结果表明，该智能体可实现多模态输入数据-运维决策的全流程思维链式解析和映射，在ISO三级振动分析师测试题上的准确率超过70%，达到人类专家级水平。在工程案例和公开数据组成的多模态数据集“热启动”评测中，相比于现有主流大模型，该模型展现出了更优异的性能。更重要的是，得益于所提出的低成本、高质量多模态思维链式大规模数据集的构建体系，以及“文本基”对于知识的包罗万象、高度凝练和易于解释的天然独特优势，该模型具备可观的拓展性和发展潜力。

关键词: 智能运维, 多模态模型, 大语言模型, 故障诊断, 智能体

Abstract: An artificial intelligence architecture—the chain-of-thought (CoT) paradigm text-based multimodal intelligent agent—for operation and maintenance (O&M) of mechanical equipment is proposed. Firstly, to address the challenge of constructing high-quality, large-scale monitoring data-to-fault mode mapping datasets in real-world engineering applications, a chain-of-thought dataset construction strategy integrating monitoring signals, mathematical features, text descriptions, and fault mode is proposed. Based on this, a signal-to-text (Sig2Txt) model driven by a signal-text data generator is developed. Subsequently, a high-quality specialized textual dataset for O&M in the mechanical equipment domain is created, and an intelligent O&M-specialized large language model is established through instruction fine-tuning on a general large language model. Finally, by organically integrating the above models based on large model intelligent agent technology and guided by the operational thinking patterns of human experts in equipment maintenance, a chain-of-thought paradigm text-based multimodal intelligent agent for intelligent O&M is formed. Testing results indicate that this model can achieve chain-of-thought parsing and mapping from multimodal input decision-making, with an accuracy exceeding 70% on ISO Category III vibration analyst test questions, thus reaching expert-level performance. In evaluations with engineering cases and publicly available multimodal datasets, the proposed model outperforms existing mainstream large models. More importantly, owing to the proposed low-cost, high-quality multimodal CoT large-scale dataset construction framework, and the unique advantages of a “text-based” approach in terms of encompassing knowledge, high-level abstraction, and interpretability, the model shows considerable scalability and development potential.

Key words: intelligent operation and maintenance, multimodal model, large language model, fault diagnosis, intelligent agent

中图分类号:

TH17

黄金凤, 王成城, 何宏亮, 王旭, 李奇, 杨康定, 王凯, 张飞斌, 秦朝烨, 褚福磊. 思维链范式智能运维文本基多模态智能体[J]. 机械工程学报, 2025, 61(23): 58-74.

HUANG Jinfeng, WANG Chengcheng, HE Hongliang, WANG Xu, LI Qi, YANG Kangding, WANG Kai, ZHANG Feibin, QIN Zhaoye, CHU Fulei. Chain-of-thought Paradigm Text-based Multimodal Intelligent Agent for Equipment Operations and Maintenance[J]. Journal of Mechanical Engineering, 2025, 61(23): 58-74.

参考文献

[1] 臧冀原,刘宇飞,王柏村,等. 面向2035的智能制造技术预见和路线图研究[J]. 机械工程学报,2022,58(4):285-308. ZANG Jiyuan,LIU Yufei,WANG Baicun,et al. Technology forecasting and roadmapping of intelligent manufacturing by 2035[J]. Journal of Mechanical Engineering,2022,58(4):285-308.
[2] 汪俊亮,高鹏捷,张洁,等. 制造大数据分析综述:内涵、方法、应用和趋势[J]. 机械工程学报,2023,59(12):1-16.WANG Junliang,GAO Pengjie,ZHANG Jie,et al. A review of manufacturing big data:connotation,methodology,application and trends[J]. Journal of Mechanical Engineering,2023,59(12):1-16.
[3] TAUHEED M,ANURAG C,SHAHAB F. Multi-sensor fault diagnosis for misalignment and unbalance detection using machine learning[J]. IEEE Transactions on Industry Applications,2023,59(5):5749-59.
[4] LEI Yaguo,YANG Bin,JIANG Xinwei,et al. Applications of machine learning to machine fault diagnosis:A review and roadmap[J]. Mechanical Systems and Signal Processing,2020,138:106587.
[5] CHEN Jiaxian,HUANG Ruyi,CHEN Zhuyun,et al. Transfer learning algorithms for bearing remaining useful life prediction:A comprehensive review from an industrial application perspective[J]. Mechanical Systems and Signal Processing,2023,193:110239.
[6] 黄包裕,张永祥.一种基于三步法的轴承故障诊断方法[J]. 振动工程学报,2024(14):51-68. HUANG Baoyu,ZHANG Yongxiang. Rolling element bearing fault diagnosis using a three-step scheme[J]. Journal of Mechanical Engineering,2024(14):51-68.
[7] 任杉,王晋,赵欣,等. 复杂产品智能设计与主动运维"双馈式"制造服务方法体系[J]. 机械工程学报,2024,60(06):127-36. REN Shan,WANG Jin,ZHAO Xin,et al. "Doubly-fed" manufacturing service of intelligent design and preventive maintenance for complex products[J]. Journal of Mechanical Engineering,2024,60(06):127-36.
[8] ZHENG Zhe,WANG Fei,GONG Guofang,et al. Intelligent technologies for construction machinery using data-driven methods[J]. Automation in Construction,2023,147:104711.
[9] DENG J,DONG W,SOCHER R,et al. Imagenet:A large-scale hierarchical image database[C]. 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE,2009:248-255.
[10] FANG A,KORNBLITH S,SCHMIDT L. Does progress on ImageNet transfer to real-world datasets[J]. Advances in Neural Information Processing Systems,2024,36:25050-25080.
[11] BROWN T B. Language models are few-shot learners[J]. Arxiv,Preprint Arxiv:2005.14165,2020.
[12] WU T,HE S,LIU J,et al. A brief overview of ChatGPT:The history,status quo and potential future development[J]. IEEE/CAA Journal of Automatica Sinica,2023,10(5):1122-1136.
[13] SILVER D,SCHRITTWIESER J,SIMONYAN K,et al. Mastering the game of go without human knowledge[J]. Nature,2017,550(7676):354-359.
[14] MIAO Q,ZHENG W,LV Y,et al. DAO to HANOI via DeSci:AI paradigm shifts from AlphaGo to ChatGPT[J]. IEEE/CAA Journal of Automatica Sinica,2023,10(04):877-897.
[15] CHIB P,SINGH P. Recent advancements in end-to-end autonomous driving using deep learning:A survey[J]. IEEE Transactions on Intelligent Vehicles,2023.
[16] HAYDARI A,YILMAZ Y. Deep reinforcement learning for intelligent transportation systems:A survey[J]. IEEE Transactions on Intelligent Transportation Systems,2020,23(1):11-32.
[17] FEDUS W,ZOPH B,and SHAZEER N,Switch trans formers:Scaling to trillion parameter models with simple and efficient sparsity[J]. Journal of Machine Learning Research,2022,23(120):1-40.
[18] RADFORD A,WU J,CHILD R,et al.,Language models are unsuper vised multitask learners[R/OL]. San Francisco:OpenAI,2019-02. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf.
[19] BROWN T. Language models are few-shot learners[J]. Arxiv,Preprint Arxiv:2005.14165,2020.
[20] OpenAI. GPT-4 Technical Report[EB/OL]. San Francisco:OpenAI,2023-03-27[2025-08-04]. https://cdn.openai.com/papers/gpt-4.pdf.
[21] ZENG A,LIU X,DU Z,et al. Glm-130b:An open bilingual pre-trained model[J]. Arxiv,Preprint Arxiv:2210.02414,2022.
[22] WEI J,TAY Y,BOMMASANI R,et al. Emergent abilities of large language models[J]. Arxiv,Preprint Arxiv,2206.07682,2022.
[23] BUBECK S,CHANDRASEKARAN V,ELDAN R,et al. Sparks of artificial general intelligence:Early experiments with gpt-4[J]. Arxiv,Preprint Arxiv:2303.12712,2023.
[24] VASWANI A,SHAZEER N,et al. Attention is all you need[J]. Arxiv,Preprint Arxiv:1706.03762,2023.
[25] HUANG Jinfeng,ZHANG Feibin,SAFAEI B,et al. The flexible tensor singular value decomposition and its applications in multisensor signal fusion processing[J]. Mechanical Systems and Signal Processing,2024,220:111662.
[26] HUANG Jinfeng,ZHANG Feibin,Coombs T,et al. The first-kind flexible tensor SVD:innovations in multi-sensor data fusion processing[J]. Nonlinear Dynamics,2025,113:6541-6559.
[27] 张贤达. 现代信号处理[M]. 清华大学出版社有限公司,2015. ZHANG Xianda. Moderm signal processing[M]. Beijing:Press of Tsinghua University,2015.
[28] FENG Z,ZUO M. Vibration signal models for fault diagnosis of planetary gearboxes[J]. Journal of Sound and Vibration,2012,331(22):4919-4939.
[29] Jorj X. McKie. PyMuPDF:A high performance Python library for data extraction,analysis,conversion & manipulation of PDF (and other) documents[EB/OL]. 2023- 05. https://github.com/pymupdf/PyMuPDF.
[30] VASILIEV Y. Natural language processing with Python and spaCy:A practical introduction[M]. No Starch Press,2020.
[31] NAYAK N,NAN Y,TROST A,et al. Learning to generate instruction tuning datasets for zero-shot task adaptation[J]. Arxiv,Preprint Arxiv:2402.18334,2024.
[32] International Organization for Standardization. ISO 18436‑2:2014 Condition monitoring and diagnostics of machines-Requirements for qualification and assessment of personnel-Part 2:Vibration condition monitoring and diagnostics[S]. Geneva:ISO,2014.
[33] LI Junan,LI Dongxu,SAVARESE S,et al. BLIP-2:bootstrapping language-image pretraining with frozen image encoders and large language models[C]//International Conference on Machine Learning,2023,19730-19742.
[34] ZHE Yutong ZHANG Rui,ZHANG Jie,et al. Llamafactory:Unified efficient fine-tuning of 100+ language models[J]. Arxiv,Preprint Arxiv:2403.13372,2024.
[35] GLM T,ZENG A,XU B,et al. Chatglm:A family of large language models from glm-130b to glm-4 all tools[J]. Arxiv,Preprint Arxiv:2406.12793,2024.
[36] HU E,SHEN Y,WALLIS P,et al. Lora:Low-rank adaptation of large language models[J]. Arxiv,Preprint Arxiv:2106.09685,2021.
[37] LOSHCHILOV I,HUTTER F. Sgdr:Stochastic gradient descent with warm restarts[J]. Arxiv,Preprint Arxiv:1608.03983,2016.
[38] ZHAO Xiaoli. Intelligent fault diagnosis of gearbox under variable working conditions with adaptive intraclass and interclass convolutional neural network[J]. IEEE Transactions on Neural Networks and Learning Systems,2023,34(9):6339-6353.
[39] XU Haifeng,WANG Xu,HUANG Jinfeng,et al. Semi-supervised multi-sensor information fusion tailored graph embedded low-rank tensor learning machine under extremely low labeled rate[J]. Information Fusion,2024,105:102222.
[40] YEUNG Y,PAUL-AJUWAPE A,TAHIRY F,et al. RoSA:A mechatronically synthesized dataset for rotodynamic system anomaly detection[J]. 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems,2022.
[41] HENG A,ZHANG Sheng,TAN A,et al. Rotating machinery prognostics:State of the art,challenges and opportunities[J]. Mechanical Systems and Signal Processing,2009,23:724-739.
[42] 王国彪,何正嘉,等,机械故障诊断基础研究"何去何从"[J]. 机械工程学报,2013,49(1):63-72. WANG Guobiao,HE Zhengjia,et al. Basic research on machinery fault diagnosis-what is the prescription[J]. Journal of Mechanical Engineering,2013,49(1):63-72.
[43] WEI Jason,WANG Xuezhi,SCHUURMANS Dale,et al. Chain-of-thought prompting elicits reasoning in large language models[J]. Advances in Neural Information Processing Systems,2022,35:24824-24837.
[44] GUO T,CHEN X,WANG Y,et al. Large language model based multi-agents:A survey of progress and challenges[J]. Arxiv,Preprint Arxiv:2402.01680,2024.
[45] WENG Lilian. Llm powered autonomous agents[EB/OL]. 2023-06. https://lilianweng.github.io/posts/2023-06-23-agent/.
[46] OSNIOsni. Introducing GPT-4o:the fastest and most powerful AI yet[EB/OL]. 2024-05. https://www.swiftask.ai/blog/gpt-4o?utm_source=chatgpt.com
[47] GLM T,ZENG A,XU B,et al. Chatglm:A family of large language models from glm-130b to glm-4 all tools[J]. Arxiv,Preprint Arxiv:2406.12793,2024.
[48] QWEN Team. Qwen2.5-LLM:Extending the boundary of LLMs[EB/OL]. 2024-09. https://qwenlm.github.io/blog/qwen2.5-llm/.
[49] SMITH W A,RANDALL R B. Rolling element bearing diagnostics using the Case Western Reserve University data:A benchmark study[J]. Mechanical Systems and Signal Processing,2015,64:100-131.
[50] LIU Dong,XIAO Zhihuai,HU Xiao,et al. Feature extraction of rotor fault based on EEMD and curve code[J]. Measurement,2019,135:712-724.
[51] MARINS M,RIBEIRO F,NETTO S,et al. Improved similarity-based modeling for the classification of rotating-machine failures[J]. Journal of the Franklin Institute,2018,355(4):1913-1930.
[52] TEAM Q. Qwq-32b:Embracing the power of reinforcement learning[EB/OL]. 2025-03. https://qwenlm.github.io/blog/qwq-32b.
[53] Artificial analysis. QwQ-32B-intelligence,performance & price analysis[EB/OL]. 2025-03. https://artificialanalysis.ai/models/qwq-32b?utm_source=chatgpt.com.
[54] OpenAI. O1 and new tools for developers[EB/OL]. 2024-12. https://openai.com/index/o1-and-new-tools-for-developers/?utm_source=chatgpt.com.

思维链范式智能运维文本基多模态智能体

Chain-of-thought Paradigm Text-based Multimodal Intelligent Agent for Equipment Operations and Maintenance

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	吴艳灵, 汤宝平, 邓蕾, 付豪. 低通筛选优化神经架构搜索的风电齿轮箱边缘侧故障诊断方法[J]. 机械工程学报, 2025, 61(7): 361-372.
[2]	雷亚国, 李熹伟, 李响, 李乃鹏, 杨彬. 面向机械设备通用健康管理的智能运维大模型[J]. 机械工程学报, 2025, 61(6): 1-13.
[3]	韩长坤, 卢威, 宋浏阳, 王华庆. 特征原子稀疏解析的多状态机械故障诊断方法[J]. 机械工程学报, 2025, 61(6): 33-42.
[4]	刘文朋, 杨绍普, 刘永强, 顾晓辉, 王久健. 轮轨激励下高速列车轴箱轴承故障诊断方法研究[J]. 机械工程学报, 2025, 61(6): 249-259.
[5]	李华, 王天杨, 张飞斌, 褚福磊. KurVMDPgram：一种用于旋转机械故障诊断的信号分解算法[J]. 机械工程学报, 2025, 61(4): 11-23.
[6]	乐珂, 李霁蒲, 陈祝云, 何国林, 邓书涵, 李巍华. 基于无源自监督域适应的旋转机械跨域新故障诊断方法[J]. 机械工程学报, 2025, 61(3): 67-76.
[7]	李嘉佳, 易茜, 冯毅雄, 朱鹏兴, 易树平. 人本智造单元中人-智能系统协同双智能体工作机制研究[J]. 机械工程学报, 2025, 61(3): 105-118.
[8]	黄思翰, 陈建鹏, 徐哲, 阎艳, 王国新. 基于大语言模型和机器视觉的智能制造系统人机自主协同作业方法[J]. 机械工程学报, 2025, 61(3): 130-141.
[9]	门志辉, 宫岛, 周劲松, 周凯. 基于格拉姆角场与并行卷积Transformer的列车传动系统故障诊断方法[J]. 机械工程学报, 2025, 61(24): 168-179.
[10]	胡济民, 罗洋, 于亦浩, 石娟娟, 黄伟国, 沈长青, 朱忠奎. 多源内部故障激励下深沟球轴承三维动态解析建模及振动分析[J]. 机械工程学报, 2025, 61(23): 96-107.
[11]	郑近德, 丁文海, 程健, 李姜宏, 桑炜. 基于故障特征显著性指数图的滚动轴承最优解调频带识别方法[J]. 机械工程学报, 2025, 61(23): 108-119.
[12]	董绍江, 夏宗佑, 邹松, 赵兴新, 雷开印. 数据缺失条件下基于对比嵌入特征生成的滚动轴承广义零样本故障诊断方法[J]. 机械工程学报, 2025, 61(23): 156-169.
[13]	唐小林, 甘炯鹏, 张振果. 横纵向耦合跟车场景下基于多智能体深度强化学习的混合动力车队协同能量管理研究[J]. 机械工程学报, 2025, 61(2): 236-246.
[14]	王玉静, 李祎然, 康守强, 刘连胜, 李玉庆, 孙宇林. 基于数字孪生的不同工况下谐波减速器故障诊断方法[J]. 机械工程学报, 2025, 61(18): 12-26.
[15]	邵海东, 肖一鸣, 钟翔, 韩特. 面向可信机械故障诊断的模型校准方法[J]. 机械工程学报, 2025, 61(17): 114-123.