多模态知识图谱与大模型混合驱动的智能概念设计框架及应用

doi:10.3901/JME.260258

摘要/Abstract

摘要： 产品设计作为将创新理念转化为具备功能性、可行性及市场适应性的关键过程，处于产品全周期的核心地位。然而，传统以人工经验为主导的产品设计范式已难以应对当前知识爆炸与需求快速变更的挑战，亟需向高效、智能且知识驱动的全流程设计体系转型。多模态知识图谱与大模型作为新一代人工智能核心技术，在非结构化知识建模、跨模态语义解析等方面具有显著优势。为此，提出多模态知识图谱与大模型混合驱动的智能概念设计框架，旨在构建从语义建模、融合到设计方案生成的全流程闭环支持。具体而言，该框架首先采用基于LLM的三级协同建模机制，对设计文本进行深度语义解析与建模；引入视觉语言模型与多视角策略，实现对公式与图形等模态数据的语义建模，构建统一多模态语义空间。其次，在知识融合阶段，通过基于LLM的跨模态实体对齐方法，实现多模态设计知识图谱构建。最后，在设计方案生成阶段，结合知识图谱增强的大语言模型，生成概念设计方案，并利用设计语义引导进行图形快速原型开发，形成全流程设计闭环。以某汽车车身结构设计为应用场景，构建多模态知识驱动的智能概念设计方案生成及应用系统。实验表明，该框架有效提升了设计效率，为产品设计的智能化发展提供了支撑。

关键词: 多模态, 知识图谱, 大模型, 智能设计, 全流程

Abstract: Product design, as a critical process that transforms innovative concepts into functional, feasible, and market-adaptive solutions, occupies a central position in the product lifecycle. However, the traditional design paradigm—primarily reliant on human expertise—has become inadequate in addressing the challenges posed by knowledge explosion and rapidly evolving demands. This context calls for a transition toward a more efficient, intelligent, and knowledge-driven full-process design system. Multimodal knowledge graphs and large language models (LLMs), as core technologies of next-generation artificial intelligence, offer notable advantages in unstructured knowledge modeling and cross-modal semantic understanding. In response to these capabilities, a hybrid-driven intelligent conceptual design framework integrating multimodal knowledge graphs and LLMs is proposed to support a full-process closed-loop workflow encompassing semantic modeling, knowledge integration, and design solution generation. Specifically, the framework begins by employing an LLM-based three-tier collaborative modeling mechanism to conduct deep semantic parsing and modeling of design-related textual data. It further incorporates visual-language models and multi-view strategies to semantically model additional modalities such as formulas and diagrams, thereby constructing a unified multimodal semantic space. In the knowledge integration phase, an LLM-based cross-modal entity alignment method is utilized to construct a comprehensive multimodal design knowledge graph. Finally, at the design solution generation stage, conceptual design solutions are generated by integrating knowledge graph-enhanced large language models, and the generated design semantics are utilized to guide rapid prototyping of graphical models, forming a full-process design closed-loop. Taking the structural design of a car body as a representative application scenario, an intelligent conceptual design solution generation and application system driven by multimodal knowledge is developed. Experimental results demonstrate that the proposed framework effectively improves design efficiency, providing support for advancing product design toward greater intelligence.

Key words: multimodal, knowledge graph, large language model, intelligent design, full-process

中图分类号:

TP319
TH17

杨波, 杜卡泽, 王时龙, 王四宝, 刘惠, 郏维强. 多模态知识图谱与大模型混合驱动的智能概念设计框架及应用[J]. 机械工程学报, 2026, 62(15): 65-85.

YANG Bo, DU Kaze, WANG Shilong, WANG Sibao, LIU Hui, JIA Weiqiang. A Hybrid-driven Framework for Intelligent Conceptual Design with Multimodal Knowledge Graphs and Large Models: Research and Applications[J]. Journal of Mechanical Engineering, 2026, 62(15): 65-85.

导出引用管理器 EndNote|Reference Manager|ProCite|BibTeX|RefWorks

链接本文: http://www.cjmenet.com.cn/CN/10.3901/JME.260258

http://www.cjmenet.com.cn/CN/Y2026/V62/I15/65

参考文献

[1] 余隋怀，王鹏超，王磊，等. 基于知识工程的产品设计研究综述[J]. 机械工程学报，2024，60(13)：216-234. YU Suihuai，WANG Pengchao，WANG Lei，et al. Research review of product design based on knowledge engineering[J]. Journal of Mechanical Engineering，2024，60(13)：216-234.
[2] LI Zhinan，LIU Zhenyu，SA Guodong，et al. Knowledge-enhanced large language models for ideation to implementation：A new paradigm in product design[J]. Applied Soft Computing，2025：113147.
[3] 郭鑫，黄泽川，王杰，等. 人机混合交互的产品设计知识图谱构建及智能推理方法[J]. 机械工程学报，2025，61(3)：142-153. GUO Xin，HUANG Zechuan，WANG Jie，et al. Knowledge graph construction and intelligent reasoning methods for human-computer hybrid interaction in product design[J]. Journal of Mechanical Engineering，2025，61(3)：142-153.
[4] 赵欣，任杉，张耿，等. 运维数据与多模态知识驱动的复杂产品改进设计新模式研究[J]. 机械工程学报，2025，61(3)：91-104. ZHAO Xin，REN Shan，ZHANG Geng，et al. New pattern of operation & maintenance data and multi-modal knowledge driven improvement design for complex products[J]. Journal of Mechanical Engineering，2025，61(3)：91-104.
[5] HUANG Yuexin，YU Suihuai，CHU Jianjie，et al. Combining deep learning with knowledge graph for design knowledge acquisition in conceptual product design[J]. CMES-Computer Modeling in Engineering and Sciences，2024，138(1)：167-200.
[6] CHENG Yuan，LI Yupeng，ZHANG Na，et al. A knowledge graph-enabled multi-domain mapping approach supporting product rapid design：A case study of new energy vehicles[J]. Advanced Engineering Informatics，2024，62：102779.
[7] HOU Yongzhu，LIU Jihong，YUE Gaofeng. 3D interpreted model：A novel product definition model by integrating and fusing a 3D annotated model and design knowledge[J]. Applied Sciences，2021，11(16)：7192.
[8] BHARADWAJ A，STARLY B. Knowledge graph construction for product designs from large CAD model repositories[J]. Advanced Engineering Informatics，2022，53：101680.
[9] GUO Liang，YAN Fu，LU Yuqian，et al. An automatic machining process decision-making system based on knowledge graph[J]. International Journal of Computer Integrated Manufacturing，2021，34(12)：1348-1369.
[10] WANG Ru，SUN Yanshao，PENG Tao，et al. A knowledge graph-aided decision guidance method for product conceptual design[J]. Journal of Engineering Design，2025，36(2)：216-255.
[11] XU Shengyang，WEI Yao，ZHENG Pai，et al. LLM enabled generative collaborative design in a mixed reality environment[J]. Journal of Manufacturing Systems，2024，74：703-715.
[12] KHAN M，SINHA S，UDDIN T，et al. Text2cad：Generating sequential CAD designs from beginner-to- expert level text prompts[J]. Advances in Neural Information Processing Systems，2024，37：7552-7579.
[13] WANG Siyu，CHEN Cailian，LE Xinyi，et al. CAD-GPT：Synthesising CAD construction sequence with spatial reasoning-enhanced multimodal LLMs[C]// Association for the Advancement of Artificial Intelligence. Proceedings of the AAAI Conference on Artificial Intelligence，February 25- March 4，2025，Pennsylvania，Philadelphia. Palo Alto，California：AAAI Press，2025：715-735.
[14] 于泽源，赵武，张凯，等. 支持产品迭代设计的知识推送方法[J]. 机械工程学报，2022，58(18)：292-302. YU Zeyuan，ZHAO Wu，ZHANG Kai，et al. Knowledge push method to support iterative product design[J]. Journal of Mechanical Engineering，2022，58(18)：292-302.
[15] HONG Yida，LI Wenqiang，LI Chuanxiao，et al. Adaptive knowledge push method of product intelligent design based on feature transfer[J]. Advanced Engineering Informatics，2024，59：102260.
[16] HAN Zhulin，WANG Jian. Knowledge enhanced graph inference network based entity-relation extraction and knowledge graph construction for industrial domain[J]. Frontiers of Engineering Management，2024，11(1)：143-158.
[17] EDGE D，TRINH H，CHENG N，et al. From local to global：A graph rag approach to query-focused summarization[J]. arXiv，2024：2404.16130.
[18] 程锦，叶虎强，谭建荣，等. 三维CAD技术研究进展及其发展趋势综述[J]. 机械工程学报，2023，59(23)：158-185. CHENG Jin，YE Huqiang，TAN Jianrong，et al. Review of research progress and development trends of 3D CAD technology[J]. Journal of Mechanical Engineering，2023，59(23)：158-185.
[19] LI Zhinan，SA Guodong，LIU Zhenyu，et al. Top-down hierarchical construction and application of a domain knowledge graph based on multimodal design information[J]. Journal of Mechanical Design，2025，147(3)：031401.
[20] PENG Cheng，SHENG Yanyan，GUI Weihua，et al. A rolling bearing fault diagnosis method based on multimodal knowledge graph[J]. IEEE Transactions on Industrial Informatics，2024，20(11)：130047-13057.
[21] LU Hengyang，LIU Tianci，CONG Rui，et al. QAIE：LLM-based quantity augmentation and information enhancement for few-shot aspect-based sentiment analysis[J]. Information Processing and Management，2025，62(1)：103917.
[22] LIANG Xinxin，WANG Zuoxu，LI Mingrui，et al. A survey of LLM-augmented knowledge graph construction and application in complex product design[J]. Procedia CIRP，2024，128：870-875.
[23] QUINONEZ-BACA L，RAMIREZ-ALONSO G，GUZMAN-PANDO A，et al. Advances in meta-learning and zero-shot learning for multi-label classification：A review[J]. Digital Signal Processing，2025，163：105220.
[24] MAO Xin，WANG Wenting，WU Yuanbin，et al. Boosting the speed of entity alignment 10×：Dual attention matching network with normalized hard sample mining[C]// International World Wide Web Conference Committee. Proceedings of the web conference 2021，April 19-23，2021，Ljubljana Slovenia. New York：ACM，2021：821-832.
[25] TRAAG V，WALTMAN L，VANECK N. From Louvain to Leiden：Guaranteeing well-connected communities[J]. Scientific Reports，2019，9(1)：5233.
[26] LIU Yuhao，YANG Maolin，JIANG Pingyu. CGAN- driven intelligent generative design of vehicle exterior shape[J]. Expert Systems with Applications，2025，274：127066.
[27] DENG Haoxuan，KHAN S，ERKOYUNCU J. An investigation on utilizing large language model for industrial computer-aided design automation[J]. Procedia CIRP，2024，128：221-226.
[28] KONG Chuipin，LEI Jialin，ZHOU Xionghui. Research on the intelligent design system for automotive panel die based on geometry and knowledge driven[J]. The International Journal of Advanced Manufacturing Technology，2024，133(1)：765-790.
[29] WU J，BIAN J W，LI X，et al. Gaussctrl：Multi-view consistent text-driven 3D gaussian splatting editing[C]// European Computer Vision Association. The 18th European Conference on Computer Vision ECCV 2024，September 29-October 4，MiCo Milano. Cham，Switzerland：Springer Nature Switzerland，2024：55-71.
[30] HU Yuelin，ZOU Futai，HAN Jiajia，et al. LLM-TIKG：Threat intelligence knowledge graph construction utilizing large language model[J]. Computers and Security，2024，145：103999.
[31] SUN Lizhuang，ZHANG Peng，GAO Fang，et al. SF-GPT：A training-free method to enhance capabilities for knowledge graph construction in LLMs[J]. Neurocomputing，2025，613：128726.
[32] ZHANG Jian，QI Shihao，DONG Yuxuan，et al. GKG-LLM：A unified framework for generalized knowledge graph construction[J]. Information Fusion，2026，128：103956.
[33] ZHU Xiangru，LI Zhixu，WANG Xiaodan，et al. Multi-modal knowledge graph construction and application：A survey[J]. IEEE Transactions on Knowledge and Data Engineering，2022，36(2)：715-735.