Research on the Construction Method of a Multi-modal Patent Data-driven Professional Large Language Model for Product Innovation Design

doi:10.3901/JME.260154

Abstract

Abstract: To overcome the problems of "information cocoon" and "weak specialization", based on the existing large language model (LLM) technology, the limited hardware resources are used to conduct research on the construction method of multimodal patent data-driven professional large language model (DPLLM), and provide an effective tool that is both universal and professional for product innovation design. First, massive patent full-text data is derived and constructed as the high-quality dataset. Then, the original data is cleaned by MinHash and multi-threaded processing technology, and converted into JSON format for pre-training. Based on the base model, the low-rank adapter (LoRA), AdamW optimizer and cosine annealing (CA) method are used for secondary pre-training to support demand analysis and design solution generation. Second, the retrieval-augmented generation (RAG) technology is used to address the shortcoming of model that can’t generate patent numbers accurately. Considering the phenomenon of "hallucination", the GPT-4o model is used to extract the functions and patent numbers information from patents, then the vector knowledge base is constructed. Combined with the vector retrieval algorithm, the pre-trained model can accurately provide the real patent number to support patent source tracing. Third, the patent drawings are extracted by the multimodal model, then reconstructed as the image database by image generation model for patent around. Finally, the pre-trained model, RAG system and image database are integrated into the Web platform to provide a complete process tool for product design, and a wheelchair product is taken as the case for verifying the effectiveness of this method.

Key words: innovative design, multimodal patents, professional LLM, pre-training, image reconstruction

CLC Number:

TB472

LIN Wenguang, SHEN Fan, XIAO Renbin. Research on the Construction Method of a Multi-modal Patent Data-driven Professional Large Language Model for Product Innovation Design[J]. Journal of Mechanical Engineering, 2026, 62(15): 86-110.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

URL: http://www.cjmenet.com.cn/EN/10.3901/JME.260154

http://www.cjmenet.com.cn/EN/Y2026/V62/I15/86

References

[1] 车万翔，窦志成，冯岩松，等. 大模型时代的自然语言处理:挑战、机遇与发展[J]. 中国科学：信息科学，2023，53(9)：1645-1687. CHE Wanxiang，DOU Zhicheng，FENG Yansong，et al. Towards a comprehensive understanding of the impact of large language models on natural language processing：Challenges，opportunities and future directions[J]. Scientia Sinica (Informationis)，2023，53(9)：1645-1687.
[2] SUH N P. Axiomatic design theory for systems[J]. Research in Engineering Design，1998，10：189-209.
[3] ORLOFF M A. TRIZ[M]. Berlin：Springer press，2006.
[4] AKAO Y，MAZUR G H. The leading edge in QFD：Past，present and future[J]. International Journal of Quality & Reliability Management，2003，20(1)：20-35.
[5] 刘晓敏，罗林辉，陈亮，等. 基于CK理论的产品仿生创新设计研究[J]. 机械工程学报，2020，56(13)：207-217. LIU Xiaomin，LUO Linhui，CHEN Liang，et al. Research on bio-inspired design based on C-K theory for product innovation[J]. Journal of Mechanical Engineering，2020，56(13)：207-217.
[6] UMEDA Y，TOMIYAMA T，YOSHIKAWA H. FBS modeling：Modeling scheme of function for conceptual design[C]// Proceedings of the 9th International Workshop on Qualitative Reasoning，Amsterdam，The Netherlands，1995：271-278.
[7] 白仲航，邓暄，于菲，等. 面向系统资源挖掘的FBS与发明原理辅助裁剪创新方法[J].机械工程学报，2023，59(15)：255-266. BAI Zhonghang，DENG Xuan，YU Fei，et al. FBS and invention principles aided trimming innovative design method for system resource mining[J]. Journal of Mechanical Engineering，2023，59(15)：255-266.
[8] 褚学宁，陈汉斯，马红占. 性能数据驱动的机械产品关键设计参数识别方法[J]. 机械工程学报，2021，57(3)：185-196. CHU Xuening，CHEN Hansi，MA Hongzhan. Identification of critical design parameter for mechanical products based on performance data[J]. Journal of Mechanical Engineering，2021，57(3)：185-196.
[9] 侯亮，林浩菁，王少杰，等. 基于运行数据驱动反向设计的复杂装备个性化定制[J]. 机械工程学报，2021，57(8)：65-80. HOU Liang，LIN Haojing，WANG Shaojie，et al. Mass personalization for complex equipment based on operating data-driven inverse design[J]. Journal of Mechanical Engineering，2021，57(8)：65-80.
[10] LIN W，WANG Y，XIAO R. A hypergraph model for product function recombination based on online reviews[J]. Journal of Engineering Design，2023，34(9)：746-777.
[11] 胡炳涛，冯毅雄，刘继红，等. 面向“互联网+”定制产品的智能适应性设计研究[J]. 机械工程学报，2023，59(12)：109-125. HU Bingtao，FENG Yixiong，LIU Jihong，et al. Research on intelligent adaptive design for “Internet+” customized products[J]. Journal of Mechanical Engineering，2023，59(12)：109-125.
[12] QU J，ZHANG Y，TANG W，et al. Developing a virtual reality healthcare product based on data-driven concepts：A case study[J]. Advanced Engineering Informatics，2023，57：102118.
[13] 罗仕鉴，郭和睿，沈诚仪，等. 智能产品用户体验设计评价研究现状与进展[J]. 计算机集成制造系统，2024，30(6)：1919-1935. LUO Shijian，GUO Herui，SHEN Chengyi，et al. Status and progress of research on user experience design evaluation for intelligent products[J]. Computer Integrated Manufacturing System，2024，30(6)：1919-1935.
[14] SONG B，LUO J，WOOD K. Data-driven platform design：Patent data and function network analysis[J]. Journal of Mechanical Design，2019，141(2)：021101.
[15] 林文广，刘晓东，肖人彬. 文本挖掘与专利数据协同驱动的产品功能扩展研究[J]. 机械工程学报，2024，60(13)：56-70. LIN Wenguang，LIU Xiaodong，XIAO Renbin. Research on product function expansion driven by text mining and patent data collaboration[J]. Journal of Mechanical Engineering，2024，60(13)：56-70.
[16] LIU W，ZHOU W. Research on solving path of negative effect of “information cocoon room” in emergency[J]. Discrete Dynamics in Nature and Society，2022(1)：1326579.
[17] 赵欣，任杉，张耿，等. 运维数据与多模态知识驱动的复杂产品改进设计新模式研究[J]. 机械工程学报，2025，61(3)：91-104. ZHAO Xin，REN Shan，ZHANG Geng，et al. New pattern of operation & maintenance data and multi-modal knowledge driven improvement design for complex products[J]. Journal of Mechanical Engineering，2025，61(3)：91-104.
[18] 柴春雷，张熙元，丁诗莹，等. 生成式人工智能驱动产品智能设计的演进路径与前景展望[J]. 家具与室内装饰，2024，31(4)：9-18. CHAI Chunlei，ZHANG Xiyuan，DING shiying，et al. The evolution path and future development trends of generative AI-Driven product intelligent design[J]. Furniture & Interior Design，2024，31(4)：9-18.
[19] MORBIDONI C. Poster：LLMs for online customer reviews analysis：Oracles or tools? Experiments with GPT 3.5[C]//Proceedings of the 15th Biannual Conference of the Italian SIGCHI Chapter，Torino，Italy，2023：1-4.
[20] GOEL T，SHAER O，DELCOURT C，et al. Preparing future designers for human-ai collaboration in persona creation[C]//Proceedings of the 2nd Annual Meeting of the Symposium on Human-Computer Interaction for Work，New York，USA，2023：1-14.
[21] ZHU Q，CHONG L，YANG M，et al. Reading users’ minds from what they say：An investigation into llm-based empathic mental inference[C]//International Design Engineering Technical Conferences and Computers and Information in Engineering Conference，Washington，DC，USA，2024，88407：V006T06A018.
[22] LOYNAZ A，STANISH C，TREEMAN-REINKE A，et al. Can an LLM learn an unseen engineering language? An empirical study of prompt engineering on chatGPT and function modeling[C]//International Design Engineering Technical Conferences and Computers and Information in Engineering Conference，ASME，2024，88353：V02BT02A045.
[23] BARAMBONES J，MORAL C，DE ANTONIO A，et al. ChatGPT for learning HCI techniques：A case study on interviews for personas[J]. IEEE Transactions on Learning Technologies，2024，17：1460-1475.
[24] CHEN L，ZUO H，CAI Z，et al. Toward controllable generative design：A conceptual design generation approach leveraging the function-behavior-structure ontology and large language models[J]. Journal of Mechanical Design，2024，146(12)：121401.
[25] 于晗，陈治源，熊熙瑞，等. 基于检索增强大语言模型的MBSE智能设计方法[J]. 图学学报，2024，45(6)：1188-1199. YU Han，CHEN Zhiyuan，XIONG Xirui，et al. Intelligent MBSE design approach based on retrieval augmented large language model[J]. Journal of Graphics，2024，45(6)：1188-1199.
[26] LI S，ZHOU X，LIU Y，et al. Agile conceptual design and validation based on multi-source product data and large language models：A review，framework，and outlook[J]. Journal of Engineering Design，2025，36(4)：473-503.
[27] 陈树生，贾苜梁，林家豪，等. 生成式模型赋能飞行器技术应用研究进展与展望[J]. 航空学报，2025，46(10)：46-99. CHEN Shusheng，JIA Muliang，LIN Jiahao，et al. Empowering aircraft technology applications with generative models：Research progress and prospects[J]. Acta Aeronautica et Astronautica Sinica，2025，46(10)：46-99.
[28] BUCHMANN R，EDER J，FILL H G，et al. Large language models：Expectations for semantics-driven systems engineering[J]. Data & Knowledge Engineering，2024，152：102324.
[29] ZHANG A，JIA W，ZOU Q，et al. Diffusion-CAD：Controllable diffusion model for generating computer-aided design models[J]. IEEE Transactions on Visualization and Computer Graphics，2025，14：1-12.
[30] XU S，WEI Y，ZHENG P，et al. LLM enabled generative collaborative design in a mixed reality environment[J]. Journal of Manufacturing Systems，2024，74：703-715.
[31] KEELER M，FUGE M，PENG A，et al. A picture or a thousand words：Design description crafting to replicate human similarity judgments in large language models[C]//International Design Engineering Technical Conferences and Computers and Information in Engineering Conference，Washington，DC，United States. ASME，2024：DETC2024-143634.
[32] CHU J M，LO H C，HSIANG J，et al. From PARIS to LE-PARIS：Toward patent response automation with recommender systems and collaborative large language models[J]. Artificial Intelligence and Law，2024：1-27.
[33] 黄思翰，陈建鹏，徐哲，等. 基于大语言模型和机器视觉的智能制造系统人机自主协同作业方法[J]. 机械工程学报，2025，61(3)：130-141. HUANG Sihan，CHEN Jianpeng，XU Zhe，et al. Human-robot autonomous collaboration method of smart manufacturing systems based on large language model and machine vision[J]. Journal of Mechanical Engineering，2025，61(3)：130-141.
[34] 马杰，孙望淳，王平辉，等. 基于相关性提示的知识图谱问答[J]. 软件学报，2025，36(9)：4056-4071. MA Jie，SUN Wangchun，WANG Pinghui，et al. Knowledge graph question answering based on relevance prompts. Journal of Software，2025，36(9)：4056-4071.
[35] LEE C，KANG B，SHIN J. Novelty-focused patent mapping for technology opportunity analysis[J]. Technological Forecasting and Social Change，2015，90：355-365.
[36] ZHANG W，JI J，ZHU J，et al. BitHash：An efficient bitwise locality sensitive hashing method with applications[J]. Knowledge-Based Systems，2016，97：40-47.
[37] HUI B，YANG J，CUI Z，et al. Qwen2. 5-coder technical report[J]. arXiv preprint arXiv：2409.12186，2024.
[38] ZHANG Y，WANG J，YU L C，et al. Personalized LoRA for human-centered text understanding[C]//Proceedings of the AAAI Conference on Artificial Intelligence，Vancouver，Canada，2024，38(17)：19588-19596.
[39] XIN C，LU Y，LIN H，et al. Beyond full fine-tuning：Harnessing the power of LoRA for multi-task instruction tuning[C]//Proceedings of the 2024 Joint International Conference on Computational Linguistics，Language Resources and Evaluation，Torino，Italia，2024：2307-2317.
[40] LOSHCHILOV I，HUTTER F. Decoupled weight decay regularization[J]. arXiv preprint arXiv：1711.05101，2017.
[41] 杨观赐，杨静，李少波，等. 基于Dopout与ADAM优化器的改进CNN算法[J]. 华中科技大学学报，2018，46(7)：122-127. YANG Guanci，YANG Jing，LI Shaobo，et al. Modified CNN algorithm based on Dropout and ADAM optimizer[J]. Journal of Huazhong University of Science and Technology，2018，46(7):122-127.
[42] ZHOU P，XIE X，LIN Z，et al. Towards understanding convergence and generalization of AdamW[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024，46(9)：6486-6493.
[43] MISHRA S，YAMASAKI T，IMAIZUMI H. Improving image classifiers for small datasets by learning rate adaptations[C]//201916th International Conference on Machine Vision Applications (MVA). IEEE，2019：1-6.
[44] LOSHCHILOV I，HUTTER F. Sgdr：Stochastic gradient descent with warm restarts[J]. arXiv preprint arXiv:1608.03983，2016.
[45] VASWANI A，SHAZEER N，PARMAR N，et al. Attention is all you need[J]. Advances in neural information processing systems，Tokyo，Japan，IEEE，2017：1-6.
[46] WANG X，WANG Z，GAO X，et al. Searching for best practices in retrieval-augmented generation[C]// Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Florida，USA. 2024：17716-17736.
[47] LEWIS P，PEREZ E，PIKTUS A，et al. Retrieval- augmented generation for knowledge-intensive nlp tasks[J]. Advances in neural information processing systems，2020，33：9459-9474.
[48] JEONG C. Generative AI service implementation using LLM application architecture：Based on RAG model and LangChain framework[J]. Journal of Intelligence and Information Systems，2023，29(4)：129-164.
[49] YANG Z，LI L，LIN K，et al. The dawn of LMMS：Preliminary explorations with GPT-4v (ision)[J]. arXiv preprint arXiv:2309.17421，2023，9(1)：1-166.
[50] YANG L，ZHANG Z，SONG Y，et al. Diffusion models：A comprehensive survey of methods and applications[J]. ACM Computing Surveys，2023，56(4)：1-39.
[51] SAHARIA C，CHAN W，CHANG H，et al. Palette：Image-to-image diffusion models[C]//ACM SIGGRAPH 2022 conference proceedings. Vancouver，Canada，2022：1-10.
[52] 刘泽润，尹宇飞，薛文灏，等. 基于扩散模型的条件引导图像生成综述[J]. 浙江大学学报，2023，50(6)：651-667. LIU Zerun，YIN Yufei，XUE Wenhao，et al. A review of conditional image generation based on diffusion models[J]. Journal of Zhejiang University，2023，50(6)：651-667.
[53] 肖人彬. 大规模个性化定制：大规模个性化的新发展[J]. 计算机集成制造系统，2023，29(12)：4215-4226. XIAO Renbin. Massive personalized customization：New development of mass personalization[J]. Computer Integrated Manufacturing Systems，2023，29(12)：4215-4226.
[54] 肖人彬，林文广. 数据驱动的产品创新设计研究[J]. 机械设计，2019，36(12)：1-9. XIAO Renbin，LIN Wenguang. Research on data-driven product innovation design[J]. Journal of Mechanical Design，2019，36(12)：1-9.