Comparative Studies of Surrogate Models Based on Multiple Evaluation Criteria

doi:10.3901/JME.2022.16.403

Abstract

Abstract: The surrogate model refers to the use of limited sample information to establish the mathematical relationship between structural input and output, and it is increasingly used in the structural and multidisciplinary design optimization of complex equipment. With the purpose of obtaining performance effect of different approximate models in place of simulation analysis or physical test, systematic comparative analysis of approximate model for 38 test functions is carried out. Research the prediction accuracy of each model is calculated with the condition of different sample sizes, different nonlinearity degrees and different dimensions, and the computational time of different test functions on the model is also analyzed. Finally, two engineering examples are used to compare the predictive performance of the different approximate models. The results show that traditional approximate model (Polynomial Regression Surface, Radial Basis Function, Support Vector Regression, Kriging) maintain better prediction performance in engineering analysis without parameter optimization of the model. The accuracy of support vector regression (SVR) is better than other algorithm under most test functions. Radial basis function (RBF) has the best robustness in high non-linearity problems while random forest (RF) shows better robustness in low non-linearity. Thus, for different regression problems using suitable models can further improve the prediction accuracy and efficiency for different situations, which has guiding significance for studying the application of regression algorithm in the field of optimization design.

Key words: surrogate model, model comparison, degree of non-linearity, multiple evaluation criteria, optimization design

CLC Number:

TH122

HE Xiwang, YANG Liangliang, RAN Renjie, ZHU Fawen, SONG Xueguan. Comparative Studies of Surrogate Models Based on Multiple Evaluation Criteria[J]. Journal of Mechanical Engineering, 2022, 58(16): 403-419.

References

[1] ZHOU Q, WANG Y, CHOI S K, et al. A robust optimization approach based on multi-fidelity metamodel[J]. Structural and Multidisciplinary Optimization, 2018, 57(2): 775-797.
[2] HAN Z H, CHEN J, ZHANG K S, et al. Aerodynamic shape optimization of natural-laminar-flow wing using surrogate-based approach[J]. AIAA Journal, 2018, 56(7): 2579-2593.
[3] ZHOU Qi, YANG Yang, SONG Xueguan, et al. Survey of multi-fidelity surrogate models and their applications in the design and optimization of engineering equipment[J]. Journal of Mechanical Engineering, 2020, 56(24): 219-245. 周奇, 杨扬, 宋学官, 等. 变可信度近似模型及其在复杂装备优化设计中的应用研究进展[J]. 机械工程学报, 2020, 56(24): 219-245.
[4] LONG Teng, LIU Jian, WANG G G, et al. Discuss on approximate optimization strategies using design of computer experiments and metamodels for flight vehicle design[J]. Journal of Mechanical Engineering, 2016, 52(14): 79-105. 龙腾, 刘建, WANG G G, 等. 基于计算试验设计与代理模型的飞行器近似优化策略探讨[J]. 机械工程学报, 2016, 52(14): 79-105.
[5] QUEIPO N V, HAFTKA R T, SHYY W, et al. Surrogate-based analysis and optimization[J]. Progress in Aerospace Sciences, 2005, 41(1): 1-28.
[6] ZHANG X, LIU W, ZHANG Y, et al. Experimental investigation and optimization design of multi-support pipeline system[J]. Chinese Journal of Mechanical Engineering, 2021, 34(1): 1-15.
[7] LÜ Xiaojiang, GU Xianguang, WANG Guorong, et al. Reliability optimization on pedestrian flexible legform impact based on ensemble of metamodel[J]. Journal of Mechanical Engineering, 2016, 52(10): 142-149. 吕晓江, 谷先广, 王国荣, 等. 基于组合近似模型的可靠性优化方法在行人柔性腿型碰撞中应用研究[J]. 机械工程学报, 2016, 52(10): 142-149.
[8] WANG G G, SHAN S. Review of metamodeling techniques in support of engineering design optimization[J]. Journal of Mechanical Design, 2007, 129(4): 370-380.
[9] HAN Zhonghua. Kriging surrogate model and its application to design optimization: A review of recent progress[J]. Acta Aeronautica ET Astronautica Sinica, 2016, 37(11): 3197-3225. 韩忠华. Kriging模型及代理优化算法研究进展[J]. 航空学报, 2016, 37(11): 3197-3225.
[10] HAN Zhonghua, XU Chenzhou, QIAO Jianling, et al. Recent progress of efficient global aerodynamic shape optimization using surrogate-based approach[J]. Acta Aeronautica ET Astronautica Sinica, 2020, 41(5): 25-26. 韩忠华, 许晨舟, 乔建领, 等. 基于代理模型的高效全局气动优化设计方法研究进展[J]. 航空学报, 2020, 41(5): 25-65.
[11] LONG Teng, LIU Jian, MENG Lingtao, et al. Development of multidisciplinary design optimization technology and its application in aerospace industry[J]. Aeronautical Manufacturing Technology, 2016(3): 24-33. 龙腾, 刘建, 孟令涛, 等. 多学科设计优化技术发展及在航空航天领域的应用[J]. 航空制造技术, 2016(3): 24-33.
[12] LONG Teng, LI Xueliang, HUANG Bo, et al. Aerodynamic and stealthy performance optimization of airfoil based on adaptive surrogate model[J]. Journal of Mechanical Engineering, 2016, 52(22): 101-111. 龙腾, 李学亮, 黄波, 等. 基于自适应代理模型的翼型气动隐身多目标优化[J]. 机械工程学报, 2016, 52(22): 101-111.
[13] CHEN W, ZHOU X, WANG H, et al. Multi-objective optimal approach for injection molding based on surrogate model and particle swarm optimization algorithm[J]. Journal of Shanghai Jiaotong University, 2010, 15(1): 88-93.
[14] JIANG P, ZHOU Q, SHAO X. Surrogate model-based engineering design and optimization[M]. Singapore: Springer, 2020.
[15] RUAN X, JIANG P, ZHOU Q, et al. Variable-fidelity probability of improvement method for efficient global optimization of expensive black-box problems[J]. Structural and Multidisciplinary Optimization, 2020, 62(6): 3021-3052.
[16] CHEN L, QIU H, GAO L, et al. Optimization of expensive black-box problems via Gradient-enhanced Kriging[J]. Computer Methods in Applied Mechanics and Engineering, 2020, 362: 112861.
[17] CAI X, QIU H, GAO L, et al. An efficient surrogate-assisted particle swarm optimization algorithm for high-dimensional expensive problems[J]. Knowledge-Based Systems, 2019, 184: 104901.
[18] HAO Peng, WANG Bo, ZOU Weiren, et al. Optimum design of lightening holes for skin-stringer structures based on RBF model[J]. Journal of Solid Rocket Technology, 2015, 38(5): 717-721. 郝鹏, 王博, 邹威任, 等. 基于RBF模型的蒙皮桁条结构减轻孔优化[J]. 固体火箭技术, 2015, 38(5): 717-721.
[19] MENG Z, ZHANG Z, ZHOU H, et al. Robust design optimization of imperfect stiffened shells using an active learning method and a hybrid surrogate model[J]. Engineering Optimization, 2020, 52(12): 2044-2061.
[20] GOEL T, HAFTKA R T, SHYY W, et al. Ensemble of surrogates[J]. Structural and Multidisciplinary Optimization, 2007, 33(3): 199-216.
[21] JIN R, CHEN W, SIMPSON T W. Comparative studies of metamodelling techniques under multiple modelling criteria[J]. Structural and Multidisciplinary Optimization, 2001, 23(1): 1-13.
[22] SONG X, SUN G, LI G, et al. Crashworthiness optimization of foam-filled tapered thin-walled structure using multiple surrogate models[J]. Structural and Multidisciplinary Optimization, 2013, 47(2): 221-231.
[23] YONDO R, ANDRéS E, VALERO E. A review on design of experiments and surrogate models in aircraft real-time and many-query aerodynamic analyses[J]. Progress in Aerospace Sciences, 2018, 96: 23-61.
[24] VOYANT C, NOTTON G, KALOGIROU S, et al. Machine learning methods for solar radiation forecasting: A review[J]. Renewable Energy, 2017, 105: 569-582.
[25] STULP F, SIGAUD O. Many regression algorithms, one unified model: A review[J]. Neural Networks, 2015, 69: 60-79.
[26] ZHAO Shuai, HUANG Yixiang, WANG Haoren, et al. Random forest and principle components analysis based on health assessment methodology for tool wear[J]. Journal of Mechanical Engineering, 2017, 53(21): 181-189. 赵帅, 黄亦翔, 王浩任, 等. 基于随机森林与主成分分析的刀具磨损评估[J]. 机械工程学报, 2017, 53(21): 181-189.
[27] DING H, YANG L, CHENG Z, et al. A remaining useful life prediction method for bearing based on deep neural networks[J]. Measurement, 2021, 172: 108878.
[28] COLKESEN I, SAHIN E K, KAVZOGLU T. Susceptibility mapping of shallow landslides using kernel-based Gaussian process, support vector machines and logistic regression[J]. Journal of African Earth Sciences, 2016, 118: 53-64.
[29] XIE Y, ZHU C, ZHOU W, et al. Evaluation of machine learning methods for formation lithology identification: A comparison of tuning processes and model performances[J]. Journal of Petroleum Science and Engineering, 2018, 160: 182-193.
[30] SLOTNICK J, KHODADOUST A, ALONSO J, et al. CFD vision 2030 study: A path to revolutionary computational aerosciences[R]. NASA/CR-2014-218178, NF1676L-18332, 2014.
[31] KHURI A I, MUKHOPADHYAY S. Response surface methodology[J]. Wiley Interdisciplinary Reviews: Computational Statistics, 2010, 2(2): 128-149.
[32] MAJDISOVA Z, SKALA V. Radial basis function approximations: comparison and applications[J]. Applied Mathematical Modelling, 2017, 51: 728-743.
[33] PARK J, SANDBERG I W. Approximation and radial-basis-function networks[J]. Neural Computation, 1993, 5(2): 305-316.
[34] FORRESTER A, SOBESTER A, KEANE A. Engineering design via surrogate modelling: a practical guide[M]. United Kingdom: John Wiley & Sons, 2008.
[35] SUN G, SONG X, BAEK S, et al. Robust optimization of foam-filled thin-walled structure based on sequential Kriging metamodel[J]. Structural and Multidisciplinary Optimization, 2014, 49(6): 897-913.
[36] LIU F, HAN Z H, ZHANG Y, et al. Surrogate-based aerodynamic shape optimization of hypersonic flows considering transonic performance[J]. Aerospace Science and Technology, 2019, 93: 105345.
[37] BOUHLEL M A, BARTOLI N, OTSMANE A, et al. Improving Kriging surrogates of high-dimensional design models by partial least squares dimension reduction[J]. Structural and Multidisciplinary Optimization, 2016, 53(5): 935-952.
[38] ZENG Feng, ZHOU Jinzhu. Optimization Strategy for dynamic metamodel integrating minimize lower confidence bound and trust region[J]. Journal of Mechanical Engineering, 2017, 53(13): 170-178. 曾锋, 周金柱. 集成最小化置信下限和信赖域的动态代理模型优化策略[J]. 机械工程学报, 2017, 53(13): 170-178.
[39] BACHOC F. Cross validation and maximum likelihood estimations of hyper-parameters of Gaussian processes with model misspecification[J]. Computational Statistics & Data Analysis, 2013, 66: 55-69.
[40] YU Mengge, PAN Zhenkuan, JIANG Rongchao, et al. Multi-objective optimization design of the high-speed train head based on the approximate model[J]. Journal of Mechanical Engineering, 2019, 55(24): 178-186. 于梦阁, 潘振宽, 蒋荣超, 等. 基于近似模型的高速列车头型多目标优化设计[J]. 机械工程学报, 2019, 55(24): 178-186.
[41] XIANG H, LI Y, LIAO H, et al. An adaptive surrogate model based on support vector regression and its application to the optimization of railway wind barriers[J]. Structural and Multidisciplinary Optimization, 2017, 55(2): 701-713.
[42] ZHOU Q, SHAO X, JIANG P, et al. An adaptive global variable fidelity metamodeling strategy using a support vector regression based scaling function[J]. Simulation Modelling Practice and Theory, 2015, 59: 18-35.
[43] SHU Xing, LIU Yonggang, SHEN Jiangwei, et al. Capacity prediction for lithium-ion batteries based on improved least squares support vector machine and Box-Cox transformation[J]. Journal of Mechanical Engineering, 2021, 57(14): 118-128. 舒星, 刘永刚, 申江卫, 等. 基于改进最小二乘支持向量机与Box-Cox变换的锂离子电池容量预测[J]. 机械工程学报, 2021, 57(14): 118-128.
[44] DE BRABANTER K, KARSMAKERS P, OJEDA F, et al. LS-SVMlab toolbox user's guide: version 1.7[M]. Belgium: Katholieke Universiteit Leuven, 2010.
[45] ZENG Qinghong, LU Detang. Curve and surface fitting based on moving least square method[J]. Journal of Engineering Graphics, 2004, 25(1): 84-89. 曾清红, 卢德唐. 基于移动最小二乘法的曲线曲面拟合[J]. 工程图学学报, 2004, 25(1): 84-89.
[46] ISHWARAN H, LU M. Standard errors and confidence intervals for variable importance in random forest regression, classification, and survival[J]. Statistics in Medicine, 2019, 38(4): 558-582.
[47] ZHOU Haijun, WANG Chaowei, ZHOU Guojing, et al. Fault diagnosis of centrifugal pump rolling bearing based on random forest[J]. China Ship Research, 2020, 15(3): 129-135. 周海军, 王超伟, 周国敬, 等. 基于随机森林的离心泵滚动轴承故障诊断[J]. 中国舰船研究, 2020, 15(3): 129-135.
[48] LIN J, QI C, WAN H, et al. Prediction of cross-tension strength of self-piercing riveted joints using finite element simulation and XGBoost algorithm[J]. Chinese Journal of Mechanical Engineering, 2021, 34(1): 1-11.
[49] JUNG C, SCHINDLER D. Modelling monthly near-surface maximum daily gust speed distributions in Southwest Germany[J]. International Journal of Climatology, 2016, 36(12): 4058-4070.
[50] VAN HEIJST D, POTHARST R, VAN WEZEL M. A support system for predicting eBay end prices[J]. Decision Support Systems, 2008, 44(4): 970-982.
[51] REN S, HE K, GIRSHICK R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-1149.
[52] AHMAD J, MUHAMMAD K, LEE M Y, et al. Endoscopic image classification and retrieval using clustered convolutional features[J]. Journal of Medical Systems, 2017, 41(12): 196.
[53] LI L, XIAO L, WANG N, et al. Text classification method based on convolutional neural network[C]//2017 3rd IEEE International Conference on Computer and Communications, December 13-16, 2017, Chengdu, China. New York: IEEE, 2017: 1985-1989.
[54] HU Niaoqing, CHEN Huipeng, CHENG Zhe, et al. Fault diagnosis for planetary gearbox based on EMD and deep convolutional neural networks[J]. Journal of Mechanical Engineering, 2019, 55(7): 9-18. 胡茑庆, 陈徽鹏, 程哲, 等. 基于经验模态分解和深度卷积神经网络的行星齿轮箱故障诊断方法[J]. 机械工程学报, 2019, 55(7): 9-18.
[55] WANG Baoyi, ZHAO Shuo, ZHANG Shaomin. Distributed power load forecasting algorithm based on cloud computing and extreme learning machine[J]. Power System Technology, 2014(2): 526-531. 王保义, 赵硕, 张少敏. 基于云计算和极限学习机的分布式电力负荷预测模型[J]. 电网技术, 2014(2): 526-531.
[56] HUANG G B, ZHU Q Y, SIEW C K. Extreme learning machine: Theory and applications[J]. Neurocomputing, 2006, 70(1-3): 489-501.
[57] SONG X, LÜ L, LI J, et al. An advanced and robust ensemble surrogate model: extended adaptive hybrid functions[J]. Journal of Mechanical Design, 2018, 140(4): 041402.
[58] TALGORN B, LE DIGABEL S, KOKKOLARAS M. Statistical surrogate formulations for simulation-based design optimization[J]. Journal of Mechanical Design, 2015, 137(2): 021405.
[59] MALLICK M, TIAN X. Analysis of Polynomial Nonlinearity Based on Measures of Nonlinearity Algorithms[J]. Sensors, 2020, 20(12): 3426.
[60] LIU Y, LI X R. Measure of nonlinearity for estimation[J]. IEEE Transactions on Signal Processing, 2015, 63(9): 2377-2388.
[61] MCKAY M D, BECKMAN R J, CONOVER W J. A comparison of three methods for selecting values of input variables in the analysis of output from a computer code[J]. Technometrics, 2000, 42(1): 55-61.
[62] GARBO A, GERMAN B. Adaptive sampling with adaptive surrogate model selection for computer experiment applications[C]//18th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, June 5-9, 2017, Denver, Colorado. Reston: AIAA. 2017: 4430.
[63] FORRESTER A I J, KEANE A J. Recent advances in surrogate-based optimization[J]. Progress in Aerospace Sciences, 2009, 45(1-3): 50-79.
[64] LAI X, WANG S, GUO Z, et al. Designing a Shape–Performance Integrated Digital Twin Based on Multiple Models and Dynamic Data: A Boom Crane Example[J]. Journal of Mechanical Design, 2021, 143(7): 071703.