Product Conceptual Sketch Generation Design Using Deep Learning

doi:10.3901/JME.2023.11.016

Abstract

Abstract: Concept sketching, as a higher order human visual cognitive activity, is an important tool to assist designers in recording, ideating, creating, and evaluating ideas and has a positive impact on the generation of innovative concepts. In order to simulate this higher order visual cognitive behaviors of designers and to achieve intelligent assistance in creative sketching, a deep learning-based design integrated framework for intelligent generation of product concept sketches is proposed, which includes two core modules: an end-to-end sketch design GAN (Sketch2Render-GAN) and a sketch-neural style transfer network (Sketch-NST). The first module implements sketch generation and rendering, while the second performs sketch style features transformation. The hand drill and bicycle helmet were used as design objects respectively, and experimental results show that the proposed approach framework can quickly obtain many innovative concept sketches and implement automatic sketch rendering and style transformation. The findings also show that the approach framework helps designers to break through design solidification at the visual perception level and increase design efficiency. Furthermore, a smart-sketch design generator (S-SDG_v0.1) was developed to facilitate human-machine design collaboration between designers and AI models, which effectively reduces the threshold of designers to apply intelligent algorithms to assist design.

Key words: visual cognition, deep learning, generative design, sketch design, GAN

CLC Number:

TG156

LI Xiong, SU Jianning, ZHANG Zhipeng. Product Conceptual Sketch Generation Design Using Deep Learning[J]. Journal of Mechanical Engineering, 2023, 59(11): 16-30.

References

[1] CHEN L，WANG P，DONG H，et al. An artificial intelligence based data-driven approach for design ideation[J]. Journal of Visual Communication and Image Representation，2019，61:10-22.
[2] MAO X，GALIL O，PARRISH Q，et al. Evidence of cognitive chunking in freehand sketching during design ideation[J]. Design Studies，2019，67:1-26.
[3] HA D，ECK D. A neural representation of sketch drawings[C]// 6th International Conference on Learning Representations (ICLR)，Vancouver，BC，Canada，April 30-May 3，2018.
[4] CAO N，YAN X，SHI Y，et al. AI-Sketcher:A deep generative model for producing high-quality sketches[J]. Proceedings of the AAAI Conference on Artificial Intelligence，2019，33(1):2564-2571.
[5] TANG Y C，HUANG J J，YAO M T，et al. A review of design intelligence:progress，problems，and challenges[J]. Frontiers of Information Technology & Electronic Engineering，2019，20(12):1595-1617.
[6] GOODFELLOW I，POUGET-ABADIE J，MIRZA M，et al. Generative adversarial nets[C]// Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge:MIT Press，2014:2672-2680.
[7] RADFORD A，METZ L，CHINTAL S. Unsupervised representation learning with deep convolutional generative adversarial networks[C]// 4th International Conference on Learning Representations (ICLR)，San Juan，Puerto Rico，May 2-4，2016.
[8] NGUYEN-PHUOC T，LI C，THEIS L，et al. HoloGAN:unsupervised learning of 3D representations from natural images[C]// 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCV)，Seoul，Korea，Oct. 27-Nov. 2，2019:7588-7597.
[9] KWON Y H，PARK M G. Predicting future frames using retrospective cycle GAN[C]// IEEE Computer Vision and Pattern Recognition (CVPR)，Long Beach，CA，June 16-20，2019:1811-1820.
[10] XU T，ZHANG P，HUANG Q，et al. AttnGA:Fine-grained text to image generation with attentional generative adversarial networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)，Salt Lake City，Utah，June 18-23，2018:1316-1324.
[11] CRESWELL A，WHITE T，DUMOULIN V，et al. Generative adversarial networks:an overview[J]. IEEE Signal Processing Magazine，2017，35(1):53-65. https://arxiv.org/pdf/1710.07035.pdf.
[12] LIU Z，GAO F，WANG Y. A Generative adversarial network for AI-aided chair design[C]// 2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR)，San Jose，California，March 28-30，2019:486-490.
[13] DAI Y，LI Y，LIU L J. New product design with automatic scheme generation[J]. Sensing and Imaging an International Journal，2019，20(1):1-16.
[14] MAO X，LI Q，XIE H，et al. Least squares generative adversarial networks[C]// 2017 IEEE International Conference on Computer Vision (ICCV)，Venice，Italy，October 22-29，2017:2813-2821.
[15] ARJOVSKY M，CHINTALA S，BOTTOU L. Wasserstein generative adversarial networks[C] //Proceedings of the 34th International Conference on Machine Learning (ICML). Sydney，Australia，August 6-11，2017:214-223.
[16] SAGE A，AGUSTSSON E，TIMOFTE R，et al. Logo synthesis and manipulation with clustered generative adversarial networks[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)，Salt Lake City，UT，USA，June 18-23，2018:5879-5888.
[17] JIN Y H，ZHANG J K，LI M J，et al. Towards the automatic anime characters creation with generative adversarial networks[J]. arXiv preprint arXiv:1708.05509，2017. https://arxiv.org/pdf/1708.05509.pdf.
[18] MIRZA M，OSINDERO S. Conditional generative adversarial nets[J]. arXiv preprint arXiv:1411.1784，2014. https://arxiv.org/pdf/1411.1784.pdf
[19] KIM T，CHA M，KIM H，et al. Learning to discover cross-domain relations with generative adversarial networks[C]// Proceedings of the 34th International Conference on Machine Learning (ICML)，Sydney NSW，Australia，August 6-11，2017:1857-1865.
[20] CHAI C，LIAO J，ZOU N，et al. A one-to-many conditional generative adversarial network framework for multiple image-to-image translations[J]. Multimedia Tools and Applications，2018，77(17):22339-22366.
[21] PAN Y，BURNAP A，HARTLEY J，et al. Deep design:product aesthetics for heterogeneous markets[C]// Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)，Halifax，August 14-17，2017:1961-1970.
[22] 刘跃中，斯托夫斯·卢迪，杨阳. 基于条件生成对抗网络的城市设计研究[J]. 建筑学报，2018(9):108-113. LIU Yuezhong，STOUFFS R，YANG Yang. Urban design process with conditional generative adversarial networks[J] Architectural Journal，2018(9):108-113.
[23] PORTILLA J，SIMONCELLI E P. A parametric texture model based on joint statistics of complex wavelet coefficients[J]. International Journal of Computer Vision，2000，40(1):49-70.
[24] HAYS J，ESSA I. Image and video based painterly animation[C]// Proceedings of the 3rd International Symposium on Non-photorealistic Animation and Rendering (NPAR)，Annecy France，June 7-9，2004，113-120.
[25] SHIH Y C，PARIS S，BARNES C，et al. Style transfer for headshot portraits[J]. ACM Transactions on Graphics，2014，33(4):1-14
[26] GATYS L A，ECKER A S，BETHGE M，et al. A neural algorithm of artistic style[J]. Journal of Vision，2016，16(12):1-16.
[27] GATYS L A，ECKER A S，BETHGE M，et al. Texture synthesis using convolutional neural networks[C]// Proceedings of the 28th International Conference on Neural Information Processing Systems，Montreal，Canada，December 7-12，2015:262-270.
[28] JOHNSON J，ALAHI A，FEI-FEI L. Perceptual losses for real-time style transfer and super-resolution[C]// 14th European Conference on Computer Vision (ECCV). Amsterdam，Netherlands，October 11-14，2016:694-711.
[29] FORBUS，KENNETH D. Vision and sketching[J]. Perception，2012，41(9):1031-1039.
[30] GULRAJANI I. AHMED F，ARJOVSKY M，et al. Improved training of Wasserstein GANs[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems，California，USA，December 4-9，2017:5769-5779.
[31] KODALI N，ABERNETHY J，HAYS J，et al. On convergence and stability of GANs[C]// 6th International Conference on Learning Representations (ICLR)，Vancouver，BC，Canada，April 30-May 3，2018.
[32] EREZ P. Sketching in design and caid:a theoretical exploration[D]. Calgary:The University of Calgary，1999:50-66.
[33] 刘征，鲁娜，吴剑锋. 基于设计认知的草图研究综述[J]. 浙江大学学报(工学版)，2010，44(12):2376-2382. LIU Zheng，LU Na，WU Jianfeng. Review of sketch based on design cognition[J]. Journal of Zhejiang University (Engineering Science)，2010，44(12):2376-2382.
[34] ISOLA P，ZHU J，ZHOU T，et al. Image-to-image translation with conditional adversarial networks[C]// Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR)，Honolulu，Hi，USA，July 21-26，2017:5967-5976.
[35] WANG T C，LIU M Y，ZHU J Y，et al. High-resolution image synthesis and semantic manipulation with conditional GANs[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)，Salt Lake City，Utah，June 18-23，2018:8798-8807.
[36] ZHU J Y，PARK T，ISOLA P，et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]// IEEE International Conference on Computer Vision (ICCV)，Venice，Italy，October 22-29，2017:2242-2251.
[37] JIN Y，CHUSILP P. Study of mental iteration in different design situations[J]. Design Studies，2006，27(1):25-55.
[38] GATYS L A，ECKER A. S，BETHGE M. Image style transfer using convolutional neural networks[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas，June 26-July 1，2016:2414-2423.
[39] HADJI I，WILDES R P. What do we understand about convolutional networks?[J]. arXiv preprint arXiv:1803.08834，2018. https://arxiv.org/pdf/1803.08834.pdf.
[40] IOFFE S，SZEGEDY C. Batch normalization:accelerating deep network training by reducing internal covariate shift[C]// Proceedings of the 32nd International Conference on International Conference on Machine Learning (ICML). Lille France，June 6-11，2015: 448-456.
[41] ULYANOV D，VEDALDI A，LEMPITSKY V. Instance normalization:The missing ingredient for fast stylization[J]. arXiv preprint arXiv:1607.08022v3， 2018. https://arxiv.org/pdf/1607.08022v3.pdf.
[42] DUMOULIN V，SHLENS J，KUDLUR M，et al. A learned representation for artistic style[C]// 5th International Conference on Learning Representations (ICLR)，Toulon，France，April 24-26，2017.
[43] HUANG X，BELONGIE S. Arbitrary style transfer in real-time with adaptive instance normalization[C]// IEEE International Conference on Computer Vision (ICCV)，Venice，Italy，October 22-29，2017:1510-1519.
[44] LIN T，MAIRE M，BELONGIE S，et al. Microsoft COCO:common objects in context[C]// European Conference on Computer Vision (ECCV)，Zürich，Switzerland，September 6-12，2014:740-755.
[45] HENDRYCKS D，GIMPEL K. Gaussian error linear units (GELUs)[J]. arXiv preprint arXiv:1606. 08415，2016. https://arxiv.org/pdf/1606.08415v4.pdf.
[46] KINGMA D，BA J. Adam:a method for stochastic optimization[C]// 3rd International Conference on Learning Representations (ICLR)，San Diego，CA，USA，May 7-9，2015.
[47] MAATEN L V D，HINTON G. Visualizing Data using t-SNE[J]. Journal of Machine Learning Research，2008，9(2605):2579-2605.
[48] LECUN Y，BOTTOU L，BENGIO Y，et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE，1998，86(11):2278-2324.
[49] LOSHCHILOV I，HUTTER F. Fixing Weight Decay Regularization in Adam[C]// 6th International Conference on Learning Representations (ICLR)，Vancouver，BC，Canada，April 30-May 3，2018.
[50] SCHON D A，WIGGINS G. Kinds of seeing and their functions in designing[J]. Design Studies，1992，13(2):135-156.
[51] 胡洁. 人工智能驱动的艺术创新[J]. 装饰，2019(11):12-17. HU Jie. Artificial intelligence driven art innovation[J]. Zhuang Shi，2019(11):12-17.
[52] HUANG F，CANNY J F. Sketchforme:composing sketched scenes from text descriptions for interactive applications[C]// Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (UIST)，New Orleans LA USA，October 20-23，2019:209-220.
[53] FORREST H，ELDON S，DAVID H，et al. Scones:towards conversational authoring of sketches[C]// Proceedings of the 25th International Conference of Intelligent User Interfaces (IUI)，Cagliari Italy，March 17-20，2020:313-323.