Hostname: page-component-848d4c4894-x24gv Total loading time: 0 Render date: 2024-05-05T05:21:29.929Z Has data issue: false hasContentIssue false
  • English
  • Français

Generative Pre-Trained Transformer for Design Concept Generation: An Exploration

Published online by Cambridge University Press:  26 May 2022

Q. Zhu  and
J. Luo
Q. Zhu*
Affiliation:
Singapore University of Technology and Design, Singapore
J. Luo
Affiliation:
Singapore University of Technology and Design, Singapore
*
qihao_zhu@mymail.sutd.edu.sg

Abstract

Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the ‘Save PDF’ action button.

Novel concepts are essential for design innovation and can be generated with the aid of data stimuli and computers. However, current generative design algorithms focus on diagrammatic or spatial concepts that are either too abstract to understand or too detailed for early phase design exploration. This paper explores the uses of generative pre-trained transformers (GPT) for natural language design concept generation. Our experiments involve the use of GPT-2 and GPT-3 for different creative reasonings in design tasks. Both show reasonably good performance for verbal design concept generation.

Keywords

early design phaseidea generationgenerative designnatural language generationgenerative pre-trained transformer
Type
Article
Information
Proceedings of the Design Society , Volume 2: DESIGN2022 , May 2022 , pp. 1825 - 1834
Creative Commons
Creative Common License - CCCreative Common License - BYCreative Common License - NCCreative Common License - ND
This is an Open Access article, distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives licence (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is unaltered and is properly cited. The written permission of Cambridge University Press must be obtained for commercial re-use or in order to create a derivative work.
Copyright
The Author(s), 2022.

References

Ahmed, S. and Boelskifte, P. (2006), “Investigations of Product Design Engineering Students Intentions and a Users Perception of Product Character”, Proceedings of Nordesign, Reykjavik, Iceland, pp. 372–381, 2006.Google Scholar
Amin-Nejad, A., Ive, J., & Velupillai, S. (2020), “Exploring transformer text generation for medical dataset augmentation”. In Proceedings of the 12th Language Resources and Evaluation Conference (pp. 4699–4708).Google Scholar
Arslan, Y., Allix, K., Veiber, L., Lothritz, C., Bissyandé, T. F., Klein, J., & Goujon, A. (2021). “A comparison of pre-trained language models for multi-class text classification in the financial domain”. In Comp. Proc. Web Conf. 2021 (260–268). 10.1145/3442442.3451375Google Scholar
Bonnardel, N., & Didier, J. (2020), “Brainstorming variants to favor creative design”. Applied Ergo., 83, 102987. 10.1016/j.apergo.2019.102987CrossRefGoogle ScholarPubMed
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., et al. . (2020), “Language models are few-shot learners”. Advances in Neural Information Processing Systems 33 (NeurIPS 2020).Google Scholar
Burnap, A., Liu, Y., Pan, Y., Lee, H., Gonzalez, R., et al, P.Y. (2016), “Estimating and exploring the product form design space using deep generative models”, in IDETC-CIE, ASME, V02AT03A013. 10.1115/DETC2016-60091Google Scholar
Camburn, B., He, Y., Raviselvam, S., Luo, J. and Wood, K. (2020), “Machine learning-based design concept evaluation”, Journal of Mechanical Design, 142(3), 031113. 10.1115/1.4045126CrossRefGoogle Scholar
Campbell, M., (2009) “A Graph Grammar Methodology for Generative Systems” [Online]. Available: http://repositories.lib.utexas.edu/handle/2152/6258. [Accessed: 10 -11 -2021].Google Scholar
Chakrabarti, A., Shea, K., Stone, R., Cagan, J., Campbell, M., et al. . (2011), “Computer-based design synthesis research: an overview”, J. Comput. Inf. Sci. Eng., 11(2). 10.1115/1.3593409CrossRefGoogle Scholar
Chiu, I. and Shu, L. (2007) “Understanding the use of language stimuli in concept generation”, in IDETC-CIE, 161–172. 10.1115/DETC2007-35772Google Scholar
Dogan, K.M., Suzuki, H., Gunpinar, E. and Kim, M.-S. (2019), “A generative sampling system for profile designs with shape constraints and user evaluation”, Computer-Aided Design, 111, 93–112. 10.1016/j.cad.2019.02.002CrossRefGoogle Scholar
Duan, J., Zhao, H., Zhou, Q., Qiu, M., & Liu, M. (2020, November). “A Study of Pre-trained Language Models in Natural Language Processing”. In 2020 IEEE International Conference on Smart Cloud (SmartCloud) (pp. 116–121). 10.1109/SmartCloud49737.2020.00030CrossRefGoogle Scholar
Fang, J. (2021), An Application of Customized GPT-2 Text Generator for Modern Content Creators, [Master Thesis], UCLA.Google Scholar
Fargnoli, M., Rovida, E. and Troisi, R. (2006), “The morphological matrix: Tool for the development of innovative design solutions”, in 4th ICAD, 1–7. 10.1109/FIE.1998.736828Google Scholar
Gatt, A. and Krahmer, E. (2018), “Survey of the state of the art in natural language generation: Core tasks, applications and evaluation”, Journal of Artificial Intelligence Research, 61, 65–170. 10.1613/jair.5477CrossRefGoogle Scholar
Gentner, D. (1983), “Structure-mapping: A theoretical framework for analogy”, Cognitive science, 7(2), 155–170. 10.1016/S0364-0213(83)80009-3CrossRefGoogle Scholar
Goldschmidt, G. and Smolkov, M. (2006), “Variances in the impact of visual stimuli on design problem solving performance”, Design studies, 27(5), 549–569. 10.1016/j.destud.2006.01.002CrossRefGoogle Scholar
Goucher-Lambert, K. and Cagan, J. (2019) 'Crowdsourcing inspiration: Using crowd generated inspirational stimuli to support designer ideation', Design studies, 61, 1–29. 10.1016/j.destud.2019.01.001CrossRefGoogle Scholar
Han, J., Forbes, H., Shi, F., Hao, J. and Schaefer, D. (2020), “A data-driven approach for creative concept generation and evaluation”, in Proceedings of the Design Society: DESIGN Conference, 167–176. 10.1017/dsd.2020.5Google Scholar
Han, J., Shi, F., Chen, L. and Childs, P.R. (2018), “The Combinator–a computer-based tool for creative idea generation based on a simulation approach”, Design Science, 4. 10.1017/dsj.2018.7CrossRefGoogle Scholar
He, Y., Camburn, B., Liu, H., Luo, J., Yang, M., et al. . (2019), “Mining and representing the concept space of existing ideas for directed ideation”, J. Mech. Des., 141(12). 10.1115/1.4044399CrossRefGoogle Scholar
Hinton, G.E. and Salakhutdinov, R.R. (2006), “Reducing the dimensionality of data with neural networks”, Science, 313(5786), 504–507. 10.1126/science.1127647CrossRefGoogle ScholarPubMed
Huang, Q., Gan, Z., Celikyilmaz, A., Wu, D., Wang, J. and He, X. (2019), “Hierarchically structured reinforcement learning for topically coherent visual story generation”, in Proceedings of the AAAI Conference on Artificial Intelligence, 8465–8472. 10.1609/aaai.v33i01.33018465CrossRefGoogle Scholar
Ilevbare, I.M., Probert, D. and Phaal, R. (2013), “A review of TRIZ, and its benefits and challenges in practice”, Technovation, Vol. 33 No. 2-3, pp. 30–37. 10.1016/j.technovation.2012.11.003CrossRefGoogle Scholar
Jin, X. and Dong, H. (2020), “New design heuristics in the digital era”, in Proceedings of the Design Society: DESIGN Conference, 607–616. 10.1017/dsd.2020.321Google Scholar
Kang, S.W. and Tucker, C.S. (2015), “Automated concept generation based on function-form synthesis”, in IDETC-CIE, ASME, V02AT03A008. 10.1115/DETC2015-47687Google Scholar
Kenny, D. (2019), 'Machine translation', In: Baker, M., & Saldanha, G. (Eds.), Routledge Encyclopedia of Translation Studies Routledge (3rd ed.), pp. 305–310. 10.4324/9781315678627CrossRefGoogle Scholar
Kenton, J. D. M. W. C., & Toutanova, L. K. (2019). “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”. In Proc. of NAACL-HLT (pp. 4171–4186).Google Scholar
Kusner, M., Sun, Y., Kolkin, N., & Weinberger, K. (2015, June). “From word embeddings to document distances”. In Int'l Conference on Machine Learning (pp. 957–966). PMLR.Google Scholar
Li, Z., Jiang, X., Shang, L. and Li, H. (2018), “Paraphrase Generation with Deep Reinforcement Learning”, in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 3865–3878.CrossRefGoogle Scholar
Luo, J., Sarica, S. and Wood, K.L. (2019), “Computer-aided design ideation using InnoGPS”, in IDETC-CIE, ASME, V02AT03A011. 10.1115/DETC2019-97587Google Scholar
Luo, J., Sarica, S. and Wood, K.L. (2021), “Guiding data-driven design ideation by knowledge distance”, Knowledge-Based Systems, 218, 106873. 10.1016/j.knosys.2021.106873Google Scholar
Nie, Z., Lin, T., Jiang, H. and Kara, L.B. (2021), “Topologygan: Topology optimization using generative adversarial networks based on physical fields over the initial domain”, J. Mech. Des., 143(3), 031715. 10.1115/1.4049533CrossRefGoogle Scholar
Oh, S., Jung, Y., Kim, S., Lee, I. and Kang, N. (2019), “Deep generative design: Integration of topology optimization and generative models”, J. Mech. Des., 141(11). 10.1115/1.4044229CrossRefGoogle Scholar
Ozsoy, M.G., Alpaslan, F.N. and Cicekli, I. (2011), “Text summarization using latent semantic analysis”, Journal of Information Science, 37(4), 405–417. 10.1177%2F0165551511408848CrossRefGoogle Scholar
Pahl, Beitz, W., Feldhusen, J., & Grote, K.-H. (2007), “Engineering Design A Systematic Approach”, Springer London. 10.1007/978-1-84628-319-2Google Scholar
Papineni, K., Roukos, S., Ward, T., & Zhu, W. J. (2002, July). “Bleu: a method for automatic evaluation of machine translation”. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics (pp. 311–318).Google Scholar
Pascanu, R., Mikolov, T. and Bengio, Y. (2013), “On the difficulty of training recurrent neural networks”, in International conference on machine learning, PMLR, 1310–1318.Google Scholar
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019), “Language models are unsupervised multitask learners”. OpenAI blog, 1(8), 9.Google Scholar
Regenwetter, L., Nobari, A.H. and Ahmed, F. (2021), “Deep Generative Models in Engineering Design: A Review”, arXiv preprint arXiv:2110.10863.Google Scholar
Ren, Y., Burnap, A. and Papalambros, P. (2013), “Quantification of perceptual design attributes using a crowd”, in DS 75-6: Proceedings of the 19th International Conference on Engineering Design (ICED13), Design for Harmonies, Vol. 6: Design Information and Knowledge, Seoul, Korea, 19-22.08. 2013.Google Scholar
Sangelkar, S. and McAdams, D.A. (2017), “Automated Graph Grammar Generation for Engineering Design With Frequent Pattern Mining”, in International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, ASME, V02AT03A006. 10.1115/DETC2017-67520Google Scholar
Sarica, S., Song, B., Luo, J. and Wood, K.L. (2021), “Idea generation with technology semantic network”, AI EDAM, 1–19. 10.1017/S0890060421000020Google Scholar
Shah, J.J., Vargas-Hernandez, N., Summers, J.D. and Kulkarni, S. (2001), “Collaborative Sketching (C-Sketch)—An idea generation technique for engineering design”, The Journal of Creative Behavior, 35(3), 168–198. 10.1002/j.2162-6057.2001.tb01045.xCrossRefGoogle Scholar
Shih, P.C., Nguyen, D.H., Hirano, S.H., Redmiles, D.F. and Hayes, G.R. (2009), “GroupMind: supporting idea generation through a collaborative mind-mapping tool”, in Proceedings of the ACM 2009 international conference on Supporting group work, 139–148. 10.1145/1531674.1531696CrossRefGoogle Scholar
Stone, R.B., Wood, K.L. and Crawford, R.H. (2000), “A heuristic method for identifying modules for product architectures”, Design studies, 21(1), 5–31. 10.1016/S0142-694X(99)00003-4CrossRefGoogle Scholar
Topal, M. O., Bas, A., & van Heerden, I. (2021), “Exploring transformers in natural language generation: GPT, BERT, and XLNET”. International Conference on Interdisciplinary Applications of AI (ICIDAAI)Google Scholar
Tschimmel, K. (2012), “Design Thinking as an effective Toolkit for Innovation”, in ISPIM Conference Proceedings, The International Society for Professional Innovation Management (ISPIM), 1.Google Scholar
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., et al. . (2017), “Attention is all you need”, in Advances in neural information processing systems, 5998–6008.Google Scholar
Viswanathan, V., Tomko, M. and Linsey, J. (2016), “A study on the effects of example familiarity and modality on design fixation”, AI EDAM, 30(2), 171–184. 10.1017/S0890060416000056Google Scholar
Vlah, D., Žavbi, R. and Vukašinović, N. (2020), “Evaluation of topology optimization and generative design tools as support for conceptual design”, in Proceedings of the Design Society: DESIGN Conference, 451–460. 10.1017/dsd.2020.165Google Scholar
Yagita, H., Tose, A., Nakajima, M., Kim, S.K. and Maeno, T. (2011), “A validation regarding effectiveness of scenario graph”, in IDETC-CIE, 385–394. 10.1115/DETC2011-48047Google Scholar
Yilmaz, S., Daly, S.R., Seifert, C.M. and Gonzalez, R. (2016), “Evidence-based design heuristics for idea generation”, Design studies, 46, 95–124. 10.1016/j.destud.2016.05.001CrossRefGoogle Scholar