Weizhi Wang
ABSTRACT
In this paper, we propose to formulate the task-oriented dialogue system as the purely natural language generation task, so as to fully leverage the large-scale pre-trained models like GPT-2 and simplify complicated delexicalization prepossessing. However, directly applying this method heavily suffers from the dialogue entity inconsistency caused by the removal of delexicalized tokens, as well as the catastrophic forgetting problem of the pre-trained model during fine-tuning, leading to unsatisfactory performance. To alleviate these problems, we design a novel GPT-Adapter-CopyNet network, which incorporates the lightweight adapter and CopyNet modules into GPT-2 to achieve better performance on transfer learning and dialogue entity generation. Experimental results conducted on the DSTC8 Track 1 benchmark and MultiWOZ dataset demonstrate that our proposed approach significantly outperforms baseline models with a remarkable performance on automatic and human evaluations.
Supplemental Material
- Yoshua Bengio, R. Ducharme, Pascal Vincent, and Christian Janvin. 2000. A Neural Probabilistic Language Model. In J. Mach. Learn. Res.Google Scholar
- Pawel Budzianowski and Ivan Vuli. 2019. Hello, It's GPT-2--How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems. arXiv preprint arXiv:1907.05774 (2019).Google Scholar
- Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, O. Ramadan, and Milica Gasic. 2018. MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling. In EMNLP.Google Scholar
- Wenhu Chen, Jianshu Chen, Pengda Qin, X. Yan, and William Yang Wang. 2019. Semantically Conditioned Dialog Response Generation via Hierarchical Disentangled Self-Attention. In ACL.Google Scholar
- Rahul Goel, Shachi Paul, and Dilek Hakkani-Tür. 2019. Hyst: A hybrid approach for flexible and accurate dialogue state tracking. arXiv preprint arXiv:1907.00883 (2019).Google Scholar
- Jiatao Gu, Z. Lu, Hang Li, and Victor O. K. Li. 2016. Incorporating Copying Mechanism in Sequence-to-Sequence Learning. ArXiv abs/1603.06393 (2016).Google Scholar
- Junliang Guo, Zhirui Zhang, Linli Xu, Boxing Chen, and Enhong Chen. 2021. Adaptive Adapters: An Efficient Way to Incorporate BERT Into Neural Machine Translation. IEEE/ACM Transactions on Audio, Speech, and Language Processing 29 (2021), 1740--1751.Google ScholarDigital Library
- Junliang Guo, Zhirui Zhang, Linli Xu, Hao-Ran Wei, Boxing Chen, and Enhong Chen. 2020. Incorporating BERT into Parallel Sequence Decoding with Adapters. Advances in Neural Information Processing Systems 33 (2020).Google Scholar
- Suchin Gururangan, Ana Marasovi?, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't Stop Pretraining: Adapt Language Models to Domains and Tasks. ArXiv abs/2004.10964 (2020).Google Scholar
- Donghoon Ham, Jeong-Gwan Lee, Youngsoo Jang, and Kee-Eung Kim. 2020. End-to-End Neural Pipeline for Goal-Oriented Dialogue Systems using GPT-2. In ACL.Google Scholar
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. 770--778.Google Scholar
- Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. arXiv preprint arXiv:2005.00796 (2020).Google Scholar
- Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. arXiv preprint arXiv:1902.00751 (2019).Google Scholar
- Seokhwan Kim, Michel Galley, Chulaka Gunasekara, Sungjin Lee, Adam Atkinson, Baolin Peng, Hannes Schulz, Jianfeng Gao, Jinchao Li, Mahmoud Adada, et al. 2019. The eighth dialog system technology challenge. arXiv preprint arXiv:1911.06394 (2019).Google Scholar
- Sungdong Kim, Sohee Yang, Gyuwan Kim, and Sang-Woo Lee. 2020. Efficient Dialogue State Tracking by Selectively Overwriting Memory. In ACL. 567--582.Google Scholar
- Young-Bum Kim, Sungjin Lee, and Karl Stratos. 2017. Onenet: Joint domain, intent, slot prediction for spoken language understanding. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 547--553.Google ScholarCross Ref
- Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
- Hung Le, Richard Socher, and Steven CH Hoi. 2019. Non-Autoregressive Dialog State Tracking. In ICLR.Google Scholar
- Sungjin Lee, Qi Zhu, Ryuichi Takanobu, Xiang Li, Y. Zhang, Zheng Zhang, Jin chao Li, Baolin Peng, Xiujun Li, Minlie Huang, and Jianfeng Gao. 2019. ConvLab: Multi-Domain End-to-End Dialog System Platform. ArXiv abs/1904.08637 (2019).Google Scholar
- Wenqiang Lei, Xisen Jin, Min-Yen Kan, Z. Ren, X. He, and D. Yin. 2018. Sequicity: Simplifying Task-oriented Dialogue Systems with Single Sequence-to-Sequence Architectures. In ACL.Google Scholar
- Weizhe Lin, B-H Tseng, and Bill Byrne. 2021. Knowledge-Aware Graph-Enhanced GPT-2 for Dialogue State Tracking. In EMNLP.Google Scholar
- Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. 2018. Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systems. In ACL.Google Scholar
- M. McCloskey and N. J. Cohen. 1989. Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem. Psychology of Learning and Motivation 24 (1989), 109--165.Google ScholarCross Ref
- Shikib Mehri, Tejas Srinivasan, and M. Eskénazi. 2019. Structured Fusion Networks for Dialog. In SIGdial.Google Scholar
- Kishore Papineni, S. Roukos, T. Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In ACL.Google Scholar
- Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Liden, and Jianfeng Gao. 2020. SOLOIST: Few-shot Task-Oriented Dialog with A Single Pre-trained Auto-regressive Model. arXiv preprint arXiv:2005.05298 (2020).Google Scholar
- Baolin Peng, Xiujun Li, Jianfeng Gao, J. Liu, and K. Wong. 2018. Integrating planning for task-completion dialogue policy learning. ArXiv abs/1801.06176 (2018).Google Scholar
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.Google Scholar
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace's Transformers: State-of-the-art Natural Language Processing. ArXiv abs/1910.03771 (2019).Google Scholar
- Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. arXiv preprint arXiv:1905.08743 (2019).Google Scholar
- Yunyi Yang, Yun-Hao Li, and Xiaojun Quan. 2020. UBAR: Towards Fully End-toEnd Task-Oriented Dialog Systems with GPT-2. ArXiv abs/2012.03539 (2020).Google Scholar
- Jian-Guo Zhang, Kazuma Hashimoto, Chien-Sheng Wu, Yao Wan, Philip S Yu, Richard Socher, and Caiming Xiong. 2019. Find or classify? dual strategy for slot-value predictions on multi-domain dialog state tracking. arXiv preprint arXiv:1910.03544 (2019).Google Scholar
- Yichi Zhang, Zhijian Ou, and Zhou Yu. 2020. Task-Oriented Dialog Systems That Consider Multiple Appropriate Responses under the Same Context. In AAAI.Google Scholar
- Zhirui Zhang, Xiujun Li, Jianfeng Gao, and Enhong Chen. 2019. Budgeted policy learning for task-oriented dialogue systems. arXiv preprint arXiv:1906.00499 (2019).Google Scholar
- Tiancheng Zhao, Kaige Xie, and M. Eskénazi. 2019. Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models. ArXiv abs/1902.08858 (2019).Google Scholar
Index Terms
- Task-Oriented Dialogue System as Natural Language Generation
Recommendations
Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generation
SIGIR '22: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information RetrievalRecently, pre-training methods have shown remarkable success in task-oriented dialog (TOD) systems. However, most existing pre-trained models for TOD focus on either dialog understanding or dialog generation, but not both. In this paper, we propose ...
Dialog-to-Actions: Building Task-Oriented Dialogue System via Action-Level Generation
SIGIR '23: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information RetrievalEnd-to-end generation-based approaches have been investigated and applied in task-oriented dialogue systems. However, in industrial scenarios, existing methods face the bottlenecks of reliability (e.g., domain-inconsistent responses, repetition problem, ...
Mutually improved response generation and dialogue summarization for multi-domain task-oriented dialogue systems
AbstractWith the development of pre-trained language models (PrLM), the research of PrLM-based multi-domain task-oriented dialogue systems (TOD) has attracted growing attention and has achieved great progress. However, most current studies suffer from ...
Comments