Language to Network: Conditional Parameter Adaptation with Natural Language Descriptions

Tian Jin, Zhun Liu, Shengjia Yan, Alexandre Eichenberger, Louis-Philippe Morency


Abstract
Transfer learning using ImageNet pre-trained models has been the de facto approach in a wide range of computer vision tasks. However, fine-tuning still requires task-specific training data. In this paper, we propose N3 (Neural Networks from Natural Language) - a new paradigm of synthesizing task-specific neural networks from language descriptions and a generic pre-trained model. N3 leverages language descriptions to generate parameter adaptations as well as a new task-specific classification layer for a pre-trained neural network, effectively “fine-tuning” the network for a new task using only language descriptions as input. To the best of our knowledge, N3 is the first method to synthesize entire neural networks from natural language. Experimental results show that N3 can out-perform previous natural-language based zero-shot learning methods across 4 different zero-shot image classification benchmarks. We also demonstrate a simple method to help identify keywords in language descriptions leveraged by N3 when synthesizing model parameters.
Anthology ID:
2020.acl-main.625
Volume:
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2020
Address:
Online
Editors:
Dan Jurafsky, Joyce Chai, Natalie Schluter, Joel Tetreault
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
6994–7007
Language:
URL:
https://aclanthology.org/2020.acl-main.625
DOI:
10.18653/v1/2020.acl-main.625
Bibkey:
Cite (ACL):
Tian Jin, Zhun Liu, Shengjia Yan, Alexandre Eichenberger, and Louis-Philippe Morency. 2020. Language to Network: Conditional Parameter Adaptation with Natural Language Descriptions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6994–7007, Online. Association for Computational Linguistics.
Cite (Informal):
Language to Network: Conditional Parameter Adaptation with Natural Language Descriptions (Jin et al., ACL 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.acl-main.625.pdf
Video:
 http://slideslive.com/38928907