Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space
Conference on Empirical Methods in Natural Language Processing(2020)
摘要
When trained effectively, the Variational Autoencoder (VAE) can be both a powerful generative model and an effective representation learning framework for natural language. In this paper, we propose the first large-scale language VAE model Optimus (Organizing sentences via Pre-Trained Modeling of a Universal Space). A universal latent embedding space for sentences is first pre-trained on large text corpus, and then fine-tuned for various language generation and understanding tasks. Compared with GPT-2, Optimus enables guided language generation from an abstract level using the latent vectors. Compared with BERT, Optimus can generalize better on low-resource language understanding tasks due to the smooth latent space structure. Extensive experimental results on a wide range of language tasks demonstrate the effectiveness of Optimus. It achieves new state-of-the-art on VAE language modeling benchmarks.
更多查看译文
关键词
sentences,latent,modeling,space,pre-trained
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络