Wake-Sleep Variational Autoencoders For Language Modeling
NEURAL INFORMATION PROCESSING, ICONIP 2017, PT I(2017)
摘要
Variational Autoencoders (VAEs) are known to easily suffer from the KL-vanishing problem when combining with powerful autoregressive models like recurrent neural networks (RNNs), which prohibits their wide application in natural language processing. In this paper, we tackle this problem by tearing the training procedure into two steps: learning effective mechanisms to encode and decode discrete tokens (wake step) and generalizing meaningful latent variables by reconstructing dreamed encodings (sleep step). The training pattern is similar to the wake-sleep algorithm: these two steps are trained alternatively until an equilibrium is achieved. We test our model in a language modeling task. The results demonstrate significant improvement over the current state-of-the-art latent variable models.
更多查看译文
关键词
Variational Autoencoder,Wake-sleep algorithm,Language modeling,Latent variable
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络