Improved Techniques for Training GANs

Vicki Cheung
Vicki Cheung
Alec Radford
Alec Radford

NIPS, pp. 2234-2242, 2016.

Cited by: 3504|Bibtex|Views180|Links
EI
Keywords:
MNISTgenerative adversarial networktraining gansevaluation metricgenerative modelMore(9+)
Weibo:
Generative adversarial networks are a promising class of generative models that has so far been held back by unstable training and by the lack of a proper evaluation metric

Abstract:

We present a variety of new architectural features and training procedures that we apply to the generative adversarial networks (GANs) framework. Using our new techniques, we achieve state-of-the-art results in semi-supervised classification on MNIST, CIFAR-10 and SVHN. The generated images are of high quality as confirmed by a visual Tur...More

Code:

Data:

0
Introduction
  • Generative adversarial networks [1] (GANs) are a class of methods for learning generative models based on game theory.
  • The goal of GANs is to train a generator network G(z; θ(G)) that produces samples from the data distribution, pdata(x), by transforming vectors of noise z as x = G(z; θ(G)).
  • The training signal for G is provided by a discriminator network D(x) that is trained to distinguish samples from the generator distribution pmodel(x) from real data.
  • Training GANs requires finding a Nash equilibrium of a non-convex game with continuous, highdimensional parameters.
  • When used to seek for a Nash equilibrium, these algorithms may fail to converge [4]
Highlights
  • Generative adversarial networks [1] (GANs) are a class of methods for learning generative models based on game theory
  • Batch normalization greatly improves optimization of neural networks, and was shown to be highly effective for DCGANs [3]. It causes the output of a neural network for an input example x to be highly dependent on several other inputs x in the same minibatch. To avoid this problem we introduce virtual batch normalization (VBN), in which each example x is normalized based on the statistics collected on a reference batch of examples that are chosen once and fixed at the start of training, and on x itself
  • Our Inception score is closely related to the objective used for training generative models in CatGAN [14]: we had less success using such an objective for training, we find it is a good metric for evaluation that correlates very
  • Generative adversarial networks are a promising class of generative models that has so far been held back by unstable training and by the lack of a proper evaluation metric
  • We apply our techniques to the problem of semi-supervised learning, achieving state-of-the-art results on a number of different data sets in computer vision
Methods
  • The authors performed semi-supervised experiments on MNIST, CIFAR-10 and SVHN, and sample generation experiments on MNIST, CIFAR-10, SVHN and ImageNet. The authors provide code to reproduce the majority of the experiments.
  • The MNIST dataset contains 60, 000 labeled images of digits.
  • The authors perform semi-supervised training with a small randomly picked fraction of these, considering setups with 20, 50, 100, and 200 labeled examples.
  • Results are averaged over 10 random subsets of labeled data, each chosen to have a balanced number of examples from each class.
  • The remaining training images are provided without labels.
Results
  • The authors achieve state-of-the-art results in semi-supervised classification on MNIST, CIFAR-10 and SVHN.
Conclusion
  • Generative adversarial networks are a promising class of generative models that has so far been held back by unstable training and by the lack of a proper evaluation metric.
  • The authors propose several techniques to stabilize training that allow them to train models that were previously untrainable.
  • The authors' proposed evaluation metric gives them a basis for comparing the quality of these models.
  • The authors apply our techniques to the problem of semi-supervised learning, achieving state-of-the-art results on a number of different data sets in computer vision.
  • The contributions made in this work are of a practical nature; the authors hope to develop a more rigorous theoretical understanding in future work
Summary
  • Generative adversarial networks [1] (GANs) are a class of methods for learning generative models based on game theory.
  • The goal of GANs is to train a generator network G(z; θ(G)) that produces samples from the data distribution, pdata(x), by transforming vectors of noise z as x = G(z; θ(G)).
  • The training signal for G is provided by a discriminator network D(x) that is trained to distinguish samples from the generator distribution pmodel(x) from real data.
  • One of the primary goals of this work is to improve the effectiveness of generative adversarial networks for semi-supervised learning.
  • Like many deep generative models, GANs have previously been applied to semi-supervised learning [13, 14], and our work can be seen as a continuation and refinement of this effort.
  • The discriminator is still required to output a single number for each example indicating how likely it is to come from the training data: The task of the discriminator is effectively still to classify single examples as real data or generated data, but it is able to use the other examples in the minibatch as side information.
  • As an alternative to human annotators, we propose an automatic method to evaluate samples, which we find to correlate well with human evaluation: We apply the Inception model1 [19] to every generated image to get the conditional label distribution p(y|x).
  • We can do semi-supervised learning with any standard classifier by adding samples from the GAN generator G to our data set, labeling them with a new “generated” class y = K + 1, and correspondingly increasing the dimension of our classifier output from K to K + 1.
  • This approach introduces an interaction between G and our classifier that we do not fully understand yet, but empirically we find that optimizing G using feature matching GAN works very well for semi-supervised learning, while training G using GAN with minibatch discrimination does not work at all.
  • Besides achieving state-of-the-art results in semi-supervised learning, the approach described above has the surprising effect of improving the quality of generated images as judged by human annotators.
  • (Right) Samples generated with semi-supervised learning using feature match- minibatch discrimination.
  • We use this data set to study semi-supervised learning, as well as to examine the visual quality of samples that can be achieved.
  • We apply our techniques to the problem of semi-supervised learning, achieving state-of-the-art results on a number of different data sets in computer vision.
  • The contributions made in this work are of a practical nature; we hope to develop a more rigorous theoretical understanding in future work
Tables
  • Table1: Number of incorrectly classified test examples for the semi-supervised setting on permutation invariant MNIST. Results are averaged over 10 seeds
  • Table2: Test error on semi-supervised CIFAR-10. Results are averaged over 10 splits of data
  • Table3: Table of Inception scores for samples generated by various models for 50, 000 images. Score highly correlates with human judgment, and the best score is achieved for natural images. Models that generate collapsed samples have relatively low score. This metric allows us to avoid relying on human evaluations. “Our methods” includes all the techniques described in this work, except for feature matching and historical averaging. The remaining experiments are ablation experiments showing that our techniques are effective. “-VBN+BN” replaces the VBN in the generator with BN, as in DCGANs. This causes a small decrease in sample quality on CIFAR. VBN is more important for ImageNet. “-L+HA” removes the labels from the training process, and adds historical averaging to compensate. HA makes it possible to still generate some recognizable objects. Without HA, sample quality is considerably reduced (see ”-L”). “-LS” removes label smoothing and incurs a noticeable drop in performance relative to “our methods.” “-MBF” removes the minibatch features and incurs a very large drop in performance, greater even than the drop resulting from removing the labels. Adding HA cannot prevent this problem
Download tables as Excel
Related work
  • Several recent papers focus on improving the stability of training and the resulting perceptual quality of GAN samples [2, 3, 5, 6]. We build on some of these techniques in this work. For instance, we use some of the “DCGAN” architectural innovations proposed in Radford et al [3], as discussed below.

    One of our proposed techniques, feature matching, discussed in Sec. 3.1, is similar in spirit to approaches that use maximum mean discrepancy [7, 8, 9] to train generator networks [10, 11]. Another of our proposed techniques, minibatch features, is based in part on ideas used for batch normalization [12], while our proposed virtual batch normalization is a direct extension of batch normalization.

    One of the primary goals of this work is to improve the effectiveness of generative adversarial networks for semi-supervised learning (improving the performance of a supervised task, in this case, classification, by learning on additional unlabeled examples). Like many deep generative models, GANs have previously been applied to semi-supervised learning [13, 14], and our work can be seen as a continuation and refinement of this effort.
Funding
  • Using our new techniques, we achieve state-of-the-art results in semi-supervised classification on MNIST, CIFAR-10 and SVHN
Reference
  • Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, et al. Generative adversarial nets. In NIPS, 2014.
    Google ScholarLocate open access versionFindings
  • Emily Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus. Deep generative image models using a laplacian pyramid of adversarial networks. arXiv preprint arXiv:1506.05751, 2015.
    Findings
  • Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
    Findings
  • Ian J Goodfellow. On distinguishability criteria for estimating generative models. arXiv preprint arXiv:1412.6515, 2014.
    Findings
  • Daniel Jiwoong Im, Chris Dongjoo Kim, Hui Jiang, and Roland Memisevic. Generating images with recurrent adversarial networks. arXiv preprint arXiv:1602.05110, 2016.
    Findings
  • Donggeun Yoo, Namil Kim, Sunggyun Park, Anthony S Paek, and In So Kweon. Pixel-level domain transfer. arXiv preprint arXiv:1603.07442, 2016.
    Findings
  • Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Scholkopf. Measuring statistical dependence with hilbert-schmidt norms. In Algorithmic learning theory, pages 63–77.
    Google ScholarLocate open access versionFindings
  • Kenji Fukumizu, Arthur Gretton, Xiaohai Sun, and Bernhard Scholkopf. Kernel measures of conditional dependence. In NIPS, volume 20, pages 489–496, 2007.
    Google ScholarLocate open access versionFindings
  • Alex Smola, Arthur Gretton, Le Song, and Bernhard Scholkopf. A hilbert space embedding for distributions. In Algorithmic learning theory, pages 13–31.
    Google ScholarLocate open access versionFindings
  • Yujia Li, Kevin Swersky, and Richard S. Zemel. Generative moment matching networks. CoRR, abs/1502.02761, 2015.
    Findings
  • Gintare Karolina Dziugaite, Daniel M Roy, and Zoubin Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. arXiv preprint arXiv:1505.03906, 2015.
    Findings
  • Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
    Findings
  • Ilya Sutskever, Rafal Jozefowicz, Karol Gregor, et al. Towards principled unsupervised learning. arXiv preprint arXiv:1511.06440, 2015.
    Findings
  • Jost Tobias Springenberg. Unsupervised and semi-supervised learning with categorical generative adversarial networks. arXiv preprint arXiv:1511.06390, 2015.
    Findings
  • Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. 2016. MIT Press.
    Google ScholarFindings
  • George W Brown. Iterative solution of games by fictitious play. Activity analysis of production and allocation, 13(1):374–376, 1951.
    Google ScholarLocate open access versionFindings
  • C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the Inception Architecture for Computer Vision. ArXiv e-prints, December 2015.
    Google ScholarFindings
  • David Warde-Farley and Ian Goodfellow. Adversarial perturbations of deep neural networks. In Tamir Hazan, George Papandreou, and Daniel Tarlow, editors, Perturbations, Optimization, and Statistics, chapter 11. 2016. Book in preparation for MIT Press.
    Google ScholarFindings
  • Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. arXiv preprint arXiv:1512.00567, 2015.
    Findings
  • Tim Salimans and Diederik P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. arXiv preprint arXiv:1602.07868, 2016.
    Findings
  • Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Neural Information Processing Systems, 2014.
    Google ScholarLocate open access versionFindings
  • Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. Distributional smoothing by virtual adversarial examples. arXiv preprint arXiv:1507.00677, 2015.
    Findings
  • Lars Maaløe, Casper Kaae Sønderby, Søren Kaae Sønderby, and Ole Winther. Auxiliary deep generative models. arXiv preprint arXiv:1602.05473, 2016.
    Findings
  • Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi-supervised learning with ladder networks. In Advances in Neural Information Processing Systems, 2015.
    Google ScholarLocate open access versionFindings
  • Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, et al. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
    Findings
  • Martın Abadi, Ashish Agarwal, Paul Barham, et al. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org.
    Google ScholarFindings
Your rating :
0

 

Tags
Comments