Improved Techniques for Training GANs

Vicki Cheung
Vicki Cheung
Alec Radford
Alec Radford

NIPS, pp. 2226-2234, 2016.

Cited by: 0|Bibtex|Views39|Links
EI
Keywords:
nash equilibriumadversarial networktraining ganssample quality is considerably reducedMechanical TurkMore(9+)
Weibo:
Generative adversarial networks are a promising class of generative models that has so far been held back by unstable training and by the lack of a proper evaluation metric

Abstract:

We present a variety of new architectural features and training procedures that we apply to the generative adversarial networks (GANs) framework. Using our new techniques, we achieve state-of-the-art results in semi-supervised classification on MNIST, CIFAR-10 and SVHN. The generated images are of high quality as confirmed by a visual Tur...More

Code:

Data:

0
Introduction
  • Generative adversarial networks [1] (GANs) are a class of methods for learning generative models based on game theory.
  • The goal of GANs is to train a generator network G(z; θ(G)) that produces samples from the data distribution, pdata(x), by transforming vectors of noise z as x = G(z; θ(G)).
  • The training signal for G is provided by a discriminator network D(x) that is trained to distinguish samples from the generator distribution pmodel(x) from real data.
  • Training GANs requires finding a Nash equilibrium of a non-convex game with continuous, highdimensional parameters.
  • When used to seek for a Nash equilibrium, these algorithms may fail to converge [4]
Highlights
  • Generative adversarial networks [1] (GANs) are a class of methods for learning generative models based on game theory
  • Batch normalization greatly improves optimization of neural networks, and was shown to be highly effective for DCGANs [3]. It causes the output of a neural network for an input example x to be highly dependent on several other inputs x in the same minibatch. To avoid this problem we introduce virtual batch normalization (VBN), in which each example x is normalized based on the statistics collected on a reference batch of examples that are chosen once and fixed at the start of training, and on x itself
  • Our Inception score is closely related to the objective used for training generative models in CatGAN [14]: we had less success using such an objective for training, we find it is a good metric for evaluation that correlates very well with human judgment
  • Generative adversarial networks are a promising class of generative models that has so far been held back by unstable training and by the lack of a proper evaluation metric
  • We apply our techniques to the problem of semi-supervised learning, achieving state-of-the-art results on a number of different data sets in computer vision
Methods
  • The authors performed semi-supervised experiments on MNIST, CIFAR-10 and SVHN, and sample generation experiments on MNIST, CIFAR-10, SVHN and ImageNet. The authors provide code to reproduce the majority of the experiments.
  • The MNIST dataset contains 60, 000 labeled images of digits.
  • The authors perform semi-supervised training with a small randomly picked fraction of these, considering setups with 20, 50, 100, and 200 labeled examples.
  • Results are averaged over 10 random subsets of labeled data, each chosen to have a balanced number of examples from each class.
  • The remaining training images are provided without labels.
Results
  • The authors achieve state-of-the-art results in semi-supervised classification on MNIST, CIFAR-10 and SVHN.
Conclusion
  • Generative adversarial networks are a promising class of generative models that has so far been held back by unstable training and by the lack of a proper evaluation metric.
  • The authors propose several techniques to stabilize training that allow them to train models that were previously untrainable.
  • The authors' proposed evaluation metric gives them a basis for comparing the quality of these models.
  • The authors apply our techniques to the problem of semi-supervised learning, achieving state-of-the-art results on a number of different data sets in computer vision.
  • The contributions made in this work are of a practical nature; the authors hope to develop a more rigorous theoretical understanding in future work
Summary
  • Introduction:

    Generative adversarial networks [1] (GANs) are a class of methods for learning generative models based on game theory.
  • The goal of GANs is to train a generator network G(z; θ(G)) that produces samples from the data distribution, pdata(x), by transforming vectors of noise z as x = G(z; θ(G)).
  • The training signal for G is provided by a discriminator network D(x) that is trained to distinguish samples from the generator distribution pmodel(x) from real data.
  • Training GANs requires finding a Nash equilibrium of a non-convex game with continuous, highdimensional parameters.
  • When used to seek for a Nash equilibrium, these algorithms may fail to converge [4]
  • Methods:

    The authors performed semi-supervised experiments on MNIST, CIFAR-10 and SVHN, and sample generation experiments on MNIST, CIFAR-10, SVHN and ImageNet. The authors provide code to reproduce the majority of the experiments.
  • The MNIST dataset contains 60, 000 labeled images of digits.
  • The authors perform semi-supervised training with a small randomly picked fraction of these, considering setups with 20, 50, 100, and 200 labeled examples.
  • Results are averaged over 10 random subsets of labeled data, each chosen to have a balanced number of examples from each class.
  • The remaining training images are provided without labels.
  • Results:

    The authors achieve state-of-the-art results in semi-supervised classification on MNIST, CIFAR-10 and SVHN.
  • Conclusion:

    Generative adversarial networks are a promising class of generative models that has so far been held back by unstable training and by the lack of a proper evaluation metric.
  • The authors propose several techniques to stabilize training that allow them to train models that were previously untrainable.
  • The authors' proposed evaluation metric gives them a basis for comparing the quality of these models.
  • The authors apply our techniques to the problem of semi-supervised learning, achieving state-of-the-art results on a number of different data sets in computer vision.
  • The contributions made in this work are of a practical nature; the authors hope to develop a more rigorous theoretical understanding in future work
Tables
  • Table1: Number of incorrectly classified test examples for the semi-supervised setting on permutation invariant MNIST. Results are averaged over 10 seeds
  • Table2: Test error on semi-supervised CIFAR-10. Results are averaged over 10 splits of data
  • Table3: Table of Inception scores for samples generated by various models for 50, 000 images
Download tables as Excel
Related work
  • Several recent papers focus on improving the stability of training and the resulting perceptual quality of GAN samples [2, 3, 5, 6]. We build on some of these techniques in this work. For instance, we use some of the “DCGAN” architectural innovations proposed in Radford et al [3], as discussed below.

    One of our proposed techniques, feature matching, discussed in Sec. 3.1, is similar in spirit to approaches that use maximum mean discrepancy [7, 8, 9] to train generator networks [10, 11].

    30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.

    Another of our proposed techniques, minibatch features, is based in part on ideas used for batch normalization [12], while our proposed virtual batch normalization is a direct extension of batch normalization.

    One of the primary goals of this work is to improve the effectiveness of generative adversarial networks for semi-supervised learning (improving the performance of a supervised task, in this case, classification, by learning on additional unlabeled examples). Like many deep generative models, GANs have previously been applied to semi-supervised learning [13, 14], and our work can be seen as a continuation and refinement of this effort. In concurrent work, Odena [15] proposes to extend GANs to predict image labels like we do in Section 5, but without our feature matching extension (Section 3.1) which we found to be critical for obtaining state-of-the-art performance.
Funding
  • Presents a variety of new architectural features and training procedures that toes the generative adversarial networks framework
  • Achieves state-of-the-art results in semi-supervised classification on MNIST, CIFAR-10 and SVHN
  • Presents ImageNet samples with unprecedented resolution and show that our methods enable the model to learn recognizable features of ImageNet classes
  • Introduces several techniques intended to encourage convergence of the GANs game
  • Does in Section 5, but without our feature matching extension which found to be critical for obtaining state-of-the-art performance
Reference
  • Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, et al. Generative adversarial nets. In NIPS, 2014.
    Google ScholarLocate open access versionFindings
  • Emily Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus. Deep generative image models using a laplacian pyramid of adversarial networks. arXiv preprint arXiv:1506.05751, 2015.
    Findings
  • Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
    Findings
  • Ian J Goodfellow. On distinguishability criteria for estimating generative models. arXiv preprint arXiv:1412.6515, 2014.
    Findings
  • Daniel Jiwoong Im, Chris Dongjoo Kim, Hui Jiang, and Roland Memisevic. Generating images with recurrent adversarial networks. arXiv preprint arXiv:1602.05110, 2016.
    Findings
  • Donggeun Yoo, Namil Kim, Sunggyun Park, Anthony S Paek, and In So Kweon. Pixel-level domain transfer. arXiv preprint arXiv:1603.07442, 2016.
    Findings
  • Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Scholkopf. Measuring statistical dependence with hilbert-schmidt norms. In Algorithmic learning theory, pages 63–77.
    Google ScholarLocate open access versionFindings
  • Kenji Fukumizu, Arthur Gretton, Xiaohai Sun, and Bernhard Scholkopf. Kernel measures of conditional dependence. In NIPS, volume 20, pages 489–496, 2007.
    Google ScholarLocate open access versionFindings
  • Alex Smola, Arthur Gretton, Le Song, and Bernhard Scholkopf. A hilbert space embedding for distributions. In Algorithmic learning theory, pages 13–31.
    Google ScholarLocate open access versionFindings
  • Yujia Li, Kevin Swersky, and Richard S. Zemel. Generative moment matching networks. CoRR, abs/1502.02761, 2015.
    Findings
  • Gintare Karolina Dziugaite, Daniel M Roy, and Zoubin Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. arXiv preprint arXiv:1505.03906, 2015.
    Findings
  • Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
    Findings
  • Ilya Sutskever, Rafal Jozefowicz, Karol Gregor, et al. Towards principled unsupervised learning. arXiv preprint arXiv:1511.06440, 2015.
    Findings
  • Jost Tobias Springenberg. Unsupervised and semi-supervised learning with categorical generative adversarial networks. arXiv preprint arXiv:1511.06390, 2015.
    Findings
  • Augustus Odena. Semi-supervised learning with generative adversarial networks. arXiv preprint arXiv:1606.01583, 2016.
    Findings
  • Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. 20MIT Press.
    Google ScholarFindings
  • George W Brown. Iterative solution of games by fictitious play. Activity analysis of production and allocation, 13(1):374–376, 1951.
    Google ScholarLocate open access versionFindings
  • C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the Inception Architecture for Computer Vision. ArXiv e-prints, December 2015.
    Google ScholarFindings
  • David Warde-Farley and Ian Goodfellow. Adversarial perturbations of deep neural networks. In Tamir Hazan, George Papandreou, and Daniel Tarlow, editors, Perturbations, Optimization, and Statistics, chapter 11. 2016. Book in preparation for MIT Press.
    Google ScholarFindings
  • Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. arXiv preprint arXiv:1512.00567, 2015.
    Findings
  • Tim Salimans and Diederik P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. arXiv preprint arXiv:1602.07868, 2016.
    Findings
  • Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Neural Information Processing Systems, 2014.
    Google ScholarLocate open access versionFindings
  • Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. Distributional smoothing by virtual adversarial examples. arXiv preprint arXiv:1507.00677, 2015.
    Findings
  • Lars Maaløe, Casper Kaae Sønderby, Søren Kaae Sønderby, and Ole Winther. Auxiliary deep generative models. arXiv preprint arXiv:1602.05473, 2016.
    Findings
  • Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi-supervised learning with ladder networks. In Advances in Neural Information Processing Systems, 2015.
    Google ScholarLocate open access versionFindings
  • Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, et al. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
    Findings
  • Junbo Zhao, Michael Mathieu, Ross Goroshin, and Yann Lecun. Stacked what-where auto-encoders. arXiv preprint arXiv:1506.02351, 2015.
    Findings
  • Martın Abadi, Ashish Agarwal, Paul Barham, et al. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org.
    Google ScholarFindings
Your rating :
0

 

Tags
Comments