USIS: A unified semantic image synthesis model trained on a single or multiple samples

Semantic image synthesis methods learn to generate new images conditioned on predefined semantic label maps. Existing methods require access to large-volume samples labeled with semantic maps, which limits their applications. We propose USIS, a Unified Semantic Image Synthesis model which can be trained on only a single or multiple pairs of images and semantic maps. Once trained, a USIS model can generate new images according to unseen semantic maps, as existing semantic image synthesis methods do. Specifically, we design a hierarchical architecture to reconstruct training samples and grad-ually learn the distributions of multi-scale patches in samples from coarse to fine. To avoid the error accu-mulation across scales, we propose a mixed training strategy to stabilize the training process. Extensive experiments on one-or multiple-sample datasets show our proposed model achieves state-of-the-art performance in terms of visual fidelity.(c) 2022 Elsevier B.V. All rights reserved.