Exploring Contextual Word-level Style Relevance for Unsupervised Style Transfer

Zhou Chulun
Zhou Chulun
Chen Liangyu
Chen Liangyu
Liu Jiachen
Liu Jiachen
Guo Sheng
Guo Sheng

ACL, pp. 7135-7144, 2020.

Cited by: 0|Bibtex|Views27|Links
EI
Keywords:
layer-wise relevance propagationoutput sentenceinput sentenceoutput wordattentional seq2seq modelMore(15+)
Weibo:
We invite 3 annotators with linguistic background to evaluate the output sentences.† The accuracy of style transfer, the preservation of original content and the fluency are the three aspects of model performance we consider

Abstract:

Unsupervised style transfer aims to change the style of an input sentence while preserving its original content without using parallel training data. In current dominant approaches, owing to the lack of fine-grained control on the influence from the target style,they are unable to yield desirable output sentences. In this paper, we prop...More

Code:

Data:

0
Introduction
  • Text style transfer is a task that changes the style of input sentences while preserving their styleindependent content.
  • Many recent models further explore this task in different ways, such as gradient-based optimization method (Liu et al, 2019), dual reinforcement learning (Luo et al, 2019), hierarchical reinforcement learning (Wu et al, 2019) and transformerbased model (Dai et al, 2019)
  • Overall, in these models, the quality of output sentences mainly depends on the content representation of input sentence and the exploitation of the target style
Highlights
  • Text style transfer is a task that changes the style of input sentences while preserving their styleindependent content
  • We explore a training approach based on layer-wise relevance propagation and denoising auto-encoding for the Seq2seq style transfer model, which enables the model to automatically predict the word-level style relevance of output sentences;
  • We invite 3 annotators with linguistic background to evaluate the output sentences.† The accuracy of style transfer (Acc), the preservation of original content (Con) and the fluency (Flu) are the three aspects of model performance we consider
  • Our model achieves the best performance on both datasets in terms of almost every aspects, except that the accuracy of style transfer score of our model is slightly lower than PoiGen on YELP
  • It may be due to the error of the pre-trained style classifier that the transfer accuracy of our model is higher than PoiGen on YELP
  • Note that the content preservation of our model is significantly higher than others, showing our word-level control preserves more
Results
  • The authors invite 3 annotators with linguistic background to evaluate the output sentences.† The accuracy of style transfer (Acc), the preservation of original content (Con) and the fluency (Flu) are the three aspects of model performance the authors consider.
  • The authors' model achieves the best performance on both datasets in terms of almost every aspects, except that the Acc score of the model is slightly lower than PoiGen on YELP.
  • It may be due to the error of the pre-trained style classifier that the transfer accuracy of the model is higher than PoiGen on YELP.
  • Note that the content preservation of the model is significantly higher than others, showing the word-level control preserves more
Conclusion
  • This paper has proposed a novel attentional Seq2seq model equipped with a neural style component for unsupervised style transfer.
  • Equipped with the style component, the model can exploit the word-level predicted style relevance for better style transfer.
  • The authors plan to adapt variational neural network to refine the style transfer model, which has shown effectiveness in other conditional text generation tasks, such as machine translation (Zhang et al, 2016; Su et al, 2018)
Summary
  • Introduction:

    Text style transfer is a task that changes the style of input sentences while preserving their styleindependent content.
  • Many recent models further explore this task in different ways, such as gradient-based optimization method (Liu et al, 2019), dual reinforcement learning (Luo et al, 2019), hierarchical reinforcement learning (Wu et al, 2019) and transformerbased model (Dai et al, 2019)
  • Overall, in these models, the quality of output sentences mainly depends on the content representation of input sentence and the exploitation of the target style
  • Results:

    The authors invite 3 annotators with linguistic background to evaluate the output sentences.† The accuracy of style transfer (Acc), the preservation of original content (Con) and the fluency (Flu) are the three aspects of model performance the authors consider.
  • The authors' model achieves the best performance on both datasets in terms of almost every aspects, except that the Acc score of the model is slightly lower than PoiGen on YELP.
  • It may be due to the error of the pre-trained style classifier that the transfer accuracy of the model is higher than PoiGen on YELP.
  • Note that the content preservation of the model is significantly higher than others, showing the word-level control preserves more
  • Conclusion:

    This paper has proposed a novel attentional Seq2seq model equipped with a neural style component for unsupervised style transfer.
  • Equipped with the style component, the model can exploit the word-level predicted style relevance for better style transfer.
  • The authors plan to adapt variational neural network to refine the style transfer model, which has shown effectiveness in other conditional text generation tasks, such as machine translation (Zhang et al, 2016; Su et al, 2018)
Tables
  • Table1: Performance of different models in YELP and GYAFC. Acc measures the percentage of output sentences that match the target style. BLEU measures the content similarity between the output and the corresponding four human references. G2 and H2 denotes the geometric mean and harmonic mean of Acc and BLEU, respectively. Numbers in bold mean that the improvement to the best performing baseline is statistically significant (t-test with p-value <0.05)
  • Table2: Human evaluation results. We show human rating (1-5) for transfer accuracy (Acc), content preservation (Con), fluency (Flu). The average ratings (Avg) are also calculated as overall scores. Numbers in bold mean that the improvement to the best performing baseline is statistically significant (t-test with p-value <0.05)
  • Table3: Ablation study results
Download tables as Excel
Related work
  • In recent years, unsupervised text style transfer has attracted increasing attention. Most of previous work (Hu et al, 2017; Shen et al, 2017; Fu et al, 2018; Prabhumoye et al, 2018; Xu et al, 2018; Li et al, 2018) aimed at producing a styleindependent content representation from the input sentence and generate the output with target style. For example, Hu et al (2017) employed a variational auto-encoder with an attribute classifier as discriminator, forcing the disentanglement of specific attributes and content in latent representation. Shen et al (2017) exploited an auto-encoder framework with an adversarial style discriminator to obtain a shared latent space cross-aligning the content of text from different styles. Based on multi-task learning and adversarial training of deep neural networks, Fu et al (2018) explored two models to learn style transfer from non-parallel data. Prabhumoye et al (2018) learned a latent representation of the input sentence to better preserve the meaning of the sentence while reducing stylistic properties. Then adversarial training and multi-task learning techniques were exploited to make the output match the desired style. Although these work has shown effectiveness to some extent, however, as analyzed by some recent work (Li et al, 2017; Lample et al, 2019), their style discriminators are prone to be fooled.
Funding
  • This work was supported by the Beijing Advanced Innovation Center for Language Resources (No TYR17002), the National Key R&D Project of China (No 2018AAA0101900), the National Natural Science Foundation of China (No 61672440), and the Scientific Research Project of National Language Committee of China (No YB135-49). Parag Jain, Abhijit Mishra, Amar Prakash Azad, and Karthik Sankaranarayanan. 2019
Reference
  • Sebastian Bach, Alexander Binder, Gregoire Montavon, Frederick Klauschen, Klaus-Robert Muller, and Wojciech Samek. 2015. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one.
    Google ScholarFindings
  • Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR 2015.
    Google ScholarLocate open access versionFindings
  • Ning Dai, Jianze Liang, Xipeng Qiu, and Xuanjing Huang. 2019. Style transformer: Unpaired text style transfer without disentangled latent representation. In Proceedings of ACL 2019.
    Google ScholarLocate open access versionFindings
  • Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2018. Style transfer in text: Exploration and evaluation. In Proceedings of AAAI 2018.
    Google ScholarLocate open access versionFindings
  • Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. Toward controlled generation of text. In Proceedings of ICML 2017.
    Google ScholarLocate open access versionFindings
  • In the future, we plan to adapt variational neural network to refine our style transfer model, which has shown effectiveness in other conditional text generation tasks, such as machine translation (Zhang et al., 2016; Su et al., 2018).
    Google ScholarLocate open access versionFindings
Your rating :
0

 

Tags
Comments