Machine Translation Testing via Syntactic Tree Pruning
CoRR(2024)
摘要
Machine translation systems have been widely adopted in our daily life,
making life easier and more convenient. Unfortunately, erroneous translations
may result in severe consequences, such as financial losses. This requires to
improve the accuracy and the reliability of machine translation systems.
However, it is challenging to test machine translation systems because of the
complexity and intractability of the underlying neural models. To tackle these
challenges, we propose a novel metamorphic testing approach by syntactic tree
pruning (STP) to validate machine translation systems. Our key insight is that
a pruned sentence should have similar crucial semantics compared with the
original sentence. Specifically, STP (1) proposes a core semantics-preserving
pruning strategy by basic sentence structure and dependency relations on the
level of syntactic tree representation; (2) generates source sentence pairs
based on the metamorphic relation; (3) reports suspicious issues whose
translations break the consistency property by a bag-of-words model. We further
evaluate STP on two state-of-the-art machine translation systems (i.e., Google
Translate and Bing Microsoft Translator) with 1,200 source sentences as inputs.
The results show that STP can accurately find 5,073 unique erroneous
translations in Google Translate and 5,100 unique erroneous translations in
Bing Microsoft Translator (400
64.5
vary in types and more than 90
techniques. There are 9,393 erroneous translations unique to STP, which is
711.9
to detect translation errors for the original sentences with a recall reaching
74.0
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要