On Using Distribution-Based Compositionality Assessment to Evaluate Compositional Generalisation in Machine Translation.
CoRR(2023)
摘要
Compositional generalisation (CG), in NLP and in machine learning more
generally, has been assessed mostly using artificial datasets. It is important
to develop benchmarks to assess CG also in real-world natural language tasks in
order to understand the abilities and limitations of systems deployed in the
wild. To this end, our GenBench Collaborative Benchmarking Task submission
utilises the distribution-based compositionality assessment (DBCA) framework to
split the Europarl translation corpus into a training and a test set in such a
way that the test set requires compositional generalisation capacity.
Specifically, the training and test sets have divergent distributions of
dependency relations, testing NMT systems' capability of translating
dependencies that they have not been trained on. This is a fully-automated
procedure to create natural language compositionality benchmarks, making it
simple and inexpensive to apply it further to other datasets and languages. The
code and data for the experiments is available at
https://github.com/aalto-speech/dbca.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要