XATU: A Fine-grained Instruction-based Benchmark for Explainable Text Updates
arxiv(2023)
摘要
Text editing is a crucial task of modifying text to better align with user
intents. However, existing text editing benchmark datasets contain only
coarse-grained instructions and lack explainability, thus resulting in outputs
that deviate from the intended changes outlined in the gold reference. To
comprehensively investigate the text editing capabilities of large language
models (LLMs), this paper introduces XATU, the first benchmark specifically
designed for fine-grained instruction-based explainable text editing. XATU
considers finer-grained text editing tasks of varying difficulty
(simplification, grammar check, fact-check, etc.), incorporating lexical,
syntactic, semantic, and knowledge-intensive edit aspects. To enhance
interpretability, we combine LLM-based annotation and human annotation,
resulting in a benchmark that includes fine-grained instructions and
gold-standard edit explanations. By evaluating existing LLMs against our
benchmark, we demonstrate the effectiveness of instruction tuning and the
impact of underlying architecture across various editing tasks. Furthermore,
extensive experimentation reveals the significant role of explanations in
fine-tuning language models for text editing tasks. The benchmark will be
open-sourced to support reproduction and facilitate future research
at .
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要