Word Order and World Knowledge
CoRR(2024)
摘要
Word order is an important concept in natural language, and in this work, we
study how word order affects the induction of world knowledge from raw text
using language models. We use word analogies to probe for such knowledge.
Specifically, in addition to the natural word order, we first respectively
extract texts of six fixed word orders from five languages and then pretrain
the language models on these texts. Finally, we analyze the experimental
results of the fixed word orders on word analogies and show that i) certain
fixed word orders consistently outperform or underperform others, though the
specifics vary across languages, and ii) the Wov2Lex hypothesis is not hold in
pre-trained language models, and the natural word order typically yields
mediocre results. The source code will be made publicly available at
https://github.com/lshowway/probing_by_analogy.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要