Attention Understands Semantic Relations.

Today, natural language processing heavily relies on pre-trained large language models. Even though such models are criticised for poor interpretability, they still yield state-of-the-art solutions for a wide range of very different tasks. While many probing studies have been conducted to measure the models awareness of grammatical knowledge, semantic probing is less popular. In this work, we introduce a probing pipeline to study how semantic relations are represented in transformer language models. We show that in this task, attention scores express the information about relations similar to the layers' output activations despite their lesser ability to represent surface cues. This supports the hypothesis that attention mechanisms focus not only on syntactic relational information but semantic as well.