STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases
CoRR(2024)
Abstract
Answering real-world user queries, such as product search, often requires
accurate retrieval of information from semi-structured knowledge bases or
databases that involve blend of unstructured (e.g., textual descriptions of
products) and structured (e.g., entity relations of products) information.
However, previous works have mostly studied textual and relational retrieval
tasks as separate topics. To address the gap, we develop STARK, a large-scale
Semi-structure retrieval benchmark on Textual and Relational Knowledge Bases.
We design a novel pipeline to synthesize natural and realistic user queries
that integrate diverse relational information and complex textual properties,
as well as their ground-truth answers. Moreover, we rigorously conduct human
evaluation to validate the quality of our benchmark, which covers a variety of
practical applications, including product recommendations, academic paper
searches, and precision medicine inquiries. Our benchmark serves as a
comprehensive testbed for evaluating the performance of retrieval systems, with
an emphasis on retrieval approaches driven by large language models (LLMs). Our
experiments suggest that the STARK datasets present significant challenges to
the current retrieval and LLM systems, indicating the demand for building more
capable retrieval systems that can handle both textual and relational aspects.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined