Ruffle Riley: Insights from Designing and Evaluating a Large Language Model-Based Conversational Tutoring System
arxiv(2024)
摘要
Conversational tutoring systems (CTSs) offer learning experiences through
interactions based on natural language. They are recognized for promoting
cognitive engagement and improving learning outcomes, especially in reasoning
tasks. Nonetheless, the cost associated with authoring CTS content is a major
obstacle to widespread adoption and to research on effective instructional
design. In this paper, we discuss and evaluate a novel type of CTS that
leverages recent advances in large language models (LLMs) in two ways: First,
the system enables AI-assisted content authoring by inducing an easily editable
tutoring script automatically from a lesson text. Second, the system automates
the script orchestration in a learning-by-teaching format via two LLM-based
agents (Ruffle Riley) acting as a student and a professor. The system allows
for free-form conversations that follow the ITS-typical inner and outer loop
structure. We evaluate Ruffle Riley's ability to support biology lessons in two
between-subject online user studies (N = 200) comparing the system to simpler
QA chatbots and reading activity. Analyzing system usage patterns,
pre/post-test scores and user experience surveys, we find that Ruffle Riley
users report high levels of engagement, understanding and perceive the offered
support as helpful. Even though Ruffle Riley users require more time to
complete the activity, we did not find significant differences in short-term
learning gains over the reading activity. Our system architecture and user
study provide various insights for designers of future CTSs. We further
open-source our system to support ongoing research on effective instructional
design of LLM-based learning technologies.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要