Desirable Characteristics for AI Teaching Assistants in Programming Education

Providing timely and personalized feedback to large numbers of students is along-standing challenge in programming courses. Relying on human teachingassistants (TAs) has been extensively studied, revealing a number of potentialshortcomings. These include inequitable access for students with low confidencewhen needing support, as well as situations where TAs provide direct solutionswithout helping students to develop their own problem-solving skills. With theadvent of powerful large language models (LLMs), digital teaching assistantsconfigured for programming contexts have emerged as an appealing and scalableway to provide instant, equitable, round-the-clock support. Although digitalTAs can provide a variety of help for programming tasks, from high-levelproblem solving advice to direct solution generation, the effectiveness of suchtools depends on their ability to promote meaningful learning experiences. Ifstudents find the guardrails implemented in digital TAs too constraining, or ifother expectations are not met, they may seek assistance in ways that do nothelp them learn. Thus, it is essential to identify the features that studentsbelieve make digital teaching assistants valuable. We deployed an LLM-powereddigital assistant in an introductory programming course and collected studentfeedback (n=813) on the characteristics of the tool they perceived to be mostimportant. Our results highlight that students value such tools for theirability to provide instant, engaging support, particularly during peak timessuch as before assessment deadlines. They also expressed a strong preferencefor features that enable them to retain autonomy in their learning journey,such as scaffolding that helps to guide them through problem-solving stepsrather than simply being shown direct solutions.