Toward Human-AI Collaboration: A Recommender System to Support CS1 Instructors to Select Problems for Assignments and Exams

Programming online judges (POJs) have been increasingly used in CS1 classes, as they allow students to practice and get quick feedback. For instructors, it is a useful tool for creating assignments and exams. However, selecting problems in POJs is time consuming. First, problems are generally not organized based on topics covered in the CS1 syllabus. Second, assessing whether problems require similar effort to be completed and map onto the same topic is a subjective and expert-dependent task. The difficulty increases if the instructor must create variations of these assessments, e.g., to avoid plagiarism. Thus, here, we research how to support CS1 instructors in the task of selecting problems, to compose one-size-fits-all or personalized assignments/exams. Our solution is to propose a novel intelligent recommender system, based on a fine-grained data-driven analysis of the students' effort on solving problems in the integrated development environment of a POJ system, and automatic detection of topics for CS1 problems, based on problem descriptions. Data collected from 2714 students are processed to support, via our artificial intelligence (AI) method recommendations, the instructors' decision-making process. We evaluated our method against the state of the art in a simple blind experiment with CS1 instructors (N < 35). Results show that our recommendations are 88% accurate, surpassing our baseline (p < 0.05). Finally, our work paves the way for novel POJ smart learning environments, wherein instructors define learning tasks (assignments/exams) supported by AI.