STOP THE (AUTOGRADER) INSANITY: Regression Penalties to Deter Autograder Overreliance

ABSTRACTAutograders are an invaluable tool for deploying assessments in large classes. However students sometimes rely on the autograder in place of careful thought for ways to improve to their solution. We sought to naturally encourage students to check their own solutions more, and hammer the grader less. To do this, we imposed a penalty each time a student's grade went down: we called these regression penalties. We assessed whether the introduction of these penalties resulted in less reliance on the autograder without hurting student performance. Encouragingly, the number of autograder submissions was reduced by roughly half while only slightly decreasing the median final grade. Students reported feeling nervous about their submissions, but noted that they checked their own solutions by testing their code far more than they would have without the penalty. Students also expressed positivity about the regression model.