Automating Systems Course Unit and Integration Testing: Experience Report

Introducing software testing has taken on a greater importance in undergraduate computer science curricula in the last several years, with many departments using JUnit or other testing frameworks in the programming sequence and software engineering courses. We have developed an automated framework for unit and integration testing and grading for our intermediate-level systems course projects. Our system--designed to test C programs--combines the Check unit testing framework, custom Bash scripts for integration testing, and the Valgrind Memcheck memory leak detection tool. Although our courses use Linux, the framework is platform-independent and has been tested on a variety of other platforms. We have used this framework for seven semesters with four different instructors as part of the computer science program at a primarily undergraduate university with an emphasis on liberal arts. We distribute both public and private tests so that students get immediate feedback on their progress without knowing the actual contents of every test. We have observed that knowing their code is not completely working motivates more students to figure out what they don't understand before the project deadline. It also gives students examples of different levels of tests to use to debug their code, encourages them to develop a deeper understanding of the project specification, and reduces student anxiety about grades.