Shaker: a tool for detecting more flaky tests faster

BSTRACTA test case that intermittently passes or fails when performed under the same version of source code and test code is said to be flaky. The presence of flaky tests wastes testing time and effort. The most popular approach in industry to detect flakiness is ReRun. The idea behind ReRun is very simple: failing test cases are re-executed many times looking for inconsistencies in the output. Despite its simplicity, the ReRun strategy is very expensive both in terms of time and in terms of computational resources. This is particularly true for contexts where thousands of test cases are performed on a daily basis. Reducing the rerunning overhead is, thus, of utmost importance. This paper presents Shaker, an open-source tool for detecting flakiness in time-constrained tests by adding noise in the execution environment. The main idea behind Shaker is to add stressing tasks that compete with the test execution for the use of resources (CPU or memory). Shaker is available as a GitHub Actions workflow that can be seamlessly integrated with any GitHub project. Alternatively, Shaker can also be used via its provided Command Line Interface. In our evaluation, Shaker was able to discover more flaky tests than ReRun and in a faster way (less re-executions); besides, our approach revealed tens of new flaky tests that went undetected by ReRun even after 50 re-executions. Thanks to its flexibility and ease of use, we believe that Shaker can be useful for both practitioners and researchers. Demo video: https://youtu.be/7-aiQwOb4rA Shaker website: https://star-rg.github.io/shake