Predicting Defects in SAP Products: A Replicated Study

Given a large body of code, how do we know where to focus our quality assurance effort? By mining the software's defect history, we can automatically learn which code features correlated with de- fects in the past—and leverage these correlations for new predic- tions: "In the past, high inheritance depth meant high number of de- fects. Since this new component also has a high inheritance depth, let us test it thoroughly". Such history-based approaches work best if the new component is similar to the components learned from. But how does learning from history perform for projects with high variability between components? We ran a study on two SAP prod- ucts involving a wide spectrum of functionality. We found that learning and predicting was accurate at package level, but not at product level. These results suggest that to learn from past de- fects, one should separate the product into component clusters with similar functionality, and make separate predictions for each clus- ter. Initial approaches to form such clusters automatically, based on similarity of metrics, showed promising accuracy.