What should your run-time configuration framework do to help developers?

The users or deployment engineers of a software system can adapt such a system to a wide range of deployment and usage scenarios by changing the value of configuration options, for example by disabling unnecessary features, tweaking performance-related parameters or specifying GUI preferences. However, the literature agrees that the flexibility of such options comes at a price: misconfigured options can lead a software system to crash in the production environment, while even in the absence of such configuration errors, a large number of configuration options makes a software system more complicated to deploy and use. In earlier work, we also found that developers who intend to make their application configurable face 22 challenges that impact their configuration engineering activities, ranging from technical to management-related or even inherent to the domain of configuration engineering. In this paper, we use a prototyping approach to derive and empirically evaluate requirements for tool support able to deal with 13 (primarily technical) configuration engineering challenges. In particular, via a set of interviews with domain experts, we identify four requirements by soliciting feedback on an incrementally evolving prototype. The resulting “Config2Code” prototype, which implements the four requirements, is then empirically evaluated via a user study involving 55 participants that comprises 10 typical configuration engineering tasks, ranging from the creation, comprehension, refactoring, and reviewing of configuration options to the quality assurance of options and debugging of configuration failures. A configuration framework satisfying the four requirements enables developers to perform more accurately and more swiftly in 70% and 60% (respectively) of the configuration engineering tasks than a state-of-the-practice framework not satisfying the requirements. Furthermore, such a framework allows to reduce the time taken for these tasks by up to 94.62%, being slower for only one task.