An exploratory study on assessing the impact of environment variations on the results of load tests.

Large-scale software systems like Amazon and healthcare.gov are used by thousands or millions of people every day. To ensure the quality of these systems, load testing is a required testing procedure in addition to the conventional functional testing techniques like unit and system integration testing. One of the important requirements of load testing is to create a field-like test environment. Unfortunately, this task is often very challenging due to reasons like security and rapid field updates. In this paper, we have conducted an exploratory study on the impact of environment variations on the results of load tests. We have run over 110 hours load tests, which examine the system's behavior under load with various changes (e.g., installing an anti-virus program) to the targeted deployment environment. We call such load tests as environment-variation-based load tests. Case studies in three open source systems have shown that there is a clear performance impact on the system's performance due to these environment changes. Different scenarios react differently to the changes in the underlying computing resources. When predicting the performance of the system under environment changes that are not previously load tested, our ensemble models out-perform (24% - 94% better) the baseline models.