Enterprise-Level Controlled Experiments at Scale: Challenges and Solutions

Online controlled experiments, known as A/B tests, have delivered tremendous value for businesses and customers. Traditionally, online controlled experiments running in search engines, websites, mobile apps, and other Business-to-Consumer (B2C) products use individual customers as randomization units. However, this may not be possible in Business-to-Business (B2B) products that are sold to, and used by, other enterprises. B2B products typically have constraints such as the need to provide identical experiences to all users within an enterprise. In other scenarios, such as features that need to be used by multiple users to be meaningful or the metrics of interest can be only obtained by showing the same experience under one enterprise, experimentation randomized by enterprise is needed. In this paper, based on experiences from Microsoft and Outreach, we introduce the concept of Enterprise Level Experiments (ELE) in software product development. We discuss the unique technical challenges in designing and analyzing ELEs and present solutions to overcome them. Furthermore, we present and discuss a real-world example of ELE run by Microsoft. Our aims in this paper are to provide practical guidance on running trustworthy experiments in B2B scenarios, and to stimulate further research in this new and important area of experimentation.