A Thousand Worlds: Scenery Specification and Generation for Simulation-Based Testing of Mobile Robot Navigation Stacks

Is mobile robot navigation a solved problem? We asked this question to 14 professional robot software engineers who work with navigation stacks of mobile, wheeled robots on a daily basis. They unanimously report that it remains challenging to ensure the performance of their mobile robots. We find that the method of choice to verify a robot's performance is to expose it to different environments under varying conditions. Unfortunately, these field tests are costly to set up and often too risky to execute. Therefore, robot software engineers want to replicate real-world environments in simulated sceneries (i) to test their navigation stack or parts of it prior to deployment; and (ii) to reproduce erroneous behaviour observed in the real world. Motivated by these insights, we have developed a domain-specific language and associated tooling which enables engineers (i) to specify sceneries of indoor environments; and (ii) to automatically generate variants thereof that resemble changes in the real world. We demonstrate how our approach enables the simulation-based replication of real-world environments and allows us to recreate erroneous robot behaviour as reported in the interviews. While performing simulation-based tests, we discovered an 8-year dormant bug in the ROS navigation stack.