Are All Tests Equal? A Comparison of Emulator and Device Testing for Mobile Usability Evaluation

As technology becomes increasingly mobile-focused, usability techniques must evolve to support users in this next era of computing. Usability testing of mobile applications and websites can be conducted with either a mobile device, or a desktop emulator that simulates the device’s content and functionality. Emulator testing offers several advantages, such as the capacity to test multiple devices in rapid succession, though device testing may offer more valid data. Although both methods are commonly implemented in mobile usability testing, little research has directly compared the findings generated from each approach. The current study examines whether emulator and device testing produce differences in eye tracking measures and established metrics that assess the ISO 9241-11 usability dimensions of effectiveness, efficiency, and satisfaction. Results indicated that mean task completion times and mean fixations per task were greater for participants in the emulator condition than for those in the device condition for two of the fourteen tasks tested. A qualitative analysis of potential drivers of these effects is presented, and implications for substituting device testing with emulator testing are discussed.