Methodology for testing and evaluation of safety analytics approaches

There has been a significant increase in the development of data-driven safety analytics (SA) approaches in recent years, geared towards improving industrial safety. In light of these advances it has become imperative to evaluate such approaches in a principled way to determine their merits and limitations. To that end, we propose an evaluation methodology underpinned by a simulated testbed that allows for a comprehensive assessment of SA approaches. While assessing such approaches with historical field data is undoubtedly important, such an assessment has limited statistical power because it corresponds to only a few realizations of an inherently stochastic process. The proposed simulation-based methodology enables validation over a large number of realizations, thereby circumventing the statistical limitations of evaluation on historical data. Moreover, a simulated testbed allows for a comparison under controlled circumstances, resulting in a fair and systematic assessment of potential long-term benefits of SA approaches. We demonstrate the utility of the proposed methodology via a case study that compares a few candidate SA approaches, which differ in the manner they assimilate field data to assess safety risk. We show that the simulation-based methodology indeed reveals useful insights and quantifies the relative merits and drawbacks of the different SA approaches, which would be otherwise difficult to objectively determine in a real-world scenario.