Meaningful machine learning robustness evaluation in real-world machine learning enabled system contexts

Applied research presented in this paper describes an approach to provide meaningful evaluation of the Machine Learning (ML) components in a Full Motion Video (FMV) Machine Learning Enabled System ( MLES). The MLES itself is not discussed in the paper. We focus on the experimental activity that has been designed to provide confidence that the MLES, when fielded under dynamic and uncertain conditions, performance will not be undermined by a lack of ML robustness. For example, to real-world changes of the same scene under differing light conditions. The paper details the technical approach and how it is applied to data, across the overall experimental pipeline, consisting of a perturbation engine, test pipeline and metric production. Data is from a small imagery dataset and the results are shown and discussed as part of a proof of concept study.