Use Cases for Evaluation of Machine Based Situation Awareness

Situation awareness (SA) is important both for human decision making and for complex automated decision making. The presumption is that improving the accuracy of SA will lead to better decisions. While there has been significant research on measuring the accuracy of human SA, there has not been as much work on machine-based SA. Unlike humans, complex systems have many levels of decision making that may operate independently and may run at very different timescales. The accuracy of SA for each decision making process, determined in isolation, need not contribute to overall system performance. Moreover, achieving more accurate SA may require devoting resources that are disproportionate to the benefits. We propose that one should focus on the net value of SA to the system rather than simply on the accuracy. In this article, we present some use cases for determining the value of machine-based SA. The purpose is to illustrate how one can quantitatively evaluate SA so that one can optimize important issues for automated decision making processes such as system performance and stability.