Simulation Awareness : Assessing Performance with Limited Simulation Instrumentation

Experts in troubleshooting are skilled at identifying important diagnostic cues and making justified inferences about problems and their causes. In a training setting, students can be assessed for the same troubleshooting skills, as long as there is clarity about the cues students use as a basis for their decisions. In a simulation-based intelligent tutoring system (ITS) where assessment is automated, this means the simulation must be transparent enough to afford an accurate picture of the cues that have been revealed to the student in the environment, in order to validate or invalidate the student’s decisions. But not all simulations can be affordably instrumented to provide the desired level of transparency. This paper describes a domain modeling and assessment approach designed to accommodate reasoning with partial knowledge in cases where there are practical limits on simulation instrumentation. The assessment approach is applied in an ITS for training information technology troubleshooting, with a free-play simulation using virtual machines to reproduce a realistic network of computers. From an experiential point of view, this is ideal for giving trainees the opportunity to perform in a realistic operational environment. However, only a subset of simulation events in the virtual machines can be feasibly collected by instrumentation, and in many cases it is only practical to monitor either student actions, or their results, but not both. The paper describes the modeling and assessment approach in this context, with examples where reductions in simulation instrumentation were achievable. We discuss the applicability of this approach for other domains and its limitations, as well as the methods used for model authoring.