Context-sensitive multi-model anomaly detection

Empirical studies reveal that the security guarantees offered by operating systems and applications routinely fail, often relinquishing control of systems, data, and networks to attackers. Intrusion detection systems (IDSs) have been proposed as mechanisms for detecting and responding to malicious activity in computer systems when such failures occur. A prominent thrust of research in this area focuses on misuse-based schemes, in which attacks are detected using pre-defined signatures. The disadvantage of this approach is that only known attacks can be detected. To overcome this problem, an additional thrust has instead advocated model-based (anomaly) detection schemes. In these systems, pre-computed models of normal system behavior are compared against observed activity to identify attacks. Since attacks are not explicitly represented by the system, these approaches have the advantage of sensitivity to novel, unforeseen attacks. This dissertation argues that an anomaly detection system composed of multiple learning-based detection models is well-suited to detect attacks with a low rate of false alarms in two domains of practical interest: system call invocations, monitored at the kernel level, and HTTP requests, monitored at the application level. We combine this approach with a mechanism that applies Bayesian networks to the problem of assigning an overall anomaly score given a collection of individual model scores. Finally, we show that by tailoring individual model instances to individual software sub-behaviors, the overall detection performance can be improved considerably.