Multi-level sandboxing techniques for execution-based stealthy malware detection

These days all kinds of malware are pervasive on the Internet. Compared to their ancestors that were commonly used for vandalism or demonstration of skills, modern malware, such as Bots, are driven by the underground economics. Often consisting of hundreds to thousands of bots, botnets are one of the most serious threats on the Internet, responsible for various attacks, such as spamming and distributed denial of service (DDoS). As web browsers are the main interface for the majority of Internet users to surf the Internet today, many of such stealthy malware seek to invade via web browsers in the form of browser helper objects (BHO) and browser toolbars. To defend against Internet malware, existing schemes mainly rely on either signature-based or anomaly-based detection approaches. Signature-based detection is effective for known malware if the malware signature has been generated. However, the effectiveness of signature-based schemes is challenged by polymorphism, metamorphism, obfuscation, encryption, and other techniques. Moreover, signature-based schemes do not work for zero-day (or unknown) malware. On the other hand, anomaly-based detection schemes seek to detect behavior patterns that do not conform to the established normal patterns. Anomaly-based detection schemes do not require malware signatures. However, modern computer software and systems are often complicated, building and analyzing a comprehensive behavior model is time consuming and even impractical. To overcome these challenges, we propose a novel execution-based approach for stealthy malware detection. In order to facilitate such run-time detection, we aim to design and implement multi-level sandboxing techniques to create controlled running environments to execute testing programs so that their behaviors can be closely observed and analyzed. First, we leverage virtual machines for OS-level sandboxing to detect bots on individual hosts. By cloning the host image to a virtual machine and screening user input on the virtual machine, the detection noise is significantly reduced. We find that a typical bot exhibits three invariant features along its onset: (1) the startup of a bot is automatic without requiring any user actions; (2) a bot must establish a command and control channel with its botmaster; and (3) a bot will perform local or remote attacks sooner or later. These invariants indicate three indispensable phases (startup, preparation, and attack) for a bot attack. Thus, we propose BotTracer to detect these three phases with the assistance of OS-level sandboxing techniques. To validate BotTracer , we implement a prototype of BotTracer based on VMware . The results show that BotTracer can successfully detect all the bots in the experiments. However, BotTracer may slightly degrade the user performance. Furthermore, advanced malware could evade BotTracer by performing virtual machine fingerprinting. Second, to overcome the limitations of OS-level sandboxes, we build Malyzer based on process-level sandboxes for malware detection. The key of Malyzer is to defeat malware anti-detection mechanisms at startup and runtime so that malware behaviors during execution can be accurately captured and distinguished. For analysis, Malyzer always starts a copy, referred to as a shadow process, of any suspicious process in the process-level sandbox by defeating all startup anti-detection mechanisms employed in the suspicious process. To defeat internal runtime anti-detection attempts, Malyzer further makes this shadow process mutually invisible to the original suspicious process. To defeat external anti-detection attempts, Malyzer makes as if the shadow process runs on a different machine to the outside. Since ultimately malware will conduct local information harvesting or dispersion, Malyzer constantly monitors the shadow process behaviors and adopts a hybrid scheme for its behavior analysis. In our experiments, Malyzer can accurately detect all malware samples that employ various anti-detection techniques. Lastly, to detect and contain malicious browser plugins, we develop sePlugin with intra-process sandboxing techniques. With an intra-process sandbox, only plugins are closely monitored for misbehavior detection without confining the entire process. This further reduces the detection overhead while maintaining transparency to end-users. Based on intra-process sandboxing techniques, we build sePlugin to enhance the security of a browser by enforcing security policies on plugins' accessing requests to the browser's internal objects and external system-level resources, such as file systems and network interfaces. sePlugin deals with both native and .NET-based plugins and its unique design renders it possible to work with commodity web browsers without requiring any modifications to the legacy browser architecture or plugin code. We implement sePlugin in Windows XP and IE8.