The threat landscape of malicious applications, or malware, is persistently growing and evolving. Malware has become one of the major offensive components of the global cybersecurity threat. Accurate understanding of malware behavior is a crucial step towards developing systems that deter, detect, and defend against malware threats. Unfortunately, the widely deployed signature-based and lightweight static-analysis-based detection techniques (Antivirus) are easily evaded by techniques commonly seen in the wild, such as code obfuscation, packing, and encryption. Recent malware detection systems are moving towards a more robust dynamic analysis approach. These systems execute suspicious samples in a controlled environment, called "sandbox", and observe malicious intent through their dynamic behavior. However, many sophisticated evasive malware samples are evading such analysis by first detecting the analysis environment and then stopping their malicious activities. Because of the sophisticated and evolving techniques used by the malware authors, so far the analysis and detection of evasive malware has been largely a manual process. This manual approach is not scalable to tens of thousands of new malware samples that we observe every day.

In this dissertation, I will present my research on scalable and automated evasive malware analysis. First, I will discuss my work on reducing the input load to a dynamic analyzer and present a static-filter called SIGMAL, which is based on image processing and machine learning algorithms. I will outline a bare metal-based approach to evasive malware analysis. Then, I will describe BAREBOX, a novel technique for improving the scalability of this approach. Next, I will propose a new algorithm for detecting deviation in the malware behaviors among different execution environments. The BARECLOUD project leverages this algorithm and builds a system for automatically detecting realworld evasive malware at a large-scale. Finally, I will present MALGENE, a first step towards the automatic extraction of evasion signatures from evasive malware samples. Given millions of incoming suspicious samples as a daily input, SIGMAL can quickly identify newer and relevant samples, BAREBOX can execute them transparently in a bare metal environment and generate behavior profiles, which BARECLOUD can use to automatically detect evasive samples. Given these evasive samples, MALGENE can automatically extract evasion signatures and cluster them according to the underlying evasion techniques.