Contemporary malicious software is crafted and deployed by sophisticated and highly organized criminal enterprises. Modern threats include advanced spyware and large-scale botnets that are designed to profit from the vulnerabilities that are inherent in the well-connected, Internet-enabled computing ecosystem. Furthermore, nation-states have begun leveraging malware in targeted attacks against their enemies to steal state secrets or even damage physical infrastructure.

A considerable security community has formed to combat the ever-evolving malware threat, inspiring an arms race between attackers and defenders. To wit, malware authors continue to improve their techniques for the delivery and concealment of malicious payloads. Meanwhile, researchers have enhanced their approaches to defense, which can be generally divided into three stages: analysis, detection, and response. While valuable work has been done in each of these areas, malware, unfortunately, still flourishes.

In this dissertation, we take a holistic approach to improving malware defense by making novel contributions to all three of the aforementioned stages. In particular, we discuss our analysis of the Torpig botnet, in which we obtained unique insights into the operations of the malicious network by taking control of it for a period of ten days. Our analysis demonstrates how sophisticated and destructive contemporary malware has become. This motivated our development of two host-based systems to detect and contain such malware. The first approach detects malware by implementing a dynamic code identity primitive. The second system contains malware by blocking malicious attempts to interact with trusted processes to carry out hostile actions. Collectively, these systems offer an effective and complementary approach to mitigating the threat of advanced malware.