Today's computing systems are in the midst of a transformation driven by proliferation of big data. Explosive growth in data-driven applications is creating unprecedented demands on communication networks. From large data centers to smartphones and tablets, networking infrastructures are struggling to keep up with both the volume of data and the complexity of protocols needed for timely and reliable data delivery. Traditional approaches to network design and management allocate network resources in a static and long-term manner. They are simply unable to handle today's massive and dynamic traffic loads.

In this dissertation, we develop new methods to design networks capable of handling the volume and unpredictable dynamics of today's data traffic. We add aspects of flexibility into existing network designs so that each network entity can obtain resources on the fly based on current needs. Following this principle, we address several high-impact problems related to big data.

Our first research area is supporting distributed computing applications in data centers, which requires timely and reliable delivery of massive data across network entities. This is highly challenging in today's data centers that deploy wired networks with a fixed amount of fiber to each network rack. Once deployed, wired networks are extremely expensive and time consuming to modify or update. This static bandwidth provisioning falls short when dealing with massive, dynamic traffic, and leads to congestion losses and application downtime. We tackle this problem by using flexible wireless links to augment (or even replace) wired networks. We are the first to identify the fundamental challenges of using wireless links in data centers, and we overcome these challenges using a new wireless primitive called 3D beamforming (bouncing 60GHz beam off the ceiling). Our solution is simple, and yet highly effective. It significantly extends wireless transmission range while reducing interference footprint. Using 3D beamforming links, we demonstrate their efficacy by building robust data and control planes to handle massive and unpredictable traffic patterns.

Our second research area focuses on spectrum management in wireless networks. For all wireless transmissions to succeed, they must obtain an adequate amount of spectrum. Historical management policies regulate spectrum statically via long-term national auctions. These traditional auctions are unable to distribute spectrum for fine-grained use (local and short-term use). Therefore they lead to highly inefficient use of spectrum, and create an artificial spectrum scarcity problem. We address this problem by building a much more flexible auction format akin to the eBay marketplace. It dynamically distributes spectrum to users based on their current demands. We are the first to show that existing mechanism designs fail in dynamic spectrum auctions because of the presence of wireless interference. We overcome these challenges by designing new dynamic spectrum auctions that allocate spectrum efficiently while achieving desired economic properties, such as resistance to different types of bidder cheating and collusion.

Finally, our research also demonstrates how big data can be used to implement new network designs in practice. To deploy dynamic spectrum auctions, we need to characterize interference conditions accurately across a large number of bidders. Using extensive network measurements, we perform the first empirical study to evaluate the usability of conflict graphs, the most widely used interference model. We study key issues long considered by the wireless community to be serious limitations of this model, and propose techniques to overcome them. Our results validate the usability of conflict graphs for dynamic spectrum auctions as well as general wireless networks. Furthermore, our work offers an efficient online solution to construct conflict graphs on the fly, thus enabling real-time dynamic spectrum auctions across a large number of users.

In summary, our work demonstrates that to support massive and highly dynamic traffic in the big-data age, network design must embed flexibility as a key element. Our research has tackled important problems in data centers and wireless networks, and the same design principle is applicable to a wide range of networking systems.