It is an obvious trend in our daily life that the world is more and more digitized and connected. It took the human race thousands of years to build the physical world that we are in today, but we have now created a ``virtual'' world in less than half a century. It is made possible because of the deepest understanding in physics and information. We can store all books that were written in human history in just several hard drives. But even then, the digital world has grown so large that it is now starting to show an impact on the physical world, reflected most noticeably by the physical size and power consumption of modern datacenters. It requires information technology to be even more efficient in every way.

Datacenters, where centralized computing is performed by thousands of servers, require enormous bandwidth for internal networking as well as input/output. In order to increase capacity per optical wavelength, coherent fiber-optic communications with advanced modulation formats has being developed to increase the spectral efficiency (SE). It is made possible by advances in electronics especially digital signal processing (DSP). High order quadrature amplitude modulation (QAM) formats which are commonly seen in radio frequency (RF) communication systems, are now at the research frontier for fiber-optic communications. Large constellation such as 256-QAM could provide 8-fold capacity increase over conventional systems with on-off keying (OOK) modulation format at the same modulation speed. However, high order QAM signals require higher signal-to-noise ratio (SNR) and are more vulnerable to fiber impairments such as nonlinearities.

In this dissertation we thoroughly studied the principle, implementation and qualification of high order QAM formats in fiber-optic communications, and we demonstrated a dual-channel 96~Gb/s 256-QAM signal transmission over 50~km standard single-mode fiber (SSMF). The link BER was 4 x 10 --4 which is below the forward error correction (FEC) threshold. We studied fiber nonlinear effects such as self-phase modulation (SPM) and cross-phase modulation (XPM) on high order QAM signals. We experimentally determined the nonlinear power thresholds for various conditions such as different transmission distances. Meanwhile, we also explored incorporating advanced modulation formats with ultra-fast optical time-division multiplexing (OTDM) systems. We demonstrated a 64-QAM OTDM experiment and discussed the technical challenges. These results are valuable to our understanding of the system limitations and will help guiding future high SE system design. This technology could enable ultra-high speed optical Ethernet beyond Terabit/s for future datacenter networks.