The focus of the dissertation is on estimation-theoretic approaches in video coding and networking, and in particular on transform and predictive coding, scalable coding and error resilient communication. One main line of research is concerned with the interactions between predictive and transform coding modules in video compression, and means to exploit such inter-dependencies to achieve joint optimality and thereby substantial performance gains. Given the unitary transforms employed in conventional motion-compensated video coding the underlying process can be equivalently modeled as a set of parallel, largely uncorrelated auto-regressive processes in the appropriate transform domain. This point of view eliminates the spatial correlation from temporal predictive coding, and hence opens the door to fully account for and exploit the effects of quantization in the motion compensated prediction loop.

The dissertation work builds on this paradigm and demonstrates how several advanced techniques are devised by incorporating information that emerges only in the transform domain within an estimation-theoretic framework, including optimal delayed decoding and transform domain motion compensation. This methodology is further leveraged to impact the problem of scalable video coding, where the enhancement layer has access to very different types of information from the current base layer reconstruction and prior enhancement layer frames, which makes optimal prediction and hence optimal coding a significant challenge. An estimation-theoretic (ET) approach for optimal scalability developed earlier at the Signal Compression Lab showed the feasibility of optimal exploitation of the various sources of information.

Taking this ET prediction framework as the initial foundation, the thesis work makes several major advances: (1) It extends the method to enable delayed enhancement layer prediction that incorporates information from future base layer reconstructed frame, often available in typical settings of scalable coding over networks; (2) It derives a solution to the longstanding challenge of applying ET prediction to spatial scalability; (3) The underlying probability model that is conditioned on all the available information, is extended to obtain optimal solutions involving other modules including quantization, entropy coding, and more.

The overall methodology thus departs from conventional video coding schemes that treat prediction, transformation, quantization, and entropy coding, as largely separate sequential functional components, and instead offers the effective means for joint optimization of entropy-constrained quantization and arithmetic coding, while fully accounting for hitherto ignored relevant factors, inherent to predictive scalable coding, including information from base layer operation and from enhancement layer motion compensated reference. Experiments demonstrate major coding performance gains over existing standard competitors as well as over the earlier ET prediction approach.

Practical deployment of video codecs often requires careful consideration of the impact of the potential channel loss during transmission over packet-based networks, as well as the interaction with subsequent error concealment. Errors due to packet losses propagate via the prediction loop, and can significantly affect the reconstruction quality. A major strategy to achieve error resilience is thus to judiciously select the prediction mode, and other encoding parameters so that the end-to-end distortion (EED) versus rate tradeoff is optimized. Central to this approach is the ability to accurately estimate at the encoder the EED which measures the distortion perceived at the decoder. The recursive optimal per-pixel estimate (ROPE) is an established EED estimation method that recursively calculates the first and second moments of reconstructed pixels via update recursions that explicitly account for the prediction, concealment, channel uncertainties, etc.

The applicability of ROPE, however, is inherently limited to account for operations that are recursive in the pixel domain. While this is sufficient for standard video coding schemes, the advanced compression techniques mentioned above effectively operate in the transform domain, which motivates the development of a ROPE-like estimation method that performs its update recursions entirely in the transform domain, and is naturally capable of capturing such transform domain operations, namely, the spectral coefficient-wise optimal recursive estimate (SCORE). It recursively computes up to second moments of decoder reconstructed transform coefficients in rough analogy to what ROPE does per-pixel. The scope of basic SCORE was then extended to encompass ET operations in the setting of scalable video coding.

The unified ET framework that incorporates SCORE to optimize coding decisions effectively achieves optimality in both prediction and EED distortion, and considerably outperforms standard codecs whose coding decisions are optimized via ROPE.