Visual and multimodal sensors are playing an increasingly important role in applications such as object recognition, activity analysis, agriculture monitoring, autonomous navigation and assisted healthcare. Typical features computed from the raw data/ measurements have significant mutual information that can be exploited towards improving the overall pattern recognition performance. In this dissertation we introduce principled methods to aggregate such information using conditional random field (CRF) models. Our aggregation is built on top of a large number of classifiers working independently on features computed from the raw data. The statistical dependencies among classifier outputs are formulated in a factorized form to characterize each class label. This technique enables the consistent prediction patterns to unfold a stronger discriminative power. In the first part of the presentation we demonstrate the effectiveness of this approach in combining information from a large number of visual features to improve the classification accuracy. Our experimental results show a significant improvement over the state-of-the-art aggregation methods on diverse image datasets. In the second application, we integrate the proposed CRF-based approach to combine multimodal measurements in a task involving a quadcopter UAV with autonomous path planning to detect and recognize the frontal view of a person. In this scenario, we utilize geographic location, time of day, view angle of the camera, altitude and magnetic heading, in addition to the video data, to estimate the body orientation in real time and automatically maneuver the UAV to maintain the frontal view of a person. We demonstrate that the contextual information implicit in the multimodal measurements can significantly reduce the vision-based computations for such real-time tasks.