The continued emergence of inexpensive sensors and storage has made the collection and processing of large quantities of visual data practical, opening up new possibilities in data exploitation and understanding. The volume of data also makes it increasingly difficult to rely solely on humans for review, requiring assistance from automated systems to use large data sources to their full potential. However, while large data has also enabled new algorithmic techniques, computer performance still lags behind that of humans. The work in this thesis addresses both sides of this problem by exploring both how automated systems can make the most of large data and how they can be refined to act more human when doing so. I will discuss video summarization as applied to a network of 11 cameras and show how our system makes the network data more accessible to human operators while also using human feedback to guide its design. A novel approach to object tracking that uses large-scale human annotation to implicitly apply human scene understanding in an automated system will also be discussed. Finally, I will present recent work in using functional magnetic resonance imaging (fMRI) to explore how quantitative human feedback can be directly collected from a subject and applied to debugging traditional computer vision algorithms to bring them closer to human capabilities.