Understanding and controlling carrier transport in conjugated polymer films and composites is critical to the development and application of plastic solar cells. Recent efforts have focused on "bulk heterojunction" structures where a conjugated polymer donor is mixed at the nanoscale with a fullerene acceptor to achieve large interfacial areas for exciton splitting. In these systems, fabrication protocols dramatically affect device efficiency and charge transport is intimately tied to film morphology through local order, domain formation, and compositional heterogeneity. We employ both far-field and confocal/near-field optical spectroscopy (absorbance, low-temperature photoluminescence, Raman) to study chain order (aggregation, pi-stacking), photo-oxidation, and local morphology in conjugated polymer (PPV and polythiophene) -- fullerene (PCBM) blends. Through quantitative analysis of exciton bandwidths, emission intensity, and vibronic lineshapes, we demonstrate that competition exists between the chemical "disordering" effect of photo-degradation and the physical "ordering" effect of aggregation, each of which dominate under different processing conditions. Large changes in photoluminescence and Raman show that PCBM begins to significantly hinder intra-chain planarization and inter-chain pi-overlap at a critical PCBM weight fraction. Furthermore, the critical weight fraction is a function of the polymer regiochemistry, occurring at lower PCBM weight fractions for a more regio-random polymer. Mild thermal annealing of blended films was seen to restore order, which results from PCBM phase segregation (lower dispersion) and growth of polymer aggregates. Spatially resolved spectral analysis of photoluminescence was also used to map fullerene diffusion and agglomeration as well as detect local changes in interfacial contact between donor and acceptor domains due to thermal annealing.