Machine Learning Applications for Atmospheric Retrieval from Earth Observing Spectroscopy with Robust Uncertainty Quantification

Abstract

Satellite estimates of column averaged CO2 (XCO2) undergo two essential post-processing steps. Bias correction suppresses systematic error introduced during retrieval, and quality filtering removes soundings whose residual error is likely unacceptable for science use. NASA's Orbiting Carbon Observatory-2 currently relies on a linear regression bias model and a rule-based filter; both requiring extensive manual tuning and degrade when biases vary non-linearly. Planned missions (e.g. ESA's CO2M) will deliver far larger data volumes, and the computational cost of physics-based optimal estimation retrieval will become a bottleneck. Automated, scalable methods with explicit uncertainty estimates are needed to keep pace. This dissertation presents novel machine-learning (ML) applications for bias correction and quality filtering of OCO-2 XCO2 estimates. The models output point corrections and predictive uncertainties, handling non-linear error structure better than the existing linear scheme. Quality filtering is framed as a multi-objective search guided by domain scientist expertise, balancing error reduction, spatial–temporal coverage, and extrapolation risk. Bayesian optimization explores uncertainty thresholds, spatial weightings, and model settings, while explainable ML diagnostics keep the procedure transparent. The resulting ternary quality flag provides tailored data streams for example, a high precision subset for carbon flux inversion modeling and broader coverage for plume studies. Complementing these post-processing advances, a conditional diffusion model trained on simulated spectra from a Radiative Transfer Model, retrieves XCO2 directly from top-of-atmosphere radiances. The approach offers competitive accuracy, supplies full predictive uncertainty, and meets the speed requirements of next-generation greenhouse gas missions. In addition to the speedup in retrieval computation time, the diffusion model applied to potentially ill-posed inverse problem can recover non-Gaussian predictive posteriors which can speedup the analysis of model discrepancy between the physics-based forward model and Nature's forward model. A novel Evidential Transformer architecture is also presented for the retrieval of methane from NASA's EMIT instrument. Through these applications to Remote Sensing this dissertation also makes several novel contributions to the field of Data Science & Analytics. 1.) Post-hoc uncertainty calibration of Bayesian Optimal Estimation retrievals. 2.) Spatially weighting of Pseudo-labels and loss for Machine Learning methods applied to Remote Sensing problems. 3.) A mixture of experts Bayesian multi-objective optimization framework that incorporates Domain Expert input (Human-in-the-Loop) for fast development of science enabling quality filters. 4.) Diffusion for fast posterior sampling of ill-posed inverse problems to enable efficient model discrepancy analysis. 5.) Injecting physics information into a conditional diffusion architecture. 6.) A novel probabilistic Transformer for the unique modality of spectroscopic data with robust uncertainty quantification.