Quantifying Covariate and Concept Shifts with Optimal Transport
This paper addresses the fundamental challenge of generalization in machine learning when the data distribution changes between training and deployment. It introduces a novel theoretical framework using optimal transport to quantify different types of distribution shifts and provides a method to estimate learning error bounds.
Reading in Plain English mode — toggle below to switch
This paper addresses the fundamental challenge of generalization in machine learning when the data distribution changes between training and deployment. It introduces a novel theoretical framework using optimal transport to quantify different types of distribution shifts and provides a method to estimate learning error bounds.
Imagine you've trained a self-driving car in sunny California, but now you need it to operate in snowy Alaska. The 'landscape' (data distribution) has shifted. This paper provides a mathematical compass to measure how much the landscape has changed and predict how well your car will perform in the new environment, even if some roads are entirely new.
How Distribution Shift Quantification Works
- 11. Identify limitations of existing concept shift definitions, especially when source and target data supports don't overlap.
- 22. Introduce a new concept, $γ^{*}\!$-concept shifts, leveraging entropic optimal transport to provide a more robust definition.
- 33. Derive a general error bound that unifies covariate and $γ^{*}\!$-concept shifts, applicable across various loss functions and label spaces.
- 44. Develop estimators for these shifts with strong theoretical guarantees (concentration guarantees).
- 55. Propose the DataShifts algorithm to practically quantify distribution shifts and estimate the error bound in real-world applications.
γ*
A novel measure of concept shift that uses entropic optimal transport to account for differences in data distributions, even when data supports do not perfectly overlap.
Consider a medical diagnostic AI trained on data from one hospital. When deployed in another hospital, patient demographics (covariate shift) or disease presentation (concept shift) might differ. This framework allows quantifying these shifts and predicting the diagnostic accuracy in the new environment, providing a rigorous tool for assessing model reliability.
The paper bridges the gap between theoretical distribution shift analysis and practical application by introducing $γ^{*}\!$-concept shifts and a unified error bound, offering a robust method to quantify and predict model performance under varying data distributions.