text: 284,3,1402382668910796810,20593623,Maggie Makar,"[1/n] New arXiv paper alert! ML models can take shortcuts, which makes them perform poorly under distribution shift. We used some ideas from causality to get models to use the right signal. W @davisblalock, @packer_ben, Yoni Halpern, & @alexdamour. <LINK> 🧵 <LINK> [2/n] Example (based on work by @shiorisagawa, @pangweikoh, et al <LINK>): Train a ResNet50 to classify water and land birds using a training distribution where most land birds appear on land backgrounds and vice versa. <LINK> [3/n] The bird in the foreground is sufficient to classify the images perfectly, but the model learns to use the background as a shortcut. If the foreground/background correlation changes at test time, performance deteriorates. <LINK> [4/n] Addressing this problem automatically is hard. But if you have labels for the shortcut feature at training time, we show that you can enforce certain causal invariances in the model's representation, using weights and an invariance penalty (based on MMD). It works! <LINK> [5/n] Importantly, the penalty reduces the complexity of the model's function space without inducing bias (since we know the causal invariances are true). In a simple linear case, the invariance penalty limits the weights of the “shortcut” factors <LINK> [6/n] Thus, even when there is no shortcut, the method is more data efficient, achieving better performance on balanced data. <LINK> [7/n] We're very excited about this work, but it's just the beginning. Lots of opportunities to use causal formalism to import domain knowledge into ML to improve robustness, efficiency, and trustworthiness! Code coming soon! [7.5/n] work also with Dan Moldovan! #errata",https://arxiv.org/abs/2105.06422,"Shortcut learning, in which models make use of easy-to-represent but unstable associations, is a major failure mode for robust machine learning. We study a flexible, causally-motivated approach to training robust predictors by discouraging the use of specific shortcuts, focusing on a common setting where a robust predictor could achieve optimal \emph{iid} generalization in principle, but is overshadowed by a shortcut predictor in practice. Our approach uses auxiliary labels, typically available at training time, to enforce conditional independences implied by the causal graph. We show both theoretically and empirically that causally-motivated regularization schemes (a) lead to more robust estimators that generalize well under distribution shift, and (b) have better finite sample efficiency compared to usual regularization schemes, even when no shortcut is present. Our analysis highlights important theoretical properties of training techniques commonly used in the causal inference, fairness, and disentanglement literatures. Our code is available at this https URL ",Causally motivated Shortcut Removal Using Auxiliary Labels,8,"['[1/n] New arXiv paper alert! ML models can take shortcuts, which makes them perform poorly under distribution shift. We used some ideas from causality to get models to use the right signal. W @davisblalock, @packer_ben, Yoni Halpern, &amp; @alexdamour. <LINK> 🧵 <LINK>', '[2/n] Example (based on work by @shiorisagawa, @pangweikoh, et al https://t.co/DwBES7bvzO): Train a ResNet50 to classify water and land birds using a training distribution where most land birds appear on land backgrounds and vice versa. https://t.co/yoinuq4FnG', '[3/n] The bird in the foreground is sufficient to classify the images perfectly, but the model learns to use the background as a shortcut. If the foreground/background correlation changes at test time, performance deteriorates. https://t.co/ux5kRGdoA0', ""[4/n] Addressing this problem automatically is hard. But if you have labels for the shortcut feature at training time, we show that you can enforce certain causal invariances in the model's representation, using weights and an invariance penalty (based on MMD). It works! https://t.co/p3JHWQCXWq"", ""[5/n] Importantly, the penalty reduces the complexity of the model's function space without inducing bias (since we know the causal invariances are true).  In a simple linear case, the invariance penalty limits the weights of the “shortcut” factors https://t.co/jBCqg3lYbd"", '[6/n] Thus, even when there is no shortcut, the method is more data efficient, achieving better performance on balanced data. https://t.co/Q3lPPrPGD3', ""[7/n] We're very excited about this work, but it's just the beginning. Lots of opportunities to use causal formalism to import domain knowledge into ML to improve robustness, efficiency, and trustworthiness! Code coming soon!"", '[7.5/n] work also with Dan Moldovan! #errata']",21,05,1664