text: 479,198,1488950496727429121,882511862524465152,Aleksander Madry,"Can we cast ML predictions as simple functions of individual training inputs? Yes! w/ @andrew_ilyas @smsampark @logan_engstrom @gpoleclerc, we introduce datamodels (<LINK>), a framework to study how data + algs -&gt; predictions. Blog: <LINK> (1/6) <LINK> We trained *hundreds of thousands* of models on random subsets of computer vision datasets using our library FFCV (<LINK>). We then used this data to fit *linear* models that can successfully predict model outputs. (2/6) <LINK> We then use datamodels to: (1) Predict data counterfactuals (i.e., what if I remove subset R from the train set?) and find that you can flip model predictions for *over 50%* of test examples on CIFAR-10 by removing only 200 (target-specific) training images (0.4% of total) (3/6) <LINK> (2) Identify similar training examples to a given test example and use this to find (a non-trivial amount of) train-test leakage in both CIFAR-10 and FMoW datasets. [Some (of many) examples images from the same scene are below.] (4/6) <LINK> (3) Use datamodels as *feature representation* and employ these embeddings to perform clustering that pinpoints *model-driven* data subpopulations (see a PCA-driven example below) and unlock a suite of graph algorithms for analyzing complex datasets. (5/6) <LINK> Overall, datamodels are a versatile tool for *model-driven* understanding of the data. The fact that a simple linear model can predict outputs of end-to-end model training suggests there is much to be learned about generalization of DNNs here too. (See Sec. 7 of the paper.) (6/6)",http://arxiv.org/abs/2202.00622,"We present a conceptual framework, datamodeling, for analyzing the behavior of a model class in terms of the training data. For any fixed ""target"" example $x$, training set $S$, and learning algorithm, a datamodel is a parameterized function $2^S \to \mathbb{R}$ that for any subset of $S' \subset S$ -- using only information about which examples of $S$ are contained in $S'$ -- predicts the outcome of training a model on $S'$ and evaluating on $x$. Despite the potential complexity of the underlying process being approximated (e.g., end-to-end training and evaluation of deep neural networks), we show that even simple linear datamodels can successfully predict model outputs. We then demonstrate that datamodels give rise to a variety of applications, such as: accurately predicting the effect of dataset counterfactuals; identifying brittle predictions; finding semantically similar examples; quantifying train-test leakage; and embedding data into a well-behaved and feature-rich representation space. Data for this paper (including pre-computed datamodels as well as raw predictions from four million trained deep neural networks) is available at this https URL . ",Datamodels: Predicting Predictions from Training Data,6,"['Can we cast ML predictions as simple functions of individual training inputs? Yes! w/ @andrew_ilyas @smsampark @logan_engstrom @gpoleclerc, we introduce datamodels (<LINK>), a framework to study how data + algs -&gt; predictions.  Blog: <LINK> (1/6) <LINK>', 'We trained *hundreds of thousands* of models on random subsets of computer vision datasets using our library FFCV (https://t.co/QWUdL50g9i). We then used this data to fit *linear* models that can successfully predict model outputs. (2/6) https://t.co/ITedgWiCQi', 'We then use datamodels to: (1) Predict data counterfactuals (i.e., what if I remove subset R from the train set?) and find that you can flip model predictions for *over 50%* of test examples on CIFAR-10 by removing only 200 (target-specific) training images (0.4% of total)  (3/6) https://t.co/SbkaAGJffl', '(2) Identify similar training examples to a given test example and use this to find (a non-trivial amount of) train-test leakage in both CIFAR-10 and FMoW datasets. [Some (of many) examples images from the same scene are below.] (4/6) https://t.co/919Ls0xlyJ', '(3) Use datamodels as *feature representation* and employ these embeddings to perform clustering that pinpoints *model-driven* data subpopulations (see a PCA-driven example below) and unlock a suite of graph algorithms for analyzing complex datasets. (5/6) https://t.co/Grbfy3ghc5', 'Overall, datamodels are a versatile tool for *model-driven* understanding of the data. The fact that a simple linear model can predict outputs of end-to-end model training suggests there is much to be learned about generalization of DNNs here too. (See Sec. 7 of the paper.) (6/6)']",22,02,1557