text:   Interventions",6,"['👋👋New ACL paper: Are VQA Systems RAD? Measuring Robustness to Augmented Data with Focused Interventions 👋👋\n\nWith: Daniel Rosenberg, @Itai_Gat, @roireichart\n\nAsks whether VQA systems understand language. \n<LINK>\n1/6', 'VQA systems seem to be constantly improving, with many different models achieving similar state-of-the-art performance. However, a more careful look reveals that they often do not understand the rich signal they are being fed with.\n\n2/6', 'To understand and measure their generalization capabilities, we look at their robustness to counterfactual augmentations.\n\nThese augmentations are designed to make a focused intervention on a specific property of the question such that the answer changes.\n\n3/6', 'Using the augmentations, we introduce a new robustness measure, Robustness to Augmented\nData (RAD), which measures the consistency\nof model predictions between original and counterfactual examples.\n\n4/6', 'We show that RAD, unlike classical accuracy measures, can quantify when state-of-the-art systems are not robust to counterfactuals. \n\nWe find substantial failure cases which reveal that current VQA systems are still brittle. \n\n5/6', 'We also connect between robustness and out-of-distribution generalization, demonstrating the predictive power of RAD for performance on unseen augmentations.\n\n6/6']",21,06,1299