text: 44,240,1434894801086910473,1014454910346285057,Daniel Coelho de Castro,"Our new study demonstrates noisy labels impacts training, eval & selection of ML models. Reannotating large datasets requires lots of expert time (eg in healthcare), so we also show data-driven strategies enable fixing more labels using fewer resources: <LINK> Great collab with Melanie Bernhardt, @RyutaroT92, Anton Schwaighofer, Kerem Tezcan, @mabmonteiro, Shruthi Bannur, @mattlungrenMD, Aditya Nori, @GlockerBen, @alvarezvalle, @ozanoktay__! @MSFTResearch That should be @mel_bernhardt! Somehow Twitter struggled to find you 😀",https://arxiv.org/abs/2109.00574,"Imperfections in data annotation, known as label noise, are detrimental to the training of machine learning models and have an often-overlooked confounding effect on the assessment of model performance. Nevertheless, employing experts to remove label noise by fully re-annotating large datasets is infeasible in resource-constrained settings, such as healthcare. This work advocates for a data-driven approach to prioritising samples for re-annotation - which we term ""active label cleaning"". We propose to rank instances according to estimated label correctness and labelling difficulty of each sample, and introduce a simulation framework to evaluate relabelling efficacy. Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection. Crucially, the proposed active label cleaning enables correcting labels up to 4 times more effectively than typical random selection in realistic conditions, making better use of experts' valuable time for improving dataset quality. ","Active label cleaning for improved dataset quality under resource