text: 38,126,1488938966485250050,226366834,Hattie Zhou,"[New paper 🚨] You’ve heard of catastrophic forgetting, but have you heard of *fortuitous* forgetting?  In this paper, we show how forgetting can be a friend of learning in artificial neural nets. With @AnkitKVani, @hugo_larochelle, @AaronCourville (1/6) <LINK> We introduce “forget-and-relearn” as a powerful paradigm for steering the learning trajectory of NNs.  In this process, the forgetting step removes undesirable information and the relearning step amplifies features that are useful under different learning conditions. (2/6) We show that many iterative training algos (e.g. LT pruning, knowledge evolution, iterated learning) can be seen as instances of forget-and-relearn, and that their success is proportional to how well the forgetting operations can selectively remove undesirable information (3/6) Thus, by designing more targeted forgetting methods, we can significantly improve the performance of these algorithms. Our analyses also illustrate how the relearning stage amplifies features that are consistently useful, and how this corresponds to better generalization. (4/6) Forget-and-relearn suggests a shift in thinking from “learning desirable information” to “forgetting undesirable information”. It’s often easier to suppress unwanted behavior than to delineate good behavior, and we hope this work will inspire new forgetting-based algorithms (5/6) Lastly, I’m really grateful for my collaborators who made this project super fun to work on, as well as for the friends who offered valuable feedback throughout this work ❤️ Excited to present this at #ICLR2022 🥳 Feel free to reach out if you want to discuss these ideas! (6/6)",http://arxiv.org/abs/2202.00155,"Forgetting is often seen as an unwanted characteristic in both human and machine learning. However, we propose that forgetting can in fact be favorable to learning. We introduce ""forget-and-relearn"" as a powerful paradigm for shaping the learning trajectories of artificial neural networks. In this process, the forgetting step selectively removes undesirable information from the model, and the relearning step reinforces features that are consistently useful under different conditions. The forget-and-relearn framework unifies many existing iterative training algorithms in the image classification and language emergence literature, and allows us to understand the success of these algorithms in terms of the disproportionate forgetting of undesirable information. We leverage this understanding to improve upon existing algorithms by designing more targeted forgetting operations. Insights from our analysis provide a coherent view on the dynamics of iterative training in neural networks and offer a clear path towards performance improvements. ",Fortuitous Forgetting in Connectionist Networks,6,"['[New paper 🚨] You’ve heard of catastrophic forgetting, but have you heard of *fortuitous* forgetting? \n\nIn this paper, we show how forgetting can be a friend of learning in artificial neural nets. With @AnkitKVani, @hugo_larochelle, @AaronCourville (1/6)\n\n<LINK>', 'We introduce “forget-and-relearn” as a powerful paradigm for steering the learning trajectory of NNs. \n\nIn this process, the forgetting step removes undesirable information and the relearning step amplifies features that are useful under different learning conditions. (2/6)', 'We show that many iterative training algos (e.g. LT pruning, knowledge evolution, iterated learning) can be seen as instances of forget-and-relearn, and that their success is proportional to how well the forgetting operations can selectively remove undesirable information\n(3/6)', 'Thus, by designing more targeted forgetting methods, we can significantly improve the performance of these algorithms.\n\nOur analyses also illustrate how the relearning stage amplifies features that are consistently useful, and how this corresponds to better generalization. (4/6)', 'Forget-and-relearn suggests a shift in thinking from “learning desirable information” to “forgetting undesirable information”. It’s often easier to suppress unwanted behavior than to delineate good behavior, and we hope this work will inspire new forgetting-based algorithms (5/6)', 'Lastly, I’m really grateful for my collaborators who made this project super fun to work on, as well as for the friends who offered valuable feedback throughout this work ❤️\n\nExcited to present this at #ICLR2022 🥳\n\nFeel free to reach out if you want to discuss these ideas! (6/6)']",22,02,1651