text: 272,185,1336544400856543236,99270209,Guillermo Valle,"I’m super excited to release this!  What do we want from a generalization theory of deep learning? We propose 7 desiderata (Ds), review how existing bounds do at them, and show that a marginal-likelihood PAC-Bayes bound does better at most Ds <LINK> The desiderata are: The predictions should scale correctly when changing 1 data complexity 2 training set size 3 architecture 4 optimizer The theory should also be 5 non-vacuous 6 efficiently computable 7 rigorous The more of these you can satisfy the better. Which ones matter more depends on the application To help us study the vast literature on generalization bounds, we classify existing types of bounds according to how many assumptions they make on the data or algorithm (This+Section 4 should also be useful as a tutorial/review the field, we hope) <LINK> The main conclusion from the review is that we want bounds which are data and algorithm dependent. We prove one such bound, which is a high-probability version of a previous PAC-Bayes bound, and is basically proportional to the marginal likelihood <LINK> The bound does well at predicting what the error does when changing data complexity <LINK> It also predicts learning curves (m is training set size) remarkably well, accross datasets and architectures! <LINK> Here we see that the exponents predicted from the bound in two different ways correlate well with the empirical learning curve exponents. <LINK> The bound is also able to capture some of the variation in generalization from changing the architecture (among several SOTA computer vision architectures), across datasets <LINK> We suggest that one of the main reasons our bound works better than previous ones may be because it works in function space which is able to capture the (statistical) properties of the behavior of neural nets, more easily than parameter space-based bounds. Finally, we discuss potential applications of the bound, including to neural architecture search (NAS). Perhaps combining the use of NNGP for NAS <LINK>, with ideas of using learning curves for NAS. btw @roydanroy this is the paper i've been mentioning to you several times, which has been ""cooming soon"" for quite a while @_vaishnavh I also talked about this paper with u in an email earlier this year @jaschasd @TheGregYang may also be interested as we use NNGPs extensively^^",https://arxiv.org/abs/2012.04115,"Generalization in deep learning has been the topic of much recent theoretical and empirical research. Here we introduce desiderata for techniques that predict generalization errors for deep learning models in supervised learning. Such predictions should 1) scale correctly with data complexity; 2) scale correctly with training set size; 3) capture differences between architectures; 4) capture differences between optimization algorithms; 5) be quantitatively not too far from the true error (in particular, be non-vacuous); 6) be efficiently computable; and 7) be rigorous. We focus on generalization error upper bounds, and introduce a categorisation of bounds depending on assumptions on the algorithm and data. We review a wide range of existing approaches, from classical VC dimension to recent PAC-Bayesian bounds, commenting on how well they perform against the desiderata. We next use a function-based picture to derive a marginal-likelihood PAC-Bayesian bound. This bound is, by one definition, optimal up to a multiplicative constant in the asymptotic limit of large training sets, as long as the learning curve follows a power law, which is typically found in practice for deep learning problems. Extensive empirical analysis demonstrates that our marginal-likelihood PAC-Bayes bound fulfills desiderata 1-3 and 5. The results for 6 and 7 are promising, but not yet fully conclusive, while only desideratum 4 is currently beyond the scope of our bound. Finally, we comment on why this function-based bound performs significantly better than current parameter-based PAC-Bayes bounds. ",Generalization bounds for deep learning,12,"['I’m super excited to release this! \n\nWhat do we want from a generalization theory of deep learning?\n\nWe propose 7 desiderata (Ds),\n\nreview how existing bounds do at them,\n\nand show that a marginal-likelihood PAC-Bayes bound does better at most Ds\n\n<LINK>', 'The desiderata are:\n\nThe predictions should scale correctly when changing\n1 data complexity\n2 training set size\n3 architecture\n4 optimizer\n\nThe theory should also be\n5 non-vacuous\n6 efficiently computable\n7 rigorous', 'The more of these you can satisfy the better. Which ones matter more depends on the application', 'To help us study the vast literature on generalization bounds, we classify existing types of bounds according to how many assumptions they make on the data or algorithm\n\n(This+Section 4 should also be useful as a tutorial/review the field, we hope) https://t.co/RMoWkh3LOp', 'The main conclusion from the review is that we want bounds which are data and algorithm dependent.\n\nWe prove one such bound, which is a high-probability version of a previous PAC-Bayes bound, and is basically proportional to the marginal likelihood https://t.co/HIswoYy92h', 'The bound does well at predicting what the error does when changing data complexity https://t.co/bM2Tz0KgW6', 'It also predicts learning curves (m is training set size) remarkably well, accross datasets and architectures! https://t.co/14YHZxvxRK', 'Here we see that the exponents predicted from the bound in two different ways correlate well with the empirical learning curve exponents. https://t.co/euEeFfnAFg', 'The bound is also able to capture some of the variation in generalization from changing the architecture (among several SOTA computer vision architectures), across datasets https://t.co/BQkXZroXJV', 'We suggest that one of the main reasons our bound works better than previous ones may be because it works in function space which is able to capture the (statistical) properties of the behavior of neural nets, more easily than parameter space-based bounds.', 'Finally, we discuss potential applications of the bound, including to neural architecture search (NAS). Perhaps combining the use of NNGP for NAS https://t.co/U3Fynzrs3E, with ideas of using learning curves for NAS.', 'btw @roydanroy this is the paper i\'ve been mentioning to you several times, which has been ""cooming soon"" for quite a while \n@_vaishnavh I also talked about this paper with u in an email earlier this year\n@jaschasd @TheGregYang may also be interested as we use NNGPs extensively^^']",20,12,2339