text: 130,41,1194712172183474176,24859650,Jan-Willem van de Meent,"1/ New work by Alican (@alicanb_) and Babak (@BabakEsmaeili10): ""Evaluating Combinatorial Generalization in Variational Autoencoders"" (<LINK>) In this paper we ask the question: ""To what extent do VAEs generalize to unseen combinations of features?""(thread) <LINK> 2/ In any dataset that is characterized by even a moderate number of factors of variation, we cannot hope to have representatives of all combinations of latent features in our training data, since the size of a ""complete"" dataset grows exponentially with the number of features. 3/ In this paper, we test what happens when test-time examples differ substantially from training examples with respect to a least one feature. We are particularly interested in the effect of model capacity on generalization. 4/ Alican and Babak trained over 3000 VAE instances to systematically evaluate the effect of network width & depth, KL regularization, the amount of training data, and the density of the training set in feature space. We were surprised by the results! <LINK> 5/ For easy generalization problems (i.e. when test set examples are similar in pixel space to training examples), increasing model capacity always improves generalization. In this regime, memorizing training data does not hurt generalization performance. <LINK> 6/ For more difficult problems we observe a range of behaviors. In this regime, increasing the layer depth can lead to data memorization, which adversely affects generalization when the test data are not similar to the training data. However this is not always the case. 7/ The level of KL regularization also plays a crucial role, as evidenced by rate-distortion analysis. 1-layer networks show a U-shaped curve as a function of the level of regularization. In 3-layer networks, decreasing the regularization paradoxically improves generalization. <LINK> 8/ Our results suggest that the generalization characteristics of VAEs are by no means straightforward. Network depth, KL regularization, and the difficulty of the problem can give rise of a range of possible behaviors, and need to be considered when evaluating generalization. 9/ In this paper, we restrict ourselves to fully-connected encoders and decoders in this work. As part of this paper, we will be releasing our Tetraminoes dataset, along with source code for all experiments. We would be excited to hear about results for different architectures! <LINK>",https://arxiv.org/abs/1911.04594,"Variational autoencoders optimize an objective that combines a reconstruction loss (the distortion) and a KL term (the rate). The rate is an upper bound on the mutual information, which is often interpreted as a regularizer that controls the degree of compression. We here examine whether inclusion of the rate also acts as an inductive bias that improves generalization. We perform rate-distortion analyses that control the strength of the rate term, the network capacity, and the difficulty of the generalization problem. Decreasing the strength of the rate paradoxically improves generalization in most settings, and reducing the mutual information typically leads to underfitting. Moreover, we show that generalization continues to improve even after the mutual information saturates, indicating that the gap on the bound (i.e. the KL divergence relative to the inference marginal) affects generalization. This suggests that the standard Gaussian prior is not an inductive bias that typically aids generalization, prompting work to understand what choices of priors improve generalization in VAEs. ",Rate-Regularization and Generalization in VAEs,9,"['1/ New work by Alican (@alicanb_) and Babak (@BabakEsmaeili10): ""Evaluating Combinatorial Generalization in Variational Autoencoders"" (<LINK>)\n\nIn this paper we ask the question:  ""To what extent do VAEs generalize to unseen combinations of features?""(thread) <LINK>', '2/ In any dataset that is characterized by even a moderate number of factors of variation, we cannot hope to have representatives of all combinations of latent features in our training data, since the size of a ""complete"" dataset grows exponentially with the number of features.', '3/ In this paper, we test what happens when test-time examples differ substantially from training examples with respect to a least one feature. We are particularly interested in the effect of model capacity on generalization.', '4/ Alican and Babak trained over 3000 VAE instances to systematically evaluate the effect of network width &amp; depth, KL regularization, the amount of training data, and the density of the training set in feature space. We were surprised by the results! https://t.co/qGEOmKEc9V', '5/ For easy generalization problems (i.e. when test set examples are similar in pixel space to training examples), increasing model capacity always improves generalization. In this regime, memorizing training data does not hurt generalization performance. https://t.co/FipiYRorev', '6/ For more difficult problems we observe a range of behaviors. In this regime, increasing the layer depth can lead to data memorization, which adversely affects generalization when the test data are not similar to the training data.  However this is not always the case.', '7/ The level of KL regularization also plays a crucial role, as evidenced by rate-distortion analysis. 1-layer networks show a U-shaped curve as a function of the level of regularization. In 3-layer networks, decreasing the regularization paradoxically improves generalization. https://t.co/Q92RNMVx9U', '8/ Our results suggest that the generalization characteristics of VAEs are by no means straightforward. Network depth, KL regularization, and the difficulty of the problem can give rise of a range of possible behaviors, and need to be considered when evaluating generalization.', '9/ In this paper, we restrict ourselves to fully-connected encoders and decoders in this work. As part of this paper, we will be releasing our Tetraminoes dataset, along with source code for all experiments. We would be excited to hear about results for different architectures! https://t.co/RaflagGjfB']",19,11,2411