text:   Limit at Initialization",6,"['There has been a flurry of work beyond the infinite-width limit. We study the infinite DEPTH-AND-WIDTH limit of ReLU nets with residual connections and see remarkable (!) agreement with STANDARD finite networks. Joint work w/ @MihaiCNica @roydanroy <LINK> <LINK>', 'How is the infinite depth-and-width limit different? In short, each layer of width (n) carries an error term of size O(1/n), and increasing depth (d) compounds the error exponentially. At the heart of the analysis is the following ""dichotomy"": https://t.co/YtVj4Gfw0Z', 'As a result, the infinite depth-and-width limit is not Gaussian. This work extends results for fully connected MLPs where the analysis is much simpler. See @MihaiCNica’s youtube video for an introduction. https://t.co/6u0z32SwW0', 'However, networks with skip connections introduce correlations between layers, which complicates the analysis. Surprising observation: with residual connections, the population of neurons is HYPOactivated, i.e., fewer than half of the ReLU units are active. https://t.co/gTGZf2NCWO', 'Our main result is a precise description of the distribution of the output of the network in the infinite depth-and-width limit. One key observation: the magnitude contains a log-Gaussian factor. The exact constants and Gaussian parameters can be found in the paper. https://t.co/gTlmcLc8wF', 'We believe this result can be extended to non-Gaussian weights. See an earlier universality result for MLPs by @BorisHanin and @MihaiCNica https://t.co/HRbunQ7og3']",21,06,1410