I like the idea of saving long-term memory in a more efficient but lower-dimensional format than short-term memory. The current <@trend@>(@Scaling Laws for Neural Language Models@) in language modelling is that more computation leads to better results, so I think that algorithms that target computation on the most relevant information are promising. I’d be interested to see (and curious if the authors tried) more continuous variants of this where older information is compressed at a higher rate than newer information, since it seems rather arbitrary to split into two FIFO queues where one has a fixed compression rate.

I’m not well calibrated on the meaning of the evaluation metrics for NLP, so I don’t have a sense of how much of an improvement this is over the *TransformerXL*. I looked through some of the example text they gave in the blog post and thought it was impressive but has clear room for improvement.

Paper: Compressive Transformers for Long-Range Sequence Modelling

A new model and dataset for long-range memory

Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Timothy P. Lillicrap

A central challenge in language modeling is capturing long-range dependencies. For example, a model needs to be able to identify the antecedent of a pronoun even if it is much earlier in the text. Existing datasets consist of news and Wikipedia articles, where articles have average lengths ranging from 27 to 3,600 words. This paper introduces a dataset of Project Gutenberg books, PG-19, where each book has a much longer average length of 69,000 words. This benchmark enables comparison of how well algorithms can make use of information that is spread out across a much larger context.

They then introduce the *Compressive Transformer*, which builds on the <@*TransformerXL*@>(@Transformer-XL: Unleashing the Potential of Attention Models@). The *TransformerXL* saves old activations into a FIFO queue, discarding them when the queue is full. The *Compressive Transformer* instead has two FIFO queues: the first stores the activations just like *TransformerXL*, but when activations are ejected, they are compressed and added to the second queue. This functions as a sort of long-term memory, storing information from a longer period of time but in a compressed format.

They try a number of types of compression function and find that it is best to use a 1D convolutional compression function with an auxiliary loss that leads to lossy compression, where information that is not attended to can be removed. The compression network and the Transformer optimize independent losses without any mixing.

They find that the *Compressive Transformer* improves on *TransformerXL* on their new PG-19 dataset and is state of the art on the already existing WikiText-103 and Enwik8 benchmarks. They also inspect where the network attends to and find that more attention is paid to the compressed memory than the oldest activations in regular memory, showing that the network is preserving some valuable information.


Because the way humans classify things are strongly influenced by our priors about how classes "should" behave, learning with limited data most likely requires some information about these priors. Semi-supervised learning that respects that data augmentation does not change the correct classification might be an efficient and scalable way to force some of these priors onto a model. Thus it seems likely that more diverse and sophisticated data augmentation could lead to further improvements in the near term. On the other hand, it seems like a lot of our priors would be very hard to capture only using automatic data augmentation, such that other methods to transfer our priors are still important.

The Quiet Semi-Supervised Revolution

Historically, semi-supervised learning that uses small amounts of labelled data combined with a lot of unlabeled data only helped when there was very little labelled data available. In this regime, both supervised and semi-supervised learning were too inaccurate to be useful. Furthermore, approaches like using a representation learnt by an autoencoder for classification empirically limited asymptotic performance. This is strange because using more data should not lead to worse performance.

Recent trends suggest that this might change soon: semi-supervised systems have begun to outperform supervised systems by larger and larger margins in the low data regime and their advantage now extends into regimes with more and more data. An important driver of this trend is the idea of using data augmentation for more consistent self-labelling.

Better semi-supervised learning might for example be useful for  federated learning which attempts to respect privacy by learning locally on (labelled) user data and sending the models trained by different users to be combined in a central server. One problem with this approach is that the central model might memorize some of the private models' idiosyncracies such that inference about the private labels is possible. Semi-supervised learning makes this harder by reducing the amount of influence private data has on the aggregate model.


Basically what's happening here is that at the interpolation threshold, the model is forced to memorize noise, and it has only one way of doing so, which need not generalize well. However, past the interpolation threshold, when the model is overparameterized, there are _many_ models that successfully memorize noise, and gradient descent "correctly" chooses one with minimum norm. This fits into the broader story being told in other papers that what's happening is that the data has noise and/or misspecification, and at the interpolation threshold it fits the noise in a way that doesn't generalize, and after the interpolation threshold it fits the noise in a way that does generalize. Here that's happening because gradient descent chooses the minimum norm estimator that fits the noise; perhaps something similar is happening with neural nets.

This explanation seems like it could explain double descent on model size and double descent on dataset size, but I don't see how it would explain double descent on training time. This would imply that gradient descent on neural nets first has to memorize noise in one particular way, and then further training "fixes" the weights to memorize noise in a different way that generalizes better. While I can't rule it out, this seems rather implausible to me. (Note that regularization is _not_ such an explanation, because regularization applies throughout training, and doesn't "come into effect" after the interpolation threshold.)

More Data Can Hurt for Linear Regression: Sample-wise Double Descent

This paper demonstrates the presence of double descent (in the size of the dataset) for _unregularized linear regression_. In particular, we assume that each data point x is a vector in independent samples from Normal(0, σ^2), and the output is y = βx + ε. Given a dataset of (x, y) pairs, we would like to estimate the unknown β, under the mean squared error loss, with no regularization.

In this setting, when the dimensionality d of the space (and thus number of parameters in β) is equal to the number of training points n, the training data points are linearly independent almost always / with probability 1, and so there will be exactly one β that solves the n linearly independent equalities of the form βx = y. However, such a β must also be fitting the noise variables ε, which means that it could be drastically overfitted, with very high norm. For example, imagine β = [1, 1], so that y = x1 + x2 + ε, and in our dataset x = (-1, 3) is mapped to y = 3 (i.e. an ε of +1), and x = (0, 1) is mapped to y = 0 (i.e. an ε of -1). Gradient descent will estimate that β = [-3, 0], which is going to generalize very poorly.

As we decrease the number of training points n, so that d > n, there are infinitely many settings of the d parameters of β that satisfy the n linearly independent equalities, and gradient descent naturally chooses the one with minimum norm (even without regularization). This limits how bad the test error can be. Similarly, as we increase the number of training points, so that d < n, there are too many constraints for β to satisfy, and so it ends up primarily modeling the signal rather than the noise, and so generalizing well.


I enjoyed this blog post a lot (though it took some time to digest it, since I know very little about generalization bounds). It constrains the ways in which we can try to explain the empirical generalization of neural networks, which I for one would love to understand. Hopefully future work will explore new avenues for understanding generalization, and hit upon a more fruitful line of inquiry.

Uniform convergence may be unable to explain generalization in deep learning

This post argues that existing generalization bounds cannot explain the empirical success of neural networks at generalizing to the test set.

"What?", you say if you're like me, "didn't we already know this? Generalization bounds depend on your hypothesis space being sufficiently small, but [neural nets can represent any reasonable function](https://en.wikipedia.org/wiki/Universal_approximation_theorem)? And even if you avoid that by considering the size of the neural net, we know that empirically [neural nets can learn randomly labeled data](https://arxiv.org/abs/1611.03530), which can never generalize; surely this means that you can't explain generalization without reference to some property of the dataset, which generalization bounds typically don't do?"

It turns out that the strategy has been to prove generalization bounds that depend on the _norm of the weights of the trained model_ (for some norm that depends on the specific bound), which gets around both these objections, since the resulting bounds are independent of the number of parameters, and depend on the trained model (which itself depends on the dataset). However, when these bounds are evaluated on a simple sphere-separation task, they _increase_ with the size of the training dataset, because the norms of the trained models increase.

Okay, but can we have a stronger argument than mere empirical results? Well, all of these bounds depend on a _uniform convergence bound_: a number that bounds the absolute difference between the train and test error for _any_ model in your hypothesis space. (I assume the recent generalization bounds only consider the hypothesis space "neural nets with norms at most K", or some suitable overapproximation of that, and this is how they get a not-obviously-vacuous generalization bound that depends on weight norms. However, I haven't actually read those papers.)

However, no matter what hypothesis space these bounds choose, to get a valid generalization bound the hypothesis space must contain (nearly) all of the models that would occur by training the neural net on a dataset sampled from the underlying distribution. What if we had the actual smallest such hypothesis space, which only contained the models that resulted from an actual training run? The authors show that, at least on the sphere-separation task, the uniform convergence bound is still extremely weak. Let's suppose we have a training dataset S. Our goal is now to find a model in the hypothesis space which has a high absolute difference between actual test error, and error in classifying S. (Recall that uniform convergence requires you to bound the absolute difference for _all_ models in your hypothesis class, not just the one trained on S.) The authors do so by creating an "adversarial" training dataset S' that also could have been sampled from the underlying distribution, and training a model on S'. This model empirically gets S almost completely wrong. Thus, this model has low test error, but high error in classifying S, which forces the uniform convergence bound to be very high.


I enjoyed reading this blogpost and like the idea of using winning tickets for transfer learning. I would have been quite surprised if they had found that the lottery ticket hypothesis was specific to image classification, as similar to pretraining, winning tickets seem to provide an inductive bias constraining the set of features that can be learnt during training to more useful ones. I do not think that further research into that direction will directly help with quickly training models for novel tasks unless the tickets can be identified very efficiently which seems like a harder optimization problem than just training a network by gradient descent.

Understanding the generalization of ‘lottery tickets’ in neural networks

The <@lottery ticket hypothesis@>(@The Lottery Ticket Hypothesis at Scale@) states that a randomly initialized dense or convolutional neural network contains (sparse) subnetworks, called "winning tickets", which can be trained to achieve performance similar to the trained base network while requiring a lot less compute.

The blogpost summarizes facebook AI's recent investigations of the generalization of winning tickets and the generality of the hypothesis. Because winning tickets are hard to find, we would like to reuse the ones we have found for similar tasks. To test whether this works, the authors trained classifiers, pruned and reset them to obtain winning tickets on different image datasets and then trained these on other datasets. Winning tickets derived from similar datasets relevantly outperform random subnetworks after training and ones derived from larger or more complex datasets generalize better. For example, tickets from ImageNet are consistently among the best and tickets from CIFAR-100 generalize better than those from CIFAR-10.

Experiments in natural language processing and reinforcement learning suggest that the lottery ticket hypothesis is not just a peculiarity of image classification: for example, the performance of a large transformer model could be recovered from a winning ticket with just a third of the original weights, whereas random tickets with that amount of weights performed quite a bit worse. The analysis of simple shallow neural networks in a student-teacher setting is used as a toy model: when a larger student network is trained to mimic a smaller teacher with the same amount of layers, **student specialization** happens: some of the student's neurons learn to imitate single neurons of the teacher. This can be seen to happen more often and faster if the student neuron is already close to the teacher neuron at initialization. If the student network is large enough, every teacher neuron will be imitated by some student neuron and these student neurons collectively form a winning ticket.


My impression is this is a big step forward in being able to embed logical reasoning in current deep learning techniques. From an engineering perspective, it seems extremely useful to be able to train systems that encorporate these layers end-to-end. It's worth being clear that in systems like these, a lot of generality is lost since part of the network is explicitly carved out for solving a particular problem of logical constraints-- it would be hard to use the same network to learn a different problem.

SATNet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver

Po-Wei Wang, Priya L. Donti, Bryan Wilder, Zico Kolter

Historically, deep learning architectures have struggled with problems that involve logical reasoning, since they often impose non-local constraints that gradient descent has a hard time learning. This paper presents a new technique, SATNet, which allows neural nets to solve logical reasoning problems by encoding them explicitly as MAXSAT-solving neural network layers. A MAXSAT problem provides a large set of logical constraints on an exponentially large set of options, and the goal is to find the option that satisfies as many logical constraints as possible. Since MaxSAT is NP-complete, the authors design a layer that solves a relaxation of the MaxSAT problem in its forward pass (that can be solved quickly, unlike MaxSAT), while the backward pass computes gradients as usual.

In experiment, SATNet is given bit representations of 9,000 9 x 9 Sudoku boards which it uses to learn the logical constraints of Sudoku, then presented with 1,000 test boards to solve. SATNet vastly outperforms traditional convolutional neural networks given the same training / test setup, achieving 98.3% test accuracy where the convolutional net achieves 0%. It performs similarly well on a "Visual" Sudoku problem where the trained network consists of initial layers that perform digit recognition followed by SATNet layers, achieving 63.2% accuracy where the convolutional net achieves 0.1%.


the transformer … “explained”?

This is an excellent explanation of the intuitions and ideas behind self-attention and the [Transformer](https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html) <@architecture@>(@Transformer-XL: Unleashing the Potential of Attention Models@).


While it certainly would have been exciting to have a paper presenting a unified (and empirically supported) theoretical understanding of the LTH, I respect the fact that this is such a purely empirical work, that tries to do one thing - designing and running clean, clear experiments - and does it well, without trying to construct explanations just for the sake of having them. We still have a ways to go in understanding the optimization dynamics underlying lottery tickets, but these seem like important and valuable data points on the road to that understanding.

Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask

Hattie Zhou, Janice Lan, Rosanne Liu, Jason Yosinski

This paper runs a series of experimental ablation studies to better understand the limits of the Lottery Ticket Hypothesis, and investigate variants of the initial pruning and masking procedure under which its effects are more and less pronounced. It is first and foremost a list of interesting results, without any central theory tying them together. These results include the observation that keeping pruned weights the same sign as their "lottery ticket" initialization seems more important than keeping their exact initial magnitudes, that taking a mixed strategy of zeroing pruned weights or freezing them at initialization can get better results, and that applying a learned 0/1 mask to a re-initialized network can get surprisingly high accuracy even without re-training.


Do we still need models or just more data and compute?

This is a response to <@The Bitter Lesson@>, that emphasizes the importance of data in addition to compute. It brings up a number of considerations that seem important to me, and is worth reading if you want to better understand my position on the bitter lesson.


Semantic Image Synthesis with Spatially-Adaptive Normalization

Taesung Park, Ming-Yu Liu, Ting-Chun Wang and Jun-Yan Zhu

This paper shows how to create somewhat realistic images specified by semantic segmentation maps. They accomplish this by modifying batch normalization. Batch normalization modifications can be quite powerful for image generation, even enough to [control style](https://arxiv.org/abs/1703.06868). Their modification is that normalization is a direct function of the semantic segmentation map throughout the network, so that the semantic segmentation map is readily available to each ResBlock. Visualizations produced by this method are [here](https://nvlabs.github.io/SPADE/).


Overall, I find this an interesting and well-articulated paper, and am curious to see future work that addresses some of the extrapolations and claims implied by this paper, particularly their claim, surprising relative to my intuitions, that increasing n_parameters will, maybe monotonically, reduce difficulty of training, because it simply increases the dimensionality of the solution set. I'm also not sure how to feel about their simply asserting that a solution exists when a network reaches 90% of baselines performance, since we may care about that "last mile" performance and it might also be the harder to reach.

Measuring the Intrinsic Dimension of Objective Landscapes

Chunyuan Li, Rosanne Liu, and Jason Yosinski

This paper proposes and defines a quantity called "intrinsic dimension", a geometrically-informed metric of how many degrees of freedom are actually needed to train a given model on a given dataset.  They calculate this by picking a set of random directions that span some subspace of dimension d, and taking gradient steps only along that lower-dimensional subspace. They consider the intrinsic dimension of a model and a dataset to be the smallest value d at which performance reaches 90% of a baseline, normally trained model on the dataset. The geometric intuition of this approach is that the dimensionality of parameter space can be, by definition, split into intrinsic dimension and its codimension, the dimension of the solution set. In this framing, higher solution set dimension (and lower intrinsic dimension) corresponds to proportionally more of the search space containing reasonable solution points, and therefore a situation where a learning agent will be more likely to find such a solution point. There are some interesting observations here that correspond with our intuitions about model trainability: on MNIST, intrinsic dimensionality for a CNN is lower than for a fully connected network, but if you randomize pixel locations, CNN's intrinsic dimension shoots up above FC, matching the intuition that CNNs are appropriate when their assumption of local structure holds.


OpenAI <@found@>(@How AI Training Scales@) that the best predictor of the maximum useful batch size was how noisy the gradient is. Presumably when you have noisy gradients, a larger batch size helps "average out" the noise across examples. Rereading their post, I notice that they mentioned the study I've summarized here and said that their results can help explain why there's so much variance in the transition points _across datasets_. However, I don't think it can explain the variance in transition points _across architectures_. Noisy gradients are typically a significant problem, and so it would be weird if the variance in transition points across architectures were explained by the noisiness of the gradient: that would imply that two architectures reach the same final performance even though one had the problem of noisy gradients while the other didn't. So there seems to be something left to explain here.

That said, I haven't looked in depth at the data, so the explanation could be very simple. For example, maybe the transition points don't vary much across architecture and vary much more across datasets, and the variance across architecture is small enough that its effect on performance is dwarfed by all the other things that can affect the performance of deep learning systems. Or perhaps while the noisiness of the gradient is a good predictor of the maximum batch size, it still only explains say 40% of the effect, and so variance across architectures is totally compatible with factors other than the gradient noise affecting the maximum batch size.

Measuring the Limits of Data Parallel Training for Neural Networks

Consider the relationship between the size of a single batch and the number of batches needed to reach a specific performance bound when using deep learning. If all that mattered for performance was the total number of examples that you take gradient steps on (i.e. the product of these two numbers), then you would expect a perfect inverse relationship between these two quantities, which would look like a line with negative slope on a log-log plot. In this case, we could scale batch sizes up arbitrarily far, and distribute them across as many machines as necessary, in order to reduce wall clock training time. A 2x increase in batch size with twice as many machines would lead to a 2x decrease in training time. However, as you make batch sizes really large, you face the problem of stale gradients: if you had updated on the first half of the batch and then computed gradients on the second half of the batch, the gradients for the second half would be "better", because they were computed with respect to a better set of parameters. When this effect becomes significant, you no longer get the nice linear scaling from parallelization.

This post studies the relationship empirically across a number of datasets, architectures, and optimization algorithms. They find that universally, there is initially an era of perfect linear scaling as you increase batch size, followed by a region of diminishing marginal returns that ultimately leads to an asymptote where increasing batch size doesn't help at all with reducing wall-clock training time. However, the transition points between these regimes vary wildly, suggesting that there may be low hanging fruit in the design of algorithms or architectures that explicitly aim to achieve very good scaling.


While OpenAI doesn't speculate on why this relationship exists, it seems to me that as you get larger batch sizes, you are improving the gradient by reducing noise by averaging over a larger batch. This predicts the results well: as the task gets harder and the noise in the gradients gets larger, there's more noise to get rid of by averaging over data points, and so there's more opportunity to have _even larger_ batch sizes.

How AI Training Scales

Sam McCandlish, Jared Kaplan and Dario Amodei

OpenAI has done an empirical investigation into the performance of AI systems, and found that the maximum useful batch size for a particular task is strongly influenced by the noise in the gradient. (Here, the noise in the gradient comes from the fact that we are using _stochastic_ gradient descent -- any difference in the gradients across batches counts as "noise".) They also found some preliminary results showing the more powerful ML techniques tend to have more gradient noise, and even a single model tends to have increased gradient noise over time as they get better at the task.


Getting neural networks to generalise to longer versions of training problems is often surprisingly difficult, so I'm impressed by the Box-World results; I would have liked to see what happened on even longer problems.

Relational Deep Reinforcement Learning

Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascanu, Matthew Botvinick, Oriol Vinyals, Peter Battaglia

This paper uses the self-attention mechanism discussed in 'Relational recurrent neural networks' to compute relationships between entities extracted from input data. The system was tested on the Box-World environment, in which an agent needs to use keys to open boxes in a certain order. It generalised very well to test environments which required much longer sequences of actions than any training examples, and improved slightly on a baseline for Starcraft mini-games.


1st Mathematical Reasoning in General Artificial Intelligence Workshop, ICLR 2021

Another interesting fact about neural net generalization! Like <@double descent@>(@Deep Double Descent@), this can’t easily be explained by appealing to the diversity model. I don’t really have a good theory for either of these phenomena, but one guess for grokking is that:

1. Functions that perfectly memorize the data without generalizing (i.e. probability 1 on the true answer and 0 elsewhere) are very complicated, nonlinear, and wonky. The memorizing functions learned by deep learning don’t get all the way there and instead assign a probability of (say) 0.95 to the true answer.
2. The correctly generalizing function is much simpler and for that reason can be easily pushed by deep learning to give a probability of 0.99 to the true answer.
3. Gradient descent quickly gets to a memorizing function, and then moves mostly randomly through the space, but once it hits upon the correctly generalizing function (or something close enough to it), it very quickly becomes confident in it, getting to probability 0.99 and then never moving very much again.

A similar theory could explain deep double descent: the worse your generalization, the more complicated, nonlinear and wonky you are, and so the more you explore to find a better generalizing function. The biggest problem with this theory is that it suggests that making the neural net larger should primarily advantage the memorizing functions, but in practice I expect it will actually advantage the correctly generalizing function. You might be able to rescue the theory by incorporating aspects of the <@lottery ticket hypothesis@>(@The Lottery Ticket Hypothesis at Scale@).

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, Vedant Misra

This paper presents an interesting empirical phenomenon with deep learning: **grokking**.

Consider tasks of the form "a ◦ b = ?", where "◦" is some operation on modular arithmetic. For example, in the task of addition mod 97, an example problem would be "32 + 77 = ?". There are exactly 97 possible operands, each of which gets its own token, and so there are 97^2 possible problems that are defined by pairs of tokens. We will train a neural net on some fraction of all possible problems and then ask how well it performs on the remaining problems it didn't see: that is, we're asking it to fill in the missing entries in the 97x97 table that defines addition mod 97.

It turns out that in these cases, the neural net memorizes the training dataset pretty quickly (in around 10^3 updates), at which point it has terrible generalization performance. However, if you continue to train it all the way out to 10^6 updates, then it will often hit a phase transition where you go from random chance to perfect generalization almost immediately. Intuitively, at the point of the phase transition, the network has "grokked" the function and can run it on new inputs as well. Some relevant details about grokking:

1. It isn't specific to group or ring operations: you also see grokking for tasks like "a/b if b is odd, otherwise a − b".
2. It is quite sensitive to the choice of hyperparameters, especially learning rate; the learning rate can only vary over about a single order of magnitude.
3. The time till perfect generalization is reduced by weight decay and by adding noise to the optimization process.
4. When you have 25-30% of possible examples as training data, a decrease of 1 percentage point leads to an increase of 40-50% in the median time to generalization.
5. As problems become more intuitively complicated, time till generalization increases (and sometimes generalization doesn't happen at all). For example, models failed to grok the task x^3 + xy^2 + y (mod 97) even when provided with 95% of the possible examples as training data.
6. Grokking mostly still happens even when adding 1,000 "outliers" (points that could be incorrectly labeled), but mostly stops happening at 2,000 "outliers".


Understanding the Lottery Ticket Hypothesis

This post summarizes work on the <@lottery ticket hypothesis@>(@The Lottery Ticket Hypothesis at Scale@), including its implications for AI alignment.


Muppet: Massive Multi-task Representations with Pre-Finetuning

Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, Sonal Gupta

This paper proposes pre-finetuning: given a language model pretrained on a large dataset, we do a second stage where we train the model to solve a large variety of tasks (around 50 in this paper), and only after that do we finetune the model on our actual task of interest. The authors show that this leads to improved results, especially on tasks where we only have limited data.


Scaling Scaling Laws with Board Games

While we've seen <@scaling laws@>(@Scaling Laws for Neural Language Models@) for compute, data, and model size, we haven't yet seen scaling laws for the _problem size_. This paper studies this case using the board game Hex, in which difficulty can be increased by scaling up the size of the board. The author applies AlphaZero to a variety of different board sizes, model sizes, RL samples, etc and finds that performance tends to be a logistic function of compute / samples used. The function can be characterized as follows:

1. Slope: In the linearly-increasing regime, you will need about 2× as much compute as your opponent to beat them 2/3 of the time.
2) Perfect play: The minimum compute needed for perfect play increases 7× for each increment in board size.
3) Takeoff: The minimum training compute needed to see any improvement over random play increases by 4× for each increment of board size.

These curves fit the data quite well. If the curves are fit to data from small board sizes and then used to predict results for large board sizes, their error is small.

Recall that AlphaZero uses MCTS to amplify the neural net policy. The depth of this MCTS determines how much compute is spent on each decision, both at training time and test time. The author finds that a 10x increase in training-time compute allows you to eliminate about 15x of test-time compute while maintaining similar performance.


I find the hypothesis the authors propose quite compelling (and this is very similar to the hypothesis that neural networks tend to be modular, which we discuss more below). Partly, this is because it has a common-sense explanation: when designing an organization, you want to put related functions in the same group to minimize the communication across groups. Here, the full network is the organization, the branches are an explicit constraint on communication, and so you want to put related functions in the same branch.

At the end of the article, the authors also suggest that there could be a connection with the way that different regions of the brain are specialized to particular tasks. I’ll go further than the authors in my speculation: it seems plausible to me that this specialization is simply the result of the brain’s learning algorithm reflecting the structure of the world through specialization. (Though it seems likely that the different areas of the brain must at least have different “architectures”, in order for the same tasks to be routed to the same brain regions across humans.) But the case of AlexNet demonstrates that in theory, the only thing you need for specialization to arise is a restriction on the communication between one part of the architecture and the other.

Branch Specialization

Chelsea Voss, Gabriel Goh, Nick Cammarata, Michael Petrov, Ludwig Schubert, Chris Olah

Neural network architectures sometimes have different "branches", where later features can depend on earlier features _within the same branch_, but cannot depend on features in parallel branches. This post presents evidence showing that in these architectures, branches often tend to specialize in particular types of features. For example:

1. The first two layers in AlexNet are split into two branches. In one branch, the first layer tends to learn black and white Gabor filters, while in the other branch, the first layer tends to learn low-frequency color detectors. This persists across retraining, or even training on a different dataset of natural images, such as Places (rather than ImageNet).
2. All 9 of the black and white vs. color detectors in mixed3a are in mixed3a_5x5 (p < 1e-8). All 30 of the curve-related features in mixed3b are in mixed3b_5x5 (p < 1e-20). There are confounds here, but also good reasons to expect that it is in fact branch specialization.

Given that branch specialization seems to be robust and consistent even across datasets, a natural hypothesis is that it is reflecting a structure that already exists. Even if you didn't have branching, it seems likely that the model would still learn very similar neurons, and it seems plausible that e.g. the weights connecting the first-layer black-and-white Gabor filters to the second-layer color detectors are effectively zero. With branching, you learn the same features in such a way that all the weights that previously were effectively zero now don't exist because they would be crossing branches. This would look like having the Gabor filters in one branch and the color detectors in the other branch.


The “typographic adversarial attack” is interesting as a phenomenon that happens, but I’m not that happy about the phrasing -- it suggests that CLIP is dumb and making an elementary mistake. It’s worth noting here that what’s happening is that CLIP is being asked to look at an image of a Granny Smith apple with a piece of paper saying “iPod” on it, and then to complete the caption “an image of ???” (or some other similar zero-shot prompt). It is quite possible that CLIP “knows” that the image contains a Granny Smith apple with a piece of paper saying “iPod”, but when asked to complete the caption with a single class from the ImageNet classes, it ends up choosing “iPod” instead of “Granny Smith”. I’d caution against saying things like “CLIP thinks it is looking at an iPod”; this seems like too strong a claim given the evidence that we have right now.

Multimodal Neurons in Artificial Neural Networks

Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, Chris Olah

[CLIP](https://openai.com/blog/clip/) is a large model that was trained to learn a separate embedding for images and text, such that the embedding for an image is maximally similar to the embedding for its caption. This paper uses feature visualization and dataset examples to analyze the vision side of the model, and shows that there are many _multimodal_ neurons. For example, there is a Spiderman neuron that responds not just to pictures of Spiderman, but also to sketches of Spiderman, and even the word "spider" (written in the image, not the caption). The neurons are quite sophisticated, activating not just on instances of the concept, but also things that are _related_ to the concept. For example, the Spiderman neuron is also activated by images of other heroes and villains from the Spiderman movies and comics. There are lots of other neurons that I won't go into, such as neurons for famous people, regions, facial emotions, religions, holidays, abstract concepts, numbers, and text.

Unsurprisingly, many of these neurons encode stereotypes that we might consider problematic: for example, there is an immigration neuron that responds to Latin America, and a terrorism neuron that responds to the Middle East.

The concepts learned by CLIP also have some notion of hierarchy and abstraction. In particular, the authors find that when they train a sparse linear classifier on top of the CLIP features, the resulting classifier has a "hierarchy" that very approximately matches the hierarchy used to organize the ImageNet classes in the first place -- despite the fact that CLIP was never trained on ImageNet at all. (I'm not sure how approximate this match is.)

As mentioned before, the neurons can respond to text in the image, and in a few cases they can even respond to text in different languages. For example, a "positivity" neuron responds to images of English "Thank You", French "Merci", German "Danke", and Spanish "Gracias". The fact that the model is so responsive to text in images means that it is actually very easy to influence its behavior. If we take an apple (originally correctly classified as a Granny Smith) and tape on a piece of paper with the word "iPod" written on it, it will now be classified as an iPod with near certainty. This constitutes a new and very easy to execute "typographic" adversarial attack.

However, not everything that CLIP is capable of can be explained with our current interpretability techniques. For example, CLIP is often able to tell whether an image is from San Francisco (and sometimes even what region within San Francisco), but the authors were not able to find a San Francisco neuron, nor did it look like there was a computation like "California + city".


As with last week’s paper, I’m happy to see more work on understanding scaling laws. For this paper, the “assumption on reality” is in which data distribution we assume the data is drawn from. However, overall I feel less compelled by this paper than with the one from last week, for two reasons: First, it seems to me that using a tabular (memorization) algorithm is probably too coarse of a model; I would guess that there are facts about neural nets that are relevant to scaling that aren’t captured by tabular algorithms. Second, I prefer the assumption that the data are drawn from a low-dimensional manifold rather than that the data are drawn from some specific distribution like a Zipf distribution (or others discussed in the paper).

Learning Curve Theory

Like [last week's highlight](https://arxiv.org/abs/2102.06701) ([AN #140](https://mailchi.mp/229fd666e06b/an-140-theoretical-models-that-predict-scaling-laws)), this paper proposes a theoretical model that could predict empirically observable scaling laws. The author considers a very simple online learning model, in which we are given a feature vector and must classify it into one of two categories. We'll also consider a very simple tabular algorithm that just memorizes the classifications of all previously seen vectors and spits out the correct classification if it has been seen before, and otherwise says "I don't know". How does the error incurred by this algorithm scale with data size?

The answer of course depends on the data distribution -- if we always see the same feature vector, then we never make an error after the first timestep, whereas if the vector is chosen uniformly at random, we'll always have maximal error. The author analyzes several possible data distributions in between these extremes.

The most interesting case is when the data is drawn from a Zipf distribution. In this case, when you order the feature vectors from most to least likely, the nth vector has probability proportional to n^(-(α+1)). Then we see a power law for the scaling, n^(-β), where β = α / (α+1). This could explain the scaling laws observed in the wild.


Obviously this is an astounding accomplishment for DeepMind (conflict of interest notice: I work at DeepMind). I feel like I should have some opinion on what this means for the future of AI systems, but unfortunately I think I don’t know enough about protein folding to have any interesting takes.

From an outside view perspective, it seems like this is an example of deep learning crushing a task that a) humans put a lot of effort into and b) humans weren’t evolutionarily designed for. This is exactly what we saw with Go, Dota and StarCraft, and so this isn’t much of an update for me. Yes, this is a case of it being used in a real-world problem rather than a synthetic game, but that doesn’t seem particularly relevant.

**Asya's opinion:** I think this is particularly interesting because this model is closer to being a source of revenue than solutions to other problems. This makes me think machine learning research might actually solve enough important problems to pay for itself in the near future.

AlphaFold: a solution to a 50-year-old grand challenge in biology

The AlphaFold team, John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Kathryn Tunyasuvunakool, Olaf Ronneberger, Russ Bates, Augustin Žídek, Alex Bridgland, Clemens Meyer, Simon A A Kohl, Anna Potapenko, Andrew J Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Martin Steinegger, Michalina Pacholska, David Silver, Oriol Vinyals, Andrew W Senior, Koray Kavukcuoglu, Pushmeet Kohli, Demis Hassabis

The newest results from <@AlphaFold@>(@AlphaFold: Using AI for scientific discovery@) on the CASP-14 assessment give it a median score of 92.4 GDT across all targets, where a score of 90 is informally considered to be competitive with results obtained from experimental methods. The system also shows some signs of real-world usability: for example, it was used earlier this year to predict the structure of two COVID proteins, which were later borne out by experimental results (that took several months to obtain, if I understand correctly).


Fast hyperparameter tuning is a pretty important aspect of models. I particularly like [population-based training](https://deepmind.com/blog/article/population-based-training-neural-networks) for this purpose, because it doesn’t require your computation to be differentiable. However, when you can make your computation differentiable, this method is probably significantly more efficient (and perhaps also more performant).

Gradient Descent: The Ultimate Optimizer

Kartik Chandra, Erik Meijer, Samantha Andow, Emilio Arroyo-Fang, Irene Dea, Johann George, Melissa Grueter, Basil Hosmer, Steffi Stumpos, Alanna Tempest, Shannon Yang

Hyperparameter tuning is an important and tedious step for most applications of machine learning. Often this can cause a project to take significantly longer, as you need to have multiple training runs with different hyperparameters in order to identify which ones work best. How can we do better?

This paper shows that in some cases, you can make the computation involving your hyperparameters differentiable, such that they too can be optimized using gradient descent _during the actual training run_. They show this for SGD and Adam (where for Adam they optimize all four hyperparameters, not just the learning rate). Since these hyperparameters are then optimized using another instantiation of gradient descent, that new instantiation also has its own hyperparameters that can once again be optimized. They show how to build an arbitrarily high "stack" of hyperparameter optimizers.

In practice, building a stack of just 3 or 4 such optimizers makes it very robust to the initial choice of parameters by a human, while only increasing the cost of training by less than 2x.


I spent most of the summary talking about the language model, but I think it's likely that the cooler thing is in fact GShard, as it will enable other very large models to do model parallelization in the future.

The improved efficiency for wide models here seems like it may go away as we become able to train even deeper models that are extremely general and so much more sample efficient than wide models.

This model technically has more parameters than GPT-3, but it’s “sparse” in that not all the inputs are used to update all the parameters. Sometimes people compare the number of parameters in a neural network to the number of synapses in the human brain to guess at when we're likely to get human-level AI. I find using this number directly to be pretty dubious, partially because, as this paper illustrates, the exact architecture of a system has a big influence on the effective power of each parameter, even within the relatively narrow domain of artificial neural networks.

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, Zhifeng Chen

This paper introduces GShard, a module that makes it easy to write parallel computation patterns with minimal changes to existing model code. GShard automatically does a lot of the work of splitting computations across machines, enabling the easy creation of much larger models than before.

The authors use GShard to train a 600 billion parameter multilingual Transformer translation model that's wide, rather than deep (36 layers). They use a "mixture of experts" model where some of the individual feed-forward networks in the Transformer are replaced with a set of feed-forward networks-- each one an "expert" in some part of the translation. The experts are distributed across different machines, and the function for sending inputs to experts is learned, with each input being sent to the top two most relevant experts. Since each expert only has to process a fraction of all the inputs, the amount of computation needed is dramatically less than if every input were fed through a single, larger network. This decrease in needed computation comes with a decrease in the amount of weight sharing done by the network.

The paper compares the 600 billion parameter model's performance to several other smaller models as well as a 96-layer deep model with only 2.3 billion parameters. For the wide networks, the authors find that in general, larger models do better, but that at some point the larger model starts doing worse for very "low-resource" languages-- languages that don't have much training data available. The authors argue that this is because the low-resource languages benefit from "positive language transfer", an effect where weights encode knowledge learned from training on other languages that can then be applied to the low-resource ones. As you increase the number of experts in the wide model past a certain point, the amount of training that each expert does decreases, so there's less positive language transfer to low-resource languages within each expert.

They also find that deeper networks are more sample efficient, reaching better test error with the same amount of training examples, but are less computationally efficient (given current constraints). The 600 billion parameter, 36-layer model takes 22.4 TPU core years and 4 days to train, reaching a score on the BLEU benchmark of 44.3. The 2.3 billion parameter, 96-layer model takes 235 TPU core years and 42 days to train, reaching a score on the BLEU benchmark of 36.9.


This was a pretty interesting read and provides evidence against large effects of overfitting on the test set. On the other hand, results like this also seem to highlight how benchmarks are mostly useful for model comparison, and how nonrobust they can be to fairly benign distributional shift.

Identifying Statistical Bias in Dataset Replication

Logan Engstrom, Andrew Ilyas, Aleksander Mądry, Shibani Santurkar, Jacob Steinhardt, Dimitris Tsipras

One way of dealing with finite and fixed test sets and the resulting possibility of overfitting on the test set is dataset replication, where one tries to closely mimic the original process of dataset creation to obtain a larger test set. This can lead to bias if the difficulty of the new test images is distributed differently than in the original test set. A previous attempt at [dataset replication on ImageNet](https://arxiv.org/abs/1902.10811) tried to get around this by measuring how often humans under time pressure correctly answered a yes/no question about an image's class (dubbed selection frequency), which can be seen as a proxy for classification difficulty.

This data was then used to sample candidate images for every class which match the distribution of difficulty in the original test set. Still, all tested models performed worse on the replicated test set than on the original. Parts of this bias can be explained by noisy measurements combined with disparities in the initial distribution of difficulty, which are likely as the original ImageNet data was prefiltered for quality. Basically, the more noisy our estimates for the difficulty are, the more the original distribution of difficulty matters. As an extreme example, imagine a class for which all images in the original test set have a selection frequency of 100%, but 90% of candidates in the new test set have a selection frequency of 50%, while only 10% are as easy to classify as the images in the original test set. Then, if we only use a single human annotator, half of the difficult images in the candidate pool are indistinguishable from the easy ones, such that most images ending up in the new test set are more difficult to classify than the original ones, even after the adjustment.

The authors then replicate the ImageNet dataset replication with varying amounts of annotators and find that the gap in accuracy between the original and the new test set progressively shrinks with reduced noise from 11.7% with one annotator to 5.7% with 40. Lastly, they discuss more sophisticated estimators for accuracy to further lower bias, which additionally decreases the accuracy gap down to around 3.5%.


I found it a bit surprising that replacing word prediction with fake discrimination would help that much, but from the ablations, it seems like this is really mostly an instrument to get a loss signal for every single word, which is a cool idea. On a more zoomed-out perspective, results like this seem to show that gains in <@algorithmic efficiency@>(@AI and Efficiency@) are not fundamentally slowing down.

Paper: ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

More Efficient NLP Model Pre-training with ELECTRA

Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning

There are two main approaches to pretraining for NLP, language models (LMs) which iteratively predict the next word in a given incomplete sentence, and masked language models (MLMs), which predict the identities of a few masked words in an otherwise complete sentence. While not just looking at the previous words (bidirectionality) can be advantageous, MLMs only learn to predict the masked words, which reduces how much is learnt from a given sentence.

The authors present an alternative approach, ELECTRA, that outperforms RoBERTa while requiring less than a third of the compute. This is achieved by changing the form of the pretraining task from predicting words to discriminating fake words: Instead of masking, some words are replaced by words generated by an MLM and the trained model has to classify these as fake. This way, we get bidirectionality, but also a more dense signal, as the model has to produce an output for every single word, not just the masked ones. While this looks similar to GANs, the generator is only trained on the usual MLM loss and is not incentivized to fool the discriminator, as GANs don't seem to work well on sequence data.


The main relevance of an approach like this is that it provides proof-of-concept that complex goals, such as shape formation, can be programmed in an embarrassingly parallel fashion amenable to deep learning methodology. This naturally has implications in multi-agent settings where communication is expensive. I'd recommend checking out the main web app which allows you to watch and interact with the CA while they're growing. They also have a [code repository](https://colab.research.google.com/github/google-research/self-organising-systems/blob/master/notebooks/growing_ca.ipynb) that is easily adaptable to training on your own patterns. For example, I grew a regenerating Patrick Star [here](https://colab.research.google.com/drive/1BuE-0ceBP7ebTmX7pP_urP-FJGvmjCpb?usp=sharing).

Growing Neural Cellular Automata

Alexander Mordvintsev, Ettore Randazzo, Eyvind Niklasson, Michael Levin

The process of an organism's shape development (morphogensis) is an active area of research. One central problem is determining how cells decide how to grow and when to stop. One popular model for investigating this is Cellular Automata (CA). These model cells as living on a grid and interacting with each other via rules generated by looking at their nearest neighbors. The authors contribute to this research direction by introducing rule-sets that depend continuously on their local surroundings. The central insight connecting CA and deep learning is that because the rule-sets are constant the update rules work similarly to a convolutional filter. This allows the authors to take advantage of methods available to train neural networks to simulate CA. Using this insight, the authors train CA that can form into images that are resistant to perturbations and deletions. In other words, the CA are capable of regeneration.


It's a good example of the push towards learning more and more complex algorithms using neural nets (in this case, planning). From a safety perspective, differentiable planning networks may be useful for modeling humans.

Universal Planning Networks

Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, Chelsea Finn

This is an architecture that has a differentiable planning module, that is, a neural network that takes in (encodings of) states or observations and produces actions. You can use this in conjunction with eg. expert demonstrations (as in imitation learning) in order to learn features that are optimized for the purpose of planning, focusing only on the details relevant to the task, unlike an auto-encoder, which must reconstruct the entire image, including irrelevant details.


This work empirically demonstrates that we now have sufficient methods and tricks in our ES toolkit that enable us to evolve machine learning algorithms from scratch. Additionally, this process produces computer code, which itself may yield to theoretical analysis furthering our knowledge of learning algorithms. I think that powerful AI systems of the future may employ such techniques to discover solutions.

**Rohin's opinion:** It’s cool to see automation of even the ML algorithms themselves. However, note that there is no deep learning or function approximation in this system: it is a simple search algorithm that is commonly used in program synthesis. In fact, though the paper is presented as an ML paper, it seems to me much more like a classic program synthesis paper that happens to be working in the domain of ML algorithms.

Without any learned heuristics, these algorithms have a hard time scaling: they can only synthesize relatively short snippets of code, since the search space grows exponentially as the code gets longer. Good results depend very strongly on being able to create a DSL in which the correct program is short. In this case, it looks like strong programs are about ~20 straight-line instructions, which seems on the more impressive side given the simplicity of the algorithm and the DSL, though they did throw a huge amount of compute at the problem.

AutoML-Zero: Evolving Machine Learning Algorithms From Scratch

Esteban Real*, Chen Liang*, David R. So, Quoc V. Le

Most previous work in the area of automated machine learning, or AutoML, has focussed on narrow search spaces that are restricted to specific parts of the machine learning pipeline, e.g. the architecture of a neural network, or the optimizer in meta-learning. These spaces are often so constrained by the hand-engineered components around them that architectures and algorithms discovered, say, by evolutionary search (ES), are only slightly better than random search (RS). This work aims to set up the problem with very weak constraints and a wide search space: a) a machine learning program has three component functions, _Setup_, _Predict_, and _Learn_, which start out empty, and b) are populated by RS or ES with procedural operations from over 50 arithmetic, trigonometric, linear algebra, probability, and pre-calculus operators.

They demonstrate that with such a vast search space, RS fares very poorly in comparison to ES. They also report that ES finds several procedures that are recognizable as useful for machine learning, such as a simple neural network, gradient descent, gradient normalization, multiplicative interactions, noise augmentation, noisy dropout and learning rate decay.


The correspondence between subnetwork stability and performance after retraining might just be an artefact of both (somewhat obviously) improving with more training. What is interesting is that small amounts of training seem to have disproportionate effects for both factors, although one should keep in mind that the same is true for the loss, at least in absolute terms.

Linear Mode Connectivity and the Lottery Ticket Hypothesis

Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, Michael Carbin

Instability analysis looks at how sensitive neural network training is to noise in SGD. A network is called stable if the test error remains approximately constant along the line connecting network weights obtained by training on differently ordered data.

The authors find that most popular networks in image classification are unstable at initialization for more challenging tasks but become stable long before convergence. They also find that <@winning tickets@>(@Understanding the generalization of 'lottery tickets' in neural networks@) found by iterative magnitude pruning are usually stable, while unstable subnetworks don't manage to match the original network's performance after training. As the original network, pruned subnetworks become more stable when they are initialized with weights from later stages of the training process. This is consistent with previous results showing that resetting subnetwork weights to states in early training leads to increased performance after retraining, compared to resetting to the initial state. While stability seems to correspond to better accuracy for subnetworks, very sparse subnetworks perform worse than the unpruned network, even if they are stable.


Paper: Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers

Speeding Up Transformer Training and Inference By Increasing Model Size

Zhuohan Li*, Eric Wallace*, Sheng Shen*, Kevin Lin*, Kurt Keutzer, Dan Klein, Joseph E. Gonzalez

This blog post and associated paper confirm the findings from <@Scaling Laws for Neural Language Models@> that the most efficient way to train Transformer-based language models is to train very large models and stop before convergence, rather than training smaller models to convergence.


Do Better ImageNet Models Transfer Better?

See [Import AI](https://jack-clark.net/2018/05/29/import-ai-96-seeing-heartbeats-with-deepphys-better-synthetic-images-via-sagan-and-spotting-pedestrians-via-a-trans-european-dataset/)


At first this paper appeared more ambitious than [Saxton et al. (2019)](https://arxiv.org/abs/1904.01557), but it ended up with more positive results, even though the papers used the same techniques. Therefore, my impression is not that we recently made rapid progress on incorporating mathematical reasoning into neural networks; rather, I now think that the tasks of integration and solving differential equations are simply well-suited for neural networks.

Deep Learning for Symbolic Mathematics

This paper demonstrates the ability of sequence-to-sequence models to outperform [computer algebra systems](https://en.wikipedia.org/wiki/Computer_algebra_system) (CAS) at the tasks of symbolic integration and solving ordinary differential equations. Since finding the derivative of a function is usually easier than integration, the authors generated a large training set by generating random mathematical expressions, and then using these expressions as the labels for their derivatives. The mathematical expressions were formulated as syntax trees, and mapped to sequences by writing them in Polish notation. These sequences were, in turn, used to train a transformer model. While their model outperformed top CAS on the training data set, and could compute answers much more quickly than the CAS could, tests of generalization were mixed: importantly, the model did not generalize extremely well to datasets that were generated using different techniques than the training dataset.


As you might have gathered last week, I'm not sold on double descent as a clear, always-present phenomenon, though it certainly is a real effect that occurs in at least some situations. So I tend not to believe counterintuitive conclusions like "larger models are simpler" that are premised on double descent.

Regardless, I expect that powerful AI systems are going to be severely underparameterized, and so I don't think it really matters that past the interpolation threshold larger models are simpler. I don't think the case for mesa optimization should depend on this; humans are certainly "underparameterized", but should count as mesa optimizers.

Inductive biases stick around

This update to Evan's <@double descent post@>(@Understanding "Deep Double Descent"@) explains why he thinks double descent is important. Specifically, Evan argues that it shows that inductive biases matter even for large, deep models. In particular, double descent shows that larger models are _simpler_ than smaller models, at least in the overparameterized setting where models are past the interpolation threshold where they can get approximately zero training error. This makes the case for <@mesa optimization@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@) stronger, since mesa optimizers are _simple_, compressed policies.


I certainly agree that neural nets have strong inductive biases that help with their generalization; a clear example of this is that neural nets can learn [randomly labeled data](https://arxiv.org/abs/1611.03530) (which can never generalize to the test set), but nonetheless when trained on correctly labeled data such nets do generalize to test data. Perhaps more surprising here is that the inductive biases help even _after_ fully capturing the data (achieving zero training loss) -- you might have thought that the data would swamp the inductive biases. This might suggest that powerful AI systems will become simpler over time (assuming an inductive bias towards simplicity). However, this is happening in the regime where the neural nets are overparameterized, so it makes sense that inductive biases would still play a large role. I expect that in contrast, powerful AI systems will be severely underparameterized, simply because of _how much data_ there is (for example, <@the largest GPT-2 model still underfits the data@>(@Better Language Models and Their Implications@)).

Understanding “Deep Double Descent”

This post explains deep double descent (in more detail than my summaries), and speculates on its relevance to AI safety. In particular, Evan believes that deep double descent shows that neural nets are providing strong inductive biases that are crucial to their performance -- even _after_ getting to ~zero training loss, the inductive biases _continue_ to do work for us, and find better models that lead to lower test loss. As a result, it seems quite important to understand the inductive biases that neural nets use, which seems particularly relevant for e.g. <@mesa optimization and pseudo alignment@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@).


I appreciate the premise of this paper, and in general feel positively towards papers that delve into a better understanding of how much of modern neural network performance is attributable to (discrete) structural architectures vs particular settings of continuous weight parameters, and I think this paper does that in a clever way by essentially marginalizing over different weight values. The framing of this paper, implicitly comparing networks used without weight training to animals with innate abilities, did make me wonder whether the architecture vs weights analogy to evolution vs learning is a sound one. Because, while it's true that the weights weren't explicitly gradient-descent trained in this paper, the network did still perform optimization based on task performance, just over a set of discrete parameters rather than continuous ones. In that context, it doesn't really seem correct to consider the resulting architectures "untrained" in a way that I think that analogy would suggest. I'd be curious to see more work in this direction that blends in ideas from meta-learning, and tries to find architectures that perform well on multiple tasks, rather than just one.

Weight Agnostic Neural Networks

Inspired by the ability of animals to perform some tasks at birth, before learning about the world, this paper tries to find network architectures that perform well over a wide range of possible model parameters. The idea here is that if an architecture performs well with different sampled weights and without training to update those weights, then the architecture itself is what's responsible for encoding the solution, rather than any particular weight configuration. The authors look for such architectures on both classification and reinforcement learning problems by employing NEAT, a evolutionary method from Neural Architecture Search that searches for the best-performing topologies within the space of possible node connections and activations. The authors find that they're able to construct architectures that do better than random on their test problems without training weights explicitly.


XLNet: Generalized Autoregressive Pretraining for Language Understanding

Zhilin Yang*, Zihang Dai*, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le

XLNet sets significantly improved state-of-the-art scores on many NLP tasks, beating out BERT. This was likely due to pretraining on significantly more data, though there are also architectural improvements.


This seems like a useful mental concept, but it seems quite difficult to effectively remedy, except through preferring off-policy methods to on-policy ones, since there isn't really a way to decompose real RL tasks into separable components the way they do in their toy example

Ray Interference: a Source of Plateaus in Deep Reinforcement Learning

Tom Schaul, Diana Borsa, Joseph Modayil, Razvan Pascanu

The authors argue that Deep RL is subject to a particular kind of training pathology called "ray interference", caused by situations where (1) there are multiple sub-tasks within a task, and the gradient update of one can decrease performance on the others, and (2) the ability to learn on a given sub-task is a function of its current performance. Performance interference can happen whenever there are shared components between notional subcomponents or subtasks, and the fact that many RL algorithms learn on-policy means that low performance might lead to little data collection in a region of parameter space, and make it harder to increase performance there in future.


I find this paper really clever and potentially quite high-impact, since Transformers are *so* widely used, and this paper could offer a substantial speedup without much theoretical loss of information. I also just enjoyed having to think more about the trade-offs between convolutions, RNNs, and transformers, and how to get access to different points along those tradeoff curves.

Generative Modeling with Sparse Transformers

I see this paper as trying to interpolate the space between convolution (fixed receptive field, number of layers needed to gain visibility to the whole sequence grows with sequence length) and attention (visibility to the entire sequence at each operation, but n^2 memory and compute scaling with sequence length, since each new element needs to query and be queried by each other element). This is done by creating  chains of operations that are more efficient, and can offer visibility to the whole sequence in k steps rather than k=1 steps, as with normal attention. An example of this is one attention step that pulls in information from the last 7 elements, and then a second that pulls in information from each 7th element back in time (the "aggregation points" of the first operation).


Machine translation used to be done in multiple stages (involving parse trees as an intermediate representation), and then it was done better using end-to-end training of a deep neural net. This looks like the beginning of the same process for speech-to-speech translation. I'm not sure how much people care about speech-to-speech translation, but **if it's an important problem, I'd expect the direct speech-to-speech systems to outperform the cascaded approach relatively soon**. I'm particularly interested to see whether you can "bootstrap" by using the cascaded approach to generate training data for the end-to-end approach, and then finetune the end-to-end approach on the direct speech-to-speech data that's available to improve performance further.

Introducing Translatotron: An End-to-End Speech-to-Speech Translation Model

This post introduces Translatotron, a system that takes speech (not text!) in one language and translates it to another language. This is in contrast to most current "cascaded" systems, which typically go from speech to text, then translate to the other language, and then go back from text to speech. While Translatotron doesn't beat current systems, it demonstrates the feasibility of this approach.


A Recipe for Training Neural Networks

This is a great post detailing how to train neural networks in practice when you want to do anything more complicated than training the most common architecture on the most common dataset. For all of you readers who are training neural nets, I strongly recommend this post; the reason I'm not summarizing it in depth is because a) it would be a really long summary and b) it's not that related to AI alignment.


MLPerf

From their overview: "The MLPerf effort aims to build a common set of benchmarks that enables the machine learning (ML) field to measure system performance for both training and inference from mobile devices to cloud services." They have a track to measure the performance of hardware and software systems that support ML models, as well as a track that aims to advance the state-of-the-art in ML models. They consider a broad set of problems (though it seems like they are all problems where some deep learning technique is state-of-the-art).


A Conservative Human Baseline Estimate for GLUE: People Still (Mostly) Beat Machines

[BERT](https://arxiv.org/abs/1810.04805) tremendously improves performance on several NLP datasets, such that it has "taken over" NLP. GLUE represents performance of NLP models across a broad range of NLP datasets. Now GLUE has human performance measurements. According to the [current GLUE leaderboard](https://gluebenchmark.com/leaderboard), the gap between human performance and models fine-tuned on GLUE datasets is a mere 4.7%. Hence many current NLP datasets are nearly "solved."


Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet

This paper proposes a bag-of-features model using patches as features, and they show that this can obtain accuracy similar to VGGNet architectures. They classify each patch and produce the final classification by a majority vote; Figure 1 of the paper tells all. In some ways this model is more interpretable than other deep architectures, as it is clear which regions activated which class. They attempt to show that, like their model, VGGNet does not use global shape information but instead uses localized features.


While performance is good on small problems, comparing every memory to every other doesn't scale well (a concern the authors also mention in their discussion). It remains to be seen how pruning older memories affects performance.

Relational recurrent neural networks

Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, Timothy Lillicrap

This paper introduces the Relational Memory Core, which allows interactions between memories stored in memory-based neural networks. It does so using a "self-attention mechanism": each memory updates its contents by attending to all other memories via several "attention heads" which focus on different features. This leads to particularly good performance on the nth-farthest task, which requires the ranking of pairwise distances between a set of vectors (91% accuracy, compared with baseline 30%), and the Mini-Pacman task.


It's worth skimming through the page to get a sense of concrete numbers for various benchmarks used in the ML community.

DAWNBench

This is a collection of statistics for time and compute costs, both for training and inference, for various common models and benchmarks.


Discriminator Rejection Sampling

Samaneh Azadi, Catherine Olsson, Trevor Darrell, Ian Goodfellow, Augustus Odena

Under simplifying assumptions, GAN training should converge to the generator modelling the true data distribution while the discriminator always outputs 0.5. In practice, at the end of training the discriminator can still distinguish between images from the generator and images from the dataset. This suggests that we can improve the generated images by only choosing the ones that the discriminator thinks are from the dataset. However, if we use a threshold (rejecting all images where the discriminator is at least X% sure it comes from the generator), then we no longer model the true underlying distribution, since some low probability images could never be generated. They instead propose a rejection sampling algorithm that still recovers the data distribution under strict assumptions, and then relax those assumptions to get a practical algorithm, and show that it improves performance.


Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

It always makes me happy to read a paper about making symbolic approaches faster using neural nets to learn heuristics. That said, I'm concerned about the evaluation in this paper -- their programs are fairly strange, often involving a huge mess of cons (make-pair), car (first) and cdr (second), and not including recursion. The symbolic approaches they evaluate against are aiming to synthesize recursive functions similar to what people write, and I wouldn't be surprised if they had heuristics that actively discouraged these big messes of cars and cdrs, since normal programs don't look like that. The programs are also primarily taking pieces of data out of an input, and then recombining them in some way -- this feels like a significantly easier task than most synthesis problems (in the sense that I could probably write a handcoded solution that performs very well on this domain only).

Neural Guided Constraint Logic Programming for Program Synthesis

Lisa Zhang, Gregory Rosenblatt, Ethan Fetaya, Renjie Liao, William E. Byrd, Matthew Might, Raquel Urtasun, Richard Zemel

In program synthesis from examples, we want to find a program consistent with a given set of input-output examples. One classic approach is to use logic programming. In logic programming, instead of writing functions that compute output = f(input), we write rules to compute relations. To encode standard functions, we would write the relation (f, i, o), which is interpreted as "computing f(i) gives o". In logic programming, you can let any variable be unknown, and the language will search for a solution. Using this you can eg. invert a function f on a specific output o, using the query (f, ?, o). To apply logic programming to program synthesis, we write an interpreter eval for the language we want to synthesize in, and pose the query (eval, ?, i, o). They consider the lambda calculus with pairs and lists as their language.

The algorithm that falls out is a recursive descent search over the possible structure of the program, that generates and checks partial constraints over the partial programs implied by the input-output examples during the search. The search has branching points where it must choose, for some as-yet-unknown part of the program, what language construct it should use (if, cons, variable, etc.) This paper attempts to use a neural net to predict what choice the search should make to find a solution, replacing some simple hand-tuned heuristics. It can be trained either using reinforcement learning (where the search choices are actions, the partial search trees are states, and the goal is to find a complete program), or through supervised learning since they know for training programs what choices are optimal. They also use a curriculum and experience replay. They evaluate against classical symbolic approaches (λ2, Escher, Myth) and RobustFill, and show that their method generalizes better to finding longer programs not seen in the training dataset.


The post doesn't really explain why this is the case, instead referencing the theory in their paper (which I haven't read). It does sound like a cool result explaining a phenomenon that I do find confusing, since RNNs should be more expressive than feedforward models. It does suggest that gradient descent is not actually good at finding the optimum of a function, if that optimum involves lots of long-term dependencies.

When Recurrent Models Don't Need to be Recurrent

Recurrent neural networks (RNNs) are able to use and update a hidden state over an entire sequence, which means that in theory it is possible for them to learn very long term dependencies in a sequence, that a feedforward model would not be able to do. For example, it would be easy to assign weights to an RNN so that on input x_n it outputs n (the length of the sequence so far), whereas a feedforward model could not learn this function. Despite this, in practice feedforward methods match and exceed the performance of RNNs on sequence modeling tasks. This post argues that this is because of gradient descent -- any stable gradient descent on RNNs can be well approximated by gradient descent on a feedforward model (both at training and inference time).


Another great example of using unsupervised learning to learn good embeddings. Also, a note -- you might wonder why I'm calling this unsupervised learning even though there's a task, with a yes/no answer, a loss function, and an iid dataset, which are hallmarks of supervised learning. The difference is that the labels for the data did not require any human annotation, and we don't care about the actual task that we're learning -- we're after the underlying embeddings that it uses to solve the task. In the previous paper on learning actionable representations, time was used to define an unsupervised learning signal in a similar way.

Objects that Sound

The key idea behind this blog post is that there is a rich source of information in videos -- the alignment between the video frames and audio frames. We can leverage this by creating a proxy task that will force the neural net to learn good representations of the video, which we can then use for other tasks. In particular, we can consider the proxy task of deciding whether a short (~1 second) video clip and audio clip are aligned or not. We don't care about this particular task, but by designing our neural net in the right way, we can ensure that the net will learn good representations of video and audio. We pass the video clip through a convolutional net, the audio clip through another convolutional net, and take the resulting vectors and use the distance between them as a measure of how dissimilar they are. There is no way for video to affect the audio or vice versa before the distance -- so the net is forced to learn to map each of them to a shared space where the distance is meaningful. Intuitively, we would expect that this shared space would have to encode the cause of both the audio and video. Once we have these embeddings (and the neural nets that generate them), we can use them for other purposes. For example, their audio encoder sets the new state-of-the-art on two audio classification benchmarks. In addition, by modifying the video encoder to output embeddings for different regions in the image, we can compute the distance between the audio embedding and the video embedding at each region, and the regions where this is highest correspond to the object that is making the sound.


Neural architecture search is diversifying, focusing on computation time in addition to accuracy now. It seems possible that we'll run into the same problems with architecture search soon, where the reward functions are complex enough that we don't get them right on the first try. What would it look like to learn from human preferences here? Perhaps we could present two models from the search to humans, along with statistics about each, and see which ones the researchers prefer? Perhaps we could run tests on the model, and then have humans provide feedback on the result? Maybe we could use feature visualization to provide feedback on whether the network is learning the "right" concepts?

MnasNet: Towards Automating the Design of Mobile Machine Learning Models

Mobile phones have strong resource constraints (memory, power usage, available compute), which makes it hard to put neural nets on them. Previously, for image classification, researchers hand designed MobileNetV2 to be fast while still achieving good accuracy. Now, using neural architecture search, researchers have found a new architecture, MnasNet, which is 1.5x faster with the same accuracy. Using the [squeeze-and-excitation](https://arxiv.org/abs/1709.01507) optimization improves it even further.


Understanding specific failure modes of large language model generation seems useful, and the detailed mathematical explanation here makes it easier to understand what exactly the problem is, and what we can do to fix it. I'd be interested to see whether we can distinguish delusions from other failure modes and measure what proportion of failures are delusions (although failure modes likely can’t be as cleanly divided as I’m implying here). However, it seems fundamentally very difficult to train using offline data in a way that the model does learn to understand its own actions as interventions, so other solutions may need to be found.

Shaking the foundations: delusions in sequence models for interaction and control

Pedro A. Ortega, Markus Kunesch, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Joel Veness, Jonas Buchli, Jonas Degrave, Bilal Piot, Julien Perolat, Tom Everitt, Corentin Tallec, Emilio Parisotto, Tom Erez, Yutian Chen, Scott Reed, Marcus Hutter, Nando de Freitas, Shane Legg

**Delusions** in language models (LMs) like GPT-3 occur when an incorrect generation early on throws the LM off the rails later. Specifically, if there is some unobserved context that influences how humans generate text that the LM is unaware of, then the LM will generate some plausible text -- and then take that text as _evidence_ about what the unobserved context must be. This can be especially likely when the desired context or task for the generation is difficult to infer from the input. In these settings the human generating the text has access to a lot more information than the model, making generation harder for the model and delusions more likely: an incorrect generation will make it more likely that the model infers the task or context incorrectly. This also applies to sequence modelling approaches in RL like <@Decision Transformer@>(@Decision Transformer: Reinforcement Learning via Sequence Modeling@) and <@Trajectory Transformer@>(@Reinforcement Learning as One Big Sequence Modeling Problem@), where incorrectly chosen actions could change the model's beliefs about optimal future actions.

This work explains this problem using tools from causality and argues that these models should act as if their previous actions are causal interventions rather than observations. However, training a model in this way requires access to a model of the environment and the expert demonstrating trajectories in an online way, and the authors don't describe a way to do this with purely offline data (it may be fundamentally impossible). The authors do argue that in settings where the context or task information can be easily extracted from the observations so far, then delusions are less likely. This points to the importance of prompt engineering, or providing context information in another way to sequence models, so that they don't delude themselves.


ANNPR 2018: Artificial Neural Networks in Pattern Recognition

The conclusions (section 8) contain a nice list of lessons learned from their case studies, emphasizing problems such as the difficulty of getting good data, the importance of reward shaping, etc.

Deep Learning in the Wild

Thilo Stadelmann, Mohammadreza Amirian, Ismail Arabaci, Marek Arnold, Gilbert François Duivesteijn, Ismail Elezi, Melanie Geiger, Stefan Lörwald, Benjamin Bruno Meier, Katharina Rombach, Lukas Tuggener

Describes how deep learning is used to solve real-world problems (eg. in industry).


The Power of Scale for Parameter-Efficient Prompt Tuning

Brian Lester, Rami Al-Rfou, Noah Constant

The highlighted paper showed that prompt programming as currently practiced depends on having a dataset on which prompts can be tested. If we have to use a large dataset anyway, then could we do better by using ML techniques like gradient descent to choose the prompt? Now, since prompts are discrete English sentences, you can't calculate gradients for them, but we know how to deal with this -- the first step of a language model is to _embed_ English words (or syllables, or bytes) into a real-valued vector, after which everything is continuous. So instead of using gradient descent to optimize the English words in the prompt, we instead optimize the embeddings directly. Another way of thinking about this is that we have our "prompt" be a sentence of (say) 50 completely new words, and then we optimize the "meaning" of those words such that the resulting sequence of 50 newly defined words becomes a good prompt for the task of interest.

The authors show that this approach significantly outperforms the method of designing prompts by hand. While it does not do as well as finetuning the full model on the task of interest, the gap between the two decreases as the size of the model increases. At ~10 billion parameters, the maximum size tested, prompt tuning and model tuning are approximately equivalent.

In addition, using a prompt is as simple as prepending the new prompt embedding to your input and running it through your model. This makes it particularly easy to do ensembling: if you have N prompts in your ensemble, then given a new input, you create a batch of size N where the ith element consists of the ith prompt followed by the input, and run that batch through your model to get your answer. (In contrast, if you had an ensemble of finetuned models, you would have to run N different large language models for each input, which can be significantly more challenging.)


This seems cool but I'm not very familiar with this area so I don't have a strong opinion. The algorithm seemed weirdly complicated to me but I think it's based on previous work, and I only spent a couple of minutes looking at it.

Glow: Better Reversible Generative Models

A generative model here means something that models the data distribution, including any underlying structure. For example, a generative model for images would let you generate new images that you hadn't seen during training. While we normally here of GANs and VAEs for current generative models, this work builds on reversible or flow-based generative models. Similarly to word vectors, we can find directions in the learned embedding space corresponding to natural categories (such as "hair color"), and manipulate an image by first encoding to the embedding space, then adding one of these directions, and then decoding it back to the manipulated image.


The experiments in this paper are well-designed. In particular, I found the side-by-side experiment setup to be a clever way to test the ideas presented in the paper. Having said that, the actual results from the experiments are mixed. The paper's strongest results are for the clusterability of pruned networks, while evidence for the clusterability of convolutional networks is quite mixed. However, pruning is not common practice. Additionally, in an intuitive sense, pruning a network seems as though it could be _defined_ in terms of clusterability notions, which limits my enthusiasm for that result.

**Rohin's opinion:** I feel like there are quite a few interesting next questions for the study of modularity in neural networks:

1. Does modularity become more obvious or pronounced as we get to larger models and more complex and realistic data?
2. How well do networks cluster when you are looking for hundreds or thousands of clusters (rather than the 12 clusters considered in this paper)?
3. To what extent is modularity a result of any training, vs. a result of the specific data being trained on? If you train a network to predict random labels, will it be less modular as a result?

One challenge here is in ensuring that the clustering algorithm used is still accurately measuring modularity in these new settings. Another challenge is whether networks are more modular just because in a bigger model there are more chances to find good cuts. (In other words, what's the default to which we should be comparing?) For example, the paper does present some results with models trained on ImageNet, and ResNet-50 gets one of the highest clusterability numbers in the paper. But the authors do mention that the clustering algorithm was less stable, and it's not clear how exactly to interpret this high clusterability number.

Clusterability in Neural Networks

Daniel Filan*, Stephen Casper*, Shlomi Hod*, Cody Wild, Andrew Critch, Stuart Russell

Neural networks are often construed as lacking internal structure. In this paper, the authors challenge the predominant view and hypothesize that neural networks are more clusterable than is suggested by chance. To investigate the claim, the authors partition the network into groups where most of the edge weight is between neurons in the same group. The authors find that the quality of these groups improves after training, as compared to randomly initialized networks. However, this only holds for certain training setups. Despite this limitation, the authors show it's possible to promote clusterability with little to no effect on accuracy.

In experiments, the authors compare the clusterability of trained networks to randomly initialized networks and trained networks with shuffled weights. They focus on multi-layer perceptrons (MLPs) and convolutional networks with dropout regularization. They also run experiments with pruned networks or networks where 'unimportant' edges are removed. They find that MLP networks have clusterable neurons at rates higher than chance, but have mixed results for convolutional networks.

The authors hypothesize that clusterability is more likely to arise when different features of the input can be computed in parallel without communication between the features (which is very similar to the hypothesis in the previous paper). To test the hypothesis, they combine examples from the datasets into pairs and then train the neural network to make a double-prediction in a side-by-side setup. Intuitively, the network would need to look at each pair separately, without any need to combine information across the two sides. They find that this setup results in increased modularity.


Weight Banding

Michael Petrov, Chelsea Voss, Ludwig Schubert, Nick Cammarata, Gabriel Goh, Chris Olah

Empirically, when training neural networks on ImageNet, we can commonly observe "weight banding" in the final layer. In other words, the neurons in the final layer pay very strong attention to the vertical position of features, and ignore the horizontal position of features. This holds across InceptionV1, ResNet50, and VGG19, though it doesn't hold for AlexNet.

If you rotate the training data by 90 degrees, then the phenomenon changes to have vertical striping, that is, we now pay strong attention to the horizontal position of features. This suggests that this phenomenon is being driven somehow by the ImageNet data.

The authors hypothesize that this is caused by the neural network needing to recover some spatial information that was reduced by the previous average pooling layer (which is not present in AlexNet). They try removing this layer, which causes the effect to go away in Inception, but not in VGG19. They seem to think that it also goes away in ResNet50, but when I look at the results, it seems like the phenomenon is still there (though not as strongly as before).

They try a bunch of other architectural interventions on a simplified architecture and find that weight banding persists across all of these.