Previously I believed that good chatbots would be hard to build, since it is challenging to find large datasets of high-quality published conversations. Given the very large dataset that the researchers were able to find, I no longer think this is a major barrier for chatbots. It's important to note that this result does not imply that a strong Turing test will soon be passed: the authors themselves note that SSA overestimates the abilities of Meena relative to humans. Since humans are often vague in their conversations, evaluating human conversation with SSA yields a relatively low score. Furthermore, a strong Turing test would involve a judge asking questions designed to trip AI systems, and we are not yet close to a system that could fool such judges.

Towards a Human-like Open-Domain Chatbot

Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, Quoc V. Le

This paper presents a chatbot called Meena that reaches near human-level performance for measures of human likeness. The authors mined social media to find 341 GB of public domain conversations, and trained an [evolved transformer](https://arxiv.org/abs/1901.11117) on those conversations. To test its performance, they devised a metric they call Sensibility and Specificity (SSA) which measures how much sense the chatbot's responses make in context, as well as whether they were specific. SSA was tightly correlated with perplexity and a subjective measure of human likeness, suggesting that optimizing for perplexity will translate to greater conversational ability. Meena substantially improved on the state of the art, including both hand-crafted bots like [Mitsuku](https://en.wikipedia.org/wiki/Mitsuku) and the neural model [DialoGPT](https://arxiv.org/abs/1911.00536), though it still falls short of human performance. You can read some conversation transcrips [here](https://github.com/google-research/google-research/blob/master/meena/meena.txt); many of the responses from Meena are very human-like.

See also [Import AI #183](https://jack-clark.net/2020/02/03/import-ai-183-curve-fitting-conversation-with-meena-gans-show-us-our-climate-change-future-and-what-compute-data-arbitrage-means/)


ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring

David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, Colin Raffel

A common criticism of deep learning is that it requires far too much training data. Some view this as a fundamental flaw that suggests we need a new approach. However, considerable data efficiency is possible with a new technique called ReMixMatch. ReMixMatch on CIFAR-10 obtains 84.92% accuracy using only 4 labeled examples per class. Using 250 labeled examples, or around 25 labeled examples per class, a ReMixMatch model on CIFAR-10 has 93.73% accuracy. This is approximately how well a vanilla ResNet does on CIFAR-10 with 50000 labeled examples. Two years ago, special techniques utilizing 250 CIFAR-10 labeled examples could enable an accuracy of approximately [53%](https://paperswithcode.com/sota/semi-supervised-image-classification-on-3). ReMixMatch builds on [MixMatch](https://arxiv.org/abs/1905.02249) and has several seemingly arbitrary design decisions, so I will refrain from describing its design. In short, deep networks do not necessarily require large labeled datasets.

And just yesterday, after this summary was first written, the [FixMatch](https://arxiv.org/abs/2001.07685) paper got even better results.


While I previously didn't think that double descent was a real phenomenon (see summaries later in this email for details), these experiments convinced me that I was wrong and in fact there is something real going on. Note that the settings studied in this work are still not fully representative of typical use of neural nets today; the label noise is the most obvious difference, but also e.g. ResNets are usually trained with higher widths than studied in this paper. So the phenomenon might not generalize to neural nets as used in practice, but nonetheless, there's _some_ real phenomenon here, which flies in the face of all of my intuitions.

The authors don't really suggest an explanation; the closest they come is speculating that at the interpolation threshold there's only ~one model that can fit the data, which may be overfit, but then as you increase further the training procedure can "choose" from the various models that all fit the data, and that "choice" leads to better generalization. But this doesn't make sense to me, because whatever is being used to "choose" the better model applies throughout training, and so even at the interpolation threshold the model should have been selected throughout training to be the type of model that generalized well. (For example, if you think that regularization is providing a simplicity bias that leads to better generalization, the regularization should also help models at the interpolation threshold, since you always regularize throughout training.)

Perhaps one explanation could be that in order for the regularization to work, there needs to be a "direction" in the space of model parameters that doesn't lead to increased training error, so that the model can move along that direction towards a simpler model. Each training data point defines a particular direction in which training error will increase. So, when the number of training points is equal to the number of parameters, the training points just barely cover all of the directions, and then as you increase the number of parameters further, that starts creating new directions that are not constrained by the training points, allowing the regularization to work much better. (In fact, the [original paper](https://arxiv.org/abs/1812.11118), summarized below, _defined_ the interpolation threshold as the point where number of parameters equals the size of the training dataset.) However, while this could explain model-wise double descent and training-set-size double descent, it's not a great explanation for epoch-wise double descent.

Paper: Deep Double Descent: Where Bigger Models and More Data Hurt

Deep Double Descent

Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, Ilya Sutskever

This blog post provides empirical evidence for the existence of the _double descent_ phenomenon, proposed in an earlier paper summarized below. Define the _effective model complexity_ (EMC) of a training procedure and a dataset to be the maximum size of training set such that the training procedure achieves a _train_ error of at most ε (they use ε = 0.1). Let's suppose you start with a small, underparameterized model with low EMC. Then initially, as you increase the EMC, the model will achieve a better fit to the data, leading to lower test error. However, once the EMC is approximately equal to the size of the actual training set, then the model can "just barely" fit the training set, and the test error can increase or decrease. Finally, as you increase the EMC even further, so that the training procedure can easily fit the training set, the test error will once again _decrease_, causing a second descent in test error. This unifies the perspectives of statistics, where larger models are predicted to overfit, leading to increasing test error with higher EMC, and modern machine learning, where the common empirical wisdom is to make models as big as possible and test error will continue decreasing.

They show that this pattern arises in a variety of simple settings. As you increase the width of a ResNet up to 64, you can observe double descent in the final test error of the trained model. In addition, if you fix a large overparameterized model and change the number of epochs for which it is trained, you see another double descent curve, which means that simply training longer can actually _correct overfitting_. Finally, if you fix a training procedure and change the size of the dataset, you can see a double descent curve as the size of the dataset decreases. This actually implies that there are points in which _more data is worse_, because the training procedure is in the critical interpolation region where test error can increase. Note that most of these results only occur when there is _label noise_ present, that is, some proportion of the training set (usually 10-20%) is given random incorrect labels. Some results still occur without label noise, but the resulting double descent peak is quite small. The authors hypothesize that label noise leads to the effect because double descent occurs when the model is misspecified, though it is not clear to me what it means for a model to be misspecified in this context.


I'm very interested in this problem, and this post does a great job of introducing it and summarizing some of the recent work. I especially appreciated the attempts at reproducing the results.

On the papers themselves, a regime where you already have ~zero training error but validation error goes _down_ as you increase model expressivity is exceedingly strange. Skimming the paper, it seems that the idea is that in the normal ML regime, you are only minimizing training error -- but once you can get the training error to zero, you can then optimize for the "simplest" model with zero training error, which by Occam's Razor-style arguments should be the best one and lead to better validation performance. This makes sense in the theoretical model that they use, but it's not clear to me how this applies to neural nets, where you aren't explicitly optimizing for simplicity after getting zero training error. (Techniques like regularization don't result in one-after-the-other optimization -- you're optimizing for both simplicity and low training error simultaneously, so you wouldn't expect this critical point at which you enter a new regime.) So I still don't understand these results. That said, given the difficulty with reproducing them, I'm not going to put too much weight on these results now.

I tried to predict the results of the last paper and correctly predicted that randomizing would always destroy performance, but predicted that resetting to initialization would be okay for _early_ layers instead of later layers. I had a couple of reasons for the wrong prediction. First, there had been a few papers that showed good results even with random features, suggesting the initial layers aren't too important, and so maybe don't get updated too much. Second, the gradient of the loss w.r.t later layers requires only a few backpropagation steps, and so probably provides a clear, consistent direction moving it far away from the initial configuration, while the gradient w.r.t earlier layers factors through the later layers which may have weird or wrong values and so might push in an unusual direction that might get cancelled out across multiple gradient updates. I skimmed the paper and it doesn't really speculate on why this happens, and my thoughts still seem reasonable to me, so this is another fact that I have yet to explain.

Are Deep Neural Networks Dramatically Overfitted?

The concepts of underfitting and overfitting, and their relation to the bias-variance tradeoff, are fundamental to standard machine learning theory. Roughly, for a fixed amount of data, there is an optimal model complexity for learning from that data: any less complex and the model won't be able to fit the data, and any more complex and it will overfit to noise in the data. This means that as you increase model complexity, training error will go down to zero, but validation error will go down and then start turning back up once the model is overfitting.

We know that neural networks are much more expressive than the theory would predict is optimal, both from theorems showing that neural networks can learn any function (including one that provides a rather tight bound on number of parameters), as well as a [paper](https://arxiv.org/abs/1611.03530) showing that neural nets can learn random noise. Yet they work well in practice, achieving good within-distribution generalization.

The post starts with a brief summary of topics that readers of this newsletter are probably familiar with: Occam's razor, the Minimum Description Length principle, Kolmogorov Complexity, and Solomonoff Induction. If you don't know these, I strongly recommend learning them if you care about understanding within-distribution generalization. The post then looks at a few recent informative papers, and tries to reproduce them.

The [first one](https://arxiv.org/abs/1812.11118) is the most surprising: they find that as you increase the model complexity, your validation error goes down and then back up, as expected, but then at some point it enters a new regime and goes down again. However, the author notes that you have to set up the experiments just right to get the smooth curves the paper got, and her own attempts at reproducing the result are not nearly as dramatic.

Another [paper](https://arxiv.org/abs/1804.08838) measures the difficulty of a task based on its "intrinsic dimension", which Cody has summarized separately in this newsletter.

The [last paper](https://arxiv.org/abs/1902.01996) looks at what happens if you (a) reset some layer's parameters to the initial parameters and (b) randomize some layer's parameters. They find that randomizing always destroys performance, but resetting to initial parameters doesn't make much of a difference for later layers, while being bad for earlier layers. This was easy to reproduce, and the findings reemerge very clearly.


These are some pretty impressive results. I'm surprised that all of this came from a single order of magnitude more data and model size, I would have expected it to take more than that. I think this lends a lot of support to the hypothesis that unsupervised learning with sufficient amounts of compute and diverse data can lead to generally capable AI systems. (See this [SlateStarCodex post](https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-general-intelligence/) for a more detailed version of this take.) This is also some evidence that we will have AI systems that can pass the Turing Test before we have general AI systems, that is, the Turing Test is not AI-complete.

Language Models are Unsupervised Multitask Learners

Better Language Models and Their Implications

Alec Radford, Jeffrey Wu, Dario Amodei, Daniela Amodei, Jack Clark, Miles Brundage,and Ilya Sutskever

OpenAI has trained a scaled up GPT model using unsupervised learning (specifically, predicting the next word given a very large context) on a very large dataset with presumably very large compute. The resulting language model can produce impressive language samples (with some cherry-picking) that to my eye are particularly good at handling long-range dependencies, which makes sense since it is based on the [Transformer](https://arxiv.org/abs/1706.03762) (see Transformer-XL entry in [AN #44](https://mailchi.mp/6bfac400a0c3/alignment-newsletter-44)). It sets new state of the art performance on 7 out of 8 language modeling tasks, including difficult datasets such as [LAMBADA](https://arxiv.org/abs/1606.06031), _without using the training data for those tasks_. It can also be used for more structured tasks by providing a particular context -- for example, to summarize a document, you can provide the document followed by "TL;DR:" in order to induce GPT-2 to "predict" a summary. (They use a different prediction algorithm in order to improve summarization results, but I suspect even with regular prediction you'd get something in the right ballpark.) On these more structured tasks, it doesn't get anywhere near the state of the art set by specialized systems -- but again, this is without any finetuning for the specific task that we are testing.

The [paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf) argues that in order to get generally capable AI systems, we will need to train them on many different tasks, as in meta-learning. However, we might expect that we need hundreds of thousands of tasks in order to learn something general, just as we need hundreds of thousands of examples in order to develop good classifiers. Prediction of the next word in natural language is particularly good for this, because in order to predict well across a huge variety of text, you need to become good at many different tasks such as question answering, summarization, and even translation. The biggest challenge is in creating a dataset that has sufficient diversity -- they do this by scraping all outbound links from Reddit with at least 3 karma.

Unusually for research, but in accordance with [its charter](https://blog.openai.com/openai-charter/) ([AN #2](https://mailchi.mp/14782876a85d/alignment-newsletter-2)), OpenAI has decided not to release the model publicly, citing the possibility of malicious uses of the model. This has been controversial, with the debate raging for days on Twitter. I haven't paid enough attention to the debate to give a reasonable summary so you'll have to rely on other sources for that.


It's quite interesting that even though the point of Transformer was to get away from recurrent structures, adding them back in leads to significant improvements. Of course, the recurrent structure is now at the higher level of segments, rather than at the word or character level. This reminds me a lot of hierarchy -- it seems like we're using the Transformer as a basic building block that works on the ~sentence level so that our RNN-like structure can deal with a higher level of abstraction (which of course also helps with vanishing/exploding gradients).

There's an interesting pattern where hierarchy and structure seem to be a good inductive bias, that let you get good performance with limited compute and data, but as those limits subside, you're better off doing something that has less bias. This would predict that as we get more data and compute, we would want larger Transformer models (i.e. longer segments) and less recurrence. It would be interesting to see if that actually holds.

Transformer-XL: Unleashing the Potential of Attention Models

Zihang Dai, Zhilin Yang and
Quoc Le, Google AI

[Transformer](https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html) architectures have become all the rage recently, showing better performance on many tasks compared to CNNs and RNNs. This post introduces Transformer-XL, an improvement on the Transformer architecture for very long sequences.

The key idea with the original Transformer architecture is to use self-attention layers to analyze sequences instead of something recurrent like an RNN, which has problems with vanishing and exploding gradients. An attention layer takes as input a query q and key-value pairs (K, V). The query q is "compared" against every key k, and that is used to decide whether to return the corresponding value v. In their particular implementation, for each key k, you take the dot product of q and k to get a "weight", which is then used to return the weighted average of all of the values. So, you can think of the attention layer as taking in a query q, and returning the "average" value corresponding to keys that are "similar" to q (since dot product is a measure of how aligned two vectors are). Typically, in an attention layer, some subset of Q, K and V will be learned. With _self-attention_, Q, K and V are all sourced from _the same place_ -- the result of the previous layer (or the input if this is the first layer). Of course, it's not exactly the output from the previous layer -- if that were the case, there would be no parameters to learn. They instead learn three _linear projections_ (i.e. matrices) that map from the output of the previous layer to Q, K and V respectively, and then feed the generated Q, K and V into a self-attention layer to compute the final output. And actually, instead of having a single set of projections, they have multiple sets that each contain three learned linear projections, that are all then used for attention, and then combined together for the next layer by another learned matrix. This is called _multi-head attention_.

Of course, with attention, you are treating your data as a set of key-value pairs, which means that the order of the key value pairs does not matter. However, the order of words in a sentence is obviously important. To allow the model to make use of position information, they augment each word and add position information to it. You could do this just by literally appending a single number to each word embedding representing its absolute position, but then it would be hard for the neural net to ask about a word that was "3 words prior". To make this easier for the net to learn, they create a vector of numbers to represent the absolute position based on sinusoids such that "go back 3 words" can be computed by a linear function, which should be easy to learn, and add _(not concatenate!)_ it elementwise to the word embedding.

This model works great when you are working with a single sentence, where you can attend over the entire sentence at once, but doesn't work as well when you are working with eg. entire documents. So far, people have simply broken up documents into segments of a particular size N and trained Transformer models over these segments. Then, at test time, for each word, they use the past N - 1 words as context and run the model over all N words to get the output. This cannot model any dependencies that have range larger than N. The Transformer-XL model fixes this issue by taking the segments that vanilla Transformers use, and adding recurrence. Now, in addition to the normal output predictions we get from segments, we also get as output a new hidden state, that is then passed in to the next segment's Transformer layer. This allows for arbitrarily far long-range dependencies. However, this screws up our position information -- each word in each segment is augmented with _absolute_ position information, but this doesn't make sense across segments, since there will now be multiple words at (say) position 2 -- one for each segment. At this point, we actually want _relative_ positions instead of absolute ones. They show how to do this -- it's quite cool but I don't know how to explain it without going into the math and this has gotten long already. Suffice it to say that they look at the interaction between arbitrary words x_i and x_j, see the terms that arise in the computation when you add absolute position embeddings to each of them, and then change the terms so that they only depend on the difference j - i, which is a relative position.

This new model is state of the art on several tasks, though I don't know what the standard benchmarks are here so I don't know how impressed I should be.


This seems like an important advance in meta-learning, as it is much more computationally efficient than MAML while still achieving similar levels of performance.

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

Reptile: A Scalable Meta-Learning Algorithm

I somehow forgot to include this in past emails, so I'm including it now. Reptile is an algorithm for meta-learning, and in this paper is applied to few-shot classification, where given a few examples of different classes, you must learn a good classification algorithm for those classes. The authors show using a Taylor expansion that [MAML](https://arxiv.org/abs/1703.03400) and Reptile have very similar gradients to first order in alpha, the step size. Their evaluation shows that for the few-shot classification case, Reptile and MAML perform similarly (though they do not evaluate on reinforcement learning tasks, as in the MAML paper).


This paper is an excellent holistic discussion of graph networks and reasons to think they are promising. I'm glad that it also mentioned the open problems, though, since I think they're pretty crucial to using graphs in deep learning, and current approaches in this area (e.g. capsule networks' dynamic control flow) aren't satisfactory.

Relational inductive biases, deep learning, and graph networks

Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, Razvan Pascanu

"Part position paper, part review, and part unification", this paper emphasises the importance of combinatorial generalisation, which is key to how humans understand the world. It argues for approaches which perform computation over discrete entities and the relations between them, such as graph networks. The authors claim that CNNs and RNNs are so successful due to relational inductive biases - for example, the bias towards local structure induced by convolutional layers. Graph networks are promising because they can express arbitrary relational biases: any nodes can be connected with any others depending on the structure of the problem. Further, since graph networks learn functions which are reused for all nodes and edges, each one can be applied to graphs of any shape and size: a form of combinatorial generalisation.

In this paper's framework, each 'graph block' does computations over an input graph and returns an output graph. The relevant part of the output might be the values of edges, or those of nodes, or 'global' properties of the overall graph. Graph blocks can be implemented by standard neural network architectures or more unusual ones such as message-passing neural networks or non-local neural networks. The authors note some major open questions: how to generate the graphs in the first place, and how to adaptively modify them during the course of computation.


I enjoyed this paper a lot; it feels like it gave me a good understanding of the programming abilities of large language models.

I was most surprised by the result that, for the synthesis task, finetuning on execution helps but finetuning on synthesis doesn’t help for the 137B model. It is possible that this is just noise, though that is more noise than I would expect for such an experiment. It could be that the finetuning dataset for synthesis was too small (it only contains 374 problems), but that dataset was sufficient for big gains on the smaller models, and I would expect that, if anything, larger models should be able to make _better_ use of small finetuning datasets, not worse.

It’s also notable that, for the 137B model, the knowledge gained from finetuning on execution successfully transferred to improve synthesis performance. While I agree that the poor execution performance implies the model doesn’t “understand” the code according to the normal usage of that term, it seems like this sort of transfer suggests a low but non-zero level on some quantitative scale of understanding.

I also found the human feedback section quite cool. However, note that the human providing the feedback often needs to understand the generated code as well as the desired algorithm, so it is plausible that it would be easier for the human to simply fix the code themselves.

Program Synthesis with Large Language Models

Jacob Austin*, Augustus Odena*, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, Charles Sutton

Can we use large language models to solve programming problems? In order to answer this question, this paper builds the Mostly Basic Python Programming (MBPP) dataset. The authors asked crowd workers to provide a short problem statement, a Python function that solves the problem, and three test cases checking correctness. On average across the 974 programs, the reference solution has 7 lines of code, suggesting the problems are fairly simple. (This is partly because you can use library functions.) They also edit a subset of 426 problems to improve their quality, for example by making the problem statement less ambiguous or making the function signature more normal.

They evaluate pretrained language models on this dataset across a range of model sizes from 0.244B to 137B parameters. (This largest model is within a factor of 2 of GPT-3.) They consider both few-shot and finetuned models. Since we have test cases that can be evaluated automatically, we can boost performance by generating lots of samples (80 in this case), evaluating them on the test cases, and then keeping the ones that succeed. They count a problem as solved if any sample passes all the test cases, and report as their primary metric the fraction of problems solved according to this definition. Note however that the test cases are not exhaustive: when they wrote more exhaustive tests for 50 of the problems, they found that about 12% of the so-called "solutions" did not pass the new tests (but conversely, 88% did). They also look at the fraction of samples which solve the problem, as a metric of the reliability or confidence of the model for a given problem.

Some of their findings:
1. Performance increases approximately log-linearly with model size. The trend is clearer and smoother by the primary metric (fraction of problems solved by at least one sample) compared to the secondary metric (fraction of samples that solve their problem).
2. Finetuning provides a roughly constant boost across model sizes. An exception: at the largest model size, finetuning provides almost no benefit, though this could just be noise.
3. It is important to provide at least one test case to the model (boosts problems solved from 43% to 55%) but after that additional test cases don't make much of a difference (an additional two examples per problem boosts performance to 59%).
4. In few-shot learning, the examples used in the prompt matter a lot. In a test of 15 randomly selected prompts for the few-shot 137B model, the worst one got ~1%, while the best one got ~59%, with the others distributed roughly uniformly between them. Ensembling all 15 prompts boosts performance to 66%.
5. In rare cases, the model overfits to the test cases. For example, in a question about checking whether the input is a Woodall number, there is only one test checking an actual Woodall number (383), and the model generates a program that simply checks whether the input is 383.
6. When choosing the best of multiple samples, you want a slightly higher temperature, in order to have more diversity of possible programs to check.
7. It is important to have high quality problem descriptions as input for the model. The 137B model solves 79% of problems in the edited dataset, but only solves 63% of the original (unedited) versions of those problems. The authors qualitatively analyze the edits on the problems that switched from unsolved to solved and find a variety of things that you would generally expect to help.

Now for the controversial question everyone loves to talk about: does the model _understand_ the meaning of the code, or is it "just learning statistical correlations"? One way to check this is to see whether the model can also _execute_ code. Specifically, we provide the ground truth code for one of the problems in the MBPP dataset along with one of the test case inputs and ask the model to predict the output for that test case. Even after finetuning for this task, the 137B model gets only 21% right. This can be boosted to 27% by also providing example test cases for the code before predicting the output for a new test case. Overall, this suggests that the model doesn't "understand" the code yet.

We can take the model finetuned for execution and see how well it does on program synthesis. (We can do this because there are different prompts for execution and synthesis.) For the 8B model, the finetuning makes basically no difference: it's equivalent to the original few-shot setting. However, for the 137B model, finetuning on execution actually leads to a small but non-trivial improvement in performance (from ~59% to ~63%, I think). This is true relative to either the few-shot or finetuned-for-synthesis setting, since they performed near-identically for the 137B model. So in fact the 137B model finetuned on execution is actually the strongest model, according to synthesis performance.

So far we've just been looking at how our model performs when taking the best of multiple samples. However, if our goal is to actually use models for program synthesis, we aren't limited to such simple tricks. Another approach is to have a human provide _feedback_ in natural language when the model's output is incorrect, and then have the model generate a new program. This feedback is very informal, for example, "Close, but you need to replace the underscore with an empty string". This provides a huge performance boost: the 137B solves ~31% of problems on its first sample; adding just a single piece of human feedback per problem boosts performance to ~55%, and having four rounds of human feedback gets you to over 65%.

The authors also introduce the MathQA-Python dataset, which provides arithmetic word problems and asks models to write programs that would output the correct answer to the problem. They only run a few experiments on this dataset, so I've mostly ignored it. The main upshot is that a finetuned 137B parameter model can solve 83.8% of problems with _some_ sample. They don't report metrics with a single sample, which seems like the more relevant metric for this dataset, but eyeballing other graphs I think it would be around 45%, which you could probably boost a little bit by decreasing the sampling temperature.


I haven't explored it yet, but this seems great, especially if you want to learn ML. I have used Colab notebooks before and recommend them highly for small projects (maybe even large ones, I'm not sure), especially if you're familiar with Jupyter notebooks.

Seedbank — discover machine learning examples

Seedbank provides interactive machine learning examples in Colab notebooks (think Jupyter notebooks in the cloud). This makes it really easy to just run example code without any setup, and even to modify it to play around with it. Google even provides a free GPU to make the training and inference faster!


I really liked the experiment demonstrating misalignment, as it seems like it accurately captures the aspects that we expect to see with existentially risky misaligned AI systems: they will “know” how to do the thing we want, they simply won’t be “motivated” to actually do it.

Evaluating Large Language Models Trained on Code

Mark Chen*, Jerry Tworek*, Heewoo Jun*, Qiming Yuan*, Henrique Ponde*, Jared Kaplan*, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, Will Guss, Alex Nichol, Igor Babuschkin, Suchir Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, Wojciech Zaremba

You've probably heard of GitHub Copilot, the programming assistant tool that can provide suggestions while you are writing code. This paper evaluates Codex, a precursor to the model underlying Copilot. There's a lot of content here; I'm only summarizing what I see as the highlights.

The core ingredient for Codex was the many, many public repositories on GitHub, which provided hundreds of millions of lines of training data. With such a large dataset, the authors were able to get good performance by training a model completely from scratch, though in practice they finetuned an existing pretrained GPT model as it converged faster while providing similar performance.

Their primary tool for evaluation is HumanEval, a collection of 164 hand-constructed Python programming problems where the model is provided with a docstring explaining what the program should do along with some unit tests, and the model must produce a correct implementation of the resulting function. Problems are not all equally difficult; an easier problem asks Codex to "increment all numbers in a list by 1" while a harder one provides a function that encodes a string of text using a transposition cipher and asks Codex to write the corresponding decryption function.

To improve performance even further, they collect a sanitized finetuning dataset of problems formatted similarly to those in HumanEval and train Codex to perform well on such problems. These models are called Codex-S. With this, we see the following results:

1. Pretrained GPT models get roughly 0%.
2. The largest 12B Codex-S model succeeds on the first try 29% of the time. (A Codex model of the same size only gets roughly 22%.)
3. There is a consistent scaling law for reduction in loss. This translates into a less consistent graph for performance on the HumanEval dataset, where once the model starts to solve at least (say) 5% of the tasks, there is a roughly linear increase in the probability of success when doubling the size of the model.
4. If instead we generate 100 samples and check whether they pass the unit tests to select the best one, then Codex-S gets 78%. If we still generate 100 samples but select the sample that has the highest mean log probability (perhaps because we don't have an exhaustive suite of unit tests), then we get 45%.

They also probe the model for bad behavior, including misalignment. In this context, they define misalignment as a case where the user wants A, but the model outputs B, and the model is both capable of outputting A and capable of distinguishing between cases where the user wants A and the user wants B.

Since Codex is trained primarily to predict the next token, it has likely learned that buggy code should be followed by more buggy code, that insecure code should be followed by more insecure code, and so on. This suggests that if the user accidentally provides examples with subtle bugs, then the model will continue to create buggy code, even though the user would want correct code. They find that exactly this effect occurs, and that the divergence between good and bad performance _increases_ as the model size increases (presumably because larger models are better able to pick up on the correlation between previous buggy code and future buggy code).


In my view, the scaling hypothesis is easily the most important hypothesis relevant to AI forecasting and AI development models, and this is the best public writeup of it that I know of. (For example, it seems to be an implicit assumption in the <@bio anchors framework@>(@Draft report on AI timelines@).) I broadly agree with the author that it’s a bit shocking how few people seem to be taking it seriously after OpenAI Five, AlphaStar, GPT-3, Copilot, etc.

I *think* this includes the AI safety space, where as far as I can tell the primary effect has been that it is even more fashionable to have shorter timelines, whereas it hasn’t affected AI safety research very much. However, I do know around 3-4 researchers who changed what they were working on based on changing their mind about the scaling hypothesis, so it’s possible there are several others I don’t know about.

As a simple example of how the scaling hypothesis affects AI safety research, it suggests that the training objective (“predict the next word”) is relatively unimportant in determining properties of the trained agent; in contrast, the dataset is much more important. This suggests that analyses based on the “reward function used to train the agent” are probably not going to be very predictive of the systems we actually build.

The Scaling Hypothesis

This post centers around the **scaling hypothesis**:

_Once we find a scalable architecture which can be applied fairly uniformly, we can simply train ever larger networks and ever more sophisticated behavior will emerge naturally as the easiest way to optimize for all the tasks and data. More powerful NNs are "just" scaled-up weak NNs, in much the same way that human brains look much like scaled-up primate brains._

Importantly, we can get this sophisticated behavior just by training on simple objectives, such as "predict the next word", as long as the data is sufficiently diverse. So, a priori, why might we expect the scaling hypothesis to be true?

The core reason is that optimal (or human-level) prediction of text really does require knowledge, reasoning, causality, etc. If you don't know how to perform addition, you are probably not going to be able to predict the next word in the sentence "Though he started with six eggs, he found another fourteen, bringing his total to \_\_\_\_". However, since any specific fact is only useful in a tiny, tiny number of cases, it only reduces the expected loss by a tiny amount. So, you'll only see models learn this sort of behavior once they have exhausted all the other "easy wins" for predicting text; this will only happen when the models and dataset are huge.

Consider a model tasked with predicting characters in text with a set of 64 characters (52 uppercase and lowercase letters, along with some punctuation). Initially it outputs random characters, assigning a probability of 1/64 to the correct character, resulting in a loss of 6 bits. Once you start training, the easiest win is to simply notice how frequent each character is; just noticing that uppercase letters are rare, spaces are common, vowels are common, etc. could get your error down to 4-5 bits. After this, it might start to learn what words actually exist; this might take 10^5 - 10^6 samples since each word is relatively rare and there are thousands of words to learn, but this is a drop in the bucket given our huge dataset. After this step, it may have also learned punctuation along the way, and might now be down to 3-4 bits. At this point, if you sample from the model, you might get correctly spelled English words, but they won't make any sense.

With further training the model now has to pick up on associations between adjacent words to make progress. Now it needs to look at things 10 characters ago to predict the next character -- a far cry from our initial letter frequencies where it didn't even need to look at other characters! For example, it might learn that "George W" tends to be followed by "ashington". It starts to learn grammar, being able to correctly put verbs in relation to subjects and objects (that are themselves nouns). It starts to notice patterns in how words like "before" and "after" are used; these can then be used to better predict words in the future; at this point it's clear that the model is starting to learn semantics. Now the loss is around 2 bits per character. A little more training and your model starts to produce sentences that sound human-like in isolation, but don't fit together: a model might start a story about a very much alive protagonist and then talk about how she is dead in the next sentence. Training is now about fixing errors like these and each such fix gains a tiny amount of accuracy -- think ten thousandths of a bit. Every further 0.1 bits you gain represents the model learning a huge amount of relevant knowledge (and correspondingly each subsequent 0.1 bits takes a much larger amount of training and data). The final few fractions of a bit are the most important and comprise most of what we call "intelligence".

(The human baseline is a loss of 0.7 bits, with lots of uncertainty on that figure.)

So far this is a clever argument, but doesn't really establish that this will work _in practice_ -- for example, maybe your model has to have 10^100 parameters to learn all of this, or maybe existing models and algorithms are not sophisticated enough to _find_ the right parameters (and instead just plateau at, say, 2 bits of loss). But recent evidence provides strong support for the scaling hypothesis:

1. The <@scaling laws@>(@Scaling Laws for Neural Language Models@) line of work demonstrated that models could be expected to reach the interesting realm of loss at amounts of compute, data, and model capacity that seemed feasible in the near future.
2. Various projects have trained large models and demonstrated that this allows them to solve tasks that they weren't explicitly trained for, often in a more human-like way and with better performance than a more supervised approach. Examples include <@GPT-3@>(@Language Models are Few-Shot Learners@), [Image GPT](https://openai.com/blog/image-gpt/), [BigGAN](https://arxiv.org/abs/1809.11096), <@AlphaStar@>(@AlphaStar: Grandmaster level in StarCraft II using multi-agent reinforcement learning@), etc. (The full post has something like 25 examples.)

The author then argues that it seems like most researchers seem to be completely ignoring this phenomenon. OpenAI is the only actor that really has the conviction needed to put a large amount of resources behind a project based on the scaling hypothesis (such as GPT-3); DeepMind seems to believe in a weaker version where we need to build a bunch of "modules" similar to those in the human brain, but that those modules can then be scaled up indefinitely. Other actors seem to not take either scaling hypothesis very seriously.


The phenomenon in this paper is pretty broadly applicable to any setting in which a real-world problem is studied in a toy domain where there is extra information available. For example, one of my projects at Berkeley involves using imitation learning on tasks where there really isn’t any reward function available, and it’s quite informative to see just how much it slows you down when you can’t just look at how much reward your final learned policy gets; research becomes much more challenging to do. This suggests that performance on existing imitation learning benchmarks is probably overstating how good we are at imitation learning, because the best models in these situations were probably validated based on the final reward obtained by the policy, which we wouldn’t normally have access to.

True Few-Shot Learning with Language Models

We can get <@GPT-3@>(@Language Models are Few-Shot Learners@) to perform useful tasks using "prompt programming", in which we design an input sentence such that the most likely continuation of that sentence would involve GPT-3 performing the task of interest. For example, to have GPT-3 answer questions well, we might say something like "The following is a transcript of a dialogue with a helpful, superintelligent, question-answering system:", followed by a few example question-answer pairs, after which we ask our questions.

Since the prompts only contain a few examples, this would seem to be an example of strong _few-shot learning_, in which an AI system can learn how to do a task after seeing a small number of examples of that task. This paper contends that while GPT-3 is capable of such few-shot learning, the results reported in various papers exaggerate this ability. Specifically, while it is true that the prompt only contains a few examples, researchers often tune their choice of prompt by looking at how well it performs on a relatively large validation set -- which of course contains many examples of performing the task, something we wouldn't expect to have in a true few-shot learning context.

To illustrate the point, the authors conduct several experiments where we start with around 12 possible prompts and must choose which to use based only on the examples given (typically 5). They test two methods for doing so:

1. Cross-validation: Given a prompt without examples, we attach 4 of the examples to the prompt and evaluate it on the last example, and average this over all possible ways of splitting up the examples.
2. Minimum description length: While cross-validation evaluates the final generalization loss on the last example after updating on previous examples, MDL samples an ordering of the examples and then evaluates the average generalization loss as you feed the examples in one-by-one (so more like an online learning setup).

On the LAMA-UHN task, the difference between a random prompt and the best prompt looks to be roughly 5-6 percentage points, regardless of model size. Using MDL or cross-validation usually gives 20-40% of the gain, so 1-2 percentage points. This suggests that on LAMA-UHN, typical prompt-based "few-shot" learning results are likely 3-5 percentage points higher than what you would expect if you were in a true few-shot setting where there is no validation set to tune on. This is all on average across tasks -- for any given task, you could recover the entire gain, or you might select a prompt that is actually worse than what you'd get from random chance.

But it may actually be worse than that. We've talked just about the prompt so far, but the validation set can also be used to improve hyperparameters, network architecture, the design of the learning algorithm etc. This could also lead to inflated results. The authors conduct one experiment with ADAPET on SuperGLUE which suggests that using the validation set to select hyperparameters can also lead to multiple percentage points of inflation.


I really enjoy Distill articles, they are consistently far more readable and understandable than typical papers (or even blog posts), even without including the interactive visualizations. This article is no exception. I didn't have particularly strong opinions on how to condition neural nets before, but now I think I will think about FiLM and how it could apply.

Feature-wise transformations

Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron Courville and Yoshua Bengio

This Distill article is about transformations on features using FiLM (feature-wise linear modulation). A FiLM layer is used to "condition" a neural network on auxiliary information, which just means providing the input to the neural network in a way that it can use it effectively. This can be used to integrate multiple sources of information -- for example, in visual question answering (VQA), the main part of the network can be an image processing pipeline, and FiLM can be used to turn the natural language question about the image into a task representation and integrate it into the pipeline, and the full network can be trained end-to-end. The FiLM layer works by first using a subnetwork to turn the auxiliary information (such as the question in VQA) into a "task representation" (a new representation chosen by the neural network), which is then used as the parameters for an affine transformation of the features in the main pipeline. Importantly, each feature is treated independently of other features, so the FiLM layer can't create interactions between features. Yet, this still works well in many different contexts.

Since it is a Distill paper, it then goes into a ton of detail about lots of interesting details, such as how architectures in a variety of ML tasks can be thought of as FiLM, how FiLM relates to other ideas such as attention, how we can often interpolate between different auxiliary information by taking a weighted combination of the corresponding task information, how conditioning through concatenation is equivalent to FiLM with only a bias and no scaling, etc.


This is a pretty cool result. I’m not sure what I would have predicted ahead of time -- the gains are small enough that I could believe I might have predicted them on a general basis of “probably training on realistic data gives you slightly better patterns of thought, so probably if you try hard enough you can find a small set of parameters to finetune that would work well”.

However, another possible line of reasoning would be “the attention heuristics learned for language would probably throw away lots of information if we applied them directly to the input tokens, and the input linear layer may not be enough to handle this issue, so probably this just destroys any good performance of the model”. I could see myself being convinced by that too.

Paper: Pretrained Transformers as Universal Computation Engines

Kevin Lu, Aditya Grover, Pieter Abbeel, Igor Mordatch

We've seen some very impressive few-shot learning results from <@GPT-3@>(@Language Models are Few-Shot Learners@) and [CLIP](https://openai.com/blog/clip/). These work by training a large Transformer model on a giant pile of data in a particular modality (such as language or images), and then we express tasks within that modality (e.g. summarization  for a language model). This paper asks the question: could those models also help with tasks in a _different_ modality? Surprisingly, the answer seems to be yes!

Specifically, the authors take the pretrained GPT-2 models and finetune on very different tasks, changing only the following parameters (which make up just ~0.1% of the model):
1. Input layer: This is a linear layer that transforms the input tokens before they go through the attention layers.
2. Output layer: This is a linear layer that uses the final representations to solve some downstream tasks.
3. Layer norm: These parameters are meant to mimic the statistics of the data distribution, and so need to be finetuned.
4. Positional embeddings. (They say that it only makes a slight difference to finetune these.)

For downstream tasks, they consider tasks like memorizing bit sequences, computing XORs, MNIST and CIFAR (where each image is represented as a sequence of 64 tokens, and each token is a 4x4 patch of the image), and protein folding. None of these tasks involve any use of natural language -- the input modality is completely different.

The headline result: these sorts of models tend to achieve similar performance as Transformer models trained from scratch on the same tasks, and better performance than models initialized with random weights and then finetuned using the method above. This suggests that _even for new data modalities_ the GPT-2 pretraining helps, suggesting that the model has learned some "universal computation" in its attention layers (hence the title). Note though that the differences from the random initialization are not that large (2-6 percentage points, except 25 percentage points in Bit Memory), suggesting that a lot of this might be the inductive bias of the Transformer architecture itself.

The rest of the paper delves into this more, running several experiments to learn more empirical facts. For example:
1. If the Transformers are pretrained on images instead of language, you do better on image tasks like CIFAR, but not as well on the other tasks.
2. Transformers do a _lot_ better than LSTMs.
3. Pretrained Transformers also learn significantly faster than randomly initialized Transformers.


This seems like a solid attack on a theory of scaling. As we discussed [last week](https://mailchi.mp/9ca67b5f5b72/an-139how-the-simplicity-of-reality-explains-the-success-of-neural-nets), it seems likely that any such theory must condition on some assumption about the “simplicity of reality”; in this paper, that assumption is that the data lies on a low-dimensional manifold within a high-dimensional observation space. This seems like a pretty natural place to start, though I do expect that it isn’t going to capture everything.

Note that many of the authors’ experiments are in teacher-student models. In these models, a large teacher neural network is first initialized to compute some random function; a student network must then learn to imitate the teacher, but has either limited data or limited parameters. The benefit is that they can precisely control factors like the intrinsic dimension d, but the downside is that it isn’t immediately clear that the insights will generalize to real-world tasks and datasets. Their experiments with more realistic tasks are less clean, though I would say that they support the theory.

[Explaining Neural Scaling Laws](https://arxiv.org/abs/2102.06701) and [A Neural Scaling Law from the Dimension of the Data Manifold](https://arxiv.org/abs/2004.10802)

Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma

We've seen lots of empirical work on <@scaling laws@>(@Scaling Laws for Neural Language Models@), but can we understand theoretically why these arise? This paper suggests two different models for how power-law scaling laws could arise, _variance-limited_ and _resolution-limited_ scaling, and argues that neural nets are typically trained in the resolution-limited setting. In both cases, we have versions that occur when the dataset size D is large and the number of parameters P is low (parameter-limited), and when D is low and P is large (data-limited).

Recall that a scaling law is a power-law equation that predicts the test loss L as a function of P and D. In this paper, we consider cases where only one of the resources is the bottleneck, so that our power laws are of the form L = kP^(-α) or L = kD^(-α), for constants k and α. (For simplicity, we're assuming that the minimum value of our loss function is zero.)

Resolution-limited scaling happens when either the dataset is too small to "resolve" (capture) the true underlying function, or when the model doesn't have enough capacity to "resolve" (fit) the training dataset. In this case, we're going to take the common ML assumption that while our observation space might be high-dimensional, the data itself comes from a low-dimensional manifold with dimension d, called the _intrinsic dimension_. We'll model our neural net as transforming the input space into a roughly d-dimensional representation of the manifold, which is then used in further processing by later layers. Thus the output of the network is a simple function over this low-dimensional representation.

Let's first consider the case where P is sufficiently large, so that we perfectly fit the training data, but D is limited. We can think of the training data as a "net" of points covering the true d-dimensional manifold. Intuitively, to halve the distance between the points (making the net "twice as fine"), we need ~2^d times as many points. Some simple algebraic manipulation tells us that distance between points would then scale as D^(-1/d).

How can we translate this to the test loss? Let's assume a simple nearest neighbor classifier where, given a test data point, we simply predict the value associated with the nearest training data point. This is equivalent to assuming that our neural net learns a piecewise constant function. In this case, for a test data point drawn from the same distribution as the training set, that data point will be "near" some training data point and our model will predict the same output as for the training data point.

Under the assumption that our test loss is sufficiently "nice", we can do a Taylor expansion of the test loss around this nearest training data point and take just the first non-zero term. Since we have perfectly fit the training data, at the training data point, the loss is zero; and since the loss is minimized, the gradient is also zero. Thus, the first non-zero term is the second-order term, which is proportional to the square of the distance. So, we expect that our scaling law will look like kD^(-2/d), that is, α = 2/d.

The above case assumes that our model learns a piecewise _constant_ function. However, neural nets with Relu activations learn piecewise _linear_ functions. For this case, we can argue that since the neural network is interpolating linearly between the training points, any deviation of the distance between the true value and the actual value should scale as D^(-2/d) instead of D^(-1/d), since the linear term is being approximated by the neural network. In this case, for loss functions like the L2 loss, which are quadratic in the distance, we get that α = 4/d.

Note that it is possible that scaling could be even faster, e.g. because the underlying manifold is simple or has some nice structure that the neural network can quickly capture. So in general, we might expect α >= 2/d and for L2 loss α >= 4/d.

What about the case when P is the bottleneck? Well, in this case, since the training data is not the bottleneck, it is presumably a sufficiently good approximation to the underlying function; and so we are just seeing whether the learned model can match the dataset. Once again, we make the assumption that the learned model gives a piecewise linear approximation, which by the same argument suggests a scaling law of X^(-α), with α >= 2/d (and α >= 4/d for the case of L2 loss), where X is the number of "parts" in the approximation. In the case of linear models, we should have X = P, but for neural networks I believe the authors suggest that we should instead have X = w, the width of the network. (One motivation is that in the infinite-width limit, neural networks behave like linear models.)

In variance-limited scaling for D, the scaling bottleneck is the randomness inherent in the sampling of the dataset from the underlying distribution. We can view the dataset as a random variable, implying that the gradient is also a random variable since it is a function of the training dataset. We can then consider the "error term" δG = G - G_inf, which is the difference between the finite-dataset gradients and the gradients for infinite data. We'll make the assumption that you're equally likely to be wrong in all directions -- if there's a dataset that makes you a bit more likely to predict A, then there's also a corresponding equally likely dataset that makes you a bit less likely to predict A. In that case, in expectation δG is zero, since on average the errors all cancel out. Since D is assumed to be large, we can apply the [law of large numbers](https://en.wikipedia.org/wiki/Law_of_large_numbers) to deduce that the variance of δG will scale as 1/D.

Let us then consider the test loss as a function of the gradients. The test loss we actually get is L(G) = L(G_inf + δG). We can now Taylor expand this to get an expansion which tells us that the quantity we care about, L(G) - L(G_inf), is of the form AδG + B(δG)^2, where A and B are constants that depend on derivatives of the test loss in the infinite dataset case. We had already concluded that E[δG] = 0, and E[(δG)^2] is just the variance and so scales as 1/D, which implies that α = 1.

Here's a slightly less mathematical and more conceptual argument for the same thing (though note that this feels like a sketchier argument overall):
1. Variance of the gradient scales as 1/D by the law of large numbers
2. Thus standard deviation scales as 1/√D
3. Thus the deviation of the empirical estimate of the gradients scales as 1/√D
4. Thus the deviation of the neural net parameters scales as 1/√D
5. Thus the deviation of the output of the final layer scales as 1/√D
6. Any linear dependence on this deviation would cancel out in expectation, since the deviation could either increase or decrease the test loss. However, quadratic dependences would add together. These would scale as (1/√D)^2, that is, 1/D.

The authors also suggest that a similar argument can be applied to argue that for parameters, the loss scales as 1/w, where w is the width of the network. This is variance-limited scaling for P. This again relies on previous results showing that neural networks behave like linear models in the limit of infinite width.

The authors use this theory to make a bunch of predictions which they can then empirically test. I'll only go through the most obvious test: independently measuring the scaling exponent α and the intrinsic dimension d, and checking whether α >= 4/d. In most cases, they find that it is quite close to equality. In the case of language modeling with GPT, they find that α is significantly larger than 4/d, which is still in accordance with the equality (though it is still relatively small -- language models just have a high intrinsic dimension). Variance-limited scaling is even easier to identify: we simply measure the scaling exponent α and check whether it is 1.


While I really enjoyed this paper, I would caution against interpreting it too broadly. If we are to interpret this as a satisfactory answer to our first empirical puzzle, we’d have to say something like “reality tends to be expressible via polynomials, and neural networks tend to learn those polynomials because that is something they can do”. As the paper itself notes, just because reality is determined with low-order Hamiltonians doesn’t mean that given a _subset_ of the information we can get by with a polynomial approximation. In addition, my guess is that if we peered into the internals of the neural networks, it would not be the case that they were calculating the sorts of polynomials that this paper talks about; rather, they would be learning some heuristics that provide some amount of evidence, and combining all these heuristics together leads to a function that is correct the majority of the time. So it’s not clear that this paper really answers our first empirical puzzle.

What I especially liked about this paper was that it analyzed the set of functions we care about (aka functions about reality) and asked what properties of reality made it such that neural networks tended to work well at approximating these functions. Note that this is similar to the common hypothesis in machine learning that the functions we are trying to learn lie on a low-dimensional manifold in a high-dimensional space. This seems like an important direction of research in understanding what neural networks do, and this paper seems like a good example of what such research could look like. I’d be excited to see similar research in the future.

Why does deep and cheap learning work so well?

Henry W. Lin, Max Tegmark, David Rolnick

We know that the success of neural networks must be at least in part due to some inductive bias (presumably towards "simplicity"), based on the following empirical observations:

1. Neural networks with mere millions of parameters work well with high-dimensional inputs such as images, despite the fact that, speaking loosely, there are exponentially more functions from images to classifications than there are functions expressible by million-parameter neural networks.
2. Neural networks learn solutions that generalize well even in the overparameterized regime, where statistical learning theory would predict that they overfit.
3. Relatedly, neural networks learn solutions that generalize well, despite the fact that they can memorize a _randomly_ labeled training dataset of the same size.

Can we say more about this inductive bias towards simplicity? This paper tackles this question from the perspective of the first empirical observation: what is it about neural networks and/or reality such that relatively small neural networks can still learn the "correct" function? We can't appeal to the fact that neural networks are universal function approximators, because that theorem doesn't put a bound on the size of the neural network. The core idea of this paper is that any function that we care to model with neural networks in practice tends to be quite simple: in particular, it can often be expressed as a polynomial plus a few extra things.

Typically, we're interested in modeling the relationship between some latent class y and some detailed observations or data x. For example, y might be a concept like "cat" or image labels more broadly, while x might be specific natural images. In this case the causal structure in reality looks like y → x. In our example, there is first an actual cat (y), and then via the physics of light and cameras we get the image of the cat (x).

Given this setup, why are functions of interest typically "just polynomials"? Well, thanks to Taylor expansions, all (smooth) functions can be expressed as infinite polynomials, so let's rephrase the question: why are they polynomials with only a few terms?

The negative log probability -ln p(x | y) is called the _Hamiltonian_ in statistical physics. There are lots of reasons you might expect that the Hamiltonian is a simple low order polynomial:
1. The Hamiltonians of several fundamental physical laws are polynomials of order 2-4. A polynomial of order d can have at most O(n^d) terms (where n is the number of input variables in the polynomial).
2. The Gaussian distribution (often created in reality thanks to the Central Limit Theorem) has a quadratic Hamiltonian (i.e. order 2).
3. Most functions of interest have a _locality_ property: things only directly affect what is in their immediate vicinity. This causes almost all of the coefficients in the Taylor series to vanish.
4. Many functions have symmetry properties that can further reduce the number of parameters needed to specify them.

One might respond that while this could be true for simple functions like predicting the sum of independent events, this wouldn't apply for the complex functions like "cat" → cat image. Here the authors appeal to _hierarchy_: in practice, the world is very hierarchical, and complex functions can usually be broken down into sequences of simpler ones. If we agree that the simple ones can be implemented with simple polynomials, then a deep neural network could simply learn the same sequence of operations (here the depth of the network is used to chain the operations one after the other).

So far we've argued that generative models p(x | y) tend to be simple polynomials. What about discriminative models p(y | x)? Well, if we can implement the Hamiltonian -ln p(x | y), then there is a simple way to get p(y | x): we simply calculate the Hamiltonian for all possible y, and then add in the prior probabilities -ln p(y) (which can be done through the bias term of the logit layer), and apply a softmax layer to the result. Indeed, the softmax layer at the end is best practice in ML for creating such models. In addition, in the case of a hierarchical sequence of steps, we can invert that sequence of steps and throw away unnecessary information at each step.

Okay, so far we've argued that the functions we care about learning can be expressed with polynomials with relatively few terms (in particular, not an exponential number of terms). What does this have to do with neural networks? It turns out that neural networks can express polynomials quite easily. In particular, the authors show:

1. Multiplication of two real numbers can be approximated arbitrarily well by a neural network with a hidden layer containing 4 neurons.
2. As a result, any given multivariate polynomial can be approximated arbitrarily well by a (potentially deep) neural network of size a little larger than 4 times the number of multiplications needed to evaluate the polynomial.

The authors also show that depth is required for the second result: for a single-layer neural network to multiply n inputs arbitrarily well, it _must_ have at least 2^n neurons (under the assumption that the nonlinear activation function is smooth).


It's extremely cool to have a mathematical characterization of the power of pre-training. I would love to see similar analyses measuring effective data transferred for tasks other than predicting the next token -- if it turns out that modest increases in model size compensate for small datasets in a wide variety of tasks, that makes a strong case that unsupervised learning will be most of the work towards transformative abilities.

Reading these scaling papers really makes me think that there's some deeper theoretical understanding of distributions and predictive models that these results are accessing, maybe encapsulated by [this theory paper](https://arxiv.org/abs/2004.10802) that I still haven't read...

**Rohin's opinion:** Like Asya, I really like the simplicity of the functional form of the scaling law, and the fits to the data seem quite good. I was quite surprised that β seemed to be independent of the pretraining distribution; I would not have predicted that in advance.

Note that despite the form of the scaling law, the effective data transferred isn’t independent of the pretraining dataset size -- that dependence effectively comes through the model size N. This is because authors use compute-optimal pretraining, and so a given model size N corresponds to a specific amount of pretraining compute, which is probably almost the same as having a specific amount of pretraining data.

I am confused by the claim that distributions that are “closer” have lower α. It does make sense that for identical distributions, we should have α = 0. However, if closer distributions have lower α, and β is independent of the distribution, then as your finetuning dataset gets sufficiently large, you actually prefer the further-away distribution! For example, once your finetuning dataset hits 10 trillion points, the scaling laws predict that you prefer to have pretrained on text, rather than 50% text and 50% code, which seems really bizarre. Possibly the scaling laws break down before that point, and in any case a _finetuning_ dataset of 10 trillion data points would be ridiculously massive, but it still seems like something needs to be explained here.

Could we use this to improve timelines estimates? I take a stab at a calculation [here](https://www.lesswrong.com/posts/KrJfoZzpSDpnrv9va/draft-report-on-ai-timelines?commentId=LqZidQ6bEtDnqP6ji); the _very rough and uncertain_ conclusion is that while transfer does seem to be a useful way to reduce compute requirements, the overall effect is not large.

Scaling Laws for Transfer

Danny Hernandez, Jared Kaplan, Tom Henighan, Sam McCandlish

This paper studies empirical scaling laws for transfer learning in language models. The authors use Transformer-based models to predict Python code by training on three different dataset curricula:
- Training from-scratch on Python code
- Pre-training on natural language, then fine-tuning on Python code
- Pre-training on natural language and non-Python code, then fine-tuning on Python code

The authors then measure the "effective data transferred" from pre-training-- if we wanted to replace all the pre-training steps with from-scratch training, maintaining the same loss, how much additional from-scratch data would we need?

They find that when the amount of data used to train is small, effective data transferred is described by a simple power-law function of **D_F**, the amount of data used for fine-tuning, and **N**, the number of parameters: **k (D_F)^α (N)^β**, for constants k, α, and β.

In their experiments, **β** doesn't change between pre-training on natural language and pre-training on a mixture of natural language and non-Python code. They hypothesize that **β** measures how the model architecture generalizes on the target distribution, and doesn't depend on the contents of the pre-training data.

The authors think that **α** is a measure of the directed proximity of the pre-training and from-scratch distributions, with smaller **α** indicating closer proximity. Measuring **α** can be done cheaply by changing the finetuning dataset size while holding the pretrained model constant, making it useful for deciding between collecting more fine-tuning data and increasing model size. For pre-training on natural language and fine-tuning on Python, **β** is about **2 * α**, so for decreasing loss, increasing the fine-tuning dataset size by a factor of **C** (e.g., 100x) would be worth approximately the same as increasing the model size by **√C** (e.g. 10x).

The authors find that pre-training on a mixture of natural language and non-Python code has a higher **k** but lower **α** than pre-training on natural language alone. The higher **k** indicates that the mixture model has better transfer performance when trained in the low data regime, while the lower **α** value means that benefits of the mixture model diminish as more data is used.

The authors also observe that:
- Not counting pre-training compute, pre-trained models are generally more compute efficient than from-scratch models when trained in the low data regime, approximately as compute efficient in the medium data regime, and less compute efficient in the high data regime (close to convergence).
- Small pre-trained models perform worse than small from-scratch models in the high data regime. The authors call this phenomenon "ossification"-- a term used to suggest that small pre-trained models may have a hard time moving away from bad initializations.
- In general, pre-trained models of a given size are compute efficient (on the frontier of loss given compute) for a large portion of their fine-tuning. From-scratch models, by contrast, are only compute efficient for a narrow window of training-- using too little compute for a given model dramatically increases loss and suggests that you should instead be using a smaller model. This makes pre-trained models in some sense "easier" to train.


I really appreciated how this paper gives a clear explanation for empirical results that I had found quite surprising. While there is still the mystery of why variance is unimodal, this helps isolate the confusing phenomenon much more precisely. The explanation of why label noise is necessary seemed quite clear to me and makes much more sense to me than the description in <@Deep Double Descent@>(@Deep Double Descent@). I was encouraged to see that most of the out-of-distribution generalization gap comes from an increase in bias, paired with the finding that deeper networks exhibit lower bias. This seems to indicate that larger models should actually get more robust, which I think will make powerful AI systems safer.

**Rohin’s opinion:** I really liked this paper for the same reasons as Nicholas above. Note that while the classical theory and this paper both talk about the bias and variance of “models” or “hypothesis classes”, what’s actually being measured is the bias and variance of a _full training procedure on a specific task and dataset_. This includes the model architecture, choice of training algorithm, choice of hyperparameters, etc. To me this is a positive: we care about what happens with actual training runs, and I wouldn’t be surprised if this ends up being significantly different from whatever happens in theory when you assume that you can find the model that best fits your training data.

I’d be excited to see more work testing the unimodal variance explanation for deep double descent: for example, can we also explain <@epochal deep double descent@>(@Deep Double Descent@) via unimodal variance?

In addition, I’d be interested in speculation on possible reasons for why variance tends to be unimodal. For example, perhaps real datasets tend to have some simple patterns that every model can quickly learn, but then have a wide variety of small patterns that they could learn in order to slightly boost performance, where different facts generalize differently. If a model with capacity C can learn N such facts, and there are F facts in total, then perhaps you could get F choose N different possible models depending on the random seed. The more possible models with different facts, the higher the variance, and so variance might then be maximized when C is such that N = F / 2, and decrease to either side of that point, giving the unimodal pattern.

(To be clear, this explanation is too high-level -- I don’t really like having to depend on the idea of “facts” when talking about machine learning -- the hope is that with speculation like this we might figure out a more mechanistic explanation.)

Rethinking Bias-Variance Trade-off for Generalization of Neural Networks

Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, Yi Ma

A fundamental result in ML theory shows that the squared error loss function can be decomposed into two components: bias and variance. Suppose that we train a model f to predict some ground truth function y. The bias measures how incorrect the model will be _in expectation over the training process_, while the variance measures how different the model's output can be over different runs of the training process. More concretely, imagine that we run a training process N times, each with a different training set drawn iid from the same underlying training distribution, to get N different models. Bias is like taking the average of these N models, and asking how far away it is from the truth. Meanwhile, variance is like the average distance from each of the N models to the average of all of the N models.

Classical ML predicts that larger models have *lower bias* but *higher variance*. This paper shows that instead, the variance of deep NNs first increases but then decreases at larger model sizes. If the bias tends to be much larger than variance, then we see monotonically decreasing total error. If the variance tends to be much larger than the bias, then loss will also look bell-shaped, initially _increasing_ as models get bigger and then decreasing. Finally, if the bias starts high, but over time is overshadowed by the variance, we get <@double descent@>(@Deep Double Descent@) curves; this explains why previous work needed to add label noise to get double descent curves (as higher label noise should lead to higher variance).

In order to estimate the variance, the authors split their data into two subsets and use these to create an unbiased estimator of the variance (effectively following a similar procedure to the one described in the first paragraph). The bias estimate can then be determined from the test loss and the estimated variance. They then test how various factors contribute to test loss. As expected, label noise increases variance. Out-of-distribution samples have higher test loss, which is driven by both bias and variance, but most of the increase comes from bias. Deeper networks sharing the same architecture have lower bias but higher variance.


For a long time, I’ve heard people quietly hypothesizing that with a sufficient diversity of tasks, regular gradient descent could lead to general reasoning abilities allowing for quick adaptation to new tasks. This is a powerful demonstration of this hypothesis.

One [critique](https://www.lesswrong.com/posts/ZHrpjDc3CepSeeBuE/gpt-3-a-disappointing-paper) is that GPT-3 still takes far too long to “identify” a task -- why does it need 50 examples of addition in order to figure out that what it should do is addition? Why isn’t 1 sufficient? It’s not like there are a bunch of other conceptions of “addition” that need to be disambiguated. I’m not sure what’s going on mechanistically, but we can infer from the paper that as language models get larger, the number of examples needed to achieve a given level of performance goes down, so it seems like there is some “strength” of general reasoning ability that goes up (see also [this commentary](https://www.gwern.net/newsletter/2020/05)). Still, it would be really interesting to figure out mechanistically how the model is “reasoning”.

This also provides some empirical evidence in support of the threat model underlying <@inner alignment concerns@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@): they are predicated on neural nets that implicitly learn to optimize. (To be clear, I think it provides empirical support for neural nets learning to “reason generally”, not neural nets learning to implicitly “perform search” in pursuit of a “mesa objective” -- see also <@Is the term mesa optimizer too narrow?@>.)

Language Models are Few-Shot Learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei

The biggest <@GPT-2 model@>(@Better Language Models and Their Implications@) had 1.5 billion parameters, and since its release people have trained language models with up to 17 billion parameters. This paper reports GPT-3 results, where the largest model has _175 billion_ parameters, a 10x increase over the previous largest language model. To get the obvious out of the way, it sets a new state of the art (SOTA) on zero-shot language modeling (evaluated only on Penn Tree Bank, as other evaluation sets were accidentally a part of their training set).

The primary focus of the paper is on analyzing the _few-shot learning_ capabilities of GPT-3. In few-shot learning, after an initial training phase, at test time models are presented with a small number of examples of a new task, and then must execute that task for new inputs. Such problems are usually solved using _meta-learning_ or _finetuning_, e.g. at test time [MAML](https://arxiv.org/abs/1703.03400) takes a few gradient steps on the new examples to produce a model finetuned for the test task. In contrast, the key hypothesis with GPT-3 is that language is so diverse, that doing well on it already requires adaptation to the input, and so the learned language model will _already be a meta-learner_. This implies that they can simply "prime" the model with examples of a task they care about, and the model can _learn_ what task is supposed to be performed, and then perform that task well.

For example, consider the task of generating a sentence using a newly made-up word whose meaning has been explained. In one notable example, the prompt for GPT-3 is:

_A "whatpu" is a small, furry animal native to Tanzania. An example of a sentence that uses the word whatpu is:_
_We were traveling in Africa and we saw these very cute whatpus._
_To do a "farduddle" means to jump up and down really fast. An example of a sentence that uses the word farduddle is:_

Given this prompt, GPT-3 generates the following example sentence for "farduddle":

_One day when I was playing tag with my little sister, she got really excited and she started doing these crazy farduddles._

The paper tests on several downstream tasks for which benchmarks exist (e.g. question answering), and reports zero-shot, one-shot, and few-shot performance on all of them. On some tasks, the few-shot version sets a new SOTA, _despite not being finetuned using the benchmark's training set_; on others, GPT-3 lags considerably behind finetuning approaches.

The paper also consistently shows that few-shot performance increases as the number of parameters increases, and the rate of increase is faster than the corresponding rate for zero-shot performance. While they don't outright say it, we might take this as suggestive evidence that as models get larger, they are more incentivized to learn "general reasoning abilities".

The most striking example of this is in arithmetic, where the smallest 6 models (up to 6.7 billion parameters) have poor performance (< 20% on 2-digit addition), then the next model (13 billion parameters) jumps to > 50% on 2-digit addition and subtraction, and the final model (175 billion parameters) achieves > 80% on 3-digit addition and subtraction and a perfect 100% on 2-digit addition (all in the few-shot regime). They explicitly look for their test problems in the training set, and find very few examples, suggesting that the model really is learning "how to do addition"; further, when it is incorrect, it tends to make mistakes like "forgetting to carry a 1".

On broader impacts, the authors talk about potential misuse, fairness and bias concerns, and energy usage concerns; and say they about these issues what you'd expect. One interesting note: "To understand how low and mid-skill actors think about language models, we have been monitoring forums and chat groups where misinformation tactics, malware distribution, and computer fraud are frequently discussed." They find that while there was significant discussion of misuse, they found no successful deployments. They also consulted with professional threat analysts about the possibility of well-resourced actors misusing the model. According to the paper: "The assessment was that language models may not be worth investing significant resources in because there has been no convincing demonstration that current language models are significantly better than current methods for generating text, and because methods for "targeting" or "controlling" the content of language models are still at a very early stage."


I took a look at the repository and found there to be ample documentation available making it easy for me to try training my own infinite-width network. The authors derive a practical way to compute the exact convolutional NTK which I find  impressive and which seems to be the main technical contribution of this paper. While the authors note that there are some conditions necessary to enter the so-called "kernel regime", in practice it seems as though you can often get away with merely large network widths. If for nothing else, I'd recommend at least perusing the notebooks they have available or taking a look at the visualization they present of a neural network converging to a Gaussian process, which relies on a subtle application of the law of large numbers.

Paper: Neural Tangents: Fast and Easy Infinite Neural Networks in Python

Fast and Easy Infinitely Wide Networks with Neural Tangents

Roman Novak*, Lechao Xiao*, Samuel S. Schoenholz*, Jiri Hron, Jaehoon Lee, Alexander A. Alemi, Jascha Sohl-Dickstein

The success of Deep Learning has led researchers to explore why they're such effective function approximators. One key insight is that increasing the width of the network layers makes it *easier* to understand. More precisely, as the width is sent to infinity the network's learning dynamics can be approximated with a Taylor expansion and become a kernel problem. This kernel has an exact form in the limit and is referred to as the neural tangent kernel (NTK). Ultimately, this allows us to model the network with a simpler model known as a Gaussian process. Unfortunately, showing this analytically is difficult and creating efficient implementations is cumbersome. **The authors address this problem by introducing "Neural Tangents", a library that makes creating infinite-width networks as easy as creating their finite counterparts with libraries such as PyTorch or TensorFlow.** They include support for convolutions with full-padding, residual-connections, feed-forward networks, and support for a variety of activation functions. Additionally, there is out-of-the-box support for CPU, GPU, and TPU. Moreover, uncertainty comparisons with finite ensembles are possible via exact Bayesian inference.


The hypothesis makes sense to me and the experiments do seem to bear out their conclusions. However, none of the results of the experiments were surprising to me and seem to me like they could be consistent with other explanations for generalization. I would be more convinced if the Coherent Gradients hypothesis made predictions that were different from other leading theories and then those turned out to be true.

Explaining Memorization and Generalization: A Large-Scale Study with Coherent Gradients

Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization

Deep neural networks trained with gradient descent do well at generalizing from their training set, but the field currently has relatively little understanding of why that is. Large networks have enough parameters to fully memorize the training set and can do so even if trained on data with entirely random labels. This allows for many functions that would fit the training set well, but not generalize, a phenomenon known as overfitting. The question is how gradient descent picks out one of a small subset of functions that will generalize well.

The *Coherent Gradients* hypothesis, introduced here and tested further in [this paper](http://arxiv.org/abs/2003.07422), is that this results from per-example gradients being averaged during gradient descent. For each example, some of the gradient points in a direction that is idiosyncratic to that example, but some of it points towards a more general solution. When the average is taken across these gradients, the more general directions reinforce each other while the example-specific directions cancel out. As a result, the training process moves faster towards more general directions.

In order to test this hypothesis, they run two experiments. First they use varying amounts of label noise (corrupting a fraction of the dataset to have random labels). They predict and find that:
1. More label noise leads to slower learning.
2. The uncorrupted examples will be learned faster.

The next experiment tests a novel form of regularization, called winsorization, where they clip the gradients on a per-example and per-parameter basis to prevent a single example from dominating the gradient, effectively curtailing the component of the gradient that is example-specific. Since the computation of per-example gradients is expensive, when scaling this up to larger networks, they instead use the median of 3 mini-batches to address outliers. Theexperiments suggest that winsorization reduces overfitting and in particular prevents neural nets from learning randomly labeled data.


The tasks they test on are _extremely_ sparse: since there are hundreds of thousands of possible labels, even if you take the top ~thousand predictions in the final layer (which corresponds to most of the computation), that’s only 1% of the total number of predictions, saving you 99% of the arithmetic you would have had to do. The input features are also very sparse: in both datasets, less than 0.06% (yes, percent) of features are non-zero. It’s cool that under such conditions you can design an algorithm that is ~an order of magnitude better on cost, but it’s not going to be “the death of NVIDIA” or anything like that — without further optimizations, SLIDE will be worse than regular Tensorflow on GPU for something like ImageNet.

I'm also not sure I agree with the 'thesis' of the paper that smart algorithms beat hardware acceleration-- it seems to me like there are large gains from investing in the combination of the two. Even if GPUs aren't optimized to run SLIDE, I can imagine specialized hardware optimized for SLIDE creating even bigger performance gains.

SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems

Beidi Chen, Tharun Medini, James Farwell, Sameh Gobriel, Charlie Tai, Anshumali Shrivastava

This paper presents an algorithmic technique called SLIDE (Sub-LInear Deep learning Engine) which takes advantage of sparsity in inputs and activations to speed up the training of large neural networks.

Suppose that activations at layer k are a_k. Then, the ith element of a_{k+1} is given by the dot product of a_k and w_i for some weight vector w_i. Call w_i the ith neuron of layer k + 1. The largest activations in a_{k+1} are the ones for whom w_i has high magnitude and points in the same direction as a_k. The core proposal of SLIDE is to only compute the largest elements of a_{k+1}, which they call the "activated neurons", and approximate all of the others are zero, allowing us to avoid a lot of computation.

In order to do this, we maintain a data structure called a _locality-sensitive hash table_, which when given an activation a_k can tell us which neurons (w_is) are most similar. We can then compute the outputs for just those neurons to get a_{k+1}. In this way, we can effectively 'sparsify' the network, calculating the activations and updating the weights of only a small subset of the neurons. This is what gives us our computational gains.

SLIDE randomly initializes weights in the network and generates the locality-sensitive hash table that maps activations to activated neurons. To take a gradient step on an input, it calculates the activated neurons in a forward pass, then backpropagates through the activated neurons, and then updates the locality-sensitive hash table. The hash table update is computationally expensive, and SLIDE uses several mechanisms to make it less costly, such as updating hash tables less frequently later in the training process since gradients are likely to change less then. Due to the sparsity, the gradients for different inputs are often changing different neurons, and so SLIDE asynchronously parallelizes gradient updates without worrying about race conditions, allowing for much better scaling with additional cores.

The paper evaluates SLIDE on large multi-label classification tasks, which must run on neural networks with extremely wide final layers. It finds that the CPUs running SLIDE are 1.8 times faster in clock-time than the GPU on the Delicious 200k dataset, and 2.7 times faster than the GPU on the Amazon-670K dataset, with an additional ~1.3x speed-up after performing cache optimization on SLIDE. Scalability tests suggest that the SLIDE CPUs beat GPU performance even when using only 8 cores. The paper claims that SLIDE's computational benefits come because the number of neurons sampled in the wide final layer is extremely small-- fewer than 0.5% of active neurons.