Fundamentally, the point here is that generalization performance is explained much more by the neural network architecture rather than the structure of stochastic gradient descent, since we can see that stochastic gradient descent tends to behave similarly to (an approximation of) random sampling. The paper talks a bunch about things like SGD being (almost) Bayesian and the neural network prior having low Kolmogorov complexity; I found these to be distractions from the main point. Beyond that, approximating the random sampling probability with a Gaussian process is a fairly delicate affair and I have concerns about the applicability to real neural networks.

One way that SGD could differ from random sampling is that SGD will typically only reach the boundary of a region with zero training error, whereas random sampling will sample uniformly within the region. However, in high dimensional settings, most of the volume is near the boundary, so this is not a big deal. I'm not aware of any work that claims SGD uniformly samples from this boundary, but it's worth considering that possibility if the experimental results hold up.

**Rohin’s opinion:** I agree with Zach above about the main point of the paper. One other thing I’d note is that SGD can’t have literally the same outcomes as random sampling, since random sampling wouldn’t display phenomena like <@double descent@>(@Deep Double Descent@). I don’t think this is in conflict with the claim of the paper, which is that _most_ of the inductive bias comes from the architecture and initialization.

[Other](https://arxiv.org/abs/1805.08522) [work](https://arxiv.org/abs/1909.11522) by the same group provides some theoretical and empirical arguments that the neural network prior does have an inductive bias towards simplicity. I find those results suggestive but not conclusive, and am far more persuaded by the paper summarized here, so I don’t expect to summarize them.

Is SGD a Bayesian Sampler? Well, almost.

Chris Mingard, Guillermo Valle-Pérez, Joar Skalse, Ard A. Louis

Neural networks have been shown empirically to generalize well in the overparameterized setting, which suggests that there is an inductive bias for the final learned function to be simple. The obvious next question: does this inductive bias come from the _architecture_ and _initialization_ of the neural network, or does it come from stochastic gradient descent (SGD)? This paper argues that it is primarily the former.

Specifically, if the inductive bias came from SGD, we would expect that bias to go away if we replaced SGD with random sampling. In random sampling, we sample an initialization of the neural network, and if it has zero training error, then we're done, otherwise we repeat.

The authors explore this hypothesis experimentally on the MNIST, Fashion-MNIST, and IMDb movie review databases. They test on variants of SGD, including Adam, Adagrad, and RMSprop. Since actually running rejection sampling for a dataset would take _way_ too much time, the authors approximate it using a Gaussian Process. This is known to be a good approximation in the large width regime.

Results show that the two probabilities are correlated over a wide order of magnitudes for different architectures, datasets, and optimization methods. While correlation isn't perfect over all scales, it tends to improve as the frequency of the function increases. In particular, the top few most likely functions tend to have highly correlated probabilities under both generation mechanisms.


Intuitively, inductive biases become less and less useful the more training data we have, but I would have thought that in the current regime CNNs have too weak rather than too strong inductive biases, so the results are surprising. What is even more surprising is how simple the model is: It does not seem to use any data augmentation, unsupervised pretraining or other tricks like noisy student-teacher training, such that there are many promising avenues for immediate improvements. Also, I would imagine that using something more sophisticated than a linear projection to embed the 16x16 patches could go a long way.

Paper: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Alexey Dosovitskiy*, Lucas Beyer*, Alexander Kolesnikov*, Dirk Weissenborn*, Xiaohua Zhai*, Neil Houlsby*, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit

This paper applies transformers to image classification in a fairly straightforward way: First, an input image is divided into 16x16 pixel patches on a grid. Then, a linear projection of the patch is combined with a learnt positional embedding and fed into a standard transformer pipeline. Lastly, a standard MLP head is applied on top of the transformer for the classification. When trained on ImageNet, this architecture overfits and does not reach SOTA performance. However, it can compete with the previous SOTA on the larger ImageNet-21k (14M images) and outcompete it on JFT (300M images), while needing four times less compute for training. By finetuning the JFT model on ImageNet, the transformer narrowly outperforms the previous best ImageNet classifier.

The positional embeddings learnt by the model look meaningful in that each is most similar to others in the same row or column. Also, some of the attention heads in early layers attend to multiple distant patches, while others are a lot more local. This means that some heads in the early layers have a wide receptive field, which is something that convolution kernels cannot achieve. Overall, given enough data, the transformer seems to be able to learn inductive biases used by CNNs without being limited to them.


Ultimately we’re going to want AI systems that learn and improve over time, even during deployment. It’s exciting to see an example of what that might look like.

Deploying Lifelong Open-Domain Dialogue Learning

Kurt Shuster*, Jack Urbanek*, Emily Dinan, Arthur Szlam, Jason Weston

Most research in natural language processing (NLP) follows a paradigm in which we first collect a dataset via crowdsourced workers, and then we train a model on this dataset to solve some task. Could we instead have _lifelong learning_, in which a model could continue learning after being deployed, getting better and better the more it is used? This paper shows one instantiation of such an approach, in a fantasy role-playing game.

The authors take the previously developed LIGHT role-playing setting, and gamify it. The human player talks to a language model while playing some role, and earns stars and badges for saying realistic things (as evaluated by another language model). Rather than paying crowdsourced workers to provide data, the authors instead merely advertise their game, which people then play for fun, reducing the cost of data acquisition. They find that in addition to reducing costs, this results in a more diverse dataset, and also leads to faster improvements in automated metrics.


I often find it's very useful to stare directly at raw data in order to understand how something works, in addition to looking at summary statistics and graphs that present a very high-level view of the data. While this isn't literally raw data (Gwern heavily designed the prompts, and somewhat curated the outputs), I think it provides an important glimpse into how GPT-3 works that you wouldn't really get from reading the <@paper@>(@Language Models are Few-Shot Learners@).

GPT-3 Creative Fiction

In Gwern's words, this is "creative writing by OpenAI's GPT-3 model, demonstrating poetry, dialogue, puns, literary parodies, and storytelling".


I am confused about the overfitting part, as most methods they try (like ResNets) don't seem to have been selected for performance on the MNIST test set. Granted, LeNet seems to degrade more than other models, but it seems like the additional test images in QMNIST are actually harder to classify. This seems especially plausible with the previous summary in mind and because the authors mention a dichotomy between the ease of classification for NIST images generated by highschoolers vs government employees but don’t seem to mention any attempts to deal with potential selection bias.

Cold Case: The Lost MNIST Digits

As the MNIST test set only contains 10,000 samples, concerns that further improvements are essentially overfitting on the test set have been voiced. Interestingly, MNIST was originally meant to have a test set of 60,000, as large as the training set, but the remaining 50,000 digits have been lost. The authors made many attempts to reconstruct the way MNIST was obtained from the NIST handwriting database as closely as possible and present QMNIST(v5) which features an additional 50,000 test images for MNIST, while the rest of the images are very close to the originals from MNIST. They test their dataset using multiple classification methods and find little difference in whether MNIST or QMNIST is used for training, but the test error on the additional 50,000 images is consistently higher than on the original 10,000 test images or their reconstruction of these. While the concerns about overuse of a test set are justified, the measured effects were mostly small and their relevance might be outweighed by the usefulness of paired differences for statistical model selection.


The AI claims made in this paper seem pretty reasonable to me, though I think the paper overstates how much deep learning aficionados disagree with them. I certainly agree for example that existing deep learning systems do not generalize well outside of their training environment, or that AI systems will need to work with abstract knowledge, or that AI systems will have to learn from external, cultural knowledge represented in natural language. And while I am perhaps not as enamored of deep learning as (say) OpenAI or DeepMind, I'm a pretty big fan of it, and try to design algorithms where deep learning can do most of the "heavy lifting".

The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence

This paper suggests a few directions which would allow us to build more _robust_ AI systems with better "understanding" of the world: specifically, it highlights **symbol manipulation, encoded knowledge, reasoning, and cognitive models** as areas of research for the next decade.

See also [Import AI #187](https://jack-clark.net/2020/03/02/import-ai-187-real-world-robot-tests-at-cvpr-all-hail-the-molecule-transformer-the-four-traits-needed-for-smarter-ai-systems/) and [Matthew Barnett's summary](https://www.lesswrong.com/posts/CeJs4rPgPtJPNqLMt/gary-marcus-four-steps-towards-robust-artificial).


I read this paper back when summarizing <@Are Deep Neural Networks Dramatically Overfitted?@> and found it uncompelling, and I'm really curious how the ML community correctly seized upon this idea as deserving of further investigation while I incorrectly dismissed it. None of the experimental results in this paper are particularly surprising to me, whereas double descent itself is quite surprising.

In the random Fourier features and decision trees experiments, there is a qualitative difference in the _learning algorithm_ before and after the interpolation threshold, that suffices to explain the curve. With the random Fourier features, we only start regularizing the model after the interpolation threshold; it is not surprising that adding regularization helps reduce test loss. With the decision trees, after the interpolation threshold, we start using ensembles; it is again not at all surprising that ensembles help reduce test error. (See also [this comment](https://www.alignmentforum.org/posts/FRv7ryoqtvSuqBxuT/understanding-deep-double-descent#6bqY4oSFncLTJgYdm).) So yeah, if you start regularizing (via L2 norm or ensembles) after the interpolation threshold, that will help your test error, but in practice we regularize throughout the training process, so this should not occur with neural nets.

The neural net experiments also have a similar flavor -- the nets before the interpolation threshold are required to reuse weights from the previous run, while the ones after the interpolation threshold do not have any such requirement. When this is removed, the results are much more muted. The authors claim that this is necessary to have clear graphs (where training risk monotonically decreases), but it's almost certainly biasing the results -- at the interpolation threshold, with weight reuse, the test squared loss is ~0.55 and test accuracy is ~80%, while without weight reuse, test squared loss is ~0.35 and test accuracy is ~85%, a _massive_ difference and probably not within the error bars.

Some speculation on what's happening here: neural net losses are nonconvex and training can get stuck in local optima. A pretty good way to get stuck in a local optimum is to initialize half your parameters to do something that does quite well while the other half are initialized randomly. So with weight reuse we might expect getting stuck in worse local optima. However, it looks like the training losses are comparable between the methods. Maybe what's happening is that with weight reuse, the half of parameters that are initialized randomly memorize the training points that the good half of the parameters can't predict, which doesn't generalize well but does get low training error. Meanwhile, without weight reuse, all of the parameters end up finding a good model that does generalize well, for whatever reason it is that neural nets do work well.

But again, note that the authors were right about double descent being a real phenomenon, while I was wrong, so take all this speculation with many grains of salt.

Reconciling modern machine learning practice and the bias-variance trade-off

Mikhail Belkin, Daniel Hsu, Siyuan Ma, Soumik Mandal

This paper first proposed double descent as a general phenomenon, and demonstrated it in three machine learning models: linear predictors over random Fourier features, fully connected neural networks with one hidden layer, and forests of decision trees. Note that they define the interpolation threshold as the point where the number of parameters equals the number of training points, rather than using something like effective model complexiy.

For linear predictors over random Fourier features, their procedure is as follows: they generate a set of random features, and then find the linear predictor that minimizes the squared loss incurred. If there are multiple predictors that achieve zero squared loss, then they choose the one with the minimum L2 norm. The double descent curve for a subset of MNIST is very pronounced and has a huge peak at the point where the number of features equals the number of training points.

For the fully connected neural networks on MNIST, they make a significant change to normal training: prior to the interpolation threshold, rather than training the networks from scratch, they train them from the final solution found for the previous (smaller) network, but after the interpolation threshold they train from scratch as normal. With this change, you see a very pronounced and clear double descent curve. However, if you always train from scratch, then it's less clear -- there's a small peak, which the authors describe as "clearly discernible", but to me it looks like it could be noise.

For decision trees, if the dataset has n training points, they learn decision trees of size up to n leaves, and then at that point (the interpolation threshold) they switch to having ensembles of decision trees (called forests) to get more expressive function classes. Once again, you can see a clear, pronounced double descent curve.


Beijing Academy of Artificial Intelligence announces 1,75 trillion parameters model, Wu Dao 2.0

There's a good chance you've heard of the new Wu Dao 2.0 language model, with over 1 trillion parameters. Unfortunately, as far as I know there is no technical writeup describing this model, so I'm going to refrain from commenting on it. You can see other people's takes in the linked LessWrong post, on [ChinAI](https://chinai.substack.com/p/chinai-145-enlightenment-via-large), and on [policy.ai](https://cset.georgetown.edu/newsletter/june-10-2021/).


It's interesting that the lottery ticket hypothesis scales; however, this paper seems quite incremental overall.

The Lottery Ticket Hypothesis at Scale

Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, Michael Carbin

The [lottery ticket hypothesis](https://arxiv.org/abs/1803.03635) is the claim that "dense, randomly-initialized, feed-forward networks contain subnetworks (winning tickets) that - when trained in isolation - reach test accuracy comparable to the original network in a similar number of iterations". This paper builds on previous work to show that winning tickets can also be found for larger networks (Resnet-50, not just Resnet-18), if those winning tickets are initialised not with their initial weights from the full network, but rather with their weights after a small amount of full-network training.


Embedded Agency (full-text version)

This is the text version of all of the previous posts in the sequence.


I'm definitely on board with the idea of curiosity-driven research, it seems important to try to find the places in which we're confused and refine our knowledge about them. I think my main point of departure is that I am less confident than (my perception of) MIRI that there is a nice, clean formulation of embedded agents and intelligence that you can write down -- I wouldn't be surprised if intelligence was relatively environment-specific. (This point was made in [Realism about rationality](https://www.lesswrong.com/posts/suxvE2ddnYMPJN9HD/realism-about-rationality) ([AN #25](https://mailchi.mp/0c5eeec28f75/alignment-newsletter-25)).) That said, I'm not particularly confident about this and think there's reasonable room for disagreement -- certainly I wouldn't want to take everyone at MIRI and have them work on application-based AI alignment research.

Embedded Curiosities

This sequence concludes with a brief note on why MIRI focuses on embedded agency. While most research in this space is presented from a motivation of mitigating AI risk, Scott has presented it more as an intellectual puzzle, something to be curious about. There aren't clear, obvious paths from the problems of embedded agency to specific failure modes. It's more that the current dualistic way of thinking about intelligence will break down with smarter agents, and it seems bad if we are still relying on these confused concepts when reasoning about our AI systems, and by default it doesn't seem like anyone will do the work of finding better concepts. For this work, it's better to have a curiosity mindset, which helps you orient towards the things you are confused about. An instrumental strategy approach (which aims to directly mitigate failure modes) is vulnerable to the urge to lean on the shaky assumptions we currently have in order to make progress.


Embedded World-Models (Addendum)

A few slides have been added to this post since my summary last week, going into more detail about the grain-of-truth problem. This problem is particularly hard because your learned world model must include the world model itself inside of it, even in the presence of an environment that can behave adversarially towards the world model. It is easy to construct deterministic paradoxes where the world model cannot be correct -- for example, in rock-paper-scissors, if your model predicts what the opponent will do and plays the action that wins against the prediction, the opponent will (if they can) predict that and play the action that beats your action, falsifying your model. While game theory solves these sorts of scenarios, it does so by splitting the agent away from the environment, in a way that is very reminiscent of the dualistic approach. Recently, reflective oracles were developed, that solve this problem by having probabilistic models that were robust to self-reference, but they still assume logical omniscience.


The first part of this post seems to rest upon an assumption that any subagents will have long-term goals that they are trying to optimize, which can cause competition between subagents. It seems possible to instead pursue subgoals under a limited amount of time, or using a restricted action space, or using only "normal" strategies. When I write this newsletter, I certainly am treating it as a subgoal -- I don't typically think about how the newsletter contributes to my overall goals, I just aim to write a good newsletter. Yet I don't recheck every word until the email is sent. Perhaps this is because that would be a new strategy I haven't used before and so I evaluate it with my overall goals, instead of just the "good newsletter" goal, or perhaps it's because my goal also has time constraints embedded in it, or something else, but in any case it seems wrong to think of newsletter-Rohin as optimizing long term preferences for writing as good a newsletter as possible.

I agree quite strongly with the second part of the post, about inner optimizers that could arise from search. Agents that maximize some long-term preferences are certainly possible, and it seems reasonably likely that a good solution to a complex problem would involve an optimizer that can adjust to different circumstances (for concreteness, perhaps imagine [OpenAI Five](https://blog.openai.com/openai-five/) ([AN #13](https://mailchi.mp/8234356e4b7f/alignment-newsletter-13))). I don't think that inner optimizers are guaranteed to show up, but it seems quite likely, and they could lead to catastrophic outcomes if they are left unchecked.

Subsystem Alignment

Any agent is likely to be built out of multiple subsystems, that could potentially have their own goals and work at cross-purposes to each other. A simple unrealistic example would be an agent composed of two parts -- a world model and a decision algorithm (akin to the setup in [World Models](https://worldmodels.github.io/) ([AN #23](https://mailchi.mp/9295c961b39b/alignment-newsletter-23))). The decision algorithm aims to cause some feature of the world model to be high. In this case, the decision algorithm could trick the world model into thinking the feature is high, instead of actually changing the world so that the feature is high (a [delusion box](https://pdfs.semanticscholar.org/7480/d1ba5b765cc21d7aa395c7ac2eba8c68b44e.pdf)).

Why not just build a monolithic agent, or build an agent whose subcomponents are all aligned with each other? One reason is that our agent may want to solve problems by splitting into subgoals. However, what then prevents the agent from optimizing the subgoal too far, to the point where it is no longer helps for the original goal? Another reason is that when we make subagents to solve simpler tasks, they shouldn't need the whole context of what we value to do their task, and so we might give them a "pointer" to the true goal that they can use if necessary. But in that case, we have introduced a level of indirection, which a [previous post](https://www.alignmentforum.org/posts/iTpLAaPamcKyjmbFC/robust-delegation) ([AN #31](https://mailchi.mp/7d0e3916e3d9/alignment-newsletter-31)) argues leads to wireheading.

Perhaps the most insidious case is search, which can produce subagents by accident. Often, it is easier to solve a problem by searching for a good solution than deriving it from first principles. (For example, machine learning is a search over functions, and often outperforms hand-designed programs.) However, when an agent searches for a good solution, the solution it finds might _itself_ be an agent optimizing some other goal that is currently correlated with the original goal, but can diverge later due to [Goodhart's law](https://www.lesswrong.com/posts/EbFABnst8LsidYs5Y/goodhart-taxonomy). If we optimize a neural net for some loss function, we might get such an inner optimizer. As an analogy, if an agent wanted to maximize reproductive fitness, they might have used evolution to do this -- but in that case humans would be inner optimizers that subvert the original agent's goals (since our goals are not to maximize reproductive fitness).


_Warning:_ Ramblings about topics I haven't thought about much.

I'm certainly confused about how humans actually make decisions -- we do seem to be able to consider counterfactuals in some reasonable way, but it does seem like these are relatively fuzzy (we can't do the counterfactual "what if 2+2=3", we can do the counterfactual "what if I took the $10", and we disagree on how to do the counterfactual "what would happen if we legalize drugs" (eg. do we assume that public opinion has changed or not?). This makes me feel pessimistic about the goal of having a "correct" counterfactual -- it seems likely that humans somehow build causal models of some aspects of the world (which do admit good counterfactuals), especially of the actions we can take, and not of others (like math), and disagreements on "correct" counterfactuals amount to disagreements on causal models. Of course, this just pushes the question down to how we build causal models -- maybe we have an inductive bias that pushes us towards simple causal models, and the world just happens to be the kind where the data you observe constrains your models significantly, such that everyone ends up inferring similar causal models.

However, if we do build something like this, it seems hard to correctly solve most decision theory problems that they consider, such as Newcomblike problems, at least if we use the intuitive notion of causality. Maybe this is okay, maybe not, I'm not sure. It definitely doesn't feel like this is resolving my confusion about how to make good decisions in general, though I could imagine that it could resolve my confusion about how to make good decisions in our actual universe (where causality seems important and "easy" to infer).

Decision Theory

The major issue with porting decision theory to the embedded agency section is that there is no longer a clear, well-defined boundary between actions and outcomes, such that we can say "_if_ I take this action, _then_ this outcome occurs". In an embedded setting, the agent is just another part of the environment, and so if the agent is reasoning about the environment, it can also reason about itself, and its reasoning can tell it something about what its actions will be. But if you know what action you are going to take, how do you properly think about the counterfactual "_what if_ I had taken this other action"?

A formalization in logic, where counterfactuals are represented by logical implication, doesn't work. If you know what your action is going to be, then the premise of the counterfactual (that you take some other action) is false, and you can conclude anything. The post gives a concrete example of a reasonable-looking agent which ends up choosing to take $5 when offered a choice between $5 and $10 because it can prove that "if I took $10, then I would get $0" (which is in fact _true_, since it took $5, and not $10!) A formalization in probability theory doesn't work, because if you condition on an alternative action that you know you won't take, you are conditioning on a probability zero event. If you say that there is always some uncertainty in which action you take, or you force the agent to always explore with some small probability, then your agent is going to reason about alternative actions under the assumption that there was some hardware failure, or that it was forced to explore -- this seems like the wrong way to reason about alternatives.

Changing tack a bit, how would we think about "_What if_ 2+2=3?" This seems like a pretty hard counterfactual for us to evaluate -- it's not clear what it means. There may just be no "correct" counterfactuals -- but in this case we still need to figure out how intelligent agents like humans successfully consider alternative actions that they are not going to take, in order to make good decisions. One approach is Updateless Decision Theory (UDT), which takes the action your earlier self would have wanted to commit to, which comes closer to viewing the problem from the outside. While it neatly resolves many of the problems in decision theory, including counterfactual mugging (described in the post), it assumes that your earlier self can foresee all outcomes, which can't happen in embedded agents because the environment is bigger than the agent and any world model can only be approximate (the subject of the next post).


This post is a great summary of the sequence to come, and is intuitive and easy to understand. I strongly recommend reading the full post -- I haven't summarized it much because it already is a good summary.

Embedded Agents

This post introduces embedded agency, which refers to the notion of an "agent" that is more realistic than the version considered in mainstream AI, which is best formalized by AIXI. An embedded agent is one that is actually a part of the environment it is acting in, as opposed to our current AI agents which model the environment as external to them. The problems around embedded agency fall into four main clusters, which future posts will talk about.


_Warning:_ Ramblings about topics I haven't thought about much.

The high-level and multi-level model problems sound similar to the problems that could arise with hierarchical reinforcement learning or hierarchical representation learning, though the emphasis here is on the inconsistencies between different levels rather than how to learn the model in the first place.

The grain of truth problem is one of the problems I am most confused about -- in machine learning, model misspecification can lead to very bad results, so it is not clear how to deal with this even approximately in practice. (Whereas with decision theory, "approximate in-practice solutions" include learning causal models on which you can construct counterfactuals, or learning from experience what sort of decisionmaking algorithm tends to work well, and these solutions do not obviously fail as you scale up.) If you learn enough to rule out _all_ of your hypotheses, as could happen with the grain of truth problem, what do you do then? If you're working in a Bayesian framework, you end up going with the hypothesis you've disproven the least, which is probably not going to get you good results. If you're working in logic, you get an error. I guess learning a model of the environment in model-based RL doesn't obviously fail if you scale up.

Embedded World-Models

In order to get optimal behavior on environments, you need to be able to model the environment in full detail, which an embedded agent cannot do. For example, AIXI is incomputable and gets optimal behavior on computable environments. If you use AIXI in an incomputable environment, it gets bounded loss on predictive accuracy compared to any _computable_ predictor, but there are no results on absolute loss on predictive accuracy, or on the optimality of actions it chooses. In general, if the environment is not in the space of hypotheses you can consider, that is your environment hypothesis space is misspecified, then many bad issues can arise (as often happens with misspecification). This is called the grain-of-truth problem, so named because you have to deal with the fact that your prior does not even have a grain of truth (the true environment hypothesis).

One approach could be to learn a small yet well-specified model of the environment, such as the laws of physics, but not be able to compute all of the consequences of that model. This gives rise to the problem of logical uncertainty, where you would like to have beliefs about facts that can be deduced or refuted from facts you already know, but you lack the ability to do this. This requires a unification of logic and probability, which is surprisingly hard.

Another consequence is that our agents will need to have high-level world models -- they need to be able to talk about things like chairs and tables as atoms, rather than thinking of everything as a quantum wavefunction. They will also have to deal with the fact that the high-level models will often conflict with models at lower levels, and that models at any level could shift and change without any change to models at other levels. An _ontological crisis_ occurs when there is a change in the level at which our values are defined, such that it is not clear how to extrapolate our values to the new model. An analogy would be if our view of the world changed such that "happiness" no longer seemed like a coherent concept.

As always, we also have problems with self-reference -- naturalized induction is the problem of learning a world model that includes the agent, and anthropic reasoning requires you to figure out how many copies of yourself exist in the world.


This is a lot closer to the problem I think about frequently (since I focus on the principal-agent problem between a human and an AI) so I have a lot of thoughts about this, but they'd take a while to untangle and explain. Hopefully, a lot of these intuitions will be written up in the second part of the value learning sequence.

Robust Delegation

Presumably, we will want to build AI systems that become more capable as time goes on, whether simply by learning more or by constructing a more intelligent successor agent (i.e. self-improvement). In both cases, the agent would like to ensure that its future self continues to apply its intelligence in pursuit of the same goals, a problem known as Vingean reflection. The main issue is that the future agent is "bigger" (more capable) than the current agent, and so the smaller agent cannot predict it. In addition, from the future agent's perspective, the current agent may be irrational, may not know what it wants, or could be made to look like it wants just about anything.

When constructing a successor agent, you face the value loading problem, where you need to specify what you want the successor agent to do, and you need to get it right because [optimization amplifies](https://www.lesswrong.com/posts/zEvqFtT4AtTztfYC4/optimization-amplifies) ([AN #13](https://mailchi.mp/8234356e4b7f/alignment-newsletter-13)) mistakes, in particular via Goodhart's Law. There's a discussion of the types of Goodhart's Law (also described in [Goodhart Taxonomy](https://www.lesserwrong.com/posts/EbFABnst8LsidYs5Y/goodhart-taxonomy)). Another issue that arises in this setting is that the successor agent could take over the representation of the reward function and make it always output the maximal value, a phenomenon called "wireheading", though this can be avoided if the agent's plan to do this is evaluated by the current utility function.

One hope is to create the successor agent from the original agent through intelligence amplification, along the lines of [iterated amplification](https://blog.openai.com/amplifying-ai-training/). However, this requires the current small agent to be able to decompose arbitrary problems, and to ensure that its proposed decomposition doesn't give rise to malign subcomputations, a problem to be described in the next post on subsystem alignment.


This is an important step forward for curiosity-driven agents. As the authors note in the paper, RND has the additional advantages of being simple to implement and flexible.

Reinforcement Learning with Prediction-Based Rewards

Researchers at OpenAI have beaten average human performance on Montezuma's Revenge using a prediction-based curiosity technique called Random Network Distillation. A network with fixed random weights evaluates each state; another network with the same architecture is trained to predict the random network's output, given its input. The agent receives an additional reward proportional to the predictor's error on its current state. The idea behind the technique is that the predictor's error will be higher on states different from those it's been trained on, and so the agent will be rewarded for exploring them.

This paper follows from their [study on curiosity](https://arxiv.org/abs/1808.04355) ([AN #20](https://mailchi.mp/d92bd0fefc83/alignment-newsletter-20)) in which a predictor was trained to predict the next state directly, and the agent was rewarded when its error was high. However, this led to high reward on states that were unpredictable due to model limitations or stochasticity (e.g. the noisy TV problem). By contrast, Random Network Distillation only requires the prediction of a deterministic function which is definitely within the class of functions representable by the predictor (since it has the same architecture as the random network).


For all that this paper's technique is a fairly straightforward merging of existing techniques (separately-prioritized demonstration and agent pools, and the off-policy SotA R2D2), its results are surprisingly impressive: the tasks tested on require long and complex chains of correct actions that would be challenging for a non-imitation based system to discover, and high levels of environment stochasticity that make a pure imitation approach difficult.

Making Efficient Use of Demonstrations to Solve Hard Exploration Problems

Caglar Gulcehre*, Tom Le Paine*, Bobak Shahriari, Misha Denil, Matt Hoffman, Hubert Soyer, Richard Tanburn, Steven Kapturowski, Neil Rabinowitz, Duncan Williams, Gabriel Barth-Maron, Ziyu Wang, Nando de Freitas, Worlds Team

This paper combines ideas from existing techniques to construct an architecture (R2D3) capable of learning to solve hard exploration problems with a small number (N~100) of demonstrations. R2D3 has two primary architectural features: its use of a recurrent head to learn Q values, and its strategy of sampling trajectories from separate pools of agent and demonstrator experience, with sampling prioritized by highest-temporal-difference-error transitions within each pool.

As the authors note, this approach is essentially an extension of an earlier paper, [Deep Q-Learning from Demonstrations](https://arxiv.org/abs/1704.03732), to use a recurrent head rather than a feed-forward one, allowing it to be more effectively deployed on partial-information environments. The authors test on 8 different environments that require long sequences of task completion to receive any reward, and find that their approach is able to reach human level performance on four of the tasks, while their baseline comparisons essentially never succeed on any task. Leveraging demonstrations can be valuable for solving these kinds of difficult exploration tasks, because demonstrator trajectories provide examples of how to achieve reward in a setting where the trajectories of a randomly exploring agent would rarely ever reach the end of the task to find positive reward.


I enjoyed reading the article and think it is a good starting point for people who want to learn more about exploration. Sadly, safe exploration where potential negative consequnces of some explorative actions are taken into account was outside of the article's scope.

Exploration Strategies in Deep Reinforcement Learning

A good exploration strategy is critical for fast reinforcement learning. This blog post presents two key problems and a wide array of strategies that have been proposed to deal with them. The **hard-exploration problem** is about sparse or deceptive rewards which make occasional random exploration next to useless. **The noisy-TV problem** is about a pitfall of directly rewarding agents for seeking novel experience: If there was a TV with unpredictable noise outputs in the environment, the agent would be rewarded for sitting in front of the TV and might not learn anything new.

Most of the discussed strategies are intrinsic reward schemes, where an additional reward is given to the agent for exploring new states. One way of doing this is count-based exploration, where the bonus reward depends on how often a state has been visited before. This can be extended to high-dimensional state spaces using density models or discretization. Another way is based on learning a predictor for features of the next state and rewarding the agent proportional to the <@predictor's error@>(@Reinforcement Learning with Prediction-Based Rewards@). An alternative is to learn multiple predictors and rewarding the agent for <@reaching states where they disagree@>(@Self-Supervised Exploration via Disagreement@). One problem with learnt predictors is that they only update slowly. This can be circumvented by combining the approach with episodic memory and a second intrinsic reward based on the distance (either euclidean or based on <@reachability@>(@Episodic Curiosity through Reachability@)) from states that were previously visited in the same episode. <@Agent57@>(@Agent57: Outperforming the human Atari benchmark@) combined this idea with a population of policies with different hyperparameters for the intrinsic reward and a meta-controller for prioritization of the most promising exploration policy.

Other strategies include basing exploration on uncertainty in Q-value estimates, learning options or "skills" that encode a wide range of different behaviours <@Variational Option Discovery Algorithms@> or using either an explicit memory or a <@goal-conditioned policy@>(@Learning Actionable Representations with Goal-Conditioned Policies@) to reach informative states and start random exploration from there.


I found this approach really elegant and clever as a way of addressing the "static TV" problem in curiosity literature. I'd be curious to see more work that introduces even stronger incentives towards diversity among the ensemble models (different architectures, even more different datasets they're trained on), to see if that amplifies the cases of model disagreement.

Self-Supervised Exploration via Disagreement

Deepak Pathak*, Dhiraj Gandhi*, Abhinav Gupta

For researchers who want to build a reinforcement learning system that can learn to explore its environment without explicit rewards, a common approach is to have the agent learn a model of the world, and incentivize it to explore places where its model has the highest error, under the theory that these represent places where it needs to interact more to collect more data and improve its world model. However, this approach suffers in cases when the environment is inherently stochastic, since in a stochastic environment  (think: sitting in front of a static TV and trying to predict the next frame), prediction error can never be brought to zero, and the agent will keep interacting even when its world model has collected enough data to converge as much as it can.  This paper proposes an alternative technique: instead of exploring in response to prediction error, learn an ensemble of bootstrapped next-state prediction models and explore in response to variance or disagreement between the models. This has a few nice properties. One is that, in cases of inherent stochasticity, all models will eventually converge to predicting the mean of the stochastic distribution, and so even though they've not brought error down to zero, the variance among models will be low, and will correctly incentivize our agent to not spend more time trying to learn. Another benefit is that since the reward is purely a function of the agent's models, it can be expressed analytically as a function of the agent's choices and trained via direct backpropogation rather than "black box reward" RL, making it more efficient.


This is easier to read than the paper and more informative than our summaries, so I'd recommend it if you were interested in the paper.

Curiosity and Procrastination in Reinforcement Learning

This blog post explains [Episodic Curiosity through Reachability](http://arxiv.org/abs/1810.02274), discussed in [AN #28](https://mailchi.mp/df2e472140b6/alignment-newsletter-28). As a reminder, this method trains a neural net to predict whether two observations were close in time to each other. Recent observations are stored in memory, and the agent is rewarded for reaching states that are predicted to be far away from any observations in memory.


This paper is a useful advance which does help address the couch potato problem, but it seems like it might still fail on similar problems. For example, suppose an agent were given a piece of paper on which it could doodle. Then states with lots of ink are far away from states with little ink, and so it might be rewarded for doodling forever (assuming a perfect model of reachability). My guess is that a model-based metric for novelty will be necessary to counter such problems - but it's also plausible that we end up using combinations of techniques like this one.

Episodic Curiosity through Reachability

Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Marc Pollefeys, Timothy Lillicrap, Sylvain Gelly

This paper addresses the "couch potato" problem for intrinsic curiousity - the fact that, if you reward an agent for observing novel or surprising states, it prefers to sit in front of a TV and keep changing channels rather than actually exploring. It proposes instead rewarding states which are difficult to reach from already-explored states (stored in episodic memory). Their agent has a separate network to estimate reachability, which is trained based on the agent's experiences (where observations few steps apart are negative examples and those many steps apart are positive examples). This method significantly outperforms the previous state of the art curiousity method on VizDoom and DMLab environments.


I’m a big fan of <@thinking harder about metrics and evaluation@>(@Evaluating the Robustness of Collaborative Agents@), and I do generally like the approach of “just look at a bunch of proxy statistics to help understand what’s happening”. However, I’m not sure how much I believe in the ones used in this paper -- with the exception of task reward, they are computed by downsampling the pixel inputs _really_ far (8x8 with each pixel taking on 4 possible values), in order to create a nice discrete distribution that they can compute objectives over. While you can downsample quite a lot without losing much important information, this seems too far to me.

I’m also a big fan of their choice of Minecraft as one of their environments. The issue with Atari is that the environments are too “linear” -- either you do the thing that causes you to score points or win the game, or you die; unsurprisingly many objectives lead to you scoring points. (See the <@large-scale study on curiosity@>(@Large-Scale Study of Curiosity-Driven Learning@).) However, on Minecraft there doesn’t seem to be much difference between the agents -- you’d be hard-pressed to tell the difference between the random agent and the trained agents based only on the values of the proxy objectives. To be fair, this may not be a problem with the objectives: it could be that the agents haven’t been trained for long enough (they were trained for 12 million steps, because the Minecraft simulator is quite slow).

Evaluating Agents without Rewards

Brendon Matusch, Jimmy Ba, Danijar Hafner

How can we evaluate algorithms for exploration? This paper suggests that we look at a variety of proxy objectives, such as reward obtained, similarity to human behavior, empowerment, and entropy of the visited state distribution.

The authors evaluate two algorithms ([ICM](https://arxiv.org/abs/1705.05363) and <@RND@>(@Reinforcement Learning with Prediction-Based Rewards@)) as well as three baselines (noop agent, random agent, and PPO) on three Atari games and the <@Minecraft TreeChop task@>(@NeurIPS 2019 Competition: The MineRL Competition on Sample Efficient Reinforcement Learning using Human Priors@), producing a list of proxy objective values for each combination. Their analysis then concludes that intrinsic objectives correlate with human behavior more strongly than task rewards do.


Planning to reach states where a lot of information is gained seems like a very promising strategy for exploration. I am not sure whether building sufficiently precise world models is always as feasible as model-free RL. If it was, misspecified rewards and similar problems would probably become easier to catch, as rollouts of a policy using a precise world model can help us predict what kind of worlds this policy produces without deployment. On the other hand, the improved capabilities for transfer learning could lead to more ubiquitous deployment of RL systems and amplify remaining failure modes, especially those stemming from <@multiagent interactions@>(@Multiparty Dynamics and Failure Modes for Machine Learning and Artificial Intelligence@).

Planning to Explore via Self-Supervised World Models

Ramanan Sekar*, Oleh Rybkin*, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, Deepak Pathak

<@PlaNet@>(@Learning Latent Dynamics for Planning from Pixels@) learns a latent world model which can be used for planning, and <@Dreamer@>(@Dream to Control: Learning Behaviors by Latent Imagination@) extends the idea by performing RL within the learned latent world model instead of requiring interaction with the environment. However, we still need to efficiently explore the real environment to obtain training data for the world model.

The authors propose to augment Dreamer with a novel exploration strategy. In addition to the learned latent world model, an ensemble of simpler one-step world models is trained and the magnitude of disagreement within the ensemble for a state is used as a proxy for the information gain for reaching that state. This is used as a (dynamically changing) intrinsic reward that can guide planning. By training Dreamer on this intrinsic reward, we can identify informative states in the real environment without having to first visit similar states as would be the case with e.g. curiosity, where the intrinsic reward is computed in retrospect.

The resulting system achieves state of the art zero-shot learning on a variety of continuous control tasks, and often comes close to the performance of agents that were trained for the specific task.


I’m not particularly impressed by this result, for a couple of reasons. Firstly, I think that exploiting determinism by resetting the environment (or even just memorising trajectories) fundamentally changes the nature of the problem posed by hard Atari games. Doing so allows us to solve them in the same ways as any other search problem - we could, for instance, just use the AlphaZero algorithm to train a value network. In addition, the headline results are generated by hand-engineering features like x-y coordinates and room number, a technique that has been eschewed by most other attempts. When you take those features away, their agent’s total reward on Pitfall falls back to 0.

Montezuma’s Revenge Solved by Go-Explore, a New Algorithm for Hard-Exploration Problems

Adrien Ecoffet,  Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune

This blog post showcases an agent which achieves high scores in Montezuma's Revenge and Pitfall by keeping track of a frontier of visited states (and the trajectories which led to them). In each training episode, a state is chosen from the frontier, the environment is reset to that state, and then the agent randomly explores further and updates the frontier. The authors argue that this addresses the tendency of intrinsic motivation algorithms to forget about promising areas they've already explored. To make state storage tractable, each state is stored as a downsampled 11x8 image.

The authors note that this solution exploits the determinism of the environment, which makes it brittle. So they then use imitation learning to learn a policy from demonstrations by the original agent. The resulting agents score many times higher than state-of-the-art on Montezuma's Revenge and Pitfall.


I really enjoyed this paper, and my summary doesn't do it justice -- it makes several other good points. I feel similarly about alignment: I feel relatively pessimistic about formal definitions of concepts like <@goal-directedness@>(@Intuitions about goal-directed behavior@) or <@safe exploration@>(@Safety Gym@), and feel much better about schemes that don't assume a formal definition of concepts and instead learn them from humans (or don't require them at all).

Another thing that jumped out at me was that their description of the non-ideal mode of theorizing focuses a _lot_ on understanding what exactly is going on, which is very similar to the concepts of interpretability and <@universality@>(@Towards formalizing universality@) in alignment.

Algorithmic Fairness from a Non-ideal Perspective

The field of fairness has aimed to develop objective metrics of fairness, which can then be optimized for in order to produce a just AI system. Unfortunately, many intuitively desirable fairness metrics are fundamentally incompatible, and cannot be simultaneously achieved except in special circumstances. Should we lose all hope for fairness?

This paper argues that the problem was that we were building _idealized_ theories, referring to a conception from political philosophy of ideal and non-ideal modes of theorizing. An ideal theory is one that describes an optimal, ideal world, and then identifies injustices by searching for discrepancies between the real world and the idealized one. This leads to three major flaws:

1. It can lead to systematic neglect of some injustices and distortions of our understanding of other injustices. For example, group parity metrics of fairness applied to college admissions would identify east Asian students as privileged relative to white students despite historical and institutional discrimination.

2. It does not offer sufficient practical guidance about what should be done, sometimes leading to misguided mitigation strategies. Consider college admissions again. A _disparate learning process_ aims to be blind to protected characteristics (like gender) while still achieving demographic parity. This forces the model to penalize features that correlate with being male. As a result, we end up rewarding women who go into female-dominated fields, and penalize women who go into male-dominated fields! This was presumably not what we wanted.

3. It does not make clear who among decision-makers is responsible for intervening to correct specific injustices.

The authors suggest that the research community move towards a non-ideal mode of theorizing, in which there is more emphasis on having a deep empirical understanding of the problem (including the various causal factors, rather than summary statistics), and using empirically-informed choices of treatments, rather than modifying ML algorithms to optimize a mathematically defined metric.


A new course to teach people about fairness in machine learning

Google has added a short section on fairness to their Machine Learning Crash Course (MLCC).


This actually feels quite related to the value alignment problem -- in general, we care about things besides fairness, and if we try to optimize directly for fairness, then we'll be giving up good outcomes on other dimensions. It's another case of Goodhart's law, where "fairness" was a proxy for "good for disadvantaged groups".

Delayed Impact of Fair Machine Learning

Lydia T. Liu, Sarah Dean, Esther Rolf, Max Simchowitz, Moritz Hardt

Consider a bank that has to choose which loan applications should be approved based on a credit score. Typically, fairness in this setting is encoded by saying that there should be some sort of parity between groups (and different criteria have been proposed for what actually should be the same). However, if you model the actual outcomes that come from the decision (namely, profit/loss to the bank _and_ changes in credit score to the applicant), you can see that standard fairness criteria lead to suboptimal outcomes. As a result, in general you want to look at the delayed impact of ML models.


I'm really interested in the techniques and results here, since there's a clear, sharp distribution shift from the training set to the test set, which is always hard to deal with. Hopefully some of the entries will have general solutions which we can adapt to other settings.

Introducing the Inclusive Images Competition

The authors write, "this competition challenges you to use Open Images, a large, multilabel, publicly-available image classification dataset that is majority-sampled from North America and Europe, to train a model that will be evaluated on images collected from a different set of geographic regions across the globe". The results will be presented at NIPS 2018 in December.


The case for building expertise to work on US AI policy, and how to do it

This in-depth career review makes the case for working on US AI policy. It starts by making a short case for why AI policy is important; and then argues that US AI policy roles in particular can be very impactful (though they would still recommend a policy position in an AI lab like DeepMind or OpenAI over a US AI policy role). It has tons of useful detail; the only reason I'm not summarizing it is because I suspect that most readers are not currently considering career choices, and if you are considering your career, you should be reading the entire article, not my summary. You could also check out [Import AI's summary](https://jack-clark.net/2019/02/04/import-ai-132-can-your-algorithm-outsmart-the-obstacle-tower-cross-domain-nlp-with-biobert-and-training-on-faceforensics-to-spot-deepfakes/).


I really like this introduction, it remains understandable while being technically accurate. It will probably be my new default article to introduce people to the problem.

The case for taking AI seriously as a threat to humanity

This is an introduction to the problem of AI safety, from the perspective that it is hard to specify the "right" goal, and that goal-driven behavior leads to convergent instrumental subgoals that will likely be dangerous. It also addresses several common initial reactions that people have.


Anyone who wants to get into AI safety (and isn't already an AI researcher) should listen to this podcast - there's a lot of useful information in it and [this career transition guide](https://80000hours.org/articles/ml-engineering-career-transition-guide/). I agree that having more research engineers is very valuable, and that it's a relatively easy transition for people with CS backgrounds to make. (I may be a little biased on this point, though, since it's also the path I'm currently taking.)

I think the issue of PhDs and mentorship is an important and complicated one. The field of AI safety is currently bottlenecked to a significant extent by the availability of mentorship, and so even a ML PhD unrelated to safety can still be very valuable if it teaches you how to do good independent research and supervise others, without requiring the time of current safety researchers. Also note that the trade-offs involve vary quite a bit. In particular, European PhDs can be significantly shorter than US ones; and the one-year Masters degrees available in the UK are a quick and easy way to transition into research engineering roles.

Concrete next steps for transitioning from CS or software engineering into ML engineering for AI safety and alignment

The fastest way into a high-impact role as a machine learning engineer, according to Catherine Olsson & Daniel Ziegler

Catherine Olsson, Daniel Ziegler, and Rob Wiblin

Catherine and Daniel both started PhDs, but left to work on AI safety (they're currently at Google Brain and OpenAI respectively). They note that AI safety teams need research engineers to do implementation work, and that talented programmers can pick up the skills required within a few months, without needing to do a PhD. The distinction between research engineers and research scientists is fairly fluid - while research engineers usually work under the direction of a research scientist, they often do similar things.

Their advice on developing the skills needed to get into good research roles is not to start with a broad theoretical focus, but rather to dive straight into the details. Read and reimplement important papers, to develop technical ML expertise. Find specific problems relevant to AI safety that you're particularly interested in, figure out what skills they require, and focus on those. They also argue that even if you want to eventually do a PhD, getting practical experience first is very useful, both technically and motivationally. While they're glad not to have finished their PhDs, doing one can provide important mentorship.

This is a long podcast and there's also much more discussion of object-level AI safety ideas, albeit mostly at an introductory level.


FAQ: Advice for AI Alignment Researchers

I've written an FAQ answering a broad range of AI alignment questions that people entering the field tend to ask me. Since it's a meta post, i.e. about how to do alignment research rather than about alignment itself, I'm not going to summarize it here.


I feel pretty unqualified to have an opinion here on how strong this effect is -- it's pretty far outside of my experience. At the very least it's a consideration we should be aware about, and Critch supports it better in the full post, so I'd recommend you read it.

Critch on career advice for junior AI-x-risk-concerned researchers

A common piece of advice for aspiring AI x-risk researchers is to work on AI capabilities research in order to skill up so they can later contribute to safety. However, Critch is worried that such researchers will rationalize their work as being "relevant to safety", leading to a false sense of security since AI researchers are now surrounded by people who are "concerned about safety", but _aren't actually doing safety research_. Note that Critch would still advise young researchers to get into grad school for AI, but to be aware of this effect and not feel any pressure to do safety research and to avoid rationalizing whatever research they are doing.


I strongly recommend this post to anyone looking to get into research -- it's a great post; I'm not summarizing it because I want this newsletter to be primarily about technical research. Even if you are not planning to do the type of research that MIRI does, I think this post presents a very different perspective on how to do research compared to the mainstream view in academia. Note though that this is _not_ the advice I'd give to someone trying to publish papers or break into academia. Also, while I'm talking about recommendations on how to do research, let me also recommend [Research as a Stochastic Decision Process](https://cs.stanford.edu/~jsteinhardt/ResearchasaStochasticDecisionProcess.html).

Alignment Research Field Guide

This post gives advice on how to get started on technical research, in particular by starting a local MIRIx research group.


Current AI Safety Roles for Software Engineers

This post and its comments summarize the AI safety roles available for software engineers (including ones that don't require ML experience).


List of AI safety courses and resources

Exactly what it says in the title.


AI Safety Career Bottlenecks Survey Responses Responses

A past survey asked for respondents' wish list of things that would be helpful and/or make them more efficient (with respect to careers in AI safety). This post provides advice for some of these wishes. If you're trying to break into AI safety work, this seems like a good source to get ideas on what to try or resources that you hadn't previously seen.


AI x-risk reduction: why I chose academia over industry

This post and its comments discuss considerations that impact whether new PhD graduates interested in reducing AI x-risk should work in academia or industry.


AGI Safety Fundamentals curriculum and application

This post presents the curriculum used in the AGI safety fundamentals course, which is meant to serve as an effective introduction to the field of AGI safety.


This is a nice concise case to make, but I think the bulk of the work is in splitting the first claim into subclaims: this is the part that is usually a sticking point (see also the next summary).

What I talk about when I talk about AI x-risk: 3 core claims I want machine learning researchers to address.

When making the case for work on AI x-risk to other ML researchers, what should we focus on? This post suggests arguing for three core claims:

1. Due to Goodhart's law, instrumental goals, and safety-performance trade-offs, the development of advanced AI increases the risk of human extinction non-trivially.
2. To mitigate this x-risk, we need to know how to build safe systems, know that we know how to build safe systems, and prevent people from building unsafe systems.
3. So, we should mitigate AI x-risk, as it is impactful, neglected, and challenging but tractable.


Fixed Point Discussion

This post discusses the various fixed point theorems from a mathematical perspective, without commenting on their importance for AI alignment.


I'm not an expert on agent foundations, so I don't have an opinion worth saying here. I'm not going to cover the posts with exercises in the newsletter -- visit the [Alignment Forum](https://www.alignmentforum.org/) for that. I probably will cover the posts about how the theorems relate to agent foundations research.

Fixed Point Exercises

Scott's advice to people who want to learn math in order to work on agent foundations is to learn all of the fixed-point theorems across the different areas of math. This sequence will present a series of exercises designed to teach fixed-point theorems, and will then talk about core ideas in the theorems and how the theorems relate to alignment research.


Even though access to compute has surely helped with increased efficiency in ways that I would not really label as algorithmic progress (for example by enabling researchers to try more different hyperparameters), the aggregated numbers seem surprisingly high. This suggests that I either had not correctly internalized what problems AI is able to solve these days, or underestimated the difficulty of solving these problems. It would be quite interesting to see whether there are similar improvements in the sample efficiency of deep reinforcement learning, as I expect this to be a major bottleneck for the application of agentic AIs in the absence of accurate simulators for real-world decision making.

AI and Efficiency

Given the <@exponential increase@>(@AI and Compute@) in compute used for state-of-the-art results in ML, one might come to think that there has been little algorithmic progress. This paper presents strong evidence against that hypothesis. We can roughly measure algorithmic progress by tracking the compute needed to achieve a concrete performance benchmark over time. Doing so yields doubling times in efficiency (time until only half of the initial compute was needed for the same performance) of around 16 months for ImageNet, which is faster than Moore's law. Other tasks like translation as well as playing Go and Dota 2 exhibit even faster doubling times over short periods. As making a task feasible for the first time arguably presents more algorithmic progress than improving the efficiency of solving an already feasible task, actual progress might be even faster than these numbers suggest. However, the amount of data points is quite limited and it is unclear if these trends will persist and whether they will generalize to other domains. Still, the authors conjecture that similar trends could be observed for tasks that received large amounts of investment and have seen substantial gains in performance.

Combining these results with the increased available compute over time, the authors estimate that the effective training compute available to the largest AI experiments has increased by a factor of 7.5 million (!) in 2018 relative to 2012.

A focus on efficiency instead of top performance allows actors with limited amounts of compute to contribute. Furthermore, models that reach a particular benchmark quickly seem like strong candidates for scaling up. This way, more efficient algorithms might act as a catalyst for further progress. There is a public [git repository](https://github.com/openai/ai-and-efficiency) to keep better track of algorithmic efficiency.


Ignoring the George Washington Bridge (which confuses both me and the authors), I’d roughly categorize the causes of these discontinuities as
- 3 of them were due to a concerted but apparently misplaced effort towards something others weren’t trying to do. These are Pyramid of Djoser, SS Great Eastern, and the Paris Gun.
- 2 of them were due to the Atlantic Ocean causing a threshold effect (as they explain in the post). These are the ICBM and the first nonstop transatlantic flight.
- 4 of them were due to a new technological breakthrough followed by increased investment and a faster rate of progress. These are the two telegraph cables, nuclear weapons, and superconductors.

Of these, the final category seems the most relevant to AGI timelines, and I could imagine AGI development following a similar trajectory, where a major breakthrough causes a large amount of investment and then we have much faster progress on AI going forward.

I was quite surprised that AlexNet did not represent a discontinuity on ImageNet performance. It is widely regarded to have kicked off deep learning in the computer vision community. I’m not sure if this is because the discontinuity metric they use doesn’t correspond with my sense of a “breakthrough”, because there were only two years of ImageNet beforehand, or because the vision community is just mistakenly attributing gradual progress to one major event.

**Rohin's opinion:** I agree with Nicholas that the final category seems most relevant to AI progress. Note though that even for this analogy to hold, you need to imagine a major AI breakthrough, since as Nicholas pointed out, these discontinuities were caused by a radically new technology (telegraph cables replacing ships, nuclear weapons replacing conventional bombs, and ceramic superconductors replacing alloy superconductors). This doesn't seem likely in worlds where progress is driven primarily by <@compute@>(@AI and Compute@), but could happen if (as academics often suggest) deep learning hits a wall and we need to find other AI algorithms to make progress.

Discontinuous progress in history: an update

One of the big questions in AI alignment is whether there will be a discontinuous AI takeoff (see <@here@>(@Clarifying some key hypotheses in AI alignment@) for some reasons why the question is decision-relevant). To get a better outside view, AI Impacts has been looking for large discontinuities in historical technological trends. A discontinuity is measured by how many years ahead of time that value is reached, relative to what would have been expected by extrapolating the trend.

They found ten 100-year discontinuous events, for example in ship size (The SS *Great Eastern*), the average speed of military payload across the Atlantic Ocean (the first ICBM), and the warmest temperature of superconduction (yttrium barium copper oxide).

There are also some interesting negative examples of discontinuities. Particularly relevant to AI are AlexNet not being a discontinuity on the ImageNet benchmark and chess performance not having any discontinuities in Elo rating.


I don’t have a strong opinion between which of these two questions is most important to focus on. It makes sense to me to work on them both in parallel since the data required is likely to be the same. My concern with this approach is that there is no clear denominator to the discontinuities they find. The case studies convince me that discontinuities **can** happen, but I really want to know the **frequency** with which they happen.

**Rohin's opinion:** Given that we want to use this to forecast AI progress, it seems like we primarily care about the second question (simulated prediction). However, it's _really hard_ to put yourselves in the shoes of someone in the past, making sure to have exactly the information that was available at the time; as a result I broadly agree with the decision to focus more on a description of what happened.

Description vs simulated prediction

AI Impacts' investigation into discontinuous progress intends to answer two questions:
1. How did tech progress happen in the past?
2. How well could it have been predicted beforehand?

These can diverge when we have different information available now than in the past. For example, we could have more information because later data clarified trends or because the information is more accessible. We might have less information because we take an outside view (looking at trends) rather than an inside view (knowing the specific bottlenecks and what might need to be overcome).

The post then outlines some tradeoffs between answering these two questions and settles on primarily focusing on the first: describing tech progress in the past.


I did know that the amount of compute used was growing fast, but a 3.5 month doubling time _for 6 years running_ is _huge_, and there's no reason to expect that it will stop now. It's also interesting to see what made it onto the graph -- there's image classification, machine translation, and neural architecture search (all of which have clear economic incentives), but some of the largest ones are by projects aiming to build AGI (AlphaGo Zero, AlphaZero, and Dota). Notably, deep reinforcement learning just barely makes it on the graph, with DQN two orders of magnitude lower than any other point on the graph. I'm really curious what deep RL could solve given AlphaGo levels of compute.

AI and Compute

Since 2012, when the deep learning revolution began with AlexNet, the amount of compute used in the largest-scale experiments has been doubling _every 3.5 months_. Initially, people started to use GPUs to scale up, but there wasn't a huge amount o f interest. In 2014-16, as interest in deep learning really began to take off, people started to use a lot of compute to get good results -- but parallelism stopped helping beyond a certain point (~100 GPUs) because the parameter updates from the data were becoming too stale. Since then, we've had algorithmic improvements that allow us to take advantage of more parallelism (huge batch sizes, architecture search, expert iteration), and this has let us scale up the amount of compute thrown at the problem.


The surprising part of the data is how fast compute has been growing. One common part of AGI timelines is whether you think compute or algorithms are the bottleneck. Assuming you had a good sense of progress in AI, but were surprised by how fast compute grew, you should update against the relative benefits of compute.

This post seems to be about the way you relate compute to AGI timelines, ignoring algorithms altogether. If you think of AGI as requiring a specific amount of compute that is determined independently of current AI progress (for example, by estimating the compute used by a human brain), then the evidence should shorten your timelines. If you instead predict how close AGI is by looking at the rate of progress in AI and extrapolating over time, then to first order this data should not affect timelines (since compute is not part of the model), and to second order it should lengthen them for the reasons in this post.

Reinterpreting “AI and Compute”

[Data](https://blog.openai.com/ai-and-compute/) from OpenAI showed that the amount of compute used by the most expensive projects had been growing exponentially with a doubling time of three months. While it is easy to interpret this trend as suggesting that we will get AGI sooner than expected, it is also possible to interpret this trend as evidence in the opposite direction. A surprisingly high rate of increase in amount of compute used suggests that we have been _overestimating_ how helpful more compute is. Since this trend [can't be sustainable over decades](https://aiimpacts.org/interpreting-ai-compute-trends/), we should expect that progress will slow down, and so this data is evidence _against_ near-term AGI.


While the previous paper focused on how we could make _maximal use_ of existing models for program synthesis, this paper is much more focused on how we can _measure_ the capabilities of models. This leads to quite a bit of difference in what they focus on: for example, the highlighted paper treats the strategy of generating multiple possible answers as a fundamental approach to study, while this paper considers it briefly in a single subsection.

Although the introductory problems in the APPS dataset seemed to me to be comparable to those in the MBPP dataset from the previous paper, models do significantly better on MBPP. A model slightly smaller than GPT-3 has a ~17% chance of solving a random MBPP problem in a single sample and ~10% if it is not given any example test cases; in contrast for introductory APPS problems GPT-3 is at 0.2%. I'm not sure whether this is because the introductory problems in APPS are harder, or if the format of the APPS problems is harder for the model to work with, or if this paper didn't do the prompt tuning that the previous paper found was crucial, or something else entirely.

Measuring Coding Challenge Competence With APPS

Dan Hendrycks*, Steven Basart*, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, Jacob Steinhardt

The APPS dataset measures programming competence by testing models the way humans are tested: we provide them with natural language descriptions of the code to be written and then evaluate whether the code they generate successfully solves the problem by testing the proposed solutions. The authors collect a dataset of 3,639 introductory problems (solvable by humans with 1-2 years of experience), 5,000 interview problems (comparable difficulty to interview questions), and 1,361 competition problems (comparable difficulty to questions in programming competitions). In addition, the test set contains 1,000 introductory problems, 3,000 interview problems, and 1,000 competition problems.

They use this benchmark to test four models: two variants of GPT-2 (0.1B params and 1.5B params), GPT-Neo (2.7B params), and GPT-3 (175B params). GPT-3 is prompted with examples; all other models are finetuned on a dataset collected from GitHub. The authors find that:

1. Finetuning makes a big difference in performance: GPT-3 only solves 0.2% of introductory problems, while the finetuned GPT-2-0.1B model solves 1% of such problems.
2. Model performance increases with size, as you would expect: GPT-Neo performs best, solving 3.9% of problems.
3. Syntax errors in generated code drop sharply as model performance improves: for introductory problems, GPT-3 has syntax errors in slightly under 40% of generations, while GPT-Neo has under 1%.
4. Performance can be improved by sampling the best of multiple generated programs: a beam search for 5 programs boosts GPT-Neo's performance from 3.9% to 5.5% on introductory problems.
5. While no model synthesizes a correct solution to a competition level program, they do sometimes generate solutions that pass some of the test cases: for example, GPT-Neo passes 6.5% of test cases.


I find the economic take on AI to be particularly interesting because it makes the “automation” frame on AI the default one, as opposed to the “superintelligent goal-directed agent” frame that we often work with in AI alignment. The critical assumption needed in this automation frame is that AI systems can automate ~every task that a human worker could do. This is what enables the positive feedback loop to work (which is the automation version of recursive self-improvement).

I generally prefer the automation frame for thinking about and predicting how AI systems are integrated into the world, while preferring the agent frame for thinking about how AI systems might cause _alignment_ problems (i.e. ignoring <@misuse and structural risks@>(@Thinking About Risks From AI: Accidents, Misuse and Structure@)). Many of my disagreements with <@CAIS@>(@Reframing Superintelligence: Comprehensive AI Services as General Intelligence@) feel like cases where I think it is appropriate to use the agent frame rather than the automation frame. I would classify several newer alignment <@risk@>(@What failure looks like@) <@stories@>(@Another (outer) alignment failure story@) as taking the same agent-based _cause_ of alignment failure as in (say) Superintelligence, but then telling a story in which the _deployment_ of the misaligned AI system is automation-based.

I think it is generally worth spending some time meditating on the growth models explored in this post, and what implications they would have for AI development (and thus for AI alignment). For example, some models emphasize that there are many different tasks and suggest (not conclusively) that we’ll have different AI systems for different tasks. In such a world, it doesn’t seem very useful to focus on teaching AI systems about humanity’s true values, as they are going to be asked to do particular tasks that are pretty divorced from these “true values”.

Note that I am not an economist. This means that there’s a higher chance than usual that I’ve accidentally inserted an erroneous claim into this summary and opinion. It is also the reason why I don’t usually summarize econ papers that are relevant to AI -- I’ve summarized this one because it’s explained at a level that I can understand. If you’re interested in this area, other papers include [Economic Growth Given Machine Intelligence](http://mason.gmu.edu/~rhanson/aigrow.pdf) and [Economic growth under transformative AI](https://globalprioritiesinstitute.org/philip-trammell-and-anton-korinek-economic-growth-under-transformative-ai/).

Could Advanced AI Drive Explosive Economic Growth?

<@Some@>(@Draft report on AI timelines@) <@previous@>(@Modeling the Human Trajectory@) <@work@>(@Semi-informative priors over AI timelines@) has suggested that by 2100 there is a non-trivial chance that AI could lead to _explosive growth_, that is, a growth rate of 30% (i.e. a doubling time of 2-3 years), 10x the current growth rate of ~3%. What does economics have to say about the matter?

This report investigates the following three stories:

1. **Ignorance story:** In this story, we don't know how growth is determined, and attempts to forecast it based on models of how growth works are likely to be wrong. Note that this is perfectly compatible with explosive growth. We know that the growth rate has increased by orders of magnitude over the past millennia; so on an ignorance story we certainly shouldn't rule out that the growth rate could increase by an order of magnitude again.

2. **Standard story:** This story focuses on the last ~century of growth, noting that the growth rate has stayed relatively constant at 2-3% per year, and thus predicting that future growth will be exponential (i.e. a constant growth rate), or possibly subexponential.

3. **Explosive story:** This story focuses on growth models with positive feedback loops, in which increased output leads to increased inputs which leads to even larger outputs, resulting in superexponential (and explosive) growth.

The author is interested in whether explosive growth is _plausible_, and so is most interested in arguments that argue for the standard story and against the ignorance or explosive stories, or vice versa. The main empirical facts we have are that the growth rate increased (maybe continuously, maybe not, it's hard to tell) until about a century ago, when it plateaued at the current level of 2-3%. What can we then learn from economic growth models?

1. Ideas-based models of economic growth suggest that growth in output is driven primarily by the rate at which we get ideas (leading to technological improvement), which in turn is driven by population size, which in turn is driven by output (completing the positive feedback cycle). This predicts increases in the growth rate _as long as_ population growth rate is increasing. A century ago, we underwent the "demographic transition" where, as we produced more output, instead of having more kids we became richer, breaking the positive feedback loop and preventing population size from growing. This fits our empirical facts well, and if we now assume that AI can also generate ideas, then the feedback loop is reestablished and we should expect explosive growth.

2. Economists have tried to find growth models that robustly predict exponential growth alongside a slowly growing population, but have mostly not found such models, suggesting that our current exponential growth might be an anomaly that will eventually change. The best explanations of exponential growth imply that future growth will be sub-exponential given that population growth is predicted to slow down.

3. Most economic growth models, including the ones in the previous point, predict explosive growth if you add in an assumption that AI systems can replace human workers.

Thus, it seems that economic growth theory suggests that explosive growth is probable, _conditional_ on the assumption that we develop AI systems that can replace arbitrary human workers.

You could object to these arguments on several grounds. The ones that the author finds partially convincing are:

1. We don't see any trends of explosive growth right now -- this suggests that we at least won't see explosive growth in the next couple of decades (though it's harder to make claims all the way out to 2100).
2. If there are a few key "bottleneck" tasks that (a) are crucial for growth and (b) can't be automated by AI, then those tasks may limit growth.
3. There may be physical limits on growth that we haven't yet encountered: for example, growth may be bottlenecked on running experiments in the real world, extracting and transporting raw materials, delays for humans to adjust to new technology, etc.

Another objection is that ideas are getting harder to find, which would surely prevent explosive growth. The author is _not_ convinced by this objection, because the growth models predicting explosive growth already take this into account, and still predict explosive growth. (Roughly, the superexponential increase in the inputs "overpowers" the exponential increase in the difficulty of finding good ideas.)


Ask and ye shall receive! In my <@last summary@>(@Scaling Laws for Autoregressive Generative Modeling@), I mentioned that I was uncertain about how cross-entropy loss translates to transformative progress that we care about, and here is an excellent post exploring just that question. I'm sure I'll end up referencing this many times in the future.

The post discusses both what benchmarks might suggest for forecasting "human equivalence" and how benchmarks might relate to economic value via concrete task automation. I agree with the tasks the author suggests for the latter, and continuing my "opinions as calls for more work" trend, I'd be interested in seeing even more work on this-- i.e. attempts to decompose tasks into a set of concrete benchmark performances which would be sufficient for economically valuable automation. [This comment thread](https://www.lesswrong.com/posts/k2SNji3jXaLGhBeYP/extrapolating-gpt-n-performance?commentId=x7cGXDCju6qsnzRfg) discusses whether current benchmarks are likely to capture a substantial portion of what is necessary for economic value, given that many jobs end up requiring a diverse portfolio of skills and reasoning ability. It seems plausible to me that AI-powered automation will be "discontinuous" in that a lot of it will be unlocked only when we have a system that's fairly general.

It seems quite noteworthy that the parameter estimates here and in the AI timelines report draft are close together, even though one is anchored to human-level benchmark performance, and the other is anchored to brain computation. That updates me in the direction of those numbers being in the right range for human-like abilities.

People interested in this post maybe also be interested in [BIG-bench](https://github.com/google/BIG-Bench), a project to crowdsource the mother of all benchmarks for language models.

Extrapolating GPT-N performance

This post describes the author's insights from extrapolating the performance of GPT on the benchmarks presented in the <@GPT-3 paper@>(@Language Models are Few-Shot Learners@). The author compares cross-entropy loss (which measures how good a model is at predicting the next token) with benchmark performance normalized to the difference between random performance and the maximum possible performance. Since <@previous work@>(@Scaling Laws for Neural Language Models@) has shown that cross-entropy loss scales smoothly with model size, data, and FLOP requirements, we can then look at the overall relationship between those inputs and benchmark performance.

The author finds that most of the benchmarks scale smoothly and similarly with respect to cross-entropy loss. Three exceptions are arithmetic, scramble (shuffling letters around the right way), and ANLI (a benchmark generated adversarially against transformer-based language models), which don't improve until the very end of the cross-entropy loss range. The author fits linear and s-shaped curves to these relationships, and guesses that:

- Performance improvements are likely to slow down closer to maximum performance, making s-curves a better progress estimate than linear curves.
- Machine learning models may use very different reasoning from humans to get good performance on a given benchmark, so human-level performance on any single benchmark would likely not be impressive, but human-level performance on almost all of them with few examples might be.
- We might care about the point where we can achieve human-level performance on all tasks with a 1 token "horizon length"-- i.e., all tasks where just 1 token is enough of a training signal to understand how a change in the model affects its performance. (See <@this AI timelines report draft@>(@Draft report on AI timelines@) for more on horizon length.) Achieving this milestone is likely to be _more_ difficult than getting to human-level performance on these benchmarks, but since scaling up GPT is just one way to do these tasks, the raw number of parameters required for this milestone could just as well be _less_ than the number of parameters that GPT needs to beat the benchmarks.
- Human-level performance on these benchmarks would likely be enough to automate lots of particular tasks with short horizon length, such as customer service, PA and RA work, and writing routine sections of code.

The author augments his s-curves graph with references to certain data, FLOP, and parameter levels, including the number of words in common crawl, the number of FLOPs that could currently be bought for $1B, the point where reading or writing one word would cost 1 cent, and the number of parameters in a transformative model according to <@this AI timelines report draft@>(@Draft report on AI timelines@). (I recommend looking at the graph of these references to see their relationship to the benchmark trends.)

Overall, the author concludes that:

- GPT-3 is in line with smooth performance on benchmarks predicted by smaller models. It sharply increases performance on arithmetic and scramble tasks, which the author thinks is because the tasks are 'narrow' in that they are easy once you understand their one trick. The author now finds it less likely that a small amount of scaling will suddenly lead to a large jump in performance on a wide range of tasks.
- Close to optimal performance on these benchmarks seems like it's at least ~3 orders of magnitude away ($1B at current prices). The author thinks more likely than not, we'd get there after increasing the training FLOP by ~5-6 orders of magnitude ($100B -$1T at current prices, $1B - $10B given estimated hardware and software improvements over the next 5 - 10 years). The author thinks this would probably not be enough to be transformative, but thinks we should prepare for that eventuality anyway.
- The number of parameters estimated for human-equivalent performance on these benchmarks (~1e15) is close to the median number of parameters given in <@this AI timelines report draft@>(@Draft report on AI timelines@), which is generated via comparison to the computation done in the human brain.


I would also recommend listening to [Jared Kaplan's talk](https://www.youtube.com/watch?v=QMqPAM_knrE) on this.

I was really excited to learn about more empirical work here. These results suggest that scaling behavior predictable with smooth power-laws is likely a feature of most generative models, not just text. I found it surprising that optimal model size given a compute budget scales the same way across data modalities-- it does seem to suggest that there's something more fundamental going on here that I don't understand (but which may be explained in [this theory paper](https://arxiv.org/abs/2004.10802) that I haven't read). It's also interesting that pretraining on a generative model (rather than training from scratch) seems to confer real benefits to scaling behavior for image classification-- this lends some support to the view that a lot of the learning that needs to happen will come from unsupervised settings.

A lot of the most salient questions around current scaling laws for me still lie in the translation between cross-entropy loss in these domains and performance on downstream tasks we care about. I feel very unsure about whether any of the fine-tuned generative models we (currently) have the data to train are likely to have transformative performance within even the next 5 orders of magnitude of compute scaling.

**Rohin's opinion:** In addition to the points Asya made above, I wanted to speculate on the implications of these scaling laws for AGI. I was particularly struck by how well these scaling laws seem to fit the data. This was also true in the case of mathematics problems, at least for the models we have so far, even though intuitively math requires “reasoning”. This suggests to me that even for tasks that require reasoning, capability will increase smoothly along a spectrum, and the term “reasoning” is simply a descriptor of a particular capability level. (An alternative position is that “reasoning” happens only to the extent that the neural net is implementing an algorithm that can justifiably be known to always output the right answer, but this sort of definition usually implies that humans are not doing reasoning, which seems like a deal-breaker.)

Note however that we haven't gotten to the level of performance that would be associated with "reasoning", so it is still _possible_ that the trends stop holding and reasoning then leads to some sort of discontinuous increase in performance. I just wouldn't bet on it.

the scaling “inconsistency”: openAI’s new insight

Scaling Laws for Autoregressive Generative Modeling

Tom Henighan*, Jared Kaplan*, Mor Katz*, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, Sam McCandlish

This paper looks at scaling laws for generative Transformer models of images (predicting pixels or parts of image encodings), videos (predicting frames of image encodings), multimodal image <-> text (predicting captions based on images or images based on captions), and mathematical problem solving (predicting answers to auto-generated questions about algebra, arithmetic, calculus, comparisons, integer properties, measurement, polynomials, and probability). The authors find that:

- Cross-entropy loss as a function of compute follows a power law + constant in all these data modalities (just as it does <@in language@>(@Scaling Laws for Neural Language Models@)). Information theoretically, this can be interpreted as scaling a 'reducible loss' which estimates the KL divergence between the true and model distributions, and an 'irreducible loss' which estimates the entropy of the true data distribution.
- Performance on ImageNet classification fine-tuned from their generative image model also follows such a power law, whereas ImageNet classification trained *from scratch* actually gets worse with sufficiently large model sizes. Interestingly, this classification power law continues even past model sizes where the generative cross-entropy loss starts bending as a result of irreducible loss. The authors conclude that approaching the irreducible loss for some dataset does not necessarily indicate diminishing returns for representation quality or semantic content.
- Optimal model size as a function of compute follows a power law with an exponent very close to ~0.7 for all data modalities they've studied so far. This implies that in the current compute regime, as compute budgets grow, it's best to devote a majority of compute towards making models bigger and a minority towards training on more data.
- Larger models perform better on extrapolating to math problems more difficult than those seen in training, but only insofar as they do better on the training distribution (no benefits to 'strong generalization').
- Larger models are able to take advantage of more multimodal information, but the scaling is extremely slow-- a 1-billion-parameter model uses 10% of the information in a caption to define an image, while using 20% of the information would require a 3-trillion-parameter model.

As in the <@language models paper@>(@Scaling Laws for Neural Language Models@), extrapolating the steep power laws found for optimally-used compute seems to eventually paradoxically result in loss lower than the bound given by shallower power laws for optimally-used training data. The authors offer a potential hypothesis for resolving this inconsistency-- in the regime of less compute and smaller model sizes, increasing model size effectively increases the amount of information you extract from each data point you train on, resulting in the steepness of the current compute law. As compute increases past a certain point, however, the amount of information extracted per data point approaches the maximum amount possible, so the curve switches to a shallower regime and marginal compute should be used increasingly on dataset increases rather than model size increases. If this hypothesis is true, we should eventually expect the scaling laws for compute to bend towards laws set by dataset size, and perhaps should think they will ultimately be set by trends for overfitting (see [this post](https://www.alignmentforum.org/posts/diutNaWF669WgEt3v/the-scaling-inconsistency-openai-s-new-insight) for another explanation of this).