Formal Inner Alignment, Prospectus

This post outlines a document that the author plans to write in the future, in which he will define the inner alignment problem formally, and suggest directions for future research. I will summarize that document when it comes out, but if you would like to influence that document, check out the post.


AXRP #4 - Risks from Learned Optimization

This podcast delves into a bunch of questions and thoughts around <@mesa optimization@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@). Here are some of the points that stood out to me (to be clear, many of these have been covered in this newsletter before, but it seemed worth it to state them again):

- A model is a mesa optimizer if it is a _mechanistic_ optimizer, that is, it is executing an algorithm that performs search for some objective.
- We need to focus on mechanistic optimizers instead of things that behave as though they are optimizing for some goal, because those two categories can have very different generalization behavior, and we are primarily interested in how they will generalize.
- Humans do seem like mesa optimizers relative to evolution (though perhaps not a central example). In particular, it seems accurate to say that humans look at different possible strategies and select the ones which have good properties, and thus we are implementing a mechanistic search algorithm.
- To reason about whether machine learning will result in these mechanistic optimizers, we need to reason about the _inductive biases_ of machine learning. We mostly don’t yet know how likely they are.
- Evan expects that in powerful neural networks there will exist a combination of neurons that encode the objective, which we might be able to find with interpretability techniques.
- Even if training on a myopic base objective, we might expect the mesa objective to be non-myopic, as the non-myopic objective "pursue X" is simpler than the myopic objective "pursue X until time T".
- We can’t rely on generalization bounds to guarantee performance, since in practice there is always some distribution shift (which invalidates those bounds).
- Although it is usually phrased in the train/test paradigm, mesa optimization is still a concern in an online learning setup, since at every time we are interested in whether the model will generalize well to the next data point it sees.
- We will probably select for simple ML models (in the sense of short description length) but not for low inference time, such that mechanistic optimizers are more likely than models that use more space (the extreme version being lookup tables).
- If you want to avoid mesa optimizers entirely (rather than aligning them), you probably need to have a pretty major change from the current practice of AI, as with STEM AI and Microscope AI (explained <@here@>(@An overview of 11 proposals for building safe advanced AI@)).
- Even in a <@CAIS scenario@>(@Reframing Superintelligence: Comprehensive AI Services as General Intelligence@) where we have (say) a thousand models doing different tasks, each of those tasks will still likely be complex enough to lead to the models being mesa optimizers.
- There are lots of mesa objectives which would lead to deceptive alignment relative to corrigible or internalized alignment, and so we should expect deceptive alignment a priori.


One of the most important approaches to improve inner alignment is to monitor the performance of your system online, and train to correct any problems. This paper shows the benefit of explicitly quantified, well-specified uncertainty: it allows you to detect problems _before they happen_ and then correct for them.

This setting has also been studied in <@delegative RL@>(@Delegative Reinforcement Learning@), though there the agent also has access to a reward signal in addition to a demonstrator.

Paper: Fully General Online Imitation Learning

Formal Solution to the Inner Alignment Problem

Michael K. Cohen, Marcus Hutter, Neel Nanda

Since we probably can’t specify a reward function by hand, one way to get an agent that does what we want is to have it imitate a human. As long as it does this faithfully, it is as safe as the human it is imitating. However, in a train-test paradigm, the resulting agent may faithfully imitate the human on the training distribution but fail catastrophically on the test distribution. (For example, a deceptive model might imitate faithfully until it has sufficient power to take over.) One solution is to never stop training, that is, use an online learning setup where the agent is constantly learning from the demonstrator.

There are a few details to iron out. The agent needs to reduce the frequency with which it queries the demonstrator (otherwise we might as well just have the demonstrator do the work). Crucially, we need to ensure that the agent will never do something that the demonstrator wouldn’t have done, because such an action could be arbitrarily bad.

This paper proposes a solution in the paradigm where we use Bayesian updating rather than gradient descent to select our model, that is, we have a prior over possible models and then when we see a demonstrator action we update our distribution appropriately. In this case, at every timestep we take the N most probable models, and only take an action a with probability p if **every** one of the N models takes that action with at least probability p. (There’s a specific rule that ensures that N decreases over time.) The total probability of all the actions will typically be less than 1 -- the remaining probability is assigned to querying the demonstrator.

The key property here is that as long as the true demonstrator is in the top N models, then the agent never autonomously takes an action with more probability than the demonstrator would. Therefore, as long as we believe the demonstrator is safe, the agent should be as well. Since the agent learns more about the demonstrator every time it queries them, over time it needs to query the demonstrator less often. Note that the higher N is, the more likely it is that the true model is one of those N models (and thus we have more safety), but also the more likely it is that we will have to query the demonstrator. This tradeoff is controlled by a hyperparameter α that implicitly determines N.


Fixing The Good Regulator Theorem

Consider a setting in which we must extract information from some data X to produce model M, so that we can later perform some task Z in a system S while only having access to M. We assume that the task depends only on S and not on X (except inasmuch as X affects S). As a concrete example, we might consider gradient descent extracting information from a training dataset (X) and encoding it in neural network weights (M), which can later be used to classify new test images (Z) taken in the world (S) without looking at the training dataset.

The key question: when is it reasonable to call M a model of S?
1. If we assume that this process is done optimally, then M must contain all information in X that is needed for optimal performance on Z.
2. If we assume that every aspect of S is important for optimal performance on Z, then M must contain all information about S that it is possible to get. Note that it is usually important that Z contains some new input (e.g. test images to be classified) to prevent M from hardcoding solutions to Z without needing to infer properties of S.
3. If we assume that M contains _no more_ information than it needs, then it must contain exactly the information about S that can be deduced from X.

It seems reasonable to say that in this case we constructed a model M of the system S from the source X "as well as possible". This post formalizes this conceptual argument and presents it as a refined version of the [Good Regulator Theorem](http://pespmc1.vub.ac.be/books/Conant_Ashby.pdf).

Returning to the neural net example, this argument suggests that since neural networks are trained on data from the world, their weights will encode information about the world and can be thought of as a model of the world.


Defining capability and alignment in gradient descent

Consider a neural network like GPT-3 trained by gradient descent on (say) the cross-entropy loss function. This loss function forms the _base objective_ that the process is optimizing for. Gradient descent typically ends up at some local minimum, global minimum, or saddle point of this base objective.

However, if we look at the gradient descent equation, θ = θ - αG, where G is the gradient, we can see that this is effectively minimizing the size of the gradients. We can think of this as the mesa objective: the gradient descent process (with an appropriate learning rate decay schedule) will eventually get G down to zero, its minimum possible value (even though it may not be at the global minimum for the base objective).

The author then proposes defining capability of an optimizer based on how well it decreases its loss function in the limit of infinite training. Meanwhile, given a base optimizer and mesa optimizer, alignment is given by the capability of the base optimizer divided by the capability of the mesa optimizer. (Since the mesa optimizer is the one that actually acts, this is effectively measuring how much progress on the mesa objective also causes progress on the true base objective.)

This has all so far assumed a fixed training setup (such as a fixed dataset and network architecture). Ideally, we would also want to talk about robustness and generalization. For this, the author introduces the notion of a “perturbation” to the training setup, and then defines [capability / alignment] [robustness / generalization] based on whether the optimization stays approximately the same when the training setup is perturbed.

It should be noted that these are all definitions about the behavior of optimizers in the infinite limit. We may also want stronger guarantees that talk about the behavior on the way to the infinite limit.


Confucianism in AI Alignment

Suppose we trained our agent to behave well on some set of training tasks. <@Mesa optimization@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@) suggests that we may still have a problem: the agent might perform poorly during deployment, because it ends up optimizing for some misaligned _mesa objective_ that only agrees with the base objective on the training distribution.

This post suggests that in any training setup in which mesa optimizers would normally be incentivized, it is not sufficient to just prevent mesa optimization from happening. The fact that mesa optimizers could have arisen means that the incentives were bad. If you somehow removed mesa optimizers from the search space, there would still be a selection pressure for agents that without any malicious intent end up using heuristics that exploit the bad incentives. As a result, we should focus on fixing the incentives, rather than on excluding mesa optimizers from the search space.


I’m glad that definitions are being made clear, especially since I usually use these terms differently than the author. In particular, as mentioned in my opinion on the highlighted paper, I expect performance to smoothly go up with additional compute, data, and model capacity, and there won’t be a clear divide between capability robustness and objective robustness. As a result, I prefer not to divide these as much as is done in this post.

Clarifying inner alignment terminology

This post clarifies the author’s definitions of various terms around inner alignment. Alignment is split into intent alignment and capability robustness, and then intent alignment is further subdivided into outer alignment and objective robustness. Inner alignment is one way of achieving objective robustness, in the specific case that you have a mesa optimizer. See the post for more details on the definitions.


Mesa-Search vs Mesa-Control

This post discusses several topics related to mesa optimization, and the ideas in it led the author to update towards thinking inner alignment problems are quite likely to occur in practice. I’m not summarizing it in detail here because it’s written from a perspective on mesa optimization that I find difficult to inhabit. However, it seems to me that this perspective is common so it seems fairly likely that the typical reader would find the post useful.


In a [comment](https://www.alignmentforum.org/posts/J9D6Bi3eFDDhCaovi/will-transparency-help-catch-deception-perhaps-not#yn5YcLnL6vs6AxxAE), Evan Hubinger makes a point I agree with: the transparency tools don't need to be able to detect all deception; they just need to prevent the model from developing deception. If deception gets added slowly (i.e. the model doesn't "suddenly" become perfectly deceptive), then this can be way easier than detecting deception in arbitrary models, and could be done by tools.

<@Relaxed adversarial training for inner alignment@>

Will transparency help catch deception? Perhaps not

<@Recent@>(@Relaxed adversarial training for inner alignment@) <@posts@>(@Chris Olah’s views on AGI safety@) have been optimistic about using transparency tools to detect deceptive behavior. This post argues that we may not want to use _transparency tools_, because then the deceptive model can simply adapt to fool the transparency tools. Instead, we need something more like an end-to-end trained deception checker that's about as smart as the deceptive model, so that the deceptive model can't fool it.


I'd be interested in work that further sketches out a scenario in which this could occur. I wrote about some particular details in [this comment](https://www.alignmentforum.org/posts/uXH4r6MmKPedk8rMA/gradient-hacking#c7R2dBHRsCAnXyk7b).

Gradient hacking

This post calls attention to the problem of **gradient hacking**, where a powerful agent being trained by gradient descent could structure its computation in such a way that it causes its gradients to update it in some particular way. For example, a mesa optimizer could structure its computation to first check whether its objective has been tampered with, and if so to fail catastrophically, so that the gradients tend to point away from tampering with the objective.


These two variants of pseudo-alignment seem useful to keep in mind, and I am optimistic that classifying risks from mesa-optimization (and AI more generally) will make them easier to understand and address.

More variations on pseudo-alignment

This post identifies two additional types of pseudo-alignment not mentioned in <@Risks from Learned Optimization@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@). **Corrigible pseudo-alignment** is a new subtype of corrigible alignment. In corrigible alignment, the mesa optimizer models the base objective and optimizes that. Corrigible pseudo-alignment occurs when the model of the base objective is a non-robust proxy for the true base objective. **Suboptimality deceptive alignment** is when deception would help the mesa-optimizer achieve its objective, but it does not yet realize this. This is particularly concerning because even if AI developers check for and prevent deception during training, the agent might become deceptive after it has been deployed.


I really like the idea of having a standardized benchmark for evaluating meta-learning algorithms. There's a lot of room for improvement in performance on the benchmark tasks and it would be cool if this incentivized algorithm development. As with any benchmark, I worry that it is too narrow to capture all the nuances of potential algorithms; I wouldn't be surprised if some meta-learning algorithm performed poorly here but did well in some other domain.

Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning

Tianhe Yu*, Deirdre Quillen*, Zhanpeng He*, Ryan Julian, Karol Hausman, Chelsea Finn, Sergey Levine

"Meta-learning" or "learning to learn" refers to the problem of transferring insight and skills from one set of tasks to be able to quickly perform well on new tasks. For example, you might want an algorithm that trains on some set of platformer games to pick up general skills that it can use to quickly learn new platformer games.

This paper introduces a new benchmark, "Meta World", for evaluating meta-learning algorithms. The benchmark consists of 50 simulated robotic manipulation tasks that require a robot arm to do a combination of reaching, pushing and grasping. The benchmark tests the ability of algorithms to learn to do a single task well, learn one multi-task policy that trains and performs well on several tasks at once, and adapt to new tasks after training on a number of other tasks. The paper argues that unlike previous meta-learning evaluations, the task distribution in this benchmark is very broad while still having enough shared structure that meta-learning is possible.

The paper evaluates existing multi-task learning and meta-learning algorithms on this new benchmark. In meta-learning, it finds that different algorithms do better depending on how much training data they're given. In multi-task learning, it finds that the algorithm that performs best uses multiple "heads", or ends of neural networks, one for each task. It also finds that algorithms that are "off-policy"-- that estimate the value of actions other than the one that the network is currently planning to take-- perform better on multi-task learning than "on-policy" algorithms.


I am not an expert in meta-learning, but my impression is that this paper removes a major bottleneck to future research on meta-learning and gives a clear direction for future work on medium-shot learning and more complex inner optimizers. I’m also particularly interested in how this will interact with <@risks from learned optimization@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@). In particular, it seems possible to me that the outer parameters θ or a subset of them might potentially encode an internal mesa-optimizer. On the other hand, they may simply be learning locations in the parameter space from which it is easy to reach useful configurations. Interpretability of weights is a difficult problem, but I'd be excited about any work that sheds light on what characteristics of θ enable it to generalize across tasks.

Meta-Learning with Implicit Gradients

Aravind Rajeswaran, Chelsea Finn, Sham Kakade, Sergey Levine

The field of meta-learning endeavors to create agents that don’t just learn, but instead learn how to learn. Concretely, the goal is to train an algorithm on a subset of tasks, such that it can get low error on a different subset of tasks with minimal training.

Model Agnostic Meta Learning (MAML) tackles this problem by finding a set of initial parameters, θ, from which it is easy to quickly learn other tasks. During training, an inner loop copies θ into parameters φ, and optimizes φ for a fixed number of steps. Then an outer loop computes the gradient of θ through the inner optimization process (e.g. backpropagating through gradient descent) and updates θ accordingly.

MAML as described above has a few downsides, which this paper addresses.
1.  The base optimizer itself must be differentiable, not just the loss function.
2.  The gradient computation requires linear compute and memory in the number of steps, and suffers from vanishing and exploding gradients as that number increases.
3.  In the inner loop, while φ is initially identical to θ, its dependence on θ fades as more steps occur.

Implicit MAML (iMAML) addresses these with two innovations. First, it adds a regularization term to keep φ close to θ, which maintains the dependence of φ on θ throughout training. Second, it computes the outer update gradient in closed form based purely on the final value of φ rather than using the entire optimization trajectory. Because the inner loop is an optimization process, the end result is an optimum, and thus has zero gradient. This leads to an implicit equation that when differentiated gives a closed form representation for the gradient of θ. This enables iMAML to work with inner optimization sequences that have more training steps, or that are not differentiable.


Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples

Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, Hugo Larochelle


Online Meta-Learning

Chelsea Finn, Aravind Rajeswaran, Sham Kakade, Sergey Levine


This paper is interesting because it's a demonstration of the power of meta-learning to formulate the true optimisation objective. Often in machine learning much research is devoted to the manual path of trying to find the correct inductive biases to solve hard problems (such as catastrophic forgetting). Instead, this paper shows we can use methods like meta-learning to learn these inductive biases (such as sparsity) automatically, by optimising directly for what we want. This relates to (and is motivated by) <@AI-Generating Algorithms@>(@AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence@). Obviously, this method still uses the neuromodulatory network as an architectural inductive bias - it'd be interesting to see whether we could somehow learn this method (or something more specific) as well, perhaps through neural architecture search or just using a larger network which has the representational capacity to perform something like the gating operation.

Paper: AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence

Learning to Continually Learn

Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O. Stanley, Jeff Clune, Nick Cheney

This paper presents the **ANML** (A Neuromodulated Meta-Learning algorithm) method for countering catastrophic forgetting in continual learning. Continual learning is a problem setting where the system is presented with several tasks in sequence, and must maintain good performance on all of them. When training on new tasks, neural networks often “forget” how to perform the previous tasks, which is called catastrophic forgetting. This makes the naive approach of just training on each task in sequence ineffective.

The paper has two main ideas. First, rather than avoiding catastrophic forgetting by using hand-crafted solutions (e.g. previous methods have encouraged sparsity), the authors use meta-learning to directly optimise for this goal. This is done by **learning a network parameterization which, after training sequentially on many tasks, will get good performance on all tasks**. This outer loop objective can be optimised for directly by taking higher order gradients (gradients of gradients). The second idea is a novel form of neuromodulation. This takes the form of a neuromodulatory (NM) network, which takes the same input as the prediction network, and gates the prediction network’s forward pass. **This provides direct control of the output of the prediction network, but also indirect control of the learning of the prediction network, as gradients will only flow through the paths which haven’t been zeroed out by the gating mechanism.**

**Their method achieves state-of-the-art results on continual learning in Omniglot**, a few-shot dataset consisting of 1623 characters, each with only 20 hand-drawn examples. The network has to learn a sequence of tasks (e.g. classifying a character) with only 15 examples, and is then tested on overall performance over all the classes it’s learned. Their network gets 60% accuracy when presented with 600 classes in a row. **A classifier trained with the same data but shuffled independently at random only gets 68% accuracy**, implying that the catastrophic forgetting of their network only cost 8 percentage points. **Their method also learns a form of sparsity in the activations of the network in a much better way than the hand-crafted methods** - while per-class activations are very sparse, no neurons are wasted, as they all still activate over the entire dataset.


I like this paper, and the techniques for meta-regularization it proposes seem to me like they're natural and will be picked up elsewhere. Penalizing model complexity to encourage more adaptive learning reminds me of arguments that <@pressure for compressed policies could create mesa-optimizers@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@) -- this feels like very weak evidence that that could indeed be the case.

Meta-Learning without Memorization

Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, Chelsea Finn

Meta-learning is a technique for leveraging data from previous tasks to enable efficient learning of new tasks. This paper proposes a solution to a problem in meta-learning which the paper calls the _memorization problem_. Imagine a meta-learning algorithm trained to look at 2D pictures of 3D objects and determine their orientation relative to a fixed canonical pose. Trained on a small number of objects, it may be easy for the algorithm to just memorize the canonical pose for each training object and then infer the orientation from the input image. However, the algorithm will perform poorly at test time because it has not seen novel objects and their canonical poses. Rather than memorizing, we would like the meta-learning algorithm to learn to _adapt_ to new tasks, guessing at rules for determining canonical poses given just a few example images of a new object.

At a high level, a meta-learning algorithm uses information from three sources when making a prediction-- the training data, the parameters learned while doing meta-training on previous tasks, and the current input. To prevent memorization, we would like the algorithm to get information about which task it's solving only from the training data, rather than memorizing it by storing it in its other information sources. To discourage this kind of memorization, the paper proposes two new kinds of regularization techniques which it calls "meta-regularization" schemes. One penalizes the amount of information that the algorithm stores in the direct relationship between input data and predicted label ("meta-regularization on activations"), and the other penalizes the amount of information that the algorithm stores in the parameters learned during meta-training ("meta-regularization on weights").

In some cases, meta-regularization on activations fails to prevent the memorization problem where meta-regularization on weights succeeds. The paper hypothesizes that this is because even a small amount of direct information between input data and predicted label is enough to store the correct prediction (e.g., a single number that is the correct orientation). That is, the correct activations will have _low information complexity_, so it is easy to store them even when information in activations is heavily penalized. On the other hand, the _function_ needed to memorize the predicted label has a _high information complexity_, so penalizing information in the weights, which store that function, successfully discourages memorization. The key insight here is that memorizing all the training examples results in a more information-theoretically complex model than task-specific adaptation, because the memorization model is a single model that must simultaneously perform well on all tasks.

Both meta-regularization techniques outperform non-regularized meta-learning techniques in several experimental set-ups, including a toy sinusoid regression problem, the pose prediction problem described above, and modified Omniglot and MiniImagenet classification tasks. They also outperform fine-tuned models and models regularized with standard regularization techniques.


Task-Agnostic Meta-Learning for Few-shot Learning

Muhammad Abdullah Jamal, Guo-Jun Qi, Mubarak Shah


Meta-learning with differentiable closed-form solvers

Luca Bertinetto, João F. Henriques, Philip H.S. Torr, Andrea Vedaldi


Overall I like this paper's simplicity and focus on understanding how meta learning systems work. I did find the reinforcement learning experiment a bit more difficult to parse and connect to the linear and nonlinear regression experiments, and, of course, there's always the question with work on simpler problems like this of whether the intuition extends to more complex ones

Meta-learners' learning dynamics are unlike learners'

We've seen evidence in prior work that meta learning models can be trained to more quickly learn tasks drawn from some task distribution, by training a model in the inner loop and optimizing against generalization error. This paper suggests that meta learning doesn't just learn new tasks faster, but has a different ordered pattern of how it masters the task. Where a "normal" learner first learns the low-frequency modes (think SGD modes, or Fourier modes) of a simple regularization task, and later the high-frequency ones, the meta learner makes progress on all the modes at the same relative rate. This meta learning behavior seems to theoretically match the way a learner would update on new information if it had the "correct" prior (i.e. the one actually used to generate the simulated tasks).


Meta-Learning: A Survey

This taxonomy of meta-learning classifies approaches by the main type of meta-data they learn from:
1. Evaluations of other models on related tasks
2. Characterisations of the tasks at hand (and a similarity metric between them)
3. The structures and parameters of related models
Vanschoren explores a number of different approaches in each category.


CAML: Fast Context Adaptation via Meta-Learning

Luisa M Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, Shimon Whiteson


Meta-learning of Sequential Strategies

Pedro A. Ortega, Jane X. Wang, Mark Rowland, Tim Genewein, Zeb Kurth-Nelson, Razvan Pascanu, Nicolas Heess, Joel Veness, Alex Pritzel, Pablo Sprechmann, Siddhant M. Jayakumar, Tom McGrath, Kevin Miller, Mohammad Azar, Ian Osband, Neil Rabinowitz, András György, Silvia Chiappa, Simon Osindero, Yee Whye Teh, Hado van Hasselt, Nando de Freitas, Matthew Botvinick, Shane Legg

This paper explains theoretically how to structure meta-learning such that it is incentivized to learn optimal solutions to sequence-prediction and decision-making tasks. The core idea is to define a distribution over tasks, and then sample a new task at the beginning of each episode that the agent must then handle. Importantly, the agent is _not told_ what the task is, and so must infer it from observations. As long as you structure the loss function appropriately, the optimal policy for the agent is to maintain a prior over the task that is updated via Bayes Rule after each observation.

Of course, since the agent is actually a neural net with memory, it does not explicitly perform Bayes Rule, but rather learns a set of weights that instantiate an update rule that effectively approximates Bayes Rule for the given task distribution. Since this update rule only needs to work on the specific task distribution being meta-trained on, it can be made significantly more efficient than a full-blown Bayes Rule, and thus can be learned by a relatively small neural net. We can think of this as the network implementing a full-blown _reasoning process_.

In the case of sequence prediction, we optimize the log probability assigned to the true outcomes. As a simple example, the agent might observe a sequence of coin flips from a single coin, where the bias of that coin is chosen at the beginning of each episode (and is not given to the agent). If the bias is drawn from a Normal distribution centered at 0.5, the agent will start out predicting 50-50 on Heads/Tails; if it then sees a Heads, it might update slightly to something like 55-45, and vice versa for Tails. In contrast, if the bias is drawn from a distribution where most of the mass is near 0 or 1, and very little mass is at 0.5, the agent will still start out predicting 50-50, but after seeing a Heads it will then update strongly to e.g. 90-10.

In the case of sequential decision-making, we are given a reward function; we simply optimize the expected reward using some traditional deep RL algorithm (the paper considers Q-learning).


I suspect that in these simple tasks the posterior distribution over the parameters θ maintained by the Bayes-optimal algorithm is a _minimal_ sufficient statistic, that is, _any_ optimal policy must have states that are sufficient to reconstruct the information stored by the Bayes-optimal algorithm. So it makes sense that, for an agent with optimal behavior, the agent’s states could be used to simulate the Bayes-optimal states. I don’t think this tells us that much about the algorithm the network is implementing.

Note that I am quite happy to see work investigating the sorts of reasoning processes that neural networks have learned. While I don’t think the specific results in this paper have told us that much, I’m excited to see this line of work scaled up to more complex tasks, where agents may not reach optimal behavior, or might do so by learning heuristics that _don’t_ encode all of the information that the Bayes-optimal algorithm would use.

Paper: Meta-trained agents implement Bayes-optimal agents

Understanding meta-trained algorithms through a Bayesian lens

Vladimir Mikulik*, Grégoire Delétang*, Tom McGrath*, Tim Genewein*, Miljan Martic, Shane Legg, Pedro A. Ortega

The previous paper suggested that meta-learning can implement optimal reasoning processes in theory. Does it work in practice? This paper sets out to answer this question by studying some simple prediction and decision-making tasks.

For prediction, we consider agents that are trained on a family of distributions (e.g. Bernoulli distributions whose parameter is chosen from a Beta distribution) to predict the probability distribution after seeing a sample generated from it. For decision-making, we consider two-armed bandit problems (where again there is a distribution over the parameters of the problem). These problems were chosen because their optimal solutions can be calculated analytically.

The authors train neural nets with memory to perform well on these tasks (as discussed in the previous paper) and find that they do indeed behave optimally, achieving effectively the best possible performance. They then try to investigate whether they are implementing the same reasoning algorithm as the analytic Bayes-optimal solution. To do this, they see whether they can train a second neural net to map the hidden states (memory) of the agent to the states in the Bayes-optimal solution, and vice versa. (One way to think of this: can you simulate the Bayes-optimal algorithm using the observation encodings from the RNN, and vice versa?)

They find that they _can_ learn a good mapping from agent states to Bayes-optimal states, but _cannot_ learn a good mapping from Bayes-optimal states to agent states. It seems likely that the agent has states that encode more information than is necessary, and so the minimal information stored by the Bayes-optimal algorithm is insufficient to reconstruct the agent states.


Unsupervised Meta-Learning for Reinforcement Learning

Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, Sergey Levine


I like how this paper analyzes the loss landscape of a particular problem, multi-task learning, and uses that knowledge to derive a new algorithm. One thing I always find tricky in ML papers is that it is hard to establish that the theory of why an algorithm works (typically shown on toy models) is also the reason it improves performance (typically shown using complex neural networks). I appreciate that this paper checks for the conditions of their theorem in the multi-task RL models that they train. That said, I think that in order to confirm that the tragic triad they describe is the mechanism by which *PCGrad* improves performance, they would require some way to toggle each element of the triad while keeping everything else fixed.

Gradient Surgery for Multi-Task Learning

Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, Chelsea Finn

In multi-task learning, an algorithm is given data from multiple tasks and tries to learn them all simultaneously, ideally sharing information across them. This paper identifies a *tragic triad* of conditions that can prevent gradient descent from finding a good minimum when all three are present:

**Conflicting gradients** occur when the gradient from one task points in a different direction from another.
**Dominating gradients** occur when the gradient from one task is much larger in magnitude than another.
**High curvature** is when the multi-task curvature is high in the direction of the gradient.

In this situation, the linear approximation of the gradient to the high curvature area leads to an overestimation of the increase in performance on the dominant gradient’s task and an underestimation of the performance degradation from the conflicting gradient’s task. I find picturing the parabola y=x^2 and seeing that a gradient descent step overestimates progress while a gradient ascent step underestimates to be helpful in understanding this.

To solve this, they propose *PCGrad*, which projects all gradients into the normal plane of the others in a pairwise fashion. Their theoretical analysis establishes convergence properties of *PCGrad*, and they empirically show it can be combined with other multi-task algorithms to improve performance and that it makes optimization easier for multi-task supervised learning and RL. They also show plots confirming that the necessary conditions for their theorems appear in these contexts.


6th ICML Workshop on Automated Machine Learning (2019)

Alpha MAML: Adaptive Model-Agnostic Meta-Learning

Harkirat Singh Behl, Atılım Güneş Baydin, Philip H. S. Torr


Meta-Learning with Hessian Free Approach in Deep Neural Nets Training


Meta-Learning with Latent Embedding Optimization

Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, Raia Hadsell


Bayesian Model-Agnostic Meta-Learning

Taesup Kim, Jaesik Yoon, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, Sungjin Ahn


Especially given the <@growth of compute@>(@AI and Compute@), this agenda seems like a natural one to pursue to get AGI. Unfortunately, it also mirrors very closely the phenomenon of <@mesa optimization@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@), with the only difference being that it is _intended_ that the method produces a powerful inner optimizer. As the paper acknowledges, this introduces several risks, and so it calls for deep engagement with AI safety researchers (but sadly it does not propose ideas on how to mitigate the risks).

Due to the vast data requirements, most of the environments would have to be simulated. I suspect that this will make the agenda harder than it may seem at first glance -- I think that the complexity of the real world was quite crucial, and that simulating environments that reach the appropriate level of complexity will be a very difficult task. (My intuition is that something like <@Neural MMO@> is nowhere near enough complexity.)

AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence

Historically, the <@bitter lesson@>(@The Bitter Lesson@) has been that approaches that leverage increasing computation for learning outperform ones that build in a lot of knowledge. The current ethos towards AGI seems to be that we will come up with a bunch of building blocks (e.g. convolutions, transformers, trust regions, GANs, active learning, curricula) that we will somehow manually combine into one complex powerful AI system. Rather than require this manual approach, we could instead apply learning once more, giving the paradigm of AI-generating algorithms, or AI-GA.

AI-GA has three pillars. The first is to **learn architectures**: this is analogous to a superpowered neural architecture search that can discover convolutions, recurrence and attention without any hardcoding. The second is to **learn the learning algorithms**, i.e. meta-learning. The third and most underexplored pillar is to learn to **generate complex and diverse environments** within which to train our agents. This is a natural extension of meta-learning: with meta-learning, you have to specify the distribution of tasks the agent should perform well on; AI-GA simply says to learn this distribution as well. <@POET@>(@Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions@) is an example of recent work in this area.

A strong reason for optimism about the AI-GA paradigm is that it mimics the way that humans arose: natural selection was a very simple algorithm that with a _lot_ of compute and a very complex and diverse environment was able to produce a general intelligence: us. Since it would need fewer building blocks (since it aims to learn everything), it could succeed faster than the manual approach, at least if the required amount of compute is not too high. It is also much more neglected than the "manual" approach.

However, there are safety concerns. Any powerful AI that comes from an AI-GA will be harder to understand, since it's produced by this vast computation where everything is learned, and so it would be hard to get an AI that is aligned with our values. In addition, with such a process it seems more likely that a powerful AI system "catches us by surprise" -- at some point the stars align and the giant computation makes one good random choice and suddenly it outputs a very powerful and sample efficient learning algorithm (aka an AGI, at least by some definitions). There is also the ethical concern that since we'd end up mimicking evolution, we might accidentally instantiate large amounts of simulated beings that can suffer (especially if the environment is competitive, as was the case with evolution).


I believe that I found this paper the _very first time_ I looked for generic new interesting papers _after_ I started thinking about this problem, which was quite the coincidence. I'm really happy that the authors wrote the paper -- it's not in their incentives (as far as I can tell), but the topic seems crucial to address.

That said, I disagree with the paper on a few counts. The authors don't acknowledge the value of HARKing -- **often it is useful to run many experiments and see what happens in order to develop a good theory**. Humans are not ideal Bayesian reasoners who can consider all hypotheses at once; we often require many observations in order to even hypothesize a theory. The authors make the point that in other fields HARKing leads to bad results, but ML is significantly different in that **we can run experiments much faster with a much higher iteration speed**.

If we were instead forced to preregister studies, as the authors suggest, **the iteration speed would drop by an order of magnitude or two**; I seriously doubt that the benefits would outweigh the cost of lower iteration speed. Instead of preregistering all experiments, maybe researchers could run experiments and observe results, formulate a theory, and then preregister an experiment that would test the theory -- but in this case I would expect that researchers end up "preregistering" experiments that are very similar to the experiments that generated the theory, such that the results are very likely to come out in support of the theory.

(This does not require any active malice on the part of the researchers -- it's natural to think of predictions of the theory in the domain where you developed the theory. For example, in <@our recent paper@>(@Learning Preferences by Looking at the World@), we explicitly designed four environments where we expected our method to work and one where it wouldn't.)

Another point: I think that **the underlying cause of HARKing is the incentive to chaise SotA**, and if I were writing this paper I would focus on that. For example, I believe that the bias towards SotA chasing causes HARKing, and not the other way around. (I'm not sure if the authors believe otherwise; the paper isn't very clear on this point.) This is also a more direct explanation of results being caused by grad student descent or hyperparameter tuning; the HARKing in such papers occur because it isn't acceptable to say "we obtained this result via grad student descent", because that would not be a contribution to the field.

Although I've been critiquing the paper, overall I find my beliefs much closer to the authors' than the "beliefs of the field". (Not the beliefs of researchers in the field: I suspect many researchers would agree that HARKing has negative effects, even though the incentives force researchers to do so in order to get papers published.) I'd be interested in exploring the topic further, but don't have enough time to do so myself -- if you're interested in building toy models of the research field and modeling the effect of interventions on the field, get in touch and we can see if it would make sense to collaborate.

HARK Side of Deep Learning -- From Grad Student Descent to Automated Machine Learning

Oguzhan Gencoglu, Mark van Gils, Esin Guldogan, Chamin Morikawa, Mehmet Süzen, Mathias Gruber, Jussi Leinonen, Heikki Huttunen

This paper focuses on the negative effects of Hypothesizing After the Results are Known (HARKing), a pattern in which researchers **first conduct experiments and view the results**, and once they have hit the bar to be publishable, **a hypothesis is constructed after the fact to explain the results**. It argues that HARKing is common in machine learning, and that this has negative effects on the field as a whole. First, improvements to state-of-the-art (SotA) may be questionable because they could have been caused by sufficient hyperparameter tuning via grad student descent, instead of the new idea in a paper to which the gain is attributed. Second, there is publication bias since only positive results are reported in conferences, which prevents us from learning from negative results. Third, hypotheses that are tailored to fit results for a single dataset or task are much less likely to generalize to new datasets or tasks. Fourth, while AutoML systems achieve good results, we cannot figure out what makes them work because the high compute requirements make ablation studies much harder to perform. Finally, they argue that we need to fix HARKing in order to achieve things like ethical AI, human-centric AI, reproducible AI, etc.


These summaries seem really good to me (probably higher quality than a typical summary that I write), but are often on topics I'm not an expert in (eg. GANs) so it's hard for me to evaluate. The one paper I knew well ([Inverse Reward Design](https://arxiv.org/abs/1711.02827)) had a good summary.

ML Writing Month May 2018

The author wrote up a summary of an ML paper every day in May, which have all been collected in this doc.


I really like this paper from the point of view of illustrating an alternative paradigm to building powerful AI systems that _isn’t_ based on scaling up neural networks. You might have picked up from the last few newsletters that I generally do expect us to build powerful AI systems by scaling up neural networks, so you might expect that I disagree with this paper. This is only partially true. I do in fact think that many of the skills mentioned in this paper will emerge by training very large neural networks on diverse datasets; indeed we’re already seeing this with <@few-shot learning@>(@Language Models are Few-Shot Learners@). However, this likely only happens at what would be truly mind-boggling amounts of compute today: in order for this to be remotely feasible, we need to have <@exponential improvements in hardware cost and algorithmic efficiency@>(@Draft report on AI timelines@). It is plausible to me that some of the needed improvements in algorithmic efficiency will come through ideas similar to the ones in this paper: for example, just as CNNs provided a useful inductive bias of translation-invariance, perhaps we get a new architecture that has an inductive bias towards compositionality or causality.

Building Machines That Learn and Think Like People

Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, Samuel J. Gershman

The core claim of this 2016 paper is that we should focus on building AI systems that work as _flexibly_ as humans do. For example, a human can learn how to play the Atari game Frostbite in just a couple of hours, way faster than typical deep RL algorithms -- and in addition, after this they will likely be able to transfer zero-shot to new reward functions, such as “lose as quickly as possible”, “maximize the number of fish”, “beat the level with as little time to spare as possible”, and so on. How can we build AI systems that mimic this feat? Deep RL certainly doesn’t get us there. Similarly, while neural networks can learn to classify digits and characters with thousands of examples, humans can learn new characters from a single example, which then allows them to perform many different tasks such as classification, generation, parsing it into different pen strokes, etc. Since the paper was written neural nets have made progress on few-shot classification, but are still quite far from the flexibility that humans display.

You might reasonably object that humans have rich priors built from years of lived experience, as well as innate knowledge baked in by evolution; in contrast, a neural network has to learn from scratch. The authors agree: in their view, the challenge is **how to imbue rich priors into artificial agents**, so that they too can exhibit these impressive behaviors that humans show. Their preferred approach is to take inspiration from human learning and intelligence as much as possible. In this paper, they identify three main ingredients to recreate that flexibility, and provide an overview of the existing literature:

1. **Developmental software:** This refers to the basic capabilities that children have, even before they learn language. These are called “intuitive theories” in cognitive science; think of “intuitive physics” and “intuitive psychology” theories.

2. **Model building:** Neural networks primarily work via _pattern matching_, but in order to get human-level flexibility, you will need to build _models_: this enables flexibility because the same model can be used for a variety of different tasks. (For example, you can reuse your understanding of the environment transitions in Frostbite when the reward function changes.) Models need to be _compositional_, that is, the representations should be capable of being composed with each other to provide new semantically meaningful representation. For example, for handwritten characters, the representation of a character should be the composition of the representations of the individual pen strokes used to make the character. The authors also highlight _causality_ and _learning to learn_ as important.

3. **Thinking fast:** One major drawback of models is that getting _conclusions_ from these models often requires slow, complex inference algorithms. But human thinking is actually quite fast; just think of how quickly we can understand a visual scene. How can we get this property as well? First, we can use approximate inference algorithms to get answers much more quickly (in fact, one line of work distills the inference algorithm into a fast neural network for even more speed). Second, we can combine model-based and model-free algorithms together; for example we might use a model-based algorithm for flexibility but then use the data generated by that algorithm to train a model-free method that can run faster.


I really like that this paper not just claims that there is a problem with reproducibility, but demonstrates this more rigorously using an experiment. More robust empirical findings seem quite important for getting to a better understanding of machine learning systems in the medium term. Since this understanding is especially important for safety relevant research, where exploratory research seems more problematic by default, I am excited for a push in that direction.

Unreproducible Research is Reproducible

Xavier Bouthillier, Cesar Laurent, Pascal Vincent

This paper argues that despite the growing popularity of sharing code, machine learning research has a problem with reproducibility. It makes the distinction between the reproducibility of **methods/results**, which can be achieved by fixing random seeds and sharing code, and the reproducibility of **findings/conclusions**, which requires that different experimental setups (or at least random seeds) lead to the same conclusion.

Several popular neural network architectures are trained on several image classification datasets several times with different random seeds determining the weight initialization and sampling of data. The relative rankings of the architectures with respect to the test accuracy are found to vary relevantly with the random seed for all data sets, as well as between data sets.

The authors then argue that while the reproducibility of methods can help with speeding up **exploratory research**, the reproducibility of findings is necessary for **empirical research** from which robust conclusions can be drawn.  They claim that exploratory research that is not based on robust findings can get inefficient, and so call for the machine learning community to do more empirical research.


The Bitter Lesson

This blog post is controversial. This is a combination summary and opinion, and so is more biased than my summaries usually are.

Much research in AI has been about embedding human knowledge in AI systems, in order to use the current limited amount of compute to achieve some outcomes. That is, we try to get our AI systems to think the way we think we think. However, this usually results in systems that work currently, but then cannot leverage the increasing computation that will be available. The bitter lesson is that methods like search and learning that can scale to more computation eventually win out, as more computation becomes available. There are many examples that will likely be familiar to readers of this newsletter, such as chess (large scale tree search), Go (large scale self play), image classification (CNNs), and speech recognition (Hidden Markov Models in the 70s, and now deep learning).

Shimon Whiteson's [take](https://twitter.com/shimon8282/status/1106534178676506624) is that in reality lots of human knowledge has been important in getting AI to do things; such as the invariances built into convolutional nets, or the MCTS and self-play algorithm underlying AlphaZero. I don't see this as opposed to Rich Sutton's point -- it seems to me that the takeaway is that we should aim to build algorithms that will be able to leverage large amounts of compute, but we can be clever and embed important knowledge in such algorithms. I think this criterion would have predicted ex-ante (i.e. before seeing the results) that much past and current research in AI was misguided, without also predicting that any of the major advances (like CNNs) were misguided.

It's worth noting that this is coming from a perspective of aiming for the most general possible capabilities for AI systems. If your goal is to instead build something that works reliably now, then it really is a good idea to embed human domain knowledge, as it does lead to a performance improvement -- you should just expect that in time the system will be replaced with a better performing system with less embedded human knowledge.

One disagreement I have is that this post doesn't acknowledge the importance of data. The AI advances we see now are ones where the data has been around for a long time (or you use simulation to get the data), and someone finally put in enough engineering effort + compute to get the data out and put it in a big enough model. That is, currently compute is <@increasing much faster@>(@AI and Compute@) than data, so the breakthroughs you see are in domains where the bottleneck was compute and not data; that doesn't mean data bottlenecks don't exist.


This is well presented and you can glean a lot from looking at the introductory sections. If you choose to dive into a particular topic, charts and methodology are presented in a clear manner with nice hyperlinking to make navigation relatively painless. There is also an [interactive](https://aiindex.stanford.edu/vibrancy/) visualization that allows for cross-country comparison according to user-defined metrics. Once again, very well presented.

2021 AI Index Report

Daniel Zhang, Saurabh Mishra, Erik Brynjolfsson, John Etchemendy, Deep Ganguli, Barbara Grosz, Terah Lyons, James Manyika, Juan Carlos Niebles, Michael Sellitto, Yoav Shoham, Jack Clark, Raymond Perrault

The AI Index Report is a project to track and distill data related to artificial intelligence. One central theme the report focuses on is the effects of COVID on AI research direction. The report highlights significant increases in spending on drug development, 4.5 times that in 2019. The report also focuses a spotlight on the relative lack of AI ethics benchmarks. This could pose a significant problem as surveillance technologies become an increasingly mature technology. Beyond these broad themes, there's data on publication trends, politics, diversity, and more in the 222-page report. Additionally, a significant amount of data is publicly available or interactive.


I enjoyed and appreciated the way this paper questions the obvious default of convolutional models for image processing, and in particular the way it highlights various reachable points (neighborhood-aware value transformation, relative position encoding, etc) on the interpolation path between weights based purely on relative distance (convolution) and weights based purely on content similarity (attention without any position representation). I'd be interested in the future to see more work in this space, exploring different places in a network architecture where content-focused or position-focused computation is most valuable.

Stand-Alone Self Attention in Vision Models

Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, Jonathon Shlens

Continuing with the more general rise of attention models across disciplines, this paper argues that attention-only models can perform comparably to convolutional networks on image classification tasks, a domain where convolution has been the reigning default method for years now.  Because attention doesn't scale parameter-wise as you increase spatial scale, this comparable performance can be achieved at a notably lower number of parameters and FLOPs. The authors perform a few interesting modifications to attention. Firstly, it's canonical, with attention, to include a representation of a pixel's position in the image, in addition to the vector storing the content of the image. In this paper, they found that storing this position information in relative terms (i.e. "how close is this pixel to the center one where attention is being calculated") performs better.

This can be seen as a sort of generalized form of convolution, where instead of having fixed weights for pixels in a kernel indexed by their relative position, attention takes both content and relative position as an input and generates a weight dynamically. Another modification is, at the lower parts of the network, to somewhat modify the attention paradigm such that the "value" at each location isn't just a neutrally transformed version of the input at that location, but rather one transformed differently according to the pixel's position relative to the anchor point where attention is being calculated. At the lower levels of the network, convolutions tend to outperform attention, but attention performs better at later layers of the network. This makes sense, the authors claim, because in early layers each individual pixel doesn't contain much content information that an attention mechanism could usefully leverage, whereas later the learned features at a given spatial location are richer, and more productively leveraged by attention.


Sparsity seems like a good objective to shoot for in order to ensure explainability. I'm less convinced that it's worthwhile for representation learning: I doubt humans have any sort of "sparse learning" bias; I think sparsity of knowledge is a natural consequence of having to understand a very complex world with a very small brain. (Whereas current ML systems only have to understand much simpler environments.)

Explainable AI, Sparse Representations, and Signals

So far, we have built AI systems that store knowledge _symbolically_ or in a _distributed fashion_ (with neural nets being the latter). While the distributed form allows us to learn knowledge and rules automatically, it is much harder to understand and interpret than symbolically represented knowledge. This post argues that the main difference is in the **sparsity** of the learned knowledge. Of course, with more "sparse" knowledge, it should be easier for us to understand the internal workings of the AI system, since we can ignore the pruned connections. However, the author also argues that sparse knowledge will help 'guide the search for models and agents that can be said to "learn" but also "reason"'. Given that AGI will likely involve finding good representations for the world (in the sense of unsupervised learning), then sparse learning can be thought of as a bias towards finding better [bases](https://en.wikipedia.org/wiki/Basis_(linear_algebra)) for world models, that are more likely to be conceptually clean and more in line with Occam's razor.

In a postscript, the author considers arguments for AI risk. Notably, there isn't any consideration of goal-directedness or alignment failures; the worry is that we will start applying superhuman AI systems to superhuman tasks, and we won't know how to deal with these situations.


BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova


I imagine that this is primarily targeted at data scientists aiming to learn about the real world, and not ML researchers, but I wouldn't be surprised if it was helpful for us as well. MNIST and ImageNet are both present, and a search for "self-driving cars" turned up some promising-looking links that I didn't investigate further.

Making it easier to discover datasets

Google has launched Dataset Search, a tool that lets you search for datasets that you could then use in research.


In <@last year’s report@>(@State of AI Report 2020@), I said that their 8 predictions seemed to be going out on a limb, and that even 67% accuracy woud be pretty impressive. This year, they scored their predictions as 5 “Yes”, 1 “Sort of”, and 2 “No”. That being said, they graded “The first 10 trillion parameter dense model” as “Yes”, I believe on the basis that Microsoft had run a couple of steps of training on a 32 trillion parameter dense model. I definitely interpreted the prediction as saying that a 10 trillion parameter model would be trained _to completion_, which I do not think happened publicly, so I’m inclined to give it a “No”. Still, this does seem like a decent track record for what seemed to me to be non-trivial predictions. This year's predictions seem similarly "out on a limb" as last year's.

This year’s report included one-slide summaries of many papers I’ve summarized before. I only found one major issue -- the slide on <@TruthfulQA@>(@TruthfulQA: Measuring How Models Mimic Human Falsehoods@) implies that larger language models are less honest _in general_, rather than being more likely to imitate human falsehoods. This is actually a pretty good track record, given the number of things they summarized where I would have noticed if there were major issues.

State of AI Report 2021

As with <@past@>(@State of AI@) <@reports@>(@State of AI Report 2020@), I’m not going to summarize the entire thing; instead you get the high-level themes that the authors identified:

1. AI is stepping up in more concrete ways, including in mission critical infrastructure.
2. AI-first approaches have taken biology by storm (and we aren’t just talking about AlphaFold).
3. Transformers have emerged as a general purpose architecture for machine learning in many domains, not just NLP.
4. Investors have taken notice, with record funding this year into AI startups, and two first ever IPOs for AI-first drug discovery companies, as well as blockbuster IPOs for data infrastructure and cybersecurity companies that help enterprises retool for the AI-first era.
5. The under-resourced AI-alignment efforts from key organisations who are advancing the overall field of AI, as well as concerns about datasets used to train AI models and bias in model evaluation benchmarks, raise important questions about how best to chart the progress of AI systems with rapidly advancing capabilities.
6. AI is now an actual arms race rather than a figurative one, with reports of recent use of autonomous weapons by various militaries.
7. Within the US-China rivalry, China's ascension in research quality and talent training is notable, with Chinese institutions now beating the most prominent Western ones.
8. There is an emergence and nationalisation of large language models.


The article does a reasonably good job of giving equal attention to different viewpoints on neural-symbolic integration. While the article does focus on the localist vs. distributed distinction, I also find it to be broadly useful. Personally, after reading the article I wonder if 'reasoning' needs to be hand-set into a neural network at all. Is there really something inherently different about reasoning such that it wouldn't just emerge from any sufficiently powerful forward predictive model? The authors make a good point regarding XAI and the importance of fidelity. I agree that it's important that our explanations specifically fit the model rather than interpret the data. However, from a performance perspective, I don't feel I have a good understanding of why the abstraction of a symbol/logic should occur outside the neural network. This leaves me thinking [**the bitter lesson**](http://www.incompleteideas.net/IncIdeas/BitterLesson.html) ([**AN #49**](https://mailchi.mp/efed27be268a/alignment-newsletter-49)) will apply to neural-symbolic approaches that try to extract symbols or apply reason using human features (containers/first-order logic).

**Rohin's opinion:** While I do think that you can get human-level reasoning (including e.g. causality) by scaling up neural networks with more diverse data and environments, this does _not_ mean that neural-symbolic methods are irrelevant. I don’t focus on them much in this newsletter because 1) they don’t seem that relevant to AI alignment in particular (just as I don’t focus much on e.g. neural architecture search) and 2) I don’t know as much about them, but this should not be taken as a prediction that they won’t matter. I agree with Zach that the bitter lesson will apply, in the sense that for a specific task as we scale up we will tend to reproduce neural-symbolic approaches with end-to-end approaches. However, it could still be the case that for the most challenging and/or diverse tasks, neural-symbolic approaches will provide useful knowledge / inductive bias that make them the best at a given time, even though vanilla neural nets could scale better (if they had the data, memory and compute).

[Logic Tensor Networks](https://arxiv.org/abs/1606.04422), [The Bitter Lesson](http://www.incompleteideas.net/IncIdeas/BitterLesson.html)

Neurosymbolic AI: The 3rd Wave

The field of neural-symbolic AI is broadly concerned with how to combine the power of discrete symbolic reasoning with the expressivity of neural networks. This article frames the relevance of neural-symbolic reasoning in the context of a big question: what are the necessary and sufficient building blocks of AI? The authors address this and argue that AI needs to have both the ability to learn from and make use of experience. In this context, the neural-symbolic approach to AI seeks to establish provable correspondences between neural models and logical representations. This would allow neural systems to generalize beyond their training distributions through neural-reasoning and would constitute significant progress towards AI.

The article surveys the last 20 years of research on neural-symbolic integration. As a survey, a number of different perspectives on neural-symbolic AI are presented. In particular, the authors tend to see neural-symbolic reasoning as divided into two camps: localist and distributed. Localist approaches assign definite identifiers to concepts while distributed representations make use of continuous-valued vectors to work with concepts. In the later parts of the article, promising approaches, current challenges, and directions for future work are discussed.

Recognizing 'patterns' in neural networks constitutes a localist approach. This relates to explainable AI (XAI) because recognizing how a given neural model makes a decision is a pre-requisite for interpretability. One justification for this approach is that codifying patterns in this way allows systems to avoid reinventing the wheel by approximating functions that are already well-known. On the other hand, converting logical relations (if-then) into representations compatible with neural models constitutes a distributed approach. One distributed method the authors highlight is the conversion of statements in first-order logic to vector embeddings. Specifically, Logic Tensor Networks generalize this method by grounding logical concepts onto tensors and then using these embeddings as constraints on the resulting logical embedding.

Despite the promising approaches to neural-symbolic reasoning, there remain many challenges. Somewhat fundamentally, formal reasoning systems tend to struggle with existential quantifiers while learning systems tend to struggle with universal quantification. Thus, the way forward is likely a combination of localist and distributed approaches. Another challenging area lies in XAI. Early methods for XAI were evaluated according to fidelity: measures of the accuracy of extracted knowledge in relation to the network rather than the data. However, many recent methods have opted to focus on explaining data rather than the internal workings of the model. This has resulted in a movement away from fidelity which the authors argue is the wrong approach.


State of AI Report 2020

The third <@State of AI@> report is out! I won’t go into details here since there is really quite a lot of information, but I recommend scrolling through the presentation to get a sense of what’s been going on. I was particularly interested in their 8 predictions for the next year: most of them seemed like they were going out on a limb, predicting something that isn’t just “the default continues”. On last year’s 6 predictions, 4 were correct, 1 was wrong, and 1 was technically wrong but quite close to being correct;  even this 67% accuracy would be pretty impressive on this year’s 8 predictions. (It does seem to me that last year’s predictions were more run-of-the-mill, but that might just be hindsight bias.)


I really liked the super detailed description of a large number of things that humans can do but current vision systems cannot do; it feels like I have a much more detailed sense now of what is missing from current approaches to vision. While the paper has a huge 491 references backing up its claims, I’m not sure how relevant all of them are. For example, the reference to the revelation principle didn’t really seem to justify the associated point. As a counterpoint, the discussion on utility functions in various fields was excellent. Unfortunately I’m not familiar enough with most of the other areas to spot check them.

I read this paper because I heard that the last author, Song-Chun Zhu, was leaving his job as a professor at UCLA to set up a research institute on “general AI” in Beijing, and I wanted to get a sense of what the institute was likely to work on. It seems like the institute will probably pursue an agenda that focuses on building the five particular facets of intelligence into AI systems, as a form of inductive bias: this is how you’d get to a “small data for big tasks” paradigm. If that’s right, it would be in stark contrast to the neural network approaches taken by most of industry, and the biology-inspired approaches taken by (say) the Human Brain Project, but it would feel quite aligned with the views of many academics (like Josh Tenenbaum, who is a coauthor on this paper).

Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense

Yixin Zhu, Tao Gao, Lifeng Fan, Siyuan Huang, Mark Edmonds, Hangxin Liu, Feng Gao, Chi Zhang, Siyuan Qi, Ying Nian Wu, Joshua B. Tenenbaum, Song-Chun Zhu

This paper argues that current computer vision research focuses too much on a “big data for small tasks” paradigm that focuses only on the “what” and “where” of images. More work should be done on a “small data for big tasks” paradigm that focuses more on the “how” and “why” of images. These “how” and “why” questions focus attention on details of an image that may not be directly present in the pixels of the image, which the authors term “dark” data (analogously to dark matter in physics, whose existence is inferred, not observed). For example, by asking why a human is holding a kettle with the spout pointing down, we can infer that the kettle contains liquid that will soon come out of the kettle, even though there are no pixels that directly correspond to the liquid.

The authors propose five important areas for further research, abbreviated FPICU, and do a literature review within each one:
1. **Functionality:** Many objects, especially those designed by humans, can be better understood by focusing on what functionalities they have.
2. **Physics:** Cognitive science has shown that humans make extensive use of _intuitive physics_ to understand the world. For example, simply reasoning about whether objects would fall can provide a lot of constraints on a visual scene; it would be weird to see an upright cup floating in the air.
3. **Intent:** The world is filled with goal-directed agents, and so understanding the world requires us to infer the goals that various agents have. This is a capability humans get very quickly -- at eighteen months of age, children can infer and imitate the intended goal of an action, even if the action fails to achieve the goal.
4. **Causality:** Much has already been written about causality; I will not bore you with it again. The authors see this as the most important factor that underlies the other four areas.
5. **Utility:** I didn’t really get how this differed from intent. The section in the paper discusses utility theory, and then talks about work that infers utility functions from behavior.


PPO-CMA: Proximal Policy Optimization with Covariance Matrix Adaptation

Perttu Hämäläinen, Amin Babadi, Xiaoxiao Ma, Jaakko Lehtinen


I've summarized the post pretty well, I think -- mainly I'd encourage you to play the Think Again game, it's pretty fun and you generate useful data for them.

Predicting Slow Judgments

Andreas Stuhlmueller, Owain Evans, Tom McGrath, Zac Kenton, Chris Cundy, Ryan Carey, Andrew Schreiber, Neal Jean, Girish Sastry

A joint project between Ought and FHI. The goal is to predict judgments that humans make after considering a question for many hours. However, that data is expensive to collect, so they also collect a lot of labels of how humans make judgments in limited time, and use those as noisy labels for the slow, deliberative judgments. They have just released Think Again (https://thinkagain.ought.org/), where you can make fast and slow judgments on Fermi estimates, political statements, or ML papers. They are especially looking for people to make judgments on ML papers. This will take an hour or two, but they'll recommend papers that they think you'll like based on your judgments of the papers they showed you.


I enjoyed this book more than I thought I would: it had a lot of novel content for me, and I liked the explanations and comparisons across different kinds of existential risks (something that I hadn't really seen a single unified perspective on), and I especially liked the constant focus on what we do and don't know -- it felt more like a research paper (albeit in a conversational style) than a popular book, and was similarly information-dense.

On the AI part specifically, I liked that one of the endnotes cashed out powerful AI systems using model-based RL: this indeed seems like the thing that is closest to the classic expected utility maximizer, so the conclusions make a bit more sense. You still have to wonder how exactly the model is learned, and how exactly the AI system becomes good at using the model to find good actions, but at least under those two assumptions you would have all the standard convergent instrumental subgoals. In contrast, with model-free RL, the default expectation is that the RL agent needs to try things multiple times before it can learn to do them again, so it's less clear how it starts doing novel things. It seems that model-based and model-free RL are pretty similar so the distinction doesn't matter in practice, but at least conceptually it's a lot easier to reason about the model-based system (at least in the context of AI risk).

Toby gives a 1 in 10 chance of existential catastrophe from AI in the next century (more than half of his total of 1 in 6), which decomposes into a 1 in 2 chance of AGI this century, and 1 in 5 of it leading to existential catastrophe. This is a bit more pessimistic than Paul's <@estimate@>(@Conversation with Paul Christiano@) of 10% EV loss (which was over all time, not just this century), which is in turn a bit more pessimistic than the 1 in 10 chance that I <@estimated@>(@Conversation with Rohin Shah@) (and am now forever anchored on), which was over all time _and_ conditional on no additional effort from longtermists. But I wouldn't read too much into this -- 10 is a nice round number, and that probably played a big role in why I chose it. I certainly don't feel calibrated enough to easily tell the difference between 1 in 5 and 1 in 20 on a question of this complexity.

I am very happy about this trend of people actually stating numbers: it's a lot easier to narrow down on the important disagreements when people put down numbers, even if they're completely made up. I'd really like to see numbers from people who have larger disagreements (as I expect would be the case with e.g. MIRI researchers).

FLI Podcast: The Precipice: Existential Risk and the Future of Humanity with Toby Ord

This book argues that humanity is in a special stage of its development: it is on the _precipice_, a narrow time during which we have enough power to destroy ourselves, but not enough wisdom to have mitigated such risks. It first argues that existential risk would be very important to reduce (for all the standard reasons), and then considers many different kinds of existential risks, finding that natural ones (asteroids, supervolcanoes, stellar explosions) are small relative to anthropogenic risks, both current (nuclear war, climate change, environmental destruction) and future (engineered pandemics, unaligned AI, dystopian scenarios). I'll focus primarily on the part about AI risk, as well as some of the comments on existential risk in general.

The AI risk presentation in the book was similar to that in [Superintelligence](https://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/1501227742): it argues for risk from goal-directed AI systems (though the terminology used in the book is different). It first demonstrates the strong progress in deep learning, and then notes that expert surveys estimate that AGI is more likely than not to arrive in the next century. It then notes that we don't know how to specify a reward function for an AI system (even with e.g. inverse reinforcement learning), and to the extent that we get it wrong, it pits us in competition against a superintelligent adversary. Ideas like switching off the AI system wouldn't work, due to convergent instrumental subgoals like survival.

It also considers some obvious objections, including the very reasonable objection that "AI researchers won't build something that will kill them". However, Toby is still worried, citing that due to the unilateralist curse unaligned AGI might still be built by the most optimistic researchers, and in any case the personal benefits to the researchers might justify the risk of misalignment to them personally (though it would not be justified for the world as a whole).

The book then spends some time discussing _risk factors_, which are things that do not _directly_ lead to existential risks, but indirectly _exacerbate_ other existential risks, making them more likely. For example, great power war seems like a risk factor: it isn't going to cause an existential catastrophe by itself, but it increases the likelihood that we use risky technologies like bioweapons and AI that could then cause an existential catastrophe.

The book also has lots of useful insights about existential risks in general, which then also apply to AI risk: for example, risks that strike sooner should be prioritized (since the later risks can be dealt with later), risks that are more sudden will be more important to focus on (since we won't be able to build support as the risk gradually comes in), and risks that are "sharper" will be more neglected since there won't be as many "warning shots".


I'm excited to see more big-picture thought about AI alignment out of DeepMind. This newsletter (and I) tend to focus a lot more on the technical alignment problem than the normative one, partly because there's more work on it, but also partly because I think it is the [more urgent problem](https://www.alignmentforum.org/posts/ZeE7EKHTFMBs8eMxn/clarifying-ai-alignment#3ECKoYzFNW2ZqS6km) (a [controversial](https://www.alignmentforum.org/posts/ZeE7EKHTFMBs8eMxn/clarifying-ai-alignment#JK9Jzvz8f4BEjmNqi) [position](https://www.alignmentforum.org/posts/w6d7XBCegc96kz4n3/the-argument-from-philosophical-difficulty)).

Artificial Intelligence, Values and Alignment

This paper from a DeepMind author considers what it would mean to align an AI system. It first makes a distinction between the _technical_ and _normative_ aspects of the AI alignment problem. Roughly, the normative aspect asks, "what should our AI systems do?", while the technical aspect asks, "given we know what our AI systems should do, how do we get them to do it?". The author argues that these two questions are interrelated and should not be solved separately: for example, the current success of deep reinforcement learning in which we _maximize expected reward_ suggests that it would be much easier to align AI to a utilitarian framework in which we _maximize expected utility_, as opposed to a deontological or Kantian framework.

The paper then explores the normative aspect, in both the single human and multiple humans case. When there's only one human, we must grapple with the problem of what to align our AI system to. The paper considers six possibilities: instructions, expressed intentions, revealed preferences, informed preferences, interests, and values, but doesn't come to a conclusion about which is best. When there are multiple humans, we must also deal with the fact that different people disagree on values. The paper analyzes three possibilities: aligning to a global notion of morality (e.g. "basic human rights"), doing what people would prefer from behind a veil of ignorance, and pursuing values that are determined by a democratic process (the domain of social choice theory).

See also [Import AI #183](https://jack-clark.net/2020/02/03/import-ai-183-curve-fitting-conversation-with-meena-gans-show-us-our-climate-change-future-and-what-compute-data-arbitrage-means/)


I look forward to this post every year. This year, it's been a stark demonstration of how much work _doesn't_ get covered in this newsletter -- while I tend to focus on the technical alignment problem, with some focus on AI governance and AI capabilities, Larks's literature review spans many organizations working on existential risk, and as such has many papers that were never covered in this newsletter. Anyone who wants to donate to an organization working on AI alignment and/or x-risk should read this post. However, if your goal is instead to figure out what the field has been up to for the last year, for the sake of building inside view models of what's happening in AI alignment, I _might_ soon write up such an overview myself, but no promises.

2019 AI Alignment Literature Review and Charity Comparison

As in [three](https://forum.effectivealtruism.org/posts/nSot23sAjoZRgaEwa/2016-ai-risk-literature-review-and-charity-comparison) [previous](https://forum.effectivealtruism.org/posts/XKwiEpWRdfWo7jy7f/2017-ai-safety-literature-review-and-charity-comparison) <@years@>(@2018 AI Alignment Literature Review and Charity Comparison@), this mammoth post goes through the work done within AI alignment from December 2018 - November 2019, from the perspective of someone trying to decide which of several AI alignment organizations to donate to. As part of this endeavor, Larks summarizes several papers that were published at various organizations, and compares them to their budget and room for more funding.


I strongly agree with 2 and 4, weakly agree with 1, 5, and 6, and disagree with 3.

I'm Buck Shlegeris, I do research and outreach at MIRI, AMA

Here are some beliefs that Buck reported that I think are particularly interesting (selected for relevance to AI safety):

1. He would probably not work on AI safety if he thought there was less than 30% chance of AGI within 50 years.
2. The ideas in <@Risks from Learned Optimization@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@) are extremely important.
3. If we build "business-as-usual ML", there will be inner alignment failures, which can't easily be fixed. In addition, the ML systems' goals may accidentally change as they self-improve, obviating any guarantees we had. The only way to solve this is to have a clearer picture of what we're doing when building these systems. _(This was a response to a question about the motivation for MIRI's research agenda, and so may not reflect his actual beliefs, but just his beliefs about MIRI's beliefs.)_
4. Different people who work on AI alignment have radically different pictures of what the development of AI will look like, what the alignment problem is, and what solutions might look like.
5. Skilled and experienced AI safety researchers seem to have a much more holistic and much more concrete mindset: they consider a solution to be composed of many parts that solve subproblems that can be put together with different relative strengths, as opposed to searching for a single overall story for everything.
6. External criticism seems relatively unimportant in AI safety, where there isn't an established research community that has already figured out what kinds of arguments are most important.