I enjoyed this sequence: I’m glad to see more analysis of what is and isn’t necessary for iterated amplification and debate to work, as well as more theoretical models of debate. I broadly agreed with the conceptual points made, with one exception: I’m not convinced that we should not allow brute force for HCH, and for similar reasons I don’t find the arguments that HCH won’t be superintelligent convincing. In particular, the hope with iterated amplification is to approximate a truly massive tree of humans, perhaps a tree containing around 2^100 (about 1e30) base agents / humans. At that scale (or even at just a measly billion (1e9) humans), I don’t expect the reasoning to look anything like what an individual human does, and approaches that are more like “brute force” seem a lot more feasible.

One might wonder why I think it is possible to approximate a tree with more base agents than there are grains of sand in the Sahara desert. Well, a perfect binary tree of depth 99 would have 1e30 nodes; thus we can roughly say that we’re approximating 99-depth-limited HCH. If we had perfect distillation, this would take 99 rounds of iterated amplification and distillation, which seems quite reasonable. Of course, we don’t have perfect distillation, but I expect that to be a relatively small constant factor on top (say 100x), which still seems pretty reasonable. (There’s more detail about how we get this implicit exponential-time computation in <@this post@>(@Factored Cognition@).)

Factored Cognition sequence

The <@Factored Cognition Hypothesis@>(@Factored Cognition@) informally states that any task can be performed by recursively decomposing the task into smaller and smaller subtasks until eventually the smallest tasks can be done by a human. This sequence aims to formalize the hypothesis to the point that it can be used to argue for the outer alignment of (idealized versions of) <@iterated amplification@>(@Supervising strong learners by amplifying weak experts@) and <@debate@>(@AI safety via debate@).

The key concept is that of an _explanation_ or _decomposition_. An explanation for some statement **s** is a list of other statements **s1, s2, … sn** along with the statement "(**s1** and **s2** and … and **sn**) implies **s**". A _debate tree_ is a tree in which for a given node **n** with statement **s**, the children of **n** form an explanation (decomposition) of **s**. The leaves of the tree should be statements that the human can verify. (Note that the full formalism has significantly more detail, e.g. a concept of the "difficulty" for the human to verify any given statement.)

We can then define an idealized version of debate, in which the first debater must produce an answer with associated explanation, and the second debater can choose any particular statement to expand further. The judge decides the winner based on whether they can confidently verify the final statement or not. Assuming optimal play, the correct (honest) answer is an equilibrium as long as:

**Ideal Debate Factored Cognition Hypothesis:** For every question, there exists a debate tree for the correct answer where every leaf can be verified by the judge.

The idealized form of iterated amplification is <@HCH@>(@Humans Consulting HCH@); the corresponding Factored Cognition Hypothesis is simply "For every question, HCH returns the correct answer". Note that the _existence_ of a debate tree is not enough to guarantee this, as HCH must also _find_ the decompositions in this debate tree. If we imagine that HCH gets access to a decomposition oracle that tells it the right decomposition to make at each node, then HCH would be similar to idealized debate. (HCH could of course simply try all possible decompositions, but we are ignoring that possibility: the decompositions that we rely on should reduce or hide complexity.)

Is the HCH version of the Factored Cognition Hypothesis true? The author tends to lean against (more specifically, that HCH would not be superintelligent), because it seems hard for HCH to find good decompositions. In particular, humans seem to improve their decompositions over time as they learn more, and also seem to improve the concepts by which they think over time, all of which are challenging for HCH to do. On the other hand, the author is cautiously optimistic about debate.


I enjoyed reading through this agenda, especially because of the inline clarifications from Paul. I found it actually more useful to see what the author initially thought and what Paul’s correction was, relative to the scenario in which the author simply made the correction and presented the final result, as by including both it makes clear what the (probably common) misunderstanding was.

My Understanding of Paul Christiano's Iterated Amplification AI Safety Research Agenda

This post provides an informal description of the full [iterated amplification](https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd) agenda, aimed at all levels of technical expertise. It is significantly more comprehensive than past descriptions.


This seems eminently sensible to me. Of course, evidence law is going to be specialized to arguments about innocence or guilt of a crime, and may not generalize to what we would like to do with debate, but it still seems like we should be able to learn some generalizable lessons.

Parallels Between AI Safety by Debate and Evidence Law

<@Debate@>(@Writeup: Progress on AI Safety via Debate@) requires us to provide a structure for a debate as well as rules for how the human judge should decide who wins. This post points out that we have an existing system that has been heavily optimized for this already: evidence law, which governs how court cases are run. A court case is high-stakes and involves two sides presenting opposing opinions; evidence law tells us how to structure these arguments and how to limit the kinds of arguments debaters can use. Evidence is generally admissible by default, but there are many exceptions, often based on the fallibility of fact-finders.

As a result, it may be fruitful to look to evidence law for how we might structure debates, and to see what types of arguments we should be looking for.


Weak HCH accesses EXP

This followup to last week's <@Alignment proposals and complexity classes@> shows that the amplification-based proposals can access EXP.


In practice our models are not going to reach the optimal loss, and humans won’t solve arbitrary polynomial-time problems, so these theorems won’t directly apply to reality. Nonetheless, this does seem like a worthwhile check to do -- it feels similar to ensuring that a deep RL algorithm has a proof of convergence under idealized assumptions, even if those assumptions won’t actually hold in reality. I have much more faith in a deep RL algorithm that started from one with a proof of convergence and then was modified based on empirical considerations.

Alignment proposals and complexity classes

The original <@debate@>(@AI safety via debate@) paper showed that any problem in PSPACE can be solved by optimal play in a debate game judged by a (problem-specific) algorithm in P. Intuitively, this is an illustration of how the mechanism of debate can take a weak ability (the ability to solve arbitrary problems in P) and amplify it into a stronger ability (the ability to solve arbitrary problems in PSPACE). One would hope that similarly, debate would allow us to amplify a human's problem-solving ability into a much stronger problem-solving ability.

This post applies this technique to several other alignment proposals. In particular, for each proposal, we assume that the "human" can be an arbitrary polynomial-time algorithm, and the AI models are optimal w.r.t their loss functions, and we ask which problems we can solve using these capabilities. The post finds that, as lower bounds, the various forms of amplification can access PSPACE, while <@market making@>(@AI safety via market making@) can access EXP. If there are untamperable pointers (so that the polynomial-time algorithm can look at objects of an arbitrary size, as long as it only looks at a polynomial-sized subset of them), then amplification and market making can access R (the set of decidable problems).


I like the simplicity of the idea "find the point at which the human no longer changes their mind", and that this is a new idea of how we can scale training of AI systems beyond human level performance. However, I’m not convinced that the training procedure given in this post would end up at this equilibrium, unless the human very specifically guided the training to do so (an assumption I don’t think we can usually make). It seems that if we were to reach the state where M stably reported the true answer to the question, then Adv would never get any reward -- but Adv could do better by randomizing what arguments it makes, so that M cannot know which arguments H will be exposed to and so can’t stably predict H’s final answer. See more details in this [thread](https://www.alignmentforum.org/posts/YWwzccGbcHMJMpT45/ai-safety-via-market-making?commentId=43im5ykxC2SPD86Po).

AI safety via market making

If you have an expert, but don't trust them to give you truthful information, how can you incentivize them to tell you the truth anyway? One [option](https://meteuphoric.com/2014/07/21/how-to-buy-a-truth-from-a-liar/) is to pay them every time they provide evidence that changes your mind, with the hope that only once you believe the truth will there be no evidence that can change your mind. This post proposes a similar scheme for AI alignment.

We train two models, M and Adv. Given a question Q, M is trained to predict what answer to Q the human will give at the end of the procedure. Adv on the other hand is trained to produce arguments that will most make M "change its mind", i.e. output a substantially different distribution over answers than it previously outputted. M can then make a new prediction. This is repeated T times, and eventually the human is given all T outputs produced by Adv, and provides their final answer (which is used to provide a gradient signal for M). After training, we throw away Adv and simply use M as our question-answering system.

One way to think about this is that M is trained to provide a prediction market on "what the human will answer", and Adv is trained to manipulate the market by providing new arguments that would change what the human says. So, once you see M providing a stable result, that should mean that the result is robust to any argument that Adv could provide, and so it is what the human would say after seeing all the arguments.

This scheme bears some resemblance to <@debate@>(@AI safety via debate@), and it can benefit from schemes that help debate, most notably <@cross-examination@>(@Writeup: Progress on AI Safety via Debate@). In particular, at every step Adv can cross-examine the previous incarnation of Adv. If the previous incarnation was deceptive, the current incarnation can demonstrate this to the human, which should cause them to disregard the previous argument. We can also add oversight, where an overseer with access to the model ensures that the model does not become non-myopic or deceptive.


AI Safety Debate and Its Applications

This post defines the components of a <@debate@>(@AI safety via debate@) game, lists some of its applications, and defines truth-seeking as the property that we want. Assuming that the agent chooses randomly from the possible Nash equilibria, the truth-promoting likelihood is the probability that the agent picks the actually correct answer. The post then shows the results of experiments on MNIST and Fashion MNIST, seeing comparable results to the original paper.


I like the idea, but the math in the post is quite hard to read (mainly due to the lack of exposition). The post also has separate procedures for amplification, distillation and iteration; I think they can be collapsed into a single more efficient procedure, which I wrote about in this [comment](https://www.alignmentforum.org/posts/jYvm4mmjvGHcPXtGL/a-concrete-proposal-for-adversarial-ida#KdqEJNmRH6vFx5NZg).

A Concrete Proposal for Adversarial IDA

This post presents a method to use an adversary to improve the sample efficiency (with respect to human feedback) of iterated amplification. The key idea is that when a question is decomposed into subquestions, the adversary is used to predict which subquestion the agent will do poorly on, and the human is only asked to resolve that subquestion. In addition to improving sample efficiency by only asking relevant questions, the resulting adversary can also be used for interpretability: for any question-answer pair, the adversary can pick out specific subquestions in the tree that are particularly likely to contain errors, which can then be reviewed.


Can HCH epistemically dominate Ramanujan?

Iterated amplification rests on the hope that we can achieve arbitrarily high capabilities with (potentially very large) trees of explicit verbal breakdowns of problems. This is often formalized as a question about [HCH](https://www.alignmentforum.org/posts/NXqs4nYXaq8q6dTTx/humans-consulting-hch) ([AN #34](https://mailchi.mp/f1947668b183/alignment-newsletter-34)). This post considers the example of Srinivasa Ramanujan, who is "famously known for solving math problems with sudden and inexplicable flashes of insight". It is not clear how HCH would be able to replicate this sort of reasoning.


AI Unsafety via Non-Zero-Sum Debate

This post points out that <@debate@>(@AI safety via debate@) relies crucially on creating a zero-sum game in order to ensure that the debaters point out flaws in each other's arguments. For example, if you modified debate so that both agents are penalized for an inconclusive debate, then an agent may decide not to point out a flaw in an argument if it believes that it has some chance of confusing the judge.


Reinforcement Learning in the Iterated Amplification Framework

This post and its comments clarify how we can use reinforcement learning for the distillation step in iterated amplification. The discussion is still happening so I don't want to summarize it yet.


This is the counterpoint to [Reliability amplification](https://www.alignmentforum.org/posts/6fMvGoyy3kgnonRNM/reliability-amplification) ([AN #44](https://mailchi.mp/6bfac400a0c3/alignment-newsletter-44)) from last week, and the same [confusion](https://www.alignmentforum.org/posts/6fMvGoyy3kgnonRNM/reliability-amplification#9HpJBuc9PBwTPtdQW) I had last week still apply, so I'm going to refrain from an opinion.

Security amplification

If we imagine humans as reasoners over natural language, there are probably some esoteric sentences that could cause "failure". For example, maybe there are unreasonably convincing arguments that cause the human to believe something, when they shouldn't have been convinced by the argument. Maybe they are tricked or threatened in a way that "shouldn't" have happened. The goal with security amplification is to make these sorts of sentences difficult to find, so that we will not come across them in practice. As with [Reliability amplification](https://www.alignmentforum.org/posts/6fMvGoyy3kgnonRNM/reliability-amplification) ([AN #44](https://mailchi.mp/6bfac400a0c3/alignment-newsletter-44)), we are trying to amplify a fast agent A into a slow agent A* that is "more secure", meaning that it is multiplicatively harder to find an input that causes a catastrophic failure.

You might expect that [capability amplification](https://www.alignmentforum.org/posts/t3AJW5jP3sk36aGoC/capability-amplification) ([AN #42](https://mailchi.mp/f6488137d76c/alignment-newsletter-42)) would also improve security, since the more capable agent would be able to notice failure modes and remove them. However, this would likely take far too long.

Instead, we can hope to achieve security amplification by making reasoning abstract and explicit, with the hope that when reasoning is explicit it becomes harder to trigger the underlying failure mode, since you have to get your attack "through" the abstract reasoning. I believe a future post will talk about this more, so I'll leave the details till then. Another option would be for the agent to act stochastically; for example, when it needs to generate a subquestion, it generates many different wordings of the subquestion and chooses one randomly. If only one of the wordings can trigger the failure, then this reduces the failure probability.


It seems like this requires the assumption that our agents have a small probability of failure on any given input. I think this makes sense of we are thinking of reliability of [corrigibility](https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd/p/fkLYhTQteAu5SinAc) ([AN #35](https://mailchi.mp/bbd47ba94e84/alignment-newsletter-35)). That said, I'm pretty confused about what problem this technique is trying to protect against, which I wrote about [in this comment](https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd/p/6fMvGoyy3kgnonRNM#9HpJBuc9PBwTPtdQW).

Reliability amplification

One hope for building an aligned AI system is to alternate [capability amplification](https://www.alignmentforum.org/posts/t3AJW5jP3sk36aGoC/capability-amplification) and [reward engineering](https://www.alignmentforum.org/posts/4nZRzoGTqg8xy5rr8/the-reward-engineering-problem) (both [AN #42](https://mailchi.mp/f6488137d76c/alignment-newsletter-42)) with semi-supervised learning in order to create an agent that becomes more and more capable with more iterations, while remaining aligned. However, our agents will likely always have a small probability of failing, and when we use capability amplification, that failure probability is also amplified (since each of the subagents could fail). If we do enough capability amplifiation, the failure probability could approach 1. This motivates the problem of _reliability amplification_, which aims to take an agent that fails with small probability ε and produce a new agent that fails with significantly lower probability. As with capability amplification, the new agent can take much more time and compute than the original. The hope would be that the combination of capability and reliability amplification would allow us to take an initial agent and make it both more capable and more reliable (even though capability amplification reduces reliability, and reliability amplification will probably reduce capability).

One simple thing to do would be to sample three agents, have them collaboratively settle on an action. If all three agents are aligned, we're fine. The chance of two or three of the agents being malign is small enough that we can ignore it. The interesting case is if one agent is malign and two are benign. If there were only one correct answer, we could simply take a majority vote and get the right answer, and we would have amplified reliability to order ε^2 (which happens when you get two or more malign agents). However, different benign agents could give different answers, since there is no one "correct" benign answer. If we were to choose randomly, as you might expect we get no benefit: we'd have a ~3ε chance of having exactly one malign agent, and a 1/3 chance of failing in that case, keeping the failure probability at ε. The hope for reducing the failure probability is if the two benign agents can "recognize" each other as benign, and so that the majority vote has a >2/3 chance of choosing one of the actions that the benign agents suggested. This does depend on all of the agents having similar capabilities -- if the malign agent is a superintelligence while the two benign agents are human-level, then it seems likely that the malign superintelligence would be able to convince the benign agents to go along with its plan.


I'm not sure how necessary it is to explicitly aim to avoid catastrophic behavior -- it seems that even a low capability [corrigible](https://ai-alignment.com/corrigibility-3039e668638) agent would still know enough to avoid catastrophic behavior in practice. However, based on [Techniques for optimizing worst-case performance](https://www.alignmentforum.org/posts/E2aZ9Xwdz3i2ghPtn/techniques-for-optimizing-worst-case-performance), summarized below, it seems like the motivation is actually to avoid catastrophic failures of corrigibility, as opposed to all catastrophes.

In fact, we can see that we can't avoid all catastrophes without some assumption on either the environment or the oracle. Suppose the environment can do anything computable, and the oracle evaluates behavior only based on outcomes (observations). In this case, for any observation that the oracle would label as catastrophic, there is an environment that regardless of the agent's action outputs that observation, and there is no agent that can always avoid catastrophe. So for this problem to be solvable, we need to either have a limit on what the environment "could do", or an oracle that judges "catastrophe" based on the agent's action in addition to outcomes. That latter option can cache out to "are the actions in this transcript knowably going to cause something bad to happen", which sounds very much like corrigibility.

Learning with catastrophes

In iterated amplification, we need to train a fast agent from a slow one produced by [amplification](https://www.alignmentforum.org/posts/t3AJW5jP3sk36aGoC/capability-amplification) ([AN #42](https://mailchi.mp/f6488137d76c/alignment-newsletter-42)). We need this training to be such that the resulting agent _never_ does anything catastrophic at test time. In iterated amplification, we do have the benefit of having a strong overseer who can give good fedback. This suggests a formalization for catastrophes. Suppose there is some oracle that can take any sequence of observations and actions and label it as catastrophic or not. How do we use this oracle to train an agent that will never produce catastrophic behavior at test time?

Given unlimited compute and unlimited access to the oracle, this problem is easy: simply search over all possible environments and ask the oracle if the agent behaves catastrophically on them. If any such behavior is found, train the agent to not perform that behavior any more. Repeat until all catastrophic behavior is eliminated. This is basically a very strong form of adversarial training.


While I agree that it would be great to optimize worst-case performance, all of these techniques feel quite difficult to do scalably and with guarantees. With adversarial training, you need to find _all_ of the ways that an agent could fail, while catastrophe could happen if the agent stumbles across _any_ of these methods. It seems plausible to me that with sufficient additional information given to the adversary we can meet this standard, but it seems very hard to _knowably_ meet this standard, i.e. to have a strong argument that we will find all of the potential issues.

With verification, the specification problem seems like a deal-breaker, unless combined with other methods: a major point with AI safety is that we can't write down a good specification for what we want. If we instead only use verification to propagate worst-case guarantees from one model to another (as the post suggests), then it seems possible in theory, but very expensive in practice: most verification techniques assume unlimited fast access to the specification, whereas our specification is very expensive to query. Of course, not much research has focused on this setting, so we can expect large gains; nonetheless you do need to somehow extrapolate the specification to all possible inputs, which seems hard to do with limited access to the specification.

Transparency also seems like it provides additional safety, rather than making any guarantees, since we probably can't get a guarantee that our transparency mechanisms can show us all possible failure modes in a way that we understand them. The argument that we can only focus on the training data makes the assumption that the AI system is not going to generalize well outside of the training dataset. While I'm sympathetic to this assumption (we really don't have good methods for generalization, and there are strong reasons to expect generalization to be near-impossible), it isn't one that I'm confident about, especially when we're talking about _general_ intelligence.

Of course, I'm still excited for more research to be done on these topics, since they do seem to cut out some additional failure modes. But if we're looking to have a semi-formal strong argument that we will have good worst-case performance, I don't see the reasons for optimism about that.

Techniques for optimizing worst-case performance

There are "benign" failures of worst-case performance, where the AI system encounters a novel situation and behaves weirdly, but not in a way that systematically disfavors human values. As I noted above, we can't get rid of all of these, but that's probably fine. We instead would like to focus on "malign" failures, where the AI system applies its intelligence in pursuit of the wrong goal. There are a few techniques that could be applied to this problem.

With adversarial training, we can have one system find inputs on which our agent fails catastrophically, and then train the agent to avoid those behaviors. The main issue here is that there will likely be some failures that aren't found.

On the other hand, verification has strong guarantees, but suffers from the problem that it is hard to know what to specify, and it is computationally expensive to actually perform verification. If we have a strong trusted overseer, eg. produced by amplification, we could use it as a very expensive specification. Alternatively, we could use a catastrophe-checker in lieu of a specification. _(Note: While I can see some ways of using catastrophe-checkers, the post seems to have a specific method in mind that I don't understand.)_

Any malignant failure must be using the intelligence of the agent somehow, and the agent is only intelligent on the training data, so if we can use strong transparency techniques on the training data, we could find such failures. However, if you use transparency for this, by default you weed out the comprehensible failures and leave in the incomprehensible ones.


As the post notes, these problems are only "easy" in the conceptual sense -- the resulting RL problems could be quite hard. I feel most confused about the third and fourth problems. Choosing a yardstick could work to aggregate reward functions, but I still worry about the issue that this tends to overweight reward functions that assign a low value to the yardstick but high value to other outcomes. With widely-varying rewards, it seems hard to importance sample high-stakes decisions, without knowing what those decisions might be. Maybe if we notice a very large reward, we instead make it lower reward, but oversample it in the future? Something like this could potentially work, but I don't see how yet.

For complex, expensive-to-evaluate rewards, Paul suggests using semi-supervised learning; this would be fine if semi-supervised learning was sufficient, but I worry that there actually isn't enough information in just a few evaluations of the reward function to narrow down on the true reward sufficiently, which means that even conceptually we will need something else.

Thoughts on reward engineering

This post digs into some of the "easy" issues with reward engineering (where we must design a good reward function for an agent, given access to a stronger overseer).

First, in order to handle outcomes over long time horizons, we need to have the reward function capture the overseer's evaluation of the long-term consequences of an action, since it isn't feasible to wait until the outcomes actually happen.

Second, since human judgments are inconsistent and unreliable, we could have the agent choose an action such that there is no other action which the overseer would evaluate as better in a _comparison_ between the two. _(This is not exactly right -- the human's comparisons could be such that this is an impossible standard. The post uses a two-player game formulation that avoids the issue, and gives the guarantee that the agent won't choose something that is unambiguously worse than another option.)_

Third, since the agent will be uncertain about the overseer's reward, it will have the equivalent of normative uncertainty -- how should it trade off between different possible reward functions the overseer could have? One option is to choose a particular yardstick, eg. how much the overseer values a minute of their time, some small amount of money, etc. and normalize all rewards to that yardstick.

Fourth, when there are decisions with very widely-varying scales of rewards, traditional algorithms don't work well. Normally we could focus on the high-stakes decisions and ignore the others, but if the high-stakes decisions occur infrequently then all decisions are about equally important. In this case, we could oversample high-stakes decisions and reduce their rewards (i.e. importance sampling) to use traditional algorithms to learn effectively without changing the overall "meaning" of the reward function. However, very rare+high-stakes decisions will probably require additional techniques.

Fifth, for sparse reward functions where most behavior is equally bad, we need to provide "hints" about what good behavior looks like. Reward shaping is the main current approach, but we do need to make sure that by the end of training we are using the true reward, not the shaped one. Lots of other information such as demonstrations can also be taken as hints that allow you to get higher reward.

Finally, the reward will likely be sufficiently complex that we cannot write it down, and so we'll need to rely on an expensive evaluation by the overseer. We will probably need semi-supervised RL in order to make this sufficiently computationally efficient.


There's a clear reason for work on capability amplification: it could be used as a part of an implementation of iterated amplification. However, this post also suggests another reason for such work -- it may help us determine where the "hard part" of AI safety lies. Does it help to assume that you have lots of time and compute, and that you have access to a weaker policy?

Certainly if you just have access to a weaker policy, this doesn't make the problem any easier. If you could take a weak policy and amplify it into a stronger policy efficiently, then you could just repeatedly apply this policy-improvement operator to some very weak base policy (say, a neural net with random weights) to solve the full problem. (In other variants, you have a much stronger aligned base policy, eg. the human policy with short inputs and over a short time horizon; in that case this assumption is more powerful.) The more interesting assumption is that you have lots of time and compute, which does seem to have a lot of potential. I feel pretty optimistic that a human thinking for a long time could reach "superhuman performance" by our current standards; capability amplification asks if we can do this in a particular structured way.

Capability amplification

Capability amplification is the problem of taking some existing policy and producing a better policy, perhaps using much more time and compute. It is a particularly interesting problem to study because it could be used to define the goals of a powerful AI system, and it could be combined with [reward engineering](https://www.alignmentforum.org/posts/4nZRzoGTqg8xy5rr8/the-reward-engineering-problem) above to create a powerful aligned system. (Capability amplification and reward engineering are analogous to amplification and distillation respectively.) In addition, capability amplification seems simpler than the general problem of "build an AI that does the right thing", because we get to start with a weak policy A rather than nothing, and were allowed to take lots of time and computation to implement the better policy. It would be useful to tell whether the "hard part" of value alignment is in capability amplification, or somewhere else.

We can evaluate capability amplification using the concepts of reachability and obstructions. A policy C is _reachable_ from another policy A if there is some chain of policies from A to C, such that at each step capability amplification takes you from the first policy to something at least as good as the second policy. Ideally, all policies would be reachable from some very simple policy. This is impossible if there exists an _obstruction_, that is a partition of policies into two sets L and H, such that it is impossible to amplify any policy in L to get a policy that is at least as good as some policy in H. Intuitively, an obstruction prevents us from getting to arbitrarily good behavior, and means that all of the policies in H are not reachable from any policy in L.

We can do further work on capability amplification. With theory, we can search for challenging obstructions, and design procedures that overcome them. With experiment, we can study capability amplification with humans (something which [Ought](https://ought.org/) is now doing).


The information asymmetry problem between H and A seems like a major issue. For me, it's the strongest argument for why transparency is a _necessary_ ingredient of a solution to alignment. The argument against imitation learning and IRL is quite strong, in the sense that it seems like you can't rely on either of them to capture the right behavior. These are stronger than the arguments against [ambitious value learning](https://www.alignmentforum.org/s/4dHMdK5TLN6xcqtyc/p/5eX8ko7GCxwR5N9mN) ([AN #31](https://mailchi.mp/7d0e3916e3d9/alignment-newsletter-31)) because here we assume that H is smarter than A, which we could not do with ambitious value learning. So it does seem to me that direct supervision (with semi-supervised techniques and robustness) is the most likely path forward to solving the reward engineering problem.

There is also the question of whether it is necessary to solve the reward engineering problem. It certainly seems necessary in order to implement iterated amplification given current systems (where the distillation step will be implemented with optimization, which means that we need a reward signal), but might not be necessary if we move away from optimization or if we build systems using some technique other than iterated amplification (though even then it seems very useful to have a good reward engineering solution).

The reward engineering problem

The reward engineering problem is the problem of designing some form of reward feedback such that if the agent learns to get high reward, then we are happy with the resulting behavior. We assume that an overseer H is providing reward for an agent A, with H smarter than A.

(Aside: Why are we assuming H is smarter, when we are typically worried about AI that is more intelligent than we are? Because we could hope to use AI tools to create this reward function, so that the human + AI team is smarter than the AI being trained. In iterated amplification, this corresponds to the distillation step.)

We could imagine three approaches: direct supervision (where H provides a reward for every action A takes), imitation learning (where A tries to mimic H, perhaps by trying to fool a discriminator that distinguishes between A and H), and inverse reinforcement learning (where we infer a reward function by observing H's behavior and use that to supervise A).

With all of these approaches, we might hope that since H is smarter than A, A will not be able to trick H with some behavior that gets high reward but is actually undesirable. However, this argument has a major flaw. When A chooses an action, the process or computation by which it chooses that action is important information, which by default (especially with neural nets) only A has access to, and so this can allow it to fool H. For example, if A is tasked with drawing original artwork, it might copy and slightly change existing artwork, which H may not realize if it cannot see how the artwork was made. We could hope to fix this problem with transparency or explanations, but this requires a lot more research.

Imitation learning and IRL have the problem that A may not be capable of doing what H does. In that case, it will be off-distribution and may have weird behavior. Direct supervision doesn't suffer from this problem, but it is very time-inefficient. This could potentially be fixed using semi-supervised learning techniques.


AlphaGo Zero and capability amplification

AlphaGo Zero works by starting with a randomly chosen policy and value network. Then, it repeatedly applies a "policy improvement" step: it runs MCTS using the policy and value networks to guide the search, which results in moves that are better than the policy and value networks used alone, and then trains the policy and value networks on those moves. Iterated amplification is very similar: it starts with a base policy, and then repeatedly applies a "policy improvement" step that consists of first amplifying the policy and then distilling it. MCTS is analogous to amplification, and training on the resulting moves is analogous to distillation.

So, if a general method of building capable agents is to constantly apply "policy improvement" steps, the key challenge is to figure out a "policy improvement" operator that preserves alignment. (We could hopefully get an aligned weak initial agent by eg. imitating a human's cognitive procedure.) In iterated amplification, we are hoping that "think longer" (as formalized by increasingly large bounded versions of [HCH](https://www.alignmentforum.org/posts/NXqs4nYXaq8q6dTTx/humans-consulting-hch) ([AN #34](https://mailchi.mp/f1947668b183/alignment-newsletter-34))) would be an alignment-preserving policy improvement operator.


The three directions of reliability/robustness, reward learning, and amplification seem great. Robustness seems particularly hard to achieve. While there is current work on adversarial training, interpretability and verification, even if all of the problems that researchers currently work on were magically solved, I don't have a story for how that leads to robustness of (say) an agent trained by iterated amplification. Normally, I'd also talk about how amplification seems to require some sort of universality assumption, but now I can just refer you to [ascription universality](https://www.alignmentforum.org/posts/M8WdeNWacMrmorNdd/towards-formalizing-universality) (described in this newsletter).

I am more conflicted about the desiderata. They seem very difficult to satisfy, and they don't seem strictly necessary to achieve good outcomes. The underlying view here is that we should aim for something that we know is _sufficient_ to achieve good outcomes, and only weaken our requirements if we find a fundamental obstacle. My main issue with this view is that even if it is true that the requirements are impossible to satisfy, it seems very hard to _know_ this, and so we may spend a lot of time trying to satisfy these requirements and most of that work ends up being useless. I can imagine that we try to figure out ways to achieve robustness for several years in order to get a secure AI system, and it turns out that this is impossible to do in a way where we know it is robust, but in practice any AI system that we train will be sufficiently robust that it never fails catastrophically. In this world, we keep trying to achieve robustness, never find a fundamental obstruction, but also never succeed at creating a secure AI system.

Another way of phrasing this is that I am pessimistic about the prospects of conceptual thinking, which seems to be the main way by which we could find a fundamental obstruction. (Theory and empirical experiments can build intuitions about what is and isn't hard, but given the complexities of the real world it seems unlikely that either would give us the sort of crystallized knowledge that Paul is aiming for.) Phrased this way, I put less credence in this opinion, because I think there are a few examples of conceptual thinking being very important, though not that many.

Directions and desiderata for AI alignment

What further research needs to be done to get iterated amplification to work? There are three main areas: robustness, reward learning, and amplification.

With reliability and robustness, we want our AI systems to never fail catastrophically, whereas the simple setup of iterated amplification only optimizes performance in the average case. This is discussed more in [Techniques for Optimizing Worst-Case Performance](https://ai-alignment.com/techniques-for-optimizing-worst-case-performance-39eafec74b99), which talks about adversarial training, interpretability, and verification.

Reward learning, as I understand it here, is the same thing as [narrow value learning](https://www.alignmentforum.org/posts/SvuLhtREMy8wRBzpC/ambitious-vs-narrow-value-learning), described in this newsletter. We've talked a lot about the challenges of reward learning, so I'll leave that part out.

Finally, deliberation and amplification is about how to train an ML system to exceed human performance. At present, the only way we have of doing this is to use reinforcement learning to optimize some simple objective. However, given that "human values" is not a simple objective, we need to figure out another way that allows us to get superhuman performance on the tasks that we actually care about. You could do this with ambitious value learning, but not with narrow value learning. Alternatively, you could try to replicate human cognition and run it longer, either by using iterated amplification, or by using inverse reinforcement learning on cognition.

Then, the post goes on to talk about desiderata for any solution to AI alignment, proposing that the solution should be secure, competitive and scalable.

A secure AI system is one that works even when the environment is behaving adversarially. This is not possible in full generality because of no-free-lunch theorems, but we could at least hope that our AI system is never actively "trying" to hurt us. This is desirable because it holds us to a relatively clear standard that we can evaluate currently, rather than trying to predict how the future will look (a notoriously hard task) and evaluating whether our AI system will work in that future. In addition, the typical iterative method of trying things out and seeing how they do may not work in the future, because technological progress may accelerate due to recursive improvement. If we instead focus on arguments and analysis that suggest that our AI systems are secure, we would not need the iterative method. Finally, there could in fact be adversaries (that themselves might be AI systems) that we need to deal with.

An AI system is competitive if it is almost as efficient and capable as the best possible AI system we could build at the same time (which may be unaligned). This is necessary because without it it becomes very likely that someone will build the unaligned version for the efficiency gains; we would need global coordination to avoid this which seems quite difficult. In addition, it is relatively easy to tell when a system is competitive, whereas if an AI system is uncompetitive, it is hard to tell how uncompetitive it is. Once again, with an uncompetitive AI system, we need to predict the future in order to tell whether or not it will be okay to rely on that system.

A scalable AI system is one that remains aligned as the underlying technologies become more and more powerful. Without scalability, we would need to predict how far a particular scheme will take us (which seems hard), and we need to continually invest in alignment research in order to "keep up" with improving capabilities, which might be particularly hard to do in the face of accelerating progress due to recursive improvement.

These are very demanding desiderata. However, while it may seem impossible to get a secure, competitive, scalable solution to AI alignment, we don't know why yet, and impossibility claims are notoriously hard to get right. In any case, even if it is impossible, it would be worthwhile to clarify exactly why these desiderata are impossible to meet, in order to figure out how to deal with the resulting problems.


This is my new favorite post/paper for explaining how iterated amplification works, since it very succinctly and clearly makes the case for iterated amplification as a strategy for generating a good training signal. I'd recommend reading the [paper](https://arxiv.org/abs/1810.08575) in full, as it makes other important points that I haven't included in the summary.

Note that it does not explain a lot of Paul's thinking. It explains one particular training method that allows you to train an AI system with a more intelligent and informed overseer.

Supervising strong learners by amplifying weak experts

This was previously covered in [AN #30](https://mailchi.mp/c1f376f3a12e/alignment-newsletter-30), I've copied the summary and opinion. This paper introduces iterated amplification, focusing on how it can be used to define a training signal for tasks that humans cannot perform or evaluate, such as designing a transit system. The key insight is that humans are capable of decomposing even very difficult tasks into slightly simpler tasks. So, in theory, we could provide ground truth labels for an arbitrarily difficult task by a huge tree of humans, each decomposing their own subquestion and handing off new subquestions to other humans, until questions are easy enough that a human can directly answer them.

We can turn this into an efficient algorithm by having the human decompose the question only once, and using the current AI system to answer the generated subquestions. If the AI isn't able to answer the subquestions, then the human will get nonsense answers. However, as long as there are questions that the human + AI system can answer but the AI alone cannot answer, the AI can learn from the answers to those questions. To reduce the reliance on human data, another model is trained to predict the decomposition that the human performs. In addition, some tasks could refer to a large context (eg. evaluating safety for a specific rocket design), so they model the human as being able to access small pieces of the context at a time.

They evaluate on simple algorithmic tasks like distance between nodes in a graph, where they can program an automated human decomposition for faster experiments, and there is a ground truth solution. They compare against supervised learning, which trains a model on the ground truth answers to questions (which iterated amplification does not have access to), and find that they can match the performance of supervised learning with only slightly more training steps.


It seems a lot easier to generalize well to novel situations if you've learned the right decision-making algorithms. Since it's way easier to learn something if you have direct supervision of it, internal supervision seems particularly interesting as a method of generalizing well. For example, in [Making Neural Programming Architectures Generalize via Recursion](https://arxiv.org/abs/1704.06611), the learned programs generalize perfectly, because they learn from execution traces, which show _how_ to solve a task, and the learned algorithm is forced to use a recursive structure. (Typical approaches to neural program induction only use input-output examples, and often fail to generalize to inputs longer than those seen during training.)

I also like the point that iterated amplification allows you to mimic an implicit exponential-time computation. This is a key reason for my optimism: it seems like well-honed human intuition beats short explicit reasoning almost always. In fact, I think it is reasonable to view human intuition as the result of iterated amplification (see [these](https://www.lesswrong.com/posts/XHMCvvhb7zTZcQAgA/argument-intuition-and-recursion) [posts](https://terrytao.wordpress.com/career-advice/theres-more-to-mathematics-than-rigour-and-proofs/)).

Factored Cognition

This was previously summarized in [AN #12](https://mailchi.mp/bcb2c6f1d507/alignment-newsletter-12). While this post describes a project from Ought, it explains many of the ideas behind iterated amplification (which is why it is in this sequence). There are two important and distinct topics addressed in this post: first, whether it is possible in principle to achieve good performance on tasks when you must use _explicit reasoning_, and second, how to use explicit reasoning to train an aligned AI system via iterated amplification.

Currently, most AI research is supervised based on _behavior_: we specify a domain and data or reward functions that identify good behavior, and hope that the AI develops a good decision-making system during training. We don't provide training data for decision-making itself. In contrast, Factored Cognition aims to make the decision-making algorithm itself explicit, in a form that could be used to train an AI system, which we might call internal supervision. Note that this is _not_ a required feature of iterated amplification, which can work with external (i.e. behavioral) supervision, as in [recursive reward modeling](https://medium.com/@deepmindsafetyresearch/scalable-agent-alignment-via-reward-modeling-bf4ab06dfd84) ([AN #34](https://mailchi.mp/f1947668b183/alignment-newsletter-34)). However, we might want to use internal supervision because then we get to control the decision-making process itself, and that is what determines whether the AI system is aligned or not.

Research currently does not use internal supervision because it requires training data on _how_ to solve the problem, whereas a reward function only needs to tell _whether_ the problem is solved. However, for sufficiently general AI systems, we could provide training data on general problem-solving, which could then be applied to many tasks. This could be competitive with the alternative of providing training data or reward functions for each task separately.

How might we make general problem-solving explicit enough that we could train AI systems to replicate it? Factored Cognition hypothesizes that we could do this by providing _decompositions_ of tasks into simpler subtasks. While this is not how humans solve tasks (for example, expert Go players use a lot of intuition), it seems plausible that we could get similar or better performance using decomposition as long as we were willing to wait a long time (for example, by implementing minimax using decomposition to solve Go). Ought wants to test this hypothesis, by studying whether humans are in fact capable of providing decompositions in challenging problem domains, such as math and programming puzzles, textbook problems, fact-checking, interactive dialog, and task prioritization. (There will likely be no ML here, just humans decomposing problems for other humans to then solve.)

The post makes this concrete by considering a particular way in which humans might provide decompositions. They develop a "cognitive workspace" where a human can take "actions" such as editing the text in the workspace, each of which can only involve a small amount of cognitive work. There are a lot of details that I won't get into, but broadly it seems to me like what you would get if you tried to create a functional programming language where the primitive forms are things that a human can do. Note that the examples in the post use task-specific decompositions for clarity but in the long term we would need general decomposition strategies.

We could train an AI system to mimic this explicit reasoning. It seems like this would achieve human performance but no more, since after all we are imitating human decision-making algorithms. However, now we can use the iterated amplification trick. If a human got to use explicit reasoning over a long time, we would expect significantly better decisions. So, we can first distill an agent A that mimics a human using explicit reasoning over a short time period, then amplify it to get H[A], where a human performs a decomposition into subquestions that are answered by A, then distill that into A', etc. The post has a nice visualization of how this allows you to mimic an implicit exponentially large tree of decompositions in polynomial time. As before, distillation could be done using imitation learning, or RL with some good inferred reward.


Benign model-free RL

This post is very similar to the previous one, just with different language: distillation is now implemented through reward modeling with robustness. The point of robustness is to ensure that the distilled agent is benign even outside of the training distribution (though it can be incompetent). There's also an analysis of the costs of the scheme. One important note is that this approach only works for model-free RL systems -- we'll need something else for eg. model-based RL, if it enables capabilities that we can't get with model-free RL.


I really like this formulation of corrigibility, which I find quite different from [MIRI's paper](https://intelligence.org/2014/10/18/new-report-corrigibility/). This seems a lot more in line with the kind of reasoning that I want from an AI system, and it seems like iterated amplification or something like it could plausibly succeed at achieving this sort of corrigible behavior.

Corrigibility

A corrigible agent is one which helps its operator, even with tasks that would change the agent itself, such as correcting mistakes in AI design. Consider a good act-based agent, which chooses actions according to our preferences over that action. Since we have a short-term preference for corrigibility, the act-based agent should be corrigible. For example, if we are trying to turn off the agent, the agent will turn off because that's what we would prefer -- it is easy to infer that the overseer would not prefer that agents stop the overseer from shutting them down. Typically we only believe that the agent would stop us from shutting it down if it makes _long-term plans_, in which case being operational is instrumentally useful, but with act-based agents the agent only optimizes for its overseer's short term preferences. One potential objection is that the notion of corrigibility is not easy to learn, but it seems not that hard to answer the question "Is the operator being misled", and in any case we can try this with simple systems, and the results should _improve_ with more capable systems, since as you get smarter you are more capable of predicting the overseer.

In addition, even if an agent has a slightly wrong notion of the overseer's values, it seems like it will _improve_ over time. It is not hard to infer that the overseer wants the agent to make its approximation of the overseer's values more accurate. So, as long as the agent has enough of the overseer's preferences to be corrigible, it will try to learn about the preferences it is wrong about and will become more and more aligned over time. In addition, any slight value drifts caused by eg. amplification will tend to be fixed over time, at least on average.


This is a great simple explanation of the scheme. I don't have much to say about the idea since I've talked about iterated amplification so much in this newsletter already.

Iterated Distillation and Amplification

This is the first in a series of four posts describing the iterated amplification framework in different ways. This post focuses on the repetition of two steps. In amplification, we take a fast aligned agent and turn it into a slow but more capable aligned agent, by allowing a human to coordinate many copies of the fast agent in order to make better decisions. In distillation, we take a slow aligned agent and turn it a fast aligned agent (perhaps by training a neural net to imitate the judgments of the slow agent). This is similar to AlphaGoZero, in which MCTS can be thought of as amplification, while distillation consists of updating the neural net to predict the outputs of the MCTS.

This allows us to get both alignment and powerful capabilities, whereas usually the two trade off against each other. High capabilities implies a sufficiently broad mandate to search for good behaviors, allowing our AI systems to find novel behaviors that we never would have thought of, which could be bad if the objective was slightly wrong. On the other hand, high alignment typically requires staying within the realm of human behavior, as in imitation learning, which prevents the AI from finding novel solutions.

In addition to distillation and amplification robustly preserving alignment, we also need to ensure that given a human as a starting point, iterated distillation and amplification can scale to arbitrary capabilities. We would also want it be about as cost-efficient as alternatives. This seems to be true at test time, when we are simply executing a learned model, but it could be that training is much more expensive.


Goal-directed behavior requires an extremely intelligent overseer in order to ensure that the agent is pointed at the correct goal (as opposed to one the overseer thinks is correct but is actually slightly wrong). I think of approval-directed agents as providing the intuition that we may only require an overseer that is slightly smarter than the agent in order to be aligned. This is because the overseer can simply "tell" the agent what actions to take, and if the agent makes a mistake, or tries to optimize a heuristic too hard, the overseer can notice and correct it interactively. (This is assuming that we solve the [informed oversight problem](https://ai-alignment.com/the-informed-oversight-problem-1b51b4f66b35) so that the agent doesn't have information that is hidden from the overseer, so "intelligence" is the main thing that matters.) Only needing a slightly smarter overseer opens up a new space of solutions where we start with a human overseer and subhuman AI system, and scale both the overseer and the AI at the same time while preserving alignment at each step.

[Approval-directed agents: overview](https://www.alignmentforum.org/posts/7Hr8t6xwuuxBTqADK/approval-directed-agents-overview) and [Approval-directed agents: details](https://www.alignmentforum.org/posts/njRjnmaBnZquqPfBa/approval-directed-agents-details)

These two posts introduce the idea of approval-directed agents, which are agents that choose actions that they believe their operator Hugh the human would most approve of, if he reflected on it for a long time. This is in contrast to the traditional approach of goal-directed behavior, which are defined by the _outcomes_ of the action.

Since the agent Arthur is no longer reasoning about how to achieve outcomes, it can no longer outperform Hugh at any given task. (If you take the move in chess that Hugh most approves of, you probably still lose to Gary Kasparov.) This is still better than Hugh performing every action himself, because Hugh can provide an expensive learning signal which is then distilled into a fast policy that Arthur executes. For example, Hugh could deliberate for a long time whenever he is asked to evaluate an action, or he could evaluate very low-level decisions that Arthur makes billions of times. We can also still achieve superhuman performance by bootstrapping (see the next summary).

The main advantage of approval-directed agents is that we avoid locking in a particular goal, decision theory, prior, etc. Arthur should be able to change any of these, as long as Hugh approves it. In essence, approval-direction allows us to delegate these hard decisions to future overseers, who will be more informed and better able to make these decisions. In addition, any misspecifications seem to cause graceful failures -- you end up with a system that is not very good at doing what Hugh wants, rather than one that works at cross purposes to him.

We might worry that _internally_ Arthur still uses goal-directed behavior in order to choose actions, and this internal goal-directed part of Arthur might become unaligned. However, we could even have internal decision-making about cognition be approval-based. Of course, eventually we reach a point where decisions are simply made -- Arthur doesn't "choose" to execute the next line of code. These sorts of things can be thought of as heuristics that have led to choosing good actions in the past, that could be changed if necessary (eg. by rewriting the code).

How might we write code that defines approval? If our agents can understand natural language, we could try defining "approval" in natural language. If they are able to reason about formally specified models, then we could try to define a process of deliberation with a simulated human. Even in the case where Arthur learns from examples, if we train Arthur to predict approval from observations and take the action with the highest approval, it seems possible that Arthur would not manipulate approval judgments (unlike AIXI).

There are also important details on how Hugh should rate -- in particular, we have to be careful to distinguish between Hugh's beliefs/information and Arthur's. For example, if Arthur thinks there's a 1% chance of a bridge collapsing if we drive over it, then Arthur shouldn't drive over it. However, if Hugh always assigns approval 1 to the optimal action and approval 0 to all other actions, and Arthur believes that Hugh knows whether the bridge will collapse, then the maximum expected approval action is to drive over the bridge.

The main issues with approval-directed agents is that it's not clear how to define them (especially from examples), whether they can be as useful as goal-directed agents, and whether approval-directed agents will have internal goal-seeking behavior that brings with it all of the problems that approval was meant to solve. It may also be a problem if some other Hugh-level intelligence gets control of the data that defines approval.


This complements the idea of approval from the previous posts nicely: while approval tells us how to build an aligned agent from a slightly smarter overseer, bootstrapping tells us how to improve the capabilities of the overseer and the agent.

Approval-directed bootstrapping

To get a very smart overseer, we can use the idea of bootstrapping. Given a weak agent, we can define a stronger agent that happens from letting the weak agent think for a long time. This strong agent can be used to oversee a slightly weaker agent that is still stronger than the original weak agent. Iterating this process allows us to reach very intelligent agents. In approval-directed agents, we can simply have Arthur ask Hugh to evaluate approval for actions, and _in the process of evaluation_ Hugh can consult Arthur. Here, the weak agent Hugh is being amplified into a stronger agent by giving him the ability to consult Arthur -- and this becomes stronger over time as Arthur becomes more capable.


This has the same nice recursive structure of bootstrapping, but without the presence of the agent. This probably makes it more amenable to formal analysis, but I think that the interactive nature of bootstrapping (and iterated amplification more generally) is quite important for ensuring good outcomes: it seems way easier to control an AI system if you can provide input constantly with feedback.

Humans Consulting HCH

Suppose we unroll the recursion in the previous bootstrapping post: in that case, we see that Hugh's evaluation of an answer can depend on a question that he asked Arthur whose answer depends on how Hugh evaluated an answer that depended on a question that he asked Arthur etc. Inspired by this structure, we can define HCH (humans consulting HCH) to be a process that answers question Q by perfectly imitating how Hugh would answer question Q, _if Hugh had access to the question-answering process_. This means Hugh is able to consult a copy of Hugh, who is able to consult a copy of Hugh, who is able to consult a copy of Hugh, ad infinitum. This is one proposal for how to formally define a human's enlightened judgment.

You could also combine this with particular ML algorithms in an attempt to define versions of those algorithms aligned with Hugh's enlightened judgment. For example, for RL algorithm A, we could define max-HCH_A to be A's chosen action when maximizing Hugh's approval after consulting max-HCH_A.


I don't really have much to say here, except that I agree with this post quite strongly.

Prosaic AI alignment

It is plausible that we can build "prosaic" AGI soon, that is, we are able to build generally intelligent systems that can outcompete humans without qualitatively new ideas about intelligence. It seems likely that this would use some variant of RL to train a neural net architecture (other approaches don't have a clear way to scale beyond human level). We could write the code for such an approach right now (see [An unaligned benchmark](https://www.alignmentforum.org/posts/ZHXutm7KpoWEj9G2s/an-unaligned-benchmark) from [AN #33](https://mailchi.mp/b6dc636f6a1b/alignment-newsletter-33)), and it's at least plausible that with enough compute and tuning this could lead to AGI. However, this is likely to be bad if implemented as stated due to the standard issues of reward gaming and Goodhart's Law. We do have some approaches to alignment such as IRL and executing natural language instructions, but neither of these are at the point where we can write down code that would plausibly lead to an aligned AI. This suggests that we should focus on figuring out how to align prosaic AI.

There are several reasons to focus on prosaic AI. First, since we know the general shape of the AI system under consideration, it is easier to think about how to align it (while ignoring details like architecture, variance reduction tricks, etc. which don't seem very relevant currently). Second, it's important, both because we may actually build prosaic AGI, and because even if we don't the insights gained will likely transfer. In addition, worlds with short AGI timelines are higher leverage, and in those worlds prosaic AI seems much more likely. The main counterargument is that aligning prosaic AGI is probably infeasible, since we need a deep understanding of intelligence to build aligned AI. However, it seems unreasonable to be confident in this, and even if it is infeasible, it is worth getting strong evidence of this fact in order change priorities around AI development, and coordinate on not building an AGI that is too powerful.


I often think that the hard part of AI alignment is actually the _strategic_ side of it -- even if we figure out how to build an aligned AI system, it doesn't help us unless the actors who actually build powerful AI systems use our proposal. From that perspective, it's very important for any aligned systems we build to be competitive with unaligned ones, and so keeping these sorts of benchmarks in mind seems like a really good idea. This particular benchmark seems good -- it's essentially the AlphaGo algorithm, except with learned dynamics (since we don't know the dynamics of the real world) and rewards (since we want to be able to specify arbitrary tasks), which seems like a good contender for "powerful AI system".

An unaligned benchmark

I previously summarized this in [Recon #5](https://mailchi.mp/33af21f908b5/reconnaissance-5), but I'll consider it in more detail now. The post argues that we could get a very powerful AI system using model-based RL with MCTS. Specifically, we learn a generative model of dynamics (sample a sequence of observations given actions), a reward model, and a policy. The policy is trained using MCTS, which uses the dynamics model and reward model to create and score rollouts. The dynamics model is trained using the actual observations and actions from the environment. The reward is trained using preferences or rankings (think something like [Deep RL from Human Preferences](https://arxiv.org/abs/1706.03741)). This is a system we could program now, and with sufficiently powerful neural nets, it could outperform humans.

However, this system would not be aligned. There could be specification failures: the AI system would be optimizing for making humans think that good outcomes are happening, which may or may not happen by actually having good outcomes. (There are a few arguments suggesting that this is likely to happen.) There could also be robustness failures: as the AI exerts more control over the environment, there is a distributional shift. This may lead to the MCTS finding previously unexplored states where the reward model accidentally assigns high reward, even though it would be a bad outcome, causing a failure. This may push the environment even more out of distribution, triggering other AI systems to fail as well.

Paul uses this and other potential AI algorithms as _benchmarks_ to beat -- we need to build aligned AI algorithms that achieve similar results as these benchmarks. The further we are from hitting the same metrics, the larger the incentive to use the unaligned AI algorithm.

Iterated amplification could potentially solve the issues with this algorithm. The key idea is to always be able to cash out the learned dynamics and reward models as the result of (a large number of) human decisions. In addition, the models need to be made robust to worst case inputs, possibly by using [these techniques](https://ai-alignment.com/techniques-for-optimizing-worst-case-performance-39eafec74b99). In order to make this work, we need to make progress on robustness, amplification, and an understanding of what bad behavior is (so that we can argue that it is easy to avoid, and iterated amplification does avoid it).


Ultimately, our goal is to build AI systems that reliably do what we want them to do. One way of decomposing this is first to _define_ the behavior that we want from an AI system, and then to figure out how to obtain that behavior, which we might call the definition-optimization decomposition. [Ambitious value learning](https://www.alignmentforum.org/posts/5eX8ko7GCxwR5N9mN/what-is-ambitious-value-learning) aims to solve the definition subproblem. I interpret this post as proposing a _different decomposition_ of the overall problem. One subproblem is how to build an AI system that is _trying_ to do what we want, and the second subproblem is how to make the AI competent enough that it _actually_ does what we want. I like this motivation-competence decomposition for a few reasons, which I've written a [long comment](https://www.alignmentforum.org/posts/ZeE7EKHTFMBs8eMxn/clarifying-ai-alignment#3ECKoYzFNW2ZqS6km) about that I strongly encourage you to read. The summary of that comment is: motivation-competence isolates the urgent part in a single subproblem (motivation), humans are an existence proof that the motivation subproblem can be solved, it is possible to apply the motivation framework to systems without lower capabilities, the safety guarantees degrade slowly and smoothly, the definition-optimization decomposition as exemplified by expected utility maximizers has generated primarily negative results, and motivation-competence allows for interaction between the AI system and humans. The major con is that the motivation-competence decomposition is informal, imprecise, and may be intractable to work on.

Clarifying "AI Alignment"

I previously summarized this in [AN #2](https://mailchi.mp/14782876a85d/alignment-newsletter-2), but I'll consider it in more detail now. As Paul uses the term, "AI alignment" refers only to the problem of figuring out how to build an AI that is _trying_ to do what humans want. In particular, an AI can be aligned but still make mistakes because of incompetence. This is not a formal definition, since we don't have a good way of talking about the "motivation" of an AI system, or about "what humans want", but Paul expects that it will correspond to some precise notion after we make more progress.


This is an interesting perspective on the AI safety problem. I really like the ethos of this post, where there isn't a huge opposition between AI capabilities and AI safety, but instead we are simply trying to figure out how to use the (helpful!) capabilities developed by AI researchers to do useful things.

If I think about this from the perspective of reducing existential risk, it seems like you also need to make the argument that AI systems are unlikely to pose an existential threat before they are human-level (a claim I mostly agree with), or that the solutions will generalize to sub-human-level AI systems.

The Steering Problem

The steering problem refers to the problem of writing a program that uses black-box human-level cognitive abilities to be as useful as a well-motivated human Hugh (that is, a human who is "trying" to be helpful). This is a conceptual problem -- we don't have black-box access to human-level cognitive abilities yet. However, we can build suitable formalizations and solve the steering problem within those formalizations, from which we can learn generalizable insights that we can apply to the problem we will actually face once we have strong AI capabilities. For example, we could formalize "human-level cognitive abilities" as Hugh-level performance on question-answering (yes-no questions in natural language), online learning (given a sequence of labeled data points, predict the label of the next data point), or embodied reinforcement learning. A program P is more useful than Hugh for X if, for every project using a simulation of Hugh to accomplish X, we can efficiently transform it into a new project which uses P to accomplish X.


Preface to the sequence on iterated amplification

This is a preface, read it if you're going to read the full posts, but not if you're only going to read these summaries.


I like this paper; it's a very clear explanation of a "recipe" used to develop preference learning algorithms (especially at CHAI and Interact, two of the labs I'm a part of). It is particularly applicable to the case where there's a separate training phase where the human gives feedback on the reward function, and a deployment phase where the agent optimizes the reward function. Things get murkier once you move to a more online setting in which the human and agent are acting simultaneously, as in <@assistance games / CIRL games@>(@Cooperative Inverse Reinforcement Learning@), where the agent can learn from pragmatic actions (see also <@the deep RL version@>(@Learning to Interactively Learn and Assist@)). In particular, while previously we could separate out the grounding of choices to trajectories, and the interpretation of those trajectories (the Boltzmann rational model), this can no longer be done in an assistance game, since the way that the agent interprets the trajectories changes what the agent does which changes the choices available to the human in the next timestep. I'd be excited for more work on understanding this setting.

Reward-rational (implicit) choice: A unifying formalism for reward learning

Hong Jun Jeon*, Smitha Milli*, Anca D. Dragan

We've got algorithms for learning preferences from <@demonstrations@>(@Modeling Interaction via the Principle of Maximum Causal Entropy@) (possibly <@ranked@>(@Ranking-Based Reward Extrapolation without Rankings@)), <@comparisons@>(@Fine-Tuning GPT-2 from Human Preferences@), <@proxy rewards@>(@Inverse Reward Design@), and even the <@observed state@>(@Learning Preferences by Looking at the World@). The insight of this paper is that these are all instances of a simple underlying formalism.

Specifically, these forms of preference learning can be described by two properties: (1) the set of choices that the human picks from and (2) how each choice corresponds to a distribution over agent trajectories. Given these properties, we assume that the human makes their choice according to a Boltzmann-rational model (where the human is more likely to choose an option if it leads to higher expected reward). We have now specified a likelihood over the choice given the reward, and we can use Bayes rule to infer a distribution over the reward given the human's choice.

Consider more exotic types of feedback, such as the human's decision to <@turn the agent off@>(@The Off-Switch Game@). Here, the human has two options: turning the agent off (corresponding to the agent staying still forever), or letting it continue (corresponding to the agent taking the trajectory that maximizes its current expected reward). If the agent has the right reward function, then the Boltzmann rational human would let it continue; as a result, if the human instead tries to turn the agent off, Bayes Rule allows the agent to infer that its belief about the reward must be wrong. Thus, even this decision of whether to turn the agent off can be captured in this framework.

The paper then shows two examples of new feedback types that can be generated from this framework: first, credit assignment, in which the human identifies a subset of the trajectory that had maximal reward, and second, meta-choice, where the choice of which _type_ of feedback to give can itself give information about the reward function.


I really like that this paper has experiments with real humans. It's definitely a problem that IRL assumes that the expert is (approximately) optimal -- this means that you can't learn where the expert is likely to be wrong, and so it is hard to exceed the expert's performance. It's very difficult to figure out how to deal with the possbility of a biased expert, and I'm happy to see work that takes a shot at it.

Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior

Siddharth Reddy, Anca D. Dragan, Sergey Levine

Inverse reinforcement learning algorithms typically assume that the demonstrations come from an expert who is approximately optimal. However, this is often not the case, at least when the experts are fallible humans. This paper considers the case where the expert has an incorrect model of the dynamics (transition function) of the environment, and proposes learning the expert's model of the dynamics to improve reward function inference. However, this leads to severe unidentifiability problems, where many models of the dynamics are compatible with the observed behavior. To overcome this, they assume that they have multiple tasks with known reward functions, which they use to infer the expert's dynamics. This is then used to infer the reward function in a new task using an adaptation of max causal entropy IRL. The dynamics can be an arbitrary neural net while the reward function is a weighted linear combination of features. They evaluate the inference of the dynamics model with real humans on Lunar Lander. Given transcripts of humans playing Lunar Lander, they infer the underlying (incorrect) dynamics model. Then, when the human takes an action, they predict which next state the human wanted to achieve, and replace the human's action with the action that would actually get close to the state the human wanted.


The standard inverse reinforcement learning assumption about humans acting optimally seems unrealistic; I think this paper provides an insightful initial step in not having that assumption and models the non-optimal version of the problem in a clean and compelling way. I think it's a noteworthy observation that this problem is very sensitive to the player's learning model, and I agree with the paper that this suggests that we should put effort into researching actual human learning strategies. I am unsure how to think about the insights here generalizing to other inverse reinforcement learning cases.

The Assistive Multi-Armed Bandit

Lawrence Chan, Dylan Hadfield-Menell, Siddhartha Srinivasa, Anca Dragan

Standard approaches for inverse reinforcement learning assume that humans are acting optimally according to their preferences, rather than learning about their preferences as time goes on. This paper tries to model the latter by introducing the _assistive multi-armed bandit_ problem.

In the standard _multi-armed bandit_ problem, a player repeatedly chooses one of several "arms" to pull, where each arm provides reward according to some unknown distribution. Imagine getting 1000 free plays on your choice of 10 different, unknown slot machines. This is a hard problem since the player must trade off between exploration (learning about some arm) and exploitation (pulling the best arm so far). In _assistive multi-armed bandit_, a robot is given the opportunity to intercept the player every round and pull an arm of its choice. If it does not intercept, it can see the arm pulled by the player but not the reward the player receives. This formalizes the notion of an AI with only partial information trying to help a learning agent optimize their reward.

The paper does some theoretical analysis of this problem as well as an experimental set-up involving a neural network and players acting according to a variety of different policies. It makes several observations about the problem:
- A player better at learning does not necessarily lead to the player-robot team performing better-- the robot can help a suboptimal player do better in accordance with how much information the player's arm pulls convey about the reward of the arm.
- A robot is best at assisting when it has the right model for how the player is learning.
- A robot that models the player as learning generally does better than a robot that does not, even if the robot has the wrong model for the player's learning.
- The problem is very sensitive to which learning model the player uses and which learning model the robot assumes. Some player learning models can only be effectively assisted when they are correctly modeled. Some robot-assumed learning models effectively assist for a variety of actual player learning models.


BAIR blog post: Cooperatively Learning Human Values

Cooperative Inverse Reinforcement Learning

Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, Stuart Russell

This paper provides a formalization of the three principles from the book, in the case where there is a single human H and a single robot R. H and R are trying to optimize the same reward function. Since both H and R are represented in the environment, it can be the _human's_ reward: that is, it is possible to reward the state where the human drinks coffee, without also rewarding the state where the robot drinks coffee. This corresponds to the first principle: that machines should optimize _our_ objectives. The second principle, that machines should initially be uncertain about our objectives, is incorporated by assuming that _only H knows the reward_, requiring R to maintain a belief over the reward. Finally, for the third principle, R needs to get information about the reward from H's behavior, and so R assumes that H will choose actions that best optimize the reward (taking into account the fact that R doesn't know the reward).

This defines a two-player game, originally called a CIRL game but now called an _assistance game_. We can compute optimal joint strategies for H and R. Since this is an _interactive_ process, H can do better than just acting optimally as if R did not exist (the assumption typically made in IRL): H can _teach_ R what the reward is. In addition, R does not simply passively listen and then act, but interleaves learning and acting, and so must manage the explore-exploit tradeoff.

See also <@Learning to Interactively Learn and Assist@>, which is inspired by this paper and does a similar thing with deep RL.


While I agree with the broad thrust of this paper, I do have one nitpick: the game ends immediately after H chooses whether or not to shut off R. In reality, if R isn't shut off, the assistance game will continue, which changes the incentives. If R can be relatively confident in the utility of _some_ action (e.g. doing nothing), then it may be a better plan for it to disable the shutdown button, and then take that action and observe H in the mean time to learn the reward. Then, after it has learned more about the reward and figured out why H wanted to shut it down, it can act well and get utility (rather than being stuck with the zero utility from being shut down). While this doesn't seem great, it's not _obviously_ bad: R ends up doing nothing until it can figure out how to actually be useful, hardly a catastrophic outcome. Really bad outcomes only come if R ends up becoming confident in the wrong reward due to some kind of misspecification, as suggested in [Incorrigibility in the CIRL Framework](http://arxiv.org/abs/1709.06275), summarized next.

The Off-Switch Game

This paper studies theoretically the impact of uncertainty over the reward on R's incentives around potential off switches. It proposes the simplest model that the authors expect to lead to generalizable results. R and H are in an assistance game, in which R goes first. R may either take an action a, getting utility u, or shut itself down, getting utility 0. In either case, the game ends immediately. Alternatively, R can choose to wait, in which case H can either shut down R, getting utility 0, or allow R to go ahead with action a, getting utility u.

If H is perfectly rational, then waiting is always an optimal action for R, since H will ensure that the team gets max(u, 0) utility. There can be other optimal actions: if R is sure that u >= 0, then taking action a is also optimal, and similarly if R is sure that u <= 0, then shutting down is also optimal. However, if H is not rational, and sometimes fails to take the utility-maximizing action (in a way R can't predict), then things get murkier. If R is sure about the value of u, then it is never optimal to wait, better to just take the action a (if u >= 0) or shut down (if u < 0) rather than let H screw it up. If R is pretty confident that u is positive, it may still decide to take action a, rather than risk that H makes the wrong decision. However, if R is very uncertain about the sign of u, then waiting becomes optimal again. In general, more uncertainty over the reward leads to more deferential behavior (allowing H to shut it off), but at a cost: R is much less able to help H when it is very uncertain about the reward.


I really like this paper as an example of how to apply the third principle. This was the paper that caused me to start thinking about how we should be thinking about the assumed vs. actual information content in things (here, the key insight is that RL typically assumes that the reward function conveys much more information than it actually does). That probably influenced the development of <@Preferences Implicit in the State of the World@>(@Learning Preferences by Looking at the World@), which is also an example of the third principle and this information-based viewpoint, as it argues that the state of the world is caused by human behavior and so contains information about human preferences.

It's worth noting that in this paper the lava avoidance is both due to the belief over the true reward, _and the risk aversion_. The agent would also avoid pots of gold in the test environment if it never saw it in the training environment. IRD only gives you the correct uncertainty over the true reward; it doesn't tell you how to use that uncertainty. You would still need safe exploration, or some other source of information, if you want to reduce the uncertainty.

Inverse Reward Design

Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart Russell, Anca Dragan

Usually, in RL, the reward function is treated as the _definition_ of optimal behavior, but this conflicts with the third principle, which says that human behavior is the ultimate source of information about human preferences. Nonetheless, reward functions clearly have some information about our preferences: how do we make it compatible with the third principle? We need to connect the reward function to human behavior somehow.

This paper proposes a simple answer: since reward designers usually make reward functions through a process of trial-and-error where they test their reward functions and see what they incentivize, the reward function _tells us about optimal behavior in the **training** environment(s)_. The authors formalize this using a Boltzmann rationality model, where the reward designer is more likely to pick a _proxy reward_ when it gives higher _true reward_ in the _training environment_ (but it doesn't matter if the proxy reward becomes decoupled from the true reward in some test environment). With this assumption connecting the human behavior (i.e. the proxy reward function) to the human preferences (i.e. the true reward function), they can then perform Bayesian inference to get a posterior distribution over the _true_ reward function.

They demonstrate that by using risk-averse planning with respect to this posterior distribution, the agent can avoid negative side effects that it has never seen before and has no information about. For example, if the agent was trained to collect gold in an environment with dirt and grass, and then it is tested in an environment with lava, the agent will know that even though the specified reward was indifferent about lava, this doesn't mean much, since _any_ weight on lava would have led to the same behavior in the training environment. Due to risk aversion, it conservatively assumes that the lava is bad, and so successfully avoids it.

See also <@Active Inverse Reward Design@>, which builds on this work.


Overall, I found the idea expressed in this paper to be well-articulated. While I think that the grid-world environment is a bit simplistic, their results are interesting. Being able to learn intent in an online manner is an important problem to solve if we’re interested in robust collaboration between humans and autonomous agents. However, the authors point out that training on pixel input fails in the majority of cases, 64% of the time, which raises concerns about how well the method would generalize to non-trivial environments.

**Rohin's opinion:** I'm excited that the ideas from [CIRL](https://arxiv.org/abs/1606.03137) are making their way to deep RL. Ultimately I expect we'll want an agent that takes all of its sensory data as evidence about "what the human wants", rather than relying on a special reward channel, or a special type of data called "comparisons" or "demonstrations", and this work takes that sort of approach.

For these simple environments, an agent trained to perform well with another artificial agent will generalize reasonably well to real humans, because there's only a few reasonable strategies for the principal to take. However, with more complex environments, when there are many ways to interact, we can't expect such generalization. (I'll have a paper and blog post coming out soon about this phenomenon.)

Learning to Interactively Learn and Assist

Mark Woodward, Chelsea Finn, Karol Hausman

[Cooperative Inverse Reinforcement Learning](https://arxiv.org/abs/1606.03137) proposed a model in which an AI assistant would help a human principal, where only the principal knows the task reward. This paper explores this idea in the context of deep reinforcement learning. In their grid-world environment, two agents move around and pick up lemons or plums. The principal is penalized for moving, but is the only one who knows whether plums or lemons should be picked up. The authors hypothesize that simply by jointly training the two agents to maximize rewards, they will automatically learn to interact in order for the assistant to learn the task, rather than requiring an explicit mechanism like comparisons or demonstrations.

Recurrent Q-networks are used for the agents, which are then trained via deep Q-learning. The authors run several experiments that show emergent interaction. In the first experiment, when the principal is penalized for moving it learns to demonstrate the task to the assistant, and then let the assistant finish the job. In the second experiment, when the assistant has a restricted field of view, it learns to follow the principal until it can infer whether the principal wants plums or lemons. In the third, they tell the assistant the task 50% of the time, and so the principal is initially unsure whether the agent needs any direction (and due to the motion cost, the principal would rather not do anything). When the agent knows the task, it performs it. When the agent doesn't know the task, it moves closer to the principal, in effect "asking" what the reward is, and the principal moves until it can see the object, and then "answers" by either moving towards the object (if it should be collected) or doing nothing (if not). Finally, the authors run an experiment using pixels as input. While they had to switch to dueling DQNs instead of vanilla DQNs, they show that the joint reward is competitive with the grid approach. They also run an experiment with human principals and show that the human/assistant pair outperforms the solo-human setup.


I'm really excited to see a project that is very directly aimed at inferring end user intent in a complex environment. This seems like a great direction for the field to move towards. I think Minecraft would also be great as a test bed for the setting in which _researchers or engineers_ (as opposed to end users) are trying to get an agent to do something: we can assume more expertise and knowledge here. Ideally, this would allow us to solve more complex tasks than can be accomplished with natural language from end users. I personally plan to do work with Minecraft along these lines.

While this project does need to infer intent, it probably won't require the sort of pragmatic understanding shown by e.g. [Cooperative IRL](https://arxiv.org/abs/1606.03137). Even understanding what the human is literally asking for in Minecraft is currently beyond our capabilities.

CraftAssist: A Framework for Dialogue-enabled Interactive Agents

Why Build an Assistant in Minecraft?

Arthur Szlam, Jonathan Gray, Kavya Srinet, Yacine Jernite, Armand Joulin, Gabriel Synnaeve, Douwe Kiela, Haonan Yu, Zhuoyuan Chen, Siddharth Goyal, Demi Guo, Danielle Rothermel, C. Lawrence Zitnick, Jason Weston

This position paper proposes a new challenge for AI research: building a bot that can provide assistance in [Minecraft](https://www.minecraft.net/en-us/) (creative mode). A [companion paper](https://research.fb.com/wp-content/uploads/2019/07/CraftAssist-A-Framework-for-Dialogue-enabled-Interactive-Agents-v3.pdf) presents an initial setup for such an agent.

The main goal here is to advance natural language understanding, intent inference and instruction following. As a result, there is no formal specification like a reward function -- in their own words, "the ultimate goal of the bot is to be a useful and fun assistant in a wide variety of tasks specified and evaluated by human players". They chose Minecraft in particular partly because it has a very rich space of _tasks_, even though the _execution_ of any given task is relatively straightforward. They script many low level policies to automate this execution in order to make learning easier (for example, they have policies to navigate to a location or to build specified structures) and focus the learning challenge on figuring out what the user wants.

The current version of the bot takes dialogue from the user and uses a neural model to parse it into an _action dictionary_ that unambiguously specifies what the agent should do -- I think this neural model is the main thing to be learned. There are a bunch of details on how the rest of the modules work as well. They have also released three datasets: a semantic parsing dataset that associates instructions with action dictionaries, a house dataset that has trajectories where a human builds a house, and a semantic segmentation dataset that labels various parts of houses.


While I don't disagree with the reasoning, I disagree with the main thrust of this post. I wrote a long [comment](https://www.alignmentforum.org/posts/BKjJJH2cRpJcAnP7T/thoughts-on-human-models#8BAMtuZfBNqyF7Jng) about it; the TL;DR is that since humans want very specific behavior out of AI systems, the AI system needs to get a lot of information from humans about what it should do, and if it understands all that information then it necessarily has a (maybe implicit) human model. In other words, if you require your AI system not to have human models, it will not be very useful, and people will use other techniques.

Thoughts on Human Models

Many approaches to AI safety involve modeling humans in some way, for example in order to correctly interpret their feedback. However, there are significant disadvantages to human modeling. First and most importantly, if we have AI systems do useful things _without_ modeling humans, then we can use human approval as a "test set": we can check whether the AI's behavior is something we approve of, and this is an _independent_ evaluation of the AI system. However, if the AI system had a human model, then it may have optimized its behavior for human approval, and so we cannot use approval as a "test set". Second, if our AI system has a catastrophic bug, it seems better if it doesn't have any human models. An AI system without human models will at worst optimize for some unrelated goal like paperclips, which at worst leads to it treating humans as obstacles and causing extinction. However, an AI system with human models with a catastrophic bug might optimize for human suffering, or having humans respond to email all day, etc. Thirdly, an AI system with human models might be simulating conscious beings that can suffer. Fourthly, since humans are agent-like, an AI system that models humans is likely to produce a subsystem that is agent-like and so dangerous.

The authors then discuss why it might be hard to avoid human models. Most notably, it is hard to see how to use a powerful AI system that avoids human models to produce a better future. In particular, human models could be particularly useful for interpreting specifications (in order to do what humans mean, as opposed to what we literally say) and for achieving performance given a specification (e.g. if we want to replicate aspects of human cognition). Another issue is that it is hard to avoid human modeling, since even "independent" tasks have some amount of information about human motivations in selecting that task.

Nevertheless, the authors would like to see more work on engineering-focused approaches to AI safety without human models, especially since this area is neglected, with very little such work currently. While MIRI does work on AI safety without human models, this is from a very theoretical perspective. In addition to technical work, we could also promote certain types of AI research that is less likely to develop human models "by default" (e.g. training AI systems in procedurally generated simulations, rather than on human-generated text and images).