A core feature of this paper is the unsupervised approach to producing question decompositions. By doing this, it's possible to augment the data-set significantly by question-crawling the data-sets which helps explain why the model has performance on-par with supervised approaches. Moreover, looking at a few decomposition examples from the model seems to indicate that relevant sub-questions are being discovered. It's worth noting that decompositions with more than two questions are unlikely due to the specific loss used in the main paper. In the appendix, the authors experiment with a different loss for the pseudo-decomposition that allows more questions in the decomposition, but it performs slightly worse than the original loss. This makes me wonder whether or not such a procedure would be useful if used recursively to create sub-sub-questions. Overall, I think the decomposition is useful for both down-stream processing and interpretation.

**Rohin's opinion:** The capabilities of methods like iterated amplification depend on the ability to solve hard questions by decomposing them into simpler questions that we already know how to answer, and then combining the results appropriately. This paper demonstrates that even a very basic unsupervised approach ("decompose into the most similar simpler questions") to decomposition can work quite well, at least for current AI systems.

In private correspondence, Ethan suggested that in the long term a semi-supervised approach would probably work best, which agrees with my intuitions.

Unsupervised Question Decomposition for Question Answering

Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, Douwe Kiela

Existing methods are proficient at simple question and answering (QA). These simple questions are called single-hop and can be answered with a single yes/no or underlined passage in the text. However, progress on the more difficult task of multi-hop QA lags behind. **This paper introduces a method that can decompose hard multi-hop questions into easier single-hop questions that existing QA systems can answer.** Since collecting labeled decompositions is hard, the authors introduce a pseudo-decomposition where multi-hop questions are matched with similar single-hop questions while making sure the single-hop questions are diverse. Following this, the model is trained to map multi-hop questions to simpler subquestions using _unsupervised_ sequence-to-sequence learning (as they found the supervised version performed worse). They show large improvement on the popular HotPot QA baseline with large improvement on out-of-domain questions due to the ability of sub-questions to help gather supporting facts that can be used to answer questions.


This seems like a reasonable algorithm to study, though I suspect there is a simpler algorithm that doesn't use debate that has the same advantages. Some other thoughts in [this thread](https://www.alignmentforum.org/posts/dJSD5RK6Qoidb3QY5/synthesizing-amplification-and-debate#PhH6BstgNf5zmZK8W).

Synthesizing amplification and debate

The distillation step in <@iterated amplification@>(@Learning Complex Goals with Iterated Amplification@) can be done using imitation learning. However, as argued in [Against Mimicry](https://ai-alignment.com/against-mimicry-6002a472fc42), if your model M is unable to do perfect imitation, there must be errors, and in this case the imitation objective doesn't necessarily incentivize a graceful failure, whereas a reward-based objective does. So, we might want to add an auxiliary reward objective. This post proposes an algorithm in which the amplified model answers a question via a <@debate@>(@AI safety via debate@). The distilled model can then be trained by a combination of imitation of the amplified model, and reinforcement learning on the reward of +1 for winning the debate and -1 for losing.


While it is interesting to see what doesn't work with feature debates, I see two problems that make it hard to generalize these results to regular debate. First, I see debate as being truth-seeking in the sense that the answer you arrive at is (in expectation) more accurate than the answer the judge would have arrived at by themselves. However, this paper wants the answers to actually be _correct_. Thus, they claim that for sufficiently complicated questions, since the debate can't reach the right answer, the debate isn't truth-seeking -- but in these cases, the answer is still in expectation more accurate than the answer the judge would come up with by themselves.

Second, feature debate doesn't allow for decomposition of the question during the debate, and doesn't allow the agents to challenge each other on particular questions. I think this limits the "expressive power" of feature debate to P, while regular debate reaches PSPACE, and is thus able to do much more than feature debate. See this [comment](https://www.alignmentforum.org/posts/RQoSCs9SePDMLJvfz/new-paper-when-is-truth-telling-favored-in-ai-debate#gCeKuJ62HmLtPB9C9) for more details.

Paper: (When) Is Truth-telling Favored in AI Debate?

(When) is Truth-telling Favored in AI debate?

<@Debate@>(@AI safety via debate@) aims to train an AI system using self-play to win "debates" which aim to convincingly answer a question, as evaluated by a human judge. The main hope is that the equilibrium behavior of this game is for the AI systems to provide true, useful information. This paper studies this in a simple theoretical setting called _feature debates_. In this environment, a "world" is sampled from some distribution, and the agents (who have perfect information) are allowed to make claims about real-valued "features" of the world, in order to answer some question about the features of the world. The judge is allowed to check the value of a single feature before declaring a winner, but otherwise knows nothing about the world.

If either agent lies about the value of a feature, the other agent can point this out, which the judge can then check; so at the very least the agents are incentivized to honestly report the values of features. However, does this mean that they will try to answer the full question truthfully? If the debate has more rounds than there are features, then it certainly does: either agent can unilaterally reveal every feature, which uniquely determines the answer to the question. However, shorter debates need not lead to truthful answers. For example, if the question is whether the first K features are all 1, then if the debate length is shorter than K, there is no way for an agent to prove that the first K features are all 1.


I think it's a cool and non-trivial result that debating agents can in fact improve model accuracy. It feels hard to extrapolate much from this narrow example to debate as a general AI safety technique. The judge model is answering multiple-choice questions rather than e.g. evaluating a detailed plan of action, and debating agents are quoting from existing text rather than generating their own potentially fallacious statements.

Finding Generalizable Evidence by Learning to Convince Q&A Models

Ethan Perez, Siddharth Karamcheti, Rob Fergus, Jason Weston, Douwe Kiela, Kyunghyun Cho

This paper tries to improve performance on multiple-choice questions about text passages using a technique similar to <@AI safety via debate@>. The set-up consists of a **judge model** and one or more **evidence agents**. First, the judge model is pretrained on samples consisting of a passage, a multiple-choice question about that passage, and the correct answer to that question. Then, in the experimental portion of the set-up, instead of looking at a full passage, the judge model looks at a subsequence of the passage created by combining the outputs from several evidence agents. Each evidence agent has been given the same passage and assigned a particular answer to the question, and must select a limited number of sentences from the passage to present to the judge model to convince it of that answer.

The paper varies several parameters in its setup, including the training process for the judge model, the questions used, the process evidence agents use to select sentences, etc. It finds that for many settings of these parameters, when judge models are tasked with generalizing from shorter passages to longer passages, or easier passages to harder passages, they do better with the new passages when assisted by the evidence agents. It also finds that the sentences given as evidence by the evidence agents are convincing to humans as well as the judge model.


What are the differences between all the iterative/recursive approaches to AI alignment?


Towards a mechanistic understanding of corrigibility

One general approach to align AI is to train _and verify_ that an AI system performs acceptably on all inputs. However, we can't do this by simply trying out all inputs, and so for verification we need to have an acceptability criterion that is a function of the "structure" of the computation, as opposed to just input-output behavior. This post investigates what this might look like if the acceptability criterion is some flavor of corrigibility, for an AI trained via amplification.


The switch from Factored Cognition to Factored Evaluation is interesting. While it does make it more relevant outside the context of AI alignment (since principal-agent problems abound outside of AI), it still seems like the major impact of Ought is on AI alignment, and I'm not sure what the difference is there. In <@iterated amplification@>(@Learning Complex Goals with Iterated Amplification@), when decomposing tasks in the Factored Cognition sense, you would use imitation learning during the distillation step, whereas with Factored Evaluation, you would use reinforcement learning to optimize the evaluation signal. The switch would be useful if you expect the reinforcement learning to work significantly better than imitation learning.

However, with Factored Evaluation, the agent that you train iteratively is one that must be good at evaluating tasks, and then you'd need another agent that actually performs the task (or you could train the same agent to do both). In contrast, with Factored Cognition you only need an agent that is performing the task. If the decompositions needed to perform the task are different from the decompositions needed to evaluate the task, then Factored Cognition would presumably have an advantage.

Delegating open-ended cognitive work

This is the latest explanation of the approach Ought is experimenting with: Factored Evaluation (in contrast to <@Factored Cognition@>). With Factored Cognition, the idea was to recursively decompose a high-level task until you reach subtasks that can be directly solved. Factored Evaluation still does recursive decomposition, but now it is aimed at _evaluating_ the work of experts, along the same lines as <@recursive reward modeling@>(@Scalable agent alignment via reward modeling@).

This shift means that Ought is attacking a very natural problem: how to effectively delegate work to experts while avoiding principal-agent problems. In particular, we want to design incentives such that untrusted experts under the incentives will be as helpful as experts intrinsically motivated to help. The experts could be human experts or advanced ML systems; ideally our incentive design would work for both.

Currently, Ought is running experiments with reading comprehension on Wikipedia articles. The experts get access to the article while the judge does not, but the judge can check whether particular quotes come from the article. They would like to move to tasks that have a greater gap between the experts and the judge (e.g. allowing the experts to use Google), and to tasks that are more subjective (e.g. whether the judge should get Lasik surgery).


I strongly agree with the general point that we can get significant safety by [improving the human policy](https://www.alignmentforum.org/posts/eD9T4kiwB6MHpySGE/the-human-side-of-interaction) ([AN #43](https://mailchi.mp/768a8130013f/alignment-newsletter-43)), especially with HCH and iterated amplification, since they depend on having good human overseers, at least initially.

HCH is not just Mechanical Turk

In [Humans Consulting HCH](https://www.alignmentforum.org/posts/NXqs4nYXaq8q6dTTx/humans-consulting-hch) (HCH) ([AN #34](https://mailchi.mp/f1947668b183/alignment-newsletter-34)) a human is asked a question and is supposed to return an answer. The human can ask subquestions, which are delegated to another copy of the human, who can ask subsubquestions, ad infinitum. This post points out that HCH has a free parameter -- the base human policy. We could imagine e.g. taking a Mechanical Turk worker and using them as the base human policy, and we could argue that HCH would give good answers in this setting as long as the worker is well-motivated, since he is using "human-like" reasoning. However, there are other alternatives. For example, in theory we could formalize a "core" of reasoning. For concreteness, suppose we implement a lookup table for "simple" questions, and then use this lookup table. We might expect this to be safe because of theorems that we proved about the lookup table, or by looking at the process by which the development team created the lookup table. In between these two extremes, we could imagine that the AI researchers train the human overseers about how to corrigibly answer questions, and then the human policy is used in HCH. This seems distinctly more likely to be safe than the first case.


If I imagine each level A[n] as maximizing the expected value of some simple utility function, I agree that it would be surprising if the result was not one of your first three cases. Intuitively, either we already have all of the friendly utility function, and we didn't need induction, or we didn't and bad things happen, which corresponds to cases 1 and 3.

But it seems like one of the main points of iterated amplification is that at least the initial levels need not be maximizing the expected value of some simple utility. In that case, there seems to be a much wider space of possible designs.

For example, we could have a system that has the epistemic state of wanting to help humans but knowing that it doesn't know how best to do that, and so asking humans for feedback and deferring to them when appropriate. Such a system with amplification might eventually learn the friendly utility function and start maximizing that, but it seems like there could be many iterations before that point, during which it is corrigible in the sense of deferring to humans and not maximizing its current conception of what is best.

I don't have a strong sense at the moment what would happen, but it seems plausible that the induction will go through and will have "actually mattered".

Disagreement with Paul: alignment induction

The amplification step in iterated distillation and amplification (IDA) requires an inductive argument saying that if the agent at level n is aligned, then so is the one at level n+1. However, in order to get the induction to work, you need to say not just that agent at level n won't take unaligned actions, but that it will also "assess the decisions of a higher agent in a way that preserves alignment (and preserves the preservation of alignment, and so on)". This seems like a much less intuitive criterion, and so getting the base case of an agent that a human can verify has this property may be too hard -- it probably has to have all of the friendly utility function in the base case itself, or perhaps it gets it after one or two iterations (if it needs to solve a few problems along the way).


I like this framing of what guarantees we want to achieve. Hopefully we can apply this to other AI systems as well.

Techniques for optimizing worst-case performance

Two guarantees

The "minimum viable product" of AI alignment research would be an AI that satisfies two guarantees -- first, that it achieves good average-case behavior (the performance guarantee), and second, that it achieves reasonable worst-case behavior (the control guarantee). There's then some discussion of how we might establish these guarantees inductively about [amplification](https://ai-alignment.com/policy-amplification-6a70cbee4f34).


I agree with the criticism that breaking tasks down into subtasks causes you to lose a lot of capability. I'm mostly confused by the analysis in terms of "small" and "large" chunks and whether there are "attacks" or not so don't want to comment on it, but I think this is addressing an important open question and I'd like to see people try to understand what's going on and have more diversity of thought here.

Can corrigibility be learned safely?

We hope that iterated distillation and amplification is sufficient to learn to be corrigible, because the subagents in the amplification procedure are given simple enough tasks that the tasks are not "attacks" that lead to incorrigible behavior. However, if we are forced to break down all of our tasks into simple subtasks that are solved without access to any of the surrounding context, then we will end up with an AI that interprets us very literally and is unable to use "common sense", which could lead to incorrigible behavior. You might think that we could get around this by learning about corrigible behavior from the environment (eg. by observing how humans interact with each other), but then we have the same problem that if you learn "large" chunks from the environment you could be attacked, and if you learn "small" chunks you will be too literal.


This feels to me like a misunderstanding of (Paul's version of) corrigibility. The post takes a perspective where it looks at the _outcomes_ of an AI acting in the world, whereas my understanding is that corrigibility is also about the _motivation_ underlying an AI's choices, regardless of what outcomes actually happen.

The limits of corrigibility

There are situations in which an AI would be able (or even required) to take some actions that can influence the human's values. For example, if a billionaire asked the AI to help him figure out how to prioritize between charities to give to, based on its answer the billionaire could move towards effective altruism, or focus more on administrative costs, or help animals instead of humans -- and all of these are "compatible" with the billionaire's current values. In such a situation, it's unclear what "corrigibility" means. It seems like the only "corrigible" behavior here is to explicitly figure out what the billionaire values, and then help him optimize those values -- but then in order to get corrigible behavior, we need to learn human values, which we know is hard.


AlphaGo Zero and Capability Amplification

A comment on the IDA-AlphaGoZero metaphor; capabilities versus alignment

Paul Christiano has [compared](https://ai-alignment.com/alphago-zero-and-capability-amplification-ede767bb8446) iterated distillation and amplification (IDA) to AlphaGo Zero. However, we usually don't think of AlphaGo Zero as having any alignment issues. Alex points out that we could think of this another way -- we could imagine that the value network represents the "goals" of AlphaGo Zero. In that case, if states get an incorrect value, that is misalignment. AlphaGo Zero corrects this misalignment through MCTS (analogous to amplification in IDA), which updates the values according to the ground truth win/loss reward (analogous to the human). This suggests that in IDA, we should be aiming for any reduction in alignment from distillation to be corrected by the next amplification step.


Making AI Safe through Debate

This hour-long podcast is a good introduction to iterated amplification and debate, from a more ML perspective than most other explanations.


Epistemology of HCH

This post identifies and explores three perspectives one can take on <@HCH@>(@Humans Consulting HCH@):

1. **Philosophical abstraction:** In this perspective, HCH is an operationalization of the concept of one’s enlightened judgment.
2. **Intermediary alignment scheme:** Here we consider HCH as a scheme that arguably would be aligned if we could build it.
3. **Model of computation:** By identifying the human in HCH with some computation primitive (e.g. arbitrary polynomial-time algorithms), we can think of HCH as a particular theoretical model of computation that can be done using that primitive.


I enjoyed this sequence: I’m glad to see more analysis of what is and isn’t necessary for iterated amplification and debate to work, as well as more theoretical models of debate. I broadly agreed with the conceptual points made, with one exception: I’m not convinced that we should not allow brute force for HCH, and for similar reasons I don’t find the arguments that HCH won’t be superintelligent convincing. In particular, the hope with iterated amplification is to approximate a truly massive tree of humans, perhaps a tree containing around 2^100 (about 1e30) base agents / humans. At that scale (or even at just a measly billion (1e9) humans), I don’t expect the reasoning to look anything like what an individual human does, and approaches that are more like “brute force” seem a lot more feasible.

One might wonder why I think it is possible to approximate a tree with more base agents than there are grains of sand in the Sahara desert. Well, a perfect binary tree of depth 99 would have 1e30 nodes; thus we can roughly say that we’re approximating 99-depth-limited HCH. If we had perfect distillation, this would take 99 rounds of iterated amplification and distillation, which seems quite reasonable. Of course, we don’t have perfect distillation, but I expect that to be a relatively small constant factor on top (say 100x), which still seems pretty reasonable. (There’s more detail about how we get this implicit exponential-time computation in <@this post@>(@Factored Cognition@).)

Factored Cognition sequence

The <@Factored Cognition Hypothesis@>(@Factored Cognition@) informally states that any task can be performed by recursively decomposing the task into smaller and smaller subtasks until eventually the smallest tasks can be done by a human. This sequence aims to formalize the hypothesis to the point that it can be used to argue for the outer alignment of (idealized versions of) <@iterated amplification@>(@Supervising strong learners by amplifying weak experts@) and <@debate@>(@AI safety via debate@).

The key concept is that of an _explanation_ or _decomposition_. An explanation for some statement **s** is a list of other statements **s1, s2, … sn** along with the statement “(**s1** and **s2** and … and **sn**) implies **s**”. A _debate tree_ is a tree in which for a given node **n** with statement **s**, the children of **n** form an explanation (decomposition) of **s**. The leaves of the tree should be statements that the human can verify. (Note that the full formalism has significantly more detail, e.g. a concept of the “difficulty” for the human to verify any given statement.)

We can then define an idealized version of debate, in which the first debater must produce an answer with associated explanation, and the second debater can choose any particular statement to expand further. The judge decides the winner based on whether they can confidently verify the final statement or not. Assuming optimal play, the correct (honest) answer is an equilibrium as long as:

**Ideal Debate Factored Cognition Hypothesis:** For every question, there exists a debate tree for the correct answer where every leaf can be verified by the judge.

The idealized form of iterated amplification is <@HCH@>(@Humans Consulting HCH@); the corresponding Factored Cognition Hypothesis is simply “For every question, HCH returns the correct answer”. Note that the _existence_ of a debate tree is not enough to guarantee this, as HCH must also _find_ the decompositions in this debate tree. If we imagine that HCH gets access to a decomposition oracle that tells it the right decomposition to make at each node, then HCH would be similar to idealized debate. (HCH could of course simply try all possible decompositions, but we are ignoring that possibility: the decompositions that we rely on should reduce or hide complexity.)

Is the HCH version of the Factored Cognition Hypothesis true? The author tends to lean against (more specifically, that HCH would not be superintelligent), because it seems hard for HCH to find good decompositions. In particular, humans seem to improve their decompositions over time as they learn more, and also seem to improve the concepts by which they think over time, all of which are challenging for HCH to do. On the other hand, the author is cautiously optimistic about debate.


I enjoyed reading through this agenda, especially because of the inline clarifications from Paul. I found it actually more useful to see what the author initially thought and what Paul’s correction was, relative to the scenario in which the author simply made the correction and presented the final result, as by including both it makes clear what the (probably common) misunderstanding was.

My Understanding of Paul Christiano's Iterated Amplification AI Safety Research Agenda

This post provides an informal description of the full [iterated amplification](https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd) agenda, aimed at all levels of technical expertise. It is significantly more comprehensive than past descriptions.


This seems eminently sensible to me. Of course, evidence law is going to be specialized to arguments about innocence or guilt of a crime, and may not generalize to what we would like to do with debate, but it still seems like we should be able to learn some generalizable lessons.

Parallels Between AI Safety by Debate and Evidence Law

<@Debate@>(@Writeup: Progress on AI Safety via Debate@) requires us to provide a structure for a debate as well as rules for how the human judge should decide who wins. This post points out that we have an existing system that has been heavily optimized for this already: evidence law, which governs how court cases are run. A court case is high-stakes and involves two sides presenting opposing opinions; evidence law tells us how to structure these arguments and how to limit the kinds of arguments debaters can use. Evidence is generally admissible by default, but there are many exceptions, often based on the fallibility of fact-finders.

As a result, it may be fruitful to look to evidence law for how we might structure debates, and to see what types of arguments we should be looking for.


Weak HCH accesses EXP

This followup to last week's <@Alignment proposals and complexity classes@> shows that the amplification-based proposals can access EXP.


In practice our models are not going to reach the optimal loss, and humans won’t solve arbitrary polynomial-time problems, so these theorems won’t directly apply to reality. Nonetheless, this does seem like a worthwhile check to do -- it feels similar to ensuring that a deep RL algorithm has a proof of convergence under idealized assumptions, even if those assumptions won’t actually hold in reality. I have much more faith in a deep RL algorithm that started from one with a proof of convergence and then was modified based on empirical considerations.

Alignment proposals and complexity classes

The original <@debate@>(@AI safety via debate@) paper showed that any problem in PSPACE can be solved by optimal play in a debate game judged by a (problem-specific) algorithm in P. Intuitively, this is an illustration of how the mechanism of debate can take a weak ability (the ability to solve arbitrary problems in P) and amplify it into a stronger ability (the ability to solve arbitrary problems in PSPACE). One would hope that similarly, debate would allow us to amplify a human’s problem-solving ability into a much stronger problem-solving ability.

This post applies this technique to several other alignment proposals. In particular, for each proposal, we assume that the “human” can be an arbitrary polynomial-time algorithm, and the AI models are optimal w.r.t their loss functions, and we ask which problems we can solve using these capabilities. The post finds that, as lower bounds, the various forms of amplification can access PSPACE, while <@market making@>(@AI safety via market making@) can access EXP. If there are untamperable pointers (so that the polynomial-time algorithm can look at objects of an arbitrary size, as long as it only looks at a polynomial-sized subset of them), then amplification and market making can access R (the set of decidable problems).


I like the simplicity of the idea "find the point at which the human no longer changes their mind", and that this is a new idea of how we can scale training of AI systems beyond human level performance. However, I’m not convinced that the training procedure given in this post would end up at this equilibrium, unless the human very specifically guided the training to do so (an assumption I don’t think we can usually make). It seems that if we were to reach the state where M stably reported the true answer to the question, then Adv would never get any reward -- but Adv could do better by randomizing what arguments it makes, so that M cannot know which arguments H will be exposed to and so can’t stably predict H’s final answer. See more details in this [thread](https://www.alignmentforum.org/posts/YWwzccGbcHMJMpT45/ai-safety-via-market-making?commentId=43im5ykxC2SPD86Po).

AI safety via market making

If you have an expert, but don’t trust them to give you truthful information, how can you incentivize them to tell you the truth anyway? One [option](https://meteuphoric.com/2014/07/21/how-to-buy-a-truth-from-a-liar/) is to pay them every time they provide evidence that changes your mind, with the hope that only once you believe the truth will there be no evidence that can change your mind. This post proposes a similar scheme for AI alignment.

We train two models, M and Adv. Given a question Q, M is trained to predict what answer to Q the human will give at the end of the procedure. Adv on the other hand is trained to produce arguments that will most make M “change its mind”, i.e. output a substantially different distribution over answers than it previously outputted. M can then make a new prediction. This is repeated T times, and eventually the human is given all T outputs produced by Adv, and provides their final answer (which is used to provide a gradient signal for M). After training, we throw away Adv and simply use M as our question-answering system.

One way to think about this is that M is trained to provide a prediction market on “what the human will answer”, and Adv is trained to manipulate the market by providing new arguments that would change what the human says. So, once you see M providing a stable result, that should mean that the result is robust to any argument that Adv could provide, and so it is what the human would say after seeing all the arguments.

This scheme bears some resemblance to <@debate@>(@AI safety via debate@), and it can benefit from schemes that help debate, most notably <@cross-examination@>(@Writeup: Progress on AI Safety via Debate@). In particular, at every step Adv can cross-examine the previous incarnation of Adv. If the previous incarnation was deceptive, the current incarnation can demonstrate this to the human, which should cause them to disregard the previous argument. We can also add oversight, where an overseer with access to the model ensures that the model does not become non-myopic or deceptive.


AI Safety Debate and Its Applications

This post defines the components of a <@debate@>(@AI safety via debate@) game, lists some of its applications, and defines truth-seeking as the property that we want. Assuming that the agent chooses randomly from the possible Nash equilibria, the truth-promoting likelihood is the probability that the agent picks the actually correct answer. The post then shows the results of experiments on MNIST and Fashion MNIST, seeing comparable results to the original paper.


I like the idea, but the math in the post is quite hard to read (mainly due to the lack of exposition). The post also has separate procedures for amplification, distillation and iteration; I think they can be collapsed into a single more efficient procedure, which I wrote about in this [comment](https://www.alignmentforum.org/posts/jYvm4mmjvGHcPXtGL/a-concrete-proposal-for-adversarial-ida#KdqEJNmRH6vFx5NZg).

A Concrete Proposal for Adversarial IDA

This post presents a method to use an adversary to improve the sample efficiency (with respect to human feedback) of iterated amplification. The key idea is that when a question is decomposed into subquestions, the adversary is used to predict which subquestion the agent will do poorly on, and the human is only asked to resolve that subquestion. In addition to improving sample efficiency by only asking relevant questions, the resulting adversary can also be used for interpretability: for any question-answer pair, the adversary can pick out specific subquestions in the tree that are particularly likely to contain errors, which can then be reviewed.


Can HCH epistemically dominate Ramanujan?

Iterated amplification rests on the hope that we can achieve arbitrarily high capabilities with (potentially very large) trees of explicit verbal breakdowns of problems. This is often formalized as a question about [HCH](https://www.alignmentforum.org/posts/NXqs4nYXaq8q6dTTx/humans-consulting-hch) ([AN #34](https://mailchi.mp/f1947668b183/alignment-newsletter-34)). This post considers the example of Srinivasa Ramanujan, who is "famously known for solving math problems with sudden and inexplicable flashes of insight". It is not clear how HCH would be able to replicate this sort of reasoning.


AI Unsafety via Non-Zero-Sum Debate

This post points out that <@debate@>(@AI safety via debate@) relies crucially on creating a zero-sum game in order to ensure that the debaters point out flaws in each other’s arguments. For example, if you modified debate so that both agents are penalized for an inconclusive debate, then an agent may decide not to point out a flaw in an argument if it believes that it has some chance of confusing the judge.


Problems with Amplification/Distillation


Thoughts on "AI safety via debate"


Reinforcement Learning in the Iterated Amplification Framework

This post and its comments clarify how we can use reinforcement learning for the distillation step in iterated amplification. The discussion is still happening so I don't want to summarize it yet.


Amplification Discussion Notes


My confusions with Paul's Agenda


This is the counterpoint to [Reliability amplification](https://www.alignmentforum.org/posts/6fMvGoyy3kgnonRNM/reliability-amplification) ([AN #44](https://mailchi.mp/6bfac400a0c3/alignment-newsletter-44)) from last week, and the same [confusion](https://www.alignmentforum.org/posts/6fMvGoyy3kgnonRNM/reliability-amplification#9HpJBuc9PBwTPtdQW) I had last week still apply, so I'm going to refrain from an opinion.

Security amplification

If we imagine humans as reasoners over natural language, there are probably some esoteric sentences that could cause "failure". For example, maybe there are unreasonably convincing arguments that cause the human to believe something, when they shouldn't have been convinced by the argument. Maybe they are tricked or threatened in a way that "shouldn't" have happened. The goal with security amplification is to make these sorts of sentences difficult to find, so that we will not come across them in practice. As with [Reliability amplification](https://www.alignmentforum.org/posts/6fMvGoyy3kgnonRNM/reliability-amplification) ([AN #44](https://mailchi.mp/6bfac400a0c3/alignment-newsletter-44)), we are trying to amplify a fast agent A into a slow agent A* that is "more secure", meaning that it is multiplicatively harder to find an input that causes a catastrophic failure.

You might expect that [capability amplification](https://www.alignmentforum.org/posts/t3AJW5jP3sk36aGoC/capability-amplification) ([AN #42](https://mailchi.mp/f6488137d76c/alignment-newsletter-42)) would also improve security, since the more capable agent would be able to notice failure modes and remove them. However, this would likely take far too long.

Instead, we can hope to achieve security amplification by making reasoning abstract and explicit, with the hope that when reasoning is explicit it becomes harder to trigger the underlying failure mode, since you have to get your attack "through" the abstract reasoning. I believe a future post will talk about this more, so I'll leave the details till then. Another option would be for the agent to act stochastically; for example, when it needs to generate a subquestion, it generates many different wordings of the subquestion and chooses one randomly. If only one of the wordings can trigger the failure, then this reduces the failure probability.


It seems like this requires the assumption that our agents have a small probability of failure on any given input. I think this makes sense of we are thinking of reliability of [corrigibility](https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd/p/fkLYhTQteAu5SinAc) ([AN #35](https://mailchi.mp/bbd47ba94e84/alignment-newsletter-35)). That said, I'm pretty confused about what problem this technique is trying to protect against, which I wrote about [in this comment](https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd/p/6fMvGoyy3kgnonRNM#9HpJBuc9PBwTPtdQW).

Reliability amplification

One hope for building an aligned AI system is to alternate [capability amplification](https://www.alignmentforum.org/posts/t3AJW5jP3sk36aGoC/capability-amplification) and [reward engineering](https://www.alignmentforum.org/posts/4nZRzoGTqg8xy5rr8/the-reward-engineering-problem) (both [AN #42](https://mailchi.mp/f6488137d76c/alignment-newsletter-42)) with semi-supervised learning in order to create an agent that becomes more and more capable with more iterations, while remaining aligned. However, our agents will likely always have a small probability of failing, and when we use capability amplification, that failure probability is also amplified (since each of the subagents could fail). If we do enough capability amplifiation, the failure probability could approach 1. This motivates the problem of _reliability amplification_, which aims to take an agent that fails with small probability ε and produce a new agent that fails with significantly lower probability. As with capability amplification, the new agent can take much more time and compute than the original. The hope would be that the combination of capability and reliability amplification would allow us to take an initial agent and make it both more capable and more reliable (even though capability amplification reduces reliability, and reliability amplification will probably reduce capability).

One simple thing to do would be to sample three agents, have them collaboratively settle on an action. If all three agents are aligned, we're fine. The chance of two or three of the agents being malign is small enough that we can ignore it. The interesting case is if one agent is malign and two are benign. If there were only one correct answer, we could simply take a majority vote and get the right answer, and we would have amplified reliability to order ε^2 (which happens when you get two or more malign agents). However, different benign agents could give different answers, since there is no one "correct" benign answer. If we were to choose randomly, as you might expect we get no benefit: we'd have a ~3ε chance of having exactly one malign agent, and a 1/3 chance of failing in that case, keeping the failure probability at ε. The hope for reducing the failure probability is if the two benign agents can "recognize" each other as benign, and so that the majority vote has a >2/3 chance of choosing one of the actions that the benign agents suggested. This does depend on all of the agents having similar capabilities -- if the malign agent is a superintelligence while the two benign agents are human-level, then it seems likely that the malign superintelligence would be able to convince the benign agents to go along with its plan.


I'm not sure how necessary it is to explicitly aim to avoid catastrophic behavior -- it seems that even a low capability [corrigible](https://ai-alignment.com/corrigibility-3039e668638) agent would still know enough to avoid catastrophic behavior in practice. However, based on [Techniques for optimizing worst-case performance](https://www.alignmentforum.org/posts/E2aZ9Xwdz3i2ghPtn/techniques-for-optimizing-worst-case-performance), summarized below, it seems like the motivation is actually to avoid catastrophic failures of corrigibility, as opposed to all catastrophes.

In fact, we can see that we can't avoid all catastrophes without some assumption on either the environment or the oracle. Suppose the environment can do anything computable, and the oracle evaluates behavior only based on outcomes (observations). In this case, for any observation that the oracle would label as catastrophic, there is an environment that regardless of the agent's action outputs that observation, and there is no agent that can always avoid catastrophe. So for this problem to be solvable, we need to either have a limit on what the environment "could do", or an oracle that judges "catastrophe" based on the agent's action in addition to outcomes. That latter option can cache out to "are the actions in this transcript knowably going to cause something bad to happen", which sounds very much like corrigibility.

Learning with catastrophes

In iterated amplification, we need to train a fast agent from a slow one produced by [amplification](https://www.alignmentforum.org/posts/t3AJW5jP3sk36aGoC/capability-amplification) ([AN #42](https://mailchi.mp/f6488137d76c/alignment-newsletter-42)). We need this training to be such that the resulting agent _never_ does anything catastrophic at test time. In iterated amplification, we do have the benefit of having a strong overseer who can give good fedback. This suggests a formalization for catastrophes. Suppose there is some oracle that can take any sequence of observations and actions and label it as catastrophic or not. How do we use this oracle to train an agent that will never produce catastrophic behavior at test time?

Given unlimited compute and unlimited access to the oracle, this problem is easy: simply search over all possible environments and ask the oracle if the agent behaves catastrophically on them. If any such behavior is found, train the agent to not perform that behavior any more. Repeat until all catastrophic behavior is eliminated. This is basically a very strong form of adversarial training.


While I agree that it would be great to optimize worst-case performance, all of these techniques feel quite difficult to do scalably and with guarantees. With adversarial training, you need to find _all_ of the ways that an agent could fail, while catastrophe could happen if the agent stumbles across _any_ of these methods. It seems plausible to me that with sufficient additional information given to the adversary we can meet this standard, but it seems very hard to _knowably_ meet this standard, i.e. to have a strong argument that we will find all of the potential issues.

With verification, the specification problem seems like a deal-breaker, unless combined with other methods: a major point with AI safety is that we can't write down a good specification for what we want. If we instead only use verification to propagate worst-case guarantees from one model to another (as the post suggests), then it seems possible in theory, but very expensive in practice: most verification techniques assume unlimited fast access to the specification, whereas our specification is very expensive to query. Of course, not much research has focused on this setting, so we can expect large gains; nonetheless you do need to somehow extrapolate the specification to all possible inputs, which seems hard to do with limited access to the specification.

Transparency also seems like it provides additional safety, rather than making any guarantees, since we probably can't get a guarantee that our transparency mechanisms can show us all possible failure modes in a way that we understand them. The argument that we can only focus on the training data makes the assumption that the AI system is not going to generalize well outside of the training dataset. While I'm sympathetic to this assumption (we really don't have good methods for generalization, and there are strong reasons to expect generalization to be near-impossible), it isn't one that I'm confident about, especially when we're talking about _general_ intelligence.

Of course, I'm still excited for more research to be done on these topics, since they do seem to cut out some additional failure modes. But if we're looking to have a semi-formal strong argument that we will have good worst-case performance, I don't see the reasons for optimism about that.

Techniques for optimizing worst-case performance

There are “benign” failures of worst-case performance, where the AI system encounters a novel situation and behaves weirdly, but not in a way that systematically disfavors human values. As I noted above, we can’t get rid of all of these, but that’s probably fine. We instead would like to focus on “malign” failures, where the AI system applies its intelligence in pursuit of the wrong goal. There are a few techniques that could be applied to this problem.

With adversarial training, we can have one system find inputs on which our agent fails catastrophically, and then train the agent to avoid those behaviors. The main issue here is that there will likely be some failures that aren't found.

On the other hand, verification has strong guarantees, but suffers from the problem that it is hard to know what to specify, and it is computationally expensive to actually perform verification. If we have a strong trusted overseer, eg. produced by amplification, we could use it as a very expensive specification. Alternatively, we could use a catastrophe-checker in lieu of a specification. _(Note: While I can see some ways of using catastrophe-checkers, the post seems to have a specific method in mind that I don't understand.)_

Any malignant failure must be using the intelligence of the agent somehow, and the agent is only intelligent on the training data, so if we can use strong transparency techniques on the training data, we could find such failures. However, if you use transparency for this, by default you weed out the comprehensible failures and leave in the incomprehensible ones.


As the post notes, these problems are only "easy" in the conceptual sense -- the resulting RL problems could be quite hard. I feel most confused about the third and fourth problems. Choosing a yardstick could work to aggregate reward functions, but I still worry about the issue that this tends to overweight reward functions that assign a low value to the yardstick but high value to other outcomes. With widely-varying rewards, it seems hard to importance sample high-stakes decisions, without knowing what those decisions might be. Maybe if we notice a very large reward, we instead make it lower reward, but oversample it in the future? Something like this could potentially work, but I don't see how yet.

For complex, expensive-to-evaluate rewards, Paul suggests using semi-supervised learning; this would be fine if semi-supervised learning was sufficient, but I worry that there actually isn't enough information in just a few evaluations of the reward function to narrow down on the true reward sufficiently, which means that even conceptually we will need something else.

Thoughts on reward engineering

This post digs into some of the "easy" issues with reward engineering (where we must design a good reward function for an agent, given access to a stronger overseer).

First, in order to handle outcomes over long time horizons, we need to have the reward function capture the overseer's evaluation of the long-term consequences of an action, since it isn't feasible to wait until the outcomes actually happen.

Second, since human judgments are inconsistent and unreliable, we could have the agent choose an action such that there is no other action which the overseer would evaluate as better in a _comparison_ between the two. _(This is not exactly right -- the human's comparisons could be such that this is an impossible standard. The post uses a two-player game formulation that avoids the issue, and gives the guarantee that the agent won't choose something that is unambiguously worse than another option.)_

Third, since the agent will be uncertain about the overseer's reward, it will have the equivalent of normative uncertainty -- how should it trade off between different possible reward functions the overseer could have? One option is to choose a particular yardstick, eg. how much the overseer values a minute of their time, some small amount of money, etc. and normalize all rewards to that yardstick.

Fourth, when there are decisions with very widely-varying scales of rewards, traditional algorithms don't work well. Normally we could focus on the high-stakes decisions and ignore the others, but if the high-stakes decisions occur infrequently then all decisions are about equally important. In this case, we could oversample high-stakes decisions and reduce their rewards (i.e. importance sampling) to use traditional algorithms to learn effectively without changing the overall "meaning" of the reward function. However, very rare+high-stakes decisions will probably require additional techniques.

Fifth, for sparse reward functions where most behavior is equally bad, we need to provide "hints" about what good behavior looks like. Reward shaping is the main current approach, but we do need to make sure that by the end of training we are using the true reward, not the shaped one. Lots of other information such as demonstrations can also be taken as hints that allow you to get higher reward.

Finally, the reward will likely be sufficiently complex that we cannot write it down, and so we'll need to rely on an expensive evaluation by the overseer. We will probably need semi-supervised RL in order to make this sufficiently computationally efficient.


There's a clear reason for work on capability amplification: it could be used as a part of an implementation of iterated amplification. However, this post also suggests another reason for such work -- it may help us determine where the "hard part" of AI safety lies. Does it help to assume that you have lots of time and compute, and that you have access to a weaker policy?

Certainly if you just have access to a weaker policy, this doesn't make the problem any easier. If you could take a weak policy and amplify it into a stronger policy efficiently, then you could just repeatedly apply this policy-improvement operator to some very weak base policy (say, a neural net with random weights) to solve the full problem. (In other variants, you have a much stronger aligned base policy, eg. the human policy with short inputs and over a short time horizon; in that case this assumption is more powerful.) The more interesting assumption is that you have lots of time and compute, which does seem to have a lot of potential. I feel pretty optimistic that a human thinking for a long time could reach "superhuman performance" by our current standards; capability amplification asks if we can do this in a particular structured way.

Capability amplification

Capability amplification is the problem of taking some existing policy and producing a better policy, perhaps using much more time and compute. It is a particularly interesting problem to study because it could be used to define the goals of a powerful AI system, and it could be combined with [reward engineering](https://www.alignmentforum.org/posts/4nZRzoGTqg8xy5rr8/the-reward-engineering-problem) above to create a powerful aligned system. (Capability amplification and reward engineering are analogous to amplification and distillation respectively.) In addition, capability amplification seems simpler than the general problem of "build an AI that does the right thing", because we get to start with a weak policy A rather than nothing, and were allowed to take lots of time and computation to implement the better policy. It would be useful to tell whether the "hard part" of value alignment is in capability amplification, or somewhere else.

We can evaluate capability amplification using the concepts of reachability and obstructions. A policy C is _reachable_ from another policy A if there is some chain of policies from A to C, such that at each step capability amplification takes you from the first policy to something at least as good as the second policy. Ideally, all policies would be reachable from some very simple policy. This is impossible if there exists an _obstruction_, that is a partition of policies into two sets L and H, such that it is impossible to amplify any policy in L to get a policy that is at least as good as some policy in H. Intuitively, an obstruction prevents us from getting to arbitrarily good behavior, and means that all of the policies in H are not reachable from any policy in L.

We can do further work on capability amplification. With theory, we can search for challenging obstructions, and design procedures that overcome them. With experiment, we can study capability amplification with humans (something which [Ought](https://ought.org/) is now doing).


The information asymmetry problem between H and A seems like a major issue. For me, it's the strongest argument for why transparency is a _necessary_ ingredient of a solution to alignment. The argument against imitation learning and IRL is quite strong, in the sense that it seems like you can't rely on either of them to capture the right behavior. These are stronger than the arguments against [ambitious value learning](https://www.alignmentforum.org/s/4dHMdK5TLN6xcqtyc/p/5eX8ko7GCxwR5N9mN) ([AN #31](https://mailchi.mp/7d0e3916e3d9/alignment-newsletter-31)) because here we assume that H is smarter than A, which we could not do with ambitious value learning. So it does seem to me that direct supervision (with semi-supervised techniques and robustness) is the most likely path forward to solving the reward engineering problem.

There is also the question of whether it is necessary to solve the reward engineering problem. It certainly seems necessary in order to implement iterated amplification given current systems (where the distillation step will be implemented with optimization, which means that we need a reward signal), but might not be necessary if we move away from optimization or if we build systems using some technique other than iterated amplification (though even then it seems very useful to have a good reward engineering solution).

The reward engineering problem

The reward engineering problem is the problem of designing some form of reward feedback such that if the agent learns to get high reward, then we are happy with the resulting behavior. We assume that an overseer H is providing reward for an agent A, with H smarter than A.

(Aside: Why are we assuming H is smarter, when we are typically worried about AI that is more intelligent than we are? Because we could hope to use AI tools to create this reward function, so that the human + AI team is smarter than the AI being trained. In iterated amplification, this corresponds to the distillation step.)

We could imagine three approaches: direct supervision (where H provides a reward for every action A takes), imitation learning (where A tries to mimic H, perhaps by trying to fool a discriminator that distinguishes between A and H), and inverse reinforcement learning (where we infer a reward function by observing H's behavior and use that to supervise A).

With all of these approaches, we might hope that since H is smarter than A, A will not be able to trick H with some behavior that gets high reward but is actually undesirable. However, this argument has a major flaw. When A chooses an action, the process or computation by which it chooses that action is important information, which by default (especially with neural nets) only A has access to, and so this can allow it to fool H. For example, if A is tasked with drawing original artwork, it might copy and slightly change existing artwork, which H may not realize if it cannot see how the artwork was made. We could hope to fix this problem with transparency or explanations, but this requires a lot more research.

Imitation learning and IRL have the problem that A may not be capable of doing what H does. In that case, it will be off-distribution and may have weird behavior. Direct supervision doesn't suffer from this problem, but it is very time-inefficient. This could potentially be fixed using semi-supervised learning techniques.


AlphaGo Zero and capability amplification

AlphaGo Zero works by starting with a randomly chosen policy and value network. Then, it repeatedly applies a "policy improvement" step: it runs MCTS using the policy and value networks to guide the search, which results in moves that are better than the policy and value networks used alone, and then trains the policy and value networks on those moves. Iterated amplification is very similar: it starts with a base policy, and then repeatedly applies a "policy improvement" step that consists of first amplifying the policy and then distilling it. MCTS is analogous to amplification, and training on the resulting moves is analogous to distillation.

So, if a general method of building capable agents is to constantly apply "policy improvement" steps, the key challenge is to figure out a "policy improvement" operator that preserves alignment. (We could hopefully get an aligned weak initial agent by eg. imitating a human's cognitive procedure.) In iterated amplification, we are hoping that "think longer" (as formalized by increasingly large bounded versions of [HCH](https://www.alignmentforum.org/posts/NXqs4nYXaq8q6dTTx/humans-consulting-hch) ([AN #34](https://mailchi.mp/f1947668b183/alignment-newsletter-34))) would be an alignment-preserving policy improvement operator.


The three directions of reliability/robustness, reward learning, and amplification seem great. Robustness seems particularly hard to achieve. While there is current work on adversarial training, interpretability and verification, even if all of the problems that researchers currently work on were magically solved, I don't have a story for how that leads to robustness of (say) an agent trained by iterated amplification. Normally, I'd also talk about how amplification seems to require some sort of universality assumption, but now I can just refer you to [ascription universality](https://www.alignmentforum.org/posts/M8WdeNWacMrmorNdd/towards-formalizing-universality) (described in this newsletter).

I am more conflicted about the desiderata. They seem very difficult to satisfy, and they don't seem strictly necessary to achieve good outcomes. The underlying view here is that we should aim for something that we know is _sufficient_ to achieve good outcomes, and only weaken our requirements if we find a fundamental obstacle. My main issue with this view is that even if it is true that the requirements are impossible to satisfy, it seems very hard to _know_ this, and so we may spend a lot of time trying to satisfy these requirements and most of that work ends up being useless. I can imagine that we try to figure out ways to achieve robustness for several years in order to get a secure AI system, and it turns out that this is impossible to do in a way where we know it is robust, but in practice any AI system that we train will be sufficiently robust that it never fails catastrophically. In this world, we keep trying to achieve robustness, never find a fundamental obstruction, but also never succeed at creating a secure AI system.

Another way of phrasing this is that I am pessimistic about the prospects of conceptual thinking, which seems to be the main way by which we could find a fundamental obstruction. (Theory and empirical experiments can build intuitions about what is and isn't hard, but given the complexities of the real world it seems unlikely that either would give us the sort of crystallized knowledge that Paul is aiming for.) Phrased this way, I put less credence in this opinion, because I think there are a few examples of conceptual thinking being very important, though not that many.

Directions and desiderata for AI alignment

What further research needs to be done to get iterated amplification to work? There are three main areas: robustness, reward learning, and amplification.

With reliability and robustness, we want our AI systems to never fail catastrophically, whereas the simple setup of iterated amplification only optimizes performance in the average case. This is discussed more in [Techniques for Optimizing Worst-Case Performance](https://ai-alignment.com/techniques-for-optimizing-worst-case-performance-39eafec74b99), which talks about adversarial training, interpretability, and verification.

Reward learning, as I understand it here, is the same thing as [narrow value learning](https://www.alignmentforum.org/posts/SvuLhtREMy8wRBzpC/ambitious-vs-narrow-value-learning), described in this newsletter. We've talked a lot about the challenges of reward learning, so I'll leave that part out.

Finally, deliberation and amplification is about how to train an ML system to exceed human performance. At present, the only way we have of doing this is to use reinforcement learning to optimize some simple objective. However, given that "human values" is not a simple objective, we need to figure out another way that allows us to get superhuman performance on the tasks that we actually care about. You could do this with ambitious value learning, but not with narrow value learning. Alternatively, you could try to replicate human cognition and run it longer, either by using iterated amplification, or by using inverse reinforcement learning on cognition.

Then, the post goes on to talk about desiderata for any solution to AI alignment, proposing that the solution should be secure, competitive and scalable.

A secure AI system is one that works even when the environment is behaving adversarially. This is not possible in full generality because of no-free-lunch theorems, but we could at least hope that our AI system is never actively "trying" to hurt us. This is desirable because it holds us to a relatively clear standard that we can evaluate currently, rather than trying to predict how the future will look (a notoriously hard task) and evaluating whether our AI system will work in that future. In addition, the typical iterative method of trying things out and seeing how they do may not work in the future, because technological progress may accelerate due to recursive improvement. If we instead focus on arguments and analysis that suggest that our AI systems are secure, we would not need the iterative method. Finally, there could in fact be adversaries (that themselves might be AI systems) that we need to deal with.

An AI system is competitive if it is almost as efficient and capable as the best possible AI system we could build at the same time (which may be unaligned). This is necessary because without it it becomes very likely that someone will build the unaligned version for the efficiency gains; we would need global coordination to avoid this which seems quite difficult. In addition, it is relatively easy to tell when a system is competitive, whereas if an AI system is uncompetitive, it is hard to tell how uncompetitive it is. Once again, with an uncompetitive AI system, we need to predict the future in order to tell whether or not it will be okay to rely on that system.

A scalable AI system is one that remains aligned as the underlying technologies become more and more powerful. Without scalability, we would need to predict how far a particular scheme will take us (which seems hard), and we need to continually invest in alignment research in order to "keep up" with improving capabilities, which might be particularly hard to do in the face of accelerating progress due to recursive improvement.

These are very demanding desiderata. However, while it may seem impossible to get a secure, competitive, scalable solution to AI alignment, we don't know why yet, and impossibility claims are notoriously hard to get right. In any case, even if it is impossible, it would be worthwhile to clarify exactly why these desiderata are impossible to meet, in order to figure out how to deal with the resulting problems.


This is my new favorite post/paper for explaining how iterated amplification works, since it very succinctly and clearly makes the case for iterated amplification as a strategy for generating a good training signal. I'd recommend reading the [paper](https://arxiv.org/abs/1810.08575) in full, as it makes other important points that I haven't included in the summary.

Note that it does not explain a lot of Paul's thinking. It explains one particular training method that allows you to train an AI system with a more intelligent and informed overseer.

Supervising strong learners by amplifying weak experts

This was previously covered in [AN #30](https://mailchi.mp/c1f376f3a12e/alignment-newsletter-30), I've copied the summary and opinion. This paper introduces iterated amplification, focusing on how it can be used to define a training signal for tasks that humans cannot perform or evaluate, such as designing a transit system. The key insight is that humans are capable of decomposing even very difficult tasks into slightly simpler tasks. So, in theory, we could provide ground truth labels for an arbitrarily difficult task by a huge tree of humans, each decomposing their own subquestion and handing off new subquestions to other humans, until questions are easy enough that a human can directly answer them.

We can turn this into an efficient algorithm by having the human decompose the question only once, and using the current AI system to answer the generated subquestions. If the AI isn't able to answer the subquestions, then the human will get nonsense answers. However, as long as there are questions that the human + AI system can answer but the AI alone cannot answer, the AI can learn from the answers to those questions. To reduce the reliance on human data, another model is trained to predict the decomposition that the human performs. In addition, some tasks could refer to a large context (eg. evaluating safety for a specific rocket design), so they model the human as being able to access small pieces of the context at a time.

They evaluate on simple algorithmic tasks like distance between nodes in a graph, where they can program an automated human decomposition for faster experiments, and there is a ground truth solution. They compare against supervised learning, which trains a model on the ground truth answers to questions (which iterated amplification does not have access to), and find that they can match the performance of supervised learning with only slightly more training steps.


It seems a lot easier to generalize well to novel situations if you've learned the right decision-making algorithms. Since it's way easier to learn something if you have direct supervision of it, internal supervision seems particularly interesting as a method of generalizing well. For example, in [Making Neural Programming Architectures Generalize via Recursion](https://arxiv.org/abs/1704.06611), the learned programs generalize perfectly, because they learn from execution traces, which show _how_ to solve a task, and the learned algorithm is forced to use a recursive structure. (Typical approaches to neural program induction only use input-output examples, and often fail to generalize to inputs longer than those seen during training.)

I also like the point that iterated amplification allows you to mimic an implicit exponential-time computation. This is a key reason for my optimism: it seems like well-honed human intuition beats short explicit reasoning almost always. In fact, I think it is reasonable to view human intuition as the result of iterated amplification (see [these](https://www.lesswrong.com/posts/XHMCvvhb7zTZcQAgA/argument-intuition-and-recursion) [posts](https://terrytao.wordpress.com/career-advice/theres-more-to-mathematics-than-rigour-and-proofs/)).

Factored Cognition

This was previously summarized in [AN #12](https://mailchi.mp/bcb2c6f1d507/alignment-newsletter-12). While this post describes a project from Ought, it explains many of the ideas behind iterated amplification (which is why it is in this sequence). There are two important and distinct topics addressed in this post: first, whether it is possible in principle to achieve good performance on tasks when you must use _explicit reasoning_, and second, how to use explicit reasoning to train an aligned AI system via iterated amplification.

Currently, most AI research is supervised based on _behavior_: we specify a domain and data or reward functions that identify good behavior, and hope that the AI develops a good decision-making system during training. We don't provide training data for decision-making itself. In contrast, Factored Cognition aims to make the decision-making algorithm itself explicit, in a form that could be used to train an AI system, which we might call internal supervision. Note that this is _not_ a required feature of iterated amplification, which can work with external (i.e. behavioral) supervision, as in [recursive reward modeling](https://medium.com/@deepmindsafetyresearch/scalable-agent-alignment-via-reward-modeling-bf4ab06dfd84) ([AN #34](https://mailchi.mp/f1947668b183/alignment-newsletter-34)). However, we might want to use internal supervision because then we get to control the decision-making process itself, and that is what determines whether the AI system is aligned or not.

Research currently does not use internal supervision because it requires training data on _how_ to solve the problem, whereas a reward function only needs to tell _whether_ the problem is solved. However, for sufficiently general AI systems, we could provide training data on general problem-solving, which could then be applied to many tasks. This could be competitive with the alternative of providing training data or reward functions for each task separately.

How might we make general problem-solving explicit enough that we could train AI systems to replicate it? Factored Cognition hypothesizes that we could do this by providing _decompositions_ of tasks into simpler subtasks. While this is not how humans solve tasks (for example, expert Go players use a lot of intuition), it seems plausible that we could get similar or better performance using decomposition as long as we were willing to wait a long time (for example, by implementing minimax using decomposition to solve Go). Ought wants to test this hypothesis, by studying whether humans are in fact capable of providing decompositions in challenging problem domains, such as math and programming puzzles, textbook problems, fact-checking, interactive dialog, and task prioritization. (There will likely be no ML here, just humans decomposing problems for other humans to then solve.)

The post makes this concrete by considering a particular way in which humans might provide decompositions. They develop a "cognitive workspace" where a human can take "actions" such as editing the text in the workspace, each of which can only involve a small amount of cognitive work. There are a lot of details that I won't get into, but broadly it seems to me like what you would get if you tried to create a functional programming language where the primitive forms are things that a human can do. Note that the examples in the post use task-specific decompositions for clarity but in the long term we would need general decomposition strategies.

We could train an AI system to mimic this explicit reasoning. It seems like this would achieve human performance but no more, since after all we are imitating human decision-making algorithms. However, now we can use the iterated amplification trick. If a human got to use explicit reasoning over a long time, we would expect significantly better decisions. So, we can first distill an agent A that mimics a human using explicit reasoning over a short time period, then amplify it to get H[A], where a human performs a decomposition into subquestions that are answered by A, then distill that into A', etc. The post has a nice visualization of how this allows you to mimic an implicit exponentially large tree of decompositions in polynomial time. As before, distillation could be done using imitation learning, or RL with some good inferred reward.


Benign model-free RL

This post is very similar to the previous one, just with different language: distillation is now implemented through reward modeling with robustness. The point of robustness is to ensure that the distilled agent is benign even outside of the training distribution (though it can be incompetent). There's also an analysis of the costs of the scheme. One important note is that this approach only works for model-free RL systems -- we'll need something else for eg. model-based RL, if it enables capabilities that we can't get with model-free RL.


I really like this formulation of corrigibility, which I find quite different from [MIRI's paper](https://intelligence.org/2014/10/18/new-report-corrigibility/). This seems a lot more in line with the kind of reasoning that I want from an AI system, and it seems like iterated amplification or something like it could plausibly succeed at achieving this sort of corrigible behavior.

Corrigibility

A corrigible agent is one which helps its operator, even with tasks that would change the agent itself, such as correcting mistakes in AI design. Consider a good act-based agent, which chooses actions according to our preferences over that action. Since we have a short-term preference for corrigibility, the act-based agent should be corrigible. For example, if we are trying to turn off the agent, the agent will turn off because that's what we would prefer -- it is easy to infer that the overseer would not prefer that agents stop the overseer from shutting them down. Typically we only believe that the agent would stop us from shutting it down if it makes _long-term plans_, in which case being operational is instrumentally useful, but with act-based agents the agent only optimizes for its overseer's short term preferences. One potential objection is that the notion of corrigibility is not easy to learn, but it seems not that hard to answer the question "Is the operator being misled", and in any case we can try this with simple systems, and the results should _improve_ with more capable systems, since as you get smarter you are more capable of predicting the overseer.

In addition, even if an agent has a slightly wrong notion of the overseer's values, it seems like it will _improve_ over time. It is not hard to infer that the overseer wants the agent to make its approximation of the overseer's values more accurate. So, as long as the agent has enough of the overseer's preferences to be corrigible, it will try to learn about the preferences it is wrong about and will become more and more aligned over time. In addition, any slight value drifts caused by eg. amplification will tend to be fixed over time, at least on average.


This is a great simple explanation of the scheme. I don't have much to say about the idea since I've talked about iterated amplification so much in this newsletter already.

Iterated Distillation and Amplification

This is the first in a series of four posts describing the iterated amplification framework in different ways. This post focuses on the repetition of two steps. In amplification, we take a fast aligned agent and turn it into a slow but more capable aligned agent, by allowing a human to coordinate many copies of the fast agent in order to make better decisions. In distillation, we take a slow aligned agent and turn it a fast aligned agent (perhaps by training a neural net to imitate the judgments of the slow agent). This is similar to AlphaGoZero, in which MCTS can be thought of as amplification, while distillation consists of updating the neural net to predict the outputs of the MCTS.

This allows us to get both alignment and powerful capabilities, whereas usually the two trade off against each other. High capabilities implies a sufficiently broad mandate to search for good behaviors, allowing our AI systems to find novel behaviors that we never would have thought of, which could be bad if the objective was slightly wrong. On the other hand, high alignment typically requires staying within the realm of human behavior, as in imitation learning, which prevents the AI from finding novel solutions.

In addition to distillation and amplification robustly preserving alignment, we also need to ensure that given a human as a starting point, iterated distillation and amplification can scale to arbitrary capabilities. We would also want it be about as cost-efficient as alternatives. This seems to be true at test time, when we are simply executing a learned model, but it could be that training is much more expensive.


Goal-directed behavior requires an extremely intelligent overseer in order to ensure that the agent is pointed at the correct goal (as opposed to one the overseer thinks is correct but is actually slightly wrong). I think of approval-directed agents as providing the intuition that we may only require an overseer that is slightly smarter than the agent in order to be aligned. This is because the overseer can simply "tell" the agent what actions to take, and if the agent makes a mistake, or tries to optimize a heuristic too hard, the overseer can notice and correct it interactively. (This is assuming that we solve the [informed oversight problem](https://ai-alignment.com/the-informed-oversight-problem-1b51b4f66b35) so that the agent doesn't have information that is hidden from the overseer, so "intelligence" is the main thing that matters.) Only needing a slightly smarter overseer opens up a new space of solutions where we start with a human overseer and subhuman AI system, and scale both the overseer and the AI at the same time while preserving alignment at each step.

[Approval-directed agents: overview](https://www.alignmentforum.org/posts/7Hr8t6xwuuxBTqADK/approval-directed-agents-overview) and [Approval-directed agents: details](https://www.alignmentforum.org/posts/njRjnmaBnZquqPfBa/approval-directed-agents-details)

These two posts introduce the idea of approval-directed agents, which are agents that choose actions that they believe their operator Hugh the human would most approve of, if he reflected on it for a long time. This is in contrast to the traditional approach of goal-directed behavior, which are defined by the _outcomes_ of the action.

Since the agent Arthur is no longer reasoning about how to achieve outcomes, it can no longer outperform Hugh at any given task. (If you take the move in chess that Hugh most approves of, you probably still lose to Gary Kasparov.) This is still better than Hugh performing every action himself, because Hugh can provide an expensive learning signal which is then distilled into a fast policy that Arthur executes. For example, Hugh could deliberate for a long time whenever he is asked to evaluate an action, or he could evaluate very low-level decisions that Arthur makes billions of times. We can also still achieve superhuman performance by bootstrapping (see the next summary).

The main advantage of approval-directed agents is that we avoid locking in a particular goal, decision theory, prior, etc. Arthur should be able to change any of these, as long as Hugh approves it. In essence, approval-direction allows us to delegate these hard decisions to future overseers, who will be more informed and better able to make these decisions. In addition, any misspecifications seem to cause graceful failures -- you end up with a system that is not very good at doing what Hugh wants, rather than one that works at cross purposes to him.

We might worry that _internally_ Arthur still uses goal-directed behavior in order to choose actions, and this internal goal-directed part of Arthur might become unaligned. However, we could even have internal decision-making about cognition be approval-based. Of course, eventually we reach a point where decisions are simply made -- Arthur doesn't "choose" to execute the next line of code. These sorts of things can be thought of as heuristics that have led to choosing good actions in the past, that could be changed if necessary (eg. by rewriting the code).

How might we write code that defines approval? If our agents can understand natural language, we could try defining "approval" in natural language. If they are able to reason about formally specified models, then we could try to define a process of deliberation with a simulated human. Even in the case where Arthur learns from examples, if we train Arthur to predict approval from observations and take the action with the highest approval, it seems possible that Arthur would not manipulate approval judgments (unlike AIXI).

There are also important details on how Hugh should rate -- in particular, we have to be careful to distinguish between Hugh's beliefs/information and Arthur's. For example, if Arthur thinks there's a 1% chance of a bridge collapsing if we drive over it, then Arthur shouldn't drive over it. However, if Hugh always assigns approval 1 to the optimal action and approval 0 to all other actions, and Arthur believes that Hugh knows whether the bridge will collapse, then the maximum expected approval action is to drive over the bridge.

The main issues with approval-directed agents is that it's not clear how to define them (especially from examples), whether they can be as useful as goal-directed agents, and whether approval-directed agents will have internal goal-seeking behavior that brings with it all of the problems that approval was meant to solve. It may also be a problem if some other Hugh-level intelligence gets control of the data that defines approval.


This complements the idea of approval from the previous posts nicely: while approval tells us how to build an aligned agent from a slightly smarter overseer, bootstrapping tells us how to improve the capabilities of the overseer and the agent.

Approval-directed bootstrapping

To get a very smart overseer, we can use the idea of bootstrapping. Given a weak agent, we can define a stronger agent that happens from letting the weak agent think for a long time. This strong agent can be used to oversee a slightly weaker agent that is still stronger than the original weak agent. Iterating this process allows us to reach very intelligent agents. In approval-directed agents, we can simply have Arthur ask Hugh to evaluate approval for actions, and _in the process of evaluation_ Hugh can consult Arthur. Here, the weak agent Hugh is being amplified into a stronger agent by giving him the ability to consult Arthur -- and this becomes stronger over time as Arthur becomes more capable.


This has the same nice recursive structure of bootstrapping, but without the presence of the agent. This probably makes it more amenable to formal analysis, but I think that the interactive nature of bootstrapping (and iterated amplification more generally) is quite important for ensuring good outcomes: it seems way easier to control an AI system if you can provide input constantly with feedback.

Humans Consulting HCH

Suppose we unroll the recursion in the previous bootstrapping post: in that case, we see that Hugh's evaluation of an answer can depend on a question that he asked Arthur whose answer depends on how Hugh evaluated an answer that depended on a question that he asked Arthur etc. Inspired by this structure, we can define HCH (humans consulting HCH) to be a process that answers question Q by perfectly imitating how Hugh would answer question Q, _if Hugh had access to the question-answering process_. This means Hugh is able to consult a copy of Hugh, who is able to consult a copy of Hugh, who is able to consult a copy of Hugh, ad infinitum. This is one proposal for how to formally define a human's enlightened judgment.

You could also combine this with particular ML algorithms in an attempt to define versions of those algorithms aligned with Hugh's enlightened judgment. For example, for RL algorithm A, we could define max-HCH_A to be A's chosen action when maximizing Hugh's approval after consulting max-HCH_A.


I don't really have much to say here, except that I agree with this post quite strongly.

Prosaic AI alignment

It is plausible that we can build "prosaic" AGI soon, that is, we are able to build generally intelligent systems that can outcompete humans without qualitatively new ideas about intelligence. It seems likely that this would use some variant of RL to train a neural net architecture (other approaches don't have a clear way to scale beyond human level). We could write the code for such an approach right now (see [An unaligned benchmark](https://www.alignmentforum.org/posts/ZHXutm7KpoWEj9G2s/an-unaligned-benchmark) from [AN #33](https://mailchi.mp/b6dc636f6a1b/alignment-newsletter-33)), and it's at least plausible that with enough compute and tuning this could lead to AGI. However, this is likely to be bad if implemented as stated due to the standard issues of reward gaming and Goodhart's Law. We do have some approaches to alignment such as IRL and executing natural language instructions, but neither of these are at the point where we can write down code that would plausibly lead to an aligned AI. This suggests that we should focus on figuring out how to align prosaic AI.

There are several reasons to focus on prosaic AI. First, since we know the general shape of the AI system under consideration, it is easier to think about how to align it (while ignoring details like architecture, variance reduction tricks, etc. which don't seem very relevant currently). Second, it's important, both because we may actually build prosaic AGI, and because even if we don't the insights gained will likely transfer. In addition, worlds with short AGI timelines are higher leverage, and in those worlds prosaic AI seems much more likely. The main counterargument is that aligning prosaic AGI is probably infeasible, since we need a deep understanding of intelligence to build aligned AI. However, it seems unreasonable to be confident in this, and even if it is infeasible, it is worth getting strong evidence of this fact in order change priorities around AI development, and coordinate on not building an AGI that is too powerful.


I often think that the hard part of AI alignment is actually the _strategic_ side of it -- even if we figure out how to build an aligned AI system, it doesn't help us unless the actors who actually build powerful AI systems use our proposal. From that perspective, it's very important for any aligned systems we build to be competitive with unaligned ones, and so keeping these sorts of benchmarks in mind seems like a really good idea. This particular benchmark seems good -- it's essentially the AlphaGo algorithm, except with learned dynamics (since we don't know the dynamics of the real world) and rewards (since we want to be able to specify arbitrary tasks), which seems like a good contender for "powerful AI system".

An unaligned benchmark

I previously summarized this in [Recon #5](https://mailchi.mp/33af21f908b5/reconnaissance-5), but I'll consider it in more detail now. The post argues that we could get a very powerful AI system using model-based RL with MCTS. Specifically, we learn a generative model of dynamics (sample a sequence of observations given actions), a reward model, and a policy. The policy is trained using MCTS, which uses the dynamics model and reward model to create and score rollouts. The dynamics model is trained using the actual observations and actions from the environment. The reward is trained using preferences or rankings (think something like [Deep RL from Human Preferences](https://arxiv.org/abs/1706.03741)). This is a system we could program now, and with sufficiently powerful neural nets, it could outperform humans.

However, this system would not be aligned. There could be specification failures: the AI system would be optimizing for making humans think that good outcomes are happening, which may or may not happen by actually having good outcomes. (There are a few arguments suggesting that this is likely to happen.) There could also be robustness failures: as the AI exerts more control over the environment, there is a distributional shift. This may lead to the MCTS finding previously unexplored states where the reward model accidentally assigns high reward, even though it would be a bad outcome, causing a failure. This may push the environment even more out of distribution, triggering other AI systems to fail as well.

Paul uses this and other potential AI algorithms as _benchmarks_ to beat -- we need to build aligned AI algorithms that achieve similar results as these benchmarks. The further we are from hitting the same metrics, the larger the incentive to use the unaligned AI algorithm.

Iterated amplification could potentially solve the issues with this algorithm. The key idea is to always be able to cash out the learned dynamics and reward models as the result of (a large number of) human decisions. In addition, the models need to be made robust to worst case inputs, possibly by using [these techniques](https://ai-alignment.com/techniques-for-optimizing-worst-case-performance-39eafec74b99). In order to make this work, we need to make progress on robustness, amplification, and an understanding of what bad behavior is (so that we can argue that it is easy to avoid, and iterated amplification does avoid it).


Ultimately, our goal is to build AI systems that reliably do what we want them to do. One way of decomposing this is first to _define_ the behavior that we want from an AI system, and then to figure out how to obtain that behavior, which we might call the definition-optimization decomposition. [Ambitious value learning](https://www.alignmentforum.org/posts/5eX8ko7GCxwR5N9mN/what-is-ambitious-value-learning) aims to solve the definition subproblem. I interpret this post as proposing a _different decomposition_ of the overall problem. One subproblem is how to build an AI system that is _trying_ to do what we want, and the second subproblem is how to make the AI competent enough that it _actually_ does what we want. I like this motivation-competence decomposition for a few reasons, which I've written a [long comment](https://www.alignmentforum.org/posts/ZeE7EKHTFMBs8eMxn/clarifying-ai-alignment#3ECKoYzFNW2ZqS6km) about that I strongly encourage you to read. The summary of that comment is: motivation-competence isolates the urgent part in a single subproblem (motivation), humans are an existence proof that the motivation subproblem can be solved, it is possible to apply the motivation framework to systems without lower capabilities, the safety guarantees degrade slowly and smoothly, the definition-optimization decomposition as exemplified by expected utility maximizers has generated primarily negative results, and motivation-competence allows for interaction between the AI system and humans. The major con is that the motivation-competence decomposition is informal, imprecise, and may be intractable to work on.

Clarifying "AI Alignment"

I previously summarized this in [AN #2](https://mailchi.mp/14782876a85d/alignment-newsletter-2), but I'll consider it in more detail now. As Paul uses the term, "AI alignment" refers only to the problem of figuring out how to build an AI that is _trying_ to do what humans want. In particular, an AI can be aligned but still make mistakes because of incompetence. This is not a formal definition, since we don't have a good way of talking about the "motivation" of an AI system, or about "what humans want", but Paul expects that it will correspond to some precise notion after we make more progress.


This is an interesting perspective on the AI safety problem. I really like the ethos of this post, where there isn't a huge opposition between AI capabilities and AI safety, but instead we are simply trying to figure out how to use the (helpful!) capabilities developed by AI researchers to do useful things.

If I think about this from the perspective of reducing existential risk, it seems like you also need to make the argument that AI systems are unlikely to pose an existential threat before they are human-level (a claim I mostly agree with), or that the solutions will generalize to sub-human-level AI systems.

The Steering Problem

The steering problem refers to the problem of writing a program that uses black-box human-level cognitive abilities to be as useful as a well-motivated human Hugh (that is, a human who is "trying" to be helpful). This is a conceptual problem -- we don't have black-box access to human-level cognitive abilities yet. However, we can build suitable formalizations and solve the steering problem within those formalizations, from which we can learn generalizable insights that we can apply to the problem we will actually face once we have strong AI capabilities. For example, we could formalize "human-level cognitive abilities" as Hugh-level performance on question-answering (yes-no questions in natural language), online learning (given a sequence of labeled data points, predict the label of the next data point), or embodied reinforcement learning. A program P is more useful than Hugh for X if, for every project using a simulation of Hugh to accomplish X, we can efficiently transform it into a new project which uses P to accomplish X.


Preface to the sequence on iterated amplification

This is a preface, read it if you're going to read the full posts, but not if you're only going to read these summaries.


I like this paper; it's a very clear explanation of a "recipe" used to develop preference learning algorithms (especially at CHAI and Interact, two of the labs I'm a part of). It is particularly applicable to the case where there's a separate training phase where the human gives feedback on the reward function, and a deployment phase where the agent optimizes the reward function. Things get murkier once you move to a more online setting in which the human and agent are acting simultaneously, as in <@assistance games / CIRL games@>(@Cooperative Inverse Reinforcement Learning@), where the agent can learn from pragmatic actions (see also <@the deep RL version@>(@Learning to Interactively Learn and Assist@)). In particular, while previously we could separate out the grounding of choices to trajectories, and the interpretation of those trajectories (the Boltzmann rational model), this can no longer be done in an assistance game, since the way that the agent interprets the trajectories changes what the agent does which changes the choices available to the human in the next timestep. I'd be excited for more work on understanding this setting.

Reward-rational (implicit) choice: A unifying formalism for reward learning

Hong Jun Jeon*, Smitha Milli*, Anca D. Dragan

We've got algorithms for learning preferences from <@demonstrations@>(@Modeling Interaction via the Principle of Maximum Causal Entropy@) (possibly <@ranked@>(@Ranking-Based Reward Extrapolation without Rankings@)), <@comparisons@>(@Fine-Tuning GPT-2 from Human Preferences@), <@proxy rewards@>(@Inverse Reward Design@), and even the <@observed state@>(@Learning Preferences by Looking at the World@). The insight of this paper is that these are all instances of a simple underlying formalism.

Specifically, these forms of preference learning can be described by two properties: (1) the set of choices that the human picks from and (2) how each choice corresponds to a distribution over agent trajectories. Given these properties, we assume that the human makes their choice according to a Boltzmann-rational model (where the human is more likely to choose an option if it leads to higher expected reward). We have now specified a likelihood over the choice given the reward, and we can use Bayes rule to infer a distribution over the reward given the human's choice.

Consider more exotic types of feedback, such as the human's decision to <@turn the agent off@>(@The Off-Switch Game@). Here, the human has two options: turning the agent off (corresponding to the agent staying still forever), or letting it continue (corresponding to the agent taking the trajectory that maximizes its current expected reward). If the agent has the right reward function, then the Boltzmann rational human would let it continue; as a result, if the human instead tries to turn the agent off, Bayes Rule allows the agent to infer that its belief about the reward must be wrong. Thus, even this decision of whether to turn the agent off can be captured in this framework.

The paper then shows two examples of new feedback types that can be generated from this framework: first, credit assignment, in which the human identifies a subset of the trajectory that had maximal reward, and second, meta-choice, where the choice of which _type_ of feedback to give can itself give information about the reward function.