The full RFP has many, many more points about these topics; it’s 8 pages of remarkably information-dense yet readable prose. If you’re at all interested in mechanistic interpretability, I recommend reading it in full.

This RFP also has the benefit of having the most obvious pathway to impact: if we understand what algorithm neural networks are running, there’s a much better chance that we can catch any problems that arise, especially ones in which the neural network is deliberately optimizing against us. It’s one of the few areas where nearly everyone agrees that further progress is especially valuable.

RFP: Interpretability

The author provides this one sentence summary: _We would like to see research building towards the ability to “reverse engineer" trained neural networks into human-understandable algorithms, enabling auditors to catch unanticipated safety problems in these models._

This RFP is primarily focused on an aspirational “intermediate” goal: to fully reverse engineer some modern neural network, such as an ImageNet classifier. (Despite the ambition, it is only an “intermediate” goal because what we would eventually need is a general method for _cheaply_ reverse engineering _any_ neural network.) The proposed areas of research are primarily inspired by the <@Circuits line of work@>(@Circuits Thread@):

1. **Discovering Features and Circuits:** This is the most obvious approach to the aspirational goal. We simply “turn the crank” using existing tools to study new features and circuits, and this fairly often yields an interesting result that makes progress towards reverse engineering a neural network.
2. **Scaling Circuits to Larger Models:** So far the largest example of reverse engineering is [curve circuits](https://distill.pub/2020/circuits/curve-circuits/), with 50K parameters. Can we find examples of structure in the neural networks that allow us to drastically reduce the amount of effort required per parameter? (As examples, see [equivariance](https://distill.pub/2020/circuits/equivariance/) and [branch specialization](https://distill.pub/2020/circuits/branch-specialization/).)
3. **Resolving Polysemanticity:** One of the core building blocks of the circuits approach is to identify a neuron with a concept, so that connections between neurons can be analyzed as connections between concepts. Unfortunately, some neurons are _polysemantic_, that is, they encode multiple different concepts. This greatly complicates analysis of the connections and circuits between these neurons. How can we deal with this potential obstacle?


While I agree that making AI systems truthful would then enable many alignment strategies, I’m actually more interested in the _methods_ by which we make AI systems truthful. Many of the ideas suggested in the RFP are ones that would apply to alignment more generally and aren’t particularly specific to truthful AI. So it seems like whatever techniques we used to build truthful AI could then be repurposed for alignment. In other words, I expect that the benefit to AI alignment of working on truthful AI is that it serves as a good test case for methods that aim to impose constraints upon an AI system. In this sense, it is a more challenging, larger version of the <@”never describe someone getting injured” challenge@>(@Redwood Research’s current project@). Note that I am only talking about how this helps AI alignment; there are also beneficial effects on society from pursuing truthful AI that I haven’t talked about here.

RFP: Truthful and honest AI

This RFP outlines research projects on [Truthful AI](https://arxiv.org/abs/2110.06674) (summarized below). They fall under three main categories:

1. Increasing clarity about “truthfulness” and “honesty”. While there are some tentative definitions of these concepts, there is still more precision to be had: for example, how do we deal with statements with ambiguous meanings, or ones involving figurative language? What is the appropriate standard for _robustly_ truthful AI? It seems too strong to require the AI system to never generate a false statement; for example it might misunderstand the meaning of a newly coined piece of jargon.
2. Creating benchmarks and tasks for Truthful AI, such as <@TruthfulQA@>(@TruthfulQA: Measuring How Models Mimic Human Falsehoods@), which checks for imitative falsehoods. This is not just meant to create a metric to improve on; it may also simply perform as a measurement. For example, we could <@experimentally evaluate whether honesty generalizes@>(@Experimentally evaluating whether honesty generalizes@), or explore how much truthfulness is reduced when adding in a task-specific objective.
3. Improving the truthfulness of models, for example by finetuning models on curated datasets of truthful utterances, finetuning on human feedback, using <@debate@>(@AI safety via debate@), etc.

Besides the societal benefits from truthful AI, building truthful AI systems can also help with AI alignment:
1. A truthful AI system can be used to supervise its own actions, by asking it whether its selected action was good.
2. A robustly truthful AI system could continue to do this after deployment, allowing for ongoing monitoring of the AI system.
3. Similarly, we could have a robustly truthful AI system supervise its own actions in hypothetical scenarios, to make it more robustly aligned.


My coauthors and I wrote this paper with the ML research community as our target audience. Here are some thoughts on this topic:

1. The document includes numerous problems that, if left unsolved, would imply that ML systems are unsafe. We need the effort of thousands of researchers to address all of them. This means that the main safety discussions cannot stay within the confines of the relatively small EA community. I think we should aim to have over one third of the ML research community work on safety problems. We need the broader community to treat AI safety at least as seriously as safety for nuclear power plants.

2. To grow the ML safety research community, we need to suggest problems that can progressively build the community and organically grow support for elevating safety standards within the existing research ecosystem. Research agendas that pertain to AGI exclusively will not scale sufficiently, and such research will simply not get enough market share in time. If we do not get the machine learning community on board with proactively mitigating risks that already exist, we will have a harder time getting them to mitigate less familiar and unprecedented risks. Rather than try to win over the community with alignment philosophy arguments, I'll try winning them over with interesting problems and try to make work towards safer systems rewarded with prestige.

3. The benefits of a larger ML safety community are numerous. They can decrease the cost of safety methods and increase the propensity to adopt them. Moreover, to ensure that ML systems have desirable properties, it is necessary to rapidly accumulate incremental improvements, but this requires substantial growth since such gains cannot be produced by just a few card-carrying x-risk researchers with the purest intentions.

4. The community will fail to grow if we ignore near-term concerns or actively exclude or sneer at people who work on problems that are useful for both near- and long-term safety (such as adversaries). The alignment community will need to stop engaging in textbook territorialism and welcome serious hypercompetent researchers who do not post on internet forums or who happen not to subscribe to effective altruism. (We include a community strategy in the Appendix.)

5. We focus on reinforcement learning but also deep learning. Most of the machine learning research community studies deep learning (e.g., text processing, vision) and does not use, say, Bellman equations or PPO. While existentially catastrophic failures will likely require competent sequential decision-making agents, the relevant problems and solutions can often be better studied outside of gridworlds and MuJoCo. There is much useful safety research to be done that does not need to be cast as a reinforcement learning problem.

6. To prevent alienating readers, we did not use phrases such as "AGI." AGI-exclusive research will not scale; for most academics and many industry researchers, it's a nonstarter. Likewise, to prevent needless dismissiveness, we kept x-risks implicit, only hinted at them, or used the phrase "permanent catastrophe."
I would have personally enjoyed discussing at length how anomaly detection is an indispensable tool for reducing x-risks from [Black Balls](https://www.nickbostrom.com/papers/vulnerable.pdf), engineered microorganisms, and deceptive ML systems.

Here are how the problems relate to x-risk:

Adversarial Robustness: This is needed for proxy gaming. ML systems encoding proxies must become more robust to optimizers, which is to say they must become more adversarially robust. We make this connection explicit at the bottom of page 9.
Black Swans and Tail Risks: It's hard to be safe without high reliability. It's not obvious we'll achieve high reliability even by the time we have systems that are superhuman in important respects. Even though MNIST is solved for typical inputs, we still do not even have an MNIST classifier for atypical inputs that is reliable! Moreover, if optimizing agents become unreliable in the face of novel or extreme events, they could start heavily optimizing the wrong thing. Models accidentally going off the rails poses an x-risk if they are sufficiently powerful (this is related to "competent errors" and "treacherous turns"). If this problem is not solved, optimizers can use these weaknesses; this is a simpler problem on the way to adversarial robustness.
Anomaly and Malicious Use Detection: This is an indispensable tool for detecting proxy gaming, [Black Balls](https://www.nickbostrom.com/papers/vulnerable.pdf), engineered microorganisms that present bio x-risks, malicious users who may misalign a model, deceptive ML systems, and rogue ML systems.
Representative Outputs: Making models honest is a way to avoid many treacherous turns.
Hidden Model Functionality: This also helps avoid treacherous turns. Backdoors is a potentially useful related problem, as it is about detecting latent but potential sharp changes in behavior.
Value Learning: Understanding utilities is difficult even for humans. Powerful optimizers will need to achieve a certain, as-of-yet unclear level of superhuman performance at learning our values.
Translating Values to Action: Successfully prodding models to optimize our values is necessary for safe outcomes.
Proxy Gaming: Obvious.
Value Clarification: This is the philosophy bot section. We will need to decide what values to pursue. If we decide poorly, we may lock in or destroy what is of value. It is also possible that there is an ongoing moral catastrophe, which we would not want to replicate across the cosmos.
Unintended Consequences: This should help models not accidentally work against our values.
ML for Cybersecurity: If you believe that AI governance is valuable and that global turbulence risks can increase risks of terrible outcomes, this section is also relevant. Even if some of the components of ML systems are safe, they can become unsafe when traditional software vulnerabilities enable others to control their behavior. Moreover, traditional software vulnerabilities may lead to the proliferation of powerful advanced models, and this may be worse than proliferating nuclear weapons.
Informed Decision Making: We want to avoid decision making based on unreliable gut reactions during a time of crisis. This reduces risks of poor governance of advanced systems.

Here are some other notes:

1. We use systems theory to motivate inner optimization as we expect this motivation will be more convincing to others.

2. Rather than having a broad call for "interpretability," we focus on specific transparency-related problems that are more tractable and neglected. (See the Appendix for a table assessing importance, tractability, and neglectedness.) For example, we include sections on making models honest and detecting emergent functionality.

3. The "External Safety" section can also be thought of as technical research for reducing "Governance" risks. For readers mostly concerned about AI risks from global turbulence, there still is technical research that can be done.

Here are some observations while writing the document:

1. Some approaches that were previously very popular are currently neglected, such as inverse reinforcement learning. This may be due to currently low tractability.

2. Five years ago, I started explicitly brainstorming the content for this document. I think it took the whole time for this document to take shape. Moreover, if this were written last fall, the document would be far more confused, since it took around a year after GPT-3 to become reoriented; writing these types of documents shortly after a paradigm shift may be too hasty.

3. When collecting feedback, it was not uncommon for "in-the-know" researchers to make opposite suggestions. Some people thought some of the problems in the Alignment section were unimportant, while others thought they were the most critical. We attempted to include most research directions.

Unsolved Problems in ML Safety

Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt

To make the case for safety to the broader machine learning research community, this paper provides a revised and expanded collection of concrete technical safety research problems, namely:
1. Robustness: Create models that are resilient to adversaries, unusual situations, and Black Swan events.
2. Monitoring: Detect malicious use, monitor predictions, and discover unexpected model functionality.
3. Alignment: Build models that represent and safely optimize hard-to-specify human values.
4. External Safety: Use ML to address risks to how ML systems are handled, including cyberwarfare and global turbulence.
Throughout, the paper attempts to clarify the problems’ motivation and provide concrete project ideas.


I am very sympathetic to the argument that we should be getting experience with aligning powerful models right now, and would be excited to see more work along these lines. As the post mentions, I personally see this sort of work as a strong baseline, and while I currently think that the conceptual work I’m doing is more important, I wouldn’t be surprised if I worked on a project in this vein within the next two years.

I especially agree with the point that this is one of the most scalable forms of research, and am personally working on a [benchmark](https://docs.google.com/document/d/18MEmQ4aA1zdZHBKG5fLeISYoFAv1IfUozWZ5JYnq-bY/edit) meant to incentivize this sort of research for similar reasons.

The case for aligning narrowly superhuman models

One argument against work on AI safety is that [it is hard to do good work without feedback loops](https://www.jefftk.com/p/why-global-poverty). So how could we get feedback loops? The most obvious approach is to actually try to align strong models right now, in order to get practice with aligning models in the future. This post fleshes out what such an approach might look like. Note that I will not be covering all of the points mentioned in the post; if you find yourself skeptical, you may want to read the full post as your question might be answered there.

The author specifically suggests that we work on **aligning narrowly superhuman models** to make them more useful. _Aligning_ a model roughly means harnessing the full capabilities of the model and orienting these full capabilities towards helping humans. For example, GPT-3 presumably “knows” a lot about medicine and health. How can we get GPT-3 to apply this knowledge as best as possible to be maximally useful in answering user questions about health?

_Narrowly superhuman_ means that the model has more knowledge or “latent capability” than either its overseers or its users. In the example above, GPT-3 almost certainly has more medical knowledge than laypeople, so it is at least narrowly superhuman at “giving medical advice” relative to laypeople. (It might even be so relative to doctors, given how broad its knowledge is.)

<@Learning to Summarize with Human Feedback@> is a good example of what this could look like: that paper attempted to “bring out” GPT-3’s latent capability to write summaries, and outperformed the reference summaries written by humans. This sort of work will be needed for any new powerful model we train, and so it has a lot of potential for growing the field of people concerned about long-term risk.

Note that the focus here is on aligning _existing_ capabilities to make a model more useful, and so simply increasing capabilities doesn’t count. As a concrete example, just scaling up the model capacity or training data or compute would _not_ count as an example of “aligning narrowly superhuman models”, even though it might make the model more useful, since scaling increases raw capabilities without improving alignment. This makes it pretty different from what profit-maximizing companies would do by default: instead of baking in domain knowledge and simply scaling up models in order to solve the easiest profitable problems (as you would do if you wanted to maximize profit), work in this research area would look for general and scalable techniques, would not be allowed to scale up models, and would select interestingly difficult problems.

Why is this a fruitful area of research? The author points out four main benefits:
1. Most importantly, the more we align systems ahead of time, the more likely that researchers will be able to put thought and consideration into new issues like treacherous turns, rather than spending all their time putting out fires.
2. We can build practical know-how and infrastructure for alignment techniques like learning from human feedback.
3. As the world gets progressively faster and crazier, we’ll have better AI assistants helping us to navigate the world.
4. It improves our chances of discovering or verifying a long-term or “full” alignment solution.

See also [MIRI’s comments](https://www.alignmentforum.org/posts/AyfDnnAdjG7HHeD3d/miri-comments-on-cotra-s-case-for-aligning-narrowly), which were more positive than I expected.


I liked this post: it’s a good exploration of what you might do if your goal was to work on technical approaches to future governance challenges; that seems valuable and I broadly agree with it (though I did have some nitpicks in [this comment](https://www.alignmentforum.org/posts/hvGoYXi2kgnS3vxqb/some-ai-research-areas-and-their-relevance-to-existential-1?commentId=LjvvW3xddPTXaequB)).

There is then an additional question of whether the best thing to do to improve AI existential safety is to work on technical approaches to governance challenges. There’s some pushback on this claim in the comments that I agree with; I recommend reading through it. It seems like the core disagreement is on the relative importance of risks: in particular, it sounds like the author thinks that existing incentives for preference learning and out-of-distribution robustness are strong enough that we mostly don’t have to worry about it, whereas governance will be much more challenging; I disagree with at least that relative ranking.

It’s possible that we agree on the strength of existing incentives -- I’ve <@claimed@>(@Conversation with Rohin Shah@) a risk of 10% for existentially bad failures of intent alignment if there is no longtermist intervention, primarily because of existing strong incentives. That could be consistent with this post, in which case we’d disagree primarily on whether the “default” governance solutions are sufficient for handling AI risk, where I’m a lot more optimistic than the author.

Some AI research areas and their relevance to existential safety

This long post explains the author’s beliefs about a variety of research topics relevant to AI existential safety. First, let’s look at some definitions.

While AI safety alone just means getting AI systems to avoid risks (including e.g. the risk of a self-driving car crashing), _AI existential safety_ means preventing AI systems from posing risks at least as bad as human extinction. _AI alignment_ on the other hand is about getting an AI system to try to / succeed at doing what a person or institution wants them to do. (The “try” version is _intent alignment_, while the “succeed” version is _impact alignment_.)

Note that AI alignment is not the same thing as AI existential safety. In addition, the author makes the stronger claim that it is insufficient to guarantee AI existential safety, because AI alignment tends to focus on situations involving a single human and a single AI system, whereas AI existential safety requires navigating systems involving multiple humans and multiple AI systems. Just as AI alignment researchers worry that work on AI capabilities for useful systems doesn’t engage enough with the difficulty of alignment, the author worries that work on alignment doesn’t engage enough with the difficulty of multiagent systems.

The author also defines _AI ethics_ as the principles that AI developers and systems should follow, and _AI governance_ as identifying _and enforcing_ norms for AI developers and systems to follow. While ethics research may focus on resolving disagreements, governance will be more focused on finding agreeable principles and putting them into practice.

Let’s now turn to how to achieve AI existential safety. The main mechanism the author sees is to _anticipate, legitimize, and fulfill governance demands_ for AI technology. Roughly, governance demands are those properties which there are social and political pressures for, such as “AI systems should be fair” or “AI systems should not lead to human extinction”. If we can _anticipate_ these demands in advance, then we can do technical work on how to _fulfill_ or meet these demands, which in turn _legitimizes_ them, that is, it makes it clearer that the demand can be fulfilled and so makes it easier to create common knowledge that it is likely to become a legal or professional standard.

We then turn to various different fields of research, which the author ranks on three axes: helpfulness to AI existential safety (including potential negative effects), educational value, and neglectedness. Note that for educational value, the author is estimating the benefits of conducting research on the topic _to the researcher_, and not to (say) the rest of the field. I’ll only focus on helpfulness to AI existential safety below, since that’s what I’m most interested in (it’s where the most disagreement is, and so where new arguments are most useful), but I do think all three axes are important.

The author ranks both preference learning and out of distribution robustness lowest on helpfulness to existential safety (1/10), primarily because companies already have a strong incentive to have robust AI systems that understand preferences.

Multiagent reinforcement learning (MARL) comes only slightly higher (2/10), because since it doesn’t involve humans its main purpose seems to be to deploy fleets of agents that may pose risks to humanity. It is possible that MARL research could help by producing <@cooperative agents@>(@Cooperative AI Workshop@), but even this carries its own risks.

Agent foundations is especially dual-use in this framing, because it can help us understand the big multiagent system of interactions, and there isn’t a restriction on how that understanding could be used. It consequently gets a low score (3/10), that is a combination of “targeted applications could be very useful” and “it could lead to powerful harmful forces”.

Minimizing side effects starts to address the challenges the author sees as important (4/10): in particular, it can allow us both to prevent accidents, where an AI system “messes up”, and it can help us prevent externalities (harms to people other than the primary stakeholders), which are one of the most challenging issues in regulating multiagent systems.

Fairness is valuable for the obvious reason: it is a particular governance demand that we have anticipated, and research on it now will help fulfill and legitimize that demand. In addition, research on fairness helps get people to think at a societal scale, and to think about the context in which AI systems are deployed. It may also help prevent centralization of power from deployment of AI systems, since that would be an unfair outcome.

The author would love it if AI/ML pivoted to frequently think about real-life humans and their desires, values and vulnerabilities. Human-robot interaction (HRI) is a great way to cause more of that to happen, and that alone is valuable enough that the author assigns it 6/10, tying it with fairness.

As we deploy more and more powerful AI systems, things will eventually happen too quickly for humans to monitor. As a result, we will need to also automate the process of governance itself. The area of computational social choice is well-posed to make this happen (7/10), though certainly current proposals are insufficient and more research is needed.

Accountability in ML is good (8/10) primarily because as we make ML systems accountable, we will likely also start to make tech companies accountable, which seems important for governance. In addition, in a <@CAIS@>(@Reframing Superintelligence: Comprehensive AI Services as General Intelligence@) scenario, better accountability mechanisms seem likely to help in ensuring that the various AI systems remain accountable, and thus safer, to human society.

Finally, interpretability is useful (8/10) for the obvious reasons: it allows developers to more accurately judge the properties of systems they build, and helps in holding developers and systems accountable. But the most important reason may be that interpretable systems can make it significantly easier for competing institutions and nations to establish cooperation around AI-heavy operations.


Both this and the previous summary illustrate an increasingly common perspective:

1. The world is not going to look like “today’s world plus a single AGI agent”: instead, we will likely have a proliferation of many different AI systems specialized for different purposes.
2. In such a world, there are a lot of different challenges that aren’t standard intent alignment.
3. We should focus on these other challenges because [a variety of reasons].

**If you have technical CS skills**, how should you prioritize between this perspective and the more classical intent alignment perspective?

**Importance.** I’ve <@estimated@>(@Conversation with Rohin Shah@) a 10% chance of existential catastrophe via a failure of intent alignment, absent intervention from longtermists to address intent alignment. Estimates vary quite a lot, even among people who have thought about the problem a lot; I’ve heard as low as < 1% and as high as 80% (though these usually don’t assume “no intervention from longtermists”).

It’s harder to estimate the importance of structural risks and extinction risks highlighted in the two summaries above, but the arguments in the previous two posts seem reasonably compelling and I think I’d be inclined to assign a similar importance to it (i.e. similar probability of causing an existential catastrophe).

Note that this means I’m disagreeing with Critch: he believes that we are far more likely to go extinct through effects unique to multi-multi dynamics; in contrast I find the argument less persuasive because we do have governance, regulations, national security etc. that would already be trying to mitigate issues that arise in multi-multi contexts, especially things that could plausibly cause extinction.

**Neglectedness.** I’ve already taken into account neglectedness outside of EA in estimating the probabilities for importance. Within EA there is already a huge amount of effort going into intent alignment, and much less in governance and multi-multi scenarios -- perhaps a difference of 1-2 orders of magnitude; the difference is even higher if we only consider people with technical CS skills.

**Tractability.** I buy the argument in Dafoe’s article that for AI governance due to our vast uncertainty we need a “metropolis” model where field-building is quite important; I think that implies that solving the full problem (at today's level of knowledge) would require a lot of work and building of expertise. In contrast, with intent alignment, we have a single technical problem with significantly less uncertainty. As a result, I expect that currently in expectation a single unit of work goes further to solving intent alignment than to solving structural risks / multi-multi problems, and so intent alignment is more tractable.

I also expect technical ideas to be a bigger portion of "the full solution" in the case of intent alignment -- as Dafoe argues, I expect that for structural risks the solution looks more like "we build expertise and this causes various societal decisions to go better" as opposed to "we figure out how to write this piece of code differently so that it does better things". This doesn't have an obvious impact on tractability -- if anything, I'd guess it argues in favor of the tractability of work on structural risks, because it seems easier to me to create prestigious experts in particular areas than to make progress on a challenging technical problem whose contours are still uncertain since it arises primarily in the future.

I suspect that I disagree with Critch here: I think he is more optimistic about technical solutions to multi-multi issues themselves being useful. In the past I think humanity has resolved such issues via governance and regulations and it doesn’t seem to have relied very much on technical research; I’d expect that trend to continue.

**Personal fit.** This is obviously important, but there isn’t much in general for me to say about it.

Once again, I should note that this is all under the assumption that you have technical CS skills. I think overall I end up pretty uncertain which of the two areas I’d advise going in (assuming personal fit was equal in both areas). However, if you are more of a generalist, I feel much more inclined to recommend choosing some subfield of AI governance, again subject to personal fit, and Critch agrees with this.

Andrew Critch on AI Research Considerations for Human Existential Safety

This podcast discusses the recent <@ARCHES@>(@AI Research Considerations for Human Existential Safety@) document, and several thoughts surrounding it. There’s a lot in here that I won’t summarize, including a bunch of stuff that was in the summary of ARCHES. I’m going to focus primarily on the (substantial) discussion of how to prioritize within the realm of possible risks related in some way to AI systems.

Firstly, let’s be clear about the goal: ensuring existential safety, that is, making sure human extinction never happens. Note the author means literal extinction, as opposed to something like “the loss of humanity’s long-term potential”, because the former is clearer. While it is not always clear whether something counts as “extinction” (what if we all become uploads?), it is a lot clearer than whether a scenario counts as a loss of potential.

Typical alignment work focuses on the “single-single” case, where a single AI system must be aligned with a single human, as in e.g. <@intent alignment@>(@Clarifying "AI Alignment"@). However, this isn’t ultimately what we care about: we care about multi-multi existential safety, that is, ensuring that when multiple AI systems act in a world with multiple humans, extinction does not happen. There are pretty significant differences between these: in particular, it’s not clear whether multi-multi “alignment” even has meaning, since it is unclear whether it makes sense to view humanity as an agent to which an AI system could be “aligned”.

Nonetheless, single-single alignment seems like an important subproblem of multi-multi existential safety: we will be delegating to AI systems in the future; it seems important that we know how to do so. How do we prioritize between single-single alignment, and the other subproblems of multi-multi existential safety? A crucial point is that single-single work will not be neglected, because companies have strong incentives to solve single-single alignment (both in the sense of optimizing for the right thing, and for being robust to distributional shift). In contrast, in multi-multi systems, it is often the case that there is a complex set of interacting effects that lead to some negative outcome, and there is no one actor to blame for the negative outcome, and as a result it doesn’t become anybody’s job to prevent that negative outcome.

For example, if you get a huge medical bill because the necessary authorization forms hadn’t been filled out, whose fault is it? Often in such cases there are many people to blame: you could blame yourself for not checking the authorization, or you could blame the doctor’s office for not sending the right forms or for not informing you that the authorization hadn’t been obtained, etc. Since it’s nobody’s job to fix such problems, they are and will remain neglected, and so work on them is more impactful.

Something like transparency is in a middle ground: it isn’t profitable yet, but probably will be soon. So, if someone were indifferent between a bunch of areas of research, the author would advise for e.g. multi-stakeholder delegation over transparency over robustness. However, the author emphasizes that it’s far more important that people work in some area of research that they find intellectually enriching and relevant to existential safety.

The podcast has lots of other points, here is an incomplete quick selection of them:

- In a multi-multi world, without good coordination you move the world in a “random” direction. There are a lot of variables which have to be set just right for humans to survive (temperature, atmospheric composition, etc) that are not as important for machines. So sufficiently powerful systems moving the world in a “random” direction will lead to human extinction.
- One response to the multi-multi challenge is to have a single group make a powerful AI system and “take over the world”. This approach is problematic since many people will oppose such a huge concentration of power. In addition, it is probably not desirable even if possible, since it reduces robustness by creating a single point of failure.
- Another suggestion is to create a powerful AI system that protects humanity (but is still uncontrollable in that humanity cannot stop its operation). The author does not like the solution much, because if we get it wrong and deploy a misaligned uncontrollable AI system, then we definitely die. The author prefers that we instead always have control over the AI systems we deploy.


This paper is a great organization and explanation of several older papers (that haven't been summarized in this newsletter because they were published before 2018 and I read them before starting this newsletter), and I wish I had read it sooner. It seems to me that the integrated Bayesian agent is the clear winner -- the only downside is the computational cost, which would be a bottleneck for any of the models considered here.

One worry I have with this sort of analysis is that the guarantees you get out of it depends quite a lot on how you model the situation. For example, let's suppose that after I sleep I wake up refreshed and more capable of intellectual work. Should I model this as "policy corruption", or as a fixed policy that takes as an input some information about how rested I am?

The Alignment Problem for Bayesian History-Based Reinforcement Learners

After forgetting its existence for quite a while, I've finally read through this technical report (which won first place in <@round 2 of the AI alignment prize@>(@Announcement: AI alignment prize round 2 winners and next round@)). It analyzes the alignment problem from an AIXI-like perspective, that is, by theoretical analysis of powerful Bayesian RL agents in an online POMDP setting.

In this setup, we have a POMDP environment, in which the environment has some underlying state, but the agent only gets observations of the state and must take actions in order to maximize rewards. The authors consider three main setups: 1) rewards are computed by a preprogrammed reward function, 2) rewards are provided by a human in the loop, and 3) rewards are provided by a _reward predictor_ which is trained interactively from human-generated data.

For each setup, they consider the various objects present in the formalism, and ask how these objects could be corrupted, misspecified, or misleading. This methodology allows them to identify several potential issues, which I won't get into as I expect most readers are familiar with them. (Examples include wireheading and threatening to harm the human unless they provide maximal reward.)

They also propose several tools that can be used to help solve misalignment. In order to prevent reward function corruption, we can have the agent _simulate_ the future trajectory, and _evaluate_ this future trajectory with the current reward, removing the incentive to corrupt the reward function. (This was later developed into <@current-RF optimization@>(@Designing agent incentives to avoid reward tampering@).)

Self-corruption awareness refers to whether or not the agent is aware that its policy can be modified. A self-corruption _unaware_ agent is one that behaves as though it's current policy function will never be changed, effectively ignoring the possibility of corruption. It is not clear which is more desirable: while a self-corruption unaware agent will be more corrigible (in the [MIRI sense](https://intelligence.org/files/Corrigibility.pdf)), it also will not preserve its utility function, as it believes that even if the utility function changes the policy will not change.

Action-observation grounding ensures that the agent only optimizes over policies that work on histories of observations and actions, preventing agents from constructing entirely new observation channels ("delusion boxes") which mislead the reward function into thinking everything is perfect.

The interactive setting in which a reward predictor is trained based on human feedback offers a new challenge: that the human data can be corrupted or manipulated. One technique to address this is to get _decoupled_ data: if your corruption is determined by the current state s, but you get feedback about some different state s', as long as s and s' aren't too correlated it is possible to mitigate potential corruptions.

Another leverage point is how we decide to use the reward predictor. We could consider the _stationary_ reward function, which evaluates simulated trajectories with the _current_ reward predictor, i.e. assuming that the reward predictor will never be updated again. If we combine this with self-corruption unawareness (so that the policy also never expects the policy to change), then the incentive to corrupt the reward predictor's data is removed. However, the resulting agent is _time-inconsistent_: it acts as though its reward never changes even though it in practice does, and so it can make a plan and start executing it, only to switch over to a new plan once the reward changes, over and over again.

The _dynamic_ reward function avoids this pitfall by evaluating the kth timestep of a simulated trajectory by also taking an expectation over future data that the reward predictor will get. This agent is no longer time-inconsistent, but it now incentivizes the agent to manipulate the data. This can be fixed by building a single integrated Bayesian agent, which maintains a single environment model that predicts both the reward function and the environment model. The resulting agent is time-consistent, utility-preserving, and has no direct incentive to manipulate the data. (This is akin to the setup in <@assistance games / CIRL@>(@Cooperative Inverse Reinforcement Learning@).)

One final approach is to use a _counterfactual_ reward function, in which the data is simulated in a counterfactual world where the agent executed some known safe default policy. This no longer depends on the current time, and is not subject to data corruption since the data comes from a hypothetical that is independent of the agent's actual policy. However, it requires a good default policy that does the necessary information-gathering actions, and requires the agent to have the ability to simulate human feedback in a counterfactual world.


I love the abstract and introduction, because of their directness at actually stating what we want and care about. I am also a big fan of the distinction between provably beneficial and reducing x-risk, and the single/multi analysis.

The human fragility argument, as applied to generally intelligent agents, is a bit tricky. One interpretation is that the "hardness" stems from the fact that you need a bunch of "bits" of knowledge / control in order to keep humans around. However, it seems like a generally intelligent AI should easily be able to keep humans around "if it wants", and so the bits already exist in the AI. (As an analogy: we make big changes to the environment, but we could easily preserve deer habitats if we wanted to.) Thus, it is really a question of what "distribution" you expect the AI system is sampled from: if you think we'll build AI systems that try to do what humanity wants, then we're probably fine, but if you think that there will be multiple AI systems that each do what their users want, but the users have conflicts, the overall system seems more "random" in its goals, and so more likely to fall into the "default" outcome of human extinction.

The research directions are very detailed, and while there are some suggestions that don't seem particularly useful to me, overall I am happy with the list. (And as the paper itself notes, what is and isn't useful depends on your models of AI development.)

AI Research Considerations for Human Existential Safety

This research agenda out of CHAI directly attacks the problem longtermists care about: **how to prevent AI-related existential catastrophe**. This is distinctly different from the notion of being "provably beneficial": a key challenge for provable beneficence is defining what we even mean by "beneficial". In contrast, there are avenues for preventing AI-caused human extinction that do not require an understanding of "beneficial": most trivially, we could coordinate to never build AI systems that could cause human extinction.

Since the focus is on the _impact_ of the AI system, the authors need a new phrase for this kind of AI system. They define a **prepotent AI system** to be one that cannot be controlled by humanity **and** has the potential to transform the world in a way that is at least as impactful as humanity as a whole. Such an AI system need not be superintelligent, or even an AGI; it may have powerful capabilities in a narrow domain such as technological autonomy, replication speed, or social acumen that enable prepotence.

By definition, a prepotent AI system is capable of transforming the world drastically. However, there are a lot of conditions that are necessary for continued human existence, and most transformations of the world will not preserve these conditions. (For example, consider the temperature of the Earth or the composition of the atmosphere.) As a result, human extinction is the _default_ outcome from deploying a prepotent AI system, and can only be prevented if the system is designed to preserve human existence with very high precision relative to the significance of its actions. They define a misaligned prepotent AI system (MPAI) as one whose deployment leads to human extinction, and so the main objective is to avert the deployment of MPAI.

The authors break down the risk of deployment of MPAI into five subcategories, depending on the beliefs, actions and goals of the developers. The AI developers could fail to predict prepotence, fail to predict misalignment, fail to coordinate with other teams on deployment of systems that aggregate to form an MPAI, accidentally (unilaterally) deploy MPAI, or intentionally (unilaterally) deploy MPAI. There are also hazardous social conditions that could increase the likelihood of risks, such as unsafe development races, economic displacement of humans, human enfeeblement, and avoidance of talking about x-risk at all.

Moving from risks to solutions, the authors categorize their research directions along three axes based on the setting they are considering. First, is there one or multiple humans; second, is there one or multiple AI systems; and third, is it helping the human(s) comprehend, instruct, or control the AI system(s). So, multi/single instruction would involve multiple humans instructing a single AI system. While we will eventually need multi/multi, the preceding cases are easier problems from which we could gain insights that help solve the general multi/multi case. Similarly, comprehension can help with instruction, and both can help with control.

The authors then go on to list 29 different research directions, which I'm not going to summarize here.


I don't know much about MIRI and Agent Foundations, but this made sense to me and felt like it clarified things for me.

My take on agent foundations: formalizing metaphilosophical competence

Argues that the point of Agent Foundations is to create conceptual clarity for fuzzy concepts that we can't formalize yet (such as logical uncertainty). We can then verify whether our ML algorithms have these desirable properties. It is decidedly _not_ a goal to build a friendly AI using modules that Agent Foundations develop.


My broad take is that I agree that analyzing neural nets is useful and more work should go into it, but I broadly disagree that this leads to reduced x-risk by increasing the likelihood that developers can look at their trained model, determine whether it is dangerous by understanding it mechanistically, and decide whether to deploy it, in a "zero-shot" way. The key difficulty here is the mechanistic transparency, which seems like far too strong a property for us to aim: I would expect the cost of making a neural network mechanistically transparent to far exceed the cost of training that neural network in the first place, and so it would be hard to get developers to mechanistically understand trained models to detect danger.

Right now for e.g. image classifiers, some people on OpenAI's Clarity team have spent multiple years understanding a single image classifier, which is orders of magnitude more expensive than training the classifier. My guess is that this will become superlinearly harder as models get bigger (and especially as models become superhuman), and so it seems quite unlikely that we could have mechanistic transparency for very complex AGI systems built out of neural nets. More details in [this comment](https://www.alignmentforum.org/posts/8GdPargak863xaebm/an-analytic-perspective-on-ai-alignment?commentId=c8vogpyjgoXEspX8z). Note that Daniel agrees that it is an open question whether this sort of mechanistic transparency is possible, and thinks that we don't have much evidence yet that it isn't.

An Analytic Perspective on AI Alignment

In this post, Daniel Filan presents an analytic perspective on how to do useful AI alignment research. His take is that in a world with powerful AGI systems similar to neural networks, it may be sufficient to be able to detect whether a system would cause bad outcomes before you deploy it on real-world systems with unknown distributions. To this end, he advocates for work on transparency that gives <@mechanistic understandings@>(@Mechanistic Transparency for Machine Learning@) of the systems in question, combined with foundational research that allows us to reason about the safety of the produced understandings.


The comments reveal a [third possibility](https://www.alignmentforum.org/posts/jYdAxH8BarPT4fqnb/a-dilemma-for-prosaic-ai-alignment#K8fRPa9NWZXdARLYN): the alignment scheme could be trained jointly alongside the cutting edge AI training. For example, we might hope that we can train a question answerer that can answer questions about anything "the model already knows", and this question answering system is trained simultaneously with the training of the model itself. I think this takes the "oomph" out of the dilemma as posed here -- it seems reasonably likely that it only takes fractionally more resources to train a question answering system on top of the model, if it only has to use knowledge "already in" the model, which would let it be competitive, while still preventing mesa optimizers from arising (if the alignment scheme does its job). Of course, it may turn out that it takes a huge amount of resources to train the question answering system, making the system uncompetitive, but that seems hard to predict given our current knowledge.

A dilemma for prosaic AI alignment

This post points out a potential problem for <@Prosaic AI alignment@>, in which we try to align AI systems built using current techniques. Consider some prosaic alignment scheme, such as <@iterated amplification@>(@Learning Complex Goals with Iterated Amplification@) or <@debate@>(@AI safety via debate@). If we try to train an AI system directly using such a scheme, it will likely be uncompetitive, since it seems likely that the most powerful AI systems will probably require cutting-edge algorithms, architectures, objectives, and environments, at least some of which will be replaced by new versions from the safety scheme. Alternatively, we could first train a general AI system, and then use our alignment scheme to finetune it into an aligned AI system. However, this runs the risk that the initial training could create a misaligned mesa optimizer, that then deliberately sabotages our finetuning efforts.


I certainly agree that in the presence of powerful adversarial optimizers, you need security to get your system to do what you want. However, we can just _not build powerful adversarial optimizers_. My preferred solution is to make sure our AI systems are [trying to do what we want](https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd/p/ZeE7EKHTFMBs8eMxn#3ECKoYzFNW2ZqS6km), so that they never become adversarial in the first place. But if for some reason we can't do that, then we could make sure AI systems don't become too powerful, or not build them at all. It seems very weird to instead say "well, the AI system is going to be adversarial and way more powerful, let's figure out how to make it secure" -- that should be the last approach, if none of the other approaches work out. (More details in [this comment](https://www.alignmentforum.org/posts/mdau2DBSMi5bWXPGA/useful-does-not-mean-secure#xccsZeboCNcNJeGas).) Note that MIRI doesn't aim for security because they expect powerful adversarial optimization -- they aim for security because _any_ optimization <@leads to extreme outcomes@>(@Optimization Amplifies@). (More details in [this comment](https://www.lesswrong.com/posts/mdau2DBSMi5bWXPGA/useful-does-not-mean-secure#a8mzXZeMLCbAY5S2J).)

Useful Does Not Mean Secure

Recently, I [suggested](https://www.alignmentforum.org/posts/X2i9dQQK3gETCyqh2/chris-olah-s-views-on-agi-safety#QyeCpFTcXsQuxpz7B) the following broad model: _The way you build things that are useful and do what you want is to understand how things work and put them together in a deliberate way. If you put things together randomly, they either won't work, or will have unintended side effects._ Under this model, relative to doing nothing, it is net positive to improve our understanding of AI systems, e.g. via transparency tools, even if it means we build powerful AI systems sooner (which reduces the time we have to solve alignment).

This post presents a counterargument: while understanding helps us make _useful_ systems, it need not help us build _secure_ systems. We need security because that is the only way to get useful systems in the presence of powerful external optimization, and the whole point of AGI is to build systems that are more powerful optimizers than we are. If you take an already-useful AI system, and you "make it more powerful", this increases the intelligence of both the useful parts and the adversarial parts. At this point, the main point of failure is if the adversarial parts "win": you now have to be robust against adversaries, which is a security property, not a usefulness property.

Under this model, transparency work need not be helpful: if the transparency tools allow you to detect some kinds of bad cognition but not others, an adversary simply makes sure that all of its adversarial cognition is the kind you can't detect. _Rohin's note: Or, if you use your transparency tools during training, you are selecting for models whose adversarial cognition is the kind you can't detect._ Then, transparency tools could increase understanding and shorten the time to powerful AI systems, _without_ improving security.


Overview of AGI Safety Research Agendas

The video from my talk at the Beneficial AGI conference has just been released. In this talk, I cover five broad safety-related areas that people are investigating: understanding the future of AI (<@embedded agency@>(@Embedded Agents@), <@CAIS@>(@Reframing Superintelligence: Comprehensive AI Services as General Intelligence@)), limiting the influence of an AI system (<@boxing@>(@Asymptotically Benign AGI@), <@impact regularization methods@>(@Designing agent incentives to avoid side effects@)), robustness (<@verification@>(@Certified Defenses against Adversarial Examples@), [red teaming](https://ai-alignment.com/red-teams-b5b6de33dc76)), helpful AI systems (<@ambitious value learning@>(@What is ambitious value learning?@), [preference learning](https://openai.com/blog/deep-reinforcement-learning-from-human-preferences/), [Cooperative IRL](https://bair.berkeley.edu/blog/2017/08/17/cooperatively-learning-human-values/), <@corrigibility@>, <@factored cognition@>, [iterated amplification](https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd), <@debate@>(@AI safety via debate@)) and <@interpretability@>(@Exploring Neural Networks with Activation Atlases@). My [podcast](https://futureoflife.org/2019/04/11/an-overview-of-technical-ai-alignment-with-rohin-shah-part-1/) ([AN #54](https://mailchi.mp/3e2f43012b07/an-54-boxing-a-finite-horizon-ai-system-to-keep-it-unambitious)) covers almost all of this and more, so you may want to listen to that instead.

[FLI's YouTube channel](https://www.youtube.com/channel/UC-rCCy3FQ-GItDimSR9lhzw)


AI Alignment Podcast: An Overview of Technical AI Alignment: [Part 1](https://futureoflife.org/2019/04/11/an-overview-of-technical-ai-alignment-with-rohin-shah-part-1/) and [Part 2](https://futureoflife.org/2019/04/25/an-overview-of-technical-ai-alignment-with-rohin-shah-part-2/)

In this podcast, I go through a large swath of research agendas around technical AI alignment. The first part is more of a description of what research agendas exist, who works on them, and what they are trying to do, while the second part delves more into the details of each approach. I'd strongly recommend listening to them if you're trying to orient yourself in the technical AI safety landscape.

Topics covered include [embedded agency](https://www.alignmentforum.org/s/Rm6oQRJJmhGCcLvxh), [value learning](https://www.alignmentforum.org/s/4dHMdK5TLN6xcqtyc), <@impact regularization methods@>(@Designing agent incentives to avoid side effects@), [iterated amplification](https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd), <@debate@>(@AI safety via debate@), <@factored cognition@>(@Factored Cognition@), <@robustness@>(@Techniques for optimizing worst-case performance@), interpretability (no canonical link, but <@activation atlases@>(@Exploring Neural Networks with Activation Atlases@) is an example), <@comprehensive AI services@>(@Reframing Superintelligence: Comprehensive AI Services as General Intelligence@), <@norm following@>(@Incomplete Contracting and AI Alignment@), and [boxing](https://www.alignmentforum.org/posts/pZhDWxDmwzuSwLjou/asymptotically-benign-agi) (this newsletter).


Summary of the Technical Safety Workshop

David identifies two broad types of AI safety work: human in the loop approaches, and theory approaches. A notable subset of the former category is methods which improve our ability to give advanced systems meaningful feedback - this includes debate, IDA, and recursive reward modeling. CIRL and CAIS are also human-in-the-loop. Meanwhile the theory category includes MIRI's work on agent foundations; side effect metrics; and verified boxing.


Comments on CAIS

This post is another summary of and response to [Comprehensive AI Services](https://www.fhi.ox.ac.uk/wp-content/uploads/Reframing_Superintelligence_FHI-TR-2019-1.1-1.pdf), after [mine](https://www.alignmentforum.org/posts/x3fNwSe5aWZb5yXEG/reframing-superintelligence-comprehensive-ai-services-as) last week ([AN #40](https://mailchi.mp/b649f32b07da/alignment-newsletter-40)). I recommend it to get a different take on an important set of ideas. It delves much more into the arguments between the CAIS perspective and the single-AGI-agent perspective than my summary did.


People sometimes expect me to be against this sort of work, because I wrote <@Coherence arguments do not imply goal-directed behavior@>. This is not true. My point in that post is that coherence arguments _alone_ are not enough, you need to combine them with some other assumption (for example, that there exists some “resource” over which the agent has no terminal preferences). I do think it is plausible that this research agenda gives us a better picture of agency that tells us something about how AI systems will behave, or something about how to better infer human values. While I am personally more excited about studying particular development paths to AGI rather than more abstract agent models, I do think this research would be more useful than other types of alignment research I have seen proposed.

Selection Theorems: A Program For Understanding Agents

This post proposes a research area for understanding agents: **selection theorems**. A selection theorem is a theorem that tells us something about agents that will be selected for in a broad class of environments. Selection theorems are helpful because (1) they can provide additional assumptions that can help with learning human values, and (2) they can tell us likely properties of the agents we build by accident (think inner alignment concerns).

As an example, [coherence arguments](https://www.alignmentforum.org/posts/RQpNHSiWaXTvDxt6R/coherent-decisions-imply-consistent-utilities) demonstrate that when an environment presents an agent with “bets” or “lotteries”, where the agent cares only about the outcomes of the bets, then any “good” agent can be represented as maximizing expected utility. (What does it mean to be “good”? This can vary, but one example would be that the agent is not subject to Dutch books, i.e. situations in which it is guaranteed to lose resources.) This can then be turned into a selection argument by combining it with something that selects for “good” agents. For example, evolution will select for agents that don’t lose resources for no gain, so humans are likely to be represented as maximizing expected utility. Unfortunately, many coherence arguments implicitly assume that the agent has no internal state, which is not true for humans, so this argument does not clearly work. As another example, our ML training procedures will likely also select for agents that don’t waste resources, which could allow us to conclude that the resulting agents can be represented as maximizing expected utility, if the agents don't have internal states.

Coherence arguments aren’t the only kind of selection theorem. The <@good(er) regulator theorem@>(@Fixing The Good Regulator Theorem@) provides a set of scenarios under which agents learn an internal “world model”. The [Kelly criterion](http://www.eecs.harvard.edu/cs286r/courses/fall10/papers/Chapter6.pdf) tells us about scenarios in which the best (most selected) agents will make bets as though they are maximizing expected log money. These and other examples are described in [this followup post](https://www.alignmentforum.org/posts/N2NebPD78ioyWHhNm/some-existing-selection-theorems).

The rest of this post elaborates on the various parts of a selection theorem and provides advice on how to make original research contributions in the area of selection theorems. Another [followup post](https://www.alignmentforum.org/posts/RuDD3aQWLDSb4eTXP/what-selection-theorems-do-we-expect-want) describes some useful properties for which the author expects there are useful selections theorems to prove.


Early on, this post reminded me of [Twenty Billion Questions](https://arxiv.org/pdf/1705.10720.pdf); even though they are concretely different, these two pieces share a conceptual thread. They both consider the measurement of multiple quantities essential for solving their problems: 20BQ for encouraging AIs to be low-impact, and this post for productive framings of ill-defined concepts and as a heads-up about potential catastrophes.

Measurement is important, and this article poignantly argues why and illustrates how. It volunteers potential ideas that can be worked on today by mainstream ML researchers, and offers up a powerful toolkit to improve one's own quality of analysis. It would be great to see more examples of this technique applied to other contentious, fuzzy concepts in ML and beyond. I'll quickly note that while there seems to be minimal interest in this from academia, measurement of optimization power has been discussed earlier in several ways, e.g. [Measuring Optimization Power](https://www.lesswrong.com/posts/Q4hLMDrFd8fbteeZ8/measuring-optimization-power), or <@the ground of optimization@>(@The ground of optimization@).

**Rohin's opinion:** I broadly agree with the perspective in this post. I feel especially optimistic about the prospects of measurement for (a) checking whether our theoretical arguments hold in practice and (b) convincing others of our positions (assuming that the arguments do hold in practice).

Measurement, Optimization, and Take-off Speed

In this blogpost, the author argues that "trying to measure pretty much anything you can think of is a good mental move that is heavily underutilized in machine learning". He motivates the value of measurement and additional metrics by (i) citing evidence from the history of science, policy-making, and engineering (e.g. x-ray crystallography contributed to rapid progress in molecular biology), (ii) describing how, conceptually, "measurement has several valuable properties" (one of which is to act as interlocking constraints that help to error-check theories), and (iii) providing anecdotes from his own research endeavours where such approaches have been productive and useful (see, e.g. <@Rethinking Bias-Variance Trade-off@>(@Rethinking Bias-Variance Trade-off for Generalization of Neural Networks@)).

He demonstrates his proposal by applying it to the notion of _optimization power_ -- an important idea that has not been measured or even framed in terms of metrics. Two metrics are offered: (a) the change (typically deterioration) of performance when trained with a perturbed objective function with respect to the original objective function, named _Outer Optimization_, and (b) the change in performance of agents during their own lifetime (but without any further parameter updates), such as the log-loss on the next sentence for a language model after it sees X number of sequences at test time, or _Inner Adaptation_. Inspired by these, the article includes research questions and possible challenges.

He concludes with the insight that take-off would depend on these two continuous processes, Outer Optimization and Inner Adaptation, that work on very different time-scales, with the former being, at this time, much quicker than the latter. However, drawing an analogy from evolution, where it took billions of years of optimization to generate creatures like humans that were exceptional at rapid adaptation, we might yet see a fast take-off were Inner Adaptation turns out to be an exponential process that dominates capabilities progress. He advocates for early, sensitive measurement of this quantity as it might be an early warning sign of imminent risks.


Transparency and AGI safety

This post identifies four different motivations for working on transparency:

1. By learning more about how current neural networks work, we can improve our forecasts for AI timelines.
2. It seems _necessary_ for inner alignment. In particular, whatever AI development model you take, it seems likely that there will be some possibility of emergent misbehavior, and there doesn’t yet seem to be a way to rule that out except via transparency.
3. A good solution to transparency would be _sufficient_ for safety, since we could at least notice when AI systems were misaligned, and then choose not to deploy them.
4. Even if AI will “go well by default”, there are still instrumental reasons for transparency, such as improving cause prioritization in EA (via point 1), and for making systems more capable and robust.

After reviewing work on <@circuits@>(@Thread: Circuits@), the post suggests a few directions for future research:

1. Investigating how modular neural networks tend to be,
2. Figuring out how to make transparency outputs more precise and less subjective,
3. Looking for circuits in other networks (i.e. not image classifiers), see e.g. <@RL vision@>(@Understanding RL Vision@),
4. Figuring out how transparency fits into an end-to-end story for AI safety.


Certainly to the extent that AI alignment requires us to “lock in” philosophical approaches, I think it is important that we consider a plurality of views for this purpose (see also <@The Argument from Philosophical Difficulty@>). I especially think this is true if our approach to alignment is to figure out “human values” and then tell an AI to maximize them. However, I’m more optimistic about other approaches to alignment; and I think they require fewer philosophical commitments, so it becomes less of an issue that the alignment community has a specific philosophical bubble. See [this comment](https://www.alignmentforum.org/posts/jS2iiDPqMvZ2tnik2/ai-alignment-philosophical-pluralism-and-the-relevance-of?commentId=zaAYniACRc29CM6sJ) for more details.

AI Alignment, Philosophical Pluralism, and the Relevance of Non-Western Philosophy

This post argues that AI alignment has specific philosophical tendencies: 1) connectionism, where knowledge is encoded in neural net weights rather than through symbols, 2) behaviorism, where we learn from data rather than using reasoning or planning, 3) Humean motivations for humans (i.e. modeling humans as reward maximizers), 4) viewing rationality as decision theoretic, that is, about maximizing expected utility, rather than also considering e.g. logic, argumentation, and dialectic, and 5) consequentialism. This could be a “philosophical bubble” caused by founder effects from the EA and rationality communities, as well as from the recent success and popularity of deep learning.

Instead, we should be aiming for philosophical plurality, where we explore other philosophical traditions as well. This would be useful because 1) we would likely find insights not available in Western philosophy, 2) we would be more robust to moral uncertainty, 3) it helps us get buy in from more actors, and 4) it is the “right” thing to do, to allow others to choose the values and ethical frameworks that matter to them.

For example, certain interpretations of Confucian philosophy hold that norms have intrinsic value, as opposed to the dominant approach in Western philosophy in which individual preferences have intrinsic value, while norms only have instrumental value. This may be very relevant for learning what an AI system should optimize. Similarly, Buddhist thought often talks about problems of ontological shifts.


This agenda, like others, is motivated by the scenario where we need to get alignment right the first time, without empirical feedback loops, both because we might be facing one-shot success or failure, and because the stakes are so high that we should aim for high reliability subject to time constraints. I put low probability on the first reason (alignment being one-shot), and it seems much less tractable, so I mostly ignore those scenarios. I agree with the second reason, but aiming for this level of rigor seems like it will take much longer than the time we actually have. Given this high level disagreement, it's hard for me to evaluate the research agenda itself.

The Learning-Theoretic AI Alignment Research Agenda

This agenda aims to create a general abstract theory of intelligence (in a manner similar to [AIXI](https://en.wikipedia.org/wiki/AIXI), but with some deficiencies removed). In particular, once we use the framework of reinforcement learning, regret bounds are a particular way of provably quantifying an agent's intelligence (though there may be other ways as well). Once we have this theory, we can ground all other AI alignment problems within it. Specifically, alignment would be formalized as a value learning protocol that achieves some regret bound. With this formalization, we can solve hard metaphilosophy problems such as "What is imperfect rationality?" through the intuitions gained from looking at the problem through the lens of value learning protocols and universal reinforcement learning.


I focus almost exclusively on mid-term safety (while thinking about long-term safety), not because I disagree with this argument, but in spite of it. I think there is a good chance that any work I do will be useless for aligning superintelligent AI because of a paradigm shift, but I do it anyway because it seems very important on short timelines, which are easier to affect; and I don't know of other approaches to take that would have a significantly higher probability of being useful for aligning superintelligent AI.

A possible stance for AI control research

Conceptual issues in AI safety: the paradigmatic gap

Lots of current work on AI safety focuses on what we can call "mid-term safety" -- the safety of AI systems that are more powerful and more broadly deployed than the ones we have today, but work using relatively similar techniques as the ones we use today. However, it seems plausible that there will be a paradigm shift in how we build AI systems, and if so it's likely that we will have a new, completely different set of mid-term concerns, rendering the previous mid-term work useless. For example, at the end of the 19th century, horse excrement was a huge public health hazard, and "mid-term safety" would likely have been about how to remove the excrement. Instead, the automobile was developed and started replacing horses, leading to new set of mid-term concerns (eg. pollution, traffic accidents), and any previous work on removing horse excrement became near-useless.


I like this post -- it seems like an accessible summary of the state of the debate so far. My opinions are already in the post, so I don’t have much to add.

Plausible cases for HRAD work, and locating the crux in the "realism about rationality" debate

This post tries to identify the possible cases for highly reliable agent design (HRAD) work to be the main priority of AI alignment. HRAD is a category of work at MIRI that aims to build a theory of intelligence and agency that can explain things like logical uncertainty and counterfactual reasoning.

The first case for HRAD work is that by becoming less confused about these phenomena, we will be able to help AGI builders predict, explain, avoid, detect, and fix safety issues and help to conceptually clarify the AI alignment problem. For this purpose, we just need _conceptual_ deconfusion -- it isn’t necessary that there must be precise equations defining what an AI system does.

The second case is that if we get a precise, mathematical theory, we can use it to build an agent that we understand “from the ground up”, rather than throwing the black box of deep learning at the problem.

The last case is that understanding how intelligence works will give us a theory that allows us to predict how _arbitrary_ agents will behave, which will be useful for AI alignment in all the ways described in the first case and <@more@>(@Theory of Ideal Agents, or of Existing Agents?@).

Looking through past discussions on the topic, the author believes that people at MIRI primarily believe in the first two cases. Meanwhile, critics (particularly me) say that it seems pretty unlikely that we can build a precise, mathematical theory, and a more conceptual but imprecise theory may help us understand reasoning better but is less likely to generalize sufficiently well to say important and non-trivial things about AI alignment for the systems we are actually building.


One of the counterarguments the podcast talks about is <@my position@>(@Conversation with Rohin Shah@) that we’ll probably learn from smaller catastrophes in order to avoid actual extinction. I just want to note that while it might sound like I disagree with Stuart on this point, I don’t think we actually do. I was arguing against the position that extinction is the default outcome (> 50% probability) while Stuart is arguing against the position that extinction is near-impossible (~0% probability). I ended up around 10%; I’d guess that if Stuart were forced to, he’d give a number similar to mine, for similar reasons as me.

The flaws that make today’s AI architecture unsafe and a new approach that could fix it

This podcast delves into many of the ideas in Stuart’s book <@Human Compatible@>(@Human Compatible: Artificial Intelligence and the Problem of Control@). Rob especially pushes on some aspects that are less talked about in the AI safety community, like the enfeeblement problem and whether we’d be locking in suboptimal values. They also discuss Stuart’s response to some counterarguments.


I am broadly sympathetic to the focus on preventing the worst outcomes and it seems plausible that extortion could play an important role in these, even though I worry more about distributional shift plus incorrigibility. Still, I am excited about the focus on cooperation, as this seems robustly useful for a wide range of scenarios and most value systems.

**Rohin's opinion:** Under a suffering-focused ethics under which s-risks far overwhelm x-risks, I think it makes sense to focus on this agenda. There don't seem to be many plausible paths to s-risks: by default, we shouldn't expect them, because it would be quite surprising for an amoral AI system to think it was particularly useful or good for humans to _suffer_, as opposed to not exist at all, and there doesn't seem to be much reason to expect an immoral AI system. Conflict and the possibility of carrying out threats are the most plausible ways by which I could see this happening, and the agenda here focuses on neglected problems in this space.

However, under other ethical systems (under which s-risks are worse than x-risks, but do not completely dwarf x-risks), I expect other technical safety research to be more impactful, because other approaches can more directly target the failure mode of an amoral AI system that doesn't care about you, which seems both more likely and more amenable to technical safety approaches (to me at least). I could imagine work on this agenda being quite important for _strategy_ research, though I am far from an expert here.

Cooperation, Conflict, and Transformative Artificial Intelligence: A Research Agenda

This agenda by the [Effective Altruism Foundation](https://ea-foundation.org/) focuses on risks of astronomical suffering (s-risks) posed by <@Transformative AI@>(@Defining and Unpacking Transformative AI@) (TAI) and especially those related to conflicts between powerful AI agents. This is because there is a very clear path from extortion and executed threats against altruistic values to s-risks.  While especially important in the context of s-risks, cooperation between AI systems is also relevant from a range of different viewpoints. The agenda covers four clusters of topics: strategy, credibility and bargaining, current AI frameworks, as well as decision theory.

The extent of cooperation failures is likely influenced by how power is distributed after the transition to TAI. At first glance, it seems like widely distributed scenarios (as <@CAIS@>(@Reframing Superintelligence: Comprehensive AI Services as General Intelligence@)) are more problematic, but related literature from international relations paints a more complicated picture. The agenda seeks a better understanding of how the distribution of power affects catastrophic risk, as well as potential levers to influence this distribution. Other topics in the strategy/governance cluster include the identification and analysis of realistic scenarios for misalignment, as well as case studies on cooperation failures in humans and how they can be affected by policy.

TAI might enable unprecedented credibility, for example by being very transparent, which is crucial for both contracts and threats. The agenda aims at better models of the effects of credibility on cooperation failures. One approach to this is open-source game theory, where agents can see other agents' source codes. Promising approaches to prevent catastrophic cooperation failures include the identification of peaceful bargaining mechanisms, as well as surrogate goals. The idea of surrogate goals is for an agent to commit to act as if it had a different goal, whenever it is threatened, in order to protect its actual goal from threats.

As some aspects of contemporary AI architectures might still be present in TAI, it can be useful to study cooperation failure in current systems. One concrete approach to enabling cooperation in social dilemmas that could be tested with contemporary systems is based on bargaining over policies combined with punishments for deviations.  Relatedly, it is worth investigating whether or not multi-agent training leads to human-like bargaining by default. This has implications on the suitability of behavioural vs classical game theory to study TAI. The behavioural game theory of human-machine interactions might also be important, especially in human-in-the-loop scenarios of TAI.

The last cluster discusses the implications of bounded computation on decision theory as well as the decision theories (implicitly) used by current agent architectures. Another focus lies on acausal reasoning and in particular the possibility of [acausal trade](https://wiki.lesswrong.com/wiki/Acausal_trade), where different correlated AI systems cooperate without any causal links between them.


Just Imitate Humans?

This post asks whether it is safe to build AI systems that just imitate humans. The comments have a lot of interesting debate.


Technical AGI safety research outside AI

This post lists 30 questions relevant to technical AI safety that could benefit from expertise outside of AI, divided into four categories: studying and understanding safety problems, solving safety problems, forecasting AI, and meta.


This post does not exhaust the possible success stories, but it gets us a lot closer to being able to look at a particular approach and ask, "Where exactly does this help us?" My guess is that most research ends up being only minimally helpful for the long run, and so I consider inquiry like this to be very useful for cause prioritization.

AI Safety "Success Stories"

It is difficult to measure the usefulness of various alignment approaches without clearly understanding what type of future they end up being useful for. This post collects "Success Stories" for AI -- disjunctive scenarios in which alignment approaches are leveraged to ensure a positive future. Whether these scenarios come to pass will depend critically on background assumptions, such as whether we can achieve global coordination, or solve the most ambitious safety issues. Mapping these success stories can help us prioritize research.


I am most excited about impact as a **regularizer** and impact as a **safety protocol**. I feel like AIs that are impact-limited at runtime (the **influence-limiter** case) are unlikely to be competitive with other AIs that have no impact penalty (this is discussed in the post). I found the argument that impact could be particularly useful for **deconfusion** uncompelling.

**Rohin's opinion**: It seems to me like the safety protocol argument is for limited actions at training time, while the influence limiter argument is for limited actions at test time. I don't really get how the regularizer is supposed to be different from these two cases -- perhaps the idea is that it is a regularizer specifically on the distribution over utility functions that the AI is optimizing? This is still confusing, I would have expected the influence limiter case to also be a change to the utility function. Like Asya, I am worried about competitiveness: see the post about [reversible changes](https://www.alignmentforum.org/posts/zrunBA8B5bmm2XZ59/reversible-changes-consider-a-bucket-of-water) below.

Four Ways An Impact Measure Could Help Alignment

Much <@recent@>(@Towards a New Impact Measure@) <@work@>(@Designing agent incentives to avoid side effects@) has focused on quantifying the effect an AI has on the world, aka measuring impact, though some are [skeptical](https://www.alignmentforum.org/posts/kCY9dYGLoThC3aG7w/best-reasons-for-pessimism-about-impact-of-impact-measures). This post presents four potential ways impact measures could help with AI alignment. _First_, impact could act as a **regularizer**: an untrained AI attempting to do value learning could have an impact penalty that prevents it from taking dangerous actions before it is confident it has learned the right utility function. _Second_, impact could act as a **safety protocol**: if our training process is dangerous, e.g. due to <@mesa optimization@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@), we can penalize impact during training to safely test models that may be misaligned. _Third_, impact could act as an **influence-limiter**: impact measures could help us construct AIs with intentionally limited scope that won’t heavily optimize the world as a side effect. _Fourth_, impact could help us with **deconfusion**: even if impact measures themselves aren't used, conceptual clarity about impact could help us gain conceptual clarity about other important concepts such as corrigibility, mild optimization, etc.


On the purposes of decision theory research

In this post, Wei Dai clarifies that he thinks decision theory research is important because it can help us learn about the nature of rationality, philosophy, and metaphilosophy; it allows us to understand potential AI failure modes; we can better understand puzzles about intelligence such as free will, logical uncertainty, counterfactuals and more; and it could improve human rationality. It is _not_ meant to find the "correct" decision theory to program into an AI, nor to create safety arguments that show that an AI system is free of "decision-theoretic" flaws.


I strongly agree with the points made in this post. I don't know to what extent researchers themselves agree with this point -- it seems like there is _a lot_ of adversarial examples research that is looking at the imperceptible perturbation case and many papers that talk about new types of adversarial examples, without really explaining why they are doing this or giving a motivation that is about unsolved research problems rather than real world settings. It's possible that researchers do think of it as a research problem and not a real world problem, but present their papers differently because they think that's necessary in order to be accepted.

The distinction between research problems and real world threat models seem to parallel the distinction between theoretical or conceptual research and engineering in AI safety. The former typically asks questions of the form "how could we do this in principle, making simplifying assumptions X, Y and Z", even though X, Y and Z are known not to hold in the real world, for the sake of having greater conceptual clarity that can later be leveraged as a solution to a real world problem. Engineering work on the other hand is typically trying to scale an approach to a more complex environment (with the eventual goal of getting to a real world problem).

Unsolved research problems vs. real-world threat models

Papers on adversarial examples often suggest that adversarial examples can lead to real world problems as their motivation. As we've <@seen@>(@Motivating the Rules of the Game for Adversarial Example Research@) <@previously@>(@Introducing the Unrestricted Adversarial Examples Challenge@), many adversarial example settings are not very realistic _threat models_ for any real world problem. For example, adversarial "stickers" that cause vision models to fail to recognize stop signs could cause an autonomous vehicle to crash... but an adversary could also just knock over the stop sign if that was their goal.

There are more compelling reasons that we might care about imperceptible perturbation adversarial examples. First, they are a proof of concept, demonstrating that our ML models are not robust and make "obvious" mistakes and so cannot be relied on. Second, they form an unsolved research problem, in which progress can be made more easily than in real settings, because it can be formalized straightforwardly (unlike realistic settings). As progress is made in this toy domain, it can be used to inform new paradigms that are closer to realistic settings. But it is _not_ meant to mimic real world settings -- in the real world, you need a threat model of what problems can arise from the outside world, which will likely suggest much more basic concerns than the "research problems", requiring solutions involving sweeping design changes rather than small fixes.


I broadly agree with the areas of AI progress identified here, though I would probably also throw in NLP, e.g. [BERT](https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html). I disagree on the details -- for example, I think that [OpenAI Five](https://blog.openai.com/openai-five/) ([AN #13](https://mailchi.mp/8234356e4b7f/alignment-newsletter-13)) was much better than I would have expected at the time and the same would have been true of AlphaStar if I hadn't already seen OpenAI Five, and the fact that they did a few things that humans can't do barely diminishes the achievement at all. (My take is pretty similar to Alex Irpan's take in his [post on AlphaStar](https://www.alexirpan.com/2019/02/22/alphastar.html).)

FLI Podcast: AI Breakthroughs and Challenges in 2018 with David Krueger and Roman Yampolskiy

Ariel Conn, David Krueger and Roman Yampolskiy

David and Roman review AI progress in 2018 and speculate about its implications. Roman identified a pattern where we see breakthroughs like [AlphaZero](https://deepmind.com/blog/alphazero-shedding-new-light-grand-games-chess-shogi-and-go/) ([AN #36](https://mailchi.mp/6751e45fbb48/alignment-newsletter-36)), [AlphaStar](https://deepmind.com/blog/alphastar-mastering-real-time-strategy-game-starcraft-ii/) ([AN #43](https://mailchi.mp/768a8130013f/alignment-newsletter-43)) and [AlphaFold](https://deepmind.com/blog/alphafold/) ([AN #36](https://mailchi.mp/6751e45fbb48/alignment-newsletter-36)) so frequently now that it no longer seems as impressive when a new one comes out. David on the other hand sounded less impressed by progress on Dota and StarCraft, since both AI systems were capable of executing actions that humans could never do (fast reaction times for Dota and high actions-per-minute for StarCraft). He also thought that these projects didn't result in any clear general algorithmic insights the way AlphaZero did.

On the deep RL + robotics side, David identified major progress in [Dactyl](https://blog.openai.com/learning-dexterity/) ([AN #18](https://mailchi.mp/51717855ea27/alignment-newsletter-18)) and [QT-Opt](https://ai.googleblog.com/2018/06/scalable-deep-reinforcement-learning.html) (which I remember reading and liking but apparently I failed to put in the newsletter). He also cited GANs as having improved significantly, and talked about feature-wise transformations in particular. Roman noted the improving performance of evolutionary algorithms.

David also noted how a lot of results were obtained by creating algorithms that could scale, and then using a huge amount of compute for them, quoting [AI and Compute](https://blog.openai.com/ai-and-compute/) ([AN #7](https://mailchi.mp/3e550712419a/alignment-newsletter-7)), [Interpreting AI Compute Trends](https://aiimpacts.org/interpreting-ai-compute-trends/) ([AN #15](https://mailchi.mp/4920e52dd61b/alignment-newsletter-15)) and [Reinterpreting AI and Compute](https://aiimpacts.org/reinterpreting-ai-and-compute/) ([AN #38](https://mailchi.mp/588354e4b91d/alignment-newsletter-38)).

On the policy side, they talked about deep fakes and the general trend that AI may be progressing to fast for us to keep up with its security implications. They do find it promising that researchers are beginning to accept that their research does have safety and security implications.

On the safety side, David noted that the main advance seemed to be with approaches using [superhuman feedback](https://www.lesswrong.com/posts/naccwaCQEEBXK7hiJ/my-use-of-the-phrase-super-human-feedback), including [debate](https://blog.openai.com/debate/) ([AN #5](https://mailchi.mp/0ae5d69de63b/alignment-newsletter-5)), [iterated amplification](https://blog.openai.com/amplifying-ai-training/) (discussed frequently in this newsletter, but that paper was in [AN #30](https://mailchi.mp/c1f376f3a12e/alignment-newsletter-30)) and [recursive reward modeling](https://medium.com/@deepmindsafetyresearch/scalable-agent-alignment-via-reward-modeling-bf4ab06dfd84) ([AN #34](https://mailchi.mp/f1947668b183/alignment-newsletter-34)). He also identified [unrestricted adversarial examples](https://ai.googleblog.com/2018/09/introducing-unrestricted-adversarial.html) ([AN #24](https://mailchi.mp/d7b5059d64ed/alignment-newsletter-24)) as an area to watch in the future.


Treacherous Turn, Simulations and Brain-Computer Interfaces


This is probably most useful for understanding the motivations behind many of Dylan's papers and how they all tie into each other, which can be hard to glean just from reading the papers. There were also a lot of framings of problems that felt useful to me that I haven't seen elsewhere.

Inverse Reinforcement Learning and Inferring Human Preference with Dylan Hadfield-Menell

A few weeks ago, Lucas Perry interviewed Dylan Hadfield-Menell on the FLI podcast about his research (which includes papers like [Cooperative Inverse Reinforcement Learning](https://arxiv.org/abs/1606.03137),  [The Off-Switch Game](https://arxiv.org/abs/1611.08219), and [Inverse Reward Design](https://arxiv.org/abs/1711.02827)). They discussed a variety of topics including the motivations behind Dylan's research, future directions, thoughts on hard problems such as corrigibility and preference aggregation, etc.


I generally agree with these arguments and the general viewpoint that we will probably solve alignment in this narrow sense. The most compelling argument to me is the second one, that we will eventually devote significant resources to the problem. This does depend on the crux that we see examples of these problems and how they could be dangerous before it is too late.

I also agree that it's much less clear whether we will solve other related problems, such as how to deal with malicious uses of AI, issues that arise when multiple superintelligent AI systems aligned with different humans start to compete, and how to ensure that humans have "good" values. I don't know if this implies that _on the margin_ it is more useful to work on the related problems. It could be that these problems are so hard that there is not much that we can do. (I'm neglecting [importance of the problem](https://concepts.effectivealtruism.org/concepts/importance-neglectedness-tractability/) here.)

Why I expect successful alignment

This post gives three arguments that we will likely solve the narrow alignment problem of having an AI system do what its operators intend it to do. First, advanced AI systems may be developed in such a way that the alignment problem doesn't even happen, at least as we currently conceive of it. For example, under the comprehensive AI services model, there are many different AI services that are superintelligent at particular tasks that can work together to accomplish complex goals, but there isn't a single unified agent to "align". Second, if it becomes obvious that alignment will be a serious problem, then we will devote a lot of resources to tackling the problem. We already see reward hacking in current systems, but it isn't sufficiently dangerous yet to merit the application of a lot of resources. Third, we have already come up with some decent approaches that seem like they could work.


This post concisely conveys a number of Vika and Jan's views, albeit without explanations for most of them. I'd encourage other safety researchers to do the same exercise, with a view to fleshing out the cruxes behind whatever disagreements come up.

Discussion on the machine learning approach to AI safety

This blog post (based on a talk at EA Global London) discusses whether current work on the machine learning approach to AI safety will remain relevant in the face of potential paradigmatic changes in ML systems. Vika and Jan rate how much they rely on each assumptions in a list drawn from [this blog post by Jon Gauthier] (http://www.foldl.me/2018/conceptual-issues-ai-safety-paradigmatic-gap/) ([AN #13](https://mailchi.mp/8234356e4b7f/alignment-newsletter-13)), and how likely each assumptions is to hold up over time. They also evaluate arguments for human-in-the-loop approaches versus problem-specific approaches.


I think it’s important to take this post as an explication of MIRI’s mindset, not as an argument for that mindset. In the former role, it’s excellent: the analogy is a fitting one in many ways. It’s worth noting, though, that the idea of only having one shot at success seems like an important component, but isn’t made explicit. Also, it’d be nice to have more clarity about the “approximately 0% chance of success” without advances in agent foundations - maybe that credence is justified under a specific model of what’s needed for AI alignment, but does it take into account model uncertainty?

The Rocket Alignment Problem

Eliezer explains the motivations behind MIRI’s work using an analogy between aligning AI and designing a rocket that can get to the moon. He portrays our current theoretical understanding of intelligence as having massive conceptual holes; MIRI is trying to clarify these fundamental confusions. Although there’s not yet any clear path from these sorts of advances to building an aligned AI, Eliezer estimates our chances of success without them as basically 0%: it’s like somebody who doesn’t understand calculus building a rocket with the intention of manually steering it on the way up.


My basic opinion is that this makes sense and agrees with my model. On the other hand, I'm not planning to switch to working on decision theory now, so perhaps I should say why. Partly it's that I have a comparative advantage at ML work, but it's also an impression that Agent Foundations will not help much with the first powerful AI systems we build. On one axis, I wouldn't be surprised if the first powerful AI systems don't look like the good reasoners that MIRI studies, and so Agent Foundations research won't apply. On another axis, Agent Foundations seems like a hard problem that we may not solve before powerful AI systems are created. I do find it plausible that to build _aligned_ AI systems that are _much_ more powerful than humans, we must understand it at the level of Agent Foundations understanding. (Though I also find the opposite statement plausible.) However, I think we will first build powerful AI systems that are not that much more powerful than humans, and that direct alignment of ML techniques will be sufficient to make that safe (even though they do pose an x-risk). (I suspect this is where my main disagreement with people at MIRI is.) We can then use those systems to help us solve Agent Foundations before we scale up.

Comment on decision theory

MIRI works on Agent Foundations because AGIs should be good reasoners, and we're currently confused about how to have good reasoning, and work on logical uncertainty and decision theory should help us resolve this confusion. If we don't resolve the confusion, it will be significantly harder to build AGI in a way that is clean, understandable and interpretable, and as a result it will be harder to understand what is happening and to fix it if anything goes wrong. This is analogous to how it seems really useful to understand Newton's law of gravitation before you try to build rockets, even though the work of figuring out Newton's law is very different from the rocket-building work.