I enjoyed reading this book; I don't usually get to read a single person's overall high-level view on the state of AI, how it could have societal impact, the argument for AI risk, potential solutions, and the need for AI governance. It's nice to see all of these areas I think about tied together into a single coherent view. While I agree with much of the book, especially the conceptual switch from the standard model of intelligent machines to Stuart's model of beneficial machines, I'm going to focus on disagreements in this opinion.

First, the book has an implied stance towards the future of AI research that I don't agree with: I could imagine that powerful AI systems end up being created by learning alone without needing the conceptual breakthroughs that Stuart outlines. This has been proposed in e.g. <@AI-GAs@>(@AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence@)), and seems to be the implicit belief that drives OpenAI and DeepMind's research agendas. This leads to differences in risk analysis and solutions: for example, the <@inner alignment problem@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@) only applies to agents arising from learning algorithms, and I suspect would not apply to Stuart's view of AI progress.

The book also gives the impression that to solve AI safety, we simply need to make sure that AI systems are optimizing the right objective, at least in the case where there is a single human and a single robot. Again, depending on how future AI systems work, that could be true, but I expect there will be other problems that need to be solved as well. I've already mentioned inner alignment; other graduate students at CHAI work on e.g. [robustness](https://adversarialpolicies.github.io/) and transparency.

The proposal for aligning AI requires us to build a model that relates human preferences to human behavior. This sounds extremely hard to get completely right. Of course, we may not need a model that is completely right: since reward uncertainty makes the agent amenable to shutdowns, it seems plausible that we can correct mistakes in the model as they come up. But it's not obvious to me that this is sufficient.

The sections on multiple humans are much more speculative and I have more disagreements there, but I expect that is simply because we haven't done enough research yet. For example, HC worries that we won't be able to use laws to prevent AIs from doing technically legal but still antisocial things for the benefit of a single human. This seems true if you imagine that a single human suddenly gets access to a superintelligent AI, but when everyone has a superintelligent AI, then the current system where humans socially penalize each other for norm violations may scale up naturally. The overall effect depends on whether AI makes it easier to violate norms, or to detect and punish norm violations.

Human Compatible: Artificial Intelligence and the Problem of Control

_Since I am aiming this summary for people who are already familiar with AI safety, my summary is substantially reorganized from the book, and skips large portions of the book that I expect will be less useful for this audience. If you are not familiar with AI safety, **note that I am skipping many arguments and counterarguments in the book that are aimed for you**. I'll refer to the book as "HC" in this newsletter._

Before we get into details of impacts and solutions to the problem of AI safety, it's important to have a model of how AI development will happen. Many estimates have been made by figuring out the amount of compute needed to run a human brain, and figuring out how long it will be until we get there. HC doesn't agree with these; it suggests the bottleneck for AI is in the algorithms rather than the hardware. We will need several conceptual breakthroughs, for example in language or common sense understanding, cumulative learning (the analog of cultural accumulation for humans), discovering hierarchy, and managing mental activity (that is, the metacognition needed to prioritize what to think about next). It's not clear how long these will take, and whether there will need to be more breakthroughs after these occur, but these seem like necessary ones.

What could happen if we do get beneficial superintelligent AI? While there is a lot of sci-fi speculation that we could do here, as a weak lower bound, it should at least be able to automate away almost all existing human labor. Assuming that superintelligent AI is very cheap, most services and many goods would become extremely cheap. Even many primary products such as food and natural resources would become cheaper, as human labor is still a significant fraction of their production cost. If we assume that this could bring up everyone's standard of life up to that of the 88th percentile American, that would result in nearly a _tenfold_ increase in world GDP per year. Assuming a 5% discount rate per year, this corresponds to $13.5 _quadrillion_ net present value. Such a giant prize removes many reasons for conflict, and should encourage everyone to cooperate to ensure we all get to keep this prize.

Of course, this doesn't mean that there aren't any problems, even with AI that does what its owner wants. Depending on who has access to powerful AI systems, we could see a rise in automated surveillance, lethal autonomous weapons, automated blackmail, fake news and behavior manipulation. Another issue that could come up is that once AI is better than humans at all tasks, we may end up delegating everything to AI, and lose autonomy, leading to _human enfeeblement_.

This all assumes that we are able to control AI. However, we should be cautious about such an endeavor -- if nothing else, we should be careful about creating entities that are more intelligent than us. After all, the gorillas probably aren't too happy about the fact that their habitat, happiness, and existence depends on our moods and whims. For this reason, HC calls this the _gorilla problem_: specifically, "the problem of whether humans can maintain their supremacy and autonomy in a world that includes machines with substantially greater intelligence". Of course, we aren't in the same position as the gorillas: we get to _design_ the more intelligent "species". But we should probably have some good arguments explaining why our design isn't going to succumb to the gorilla problem. This is especially important in the case of a fast intelligence explosion, or _hard takeoff_, because in that scenario we do not get any time to react and solve any problems that arise.

Do we have such an argument right now? Not really, and in fact there's an argument that we _will_ succumb to the gorilla problem. The vast majority of research in AI and related fields assumes that there is some definite, known _specification_ or _objective_ that must be optimized. In RL, we optimize the _reward function_; in search, we look for states matching a _goal criterion_; in statistics, we minimize _expected loss_; in control theory, we minimize the _cost function_ (typically deviation from some desired behavior); in economics, we design mechanisms and policies to maximize the _utility_ of individuals, _welfare_ of groups, or _profit_ of corporations. This leads HC to propose the following standard model of machine intelligence: _Machines are intelligent to the extent that their actions can be expected to achieve their objectives._ However, if we put in the wrong objective, the machine's obstinate pursuit of that objective would lead to outcomes we won't like.

Consider for example the content selection algorithms used by social media, typically maximizing some measure of engagement, like click-through. Despite their lack of intelligence, such algorithms end up changing the user's preference so that they become more predictable, since more predictable users can be given items they are more likely to click on. In practice, this means that users are pushed to become more extreme in their political views. Arguably, these algorithms have already caused much damage to the world.

So the problem is that we don't know how to put our objectives inside of the AI system so that when it optimizes its objective, the results are good for us. Stuart calls this the "King Midas" problem: as the legend goes, King Midas wished that everything he touched would turn to gold, not realizing that "everything" included his daughter and his food, a classic case of a <@badly specified objective@>(@Specification gaming examples in AI@). In some sense, we've known about this problem for a long time, both from King Midas's tale, and in stories about genies, where the characters inevitably want to undo their wishes.

You might think that we could simply turn off the power to the AI, but that won't work, because for almost any definite goal, the AI has an incentive to stay operational, just because that is necessary for it to achieve its goal. This is captured in what may be Stuart's most famous quote: _you can't fetch the coffee if you're dead_. This is one of a few worrisome [convergent instrumental subgoals](https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf).

What went wrong? The problem was the way we evaluated machine intelligence, which doesn't take into account the fact that machines should be useful for _us_. HC proposes: _Machines are **beneficial** to the extent that **their** actions can be expected to achieve **our** objectives_. But with this definition, instead of our AI systems optimizing a definite, wrong objective, they will _also_ be uncertain about the objective, since we ourselves don't know what our objectives are. HC expands on this by proposing three principles for the design of AI systems, that I'll quote here in full:

1. _The machine’s only objective is to maximize the realization of human preferences._
2. _The machine is initially uncertain about what those preferences are._
3. _The ultimate source of information about human preferences is human behavior._

[Cooperative Inverse Reinforcement Learning](https://arxiv.org/abs/1606.03137) provides a formal model of an _assistance game_ that showcases these principles. You might worry that an AI system that is uncertain about its objective will not be as useful as one that knows the objective, but actually this uncertainty is a feature, not a bug: it leads to AI systems that are deferential, that ask for clarifying information, and that try to learn human preferences. [The Off-Switch Game](https://arxiv.org/abs/1611.08219) shows that because the AI is uncertain about the reward, it will let itself be shut off. These papers are discussed later in this newsletter.

So that's the proposed solution. You might worry that the proposed solution is quite challenging: after all, it requires a shift in the entire way we do AI. What if the standard model of AI can deliver more results, even if just because more people work on it? Here, HC is optimistic: the big issue with the standard model is that it is not very good at learning our preferences, and there's a huge economic pressure to learn preferences. For example, I would pay a lot of money for an AI assistant that accurately learns my preferences for meeting times, and schedules them completely autonomously.

Another research challenge is how to actually put principle 3 into practice: it requires us to connect human behavior to human preferences. [Inverse Reward Design](https://arxiv.org/abs/1711.02827) and <@Preferences Implicit in the State of the World@>(@Learning Preferences by Looking at the World@) are example papers that tackle portions of this. However, there are _lots_ of subtleties in this connection. We need to use _Gricean semantics_ for language: when we say X, we do not mean the literal meaning of X: the agent must also take into account the fact that we bothered to say X, and that we didn't say Y. For example, I'm only going to ask for the agent to buy a cup of coffee if I believe that there is a place to buy reasonably priced coffee nearby. If those beliefs happen to be wrong, the agent should ask for clarification, rather than trudge hundreds of miles or pay hundreds of dollars to ensure I get my cup of coffee.

Another problem with inferring preferences from behavior is that humans are nearly always in some deeply nested plan, and many actions don't even occur to us. Right now I'm writing this summary, and not considering whether I should become a fireman. I'm not writing this summary because I just ran a calculation showing that this would best achieve my preferences, I'm doing it because it's a subpart of the overall plan of writing this bonus newsletter, which itself is a subpart of other plans. The connection to my preferences is very far up. How do we deal with that fact?

There are perhaps more fundamental challenges with the notion of "preferences" itself. For example, our _experiencing self_ and our _remembering self_ may have different preferences -- if so, which one should our agent optimize for? In addition, our preferences often change over time: should our agent optimize for our current preferences, even if it knows that they will predictably change in the future? This one could potentially be solved by learning _meta-preferences_ that dictate what kinds of preference change processes are acceptable.

All of these issues suggest that we need work across many fields (such as AI, cognitive science, psychology, and neuroscience) to reverse-engineer human cognition, so that we can put principle 3 into action and create a model that shows how human behavior arises from human preferences.

So far, we've been talking about the case with a single human. But of course, there are going to be multiple humans: how do we deal with that? As a baseline, we could imagine that every human gets their own agent that optimizes for their preferences. However, this will differentially benefit people who care less about other people's welfare, since their agents have access to many potential plans that wouldn't be available to an agent for someone who cared about other people. For example, if Harriet was going to be late for a meeting with Ivan, her AI agent might arrange for Ivan to be even later.

What if we had laws that prevented AI systems from acting in such antisocial ways? It seems likely that superintelligent AI would be able to find loopholes in such laws, so that they do things that are strictly legal but still antisocial, e.g. line-cutting. (This problem is similar to the problem that we can't just write down what we want and have AI optimize it.)

What if we made our AI systems utilitarian (assuming we figured out some acceptable method of comparing utilities across people)? Then we get the "Somalia problem": agents will end up going to Somalia to help the worse-off people there, and so no one would ever buy such an agent.

Overall, it's not obvious how we deal with the transition from a single human to multiple humans. While HC focuses on a potential solution for the single human / single agent case, there is still much more to be said and done to account for the impact of AI on all of humanity. To quote HC, "There is really no analog in our present world to the relationship we will have with beneficial intelligent machines in the future. It remains to be seen how the endgame turns out."


AI Alignment Podcast: Human Compatible: Artificial Intelligence and the Problem of Control

This podcast covers some of the main ideas from the book, which I'll ignore for this summary. It also talks a bit about the motivations for the book. Stuart has three audiences in mind. He wants to explain to laypeople what AI is and why it matters. He wants to convince AI researchers that they should be working in this new model of beneficial AI that optimizes for our objectives, rather than the standard model of intelligent AI that optimizes for its objectives. Finally, he wants to recruit academics in other fields to help connect human behavior to human preferences (principle 3), as well as to figure out how to deal with multiple humans.

Stuart also points out that his book has two main differences from Superintelligence and Life 3.0: first, his book explains how existing AI techniques work (and in particular it explains the standard model), and second, it proposes a technical solution to the problem (the three principles).


Alignment Newsletter One Year Retrospective

The Alignment Newsletter is one year old! I've written a retrospective of the newsletter's impact over the last year, with a lot of open questions about what the newsletter should look like in the future. Please help me figure out the answers by *taking this 3-minute [survey](https://docs.google.com/forms/d/e/1FAIpQLSepm8ATtun9pvLGL9MXhk6Z59hO14huEVS6qgqtl3KS6n5SIQ/viewform?usp=sf_link)*, and if you're feeling particularly generous with your time, read the retrospective and tell me your opinions in the comments!


The exponential blowup in potential loss with multiple actions would make this prohibitive, but of course you could instead view the full sequence of actions (i.e. trajectory) as a mega-action, and quantilize over this mega-action. In this case, a one-millionth-quantilizer could choose from among the million best plans that a human would make (assuming a well-specified U), and any unintended consequences (that were intentionally chosen by the quantilizer) would have to be ones that a human had a one-in-a-million chance of causing to occur, which quite plausibly excludes really bad outcomes.

Phrased this way, quantilization feels like an amplification of a human policy. Unlike the amplification in iterated amplification, it does _not_ try to preserve alignment, it simply tries to bound how far away from alignment the resulting policy can diverge. As a result, you can't iterate quantilization to get arbitrarily good capabilities. You might hope that humans could learn from powerful AI systems, grow more capable themselves (while remaining as safe as they were before), and then the next quantilizers would be more powerful.

It's worth noting that the theorem in the paper shows that, to the extent that you think quantilization is insufficient for AI alignment, you need to make some other assumption, or find some other source of information, in order to do better, since quantilization is optimal for its particular setup. For example, you could try to assume that U is at least somewhat reasonable and not pathologically bad; or you could assume an interactive setting where the human can notice and correct for any issues with the U-maximizing plan before it is executed; or you could not have U at all and exceed human performance through some other technique.

I'm not very worried about the issue that quantilization could forgo benefits that the human policy had. It seems that even if this happens, we could notice this, turn off the quantilizer, and fix the utility function U so that it no longer ignores those benefits. (We wouldn't be able to prevent the quantilizer from forgoing benefits of our policy that we didn't know about, but that seems okay to me.)

[Quantilizers: A Safer Alternative to Maximizers for Limited Optimization](https://intelligence.org/files/QuantilizersSaferAlternative.pdf) and [When to use quantilization](https://www.alignmentforum.org/posts/Rs6vZCrnQFWQ4p37P/when-to-use-quantilization)

A key worry with AI alignment is that if we maximize expected utility for some utility function chosen by hand, we will likely get unintended side effects that score highly by the utility function but are nevertheless not what we intended. We might hope to leverage human feedback to solve this: in particular, an AI system that simply mimics human actions would often be desirable. However, mimicry can only achieve human performance, and cannot improve upon it. The first link is a 2015 paper that introduces quantilization, which interpolates between these two extremes to improve upon human performance while bounding the potential (expected) loss from unintended side effects.

In particular, let's suppose that humans have some policy γ (i.e. probability distribution over actions). We evaluate utility or performance using a utility function U, but we _do not assume_ it is well-specified -- U can be any function, including one we would not want to maximize. Our goal is to design a policy π that gets higher expected U than γ (reflecting our hope that U measures utility well) without doing too much worse than γ in the worst case when U was as badly designed as possible. We'll consider a one-shot case: π is used to select an action once, and then the game is over.

The core idea behind quantilization is simple: if our policy only does things that the human might have done, any expected loss it incurs corresponds to some loss that the human could incur. So, let's take our human policy γ, keep only the top q-fraction of γ (as evaluated by U), and then sample an action from there. This defines our policy π_q, also called a q-quantilizer. For example, suppose the human would choose A with probability 0.25, B with probability 0.5, and C with probability 0.25, and U(A) > U(B) > U(C). Then a (1/4)-quantilizer would choose A with certainty, a (1/2)-quantilizer would choose randomly between A and B, and a (3/8)-quantilizer would choose A twice as often as B.

Note that even if π_q places all of its probability on the worst possible actions (because U was badly designed), by construction the human had probability q of doing the same thing as π_q, and so the expected cost of π_q can be at most (1/q) times larger than the expected cost of the human policy γ. In fact, if we have no other knowledge, a q-quantilizer maximizes expected utility U subject to the constraint of never doing worse than (1/q) times as bad as γ.

As soon as you move to the setting with multiple actions, if you choose each action using quantilization, then your worst case bound is exponential in the number of actions. If you assume the cost for each action is independent, you recover the guarantees, but this is not a realistic assumption (as the authors note). Long-term plans are very good or very bad _because_ all of the actions build on each other to achieve some goal, so the costs are not going to be independent.

The second link is a recent post that further analyzes quantilization. It points out that the theoretical analysis in the paper assumes that deviating from the human policy can only lead to costs. If the deviation could also forgo benefits, then the theorems no longer apply. Perhaps the bottom (1-q) of actions actually have strong positive benefits, that we failed to model in U. Then by eliminating those actions, we may have lost arbitrary amounts of expected value. In these situations, the only way to bound the expected regret is by exact mimicry. It also points out that if you are aiming to simultaneously do well both on U and the worst-case bound, then typically imitation will be better since adding any optimization can drastically weaken the worst-case bound and usually will not make U better by the same amount. Quantilization makes sense when there is a "sweet-spot of actions that are pretty common but substantially outperform imitation".


This post, and its accompanying paper, seems very sensible to me. While I have some doubts about how informative human debate data will be about superhuman debaters, it certainly seems worth trying to gain more empirical information. Note that while the paper primarily discusses Debate, I think that many of its arguments are applicable to any human-in-the-loop safety methods (and probably others too). Currently I think Ought is the safety group focusing most on collecting human data, but I look forward to seeing other researchers doing so.

AI Safety Needs Social Scientists

One approach to AI safety is to "ask humans a large number of questions about what they want, train an ML model of their values, and optimize the AI system to do well according to the learned values". However, humans give answers that are limited, biased and often in disagreement with each other, and so AI safety needs social scientists to figure out how to improve this data - which eventually may be gathered from thousands or millions of people. Of particular importance is the ability to design rigorous experiments, drawing from an interdisciplinary understanding of human cognition and behaviour. The authors discuss [Debate](https://blog.openai.com/debate/) ([AN #5](https://mailchi.mp/0ae5d69de63b/alignment-newsletter-5)) as a case study of a safety technique whose success depends on empirical questions such as: how skilled are humans as judges by default? Can we train people to be better judges? Are there ways to restrict debate to make it easier to judge?

There are a couple of key premises underlying this argument. The first is that, despite human biases, there are correct answers to questions about human values - perhaps defined as the answer we would endorse if given all relevant information and unlimited time to think. However, it’s not necessary for AIs to always find those answers, as long as they are able to recognise cases in which they’re uncertain and do nothing (while there are some cases in which inaction can cause harm, such as a self-driving car ceasing to steer mid-journey, it seems that the most worrying long-term catastrophes can be avoided by inaction). Another reason for optimism is that even incomplete or negative results from social science experiments may be useful in informing technical safety research going forward. However, in some cases the systems we're trying to reason about are very different from anything we can test now - for example, AI debaters that are much stronger than humans.


I don't think that the two methods are very different, and I expect that if you can control for "optimization power", the two methods would be about equally susceptible to Goodhart's Law. (In any given experiment, one will be better than the other, for reasons that depend on the experiment, but averaged across experiments I don't expect to see a clear winner.) However, I do think that gradient descent is very powerful at optimization, and it's hard to imagine the astronomically large random search that would compare with it, and so in any practical application gradient descent will lead to more Goodharting (and more overfitting) than random search. (It will also perform better, since it won't underfit, as random search would.)

One of the answers to this question talks about some experimental evidence, where they find that they can get different results with a relatively minor change to the experimental procedure, which I think is weak evidence for this position.

How does Gradient Descent Interact with Goodhart?

Scott often thinks about optimization using a simple proxy of "sample N points and choose the one with the highest value", where larger N corresponds to more powerful optimization. However, this seems to be a poor model for what gradient descent actually does, and it seems valuable to understand the difference (or to find out that there isn't any significant difference). A particularly interesting subquestion is whether [Goodhart's Law](https://www.alignmentforum.org/posts/EbFABnst8LsidYs5Y/goodhart-taxonomy) behaves differently for gradient descent vs. random search.


I don't currently think very much about timelines, intelligence explosions, and other questions that AI Impacts thinks about, but it seems very plausible to me that these could be extremely important. (I do think about discontinuities in progress and am very glad I read the [AI Impacts post](https://aiimpacts.org/likelihood-of-discontinuous-progress-around-the-development-of-agi/) on the subject.) One point that the interview brings up is that there are very few (perhaps two?) full time equivalents working on predicting the future of AI, while there are many people working on technical AI safety, so the former is more neglected. I'm not sure I agree with this -- the number of full time equivalents doing technical AI alignment research seems quite small (on the order of 50 people). However, I do see _many_ people who are trying to skill up so that they can do technical AI alignment research, and none who want to do better prediction, and that seems clearly wrong. I would guess that there are several readers of this newsletter who want to do technical AI alignment research, but who would have more impact if they worked in an adjacent area, such as prediction as at AI Impacts, or policy and strategy work, or in better tools and communication. Even though I'm well-placed to do technical research, I still think that common knowledge of research is a big enough bottleneck that I spend a lot of time on this newsletter. It seems likely that there is someone else who would do a better job than me, but who is set on technical safety research even though they wouldn't be as good. So I guess if you are still trying to figure out how to best help with AI alignment, or are about to start training up to do technical research, please do listen to this podcast and consider that alternative route, and various others as well. The goal is not to figure out which question is the most important, so that you can try to solve it. You'll likely do better by considering the field as a whole, and asking which area you would be in if someone optimally assigned people in the field to tasks.

80K podcast with Katja Grace

Rob Wiblin interviewed Katja Grace of AI Impacts about her work predicting the future of AI. My main takeaway was that there are many important questions in this space that almost no one is trying to answer, and that we haven't made a good enough attempt yet to conclude that it's too hard to do, so we should put more time into it. If you haven't seen AI Impacts' work before, you can get some of the most interesting results (at a high level) from listening to this podcast. There's a ton of detail in the podcast -- too much for me to summarize here.


I especially liked this series for the argument that 2% economic growth very likely cannot last much longer, providing quite a strong argument for the importance of this century, without relying at all on controversial facts about AI. At least personally I was previously uneasy about how “grand” or “extraordinary” AGI claims tend to be, and whether I should be far more skeptical of them as a result. I feel significantly more comfortable with these claims after seeing this argument.

Note though that it does not defuse all such uneasiness -- you can still look at how early we appear to be (given the billions of years of civilization that could remain in the future), and conclude that the simulation hypothesis is true, or that there is a Great Filter in our future that will drive us extinct with near-certainty. In such situations there would be no extraordinary impact to be had today by working on AI risk.

The "most important century" series

In some sense, it is really weird for us to claim that there is a non-trivial chance that in the near future, we might build [transformative AI](https://www.openphilanthropy.org/blog/some-background-our-views-regarding-advanced-artificial-intelligence) and either (1) go extinct or (2) exceed a growth rate of (say) 100% per year. It feels like an extraordinary claim, and thus should require extraordinary evidence. One way of cashing this out: if the claim were true, this century would be the most important century, with the most opportunity for individuals to have an impact. Given the sheer number of centuries there are, this is an extraordinary claim; it should really have extraordinary evidence. This series argues that while the claim does seem extraordinary, _all_ views seem extraordinary -- there isn’t some default baseline view that is “ordinary” to which we should be assigning most of our probability.

Specifically, consider three possibilities for the long-run future:
1. **Radical:** We will have a productivity explosion by 2100, which will enable us to become technologically mature. Think of a civilization that sends spacecraft throughout the galaxy, builds permanent settlements on other planets, harvests large fractions of the energy output from stars, etc.
2. **Conservative:** We get to a technologically mature civilization, but it takes hundreds or thousands of years. Let’s say even 100,000 years to be ultra conservative.
3. **Skeptical:** We never become technologically mature for some reason. Perhaps we run into fundamental technological limits, or we choose not to expand into the galaxy, or we’re in a simulation, etc.
It’s pretty clear why the radical view is extraordinary. What about the other two?

The conservative view implies that we are currently in the most important 100,000-year period. Given that life is billions of years old, and would presumably continue for billions of years to come once we reach a stable galaxy-wide civilization, that would make this the most important 100,000 year period out of tens of thousands of such periods. Thus the conservative view is also extraordinary, for the same reason that the radical view is extraordinary (albeit it is perhaps only half as extraordinary as the radical view).

The skeptical view by itself does not seem obviously extraordinary. However, while you could assign 70% probability to the skeptical view, it seems unreasonable to assign 99% probability to such a view -- that suggests some very strong or confident claims about what prevents us from colonizing the galaxy, which we probably shouldn’t have given our current knowledge. So, we need to have a non-trivial chunk of probability on the other views, which still opens us up to critique of having extraordinary claims.

Okay, so we’ve established that we should at least be willing to say something as extreme as “there’s a non-trivial chance we’re in the most important 100,000-year period”. Can we tighten the argument, to talk about the most important _century_? In fact, we can, by looking at the economic growth rate.

You are probably aware that the US economy grows around 2-3% per year (after adjusting for inflation), so a business-as-usual, non-crazy, default view might be to expect this to continue. You are probably also aware that exponential growth can grow _very_ quickly. At the lower end of 2% per year, the economy would double every ~35 years. If this continued for 8200 years, **we'd need to be sustaining multiple economies as big as today's entire world economy _per atom in the galaxy_**. While this is not a priori impossible, it seems quite unlikely to happen. This suggests that we’re in one of fewer than 82 centuries that will have growth rates at 2% or larger, making it far less “extraordinary” to claim that we’re in the most important one, especially if you believe that growth rates are well correlated with change and ability to have impact.

The actual radical view that the author places non-trivial probability on is one we’ve seen before in this newsletter: it is one in which there is  automation of science and technology through advanced AI or whole brain emulations or other possibilities. This allows technology to substitute for human labor in the economy, which produces a positive feedback loop as the output of the economy is ploughed back into the economy creating superexponential growth and a “productivity explosion”, where the growth rate increases far _beyond_ 2%. The series summarizes and connects together <@many@>(@Modeling the Human Trajectory@), <@past@>(@Could Advanced AI Drive Explosive Economic Growth?@), <@Open@>(@Draft report on AI timelines@), <@Phil@>(@How Much Computational Power It Takes to Match the Human Brain@) <@analyses@>(@Semi-informative priors over AI timelines@), which I won't be summarizing here (since we've summarized these analyses previously). While this is a more specific and “extraordinary” claim than even the claim that we live in the most important century, it seems like it should not be seen as so extraordinary given the arguments above.

This series also argues for a few other points important to longtermism, which I’ll copy here:
1. **The long-run future is radically unfamiliar.** Enough advances in technology could lead to a long-lasting, galaxy-wide civilization that could be a radical utopia, dystopia, or anything in between.
2. **The long-run future could come much faster than we think**, due to a possible AI-driven productivity explosion. (I briefly mentioned this above, but the full series devotes much more space and many more arguments to this point.)
3. We, the people living in this century, have the chance to have a huge impact on huge numbers of people to come - if we can make sense of the situation enough to find helpful actions. But right now, **we aren't ready for this.**


I strongly agree with this paper. Mathiness in particular is really annoying; often when I spend the time to actually understand the math in a paper, I come away disappointed at how it is saying something trivial or unimportant, and at this point I typically ignore the theorems unless I can't understand what the paper is saying without them. It's also really helpful to have ablation studies -- in fact, for last week's Capture the Flag paper, I probably would have written off the learned reward shaping as unimportant if the ablation study wasn't there to show it was important, after which I dug deeper and figured out what I had misunderstood. And suggestive language has in the past convinced me to read a paper, and then be surprised when the paper ended, because the actual content of the paper contained so much less than I expected. I'm a big fan of the recommendation to reviewers -- while it seems so obvious in hindsight, I've never actually asked myself that question when reviewing a paper.

Troubling Trends in ML Scholarship

This is a position paper arguing that ML research would benefit from more rigor, as part of the ICML debates. It identifies four trends in ML papers. First, papers often don't make clear whether they are providing an (authoritative) explanation or a speculation, in which case speculations can accidentally be cited as proven facts in other papers. Second, researchers often don't perform ablation studies, which makes it hard to figure out whether performance gains come from eg. a better algorithm or hyperparameter tuning. Third, papers often include math for the sake of conveying technical depth and impressiveness, not actual exposition, including eg. spurious theorems that are not particularly related to the main claims of the paper. Fourth, papers often misuse language by using suggestive definitions (eg. "curiosity", "fear"), overloading existing terminology, and suitcase words (words with combine many different meanings into one, leading to a very vague concept). The authors speculate on the causes (which I'm not summarizing) and have some suggestions for the community. For authors, they recommend asking what worked, and why, rather than just quantifying performance. For reviewers, they recommend asking "Might I have accepted this paper if the authors had done a worse job?” For example, if the authors hadn't done the ablation study that showed that two things didn't work, and instead just showed a combination of methods that gave a performance improvement, would I have accepted the paper?


Editorial policy for the Alignment Newsletter

In the survey I ran about a month ago, a couple of people suggested that I should clarify my editorial policy, especially since it has drifted since the newsletter was created. Note that I don’t view what I’m writing here as a policy that I am committing to. This is more like a description of how I currently make editorial decisions in practice, and it may change in the future.

I generally try to only summarize “high quality” articles. Here, "high quality" means that the article presents some conceptually new thing not previously sent in the newsletter, and there is decent evidence convincing me that this new thing is true / useful / worth considering. (Yes, novelty is one of my criteria. I could imagine summarizing e.g. a replication of some result if I wasn’t that confident of the original result, but I usually wouldn’t.)

Throughout the history of the newsletter, when deciding whether or not to summarize an article, I have also looked for some plausible pathway by which the new knowledge might be useful to an alignment researcher. Initially, there was a pretty small set of subfields that seemed particularly relevant (especially reward learning) and I tried to cover most high-quality work within those areas. (I cover progress in ML because it seems like a good model of ML / AGI development should be very useful for alignment research.)

However, over time as I learned more, I became more excited about a large variety of subfields. There’s basically no hope for me to keep up with all of the subfields, so now I rely a lot more on quick intuitive judgments about how exciting I expect a particular paper to be, and many high quality articles that are relevant to AI alignment never get summarized. I currently still try to cover almost every new high quality paper or post that _directly_ talks about AI alignment (as opposed to just being relevant).

Highlights are different. The main question I ask myself when deciding whether or not to highlight an article is: “Does it seem useful for _most_ technical alignment researchers to read this?” Note that this is very different from an evaluation of how impactful or high quality the article is: a paper that talks about all the tips and tricks you need to get learning from human feedback to work in practice could be very impactful and high quality, but probably still wouldn’t be highlighted because many technical researchers don’t work with systems that learn from human feedback, and so won’t read it. On the other hand, this editorial policy probably isn’t that impactful, but it seems particularly useful for my readers to read (so that you know what you are and aren’t getting with this newsletter).

A summary is where I say things that the authors would agree with. Usually, I strip out things that the authors said that I think are wrong. The exception is when the thing I believe is wrong is a central point of the article, in which case I will put it in the summary even though I don’t believe it. Typically I will then mention the disagreement in the opinion (though this doesn’t always happen, e.g. if I’ve mentioned the disagreement in previous newsletters, or if it would be very involved to explain why I disagree). I often give authors a chance to comment on the summaries + opinions, and usually authors are happy overall but might have some fairly specific nitpicks.

An opinion is where I say things that I believe that the authors may or may not believe.


Alignment Newsletter Three Year Retrospective

It’s (two days until) the third birthday of this newsletter! In this post, I reflect on the two years since the <@previous retrospective@>(@Alignment Newsletter One Year Retrospective@). There aren’t any major takeaways, so I won’t summarize all of it here. Please do **take [this 2 minute survey](https://docs.google.com/forms/d/e/1FAIpQLScRyiBrZoR_AxcHUlbGbp62dZUx36UuF_zAxO_ky948d-MwSw/viewform?usp=sf_link)** though. I’ll also copy over the “Advice to readers” section from the post:

**Don’t treat [newsletter entries] as an evaluation of people’s work.** As I mentioned above, I’m selecting articles based in part on how well they fit into my understanding of AI alignment. This is a poor method for evaluating other people’s work. Even if you defer to me completely and ignore everyone else’s views, it still would not be a good method, because often I am mistaken about how important the work is even on my own understanding of AI alignment. Almost always, my opinion about a paper I feel meh about will go up after talking to the authors about the work.

I also select articles based on how useful I think it would be for other AI alignment researchers to learn about the ideas presented. (This is especially true for the choice of what to highlight.) This can be very different from how useful the ideas are to the world (which is what I’d want out of an evaluation): incremental progress on some known subproblem like learning from human feedback could be very important, but still not worth telling other AI alignment researchers about.

**Consider reading just the highlights section.** If you’re very busy, or you find yourself just not reading the newsletter each week because it’s too long, I recommend just reading the highlights section. I select pretty strongly for “does this seem good for researchers to know?” when choosing the highlight(s).

**If you’re busy, consider using the [spreadsheet database](https://docs.google.com/spreadsheets/d/1PwWbWZ6FPqAgZWOoOcXM8N_tUCuxpEyMbN1NYYC02aM/edit#gid=0) as your primary mode of interaction.** Specifically, rather than reading the newsletter each week, you could instead keep the database open, and whenever you see a vaguely interesting new paper, you can check (via Ctrl+F) whether it has already been summarized, and if so you can read that summary. (Even I use the database in this way, though I usually know whether or not I’ve already summarized the paper before, rather than having to check.)

Also, there may be a nicer UI to interact with this database in the near future :)


The five criteria seem pretty good to me as a description of what people mean when they say that a system is goal-directed. It is less clear to me that all five criteria are important for making the case for AI risk (which is why I care about a definition of goal-directedness); in particular it doesn’t seem to me like the explainability property is important for such an argument (see also [this comment](https://www.alignmentforum.org/posts/EnN7cm3KaRrEAuWfa/comment-on-coherence-arguments-do-not-imply-goal-directed?commentId=CsRXodmiBfZ9wCZwr)).

Note that it can still be the case that as a research strategy it is useful to search for definitions that satisfy these five criteria; it is just that in evaluating which definition to use I would choose the one that makes the AI risk argument work best. (See also [Against the Backward Approach to Goal-Directedness](https://www.alignmentforum.org/posts/adKSWktLbxfihDANM/against-the-backward-approach-to-goal-directedness).)

Literature Review on Goal-Directedness

Adam Shimi, Michele Campolo, Joe Collman

This post extracts five different concepts that have been identified in the literature as properties of goal-directed systems:

1. **Restricted space of goals:** The space of goals should not be too expansive, since otherwise goal-directedness can <@become vacuous@>(@Coherence arguments do not imply goal-directed behavior@) (e.g. if we allow arbitrary functions over world-histories with no additional assumptions).
2. **Explainability:** A system should be described as goal-directed when doing so improves our ability to _explain_ the system’s behavior and _predict_ what it will do.
3. **Generalization:** A goal-directed system should adapt its behavior in the face of changes to its environment, such that it continues to pursue its goal.
4. **Far-sighted:** A goal-directed system should consider the long-term consequences of its actions.
5. **Efficient:** The more goal-directed a system is, the more efficiently it should achieve its goal.

The concepts of goal-directedness, optimization, and agency seem to have significant overlap, but there are differences in the ways the terms are used.

The authors then compare multiple proposals on these criteria:

1. The _intentional stance_ says that we should model a system as goal-directed when it helps us better explain the system’s behavior, performing well on explainability and generalization. It could easily be extended to include far-sightedness as well. A more efficient system for some goal will be easier to explain via the intentional stance, so it does well on that criterion too. And not every possible function can be a goal, since many are very complicated and thus would not be better explanations of behavior. However, the biggest issue is that the intentional stance cannot be easily formalized.
2. One possible formalization of the intentional stance is to say that a system is goal-directed when we can better explain the system’s behavior as maximizing a specific utility function, relative to explaining it using an input-output mapping (see <@Agents and Devices: A Relative Definition of Agency@>). This also does well on all five criteria.
3. <@AGI safety from first principles@> proposes another set of criteria that have a lot of overlap with the five criteria above.
4. A [definition based off of Kolmogorov complexity](https://www.alignmentforum.org/posts/dPmmuaz9szk26BkmD/vanessa-kosoy-s-shortform?commentId=Tg7A7rSYQSZPASm9s) works well, though it doesn’t require far-sightedness.


Announcing AXRP, the AI X-risk Research Podcast

Episodes of this new podcast will involve Daniel interviewing a researcher about a paper they’ve written, talking about the ideas in the paper and why they matter. Three episodes have already been released; I summarize them later in this newsletter.


I look forward to this post every year. It continues to be a stark demonstration of how much work _doesn't_ get covered in this newsletter -- while I tend to focus on the technical alignment problem, with some focus on AI governance and AI capabilities, this literature review spans many organizations working on existential risk, and as such has many papers that were never covered in this newsletter. Anyone who wants to donate to an organization working on AI alignment and/or x-risk should read this post.

Last year I mentioned I might write an overview for the sake of building inside view models (rather than donation decisions), this went out [shortly afterward](https://www.alignmentforum.org/posts/dKxX76SCfCvceJXHv/ai-alignment-2018-19-review) ([AN #84](https://mailchi.mp/1af38085edc5/an-84-reviewing-ai-alignment-work-in-2018-19)). I don't expect to write a similar post this year, partly because I think last year's post is still quite good as an overview of the discussion that's been happening.

2020 AI Alignment Literature Review and Charity Comparison

<@The tradition continues@>(@2019 AI Alignment Literature Review and Charity Comparison@)! I'll say nearly the same thing as I did last year:

This mammoth post goes through the work done within AI alignment from December 2019 - November 2020, from the perspective of someone trying to decide which of several AI alignment organizations to donate to. As part of this endeavor, Larks summarizes a ton of papers that were published at various organizations, and compares them to their budget and room for more funding.


I like this project and analysis -- it’s a different view on the landscape of technical AI safety than I usually get to see. I especially recommend reading it if you want to get a sense of the people and organizations comprising the technical AI safety field; I’m not going into detail here because I mostly try to focus on the object level issues in this newsletter.

TAI Safety Bibliographic Database

Related to the previous summary, we also have a database of a bunch of papers on transformative AI safety, that has attempted to have comprehensive coverage of papers motivated by safety at organizations with a significant safety focus within the years 2016-20, but also includes other stuff such as blog posts, content from earlier years, etc. There’s a bunch of analysis as well that I won’t go into.


I like this sequence: I think it’s a good “updated case” for AI risk that focuses on the situation in which intelligent AI systems arise through training of ML models. The points it makes are somewhat different from the ones I would make if I were writing such a case, but I think they are still sufficient to make the case that humanity has work to do if we are to ensure that AI systems we build are aligned.

AGI safety from first principles

This sequence presents the author’s personal view on the current best arguments for AI risk, explained from first principles (that is, without taking any previous claims for granted). The argument is a specific instantiation of the _second species argument_ that sufficiently intelligent AI systems could become the most intelligent species, in which case humans could lose the ability to create a valuable and worthwhile future.

We should clarify what we mean by superintelligence, and how it might arise. The author considers intelligence as quantifying simply whether a system “could” perform a wide range of tasks, separately from whether it is motivated to actually perform those tasks. In this case, we could imagine two rough types of intelligence. The first type, epitomized by most current AI systems, trains an AI system to perform many different tasks, so that it is then able to perform all of those tasks; however, it cannot perform tasks it has not been trained on. The second type, epitomized by human intelligence and <@GPT-3@>(@Language Models are Few-Shot Learners@), trains AI systems in a task-agnostic way, such that they develop general cognitive skills that allow them to solve new tasks quickly, perhaps with a small amount of training data. This second type seems particularly necessary for tasks where data is scarce, such as the task of being a CEO of a company. Note that these two types should be thought of as defining a spectrum, not a binary distinction, since the type of a particular system depends on how you define your space of “tasks”.

How might we get AI systems that are more intelligent than humans? Besides improved algorithms, compute and data, we will likely also see that _interactions_ between AI systems will be crucial to their capabilities. For example, since AI systems are easily replicated, we could get a _collective_ superintelligence via a collection of replicated AI systems working together and learning from each other. In addition, the process of creation of AI systems will be far better understood than that of human evolution, and AI systems will be easier to directly modify, allowing for AI systems to recursively improve their own training process (complementing human researchers) much more effectively than humans can improve themselves or their children.

The second species argument relies on the argument that superintelligent AI systems will gain power over humans, which is usually justified by arguing that the AI system will be goal-directed. Making this argument more formal is challenging: the EU maximizer framework <@doesn’t work for this purpose@>(@Coherent behaviour in the real world is an incoherent concept@) and applying the [intentional stance](https://en.wikipedia.org/wiki/Intentional_stance) only helps when you have some prior information about what goals the AI system might have, which begs the question.

The author decides to instead consider a more conceptual, less formal notion of agency, in which a system is more goal-directed the more its cognition has the following properties: (1) self-awareness, (2) planning, (3) judging actions or plans by their consequences, (4) being sensitive to consequences over large distances and long time horizons, (5) internal coherence, and (6) flexibility and adaptability. (Note that this can apply to a single unified model or a collective AI system.) It’s pretty hard to say whether current training regimes will lead to the development of these capabilities, but one argument for it is that many of these capabilities may end up being necessary prerequisites to training AI agents to do intellectual work.

Another potential framework is to identify a goal as some concept learned by the AI system, that then generalizes in such a way that the AI system pursues it over longer time horizons. In this case, we need to predict what concepts an AI system will learn and how likely it is that they generalize in this way. Unfortunately, we don’t yet know how to do this.

What does alignment look like? The author uses <@intent alignment@>(@Clarifying "AI Alignment"@), that is, the AI system should be “trying to do what the human wants it to do”, in order to rule out the cases where the AI system causes bad outcomes through incompetence where it didn’t know what it was supposed to do. Rather than focusing on the outer and inner alignment decomposition, the author prefers to take a holistic view in which the choice of reward function is just one (albeit quite important) tool in the overall project of choosing a training process that shapes the AI system towards safety (either by making it not agentic, or by shaping its motivations so that the agent is intent aligned).

Given that we’ll be trying to build aligned systems, why might we still get an existential catastrophe? First, a failure of alignment is still reasonably likely, since (1) good behavior is hard to identify, (2) human values are complex, (3) influence-seeking may be a useful subgoal during training, and thus incentivized, (4) it is hard to generate training data to disambiguate between different possible goals, (5) while interpretability could help it seems quite challenging. Then, given a failure of alignment, the AI systems could seize control via the mechanisms suggested in <@What failure looks like@> and [Superintelligence](https://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/1501227742). How likely this is depends on factors like (1) takeoff speed, (2) how easily we can understand what AI systems are doing, (3) how constrained AI systems are at deployment, and (4) how well humanity can coordinate.


I really enjoyed this book, primarily because of the tracing of the intellectual history of various ideas. While I knew of most of these ideas, and sometimes also who initially came up with the ideas, it's much more engaging to read the detailed stories of _how_ that person came to develop the idea; Brian's book delivers this again and again, functioning like a well-organized literature survey that is also fun to read because of its great storytelling. I struggled a fair amount in writing this summary, because I kept wanting to somehow communicate the writing style; in the end I decided not to do it and to instead give a few examples of passages from the book in [this post](https://www.alignmentforum.org/posts/gYfgWSxCpFdk2cZfE/the-alignment-problem-machine-learning-and-human-values).

The Alignment Problem

This book starts off with an explanation of machine learning and problems that we can currently see with it, including detailed stories and analysis of:

- The [gorilla misclassification incident](https://twitter.com/jackyalcine/status/615329515909156865)
- The [faulty reward in CoastRunners](https://openai.com/blog/faulty-reward-functions/)
- The [gender bias in language models](https://arxiv.org/abs/1607.06520)
- The [failure of facial recognition models on minorities](https://www.ted.com/talks/joy_buolamwini_how_i_m_fighting_bias_in_algorithms)
- The [COMPAS](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing) [controversy](https://www.documentcloud.org/documents/2998391-ProPublica-Commentary-Final-070616.html) (leading up to [impossibility results in fairness](https://www.chrisstucchio.com/pubs/slides/crunchconf_2018/slides.pdf))
- The [neural net that thought asthma reduced the risk of pneumonia](https://www.microsoft.com/en-us/research/wp-content/uploads/2017/06/KDD2015FinalDraftIntelligibleModels4HealthCare_igt143e-caruanaA.pdf)

It then moves on to agency and reinforcement learning, covering from a more historical and academic perspective how we have arrived at such ideas as temporal difference learning, reward shaping, curriculum design, and curiosity, across the fields of machine learning, behavioral psychology, and neuroscience. While the connections aren't always explicit, a knowledgeable reader can connect the academic examples given in these chapters to the ideas of <@specification gaming@>(@Specification gaming: the flip side of AI ingenuity@) and <@mesa optimization@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@) that we talk about frequently in this newsletter. Chapter 5 especially highlights that agent design is not just a matter of specifying a reward: often, rewards will do ~nothing, and the main requirement to get a competent agent is to provide good _shaping rewards_ or a good _curriculum_. Just as in the previous part, Brian traces the intellectual history of these ideas, providing detailed stories of (for example):

- BF Skinner's experiments in [training pigeons](https://psycnet.apa.org/record/1961-01933-001)
- The invention of the [perceptron](https://psycnet.apa.org/record/1959-09865-001)
- The success of [TD-Gammon](https://www.aaai.org/Papers/Symposia/Fall/1993/FS-93-02/FS93-02-003.pdf), and later [AlphaGo Zero](https://deepmind.com/blog/article/alphago-zero-starting-scratch)

The final part, titled "Normativity", delves much more deeply into the alignment problem. While the previous two parts are partially organized around AI capabilities -- how to get AI systems that optimize for _their_ objectives -- this last one tackles head on the problem that we want AI systems that optimize for _our_ (often-unknown) objectives, covering such topics as imitation learning, inverse reinforcement learning, learning from preferences, iterated amplification, impact regularization, calibrated uncertainty estimates, and moral uncertainty.


Engineering a Safer World

I recently read [Engineering a Safer World](https://static1.squarespace.com/static/53b78765e4b0949940758017/t/57d87eb6d2b8571af3501b26/1473898764674/Engineering_a_Safer_World+Nancy+Leveson.pdf) by Nancy G. Leveson, at [Joshua Achiam’s recommendation](https://twitter.com/jachiam0/status/1285449712573665281), and really enjoyed it, so get ready for another book summary! I’m not very happy with the summary I have -- it feels less compelling than the book, partly because the book provides a ton of examples that I don’t have the space to do -- but hopefully it is enough to get the key points across.

The main motivation of this book is to figure out how we can improve safety engineering. Its primary thesis is that the existing methods used in engineering are insufficient for the current challenges, and must be replaced by a method the author favors called STAMP. Note that the book is primarily concerned with mechanical systems that may also have computerized automation (think aerospace, chemical, and mechanical engineering); the conclusions should not be expected to apply directly to AI.

**The standard model of safety engineering and its deficiencies**

Historically, safety engineering has been developed as a reaction to the high level of accidents we had in the past, and as a result focused on the easiest gains first. In particular, there were a lot of gains to be had simply by ensuring that _machines didn’t break_. _(Rohin’s note: I’m editorializing a bit here, the author doesn’t explicitly say this but I think she believes it.)_ This led to a focus on _reliability_: given a specification for how a machine should operate, we aim to decrease the probability that the machine fails to meet that specification. For example, the specification for a water tank would be to contain the water up to a given pressure, and one way to improve the reliability of the tank would be to use a stronger material or make a thicker tank to make it less likely that the tank ruptures.

Under this model, an accident happens when a machine fails to meet its specification. So, we can analyze the accident by looking at what went wrong, and tracing back the physical causes to the first point at which a specification was not met, giving us a _root cause_ that can show us what we need to fix in order to prevent similar accidents in the future. We can call this sort of analysis an _event chain_ analysis.

However, in the last few decades there have been quite a few changes that make this model worse than it once was. The pace of technological change has risen, making it harder to learn from experience. The systems we build have become complex enough that there is a lot more _coupling_ or interaction effects between parts of the system that we could fail to account for.  Relatedly, the risks we face are getting large enough that we aren’t willing to tolerate even a _single_ accident. Human operators (e.g. factory workers) are no longer able to rely on easily understood and predictable mechanical systems, instead having to work with computerized automation which they cannot understand as well. At this point, event chain analysis and safety-via-reliability are no longer sufficient for safety engineering.

Consider for example the [Flight 965](https://en.wikipedia.org/wiki/American_Airlines_Flight_965) accident. In this case, the pilots got clearance to fly towards the Rozo waypoint in their descent, listed as (R) on their (paper) approach charts. One of the pilots pressed R in the flight management system (FMS), which brought up a list of waypoints that did _not_ include Rozo, and executed the first one (presumably believing that Rozo, being the closest waypoint, would show up first). As a result, the plane turned towards the selected waypoint, and crashed into a mountain.

The accident report for this incident placed the blame squarely on the pilots, firstly for not planning an appropriate path, and secondly for not having situational awareness of the terrain and that they needed to discontinue their approach. But most interestingly, the report blames the pilots for not reverting to basic radio navigation when the FMS became confusing. The author argues that the design of the automation was also flawed in this case, as the FMS stopped displaying the intermediate fixes to the chosen route, and the FMS’s navigational information used a different naming convention than the one in the approach charts. Surely this also contributed to the loss? In fact, in lawsuit appeals, the software manufacturer was held to be 17% liable.

However, the author argues that this is the exception, not the rule: typically event chain analysis proceeds until a human operator is found who did something unexpected, and then the blame can safely be placed on them. Operators are expected to “use common sense” to deviate from procedures when the procedures are unsafe, but when an accident happens, blame is placed on them for deviating from procedures. This is often very politically convenient, and is especially easy to justify thanks to hindsight bias, where we can identify exactly the right information and cues that the operator “should have” paid attention to, ignoring that in the moment there were probably many confusing cues and it was far from obvious which information to pay attention to. My favorite example has to be this quote from an accident report:

“Interviews with operations personnel did not produce a clear reason why the response to the [gas] alarm took 31 minutes. The only explanation was that there was not a sense of urgency since, in their experience, previous [gas] alarms were attributed to minor releases that did not require a unit evacuation.”

It is rare that I see such a clear example of a self-refuting paragraph. In the author’s words, “this statement is puzzling, because the statement itself provides a clear explanation for the behavior, that is, the previous experience”. It definitely sounds like the investigators searched backwards through the causal chain, found a situation where a human deviated from protocol, and decided to assign blame there.

This isn’t just a failure of the accident investigation -- the entire premise of some “root cause” in an event chain analysis implies that the investigators must end up choosing some particular point to label as The Root Cause, and such a decision is inevitably going to be determined more by the particular analysts involved rather than by features of the accident.

**Towards a new approach**

How might we fix the deficiencies of standard safety engineering? The author identifies several major changes in assumptions that are necessary for a new approach:

1. Blame is the enemy of safety. Safety engineering should focus on system behavior as a whole, where interventions can be made at many points on different levels, rather than seeking to identify a single intervention point.

2. Reliability (having machines meet their specifications) is neither necessary nor sufficient for safety (not having bad outcomes). Increased reliability can lead to decreased safety: if we increase the reliability of the water tank by making it using a stronger material, we may decrease the risk of rupture, but we may dramatically increase the harm when a rupture occurs since the water will be at a much higher pressure. This applies to software as well: highly reliable software need not be safe, as its specifications may not be correct.

3. Accidents involve the entire sociotechnical system, for which an event chain model is insufficient. Interventions on the sociological level (e.g. make it easy for low-level operators to report problems) should be considered part of the remit of safety engineering.

4. Major accidents are not caused by simultaneous occurrence of random chance events. Particularly egregious examples come from probabilistic risk analysis, where failures of different subsystems are often assumed to be independent, neglecting the possibility of a common cause, whether physical (e.g. multiple subsystems failing during a power outage) or sociological (e.g. multiple safety features being disabled as part of cost-cutting measures). In addition, systems tend to migrate towards higher risk over time, because environmental circumstances change, and operational practices diverge from the designed practices as they adapt to the new circumstances, or simply to be more efficient.

5. Operator behavior is a product of the environment in which it occurs. To improve safety, we must change the environment rather than the human. For example, if an accident occurs and an operator didn’t notice a warning light that could have let them prevent it, the solution is not to tell the operators to “pay more attention” -- that approach is doomed to fail.

**A detour into systems theory**

The new model proposed by the author is based on systems theory, so let’s take a moment to describe it. Consider possible systems that we may want to analyze:

First, there are some systems with _organized simplicity_, in which it is possible to decompose the system into several subsystems, analyze each of the subsystems independently, and then combine the results relatively easily to reach overall conclusions. We might think of these as systems in which analytic reduction is a good problem-solving strategy. _Rohin’s note: Importantly, this is different from the philosophical question of whether there exist phenomena that cannot be reduced to e.g. physics: that is a question about whether reduction is in principle possible, whereas this criterion is about whether such reduction is an effective strategy for a computationally bounded reasoner._ Most of physics would be considered to have organized simplicity.

Second, there are systems with _unorganized complexity_, where there is not enough underlying structure for analytic reduction into subsystems to be a useful tool. However, in such systems the behavior of individual elements of the system is sufficiently random (or at least, well-modeled as random) that statistics can be applied to it, and then the law of large numbers allows us to understand the system as an aggregate. A central example would be statistical mechanics, where we cannot say much about the motion of individual particles in a gas, but we can say quite a lot about the macroscopic behavior of the gas as a whole.

Systems theory deals with systems that have _organized complexity_. Such systems have enough organization and structure that we cannot apply statistics to it (or equivalently, the assumption of randomness is too incorrect), and are also sufficiently complex that analytic reduction is not a good technique (e.g. perhaps any potential decomposition into subsystems would be dominated by combinatorially many interaction effects between subsystems). Sociological systems are central examples of such systems: the individual components (humans) are very much not random, but neither are their interactions governed by simple laws as would be needed for analytic reduction. While systems theory cannot provide nearly the same level of precision as statistics or physics, it does provide useful concepts for thinking about such systems.

The first main concept in systems theory is that of _hierarchy and emergence_. The idea here is that systems with organized complexity can be decomposed into several hierarchical levels, with each level built “on top of” the previous one. For example, companies are built on top of teams which are built on top of individual employees. The behavior of components in a particular layer is described by some “language” that is well-suited for that layer. For example, we might talk about individual employees based on their job description, their career goals, their relationship with their manager, and so on, but we might talk about companies based on their overall direction and strategy, the desires of their customer base, the pressures from regulators, and so on.

_Emergence_ refers to the phenomenon that there can be properties of higher levels arising from lawful interactions at lower levels that nonetheless are meaningless in the language appropriate for the lower levels. For example, it is quite meaningful to say that the pressures on a company from government regulation caused them to (say) add captions to their videos, but if we look at the specific engineer who integrated the speech recognition software into the pipeline, we would presumably say “she integrated the speech recognition into the pipeline because she had previously worked with the code” rather than “she integrated it because government regulations told her to do so”. As another example, safety is an emergent system property, while reliability is not.

The second main concept is that of _control_. We are usually not satisfied with just understanding the behavior of systems; we also want to make changes to it (as in the case of making them safer). In systems theory, this is thought of as _control_, where we impose some sort of _constraint_ on possible system behavior at some level. For example, employee training is a potential control action that could aim to enforce the constraint that every employee knows what to do in an emergency. An effective controller requires a goal, a set of actions to take, a model of the system, and some way to sense the state of the system.

**STAMP: A new model underlying safety engineering**

The author then introduces a new model called Systems-Theoretic Accident Model and Processes (STAMP), which aims to present a framework for understanding how accidents occur (which can allow us to prevent them and/or learn from them). It contains three main components:

_Safety constraints_: In systems theory, a constraint is the equivalent of a specification, so these are just the safety-relevant specifications. Note that such specifications can be found at all levels of the hierarchy.

_Hierarchical safety controllers_: We use _controllers_ to enforce safety constraints at any given level. A control algorithm may be implemented by a mechanical system, a computerized system, or humans, and can exist at any level of the hierarchy. A controller at level N will typically depend on constraints at level N - 1, and thus the design of this controller influences which safety constraints are placed at level N - 1.

_Process models_: An effective controller must have a model of the process it is controlling. Many accidents are the result of a mismatch between the actual process and the process model of the controller.

This framework can be applied towards several different tasks, and in all cases the steps are fairly similar: identify the safety constraints you want, design or identify the controllers enforcing those constraints, and then do some sort of generic reasoning with these components.

If an accident occurs, then at the highest level, either the control algorithm(s) failed to enforce the safety constraints, or the control actions were sent correctly but were not followed. In the latter case, the controllers at the lower level should then be analyzed to see why the control actions were not followed. Ultimately, this leads to an analysis on multiple levels, which can identify several things that went wrong rather than one Root Cause, that can all be fixed to improve safety in the future.

**Organizational safety**

So far we’ve covered roughly chapters 1-4 of the book. I’ll now jump straight to chapter 13, which seems particularly important and relevant, as it deals with how organizational structure and management should be designed to support safety.

One major point that the author makes is that safety _is_ cost-effective for _long-term_ performance as long as it is designed into the system from the start, rather than added on at the last minute. Performance pressure on the other hand inevitably leads to cuts in safety.

In order to actually get safety designed into the system from the start, it is crucial that top management demonstrates a strong commitment to safety, as without this employees will inevitably cut corners on safety as they will believe it is in their incentives to do so. Other important factors include a concrete corporate safety policy, as well as a strong corporate safety culture. It is important that safety is part of the design process, rather than tacked on at the end. In the author’s words, _putting safety into the quality assurance organization is the worst place for it. [...] It sets up the expectation that safety is an after-the-fact or auditing activity only._

In addition, it is important that information can flow well. Going from the bottom to the top, it should be possible for low-level operators to report potential problems in a way that they are actually acted on and the relevant information reaches top management.  From the top to the bottom, safety information and training should be easily available and accessible to employees when they need it.

It is also important to have controls to prevent the general tendency of systems to migrate towards higher risk, e.g. by relaxing safety requirements as time passes without any incidents. The next chapter describes SUBSAFE, the author’s example of a well-run safety program, in which the control is for everyone to periodically watch a video reminding them of the importance of their particular safety work (in particular, the video shows the loss of the USS Thresher, an event that caused SUBSAFE to be created).

Perhaps obviously, it is important for an organization to have a dedicated safety team. This is in contrast to making everyone responsible for safety. In the author’s words: _While, of course, everyone should try to behave safely and to achieve safety goals, someone has to be assigned responsibility for ensuring that the goals are achieved._

If you start by designing for safety, it is cost-effective, not opposed to long-term money-maximizing. Once there is performance pressure, then you see cuts in safety. Also sometimes people fix symptoms instead of underlying causes, and then they just keep seeing symptoms forever and conclude they are inevitable.

**Miscellaneous notes**

The remaining chapters of the book apply STAMP in a bunch of different areas with many examples, including an entire chapter devoted to the STAMP treatment of a friendly fire accident. I also really liked the discussion of human factors in the book, but decided not to summarize it as this has already gotten quite long.

**Summary of the summary**

I’ll conclude with a quote from the book’s epilogue:

_What seems to distinguish those experiencing success is that they:_
_1. Take a systems approach to safety in both development and operations_
_2. Have instituted a learning culture where they have effective learning from events_
_3. Have established safety as a priority and understand that their long-term success depends on it_

**Relationship to AI safety**

A primary motivation for thinking about AI is that it would be very impactful for our society, and very impactful technologies need not have good impacts. “Society” clearly falls into the “organized complexity” class of systems, and so I expect that the ideas of safety constraints and hierarchical control algorithms will be useful ways to think about possible impacts of AI on society. For example, if we want to think about the possibility of AI systems differentially improving technical progress over “wisdom”, such that we get dangerous technologies before we’re ready for them, we may want to sketch out hierarchical “controllers” at the societal level that could solve this problem. Ideally these would eventually turn into constraints on the AI systems that we build, e.g. “AI systems should report potentially impactful new technologies to such-and-such committee”. I see the AI governance field as doing this sort of work using different terminology.

Technical AI alignment (in the sense of <@intent alignment@>(@Clarifying "AI Alignment"@)) does not seem to benefit as much from this sort of an approach. The main issue is that we are often considering a fairly unitary system (such as a neural net, or the mathematical model of expected utility maximization) to which the hierarchical assumption of systems theory does not really apply.

To be clear, I _do_ think that there in fact is some hierarchy. For example, in image classifiers where low levels involve edge detectors while high levels involve dog-face detectors. However, we do not have the language to talk about these hierarchies, nor the algorithms to control the intermediate layers. While <@Circuits@>(@Thread: Circuits@) is illustrating this hierarchy for image classifiers, it does not give us a language that we can (currently) use to talk about advanced AI systems. As a result, we are reduced to focusing on the incentives we provide to the AI system, or speculating on the levels of hierarchy that might be internal to advanced AI systems, neither of which seem particularly conducive to good work.

In the language of this book, I work on intent alignment because I expect that the ability to enforce the constraint “the AI system tries to do what its operator wants” will be a very useful building block for enforcing whatever societal safety constraints we eventually settle on, and it seems possible to make progress on it today. There are several arguments for risk that this ignores (see e.g. <@here@>(@The Main Sources of AI Risk?@) and <@here@>(@AI Research Considerations for Human Existential Safety@)); for some of these other risks, the argument is that we can handle those using similar mechanisms as we have before (e.g. governance, democracy, police, etc), _as long as_ we have handled intent alignment.


I'm glad this was finally written down -- I've been calling this the "induction hammer" and have used it a lot in my own thinking. Thinking about this sort of a model, and in particular what kinds of properties we could best preserve inductively, has been quite helpful for me.

A general model of safety-oriented AI development

A general model for developing safe powerful AI systems is to have a team of humans and AIs, which continually develops and adds more AIs to the team, while inductively preserving alignment.


I refer to convergent instrumental subgoals quite often in this newsletter, so it seemed like I should have a summary of it. I especially like this paper because it holds up pretty well 12 years later. Even though I’ve <@critiqued@>(@Conclusion to the sequence on value learning@) the idea that powerful AI systems must be expected utility maximizers, I still find myself agreeing with this paper, because it _assumes_ a goal-directed agent and reasons from there, rather than trying to argue that powerful AI systems must be goal-directed. Given that assumption, I agree with the conclusions drawn here.

The Basic AI Drives

This paper from 2008 introduces convergent instrumental subgoals: the subgoals that an AI system will have “by default”, unless care is taken to avoid them. For this paper, an AI system is a system that “has goals which it tries to accomplish by acting in the world”, i.e. it assumes that the system is <@goal-directed@>(@Intuitions about goal-directed behavior@).

It starts by arguing that a sufficiently powerful goal-directed AI system will want to self-improve, as that could help it achieve its goals better in the (presumably long) future. In particular, it will want to become “rational”, in the sense that it will want to maximize its _expected_ utility, where the utility function is determined by its goal. (The justification for this is the VNM theorem, and the various Dutch book arguments that support Bayesianism and expected utility maximization.)

However, not all modifications would be good for the AI system. In particular, it will very strongly want to preserve its utility function, as that determines what it will (try to) accomplish in the future, and any change in the utility function would be a disaster from the perspective of the current utility function. Similarly, it will want to protect itself from harm, that is, it has a survival incentive, because it can’t accomplish its goal if it’s dead.

The final instrumental subgoal is to acquire resources and use them efficiently in pursuit of its goal, because almost by definition resources are useful for a wide variety of goals, including (probably) the AI system’s goal.


While I agree that the COVID response was worse than it could have been, I think there are several important disanalogies between the COVID-19 pandemic and a soft takeoff scenario, which I elaborate on in [this comment](https://www.alignmentforum.org/posts/wTKjRFeSjKLDSWyww/possible-takeaways-from-the-coronavirus-pandemic-for-slow-ai?commentId=pKNQ9oq72HFS2JrdH). First, with COVID there were many novel problems, which I don’t expect with AI. Second, I expect a longer time period over which decisions can be made for AI alignment. Finally, with AI alignment, we have the option of preventing problems from ever arising, which is not really an option with pandemics. See also [this post](https://www.lesswrong.com/posts/EgdHK523ZM4zPiX5q/coronavirus-as-a-test-run-for-x-risks#Implications_for_X_risks).

Possible takeaways from the coronavirus pandemic for slow AI takeoff

The COVID-19 pandemic is an example of a large risk that humanity faced. What lessons can we learn for AI alignment? This post argues that the pandemic is an example of the sort of situation we can expect in a slow takeoff scenario, since we had the opportunity to learn from experience, act on warning signs, and reach a timely consensus that there is a serious problem. However, while we could have learned from previous epidemics like SARS, we failed to generalize the lessons from SARS. Despite warning signs of a pandemic in February, many countries wasted a month when they could have been stocking up on PPE and testing capacity. We had no consensus that COVID-19 was a problem, with articles dismissing it as no worse than the flu as late as March.

All of these problems could also happen with slow takeoff: we may fail to generalize from narrow AI systems to more general AI systems; we might not act on warning signs; and we may not believe that powerful AI is on the horizon until it is too late. The conclusion is “unless more competent institutions are in place by the time general AI arrives, it is not clear to me that slow takeoff would be much safer than fast takeoff”.


I’m glad this post exists: while most of the specific proposals could be found by patching together content spread across other blog posts, there was a severe lack of a single article laying out a full picture for even one proposal, let alone all eleven in this post.

I usually don’t think about outer alignment as what happens with optimal policies, as assumed in this post -- when you’re talking about loss functions _in the real world_ (as I think this post is trying to do), _optimal_ behavior can be weird and unintuitive, in ways that may not actually matter. For example, arguably for any loss function, the optimal policy is to hack the loss function so that it always outputs zero (or perhaps negative infinity).

An overview of 11 proposals for building safe advanced AI

This post describes eleven “full” AI alignment proposals (where the goal is to build a powerful, beneficial AI system using current techniques), and evaluates them on four axes:

1. **Outer alignment:** Would the optimal policy for the specified loss function be aligned with us? See also [this post](https://www.alignmentforum.org/posts/33EKjmAdKFn3pbKPJ/outer-alignment-and-imitative-amplification).
2. **Inner alignment:** Will the model that is _actually produced_ by the training process be aligned with us?
3. **Training competitiveness:** Is this an efficient way to train a powerful AI system? More concretely, if one team had a “reasonable lead” over other teams, would they keep at least some of the lead if they used this algorithm?
4. **Performance competitiveness:** Will the trained model have good performance (relative to other models that could be trained)?

Seven of the eleven proposals are of the form “recursive outer alignment technique” plus “<@technique for robustness@>(@Worst-case guarantees (Revisited)@)”. The recursive outer alignment technique is either <@debate@>(@AI safety via debate@), <@recursive reward modeling@>(@Scalable agent alignment via reward modeling@), or some flavor of <@amplification@>(@Capability amplification@). The technique for robustness is either transparency tools to “peer inside the model”, <@relaxed adversarial training@>(@Relaxed adversarial training for inner alignment@), or intermittent oversight by a competent supervisor. An additional two proposals are of the form “non-recursive outer alignment technique” plus “technique for robustness” -- the non-recursive techniques are vanilla reinforcement learning in a multiagent environment, and narrow reward learning.

Another proposal is Microscope AI, in which we train AI systems to simply understand vast quantities of data, and then by peering into the AI system we can learn the insights that the AI system learned, leading to a lot of value. We wouldn’t have the AI system act in the world, thus eliminating a large swath of potential bad outcomes. Finally, we have STEM AI, where we try to build an AI system that operates in a sandbox and is very good at science and engineering, but doesn’t know much about humans. Intuitively, such a system would be very unlikely to deceive us (and probably would be incapable of doing so).

The post contains a lot of additional content that I didn’t do justice to in this summary. In particular, I’ve said nothing about the analysis of each of these proposals on the four axes listed above; the full post talks about all 44 combinations.


As one of the people interviewed, these seem like the right high-level takeaways to me.

Takeaways from safety by default interviews

This post lists three key takeaways from AI Impacts' conversations with "optimistic" researchers (summarized mainly in [AN #80](https://mailchi.mp/b3dc916ac7e2/an-80-why-ai-risk-might-be-solved-without-additional-intervention-from-longtermists) with one in [AN #63](https://mailchi.mp/533c646a4b21/an-63how-architecture-search-meta-learning-and-environment-design-could-lead-to-general-intelligence)). I'll just name the takeaways here, see the post for more details:

1. Relative optimism in AI often comes from the belief that AGI will be developed gradually, and problems will be fixed as they are found rather than neglected.
2. Many of the arguments I heard around relative optimism weren’t based on inside-view technical arguments.
3. There are lots of calls for individuals with views around AI risk to engage with each other and understand the reasoning behind fundamental disagreements.


I enjoyed this post, and recommend reading it in full if you are interested in AI risk because of effective altruism. (I've kept the summary relatively short because not all of my readers care about effective altruism.) My personal cruxes and story of impact are actually fairly different: in particular, while this post sees the impact of research as coming from solving the technical alignment problem, I care about other sources of impact as well. See [this comment](https://forum.effectivealtruism.org/posts/Ayu5im98u8FeMWoBZ/my-personal-cruxes-for-working-on-ai-safety#vWuF8NCs7JQWv9XM3) for details.

My personal cruxes for working on AI safety

This post describes how Buck's cause prioritization within an [effective altruism](https://www.effectivealtruism.org/) framework leads him to work on AI risk. The case can be broken down into a conjunction of five cruxes. Specifically, the story for impact is that 1) AGI would be a big deal if it were created, 2) has a decent chance of being created soon, before any other "big deal" technology is created, and 3) poses an alignment problem for which we can think ahead in order to solve, and it's potentially valuable to do so even given the fact that people might try to solve this later. His research 4) would be put into practice if it solved the problem and 5) makes progress on solving the problem.


I can remember being critical of AI risk myself for similar reasons and think that it is important to be careful with the framing of pitches to avoid these heuristics from firing. This is not to say that we should avoid criticism of the idea of AI risk, but criticism is a lot more helpful if it comes from people who have actually engaged with the arguments.

**Rohin's opinion:** Even after knowing the arguments, I find six of the heuristics quite compelling: technology doomsayers have usually been wrong in the past, there isn't a concrete threat model, it's not empirically testable, it's too extreme, it isn't well grounded in my experience with existing AI systems, and it's too far off to do useful work now. All six make me distinctly more skeptical of AI risk.

A list of good heuristics that the case for AI x-risk fails

Because human attention is limited and a lot of people try to convince us of the importance of their favourite cause, we cannot engage with everyone’s arguments in detail. Thus we have to rely on heuristics to filter out insensible arguments. Depending on the form of exposure, the case for AI risks can fail on many of these generally useful heuristics, eight of which are detailed in this post. Given this outside view perspective, it is unclear whether we should actually expect ML researchers to spend time evaluating the arguments for AI risk.


Developing transparency tools is currently my best guess for how we can avoid deception and catastrophic planning in our AI systems. I'm most excited about applying transparency techniques via the first and third routes, which primarily help us audit our models. I'm more pessimistic about the fourth approach because it predictably involves restructuring the incentives for machine learning as a field, which is quite difficult. My opinion might be different if we could somehow coordinate the development of these technologies.

Chris Olah’s views on AGI safety

This post is Evan's best attempt to summarize [Chris Olah](https://colah.github.io/about.html)'s views on how transparency is a vital component for building safe artificial intelligence, which he distinguishes into four separate approaches:

First, we can apply interpretability to audit our neural networks, or in other words, catch problematic reasoning in our models. Second, transparency can help safety by allowing researchers to deliberately structure their models in ways that systematically work, rather than using machine learning as a black box. Third, understanding transparency allows us to directly incentivize for transparency in model design and decisions -- similar to how we grade humans on their reasoning (not just the correct answer) by having them show their work. Fourth, transparency might allow us to reorient the field of AI towards microscope AI: AI that gives us new ways of understanding the world, enabling us to be more capable, without itself taking autonomous actions.

Chris expects that his main disagreement with others is whether good transparency is possible as models become more complex. He hypothesizes that as models become more advanced, they will counterintuitively become more interpretable, as they will begin using more crisp human-relatable abstractions. Finally, Chris recognizes that his view implies that we might have to re-align the ML community, but he remains optimistic because he believes there's a lot of low-hanging fruit, research into interpretability allows low-budget labs to remain competitive, and interpretability is aligned with the scientific virtue to understand our tools.


Clarifying some key hypotheses in AI alignment

This post (that I contributed to) introduces a diagram that maps out important and controversial hypotheses for AI alignment. The goal is to help researchers identify and more productively discuss their disagreements.