I strongly agree with this paper. Mathiness in particular is really annoying; often when I spend the time to actually understand the math in a paper, I come away disappointed at how it is saying something trivial or unimportant, and at this point I typically ignore the theorems unless I can't understand what the paper is saying without them. It's also really helpful to have ablation studies -- in fact, for last week's Capture the Flag paper, I probably would have written off the learned reward shaping as unimportant if the ablation study wasn't there to show it was important, after which I dug deeper and figured out what I had misunderstood. And suggestive language has in the past convinced me to read a paper, and then be surprised when the paper ended, because the actual content of the paper contained so much less than I expected. I'm a big fan of the recommendation to reviewers -- while it seems so obvious in hindsight, I've never actually asked myself that question when reviewing a paper.

Troubling Trends in ML Scholarship

This is a position paper arguing that ML research would benefit from more rigor, as part of the ICML debates. It identifies four trends in ML papers. First, papers often don't make clear whether they are providing an (authoritative) explanation or a speculation, in which case speculations can accidentally be cited as proven facts in other papers. Second, researchers often don't perform ablation studies, which makes it hard to figure out whether performance gains come from eg. a better algorithm or hyperparameter tuning. Third, papers often include math for the sake of conveying technical depth and impressiveness, not actual exposition, including eg. spurious theorems that are not particularly related to the main claims of the paper. Fourth, papers often misuse language by using suggestive definitions (eg. "curiosity", "fear"), overloading existing terminology, and suitcase words (words with combine many different meanings into one, leading to a very vague concept). The authors speculate on the causes (which I'm not summarizing) and have some suggestions for the community. For authors, they recommend asking what worked, and why, rather than just quantifying performance. For reviewers, they recommend asking "Might I have accepted this paper if the authors had done a worse job?" For example, if the authors hadn't done the ablation study that showed that two things didn't work, and instead just showed a combination of methods that gave a performance improvement, would I have accepted the paper?


Editorial policy for the Alignment Newsletter

In the survey I ran about a month ago, a couple of people suggested that I should clarify my editorial policy, especially since it has drifted since the newsletter was created. Note that I don't view what I'm writing here as a policy that I am committing to. This is more like a description of how I currently make editorial decisions in practice, and it may change in the future.

I generally try to only summarize "high quality" articles. Here, "high quality" means that the article presents some conceptually new thing not previously sent in the newsletter, and there is decent evidence convincing me that this new thing is true / useful / worth considering. (Yes, novelty is one of my criteria. I could imagine summarizing e.g. a replication of some result if I wasn't that confident of the original result, but I usually wouldn't.)

Throughout the history of the newsletter, when deciding whether or not to summarize an article, I have also looked for some plausible pathway by which the new knowledge might be useful to an alignment researcher. Initially, there was a pretty small set of subfields that seemed particularly relevant (especially reward learning) and I tried to cover most high-quality work within those areas. (I cover progress in ML because it seems like a good model of ML / AGI development should be very useful for alignment research.)

However, over time as I learned more, I became more excited about a large variety of subfields. There's basically no hope for me to keep up with all of the subfields, so now I rely a lot more on quick intuitive judgments about how exciting I expect a particular paper to be, and many high quality articles that are relevant to AI alignment never get summarized. I currently still try to cover almost every new high quality paper or post that _directly_ talks about AI alignment (as opposed to just being relevant).

Highlights are different. The main question I ask myself when deciding whether or not to highlight an article is: "Does it seem useful for _most_ technical alignment researchers to read this?" Note that this is very different from an evaluation of how impactful or high quality the article is: a paper that talks about all the tips and tricks you need to get learning from human feedback to work in practice could be very impactful and high quality, but probably still wouldn't be highlighted because many technical researchers don't work with systems that learn from human feedback, and so won't read it. On the other hand, this editorial policy probably isn't that impactful, but it seems particularly useful for my readers to read (so that you know what you are and aren't getting with this newsletter).

A summary is where I say things that the authors would agree with. Usually, I strip out things that the authors said that I think are wrong. The exception is when the thing I believe is wrong is a central point of the article, in which case I will put it in the summary even though I don't believe it. Typically I will then mention the disagreement in the opinion (though this doesn't always happen, e.g. if I've mentioned the disagreement in previous newsletters, or if it would be very involved to explain why I disagree). I often give authors a chance to comment on the summaries + opinions, and usually authors are happy overall but might have some fairly specific nitpicks.

An opinion is where I say things that I believe that the authors may or may not believe.


Alignment Newsletter Three Year Retrospective

It's (two days until) the third birthday of this newsletter! In this post, I reflect on the two years since the <@previous retrospective@>(@Alignment Newsletter One Year Retrospective@). There aren't any major takeaways, so I won't summarize all of it here. Please do **take [this 2 minute survey](https://docs.google.com/forms/d/e/1FAIpQLScRyiBrZoR_AxcHUlbGbp62dZUx36UuF_zAxO_ky948d-MwSw/viewform?usp=sf_link)** though. I'll also copy over the "Advice to readers" section from the post:

**Don't treat [newsletter entries] as an evaluation of people's work.** As I mentioned above, I'm selecting articles based in part on how well they fit into my understanding of AI alignment. This is a poor method for evaluating other people's work. Even if you defer to me completely and ignore everyone else's views, it still would not be a good method, because often I am mistaken about how important the work is even on my own understanding of AI alignment. Almost always, my opinion about a paper I feel meh about will go up after talking to the authors about the work.

I also select articles based on how useful I think it would be for other AI alignment researchers to learn about the ideas presented. (This is especially true for the choice of what to highlight.) This can be very different from how useful the ideas are to the world (which is what I'd want out of an evaluation): incremental progress on some known subproblem like learning from human feedback could be very important, but still not worth telling other AI alignment researchers about.

**Consider reading just the highlights section.** If you're very busy, or you find yourself just not reading the newsletter each week because it's too long, I recommend just reading the highlights section. I select pretty strongly for "does this seem good for researchers to know?" when choosing the highlight(s).

**If you're busy, consider using the [spreadsheet database](https://docs.google.com/spreadsheets/d/1PwWbWZ6FPqAgZWOoOcXM8N_tUCuxpEyMbN1NYYC02aM/edit#gid=0) as your primary mode of interaction.** Specifically, rather than reading the newsletter each week, you could instead keep the database open, and whenever you see a vaguely interesting new paper, you can check (via Ctrl+F) whether it has already been summarized, and if so you can read that summary. (Even I use the database in this way, though I usually know whether or not I've already summarized the paper before, rather than having to check.)

Also, there may be a nicer UI to interact with this database in the near future :)


The five criteria seem pretty good to me as a description of what people mean when they say that a system is goal-directed. It is less clear to me that all five criteria are important for making the case for AI risk (which is why I care about a definition of goal-directedness); in particular it doesn’t seem to me like the explainability property is important for such an argument (see also [this comment](https://www.alignmentforum.org/posts/EnN7cm3KaRrEAuWfa/comment-on-coherence-arguments-do-not-imply-goal-directed?commentId=CsRXodmiBfZ9wCZwr)).

Note that it can still be the case that as a research strategy it is useful to search for definitions that satisfy these five criteria; it is just that in evaluating which definition to use I would choose the one that makes the AI risk argument work best. (See also [Against the Backward Approach to Goal-Directedness](https://www.alignmentforum.org/posts/adKSWktLbxfihDANM/against-the-backward-approach-to-goal-directedness).)

Literature Review on Goal-Directedness

Adam Shimi, Michele Campolo, Joe Collman

This post extracts five different concepts that have been identified in the literature as properties of goal-directed systems:

1. **Restricted space of goals:** The space of goals should not be too expansive, since otherwise goal-directedness can <@become vacuous@>(@Coherence arguments do not imply goal-directed behavior@) (e.g. if we allow arbitrary functions over world-histories with no additional assumptions).
2. **Explainability:** A system should be described as goal-directed when doing so improves our ability to _explain_ the system's behavior and _predict_ what it will do.
3. **Generalization:** A goal-directed system should adapt its behavior in the face of changes to its environment, such that it continues to pursue its goal.
4. **Far-sighted:** A goal-directed system should consider the long-term consequences of its actions.
5. **Efficient:** The more goal-directed a system is, the more efficiently it should achieve its goal.

The concepts of goal-directedness, optimization, and agency seem to have significant overlap, but there are differences in the ways the terms are used.

The authors then compare multiple proposals on these criteria:

1. The _intentional stance_ says that we should model a system as goal-directed when it helps us better explain the system's behavior, performing well on explainability and generalization. It could easily be extended to include far-sightedness as well. A more efficient system for some goal will be easier to explain via the intentional stance, so it does well on that criterion too. And not every possible function can be a goal, since many are very complicated and thus would not be better explanations of behavior. However, the biggest issue is that the intentional stance cannot be easily formalized.
2. One possible formalization of the intentional stance is to say that a system is goal-directed when we can better explain the system's behavior as maximizing a specific utility function, relative to explaining it using an input-output mapping (see <@Agents and Devices: A Relative Definition of Agency@>). This also does well on all five criteria.
3. <@AGI safety from first principles@> proposes another set of criteria that have a lot of overlap with the five criteria above.
4. A [definition based off of Kolmogorov complexity](https://www.alignmentforum.org/posts/dPmmuaz9szk26BkmD/vanessa-kosoy-s-shortform?commentId=Tg7A7rSYQSZPASm9s) works well, though it doesn't require far-sightedness.


Announcing AXRP, the AI X-risk Research Podcast

Episodes of this new podcast will involve Daniel interviewing a researcher about a paper they've written, talking about the ideas in the paper and why they matter. Three episodes have already been released; I summarize them later in this newsletter.


I look forward to this post every year. It continues to be a stark demonstration of how much work _doesn't_ get covered in this newsletter -- while I tend to focus on the technical alignment problem, with some focus on AI governance and AI capabilities, this literature review spans many organizations working on existential risk, and as such has many papers that were never covered in this newsletter. Anyone who wants to donate to an organization working on AI alignment and/or x-risk should read this post.

Last year I mentioned I might write an overview for the sake of building inside view models (rather than donation decisions), this went out [shortly afterward](https://www.alignmentforum.org/posts/dKxX76SCfCvceJXHv/ai-alignment-2018-19-review) ([AN #84](https://mailchi.mp/1af38085edc5/an-84-reviewing-ai-alignment-work-in-2018-19)). I don't expect to write a similar post this year, partly because I think last year's post is still quite good as an overview of the discussion that's been happening.

2020 AI Alignment Literature Review and Charity Comparison

<@The tradition continues@>(@2019 AI Alignment Literature Review and Charity Comparison@)! I'll say nearly the same thing as I did last year:

This mammoth post goes through the work done within AI alignment from December 2019 - November 2020, from the perspective of someone trying to decide which of several AI alignment organizations to donate to. As part of this endeavor, Larks summarizes a ton of papers that were published at various organizations, and compares them to their budget and room for more funding.


I like this project and analysis -- it’s a different view on the landscape of technical AI safety than I usually get to see. I especially recommend reading it if you want to get a sense of the people and organizations comprising the technical AI safety field; I’m not going into detail here because I mostly try to focus on the object level issues in this newsletter.

TAI Safety Bibliographic Database

Related to the previous summary, we also have a database of a bunch of papers on transformative AI safety, that has attempted to have comprehensive coverage of papers motivated by safety at organizations with a significant safety focus within the years 2016-20, but also includes other stuff such as blog posts, content from earlier years, etc. There's a bunch of analysis as well that I won't go into.


I like this sequence: I think it’s a good “updated case” for AI risk that focuses on the situation in which intelligent AI systems arise through training of ML models. The points it makes are somewhat different from the ones I would make if I were writing such a case, but I think they are still sufficient to make the case that humanity has work to do if we are to ensure that AI systems we build are aligned.

AGI safety from first principles

This sequence presents the author's personal view on the current best arguments for AI risk, explained from first principles (that is, without taking any previous claims for granted). The argument is a specific instantiation of the _second species argument_ that sufficiently intelligent AI systems could become the most intelligent species, in which case humans could lose the ability to create a valuable and worthwhile future.

We should clarify what we mean by superintelligence, and how it might arise. The author considers intelligence as quantifying simply whether a system "could" perform a wide range of tasks, separately from whether it is motivated to actually perform those tasks. In this case, we could imagine two rough types of intelligence. The first type, epitomized by most current AI systems, trains an AI system to perform many different tasks, so that it is then able to perform all of those tasks; however, it cannot perform tasks it has not been trained on. The second type, epitomized by human intelligence and <@GPT-3@>(@Language Models are Few-Shot Learners@), trains AI systems in a task-agnostic way, such that they develop general cognitive skills that allow them to solve new tasks quickly, perhaps with a small amount of training data. This second type seems particularly necessary for tasks where data is scarce, such as the task of being a CEO of a company. Note that these two types should be thought of as defining a spectrum, not a binary distinction, since the type of a particular system depends on how you define your space of "tasks".

How might we get AI systems that are more intelligent than humans? Besides improved algorithms, compute and data, we will likely also see that _interactions_ between AI systems will be crucial to their capabilities. For example, since AI systems are easily replicated, we could get a _collective_ superintelligence via a collection of replicated AI systems working together and learning from each other. In addition, the process of creation of AI systems will be far better understood than that of human evolution, and AI systems will be easier to directly modify, allowing for AI systems to recursively improve their own training process (complementing human researchers) much more effectively than humans can improve themselves or their children.

The second species argument relies on the argument that superintelligent AI systems will gain power over humans, which is usually justified by arguing that the AI system will be goal-directed. Making this argument more formal is challenging: the EU maximizer framework <@doesn't work for this purpose@>(@Coherent behaviour in the real world is an incoherent concept@) and applying the [intentional stance](https://en.wikipedia.org/wiki/Intentional_stance) only helps when you have some prior information about what goals the AI system might have, which begs the question.

The author decides to instead consider a more conceptual, less formal notion of agency, in which a system is more goal-directed the more its cognition has the following properties: (1) self-awareness, (2) planning, (3) judging actions or plans by their consequences, (4) being sensitive to consequences over large distances and long time horizons, (5) internal coherence, and (6) flexibility and adaptability. (Note that this can apply to a single unified model or a collective AI system.) It's pretty hard to say whether current training regimes will lead to the development of these capabilities, but one argument for it is that many of these capabilities may end up being necessary prerequisites to training AI agents to do intellectual work.

Another potential framework is to identify a goal as some concept learned by the AI system, that then generalizes in such a way that the AI system pursues it over longer time horizons. In this case, we need to predict what concepts an AI system will learn and how likely it is that they generalize in this way. Unfortunately, we don't yet know how to do this.

What does alignment look like? The author uses <@intent alignment@>(@Clarifying "AI Alignment"@), that is, the AI system should be "trying to do what the human wants it to do", in order to rule out the cases where the AI system causes bad outcomes through incompetence where it didn't know what it was supposed to do. Rather than focusing on the outer and inner alignment decomposition, the author prefers to take a holistic view in which the choice of reward function is just one (albeit quite important) tool in the overall project of choosing a training process that shapes the AI system towards safety (either by making it not agentic, or by shaping its motivations so that the agent is intent aligned).

Given that we'll be trying to build aligned systems, why might we still get an existential catastrophe? First, a failure of alignment is still reasonably likely, since (1) good behavior is hard to identify, (2) human values are complex, (3) influence-seeking may be a useful subgoal during training, and thus incentivized, (4) it is hard to generate training data to disambiguate between different possible goals, (5) while interpretability could help it seems quite challenging. Then, given a failure of alignment, the AI systems could seize control via the mechanisms suggested in <@What failure looks like@> and [Superintelligence](https://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/<PHONE>). How likely this is depends on factors like (1) takeoff speed, (2) how easily we can understand what AI systems are doing, (3) how constrained AI systems are at deployment, and (4) how well humanity can coordinate.


I really enjoyed this book, primarily because of the tracing of the intellectual history of various ideas. While I knew of most of these ideas, and sometimes also who initially came up with the ideas, it's much more engaging to read the detailed stories of _how_ that person came to develop the idea; Brian's book delivers this again and again, functioning like a well-organized literature survey that is also fun to read because of its great storytelling. I struggled a fair amount in writing this summary, because I kept wanting to somehow communicate the writing style; in the end I decided not to do it and to instead give a few examples of passages from the book in [this post](https://www.alignmentforum.org/posts/gYfgWSxCpFdk2cZfE/the-alignment-problem-machine-learning-and-human-values).

The Alignment Problem

This book starts off with an explanation of machine learning and problems that we can currently see with it, including detailed stories and analysis of:

- The [gorilla misclassification incident](https://twitter.com/jackyalcine/status/615329515909156865)
- The [faulty reward in CoastRunners](https://openai.com/blog/faulty-reward-functions/)
- The [gender bias in language models](https://arxiv.org/abs/1607.06520)
- The [failure of facial recognition models on minorities](https://www.ted.com/talks/joy_buolamwini_how_i_m_fighting_bias_in_algorithms)
- The [COMPAS](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing) [controversy](https://www.documentcloud.org/documents/<PHONE>-ProPublica-Commentary-Final-070616.html) (leading up to [impossibility results in fairness](https://www.chrisstucchio.com/pubs/slides/crunchconf_2018/slides.pdf))
- The [neural net that thought asthma reduced the risk of pneumonia](https://www.microsoft.com/en-us/research/wp-content/uploads/2017/06/KDD2015FinalDraftIntelligibleModels4HealthCare_igt143e-caruanaA.pdf)

It then moves on to agency and reinforcement learning, covering from a more historical and academic perspective how we have arrived at such ideas as temporal difference learning, reward shaping, curriculum design, and curiosity, across the fields of machine learning, behavioral psychology, and neuroscience. While the connections aren't always explicit, a knowledgeable reader can connect the academic examples given in these chapters to the ideas of <@specification gaming@>(@Specification gaming: the flip side of AI ingenuity@) and <@mesa optimization@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@) that we talk about frequently in this newsletter. Chapter 5 especially highlights that agent design is not just a matter of specifying a reward: often, rewards will do ~nothing, and the main requirement to get a competent agent is to provide good _shaping rewards_ or a good _curriculum_. Just as in the previous part, Brian traces the intellectual history of these ideas, providing detailed stories of (for example):

- BF Skinner's experiments in [training pigeons](https://psycnet.apa.org/record/1961-01933-001)
- The invention of the [perceptron](https://psycnet.apa.org/record/1959-09865-001)
- The success of [TD-Gammon](https://www.aaai.org/Papers/Symposia/Fall/1993/FS-93-02/FS93-02-003.pdf), and later [AlphaGo Zero](https://deepmind.com/blog/article/alphago-zero-starting-scratch)

The final part, titled "Normativity", delves much more deeply into the alignment problem. While the previous two parts are partially organized around AI capabilities -- how to get AI systems that optimize for _their_ objectives -- this last one tackles head on the problem that we want AI systems that optimize for _our_ (often-unknown) objectives, covering such topics as imitation learning, inverse reinforcement learning, learning from preferences, iterated amplification, impact regularization, calibrated uncertainty estimates, and moral uncertainty.


Engineering a Safer World

I recently read [Engineering a Safer World](https://static1.squarespace.com/static/53b78765e4b0949940758017/t/57d87eb6d2b8571af3501b26/1473898764674/Engineering_a_Safer_World+Nancy+Leveson.pdf) by Nancy G. Leveson, at [Joshua Achiam's recommendation](https://twitter.com/jachiam0/status/1285449712573665281), and really enjoyed it, so get ready for another book summary! I'm not very happy with the summary I have -- it feels less compelling than the book, partly because the book provides a ton of examples that I don't have the space to do -- but hopefully it is enough to get the key points across.

The main motivation of this book is to figure out how we can improve safety engineering. Its primary thesis is that the existing methods used in engineering are insufficient for the current challenges, and must be replaced by a method the author favors called STAMP. Note that the book is primarily concerned with mechanical systems that may also have computerized automation (think aerospace, chemical, and mechanical engineering); the conclusions should not be expected to apply directly to AI.

**The standard model of safety engineering and its deficiencies**

Historically, safety engineering has been developed as a reaction to the high level of accidents we had in the past, and as a result focused on the easiest gains first. In particular, there were a lot of gains to be had simply by ensuring that _machines didn't break_. _(Rohin's note: I'm editorializing a bit here, the author doesn't explicitly say this but I think she believes it.)_ This led to a focus on _reliability_: given a specification for how a machine should operate, we aim to decrease the probability that the machine fails to meet that specification. For example, the specification for a water tank would be to contain the water up to a given pressure, and one way to improve the reliability of the tank would be to use a stronger material or make a thicker tank to make it less likely that the tank ruptures.

Under this model, an accident happens when a machine fails to meet its specification. So, we can analyze the accident by looking at what went wrong, and tracing back the physical causes to the first point at which a specification was not met, giving us a _root cause_ that can show us what we need to fix in order to prevent similar accidents in the future. We can call this sort of analysis an _event chain_ analysis.

However, in the last few decades there have been quite a few changes that make this model worse than it once was. The pace of technological change has risen, making it harder to learn from experience. The systems we build have become complex enough that there is a lot more _coupling_ or interaction effects between parts of the system that we could fail to account for.  Relatedly, the risks we face are getting large enough that we aren't willing to tolerate even a _single_ accident. Human operators (e.g. factory workers) are no longer able to rely on easily understood and predictable mechanical systems, instead having to work with computerized automation which they cannot understand as well. At this point, event chain analysis and safety-via-reliability are no longer sufficient for safety engineering.

Consider for example the [Flight 965](https://en.wikipedia.org/wiki/American_Airlines_Flight_965) accident. In this case, the pilots got clearance to fly towards the Rozo waypoint in their descent, listed as (R) on their (paper) approach charts. One of the pilots pressed R in the flight management system (FMS), which brought up a list of waypoints that did _not_ include Rozo, and executed the first one (presumably believing that Rozo, being the closest waypoint, would show up first). As a result, the plane turned towards the selected waypoint, and crashed into a mountain.

The accident report for this incident placed the blame squarely on the pilots, firstly for not planning an appropriate path, and secondly for not having situational awareness of the terrain and that they needed to discontinue their approach. But most interestingly, the report blames the pilots for not reverting to basic radio navigation when the FMS became confusing. The author argues that the design of the automation was also flawed in this case, as the FMS stopped displaying the intermediate fixes to the chosen route, and the FMS's navigational information used a different naming convention than the one in the approach charts. Surely this also contributed to the loss? In fact, in lawsuit appeals, the software manufacturer was held to be 17% liable.

However, the author argues that this is the exception, not the rule: typically event chain analysis proceeds until a human operator is found who did something unexpected, and then the blame can safely be placed on them. Operators are expected to "use common sense" to deviate from procedures when the procedures are unsafe, but when an accident happens, blame is placed on them for deviating from procedures. This is often very politically convenient, and is especially easy to justify thanks to hindsight bias, where we can identify exactly the right information and cues that the operator "should have" paid attention to, ignoring that in the moment there were probably many confusing cues and it was far from obvious which information to pay attention to. My favorite example has to be this quote from an accident report:

"Interviews with operations personnel did not produce a clear reason why the response to the [gas] alarm took 31 minutes. The only explanation was that there was not a sense of urgency since, in their experience, previous [gas] alarms were attributed to minor releases that did not require a unit evacuation."

It is rare that I see such a clear example of a self-refuting paragraph. In the author's words, "this statement is puzzling, because the statement itself provides a clear explanation for the behavior, that is, the previous experience". It definitely sounds like the investigators searched backwards through the causal chain, found a situation where a human deviated from protocol, and decided to assign blame there.

This isn't just a failure of the accident investigation -- the entire premise of some "root cause" in an event chain analysis implies that the investigators must end up choosing some particular point to label as The Root Cause, and such a decision is inevitably going to be determined more by the particular analysts involved rather than by features of the accident.

**Towards a new approach**

How might we fix the deficiencies of standard safety engineering? The author identifies several major changes in assumptions that are necessary for a new approach:

1. Blame is the enemy of safety. Safety engineering should focus on system behavior as a whole, where interventions can be made at many points on different levels, rather than seeking to identify a single intervention point.

2. Reliability (having machines meet their specifications) is neither necessary nor sufficient for safety (not having bad outcomes). Increased reliability can lead to decreased safety: if we increase the reliability of the water tank by making it using a stronger material, we may decrease the risk of rupture, but we may dramatically increase the harm when a rupture occurs since the water will be at a much higher pressure. This applies to software as well: highly reliable software need not be safe, as its specifications may not be correct.

3. Accidents involve the entire sociotechnical system, for which an event chain model is insufficient. Interventions on the sociological level (e.g. make it easy for low-level operators to report problems) should be considered part of the remit of safety engineering.

4. Major accidents are not caused by simultaneous occurrence of random chance events. Particularly egregious examples come from probabilistic risk analysis, where failures of different subsystems are often assumed to be independent, neglecting the possibility of a common cause, whether physical (e.g. multiple subsystems failing during a power outage) or sociological (e.g. multiple safety features being disabled as part of cost-cutting measures). In addition, systems tend to migrate towards higher risk over time, because environmental circumstances change, and operational practices diverge from the designed practices as they adapt to the new circumstances, or simply to be more efficient.

5. Operator behavior is a product of the environment in which it occurs. To improve safety, we must change the environment rather than the human. For example, if an accident occurs and an operator didn't notice a warning light that could have let them prevent it, the solution is not to tell the operators to "pay more attention" -- that approach is doomed to fail.

**A detour into systems theory**

The new model proposed by the author is based on systems theory, so let's take a moment to describe it. Consider possible systems that we may want to analyze:

First, there are some systems with _organized simplicity_, in which it is possible to decompose the system into several subsystems, analyze each of the subsystems independently, and then combine the results relatively easily to reach overall conclusions. We might think of these as systems in which analytic reduction is a good problem-solving strategy. _Rohin's note: Importantly, this is different from the philosophical question of whether there exist phenomena that cannot be reduced to e.g. physics: that is a question about whether reduction is in principle possible, whereas this criterion is about whether such reduction is an effective strategy for a computationally bounded reasoner._ Most of physics would be considered to have organized simplicity.

Second, there are systems with _unorganized complexity_, where there is not enough underlying structure for analytic reduction into subsystems to be a useful tool. However, in such systems the behavior of individual elements of the system is sufficiently random (or at least, well-modeled as random) that statistics can be applied to it, and then the law of large numbers allows us to understand the system as an aggregate. A central example would be statistical mechanics, where we cannot say much about the motion of individual particles in a gas, but we can say quite a lot about the macroscopic behavior of the gas as a whole.

Systems theory deals with systems that have _organized complexity_. Such systems have enough organization and structure that we cannot apply statistics to it (or equivalently, the assumption of randomness is too incorrect), and are also sufficiently complex that analytic reduction is not a good technique (e.g. perhaps any potential decomposition into subsystems would be dominated by combinatorially many interaction effects between subsystems). Sociological systems are central examples of such systems: the individual components (humans) are very much not random, but neither are their interactions governed by simple laws as would be needed for analytic reduction. While systems theory cannot provide nearly the same level of precision as statistics or physics, it does provide useful concepts for thinking about such systems.

The first main concept in systems theory is that of _hierarchy and emergence_. The idea here is that systems with organized complexity can be decomposed into several hierarchical levels, with each level built "on top of" the previous one. For example, companies are built on top of teams which are built on top of individual employees. The behavior of components in a particular layer is described by some "language" that is well-suited for that layer. For example, we might talk about individual employees based on their job description, their career goals, their relationship with their manager, and so on, but we might talk about companies based on their overall direction and strategy, the desires of their customer base, the pressures from regulators, and so on.

_Emergence_ refers to the phenomenon that there can be properties of higher levels arising from lawful interactions at lower levels that nonetheless are meaningless in the language appropriate for the lower levels. For example, it is quite meaningful to say that the pressures on a company from government regulation caused them to (say) add captions to their videos, but if we look at the specific engineer who integrated the speech recognition software into the pipeline, we would presumably say "she integrated the speech recognition into the pipeline because she had previously worked with the code" rather than "she integrated it because government regulations told her to do so". As another example, safety is an emergent system property, while reliability is not.

The second main concept is that of _control_. We are usually not satisfied with just understanding the behavior of systems; we also want to make changes to it (as in the case of making them safer). In systems theory, this is thought of as _control_, where we impose some sort of _constraint_ on possible system behavior at some level. For example, employee training is a potential control action that could aim to enforce the constraint that every employee knows what to do in an emergency. An effective controller requires a goal, a set of actions to take, a model of the system, and some way to sense the state of the system.

**STAMP: A new model underlying safety engineering**

The author then introduces a new model called Systems-Theoretic Accident Model and Processes (STAMP), which aims to present a framework for understanding how accidents occur (which can allow us to prevent them and/or learn from them). It contains three main components:

_Safety constraints_: In systems theory, a constraint is the equivalent of a specification, so these are just the safety-relevant specifications. Note that such specifications can be found at all levels of the hierarchy.

_Hierarchical safety controllers_: We use _controllers_ to enforce safety constraints at any given level. A control algorithm may be implemented by a mechanical system, a computerized system, or humans, and can exist at any level of the hierarchy. A controller at level N will typically depend on constraints at level N - 1, and thus the design of this controller influences which safety constraints are placed at level N - 1.

_Process models_: An effective controller must have a model of the process it is controlling. Many accidents are the result of a mismatch between the actual process and the process model of the controller.

This framework can be applied towards several different tasks, and in all cases the steps are fairly similar: identify the safety constraints you want, design or identify the controllers enforcing those constraints, and then do some sort of generic reasoning with these components.

If an accident occurs, then at the highest level, either the control algorithm(s) failed to enforce the safety constraints, or the control actions were sent correctly but were not followed. In the latter case, the controllers at the lower level should then be analyzed to see why the control actions were not followed. Ultimately, this leads to an analysis on multiple levels, which can identify several things that went wrong rather than one Root Cause, that can all be fixed to improve safety in the future.

**Organizational safety**

So far we've covered roughly chapters 1-4 of the book. I'll now jump straight to chapter 13, which seems particularly important and relevant, as it deals with how organizational structure and management should be designed to support safety.

One major point that the author makes is that safety _is_ cost-effective for _long-term_ performance as long as it is designed into the system from the start, rather than added on at the last minute. Performance pressure on the other hand inevitably leads to cuts in safety.

In order to actually get safety designed into the system from the start, it is crucial that top management demonstrates a strong commitment to safety, as without this employees will inevitably cut corners on safety as they will believe it is in their incentives to do so. Other important factors include a concrete corporate safety policy, as well as a strong corporate safety culture. It is important that safety is part of the design process, rather than tacked on at the end. In the author's words, _putting safety into the quality assurance organization is the worst place for it. [...] It sets up the expectation that safety is an after-the-fact or auditing activity only._

In addition, it is important that information can flow well. Going from the bottom to the top, it should be possible for low-level operators to report potential problems in a way that they are actually acted on and the relevant information reaches top management.  From the top to the bottom, safety information and training should be easily available and accessible to employees when they need it.

It is also important to have controls to prevent the general tendency of systems to migrate towards higher risk, e.g. by relaxing safety requirements as time passes without any incidents. The next chapter describes SUBSAFE, the author's example of a well-run safety program, in which the control is for everyone to periodically watch a video reminding them of the importance of their particular safety work (in particular, the video shows the loss of the USS Thresher, an event that caused SUBSAFE to be created).

Perhaps obviously, it is important for an organization to have a dedicated safety team. This is in contrast to making everyone responsible for safety. In the author's words: _While, of course, everyone should try to behave safely and to achieve safety goals, someone has to be assigned responsibility for ensuring that the goals are achieved._

If you start by designing for safety, it is cost-effective, not opposed to long-term money-maximizing. Once there is performance pressure, then you see cuts in safety. Also sometimes people fix symptoms instead of underlying causes, and then they just keep seeing symptoms forever and conclude they are inevitable.

**Miscellaneous notes**

The remaining chapters of the book apply STAMP in a bunch of different areas with many examples, including an entire chapter devoted to the STAMP treatment of a friendly fire accident. I also really liked the discussion of human factors in the book, but decided not to summarize it as this has already gotten quite long.

**Summary of the summary**

I'll conclude with a quote from the book's epilogue:

_What seems to distinguish those experiencing success is that they:_
_1. Take a systems approach to safety in both development and operations_
_2. Have instituted a learning culture where they have effective learning from events_
_3. Have established safety as a priority and understand that their long-term success depends on it_

**Relationship to AI safety**

A primary motivation for thinking about AI is that it would be very impactful for our society, and very impactful technologies need not have good impacts. "Society" clearly falls into the "organized complexity" class of systems, and so I expect that the ideas of safety constraints and hierarchical control algorithms will be useful ways to think about possible impacts of AI on society. For example, if we want to think about the possibility of AI systems differentially improving technical progress over "wisdom", such that we get dangerous technologies before we're ready for them, we may want to sketch out hierarchical "controllers" at the societal level that could solve this problem. Ideally these would eventually turn into constraints on the AI systems that we build, e.g. "AI systems should report potentially impactful new technologies to such-and-such committee". I see the AI governance field as doing this sort of work using different terminology.

Technical AI alignment (in the sense of <@intent alignment@>(@Clarifying "AI Alignment"@)) does not seem to benefit as much from this sort of an approach. The main issue is that we are often considering a fairly unitary system (such as a neural net, or the mathematical model of expected utility maximization) to which the hierarchical assumption of systems theory does not really apply.

To be clear, I _do_ think that there in fact is some hierarchy. For example, in image classifiers where low levels involve edge detectors while high levels involve dog-face detectors. However, we do not have the language to talk about these hierarchies, nor the algorithms to control the intermediate layers. While <@Circuits@>(@Thread: Circuits@) is illustrating this hierarchy for image classifiers, it does not give us a language that we can (currently) use to talk about advanced AI systems. As a result, we are reduced to focusing on the incentives we provide to the AI system, or speculating on the levels of hierarchy that might be internal to advanced AI systems, neither of which seem particularly conducive to good work.

In the language of this book, I work on intent alignment because I expect that the ability to enforce the constraint "the AI system tries to do what its operator wants" will be a very useful building block for enforcing whatever societal safety constraints we eventually settle on, and it seems possible to make progress on it today. There are several arguments for risk that this ignores (see e.g. <@here@>(@The Main Sources of AI Risk?@) and <@here@>(@AI Research Considerations for Human Existential Safety@)); for some of these other risks, the argument is that we can handle those using similar mechanisms as we have before (e.g. governance, democracy, police, etc), _as long as_ we have handled intent alignment.


I'm glad this was finally written down -- I've been calling this the "induction hammer" and have used it a lot in my own thinking. Thinking about this sort of a model, and in particular what kinds of properties we could best preserve inductively, has been quite helpful for me.

A general model of safety-oriented AI development

A general model for developing safe powerful AI systems is to have a team of humans and AIs, which continually develops and adds more AIs to the team, while inductively preserving alignment.


I refer to convergent instrumental subgoals quite often in this newsletter, so it seemed like I should have a summary of it. I especially like this paper because it holds up pretty well 12 years later. Even though I’ve <@critiqued@>(@Conclusion to the sequence on value learning@) the idea that powerful AI systems must be expected utility maximizers, I still find myself agreeing with this paper, because it _assumes_ a goal-directed agent and reasons from there, rather than trying to argue that powerful AI systems must be goal-directed. Given that assumption, I agree with the conclusions drawn here.

The Basic AI Drives

This paper from 2008 introduces convergent instrumental subgoals: the subgoals that an AI system will have "by default", unless care is taken to avoid them. For this paper, an AI system is a system that "has goals which it tries to accomplish by acting in the world", i.e. it assumes that the system is <@goal-directed@>(@Intuitions about goal-directed behavior@).

It starts by arguing that a sufficiently powerful goal-directed AI system will want to self-improve, as that could help it achieve its goals better in the (presumably long) future. In particular, it will want to become "rational", in the sense that it will want to maximize its _expected_ utility, where the utility function is determined by its goal. (The justification for this is the VNM theorem, and the various Dutch book arguments that support Bayesianism and expected utility maximization.)

However, not all modifications would be good for the AI system. In particular, it will very strongly want to preserve its utility function, as that determines what it will (try to) accomplish in the future, and any change in the utility function would be a disaster from the perspective of the current utility function. Similarly, it will want to protect itself from harm, that is, it has a survival incentive, because it can't accomplish its goal if it's dead.

The final instrumental subgoal is to acquire resources and use them efficiently in pursuit of its goal, because almost by definition resources are useful for a wide variety of goals, including (probably) the AI system's goal.


While I agree that the COVID response was worse than it could have been, I think there are several important disanalogies between the COVID-19 pandemic and a soft takeoff scenario, which I elaborate on in [this comment](https://www.alignmentforum.org/posts/wTKjRFeSjKLDSWyww/possible-takeaways-from-the-coronavirus-pandemic-for-slow-ai?commentId=pKNQ9oq72HFS2JrdH). First, with COVID there were many novel problems, which I don’t expect with AI. Second, I expect a longer time period over which decisions can be made for AI alignment. Finally, with AI alignment, we have the option of preventing problems from ever arising, which is not really an option with pandemics. See also [this post](https://www.lesswrong.com/posts/EgdHK523ZM4zPiX5q/coronavirus-as-a-test-run-for-x-risks#Implications_for_X_risks).

Possible takeaways from the coronavirus pandemic for slow AI takeoff

The COVID-19 pandemic is an example of a large risk that humanity faced. What lessons can we learn for AI alignment? This post argues that the pandemic is an example of the sort of situation we can expect in a slow takeoff scenario, since we had the opportunity to learn from experience, act on warning signs, and reach a timely consensus that there is a serious problem. However, while we could have learned from previous epidemics like SARS, we failed to generalize the lessons from SARS. Despite warning signs of a pandemic in February, many countries wasted a month when they could have been stocking up on PPE and testing capacity. We had no consensus that COVID-19 was a problem, with articles dismissing it as no worse than the flu as late as March.

All of these problems could also happen with slow takeoff: we may fail to generalize from narrow AI systems to more general AI systems; we might not act on warning signs; and we may not believe that powerful AI is on the horizon until it is too late. The conclusion is "unless more competent institutions are in place by the time general AI arrives, it is not clear to me that slow takeoff would be much safer than fast takeoff".


I’m glad this post exists: while most of the specific proposals could be found by patching together content spread across other blog posts, there was a severe lack of a single article laying out a full picture for even one proposal, let alone all eleven in this post.

I usually don’t think about outer alignment as what happens with optimal policies, as assumed in this post -- when you’re talking about loss functions _in the real world_ (as I think this post is trying to do), _optimal_ behavior can be weird and unintuitive, in ways that may not actually matter. For example, arguably for any loss function, the optimal policy is to hack the loss function so that it always outputs zero (or perhaps negative infinity).

An overview of 11 proposals for building safe advanced AI

This post describes eleven "full" AI alignment proposals (where the goal is to build a powerful, beneficial AI system using current techniques), and evaluates them on four axes:

1. **Outer alignment:** Would the optimal policy for the specified loss function be aligned with us? See also [this post](https://www.alignmentforum.org/posts/33EKjmAdKFn3pbKPJ/outer-alignment-and-imitative-amplification).
2. **Inner alignment:** Will the model that is _actually produced_ by the training process be aligned with us?
3. **Training competitiveness:** Is this an efficient way to train a powerful AI system? More concretely, if one team had a "reasonable lead" over other teams, would they keep at least some of the lead if they used this algorithm?
4. **Performance competitiveness:** Will the trained model have good performance (relative to other models that could be trained)?

Seven of the eleven proposals are of the form "recursive outer alignment technique" plus "<@technique for robustness@>(@Worst-case guarantees (Revisited)@)". The recursive outer alignment technique is either <@debate@>(@AI safety via debate@), <@recursive reward modeling@>(@Scalable agent alignment via reward modeling@), or some flavor of <@amplification@>(@Capability amplification@). The technique for robustness is either transparency tools to "peer inside the model", <@relaxed adversarial training@>(@Relaxed adversarial training for inner alignment@), or intermittent oversight by a competent supervisor. An additional two proposals are of the form "non-recursive outer alignment technique" plus "technique for robustness" -- the non-recursive techniques are vanilla reinforcement learning in a multiagent environment, and narrow reward learning.

Another proposal is Microscope AI, in which we train AI systems to simply understand vast quantities of data, and then by peering into the AI system we can learn the insights that the AI system learned, leading to a lot of value. We wouldn't have the AI system act in the world, thus eliminating a large swath of potential bad outcomes. Finally, we have STEM AI, where we try to build an AI system that operates in a sandbox and is very good at science and engineering, but doesn't know much about humans. Intuitively, such a system would be very unlikely to deceive us (and probably would be incapable of doing so).

The post contains a lot of additional content that I didn't do justice to in this summary. In particular, I've said nothing about the analysis of each of these proposals on the four axes listed above; the full post talks about all 44 combinations.


As one of the people interviewed, these seem like the right high-level takeaways to me.

Takeaways from safety by default interviews

This post lists three key takeaways from AI Impacts' conversations with "optimistic" researchers (summarized mainly in [AN #80](https://mailchi.mp/b3dc916ac7e2/an-80-why-ai-risk-might-be-solved-without-additional-intervention-from-longtermists) with one in [AN #63](https://mailchi.mp/533c646a4b21/an-63how-architecture-search-meta-learning-and-environment-design-could-lead-to-general-intelligence)). I'll just name the takeaways here, see the post for more details:

1. Relative optimism in AI often comes from the belief that AGI will be developed gradually, and problems will be fixed as they are found rather than neglected.
2. Many of the arguments I heard around relative optimism weren't based on inside-view technical arguments.
3. There are lots of calls for individuals with views around AI risk to engage with each other and understand the reasoning behind fundamental disagreements.


I enjoyed this post, and recommend reading it in full if you are interested in AI risk because of effective altruism. (I've kept the summary relatively short because not all of my readers care about effective altruism.) My personal cruxes and story of impact are actually fairly different: in particular, while this post sees the impact of research as coming from solving the technical alignment problem, I care about other sources of impact as well. See [this comment](https://forum.effectivealtruism.org/posts/Ayu5im98u8FeMWoBZ/my-personal-cruxes-for-working-on-ai-safety#vWuF8NCs7JQWv9XM3) for details.

My personal cruxes for working on AI safety

This post describes how Buck's cause prioritization within an [effective altruism](https://www.effectivealtruism.org/) framework leads him to work on AI risk. The case can be broken down into a conjunction of five cruxes. Specifically, the story for impact is that 1) AGI would be a big deal if it were created, 2) has a decent chance of being created soon, before any other "big deal" technology is created, and 3) poses an alignment problem for which we can think ahead in order to solve, and it's potentially valuable to do so even given the fact that people might try to solve this later. His research 4) would be put into practice if it solved the problem and 5) makes progress on solving the problem.


I can remember being critical of AI risk myself for similar reasons and think that it is important to be careful with the framing of pitches to avoid these heuristics from firing. This is not to say that we should avoid criticism of the idea of AI risk, but criticism is a lot more helpful if it comes from people who have actually engaged with the arguments.

**Rohin's opinion:** Even after knowing the arguments, I find six of the heuristics quite compelling: technology doomsayers have usually been wrong in the past, there isn't a concrete threat model, it's not empirically testable, it's too extreme, it isn't well grounded in my experience with existing AI systems, and it's too far off to do useful work now. All six make me distinctly more skeptical of AI risk.

A list of good heuristics that the case for AI x-risk fails

Because human attention is limited and a lot of people try to convince us of the importance of their favourite cause, we cannot engage with everyone's arguments in detail. Thus we have to rely on heuristics to filter out insensible arguments. Depending on the form of exposure, the case for AI risks can fail on many of these generally useful heuristics, eight of which are detailed in this post. Given this outside view perspective, it is unclear whether we should actually expect ML researchers to spend time evaluating the arguments for AI risk.


Developing transparency tools is currently my best guess for how we can avoid deception and catastrophic planning in our AI systems. I'm most excited about applying transparency techniques via the first and third routes, which primarily help us audit our models. I'm more pessimistic about the fourth approach because it predictably involves restructuring the incentives for machine learning as a field, which is quite difficult. My opinion might be different if we could somehow coordinate the development of these technologies.

Chris Olah’s views on AGI safety

This post is Evan's best attempt to summarize [Chris Olah](https://colah.github.io/about.html)'s views on how transparency is a vital component for building safe artificial intelligence, which he distinguishes into four separate approaches:

First, we can apply interpretability to audit our neural networks, or in other words, catch problematic reasoning in our models. Second, transparency can help safety by allowing researchers to deliberately structure their models in ways that systematically work, rather than using machine learning as a black box. Third, understanding transparency allows us to directly incentivize for transparency in model design and decisions -- similar to how we grade humans on their reasoning (not just the correct answer) by having them show their work. Fourth, transparency might allow us to reorient the field of AI towards microscope AI: AI that gives us new ways of understanding the world, enabling us to be more capable, without itself taking autonomous actions.

Chris expects that his main disagreement with others is whether good transparency is possible as models become more complex. He hypothesizes that as models become more advanced, they will counterintuitively become more interpretable, as they will begin using more crisp human-relatable abstractions. Finally, Chris recognizes that his view implies that we might have to re-align the ML community, but he remains optimistic because he believes there's a lot of low-hanging fruit, research into interpretability allows low-budget labs to remain competitive, and interpretability is aligned with the scientific virtue to understand our tools.


Clarifying some key hypotheses in AI alignment

This post (that I contributed to) introduces a diagram that maps out important and controversial hypotheses for AI alignment. The goal is to help researchers identify and more productively discuss their disagreements.


I do agree that in the framework outlined in this post (the "ideal" being just a search over "feasible" strategies) the ideal solution doesn't give you much insight, but I don't think this is fully true of eg. Bayes rule. I do think that understanding Bayes rule can help you [make better decisions](https://80000hours.org/podcast/episodes/spencer-greenberg-bayesian-updating/), because it gives you a quantitative framework of how to work with hypotheses and evidence, which even simple feasible strategies can use. (Although I do think that logically-omniscient Bayes does not add much over regular Bayes rule from the perspective of suggesting a feasible strategy to use -- but in the world where logically-omniscient Bayes came first, it would have been helpful to derive the heuristic.) In the framework of the post, this corresponds to the choice of "weight" assigned to each hypothesis, and this is useful because feasible strategies do still look like search (but instead of searching over all hypotheses, you search over a very restricted subset of them). So overall I think I agree with the general thrust of the post, but don't agree with the original strong claim that 'grappling with embeddedness properly will inevitably make theories of this general type irrelevant or useless, so that "a theory like this, except for embedded agents" is not a thing that we can reasonably want'.

When does rationality-as-search have nontrivial implications?

Many theories of idealized intelligence, such as Solomonoff induction, logical inductors and Bayesian reasoning, involve a large search over a space of strategies and using the best-performing one, or a weighted combination where the weights depend on past performance. However, the procedure that involves the large search is not itself part of the space of strategies -- for example, Solomonoff induction searches over the space of computable programs to achieve near-optimality at prediction tasks relative to any computable program, but is itself uncomputable. When we want to actually implement a strategy, we have to choose one of the options from our set, rather than the infeasible idealized version, and the idealized version doesn't help us do this. It would be like saying that a chess expert is approximating the rule "consult all possible chess players weighted by past performance" -- it's true that these will look similar _behaviorally_, but they look very different _algorithmically_, which is what we actually care about for building systems.


This is an interesting theoretical analysis, and I'm surprised that the traditional ML approach seems to do so well in a context it wasn't designed for. I'm not sure about the part where it would converge to a fixed point of the function g, I've written a rambly comment on the post trying to explain more.

Standard ML Oracles vs Counterfactual ones

_(Note: This summary has more of my interpretation than usual.)_ Consider the setting where an AI system is predicting some variable y_t = f(x_t), but we will use the AI's output to make decisions that could affect the true value of y_t. Let's call the AI's prediction z_t, and have y_t = g(x_t, z_t), where g captures how humans use z_t to affect the value of y_t. The traditional ML approach would be to find the function f that minimizes the distance between y and f(x) on past examples, but this does not typically account for y_t depending on z_t. We would expect that it would converge to outputting a fixed point of g (so that y_t = z_t = g(x_t, z_t)), since that would minimize its loss. This would generally perform well; while manipulative predictions z_t are possible, they are unlikely. The main issue is that since the system does not get to observe z_t (since that is what it is predicting), it cannot model the true causal formulation, and has to resort to complex hypotheses that approximate it. This can lead to overfitting that can't be simply solved by regularization or simplicity priors. Instead, we could use a counterfactual oracle, which reifies the prediction z_t and then outputs the z_t that minimizes the distance between z_t and y_t, which allows it to model the causal connection between y_t = g(x_t, z_t).


Rohin Shah on the State of AGI Safety Research in 2021

As in [previous](https://futureoflife.org/2019/04/11/an-overview-of-technical-ai-alignment-with-rohin-shah-part-1/) [years](https://futureoflife.org/2019/04/25/an-overview-of-technical-ai-alignment-with-rohin-shah-part-2/) ([AN #54](https://mailchi.mp/3e2f43012b07/an-54-boxing-a-finite-horizon-ai-system-to-keep-it-unambitious)), on this FLI podcast I talk about the state of the field. Relative to previous years, this podcast is a bit more introductory, and focuses a bit more on what I find interesting rather than what the field as a whole would consider interesting.


I enjoyed this article quite a lot; it is _really_ thorough. I do see a lot of my own work as pushing on some of these more incremental methods for increasing caution, though I think of it more as a combination of generating more or better evidence and communicating arguments in a manner more suited to a particular audience. Perhaps I will think of new strategies that aim to reduce fear shame instead.

Beyond fire alarms: freeing the groupstruck

It has been claimed that [there's no fire alarm for AGI](https://intelligence.org/2017/10/13/fire-alarm/), that is, there will be no specific moment or event at which AGI risk becomes sufficiently obvious and agreed upon, so that freaking out about AGI becomes socially acceptable rather than embarrassing. People often implicitly argue for waiting for an (unspecified) future event that tells us AGI is near, after which everyone will know that it's okay to work on AGI alignment. This seems particularly bad if no such future event (i.e. fire alarm) exists.

This post argues that this is not in fact the implicit strategy that people typically use to evaluate and respond to risks. In particular, it is too discrete. Instead, people perform "the normal dance of accumulating evidence and escalating discussion and brave people calling the problem early and eating the potential embarrassment". As a result, the existence of a "fire alarm" is not particularly important.

Note that the author does agree that there is some important bias at play here. The original fire alarm post is implicitly considering a _fear shame hypothesis_: people tend to be less cautious in public because they expect to be negatively judged for looking scared. The author ends up concluding that there is something broader going on and proposes a few possibilities, many of which still suggest that people will tend to be less cautious around risks when they are observed.

Some points made in the very detailed, 15,000-word article:

1. Literal fire alarms don't work by creating common knowledge, or by providing evidence of a fire. People frequently ignore fire alarms. In [one experiment](https://www.youtube.com/watch?v=MtX-10c3fT0), participants continued to fill out questionnaires while a fire alarm rang, often assuming that someone will lead them outside if it is important.

2. They probably instead work by a variety of mechanisms, some of which are related to the fear shame hypothesis. Sometimes they provide objective evidence that is easier to use as a justification for caution than a personal guess. Sometimes they act as an excuse for cautious or fearful people to leave, without the implication that those people are afraid. Sometimes they act as a source of authority for a course of action (leaving the building).

3. Most of these mechanisms are amenable to partial or incremental effects, and in particular can happen with AGI risk. There are many people who have already boldly claimed that AGI risk is a problem. There exists person-independent evidence; for example, surveys of AI researchers suggest a 5% chance of extinction.

4. For other risks, there does not seem to have been a single discrete moment at which it became acceptable to worry about them (i.e. no "fire alarm"). This includes risks where there has been a lot of caution, such as climate change, the ozone hole, recombinant DNA, COVID, and nuclear weapons.

5. We could think about _building_ fire alarms; many of the mechanisms above are social ones rather than empirical facts about the world. This could be one out of many strategies that we employ against the general bias towards incaution (the post suggests 16).


I know I already harped on this in the summary, but these numbers are ridiculously non-robust and involve tons of selection biases. You probably shouldn’t conclude much from them about how much risk from AI there really is. Don’t be the person who links to this survey with the quote “experts predict 30% chance of doom from AI”.

"Existential risk from AI" survey results

This post reports on the results of a survey sent to about 117 people working on long-term AI risk (of which 44 responded), asking about the magnitude of the risk from AI systems. I'd recommend reading the exact questions asked, since the results could be quite sensitive to the exact wording, and as an added bonus you can see the visualization of the responses. In addition, respondents expressed _a lot_ of uncertainty in their qualitative comments. And of course, there are all sorts of selection effects that make the results hard to interpret.

Keeping those caveats in mind, the headline numbers are that respondents assigned a median probability of 20% to x-risk caused due to a lack of enough technical research, and 30% to x-risk caused due to a failure of AI systems to do what the people deploying them intended, with huge variation (for example, there are data points at both ~1% and ~99%).


The notion of “natural abstractions” seems quite important to me. There are at least some weak versions of the hypothesis that seem obviously true: for example, if you ask GPT-3 some new type of question it has never seen before, you can predict pretty confidently that it is still going to respond with real words rather than a string of random characters. This is effectively because you expect that GPT-3 has learned the “natural abstraction” of the words used in English and that it uses this natural abstraction to drive its output (leaving aside the cases where it must produce output in some other language).

The version of the natural abstraction hypothesis investigated here seems a lot stronger and I’m excited to see how the project turns out. I expect the author will post several short updates over time; I probably won’t cover each of these individually and so if you want to follow it in real time I recommend following it on the Alignment Forum.

Section 1.3 of [this version of the paper](https://mltheory.org/dg_short.pdf)

Testing The Natural Abstraction Hypothesis: Project Intro

We've previously seen some discussion about <@abstraction@>(@Public Static: What is Abstraction?@), and some [claims](https://www.lesswrong.com/posts/wopE4nT28ausKGKwt/classification-of-ai-alignment-research-deconfusion-good?commentId=cKNrWxfxRgENS2EKX) that there are "natural" abstractions, or that AI systems will <@tend@>(@Chris Olah's views on AGI safety@) to <@learn@>(@Conversation with Rohin Shah@) increasingly human-like abstractions (at least up to a point). To make this more crisp, given a system, let's consider the information (abstraction) of the system that is relevant for predicting parts of the world that are "far away". Then, the **natural abstraction hypothesis** states that:

1. This information is much lower-dimensional than the system itself.
2. These low-dimensional summaries are exactly the high-level abstract objects/concepts typically used by humans.
3. These abstractions are "natural", that is, a wide variety of cognitive architectures will learn to use approximately the same concepts to reason about the world.

For example, to predict the effect of a gas in a larger system, you typically just need to know its temperature, pressure, and volume, rather than the exact positions and velocities of each molecule of the gas. The natural abstraction hypothesis predicts that many cognitive architectures would all converge to using these concepts to reason about gases.

If the natural abstraction hypothesis were true, it could make AI alignment dramatically simpler, as our AI systems would learn to use approximately the same concepts as us, which can help us both to "aim" our AI systems at the right goal, and to peer into our AI systems to figure out what exactly they are doing. So, this new project aims to test whether the natural abstraction hypothesis is true.

The first two claims will likely be tested empirically. We can build low-level simulations of interesting systems, and then compute what summary is useful for predicting its effects on "far away" things. We can then ask how low-dimensional that summary is (to test (1)), and whether it corresponds to human concepts (to test (2)).

A [followup post](https://www.alignmentforum.org/posts/f6oWbqxEwktfPrKJw/computing-natural-abstractions-linear-approximation) illustrates this in the case of a linear-Gaussian Bayesian network with randomly chosen graph structure. In this case, we take two regions of 110 nodes that are far apart from each other, and operationalize the relevant information between the two as the covariance matrix between the two regions. It turns out that this covariance matrix has about 3-10 "dimensions" (depending on exactly how you count), supporting claim (1). (And in fact, if you now compare to another neighborhood, two of the three "dimensions" remain the same!) Unfortunately, this doesn't give much evidence about (2) since humans don't have good concepts for parts of linear-Gaussian Bayesian networks with randomly chosen graph structure.

While (3) can also be tested empirically through simulation, we would hope that we can also prove theorems that state that nearly all cognitive architectures from some class of models would learn the same concepts in some appropriate types of environments.

To quote the author, "the holy grail of the project would be a system which provably learns all learnable abstractions in a fairly general class of environments, and represents those abstractions in a legible way. In other words: it would be a standardized tool for measuring abstractions. Stick it in some environment, and it finds the abstractions in that environment and presents a standard representation of them."


I'm really like having a candidate algorithm in mind when reasoning about alignment. It is a lot more concrete, which makes it easier to make progress and not get lost, relative to generic reasoning from just the assumption that the AI system is superintelligent.

I'm less clear on how exactly you move between the two steps -- from my perspective, there is a core reason for worry, which is something like "you can't fully control what patterns of thought your algorithms learn, and how they'll behave in new circumstances", and it feels like you could always apply that as your step 2. Our algorithms are instead meant to chip away at the problem, by continually increasing our control over these patterns of thought. It seems like the author has a better-defined sense of what does and doesn't count as a valid step 2, and that makes this methodology more fruitful for him than it would be for me. More discussion [here](https://www.alignmentforum.org/posts/EF5M6CmKRd6qZk27Z/my-research-methodology?commentId=8Hq4GJtnPzpoALNtk).

My research methodology

This post outlines a simple methodology for making progress on AI alignment. The core idea is to alternate between two steps:

1. Come up with some alignment algorithm that solves the issues identified so far
2. Try to find some plausible situation in which either a) the resulting AI system is misaligned or b) the AI system is not competitive.

This is all done conceptually, so step 2 can involve fairly exotic scenarios that probably won't happen. Given such a scenario, we need to argue why no failure in the same class as that scenario will happen, or we need to go back to step 1 and come up with a new algorithm.
This methodology could play out as follows:

Step 1: RL with a handcoded reward function.
Step 2: This is vulnerable to <@specification gaming@>(@Specification gaming examples in AI@).
Step 1: RL from human preferences over behavior, or other forms of human feedback.
Step 2: The system might still pursue actions that are bad that humans can't recognize as bad. For example, it might write a well researched report on whether fetuses are moral patients, which intuitively seems good (assuming the research is good). However, this would be quite bad if the AI wrote the report because it calculated that it would increase partisanship leading to civil war.
Step 1: Use iterated amplification to construct a feedback signal that is "smarter" than the AI system it is training.
Step 2: The system might pick up on <@inaccessible information@>(@Inaccessible information@) that the amplified overseer cannot find. For example, it might be able to learn a language just by staring at a large pile of data in that language, and then seek power whenever working in that language, and the amplified overseer may not be able to detect this.
Step 1: Use <@imitative generalization@>(@Imitative Generalisation (AKA 'Learning the Prior')@) so that the human overseer can leverage facts that can be learned by induction / pattern matching, which neural nets are great at.
Step 2: Since imitative generalization ends up learning a description of facts for some dataset, it may learn low-level facts useful for prediction on the dataset, while not including the high-level facts that tell us how the low-level facts connect to things we care about.

The post also talks about various possible objections you might have, which I'm not going to summarize here.


On the alignment problem

This 80,000 Hours podcast goes over many of the examples from Brian's book, <@The Alignment Problem@>. I recommend listening to it if you aren't going to read the book itself; the examples and stories are fascinating. (Though note I only skimmed through the podcast.)


Note that the explanation of corrigibility given here is in accord with the usage in [this MIRI paper](https://intelligence.org/2014/10/18/new-report-corrigibility/), but not to the usage in the <@iterated amplification sequence@>(@Corrigibility@), where it refers to a broader concept. The broader concept might roughly be defined as “an AI is corrigible when it leaves its user ‘in control’”; see the linked post for examples of what ‘in control’ involves. (Here also you can have both an impact- and intent-based version of the definition.)

On the model that AI risk is caused by utility maximizers pursuing the wrong reward function, I agree that non-obstruction is a useful goal to aim for, and the resulting approaches (mild optimization, low impact, corrigibility as defined here) make sense to pursue. I <@do not like this model much@>(@Conclusion to the sequence on value learning@), but that’s (probably?) a minority view.

Non-Obstruction: A Simple Concept Motivating Corrigibility

The <@Reframing Impact sequence@>(@Reframing Impact - Part 1@) suggests that it is useful to think about how well we could pursue a _range_ of possible goals; this is called the _attainable utility (AU) landscape_. We might think of a superintelligent AI maximizing utility function U as causing this landscape to become "spiky" -- the value for U will go up, but the value for all other goals will go down. If we get this sort of spikiness for an incorrect U, then the true objective will have a very low value.

Thus, a natural objective for AI alignment research is to reduce spikiness. Specifically, we can aim for _non-obstruction_: turning the AI on does not decrease the attainable utility for _any_ goal in our range of possible goals. Mild optimization (such as [quantilization](https://intelligence.org/files/QuantilizersSaferAlternative.pdf) ([AN #48](https://mailchi.mp/3091c6e9405c/alignment-newsletter-48))) reduces spikiness by reducing the amount of optimization that an AI performs. Impact regularization aims to find an objective that, when maximized, does not lead to too much spikiness.

One particular strategy for non-obstruction would be to build an AI system that does not manipulate us, and allows us to correct it (i.e. modify its policy). Then, no matter what our goal is, if the AI system starts to do things we don't like, we would be able to correct it. As a result, such an AI system would be highly non-obstructive. This property where we can correct the AI system is [corrigibility](https://intelligence.org/2014/10/18/new-report-corrigibility/). Thus, corrigibility can be thought of as a particular strategy for achieving non-obstruction.

It should be noted that all of the discussion so far is based on _actual outcomes in the world_, rather than what the agent was trying to do. That is, all of the concepts so far are based on _impact_ rather than _intent_.


If we cannot even measure AI progress and the impacts of AI systems right now, how are we supposed to accurately forecast them? As better forecasts are crucial for prioritizing the right problems and solutions, I am glad that the measurement of AI progress and impacts is getting broader attention. While the focus on developing countries might be less important from the perspective of AI existential safety, increased attention on measuring AI from a diverse set of communities is likely very useful for bringing the topic to the mainstream.

Measurement in AI Policy: Opportunities and Challenges

Saurabh Mishra, Jack Clark, C. Raymond Perrault

This paper is itself a summary of a 2019 Stanford workshop on the measurement of AI systems and contains summaries of all of the 33 talks given at the workshop. The workshop featured three in-depth breakout sessions, one on R&D and performance, one on economic impact and policy, and one on AI for sustainable development and human rights. Based on the discussions,  the authors identify six central problems in measuring AI progress and the impacts of AI systems:

First, the exact definition of AI is hard to get down given the ongoing evolution of the field. The lack of clear definitions makes it tricky to combine results on different aspects like investments into "AI" and the effects of "AI" on productivity both with each other and across different countries or sectors.

Second, measuring progress in AI is hard for a variety of reasons: We don't just care about narrow benchmark performance but about many factors like robustness, transferability and compute-efficiency, and it is not clear how the tradeoff between performance and these factors should look like. Apart from that, progress on popular benchmarks might be faster than overall progress as methods overfit to the benchmark, and the rise and fall of benchmark popularity make it hard to track progress over longer time intervals. Still, focusing on specific benchmarks and subdomains seems like an important first step.

Third, bibliometric data is an important tool for better understanding the role of different actors in a scientific field. More precise definitions of AI could help with getting better bibliometric data and such data could shine some light on aspects like the lifecycle of AI techniques and the demographics of AI researchers.

Fourth, we would like to measure the impact of AI on the economy, especially on inequality and the labour market. This requires a better understanding of the relationship between inputs like skilled workers and data, and outputs, which is difficult to obtain because many of the involved factors are quite intangible and effects on outputs can be quite delayed. Short-term indicators that are strong predictors of longer-term effects would be very useful in this context. Lastly, even figuring out which businesses are deploying AI can be hard, especially if the applications are inward-focused.

The fifth problem is concerned with the measurement of societal impacts of AI with a special focus on developing countries: While a large number of metrics for impacts of AI systems on human rights and the UN's sustainable development goals have been proposed, there is little data on the deployment of AI systems for social good and in developing countries, so far.

Sixth, there is a need for better assessment of risks posed by and other negative impacts of AI systems, both before and after deployment. To that extent, a better understanding of risks posed by general classes of applications like autonomous weapons, surveillance and fake videos would be helpful. One barrier here is that many of the riskier applications are in the domain of governmental action such that detailed information is often classified.


I broadly agree with the perspective in this post: in particular, I think we really should have more optimism because of the tendency of neural nets to learn “natural abstractions”. There is structure and regularity in the world and neural nets often capture it (despite being able to memorize random noise); if we train neural nets on a bunch of human-relevant data it really should learn a lot about humans, including what we care about.

However, I am less optimistic than the author about the specific path presented here (and he only assigns 10% chance to it). In particular, while I do think human values are a “real” thing that a neural net will pick up on, I don’t think that they are well-defined enough to align an AI system arbitrarily far into the future: our values do not say what to do in all possible situations; to see this we need only to look at the vast disagreements among moral philosophers (who often focus on esoteric situations). If an AI system were to internalize and optimize our current system of values, as the world changed the AI system would probably become less and less aligned with humans. We could instead talk about an AI system that has internalized both current human values and the process by which they are constructed, but that feels much less like a natural abstraction to me.

I _am_ optimistic about a very similar path, in which instead of training the system to pursue (a proxy for) human values, we train the system to pursue some “meta” specification like “be helpful to the user / humanity” or “do what we want on reflection”. It seems to me that “being helpful” is also a natural abstraction, and it seems more likely that an AI system pursuing this specification would continue to be beneficial as the world (and human values) changed drastically.

Alignment By Default

I liked the author's summary, so I've reproduced it with minor stylistic changes:

A low-level model of some humans has everything there is to know about human values embedded within it, in exactly the same way that human values are embedded in physical humans. The embedding, however, is nontrivial. Thus, predictive power alone is not sufficient to define human values. The missing part is the embedding of values within the model.

However, this also applies if we replace the phrase "human values" with "trees". Yet we have a whole class of neural networks in which a simple embedding lights up in response to trees. This is because trees are a natural abstraction, and we should expect to see real systems trained for predictive power use natural abstractions internally.

Human values are a little different from trees: they're a property of an abstract object (humans) rather than an abstract object themselves. Nonetheless, the author still expects that a broad class of systems trained for predictive power will end up with simple embeddings of human values (~70% chance).

Since an unsupervised learner has a simple embedding of human values, a supervised/reinforcement learner can easily score well on values-proxy-tasks by directly using that model of human values. In other words, the system uses an actual model of human values as a proxy for our proxy of human values (~10-20% chance). This is what is meant by _alignment by default_.

When this works, it's basically a best-case scenario, so we can safely use the system to design a successor without worrying about amplification of alignment errors (among other things).


I agree with what I see as the key point of this post: myopic training does not mean that the resulting agent will have myopic cognition. However, I don’t think this means myopic training is useless. According to me, the main benefit of myopic training is that small errors in reward specification for regular RL can incentivize catastrophic outcomes, while small errors in approval feedback for myopic RL are unlikely to incentivize catastrophic outcomes. (This is because “simple” rewards that we specify often lead to <@convergent instrumental subgoals@>(@The Basic AI Drives@), which need not be the case for approval feedback.) More details in [this comment](https://www.alignmentforum.org/posts/GqxuDtZvfgL2bEQ5v/arguments-against-myopic-training?commentId=TYS4QPTLnY38GeEhg).

Arguments against myopic training

[Several](https://www.alignmentforum.org/posts/7Hr8t6xwuuxBTqADK/approval-directed-agents-overview) ([AN #34](https://mailchi.mp/f1947668b183/alignment-newsletter-34)) <@proposals@>(@An overview of 11 proposals for building safe advanced AI@) in AI alignment involve some form of myopic training, in which an AI system is trained to take actions that only maximize the feedback signal in the **next timestep** (rather than e.g. across an episode, or across all time, as with typical reward signals). In order for this to work, the feedback signal needs to take into account the future consequences of the AI system's action, in order to incentivize good behavior, and so providing feedback becomes more challenging.

This post argues that there don't seem to be any major benefits of myopic training, and so it is not worth the cost we pay in having to provide more challenging feedback. In particular, myopic training does not necessarily lead to "myopic cognition", in which the agent doesn't think about long-term consequences when choosing an action. To see this, consider the case where we know the ideal reward function R*. In that case, the best feedback to give for myopic training is the optimal Q-function Q*. However, regardless of whether we do regular training with R* or myopic training with Q*, the agent would do well if it estimates Q* in order to select the right action to take, which in turn will likely require reasoning about long-term consequences of its actions. So there doesn't seem to be a strong reason to expect myopic training to lead to myopic cognition, if we give feedback that depends on (our predictions of) long-term consequences. In fact, for any approval feedback we may give, there is an equivalent reward feedback that would incentivize the same optimal policy.

Another argument for myopic training is that it prevents reward tampering and manipulation of the supervisor. The author doesn't find this compelling. In the case of reward tampering, it seems that agents would not catastrophically tamper with their reward "by accident", as tampering is difficult to do, and so they would only do so intentionally, in which case it is important for us to prevent those intentions from arising, for which we shouldn't expect myopic training to help very much. In the case of manipulating the supervisor, he argues that in the case of myopic training, the supervisor will have to think about the future outputs of the agent in order to be competitive, which could lead to manipulation anyway.


It’s interesting how much of the alignment problem manifests itself even when you assume that the human utility function is known, but the feature sets used by the human and agent are different. The only piece of the argument missing from this paper is that with sufficiently capable agents, the agent will actually be _adversarial_ towards the human because of [convergent instrumental subgoals](https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf), and that argument can be made in this framework.

Unfortunately, both of their methods for producing alignment don’t scale well, as they admit in the paper. (The second method in particular is kind of like hardcoding the policy, similarly to the construction <@here@>(@Coherence arguments do not imply goal-directed behavior@).)

User-Agent Value Alignment

This paper **from 2002** investigates what it would take to align an artificial agent with a human principal, under the assumption that the human utility function is known, but that the agent reward and human utility might be computed from different feature sets fA and fH. In this case, it is possible that the agent reward cannot capture all of the effects that the human cares about, leading to misalignment.

They introduce the concept of _graphical value alignment_, in which the only way that the agent's actions can affect fH is through fA. In this case, we can establish _functional value alignment_ (in which the agent's optimal policy also maximizes human utility), by setting the agent's reward for any specific fA to be the expectation (over fH) of the utility of fH, given fA. Note that the graphical criterion is _very strong_: it requires that _none_ of the agent's unobserved effects matter at all to the human.

They suggest two methods for establishing alignment. First, we can define additional agent features (perhaps requiring additional sensors), until all of the effects on fH are captured by fA. However, this would be very difficult, if not impossible. Second, we can include all agent actions and observations as agent features, since any effect of the agent's choice of policy on fH depends only on the observations made and actions taken. Of course, to achieve functional value alignment we would then have to have a good understanding of the expected human utility for every action given any observation, which is also hard.

They also briefly discuss the relationship between aligned agents and capable agents: a stone is aligned with you (per their definition), but also entirely useless. An interesting quote: _"Note that it might be harder to establish alignment with more competent agents because their skills afford many more pathways for adverse effects. This is a somewhat troubling thought."_


One particularly interesting finding is that so far, in all of the cases they've looked at, there was a lot of feedback available to develop a solution. The post notes that this could be interpreted in two ways. First, since feedback is abundant in real-world problems, we should expect feedback for AI alignment as well ([the optimistic take](https://mailchi.mp/b3dc916ac7e2/an-80-why-ai-risk-might-be-solved-without-additional-intervention-from-longtermists)). Second, since AI alignment has no opportunity for feedback, it is unlike other problems (<@the pessimistic take@>(@The Rocket Alignment Problem@)). I would add a third option: that any real-world problem without feedback is extremely hard to solve, and so we wouldn't generate any hypotheses of actions that were prescient for such problems (in which case AI risk is not special amongst problems, it is just an instance of a very difficult problem).

Survey of prescient actions

AI Impacts is looking into other examples in history where people took actions in order to address a complex, novel, severe future problem, and in hindsight we recognize those actions as prescient. Ideally we could learn lessons for AI alignment from such cases. The survey is so far very preliminary, so I'll summarize it later when it has been further developed, but I thought I'd send it along if you wanted to follow along (I found the six cases they've identified quite interesting).


Bayesian Evolving-to-Extinction

Consider a Bayesian learner, that updates the weights of various hypotheses using Bayes Rule. If the hypotheses can influence future events and predictions (for example, maybe it can write out logs, which influence what questions are asked in the future), then hypotheses that affect the future in a way that only they can predict will be selected for by Bayes Rule, rather than hypotheses that straightforwardly predict the future without trying to influence it. In some sense, this is "myopic" behavior on the part of Bayesian updating: Bayes Rule only optimizes per-hypothesis, without taking into account the effect on overall future accuracy. This phenomenon could also apply to neural nets if the <@lottery ticket hypothesis@>(@The Lottery Ticket Hypothesis: Training Pruned Neural Networks@) holds: in this case each "ticket" can be thought of as a competing hypothesis.


I liked this sequence. Often when presented with a potential problem in AI safety, I ask myself why the problem doesn't also apply to humans, and how humans have managed to solve the problem. This sequence was primarily this sort of reasoning, and I think it did a good job of highlighting how with sufficient conservatism it seems plausible that many problems are not that bad if the AI is well-intentioned, even if it has very little information, or finds it hard to communicate with humans, or has the wrong abstractions.

If I were a well-intentioned AI

This sequence takes on the perspective of an AI system that is well-intentioned, but lacking information about what humans want. The hope is to find what good AI reasoning might look like, and hopefully use this to derive insights for safety. The sequence considers Goodhart problems, adversarial examples, distribution shift, subagent problems, etc.