### **EleutherAI Alignment 101** **Curriculum**

Overview
--------

Meeting Times: {Fill-in}

The first iteration of the reading group has [[concluded]{.ul}](https://docs.google.com/document/d/1W-V7hCr9JEnoi_SG6occL8SztkcmXhW9-FbP7tOki6c/edit?usp=sharing). Future iterations will probably happen at some point.

Each week the group will meet for 1.5 hours to discuss the readings and exercises. Broadly speaking, the first half of the course explores the motivations and arguments underpinning the field of AGI safety, while the second half focuses on proposals for technical solutions.

The main focus each week will be on the core readings and one exercise **of your choice** out of the two exercises listed, for which you should allocate **around 2 hours preparation time.** Most people find some concepts from the readings confusing, but **that's totally fine** - resolving those uncertainties is what the discussion groups are for. Approximate times taken to read each piece in depth are listed next to them. **Note that in some cases only a small section of the linked reading is assigned**. In several cases, blog posts about machine learning papers are listed instead of the papers themselves; you're only expected to read the blog posts, but for those with strong ML backgrounds reading the paper versions might be worthwhile.

This curriculum is forked from Richard Ngo's AGI Safety Fundamentals curriculum, but has since accumulated a lot of changes.

> [EleutherAI Alignment 101 Curriculum](#eleutherai-alignment-101-curriculum) 1

**[Overview](#overview) 1**

**[Full curriculum](#full-curriculum) 2**

> [Week 1: Intro and Motivation](#week-1-intro-and-motivation) 2
>
> [Week 2: Goals and misalignment: Outer Alignment](#week-2-goals-and-misalignment-outer-alignment) 5
>
> [Week 3: Goals and misalignment: Inner Alignment](#week-3-goals-and-misalignment-inner-alignment) 7
>
> [Week ?: Timelines and Threat models](#week-timelines-and-threat-models) 10
>
> [Week 4: Learning from humans](#week-4-learning-from-humans) 12
>
> [Week 5: Factored Cognition for Outer Alignment](#week-5-factored-cognition-for-outer-alignment) 16
>
> [Week 6: Interpretability & ELK](#week-6-interpretability-elk) 19
>
> [Week 7: Agent Foundations and Embedded Agency](#week-7-agent-foundations-and-embedded-agency) 21
>
> [Week 8: Wrap Up & Bigger Picture](#week-8-wrap-up-bigger-picture) 23
>
> [Week TBD: AI governance, and careers in alignment research](#week-tbd-ai-governance-and-careers-in-alignment-research) 24
>
> [Week TBD (four weeks later): Projects](#week-tbd-four-weeks-later-projects) 27
>
> [Tentative Ideas](#tentative-ideas) 27
>
> [Projects overview](#projects-overview) 27
>
> [Timings](#timings) 27
>
> [Format](#format) 27
>
> [Ideas](#ideas) 27
>
> [Further resources](#further-resources) 29

Full curriculum
---------------

### Week 1: Intro and Motivation

The first week focuses on a brief introduction to the basic ideas of AGI and is intended to provide motivation for why we care about alignment and AGI. TODO: write more here

Slides: [[Week 1.5 EAI-ARG]{.ul}](https://docs.google.com/presentation/d/1cc0oUOItfOyQYcPBp7I9-zHFEUfrpcoZSCE3EVsiJXQ/edit?usp=sharing)

TODO: Notebooks for each week

Core readings:

1.  [[Deadly Truth of General AI (Rob Miles, 2015)]{.ul}](https://www.youtube.com/watch?v=tcdVC4e6EV4) (10 mins)

2.  [["There's no fire alarm for AGI" (Yudkowsky, 2017)]{.ul}](https://intelligence.org/2017/10/13/fire-alarm/) (35 mins)

    a.  Provides the case for why we should work on alignment even if it's very hard to forecast when AGI will happen, and why we shouldn't say "we'll start working on AGI once it's closer".

3.  [[The Bitter Lesson (Sutton, 2019)]{.ul}](http://www.incompleteideas.net/IncIdeas/BitterLesson.html) (15 mins)

    a.  The harsh reality of recent AI progress is that general methods which scale with increased compute outperform (expert) human-knowledge-based approaches.

4.  [[AGI safety from first principles (Ngo, 2020)]{.ul}](https://drive.google.com/file/d/1uK7NhdSKprQKZnRjU58X7NLA1auXlWHt/view) **(from section 1 to end of 2.1)** (20 mins)

    -   Narrow task-based AI and general AI form the tails of a spectrum of artificially intelligent systems. This short read distinguishes their features.

5.  [["We choose to align AI" (Wentworth, 2022)]{.ul}](https://www.lesswrong.com/posts/BseaxjsiDPKvGtDrm/we-choose-to-align-ai) (5 mins)

    -   A fun and short motivational post

Further readings for Week 1:

1.  [[AI: racing towards the brink (Harris and Yudkowsky, 2018)]{.ul}](https://intelligence.org/2018/02/28/sam-harris-and-eliezer-yudkowsky/) (110 mins) ([[audio here]{.ul}](https://www.youtube.com/watch?v=aS9cK0J5JXI))

    -   Transcript of a podcast conversation between Sam Harris and Eliezer Yudkowsky. Probably the best standalone resource for introducing AGI risk; covers many of the topics from this week and next week.

2.  [[General intelligence (Yudkowsky, 2017)]{.ul}](https://arbital.com/p/general_intelligence/) (25 mins)

    -   Yudkowsky provides a compelling explanation of the importance of the concept of general intelligence.

3.  More is different for AI (Steinhardt, 2022) ([[introduction]{.ul}](https://bounded-regret.ghost.io/more-is-different-for-ai/), [[second post]{.ul}](https://bounded-regret.ghost.io/future-ml-systems-will-be-qualitatively-different/), [[third post]{.ul}](https://bounded-regret.ghost.io/thought-experiments-provide-a-third-anchor/)) (20 mins)

    -   Steinhardt argues that, in machine learning, novel behaviours tend to emerge at larger scales, which are difficult to predict using standard approaches. Particularly recommended for those who are concerned that discussions of AGI are ungrounded or speculative.

4.  [[The power of intelligence (Yudkowsky, 2007)]{.ul}](https://intelligence.org/2007/07/10/the-power-of-intelligence/) (10 mins)

    -   One possible objection is that AGI is a confused concept, because "intelligence" is not a single unified property. Yudkowsky addresses this point, again drawing on the analogy from humans.

5.  [[Summary of Drexler's Reframing Superintelligence report (Shah, 2019)]{.ul}](https://www.alignmentforum.org/posts/x3fNwSe5aWZb5yXEG/reframing-superintelligence-comprehensive-ai-services-as) (10 mins)

    -   It's also worth considering ways the field of AI might advance without leading to AGI, e.g. via developing many narrow AIs which can each perform a small set of tasks very well. Drexler lays out one way this might happen in his Reframing Superintelligence report (summarised by Shah).

6.  [[Understanding human intelligence through human limitations (Griffiths, 2020)]{.ul}](https://arxiv.org/abs/2009.14050) (40 mins)

    -   Griffiths provides a framework for thinking about ways in which machine intelligence might differ from human intelligence.

7.  [[AI and compute: how much longer can computing power drive AI progress? (Lohn and Musser, 2022)]{.ul}](https://cset.georgetown.edu/wp-content/uploads/AI-and-Compute-How-Much-Longer-Can-Computing-Power-Drive-Artificial-Intelligence-Progress.pdf) (30 mins)

    -   This and the next two readings focus on forecasting progress in AI, via looking at trends in compute and algorithms, and surveying expert opinions.

8.  [[AI and efficiency (Hernandez and Brown, 2020)]{.ul}](https://openai.com/blog/ai-and-efficiency/) (15 mins)

    -   See above.

9.  [[When will AI exceed human performance? Evidence from AI experts (Grace et al., 2017)]{.ul}](https://arxiv.org/pdf/1705.08807.pdf) (15 mins)

    -   See above.

Exercises:

1.  A crucial feature of AGI is that it will possess cognitive skills which are useful across a range of tasks, rather than just the tasks it was trained to perform. Which cognitive skills did humans evolve because they were useful in our ancestral environments, which have remained useful in our modern environment? Which have become less useful?

2.  What are the most plausible ways for the hypothesis "we will eventually build AGIs which have transformative impacts on the world" to be false? How likely are they?

Notes:

1.  Instead of AGI, some people use the terms "human-level AI" or "strong AI". "Superintelligence" refers to AGI which is far beyond human-level intelligence. The opposite of general AI is called *narrow* AI. In his "Most important century" series, Karnofsky focuses on AI which will automate the process of scientific and technological advancement (which he gives the acronym PASTA) - this seems closely related to the concept of AGI, but without some additional connotations that the latter carries.

2.  Most of the content in this curriculum doesn't depend on strong claims about when AGI will arise, so try to avoid focusing disproportionately on the reading about timelines during discussions. However, I expect that it would be useful for participants to consider which evidence would change their current expectations in either direction. Note that the forecasts produced by the biological anchors method are fairly consistent with the survey of expert opinions carried out by Grace et al. (2017).

Discussion prompts:

1.  Ngo (2020) opens with a definition of intelligence as the ability to perform well at a wide range of cognitive tasks. What are some advantages and disadvantages of this definition?

2.  Here's an alternative way of describing general intelligence: whatever mental skills humans have that allow us to build technology and civilization (in contrast to other animals). What do you think about this characterisation?

3.  One intuition for how to think about very smart AIs: imagine speeding up human intellectual development by a factor of X. What do you expect a human civilization to know by 2100 or 2200? If an AI could do the same quality of research, but 10 or 100 times faster, how would you use it?

4.  How frequently do humans build technologies where some of the details of why they work aren't understood by anyone? Would it be very surprising if we built AGI without understanding very much about how its thinking process works?

5.  Thinking about AGI involves reasoning about entities smarter than us, and a future technology that doesn't exist yet. What problems does this introduce, and how should we respond to them?

###

### Week 2: Goals and misalignment: Outer Alignment

This week we'll focus on how and why AGIs might develop goals that are *misaligned* with those of humans, in particular when they've been trained using machine learning. We cover several core ideas from alignment: instrumental convergence, the orthogonality thesis, and Goodhart's Law.

Slides: [[Week 2 EAI-ARG]{.ul}](https://docs.google.com/presentation/d/1kRnZxYRlJcJIIESEw5wINxWuHf0mben-Yih1sDjKdiU/edit?usp=sharing)

Core readings:

1.  [[Why Would AI Want to do Bad Things? Instrumental Convergence]{.ul}](https://www.youtube.com/watch?v=ZeecOKBus3Q) (10 mins)

2.  [[Intelligence and Stupidity: The Orthogonality Thesis]{.ul}](https://www.youtube.com/watch?v=hEUO6pjwFOo) (15 mins)

-   **Text alternative for the two videos above, for those who don't like videos:**

    -   [[Superintelligence, Chapter 7: The superintelligent will (Bostrom, 2014)]{.ul}](https://drive.google.com/file/d/1FVl9W2gW5_8ODYNZJ4nuFg79Z-_xkHkJ/view?usp=sharing) (35 mins)

```{=html}
<!-- -->
```
-   Bostrom outlines 2 theses on the relationship between intelligence and motivation in an artificial agent; namely the orthogonality thesis and instrumental convergence.

3.  [[AI "Stop Button" Problem - Computerphile]{.ul}](https://www.youtube.com/watch?v=3TYT1QfdfsM) (20 mins)

-   **Text alternative for the video above, for those who don't like videos:**

    -   **todo:**

4.  [[Specification gaming: the flip side of AI ingenuity (Krakovna et al., 2020)]{.ul}](https://medium.com/@deepmindsafetyresearch/specification-gaming-the-flip-side-of-ai-ingenuity-c85bdb0deeb4) (15 mins)

    -   Specification Gaming is one example of Goodhart's Law in action in practice

Exercises:

1.  By some definitions, a chess AI has the goal of winning. When is it useful to describe it that way? What are the key differences between human goals and the "goals" of a chess AI?

Notes:

Discussion prompts:

1.  Christiano (2018) defined alignment as follows: "an AI A is aligned with an operator H if A is trying to do what H wants it to do". Some questions about this:

    a.  What's the most natural way to interpret "what the human wants" - what they say, or what they think, or what they would think if they thought about it for much longer?

    b.  How should we define an AI being aligned to a group of humans, rather than an individual?

2.  Does it make sense to talk about corporations and countries having goals? Does it matter that these consist of many different people, or can we treat them as agents with goals in a similar way to individual humans?

3.  Did Bostrom miss any important convergent instrumental goals? (His current list: self-preservation, goal-content integrity, cognitive enhancement, technological perfection, resource acquisition.) One way of thinking about this might be to consider which goals humans regularly pursue and why.

4.  By some definitions, a chess AI has the goal of winning. When is it useful to describe it that way? What are the key differences between human goals and the "goals" of a chess AI?

    a.  The same questions, but for corporations and countries instead of chess AIs. Does it matter that these consist of many different people, or can we treat them as agents with goals in a similar way to individual humans?

5.  Suppose that we want to build a highly intelligent AGI that is myopic, in the sense that it only cares about what happens over the next day or week. Would such an agent still have convergent instrumental goals? What factors might make it easier or harder to train a myopic AGI than a non-myopic AGI?

###

### Week 3: Goals and misalignment: Inner Alignment

When the models learned by our ML systems are themselves optimizers, it's possible for their objective to become misaligned with our objective.

Hubinger et al. (2019a) argue that even an agent trained on the "right" reward function might acquire undesirable goals - the problem of *inner alignment*. Carlsmith (2021) explores in more detail what it means for an agent to be goal-directed in a worrying way, and gives reasons why such agents seem likely to arise.

In the worst case, inner misalignment can lead to deceptive alignment, where our models pretend to be aligned so we approve of them, before performing a treacherous turn in deployment.

![](media/image2.png){width="4.076560586176728in" height="2.6131791338582677in"}

Slides: [[Week 2.5 EAI-ARG]{.ul}](https://docs.google.com/presentation/d/10zHGheCrnoCsMmy54JMaJUayJmu2QtoDx-HiqBkvM9Q/edit?usp=sharing)

Core readings:

1.  [[The OTHER AI Alignment Problem: Mesa-Optimizers and Inner Alignment]{.ul}](https://www.youtube.com/watch?v=bJLcIBixGj8&feature=youtu.be) (25 mins)

    -   This video gives a more accessible introduction to the inner alignment problem, as discussed in Hubinger et al. (2019a).

2.  [[Deceptive Misaligned Mesa-Optimisers? It's More Likely Than You Think\...]{.ul}](https://www.youtube.com/watch?v=IeWljQw3UgQ) (10 min)

3.  [[Introduction to Risks from Learned Optimisation (Hubinger et al., 2019a)]{.ul}](https://www.alignmentforum.org/posts/FkgsxrGf3QxhfLWHG/risks-from-learned-optimization-introduction) (20 mins)

4.  [[Risks from Learned Optimisation: The Inner Alignment Problem (Hubinger et al., 2019b)]{.ul}](https://www.alignmentforum.org/s/r9tYkB2a8Fp4DN8yB/p/pL56xPoniLvtMDQ4J) (25 mins)

    -   Follows on from Hubinger et al. (2019a) to explain the inner alignment problem in more depth.

5.  [[Risks from Learned Optimisation: Deceptive alignment (Hubinger et al., 2019)]{.ul}](https://www.alignmentforum.org/posts/zthDPAjh9w6Ytbeks/deceptive-alignment) (30 mins)

Further readings for both week 2 and 2.5:

6.  [[Why alignment could be hard with modern deep learning (Cotra, 2021)]{.ul}](https://www.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/) (25 mins)

    -   Cotra presents one broad framing for why achieving alignment might be hard, tying together the ideas from the core readings in a more accessible way.

7.  [[Optimal policies tend to seek power (Turner et al., 2021)]{.ul}](https://neurips.cc/virtual/2021/poster/28400) (15 mins)

    -   Turner et al. help to flesh out the arguments from Bostrom (2014) by formalising the notion of power-seeking in the reinforcement learning context, and proving that many agents end up power-seeking. (See also the corresponding [[blog post]{.ul}](https://www.alignmentforum.org/posts/6DuJxY8X45Sco4bS2/seeking-power-is-often-convergently-instrumental-in-mdps) and [[paper]{.ul}](https://arxiv.org/abs/1912.01683).)

8.  [[Distinguishing claims about training vs deployment (Ngo, 2021)]{.ul}](https://www.alignmentforum.org/posts/L9HcyaiWBLYe7vXid/distinguishing-claims-about-training-vs-deployment) (15 mins)

    -   Ngo updates many of the concepts in Bostrom (2014) to reflect the context of modern machine learning.

9.  [[Objective robustness in deep reinforcement learning (Koch et al., 2021)]{.ul}](https://arxiv.org/abs/2105.14111) (30 mins)

    -   Koch et al. provide some toy examples where agents learn to score highly on proxies for their training reward function, rather than generalising their intended objective to new environments.

10. [[AGI safety from first principles (Ngo, 2020)]{.ul}](https://drive.google.com/file/d/1uK7NhdSKprQKZnRjU58X7NLA1auXlWHt/view) **(only section 3: Goals and Agency)** (30 mins)

    -   This and the next reading explore the concept of goal-directedness. Their arguments are far from conclusive, but do suggest that building goal-directed agents will be the default outcome unless we specifically try to do otherwise.

11. [[Why tool AIs want to be agent AIs (Branwen, 2016)]{.ul}](https://www.gwern.net/Tool-AI) (45 mins)

    -   See above.

12. [[Ngo and Yudkowsky on alignment difficulty (Ngo and Yudkowsky, 2021)]{.ul}](https://www.alignmentforum.org/posts/7im8at9PmhbT4JHsW/ngo-and-yudkowsky-on-alignment-difficulty) (150 mins)

    -   Ngo and Yudkowsky have an in-depth debate on the underlying reasons we might expect alignment to be easy or hard. The debate is very long, and takes place at a high level of abstraction - it's mainly recommended for those who are already very comfortable with the other arguments discussed this week.

13. [[Is power-seeking AI an existential risk? (Carlsmith, 2021)]{.ul}](https://docs.google.com/document/d/1smaI1lagHHcrhoi6ohdq3TYIZv0eNWWZMPEy8C8byYg/edit#heading=h.lvsab2uecgk4) **(only sections 2: Timelines and 3: Incentives)** (25 mins)

14. [[Clarifying "AI alignment" (Christiano, 2018)]{.ul}](https://ai-alignment.com/clarifying-ai-alignment-cec47cd69dd6) (10 mins)

15. [[Inner Alignment: Explain Like I'm 12 Edition (Rafael Harth, 2020)]{.ul}](https://www.lesswrong.com/posts/AHhCrJ2KpTjsCSwbt/inner-alignment-explain-like-i-m-12-edition) (15 mins)

16. [[Deceptively Aligned Mesa-Optimizers: It's Not Funny If I Have To Explain It]{.ul}](https://astralcodexten.substack.com/p/deceptively-aligned-mesa-optimizers)

17. [[Goodhart Taxonomy - LessWrong]{.ul}](https://www.lesswrong.com/posts/EbFABnst8LsidYs5Y/goodhart-taxonomy) (week 2 further reading)

Exercises:

1.  Why is it not appropriate to describe the specification gaming agents from last week's reading in Krakovna et al. (2020) as displaying inner alignment failures?

Notes:

1.  The core idea of inner alignment is that, although the reward function is used to update the agent's behaviour based on how well it performs tasks during training, agents don't need to refer to the reward function while carrying out any given task (e.g. playing an individual game of Starcraft). So the motivations which drive the agent's behaviour during tasks need not be closely related to their reward function. The best thought experiments to help understand this are cases where the reward function is strongly correlated with some proxy objective - e.g. rewarding agents for surviving, which then leads them to acquire the goal of eating food (as humans did). More generally the example of humans is a very useful one when discussing inner alignment.

Discussion prompts:

1.  To what extent are humans inner-misaligned with respect to evolution? How can you tell, and what might similar indicators look like in AGIs?

###

### Week ?: Timelines and Threat models

How might misaligned AGIs cause existential catastrophes, and how might we stop them? Two threat models are outlined in Christiano (2019) - the first focusing on outer misalignment, the second on inner misalignment. Muehlhauser and Salamon (2012) outline a core intuition for why we might be unable to prevent these risks: that progress in AI will at some point speed up dramatically. A third key intuition - that misaligned agents will try to deceive humans - is explored by Hubinger et al. (2019).

How might we prevent these scenarios? Christiano (2020) gives a broad overview of the landscape of different contributions to making AIs aligned, with a particular focus on some of the techniques we'll be covering in later weeks.

Slides: [[Week 3 EAI-ARG]{.ul}](https://docs.google.com/presentation/d/1SC8WWOoeJIyROiWvP3zMasY3FHBvwS_cL_be9hAoxdU/edit?usp=sharing)

Core readings:

1.  [[What failure looks like (Christiano, 2019)]{.ul}](https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-failure-looks-like) (20 mins)

2.  [[Intelligence explosion: evidence and import (Muehlhauser and Salamon, 2012)]{.ul}](https://drive.google.com/file/d/1QxMuScnYvyq-XmxYeqBRHKz7cZoOosHr/view?usp=sharing) **(only pages 10-15)** (15 mins)

3.  [[AI alignment landscape (Christiano, 2020)]{.ul}](https://forum.effectivealtruism.org/posts/63stBTw3WAW6k45dY/paul-christiano-current-work-in-ai-alignment) (35 mins)

Further readings:

1.  [[Unsolved problems in ML safety (Hendrycks et al., 2021)]{.ul}](https://arxiv.org/abs/2109.13916) (50 mins)

    -   Hendryks et al. provide an overview of open problems in safety which focuses more on links to mainstream ML.

2.  [[Takeoff speeds (Christiano, 2018)]{.ul}](https://sideways-view.com/2018/02/24/takeoff-speeds/) (35 mins)

    -   In response to Yudkowsky's (2015) argument that there will be a sharp "intelligence explosion", Christiano argues that the rate of progress will instead increase continuously over time. However, there is less distance between these positions than there may seem: Christiano still expects self-improving AI to eventually cause incredibly rapid growth.

3.  [[Clarifying "What failure looks like" (part 1) (Clarke, 2020)]{.ul}](https://www.lesswrong.com/posts/v6Q7T335KCMxujhZu/clarifying-what-failure-looks-like-part-1) (30 mins)

    -   Part 1 of What Failure Looks Like depends heavily on coordination failures driven by capitalism, in a way that is quite different from earlier AI risk narratives from Bostrom and Yudkowsky. Clarke (2020) clarifies this scenario and its underlying assumptions.

4.  [[What multipolar failure looks like (Critch, 2021)]{.ul}](https://www.lesswrong.com/posts/LpM3EAakwYdS6aRKf/what-multipolar-failure-looks-like-and-robust-agent-agnostic) (45 mins)

    -   This and the next reading give further threat scenarios also motivated by the possibility of serious coordination failures.

5.  [[Another outer alignment failure story (Christiano, 2021)]{.ul}](https://www.lesswrong.com/posts/AyNHoTWWAJ5eb99ji/another-outer-alignment-failure-story) (20 mins)

    -   See above.

6.  [[Long-term growth as a sequence of exponential modes (Hanson, 2000)]{.ul}](https://mason.gmu.edu/~rhanson/longgrow.pdf) (40 mins)

    -   Hanson uses historical data to predict a shift to a new "growth mode" in which the economy doubles every few weeks, which he considers a more plausible outcome of AI progress than an intelligence explosion. Again, although there has been [[substantial debate between Hanson and Yudkowsky]{.ul}](https://intelligence.org/ai-foom-debate/) about the plausibility of an intelligence explosion, don't overestimate the extent of their disagreement: both of them expect technological progress to speed up to a much greater extent than most forecasters.

7.  [[Eight claims about multi-agent AGI safety (Ngo, 2019)]{.ul}](https://www.alignmentforum.org/posts/dSAJdi99XmqftqXXq/eight-claims-about-multi-agent-agi-safety) (10 mins)

    -   Ngo distinguishes eight different claims about how multi-agent dynamics might affect the safety of either training or deploying AGIs.

8.  [[Value is fragile (Yudkowsky, 2009)]{.ul}](https://www.lesswrong.com/posts/GNnHHmm8EzePmKzPk/value-is-fragile) (15 mins)

    -   Yudkowsky argues that even small inaccuracies in aligning AIs to human values may lead to catastrophic consequences.

Exercises:

1.  Christiano's "influence-seeking systems" threat model in *What Failure Looks Like* is in some ways analogous to profit-seeking companies. What are the most important mechanisms preventing companies from catastrophic misbehaviour? Which of those would and wouldn't apply to influence-seeking AIs?

2.  What are the individual tasks involved in machine learning research (or some other type of research important for technological progress)? Identify the parts of the process which have already been automated, the parts of the process which seem like they could plausibly soon be automated, and the parts of the process which seem hardest to automate.

Discussion prompts:

1.  What are the biggest vulnerabilities in human civilisation that might be exploited by misaligned AGIs? To what extent do they depend on the development of other technologies more powerful than those which exist today?

2.  Does the distinction between "paying the alignment tax" and "reducing the alignment tax" make sense to you? Give a concrete example of each case. Are there activities which fall into both of these categories, or are ambiguous between them?

3.  Most of the readings so far have been framed in the current paradigm of deep learning. Is this reasonable? To what extent are they undermined by the possibility of future paradigm shifts in AI?

###

### Week 4: Learning from humans

This week, we look at four techniques for training AIs on human data (all falling under "learn from teacher" in [[Christiano's AI alignment landscape]{.ul}](https://ai-alignment.com/ai-alignment-landscape-d3773c37ae38) from last week). From a safety perspective, each of them improves on standard reinforcement learning techniques in some ways, but also has weaknesses which prevent it from solving the whole alignment problem. Next week, we'll look at some ways to make these techniques more powerful and scalable; this week focuses on understanding each of them. Participants who are already familiar with these techniques should read some of the further readings instead.

The first technique, behavioural cloning, is essentially an extension of supervised learning to settings where an AI must take actions over time - as discussed by Levine (2021). The second, reward modelling, allows humans to give feedback on the behaviour of reinforcement learning agents, which is then used to determine the rewards they receive; this is used by Christiano et al. (2017) and Steinnon et al. (2020). The third, inverse reinforcement learning (IRL for short), attempts to identify what goals a human is pursuing based on their behaviour.

A notable variant of IRL is *cooperative* IRL (CIRL for short), introduced by Hadfield-Menell et al. (2016). CIRL focuses on cases where the human and AI interact in a shared environment, and therefore the best strategy for the human is often to help the AI learn what goal the human is pursuing.

Finally, Christiano (2015) argues that learning human values is likely a hard enough problem that the techniques discussed so far won't be sufficient to solve it - a possibility which motivates the techniques discussed in subsequent weeks. While these techniques are flawed, understanding them is nonetheless important as background knowledge.

Slides: [[Week 4 EAI-ARG]{.ul}](https://docs.google.com/presentation/d/1aC8uZxLE1kiUqu0dZYoUL4XTNnmu41aMkg8dWuACL4s/edit?usp=sharing)

Core readings:

1.  [[Imitation learning lecture: part 1 (Levine, 2021a)]{.ul}](https://youtu.be/kGc8jOy5_zY) (20 mins)

2.  [[Training AI Without Writing A Reward Function, with Reward Modelling]{.ul}](https://www.youtube.com/watch?v=PYylPRX6z4Q)

-   **Text alternatives for the video above, for people who don't like videos:**

    -   [[Deep RL from human preferences blog post (Christiano et al., 2017)]{.ul}](https://openai.com/blog/deep-reinforcement-learning-from-human-preferences/) (10 mins)

        -   [[Learning to summarise with human feedback blog post (Stiennon et al., 2020)]{.ul}](https://openai.com/blog/learning-to-summarize-with-human-feedback/) (20 mins)

3.  Inverse reinforcement learning (IRL)

    a.  For those who don't already understand IRL:

        -   [[Inverse reinforcement learning example (Udacity, 2016)]{.ul}](https://www.youtube.com/watch?v=h7uGyBcIeII) (5 mins)

        -   [[Learning from humans: what is inverse reinforcement learning? (Alexander, 2018)]{.ul}](https://thegradient.pub/learning-from-humans-what-is-inverse-reinforcement-learning/) (25 mins)

    b.  For those who already understand IRL:

        -   [[Cooperative inverse reinforcement learning (Hadfield-Menell et al., 2016)]{.ul}](https://arxiv.org/abs/1606.03137) (40 mins) and/or [[Cooperatively Learning Human Values -- The Berkeley Artificial Intelligence Research Blog]{.ul}](https://bair.berkeley.edu/blog/2017/08/17/cooperatively-learning-human-values/) (10 mins)

4.  [[The easy goal inference problem is still hard (Christiano, 2015)]{.ul}](https://www.alignmentforum.org/s/4dHMdK5TLN6xcqtyc/p/h9DesGT3WT9u2k7Hr) (10 mins)

5.  [[Humans aren't agents - what then for value learning?]{.ul}](https://www.lesswrong.com/posts/DsEuRrsenZ6piGpE6/humans-aren-t-agents-what-then-for-value-learning) (5 mins)

Further readings:

1.  [[Reward-rational (implicit) choice: a unifying formalism for reward learning (Jeon et al., 2020)]{.ul}](https://arxiv.org/abs/2002.04833) (60 mins)

    -   The task of aiming to identify human preferences from human data is known as *reward learning*. Both reward modelling and inverse reinforcement learning are examples of reward learning, using different types of data. In response to the proliferation of different types of reward learning, Jeon et al. (2020) proposes a unifying framework.

2.  [[Humans can be assigned any values whatsoever (Armstrong, 2018)]{.ul}](https://www.alignmentforum.org/posts/ANupXf8XfZo2EJxGv/humans-can-be-assigned-any-values-whatsoever) (15 mins)

    -   A key challenge for reward learning is to account for ways in which humans are less than perfectly rational. Armstrong argues that this will be difficult, because there are many possible combinations of preferences and biases that can lead to any given behaviour, and the simplest is not necessarily the most accurate.

3.  [[Learning the preferences of bounded agents (Evans et al., 2015)]{.ul}](https://stuhlmueller.org/papers/preferences-nipsworkshop2015.pdf) (25 mins)

    -   Evans et al. discuss a few biases that humans display, and ways to account for them when learning values.

4.  [[An EPIC way to evaluate reward functions (Gleave et al., 2021)]{.ul}](https://deepmindsafetyresearch.medium.com/an-epic-way-to-evaluate-reward-functions-c2c6d41b61cc) (15 mins) (see also [[a recorded presentation]{.ul}](https://slideslive.com/38953511/quantifying-differences-in-reward-functions))

    -   Gleave et al. provide a way to evaluate the quality of learned reward functions.

5.  [[Learning human objectives by evaluating hypothetical behaviours (Reddy et al., 2020)]{.ul}](https://deepmind.com/blog/article/learning-human-objectives-by-evaluating-hypothetical-behaviours) (10 mins)

    -   Reddy et al. present a technique that allows agents to learn about unsafe actions without actually taking them, by having humans evaluate hypothetical behaviors.

6.  [[Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations (Brown et al., 2019)]{.ul}](http://proceedings.mlr.press/v100/brown20a/brown20a.pdf) (40 mins)

    -   Brown et al. provide a clever technique to allow imitation learning agents to surpass the performance of human demonstrators.

7.  [[Summary of assistance games (Flint, 2020)]{.ul}](https://www.alignmentforum.org/posts/qPoaA5ZSedivA4xJa/our-take-on-chai-s-research-agenda-in-under-1500-words) (15 mins)

    -   CIRL is just one example of a broader framework called *assistance* primarily researched by Berkeley's Center for Human-Compatible AI (CHAI for short), which features ongoing interactions between humans and AIs - as explained by Flint.

8.  [[A general language assistant as a laboratory for alignment (Askell et al., 2021)]{.ul}](https://arxiv.org/abs/2112.00861) (sections 1 and 2) (40 mins)

    -   Askell et al. focus on another way of learning from humans: having humans design prompts which encourage aligned behaviour, and then fine-tuning on those prompts (via a method they call context distillation).

9.  [[The MineRL BASALT Competition on Learning from Human Feedback (Shah et al. 2021)]{.ul}](https://arxiv.org/abs/2107.01969) (only section 1) (25 mins)

    -   Shah et al. present a competition to train agents using human feedback to perform complex tasks in a Minecraft environment.

10. [[Stop Button Solution? - Computerphile]{.ul}](https://www.youtube.com/watch?v=9nktr1MgS-A)

11. [[Scalable agent alignment via reward modeling: a research direction]{.ul}](https://arxiv.org/pdf/1811.07871.pdf)

    -   Provides a good lay of the land for reward modeling approaches circa 2018 and discusses the limitations of alternative value alignment approaches (see Section 7).

Exercises:

1.  Imagine using reward modelling, as described in the second reading from this week, to train an AI to perform a complex task like building a castle in Minecraft. What sort of problems would you encounter?

2.  Stiennon et al. (2020) note that "optimizing our reward model eventually leads to sample quality degradation". Explain why the curves in the corresponding graph are shaped the way they are. How could we prevent performance from decreasing so much?

Notes:

1.  Learning how to perform well on a task based on examples of human performance on that task is known as *imitation learning*. Behavioural cloning is the simplest type of imitation learning. Using IRL to learn a human reward function, then training an agent on it, is a more complex type of imitation learning.

2.  The techniques discussed this week showcase a tradeoff between power and alignment: behavioural cloning provides the fewest incentives for misbehaviour, but is also hardest to use to go beyond human-level ability. Whereas reward modelling can reward agents for unexpected behaviour that leads to good outcomes (as long as humans can recognise them) - but this also means that those agents might find and be rewarded for manipulative or deceptive actions. Christiano et al. (2017) provide an example of an agent learning to deceive the human evaluator; and Stiennon et al. (2020) provide an example of an agent learning to "deceive" its reward model. Lastly, while IRL could in theory be used even for tasks that humans can't evaluate, it relies most heavily on assumptions about human rationality in order to align agents.

Discussion prompts:

1.  What are the key similarities and differences between behavioural cloning, reward modelling, and inverse reinforcement learning?

2.  What types of human preferences can these techniques most easily learn? What types would be hardest to learn?

3.  How might using reward modelling lead to misaligned AGIs? What are some of the fundamental limitations of reward modelling and inverse reinforcement learning that prevent them from scaling to AGI?

###

### Week 5: Factored Cognition for Outer Alignment

One category of research direction in technical AGI safety involves training AIs to do complex tasks by decomposing those tasks into simpler ones where humans can more easily understand and evaluate AI behavior. This week we'll cover three closely-related algorithms (all falling under "build a better teacher" in [[Christiano's AI alignment landscape]{.ul}](https://forum.effectivealtruism.org/posts/63stBTw3WAW6k45dY/paul-christiano-current-work-in-ai-alignment)). The extent to which these approaches scale to AGI is an open question in alignment.

Wu et al. (2021) applies reward modelling recursively in order to solve more difficult tasks. Recursive reward modelling can be considered a special case of a more general class of techniques called *iterated amplification* (also known as *iterated distillation and amplification/IDA*), which is described in Ought (2019). A more technical description of iterated amplification is given by Christiano et al. (2018), along with some small-scale experiments. One useful way of thinking about IDA-like techniques is that, in the limit, it aims to instantiate HCH, a theoretical structure described by Christiano (2016). Whether HCH is aligned is a topic of dispute between different alignment researchers.

The third technique we'll discuss this week is *Debate*, as proposed by Irving and Amodei (2018). Unlike the other two techniques, Debate focuses on evaluating claims made by language models, rather than supervising AI behaviour over time. You'll spend some time during this week's session trying out [[a toy implementation of Debate]{.ul}](https://debate-game.openai.com/) (as explained in the curriculum notes).

Slides: [[Week 5 EAI-ARG]{.ul}](https://docs.google.com/presentation/d/1rfYQUn2uEgc4cn6DZyEt69oHi9WK1KB8ZLqVwAXJ014/edit?usp=sharing)

Core readings:

1.  [[A guide to Iterated Amplification & Debate]{.ul}](https://www.lesswrong.com/posts/vhfATmAoJcN8RqGg6/a-guide-to-iterated-amplification-and-debate) (30 mins)

2.  Factored cognition (Ought, 2019) ([[introduction]{.ul}](https://ought.org/research/factored-cognition) and [[scalability section]{.ul}](https://ought.org/research/factored-cognition/scalability)) (20 mins)

3.  [[Supervising strong learners by amplifying weak experts (Christiano et al., 2018)]{.ul}](https://arxiv.org/abs/1810.08575) (35 mins)

4.  [[AI safety via debate blog post (Irving and Amodei, 2018)]{.ul}](https://openai.com/blog/debate/) (15 mins)

5.  [[Humans consulting HCH (Christiano, 2016)]{.ul}](https://ai-alignment.com/humans-consulting-hch-f893f6051455) (5 mins)

```{=html}
<!-- -->
```
6.  [[Scalable agent alignment via reward modeling \| by DeepMind Safety Research \| Medium]{.ul}](https://deepmindsafetyresearch.medium.com/scalable-agent-alignment-via-reward-modeling-bf4ab06dfd84)

Further readings:

7.  [[Iterated Distillation and Amplification (Cotra, 2018)]{.ul}](https://ai-alignment.com/iterated-distillation-and-amplification-157debfd1616) (20 mins)

    -   Another way of understanding Iterated Amplification is by analogy to AlphaGo: as Cotra discusses, AlphaGo's tree search is an amplification step which is then distilled into its policy network.

8.  [[An overview of 11 proposals for building safe advanced AI (Hubinger, 2020)]{.ul}](https://www.alignmentforum.org/posts/fRsjBseRuvRhMPPE5/an-overview-of-11-proposals-for-building-safe-advanced-ai) (introduction then proposals 3, 8, and 9. Proposals 2 and 7 may be useful as background for these.) (25 mins)

    -   Iterated amplification doesn't specify *how* a human amplified by a model M should supervise the training of the next version of M. In theory we could use any of the techniques discussed last week - for example behavioural cloning, or reward modelling. Hubinger discusses several different options.

9.  [[Progress on AI safety via debate (Barnes et al., 2020)]{.ul}](https://www.alignmentforum.org/posts/Br4xDbYu4Frwrb64a/writeup-progress-on-ai-safety-via-debate-1) (60 mins)

    -   Hubinger's (2020) discussion of Debate mentions the cross-examination variant of Debate, intended to make strategic ambiguity more difficult. Barnes et al. give more details on this, and some interesting human debate experiments.

10. [[Debate update: obfuscated arguments problem (Barnes and Christiano, 2020)]{.ul}](https://www.alignmentforum.org/posts/PJLABqQ962hZEqhdB/debate-update-obfuscated-arguments-problem) (15 mins)

    -   Barnes and Christiano discuss one of the key difficulties with applying Debate in practice.

11. [[Scalable agent alignment via reward modelling (Leike et al., 2018)]{.ul}](https://arxiv.org/abs/1811.07871) (80 mins)

    -   Leike et al. provide a research agenda for how reward modelling might scale up to solving the alignment problem, which describes recursive reward modelling as well as various challenges which will need to be solved.

12. [[Challenges to Christiano's capability amplification proposal - LessWrong]{.ul}](https://www.lesswrong.com/posts/S7csET9CgBtpi7sCh/challenges-to-christiano-s-capability-amplification-proposal) (35 min)

13. [[How to Keep Improving When You're Better Than Any Teacher - Iterated Distillation and Amplification]{.ul}](https://www.youtube.com/watch?v=v9M2Ho9I9Qo)

14. [[Recursively summarising books with human feedback (Wu et al., 2021)]{.ul}](https://arxiv.org/abs/2109.10862) **(ending after section 4.1.2: Findings)** (35 mins)

Exercises:

1.  Wu et al. (2021) use a combination of behavioural cloning and reinforcement learning to train a summarisation model; this combination was also used to train AlphaGo and AlphaStar. Explain why that's better than using either by itself.

2.  A complex task like running a factory can be broken down into subtasks in a fairly straightforward way, allowing a large team of workers to perform much better than even an exceptionally talented individual. Describe a task where teams have much less of an advantage over the best individuals. Why doesn't your task benefit as much from being broken down into subtasks? How might we change that?

Notes:

1.  During this week's discussion session, try playing [[OpenAI's implementation of the Debate game]{.ul}](https://debate-game.openai.com/). The instructions on the linked page are fairly straightforward, and each game should be fairly quick. Note in particular the example GIF on the webpage, and the instructions that "the debaters should take turns, restrict themselves to short statements, and not talk too fast (otherwise, the honest player wins too easily)."

2.  What makes AI Debate different from debates between humans? One crucial point is that in debates between humans, we prioritise the most important or impactful claims made - whereas *any* incorrect statement from an AI debater loses them the debate. This is a demanding standard (aimed at making debates between superhuman debaters easier to judge).

Discussion prompts:

1.  To what extent does the honest debater have an advantage in Debate? How might we modify the rules to give the honest debater a bigger advantage?

    a.  One approach which has been suggested is called "cross-examination", which involves saving copies of each of the AI debaters throughout the debate, so that it's possible to go back later on and ask them to clarify what they meant (of course this can't be done with humans, since we can't just copy ourselves).

2.  Recursive reward modelling is one type of iterated amplification. Another is "imitative amplification", where we use behavioural cloning rather than reward modelling at each step. How should we expect them to differ?

3.  What might Debate look like when applied to complex technical questions?

4.  Debate is limited to question-answering, and can't train agents to take actions. How important do you expect this limitation to be?

###

### Week 6: Interpretability & ELK

Our current methods of training capable neural networks give us very little insight into how or why they function. In this week we cover two broad research directions which aim to change this by developing a more scientific understanding of machine cognition, with the long-term goal of solving (or dissolving) the inner alignment problem.

The first is work on mechanistic interpretability, most notably pursued by Olah et al. (2020), which studies neural circuits to identify the functions they carry out. As background, first read Olah et al. (2017) for a discussion of how the techniques of feature visualisation work. Olah's overall perspective on how this research contributes to AGI safety is summarised by Hubinger et al. (2019).

Finally, ELK is at the forefront of Paul Christiano's research agenda and is an important unifying framework behind many of the concepts so far.

[[https://twitter.com/nabla_theta/status/1502106478890479619]{.ul}](https://twitter.com/nabla_theta/status/1502106478890479619)

Slides: [[Week 6 EAI-ARG]{.ul}](https://docs.google.com/presentation/d/1PgXyJ_xfOHUQwo2jcNfmM2LEoCthopRQ7mf_PkePOR0/edit?usp=sharing)

Core readings:

1.  [[Feature visualisation (Olah et al, 2017)]{.ul}](https://distill.pub/2017/feature-visualization/) (20 mins)

2.  [[Zoom In: an introduction to circuits (Olah et al., 2020)]{.ul}](https://distill.pub/2020/circuits/zoom-in/) (35 mins)

3.  [[Chris Olah's views on AGI safety (Hubinger, 2019)]{.ul}](https://www.alignmentforum.org/posts/X2i9dQQK3gETCyqh2/chris-olah-s-views-on-agi-safety) (20 mins)

4.  [[A Mathematical Framework for Transformer Circuits]{.ul}](https://transformer-circuits.pub/2021/framework/index.html) (65 mins)

5.  [[Eliciting latent knowledge (Christiano et al., 2021)]{.ul}](https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit#) **(up to the end of the Ontology Identification section on page 38, plus the [[Mechanistic Interpretability]{.ul}](https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit#heading=h.w3tt0jawudyv) appendix section)** (60 mins)

Further readings:

1.  [[Thread: Circuits (Cammarata et al., 2020)]{.ul}](https://distill.pub/2020/circuits/)

    -   A series of short articles building on Zoom In, exploring different circuits in the InceptionV1 vision network.

2.  [[A mathematical framework for transformer circuits (Elhage et al., 2021)]{.ul}](https://transformer-circuits.pub/2021/framework/index.html) (90 mins)

    -   Elhage et al. build on previous circuits work to analyse transformers, the neural network architecture used by most of today's cutting-edge models. For a deeper dive into the topic, see the [[associated videos]{.ul}](https://transformer-circuits.pub/2021/videos/index.html).

3.  [[Rewriting a deep generative model (Bau et al., 2020)]{.ul}](https://rewriting.csail.mit.edu/) (20 mins)

    -   Bau et al. find a way to change individual associations within a neural network, which allows them to replace specific components of an image. For work along similar lines in language models, [[see here]{.ul}](https://openreview.net/forum?id=mMECu_poAs).

4.  [[Value loading in the human brain: a worked example (Byrnes, 2021)]{.ul}](https://www.alignmentforum.org/posts/iMM6dvHzco6jBMFMX/value-loading-in-the-human-brain-a-worked-example) (25 mins)

    -   In addition to interpretability research on neural networks, another approach to developing more interpretable AI involves studying human and animal brains. Byrnes gives an example of applying ideas from neuroscience to better understand AI.

5.  [[Interpretability beyond feature attribution: quantitative testing with concept attribution vectors (Kim et al., 2018)]{.ul}](https://arxiv.org/abs/1711.11279) (35 mins)

    -   Kim et al. introduce a technique for interpreting a neural net's internal state in terms of human concepts.

6.  [[Clusterability in neural networks (Filan et al., 2021)]{.ul}](https://arxiv.org/abs/2103.03386) (25 mins)

    -   Filan et al. present a technique for identifying modular structure within neural networks, and demonstrate that this type of structure arises during training.

7.  [[A Longlist of Theories of Impact for Interpretability - LessWrong (Neel Nanda. 2022)]{.ul}](https://www.lesswrong.com/posts/uK6sQCNMw8WKzJeCQ/a-longlist-of-theories-of-impact-for-interpretability)

Exercises:

-   Interpretability work on artificial neural networks is closely related to interpretability work on biological neural networks (aka brains). Describe two ways in which the former is easier than the latter, and two ways in which it's harder.

-   Think about why a complexity penalty on the reporter in ELK to disincentivize human simulation would not necessarily work in the worst case. Check your answer with the [[complexity penalty section]{.ul}](https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit#heading=h.lltpmkloasiz) in the report (no peeking! Coming up with the answer on your own is important).

Notes:

1.  Hubinger (2019) discusses the possibility of giving feedback on the process by which an agent chooses actions - in other words, supervising not just its behaviour but also its thoughts. Some researchers expect that this type of feedback will be crucial in making those scalable oversight techniques work for very advanced agents - but we currently have little idea how to do so at scale.

Discussion prompts:

-   The second core reading discusses how Chris Olah wants machine learning to be "a field which focuses on deliberate design where understanding models is prioritized and the way that people make progress is through deeply understanding their systems". How plausible/feasible is this? What might it look like to deeply understand the development of neural networks, apart from the sort of interpretability work discussed this week?

-   Were you surprised by the results and claims in Zoom In? Do you believe the Circuits hypothesis? If true, what are its most important implications?

-   In what ways are ELK and interpretability connected? If ELK is difficult in practice, what does this mean for interpretability?

###

### Week 7: Agent Foundations and Embedded Agency

The agent foundations research agenda pursued by the Machine Intelligence Research Institute (MIRI) aims to create rigorous mathematical frameworks to describe how AIs should reason about their real-world environments. Wentworth (2022) provides some motivation for the general class of approaches, Soares (2015) gives a high-level explanation of their approach, and Demski and Garrabrant (2018) identify a range of open problems and links between them.

Slides: [[Week 7 EAI-ARG]{.ul}](https://docs.google.com/presentation/d/1Jt3Ngiil8lPRuLdyBSjMnqXg_pw2aifaNyKIC2IPcw8/edit?usp=sharing)

Core readings:

1.  [[Why Agent Foundations? An Overly Abstract Explanation (Wentworth, 2022)]{.ul}](https://www.lesswrong.com/posts/FWvzwCDRgcjb9sigb/why-agent-foundations-an-overly-abstract-explanation) (15 mins)

2.  [[Embedded agency (Demski and Garrabrant, 2018)]{.ul}](https://intelligence.org/2018/10/29/embedded-agents/) (60 mins; **read section 1 carefully and skim the rest**)

    -   The important takeaway from this is to get a feel for the kinds of issues that come up with the full alignment problem. These are issues that are typically neglected in more prosaic work, but having an understanding of the things that make the full alignment problem hard is useful background knowledge.

    -   I strongly recommend reading this in its entirety if you have the time (several hours), though be warned, there's a lot of meat in there.

Further Readings (warning: some of these posts are pretty intense):

8.  [[The rocket alignment problem (Yudkowsky, 2018)]{.ul}](https://intelligence.org/2018/10/03/rocket-alignment/) (35 mins)

    -   The underlying intuition driving MIRI's approach is that the alignment problem is very difficult, and therefore rigorous mathematical frameworks will be needed in order to solve it. Yudkowsky uses an extended analogy between building AGI and building rockets to convey this intuition.

9.  [[Logical induction blog post (Garrabrant et al., 2016)]{.ul}](https://intelligence.org/2016/09/12/new-paper-logical-induction/) (15 mins)

    -   This is the biggest result in agent foundations research thus far; it provides an algorithm for assigning probabilities to logical statements (like mathematical claims).

10. [[Parametric Bounded Löb's Theorem and Robust Cooperation of Bounded Agents (Critch, 2016)]{.ul}](https://arxiv.org/abs/1602.04184) (40 mins)

    -   Critch finds an algorithm by which agents which can observe each other's source code can reliably cooperate.

11. [[An introduction to the infra-Bayesianism sequence (Kosoy and Diffractor, 2020)]{.ul}](https://www.alignmentforum.org/posts/zB4f7QqKhBHa5b37a/introduction-to-the-infra-bayesianism-sequence) (30 mins)

    -   Kosoy and Diffractor present a formalism for bayesian inference with incomplete models. This post is extremely intense

12. [[Progress on causal influence diagrams (Everitt et al., 2021)]{.ul}](https://www.alignmentforum.org/posts/Cd7Hw492RqooYgQAS/progress-on-causal-influence-diagrams) (15 mins)

    -   Everitt et al. formally describe the incentives of reinforcement learning agents using causal influence diagrams.

13. [[Intuitive Introduction to Functional Decision Theory]{.ul}](https://www.lesswrong.com/s/q2WQMoQSexx7xqqZR) (30 mins)

14. [[MIRI's approach (Soares, 2015)]{.ul}](https://intelligence.org/2015/07/27/miris-approach/) (25 mins)

Exercises:

1.  Find another example of real world True Names in mathematics, science, etc other than the mutual information example mentioned in the Wentworth post. Compare it to potentially goodhartable versions of the same concept.

Discussion prompts:

1.  What are the best examples throughout history of scientists discovering mathematical formalisms that allowed them to deeply understand a phenomenon that they were previously confused about? (The easiest examples are from physics; how about others from outside physics?) To what extent should these make us optimistic about agent foundations research developing a mathematical understanding of intelligence?

###

### Week 8: Wrap Up & Bigger Picture

6.  [[Existential Risk: Analyzing Human Extinction Scenarios and Related Events (X-Risks) (Bostrom, 2002]{.ul}](https://www.nickbostrom.com/existential/risks.pdf)) (**up to and including section 3)** (20 min)

    a.  Understanding the magnitude of x-risks is important to understanding why alignment is so important.

7.  [[Risks of Astronomical Suffering (S-Risks)]{.ul}](https://www.lesswrong.com/tag/risks-of-astronomical-suffering-s-risks) (5 min)

    a.  S-risks are the worst case scenario of misalignment - even worse than x-risks.

8.  [["There's No Rule That Says We'll Make It" (Miles, 2022)]{.ul}](https://www.youtube.com/watch?v=JD_iA7imAPs) (10 mins)

    a.  Lots of people understand on an intellectual level that things aren't guaranteed to work out, but don't really *internalize* it. We could fail at alignment.

9.  [["Most important century" series summary (Karnofsky, 2021b)]{.ul}](https://www.cold-takes.com/most-important-century/#Summary) (15 mins)

10. [["Critch on career advice for junior AI-x-risk-concerned researchers"]{.ul}](https://www.lesswrong.com/posts/7uJnA3XDpTgemRH2c/critch-on-career-advice-for-junior-ai-x-risk-concerned) (5 mins)

    a.  A short caution against "just standing nearby" AI capabilities research as a common failure case for junior x-risk-concerned researchers.

```{=html}
<!-- -->
```
1.  [["It Looks Like You're Trying To Take Over The World" (gwern, 2022)]{.ul}](https://www.gwern.net/Clippy) (20 mins)

    -   An entertaining short story that outlines one potential way misalignment and takeoff could happen with more prosaic methods.

###

### ~~Week **TBD**: AI governance, and careers in alignment research~~

~~The last week of curriculum content is split between looking into the field of AI governance, and thinking about next steps for pursuing careers in alignment research. For the latter, see Ngo (forthcoming). For the former, start with Dafoe (2020), which gives a thorough overview of AI governance and ways in which it might be important, particularly focusing on the framing of AI governance as field-building. An alternative framing - of AI governance as an attempt to prevent cooperation failures - is explored by Clifton (2019). Finally, Khan (2021), Macro Polo (2020), and Shevlane (2022) give brief introductions to three key factors affecting the AI strategic landscape.~~

~~In the taxonomy of AI governance given by Clarke (2022) in the optional readings (diagram below) this week's governance readings focus on strategy research, tactics research and field-building, not on developing, advocating or implementing specific policies. Those interested in exploring AI governance in more detail, including looking into individual policies, should look at [[the curriculum for the parallel AI governance track of this course]{.ul}](https://docs.google.com/document/d/1F4lq6yB9SCINuo190MeTSHXGfF5PnPk693JToszRttY/edit?usp=sharing).~~

![](media/image1.png){width="6.5in" height="1.1805555555555556in"}

~~Core readings:~~

1.  ~~Placeholder: reading on careers in alignment (to be added later) (Ngo, forthcoming)~~

2.  ~~[[AI Governance: Opportunity and Theory of Impact (Dafoe, 2020)]{.ul}](https://forum.effectivealtruism.org/posts/42reWndoTEhFqu6T8/ai-governance-opportunity-and-theory-of-impact) (25 mins)~~

3.  ~~[[Cooperation, conflict and transformative AI: sections 1 & 2 (Clifton, 2019)]{.ul}](https://www.alignmentforum.org/s/p947tK8CoBbdpPtyK/p/KMocAf9jnAKc2jXri) (25 mins)~~

4.  ~~[[The semiconductor supply chain (Khan, 2021)]{.ul}](https://cset.georgetown.edu/publication/the-semiconductor-supply-chain/) **(up to page 15)** (15 mins)~~

5.  ~~[[The global AI talent tracker (Macro Polo, 2020)]{.ul}](https://macropolo.org/digital-projects/the-global-ai-talent-tracker/) (5 mins)~~

6.  ~~[[Sharing powerful AI models (Shevlane, 2022)]{.ul}](https://www.governance.ai/post/sharing-powerful-ai-models) (10 mins)~~

~~Further readings:~~

*~~On strategic AI governance considerations:~~*

1.  ~~[[Deciphering China's AI dream (Ding, 2018)]{.ul}](https://www.fhi.ox.ac.uk/wp-content/uploads/Deciphering_Chinas_AI-Dream.pdf) (95 mins) ([[see also his podcast on this topic]{.ul}](https://80000hours.org/podcast/episodes/jeffrey-ding-china-ai-dream/))~~

    -   ~~Ding gives an overview of Chinese AI policy, one of the key factors affecting the landscape of possible approaches to AI governance.~~

2.  ~~[[Leo Szilard and the danger of nuclear weapons: a case study in risk mitigation (Grace, 2015)]{.ul}](https://intelligence.org/files/SzilardNuclearWeapons.pdf) (60 mins)~~

    -   ~~Grace (2015) discusses the analogies between attempts to prevent misuse of nuclear weapons, and attempts to ensure good outcomes from AGI.~~

3.  ~~[[AI, the space race, and prestige (Barnhart, 2021)]{.ul}](https://docs.google.com/document/d/1T4FIbV32pHow72pd_aOrGPJuV8YZ8PwP5Mhwjt6ppy4/edit#) **(all except section II: Exacerbating Conditions)** (40 mins)~~

    -   ~~Barnhart (2021) provides a case study of the space race, identifying aspects which are analogous to our current situation.~~

4.  ~~[[The vulnerable world hypothesis (Bostrom, 2019)]{.ul}](https://www.nickbostrom.com/papers/vulnerable.pdf) (ending at the start of the section on 'Preventive policing') (60 mins)~~

    -   ~~Bostrom provides a background framing for thinking about technological risks: the process of randomly sampling new technologies, some of which might prove catastrophic.~~

5.  ~~[[Sharing the world with digital minds (Shulman and Bostrom, 2020)]{.ul}](https://nickbostrom.com/papers/digital-minds.pdf) (50 mins)~~

    -   ~~Shulman and Bostrom explore the moral status of digital minds, and the possibility that their experiences may be of great moral importance.~~

6.  ~~[[AI Governance: a research agenda (Dafoe, 2018)]{.ul}](https://www.fhi.ox.ac.uk/wp-content/uploads/GovAI-Agenda.pdf) (120 mins)~~

    -   ~~Dafoe outlines an overarching research agenda linking many areas of AI governance.~~

*~~On useful approaches to AI governance:~~*

7.  ~~[[Cooperative AI: machines must learn to find common ground (Dafoe et al., 2021)]{.ul}](https://www.nature.com/articles/d41586-021-01170-0) (15 mins)~~

    -   ~~This and the next reading describe two research directions which mix technical work and governance work: cooperative AI, and truthful AI.~~

8.  ~~[[Truthful AI: Developing and governing AI that does not lie (Evans et al., 2021)]{.ul}](https://www.alignmentforum.org/posts/aBixCPqSnTsPsTJBQ/truthful-ai-developing-and-governing-ai-that-does-not-lie) (20 mins)~~

    -   ~~See above.~~

9.  ~~[[Some AI governance research ideas (Anderljung and Carlier, 2021)]{.ul}](https://docs.google.com/document/d/13LJhP3ksrcEBKxYFG5GkJaC2UoxHKUYAHCRdRlpePEc/edit) (60 mins)~~

    -   ~~This and the next two readings provide lists of research directions which have either been promising so far, or which may be useful to look into in the future.~~

10. ~~[[Our AI governance grantmaking so far (Muehlhauser, 2020)]{.ul}](https://www.openphilanthropy.org/blog/ai-governance-grantmaking) (15 mins)~~

    -   ~~See above.~~

11. ~~[[The longtermist AI governance landscape: a basic overview (Clarke, 2022)]{.ul}](https://forum.effectivealtruism.org/posts/ydpo7LcJWhrr2GJrx/the-longtermist-ai-governance-landscape-a-basic-overview) (15 mins)~~

    -   ~~See above.~~

~~Exercises:~~

1.  ~~Explain the importance of the ability to make credible commitments for Clifton's (2019) game-theoretic analysis of cooperation failures.~~

2.  ~~In what ways has humanity's response to other threats apart from AI (e.g. nuclear weapons, pandemics) been better than we would have expected beforehand? In what ways has it been worse? What can we learn from this?~~

~~Notes:~~

1.  ~~"Accident" risks, as discussed in Dafoe (2020), include the standard risks due to misalignment which we've been discussing for most of the course. I don't usually use the term, because "deliberate" misbehaviour from AIs is quite different from standard accidents.~~

~~Discussion prompts:~~

1.  ~~How worried are you about misuse vs structural vs accident risk?~~

2.  ~~Do you expect AGI to be developed by a government or corporation (or something else)? What are the key ways that this difference would affect AI governance?~~

3.  ~~What are the main ways in which technical work could make AI governance easier or harder?~~

4.  ~~What are the biggest ways you expect AI to impact the world in the next 10 years? How will these affect policy responses aimed at the decade after that?~~

5.  ~~How likely do you think it is that we build and deploy many AIs that have net negative conscious experiences (as discussed in Shulman and Bostrom (2020))?~~

6.  ~~It seems important for regulators and policy-makers to have a good technical understanding of AI and its implications. In what cases should people with technical AI backgrounds consider entering these fields?~~

###

### Week **TBD** (four weeks later): Projects

#### Tentative Ideas

From Leo on EAI discord: "at the end of the 8 weeks we let everyone pick a direction they're interested in learning more about, give them one or two months to just read up on as much as possible about what they're interested in. this could also double as flex time for people who fall behind on the reading or join late to catch up. probably multiple people per direction for redundancy because even at that point there's going to be some flaking, and so all the people working on the same thing will work together. of course throughout this thing we can have mentorship and stuff to make sure nobody gets stuck. maybe as an intermediate product we have blog posts that we then synthesize into a paper"

#### Projects overview

The final part of the AGI safety fundamentals course will be projects where you get to dig into something related to the course. The project is a chance for you to explore your interests, so try to find something you're excited about! The goal of this project is to help you practice taking an intellectually productive stance towards AGI safety - to go beyond just reading and discussing existing ideas, and take a tangible step towards contributing to the field yourself. This is particularly valuable because it's such a new field, with lots of room to explore.

#### Timings

We've allocated four weeks between the last week of the curriculum content and the sessions where people present their projects. As a rough guide, spending 10-15 hours on the project during that time seems reasonable, but we're happy for participants to be flexible about spending more or less time on it. You may find it useful to write up a rough project proposal in the first week of working on your project and to send it to your cohort for feedback.

#### Format

The format of the project is very flexible. The default project will probably be a piece of writing, roughly the length and scope of a typical blog post; we'd encourage participants to put these online after finishing them (although this is entirely optional). We'd also be happy for people to spend time gaining familiarity with machine learning (e.g. via the practical projects discussed below). Projects in the form of presentations are also possible, but we slightly discourage them; we'd prefer if you spend more time creating some piece of writing or code, then just casually talk through it with your cohort, rather than spending time on trying to make a polished presentation. We expect most projects to be individual ones; but feel free to do a collaborative project if you'd like to.

#### Ideas

Some project ideas (all just suggestions; feel free to design your own):

-   Pick a reading you found difficult but insightful. Distill down the key ideas and write up an accessible summary of the arguments, their weaknesses and your overall view. (Here are some examples of this being done well for [[Iterated Amplification]{.ul}](https://www.lesswrong.com/posts/PT8vSxsusqWuN7JXp/my-understanding-of-paul-christiano-s-iterated-amplification) and [[Inner Alignment]{.ul}](https://www.lesswrong.com/s/BRrcuwiF2SYWTQfqA/p/AHhCrJ2KpTjsCSwbt), although we don't expect projects to be as comprehensive as these.)

-   Pick a reading from the curriculum that you disagreed with, and critique it.

-   Pick one of the weeks of the curriculum; summarise and evaluate the key claims made in the core readings for that week and your key takeaways.

-   Make a set of forecasts about the future of AI, in a way that's concrete enough that you will be able to judge whether you were right or not, and predict what would significantly change your mind.

-   Practical projects:

    -   (For those without much ML experience): Train a neural network on some standard datasets. For help, see the [[fast.ai course]{.ul}](https://course.fast.ai/) or the [[PyTorch tutorials]{.ul}](https://pytorch.org/tutorials/).

    -   (For those with ML experience but without much RL experience): Train a deep reinforcement learning agent on a standard environment. For help, see [[Spinning Up in Deep RL]{.ul}](https://spinningup.openai.com/en/latest/).

    -   Download a large language model (e.g. [[GPT-2]{.ul}](https://lambdalabs.com/blog/run-openais-new-gpt-2-text-generator-code-with-your-gpu/) or [[GPT-J-6B]{.ul}](https://github.com/kingoflolz/mesh-transformer-jax/)) or image-generating model (e.g. [[VQGAN+CLIP]{.ul}](https://docs.google.com/document/d/1Lu7XPRKlNhBQjcKr8k8qRzUzbBW7kzxb5Vu72GMRn2E/edit)), and try to find alignment failures - cases where the model is *capable* of doing what you intend, but doesn't. As one example (discussed in section 7.2 of [[this paper]{.ul}](https://arxiv.org/abs/2107.03374)), when the user gives it a prompt containing a subtle bug, the Codex language model may "deliberately" introduce further bugs into the code it writes, in order to match the style of the user prompt.

    -   (For those with extensive ML and RL experience, looking for a longer project): Replicate the [[TREX paper]{.ul}](https://arxiv.org/abs/1904.06387) (easier) or the [[Deep Reinforcement Learning from Human Preferences paper]{.ul}](https://arxiv.org/pdf/1706.03741.pdf) (harder) in a simpler environment (e.g. [[cartpole]{.ul}](http://gym.openai.com/envs/CartPole-v1/)). See if you can train the agent to do something in that environment which you can't write an explicit reward function for.

-   If you're considering a career in machine learning, put together a career plan for what that might look like, with a particular focus on the most important skills for you to acquire.

-   Pick a key underlying belief which would impact your AGI safety research interests, or whether to research AGI safety at all. Review the literature around this question, and write up a post giving *your* overall views on it, and the strongest arguments for and against.

    -   E.g. 'AGI is likely within the next 50 years' or 'Iterated Amplification is likely to produce competitive and aligned AGI'.

###

### Further resources

ML courses (free online):

-   [[Fast.ai]{.ul}](https://www.fast.ai/) courses

-   [[Stanford computer vision course (2017)]{.ul}](http://cs231n.stanford.edu/2017/syllabus.html)

-   [[NYU deep learning course]{.ul}](https://atcold.github.io/pytorch-Deep-Learning/)

-   [[Spinning up in deep RL]{.ul}](https://spinningup.openai.com/en/latest/index.html)

ML textbooks (all free online except the first):

-   [[Grokking deep learning (Trask)]{.ul}](https://www.manning.com/books/grokking-deep-learning)

-   [[Neural networks and deep learning (Nielsen)]{.ul}](http://neuralnetworksanddeeplearning.com/)

-   [[Deep Learning (Goodfellow, Bengio and Courville)]{.ul}](https://www.deeplearningbook.org/)

-   [[Reinforcement learning: an introduction (Sutton and Barto, 2nd edition)]{.ul}](http://incompleteideas.net/book/the-book-2nd.html)

-   [[Mathematics for machine learning (Deisenroth, Faisal and Ongg)]{.ul}](https://mml-book.github.io/)

-   [[Notes on contemporary machine learning for physicists (Kaplan)]{.ul}](https://sites.krieger.jhu.edu/jared-kaplan/files/2019/04/ContemporaryMLforPhysicists.pdf)

AI safety resources:

-   [[Annotated Bibliography of Recommended Materials]{.ul}](https://humancompatible.ai/bibliography) - UC Berkeley Center for Human-Compatible AI (CHAI)

```{=html}
<!-- -->
```
-   [[Alignment forum curated sequences]{.ul}](https://www.alignmentforum.org/library)

-   [[Alignment newsletter]{.ul}](https://rohinshah.com/alignment-newsletter/) - Rohin Shah

-   [[AI safety videos]{.ul}](https://www.youtube.com/channel/UCLB7AzTwc6VFZrBsO2ucBMg) - Rob Miles

-   [[Lots of Links]{.ul}](https://www.aisafetysupport.org/resources/lots-of-links) - AI Safety Support

-   [[Longer list of further resources]{.ul}](https://www.eacambridge.org/agi-further-resources) (including financial support, career resources, etc)

Other:

-   [[GB Hamming, "You and Your Research" (June 6, 1995)]{.ul}](https://www.youtube.com/watch?v=a1zDuOPkMSw)

-   [["How To Get Into Independent Research On Alignment/Agency" (Wentworth, 2021)]{.ul}](https://www.lesswrong.com/posts/P3Yt66Wh5g7SbkKuT/how-to-get-into-independent-research-on-alignment-agency) (20 mins)

/Users/janhendrikkirchner/code/2022/10/alignment-research-dataset/align_data/common/../../data/raw/md_ebooks/EAI 2022 AGI Safety Fundamentals alignment curriculum-by EleutherAI, Richard Ngo-date 2022-05-16.md