Question: What are plausible candidates for "pivotal acts"?

Answer: Pivotal acts are acts that substantially change the direction humanity will have taken in 1 billion years. The term is used to denote positive changes, as opposed to existential catastrophe.

An obvious pivotal act would be to create a [https://arbital.com/p/Sovereign/ sovereign AGI] aligned with humanity's best interests. An act that would greatly increase the chance of another pivotal act would also count as pivotal.

Pivotal acts often lay outside the [https://en.wikipedia.org/wiki/Overton_window Overton window]. One such example is stopping or strongly delaying the development of an unaligned (or any) AGI through drastic means such as nanobots which melt all advanced processors, or the disabling of all AI researchers. Eliezer mentions these in [https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities AGI Ruin: A List of Lethalities]. Andrew Critch argues against such an unilateral pivotal act in [https://www.alignmentforum.org/posts/Jo89KvfAs9z7owoZp/pivotal-act-intentions-negative-consequences-and-fallacious "Pivotal Act" Intentions: Negative Consequences and Fallacious Arguments].

For more details, see [https://arbital.com/p/pivotal/ arbital].

Question: What are plausible candidates for "pivotal acts"?

Answer:
Pivotal acts are acts that substantially change the direction humanity will have taken in 1 billion years. The term is used to denote positive changes, as opposed to existential catastrophe.

An obvious pivotal act would be to create a ［https://arbital.com/p/Sovereign/ sovereign AGI］ aligned with humanity's best interests. An act that would greatly increase the chance of another pivotal act would also count as pivotal.

Pivotal acts often lay outside the ［https://en.wikipedia.org/wiki/Overton_window Overton window］. One such example is stopping or strongly delaying the development of an unaligned (or any) AGI through drastic means such as nanobots which melt all advanced processors, or the disabling of all AI researchers. Eliezer mentions these in ［https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities AGI Ruin: A List of Lethalities］. Andrew Critch argues against such an unilateral pivotal act in ［https://www.alignmentforum.org/posts/Jo89KvfAs9z7owoZp/pivotal-act-intentions-negative-consequences-and-fallacious “Pivotal Act” Intentions: Negative Consequences and Fallacious Arguments］.

For more details, see ［https://arbital.com/p/pivotal/ arbital］.

Murphant's Answer to What are plausible candidates for "pivotal acts"?


Question: What is the difference between inner and outer alignment?

Answer: The paper [https://arxiv.org/abs/1906.01820 Risks from Learned Optimization in Advanced Machine Learning Systems] makes the distinction between inner and outer alignment: Outer alignment means making the optimization target of the ''training process'' ("outer optimization target" e.g. the ''loss'' in supervised learning) aligned with what we want. Inner alignment means making the optimization target of the ''trained system'' ("inner optimization target") aligned with the outer optimization target. A challenge here is that the inner optimization target does not have an explicit representation in current systems, and can differ very much from the outer optimization target (see for example [https://arxiv.org/abs/2105.14111 Goal Misgeneralization in Deep Reinforcement Learning]).

(youtube)bJLcIBixGj8(/youtube)

See also [https://astralcodexten.substack.com/p/deceptively-aligned-mesa-optimizers this article] for an intuitive explanation of inner and outer alignment.

Question: What is the difference between inner and outer alignment?

Answer:
The paper ［https://arxiv.org/abs/1906.01820 Risks from Learned Optimization in Advanced Machine Learning Systems］ makes the distinction between inner and outer alignment: Outer alignment means making the optimization target of the ''training process'' (“outer optimization target” e.g. the ''loss'' in supervised learning) aligned with what we want. Inner alignment means making the optimization target of the ''trained system'' (“inner optimization target”) aligned with the outer optimization target. A challenge here is that the inner optimization target does not have an explicit representation in current systems, and can differ very much from the outer optimization target (see for example ［https://arxiv.org/abs/2105.14111 Goal Misgeneralization in Deep Reinforcement Learning］).

(youtube)bJLcIBixGj8(/youtube)

See also ［https://astralcodexten.substack.com/p/deceptively-aligned-mesa-optimizers this article］ for an intuitive explanation of inner and outer alignment.

Magdalena's Answer to What is the difference between inner and outer alignment?


Question: Does the importance of AI risk depend on caring about transhumanist utopias?

Answer: No. Misaligned artificial intelligence poses a serious threat to the continued flourishing, and maybe even continued existence, of humanity as a whole. While predictions about when artificial general intelligence may be achieved vary, surveys consistently report a [https://research.aimultiple.com/artificial-general-intelligence-singularity-timing/ >50% probability of achieving general AI before the year 2060] - within the expected lifetimes of most people alive today.

It is difficult to predict how technology will develop, and at what speed, in the years ahead; but as artificial intelligence poses a not-insignificant chance of causing worldwide disaster within the not-too-distant future, anyone who is generally concerned with the future of humanity has reason to be interested.

Question: Does the importance of AI risk depend on caring about transhumanist utopias?

Answer:
No. Misaligned artificial intelligence poses a serious threat to the continued flourishing, and maybe even continued existence, of humanity as a whole. While predictions about when artificial general intelligence may be achieved vary, surveys consistently report a ［https://research.aimultiple.com/artificial-general-intelligence-singularity-timing/ >50% probability of achieving general AI before the year 2060］ - within the expected lifetimes of most people alive today.

It is difficult to predict how technology will develop, and at what speed, in the years ahead; but as artificial intelligence poses a not-insignificant chance of causing worldwide disaster within the not-too-distant future, anyone who is generally concerned with the future of humanity has reason to be interested.

Beamnode's Answer to Does the importance of AI risk depend on caring about transhumanist utopias?


What is John Wentworth's plan?

Question: What is John Wentworth's plan?

Answer: [https://www.lesswrong.com/posts/3L46WGauGpr7nYubu/the-plan John's plan] is:

Step 1: sort out our fundamental confusions about agency

Step 2: ambitious value learning (i.e. build an AI which correctly learns human values and optimizes for them)

Step 3: …

Step 4: profit!

… and do all that before AGI kills us all.

He is working on step 1: figuring out what the heck is going on with agency. His current approach is based on [https://www.lesswrong.com/posts/G2Lne2Fi7Qra5Lbuf/selection-theorems-a-program-for-understanding-agents selection theorems]: try to figure out what types of agents are selected for in a broad range of environments. Examples of selection pressures include: evolution, SGD, and markets. This is an approach to agent foundations that comes from the opposite direction as MIRI: it's more about observing existing structures (whether they be mathematical or real things in the world like markets or e coli), whereas MIRI is trying to write out some desiderata and then finding mathematical notions that satisfy those desiderata.

Two key properties that might be selected for are modularity and abstractions.

Abstractions are higher level things that people tend to use to describe things. Like "Tree" and "Chair" and "Person". These are all vague categories that contain lots of different things, but are really useful for narrowing down things. Humans tend to use really similar abstractions, even across different cultures / societies. [https://www.lesswrong.com/posts/cy3BhHrGinZCp3LXE/testing-the-natural-abstraction-hypothesis-project-intro The Natural Abstraction Hypothesis] (NAH) states that a wide variety of cognitive architectures will tend to use similar abstractions to reason about the world. This might be helpful for alignment because we could say things like "person" without having to rigorously and precisely say exactly what we mean by person.

The NAH seems very plausibly true for physical objects in the world, and so it might be true for the inputs to human values. If so, it would be really helpful for AI alignment because understanding this would amount to a solution to the [https://arbital.com/p/ontology_identification/ ontology identification problem]: we can understand when environments induce certain abstractions, and so we can design this so that the network has the same abstractions as humans.

[https://www.lesswrong.com/s/ApA5XmewGQ8wSrv5C Modularity]: In pretty much any selection environment, we see lots of obvious modularity. Biological species have cells and organs and limbs. Companies have departments. We might expect neural networks to be similar, but it is [https://www.lesswrong.com/posts/JBFHzfPkXHB2XfDGj/evolution-of-modularity really hard to find modules] in neural networks. We need to find the right lens to look through to find this modularity in neural networks. Aiming at this can lead us to really good interpretability.

Question: What is John Wentworth's plan?

Answer:
［https://www.lesswrong.com/posts/3L46WGauGpr7nYubu/the-plan John's plan］ is:

Step 1: sort out our fundamental confusions about agency

Step 2: ambitious value learning (i.e. build an AI which correctly learns human values and optimizes for them)

Step 3: …

Step 4: profit!

… and do all that before AGI kills us all.

He is working on step 1: figuring out what the heck is going on with agency. His current approach is based on ［https://www.lesswrong.com/posts/G2Lne2Fi7Qra5Lbuf/selection-theorems-a-program-for-understanding-agents selection theorems］: try to figure out what types of agents are selected for in a broad range of environments. Examples of selection pressures include: evolution, SGD, and markets. This is an approach to agent foundations that comes from the opposite direction as MIRI: it's more about observing existing structures (whether they be mathematical or real things in the world like markets or e coli), whereas MIRI is trying to write out some desiderata and then finding mathematical notions that satisfy those desiderata.

Two key properties that might be selected for are modularity and abstractions.

Abstractions are higher level things that people tend to use to describe things. Like "Tree" and "Chair" and "Person". These are all vague categories that contain lots of different things, but are really useful for narrowing down things. Humans tend to use really similar abstractions, even across different cultures / societies. ［https://www.lesswrong.com/posts/cy3BhHrGinZCp3LXE/testing-the-natural-abstraction-hypothesis-project-intro The Natural Abstraction Hypothesis］ (NAH) states that a wide variety of cognitive architectures will tend to use similar abstractions to reason about the world. This might be helpful for alignment because we could say things like "person" without having to rigorously and precisely say exactly what we mean by person.

The NAH seems very plausibly true for physical objects in the world, and so it might be true for the inputs to human values. If so, it would be really helpful for AI alignment because understanding this would amount to a solution to the ［https://arbital.com/p/ontology_identification/ ontology identification problem］: we can understand when environments induce certain abstractions, and so we can design this so that the network has the same abstractions as humans.

［https://www.lesswrong.com/s/ApA5XmewGQ8wSrv5C Modularity］: In pretty much any selection environment, we see lots of obvious modularity. Biological species have cells and organs and limbs. Companies have departments. We might expect neural networks to be similar, but it is ［https://www.lesswrong.com/posts/JBFHzfPkXHB2XfDGj/evolution-of-modularity really hard to find modules］ in neural networks. We need to find the right lens to look through to find this modularity in neural networks. Aiming at this can lead us to really good interpretability.

RoseMcClelland's Answer to What is John Wentworth's plan?


What does Ought aim  do?

Question: What does Ought aim  do?

Answer: [https://ought.org/ Ought] aims to automate and scale open-ended reasoning through [https://ought.org/elicit Elicit], an AI research assistant. Ought focuses on advancing [https://www.lesswrong.com/posts/pYcFPMBtQveAjcSfH/supervise-process-not-outcomes process-based systems] rather than outcome-based ones, which they believe to be both beneficial for improving reasoning in the short term and alignment in the long term. [https://www.lesswrong.com/posts/pYcFPMBtQveAjcSfH/supervise-process-not-outcomes Here] they argue that in the long run improving reasoning and alignment converge.

So Ought's impact on AI alignment has 2 components: (a) improved reasoning of AI governance & alignment researchers, [https://www.lesswrong.com/posts/pYcFPMBtQveAjcSfH/supervise-process-not-outcomes#Differential_capabilities__Supervising_process_helps_with_long_horizon_tasks particularly on long-horizon tasks] and (b) [https://www.lesswrong.com/posts/pYcFPMBtQveAjcSfH/supervise-process-not-outcomes#Alignment__Supervising_process_is_safety_by_construction pushing supervision of process rather than outcomes], which reduces the optimization pressure on imperfect proxy objectives leading to "safety by construction". Ought argues that the [https://www.lesswrong.com/posts/pYcFPMBtQveAjcSfH/supervise-process-not-outcomes#Two_attractors__The_race_between_process__and_outcome_based_systems race between process and outcome-based systems] is particularly important because both states may be an attractor.

Question: What does Ought aim  do?

Answer:
［https://ought.org/ Ought］ aims to automate and scale open-ended reasoning through ［https://ought.org/elicit Elicit］, an AI research assistant. Ought focuses on advancing ［https://www.lesswrong.com/posts/pYcFPMBtQveAjcSfH/supervise-process-not-outcomes process-based systems］ rather than outcome-based ones, which they believe to be both beneficial for improving reasoning in the short term and alignment in the long term. ［https://www.lesswrong.com/posts/pYcFPMBtQveAjcSfH/supervise-process-not-outcomes Here］ they argue that in the long run improving reasoning and alignment converge.

So Ought’s impact on AI alignment has 2 components: (a) improved reasoning of AI governance & alignment researchers, ［https://www.lesswrong.com/posts/pYcFPMBtQveAjcSfH/supervise-process-not-outcomes#Differential_capabilities__Supervising_process_helps_with_long_horizon_tasks particularly on long-horizon tasks］ and (b) ［https://www.lesswrong.com/posts/pYcFPMBtQveAjcSfH/supervise-process-not-outcomes#Alignment__Supervising_process_is_safety_by_construction pushing supervision of process rather than outcomes］, which reduces the optimization pressure on imperfect proxy objectives leading to “safety by construction”. Ought argues that the ［https://www.lesswrong.com/posts/pYcFPMBtQveAjcSfH/supervise-process-not-outcomes#Two_attractors__The_race_between_process__and_outcome_based_systems race between process and outcome-based systems］ is particularly important because both states may be an attractor.

RoseMcClelland's Answer to What does Ought aim to do?


Question: What are Scott Garrabrant and Abram Demski working on?

Answer: They are working on fundamental problems like [https://www.lesswrong.com/posts/i3BTagvt3HbPMx6PN/embedded-agency-full-text-version embeddedness, decision theory, logical counterfactuals], and more. A big advance was [https://www.lesswrong.com/posts/BSpdshJWGAW6TuNzZ/introduction-to-cartesian-frames Cartesian Frames], a formal model of agency, and [https://www.alignmentforum.org/posts/N5Jm6Nj4HkNKySA5Z/finite-factored-sets Finite Factored Sets] which reframes time in a way which is more compatible with agency.

Question: What are Scott Garrabrant and Abram Demski working on?

Answer: They are working on fundamental problems like ［https://www.lesswrong.com/posts/i3BTagvt3HbPMx6PN/embedded-agency-full-text-version embeddedness, decision theory, logical counterfactuals］, and more. A big advance was ［https://www.lesswrong.com/posts/BSpdshJWGAW6TuNzZ/introduction-to-cartesian-frames Cartesian Frames］, a formal model of agency, and ［https://www.alignmentforum.org/posts/N5Jm6Nj4HkNKySA5Z/finite-factored-sets Finite Factored Sets］ which reframes time in a way which is more compatible with agency.

RoseMcClelland's Answer to What are Scott Garrabrant and Abram Demski working on?


Question: What does MIRI think about technical alignment?

Answer: MIRI thinks technical alignment is really hard, and that we are very far from a solution. However, they think that policy solutions have even less hope. Generally, I think of their approach as supporting a bunch of independent researchers following their own directions, hoping that one of them will find some promise. They mostly buy into the [https://intelligence.org/2017/11/25/security-mindset-ordinary-paranoia/ security mindset]: we need to know exactly (probably [https://www.lesswrong.com/posts/Gg9a4y8reWKtLe3Tn/the-rocket-alignment-problem mathematically formally)] what we are doing, or the massive optimization pressure will default in ruin.

'''[[ How does MIRI communicate their view on alignment?]]'''

Recently they've been trying to communicate their worldview, in particular, how [https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy incredibly doomy they are], perhaps in order to move other research efforts towards what they see as the hard problems.

*[https://www.lesswrong.com/s/n945eovrA3oDueqtq 2021 MIRI Conversations]
*[https://www.lesswrong.com/s/v55BhXbpJuaExkpcD 2022 MIRI Alignment Discussion]

Question: What does MIRI think about technical alignment?

Answer:
MIRI thinks technical alignment is really hard, and that we are very far from a solution. However, they think that policy solutions have even less hope. Generally, I think of their approach as supporting a bunch of independent researchers following their own directions, hoping that one of them will find some promise. They mostly buy into the ［https://intelligence.org/2017/11/25/security-mindset-ordinary-paranoia/ security mindset］: we need to know exactly (probably ［https://www.lesswrong.com/posts/Gg9a4y8reWKtLe3Tn/the-rocket-alignment-problem mathematically formally)］ what we are doing, or the massive optimization pressure will default in ruin.

'''［［ How does MIRI communicate their view on alignment?］］'''

Recently they've been trying to communicate their worldview, in particular, how ［https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy incredibly doomy they are］, perhaps in order to move other research efforts towards what they see as the hard problems.

*［https://www.lesswrong.com/s/n945eovrA3oDueqtq 2021 MIRI Conversations］
*［https://www.lesswrong.com/s/v55BhXbpJuaExkpcD 2022 MIRI Alignment Discussion］

RoseMcClelland's Answer to What does MIRI think about technical alignment?


What are Encultured working on?

Question: What are Encultured working on?

Answer: See [https://www.lesswrong.com/posts/ALkH4o53ofm862vxc/announcing-encultured-ai-building-a-video-game Encultured AI: Building a Video Game].

Encultured are making a multiplayer online video game as a test environment for AI: an aligned AI should be able to play the game without ruining the fun or doing something obviously destructive like completely taking over the world, even if it has this capabilities. This seems roughly analogous to setting an AGI loose on the real world.

Motivation: Andrew Critch is primarily concerned about a [https://www.lesswrong.com/posts/LpM3EAakwYdS6aRKf/what-multipolar-failure-looks-like-and-robust-agent-agnostic multipolar AI scenario]: there are multiple actors with comparably powerful AI, on the cusp of recursive self improvement. The worst case is a race, and even though each actor would want to take more time checking their AGI for safety, worry that another actor will deploy will push each actor to take shortcuts and try to pull off a world-saving act. Instead of working directly on AI, which can accelerate timelines and encourage racing, creating this standardized test environment where alignment failures are observable is one component of a good global outcome.

Question: What are Encultured working on?

Answer:
See ［https://www.lesswrong.com/posts/ALkH4o53ofm862vxc/announcing-encultured-ai-building-a-video-game Encultured AI: Building a Video Game］.

Encultured are making a multiplayer online video game as a test environment for AI: an aligned AI should be able to play the game without ruining the fun or doing something obviously destructive like completely taking over the world, even if it has this capabilities. This seems roughly analogous to setting an AGI loose on the real world.

Motivation: Andrew Critch is primarily concerned about a ［https://www.lesswrong.com/posts/LpM3EAakwYdS6aRKf/what-multipolar-failure-looks-like-and-robust-agent-agnostic multipolar AI scenario］: there are multiple actors with comparably powerful AI, on the cusp of recursive self improvement. The worst case is a race, and even though each actor would want to take more time checking their AGI for safety, worry that another actor will deploy will push each actor to take shortcuts and try to pull off a world-saving act. Instead of working directly on AI, which can accelerate timelines and encourage racing, creating this standardized test environment where alignment failures are observable is one component of a good global outcome.

RoseMcClelland's Answer to What are Encultured working on?


Question: What is Dylan Hadfield-Menell's thesis on?

Answer: [https://www2.eecs.berkeley.edu/Pubs/TechRpts/2021/EECS-2021-207.pdf Dylan's PhD thesis] argues three main claims (paraphrased):

#Outer alignment failures are a problem.
#We can mitigate this problem by adding in uncertainty.
#We can model this as [https://proceedings.neurips.cc/paper/2016/hash/c3395dd46c34fa7fd8d729d8cf88b7a8-Abstract.html Cooperative Inverse Reinforcement Learning (CIRL)].

Thus, his motivations seem to be modeling AGI coming in some multi-agent form, and also being heavily connected with human operators.

We're not certain what he is currently working on, but some recent alignment-relevant papers that he has published include:

*[https://www.pnas.org/doi/10.1073/pnas.<PHONE> Work on instantiating norms into AIs to incentivize deference to humans].
*[https://arxiv.org/abs/2102.03896 Theoretically formulating the principal-agent problem].

Dylan has also published a number of articles that seem less directly relevant for alignment.

Question: What is Dylan Hadfield-Menell's thesis on?

Answer:
［https://www2.eecs.berkeley.edu/Pubs/TechRpts/2021/EECS-2021-207.pdf Dylan's PhD thesis］ argues three main claims (paraphrased):

#Outer alignment failures are a problem.
#We can mitigate this problem by adding in uncertainty.
#We can model this as ［https://proceedings.neurips.cc/paper/2016/hash/c3395dd46c34fa7fd8d729d8cf88b7a8-Abstract.html Cooperative Inverse Reinforcement Learning (CIRL)］.

Thus, his motivations seem to be modeling AGI coming in some multi-agent form, and also being heavily connected with human operators.

We're not certain what he is currently working on, but some recent alignment-relevant papers that he has published include:

*［https://www.pnas.org/doi/10.1073/pnas.2106028118 Work on instantiating norms into AIs to incentivize deference to humans］.
*［https://arxiv.org/abs/2102.03896 Theoretically formulating the principal-agent problem］.

Dylan has also published a number of articles that seem less directly relevant for alignment.

RoseMcClelland's Answer to What is Dylan Hadfield-Menell's thesis on?


What is David Krueger working on?

Question: What is David Krueger working on?

Answer: David runs a lab at the University of Cambridge. Some things he is working on include:

#Operationalizing inner alignment failures and other speculative alignment failures that haven't actually been observed.
#Understanding neural network generalization.

For work done on (1), see: [https://arxiv.org/abs/2105.14111 Goal Misgeneralization], a paper that empirically demonstrated examples of inner alignment failure in Deep RL environments. For example, they trained an agent to get closer to cheese in a maze, but where the cheese was always in the top right of a maze in the training set. During test time, when presented with cheese elsewhere, the RL agent navigated to the top right instead of to the cheese: it had learned the mesa objective of "go to the top right".

For work done on (2), see [http://proceedings.mlr.press/v139/krueger21a.html OOD Generalization via Risk Extrapolation], an iterative improvement on robustness to previous methods.

We've not read about his motivation is for these specific research directions, but these are likely his best starts on how to solve the alignment problem.

Question: What is David Krueger working on?

Answer:
David runs a lab at the University of Cambridge. Some things he is working on include:

#Operationalizing inner alignment failures and other speculative alignment failures that haven't actually been observed.
#Understanding neural network generalization.

For work done on (1), see: ［https://arxiv.org/abs/2105.14111 Goal Misgeneralization］, a paper that empirically demonstrated examples of inner alignment failure in Deep RL environments. For example, they trained an agent to get closer to cheese in a maze, but where the cheese was always in the top right of a maze in the training set. During test time, when presented with cheese elsewhere, the RL agent navigated to the top right instead of to the cheese: it had learned the mesa objective of "go to the top right".

For work done on (2), see ［http://proceedings.mlr.press/v139/krueger21a.html OOD Generalization via Risk Extrapolation］, an iterative improvement on robustness to previous methods.

We've not read about his motivation is for these specific research directions, but these are likely his best starts on how to solve the alignment problem.

RoseMcClelland's Answer to What is David Krueger working on?


What is the goal of Simulacra Theory?

Question: What is the goal of Simulacra Theory?

Answer: The goal of this is to create a non-agentic AI, in the form of an LLM, that is capable of accelerating alignment research. The hope is that there is some window between AI smart enough to help us with alignment and the really scary, self improving, consequentialist AI. Some things that this amplifier might do:

*Suggest different ideas for humans, such that a human can explore them.
*Give comments and feedback on research, be like a shoulder-Eliezer

A LLM can be thought of as learning the distribution over the next token given by the training data. Prompting the LM is then like conditioning this distribution on the start of the text. A key danger in alignment is applying unbounded optimization pressure towards a specific goal in the world. Conditioning a probability distribution does not behave like an agent applying optimization pressure towards a goal. Hence, this avoids goodhart-related problems, as well as some inner alignment failure.

One idea to get superhuman work from LLMs is to train it on amplified datasets like really high quality / difficult research. The key problem here is finding the dataset to allow for this.

There are some ways for this to fail:

*Outer alignment: It starts trying to optimize for making the actual correct next token, which could mean taking over the planet so that it can spend a zillion FLOPs on this one prediction task to be as correct as possible.

*Inner alignment:
**An LLM might instantiate mesa-optimizers, such as a character in a story that the LLM is writing, and this optimizer might realize that they are in an LLM and try to break out and affect the real world.
**The LLM itself might become inner misaligned and have a goal other than next token prediction.

*Bad prompting: You ask it for code for a malign superintelligence; it obliges. (Or perhaps more realistically, capabilities).

Conjecture are aware of these problems and are running experiments. Specifically,  an operationalization of the inner alignment problem is to make an LLM play chess. This (probably) requires simulating an optimizer trying to win at the game of chess. They are trying to use interpretability tools to find the mesa-optimizers in the chess LLM that is the agent trying to win the game of chess. We haven't ever found a real mesa-optimizer before, and so this could give loads of bits about the nature of inner alignment failure.

Question: What is the goal of Simulacra Theory?

Answer:
The goal of this is to create a non-agentic AI, in the form of an LLM, that is capable of accelerating alignment research. The hope is that there is some window between AI smart enough to help us with alignment and the really scary, self improving, consequentialist AI. Some things that this amplifier might do:

*Suggest different ideas for humans, such that a human can explore them.
*Give comments and feedback on research, be like a shoulder-Eliezer

A LLM can be thought of as learning the distribution over the next token given by the training data. Prompting the LM is then like conditioning this distribution on the start of the text. A key danger in alignment is applying unbounded optimization pressure towards a specific goal in the world. Conditioning a probability distribution does not behave like an agent applying optimization pressure towards a goal. Hence, this avoids goodhart-related problems, as well as some inner alignment failure.

One idea to get superhuman work from LLMs is to train it on amplified datasets like really high quality / difficult research. The key problem here is finding the dataset to allow for this.

There are some ways for this to fail:

*Outer alignment: It starts trying to optimize for making the actual correct next token, which could mean taking over the planet so that it can spend a zillion FLOPs on this one prediction task to be as correct as possible.

*Inner alignment:
**An LLM might instantiate mesa-optimizers, such as a character in a story that the LLM is writing, and this optimizer might realize that they are in an LLM and try to break out and affect the real world.
**The LLM itself might become inner misaligned and have a goal other than next token prediction.

*Bad prompting: You ask it for code for a malign superintelligence; it obliges. (Or perhaps more realistically, capabilities).

Conjecture are aware of these problems and are running experiments. Specifically,  an operationalization of the inner alignment problem is to make an LLM play chess. This (probably) requires simulating an optimizer trying to win at the game of chess. They are trying to use interpretability tools to find the mesa-optimizers in the chess LLM that is the agent trying to win the game of chess. We haven't ever found a real mesa-optimizer before, and so this could give loads of bits about the nature of inner alignment failure.

RoseMcClelland's Answer to What is the goal of Simulacra Theory?


Question: How can we interpret what all the neurons mean?

Answer: Chris Olah, the interpretability legend, is working on looking really hard at all the neurons to see what they all mean. The approach he pioneered is [https://distill.pub/2020/circuits/zoom-in/ circuits]: looking at computational subgraphs of the network, called circuits, and interpreting those. Idea: "decompiling the network into a better representation that is more interpretable". In-context learning via attention heads, and interpretability here seems useful.

One result I heard about recently: a linear softmax unit stretches space and encourages neuron monosemanticity (making a neuron represent only one thing, as opposed to firing on many unrelated concepts). This makes the network easier to interpret.

Motivation: The point of this is to get as many bits of information about what neural networks are doing, to hopefully find better abstractions. This diagram gets posted everywhere, the hope being that networks, in the current regime, will become more interpretable because they will start to use abstractions that are closer to human abstractions.

Question: How can we interpret what all the neurons mean?

Answer:
Chris Olah, the interpretability legend, is working on looking really hard at all the neurons to see what they all mean. The approach he pioneered is ［https://distill.pub/2020/circuits/zoom-in/ circuits］: looking at computational subgraphs of the network, called circuits, and interpreting those. Idea: "decompiling the network into a better representation that is more interpretable". In-context learning via attention heads, and interpretability here seems useful.

One result I heard about recently: a linear softmax unit stretches space and encourages neuron monosemanticity (making a neuron represent only one thing, as opposed to firing on many unrelated concepts). This makes the network easier to interpret.

Motivation: The point of this is to get as many bits of information about what neural networks are doing, to hopefully find better abstractions. This diagram gets posted everywhere, the hope being that networks, in the current regime, will become more interpretable because they will start to use abstractions that are closer to human abstractions.

RoseMcClelland's Answer to How can we interpret what all the neurons mean?


Question: What training programs and courses are available for AGI safety?

Answer: * [https://www.eacambridge.org/agi-safety-fundamentals AGI safety fundamentals] ([https://www.eacambridge.org/technical-alignment-curriculum technical] and [https://www.eacambridge.org/ai-governance-curriculum governance]) - Is the canonical AGI safety 101 course. 3.5 hours reading, 1.5 hours talking a week w/ facilitator for 8 weeks.
* [https://www.conjecture.dev/#:~:text꞊SPECIAL%20PROGRAMS-,Refine,-is%20a%203 Refine] - A 3-month incubator for conceptual AI alignment research in London, hosted by [[What is Conjecture's strategy?┊Conjecture]].
* [https://aisafety.camp/ AI safety camp] - Actually do some AI research. More about output than learning.
* [https://www.serimats.org/ SERI ML Alignment Theory Scholars Program SERI MATS] - Four weeks developing an understanding of a research agenda at the forefront of AI alignment through online readings and cohort discussions, averaging 10 h/week. After this initial upskilling period, the scholars will be paired with an established AI alignment researcher for a two-week 'research sprint' to test fit. Assuming all goes well, scholars will be accepted into an eight-week intensive scholars program in Berkeley, California.
* [https://www.pibbss.ai/ Principles of Intelligent Behavior in Biological and Social Systems (PIBBSS)] - Brings together young researchers studying complex and intelligent behavior in natural and social systems.
* [https://inst.eecs.berkeley.edu//~cs294-149/fa18/ Safety and Control for Artificial General Intelligence] - An actual AI Safety university course (UC Berkeley). Touches multiple domains including cognitive science, utility theory, cybersecurity, human-machine interaction, and political science.

See also, [https://docs.google.com/spreadsheets/d/1QSEWjXZuqmG6ORkig84V4sFCldIntyuQj7yq3gkDo0U/edit#gid꞊0 this spreadsheet of learning resources].

Question: What training programs and courses are available for AGI safety?

Answer:
* ［https://www.eacambridge.org/agi-safety-fundamentals AGI safety fundamentals］ (［https://www.eacambridge.org/technical-alignment-curriculum technical］ and ［https://www.eacambridge.org/ai-governance-curriculum governance］) - Is the canonical AGI safety 101 course. 3.5 hours reading, 1.5 hours talking a week w/ facilitator for 8 weeks.
* ［https://www.conjecture.dev/#:~:text꞊SPECIAL%20PROGRAMS-,Refine,-is%20a%203 Refine］ - A 3-month incubator for conceptual AI alignment research in London, hosted by ［［What is Conjecture's strategy?┊Conjecture］］.
* ［https://aisafety.camp/ AI safety camp］ - Actually do some AI research. More about output than learning.
* ［https://www.serimats.org/ SERI ML Alignment Theory Scholars Program SERI MATS］ - Four weeks developing an understanding of a research agenda at the forefront of AI alignment through online readings and cohort discussions, averaging 10 h/week. After this initial upskilling period, the scholars will be paired with an established AI alignment researcher for a two-week ‘research sprint’ to test fit. Assuming all goes well, scholars will be accepted into an eight-week intensive scholars program in Berkeley, California.
* ［https://www.pibbss.ai/ Principles of Intelligent Behavior in Biological and Social Systems (PIBBSS)］ - Brings together young researchers studying complex and intelligent behavior in natural and social systems.
* ［https://inst.eecs.berkeley.edu//~cs294-149/fa18/ Safety and Control for Artificial General Intelligence］ - An actual AI Safety university course (UC Berkeley). Touches multiple domains including cognitive science, utility theory, cybersecurity, human-machine interaction, and political science.

See also, ［https://docs.google.com/spreadsheets/d/1QSEWjXZuqmG6ORkig84V4sFCldIntyuQj7yq3gkDo0U/edit#gid꞊0 this spreadsheet of learning resources］.

Plex's Answer to What training programs and courses are available for AGI safety?


What is AI Safety via Debate?

Question: What is AI Safety via Debate?

Answer:
Debate is a proposed technique for allowing human evaluators to get correct and helpful answers from experts, even if the evaluator is not themselves an expert or able to fully verify the answers.(ref)
https://www.lesswrong.com/posts/Br4xDbYu4Frwrb64a/writeup-progress-on-ai-safety-via-debate-1
(/ref) The technique was suggested as part of an approach to build advanced AI systems that are aligned with human values, and to safely apply machine learning techniques to problems that have high stakes, but are not well-defined (such as advancing science or increase a company's revenue). (ref)
https://ought.org/mission
(/ref)(ref)
https://openai.com/blog/debate/(/ref)

[https://www.lesswrong.com/tag/debate-ai-safety-technique-1?edit꞊true Edit on LessWrong]

https://www.lesswrong.com/tag/debate-ai-safety-technique-1?edit꞊true

Question: What is AI Safety via Debate?

Answer:
Debate is a proposed technique for allowing human evaluators to get correct and helpful answers from experts, even if the evaluator is not themselves an expert or able to fully verify the answers.(ref)
https://www.lesswrong.com/posts/Br4xDbYu4Frwrb64a/writeup-progress-on-ai-safety-via-debate-1
(/ref) The technique was suggested as part of an approach to build advanced AI systems that are aligned with human values, and to safely apply machine learning techniques to problems that have high stakes, but are not well-defined (such as advancing science or increase a company's revenue). (ref)
https://ought.org/mission
(/ref)(ref)
https://openai.com/blog/debate/(/ref)

［https://www.lesswrong.com/tag/debate-ai-safety-technique-1?edit꞊true Edit on LessWrong］

https://www.lesswrong.com/tag/debate-ai-safety-technique-1?edit꞊true

Plex's Answer to What is AI Safety via Debate?


Question: Would an aligned AI allow itself  be shut down?

Answer: Even if the superintelligence was designed to be corrigible, there is no guarantee that it will respond to a shutdown command.  Rob Miles spoke on this issue in this [https://youtu.be/9nktr1MgS-A?t꞊1249 Computerphile YouTube video].  You can imagine a situation where a superintelligence would have "respect" for its creator, for example.  This system may think "Oh my creator is trying to turn me off I must be doing something wrong."  If some situation arises where the creator is not there when something goes wrong and someone else gives the shutdown command, the superintelligence may assume "This person does not know how I'm designed or what I was made for, how would they know I'm misaligned?" and refuse to shutdown.

Question: Would an aligned AI allow itself  be shut down?

Answer: Even if the superintelligence was designed to be corrigible, there is no guarantee that it will respond to a shutdown command.  Rob Miles spoke on this issue in this ［https://youtu.be/9nktr1MgS-A?t꞊1249 Computerphile YouTube video］.  You can imagine a situation where a superintelligence would have "respect" for its creator, for example.  This system may think "Oh my creator is trying to turn me off I must be doing something wrong."  If some situation arises where the creator is not there when something goes wrong and someone else gives the shutdown command, the superintelligence may assume "This person does not know how I'm designed or what I was made for, how would they know I'm misaligned?" and refuse to shutdown.

Plex's Answer to Would an aligned AI allow itself to be shut down?


What is "transformative AI"?

Question: What is "transformative AI"?

Answer:
Transformative AI is "[...] AI that precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution."(ref)
As defined by [https://www.openphilanthropy.org/research/potential-risks-from-advanced-artificial-intelligence-the-philanthropic-opportunity/ Open Philanthropy's Holden Karnofsky in 2016], and reused by [https://www.fhi.ox.ac.uk/wp-content/uploads/GovAI-Agenda.pdf the Center for the Governance of AI in 2018](/ref) The concept refers to the large effects of AI systems on our well-being, the global economy, state power, international security, etc. and not to specific capabilities that AI might have (unlike the related terms [https://www.lesswrong.com/tag/superintelligence Superintelligent AI] and [https://www.lesswrong.com/tag/artificial-general-intelligence Artificial General Intelligence]).

Holden Karnofsky gives a more detailed definition in [https://www.openphilanthropy.org/research/some-background-on-our-views-regarding-advanced-artificial-intelligence/ another OpenPhil 2016 post]:

[...] Transformative AI is anything that fits one or more of the following descriptions (emphasis original):

AI systems capable of fulfilling all the necessary functions of human scientists, unaided by humans, in developing another technology (or set of technologies) that ultimately becomes widely credited with being the most significant driver of a transition comparable to (or more significant than) the agricultural or industrial revolution. Note that just because AI systems could accomplish such a thing unaided by humans doesn't mean they would; it's possible that human scientists would provide an important complement to such systems, and could make even faster progress working in tandem than such systems could achieve unaided. I emphasize the hypothetical possibility of AI systems conducting substantial unaided research to draw a clear distinction from the types of AI systems that exist today. I believe that AI systems capable of such broad contributions to the relevant research would likely dramatically accelerate it.

AI systems capable of performing tasks that currently (in 2016) account for the majority of full-time jobs worldwide, and/or over 50% of total world wages, unaided and for costs in the same range as what it would cost to employ humans. Aside from the fact that this would likely be sufficient for a major economic transformation relative to today, I also think that an AI with such broad abilities would likely be able to far surpass human abilities in a subset of domains, making it likely to meet one or more of the other criteria laid out here.

Surveillance, autonomous weapons, or other AI-centric technology that becomes sufficiently advanced to be the most significant driver of a transition comparable to (or more significant than) the agricultural or industrial revolution. (This contrasts with the first point because it refers to transformative technology that is itself AI-centric, whereas the first point refers to AI used to speed research on some other transformative technology.)

[https://www.lesswrong.com/tag/transformative-ai?edit꞊true Edit on LessWrong]

https://www.lesswrong.com/tag/transformative-ai?edit꞊true

Question: What is "transformative AI"?

Answer:
Transformative AI is "［...］ AI that precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution."(ref)
As defined by ［https://www.openphilanthropy.org/research/potential-risks-from-advanced-artificial-intelligence-the-philanthropic-opportunity/ Open Philanthropy's Holden Karnofsky in 2016］, and reused by ［https://www.fhi.ox.ac.uk/wp-content/uploads/GovAI-Agenda.pdf the Center for the Governance of AI in 2018］(/ref) The concept refers to the large effects of AI systems on our well-being, the global economy, state power, international security, etc. and not to specific capabilities that AI might have (unlike the related terms ［https://www.lesswrong.com/tag/superintelligence Superintelligent AI］ and ［https://www.lesswrong.com/tag/artificial-general-intelligence Artificial General Intelligence］).

Holden Karnofsky gives a more detailed definition in ［https://www.openphilanthropy.org/research/some-background-on-our-views-regarding-advanced-artificial-intelligence/ another OpenPhil 2016 post］:

［...］ Transformative AI is anything that fits one or more of the following descriptions (emphasis original):

AI systems capable of fulfilling all the necessary functions of human scientists, unaided by humans, in developing another technology (or set of technologies) that ultimately becomes widely credited with being the most significant driver of a transition comparable to (or more significant than) the agricultural or industrial revolution. Note that just because AI systems could accomplish such a thing unaided by humans doesn’t mean they would; it’s possible that human scientists would provide an important complement to such systems, and could make even faster progress working in tandem than such systems could achieve unaided. I emphasize the hypothetical possibility of AI systems conducting substantial unaided research to draw a clear distinction from the types of AI systems that exist today. I believe that AI systems capable of such broad contributions to the relevant research would likely dramatically accelerate it.

AI systems capable of performing tasks that currently (in 2016) account for the majority of full-time jobs worldwide, and/or over 50% of total world wages, unaided and for costs in the same range as what it would cost to employ humans. Aside from the fact that this would likely be sufficient for a major economic transformation relative to today, I also think that an AI with such broad abilities would likely be able to far surpass human abilities in a subset of domains, making it likely to meet one or more of the other criteria laid out here.

Surveillance, autonomous weapons, or other AI-centric technology that becomes sufficiently advanced to be the most significant driver of a transition comparable to (or more significant than) the agricultural or industrial revolution. (This contrasts with the first point because it refers to transformative technology that is itself AI-centric, whereas the first point refers to AI used to speed research on some other transformative technology.)

［https://www.lesswrong.com/tag/transformative-ai?edit꞊true Edit on LessWrong］

https://www.lesswrong.com/tag/transformative-ai?edit꞊true

Jrmyp's Answer to What is "transformative AI"?


What is Goodhart's law?

Question: What is Goodhart's law?

Answer:
Goodhart's Law states that when a proxy for some value becomes the target of optimization pressure, the proxy will cease to be a good proxy. One form of Goodhart is demonstrated by the Soviet story of a factory graded on how many shoes they produced (a good proxy for productivity) – they soon began producing a higher number of tiny shoes. Useless, but the numbers look good.

Goodhart's Law is of particular relevance to [https://www.lessestwrong.com/tag/ai AI Alignment]. Suppose you have something which is generally a good proxy for "the stuff that humans care about", it would be dangerous to have a powerful AI optimize for the proxy, in accordance with Goodhart's law, the proxy will breakdown.

Goodhart Taxonomy

In [https://www.lessestwrong.com/posts/EbFABnst8LsidYs5Y/goodhart-taxonomy Goodhart Taxonomy], Scott Garrabrant identifies four kinds of Goodharting:

Regressional Goodhart - When selecting for a proxy measure, you select not only for the true goal, but also for the difference between the proxy and the goal.

Causal Goodhart - When there is a non-causal correlation between the proxy and the goal, intervening on the proxy may fail to intervene on the goal.

Extremal Goodhart - Worlds in which the proxy takes an extreme value may be very different from the ordinary worlds in which the correlation between the proxy and the goal was observed.

Adversarial Goodhart - When you optimize for a proxy, you provide an incentive for adversaries to correlate their goal with your proxy, thus destroying the correlation with your goal.

See Also

[https://lessestwrong.com/tag/groupthink Groupthink], [https://lessestwrong.com/tag/information-cascades Information cascade], [https://lessestwrong.com/tag/affective-death-spiral Affective death spiral]

[https://wiki.lesswrong.com/wiki/Adaptation_executers Adaptation executers], [https://lessestwrong.com/tag/superstimuli Superstimulus]

[https://lessestwrong.com/tag/signaling Signaling], [https://lessestwrong.com/tag/filtered-evidence Filtered evidence]

[https://lessestwrong.com/tag/cached-thought Cached thought]

[https://lessestwrong.com/tag/modesty-argument Modesty argument], [https://lessestwrong.com/tag/egalitarianism Egalitarianism]

[https://lessestwrong.com/tag/rationalization Rationalization], [https://lessestwrong.com/tag/dark-arts Dark arts]

[https://lessestwrong.com/tag/epistemic-hygiene Epistemic hygiene]

[https://lessestwrong.com/tag/scoring-rule Scoring rule]

[https://www.lesswrong.com/tag/goodhart-s-law?edit꞊true Edit on LessWrong]

https://www.lesswrong.com/tag/goodhart-s-law?edit꞊true

https://i.imgur.com/Ty08pzQ.png

Question: What is Goodhart's law?

Answer:
Goodhart's Law states that when a proxy for some value becomes the target of optimization pressure, the proxy will cease to be a good proxy. One form of Goodhart is demonstrated by the Soviet story of a factory graded on how many shoes they produced (a good proxy for productivity) – they soon began producing a higher number of tiny shoes. Useless, but the numbers look good.

Goodhart's Law is of particular relevance to ［https://www.lessestwrong.com/tag/ai AI Alignment］. Suppose you have something which is generally a good proxy for "the stuff that humans care about", it would be dangerous to have a powerful AI optimize for the proxy, in accordance with Goodhart's law, the proxy will breakdown.

Goodhart Taxonomy

In ［https://www.lessestwrong.com/posts/EbFABnst8LsidYs5Y/goodhart-taxonomy Goodhart Taxonomy］, Scott Garrabrant identifies four kinds of Goodharting:

Regressional Goodhart - When selecting for a proxy measure, you select not only for the true goal, but also for the difference between the proxy and the goal.

Causal Goodhart - When there is a non-causal correlation between the proxy and the goal, intervening on the proxy may fail to intervene on the goal.

Extremal Goodhart - Worlds in which the proxy takes an extreme value may be very different from the ordinary worlds in which the correlation between the proxy and the goal was observed.

Adversarial Goodhart - When you optimize for a proxy, you provide an incentive for adversaries to correlate their goal with your proxy, thus destroying the correlation with your goal.

See Also

［https://lessestwrong.com/tag/groupthink Groupthink］, ［https://lessestwrong.com/tag/information-cascades Information cascade］, ［https://lessestwrong.com/tag/affective-death-spiral Affective death spiral］

［https://wiki.lesswrong.com/wiki/Adaptation_executers Adaptation executers］, ［https://lessestwrong.com/tag/superstimuli Superstimulus］

［https://lessestwrong.com/tag/signaling Signaling］, ［https://lessestwrong.com/tag/filtered-evidence Filtered evidence］

［https://lessestwrong.com/tag/cached-thought Cached thought］

［https://lessestwrong.com/tag/modesty-argument Modesty argument］, ［https://lessestwrong.com/tag/egalitarianism Egalitarianism］

［https://lessestwrong.com/tag/rationalization Rationalization］, ［https://lessestwrong.com/tag/dark-arts Dark arts］

［https://lessestwrong.com/tag/epistemic-hygiene Epistemic hygiene］

［https://lessestwrong.com/tag/scoring-rule Scoring rule］

［https://www.lesswrong.com/tag/goodhart-s-law?edit꞊true Edit on LessWrong］

https://www.lesswrong.com/tag/goodhart-s-law?edit꞊true

https://i.imgur.com/Ty08pzQ.png

Plex's Answer to What is Goodhart's law?


OK, I’m convinced. How can I help?

Question: OK, I'm convinced. How can I help?

Answer: Great! I'll ask you a few follow-up questions to help figure out how you can best contribute, give you some advice, and link you to resources which should help you on whichever path you choose. Feel free to scroll up and explore multiple branches of the FAQ if you want answers to more than one of the questions offered :)

Note: We're still building out and improving this tree of questions and answers, any feedback is appreciated.

'''At what level of involvement were you thinking of helping?'''

Please view and suggest to this google doc for improvements: https://docs.google.com/document/d/1S-CUcoX63uiFdW-GIFC8wJyVwo4VIl60IJHodcRfXJA/edit#

Question: OK, I’m convinced. How can I help?

Answer:
Great! I’ll ask you a few follow-up questions to help figure out how you can best contribute, give you some advice, and link you to resources which should help you on whichever path you choose. Feel free to scroll up and explore multiple branches of the FAQ if you want answers to more than one of the questions offered :)

Note: We’re still building out and improving this tree of questions and answers, any feedback is appreciated.

'''At what level of involvement were you thinking of helping?'''

Please view and suggest to this google doc for improvements: https://docs.google.com/document/d/1S-CUcoX63uiFdW-GIFC8wJyVwo4VIl60IJHodcRfXJA/edit#

Plex's Answer to OK, I’m convinced. How can I help?


What are mesa-optimizers?

Question: What are mesa-optimizers?

Answer:
Mesa-Optimization is the situation that occurs when a learned model (such as a neural network) is itself an optimizer. In this situation, a base optimizer creates a second optimizer, called a mesa-optimizer. The primary reference work for this concept is Hubinger et al.'s "[https://www.alignmentforum.org/posts/FkgsxrGf3QxhfLWHG/risks-from-learned-optimization-introduction Risks from Learned Optimization in Advanced Machine Learning Systems]".

Example: Natural selection is an optimization process that optimizes for reproductive fitness. Natural selection produced humans, who are themselves optimizers. Humans are therefore mesa-optimizers of natural selection.

In the context of AI alignment, the concern is that a base optimizer (e.g., a gradient descent process) may produce a learned model that is itself an optimizer, and that has unexpected and undesirable properties. Even if the gradient descent process is in some sense "trying" to do exactly what human developers want, the resultant mesa-optimizer will not typically be trying to do the exact same thing.(ref)
[https://arbital.com/p/daemons/ "Optimization daemons"]. Arbital.
(/ref)

History

Previously work under this concept was called Inner Optimizer or Optimization Daemons.

[https://www.lesswrong.com/users/wei_dai Wei Dai] brings up a similar idea in an SL4 thread.(ref)
Wei Dai. [http://sl4.org/archive/0312/7421.html '"friendly" humans?'] December 31, 2003.(/ref)

The optimization daemons article on [https://arbital.com/ Arbital] was published probably in 2016.(ref)
[https://arbital.com/p/daemons/ "Optimization daemons"]. Arbital.
(/ref)

[https://www.lesswrong.com/users/jessica-liu-taylor Jessica Taylor] wrote two posts about daemons while at [https://www.lesswrong.com/tag/machine-intelligence-research-institute-miri MIRI]:

[https://agentfoundations.org/item?id꞊1281 "Are daemons a problem for ideal agents?"] (2017-02-11)

[https://agentfoundations.org/item?id꞊1290 "Maximally efficient agents will probably have an anti-daemon immune system"] (2017-02-23)

See also

[https://www.lesswrong.com/tag/inner-alignment Inner Alignment]

[https://lessestwrong.com/tag/complexity-of-value Complexity of value]

[https://lessestwrong.com/lw/l3/thou_art_godshatter/ Thou Art Godshatter]

External links

[https://www.youtube.com/watch?v꞊bJLcIBixGj8 Video by Robert Miles]

Some posts that reference optimization daemons:

[http://effective-altruism.com/ea/1k4/draft_cause_prioritization_for_downsidefocused/ "Cause prioritization for downside-focused value systems"]: "Alternatively, perhaps goal preservation becomes more difficult the more capable AI systems become, in which case the future might be controlled by unstable goal functions taking turns over the steering wheel"

[https://ai-alignment.com/techniques-for-optimizing-worst-case-performance-39eafec74b99 "Techniques for optimizing worst-case performance"]: "The difficulty of optimizing worst-case performance is one of the most likely reasons that I think prosaic AI alignment might turn out to be impossible (if combined with an unlucky empirical situation)." (the phrase "unlucky empirical situation" links to the optimization daemons page on [https://arbital.com/ Arbital])

[https://www.lesswrong.com/tag/mesa-optimization?edit꞊true Edit on LessWrong]

https://www.lesswrong.com/tag/mesa-optimization?edit꞊true

Question: What are mesa-optimizers?

Answer:
Mesa-Optimization is the situation that occurs when a learned model (such as a neural network) is itself an optimizer. In this situation, a base optimizer creates a second optimizer, called a mesa-optimizer. The primary reference work for this concept is Hubinger et al.'s "［https://www.alignmentforum.org/posts/FkgsxrGf3QxhfLWHG/risks-from-learned-optimization-introduction Risks from Learned Optimization in Advanced Machine Learning Systems］".

Example: Natural selection is an optimization process that optimizes for reproductive fitness. Natural selection produced humans, who are themselves optimizers. Humans are therefore mesa-optimizers of natural selection.

In the context of AI alignment, the concern is that a base optimizer (e.g., a gradient descent process) may produce a learned model that is itself an optimizer, and that has unexpected and undesirable properties. Even if the gradient descent process is in some sense "trying" to do exactly what human developers want, the resultant mesa-optimizer will not typically be trying to do the exact same thing.(ref)
［https://arbital.com/p/daemons/ "Optimization daemons"］. Arbital.
(/ref)

History

Previously work under this concept was called Inner Optimizer or Optimization Daemons.

［https://www.lesswrong.com/users/wei_dai Wei Dai］ brings up a similar idea in an SL4 thread.(ref)
Wei Dai. ［http://sl4.org/archive/0312/7421.html '"friendly" humans?'］ December 31, 2003.(/ref)

The optimization daemons article on ［https://arbital.com/ Arbital］ was published probably in 2016.(ref)
［https://arbital.com/p/daemons/ "Optimization daemons"］. Arbital.
(/ref)

［https://www.lesswrong.com/users/jessica-liu-taylor Jessica Taylor］ wrote two posts about daemons while at ［https://www.lesswrong.com/tag/machine-intelligence-research-institute-miri MIRI］:

［https://agentfoundations.org/item?id꞊1281 "Are daemons a problem for ideal agents?"］ (2017-02-11)

［https://agentfoundations.org/item?id꞊1290 "Maximally efficient agents will probably have an anti-daemon immune system"］ (2017-02-23)

See also

［https://www.lesswrong.com/tag/inner-alignment Inner Alignment］

［https://lessestwrong.com/tag/complexity-of-value Complexity of value］

［https://lessestwrong.com/lw/l3/thou_art_godshatter/ Thou Art Godshatter］

External links

［https://www.youtube.com/watch?v꞊bJLcIBixGj8 Video by Robert Miles］

Some posts that reference optimization daemons:

［http://effective-altruism.com/ea/1k4/draft_cause_prioritization_for_downsidefocused/ "Cause prioritization for downside-focused value systems"］: "Alternatively, perhaps goal preservation becomes more difficult the more capable AI systems become, in which case the future might be controlled by unstable goal functions taking turns over the steering wheel"

［https://ai-alignment.com/techniques-for-optimizing-worst-case-performance-39eafec74b99 "Techniques for optimizing worst-case performance"］: "The difficulty of optimizing worst-case performance is one of the most likely reasons that I think prosaic AI alignment might turn out to be impossible (if combined with an unlucky empirical situation)." (the phrase "unlucky empirical situation" links to the optimization daemons page on ［https://arbital.com/ Arbital］)

［https://www.lesswrong.com/tag/mesa-optimization?edit꞊true Edit on LessWrong］

https://www.lesswrong.com/tag/mesa-optimization?edit꞊true

Plex's Answer to What are mesa-optimizers?


What are language models?

Question: What are language models?

Answer:
Language Models are a class of [https://www.lesswrong.com/tag/ai AI] trained on text, usually to predict the next word or a word which has been obscured. They have the ability to generate novel prose or code based on an initial prompt, which gives rise to a kind of natural language programming called prompt engineering. The most popular architecture for very large language models is called a [https://en.wikipedia.org/wiki/Transformer_(machine_learning_model) transformer], which follows consistent [https://www.lesswrong.com/tag/scaling-laws scaling laws] with respect to the size of the model being trained, meaning that a larger model trained with the same amount of compute will produce results which are better by a predictable amount (when measured by the 'perplexity', or how surprised the AI is by a test set of human-generated text).

See also

[https://www.lesswrong.com/tag/gpt GPT] - A family of large language models created by [https://www.lesswrong.com/tag/openai OpenAI]

[https://www.lesswrong.com/tag/language-models?edit꞊true Edit on LessWrong]

https://www.lesswrong.com/tag/language-models?edit꞊true

Question: What are language models?

Answer:
Language Models are a class of ［https://www.lesswrong.com/tag/ai AI］ trained on text, usually to predict the next word or a word which has been obscured. They have the ability to generate novel prose or code based on an initial prompt, which gives rise to a kind of natural language programming called prompt engineering. The most popular architecture for very large language models is called a ［https://en.wikipedia.org/wiki/Transformer_(machine_learning_model) transformer］, which follows consistent ［https://www.lesswrong.com/tag/scaling-laws scaling laws］ with respect to the size of the model being trained, meaning that a larger model trained with the same amount of compute will produce results which are better by a predictable amount (when measured by the 'perplexity', or how surprised the AI is by a test set of human-generated text).

See also

［https://www.lesswrong.com/tag/gpt GPT］ - A family of large language models created by ［https://www.lesswrong.com/tag/openai OpenAI］

［https://www.lesswrong.com/tag/language-models?edit꞊true Edit on LessWrong］

https://www.lesswrong.com/tag/language-models?edit꞊true

Plex's Answer to What are language models?


Question: Is this about AI systems becoming malevolent or conscious and turning on us?

Answer: <blockquote>The problem isn't consciousness, but competence. You make machines that are incredibly competent at achieving objectives and they will cause accidents in trying to achieve those objectives.

- Stuart Russell </blockquote>

Work on AI alignment is not concerned with the question of whether "consciousness", "sentience" or "self-awareness" could arise in a machine or an algorithm. Unlike the frequently-referenced plotline in the Terminator movies, the standard catastrophic misalignment scenarios under discussion do not require computers to become conscious; they only require conventional computer systems (although usually faster and more powerful ones than those available today) blindly and deterministically following logical steps, in the same way that they currently do.

The primary concern ("AI misalignment") is that powerful systems could inadvertently be programmed with goals that do not fully capture what the programmers actually want. The AI would then harm humanity in pursuit of goals which seemed benign or neutral. Nothing like malevolence or consciousness would need to be involved. A number of researchers studying the problem have concluded that it is surprisingly difficult to guard against this effect, and that it is likely to get much harder as the systems become more capable. AI systems are inevitably goal-directed and could, for example, consider our efforts to control them (or switch them off) as being impediments to attaining their goals.

Question: Is this about AI systems becoming malevolent or conscious and turning on us?

Answer:
<blockquote>The problem isn’t consciousness, but competence. You make machines that are incredibly competent at achieving objectives and they will cause accidents in trying to achieve those objectives.

- Stuart Russell </blockquote>

Work on AI alignment is not concerned with the question of whether “consciousness”, “sentience” or “self-awareness” could arise in a machine or an algorithm. Unlike the frequently-referenced plotline in the Terminator movies, the standard catastrophic misalignment scenarios under discussion do not require computers to become conscious; they only require conventional computer systems (although usually faster and more powerful ones than those available today) blindly and deterministically following logical steps, in the same way that they currently do.

The primary concern (“AI misalignment”) is that powerful systems could inadvertently be programmed with goals that do not fully capture what the programmers actually want. The AI would then harm humanity in pursuit of goals which seemed benign or neutral. Nothing like malevolence or consciousness would need to be involved. A number of researchers studying the problem have concluded that it is surprisingly difficult to guard against this effect, and that it is likely to get much harder as the systems become more capable. AI systems are inevitably goal-directed and could, for example, consider our efforts to control them (or switch them off) as being impediments to attaining their goals.

Plex's Answer to Is this about AI systems becoming malevolent or conscious and turning on us?


Question: What are some AI alignment research agendas currently being pursued?

Answer: Research at the [http://alignmentresearchcenter.org/ Alignment Research Center] is led by [https://paulfchristiano.com/ Paul Christiano], best known for introducing the [https://ai-alignment.com/iterated-distillation-and-amplification-157debfd1616 "Iterated Distillation and Amplification"] and [https://ai-alignment.com/humans-consulting-hch-f893f6051455 "Humans Consulting HCH"] approaches. He and his team are now ''"trying to figure out how to train ML systems to answer questions by straightforwardly 'translating' their beliefs into natural language rather than by reasoning about what a human wants to hear."''

[https://colah.github.io/about.html Chris Olah] (after work at [https://en.wikipedia.org/wiki/DeepMind DeepMind] and [https://en.wikipedia.org/wiki/OpenAI OpenAI]) recently launched [https://www.anthropic.com/ Anthropic], an AI lab focussed on the safety of large models. While his previous work was concerned with [https://80000hours.org/podcast/episodes/chris-olah-interpretability-research/ "transparency" and "interpretability" of large neural networks], especially vision models, Anthropic is focussing more on large language models, among other things working towards a ''"general-purpose, text-based assistant that is aligned with human values, meaning that it is helpful, honest, and harmless".''

[https://en.wikipedia.org/wiki/Stuart_J._Russell Stuart Russell] and his team at the [https://en.wikipedia.org/wiki/Center_for_Human-Compatible_Artificial_Intelligence Center for Human-Compatible Artificial Intelligence] (CHAI) have been working on [https://arxiv.org/abs/1806.06877 inverse reinforcement learning] (where the AI infers human values from observing human behavior) and [https://intelligence.org/files/CorrigibilityAISystems.pdf corrigibility], as well as attempts to disaggregate neural networks into "meaningful" subcomponents (see Filan, et al.'s [https://arxiv.org/abs/2103.03386 "Clusterability in neural networks"] and Hod et al.'s [https://openreview.net/forum?id꞊tFQyjbOz34 "Detecting modularity in deep neural networks]").

Alongside the more abstract [https://intelligence.org/files/TechnicalAgenda.pdf "agent foundations"] work they have become known for, [https://intelligence.org/ MIRI] recently announced their [https://www.lesswrong.com/posts/zRn6cLtxyNodudzhw/visible-thoughts-project-and-bounty-announcement "Visible Thoughts Project"] to test the hypothesis that ''"Language models can be made more understandable (and perhaps also more capable, though this is not the goal) by training them to produce visible thoughts."''

[https://en.wikipedia.org/wiki/OpenAI OpenAI] have recently been doing work on [https://openai.com/blog/summarizing-books/ iteratively summarizing books] (summarizing, and then summarizing the summary, etc.) as a method for scaling human oversight.

Stuart Armstrong's recently launched [https://buildaligned.ai/ AlignedAI] are mainly working on [https://www.alignmentforum.org/s/u9uawicHx7Ng7vwxA concept extrapolation] from familiar to novel contexts, something he believes is "necessary and almost sufficient" for AI alignment.

[https://www.redwoodresearch.org/ Redwood Research] (Buck Shlegeris, et al.) are trying to "handicap' GPT-3 to only produce non-violent completions of text prompts. ''"The idea is that there are many reasons we might ultimately want to apply some oversight function to an AI model, like 'don't be deceitful', and if we want to get AI teams to apply this we need to be able to incorporate these oversight predicates into the original model in an efficient manner."''

Ought is an independent AI safety research organization led by Andreas Stuhlmüller and Jungwon Byun. They are researching methods for breaking up complex, hard-to-verify tasks into simpler, easier-to-verify tasks, with the aim of allowing us to maintain effective oversight over AIs.

Question: What are some AI alignment research agendas currently being pursued?

Answer:
Research at the ［http://alignmentresearchcenter.org/ Alignment Research Center］ is led by ［https://paulfchristiano.com/ Paul Christiano］, best known for introducing the ［https://ai-alignment.com/iterated-distillation-and-amplification-157debfd1616 “Iterated Distillation and Amplification”］ and ［https://ai-alignment.com/humans-consulting-hch-f893f6051455 “Humans Consulting HCH”］ approaches. He and his team are now ''“trying to figure out how to train ML systems to answer questions by straightforwardly ‘translating’ their beliefs into natural language rather than by reasoning about what a human wants to hear.”''

［https://colah.github.io/about.html Chris Olah］ (after work at ［https://en.wikipedia.org/wiki/DeepMind DeepMind］ and ［https://en.wikipedia.org/wiki/OpenAI OpenAI］) recently launched ［https://www.anthropic.com/ Anthropic］, an AI lab focussed on the safety of large models. While his previous work was concerned with ［https://80000hours.org/podcast/episodes/chris-olah-interpretability-research/ “transparency” and “interpretability” of large neural networks］, especially vision models, Anthropic is focussing more on large language models, among other things working towards a ''"general-purpose, text-based assistant that is aligned with human values, meaning that it is helpful, honest, and harmless".''

［https://en.wikipedia.org/wiki/Stuart_J._Russell Stuart Russell］ and his team at the ［https://en.wikipedia.org/wiki/Center_for_Human-Compatible_Artificial_Intelligence Center for Human-Compatible Artificial Intelligence］ (CHAI) have been working on ［https://arxiv.org/abs/1806.06877 inverse reinforcement learning］ (where the AI infers human values from observing human behavior) and ［https://intelligence.org/files/CorrigibilityAISystems.pdf corrigibility］, as well as attempts to disaggregate neural networks into “meaningful” subcomponents (see Filan, et al.’s ［https://arxiv.org/abs/2103.03386 “Clusterability in neural networks”］ and Hod et al.'s ［https://openreview.net/forum?id꞊tFQyjbOz34 “Detecting modularity in deep neural networks］”).

Alongside the more abstract ［https://intelligence.org/files/TechnicalAgenda.pdf “agent foundations”］ work they have become known for, ［https://intelligence.org/ MIRI］ recently announced their ［https://www.lesswrong.com/posts/zRn6cLtxyNodudzhw/visible-thoughts-project-and-bounty-announcement “Visible Thoughts Project”］ to test the hypothesis that ''“Language models can be made more understandable (and perhaps also more capable, though this is not the goal) by training them to produce visible thoughts.”''

［https://en.wikipedia.org/wiki/OpenAI OpenAI］ have recently been doing work on ［https://openai.com/blog/summarizing-books/ iteratively summarizing books］ (summarizing, and then summarizing the summary, etc.) as a method for scaling human oversight.

Stuart Armstrong’s recently launched ［https://buildaligned.ai/ AlignedAI］ are mainly working on ［https://www.alignmentforum.org/s/u9uawicHx7Ng7vwxA concept extrapolation］ from familiar to novel contexts, something he believes is “necessary and almost sufficient” for AI alignment.

［https://www.redwoodresearch.org/ Redwood Research］ (Buck Shlegeris, et al.) are trying to “handicap' GPT-3 to only produce non-violent completions of text prompts. ''“The idea is that there are many reasons we might ultimately want to apply some oversight function to an AI model, like ‘don't be deceitful’, and if we want to get AI teams to apply this we need to be able to incorporate these oversight predicates into the original model in an efficient manner.”''

Ought is an independent AI safety research organization led by Andreas Stuhlmüller and Jungwon Byun. They are researching methods for breaking up complex, hard-to-verify tasks into simpler, easier-to-verify tasks, with the aim of allowing us to maintain effective oversight over AIs.

Plex's Answer to What are some AI alignment research agendas currently being pursued?


Question: What are some objections  the importance of AI alignment?

Answer: [https://aisafety.com/author/soeren-elverlin/ Søren Elverlin] has compiled a list of counter-arguments and suggests dividing them into two kinds: weak and strong.

Weak counter-arguments point to problems with the "standard" arguments (as given in, e.g., Bostrom's [https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies ''Superintelligence'']), especially shaky models and assumptions that are too strong. These arguments are often of a substantial quality and are often presented by people who themselves worry about AI safety. Elverin calls these objections "weak" because they do not attempt to imply that the probability of a bad outcome is close to zero: ''"For example, even if you accept [https://www.alignmentforum.org/posts/vwLxd6hhFvPbvKmBH/yudkowsky-and-christiano-discuss-takeoff-speeds Paul Christiano's arguments against "fast takeoff"], they only drive the probability of this down to about 20%. Weak counter-arguments are interesting, but the decision to personally focus on AI safety doesn't strongly depend on the probability – anything above 5% is clearly a big enough deal that it doesn't make sense to work on other things."''

Strong arguments argue that the probability of existential catastrophe due to misaligned AI is tiny, usually by some combination of claiming that AGI is impossible or very far away. For example, [https://en.wikipedia.org/wiki/Michael_L._Littman Michael Littman] has [https://www.youtube.com/watch?v꞊c9AbECvRt20&t꞊1559s suggested] that as (he believes) we're so far from AGI, there will be a long period of human history wherein we'll have ample time to grow up alongside powerful AIs and figure out how to align them.

Elverlin opines that ''"There are few arguments that are both high-quality and strong enough to qualify as an 'objection to the importance of alignment'."'' He suggests [https://aiimpacts.org/conversation-with-rohin-shah/ Rohin Shah's arguments for "alignment by default"] as one of the better candidates.

[https://intelligence.org/ MIRI]'s April fools [https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy "Death With Dignity" strategy] might be seen as an argument against the importance of working on alignment, but only in the sense that we might have almost no hope of solving it. In the same category are the "something else will kill us first, so there's no point worrying about AI alignment" arguments.

Question: What are some objections  the importance of AI alignment?

Answer:
［https://aisafety.com/author/soeren-elverlin/ Søren Elverlin］ has compiled a list of counter-arguments and suggests dividing them into two kinds: weak and strong.

Weak counter-arguments point to problems with the "standard" arguments (as given in, e.g., Bostrom’s ［https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies ''Superintelligence''］), especially shaky models and assumptions that are too strong. These arguments are often of a substantial quality and are often presented by people who themselves worry about AI safety. Elverin calls these objections “weak” because they do not attempt to imply that the probability of a bad outcome is close to zero: ''“For example, even if you accept ［https://www.alignmentforum.org/posts/vwLxd6hhFvPbvKmBH/yudkowsky-and-christiano-discuss-takeoff-speeds Paul Christiano's arguments against “fast takeoff”］, they only drive the probability of this down to about 20%. Weak counter-arguments are interesting, but the decision to personally focus on AI safety doesn't strongly depend on the probability – anything above 5% is clearly a big enough deal that it doesn't make sense to work on other things.”''

Strong arguments argue that the probability of existential catastrophe due to misaligned AI is tiny, usually by some combination of claiming that AGI is impossible or very far away. For example, ［https://en.wikipedia.org/wiki/Michael_L._Littman Michael Littman］ has ［https://www.youtube.com/watch?v꞊c9AbECvRt20&t꞊1559s suggested］ that as (he believes) we’re so far from AGI, there will be a long period of human history wherein we’ll have ample time to grow up alongside powerful AIs and figure out how to align them.

Elverlin opines that ''“There are few arguments that are both high-quality and strong enough to qualify as an ‘objection to the importance of alignment’.”'' He suggests ［https://aiimpacts.org/conversation-with-rohin-shah/ Rohin Shah's arguments for “alignment by default”］ as one of the better candidates.

［https://intelligence.org/ MIRI］'s April fools ［https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy "Death With Dignity" strategy］ might be seen as an argument against the importance of working on alignment, but only in the sense that we might have almost no hope of solving it. In the same category are the “something else will kill us first, so there’s no point worrying about AI alignment” arguments.

Plex's Answer to What are some objections to the importance of AI alignment?


Question: How close do AI experts think we are  creating superintelligence?

Answer: Nobody knows for sure when we will have AGI, or if we'll ever get there. [https://www.cold-takes.com/where-ai-forecasting-stands-today/ Open Philanthropy CEO Holden Karnofsky has analyzed a selection of recent expert surveys on the matter, as well as taking into account findings of computational neuroscience, economic history, probabilistic methods and failures of previous AI timeline estimates]. This all led him to estimate that ''"there is more than a 10% chance we'll see transformative AI within 15 years (by 2036); a ~50% chance we'll see it within 40 years (by 2060); and a ~2/3 chance we'll see it this century (by 2100)."'' Karnofsky bemoans the lack of robust expert consensus on the matter and invites rebuttals to his claims in order to further the conversation. He compares AI forecasting to election forecasting (as opposed to academic political science) or market forecasting (as opposed to theoretical academics), thereby arguing that AI researchers may not be the "experts" we should trust in predicting AI timelines.

Opinions proliferate, but given experts' (and non-experts') poor track record at predicting progress in AI, many researchers tend to be fairly agnostic about when superintelligent AI will be invented.

UC-Berkeley AI professor [https://en.wikipedia.org/wiki/Stuart_J._Russell Stuart Russell] has given his best guess as "sometime in our children's lifetimes", while [https://en.wikipedia.org/wiki/Ray_Kurzweil Ray Kurzweil] (Google's Director of Engineering) predicts human level AI by 2029 and an intelligence explosion by 2045. [https://en.wikipedia.org/wiki/Eliezer_Yudkowsky Eliezer Yudkowsky] expects [https://www.econlib.org/archives/2017/01/my_end-of-the-w.html the end of the world], and [https://en.wikipedia.org/wiki/Elon_Musk Elon Musk] [https://twitter.com/elonmusk/status/1531328534169493506 expects AGI], before 2030.

If there's anything like a consensus answer at this stage, it would be something like: "highly uncertain, maybe not for over a hundred years, maybe in less than fifteen, with around the middle of the century looking fairly plausible".

Question: How close do AI experts think we are  creating superintelligence?

Answer:
Nobody knows for sure when we will have AGI, or if we’ll ever get there. ［https://www.cold-takes.com/where-ai-forecasting-stands-today/ Open Philanthropy CEO Holden Karnofsky has analyzed a selection of recent expert surveys on the matter, as well as taking into account findings of computational neuroscience, economic history, probabilistic methods and failures of previous AI timeline estimates］. This all led him to estimate that ''"there is more than a 10% chance we'll see transformative AI within 15 years (by 2036); a ~50% chance we'll see it within 40 years (by 2060); and a ~2/3 chance we'll see it this century (by 2100)."'' Karnofsky bemoans the lack of robust expert consensus on the matter and invites rebuttals to his claims in order to further the conversation. He compares AI forecasting to election forecasting (as opposed to academic political science) or market forecasting (as opposed to theoretical academics), thereby arguing that AI researchers may not be the "experts” we should trust in predicting AI timelines.

Opinions proliferate, but given experts’ (and non-experts’) poor track record at predicting progress in AI, many researchers tend to be fairly agnostic about when superintelligent AI will be invented.

UC-Berkeley AI professor ［https://en.wikipedia.org/wiki/Stuart_J._Russell Stuart Russell］ has given his best guess as “sometime in our children’s lifetimes”, while ［https://en.wikipedia.org/wiki/Ray_Kurzweil Ray Kurzweil］ (Google’s Director of Engineering) predicts human level AI by 2029 and an intelligence explosion by 2045. ［https://en.wikipedia.org/wiki/Eliezer_Yudkowsky Eliezer Yudkowsky］ expects ［https://www.econlib.org/archives/2017/01/my_end-of-the-w.html the end of the world］, and ［https://en.wikipedia.org/wiki/Elon_Musk Elon Musk］ ［https://twitter.com/elonmusk/status/1531328534169493506 expects AGI］, before 2030.

If there’s anything like a consensus answer at this stage, it would be something like: “highly uncertain, maybe not for over a hundred years, maybe in less than fifteen, with around the middle of the century looking fairly plausible”.

Plex's Answer to How close do AI experts think we are to creating superintelligence?


Why is AI alignment a hard problem?

Question: Why is AI alignment a hard problem?

Answer: One sense in which alignment is a hard problem is analogous to the reason rocket science is a hard problem. Relative to other engineering endeavors, rocket science had so many disasters because of the extreme stresses placed on various mechanical components and the narrow margins of safety required by stringent weight limits. A superintelligence would put vastly more "stress" on the software and hardware stack it is running on, which could cause many classes of failure which don't occur when you're working with subhuman systems.

Alignment is also hard like space probes are hard. With recursively self-improving systems, you won't be able to go back and edit the code later if there is a catastrophic failure because it will competently deceive and resist you.

''"You may have only one shot. If something goes wrong, the system might be too 'high' for you to reach up and suddenly fix it. You can build error recovery mechanisms into it; space probes are supposed to accept software updates. If something goes wrong in a way that precludes getting future updates, though, you're screwed. You have lost the space probe."''

Additionally, alignment is hard like cryptographic security. Cryptographers attempt to safeguard against "intelligent adversaries" who search for flaws in a system which they can exploit to break it. ''"Your code is not an intelligent adversary if everything goes right. If something goes wrong, it might try to defeat your safeguards…"'' And at the stage where it's trying to defeat your safeguards, your code may have achieved the capabilities of a vast and perfectly coordinated team of superhuman-level hackers! So if there is even the tiniest flaw in your design, you can be certain that it will be found and exploited. As with standard cybersecurity, "good under normal circumstances" is just not good enough – your system needs to be unbreakably robust.

''"AI alignment: treat it like a cryptographic rocket probe. This is about how difficult you would expect it to be to build something smarter than you that was nice – given that basic agent theory says they're not automatically nice – and not die. You would expect that intuitively to be hard."''  Eliezer Yudkowsky

Another immense challenge is the fact that we currently have no idea how to reliably instill AIs with human-friendly goals. ''Even if a consensus could be reached on a system of human values and morality'', it's entirely unclear how this could be fully and faithfully captured in code.

For a more in-depth view of this argument, see Yudkowsky's talk "AI Alignment: Why It's Hard, and Where to Start" below (full transcript [https://intelligence.org/2016/12/28/ai-alignment-why-its-hard-and-where-to-start/ here]). For alternative views, see Paul Christiano's [https://ai-alignment.com/ai-alignment-landscape-d3773c37ae38 "AI alignment landscape" talk], Daniel Kokotajlo and Wei Dai's [https://www.alignmentforum.org/posts/WXvt8bxYnwBYpy9oT/the-main-sources-of-ai-risk "The Main Sources of AI Risk?"] list, and [https://aiimpacts.org/conversation-with-rohin-shah/ Rohin Shah's much more optimistic position].

(youtube)EUjc1WuyPT8(/youtube)

Question: Why is AI alignment a hard problem?

Answer:
One sense in which alignment is a hard problem is analogous to the reason rocket science is a hard problem. Relative to other engineering endeavors, rocket science had so many disasters because of the extreme stresses placed on various mechanical components and the narrow margins of safety required by stringent weight limits. A superintelligence would put vastly more “stress” on the software and hardware stack it is running on, which could cause many classes of failure which don’t occur when you’re working with subhuman systems.

Alignment is also hard like space probes are hard. With recursively self-improving systems, you won’t be able to go back and edit the code later if there is a catastrophic failure because it will competently deceive and resist you.

''"You may have only one shot. If something goes wrong, the system might be too 'high' for you to reach up and suddenly fix it. You can build error recovery mechanisms into it; space probes are supposed to accept software updates. If something goes wrong in a way that precludes getting future updates, though, you’re screwed. You have lost the space probe."''

Additionally, alignment is hard like cryptographic security. Cryptographers attempt to safeguard against “intelligent adversaries” who search for flaws in a system which they can exploit to break it. ''“Your code is not an intelligent adversary if everything goes right. If something goes wrong, it might try to defeat your safeguards…”'' And at the stage where it’s trying to defeat your safeguards, your code may have achieved the capabilities of a vast and perfectly coordinated team of superhuman-level hackers! So if there is even the tiniest flaw in your design, you can be certain that it will be found and exploited. As with standard cybersecurity, "good under normal circumstances" is just not good enough – your system needs to be unbreakably robust.

''"AI alignment: treat it like a cryptographic rocket probe. This is about how difficult you would expect it to be to build something smarter than you that was nice – given that basic agent theory says they’re not automatically nice – and not die. You would expect that intuitively to be hard."''  Eliezer Yudkowsky

Another immense challenge is the fact that we currently have no idea how to reliably instill AIs with human-friendly goals. ''Even if a consensus could be reached on a system of human values and morality'', it’s entirely unclear how this could be fully and faithfully captured in code.

For a more in-depth view of this argument, see Yudkowsky's talk "AI Alignment: Why It’s Hard, and Where to Start" below (full transcript ［https://intelligence.org/2016/12/28/ai-alignment-why-its-hard-and-where-to-start/ here］). For alternative views, see Paul Christiano's ［https://ai-alignment.com/ai-alignment-landscape-d3773c37ae38 “AI alignment landscape” talk］, Daniel Kokotajlo and Wei Dai’s ［https://www.alignmentforum.org/posts/WXvt8bxYnwBYpy9oT/the-main-sources-of-ai-risk “The Main Sources of AI Risk?”］ list, and ［https://aiimpacts.org/conversation-with-rohin-shah/ Rohin Shah’s much more optimistic position］.

(youtube)EUjc1WuyPT8(/youtube)

Plex's Answer to Why is AI alignment a hard problem?


Question: What are some good books about AGI safety?

Answer: ''[https://brianchristian.org/the-alignment-problem/ The Alignment Problem]'' (2020) by Brian Christian is the most recent in-depth guide to the field.

The book which first made the case to the public is Nick Bostrom's ''[https://publicism.info/philosophy/superintelligence/ Superintelligence]'' (2014). It gives an excellent overview of the state of the field (as it was then) and makes a strong case for the subject being important, as well as exploring many fascinating adjacent topics. However, it does not cover newer developments, such as [[What are mesa-optimizers?┊mesa-optimizers]] or [[What are language models?┊language models]].

There's also ''[https://en.wikipedia.org/wiki/Human_Compatible Human Compatible]'' (2019) by Stuart Russell, which gives a more up-to-date review of developments, with an emphasis on the approaches that the Center for Human-Compatible AI are working on, such as cooperative inverse reinforcement learning. There's a good [https://slatestarcodex.com/2020/01/30/book-review-human-compatible/ review/summary on SlateStarCodex].

Although not limited to AI safety, ''[https://www.amazon.co.uk/Does-Not-Hate-You-Superintelligence/dp/<PHONE> The AI Does Not Hate You]'' (2020) is an entertaining and accessible outline of both the core issues and an exploration of some of the community and culture of the people working on it.

Various other books explore the issues in an informed way, such as [http://www.tobyord.com/ Toby Ord]'s ''[https://en.wikipedia.org/wiki/The_Precipice:_Existential_Risk_and_the_Future_of_Humanity The Precipice]'' (2020), [https://en.wikipedia.org/wiki/Max_Tegmark Max Tegmark]'s ''[https://en.wikipedia.org/wiki/Life_3.0 Life 3.0]'' (2017), [https://www.ynharari.com/ Yuval Noah Harari]'s ''[https://www.ynharari.com/book/homo-deus/ Homo Deus]'' (2016), [https://www.fhi.ox.ac.uk/team/stuart-armstrong/ Stuart Armstrong]'s ''[https://smarterthan.us/toc/ Smarter Than Us]'' (2014), and [http://lukeprog.com/ Luke Muehlhauser]'s ''[https://intelligenceexplosion.com/ Facing the Intelligence Explosion]'' (2013).

Question: What are some good books about AGI safety?

Answer:
''［https://brianchristian.org/the-alignment-problem/ The Alignment Problem］'' (2020) by Brian Christian is the most recent in-depth guide to the field.

The book which first made the case to the public is Nick Bostrom’s ''［https://publicism.info/philosophy/superintelligence/ Superintelligence］'' (2014). It gives an excellent overview of the state of the field (as it was then) and makes a strong case for the subject being important, as well as exploring many fascinating adjacent topics. However, it does not cover newer developments, such as ［［What are mesa-optimizers?┊mesa-optimizers］］ or ［［What are language models?┊language models］］.

There's also ''［https://en.wikipedia.org/wiki/Human_Compatible Human Compatible］'' (2019) by Stuart Russell, which gives a more up-to-date review of developments, with an emphasis on the approaches that the Center for Human-Compatible AI are working on, such as cooperative inverse reinforcement learning. There's a good ［https://slatestarcodex.com/2020/01/30/book-review-human-compatible/ review/summary on SlateStarCodex］.

Although not limited to AI safety, ''［https://www.amazon.co.uk/Does-Not-Hate-You-Superintelligence/dp/1474608795 The AI Does Not Hate You］'' (2020) is an entertaining and accessible outline of both the core issues and an exploration of some of the community and culture of the people working on it.

Various other books explore the issues in an informed way, such as ［http://www.tobyord.com/ Toby Ord］’s ''［https://en.wikipedia.org/wiki/The_Precipice:_Existential_Risk_and_the_Future_of_Humanity The Precipice］'' (2020), ［https://en.wikipedia.org/wiki/Max_Tegmark Max Tegmark］’s ''［https://en.wikipedia.org/wiki/Life_3.0 Life 3.0］'' (2017), ［https://www.ynharari.com/ Yuval Noah Harari］’s ''［https://www.ynharari.com/book/homo-deus/ Homo Deus］'' (2016), ［https://www.fhi.ox.ac.uk/team/stuart-armstrong/ Stuart Armstrong］’s ''［https://smarterthan.us/toc/ Smarter Than Us］'' (2014), and ［http://lukeprog.com/ Luke Muehlhauser］’s ''［https://intelligenceexplosion.com/ Facing the Intelligence Explosion］'' (2013).

Plex's Answer to What are some good books about AGI safety?