Question: What are the differences between "AI safety", "AGI safety", "AI alignment" and "AI existential safety"?

Answer: AI alignment is the research field focused on trying to give us the tools to align AIs to specific goals, such as human values. This is crucial when they are highly competent, as a misaligned superintelligence could be the end of human civilization.

AGI safety is the field trying to make sure that when we build Artificial General Intelligences they are safe and do not harm humanity. It overlaps with AI alignment strongly, in that misalignment of AI would be the main cause of unsafe behavior in AGIs, but also includes misuse and other governance issues.

AI existential safety is a slightly broader term than AGI safety, including AI risks which pose an existential threat without necessarily being as general as humans.

AI safety was originally used by the existential risk reduction movement for the work done to reduce the risks of misaligned superintelligence, but has also been adopted by researchers and others studying nearer term and less catastrophic risks from AI in recent years.

Question: What are the differences between “AI safety”, “AGI safety”, “AI alignment” and “AI existential safety”?

Answer:
AI alignment is the research field focused on trying to give us the tools to align AIs to specific goals, such as human values. This is crucial when they are highly competent, as a misaligned superintelligence could be the end of human civilization.

AGI safety is the field trying to make sure that when we build Artificial General Intelligences they are safe and do not harm humanity. It overlaps with AI alignment strongly, in that misalignment of AI would be the main cause of unsafe behavior in AGIs, but also includes misuse and other governance issues.

AI existential safety is a slightly broader term than AGI safety, including AI risks which pose an existential threat without necessarily being as general as humans.

AI safety was originally used by the existential risk reduction movement for the work done to reduce the risks of misaligned superintelligence, but has also been adopted by researchers and others studying nearer term and less catastrophic risks from AI in recent years.

Plex's Answer to What are the differences between “AI safety”, “AGI safety”, “AI alignment” and “AI existential safety”?


Question: How might we get from Artificial General Intelligence  a Superintelligent system?

Answer: Once a system is at least as capable as top human at AI research, it would tend to become the driver of its own development and initiate a process of recursive self-improvement known as the [[What is an "intelligence explosion"?┊intelligence explosion]], leading to an extremely powerful system. A general framing of this process is [https://www.openphilanthropy.org/ Open Philanthropy]'s [https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#:~:text꞊Process%20for%20Automating%20Scientific%20and%20Technological%20Advancement%2C%20or%20PASTA Process for Automating Scientific and Technological Advancement (PASTA)].

There is [https://sideways-view.com/2018/02/24/takeoff-speeds/ much] [https://astralcodexten.substack.com/p/yudkowsky-contra-christiano-on-ai debate] about whether there would be a notable period where the AI was partially driving its own development, with humans being gradually less and less important, or whether the transition to AI automated AI capability research would be sudden. However, the core idea that there is ''some'' threshold of capabilities beyond which a system would begin to rapidly ascend is hard to reasonably dispute, and is a significant consideration for developing alignment strategies.

Question: How might we get from Artificial General Intelligence  a Superintelligent system?

Answer:
Once a system is at least as capable as top human at AI research, it would tend to become the driver of its own development and initiate a process of recursive self-improvement known as the ［［What is an "intelligence explosion"?┊intelligence explosion］］, leading to an extremely powerful system. A general framing of this process is ［https://www.openphilanthropy.org/ Open Philanthropy］'s ［https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#:~:text꞊Process%20for%20Automating%20Scientific%20and%20Technological%20Advancement%2C%20or%20PASTA Process for Automating Scientific and Technological Advancement (PASTA)］.

There is ［https://sideways-view.com/2018/02/24/takeoff-speeds/ much］ ［https://astralcodexten.substack.com/p/yudkowsky-contra-christiano-on-ai debate］ about whether there would be a notable period where the AI was partially driving its own development, with humans being gradually less and less important, or whether the transition to AI automated AI capability research would be sudden. However, the core idea that there is ''some'' threshold of capabilities beyond which a system would begin to rapidly ascend is hard to reasonably dispute, and is a significant consideration for developing alignment strategies.

Plex's Answer to How might we get from Artificial General Intelligence to a Superintelligent system?


Question: Why would we only get one chance  align a superintelligence?

Answer: An AGI which has [[How might we get from Artificial General Intelligence to a Superintelligent system?┊recursively self-improved into a superintelligence]] would be capable of either resisting our attempts to modify incorrectly specified goals, or realizing it was still weaker than us and acting [[Would we know if an AGI was misaligned?┊deceptively aligned]] until it was highly sure it could win in a confrontation. AGI would likely prevent a human from shutting it down unless the AGI was designed to be [https://www.lesswrong.com/tag/corrigibility corrigible].  See [[Why can't we just turn the AI off if it starts to misbehave?]] for more information.

Question: Why would we only get one chance  align a superintelligence?

Answer: An AGI which has ［［How might we get from Artificial General Intelligence to a Superintelligent system?┊recursively self-improved into a superintelligence］］ would be capable of either resisting our attempts to modify incorrectly specified goals, or realizing it was still weaker than us and acting ［［Would we know if an AGI was misaligned?┊deceptively aligned］］ until it was highly sure it could win in a confrontation. AGI would likely prevent a human from shutting it down unless the AGI was designed to be ［https://www.lesswrong.com/tag/corrigibility corrigible］.  See ［［Why can't we just turn the AI off if it starts to misbehave?］］ for more information.

TJ6K's Answer to Why would we only get one chance to align a superintelligence?


Question: I'm interested in working on AI safety. What should I do?

Answer: AI Safety Support [https://calendly.com/aiss offers free calls] to advise people interested in a career in AI Safety, so that's a great place to start. We're working on creating a bunch of detailed information for Stampy to use, but in the meantime check out these resources:

* [https://www.eacambridge.org/agi-safety-fundamentals EA Cambridge AGI Safety Fundamentals curriculum]
* [https://80000hours.org/articles/ai-safety-syllabus/ 80,000 Hours AI safety syllabus]
* [https://docs.google.com/document/d/1RFo7_9JVmt0z8RPwUjB-mUMgCMoUQmsaj2CM5aHvxCw/edit Adam Gleave's Careers in Beneficial AI Research document]
* [https://rohinshah.com/faq-career-advice-for-ai-alignment-researchers/ Rohin Shah's FAQ on career advice for AI alignment researchers]
* [https://www.aisafetysupport.org/ AI Safety Support] has lots of other good resources, such as their [https://www.aisafetysupport.org/resources/lots-of-links links page], [https://www.google.com/url?q꞊https%3A%2F%2Fjoin.slack.com%2Ft%2Fai-alignment%2Fshared_invite%2Fzt-fkgwbd2b-kK50z~BbVclOZMM9UP44gw&sa꞊D&sntz꞊1&usg꞊AFQjCNEIiKykU7SJ9LhJBoE3FFaOFOhOSA slack], [https://www.aisafetysupport.org/newsletter newsletter], and  [https://www.aisafetysupport.org/events/online-events-calendar events calendar].
* [https://docs.google.com/spreadsheets/d/1JyxrfFFrzaQsS3AQ4qJ2aOLGj1aSkBaxkpZCqBX9BOY/edit#gid꞊0 Safety-aligned research training programs (under construction).]

Question: I'm interested in working on AI safety. What should I do?

Answer:
AI Safety Support ［https://calendly.com/aiss offers free calls］ to advise people interested in a career in AI Safety, so that's a great place to start. We're working on creating a bunch of detailed information for Stampy to use, but in the meantime check out these resources:

* ［https://www.eacambridge.org/agi-safety-fundamentals EA Cambridge AGI Safety Fundamentals curriculum］
* ［https://80000hours.org/articles/ai-safety-syllabus/ 80,000 Hours AI safety syllabus］
* ［https://docs.google.com/document/d/1RFo7_9JVmt0z8RPwUjB-mUMgCMoUQmsaj2CM5aHvxCw/edit Adam Gleave's Careers in Beneficial AI Research document］
* ［https://rohinshah.com/faq-career-advice-for-ai-alignment-researchers/ Rohin Shah's FAQ on career advice for AI alignment researchers］
* ［https://www.aisafetysupport.org/ AI Safety Support］ has lots of other good resources, such as their ［https://www.aisafetysupport.org/resources/lots-of-links links page］, ［https://www.google.com/url?q꞊https%3A%2F%2Fjoin.slack.com%2Ft%2Fai-alignment%2Fshared_invite%2Fzt-fkgwbd2b-kK50z~BbVclOZMM9UP44gw&sa꞊D&sntz꞊1&usg꞊AFQjCNEIiKykU7SJ9LhJBoE3FFaOFOhOSA slack］, ［https://www.aisafetysupport.org/newsletter newsletter］, and  ［https://www.aisafetysupport.org/events/online-events-calendar events calendar］.
* ［https://docs.google.com/spreadsheets/d/1JyxrfFFrzaQsS3AQ4qJ2aOLGj1aSkBaxkpZCqBX9BOY/edit#gid꞊0 Safety-aligned research training programs (under construction).］

Filip's Answer to I'm interested in working on AI safety. What should I do?


Question: Can you give an AI a goal which involves "minimally impacting the world"?

Answer: This is actually an active area of AI alignment research, called "Impact Measures"! It's not trivial to formalize in a way which won't predictably go wrong (entropy minimization likely leads to an AI which tries really hard to put out all the stars ASAP since they produce so much entropy, for example), but progress is being made. You can read about it on the [https://www.alignmentforum.org/tag/impact-measures Alignment Forum tag], or watch Rob's videos [http://youtu.be/lqJUIqZNzP8 Avoiding Negative Side Effects] and [http://youtu.be/S_Sd_S8jwP0 Avoiding Positive Side Effects]

Question: Can you give an AI a goal which involves “minimally impacting the world”?

Answer: This is actually an active area of AI alignment research, called "Impact Measures"! It's not trivial to formalize in a way which won't predictably go wrong (entropy minimization likely leads to an AI which tries really hard to put out all the stars ASAP since they produce so much entropy, for example), but progress is being made. You can read about it on the ［https://www.alignmentforum.org/tag/impact-measures Alignment Forum tag］, or watch Rob's videos ［http://youtu.be/lqJUIqZNzP8 Avoiding Negative Side Effects］ and ［http://youtu.be/S_Sd_S8jwP0 Avoiding Positive Side Effects］

Robertskmiles's Answer to Can you give an AI a goal which involves “minimally impacting the world”?


Question: Is it possible  block an AI from doing certain things on the Internet?

Answer: Once an AGI has access to the internet it would be very challenging to meaningfully restrict it from doing things online which it wants to. There are too many options to bypass blocks we may put in place.

It may be possible to design it so that it does not want to do dangerous things in the first place, or perhaps to set up tripwires so that we notice that it's trying to do a dangerous thing, though that relies on it not noticing or bypassing the tripwire so should not be the only layer of security.

Question: Is it possible  block an AI from doing certain things on the Internet?

Answer:
Once an AGI has access to the internet it would be very challenging to meaningfully restrict it from doing things online which it wants to. There are too many options to bypass blocks we may put in place.

It may be possible to design it so that it does not want to do dangerous things in the first place, or perhaps to set up tripwires so that we notice that it’s trying to do a dangerous thing, though that relies on it not noticing or bypassing the tripwire so should not be the only layer of security.

Plex's Answer to Is it possible to block an AI from doing certain things on the Internet?


Question: How quickly could an AI go from the first indications of problems  an unrecoverable disaster?

Answer: If the AI system was deceptively aligned (i.e. pretending to be nice until it was in control of the situation) or had been in stealth mode while getting things in place for a takeover, quite possibly within hours. We may get more warning with weaker systems, if the AGI does not feel at all threatened by us, or if a [https://www.lesswrong.com/posts/AyNHoTWWAJ5eb99ji/another-outer-alignment-failure-story complex ecosystem of AI systems is built over time and we gradually lose control].

Paul Christiano writes [https://www.lesswrong.com/posts/AyNHoTWWAJ5eb99ji/another-outer-alignment-failure-story a story of alignment failure] which shows a relatively fast transition.

Question: How quickly could an AI go from the first indications of problems  an unrecoverable disaster?

Answer:
If the AI system was deceptively aligned (i.e. pretending to be nice until it was in control of the situation) or had been in stealth mode while getting things in place for a takeover, quite possibly within hours. We may get more warning with weaker systems, if the AGI does not feel at all threatened by us, or if a ［https://www.lesswrong.com/posts/AyNHoTWWAJ5eb99ji/another-outer-alignment-failure-story complex ecosystem of AI systems is built over time and we gradually lose control］.

Paul Christiano writes ［https://www.lesswrong.com/posts/AyNHoTWWAJ5eb99ji/another-outer-alignment-failure-story a story of alignment failure］ which shows a relatively fast transition.

Plex's Answer to How quickly could an AI go from the first indications of problems to an unrecoverable disaster?


How doomed is humanity?

Question: How doomed is humanity?

Answer: The opinions from experts are all over the place, according to [https://www.lesswrong.com/posts/QvwSr5LsxyDeaPK5s/existential-risk-from-ai-survey-results this 2021 survey]. Someone has also collected a [https://docs.google.com/spreadsheets/d/1W10B6NJjicD8O0STPiT3tNV3oFnT8YsfjmtYR8RO_RI/edit#gid꞊0 database of existential risk estimates].

On the pessimistic end you find people like Eliezer Yudkowsky, [https://forum.effectivealtruism.org/posts/bGBm2yTiLEwwCbL6w/discussion-with-eliezer-yudkowsky-on-agi-interventions who said]: "I consider the present gameboard to look incredibly grim, and I don't actually see a way out through hard work alone. We can hope there's a miracle that violates some aspect of my background model, and we can try to prepare for that unknown miracle; preparing for an unknown miracle probably looks like "Trying to die with more dignity on the mainline" (because if you can die with more dignity on the mainline, you are better positioned to take advantage of a miracle if it occurs)."

While at the optimistic end you have people like Ben Garfinkel who put the probability at more like 0.1-1% for AI causing an existential catastrophe in the next century, with most people lying somewhere in the middle.

Question: How doomed is humanity?

Answer:
The opinions from experts are all over the place, according to ［https://www.lesswrong.com/posts/QvwSr5LsxyDeaPK5s/existential-risk-from-ai-survey-results this 2021 survey］. Someone has also collected a ［https://docs.google.com/spreadsheets/d/1W10B6NJjicD8O0STPiT3tNV3oFnT8YsfjmtYR8RO_RI/edit#gid꞊0 database of existential risk estimates］.

On the pessimistic end you find people like Eliezer Yudkowsky, ［https://forum.effectivealtruism.org/posts/bGBm2yTiLEwwCbL6w/discussion-with-eliezer-yudkowsky-on-agi-interventions who said］: "I consider the present gameboard to look incredibly grim, and I don't actually see a way out through hard work alone. We can hope there's a miracle that violates some aspect of my background model, and we can try to prepare for that unknown miracle; preparing for an unknown miracle probably looks like "Trying to die with more dignity on the mainline" (because if you can die with more dignity on the mainline, you are better positioned to take advantage of a miracle if it occurs)."

While at the optimistic end you have people like Ben Garfinkel who put the probability at more like 0.1-1% for AI causing an existential catastrophe in the next century, with most people lying somewhere in the middle.

Plex's Answer to How doomed is humanity?


Question: Why is the future of AI suddenly in the news? What has changed?

Answer: In previous decades, AI research had proceeded more slowly than some experts predicted. According to experts in the field, however, this trend has reversed in the past 5 years or so. AI researchers have been repeatedly surprised by, for example, the effectiveness of new visual and speech recognition systems. AI systems can solve CAPTCHAs that were specifically devised to foil AIs, translate spoken text on-the-fly, and teach themselves how to play games they have neither seen before nor been programmed to play. Moreover, the real-world value of this effectiveness has prompted massive investment by large tech firms such as Google, Facebook, and IBM, creating a positive feedback cycle that could dramatically speed progress.

Question: Why is the future of AI suddenly in the news? What has changed?

Answer: In previous decades, AI research had proceeded more slowly than some experts predicted. According to experts in the field, however, this trend has reversed in the past 5 years or so. AI researchers have been repeatedly surprised by, for example, the effectiveness of new visual and speech recognition systems. AI systems can solve CAPTCHAs that were specifically devised to foil AIs, translate spoken text on-the-fly, and teach themselves how to play games they have neither seen before nor been programmed to play. Moreover, the real-world value of this effectiveness has prompted massive investment by large tech firms such as Google, Facebook, and IBM, creating a positive feedback cycle that could dramatically speed progress.

Answer to Why is the future of AI suddenly in the news? What has changed?


Question: What is interpretability and what approaches are there?

Answer: Interpretability is about making machine learning (ML) systems easier to understand. It is hard because the computations of current ML systems often depend on billions of parameters which they learnt from data. Areas of research for making current ML models more understandable are ''mechanistic interpretability'', ''finding important input features'', ''explaining by examples'', ''natural language explanations'', and using ML architectures which are ''intrinsically interpretable''.

# '''Mechanistic interpretability''' is about interpreting an ML model's internal representations. A very simple way to do this is [https://towardsdatascience.com/every-ml-engineer-needs-to-know-neural-network-interpretability-afea2ac0824e activation maximization]: optimize the input such that one particular neuron is activated a lot. This optimized input is an indicator of the concept which the neuron represents. Work that is central for mechanistic interpretability is the [https://distill.pub/2020/circuits/zoom-in/ circuits thread], which focuses on interpreting the algorithms implemented by subgraphs (circuits) of neural networks. There is also work on [https://transformer-circuits.pub/2021/framework/index.html circuits in transformers] in particular. Mechanistic Interpretability has the drawback that [https://www.greaterwrong.com/posts/qXtbBAxmFkAQLQEJE/interpretability-tool-ness-alignment-corrigibility-are-not interpretability is not composable], i.e. even if we understand all the components of a system, it doesn't mean that we understand the whole. However, there may still be a way of hierarchically decomposing a system in a way that allows us to understand each layer of abstraction of it, and thus understanding the whole. <p style꞊"text-align:center">https://i.imgur.com/nGDvldz.png <br>  Feature visualization of a neuron that corresponds to dog-like features. [https://distill.pub/2020/circuits/zoom-in/ image source] </p>
# The idea of '''finding important input features''' is to find out which input features are most relevant for the output. In the case of image classification, we can highlight the relevant features with a heatmap, which is called [https://arxiv.org/abs/<PHONE> saliency map]). A very simple way to do this is to take the derivative of the output with regard to the different parts of the input. This derivative denotes how much the output changes if we change a particular part of the input, i.e. how important that part of the input is for the output. Saliency maps can be useful to notice cases in which an image classifier learns to use features it should not use. For example, the paper ''[https://www.nature.com/articles/s41467-019-08987-4 Unmasking Clever Hans predictors and assessing what machines really learn]'' used saliency maps to show that a horse-classifying image classifier was not using the image parts that contained the horse at all, but rather relied on the name of the photographer printed in a corner, because one of the photographers primarily took photos of horses. [image of horse thing, maybe see thesis] However, many of the common saliency methods fail basic [https://arxiv.org/abs/1810.03292 sanity checks], such as the saliency maps almost not changing when the model weights are randomized. Therefore, saliency maps are not sufficient for a reliable understanding of ML systems.
# '''Explanation by examples''' means showing examples in the training data that have similar features, such as in the paper ''[https://arxiv.org/abs/1806.10574 This Looks Like That: Deep Learning for Interpretable Image Recognition]''.
# '''Natural Language Explanations''' are sentences describing a model's reasons for its outputs. For example, in the paper ''[https://xfgao.github.io/xCookingWeb/ Joint Mind Modeling for Explanation Generation in Complex Human-Robot Collaborative Tasks]'' a human and an AI play a virtual cooking game together, and the AI explains its plans in natural language. They find that with the explanations the human-AI team performs significantly better.
# However, natural language explanations, as well as finding important features and explanation by examples are ''post-hoc'' explanations: They are generated after the fact, and are therefore likely to not be ''faithful'' (i.e. not accurately describe a model's decision process). '''Interpretable architectures''' are architectures which are simple enough to be understandable without additional tools. Cynthia Rudin is a central researcher [https://arxiv.org/abs/1811.10154 arguing for using interpretable architectures] in high-stakes situations. However, using interpretable architecutures usually comes with a significant cost to model performance.

You can read more about different approaches in [https://www.alignmentforum.org/posts/GEPX7jgLMB8vR2qaK/opinions-on-interpretable-machine-learning-and-70-summaries this overview article] which summarizes more than 70 interpretability-related papers, and in the free online book ''[https://christophm.github.io/interpretable-ml-book/ A Guide for Making Black Box Models Explainable]''.

Question: What is interpretability and what approaches are there?

Answer:
Interpretability is about making machine learning (ML) systems easier to understand. It is hard because the computations of current ML systems often depend on billions of parameters which they learnt from data. Areas of research for making current ML models more understandable are ''mechanistic interpretability'', ''finding important input features'', ''explaining by examples'', ''natural language explanations'', and using ML architectures which are ''intrinsically interpretable''.

# '''Mechanistic interpretability''' is about interpreting an ML model’s internal representations. A very simple way to do this is ［https://towardsdatascience.com/every-ml-engineer-needs-to-know-neural-network-interpretability-afea2ac0824e activation maximization］: optimize the input such that one particular neuron is activated a lot. This optimized input is an indicator of the concept which the neuron represents. Work that is central for mechanistic interpretability is the ［https://distill.pub/2020/circuits/zoom-in/ circuits thread］, which focuses on interpreting the algorithms implemented by subgraphs (circuits) of neural networks. There is also work on ［https://transformer-circuits.pub/2021/framework/index.html circuits in transformers］ in particular. Mechanistic Interpretability has the drawback that ［https://www.greaterwrong.com/posts/qXtbBAxmFkAQLQEJE/interpretability-tool-ness-alignment-corrigibility-are-not interpretability is not composable］, i.e. even if we understand all the components of a system, it doesn’t mean that we understand the whole. However, there may still be a way of hierarchically decomposing a system in a way that allows us to understand each layer of abstraction of it, and thus understanding the whole. <p style꞊"text-align:center">https://i.imgur.com/nGDvldz.png <br>  Feature visualization of a neuron that corresponds to dog-like features. ［https://distill.pub/2020/circuits/zoom-in/ image source］ </p>
# The idea of '''finding important input features''' is to find out which input features are most relevant for the output. In the case of image classification, we can highlight the relevant features with a heatmap, which is called ［https://arxiv.org/abs/1312.6034 saliency map］). A very simple way to do this is to take the derivative of the output with regard to the different parts of the input. This derivative denotes how much the output changes if we change a particular part of the input, i.e. how important that part of the input is for the output. Saliency maps can be useful to notice cases in which an image classifier learns to use features it should not use. For example, the paper ''［https://www.nature.com/articles/s41467-019-08987-4 Unmasking Clever Hans predictors and assessing what machines really learn］'' used saliency maps to show that a horse-classifying image classifier was not using the image parts that contained the horse at all, but rather relied on the name of the photographer printed in a corner, because one of the photographers primarily took photos of horses. ［image of horse thing, maybe see thesis］ However, many of the common saliency methods fail basic ［https://arxiv.org/abs/1810.03292 sanity checks］, such as the saliency maps almost not changing when the model weights are randomized. Therefore, saliency maps are not sufficient for a reliable understanding of ML systems.
# '''Explanation by examples''' means showing examples in the training data that have similar features, such as in the paper ''［https://arxiv.org/abs/1806.10574 This Looks Like That: Deep Learning for Interpretable Image Recognition］''.
# '''Natural Language Explanations''' are sentences describing a model’s reasons for its outputs. For example, in the paper ''［https://xfgao.github.io/xCookingWeb/ Joint Mind Modeling for Explanation Generation in Complex Human-Robot Collaborative Tasks］'' a human and an AI play a virtual cooking game together, and the AI explains its plans in natural language. They find that with the explanations the human-AI team performs significantly better.
# However, natural language explanations, as well as finding important features and explanation by examples are ''post-hoc'' explanations: They are generated after the fact, and are therefore likely to not be ''faithful'' (i.e. not accurately describe a model’s decision process). '''Interpretable architectures''' are architectures which are simple enough to be understandable without additional tools. Cynthia Rudin is a central researcher ［https://arxiv.org/abs/1811.10154 arguing for using interpretable architectures］ in high-stakes situations. However, using interpretable architecutures usually comes with a significant cost to model performance.

You can read more about different approaches in ［https://www.alignmentforum.org/posts/GEPX7jgLMB8vR2qaK/opinions-on-interpretable-machine-learning-and-70-summaries this overview article］ which summarizes more than 70 interpretability-related papers, and in the free online book ''［https://christophm.github.io/interpretable-ml-book/ A Guide for Making Black Box Models Explainable］''.

Magdalena's Answer to What is interpretability and what approaches are there?


Question: What would a good future with AGI look like?

Answer: As technology continues to improve, one thing is certain: the future is going to look like science fiction. Doubly so once superhuman AI ("[https://en.wikipedia.org/wiki/Artificial_general_intelligence AGI]") is invented, because we can expect the AGI to produce technological improvements at a superhuman rate, eventually approaching the physical limits in terms of how small machines can be miniaturized, how fast they can compute, how energy-efficient they can be, etc.

Today's world is lacking in many ways, so given these increasingly powerful tools, it seems likely that whoever controls those tools will use them to make increasingly large (and increasingly sci-fi-sounding) improvements to the world. If (and that's a big if!) humanity retains control of the AGI, we could use these amazing technologies to stop climate change, colonize other planets, solve world hunger, cure cancer and every other disease, even eliminate aging and death.

For more inspiration, here are some stories painting what a bright, AGI-powered future could look like:
* The winners of the [https://worldbuild.ai/ FHI Worldbuilding contest]
* [https://www.lesswrong.com/posts/Ybp6Wg6yy9DWRcBiR/the-adventure-a-new-utopia-story Stuart Armstrong's short story "The Adventure"]
* [https://en.wikipedia.org/wiki/Culture_series Iain M. Banks's Culture novels]

Question: What would a good future with AGI look like?

Answer:
As technology continues to improve, one thing is certain: the future is going to look like science fiction. Doubly so once superhuman AI ("［https://en.wikipedia.org/wiki/Artificial_general_intelligence AGI］") is invented, because we can expect the AGI to produce technological improvements at a superhuman rate, eventually approaching the physical limits in terms of how small machines can be miniaturized, how fast they can compute, how energy-efficient they can be, etc.

Today's world is lacking in many ways, so given these increasingly powerful tools, it seems likely that whoever controls those tools will use them to make increasingly large (and increasingly sci-fi-sounding) improvements to the world. If (and that's a big if!) humanity retains control of the AGI, we could use these amazing technologies to stop climate change, colonize other planets, solve world hunger, cure cancer and every other disease, even eliminate aging and death.

For more inspiration, here are some stories painting what a bright, AGI-powered future could look like:
* The winners of the ［https://worldbuild.ai/ FHI Worldbuilding contest］
* ［https://www.lesswrong.com/posts/Ybp6Wg6yy9DWRcBiR/the-adventure-a-new-utopia-story Stuart Armstrong's short story "The Adventure"］
* ［https://en.wikipedia.org/wiki/Culture_series Iain M. Banks's Culture novels］

Gelisam's Answer to What would a good future with AGI look like?


Question: What are "scaling laws" and how are they relevant  safety?

Answer: '''Scaling laws''' are observed trends on the performance of large machine learning models.

In the field of ML, better performance is usually achieved through better algorithms, better inputs, or using larger amounts of parameters, computing power, or data. Since the 2010s, advances in deep learning have shown experimentally that the easier and faster returns come from '''scaling''', an observation that has been described by Richard Sutton as the ''[http://www.incompleteideas.net/IncIdeas/BitterLesson.html bitter lesson]''.

While deep learning as a field has long struggled to scale models up while retaining learning capability (with such problems as [https://en.wikipedia.org/wiki/Catastrophic_interference catastrophic interference]), more recent methods, especially the Transformer model architecture, were able to ''just work'' by feeding them more data, and as the meme goes, [https://www.gwern.net/images/rl/2017-12-24-meme-nnlayers-alphagozero.jpg stacking more layers].

More surprisingly, performance (in terms of absolute likelihood loss, a standard measure) appeared to increase ''smoothly'' with compute, or dataset size, or parameter count. Which gave rise to '''scaling laws''', the trend lines suggested by performance gains, from which returns on data/compute/time investment could be extrapolated.

A companion to this purely descriptive law (no strong theoretical explanation of the phenomenon has been found yet), is the '''scaling hypothesis''', which [https://www.gwern.net/Scaling-hypothesis#scaling-hypothesis Gwern Branwen describes]:

The ''strong scaling hypothesis'' is that, once we find a scalable architecture like self-attention or convolutions, [...] we can simply train ever larger [neural networks] and ever more sophisticated behavior will emerge naturally as the easiest way to optimize for all the tasks & data.

The scaling laws, if the above hypothesis holds, become highly relevant to safety insofar capability gains become conceptually easier to achieve: no need for clever designs to solve a given task, just throw more processing at it and it will eventually yield. As [https://ai-alignment.com/prosaic-ai-control-b959644d79c2 Paul Christiano observes]:

It now seems possible that we could build "prosaic" AGI, which can replicate human behavior but doesn't involve qualitatively new ideas about "how intelligence works".

While the scaling laws still hold experimentally at the time of this writing (July 2022), whether they'll continue up to safety-relevant capabilities is still an open problem.

Question: What are "scaling laws" and how are they relevant  safety?

Answer:
'''Scaling laws''' are observed trends on the performance of large machine learning models.

In the field of ML, better performance is usually achieved through better algorithms, better inputs, or using larger amounts of parameters, computing power, or data. Since the 2010s, advances in deep learning have shown experimentally that the easier and faster returns come from '''scaling''', an observation that has been described by Richard Sutton as the ''［http://www.incompleteideas.net/IncIdeas/BitterLesson.html bitter lesson］''.

While deep learning as a field has long struggled to scale models up while retaining learning capability (with such problems as ［https://en.wikipedia.org/wiki/Catastrophic_interference catastrophic interference］), more recent methods, especially the Transformer model architecture, were able to ''just work'' by feeding them more data, and as the meme goes, ［https://www.gwern.net/images/rl/2017-12-24-meme-nnlayers-alphagozero.jpg stacking more layers］.

More surprisingly, performance (in terms of absolute likelihood loss, a standard measure) appeared to increase ''smoothly'' with compute, or dataset size, or parameter count. Which gave rise to '''scaling laws''', the trend lines suggested by performance gains, from which returns on data/compute/time investment could be extrapolated.

A companion to this purely descriptive law (no strong theoretical explanation of the phenomenon has been found yet), is the '''scaling hypothesis''', which ［https://www.gwern.net/Scaling-hypothesis#scaling-hypothesis Gwern Branwen describes］:

The ''strong scaling hypothesis'' is that, once we find a scalable architecture like self-attention or convolutions, ［...］ we can simply train ever larger ［neural networks］ and ever more sophisticated behavior will emerge naturally as the easiest way to optimize for all the tasks & data.

The scaling laws, if the above hypothesis holds, become highly relevant to safety insofar capability gains become conceptually easier to achieve: no need for clever designs to solve a given task, just throw more processing at it and it will eventually yield. As ［https://ai-alignment.com/prosaic-ai-control-b959644d79c2 Paul Christiano observes］:

It now seems possible that we could build “prosaic” AGI, which can replicate human behavior but doesn’t involve qualitatively new ideas about “how intelligence works”.

While the scaling laws still hold experimentally at the time of this writing (July 2022), whether they'll continue up to safety-relevant capabilities is still an open problem.

Jrmyp's Answer to What are "scaling laws" and how are they relevant to safety?


Plex's Answer to What is "HCH"?

What is "HCH"?

Question: What is "HCH"?

Answer:
Humans Consulting HCH (HCH) is a recursive acronym describing a setup where humans can consult simulations of themselves to help answer questions. It is a concept used in discussion of the [https://www.lesswrong.com/tag/iterated-amplification iterated amplification] proposal to solve the alignment problem.

It was first described by Paul Christiano in his post [https://www.lesswrong.com/posts/NXqs4nYXaq8q6dTTx/humans-consulting-hch Humans Consulting HCH]:

Consider a human Hugh who has access to a question-answering machine. Suppose the machine answers question Q by perfectly imitating how Hugh would answer question Q, if Hugh had access to the question-answering machine.

That is, Hugh is able to consult a copy of Hugh, who is able to consult a copy of Hugh, who is able to consult a copy of Hugh…

Let's call this process HCH, for "Humans Consulting HCH."

[https://www.lesswrong.com/tag/humans-consulting-hch?edit꞊true Edit on LessWrong]

https://www.lesswrong.com/tag/humans-consulting-hch?edit꞊true

Question: What is "HCH"?

Answer:
Humans Consulting HCH (HCH) is a recursive acronym describing a setup where humans can consult simulations of themselves to help answer questions. It is a concept used in discussion of the ［https://www.lesswrong.com/tag/iterated-amplification iterated amplification］ proposal to solve the alignment problem.

It was first described by Paul Christiano in his post ［https://www.lesswrong.com/posts/NXqs4nYXaq8q6dTTx/humans-consulting-hch Humans Consulting HCH］:

Consider a human Hugh who has access to a question-answering machine. Suppose the machine answers question Q by perfectly imitating how Hugh would answer question Q, if Hugh had access to the question-answering machine.

That is, Hugh is able to consult a copy of Hugh, who is able to consult a copy of Hugh, who is able to consult a copy of Hugh…

Let’s call this process HCH, for “Humans Consulting HCH.”

［https://www.lesswrong.com/tag/humans-consulting-hch?edit꞊true Edit on LessWrong］

https://www.lesswrong.com/tag/humans-consulting-hch?edit꞊true


Question: How might things go wrong with AI even without an agentic superintelligence?

Answer: Failures can happen with narrow non-agentic systems, mostly from humans not anticipating safety-relevant decisions made too quickly to react, much like in the [https://en.wikipedia.org/wiki/2010_flash_crash 2010 flash crash].

A helpful metaphor draws on self-driving cars. By relying more and more on an automated process to make decisions, people become worse drivers as they're not training themselves to react to the unexpected; then the unexpected happens, the software system itself reacts in an unsafe way and the human is too slow to regain control.

This generalizes to broader tasks. A human using a powerful system to make better decisions (say, as the CEO of a company) might not understand those very well, get trapped into an equilibrium without realizing it and essentially losing control over the entire process.

More detailed examples in this vein are described by Paul Christiano in ''[https://www.lesswrong.com/posts/HBxe6wdjxK239zajf/what-failure-looks-like What failure looks like]''.

Another source of failures is AI-mediated stable totalitarianism. The limiting factor in current pervasive surveillance, police and armed forces is manpower; the use of drones and other automated tools decreases the need for personnel to ensure security and extract resources.

As capabilities improve, political dissent could become impossible, checks and balances would break down as [https://www.youtube.com/watch?v꞊rStL7niR7gs a minimal number of key actors is needed to stay in power].

Question: How might things go wrong with AI even without an agentic superintelligence?

Answer:
Failures can happen with narrow non-agentic systems, mostly from humans not anticipating safety-relevant decisions made too quickly to react, much like in the ［https://en.wikipedia.org/wiki/2010_flash_crash 2010 flash crash］.

A helpful metaphor draws on self-driving cars. By relying more and more on an automated process to make decisions, people become worse drivers as they’re not training themselves to react to the unexpected; then the unexpected happens, the software system itself reacts in an unsafe way and the human is too slow to regain control.

This generalizes to broader tasks. A human using a powerful system to make better decisions (say, as the CEO of a company) might not understand those very well, get trapped into an equilibrium without realizing it and essentially losing control over the entire process.

More detailed examples in this vein are described by Paul Christiano in ''［https://www.lesswrong.com/posts/HBxe6wdjxK239zajf/what-failure-looks-like What failure looks like］''.

Another source of failures is AI-mediated stable totalitarianism. The limiting factor in current pervasive surveillance, police and armed forces is manpower; the use of drones and other automated tools decreases the need for personnel to ensure security and extract resources.

As capabilities improve, political dissent could become impossible, checks and balances would break down as ［https://www.youtube.com/watch?v꞊rStL7niR7gs a minimal number of key actors is needed to stay in power］.

Jrmyp's Answer to How might things go wrong with AI even without an agentic superintelligence?


Question: Why might we expect a superintelligence  be hostile by default?

Answer: The argument goes: computers only do what we command them; no more, no less. So it might be bad if terrorists or enemy countries develop superintelligence first. But if we develop superintelligence first there's no problem. Just command it to do the things we want, right?
Suppose we wanted a superintelligence to cure cancer. How might we specify the goal "cure cancer"? We couldn't guide it through every individual step; if we knew every individual step, then we could cure cancer ourselves. Instead, we would have to give it a final goal of curing cancer, and trust the superintelligence to come up with intermediate actions that furthered that goal. For example, a superintelligence might decide that the first step to curing cancer was learning more about protein folding, and set up some experiments to investigate protein folding patterns.

A superintelligence would also need some level of common sense to decide which of various strategies to pursue. Suppose that investigating protein folding was very likely to cure 50% of cancers, but investigating genetic engineering was moderately likely to cure 90% of cancers. Which should the AI pursue? Presumably it would need some way to balance considerations like curing as much cancer as possible, as quickly as possible, with as high a probability of success as possible.

But a goal specified in this way would be very dangerous. Humans instinctively balance thousands of different considerations in everything they do; so far this hypothetical AI is only balancing three (least cancer, quickest results, highest probability). To a human, it would seem maniacally, even psychopathically, obsessed with cancer curing. If this were truly its goal structure, it would go wrong in almost comical ways. This type of problem, [https://deepmind.com/blog/article/Specification-gaming-the-flip-side-of-AI-ingenuity specification gaming], has been observed in many AI systems.

If your only goal is "curing cancer", and you lack humans' instinct for the thousands of other important considerations, a relatively easy solution might be to hack into a nuclear base, launch all of its missiles, and kill everyone in the world. This satisfies all the AI's goals. It reduces cancer down to zero (which is better than medicines which work only some of the time). It's very fast (which is better than medicines which might take a long time to invent and distribute). And it has a high probability of success (medicines might or might not work; nukes definitely do).

So simple goal architectures are likely to go very wrong unless tempered by common sense and a broader understanding of what we do and do not value.

Even if we do train the AI on an actually desirable goal, there is also the risk of the AI actually learning a different and undesirable objective. This problem is called  [https://www.youtube.com/watch?v꞊bJLcIBixGj8 inner alignment].

Question: Why might we expect a superintelligence  be hostile by default?

Answer:
The argument goes: computers only do what we command them; no more, no less. So it might be bad if terrorists or enemy countries develop superintelligence first. But if we develop superintelligence first there’s no problem. Just command it to do the things we want, right?
Suppose we wanted a superintelligence to cure cancer. How might we specify the goal “cure cancer”? We couldn’t guide it through every individual step; if we knew every individual step, then we could cure cancer ourselves. Instead, we would have to give it a final goal of curing cancer, and trust the superintelligence to come up with intermediate actions that furthered that goal. For example, a superintelligence might decide that the first step to curing cancer was learning more about protein folding, and set up some experiments to investigate protein folding patterns.

A superintelligence would also need some level of common sense to decide which of various strategies to pursue. Suppose that investigating protein folding was very likely to cure 50% of cancers, but investigating genetic engineering was moderately likely to cure 90% of cancers. Which should the AI pursue? Presumably it would need some way to balance considerations like curing as much cancer as possible, as quickly as possible, with as high a probability of success as possible.

But a goal specified in this way would be very dangerous. Humans instinctively balance thousands of different considerations in everything they do; so far this hypothetical AI is only balancing three (least cancer, quickest results, highest probability). To a human, it would seem maniacally, even psychopathically, obsessed with cancer curing. If this were truly its goal structure, it would go wrong in almost comical ways. This type of problem, ［https://deepmind.com/blog/article/Specification-gaming-the-flip-side-of-AI-ingenuity specification gaming］, has been observed in many AI systems.

If your only goal is “curing cancer”, and you lack humans’ instinct for the thousands of other important considerations, a relatively easy solution might be to hack into a nuclear base, launch all of its missiles, and kill everyone in the world. This satisfies all the AI’s goals. It reduces cancer down to zero (which is better than medicines which work only some of the time). It’s very fast (which is better than medicines which might take a long time to invent and distribute). And it has a high probability of success (medicines might or might not work; nukes definitely do).

So simple goal architectures are likely to go very wrong unless tempered by common sense and a broader understanding of what we do and do not value.

Even if we do train the AI on an actually desirable goal, there is also the risk of the AI actually learning a different and undesirable objective. This problem is called  ［https://www.youtube.com/watch?v꞊bJLcIBixGj8 inner alignment］.

Answer to Why might we expect a superintelligence to be hostile by default?


How can I join the Stampy dev team?

Question: How can I join the Stampy dev team?

Answer: The development team works on [https://github.com/StampyAI multiple projects] in support of Stampy.  Currently, these projects include:

* [https://github.com/StampyAI/stampy-ui Stampy UI], which is made mostly in TypeScript.
* The [https://github.com/StampyAI/stampys_wiki Stampy Wiki], which is made mostly in PHP and JavaScript.
* The Stampy Bot, which is made in Python.

However, even if you don't specialize in any of these areas, do reach out if you would like to help.

To join, please contact our Project Manager, plex.  You can reach him on discord at plex#1874.  He will be able to point your skills in the right direction to help in the most effective way possible.

Question: How can I join the Stampy dev team?

Answer:
The development team works on ［https://github.com/StampyAI multiple projects］ in support of Stampy.  Currently, these projects include:

* ［https://github.com/StampyAI/stampy-ui Stampy UI］, which is made mostly in TypeScript.
* The ［https://github.com/StampyAI/stampys_wiki Stampy Wiki］, which is made mostly in PHP and JavaScript.
* The Stampy Bot, which is made in Python.

However, even if you don’t specialize in any of these areas, do reach out if you would like to help.

To join, please contact our Project Manager, plex.  You can reach him on discord at plex#1874.  He will be able to point your skills in the right direction to help in the most effective way possible.

Tayler6000's Answer to How can I join the Stampy dev team?


Question: What is artificial general intelligence safety / AI alignment?

Answer: ''AI alignment'' is a field that is focused on causing the goals of future [https://en.wikipedia.org/wiki/Superintelligence superintelligent artificial systems]
to align with [https://www.researchgate.net/publication/347891524_Literature_Review_What_AI_Safety_Researchers_Have_Written_About_the_Nature_of_Human_Values human values], meaning that they would behave in a way which was compatible with our survival and flourishing. This may be an [https://intelligence.org/2016/12/28/ai-alignment-why-its-hard-and-where-to-start/ extremely hard problem], especially with [https://www.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/ deep learning], and is likely to determine the outcome of the [https://www.cold-takes.com/all-possible-views-about-humanitys-future-are-wild/ most important century]. Alignment research is strongly interdisciplinary and can include computer science, mathematics, neuroscience, philosophy, and social sciences.

''AGI safety'' is a related concept which strongly overlaps with AI alignment. AGI safety is concerned with making sure that building AGI systems doesn't cause things to go badly wrong, and the main way in which things can go badly wrong is through misalignment. AGI safety includes policy work that prevents the building of dangerous AGI systems, or reduces misuse risks from AGI systems aligned to actors who don't have humanity's best interests in mind.

Question: What is artificial general intelligence safety / AI alignment?

Answer:
''AI alignment'' is a field that is focused on causing the goals of future ［https://en.wikipedia.org/wiki/Superintelligence superintelligent artificial systems］
to align with ［https://www.researchgate.net/publication/347891524_Literature_Review_What_AI_Safety_Researchers_Have_Written_About_the_Nature_of_Human_Values human values］, meaning that they would behave in a way which was compatible with our survival and flourishing. This may be an ［https://intelligence.org/2016/12/28/ai-alignment-why-its-hard-and-where-to-start/ extremely hard problem］, especially with ［https://www.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/ deep learning］, and is likely to determine the outcome of the ［https://www.cold-takes.com/all-possible-views-about-humanitys-future-are-wild/ most important century］. Alignment research is strongly interdisciplinary and can include computer science, mathematics, neuroscience, philosophy, and social sciences.

''AGI safety'' is a related concept which strongly overlaps with AI alignment. AGI safety is concerned with making sure that building AGI systems doesn’t cause things to go badly wrong, and the main way in which things can go badly wrong is through misalignment. AGI safety includes policy work that prevents the building of dangerous AGI systems, or reduces misuse risks from AGI systems aligned to actors who don’t have humanity’s best interests in mind.

Luca's Answer to What is artificial general intelligence safety / AI alignment?


Question: Would it improve the safety of quantilizers  cut off the top few percent of the distribution?

Answer: This is a really interesting question! Because, yeah it certainly seems to me that doing something like this would at least help, but it's not mentioned in the paper the video is based on. So I asked the author of the paper, and she said "It wouldn't improve the security guarantee in the paper, so it wasn't discussed.  Like, there's a plausible case that it's helpful, but nothing like a proof that it is".
To explain this I need to talk about something I gloss over in the video, which is that the quantilizer isn't really something you can actually build. The systems we study in AI Safety tend to fall somewhere on a spectrum from "real, practical AI system that is so messy and complex that it's hard to really think about or draw any solid conclusions from" on one end, to "mathematical formalism that we can prove beautiful theorems about but not actually build" on the other, and quantilizers are pretty far towards the 'mathematical' end. It's not practical to run an expected utility calculation on every possible action like that, for one thing. But, proving things about quantilizers gives us insight into how more practical AI systems may behave, or we may be able to build approximations of quantilizers, etc.
So it's like, if we built something that was quantilizer-like, using a sensible human utility function and a good choice of safe distribution, this idea would probably help make it safer. BUT you can't prove that mathematically, without making probably a lot of extra assumptions about the utility function and/or the action distribution. So it's a potentially good idea that's nonetheless hard to express within the framework in which the quantilizer exists.
TL;DR: This is likely a good idea! But can we prove it?

Question: Would it improve the safety of quantilizers  cut off the top few percent of the distribution?

Answer:
This is a really interesting question! Because, yeah it certainly seems to me that doing something like this would at least help, but it's not mentioned in the paper the video is based on. So I asked the author of the paper, and she said "It wouldn't improve the security guarantee in the paper, so it wasn't discussed.  Like, there's a plausible case that it's helpful, but nothing like a proof that it is".
To explain this I need to talk about something I gloss over in the video, which is that the quantilizer isn't really something you can actually build. The systems we study in AI Safety tend to fall somewhere on a spectrum from "real, practical AI system that is so messy and complex that it's hard to really think about or draw any solid conclusions from" on one end, to "mathematical formalism that we can prove beautiful theorems about but not actually build" on the other, and quantilizers are pretty far towards the 'mathematical' end. It's not practical to run an expected utility calculation on every possible action like that, for one thing. But, proving things about quantilizers gives us insight into how more practical AI systems may behave, or we may be able to build approximations of quantilizers, etc.
So it's like, if we built something that was quantilizer-like, using a sensible human utility function and a good choice of safe distribution, this idea would probably help make it safer. BUT you can't prove that mathematically, without making probably a lot of extra assumptions about the utility function and/or the action distribution. So it's a potentially good idea that's nonetheless hard to express within the framework in which the quantilizer exists.
TL;DR: This is likely a good idea! But can we prove it?

Robertskmiles's Answer to Would it improve the safety of quantilizers to cut off the top few percent of the distribution?


Why is AGI safety a hard problem?

Question: Why is AGI safety a hard problem?

Answer: There's the "we never figure out how to reliably instill AIs with human friendly goals" filter, which seems pretty challenging, especially with [https://www.youtube.com/watch?v꞊bJLcIBixGj8 inner alignment], solving morality in a way which is possible to code up, interpretability, etc.

There's the "race dynamics mean that even though we know how to build the thing safely the first group to cross the recursive self-improvement line ends up not implementing it safely" which is potentially made worse by the twin issues of "maybe robustly aligned AIs are much harder to build" and "maybe robustly aligned AIs are much less compute efficient".

There's the "we solved the previous problems but writing perfectly reliably code in a whole new domain is hard and there is some fatal bug which we don't find until too late" filter. The paper [https://arxiv.org/abs/1701.04739 The Pursuit of Exploitable Bugs in Machine Learning] explores this.

For a much more in depth analysis, see [https://ai-alignment.com/ai-alignment-landscape-d3773c37ae38 Paul Christiano's AI Alignment Landscape] talk and [https://www.alignmentforum.org/posts/WXvt8bxYnwBYpy9oT/the-main-sources-of-ai-risk The Main Sources of AI Risk?].

Question: Why is AGI safety a hard problem?

Answer:
There's the "we never figure out how to reliably instill AIs with human friendly goals" filter, which seems pretty challenging, especially with ［https://www.youtube.com/watch?v꞊bJLcIBixGj8 inner alignment］, solving morality in a way which is possible to code up, interpretability, etc.

There's the "race dynamics mean that even though we know how to build the thing safely the first group to cross the recursive self-improvement line ends up not implementing it safely" which is potentially made worse by the twin issues of "maybe robustly aligned AIs are much harder to build" and "maybe robustly aligned AIs are much less compute efficient".

There's the "we solved the previous problems but writing perfectly reliably code in a whole new domain is hard and there is some fatal bug which we don't find until too late" filter. The paper ［https://arxiv.org/abs/1701.04739 The Pursuit of Exploitable Bugs in Machine Learning］ explores this.

For a much more in depth analysis, see ［https://ai-alignment.com/ai-alignment-landscape-d3773c37ae38 Paul Christiano's AI Alignment Landscape］ talk and ［https://www.alignmentforum.org/posts/WXvt8bxYnwBYpy9oT/the-main-sources-of-ai-risk The Main Sources of AI Risk?］.

Plex's Answer to Why is AGI safety a hard problem?


Question: Why can't we just use Asimov's Three Laws of Robotics?

Answer: Isaac Asimov wrote those laws as a plot device for science fiction novels. Every story in the I, Robot series details a way that the laws can go wrong and be misinterpreted by robots. The laws are not a solution because they are an overly-simple set of natural language instructions that don't have clearly defined terms and don't factor in all edge-case scenarios.

Question: Why can’t we just use Asimov’s Three Laws of Robotics?

Answer: Isaac Asimov wrote those laws as a plot device for science fiction novels. Every story in the I, Robot series details a way that the laws can go wrong and be misinterpreted by robots. The laws are not a solution because they are an overly-simple set of natural language instructions that don’t have clearly defined terms and don’t factor in all edge-case scenarios.

Answer to Why can’t we just use Asimov’s Three Laws of Robotics?


Question: Why can't we just turn the AI off if it starts  misbehave?

Answer: We could shut down weaker systems, and this would be a useful guardrail against certain types of problem caused by narrow AI. However, once an AGI establishes itself, we could not unless it was [https://www.lesswrong.com/tag/corrigibility corrigible] and willing to let humans adjust it. There may be a period in the early stages of an AGI's development where it would be trying very hard to convince us that we should not shut it down and/or hiding itself and/or recursively self-improving and/or making copies of itself onto every server on earth.

Instrumental Convergence and the Stop Button Problem are the key reasons it would not be simple to shut down a non corrigible advanced system. If the AI wants to collect stamps, being turned off means it gets less stamps, so even without an explicit goal of not being turned off it has an instrumental reason to avoid being turned off (e.g. once it acquires a detailed world model and general intelligence, it is likely to realise that by playing nice and pretending to be aligned if you have the power to turn it off, establishing control over any system we put in place to shut it down, and eliminating us if it has the power to reliably do so and we would otherwise pose a threat).

(youtube)ZeecOKBus3Q(/youtube)
(youtube)3TYT1QfdfsM(/youtube)

Question: Why can't we just turn the AI off if it starts  misbehave?

Answer:
We could shut down weaker systems, and this would be a useful guardrail against certain types of problem caused by narrow AI. However, once an AGI establishes itself, we could not unless it was ［https://www.lesswrong.com/tag/corrigibility corrigible］ and willing to let humans adjust it. There may be a period in the early stages of an AGI's development where it would be trying very hard to convince us that we should not shut it down and/or hiding itself and/or recursively self-improving and/or making copies of itself onto every server on earth.

Instrumental Convergence and the Stop Button Problem are the key reasons it would not be simple to shut down a non corrigible advanced system. If the AI wants to collect stamps, being turned off means it gets less stamps, so even without an explicit goal of not being turned off it has an instrumental reason to avoid being turned off (e.g. once it acquires a detailed world model and general intelligence, it is likely to realise that by playing nice and pretending to be aligned if you have the power to turn it off, establishing control over any system we put in place to shut it down, and eliminating us if it has the power to reliably do so and we would otherwise pose a threat).

(youtube)ZeecOKBus3Q(/youtube)
(youtube)3TYT1QfdfsM(/youtube)

Plex's Answer to Why can't we just turn the AI off if it starts to misbehave?


Where can I learn about AI alignment?

Question: Where can I learn about AI alignment?

Answer: If you like interactive FAQs, you're in the right place already! Joking aside, some great entry points are the [https://www.youtube.com/watch?v꞊tlS5Y2vm02c&list꞊PLCRVRLd2RhZTpdUdEzJjo3qhmX3y3skWA&index꞊1 AI alignment playlist] on YouTube, "[https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html The Road to Superintelligence]" and "[https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html Our Immortality or Extinction]" posts on WaitBuyWhy for a fun, accessible introduction, and ''Vox's'' "[https://www.vox.com/future-perfect/2018/12/21/<PHONE>/ai-artificial-intelligence-machine-learning-safety-alignment The case for taking AI seriously as a threat to humanity]" as a high-quality mainstream explainer piece.

The free online [https://www.eacambridge.org/agi-safety-fundamentals Cambridge course on AGI Safety Fundamentals] provides a strong grounding in much of the field and a cohort + mentor to learn with.

There are many resources in this post on [https://forum.effectivealtruism.org/posts/S7dhJR5TDwPb5jypG/levelling-up-in-ai-safety-research-engineering Levelling Up in AI Safety Research Engineering] with a list of other guides at the bottom. There is also a [https://twitter.com/FreshMangoLassi/status/1575138148937498625 twitter thread] here with some programs for upskilling and some for safety-specific learning.

The [https://rohinshah.com/alignment-newsletter/ Alignment Newsletter] ([https://alignment-newsletter.libsyn.com/ podcast]), [https://www.alignmentforum.org/ Alignment Forum], and [https://www.reddit.com/r/ControlProblem/ AGI Control Problem Subreddit] are great for keeping up with latest developments.

Question: Where can I learn about AI alignment?

Answer:
If you like interactive FAQs, you're in the right place already! Joking aside, some great entry points are the ［https://www.youtube.com/watch?v꞊tlS5Y2vm02c&list꞊PLCRVRLd2RhZTpdUdEzJjo3qhmX3y3skWA&index꞊1 AI alignment playlist］ on YouTube, “［https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html The Road to Superintelligence］” and “［https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html Our Immortality or Extinction］” posts on WaitBuyWhy for a fun, accessible introduction, and ''Vox's'' “［https://www.vox.com/future-perfect/2018/12/21/18126576/ai-artificial-intelligence-machine-learning-safety-alignment The case for taking AI seriously as a threat to humanity］” as a high-quality mainstream explainer piece.

The free online ［https://www.eacambridge.org/agi-safety-fundamentals Cambridge course on AGI Safety Fundamentals］ provides a strong grounding in much of the field and a cohort + mentor to learn with.

There are many resources in this post on ［https://forum.effectivealtruism.org/posts/S7dhJR5TDwPb5jypG/levelling-up-in-ai-safety-research-engineering Levelling Up in AI Safety Research Engineering］ with a list of other guides at the bottom. There is also a ［https://twitter.com/FreshMangoLassi/status/1575138148937498625 twitter thread］ here with some programs for upskilling and some for safety-specific learning.

The ［https://rohinshah.com/alignment-newsletter/ Alignment Newsletter］ (［https://alignment-newsletter.libsyn.com/ podcast］), ［https://www.alignmentforum.org/ Alignment Forum］, and ［https://www.reddit.com/r/ControlProblem/ AGI Control Problem Subreddit］ are great for keeping up with latest developments.

Plex's Answer to Where can I learn about AI alignment?


Question: Where can I find people  talk  about AI alignment?

Answer: You can join:
*A local student or meetup [https://www.lesswrong.com/community LessWrong] or [https://forum.effectivealtruism.org/community Effective Altruism] group (or [https://www.effectivealtruism.org/groups start one]!)
*[https://www.eacambridge.org/agi-safety-fundamentals AGI Safety Fundamentals] which gives you a cohort to learn alongside and mentorship
*[https://discord.com/channels/677546901339504640 Rob Miles' Discord]
*[https://discord.com/invite/wz4MpRec4A Eleuther AI Discord]
*[https://ai-alignment.slack.com/join/shared_invite/zt-fkgwbd2b-kK50z~BbVclOZMM9UP44gw#/shared-invite/email AI Safety Slack]
*The relevant discussion threads on the [https://astralcodexten.substack.com/ ''Astral Codex Ten'' Substack], which sometimes discusses alignment.
*[https://discord.com/invite/RTKtdut ACX Discord]
*[https://www.reddit.com/r/slatestarcodex/ ACX Subreddit]

Or book free calls with [https://www.aisafetysupport.org/ AI Safety Support].

Question: Where can I find people  talk  about AI alignment?

Answer:
You can join:
*A local student or meetup ［https://www.lesswrong.com/community LessWrong］ or ［https://forum.effectivealtruism.org/community Effective Altruism］ group (or ［https://www.effectivealtruism.org/groups start one］!)
*［https://www.eacambridge.org/agi-safety-fundamentals AGI Safety Fundamentals］ which gives you a cohort to learn alongside and mentorship
*［https://discord.com/channels/677546901339504640 Rob Miles’ Discord］
*［https://discord.com/invite/wz4MpRec4A Eleuther AI Discord］
*［https://ai-alignment.slack.com/join/shared_invite/zt-fkgwbd2b-kK50z~BbVclOZMM9UP44gw#/shared-invite/email AI Safety Slack］
*The relevant discussion threads on the ［https://astralcodexten.substack.com/ ''Astral Codex Ten'' Substack］, which sometimes discusses alignment.
*［https://discord.com/invite/RTKtdut ACX Discord］
*［https://www.reddit.com/r/slatestarcodex/ ACX Subreddit］

Or book free calls with ［https://www.aisafetysupport.org/ AI Safety Support］.

Plex's Answer to Where can I find people to talk to about AI alignment?


What is the "long reflection"?

Question: What is the "long reflection"?

Answer:
The long reflection is a hypothesized period of time during which humanity works out how best to realize its long-term potential.

Some effective altruists, including [https://forum.effectivealtruism.org/tag/toby-ord Toby Ord] and [https://forum.effectivealtruism.org/tag/william-macaskill William MacAskill], have argued that, if humanity succeeds in eliminating [https://forum.effectivealtruism.org/tag/existential-risk existential risk] or reducing it to acceptable levels, it should not immediately embark on an ambitious and potentially irreversible project of arranging the [https://forum.effectivealtruism.org/tag/universe-s-resources universe's resources] in accordance to its values, but ought instead to spend considerable time— "centuries (or more)";(ref)
Ord, Toby (2020) [https://en.wikipedia.org/wiki/Special:BookSources/<PHONE> The Precipice: Existential Risk and the Future of Humanity], London: Bloomsbury Publishing.
(/ref) "perhaps tens of thousands of years";(ref)
Greaves, Hilary et al. (2019) [https://globalprioritiesinstitute.org/wp-content/uploads/2017/12/gpi-research-agenda.pdf A research agenda for the Global Priorities Institute], Oxford.
(/ref) "thousands or millions of years";(ref)
Dai, Wei (2019) [https://www.lesswrong.com/posts/w6d7XBCegc96kz4n3/the-argument-from-philosophical-difficulty The argument from philosophical difficulty], LessWrong, February 9.
(/ref) "[p]erhaps... a million years"(ref)
William MacAskill, in Perry, Lucas (2018) [https://futureoflife.org/2018/09/17/moral-uncertainty-and-the-path-to-ai-alignment-with-william-macaskill/ AI alignment podcast: moral uncertainty and the path to AI alignment with William MacAskill], AI Alignment podcast, September 17.
(/ref)—figuring out what is in fact of value. The long reflection may thus be seen as an intermediate stage in a rational long-term human developmental trajectory, following an initial stage of [https://forum.effectivealtruism.org/tag/existential-security existential security] when existential risk is drastically reduced and followed by a final stage when humanity's potential is fully realized.(ref)
Ord, Toby (2020) [https://en.wikipedia.org/wiki/Special:BookSources/<PHONE> The Precipice: Existential Risk and the Future of Humanity], London: Bloomsbury Publishing.
(/ref)

Criticism

The idea of a long reflection has been criticized on the grounds that virtually eliminating all existential risk will almost certainly require taking a variety of large-scale, irreversible decisions—related to [https://forum.effectivealtruism.org/tag/space-colonization space colonization], [https://forum.effectivealtruism.org/tag/global-governance global governance], [https://forum.effectivealtruism.org/tag/cognitive-enhancement cognitive enhancement], and so on—which are precisely the decisions meant to be discussed during the long reflection.(ref)
Stocker, Felix (2020) [https://www.felixstocker.com/blog/reflecting-on-the-long-reflection Reflecting on the long reflection], Felix Stocker's Blog, August 14.
(/ref)(ref)
Hanson, Robin (2021) [https://www.overcomingbias.com/2021/10/long-reflection-is-crazy-bad-idea.html 'Long reflection' is crazy bad idea], Overcoming Bias, October 20.(/ref) Since there are pervasive and inescapable tradeoffs between reducing existential risk and retaining moral option value, it may be argued that it does not make sense to frame humanity's long-term strategic picture as one consisting of two distinct stages, with one taking precedence over the other.

Further reading

Aird, Michael (2020) [https://forum.effectivealtruism.org/posts/H2zno3ggRJaph9P6c/quotes-about-the-long-reflection?commentId꞊z2ybSC353mPHpCjbn Collection of sources that are highly relevant to the idea of the Long Reflection], Effective Altruism Forum, June 20.
Many additional resources on this topic.

Wiblin, Robert & Keiran Harris (2018) [https://80000hours.org/podcast/episodes/will-macaskill-moral-philosophy/ Our descendants will probably see us as moral monsters. what should we do about that?], 80,000 Hours, January 19.
Interview with William MacAskill about the long reflection and other topics.

Related entries

[https://forum.effectivealtruism.org/tag/dystopia dystopia] ┊ [https://forum.effectivealtruism.org/tag/existential-risk existential risk] ┊ [https://forum.effectivealtruism.org/tag/existential-security existential security] ┊ [https://forum.effectivealtruism.org/tag/long-term-future long-term future] ┊ [https://forum.effectivealtruism.org/tag/longtermism longtermism] ┊ [https://forum.effectivealtruism.org/topics/longtermist-institutional-reform longtermist institutional reform] ┊ [https://forum.effectivealtruism.org/tag/moral-uncertainty moral uncertainty] ┊ [https://forum.effectivealtruism.org/tag/normative-ethics normative ethics] ┊ [https://forum.effectivealtruism.org/tag/value-lock-in value lock-in]

[https://forum.effectivealtruism.org/long-reflection?edit꞊true Edit]

https://forum.effectivealtruism.org/long-reflection?edit꞊true

Question: What is the "long reflection"?

Answer:
The long reflection is a hypothesized period of time during which humanity works out how best to realize its long-term potential.

Some effective altruists, including ［https://forum.effectivealtruism.org/tag/toby-ord Toby Ord］ and ［https://forum.effectivealtruism.org/tag/william-macaskill William MacAskill］, have argued that, if humanity succeeds in eliminating ［https://forum.effectivealtruism.org/tag/existential-risk existential risk］ or reducing it to acceptable levels, it should not immediately embark on an ambitious and potentially irreversible project of arranging the ［https://forum.effectivealtruism.org/tag/universe-s-resources universe's resources］ in accordance to its values, but ought instead to spend considerable time— "centuries (or more)";(ref)
Ord, Toby (2020) ［https://en.wikipedia.org/wiki/Special:BookSources/1526600218 The Precipice: Existential Risk and the Future of Humanity］, London: Bloomsbury Publishing.
(/ref) "perhaps tens of thousands of years";(ref)
Greaves, Hilary et al. (2019) ［https://globalprioritiesinstitute.org/wp-content/uploads/2017/12/gpi-research-agenda.pdf A research agenda for the Global Priorities Institute］, Oxford.
(/ref) "thousands or millions of years";(ref)
Dai, Wei (2019) ［https://www.lesswrong.com/posts/w6d7XBCegc96kz4n3/the-argument-from-philosophical-difficulty The argument from philosophical difficulty］, LessWrong, February 9.
(/ref) "［p］erhaps... a million years"(ref)
William MacAskill, in Perry, Lucas (2018) ［https://futureoflife.org/2018/09/17/moral-uncertainty-and-the-path-to-ai-alignment-with-william-macaskill/ AI alignment podcast: moral uncertainty and the path to AI alignment with William MacAskill］, AI Alignment podcast, September 17.
(/ref)—figuring out what is in fact of value. The long reflection may thus be seen as an intermediate stage in a rational long-term human developmental trajectory, following an initial stage of ［https://forum.effectivealtruism.org/tag/existential-security existential security］ when existential risk is drastically reduced and followed by a final stage when humanity's potential is fully realized.(ref)
Ord, Toby (2020) ［https://en.wikipedia.org/wiki/Special:BookSources/1526600218 The Precipice: Existential Risk and the Future of Humanity］, London: Bloomsbury Publishing.
(/ref)

Criticism

The idea of a long reflection has been criticized on the grounds that virtually eliminating all existential risk will almost certainly require taking a variety of large-scale, irreversible decisions—related to ［https://forum.effectivealtruism.org/tag/space-colonization space colonization］, ［https://forum.effectivealtruism.org/tag/global-governance global governance］, ［https://forum.effectivealtruism.org/tag/cognitive-enhancement cognitive enhancement］, and so on—which are precisely the decisions meant to be discussed during the long reflection.(ref)
Stocker, Felix (2020) ［https://www.felixstocker.com/blog/reflecting-on-the-long-reflection Reflecting on the long reflection］, Felix Stocker’s Blog, August 14.
(/ref)(ref)
Hanson, Robin (2021) ［https://www.overcomingbias.com/2021/10/long-reflection-is-crazy-bad-idea.html ‘Long reflection’ is crazy bad idea］, Overcoming Bias, October 20.(/ref) Since there are pervasive and inescapable tradeoffs between reducing existential risk and retaining moral option value, it may be argued that it does not make sense to frame humanity's long-term strategic picture as one consisting of two distinct stages, with one taking precedence over the other.

Further reading

Aird, Michael (2020) ［https://forum.effectivealtruism.org/posts/H2zno3ggRJaph9P6c/quotes-about-the-long-reflection?commentId꞊z2ybSC353mPHpCjbn Collection of sources that are highly relevant to the idea of the Long Reflection］, Effective Altruism Forum, June 20.
Many additional resources on this topic.

Wiblin, Robert & Keiran Harris (2018) ［https://80000hours.org/podcast/episodes/will-macaskill-moral-philosophy/ Our descendants will probably see us as moral monsters. what should we do about that?］, 80,000 Hours, January 19.
Interview with William MacAskill about the long reflection and other topics.

Related entries

［https://forum.effectivealtruism.org/tag/dystopia dystopia］ ┊ ［https://forum.effectivealtruism.org/tag/existential-risk existential risk］ ┊ ［https://forum.effectivealtruism.org/tag/existential-security existential security］ ┊ ［https://forum.effectivealtruism.org/tag/long-term-future long-term future］ ┊ ［https://forum.effectivealtruism.org/tag/longtermism longtermism］ ┊ ［https://forum.effectivealtruism.org/topics/longtermist-institutional-reform longtermist institutional reform］ ┊ ［https://forum.effectivealtruism.org/tag/moral-uncertainty moral uncertainty］ ┊ ［https://forum.effectivealtruism.org/tag/normative-ethics normative ethics］ ┊ ［https://forum.effectivealtruism.org/tag/value-lock-in value lock-in］

［https://forum.effectivealtruism.org/long-reflection?edit꞊true Edit］

https://forum.effectivealtruism.org/long-reflection?edit꞊true

Linnea's Answer to What is the "long reflection"?


What is meant by "AI takeoff"?

Question: What is meant by "AI takeoff"?

Answer:
AI Takeoff refers to the process of an [https://www.lesswrong.com/tag/artificial-general-intelligence Artificial General Intelligence] going from a certain threshold of capability (often discussed as "human-level") to being super-intelligent and capable enough to control the fate of civilization. There has been much debate about whether AI takeoff is more likely to be slow vs fast, i.e., "soft" vs "hard".

See also: [https://www.lesswrong.com/tag/ai-timelines AI Timelines], [https://www.lesswrong.com/tag/seed-ai Seed AI], [https://www.lesswrong.com/tag/singularity Singularity], [https://www.lesswrong.com/tag/intelligence-explosion Intelligence explosion], [https://www.lesswrong.com/tag/recursive-self-improvement Recursive self-improvement]

AI takeoff is sometimes casually referred to as AI FOOM.

Soft takeoff

A soft takeoff refers to an AGI that would self-improve over a period of years or decades. This could be due to either the learning algorithm being too demanding for the hardware or because the AI relies on experiencing feedback from the real-world that would have to be played out in real-time. Possible methods that could deliver a soft takeoff, by slowly building on human-level intelligence, are [https://www.lesswrong.com/tag/whole-brain-emulation Whole brain emulation], [https://www.lesswrong.com/tag/nootropics-and-other-cognitive-enhancement Biological Cognitive Enhancement], and software-based strong AGI [[https://www.lesswrong.com/tag/ai-takeoff?revision꞊0.0.24&lw_source꞊import_sheet#fn1 1]]. By maintaining control of the AGI's ascent it should be easier for a [https://wiki.lesswrong.com/wiki/Friendly_AI Friendly AI] to emerge.

Vernor Vinge, Hans Moravec and have all expressed the view that soft takeoff is preferable to a hard takeoff as it would be both safer and easier to engineer.

Hard takeoff

A hard takeoff (or an AI going "FOOM" [[https://www.lesswrong.com/tag/ai-takeoff?revision꞊0.0.24&lw_source꞊import_sheet#fn2 2]]) refers to AGI expansion in a matter of minutes, days, or months. It is a fast, abruptly, local increase in capability. This scenario is widely considered much more precarious, as this involves an AGI rapidly ascending in power without human control. This may result in unexpected or undesired behavior (i.e. [https://wiki.lesswrong.com/wiki/Unfriendly_AI Unfriendly AI]). It is one of the main ideas supporting the [https://www.lesswrong.com/tag/intelligence-explosion Intelligence explosion] hypothesis.

The feasibility of hard takeoff has been addressed by Hugo de Garis, [https://www.lesswrong.com/tag/eliezer-yudkowsky Eliezer Yudkowsky], [https://www.lesswrong.com/tag/ben-goertzel Ben Goertzel], [https://www.lesswrong.com/tag/nick-bostrom Nick Bostrom], and Michael Anissimov. It is widely agreed that a hard takeoff is something to be avoided due to the risks. Yudkowsky points out several possibilities that would make a hard takeoff more likely than a soft takeoff such as the existence of large [https://www.lesswrong.com/tag/computing-overhang resources overhangs] or the fact that small improvements seem to have a large impact in a mind's general intelligence (i.e.: the small genetic difference between humans and chimps lead to huge increases in capability) [[https://www.lesswrong.com/tag/ai-takeoff?revision꞊0.0.24&lw_source꞊import_sheet#fn3 3]].

Notable posts

[https://www.lesswrong.com/lw/wf/hard_takeoff/ Hard Takeoff] by Eliezer Yudkowsky

External links

[http://www.kurzweilai.net/the-age-of-virtuous-machines The Age of Virtuous Machines] by J. Storrs Hall President of The Foresight Institute

[http://multiverseaccordingtoben.blogspot.co.uk/2011/01/hard-takeoff-hypothesis.html Hard take off Hypothesis] by Ben Goertzel.

[http://www.acceleratingfuture.com/michael/blog/2011/05/hard-takeoff-sources/ Extensive archive of Hard takeoff Essays] from Accelerating Future

[http://www-rohan.sdsu.edu/faculty/vinge/misc/ac2005/ Can we avoid a hard take off?] by Vernor Vinge

[http://www.amazon.co.uk/Robot-Mere-Machine-Transcendent-Mind/dp/<PHONE> Robot: Mere Machine to Transcendent Mind] by Hans Moravec

[http://www.amazon.co.uk/The-Singularity-Near-Raymond-Kurzweil/dp/<PHONE>/ref꞊sr_1_1?s꞊books&ie꞊UTF8&qid꞊<PHONE>&sr꞊1-1 The Singularity is Near] by Ray Kurzweil

References

[http://www.aleph.se/andart/archives/2010/10/why_early_singularities_are_softer.html↩ http://www.aleph.se/andart/archives/2010/10/why_early_singularities_are_softer.html][https://www.lesswrong.com/tag/ai-takeoff?revision꞊0.0.24&lw_source꞊import_sheet#fnref1 ↩]

[http://lesswrong.com/lw/63t/requirements_for_ai_to_go_foom/↩ http://lesswrong.com/lw/63t/requirements_for_ai_to_go_foom/][https://www.lesswrong.com/tag/ai-takeoff?revision꞊0.0.24&lw_source꞊import_sheet#fnref2 ↩]

[https://www.lesswrong.com/lw/wf/hard_takeoff/ http://lesswrong.com/lw/wf/hard_takeoff/][http://lesswrong.com/lw/wf/hard_takeoff/↩ ↩]

[https://www.lesswrong.com/tag/ai-takeoff?edit꞊true Edit on LessWrong]

https://www.lesswrong.com/tag/ai-takeoff?edit꞊true

Question: What is meant by "AI takeoff"?

Answer:
AI Takeoff refers to the process of an ［https://www.lesswrong.com/tag/artificial-general-intelligence Artificial General Intelligence］ going from a certain threshold of capability (often discussed as "human-level") to being super-intelligent and capable enough to control the fate of civilization. There has been much debate about whether AI takeoff is more likely to be slow vs fast, i.e., "soft" vs "hard".

See also: ［https://www.lesswrong.com/tag/ai-timelines AI Timelines］, ［https://www.lesswrong.com/tag/seed-ai Seed AI］, ［https://www.lesswrong.com/tag/singularity Singularity］, ［https://www.lesswrong.com/tag/intelligence-explosion Intelligence explosion］, ［https://www.lesswrong.com/tag/recursive-self-improvement Recursive self-improvement］

AI takeoff is sometimes casually referred to as AI FOOM.

Soft takeoff

A soft takeoff refers to an AGI that would self-improve over a period of years or decades. This could be due to either the learning algorithm being too demanding for the hardware or because the AI relies on experiencing feedback from the real-world that would have to be played out in real-time. Possible methods that could deliver a soft takeoff, by slowly building on human-level intelligence, are ［https://www.lesswrong.com/tag/whole-brain-emulation Whole brain emulation］, ［https://www.lesswrong.com/tag/nootropics-and-other-cognitive-enhancement Biological Cognitive Enhancement］, and software-based strong AGI ［［https://www.lesswrong.com/tag/ai-takeoff?revision꞊0.0.24&lw_source꞊import_sheet#fn1 1］］. By maintaining control of the AGI's ascent it should be easier for a ［https://wiki.lesswrong.com/wiki/Friendly_AI Friendly AI］ to emerge.

Vernor Vinge, Hans Moravec and have all expressed the view that soft takeoff is preferable to a hard takeoff as it would be both safer and easier to engineer.

Hard takeoff

A hard takeoff (or an AI going "FOOM" ［［https://www.lesswrong.com/tag/ai-takeoff?revision꞊0.0.24&lw_source꞊import_sheet#fn2 2］］) refers to AGI expansion in a matter of minutes, days, or months. It is a fast, abruptly, local increase in capability. This scenario is widely considered much more precarious, as this involves an AGI rapidly ascending in power without human control. This may result in unexpected or undesired behavior (i.e. ［https://wiki.lesswrong.com/wiki/Unfriendly_AI Unfriendly AI］). It is one of the main ideas supporting the ［https://www.lesswrong.com/tag/intelligence-explosion Intelligence explosion］ hypothesis.

The feasibility of hard takeoff has been addressed by Hugo de Garis, ［https://www.lesswrong.com/tag/eliezer-yudkowsky Eliezer Yudkowsky］, ［https://www.lesswrong.com/tag/ben-goertzel Ben Goertzel］, ［https://www.lesswrong.com/tag/nick-bostrom Nick Bostrom］, and Michael Anissimov. It is widely agreed that a hard takeoff is something to be avoided due to the risks. Yudkowsky points out several possibilities that would make a hard takeoff more likely than a soft takeoff such as the existence of large ［https://www.lesswrong.com/tag/computing-overhang resources overhangs］ or the fact that small improvements seem to have a large impact in a mind's general intelligence (i.e.: the small genetic difference between humans and chimps lead to huge increases in capability) ［［https://www.lesswrong.com/tag/ai-takeoff?revision꞊0.0.24&lw_source꞊import_sheet#fn3 3］］.

Notable posts

［https://www.lesswrong.com/lw/wf/hard_takeoff/ Hard Takeoff］ by Eliezer Yudkowsky

External links

［http://www.kurzweilai.net/the-age-of-virtuous-machines The Age of Virtuous Machines］ by J. Storrs Hall President of The Foresight Institute

［http://multiverseaccordingtoben.blogspot.co.uk/2011/01/hard-takeoff-hypothesis.html Hard take off Hypothesis］ by Ben Goertzel.

［http://www.acceleratingfuture.com/michael/blog/2011/05/hard-takeoff-sources/ Extensive archive of Hard takeoff Essays］ from Accelerating Future

［http://www-rohan.sdsu.edu/faculty/vinge/misc/ac2005/ Can we avoid a hard take off?］ by Vernor Vinge

［http://www.amazon.co.uk/Robot-Mere-Machine-Transcendent-Mind/dp/0195136306 Robot: Mere Machine to Transcendent Mind］ by Hans Moravec

［http://www.amazon.co.uk/The-Singularity-Near-Raymond-Kurzweil/dp/0715635611/ref꞊sr_1_1?s꞊books&ie꞊UTF8&qid꞊1339495098&sr꞊1-1 The Singularity is Near］ by Ray Kurzweil

References

［http://www.aleph.se/andart/archives/2010/10/why_early_singularities_are_softer.html↩ http://www.aleph.se/andart/archives/2010/10/why_early_singularities_are_softer.html］［https://www.lesswrong.com/tag/ai-takeoff?revision꞊0.0.24&lw_source꞊import_sheet#fnref1 ↩］

［http://lesswrong.com/lw/63t/requirements_for_ai_to_go_foom/↩ http://lesswrong.com/lw/63t/requirements_for_ai_to_go_foom/］［https://www.lesswrong.com/tag/ai-takeoff?revision꞊0.0.24&lw_source꞊import_sheet#fnref2 ↩］

［https://www.lesswrong.com/lw/wf/hard_takeoff/ http://lesswrong.com/lw/wf/hard_takeoff/］［http://lesswrong.com/lw/wf/hard_takeoff/↩ ↩］

［https://www.lesswrong.com/tag/ai-takeoff?edit꞊true Edit on LessWrong］

https://www.lesswrong.com/tag/ai-takeoff?edit꞊true

Linnea's Answer to What is meant by "AI takeoff"?


What are "human values"?

Question: What are "human values"?

Answer:
Human Values are the things we care about, and would want an aligned superintelligence to look after and support. It is suspected that true human values are [https://www.lesswrong.com/tag/complexity-of-value highly complex], and could be extrapolated into a wide variety of forms.

[https://www.lesswrong.com/tag/human-values?edit꞊true Edit on LessWrong]

https://www.lesswrong.com/tag/human-values?edit꞊true

Question: What are "human values"?

Answer:
Human Values are the things we care about, and would want an aligned superintelligence to look after and support. It is suspected that true human values are ［https://www.lesswrong.com/tag/complexity-of-value highly complex］, and could be extrapolated into a wide variety of forms.

［https://www.lesswrong.com/tag/human-values?edit꞊true Edit on LessWrong］

https://www.lesswrong.com/tag/human-values?edit꞊true

Linnea's Answer to What are "human values"?


Question: Superintelligence sounds like science fiction. Do people think about this in the real world?

Answer: Many of the people with the deepest understanding of artificial intelligence are concerned about the risks of unaligned superintelligence. In 2014, Google bought world-leading artificial intelligence startup [https://en.wikipedia.org/wiki/DeepMind DeepMind] for $400 million; DeepMind added the condition that Google promise to set up an AI Ethics Board. DeepMind cofounder Shane Legg has said in interviews that he believes superintelligent AI will be ''"something approaching absolute power"'' and ''"the number one risk for this century".''

[https://en.wikipedia.org/wiki/Stuart_J._Russell#Career_and_research Stuart Russell], Professor of Computer Science at Berkeley, author of the standard AI textbook, and world-famous AI expert, warns of ''"species-ending problems"'' and wants his field to pivot to make superintelligence-related risks a central concern. He went so far as to write [https://en.wikipedia.org/wiki/Human_Compatible Human Compatible], a book focused on bringing attention to the dangers of artificial intelligence and the need for more work to address them.

Many other science and technology leaders agree. Late astrophysicist [https://en.wikipedia.org/wiki/Stephen_Hawking#Future_of_humanity Stephen Hawking] said that superintelligence ''"could spell the end of the human race."'' Tech billionaire [https://en.wikipedia.org/wiki/Bill_Gates#Post-Microsoft Bill Gates] describes himself as ''"in the camp that is concerned about superintelligence…I don't understand why some people are not concerned".'' Oxford Professor [https://www.theguardian.com/technology/2016/jun/12/nick-bostrom-artificial-intelligence-machine Nick Bostrom], who has been studying AI risks for over 20 years, has said: ''"Superintelligence is a challenge for which we are not ready now and will not be ready for a long time."''

[https://en.wikipedia.org/wiki/Holden_Karnofsky Holden Karnofsky], the CEO of [https://www.openphilanthropy.org/ Open Philanthropy], has written a carefully reasoned account of why transformative artificial intelligence means that this might be [https://www.cold-takes.com/most-important-century/ the most important century].

Question: Superintelligence sounds like science fiction. Do people think about this in the real world?

Answer:
Many of the people with the deepest understanding of artificial intelligence are concerned about the risks of unaligned superintelligence. In 2014, Google bought world-leading artificial intelligence startup ［https://en.wikipedia.org/wiki/DeepMind DeepMind］ for $400 million; DeepMind added the condition that Google promise to set up an AI Ethics Board. DeepMind cofounder Shane Legg has said in interviews that he believes superintelligent AI will be ''“something approaching absolute power”'' and ''“the number one risk for this century”.''

［https://en.wikipedia.org/wiki/Stuart_J._Russell#Career_and_research Stuart Russell］, Professor of Computer Science at Berkeley, author of the standard AI textbook, and world-famous AI expert, warns of ''“species-ending problems”'' and wants his field to pivot to make superintelligence-related risks a central concern. He went so far as to write ［https://en.wikipedia.org/wiki/Human_Compatible Human Compatible］, a book focused on bringing attention to the dangers of artificial intelligence and the need for more work to address them.

Many other science and technology leaders agree. Late astrophysicist ［https://en.wikipedia.org/wiki/Stephen_Hawking#Future_of_humanity Stephen Hawking］ said that superintelligence ''“could spell the end of the human race.”'' Tech billionaire ［https://en.wikipedia.org/wiki/Bill_Gates#Post-Microsoft Bill Gates］ describes himself as ''“in the camp that is concerned about superintelligence…I don’t understand why some people are not concerned”.'' Oxford Professor ［https://www.theguardian.com/technology/2016/jun/12/nick-bostrom-artificial-intelligence-machine Nick Bostrom］, who has been studying AI risks for over 20 years, has said: ''“Superintelligence is a challenge for which we are not ready now and will not be ready for a long time.”''

［https://en.wikipedia.org/wiki/Holden_Karnofsky Holden Karnofsky］, the CEO of ［https://www.openphilanthropy.org/ Open Philanthropy］, has written a carefully reasoned account of why transformative artificial intelligence means that this might be ［https://www.cold-takes.com/most-important-century/ the most important century］.

Answer to Superintelligence sounds like science fiction. Do people think about this in the real world?


Question: Isn't AI just a tool like any other? Won't it just do what we tell it to?

Answer: It likely will – however, intelligence is, by many definitions, the ability to figure out how to accomplish goals. Even in today's advanced AI systems, the builders assign the goal but don't tell the AI exactly how to accomplish it, nor necessarily predict in detail how it will be done; indeed those systems often solve problems in creative, unpredictable ways. Thus the thing that makes such systems intelligent is precisely what can make them difficult to predict and control. They may therefore attain the goal we set them via means inconsistent with our preferences.

Question: Isn’t AI just a tool like any other? Won’t it just do what we tell it to?

Answer: It likely will – however, intelligence is, by many definitions, the ability to figure out how to accomplish goals. Even in today’s advanced AI systems, the builders assign the goal but don’t tell the AI exactly how to accomplish it, nor necessarily predict in detail how it will be done; indeed those systems often solve problems in creative, unpredictable ways. Thus the thing that makes such systems intelligent is precisely what can make them difficult to predict and control. They may therefore attain the goal we set them via means inconsistent with our preferences.

Answer to Isn’t AI just a tool like any other? Won’t it just do what we tell it to?


Is there a danger in anthropomorphizing AI’s and trying  understand them in human terms?

Question: Is there a danger in anthropomorphizing AI's and trying  understand them in human terms?

Answer: Using some human-related metaphors (e.g. what an AGI 'wants' or 'believes') is almost unavoidable, as our language is built around experiences with humans, but we should be aware that these may lead us astray.

Many paths to AGI would result in a mind very different from a human or animal, and it would be hard to predict in detail how it would act. We should not trust intuitions trained on humans to predict what an AGI or superintelligence would do. High fidelity Whole Brain Emulations are one exception, where we would expect the system to at least initially be fairly human, but it may diverge depending on its environment and what modifications are applied to it.

There has been some discussion about how language models trained on lots of human-written text seem likely to pick up human concepts and think in a somewhat human way, and how we could [https://www.lesswrong.com/posts/Nwgdq6kHke5LY692J/alignment-by-default use this to improve alignment].

Question: Is there a danger in anthropomorphizing AI’s and trying  understand them in human terms?

Answer:
Using some human-related metaphors (e.g. what an AGI ‘wants’ or ‘believes’) is almost unavoidable, as our language is built around experiences with humans, but we should be aware that these may lead us astray.

Many paths to AGI would result in a mind very different from a human or animal, and it would be hard to predict in detail how it would act. We should not trust intuitions trained on humans to predict what an AGI or superintelligence would do. High fidelity Whole Brain Emulations are one exception, where we would expect the system to at least initially be fairly human, but it may diverge depending on its environment and what modifications are applied to it.

There has been some discussion about how language models trained on lots of human-written text seem likely to pick up human concepts and think in a somewhat human way, and how we could ［https://www.lesswrong.com/posts/Nwgdq6kHke5LY692J/alignment-by-default use this to improve alignment］.

Plex's Answer to Is there a danger in anthropomorphizing AI’s and trying to understand them in human terms?