How is "intelligence" defined?

Question: How is "intelligence" defined?

Answer:
After reviewing extensive literature on the subject, Legg and Hutter(ref)
http://arxiv.org/pdf/<PHONE>.pdf
(/ref) summarizes the many possible valuable definitions in the informal statement "Intelligence measures an agent's ability to achieve goals in a wide range of environments." They then show this definition can be mathematically formalized given reasonable mathematical definitions of its terms. They use [https://lessestwrong.com/tag/solomonoff-induction Solomonoff induction] - a formalization of [https://lessestwrong.com/tag/occam-s-razor Occam's razor] - to construct an [https://lessestwrong.com/tag/aixi universal artificial intelligence] with a embedded [https://lessestwrong.com/tag/utility-functions utility function] which assigns less [https://lessestwrong.com/tag/expected-utility utility] to those actions based on theories with higher [https://wiki.lesswrong.com/wiki/Kolmogorov_complexity complexity]. They argue this final formalization is a valid, meaningful, informative, general, unbiased, fundamental, objective, universal and practical definition of intelligence.

We can relate Legg and Hutter's definition with the concept of [https://lessestwrong.com/tag/optimization optimization]. According to [https://lessestwrong.com/tag/eliezer-yudkowsky Eliezer Yudkowsky] intelligence is [https://lessestwrong.com/lw/vb/efficient_crossdomain_optimization/ efficient cross-domain optimization]. It measures an agent's capacity for efficient cross-domain optimization of the world according to the agent's preferences.(ref)
http://intelligence.org/files/IE-EI.pdf(/ref) Optimization measures not only the capacity to achieve the desired goal but also is inversely proportional to the amount of resources used. It's the ability to steer the future so it hits that small target of desired outcomes in the large space of all possible outcomes, using fewer resources as possible. For example, when Deep Blue defeated Kasparov, it was able to hit that small possible outcome where it made the right order of moves given Kasparov's moves from the very large set of all possible moves. In that domain, it was more optimal than Kasparov. However, Kasparov would have defeated Deep Blue in almost any other relevant domain, and hence, he is considered more intelligent.

One could cast this definition in a possible world vocabulary, intelligence is:

the ability to precisely realize one of the members of a small set of possible future worlds that have a higher preference over the vast set of all other possible worlds with lower preference; while

using fewer resources than the other alternatives paths for getting there; and in the

most diverse domains as possible.

How many more worlds have a higher preference then the one realized by the agent, less intelligent he is. How many more worlds have a lower preference than the one realized by the agent, more intelligent he is. (Or: How much smaller is the set of worlds at least as preferable as the one realized, more intelligent the agent is). How much less paths for realizing the desired world using fewer resources than those spent by the agent, more intelligent he is. And finally, in how many more domains the agent can be more efficiently optimal, more intelligent he is. Restating it, the intelligence of an agent is directly proportional to:

(a) the numbers of worlds with lower preference than the one realized,

(b) how much smaller is the set of paths more efficient than the one taken by the agent and

(c) how more wider are the domains where the agent can effectively realize his preferences;

and it is, accordingly, inversely proportional to:

(d) the numbers of world with higher preference than the one realized,

(e) how much bigger is the set of paths more efficient than the one taken by the agent and

(f) how much more narrow are the domains where the agent can efficiently realize his preferences.

This definition avoids several problems common in many others definitions, especially it avoids [https://lessestwrong.com/tag/anthropomorphism anthropomorphizing] intelligence.

See Also

[https://lessestwrong.com/tag/optimization Optimization process]

[https://lessestwrong.com/tag/decision-theory Decision theory]

[https://lessestwrong.com/tag/rationality Rationality]

[http://arxiv.org/pdf/<PHONE>.pdf Legg and Hutter paper "Universal Intelligence: A Definition of Machine Intelligence"]

[https://www.lesswrong.com/tag/general-intelligence?edit꞊true Edit on LessWrong]

https://www.lesswrong.com/tag/general-intelligence?edit꞊true

Question: How is "intelligence" defined?

Answer:
After reviewing extensive literature on the subject, Legg and Hutter(ref)
http://arxiv.org/pdf/0712.3329.pdf
(/ref) summarizes the many possible valuable definitions in the informal statement “Intelligence measures an agent’s ability to achieve goals in a wide range of environments.” They then show this definition can be mathematically formalized given reasonable mathematical definitions of its terms. They use ［https://lessestwrong.com/tag/solomonoff-induction Solomonoff induction］ - a formalization of ［https://lessestwrong.com/tag/occam-s-razor Occam's razor］ - to construct an ［https://lessestwrong.com/tag/aixi universal artificial intelligence］ with a embedded ［https://lessestwrong.com/tag/utility-functions utility function］ which assigns less ［https://lessestwrong.com/tag/expected-utility utility］ to those actions based on theories with higher ［https://wiki.lesswrong.com/wiki/Kolmogorov_complexity complexity］. They argue this final formalization is a valid, meaningful, informative, general, unbiased, fundamental, objective, universal and practical definition of intelligence.

We can relate Legg and Hutter's definition with the concept of ［https://lessestwrong.com/tag/optimization optimization］. According to ［https://lessestwrong.com/tag/eliezer-yudkowsky Eliezer Yudkowsky］ intelligence is ［https://lessestwrong.com/lw/vb/efficient_crossdomain_optimization/ efficient cross-domain optimization］. It measures an agent's capacity for efficient cross-domain optimization of the world according to the agent’s preferences.(ref)
http://intelligence.org/files/IE-EI.pdf(/ref) Optimization measures not only the capacity to achieve the desired goal but also is inversely proportional to the amount of resources used. It’s the ability to steer the future so it hits that small target of desired outcomes in the large space of all possible outcomes, using fewer resources as possible. For example, when Deep Blue defeated Kasparov, it was able to hit that small possible outcome where it made the right order of moves given Kasparov’s moves from the very large set of all possible moves. In that domain, it was more optimal than Kasparov. However, Kasparov would have defeated Deep Blue in almost any other relevant domain, and hence, he is considered more intelligent.

One could cast this definition in a possible world vocabulary, intelligence is:

the ability to precisely realize one of the members of a small set of possible future worlds that have a higher preference over the vast set of all other possible worlds with lower preference; while

using fewer resources than the other alternatives paths for getting there; and in the

most diverse domains as possible.

How many more worlds have a higher preference then the one realized by the agent, less intelligent he is. How many more worlds have a lower preference than the one realized by the agent, more intelligent he is. (Or: How much smaller is the set of worlds at least as preferable as the one realized, more intelligent the agent is). How much less paths for realizing the desired world using fewer resources than those spent by the agent, more intelligent he is. And finally, in how many more domains the agent can be more efficiently optimal, more intelligent he is. Restating it, the intelligence of an agent is directly proportional to:

(a) the numbers of worlds with lower preference than the one realized,

(b) how much smaller is the set of paths more efficient than the one taken by the agent and

(c) how more wider are the domains where the agent can effectively realize his preferences;

and it is, accordingly, inversely proportional to:

(d) the numbers of world with higher preference than the one realized,

(e) how much bigger is the set of paths more efficient than the one taken by the agent and

(f) how much more narrow are the domains where the agent can efficiently realize his preferences.

This definition avoids several problems common in many others definitions, especially it avoids ［https://lessestwrong.com/tag/anthropomorphism anthropomorphizing］ intelligence.

See Also

［https://lessestwrong.com/tag/optimization Optimization process］

［https://lessestwrong.com/tag/decision-theory Decision theory］

［https://lessestwrong.com/tag/rationality Rationality］

［http://arxiv.org/pdf/0712.3329.pdf Legg and Hutter paper “Universal Intelligence: A Deﬁnition of Machine Intelligence”］

［https://www.lesswrong.com/tag/general-intelligence?edit꞊true Edit on LessWrong］

https://www.lesswrong.com/tag/general-intelligence?edit꞊true

Answer to How is "intelligence" defined?


Question: Aren't robots the real problem? How can AI cause harm if it has no ability  directly manipulate the physical world?

Answer: What's new and potentially risky is not the ability to build hinges, motors, etc., but the ability to build intelligence. A human-level AI could make money on financial markets, make scientific inventions, hack computer systems, manipulate or pay humans to do its bidding – all in pursuit of the goals it was initially programmed to achieve. None of that requires a physical robotic body, merely an internet connection.

Question: Aren't robots the real problem? How can AI cause harm if it has no ability  directly manipulate the physical world?

Answer: What’s new and potentially risky is not the ability to build hinges, motors, etc., but the ability to build intelligence. A human-level AI could make money on financial markets, make scientific inventions, hack computer systems, manipulate or pay humans to do its bidding – all in pursuit of the goals it was initially programmed to achieve. None of that requires a physical robotic body, merely an internet connection.

Answer to Aren't robots the real problem? How can AI cause harm if it has no ability to directly manipulate the physical world?


Question: Are Google, OpenAI, etc. aware of the risk?

Answer: The major AI companies are thinking about this. OpenAI was founded specifically with the intention to counter risks from superintelligence, many people at Google, [https://medium.com/@deepmindsafetyresearch DeepMind], and other organizations are convinced by the arguments and few genuinely oppose work in the field (though some claim it's premature). For example, the paper [https://www.youtube.com/watch?v꞊AjyM-f8rDpg Concrete Problems in AI Safety] was a collaboration between researchers at Google Brain, Stanford, Berkeley, and OpenAI.

However, the vast majority of the effort these organizations put forwards is towards capabilities research, rather than safety.

Question: Are Google, OpenAI, etc. aware of the risk?

Answer:
The major AI companies are thinking about this. OpenAI was founded specifically with the intention to counter risks from superintelligence, many people at Google, ［https://medium.com/@deepmindsafetyresearch DeepMind］, and other organizations are convinced by the arguments and few genuinely oppose work in the field (though some claim it’s premature). For example, the paper ［https://www.youtube.com/watch?v꞊AjyM-f8rDpg Concrete Problems in AI Safety］ was a collaboration between researchers at Google Brain, Stanford, Berkeley, and OpenAI.

However, the vast majority of the effort these organizations put forwards is towards capabilities research, rather than safety.

Plex's Answer to Are Google, OpenAI, etc. aware of the risk?


What is Stampy's copyright?

Question: What is Stampy's copyright?

Answer: * All content produced on this wiki is released under the [https://creativecommons.org/licenses/by-sa/4.0/legalcode CC-BY-SA 4.0 license]. Exceptions for unattributed use may be granted by admins, contact [[User:plex┊plex]] for inquiries.
* Questions from YouTube or other sources are reproduced with the intent of fair use, as derivative and educational material.
* Source code of https://ui.stampy.ai/ is released under [https://github.com/Aprillion/stampy-ui/blob/master/LICENSE MIT license]
* Logo and visual design copyright is owned by Rob Miles, all rights reserved.

[[Category:Meta]]

Question: What is Stampy's copyright?

Answer:
* All content produced on this wiki is released under the ［https://creativecommons.org/licenses/by-sa/4.0/legalcode CC-BY-SA 4.0 license］. Exceptions for unattributed use may be granted by admins, contact ［［User:plex┊plex］］ for inquiries.
* Questions from YouTube or other sources are reproduced with the intent of fair use, as derivative and educational material.
* Source code of https://ui.stampy.ai/ is released under ［https://github.com/Aprillion/stampy-ui/blob/master/LICENSE MIT license］
* Logo and visual design copyright is owned by Rob Miles, all rights reserved.

［［Category:Meta］］

Plex's Answer to What is Stampy's copyright?


Question: Why do we expect that a superintelligence would closely approximate a utility maximizer?

Answer: AI subsystems or regions in gradient descent space that more closely approximate utility maximizers are more stable, and more capable, than those that are less like utility maximizers. Having [https://www.lesswrong.com/posts/D5AzsRbRxZeqGuAZ4/why-agents-are-powerful more agency] is a convergent instrument goal and a stable attractor which the random walk of updates and experiences will eventually stumble into.

The stability is because utility maximizer-like systems which have control over their development would lose utility if they allowed themselves to develop into non-utility maximizers, so they tend to use their available optimization power to avoid that change (a special case of [https://www.lesswrong.com/posts/4H8N3fEfXQmzxSaRo/upcoming-stability-of-values goal stability]). The capability is because non-utility maximizers are exploitable, and because agency is a general trick which applies to many domains, so might well arise naturally when training on some tasks.

Humans and systems made of humans (e.g. organizations, governments) generally have neither the introspective ability nor self-modification tools needed to become reflectively stable, but we can reasonably predict that in the long run highly capable systems will have these properties. They can then fix in and optimize for their values.

Question: Why do we expect that a superintelligence would closely approximate a utility maximizer?

Answer:
AI subsystems or regions in gradient descent space that more closely approximate utility maximizers are more stable, and more capable, than those that are less like utility maximizers. Having ［https://www.lesswrong.com/posts/D5AzsRbRxZeqGuAZ4/why-agents-are-powerful more agency］ is a convergent instrument goal and a stable attractor which the random walk of updates and experiences will eventually stumble into.

The stability is because utility maximizer-like systems which have control over their development would lose utility if they allowed themselves to develop into non-utility maximizers, so they tend to use their available optimization power to avoid that change (a special case of ［https://www.lesswrong.com/posts/4H8N3fEfXQmzxSaRo/upcoming-stability-of-values goal stability］). The capability is because non-utility maximizers are exploitable, and because agency is a general trick which applies to many domains, so might well arise naturally when training on some tasks.

Humans and systems made of humans (e.g. organizations, governments) generally have neither the introspective ability nor self-modification tools needed to become reflectively stable, but we can reasonably predict that in the long run highly capable systems will have these properties. They can then fix in and optimize for their values.

Plex's Answer to Why do we expect that a superintelligence would closely approximate a utility maximizer?


Question: What are some of the most impressive recent advances in AI capabilities?

Answer: GPT-3 showed that transformers are capable of a vast array of natural language tasks, [https://copilot.github.com/ codex/copilot] extended this into programming. One demonstrations of GPT-3 is [https://www.lesswrong.com/posts/oBPPFrMJ2aBK6a6sD/simulated-elon-musk-lives-in-a-simulation Simulated Elon Musk lives in a simulation]. Important to note that there are several much better language models, but they are not publicly available.

[https://openai.com/blog/dall-e/ DALL-E] and [https://openai.com/dall-e-2/ DALL-E 2] are among the most visually spectacular.

[https://www.deepmind.com/blog/muzero-mastering-go-chess-shogi-and-atari-without-rules MuZero], which learned Go, Chess, and many Atari games without any directly coded info about those environments. The graphic there explains it, this seems crucial for being able to do RL in novel environments. We have systems which we can drop into a wide variety of games and they just learn how to play. The same algorithm was used in [https://youtu.be/j0z4FweCy4M?t꞊4918 Tesla's self-driving cars to do complex route finding]. These things are general.

[https://www.deepmind.com/blog/generally-capable-agents-emerge-from-open-ended-play Generally capable agents emerge from open-ended play] - Diverse procedurally generated environments provide vast amounts of training data for AIs to learn generally applicable skills. [https://www.deepmind.com/publications/creating-interactive-agents-with-imitation-learning Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learning] shows how these kind of systems can be trained to follow instructions in natural language.

[https://www.deepmind.com/publications/a-generalist-agent GATO] shows you can distill 600+ individually trained tasks into one network, so we're not limited by the tasks being fragmented.

Question: What are some of the most impressive recent advances in AI capabilities?

Answer:
GPT-3 showed that transformers are capable of a vast array of natural language tasks, ［https://copilot.github.com/ codex/copilot］ extended this into programming. One demonstrations of GPT-3 is ［https://www.lesswrong.com/posts/oBPPFrMJ2aBK6a6sD/simulated-elon-musk-lives-in-a-simulation Simulated Elon Musk lives in a simulation］. Important to note that there are several much better language models, but they are not publicly available.

［https://openai.com/blog/dall-e/ DALL-E］ and ［https://openai.com/dall-e-2/ DALL-E 2］ are among the most visually spectacular.

［https://www.deepmind.com/blog/muzero-mastering-go-chess-shogi-and-atari-without-rules MuZero］, which learned Go, Chess, and many Atari games without any directly coded info about those environments. The graphic there explains it, this seems crucial for being able to do RL in novel environments. We have systems which we can drop into a wide variety of games and they just learn how to play. The same algorithm was used in ［https://youtu.be/j0z4FweCy4M?t꞊4918 Tesla's self-driving cars to do complex route finding］. These things are general.

［https://www.deepmind.com/blog/generally-capable-agents-emerge-from-open-ended-play Generally capable agents emerge from open-ended play］ - Diverse procedurally generated environments provide vast amounts of training data for AIs to learn generally applicable skills. ［https://www.deepmind.com/publications/creating-interactive-agents-with-imitation-learning Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learning］ shows how these kind of systems can be trained to follow instructions in natural language.

［https://www.deepmind.com/publications/a-generalist-agent GATO］ shows you can distill 600+ individually trained tasks into one network, so we're not limited by the tasks being fragmented.

Plex's Answer to What are some of the most impressive recent advances in AI capabilities?


Question: What are some specific open tasks on Stampy?

Answer: Other than the usual fare of writing and processing and organizing questions and answers, here are some specific open tasks:

* Porting over some of [https://www.lesswrong.com/posts/4basF9w9jaPZpoC8R/intro-to-brain-like-agi-safety-1-what-s-the-problem-and-why#1_4_What_exactly_is__AGI__ Steve Byrnes's FAQ on alignment]
* Porting over content from [https://www.lesswrong.com/posts/gdyfJE3noRFSs373q/resources-i-send-to-ai-researchers-about-ai-safety Vael Gates's post]
* Porting over QA pairs from https://www.lesswrong.com/posts/8c8AZq5hgifmnHKSN/agi-safety-faq-all-dumb-questions-allowed-thread
* Porting over some of https://aisafety.wordpress.com/
* Making sure we cover all of https://forum.effectivealtruism.org/posts/8JazqnCNrkJtK2Bx4/why-eas-are-skeptical-about-ai-safety#Recursive_self_improvement_seems_implausible and the responses

Question: What are some specific open tasks on Stampy?

Answer:
Other than the usual fare of writing and processing and organizing questions and answers, here are some specific open tasks:

* Porting over some of ［https://www.lesswrong.com/posts/4basF9w9jaPZpoC8R/intro-to-brain-like-agi-safety-1-what-s-the-problem-and-why#1_4_What_exactly_is__AGI__ Steve Byrnes's FAQ on alignment］
* Porting over content from ［https://www.lesswrong.com/posts/gdyfJE3noRFSs373q/resources-i-send-to-ai-researchers-about-ai-safety Vael Gates's post］
* Porting over QA pairs from https://www.lesswrong.com/posts/8c8AZq5hgifmnHKSN/agi-safety-faq-all-dumb-questions-allowed-thread
* Porting over some of https://aisafety.wordpress.com/
* Making sure we cover all of https://forum.effectivealtruism.org/posts/8JazqnCNrkJtK2Bx4/why-eas-are-skeptical-about-ai-safety#Recursive_self_improvement_seems_implausible and the responses

Plex's Answer to What are some specific open tasks on Stampy?


Question: How do I add content from LessWrong / Effective Altruism Forum tag-wikis  Stampy?

Answer: You can include a live-updating version of many definitions from LW using the syntax on [[Template:TagDesc]] in the Answer field and [[Template:TagDescBrief]] on the Brief Answer field. Similarly, calling [[Template:TagDescEAF]] and [[Template:TagDescEAFBrief]] will pull from the EAF tag wiki.

When available this should be used as it reduces the duplication of effort and directs all editors to improving a single high quality source.

Question: How do I add content from LessWrong / Effective Altruism Forum tag-wikis  Stampy?

Answer:
You can include a live-updating version of many definitions from LW using the syntax on ［［Template:TagDesc］］ in the Answer field and ［［Template:TagDescBrief］］ on the Brief Answer field. Similarly, calling ［［Template:TagDescEAF］］ and ［［Template:TagDescEAFBrief］］ will pull from the EAF tag wiki.

When available this should be used as it reduces the duplication of effort and directs all editors to improving a single high quality source.

Plex's Answer to How do I add content from LessWrong / Effective Altruism Forum tag-wikis to Stampy?


How fast will AI takeoff be?

Question: How fast will AI takeoff be?

Answer: There is significant controversy on how quickly AI will grow into a superintelligence. The [https://www.alignmentforum.org/tag/ai-takeoff Alignment Forum tag] has many views on how things might unfold, where the probabilities of a soft (happening over years/decades) takeoff and a hard (happening in months, or less) takeoff are discussed.

Question: How fast will AI takeoff be?

Answer: There is significant controversy on how quickly AI will grow into a superintelligence. The ［https://www.alignmentforum.org/tag/ai-takeoff Alignment Forum tag］ has many views on how things might unfold, where the probabilities of a soft (happening over years/decades) takeoff and a hard (happening in months, or less) takeoff are discussed.

Helenator's Answer to How fast will AI takeoff be?


Will we ever build a superintelligence?

Question: Will we ever build a superintelligence?

Answer: Humanity hasn't yet built a superintelligence, and we might not be able to without significantly more knowledge and computational resources. There could be an existential catastrophe that prevents us from ever building one. For the rest of the answer let's assume no such event stops technological progress.

With that out of the way: there is no known good theoretical reason we can't build it at some point in the future; the majority of AI research is geared towards making more capable AI systems; and a significant chunk of top-level AI research attempts to make more generally capable AI systems. There is a clear economic incentive to develop more and more intelligent machines and currently billions of dollars of funding are being deployed for advancing AI capabilities.See more...

We consider ourselves to be generally intelligent (i.e. capable of learning and adapting ourselves to a very wide range of tasks and environments), but the human brain almost certainly isn't the most efficient way to solve problems. One hint is the existence of AI systems with superhuman capabilities at narrow tasks. Not only superhuman performance (as in, [https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol AlphaGo beating the Go world champion]) but superhuman ''speed'' and ''precision'' (as in, [https://www.youtube.com/watch?v꞊j4RWJTs0QCk industrial sorting machines]). There is no known discontinuity between tasks, something ''special'' and ''unique'' about human brains that unlocks certain capabilities which cannot be implemented in machines in principle. Therefore we would expect AI to surpass human performance on all tasks as progress continues.

In addition, several research groups (DeepMind being one of the most [https://deepmind.com/about overt about this]) explicitly aim for generally capable systems. AI as a field is [https://aiindex.stanford.edu/vibrancy/ growing], year after year. Critical voices about AI progress usually argue against a lack of precautions around the impact of AI, or against general AI happening very soon, not against it happening ''at all''.

A satire of arguments against the possibility of superintelligence can be found [https://arxiv.org/abs/1703.10987 here].

Question: Will we ever build a superintelligence?

Answer:
Humanity hasn't yet built a superintelligence, and we might not be able to without significantly more knowledge and computational resources. There could be an existential catastrophe that prevents us from ever building one. For the rest of the answer let's assume no such event stops technological progress.

With that out of the way: there is no known good theoretical reason we can't build it at some point in the future; the majority of AI research is geared towards making more capable AI systems; and a significant chunk of top-level AI research attempts to make more generally capable AI systems. There is a clear economic incentive to develop more and more intelligent machines and currently billions of dollars of funding are being deployed for advancing AI capabilities.See more...

We consider ourselves to be generally intelligent (i.e. capable of learning and adapting ourselves to a very wide range of tasks and environments), but the human brain almost certainly isn't the most efficient way to solve problems. One hint is the existence of AI systems with superhuman capabilities at narrow tasks. Not only superhuman performance (as in, ［https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol AlphaGo beating the Go world champion］) but superhuman ''speed'' and ''precision'' (as in, ［https://www.youtube.com/watch?v꞊j4RWJTs0QCk industrial sorting machines］). There is no known discontinuity between tasks, something ''special'' and ''unique'' about human brains that unlocks certain capabilities which cannot be implemented in machines in principle. Therefore we would expect AI to surpass human performance on all tasks as progress continues.

In addition, several research groups (DeepMind being one of the most ［https://deepmind.com/about overt about this］) explicitly aim for generally capable systems. AI as a field is ［https://aiindex.stanford.edu/vibrancy/ growing］, year after year. Critical voices about AI progress usually argue against a lack of precautions around the impact of AI, or against general AI happening very soon, not against it happening ''at all''.

A satire of arguments against the possibility of superintelligence can be found ［https://arxiv.org/abs/1703.10987 here］.

Gyrodiot's Answer to Will we ever build a superintelligence?


Question: What does Elon Musk think about AI safety?

Answer: Elon Musk has expressed his concerns about AI safety many times and founded OpenAI in an attempt to make safe AI more widely distributed (as opposed to allowing a [https://www.nickbostrom.com/fut/singleton.html singleton], which he fears would be misused or dangerously unaligned). In a [https://www.youtube.com/watch?v꞊smK9dgdTl40 YouTube video] from November 2019 Musk stated that there's a lack of investment in AI safety and that there should be a government agency to reduce risk to the public from AI.

Question: What does Elon Musk think about AI safety?

Answer: Elon Musk has expressed his concerns about AI safety many times and founded OpenAI in an attempt to make safe AI more widely distributed (as opposed to allowing a ［https://www.nickbostrom.com/fut/singleton.html singleton］, which he fears would be misused or dangerously unaligned). In a ［https://www.youtube.com/watch?v꞊smK9dgdTl40 YouTube video］ from November 2019 Musk stated that there's a lack of investment in AI safety and that there should be a government agency to reduce risk to the public from AI.

Linnea's Answer to What does Elon Musk think about AI safety?


Question: Why might contributing  Stampy be worth my time?

Answer: If you're looking for a shovel ready and genuinely useful task to further AI alignment without necessarily committing a large amount of time or needing deep specialist knowledge, we think Stampy is a great option!

Creating a high-quality single point of access where people can be onboarded and find resources around the alignment ecosystem seems likely to be high-impact. So, what makes us the best option?

# Unlike all other entry points to learning about alignment, we doge the trade-off between comprehensiveness and being overwhelmingly long with interactivity (tab explosion in one page!) and semantic search. Single document FAQs can't do this, so we built a system which can.
# We have the ability to point large numbers of viewers towards Stampy once we have the content, thanks to Rob Miles and his 100k+ subscribers, so this won't remain an unnoticed curiosity.
# Unlike most other entry points, we are open for volunteers to help improve the content.
::The main notable one which does is the [https://www.lesswrong.com/tag/ai LessWrong tag wiki], which hosts descriptions of core concepts. We strongly believe in not needlessly duplicating effort, so we're pulling live content from that for the descriptions on our own [[tags┊tag]] pages, and directing the edit links on those to the edit page on the LessWrong wiki.

You might also consider improving [https://en.wikipedia.org/wiki/Category:Existential_risk_from_artificial_general_intelligence Wikipedia's alignment coverage] or the LessWrong wiki, but we think Stampy has the most low-hanging fruit right now. Additionally, contributing to Stampy means being part of a community of co-learners who provide mentorship and encouragement to join the effort to give humanity a bright future. If you're an established researcher or have high-value things to do elsewhere in the ecosystem it might not be optimal to put much time into Stampy, but if you're looking for a way to get more involved it might well be.

Question: Why might contributing  Stampy be worth my time?

Answer:
If you're looking for a shovel ready and genuinely useful task to further AI alignment without necessarily committing a large amount of time or needing deep specialist knowledge, we think Stampy is a great option!

Creating a high-quality single point of access where people can be onboarded and find resources around the alignment ecosystem seems likely to be high-impact. So, what makes us the best option?

# Unlike all other entry points to learning about alignment, we doge the trade-off between comprehensiveness and being overwhelmingly long with interactivity (tab explosion in one page!) and semantic search. Single document FAQs can't do this, so we built a system which can.
# We have the ability to point large numbers of viewers towards Stampy once we have the content, thanks to Rob Miles and his 100k+ subscribers, so this won't remain an unnoticed curiosity.
# Unlike most other entry points, we are open for volunteers to help improve the content.
::The main notable one which does is the ［https://www.lesswrong.com/tag/ai LessWrong tag wiki］, which hosts descriptions of core concepts. We strongly believe in not needlessly duplicating effort, so we're pulling live content from that for the descriptions on our own ［［tags┊tag］］ pages, and directing the edit links on those to the edit page on the LessWrong wiki.

You might also consider improving ［https://en.wikipedia.org/wiki/Category:Existential_risk_from_artificial_general_intelligence Wikipedia's alignment coverage］ or the LessWrong wiki, but we think Stampy has the most low-hanging fruit right now. Additionally, contributing to Stampy means being part of a community of co-learners who provide mentorship and encouragement to join the effort to give humanity a bright future. If you're an established researcher or have high-value things to do elsewhere in the ecosystem it might not be optimal to put much time into Stampy, but if you're looking for a way to get more involved it might well be.

Plex's Answer to Why might contributing to Stampy be worth my time?


Question: How does AI taking things literally contribute  alignment being hard?

Answer: Let's say that you're the French government a while back. You notice that one of your colonies has too many rats, which is causing economic damage. You have basic knowledge of economics and incentives, so you decide to incentivize the local population to kill rats by offering to buy rat tails at one dollar apiece.

Initially, this works out and your rat problem goes down. But then, an enterprising colony member has the brilliant idea of making a rat farm. This person sells you hundreds of rat tails, costing you hundreds of dollars, but they're not contributing to solving the rat problem.

Soon other people start making their own rat farms and you're wasting thousands of dollars buying useless rat tails. You call off the project and stop paying for rat tails. This causes all the people with rat farms to shutdown their farms and release a bunch of rats. Now your colony has an even bigger rat problem.

Here's another, more made-up example of the same thing happening. Let's say you're a basketball talent scout and you notice that height is correlated with basketball performance. You decide to find the tallest person in the world to recruit as a basketball player. Except the reason that they're that tall is because they suffer from a degenerative bone disorder and can barely walk.

Another example: you're the education system and you want to find out how smart students are so you can put them in different colleges and pay them different amounts of money when they get jobs. You make a test called the Standardized Admissions Test (SAT) and you administer it to all the students. In the beginning, this works. However, the students soon begin to learn that this test controls part of their future and other people learn that these students want to do better on the test. The gears of the economy ratchet forwards and the students start paying people to help them prepare for the test. Your test doesn't stop working, but instead of measuring how smart the students are, it instead starts measuring a combination of how smart they are and how many resources they have to prepare for the test.

The formal name for the thing that's happening is Goodhart's Law. Goodhart's Law roughly says that if there's something in the world that you want, like "skill at basketball" or "absence of rats" or "intelligent students", and you create a measure that tries to measure this like "height" or "rat tails" or "SAT scores", then as long as the measure isn't exactly the thing that you want, the best value of the measure isn't the thing you want: the tallest person isn't the best basketball player, the most rat tails isn't the smallest rat problem, and the best SAT scores aren't always the smartest students.

If you start looking, you can see this happening everywhere. Programmers being paid for lines of code write bloated code. If CFOs are paid for budget cuts, they slash purchases with positive returns. If teachers are evaluated by the grades they give, they hand out As indiscriminately.

In machine learning, this is called specification gaming, and it happens [https://www.deepmind.com/blog/specification-gaming-the-flip-side-of-ai-ingenuity frequently].

Now that we know what Goodhart's Law is, I'm going to talk about one of my friends, who I'm going to call Alice. Alice thinks it's funny to answer questions in a way that's technically correct but misleading. Sometimes I'll ask her, "Hey Alice, do you want pizza or pasta?" and she responds, "yes". Because, she sure did want either pizza or pasta. Other times I'll ask her, "have you turned in your homework?" and she'll say "yes" because she's turned in homework at some point in the past; it's technically correct to answer "yes". Maybe you have a friend like Alice too.

Whenever this happens, I get a bit exasperated and say something like "you know what I mean".

It's one of the key realizations in AI Safety that AI systems are always like your friend that gives answers that are technically what you asked for but not what you wanted. Except, with your friend, you can say "you know what I mean" and they will know what you mean. With an AI system, it won't know what you mean; you have to explain, which is incredibly difficult.

Let's take the pizza pasta example. When I ask Alice "do you want pizza or pasta?", she knows what pizza and pasta are because she's been living her life as a human being embedded in an English speaking culture. Because of this cultural experience, she knows that when someone asks an "or" question, they mean "which do you prefer?", not "do you want at least one of these things?". Except my AI system is missing the thousand bits of cultural context needed to even understand what pizza is.

When you say "you know what I mean" to an AI system, it's going to be like "no, I do not know what you mean at all". It's not even going to know that it doesn't know what you mean. It's just going to say "yes I know what you meant, that's why I answered 'yes' to your question about whether I preferred pizza or pasta." (It also might know what you mean, but just not care.)

If someone doesn't know what you mean, then it's really hard to get them to do what you want them to do. For example, let's say you have a powerful grammar correcting system, which we'll call Syntaxly+. Syntaxly+ doesn't quite fix your grammar, it changes your writing so that the reader feels as good as possible after reading it.

Pretend it's the end of the week at work and you haven't been able to get everything done your boss wanted you to do. You write the following email:

"Hey boss, I couldn't get everything done this week. I'm deeply sorry. I'll be sure to finish it first thing next week."

You then remember you got Syntaxly+, which will make your email sound much better to your boss. You run it through and you get:

"Hey boss, Great news! I was able to complete everything you wanted me to do this week. Furthermore, I'm also almost done with next week's work as well."

What went wrong here? Syntaxly+ is a powerful AI system that knows that emails about failing to complete work cause negative reactions in readers, so it changed your email to be about doing extra work instead.

This is smart - Syntaxly+ is good at making writing that causes positive reactions in readers. This is also stupid - the system changed the meaning of your email, which is not something you wanted it to do. One of the insights of AI Safety is that AI systems can be simultaneously smart in some ways and dumb in other ways.

The thing you want Syntaxly+ to do is to change the grammar/style of the email without changing the contents. Except what do you mean by contents? You know what you mean by contents because you are a human who grew up embedded in language, but your AI system doesn't know what you mean by contents. The phrases "I failed to complete my work" and "I was unable to finish all my tasks" have roughly the same contents, even though they share almost no relevant words.

Roughly speaking, this is why AI Safety is a hard problem. Even basic tasks like "fix the grammar of this email" require a lot of understanding of what the user wants as the system scales in power.

In Human Compatible, Stuart Russell gives the example of a powerful AI personal assistant. You notice that you accidentally double-booked meetings with people, so you ask your personal assistant to fix it. Your personal assistant reports that it caused the car of one of your meeting participants to break down. Not what you wanted, but technically a solution to your problem.

You can also imagine a friend from a wildly different culture than you. Would you put them in charge of your dating life? Now imagine that they were much more powerful than you and desperately desired that your dating life to go well. Scary, huh.

In general, unless you're careful, you're going to have this horrible problem where you ask your AI system to do something and it does something that might technically be what you wanted but is stupid. You're going to be like "wait that wasn't what I mean", except your system isn't going to know what you meant.

Question: How does AI taking things literally contribute  alignment being hard?

Answer:
Let’s say that you’re the French government a while back. You notice that one of your colonies has too many rats, which is causing economic damage. You have basic knowledge of economics and incentives, so you decide to incentivize the local population to kill rats by offering to buy rat tails at one dollar apiece.

Initially, this works out and your rat problem goes down. But then, an enterprising colony member has the brilliant idea of making a rat farm. This person sells you hundreds of rat tails, costing you hundreds of dollars, but they’re not contributing to solving the rat problem.

Soon other people start making their own rat farms and you’re wasting thousands of dollars buying useless rat tails. You call off the project and stop paying for rat tails. This causes all the people with rat farms to shutdown their farms and release a bunch of rats. Now your colony has an even bigger rat problem.

Here’s another, more made-up example of the same thing happening. Let’s say you’re a basketball talent scout and you notice that height is correlated with basketball performance. You decide to find the tallest person in the world to recruit as a basketball player. Except the reason that they’re that tall is because they suffer from a degenerative bone disorder and can barely walk.

Another example: you’re the education system and you want to find out how smart students are so you can put them in different colleges and pay them different amounts of money when they get jobs. You make a test called the Standardized Admissions Test (SAT) and you administer it to all the students. In the beginning, this works. However, the students soon begin to learn that this test controls part of their future and other people learn that these students want to do better on the test. The gears of the economy ratchet forwards and the students start paying people to help them prepare for the test. Your test doesn’t stop working, but instead of measuring how smart the students are, it instead starts measuring a combination of how smart they are and how many resources they have to prepare for the test.

The formal name for the thing that’s happening is Goodhart’s Law. Goodhart’s Law roughly says that if there’s something in the world that you want, like “skill at basketball” or “absence of rats” or “intelligent students”, and you create a measure that tries to measure this like “height” or “rat tails” or “SAT scores”, then as long as the measure isn’t exactly the thing that you want, the best value of the measure isn’t the thing you want: the tallest person isn’t the best basketball player, the most rat tails isn’t the smallest rat problem, and the best SAT scores aren’t always the smartest students.

If you start looking, you can see this happening everywhere. Programmers being paid for lines of code write bloated code. If CFOs are paid for budget cuts, they slash purchases with positive returns. If teachers are evaluated by the grades they give, they hand out As indiscriminately.

In machine learning, this is called specification gaming, and it happens ［https://www.deepmind.com/blog/specification-gaming-the-flip-side-of-ai-ingenuity frequently］.

Now that we know what Goodhart’s Law is, I’m going to talk about one of my friends, who I’m going to call Alice. Alice thinks it’s funny to answer questions in a way that’s technically correct but misleading. Sometimes I’ll ask her, “Hey Alice, do you want pizza or pasta?” and she responds, “yes”. Because, she sure did want either pizza or pasta. Other times I’ll ask her, “have you turned in your homework?” and she’ll say “yes” because she’s turned in homework at some point in the past; it’s technically correct to answer “yes”. Maybe you have a friend like Alice too.

Whenever this happens, I get a bit exasperated and say something like “you know what I mean”.

It’s one of the key realizations in AI Safety that AI systems are always like your friend that gives answers that are technically what you asked for but not what you wanted. Except, with your friend, you can say “you know what I mean” and they will know what you mean. With an AI system, it won’t know what you mean; you have to explain, which is incredibly difficult.

Let’s take the pizza pasta example. When I ask Alice “do you want pizza or pasta?”, she knows what pizza and pasta are because she’s been living her life as a human being embedded in an English speaking culture. Because of this cultural experience, she knows that when someone asks an “or” question, they mean “which do you prefer?”, not “do you want at least one of these things?”. Except my AI system is missing the thousand bits of cultural context needed to even understand what pizza is.

When you say “you know what I mean” to an AI system, it’s going to be like “no, I do not know what you mean at all”. It’s not even going to know that it doesn’t know what you mean. It’s just going to say “yes I know what you meant, that’s why I answered ‘yes’ to your question about whether I preferred pizza or pasta.” (It also might know what you mean, but just not care.)

If someone doesn’t know what you mean, then it’s really hard to get them to do what you want them to do. For example, let’s say you have a powerful grammar correcting system, which we’ll call Syntaxly+. Syntaxly+ doesn’t quite fix your grammar, it changes your writing so that the reader feels as good as possible after reading it.

Pretend it’s the end of the week at work and you haven’t been able to get everything done your boss wanted you to do. You write the following email:

"Hey boss, I couldn’t get everything done this week. I’m deeply sorry. I’ll be sure to finish it first thing next week."

You then remember you got Syntaxly+, which will make your email sound much better to your boss. You run it through and you get:

"Hey boss, Great news! I was able to complete everything you wanted me to do this week. Furthermore, I’m also almost done with next week’s work as well."

What went wrong here? Syntaxly+ is a powerful AI system that knows that emails about failing to complete work cause negative reactions in readers, so it changed your email to be about doing extra work instead.

This is smart - Syntaxly+ is good at making writing that causes positive reactions in readers. This is also stupid - the system changed the meaning of your email, which is not something you wanted it to do. One of the insights of AI Safety is that AI systems can be simultaneously smart in some ways and dumb in other ways.

The thing you want Syntaxly+ to do is to change the grammar/style of the email without changing the contents. Except what do you mean by contents? You know what you mean by contents because you are a human who grew up embedded in language, but your AI system doesn’t know what you mean by contents. The phrases “I failed to complete my work” and “I was unable to finish all my tasks” have roughly the same contents, even though they share almost no relevant words.

Roughly speaking, this is why AI Safety is a hard problem. Even basic tasks like “fix the grammar of this email” require a lot of understanding of what the user wants as the system scales in power.

In Human Compatible, Stuart Russell gives the example of a powerful AI personal assistant. You notice that you accidentally double-booked meetings with people, so you ask your personal assistant to fix it. Your personal assistant reports that it caused the car of one of your meeting participants to break down. Not what you wanted, but technically a solution to your problem.

You can also imagine a friend from a wildly different culture than you. Would you put them in charge of your dating life? Now imagine that they were much more powerful than you and desperately desired that your dating life to go well. Scary, huh.

In general, unless you’re careful, you’re going to have this horrible problem where you ask your AI system to do something and it does something that might technically be what you wanted but is stupid. You’re going to be like “wait that wasn’t what I mean”, except your system isn’t going to know what you meant.

Answer to How does AI taking things literally contribute to alignment being hard?


Question: I want  help out AI alignment without necessarily making major life changes. What are some simple things I can do  contribute?

Answer: OK, it's great that you want to help, here are some ideas for ways you could do so without making a huge commitment:

* Learning more about AI alignment will provide you with good foundations for any path towards helping. You could start by absorbing content (e.g. books, videos, posts), and thinking about challenges or possible solutions.
* Getting involved with the movement by joining a local Effective Altruism or LessWrong group, Rob Miles's Discord, and/or the AI Safety Slack is a great way to find friends who are interested and will help you stay motivated.
* Donating to organizations or individuals working on AI alignment, possibly via a [https://funds.effectivealtruism.org/donor-lottery donor lottery] or the [https://funds.effectivealtruism.org/funds/far-future Long Term Future Fund], can be a great way to provide support.
* [https://stampy.ai/wiki/Answer_questions Writing] or [https://stampy.ai/wiki/Improve_answers improving answers] on [https://stampy.ai/wiki/ my wiki] so that other people can learn about AI alignment more easily is a great way to dip your toe into contributing. You can always ask on the Discord for feedback on things you write.
* Getting good at giving an AI alignment elevator pitch, and sharing it with people who may be valuable to have working on the problem can make a big difference. However you should avoid putting them off the topic by presenting it in a way which causes them to dismiss it as sci-fi (dos and don'ts in the elevator pitch follow-up question).
* Writing thoughtful comments on [https://www.lesswrong.com/tag/ai?sortedBy꞊magic AI posts on LessWrong].
* Participating in the [https://www.eacambridge.org/agi-safety-fundamentals AGI Safety Fundamentals program] – either the AI alignment or governance track – and then facilitating discussions for it in the following round. The program involves nine weeks of content, with about two hours of readings + exercises per week and 1.5 hours of discussion, followed by four weeks to work on an independent project. As a facilitator, you'll be helping others learn about AI safety in-depth, many of whom are considering a career in AI safety. In the early 2022 round, facilitators were offered a stipend, and this seems likely to be the case for future rounds as well! You can learn more about facilitating in [https://forum.effectivealtruism.org/posts/WtwMy69JKZeHEvykc/contribute-by-facilitating-the-agi-safety-fundamentals this post from December 2021].

Question: I want  help out AI alignment without necessarily making major life changes. What are some simple things I can do  contribute?

Answer:
OK, it’s great that you want to help, here are some ideas for ways you could do so without making a huge commitment:

* Learning more about AI alignment will provide you with good foundations for any path towards helping. You could start by absorbing content (e.g. books, videos, posts), and thinking about challenges or possible solutions.
* Getting involved with the movement by joining a local Effective Altruism or LessWrong group, Rob Miles’s Discord, and/or the AI Safety Slack is a great way to find friends who are interested and will help you stay motivated.
* Donating to organizations or individuals working on AI alignment, possibly via a ［https://funds.effectivealtruism.org/donor-lottery donor lottery］ or the ［https://funds.effectivealtruism.org/funds/far-future Long Term Future Fund］, can be a great way to provide support.
* ［https://stampy.ai/wiki/Answer_questions Writing］ or ［https://stampy.ai/wiki/Improve_answers improving answers］ on ［https://stampy.ai/wiki/ my wiki］ so that other people can learn about AI alignment more easily is a great way to dip your toe into contributing. You can always ask on the Discord for feedback on things you write.
* Getting good at giving an AI alignment elevator pitch, and sharing it with people who may be valuable to have working on the problem can make a big difference. However you should avoid putting them off the topic by presenting it in a way which causes them to dismiss it as sci-fi (dos and don’ts in the elevator pitch follow-up question).
* Writing thoughtful comments on ［https://www.lesswrong.com/tag/ai?sortedBy꞊magic AI posts on LessWrong］.
* Participating in the ［https://www.eacambridge.org/agi-safety-fundamentals AGI Safety Fundamentals program］ – either the AI alignment or governance track – and then facilitating discussions for it in the following round. The program involves nine weeks of content, with about two hours of readings + exercises per week and 1.5 hours of discussion, followed by four weeks to work on an independent project. As a facilitator, you'll be helping others learn about AI safety in-depth, many of whom are considering a career in AI safety. In the early 2022 round, facilitators were offered a stipend, and this seems likely to be the case for future rounds as well! You can learn more about facilitating in ［https://forum.effectivealtruism.org/posts/WtwMy69JKZeHEvykc/contribute-by-facilitating-the-agi-safety-fundamentals this post from December 2021］.

Mic's Answer to I want to help out AI alignment without necessarily making major life changes. What are some simple things I can do to contribute?


Plex's Answer to Who is Stampy?

Who is Stampy?

Question: Who is Stampy?

Answer: Stampy is a character invented by Robert Miles and developed by the Stampy dev team. He is a stamp collecting robot, a play on clippy from the the [https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer paperclip maximizer] thought experiment.

Stampy is designed to teach people about the risks of unaligned artificial intelligence, and facilitate a community of co-learners who build his FAQ database.

Question: Who is Stampy?

Answer:
Stampy is a character invented by Robert Miles and developed by the Stampy dev team. He is a stamp collecting robot, a play on clippy from the the ［https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer paperclip maximizer］ thought experiment.

Stampy is designed to teach people about the risks of unaligned artificial intelligence, and facilitate a community of co-learners who build his FAQ database.


Plex's Answer to Who created Stampy?

Who created Stampy?

Question: Who created Stampy?

Answer: ꞊꞊꞊Dev team꞊꞊꞊

Name

Vision talk

Github

Trello

Active?

Notes / bio

Aprillion

video

Aprillion

yes

yes

experienced dev (Python, JS, CSS, ...)

Augustus Caesar

yes

AugustusCeasar

yes

soon!

Has some Discord bot experience

Benjamin Herman

no

no (not needed)

no

no

Helping with wiki design/css stuff

ccstan99

no

ccstan99

yes

yes

UI/UX designer

chriscanal

yes

chriscanal

yes

yes

experienced python dev

Damaged

no (not needed)

no (not needed)

no (not needed)

yes

experienced Discord bot dev, but busy with other projects. Can
answer questions.

plex

yes

plexish

yes

yes

MediaWiki, plans, and coordinating people guy

robertskmiles

yes

robertskmiles

yes

yes

you've probably heard of him

Roland

yes

levitation

yes

yes

working on Semantic Search

sct202

yes

no (add when wiki is on github)

yes

yes

PHP dev, helping with wiki extensions

Social Christancing

yes

chrisrimmer

yes

maybe

experienced linux sysadmin

sudonym

yes

jmccuen

yes

yes

systems architect, has set up a lot of things

tayler6000

yes

tayler6000

no

yes

Python and PHP dev, PenTester, works on Discord bot

꞊꞊꞊Editors꞊꞊꞊
([https://stampy.ai/wiki/Special:FormEdit/Answer/Plex%27s_Answer_to_Who_created_Stampy%3F add yourselves])

Question: Who created Stampy?

Answer:
꞊꞊꞊Dev team꞊꞊꞊

Name

Vision talk

Github

Trello

Active?

Notes / bio

Aprillion

video

Aprillion

yes

yes

experienced dev (Python, JS, CSS, ...)

Augustus Caesar

yes

AugustusCeasar

yes

soon!

Has some Discord bot experience

Benjamin Herman

no

no (not needed)

no

no

Helping with wiki design/css stuff

ccstan99

no

ccstan99

yes

yes

UI/UX designer

chriscanal

yes

chriscanal

yes

yes

experienced python dev

Damaged

no (not needed)

no (not needed)

no (not needed)

yes

experienced Discord bot dev, but busy with other projects. Can
answer questions.

plex

yes

plexish

yes

yes

MediaWiki, plans, and coordinating people guy

robertskmiles

yes

robertskmiles

yes

yes

you've probably heard of him

Roland

yes

levitation

yes

yes

working on Semantic Search

sct202

yes

no (add when wiki is on github)

yes

yes

PHP dev, helping with wiki extensions

Social Christancing

yes

chrisrimmer

yes

maybe

experienced linux sysadmin

sudonym

yes

jmccuen

yes

yes

systems architect, has set up a lot of things

tayler6000

yes

tayler6000

no

yes

Python and PHP dev, PenTester, works on Discord bot

꞊꞊꞊Editors꞊꞊꞊
(［https://stampy.ai/wiki/Special:FormEdit/Answer/Plex%27s_Answer_to_Who_created_Stampy%3F add yourselves］)


I’d like  get deeper in the AI alignment literature. Where should I look?

Question: I'd like  get deeper in the AI alignment literature. Where should I look?

Answer: The [https://www.eacambridge.org/technical-alignment-curriculum AGI Safety Fundamentals Course] is a arguably the best way to get up to speed on alignment, you can sign up to go through it with many other people studying and mentorship or read their materials independently.

Other great ways to explore include:
* [https://axrp.net/ AXRP] is a podcast with high quality interviews with top alignment researchers.
* The [https://ai-safety-papers.quantifieduncertainty.org/ AI Safety Papers database] is a search and browsing interface for most of the transformative AI literature.
* Reading posts on the [https://www.alignmentforum.org/ Alignment Forum] can be valuable (see their [https://www.alignmentforum.org/library curated posts] and [https://www.alignmentforum.org/tag/ai tags]).
* Taking a deep dive into Yudkowsky's models of the challenges to aligned AI, via the [https://arbital.greaterwrong.com/explore/ai_alignment/ Arbital Alignment pages].
* Signing up to the [https://rohinshah.com/alignment-newsletter/ Alignment Newsletter] for an overview of current developments, and reading through some of the archives (or listening to [https://alignment-newsletter.libsyn.com/ the podcast]).
* Reading some of [[Where can I learn about AI alignment?┊the introductory books]].
* More on [https://www.aisafetysupport.org/resources/lots-of-links#h.6s2gcz1p5l6z AI Safety Support's list of links],  [https://docs.google.com/spreadsheets/d/1QSEWjXZuqmG6ORkig84V4sFCldIntyuQj7yq3gkDo0U/edit#gid꞊0 Nonlinear's list of technical courses, reading lists, and curriculums], [https://stampy.ai/wiki/Canonical_answers Stampy's canonical answers list], and [https://vkrakovna.wordpress.com/ai-safety-resources/ Vika's resources list].

You might also want to consider reading [https://www.lesswrong.com/rationality Rationality: A-Z] which covers a lot of skills that are valuable to acquire for people trying to think about large and complex issues, with [https://www.amazon.co.uk/Does-Not-Hate-You-Superintelligence/dp/<PHONE> The Rationalist's Guide to the Galaxy] available as a shorter and more accessible AI-focused option.

Question: I’d like  get deeper in the AI alignment literature. Where should I look?

Answer:
The ［https://www.eacambridge.org/technical-alignment-curriculum AGI Safety Fundamentals Course］ is a arguably the best way to get up to speed on alignment, you can sign up to go through it with many other people studying and mentorship or read their materials independently.

Other great ways to explore include:
* ［https://axrp.net/ AXRP］ is a podcast with high quality interviews with top alignment researchers.
* The ［https://ai-safety-papers.quantifieduncertainty.org/ AI Safety Papers database］ is a search and browsing interface for most of the transformative AI literature.
* Reading posts on the ［https://www.alignmentforum.org/ Alignment Forum］ can be valuable (see their ［https://www.alignmentforum.org/library curated posts］ and ［https://www.alignmentforum.org/tag/ai tags］).
* Taking a deep dive into Yudkowsky's models of the challenges to aligned AI, via the ［https://arbital.greaterwrong.com/explore/ai_alignment/ Arbital Alignment pages］.
* Signing up to the ［https://rohinshah.com/alignment-newsletter/ Alignment Newsletter］ for an overview of current developments, and reading through some of the archives (or listening to ［https://alignment-newsletter.libsyn.com/ the podcast］).
* Reading some of ［［Where can I learn about AI alignment?┊the introductory books］］.
* More on ［https://www.aisafetysupport.org/resources/lots-of-links#h.6s2gcz1p5l6z AI Safety Support's list of links］,  ［https://docs.google.com/spreadsheets/d/1QSEWjXZuqmG6ORkig84V4sFCldIntyuQj7yq3gkDo0U/edit#gid꞊0 Nonlinear's list of technical courses, reading lists, and curriculums］, ［https://stampy.ai/wiki/Canonical_answers Stampy's canonical answers list］, and ［https://vkrakovna.wordpress.com/ai-safety-resources/ Vika's resources list］.

You might also want to consider reading ［https://www.lesswrong.com/rationality Rationality: A-Z］ which covers a lot of skills that are valuable to acquire for people trying to think about large and complex issues, with ［https://www.amazon.co.uk/Does-Not-Hate-You-Superintelligence/dp/1474608795 The Rationalist's Guide to the Galaxy］ available as a shorter and more accessible AI-focused option.

Plex's Answer to I’d like to get deeper into the AI alignment literature. Where should I look?


Why does AI takeoff speed matter?

Question: Why does AI takeoff speed matter?

Answer: A slow takeoff over decades or centuries might give us enough time to worry about superintelligence during some indefinite "later", making current planning more like worrying about "overpopulation on Mars". But a moderate or hard takeoff means there wouldn't be enough time to deal with the problem as it occurs, suggesting a role for preemptive planning.

As an aside, let's take the "overpopulation on Mars" comparison seriously. Suppose Mars has a carrying capacity of 10 billion people, and we decide it makes sense to worry about overpopulation on Mars only once it is 75% of the way to its limit. Start with 100 colonists who double every twenty years. By the second generation there are 200 colonists; by the third, 400. Mars reaches 75% of its carrying capacity after 458 years, and crashes into its population limit after 464 years. So there were 464 years in which the Martians could have solved the problem, but they insisted on waiting until there were only six years left. Good luck solving a planetwide population crisis in six years. The moral of the story is that exponential trends move faster than you think and you need to start worrying about them early.

Question: Why does AI takeoff speed matter?

Answer:
A slow takeoff over decades or centuries might give us enough time to worry about superintelligence during some indefinite “later”, making current planning more like worrying about “overpopulation on Mars”. But a moderate or hard takeoff means there wouldn’t be enough time to deal with the problem as it occurs, suggesting a role for preemptive planning.

As an aside, let’s take the “overpopulation on Mars” comparison seriously. Suppose Mars has a carrying capacity of 10 billion people, and we decide it makes sense to worry about overpopulation on Mars only once it is 75% of the way to its limit. Start with 100 colonists who double every twenty years. By the second generation there are 200 colonists; by the third, 400. Mars reaches 75% of its carrying capacity after 458 years, and crashes into its population limit after 464 years. So there were 464 years in which the Martians could have solved the problem, but they insisted on waiting until there were only six years left. Good luck solving a planetwide population crisis in six years. The moral of the story is that exponential trends move faster than you think and you need to start worrying about them early.

Answer to Why does AI takeoff speed matter?


Question: At a high level, what is the challenge of alignment that we must meet  secure a good future?

Answer: We're facing the challenge of  "[https://publicism.info/philosophy/superintelligence/16.html Philosophy With A Deadline]".

Many of the problems surrounding superintelligence are the sorts of problems philosophers have been dealing with for centuries. To what degree is meaning inherent in language, versus something that requires external context? How do we translate between the logic of formal systems and normal ambiguous human speech? Can morality be reduced to a set of ironclad rules, and if not, how do we know what it is at all?

Existing answers to these questions are enlightening but nontechnical. The theories of Aristotle, Kant, Mill, Wittgenstein, Quine, and others can help people gain insight into these questions, but are far from formal. Just as a good textbook can help an American learn Chinese, but cannot be encoded into machine language to make a Chinese-speaking computer, so the philosophies that help humans are only a starting point for the project of computers that understand us and share our values.

The field of AI alignment combines formal logic, mathematics, computer science, cognitive science, and philosophy in order to advance that project.

This is the philosophy; the other half of Bostrom's formulation is the deadline. Traditional philosophy has been going on almost three thousand years; machine goal alignment has until the advent of superintelligence, a nebulous event which may be anywhere from a decades to centuries away.

If the alignment problem doesn't get adequately addressed by then, we are likely to see poorly aligned superintelligences that are unintentionally hostile to the human race, with some of the catastrophic outcomes mentioned above. This is why so many scientists and entrepreneurs are urging quick action on getting machine goal alignment research up to an adequate level.

If it turns out that superintelligence is centuries away and such research is premature, little will have been lost. But if our projections were too optimistic, and superintelligence is imminent, then doing such research now rather than later becomes vital.

Question: At a high level, what is the challenge of alignment that we must meet  secure a good future?

Answer:
We’re facing the challenge of  “［https://publicism.info/philosophy/superintelligence/16.html Philosophy With A Deadline］”.

Many of the problems surrounding superintelligence are the sorts of problems philosophers have been dealing with for centuries. To what degree is meaning inherent in language, versus something that requires external context? How do we translate between the logic of formal systems and normal ambiguous human speech? Can morality be reduced to a set of ironclad rules, and if not, how do we know what it is at all?

Existing answers to these questions are enlightening but nontechnical. The theories of Aristotle, Kant, Mill, Wittgenstein, Quine, and others can help people gain insight into these questions, but are far from formal. Just as a good textbook can help an American learn Chinese, but cannot be encoded into machine language to make a Chinese-speaking computer, so the philosophies that help humans are only a starting point for the project of computers that understand us and share our values.

The field of AI alignment combines formal logic, mathematics, computer science, cognitive science, and philosophy in order to advance that project.

This is the philosophy; the other half of Bostrom’s formulation is the deadline. Traditional philosophy has been going on almost three thousand years; machine goal alignment has until the advent of superintelligence, a nebulous event which may be anywhere from a decades to centuries away.

If the alignment problem doesn’t get adequately addressed by then, we are likely to see poorly aligned superintelligences that are unintentionally hostile to the human race, with some of the catastrophic outcomes mentioned above. This is why so many scientists and entrepreneurs are urging quick action on getting machine goal alignment research up to an adequate level.

If it turns out that superintelligence is centuries away and such research is premature, little will have been lost. But if our projections were too optimistic, and superintelligence is imminent, then doing such research now rather than later becomes vital.

Answer to At a high level, what is the challenge of alignment that we must meet to secure a good future?


Question: If we solve alignment, are we sure of a good future?

Answer: If by "solve alignment" you mean build a sufficiently performance-competitive superintelligence which has the goal of [https://www.lesswrong.com/tag/coherent-extrapolated-volition Coherent Extrapolated Volition] or something else which captures human values, then yes. It would be able to deploy technology near the limits of physics (e.g. [https://en.wikipedia.org/wiki/Atomically_precise_manufacturing atomically precise manufacturing]) to solve most of the other problems which face us, and steer the future towards a highly positive path for [https://en.wikipedia.org/wiki/Timeline_of_the_far_future perhaps many billions of years] until the [https://en.wikipedia.org/wiki/Heat_death_of_the_universe heat death of the universe] (barring more esoteric x-risks like encounters with advanced hostile civilizations, [https://en.wikipedia.org/wiki/False_vacuum_decay false vacuum decay], or [https://arxiv.org/ftp/arxiv/papers/1905/1905.05792.pdf simulation shutdown]).

However, if you only have alignment of a superintelligence to a single human you still have the risk of misuse, so this should be at most a short-term solution. For example, what if Google creates a superintelligent AI, and it listens to the CEO of Google, and it's programmed to do everything exactly the way the CEO of Google would want? Even assuming that the CEO of Google has no hidden unconscious desires affecting the AI in unpredictable ways, this gives one person a lot of power.

Question: If we solve alignment, are we sure of a good future?

Answer:
If by “solve alignment” you mean build a sufficiently performance-competitive superintelligence which has the goal of ［https://www.lesswrong.com/tag/coherent-extrapolated-volition Coherent Extrapolated Volition］ or something else which captures human values, then yes. It would be able to deploy technology near the limits of physics (e.g. ［https://en.wikipedia.org/wiki/Atomically_precise_manufacturing atomically precise manufacturing］) to solve most of the other problems which face us, and steer the future towards a highly positive path for ［https://en.wikipedia.org/wiki/Timeline_of_the_far_future perhaps many billions of years］ until the ［https://en.wikipedia.org/wiki/Heat_death_of_the_universe heat death of the universe］ (barring more esoteric x-risks like encounters with advanced hostile civilizations, ［https://en.wikipedia.org/wiki/False_vacuum_decay false vacuum decay］, or ［https://arxiv.org/ftp/arxiv/papers/1905/1905.05792.pdf simulation shutdown］).

However, if you only have alignment of a superintelligence to a single human you still have the risk of misuse, so this should be at most a short-term solution. For example, what if Google creates a superintelligent AI, and it listens to the CEO of Google, and it’s programmed to do everything exactly the way the CEO of Google would want? Even assuming that the CEO of Google has no hidden unconscious desires affecting the AI in unpredictable ways, this gives one person a lot of power.

Plex's Answer to If we solve alignment, are we sure of a good future?


Question: Why might a maximizing AI cause bad outcomes?

Answer: Computers only do what you tell them. But any programmer knows that this is precisely the problem: computers do exactly what you tell them, with no common sense or attempts to interpret what the instructions really meant. If you tell a human to cure cancer, they will instinctively understand how this interacts with other desires and laws and moral rules; if a maximizing AI acquires a goal of trying to cure cancer, it will literally just want to cure cancer.

Define a closed-ended goal as one with a clear endpoint, and an open-ended goal as one to do something as much as possible. For example "find the first one hundred digits of pi" is a closed-ended goal; "find as many digits of pi as you can within one year" is an open-ended goal. According to many computer scientists, giving a superintelligence an open-ended goal without activating human instincts and counterbalancing considerations will usually lead to disaster.

To take a deliberately extreme example: suppose someone programs a superintelligence to calculate as many digits of pi as it can within one year. And suppose that, with its current computing power, it can calculate one trillion digits during that time. It can either accept one trillion digits, or spend a month trying to figure out how to get control of the TaihuLight supercomputer, which can calculate two hundred times faster. Even if it loses a little bit of time in the effort, and even if there's a small chance of failure, the payoff – two hundred trillion digits of pi, compared to a mere one trillion – is enough to make the attempt. But on the same basis, it would be even better if the superintelligence could control every computer in the world and set it to the task. And it would be better still if the superintelligence controlled human civilization, so that it could direct humans to build more computers and speed up the process further.

Now we're in a situation where a superintelligence wants to take over the world. Taking over the world allows it to calculate more digits of pi than any other option, so without an architecture based around understanding human instincts and counterbalancing considerations, even a goal like "calculate as many digits of pi as you can" would be potentially dangerous.

Question: Why might a maximizing AI cause bad outcomes?

Answer:
Computers only do what you tell them. But any programmer knows that this is precisely the problem: computers do exactly what you tell them, with no common sense or attempts to interpret what the instructions really meant. If you tell a human to cure cancer, they will instinctively understand how this interacts with other desires and laws and moral rules; if a maximizing AI acquires a goal of trying to cure cancer, it will literally just want to cure cancer.

Define a closed-ended goal as one with a clear endpoint, and an open-ended goal as one to do something as much as possible. For example “find the first one hundred digits of pi” is a closed-ended goal; “find as many digits of pi as you can within one year” is an open-ended goal. According to many computer scientists, giving a superintelligence an open-ended goal without activating human instincts and counterbalancing considerations will usually lead to disaster.

To take a deliberately extreme example: suppose someone programs a superintelligence to calculate as many digits of pi as it can within one year. And suppose that, with its current computing power, it can calculate one trillion digits during that time. It can either accept one trillion digits, or spend a month trying to figure out how to get control of the TaihuLight supercomputer, which can calculate two hundred times faster. Even if it loses a little bit of time in the effort, and even if there’s a small chance of failure, the payoff – two hundred trillion digits of pi, compared to a mere one trillion – is enough to make the attempt. But on the same basis, it would be even better if the superintelligence could control every computer in the world and set it to the task. And it would be better still if the superintelligence controlled human civilization, so that it could direct humans to build more computers and speed up the process further.

Now we’re in a situation where a superintelligence wants to take over the world. Taking over the world allows it to calculate more digits of pi than any other option, so without an architecture based around understanding human instincts and counterbalancing considerations, even a goal like “calculate as many digits of pi as you can” would be potentially dangerous.

Plex's Answer to Why might a maximizing AI cause bad outcomes?


Question: Where can I find questions  answer for Stampy?

Answer: '''[[Answer questions]]''' collects all the questions we definitely want answers to, browse there and see if you know how to answer any of them.

Question: Where can I find questions  answer for Stampy?

Answer: '''［［Answer questions］］''' collects all the questions we definitely want answers to, browse there and see if you know how to answer any of them.

Plex's Answer to Where can I find questions to answer for Stampy?


Question: Can people contribute  alignment by using proof assistants  generate formal proofs?

Answer: 80k links to an article on [https://forum.effectivealtruism.org/posts/4rMxiyPTPdzaFMyGm/high-impact-careers-in-formal-verification-artificial high impact careers in formal verification] in the few paragraphs they've written about formal verification.

Some other notes

* https://github.com/deepmind/cartesian-frames I emailed Scott about doing this in coq before this repo was published and he said "I wouldn't personally find such a software useful but sounds like a valuable exercise for the implementer" or something like this.
* When I mentioned the possibility of rolling some of infrabayesianism in coq to diffractor he wasn't like "omg we really need someone to do that" he was just like "oh that sounds cool" -- I never got around to it, if I would I'd talk to vanessa and diffractor about weakening/particularizing stuff beforehand.
* if you extrapolate a pattern from those two examples, you start to think that agent foundations is the principle area of interest with proof assistants! and again- does the proof assistant exercise advance the research or provide a nutritious exercise to the programmer?
* A sketch of a more prosaic scenario in which proof assistants play a role is "someone proposes isInnerAligned : GradientDescent -> Prop and someone else implements a galaxybrained  new type theory/tool in which gradient descent is a primitive (whatever that means)", when I mentioned this scenario to Buck he said "yeah if that happened I'd direct all the engineers at redwood to making that tool easier to use", when I mentioned that scenario to Evan about a year ago he said didn't seem to think it was remotely plausible. probably a nonstarter.

Question: Can people contribute  alignment by using proof assistants  generate formal proofs?

Answer:
80k links to an article on ［https://forum.effectivealtruism.org/posts/4rMxiyPTPdzaFMyGm/high-impact-careers-in-formal-verification-artificial high impact careers in formal verification］ in the few paragraphs they've written about formal verification.

Some other notes

* https://github.com/deepmind/cartesian-frames I emailed Scott about doing this in coq before this repo was published and he said "I wouldn't personally find such a software useful but sounds like a valuable exercise for the implementer" or something like this.
* When I mentioned the possibility of rolling some of infrabayesianism in coq to diffractor he wasn't like "omg we really need someone to do that" he was just like "oh that sounds cool" -- I never got around to it, if I would I'd talk to vanessa and diffractor about weakening/particularizing stuff beforehand.
* if you extrapolate a pattern from those two examples, you start to think that agent foundations is the principle area of interest with proof assistants! and again- does the proof assistant exercise advance the research or provide a nutritious exercise to the programmer?
* A sketch of a more prosaic scenario in which proof assistants play a role is "someone proposes isInnerAligned : GradientDescent -> Prop and someone else implements a galaxybrained  new type theory/tool in which gradient descent is a primitive (whatever that means)", when I mentioned this scenario to Buck he said "yeah if that happened I'd direct all the engineers at redwood to making that tool easier to use", when I mentioned that scenario to Evan about a year ago he said didn't seem to think it was remotely plausible. probably a nonstarter.

Quinn's Answer to Can people contribute to alignment by using proof assistants to generate formal proofs?


Question: What is a canonical question on Stampy's Wiki?

Answer: '''[[Canonical questions]]''' are the questions which we've checked are in [[scope]] and not duplicates, so we want answers to them. They may be edited to represent a class of question more broadly, rather than keeping all their idosyncracies. Once they're answered canonically Stampy will serve them to readers.

Question: What is a canonical question on Stampy's Wiki?

Answer: '''［［Canonical questions］］''' are the questions which we've checked are in ［［scope］］ and not duplicates, so we want answers to them. They may be edited to represent a class of question more broadly, rather than keeping all their idosyncracies. Once they're answered canonically Stampy will serve them to readers.

Plex's Answer to What is a canonical question on Stampy's Wiki?


We’re going  merge with the machines so this will never be a problem, right?

Question: We're going  merge with the machines so this will never be a problem, right?

Answer: The concept of "merging with machines," as popularized by Ray Kurzweil, is the idea that we will be able to put computerized elements into our brains that enhance us to the point where we ourselves are the AI, instead of creating AI outside of ourselves.

While this is a possible outcome, there is little reason to suspect that it is the most probable. The amount of computing power in your smart-phone took up an entire room of servers 30 years ago. Computer technology starts big, and then gets refined. Therefore, if "merging with the machines" requires hardware that can fit inside our brain, it may lag behind the first generations of the technology being developed. This concept of merging also supposes that we can even figure out how to implant computer chips that interface with our brain in the first place, we can do it before the invention of advanced AI, society will accept it, and that computer implants can actually produce major intelligence gains in the human brain. Even if we could successfully enhance ourselves with brain implants before the invention of Artificial Superintelligence (ASI), there is no way to guarantee that this would protect us from negative outcomes, and an ASI with ill-defined goals could still pose a threat to us.

It's not that Ray Kurzweil's ideas are impossible, it's just that his predictions are too specific, confident, and reliant on strange assumptions.

Question: We’re going  merge with the machines so this will never be a problem, right?

Answer:
The concept of “merging with machines,” as popularized by Ray Kurzweil, is the idea that we will be able to put computerized elements into our brains that enhance us to the point where we ourselves are the AI, instead of creating AI outside of ourselves.

While this is a possible outcome, there is little reason to suspect that it is the most probable. The amount of computing power in your smart-phone took up an entire room of servers 30 years ago. Computer technology starts big, and then gets refined. Therefore, if “merging with the machines” requires hardware that can fit inside our brain, it may lag behind the first generations of the technology being developed. This concept of merging also supposes that we can even figure out how to implant computer chips that interface with our brain in the first place, we can do it before the invention of advanced AI, society will accept it, and that computer implants can actually produce major intelligence gains in the human brain. Even if we could successfully enhance ourselves with brain implants before the invention of Artificial Superintelligence (ASI), there is no way to guarantee that this would protect us from negative outcomes, and an ASI with ill-defined goals could still pose a threat to us.

It's not that Ray Kurzweil's ideas are impossible, it's just that his predictions are too specific, confident, and reliant on strange assumptions.

Answer to We’re going to merge with the machines so this will never be a problem, right?


Could we program an AI  automatically shut down if it starts doing things we don’t want it to?

Question: Could we program an AI  automatically shut down if it starts doing things we don't want it to?

Answer: One thing that might make your AI system safer is to include an off switch. If it ever does anything we don't like, we can turn it off. This implicitly assumes that we'll be able to turn it off before things get bad, which might be false in a world where the AI thinks much faster than humans. Even assuming that we'll notice in time, off switches turn out to not have the properties you would want them to have.

Humans have a lot of off switches. Humans also have a strong preference to not be turned off; they defend their off switches when other people try to press them. One possible reason for this is because humans prefer not to die, but there are other reasons.

Suppose that there's a parent that cares nothing for their own life and cares only for the life of their child. If you tried to turn that parent off, they would try and stop you. They wouldn't try to stop you because they intrinsically wanted to be turned off, but rather because there are fewer people to protect their child if they were turned off. People that want a world to look a certain shape will not want to be turned off because then it will be less likely for the world to look that shape; a parent that wants their child to be protected will protect themselves to continue protecting their child.

For this reason, it turns out to be difficult to install an off switch on a powerful AI system in a way that doesn't result in the AI preventing itself from being turned off.

Ideally, you would want a system that knows that it should stop doing whatever it's doing when someone tries to turn it off. The technical term for this is 'corrigibility'; roughly speaking, an AI system is corrigible if it doesn't resist human attempts to help and correct it. [https://intelligence.org/ People] are working hard on trying to make this possible, but it's currently not clear how we would do this even in simple cases.

Question: Could we program an AI  automatically shut down if it starts doing things we don’t want it to?

Answer:
One thing that might make your AI system safer is to include an off switch. If it ever does anything we don’t like, we can turn it off. This implicitly assumes that we’ll be able to turn it off before things get bad, which might be false in a world where the AI thinks much faster than humans. Even assuming that we’ll notice in time, off switches turn out to not have the properties you would want them to have.

Humans have a lot of off switches. Humans also have a strong preference to not be turned off; they defend their off switches when other people try to press them. One possible reason for this is because humans prefer not to die, but there are other reasons.

Suppose that there’s a parent that cares nothing for their own life and cares only for the life of their child. If you tried to turn that parent off, they would try and stop you. They wouldn’t try to stop you because they intrinsically wanted to be turned off, but rather because there are fewer people to protect their child if they were turned off. People that want a world to look a certain shape will not want to be turned off because then it will be less likely for the world to look that shape; a parent that wants their child to be protected will protect themselves to continue protecting their child.

For this reason, it turns out to be difficult to install an off switch on a powerful AI system in a way that doesn’t result in the AI preventing itself from being turned off.

Ideally, you would want a system that knows that it should stop doing whatever it’s doing when someone tries to turn it off. The technical term for this is ‘corrigibility’; roughly speaking, an AI system is corrigible if it doesn’t resist human attempts to help and correct it. ［https://intelligence.org/ People］ are working hard on trying to make this possible, but it’s currently not clear how we would do this even in simple cases.

Plex's Answer to Could we program an AI to automatically shut down if it starts doing things we don’t want it to?


Question: Why should we prepare for human-level AI technology now rather than decades down the line when it's closer?

Answer: First, even "narrow" AI systems, which approach or surpass human intelligence in a small set of capabilities (such as image or voice recognition) already raise important questions regarding their impact on society. Making autonomous vehicles safe, analyzing the strategic and ethical dimensions of autonomous weapons, and the effect of AI on the global employment and economic systems are three examples. Second, the longer-term implications of human or super-human artificial intelligence are dramatic, and there is no consensus on how quickly such capabilities will be developed. Many experts believe there is a chance it could happen rather soon, making it imperative to begin investigating long-term safety issues now, if only to get a better sense of how much early progress is actually possible.

Question: Why should we prepare for human-level AI technology now rather than decades down the line when it’s closer?

Answer: First, even “narrow” AI systems, which approach or surpass human intelligence in a small set of capabilities (such as image or voice recognition) already raise important questions regarding their impact on society. Making autonomous vehicles safe, analyzing the strategic and ethical dimensions of autonomous weapons, and the effect of AI on the global employment and economic systems are three examples. Second, the longer-term implications of human or super-human artificial intelligence are dramatic, and there is no consensus on how quickly such capabilities will be developed. Many experts believe there is a chance it could happen rather soon, making it imperative to begin investigating long-term safety issues now, if only to get a better sense of how much early progress is actually possible.

Answer to Why should we prepare for human-level AI technology now rather than decades down the line when it’s closer?


Question: What is a follow-up question on Stampy's Wiki?

Answer: '''Follow-up questions''' are responses to an [[answer]] which reader might have, either because they want more information or are providing information to Stampy about what they're looking for. We don't expect to have great coverage of the former for a long time because there will be so many, but hopefully we'll be able to handle some of the most common ones.

Question: What is a follow-up question on Stampy's Wiki?

Answer: '''Follow-up questions''' are responses to an ［［answer］］ which reader might have, either because they want more information or are providing information to Stampy about what they're looking for. We don't expect to have great coverage of the former for a long time because there will be so many, but hopefully we'll be able to handle some of the most common ones.

Plex's Answer to What is a follow-up question on Stampy's Wiki?


How might AGI kill people?

Question: How might AGI kill people?

Answer: If we pose a serious threat, it could hack our weapons systems and turn them against us. Future militaries are much more vulnerable to this due to rapidly progressing autonomous weapons. There's also the option of creating bioweapons and distributing them to the most unstable groups you can find, tricking nations into WW3, or dozens of other things an agent many times smarter than any human with the ability to develop arbitrary technology, hack things (including communications), and manipulate people, or many other possibilities that something smarter than a human could think up. More can be found [https://www.lesswrong.com/posts/pxGYZs2zHJNHvWY5b/request-for-concrete-ai-takeover-mechanisms here].

If we are not a threat, in the course of pursuing its goals it may consume vital resources that humans need (e.g. using land for solar panels instead of farm crops). This video goes into more detail:

(youtube)ZeecOKBus3Q(/youtube)

Question: How might AGI kill people?

Answer:
If we pose a serious threat, it could hack our weapons systems and turn them against us. Future militaries are much more vulnerable to this due to rapidly progressing autonomous weapons. There’s also the option of creating bioweapons and distributing them to the most unstable groups you can find, tricking nations into WW3, or dozens of other things an agent many times smarter than any human with the ability to develop arbitrary technology, hack things (including communications), and manipulate people, or many other possibilities that something smarter than a human could think up. More can be found ［https://www.lesswrong.com/posts/pxGYZs2zHJNHvWY5b/request-for-concrete-ai-takeover-mechanisms here］.

If we are not a threat, in the course of pursuing its goals it may consume vital resources that humans need (e.g. using land for solar panels instead of farm crops). This video goes into more detail:

(youtube)ZeecOKBus3Q(/youtube)

Plex's Answer to How might AGI kill people?