Question: Wouldn't it be a good thing for humanity  die out?

Answer: In the words of [https://mindingourway.com/a-torch-in-darkness/ Nate Soares]:

I don't expect humanity to survive much longer.

Often, when someone learns this, they say:

"Eh, I think that would be all right."

So allow me to make this very clear: it would not be "all right."

Imagine a little girl running into the road to save her pet dog. Imagine she succeeds, only to be hit by a car herself. Imagine she lives only long enough to die in pain.

Though you may imagine this thing, you cannot feel the full tragedy. You can't comprehend the rich inner life of that child. You can't understand her potential; your mind is not itself large enough to contain the sadness of an entire life cut short.

You can only catch a glimpse of what is lost—

—when one single human being dies.

Now tell me again how it would be "all right" if every single person were to die at once.

Many people, when they picture the end of humankind, pattern match the idea to some romantic tragedy, where humans, with all their hate and all their avarice, had been unworthy of the stars since the very beginning, and deserved their fate. A sad but poignant ending to our tale.

And indeed, there are many parts of human nature that I hope we leave behind before we venture to the heavens. But in our nature is also everything worth bringing with us. Beauty and curiosity and love, a capacity for fun and growth and joy: these are our birthright, ours to bring into the barren night above.

Calamities seem more salient when unpacked. It is far harder to kill a hundred people in their sleep, with a knife, than it is to order a nuclear bomb dropped on Hiroshima. Your brain can't multiply, you see: it can only look at a hypothetical image of a broken city and decide it's not that bad. It can only conjure an image of a barren planet and say "eh, we had it coming."

But if you unpack the scenario, if you try to comprehend all the lives snuffed out, all the children killed, the final spark of human joy and curiosity extinguished, all our potential squandered…

I promise you that the extermination of humankind would be horrific.

Question: Wouldn't it be a good thing for humanity  die out?

Answer:
In the words of ［https://mindingourway.com/a-torch-in-darkness/ Nate Soares］:

I don’t expect humanity to survive much longer.

Often, when someone learns this, they say:

"Eh, I think that would be all right."

So allow me to make this very clear: it would not be "all right."

Imagine a little girl running into the road to save her pet dog. Imagine she succeeds, only to be hit by a car herself. Imagine she lives only long enough to die in pain.

Though you may imagine this thing, you cannot feel the full tragedy. You can’t comprehend the rich inner life of that child. You can’t understand her potential; your mind is not itself large enough to contain the sadness of an entire life cut short.

You can only catch a glimpse of what is lost—

—when one single human being dies.

Now tell me again how it would be "all right" if every single person were to die at once.

Many people, when they picture the end of humankind, pattern match the idea to some romantic tragedy, where humans, with all their hate and all their avarice, had been unworthy of the stars since the very beginning, and deserved their fate. A sad but poignant ending to our tale.

And indeed, there are many parts of human nature that I hope we leave behind before we venture to the heavens. But in our nature is also everything worth bringing with us. Beauty and curiosity and love, a capacity for fun and growth and joy: these are our birthright, ours to bring into the barren night above.

Calamities seem more salient when unpacked. It is far harder to kill a hundred people in their sleep, with a knife, than it is to order a nuclear bomb dropped on Hiroshima. Your brain can’t multiply, you see: it can only look at a hypothetical image of a broken city and decide it’s not that bad. It can only conjure an image of a barren planet and say "eh, we had it coming."

But if you unpack the scenario, if you try to comprehend all the lives snuffed out, all the children killed, the final spark of human joy and curiosity extinguished, all our potential squandered…

I promise you that the extermination of humankind would be horrific.

Plex's Answer to Wouldn't it be a good thing for humanity to die out?


Question: Why might a superintelligent AI be dangerous?

Answer: A commonly heard argument goes: yes, a superintelligent AI might be far smarter than Einstein, but it's still just one program, sitting in a supercomputer somewhere. That could be bad if an enemy government controls it and asks it to help invent superweapons – but then the problem is the enemy government, not the AI ''per se''. Is there any reason to be afraid of the AI itself? Suppose the AI did appear to be hostile, suppose it even wanted to take over the world: why should we think it has any chance of doing so?

There are numerous carefully thought-out AGI-related scenarios which could result in the accidental extinction of humanity. But rather than focussing on any of these individually, it might be more helpful to think in general terms.

"Transistors can fire about 10 million times faster than human brain cells, so it's possible we'll eventually have digital minds operating 10 million times faster than us, meaning from a decision-making perspective we'd look to them like stationary objects, like plants or rocks...  To give you a sense, [https://vimeo.com/83664407 here]'s what humans look like when slowed down by only around 100x."''

Watch that, and now try to imagine advanced AI technology running for a single year around the world, making decisions and taking actions 10 million times faster than we can. That year for us becomes 10 million subjective years for the AI, in which "...there are these nearly-stationary plant-like or rock-like "human" objects around that could easily be taken apart for, say, biofuel or carbon atoms, if you could just get started building a human-disassembler. Visualizing things this way, you can start to see all the ways that a digital civilization can develop very quickly into a situation where there are no humans left alive, just as human civilization doesn't show much regard for plants or wildlife or insects."
[https://acritch.com/ Andrew Critch] - [https://www.lesswrong.com/posts/Ccsx339LE9Jhoii9K/slow-motion-videos-as-ai-risk-intuition-pumps Slow Motion Videos as AI Risk Intuition Pumps]

And even putting aside these issues of speed and subjective time, the difference in (intelligence-based) power-to-manipulate-the-world between a self-improving superintelligent AGI and humanity could be far more extreme than the difference in such power between humanity and insects.

"[https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/ AI Could Defeat All Of Us Combined]" is a more in-depth argument by the CEO of [https://en.wikipedia.org/wiki/Open_Philanthropy_(organization) Open Philanthropy].

Question: Why might a superintelligent AI be dangerous?

Answer:
A commonly heard argument goes: yes, a superintelligent AI might be far smarter than Einstein, but it’s still just one program, sitting in a supercomputer somewhere. That could be bad if an enemy government controls it and asks it to help invent superweapons – but then the problem is the enemy government, not the AI ''per se''. Is there any reason to be afraid of the AI itself? Suppose the AI did appear to be hostile, suppose it even wanted to take over the world: why should we think it has any chance of doing so?

There are numerous carefully thought-out AGI-related scenarios which could result in the accidental extinction of humanity. But rather than focussing on any of these individually, it might be more helpful to think in general terms.

"Transistors can fire about 10 million times faster than human brain cells, so it's possible we'll eventually have digital minds operating 10 million times faster than us, meaning from a decision-making perspective we'd look to them like stationary objects, like plants or rocks...  To give you a sense, ［https://vimeo.com/83664407 here］'s what humans look like when slowed down by only around 100x."''

Watch that, and now try to imagine advanced AI technology running for a single year around the world, making decisions and taking actions 10 million times faster than we can. That year for us becomes 10 million subjective years for the AI, in which "...there are these nearly-stationary plant-like or rock-like "human" objects around that could easily be taken apart for, say, biofuel or carbon atoms, if you could just get started building a human-disassembler. Visualizing things this way, you can start to see all the ways that a digital civilization can develop very quickly into a situation where there are no humans left alive, just as human civilization doesn't show much regard for plants or wildlife or insects."
［https://acritch.com/ Andrew Critch］ - ［https://www.lesswrong.com/posts/Ccsx339LE9Jhoii9K/slow-motion-videos-as-ai-risk-intuition-pumps Slow Motion Videos as AI Risk Intuition Pumps］

And even putting aside these issues of speed and subjective time, the difference in (intelligence-based) power-to-manipulate-the-world between a self-improving superintelligent AGI and humanity could be far more extreme than the difference in such power between humanity and insects.

“［https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/ AI Could Defeat All Of Us Combined］” is a more in-depth argument by the CEO of ［https://en.wikipedia.org/wiki/Open_Philanthropy_(organization) Open Philanthropy］.

Answer to Why might a superintelligent AI be dangerous?


Question: Why can't we just make a "child AI" and raise it?

Answer: A potential solution is to create an AI that has the same values and morality as a human by creating a child AI and raising it. There's nothing intrinsically flawed with this procedure. However, this suggestion is deceptive because it sounds simpler than it is.

If you get a chimpanzee baby and raise it in a human family, it does not learn to speak a human language. Human babies can grow into adult humans because the babies have specific properties, e.g. a prebuilt language module that gets activated during childhood.

In order to make a child AI that has the potential to turn into the type of adult AI we would find acceptable, the child AI has to have specific properties. The task of building a child AI with these properties involves building a system that can interpret what humans mean when we try to teach the child to do various tasks. [https://humancompatible.ai/ People] are currently working on ways to program agents that can cooperatively interact with humans to learn what they want.

Question: Why can't we just make a "child AI" and raise it?

Answer:
A potential solution is to create an AI that has the same values and morality as a human by creating a child AI and raising it. There’s nothing intrinsically flawed with this procedure. However, this suggestion is deceptive because it sounds simpler than it is.

If you get a chimpanzee baby and raise it in a human family, it does not learn to speak a human language. Human babies can grow into adult humans because the babies have specific properties, e.g. a prebuilt language module that gets activated during childhood.

In order to make a child AI that has the potential to turn into the type of adult AI we would find acceptable, the child AI has to have specific properties. The task of building a child AI with these properties involves building a system that can interpret what humans mean when we try to teach the child to do various tasks. ［https://humancompatible.ai/ People］ are currently working on ways to program agents that can cooperatively interact with humans to learn what they want.

Answer to Why can't we just make a "child AI" and raise it?


What is causal decision theory?

Question: What is causal decision theory?

Answer:
Causal Decision Theory – CDT – is a branch of [https://www.lesswrong.com/tag/decision-theory decision theory] which advises an agent to take actions which maximize the causal consequences on the probability of desired outcomes [#fn1 1]. As any branch of decision theory, it prescribes taking the action that maximizes [https://www.lesswrong.com/tag/expected-utility expected utility], i.e the action which maximizes the sum of the utility obtained in each outcome weighted by the probability of that outcome occurring, given your action. Different decision theories correspond to different ways of construing this dependence between actions and outcomes. CDT focuses on the causal relations between one's actions and outcomes, whilst [https://www.lesswrong.com/tag/evidential-decision-theory Evidential Decision Theory] – EDT - concerns itself with what an action indicates about the world (which is operationalized by the conditional probability). That is, according to CDT, a rational agent should track the available causal relations linking his actions to the desired outcome and take the action which will better enhance the chances of the desired outcome.

One usual example where EDT and CDT commonly diverge is the [https://www.lesswrong.com/tag/smoking-lesion Smoking lesion]: "Smoking is strongly correlated with lung cancer, but in the world of the Smoker's Lesion this correlation is understood to be the result of a common cause: a genetic lesion that tends to cause both smoking and cancer. Once we fix the presence or absence of the lesion, there is no additional correlation between smoking and cancer. Suppose you prefer smoking without cancer to not smoking without cancer, and prefer smoking with cancer to not smoking with cancer. Should you smoke?" CDT would recommend smoking since there is no causal connection between smoking and cancer. They are both caused by a gene, but have no causal direct connection with each other. EDT, on the other hand, would recommend against smoking, since smoking is an evidence for having the mentioned gene and thus should be avoided.

The core aspect of CDT is mathematically represented by the fact it uses probabilities of conditionals in place of conditional probabilities [#fn2 2]. The probability of a conditional is the probability of the whole conditional being true, where the conditional probability is the probability of the consequent given the antecedent. A conditional probability of B given A - P(B┊A) -, simply implies the [https://www.lesswrong.com/tag/bayesian-probability Bayesian probability] of the event B happening given we known A happened, it's used in EDT. The probability of conditionals – P(A > B) - refers to the probability that the conditional 'A implies B' is true, it is the probability of the contrafactual 'If A, then B' be the case. Since contrafactual analysis is the key tool used to speak about causality, probability of conditionals are said to mirror causal relations. In most cases these two probabilities track each other, and CDT and EDT give the same answers. However, some particular problems have arisen where their predictions for rational action diverge such as the [https://www.lesswrong.com/tag/smoking-lesion Smoking lesion] problem – where CDT seems to give a more reasonable prescription – and [https://www.lesswrong.com/tag/newcomb-s-problem Newcomb's problem] – where CDT seems unreasonable. David Lewis proved [#fn3 3] it's impossible to probabilities of conditionals to always track conditional probabilities. Hence, evidential relations aren't the same as causal relations and CDT and EDT will always diverge in some cases.

References

[http://plato.stanford.edu/entries/decision-causal/ http://plato.stanford.edu/entries/decision-causal/]

Lewis, David. (1981) "Causal Decision Theory," Australasian Journal of Philosophy 59 (1981): 5- 30.

Lewis, D. (1976), "Probabilities of conditionals and conditional probabilities", The Philosophical Review (Duke University Press) 85 (3): 297–315

See also

[https://www.lesswrong.com/tag/decision-theory Decision theory]

[https://www.lesswrong.com/tag/evidential-decision-theory Evidential Decision Theory]

[https://www.lesswrong.com/tag/causal-decision-theory?edit꞊true Edit on LessWrong]

https://www.lesswrong.com/tag/causal-decision-theory?edit꞊true

Question: What is causal decision theory?

Answer:
Causal Decision Theory – CDT – is a branch of ［https://www.lesswrong.com/tag/decision-theory decision theory］ which advises an agent to take actions which maximize the causal consequences on the probability of desired outcomes ［#fn1 1］. As any branch of decision theory, it prescribes taking the action that maximizes ［https://www.lesswrong.com/tag/expected-utility expected utility］, i.e the action which maximizes the sum of the utility obtained in each outcome weighted by the probability of that outcome occurring, given your action. Different decision theories correspond to different ways of construing this dependence between actions and outcomes. CDT focuses on the causal relations between one’s actions and outcomes, whilst ［https://www.lesswrong.com/tag/evidential-decision-theory Evidential Decision Theory］ – EDT - concerns itself with what an action indicates about the world (which is operationalized by the conditional probability). That is, according to CDT, a rational agent should track the available causal relations linking his actions to the desired outcome and take the action which will better enhance the chances of the desired outcome.

One usual example where EDT and CDT commonly diverge is the ［https://www.lesswrong.com/tag/smoking-lesion Smoking lesion］: “Smoking is strongly correlated with lung cancer, but in the world of the Smoker's Lesion this correlation is understood to be the result of a common cause: a genetic lesion that tends to cause both smoking and cancer. Once we fix the presence or absence of the lesion, there is no additional correlation between smoking and cancer. Suppose you prefer smoking without cancer to not smoking without cancer, and prefer smoking with cancer to not smoking with cancer. Should you smoke?” CDT would recommend smoking since there is no causal connection between smoking and cancer. They are both caused by a gene, but have no causal direct connection with each other. EDT, on the other hand, would recommend against smoking, since smoking is an evidence for having the mentioned gene and thus should be avoided.

The core aspect of CDT is mathematically represented by the fact it uses probabilities of conditionals in place of conditional probabilities ［#fn2 2］. The probability of a conditional is the probability of the whole conditional being true, where the conditional probability is the probability of the consequent given the antecedent. A conditional probability of B given A - P(B┊A) -, simply implies the ［https://www.lesswrong.com/tag/bayesian-probability Bayesian probability］ of the event B happening given we known A happened, it’s used in EDT. The probability of conditionals – P(A > B) - refers to the probability that the conditional 'A implies B' is true, it is the probability of the contrafactual ‘If A, then B’ be the case. Since contrafactual analysis is the key tool used to speak about causality, probability of conditionals are said to mirror causal relations. In most cases these two probabilities track each other, and CDT and EDT give the same answers. However, some particular problems have arisen where their predictions for rational action diverge such as the ［https://www.lesswrong.com/tag/smoking-lesion Smoking lesion］ problem – where CDT seems to give a more reasonable prescription – and ［https://www.lesswrong.com/tag/newcomb-s-problem Newcomb's problem］ – where CDT seems unreasonable. David Lewis proved ［#fn3 3］ it's impossible to probabilities of conditionals to always track conditional probabilities. Hence, evidential relations aren’t the same as causal relations and CDT and EDT will always diverge in some cases.

References

［http://plato.stanford.edu/entries/decision-causal/ http://plato.stanford.edu/entries/decision-causal/］

Lewis, David. (1981) "Causal Decision Theory," Australasian Journal of Philosophy 59 (1981): 5- 30.

Lewis, D. (1976), "Probabilities of conditionals and conditional probabilities", The Philosophical Review (Duke University Press) 85 (3): 297–315

See also

［https://www.lesswrong.com/tag/decision-theory Decision theory］

［https://www.lesswrong.com/tag/evidential-decision-theory Evidential Decision Theory］

［https://www.lesswrong.com/tag/causal-decision-theory?edit꞊true Edit on LessWrong］

https://www.lesswrong.com/tag/causal-decision-theory?edit꞊true

Linnea's Answer to What is causal decision theory?


Linnea's Answer to What is an "s-risk"?

What is an "s-risk"?

Question: What is an "s-risk"?

Answer:
(Astronomical) suffering risks, also known as s-risks, are risks of the creation of intense suffering in the far future on an astronomical scale, vastly exceeding all suffering that has existed on Earth so far.

S-risks are an example of [https://www.lesswrong.com/tag/existential-risk existential risk] (also known as x-risks) according to Nick Bostrom's original definition, as they threaten to "permanently and drastically curtail [Earth-originating intelligent life's] potential". Most existential risks are of the form "event E happens which drastically reduces the number of conscious experiences in the future". S-risks therefore serve as a useful reminder that some x-risks are scary because they cause bad experiences, and not just because they prevent good ones.

Within the space of x-risks, we can distinguish x-risks that are s-risks, x-risks involving human extinction, x-risks that involve immense suffering and human extinction, and x-risks that involve neither. For example:

 	extinction risk	non-extinction risk

suffering risk	Misaligned AGI wipes out humans, simulates many suffering alien civilizations.	Misaligned AGI tiles the universe with experiences of severe suffering.

non-suffering risk	Misaligned AGI wipes out humans.	Misaligned AGI keeps humans as "pets," limiting growth but not causing immense suffering.

A related concept is [https://arbital.com/p/hyperexistential_separation/ hyperexistential risk], the risk of "fates worse than death" on an astronomical scale. It is not clear whether all hyperexistential risks are s-risks per se. But arguably all s-risks are hyperexistential, since "tiling the universe with experiences of severe suffering" would likely be worse than death.

There are two [https://wiki.lesswrong.com/wiki/EA EA] organizations with s-risk prevention research as their primary focus: the [https://www.lesswrong.com/tag/center-on-long-term-risk-clr Center on Long-Term Risk] (CLR) and the [https://centerforreducingsuffering.org/ Center for Reducing Suffering]. Much of CLR's work is on suffering-focused [https://wiki.lesswrong.com/wiki/AI_safety AI safety] and [https://www.lesswrong.com/tag/crucial-considerations crucial considerations]. Although to a much lesser extent, the [https://www.lesswrong.com/tag/machine-intelligence-research-institute-miri Machine Intelligence Research Institute] and [https://www.lesswrong.com/tag/future-of-humanity-institute-fhi Future of Humanity Institute] have investigated strategies to prevent s-risks too.

Another approach to reducing s-risk is to "expand the moral circle" [https://magnusvinding.com/2018/09/04/moral-circle-expansion-might-increase-future-suffering/ together] with raising concern for suffering, so that future (post)human civilizations and AI are less likely to [https://www.lesswrong.com/tag/instrumental-value instrumentally] cause suffering to non-human minds such as animals or digital sentience. [http://www.sentienceinstitute.org/ Sentience Institute] works on this value-spreading problem.

See also

[https://www.lesswrong.com/tag/center-on-long-term-risk-clr Center on Long-Term Risk]

[https://www.lesswrong.com/tag/existential-risk Existential risk]

[https://www.lesswrong.com/tag/abolitionism Abolitionism]

[https://wiki.lesswrong.com/wiki/Mind_crime Mind crime]

[https://www.lesswrong.com/tag/utilitarianism Utilitarianism], [https://www.lesswrong.com/tag/hedonism Hedonism]

External links

[https://foundational-research.org/reducing-risks-of-astronomical-suffering-a-neglected-global-priority/ Reducing Risks of Astronomical Suffering: A Neglected Global Priority (FRI)]

[https://foundational-research.org/s-risks-talk-eag-boston-2017/ Introductory talk on s-risks (FRI)]

[https://foundational-research.org/risks-of-astronomical-future-suffering/ Risks of Astronomical Future Suffering (FRI)]

[https://foundational-research.org/files/suffering-focused-ai-safety.pdf Suffering-focused AI safety: Why "fail-safe" measures might be our top intervention PDF (FRI)]

[https://foundational-research.org/artificial-intelligence-and-its-implications-for-future-suffering Artificial Intelligence and Its Implications for Future Suffering (FRI)]

[https://sentience-politics.org/expanding-moral-circle-reduce-suffering-far-future/ Expanding our moral circle to reduce suffering in the far future (Sentience Politics)]

[https://sentience-politics.org/philosophy/the-importance-of-the-future/ The Importance of the Far Future (Sentience Politics)]

[https://www.lesswrong.com/tag/risks-of-astronomical-suffering-s-risks?edit꞊true Edit on LessWrong]

https://www.lesswrong.com/tag/risks-of-astronomical-suffering-s-risks?edit꞊true

Question: What is an "s-risk"?

Answer:
(Astronomical) suffering risks, also known as s-risks, are risks of the creation of intense suffering in the far future on an astronomical scale, vastly exceeding all suffering that has existed on Earth so far.

S-risks are an example of ［https://www.lesswrong.com/tag/existential-risk existential risk］ (also known as x-risks) according to Nick Bostrom's original definition, as they threaten to "permanently and drastically curtail ［Earth-originating intelligent life's］ potential". Most existential risks are of the form "event E happens which drastically reduces the number of conscious experiences in the future". S-risks therefore serve as a useful reminder that some x-risks are scary because they cause bad experiences, and not just because they prevent good ones.

Within the space of x-risks, we can distinguish x-risks that are s-risks, x-risks involving human extinction, x-risks that involve immense suffering and human extinction, and x-risks that involve neither. For example:

 	extinction risk	non-extinction risk

suffering risk	Misaligned AGI wipes out humans, simulates many suffering alien civilizations.	Misaligned AGI tiles the universe with experiences of severe suffering.

non-suffering risk	Misaligned AGI wipes out humans.	Misaligned AGI keeps humans as "pets," limiting growth but not causing immense suffering.

A related concept is ［https://arbital.com/p/hyperexistential_separation/ hyperexistential risk］, the risk of "fates worse than death" on an astronomical scale. It is not clear whether all hyperexistential risks are s-risks per se. But arguably all s-risks are hyperexistential, since "tiling the universe with experiences of severe suffering" would likely be worse than death.

There are two ［https://wiki.lesswrong.com/wiki/EA EA］ organizations with s-risk prevention research as their primary focus: the ［https://www.lesswrong.com/tag/center-on-long-term-risk-clr Center on Long-Term Risk］ (CLR) and the ［https://centerforreducingsuffering.org/ Center for Reducing Suffering］. Much of CLR's work is on suffering-focused ［https://wiki.lesswrong.com/wiki/AI_safety AI safety］ and ［https://www.lesswrong.com/tag/crucial-considerations crucial considerations］. Although to a much lesser extent, the ［https://www.lesswrong.com/tag/machine-intelligence-research-institute-miri Machine Intelligence Research Institute］ and ［https://www.lesswrong.com/tag/future-of-humanity-institute-fhi Future of Humanity Institute］ have investigated strategies to prevent s-risks too.

Another approach to reducing s-risk is to "expand the moral circle" ［https://magnusvinding.com/2018/09/04/moral-circle-expansion-might-increase-future-suffering/ together］ with raising concern for suffering, so that future (post)human civilizations and AI are less likely to ［https://www.lesswrong.com/tag/instrumental-value instrumentally］ cause suffering to non-human minds such as animals or digital sentience. ［http://www.sentienceinstitute.org/ Sentience Institute］ works on this value-spreading problem.

See also

［https://www.lesswrong.com/tag/center-on-long-term-risk-clr Center on Long-Term Risk］

［https://www.lesswrong.com/tag/existential-risk Existential risk］

［https://www.lesswrong.com/tag/abolitionism Abolitionism］

［https://wiki.lesswrong.com/wiki/Mind_crime Mind crime］

［https://www.lesswrong.com/tag/utilitarianism Utilitarianism］, ［https://www.lesswrong.com/tag/hedonism Hedonism］

External links

［https://foundational-research.org/reducing-risks-of-astronomical-suffering-a-neglected-global-priority/ Reducing Risks of Astronomical Suffering: A Neglected Global Priority (FRI)］

［https://foundational-research.org/s-risks-talk-eag-boston-2017/ Introductory talk on s-risks (FRI)］

［https://foundational-research.org/risks-of-astronomical-future-suffering/ Risks of Astronomical Future Suffering (FRI)］

［https://foundational-research.org/files/suffering-focused-ai-safety.pdf Suffering-focused AI safety: Why "fail-safe" measures might be our top intervention PDF (FRI)］

［https://foundational-research.org/artificial-intelligence-and-its-implications-for-future-suffering Artificial Intelligence and Its Implications for Future Suffering (FRI)］

［https://sentience-politics.org/expanding-moral-circle-reduce-suffering-far-future/ Expanding our moral circle to reduce suffering in the far future (Sentience Politics)］

［https://sentience-politics.org/philosophy/the-importance-of-the-future/ The Importance of the Far Future (Sentience Politics)］

［https://www.lesswrong.com/tag/risks-of-astronomical-suffering-s-risks?edit꞊true Edit on LessWrong］

https://www.lesswrong.com/tag/risks-of-astronomical-suffering-s-risks?edit꞊true


What is an "intelligence explosion"?

Question: What is an "intelligence explosion"?

Answer: The intelligence explosion idea was expressed by statistician [http://www.incompleteideas.net/papers/Good65ultraintelligent.pdf I.J. Good in 1965]:

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion', and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make.

The argument is this: Every year, computers surpass human abilities in new ways. A program written in 1956 was able to prove mathematical theorems, and [http://www.cs.cornell.edu/courses/cs4860/2012fa/MacKenzie-TheAutomationOfProof.pdf found a more elegant proof] for one of them than Russell and Whitehead had given in ''Principia Mathematica''. By the late 1990s, 'expert systems' had surpassed human skill for a [http://www.amazon.com/dp/<PHONE>/ wide range of tasks]. In 1997, IBM's Deep Blue computer beat the world chess champion, and in 2011, IBM's Watson computer beat the best human players at a much more complicated game: [http://www.nytimes.com/2011/02/17/science/17jeopardy-watson.html?_r꞊2&ref꞊homepage&src꞊me&pagewanted꞊all Jeopardy!]. Recently, [http://commonsenseatheism.com/wp-content/uploads/2011/02/King-The-Automation-of-Science.pdf a robot named Adam] was programmed with our scientific knowledge about yeast, then posed its own hypotheses, tested them, and assessed the results.

Computers remain far short of human intelligence, but the resources that aid AI design are accumulating (including hardware, large datasets, neuroscience knowledge, and AI theory). We may one day design a machine that surpasses human skill at designing artificial intelligences. After that, this machine could improve its own intelligence faster and better than humans can, which would make it even more skilled at improving its own intelligence. This could continue in a positive feedback loop such that the machine quickly becomes vastly more intelligent than the smartest human being on Earth: an 'intelligence explosion' resulting in a machine superintelligence.

This is what is meant by the 'intelligence explosion' in this FAQ.

See also:

* Vinge, [http://www-rohan.sdsu.edu/faculty/vinge/misc/singularity.html The Coming Technological Singularity]
* Wikipedia, [http://en.wikipedia.org/wiki/Technological_singularity Technological Singularity]
* Chalmers, [http://commonsenseatheism.com/wp-content/uploads/2011/01/Chalmers-The-Singularity-a-philosophical-analysis.pdf The Singularity: A Philosophical Analysis]

Question: What is an "intelligence explosion"?

Answer:
The intelligence explosion idea was expressed by statistician ［http://www.incompleteideas.net/papers/Good65ultraintelligent.pdf I.J. Good in 1965］:

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion’, and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make.

The argument is this: Every year, computers surpass human abilities in new ways. A program written in 1956 was able to prove mathematical theorems, and ［http://www.cs.cornell.edu/courses/cs4860/2012fa/MacKenzie-TheAutomationOfProof.pdf found a more elegant proof］ for one of them than Russell and Whitehead had given in ''Principia Mathematica''. By the late 1990s, ‘expert systems’ had surpassed human skill for a ［http://www.amazon.com/dp/0521122937/ wide range of tasks］. In 1997, IBM’s Deep Blue computer beat the world chess champion, and in 2011, IBM’s Watson computer beat the best human players at a much more complicated game: ［http://www.nytimes.com/2011/02/17/science/17jeopardy-watson.html?_r꞊2&ref꞊homepage&src꞊me&pagewanted꞊all Jeopardy!］. Recently, ［http://commonsenseatheism.com/wp-content/uploads/2011/02/King-The-Automation-of-Science.pdf a robot named Adam］ was programmed with our scientific knowledge about yeast, then posed its own hypotheses, tested them, and assessed the results.

Computers remain far short of human intelligence, but the resources that aid AI design are accumulating (including hardware, large datasets, neuroscience knowledge, and AI theory). We may one day design a machine that surpasses human skill at designing artificial intelligences. After that, this machine could improve its own intelligence faster and better than humans can, which would make it even more skilled at improving its own intelligence. This could continue in a positive feedback loop such that the machine quickly becomes vastly more intelligent than the smartest human being on Earth: an ‘intelligence explosion’ resulting in a machine superintelligence.

This is what is meant by the ‘intelligence explosion’ in this FAQ.

See also:

* Vinge, ［http://www-rohan.sdsu.edu/faculty/vinge/misc/singularity.html The Coming Technological Singularity］
* Wikipedia, ［http://en.wikipedia.org/wiki/Technological_singularity Technological Singularity］
* Chalmers, ［http://commonsenseatheism.com/wp-content/uploads/2011/01/Chalmers-The-Singularity-a-philosophical-analysis.pdf The Singularity: A Philosophical Analysis］

Answer to What is an "intelligence explosion"?


Question: What harm could a single superintelligence do when it took so many humans  build civilization?

Answer: Superintelligence has an advantage that an early human didn't – the entire context of human civilization and technology, there for it to manipulate socially or technologically.

Question: What harm could a single superintelligence do when it took so many humans  build civilization?

Answer: Superintelligence has an advantage that an early human didn’t – the entire context of human civilization and technology, there for it to manipulate socially or technologically.

Answer to What harm could a single superintelligence do when it took so many humans to build civilization?


Question: How might non-agentic GPT-style AI cause an "intelligence explosion" or otherwise contribute  existential risk?

Answer: One threat model which includes a GPT component is [https://www.lesswrong.com/posts/zzXawbXDwCZobwF9D/my-agi-threat-model-misaligned-model-based-rl-agent Misaligned Model-Based RL Agent]. It suggests that a reinforcement learner attached to a GPT-style world model could lead to an existential risk, with the RL agent being the optimizer which uses the world model to be much more effective at achieving its goals.

Another possibility is that a sufficiently powerful world model [https://www.lesswrong.com/posts/BGD5J2KAoNmpPMzMQ/why-gpt-wants-to-mesa-optimize-and-how-we-might-change-this may develop mesa optimizers] which could influence the world via the outputs of the model to achieve the mesa objective (perhaps by causing an optimizer to be created with goals aligned to it), though this is somewhat speculative.

Question: How might non-agentic GPT-style AI cause an "intelligence explosion" or otherwise contribute  existential risk?

Answer:
One threat model which includes a GPT component is ［https://www.lesswrong.com/posts/zzXawbXDwCZobwF9D/my-agi-threat-model-misaligned-model-based-rl-agent Misaligned Model-Based RL Agent］. It suggests that a reinforcement learner attached to a GPT-style world model could lead to an existential risk, with the RL agent being the optimizer which uses the world model to be much more effective at achieving its goals.

Another possibility is that a sufficiently powerful world model ［https://www.lesswrong.com/posts/BGD5J2KAoNmpPMzMQ/why-gpt-wants-to-mesa-optimize-and-how-we-might-change-this may develop mesa optimizers］ which could influence the world via the outputs of the model to achieve the mesa objective (perhaps by causing an optimizer to be created with goals aligned to it), though this is somewhat speculative.

Plex's Answer to How might non-agentic GPT-style AI cause an "intelligence explosion" or otherwise contribute to existential risk?


Question: Can't we just tell an AI  do what we want?

Answer: If we could, it would solve a large part of the alignment problem.

The challenge is, how do we code this? Converting something to formal mathematics that can be understood by a computer program is much harder than just saying it in natural language, and proposed AI goal architectures are no exception. Complicated computer programs are usually the result of months of testing and debugging. But this one will be more complicated than any ever attempted before, and live tests are impossible: a superintelligence with a buggy goal system will display goal stability and try to prevent its programmers from discovering or changing the error.

Question: Can't we just tell an AI  do what we want?

Answer:
If we could, it would solve a large part of the alignment problem.

The challenge is, how do we code this? Converting something to formal mathematics that can be understood by a computer program is much harder than just saying it in natural language, and proposed AI goal architectures are no exception. Complicated computer programs are usually the result of months of testing and debugging. But this one will be more complicated than any ever attempted before, and live tests are impossible: a superintelligence with a buggy goal system will display goal stability and try to prevent its programmers from discovering or changing the error.

Answer to Can't we just tell an AI to do what we want?


Question: Any AI will be a computer program. Why wouldn't it just do what it's programmed  do?

Answer: While it is true that a computer program always will do exactly what it is programmed to do, a big issue is that it is difficult to ensure that this is the same as what you intended it to do. Even small computer programs have bugs or glitches, and when programs become as complicated as AGIs will be, it becomes exceedingly difficult to anticipate how the program will behave when ran. This is the problem of AI alignment in a nutshell.

Nick Boström created the famous [https://www.lesswrong.com/tag/paperclip-maximizer paperclip maximizer] thought experiment to illustrate this point. Imagine you are an industrialist who owns a paperclip factory, and imagine you've just received a superintelligent AGI to work for you. You instruct the AGI to "produce as many paperclips as possible". If you've given the AGI no further instructions, the AGI will immediately acquire several instrumental goals.

# It will want to prevent you from turning itself off (If you turn off the AI, this will reduce the amount of paperclips it can produce)
# It will want to acquire as much power and resources for itself as possible (because the more resources it has access to, the more paperclips it can produce)
# It will eventually want to turn the entire universe into a paperclips including you and all other humans, as this is the state of the world that maximizes the amount of paper clips produced.

These consequences might be seen as undesirable by the industrialist, as the only reason the industrialist wanted paperclips in the first place, presumably was so he/she could sell them and make money. However, the AGI only did exactly what it was told to. The issue was that what the AGI was instructed to do, lead to it doing things the industrialist did not anticipate (and did not want).

Some good videos that explore this issue more in depth:<br/>
(youtube)tcdVC4e6EV4(/youtube)
(youtube)hEUO6pjwFOo(/youtube)

Question: Any AI will be a computer program. Why wouldn't it just do what it's programmed  do?

Answer:
While it is true that a computer program always will do exactly what it is programmed to do, a big issue is that it is difficult to ensure that this is the same as what you intended it to do. Even small computer programs have bugs or glitches, and when programs become as complicated as AGIs will be, it becomes exceedingly difficult to anticipate how the program will behave when ran. This is the problem of AI alignment in a nutshell.

Nick Boström created the famous ［https://www.lesswrong.com/tag/paperclip-maximizer paperclip maximizer］ thought experiment to illustrate this point. Imagine you are an industrialist who owns a paperclip factory, and imagine you've just received a superintelligent AGI to work for you. You instruct the AGI to "produce as many paperclips as possible". If you've given the AGI no further instructions, the AGI will immediately acquire several instrumental goals.

# It will want to prevent you from turning itself off (If you turn off the AI, this will reduce the amount of paperclips it can produce)
# It will want to acquire as much power and resources for itself as possible (because the more resources it has access to, the more paperclips it can produce)
# It will eventually want to turn the entire universe into a paperclips including you and all other humans, as this is the state of the world that maximizes the amount of paper clips produced.

These consequences might be seen as undesirable by the industrialist, as the only reason the industrialist wanted paperclips in the first place, presumably was so he/she could sell them and make money. However, the AGI only did exactly what it was told to. The issue was that what the AGI was instructed to do, lead to it doing things the industrialist did not anticipate (and did not want).

Some good videos that explore this issue more in depth:<br/>
(youtube)tcdVC4e6EV4(/youtube)
(youtube)hEUO6pjwFOo(/youtube)

ElloMelon's Answer to Any AI will be a computer program. Why wouldn't it just do what it's programmed to do?


Question: Where can I find all the features of Stampy's Wiki?

Answer: The [[Editor portal]] collects them all in one place.  Details on how to use each feature are on the individual pages.

<big>'''[[Get involved]]'''</big>

'''<big>Questions</big>'''

* '''[[Questions]]''' (262) - List of all questions except those from YouTube.
* '''[[Add question]]''' - Form to add new questions.
* '''[[Answer questions]]''' (129) - List of questions which we want answers to, ordered by quality.
* '''[[Curate questions]]''' (76) - List of questions which we want answers to, with easy buttons for setting quality.
* '''[[Review questions]]''' (19) - Review new incoming questions, to check for duplicates, quality, and relevance.
* '''[[Untagged questions]]''' (73) - Questions without any tags.
* '''[[Prioritize YouTube questions]]''' (222) - Prioritize questions from YouTube, so Stampy posts good ones when you ask.
* '''[[Canonical questions]]''' (347) - List of all canonical questions.
* [[Non-canonical questions]] (7) - List of all non-canonical questions.
* [[Questions from YouTube]] (2,940) - List of all questions from YouTube.

'''<big>Answers</big>'''

* '''[[Answer questions┊Write answers]]''' (129) - Write answers to unanswered questions.
* '''[[Answers]]''' (162) - All the answers except those directed at questions from YouTube.
* '''[[Canonical answers]]''' (168) - List of all canonical answers.
* [[Non-canonical answers]] (68) - List of all non-canonical answers.
* [[YouTube answers]] (267) - List of all answers responding to questions from YouTube.

'''<big>Review answers</big>'''

* '''[[Recent answers]]''' (23) - The most recent answers.
* '''[[Potentially canonical answers]]''' (46) - Answers to canonical questions which don't already have a canonical answer.
* '''[[Non-canonical answers to canonical questions]]''' (46)
* '''[[Recent YouTube answers]]''' (1) - The most recent answers to YouTube questions.

'''<big>Improve answers</big>'''

* '''[[Orphan answers]]''' (171)
* '''[[Wants related]]''' (109)
* '''[[Untagged]]''' (7)
* '''[[Outdated answers]]''' (4)
* '''[[Wants brief]]''' (43)
* '''[[Answers in need of work]]''' (5)
* '''[[Canonical answers with low stamps]]''' (171) - Canonical answers which are not yet highly rated and might need improvement.

'''<big>Recent activity</big>'''

* '''[[Special:RecentChanges┊Recent changes]]''' - What's changed on the wiki recently.
* '''[https://stampy.ai/wiki/Special:RecentChanges?namespace꞊2600&limit꞊50&days꞊100 Recent comments]''' - Recent commenting activity.

'''<big>Pages to create</big>'''

* '''[[Create tags]]''' (55)
* '''[[Create questions]]''' (28)

'''<big>Content</big>'''

* '''[[Tags]]''' (113)
* '''[[Videos]]''' (46)
* '''[[Channels]]''' (3)
* '''[[:Category:Templates┊Templates]]''' (62)

'''<big>External</big>'''

* '''[https://discord.gg/cEzKz8QCpa Stampy's Public Discord]''' - Ask there for an invite to the real one, until OpenAI approves our chatbot for a public Discord
* '''[https://wikiapiary.com/wiki/Stampy%27s_Wiki Wiki stats]''' - Graphs over time of active users, edits, pages, response time, etc
* '''[https://drive.google.com/drive/folders/1F9VUp84J32e3_jhvKog3zgpDXoDw4BdM?usp꞊sharing Google Drive]''' - Folder with Stampy-related documents

'''<big>UI controls</big>'''

* '''[[Initial questions]]'''
* '''[[UI intro]]'''

'''<big>To-do list</big>'''

'''[[What are some specific open tasks on Stampy?]]''' '"`UNIQ--item-653--QINU`"' '"`UNIQ--item-654--QINU`"'

Show your endorsement of this answer by giving it a stamp of approval!

Other than the usual fare of writing and processing and organizing questions and answers, here are some specific open tasks:

* Porting over some of [https://www.lesswrong.com/posts/4basF9w9jaPZpoC8R/intro-to-brain-like-agi-safety-1-what-s-the-problem-and-why#1_4_What_exactly_is__AGI__ Steve Byrnes's FAQ on alignment]
* Porting over content from [https://www.lesswrong.com/posts/gdyfJE3noRFSs373q/resources-i-send-to-ai-researchers-about-ai-safety Vael Gates's post]
* Porting over QA pairs from https://www.lesswrong.com/posts/8c8AZq5hgifmnHKSN/agi-safety-faq-all-dumb-questions-allowed-thread
* Porting over some of https://aisafety.wordpress.com/
* Making sure we cover all of https://forum.effectivealtruism.org/posts/8JazqnCNrkJtK2Bx4/why-eas-are-skeptical-about-ai-safety#Recursive_self_improvement_seems_implausible and the responses
'"`UNIQ--references-0000028F-QINU`"'

__NOTOC__ __NOEDITSECTION__

Question: Where can I find all the features of Stampy's Wiki?

Answer:
The ［［Editor portal］］ collects them all in one place.  Details on how to use each feature are on the individual pages.

<big>'''［［Get involved］］'''</big>

'''<big>Questions</big>'''

* '''［［Questions］］''' (262) - List of all questions except those from YouTube.
* '''［［Add question］］''' - Form to add new questions.
* '''［［Answer questions］］''' (129) - List of questions which we want answers to, ordered by quality.
* '''［［Curate questions］］''' (76) - List of questions which we want answers to, with easy buttons for setting quality.
* '''［［Review questions］］''' (19) - Review new incoming questions, to check for duplicates, quality, and relevance.
* '''［［Untagged questions］］''' (73) - Questions without any tags.
* '''［［Prioritize YouTube questions］］''' (222) - Prioritize questions from YouTube, so Stampy posts good ones when you ask.
* '''［［Canonical questions］］''' (347) - List of all canonical questions.
* ［［Non-canonical questions］］ (7) - List of all non-canonical questions.
* ［［Questions from YouTube］］ (2,940) - List of all questions from YouTube.

'''<big>Answers</big>'''

* '''［［Answer questions┊Write answers］］''' (129) - Write answers to unanswered questions.
* '''［［Answers］］''' (162) - All the answers except those directed at questions from YouTube.
* '''［［Canonical answers］］''' (168) - List of all canonical answers.
* ［［Non-canonical answers］］ (68) - List of all non-canonical answers.
* ［［YouTube answers］］ (267) - List of all answers responding to questions from YouTube.

'''<big>Review answers</big>'''

* '''［［Recent answers］］''' (23) - The most recent answers.
* '''［［Potentially canonical answers］］''' (46) - Answers to canonical questions which don't already have a canonical answer.
* '''［［Non-canonical answers to canonical questions］］''' (46)
* '''［［Recent YouTube answers］］''' (1) - The most recent answers to YouTube questions.

'''<big>Improve answers</big>'''

* '''［［Orphan answers］］''' (171)
* '''［［Wants related］］''' (109)
* '''［［Untagged］］''' (7)
* '''［［Outdated answers］］''' (4)
* '''［［Wants brief］］''' (43)
* '''［［Answers in need of work］］''' (5)
* '''［［Canonical answers with low stamps］］''' (171) - Canonical answers which are not yet highly rated and might need improvement.

'''<big>Recent activity</big>'''

* '''［［Special:RecentChanges┊Recent changes］］''' - What's changed on the wiki recently.
* '''［https://stampy.ai/wiki/Special:RecentChanges?namespace꞊2600&limit꞊50&days꞊100 Recent comments］''' - Recent commenting activity.

'''<big>Pages to create</big>'''

* '''［［Create tags］］''' (55)
* '''［［Create questions］］''' (28)

'''<big>Content</big>'''

* '''［［Tags］］''' (113)
* '''［［Videos］］''' (46)
* '''［［Channels］］''' (3)
* '''［［:Category:Templates┊Templates］］''' (62)

'''<big>External</big>'''

* '''［https://discord.gg/cEzKz8QCpa Stampy's Public Discord］''' - Ask there for an invite to the real one, until OpenAI approves our chatbot for a public Discord
* '''［https://wikiapiary.com/wiki/Stampy%27s_Wiki Wiki stats］''' - Graphs over time of active users, edits, pages, response time, etc
* '''［https://drive.google.com/drive/folders/1F9VUp84J32e3_jhvKog3zgpDXoDw4BdM?usp꞊sharing Google Drive］''' - Folder with Stampy-related documents

'''<big>UI controls</big>'''

* '''［［Initial questions］］'''
* '''［［UI intro］］'''

'''<big>To-do list</big>'''

'''［［What are some specific open tasks on Stampy?］］''' '"`UNIQ--item-653--QINU`"' '"`UNIQ--item-654--QINU`"'

Show your endorsement of this answer by giving it a stamp of approval!

Other than the usual fare of writing and processing and organizing questions and answers, here are some specific open tasks:

* Porting over some of ［https://www.lesswrong.com/posts/4basF9w9jaPZpoC8R/intro-to-brain-like-agi-safety-1-what-s-the-problem-and-why#1_4_What_exactly_is__AGI__ Steve Byrnes's FAQ on alignment］
* Porting over content from ［https://www.lesswrong.com/posts/gdyfJE3noRFSs373q/resources-i-send-to-ai-researchers-about-ai-safety Vael Gates's post］
* Porting over QA pairs from https://www.lesswrong.com/posts/8c8AZq5hgifmnHKSN/agi-safety-faq-all-dumb-questions-allowed-thread
* Porting over some of https://aisafety.wordpress.com/
* Making sure we cover all of https://forum.effectivealtruism.org/posts/8JazqnCNrkJtK2Bx4/why-eas-are-skeptical-about-ai-safety#Recursive_self_improvement_seems_implausible and the responses
'"`UNIQ--references-0000028F-QINU`"'

__NOTOC__ __NOEDITSECTION__

Plex's Answer to Where can I find all the features of Stampy's Wiki?


Question: What would a good solution  AI alignment look like?

Answer: An actually good solution to AI alignment might look like a superintelligence that understands, agrees with, and deeply believes in human morality.

You wouldn't have to command a superintelligence like this to cure cancer; it would already want to cure cancer, for the same reasons you do. But it would also be able to compare the costs and benefits of curing cancer with those of other uses of its time, like solving global warming or discovering new physics. It wouldn't have any urge to cure cancer by nuking the world, for the same reason you don't have any urge to cure cancer by nuking the world – because your goal isn't to "cure cancer", per se, it's to improve the lives of people everywhere. Curing cancer the normal way accomplishes that; nuking the world doesn't.
This sort of solution would mean we're no longer fighting against the AI – trying to come up with rules so smart that it couldn't find loopholes. We would be on the same side, both wanting the same thing.

It would also mean that the CEO of Google (or the head of the US military, or Vladimir Putin) couldn't use the AI to take over the world for themselves. The AI would have its own values and be able to agree or disagree with anybody, including its creators.

It might not make sense to talk about "commanding" such an AI. After all, any command would have to go through its moral system. Certainly it would reject a command to nuke the world. But it might also reject a command to cure cancer, if it thought that solving global warming was a higher priority. For that matter, why would one want to command this AI? It values the same things you value, but it's much smarter than you and much better at figuring out how to achieve them. Just turn it on and let it do its thing.

We could still treat this AI as having an open-ended maximizing goal. The goal would be something like "Try to make the world a better place according to the values and wishes of the people in it."

The only problem with this is that human morality is very complicated, so much so that philosophers have been arguing about it for thousands of years without much progress, let alone anything specific enough to enter into a computer. Different cultures and individuals have different moral codes, such that a superintelligence following the morality of the King of Saudi Arabia might not be acceptable to the average American, and vice versa.

One solution might be to give the AI an understanding of what we mean by morality – "that thing that makes intuitive sense to humans but is hard to explain", and then ask it to use its superintelligence to fill in the details. Needless to say, this suffers from various problems – it has potential loopholes, it's hard to code, and a single bug might be disastrous – but if it worked, it would be one of the few genuinely satisfying ways to design a goal architecture.

Question: What would a good solution  AI alignment look like?

Answer:
An actually good solution to AI alignment might look like a superintelligence that understands, agrees with, and deeply believes in human morality.

You wouldn’t have to command a superintelligence like this to cure cancer; it would already want to cure cancer, for the same reasons you do. But it would also be able to compare the costs and benefits of curing cancer with those of other uses of its time, like solving global warming or discovering new physics. It wouldn’t have any urge to cure cancer by nuking the world, for the same reason you don’t have any urge to cure cancer by nuking the world – because your goal isn’t to “cure cancer”, per se, it’s to improve the lives of people everywhere. Curing cancer the normal way accomplishes that; nuking the world doesn’t.
This sort of solution would mean we’re no longer fighting against the AI – trying to come up with rules so smart that it couldn’t find loopholes. We would be on the same side, both wanting the same thing.

It would also mean that the CEO of Google (or the head of the US military, or Vladimir Putin) couldn’t use the AI to take over the world for themselves. The AI would have its own values and be able to agree or disagree with anybody, including its creators.

It might not make sense to talk about “commanding” such an AI. After all, any command would have to go through its moral system. Certainly it would reject a command to nuke the world. But it might also reject a command to cure cancer, if it thought that solving global warming was a higher priority. For that matter, why would one want to command this AI? It values the same things you value, but it’s much smarter than you and much better at figuring out how to achieve them. Just turn it on and let it do its thing.

We could still treat this AI as having an open-ended maximizing goal. The goal would be something like “Try to make the world a better place according to the values and wishes of the people in it.”

The only problem with this is that human morality is very complicated, so much so that philosophers have been arguing about it for thousands of years without much progress, let alone anything specific enough to enter into a computer. Different cultures and individuals have different moral codes, such that a superintelligence following the morality of the King of Saudi Arabia might not be acceptable to the average American, and vice versa.

One solution might be to give the AI an understanding of what we mean by morality – “that thing that makes intuitive sense to humans but is hard to explain”, and then ask it to use its superintelligence to fill in the details. Needless to say, this suffers from various problems – it has potential loopholes, it’s hard to code, and a single bug might be disastrous – but if it worked, it would be one of the few genuinely satisfying ways to design a goal architecture.

Answer to What would a good solution to AI alignment look like?


What is "superintelligence"?

Question: What is "superintelligence"?

Answer: A superintelligence is a mind that is much more intelligent than any human. Most of the time, it's used to discuss hypothetical future AIs.

Question: What is "superintelligence"?

Answer: A superintelligence is a mind that is much more intelligent than any human. Most of the time, it’s used to discuss hypothetical future AIs.

Scott's Answer to What is "superintelligence"?


Plex's Answer to What is "narrow AI"?

What is "narrow AI"?

Question: What is "narrow AI"?

Answer:
A Narrow AI is capable of operating only in a relatively limited domain, such as chess or driving, rather than capable of learning a broad range of tasks like a human or an [https://www.lesswrong.com/tag/artificial-general-intelligence Artificial General Intelligence]. Narrow vs General is not a perfectly binary classification, there are degrees of generality with, for example, large language models having a fairly large degree of generality (as the domain of text is large) without being as general as a human, and we may eventually build systems that are significantly more general than humans.

Question: What is "narrow AI"?

Answer: A Narrow AI is capable of operating only in a relatively limited domain, such as chess or driving, rather than capable of learning a broad range of tasks like a human or an ［https://www.lesswrong.com/tag/artificial-general-intelligence Artificial General Intelligence］. Narrow vs General is not a perfectly binary classification, there are degrees of generality with, for example, large language models having a fairly large degree of generality (as the domain of text is large) without being as general as a human, and we may eventually build systems that are significantly more general than humans.


Linnea's Answer to What is "hedonium"?

What is "hedonium"?

Question: What is "hedonium"?

Answer:
Orgasmium (also known as hedonium) is a homogeneous substance with limited consciousness, which is in a constant state of supreme bliss. An AI programmed to "maximize happiness" might simply tile the universe with orgasmium. Some who believe this consider it a good thing; others do not. Those who do not, use its undesirability to argue that not all terminal values reduce to "happiness" or some simple analogue. Hedonium is the [https://www.lesswrong.com/tag/hedonism hedonistic] [https://www.lesswrong.com/tag/utilitarianism utilitarian]'s version of [https://www.lesswrong.com/tag/utilitronium utilitronium].

Blog posts

[http://lesswrong.com/lw/wv/prolegomena_to_a_theory_of_fun/ Prolegomena to a Theory of Fun]

[http://lesswrong.com/lw/xr/in_praise_of_boredom/ In Praise of Boredom]

[https://reflectivedisequilibrium.blogspot.com/2012/03/are-pain-and-pleasure-equally-energy.html Are pain and pleasure equally energy-efficient?]

See also

[https://www.goodreads.com/quotes/<PHONE>-consider-an-ai-that-has-hedonism-as-its-final-goal Quote from Superintelligence]

[https://www.lesswrong.com/tag/fun-theory Fun theory]

[https://www.lesswrong.com/tag/complexity-of-value Complexity of value]

[https://www.lesswrong.com/tag/utilitronium Utilitronium]

[https://www.lesswrong.com/tag/orgasmium?edit꞊true Edit on LessWrong]

https://www.lesswrong.com/tag/orgasmium?edit꞊true

Question: What is "hedonium"?

Answer:
Orgasmium (also known as hedonium) is a homogeneous substance with limited consciousness, which is in a constant state of supreme bliss. An AI programmed to "maximize happiness" might simply tile the universe with orgasmium. Some who believe this consider it a good thing; others do not. Those who do not, use its undesirability to argue that not all terminal values reduce to "happiness" or some simple analogue. Hedonium is the ［https://www.lesswrong.com/tag/hedonism hedonistic］ ［https://www.lesswrong.com/tag/utilitarianism utilitarian］'s version of ［https://www.lesswrong.com/tag/utilitronium utilitronium］.

Blog posts

［http://lesswrong.com/lw/wv/prolegomena_to_a_theory_of_fun/ Prolegomena to a Theory of Fun］

［http://lesswrong.com/lw/xr/in_praise_of_boredom/ In Praise of Boredom］

［https://reflectivedisequilibrium.blogspot.com/2012/03/are-pain-and-pleasure-equally-energy.html Are pain and pleasure equally energy-efficient?］

See also

［https://www.goodreads.com/quotes/1413237-consider-an-ai-that-has-hedonism-as-its-final-goal Quote from Superintelligence］

［https://www.lesswrong.com/tag/fun-theory Fun theory］

［https://www.lesswrong.com/tag/complexity-of-value Complexity of value］

［https://www.lesswrong.com/tag/utilitronium Utilitronium］

［https://www.lesswrong.com/tag/orgasmium?edit꞊true Edit on LessWrong］

https://www.lesswrong.com/tag/orgasmium?edit꞊true


What is "functional decision theory"?

Question: What is "functional decision theory"?

Answer:
Functional Decision Theory is a [https://www.lesswrong.com/tag/decision-theory decision theory] described by Eliezer Yudkowsky and Nate Soares which says that agents should treat one's decision as the output of a fixed mathematical function that answers the question, "Which output of this very function would yield the best outcome?". It is a replacement of [https://www.lesswrong.com/tag/timeless-decision-theory Timeless Decision Theory], and it outperforms other decision theories such as [https://www.lesswrong.com/tag/causal-decision-theory Causal Decision Theory] (CDT) and [https://www.lesswrong.com/tag/evidential-decision-theory Evidential Decision Theory] (EDT). For example, it does better than CDT on [https://www.lesswrong.com/tag/newcomb-s-problem Newcomb's Problem], better than EDT on the [https://www.lesswrong.com/tag/smoking-lesion smoking lesion problem], and better than both in [https://www.lesswrong.com/tag/parfits-hitchhiker Parfit's hitchhiker problem].

In Newcomb's Problem, an FDT agent reasons that Omega must have used some kind of model of her decision procedure in order to make an accurate prediction of her behavior. Omega's model and the agent are therefore both calculating the same function (the agent's decision procedure): they are subjunctively dependent on that function. Given perfect prediction by Omega, there are therefore only two outcomes in Newcomb's Problem: either the agent one-boxes and Omega predicted it (because its model also one-boxed), or the agent two-boxes and Omega predicted that. Because one-boxing then results in a million and two-boxing only in a thousand dollars, the FDT agent one-boxes.

External links:

[https://intelligence.org/2017/10/22/fdt Functional decision theory: A new theory of instrumental rationality]

[https://intelligence.org/2017/03/18/new-paper-cheating-death-in-damascus/ Cheating Death in Damascus]

[https://intelligence.org/2017/04/07/decisions-are-for-making-bad-outcomes-inconsistent/ Decisions are for making bad outcomes inconsistent]

[https://www.umsu.de/wo/2018/688 On Functional Decision Theory] by Wolfgang Schwarz

See Also:

[https://www.lesswrong.com/tag/timeless-decision-theory Timeless Decision Theory]

[https://www.lesswrong.com/tag/updateless-decision-theory Updateless Decision Theory]

[https://www.lesswrong.com/tag/functional-decision-theory?edit꞊true Edit on LessWrong]

https://www.lesswrong.com/tag/functional-decision-theory?edit꞊true

Question: What is "functional decision theory"?

Answer:
Functional Decision Theory is a ［https://www.lesswrong.com/tag/decision-theory decision theory］ described by Eliezer Yudkowsky and Nate Soares which says that agents should treat one’s decision as the output of a ﬁxed mathematical function that answers the question, “Which output of this very function would yield the best outcome?”. It is a replacement of ［https://www.lesswrong.com/tag/timeless-decision-theory Timeless Decision Theory］, and it outperforms other decision theories such as ［https://www.lesswrong.com/tag/causal-decision-theory Causal Decision Theory］ (CDT) and ［https://www.lesswrong.com/tag/evidential-decision-theory Evidential Decision Theory］ (EDT). For example, it does better than CDT on ［https://www.lesswrong.com/tag/newcomb-s-problem Newcomb's Problem］, better than EDT on the ［https://www.lesswrong.com/tag/smoking-lesion smoking lesion problem］, and better than both in ［https://www.lesswrong.com/tag/parfits-hitchhiker Parﬁt’s hitchhiker problem］.

In Newcomb's Problem, an FDT agent reasons that Omega must have used some kind of model of her decision procedure in order to make an accurate prediction of her behavior. Omega's model and the agent are therefore both calculating the same function (the agent's decision procedure): they are subjunctively dependent on that function. Given perfect prediction by Omega, there are therefore only two outcomes in Newcomb's Problem: either the agent one-boxes and Omega predicted it (because its model also one-boxed), or the agent two-boxes and Omega predicted that. Because one-boxing then results in a million and two-boxing only in a thousand dollars, the FDT agent one-boxes.

External links:

［https://intelligence.org/2017/10/22/fdt Functional decision theory: A new theory of instrumental rationality］

［https://intelligence.org/2017/03/18/new-paper-cheating-death-in-damascus/ Cheating Death in Damascus］

［https://intelligence.org/2017/04/07/decisions-are-for-making-bad-outcomes-inconsistent/ Decisions are for making bad outcomes inconsistent］

［https://www.umsu.de/wo/2018/688 On Functional Decision Theory］ by Wolfgang Schwarz

See Also:

［https://www.lesswrong.com/tag/timeless-decision-theory Timeless Decision Theory］

［https://www.lesswrong.com/tag/updateless-decision-theory Updateless Decision Theory］

［https://www.lesswrong.com/tag/functional-decision-theory?edit꞊true Edit on LessWrong］

https://www.lesswrong.com/tag/functional-decision-theory?edit꞊true

Linnea's Answer to What is "functional decision theory"?


What is "evidential decision theory"?

Question: What is "evidential decision theory"?

Answer:
Evidential Decision Theory – EDT – is a branch of [https://www.lesswrong.com/tag/decision-theory decision theory] which advises an agent to take actions which, conditional on it happening, maximizes the chances of the desired outcome. As any branch of decision theory, it prescribes taking the action that maximizes [https://www.lesswrong.com/tag/utility utility], that which utility equals or exceeds the utility of every other option. The utility of each action is measured by the [https://www.lesswrong.com/tag/expected-utility expected utility], the averaged by probabilities sum of the utility of each of its possible results. How the actions can influence the probabilities differ between the branches. [https://www.lesswrong.com/tag/causal-decision-theory Causal Decision Theory] – CDT – says only through causal process one can influence the chances of the desired outcome [#fn1 1]. EDT, on the other hand, requires no causal connection, the action only have to be a [https://www.lesswrong.com/tag/bayesianism Bayesian] evidence for the desired outcome. Some critics say it recommends auspiciousness over causal efficacy[#fn2 2].

One usual example where EDT and CDT are often said to diverge is the [https://www.lesswrong.com/tag/smoking-lesion Smoking lesion]: "Smoking is strongly correlated with lung cancer, but in the world of the Smoker's Lesion this correlation is understood to be the result of a common cause: a genetic lesion that tends to cause both smoking and cancer. Once we fix the presence or absence of the lesion, there is no additional correlation between smoking and cancer. Suppose you prefer smoking without cancer to not smoking without cancer, and prefer smoking with cancer to not smoking with cancer. Should you smoke?" CDT would recommend smoking since there is no causal connection between smoking and cancer. They are both caused by a gene, but have no causal direct connection with each other. Naive EDT, on the other hand, would recommend against smoking, since smoking is an evidence for having the mentioned gene and thus should be avoided. However, a more sophisticated agent following the recommendations of EDT would recognize that if they observe that they have the desire to smoke, then actually smoking or not would provide no more evidence for having cancer; that is, the "tickle" [https://www.lesswrong.com/tag/screening-off-evidence screens off] smoking from cancer. (This is known as the tickle defence.)

CDT uses probabilities of conditionals and contrafactual dependence to calculate the expected utility of an action – which track causal relations -, whereas EDT simply uses conditional probabilities. The probability of a conditional is the probability of the whole conditional being true, where the conditional probability is the probability of the consequent given the antecedent. A conditional probability of B given A - P(B┊A) -, simply implies the Bayesian probability of the event B happening given we known A happened, it's used in EDT. The probability of conditionals – P(A > B) - refers to the probability that the conditional 'A implies B' is true, it is the probability of the contrafactual 'If A, then B' be the case. Since contrafactual analysis is the key tool used to speak about causality, probability of conditionals are said to mirror causal relations. In most usual cases these two probabilities are the same. However, David Lewis proved [#fn3 3] its' impossible to probabilities of conditionals to always track conditional probabilities. Hence evidential relations aren't the same as causal relations and CDT and EDT will diverge depending on the problem. In some cases, EDT gives a better answers then CDT, such as the [https://www.lesswrong.com/tag/newcomb-s-problem Newcomb's problem], whereas in the [https://www.lesswrong.com/tag/smoking-lesion Smoking lesion] problem where CDT seems to give a more reasonable prescription (modulo the tickle defence).

References

[http://plato.stanford.edu/entries/decision-causal/ http://plato.stanford.edu/entries/decision-causal/][#fnref1 ↩]

Joyce, J.M. (1999), The foundations of causal decision theory, p. 146[#fnref2 ↩]

Lewis, D. (1976), "Probabilities of conditionals and conditional probabilities", The Philosophical Review (Duke University Press) 85 (3): 297–315[#fnref3 ↩]

Caspar Oesterheld, "[https://www.andrew.cmu.edu/user/coesterh/TickleDefenseIntro.pdf Understanding the Tickle Defense in Decision Theory]"

Ahmed, Arif. (2014), "Evidence, Decision and Causality" (Cambridge University Press)

Blog posts

[https://agentfoundations.org/item?id꞊1525 Smoking Lesion Steelman] by Abram Demski

[http://lesswrong.com/lw/gu1/decision_theory_faq/ Decision Theory FAQ] by Luke Muehlhauser

[https://casparoesterheld.files.wordpress.com/2016/12/almond_edt_1.pdf On Causation and Correlation Part 1]

[http://lesswrong.com/lw/men/twoboxing_smoking_and_chewing_gum_in_medical/ Two-boxing, smoking and chewing gum in Medical Newcomb problems] by Caspar Oesterheld

[http://lesswrong.com/r/discussion/lw/oih/did_edt_get_it_right_all_along_introducing_yet/ Did EDT get it right all along? Introducing yet another medical Newcomb problem] by Johannes Treutlein

[https://casparoesterheld.com/2017/02/06/betting-on-the-past-by-arif-ahmed/ "Betting on the Past" by Arif Ahmed] by Johannes Treutlein

[https://agentfoundations.org/item?id꞊92 Why conditioning on "the agent takes action a" isn't enough] by Nate Soares

[https://casparoesterheld.com/overview-why-we-think-that-the-smoking-lesion-does-not-refute-edt/ Overview: Why the Smoking Lesion does not refute EDT]

See also

[https://www.lesswrong.com/tag/decision-theory Decision theory]

[https://www.lesswrong.com/tag/causal-decision-theory Causal decision theory]

MacAskill, W. et al. (2021), "[https://philpapers.org/rec/MACTEW-2 The Evidentialist's Wager]"

'''[https://www.lesswrong.com/tag/evidential-decision-theory?edit꞊true Edit on LessWrong]'''

https://www.lesswrong.com/tag/evidential-decision-theory?edit꞊true

Question: What is "evidential decision theory"?

Answer:
Evidential Decision Theory – EDT – is a branch of ［https://www.lesswrong.com/tag/decision-theory decision theory］ which advises an agent to take actions which, conditional on it happening, maximizes the chances of the desired outcome. As any branch of decision theory, it prescribes taking the action that maximizes ［https://www.lesswrong.com/tag/utility utility］, that which utility equals or exceeds the utility of every other option. The utility of each action is measured by the ［https://www.lesswrong.com/tag/expected-utility expected utility］, the averaged by probabilities sum of the utility of each of its possible results. How the actions can influence the probabilities differ between the branches. ［https://www.lesswrong.com/tag/causal-decision-theory Causal Decision Theory］ – CDT – says only through causal process one can influence the chances of the desired outcome ［#fn1 1］. EDT, on the other hand, requires no causal connection, the action only have to be a ［https://www.lesswrong.com/tag/bayesianism Bayesian］ evidence for the desired outcome. Some critics say it recommends auspiciousness over causal efficacy［#fn2 2］.

One usual example where EDT and CDT are often said to diverge is the ［https://www.lesswrong.com/tag/smoking-lesion Smoking lesion］: “Smoking is strongly correlated with lung cancer, but in the world of the Smoker's Lesion this correlation is understood to be the result of a common cause: a genetic lesion that tends to cause both smoking and cancer. Once we fix the presence or absence of the lesion, there is no additional correlation between smoking and cancer. Suppose you prefer smoking without cancer to not smoking without cancer, and prefer smoking with cancer to not smoking with cancer. Should you smoke?” CDT would recommend smoking since there is no causal connection between smoking and cancer. They are both caused by a gene, but have no causal direct connection with each other. Naive EDT, on the other hand, would recommend against smoking, since smoking is an evidence for having the mentioned gene and thus should be avoided. However, a more sophisticated agent following the recommendations of EDT would recognize that if they observe that they have the desire to smoke, then actually smoking or not would provide no more evidence for having cancer; that is, the "tickle" ［https://www.lesswrong.com/tag/screening-off-evidence screens off］ smoking from cancer. (This is known as the tickle defence.)

CDT uses probabilities of conditionals and contrafactual dependence to calculate the expected utility of an action – which track causal relations -, whereas EDT simply uses conditional probabilities. The probability of a conditional is the probability of the whole conditional being true, where the conditional probability is the probability of the consequent given the antecedent. A conditional probability of B given A - P(B┊A) -, simply implies the Bayesian probability of the event B happening given we known A happened, it’s used in EDT. The probability of conditionals – P(A > B) - refers to the probability that the conditional 'A implies B' is true, it is the probability of the contrafactual ‘If A, then B’ be the case. Since contrafactual analysis is the key tool used to speak about causality, probability of conditionals are said to mirror causal relations. In most usual cases these two probabilities are the same. However, David Lewis proved ［#fn3 3］ its’ impossible to probabilities of conditionals to always track conditional probabilities. Hence evidential relations aren’t the same as causal relations and CDT and EDT will diverge depending on the problem. In some cases, EDT gives a better answers then CDT, such as the ［https://www.lesswrong.com/tag/newcomb-s-problem Newcomb's problem］, whereas in the ［https://www.lesswrong.com/tag/smoking-lesion Smoking lesion］ problem where CDT seems to give a more reasonable prescription (modulo the tickle defence).

References

［http://plato.stanford.edu/entries/decision-causal/ http://plato.stanford.edu/entries/decision-causal/］［#fnref1 ↩］

Joyce, J.M. (1999), The foundations of causal decision theory, p. 146［#fnref2 ↩］

Lewis, D. (1976), "Probabilities of conditionals and conditional probabilities", The Philosophical Review (Duke University Press) 85 (3): 297–315［#fnref3 ↩］

Caspar Oesterheld, "［https://www.andrew.cmu.edu/user/coesterh/TickleDefenseIntro.pdf Understanding the Tickle Defense in Decision Theory］"

Ahmed, Arif. (2014), "Evidence, Decision and Causality" (Cambridge University Press)

Blog posts

［https://agentfoundations.org/item?id꞊1525 Smoking Lesion Steelman］ by Abram Demski

［http://lesswrong.com/lw/gu1/decision_theory_faq/ Decision Theory FAQ］ by Luke Muehlhauser

［https://casparoesterheld.files.wordpress.com/2016/12/almond_edt_1.pdf On Causation and Correlation Part 1］

［http://lesswrong.com/lw/men/twoboxing_smoking_and_chewing_gum_in_medical/ Two-boxing, smoking and chewing gum in Medical Newcomb problems］ by Caspar Oesterheld

［http://lesswrong.com/r/discussion/lw/oih/did_edt_get_it_right_all_along_introducing_yet/ Did EDT get it right all along? Introducing yet another medical Newcomb problem］ by Johannes Treutlein

［https://casparoesterheld.com/2017/02/06/betting-on-the-past-by-arif-ahmed/ "Betting on the Past" by Arif Ahmed］ by Johannes Treutlein

［https://agentfoundations.org/item?id꞊92 Why conditioning on "the agent takes action a" isn't enough］ by Nate Soares

［https://casparoesterheld.com/overview-why-we-think-that-the-smoking-lesion-does-not-refute-edt/ Overview: Why the Smoking Lesion does not refute EDT］

See also

［https://www.lesswrong.com/tag/decision-theory Decision theory］

［https://www.lesswrong.com/tag/causal-decision-theory Causal decision theory］

MacAskill, W. et al. (2021), "［https://philpapers.org/rec/MACTEW-2 The Evidentialist’s Wager］"

'''［https://www.lesswrong.com/tag/evidential-decision-theory?edit꞊true Edit on LessWrong］'''

https://www.lesswrong.com/tag/evidential-decision-theory?edit꞊true

Linnea's Answer to What is "evidential decision theory"?


Question: What is "biological cognitive enhancement"?

Answer: There may be genes or molecules that can be modified to improve general intelligence. Researchers [https://pubmed.ncbi.nlm.nih.gov/<PHONE>/ have already done this in mice]: they over-expressed the NR2B gene, which improved those mice's memory beyond that of any other mice of any mouse species. Biological cognitive enhancement in humans may cause an intelligence explosion to occur more quickly than it otherwise would.

See also:
*Bostrom & Sandberg, [http://www.nickbostrom.com/cognitive.pdf Cognitive Enhancement: Methods, Ethics, Regulatory Challenges]

Question: What is "biological cognitive enhancement"?

Answer:
There may be genes or molecules that can be modified to improve general intelligence. Researchers ［https://pubmed.ncbi.nlm.nih.gov/10485705/ have already done this in mice］: they over-expressed the NR2B gene, which improved those mice’s memory beyond that of any other mice of any mouse species. Biological cognitive enhancement in humans may cause an intelligence explosion to occur more quickly than it otherwise would.

See also:
*Bostrom & Sandberg, ［http://www.nickbostrom.com/cognitive.pdf Cognitive Enhancement: Methods, Ethics, Regulatory Challenges］

Answer to What is "biological cognitive enhancement"?


What can I do  contribute  AI safety?

Question: What can I do  contribute  AI safety?

Answer: It's pretty dependent on what skills you have and what resources you have access to. The largest option is to pursue a [https://80000hours.org/career-reviews/artificial-intelligence-risk-research/ career in AI Safety research]. Another large option is to pursue a career in [https://80000hours.org/articles/ai-policy-guide/ AI policy], which you might think is even more important than doing technical research.

Smaller options include donating money to relevant organizations, talking about AI Safety as a plausible career path to other people or considering the problem in your spare time.

It's possible that your particular set of skills/resources are not suited to this problem. Unluckily, there are [https://concepts.effectivealtruism.org/concepts/existential-risks/ many more problems] that are of similar levels of importance.

Question: What can I do  contribute  AI safety?

Answer:
It’s pretty dependent on what skills you have and what resources you have access to. The largest option is to pursue a ［https://80000hours.org/career-reviews/artificial-intelligence-risk-research/ career in AI Safety research］. Another large option is to pursue a career in ［https://80000hours.org/articles/ai-policy-guide/ AI policy］, which you might think is even more important than doing technical research.

Smaller options include donating money to relevant organizations, talking about AI Safety as a plausible career path to other people or considering the problem in your spare time.

It’s possible that your particular set of skills/resources are not suited to this problem. Unluckily, there are ［https://concepts.effectivealtruism.org/concepts/existential-risks/ many more problems］ that are of similar levels of importance.

Answer to What can I do to contribute to AI safety?


Question: What are some good resources on AI alignment?

Answer: These are good sources for understanding AI alignment and linking to when editing Stampy!

* [https://www.youtube.com/c/RobertMilesAI/videos Rob's YouTube videos] ([https://www.youtube.com/watch?v꞊tlS5Y2vm02c&list꞊PLqL14ZxTTA4fRMts7Af2G8t4Rp17e8MdS&index꞊4 Computerphile appearances])
* [https://ai-safety-papers.quantifieduncertainty.org/ AI Safety Papers database] - Search and interface for the [https://www.lesswrong.com/posts/4DegbDJJiMX2b3EKm/tai-safety-bibliographic-database TAI Safety Bibliography]
* [https://www.eacambridge.org/agi-safety-fundamentals AGI Safety Fundamentals Course]
* [https://www.alignmentforum.org/tags/ Alignment Forum] tags
* [https://rohinshah.com/alignment-newsletter/ The Alignment Newsletter] (and [https://docs.google.com/spreadsheets/d/1PwWbWZ6FPqAgZWOoOcXM8N_tUCuxpEyMbN1NYYC02aM/edit#gid꞊0 database sheet])
* Chapters of [https://publicism.info/philosophy/superintelligence/ Bostrom's Superintelligence online] - [https://www.nickbostrom.com/views/superintelligence.pdf Initial paper which Superintelligence grew from]
* [https://arbital.greaterwrong.com/explore/ai_alignment/ AI Alignment pages on Arbital]
* Much more on [https://www.aisafetysupport.org/resources/lots-of-links AI Safety Support] (feel free to integrate useful things from there to here)
* [https://vkrakovna.wordpress.com/ai-safety-resources/ Vika's resources list]
* [https://docs.google.com/spreadsheets/d/1QSEWjXZuqmG6ORkig84V4sFCldIntyuQj7yq3gkDo0U/edit#gid꞊0 AI safety technical courses, reading lists, and curriculums]
* [https://aisafety.wordpress.com/ AI Safety Intro blog]
* [https://stampy.ai/wiki/Canonical_answers Stampy's canonical answers list] - This includes updated versions of various [[Imported FAQs┊FAQs imported with permission]]:
** [https://www.lesswrong.com/posts/LTtNXM9shNM9AC2mp/superintelligence-faq Scott Alexander's Superintelligence FAQ]
** [https://futureoflife.org/ai-faqs/ FLI's FAQ]
** [https://intelligence.org/faq/ MIRI's FAQ]
** [https://intelligence.org/ie-faq/ MIRI's Intelligence Explosion FAQ]
** [https://rohinshah.com/faq-career-advice-for-ai-alignment-researchers/ Advice for AI Alignment Researchers]
** [https://www.reddit.com/r/ControlProblem/wiki/faq r/ControlProblem's FAQ]
** [https://markxu.com/ai-safety-faqs Mark Xu's FAQ]
** [https://aisafety.wordpress.com/ AI safety blog] - Not yet imported.

Question: What are some good resources on AI alignment?

Answer:
These are good sources for understanding AI alignment and linking to when editing Stampy!

* ［https://www.youtube.com/c/RobertMilesAI/videos Rob's YouTube videos］ (［https://www.youtube.com/watch?v꞊tlS5Y2vm02c&list꞊PLqL14ZxTTA4fRMts7Af2G8t4Rp17e8MdS&index꞊4 Computerphile appearances］)
* ［https://ai-safety-papers.quantifieduncertainty.org/ AI Safety Papers database］ - Search and interface for the ［https://www.lesswrong.com/posts/4DegbDJJiMX2b3EKm/tai-safety-bibliographic-database TAI Safety Bibliography］
* ［https://www.eacambridge.org/agi-safety-fundamentals AGI Safety Fundamentals Course］
* ［https://www.alignmentforum.org/tags/ Alignment Forum］ tags
* ［https://rohinshah.com/alignment-newsletter/ The Alignment Newsletter］ (and ［https://docs.google.com/spreadsheets/d/1PwWbWZ6FPqAgZWOoOcXM8N_tUCuxpEyMbN1NYYC02aM/edit#gid꞊0 database sheet］)
* Chapters of ［https://publicism.info/philosophy/superintelligence/ Bostrom's Superintelligence online］ - ［https://www.nickbostrom.com/views/superintelligence.pdf Initial paper which Superintelligence grew from］
* ［https://arbital.greaterwrong.com/explore/ai_alignment/ AI Alignment pages on Arbital］
* Much more on ［https://www.aisafetysupport.org/resources/lots-of-links AI Safety Support］ (feel free to integrate useful things from there to here)
* ［https://vkrakovna.wordpress.com/ai-safety-resources/ Vika's resources list］
* ［https://docs.google.com/spreadsheets/d/1QSEWjXZuqmG6ORkig84V4sFCldIntyuQj7yq3gkDo0U/edit#gid꞊0 AI safety technical courses, reading lists, and curriculums］
* ［https://aisafety.wordpress.com/ AI Safety Intro blog］
* ［https://stampy.ai/wiki/Canonical_answers Stampy's canonical answers list］ - This includes updated versions of various ［［Imported FAQs┊FAQs imported with permission］］:
** ［https://www.lesswrong.com/posts/LTtNXM9shNM9AC2mp/superintelligence-faq Scott Alexander's Superintelligence FAQ］
** ［https://futureoflife.org/ai-faqs/ FLI's FAQ］
** ［https://intelligence.org/faq/ MIRI's FAQ］
** ［https://intelligence.org/ie-faq/ MIRI's Intelligence Explosion FAQ］
** ［https://rohinshah.com/faq-career-advice-for-ai-alignment-researchers/ Advice for AI Alignment Researchers］
** ［https://www.reddit.com/r/ControlProblem/wiki/faq r/ControlProblem's FAQ］
** ［https://markxu.com/ai-safety-faqs Mark Xu's FAQ］
** ［https://aisafety.wordpress.com/ AI safety blog］ - Not yet imported.

Plex's Answer to What are some good resources on AI alignment?


Question: Might an "intelligence explosion" never occur?

Answer: [http://www.amazon.com/dp/<PHONE>/ Dreyfus] and [http://www.amazon.com/dp/<PHONE>/ Penrose] have argued that human cognitive abilities can't be emulated by a computational machine. [http://citeseerx.ist.psu.edu/viewdoc/download?doi꞊10.1.1.120.749&rep꞊rep1&type꞊pdf Searle] and [http://citeseerx.ist.psu.edu/viewdoc/download?doi꞊10.1.1.4.5828&rep꞊rep1&type꞊pdf Block] argue that certain kinds of machines cannot have a mind (consciousness, intentionality, etc.). But these objections [http://consc.net/papers/singularity.pdf need not concern] those who predict an intelligence explosion.

We can reply to Dreyfus and Penrose by noting that an intelligence explosion does not require an AI to be a classical computational system. And we can reply to Searle and Block by noting that an intelligence explosion does not depend on machines having consciousness or other properties of 'mind', only that it be able to solve problems better than humans can in a wide variety of unpredictable environments. As Edsger Dijkstra once said, the question of whether a machine can 'really' think is "no more interesting than the question of whether a submarine can swim."

[http://sethbaum.com/ac/2011_AI-Experts.pdf Others] who are pessimistic about an intelligence explosion occurring within the next few centuries don't have a specific objection but instead think there are hidden obstacles that will reveal themselves and slow or halt progress toward machine superintelligence.

Finally, a global catastrophe like nuclear war or a large asteroid impact could so damage human civilization that the intelligence explosion never occurs. Or, [https://oxford.universitypressscholarship.com/view/10.1093/oso/9780198570509.<PHONE>/isbn-9780198570509-book-part-29 a stable and global totalitarianism] could prevent the technological development required for an intelligence explosion to occur.

Question: Might an "intelligence explosion" never occur?

Answer:
［http://www.amazon.com/dp/0060110821/ Dreyfus］ and ［http://www.amazon.com/dp/0195106466/ Penrose］ have argued that human cognitive abilities can’t be emulated by a computational machine. ［http://citeseerx.ist.psu.edu/viewdoc/download?doi꞊10.1.1.120.749&rep꞊rep1&type꞊pdf Searle］ and ［http://citeseerx.ist.psu.edu/viewdoc/download?doi꞊10.1.1.4.5828&rep꞊rep1&type꞊pdf Block］ argue that certain kinds of machines cannot have a mind (consciousness, intentionality, etc.). But these objections ［http://consc.net/papers/singularity.pdf need not concern］ those who predict an intelligence explosion.

We can reply to Dreyfus and Penrose by noting that an intelligence explosion does not require an AI to be a classical computational system. And we can reply to Searle and Block by noting that an intelligence explosion does not depend on machines having consciousness or other properties of ‘mind’, only that it be able to solve problems better than humans can in a wide variety of unpredictable environments. As Edsger Dijkstra once said, the question of whether a machine can ‘really’ think is “no more interesting than the question of whether a submarine can swim.”

［http://sethbaum.com/ac/2011_AI-Experts.pdf Others］ who are pessimistic about an intelligence explosion occurring within the next few centuries don’t have a specific objection but instead think there are hidden obstacles that will reveal themselves and slow or halt progress toward machine superintelligence.

Finally, a global catastrophe like nuclear war or a large asteroid impact could so damage human civilization that the intelligence explosion never occurs. Or, ［https://oxford.universitypressscholarship.com/view/10.1093/oso/9780198570509.001.0001/isbn-9780198570509-book-part-29 a stable and global totalitarianism］ could prevent the technological development required for an intelligence explosion to occur.

Answer to Might an "intelligence explosion" never occur?


Question: Is the focus on the existential threat of superintelligent AI diverting too much attention from more pressing debates about AI in surveillance and the battlefield, and its potential effects on the economy?

Answer: The near term and long term aspects of AI safety are both very important to work on. Research into superintelligence is an important part of the open letter, but the actual concern is very different from the Terminator-like scenarios that most media outlets round off this issue to. A much more likely scenario is a superintelligent system with neutral or benevolent goals that is misspecified in a dangerous way. Robust design of superintelligent systems is a complex interdisciplinary research challenge that will likely take decades, so it is very important to begin the research now, and a large part of the purpose of our research program is to make that happen. That said, the alarmist media framing of the issues is hardly useful for making progress in either the near term or long term domain.

Question: Is the focus on the existential threat of superintelligent AI diverting too much attention from more pressing debates about AI in surveillance and the battlefield, and its potential effects on the economy?

Answer: The near term and long term aspects of AI safety are both very important to work on. Research into superintelligence is an important part of the open letter, but the actual concern is very different from the Terminator-like scenarios that most media outlets round off this issue to. A much more likely scenario is a superintelligent system with neutral or benevolent goals that is misspecified in a dangerous way. Robust design of superintelligent systems is a complex interdisciplinary research challenge that will likely take decades, so it is very important to begin the research now, and a large part of the purpose of our research program is to make that happen. That said, the alarmist media framing of the issues is hardly useful for making progress in either the near term or long term domain.

Answer to Is the focus on the existential threat of superintelligent AI diverting too much attention from more pressing debates about AI in surveillance and the battlefield, and its potential effects on the economy?