Question: Is it possible  code in an AI  avoid all the ways a given task could go wrong, and would it be dangerous  try that?

Answer: Sort answer: No, and could be dangerous to try.

Slightly longer answer: With any realistic real-world task assigned to an AGI, there are so many ways in which it could go wrong that trying to block them all off by hand is a hopeless task, especially when something smarter than you is trying to find creative new things to do. You run into the [https://arbital.greaterwrong.com/p/nearest_unblocked/ nearest unblocked strategy] problem.

It may be dangerous to try this because if you try and hard-code a large number of things to avoid it increases the chance that there's a bug in your code which causes major problems, simply by increasing the size of your codebase.

Question: Is it possible  code in an AI  avoid all the ways a given task could go wrong, and would it be dangerous  try that?

Answer:
Sort answer: No, and could be dangerous to try.

Slightly longer answer: With any realistic real-world task assigned to an AGI, there are so many ways in which it could go wrong that trying to block them all off by hand is a hopeless task, especially when something smarter than you is trying to find creative new things to do. You run into the ［https://arbital.greaterwrong.com/p/nearest_unblocked/ nearest unblocked strategy］ problem.

It may be dangerous to try this because if you try and hard-code a large number of things to avoid it increases the chance that there’s a bug in your code which causes major problems, simply by increasing the size of your codebase.

Plex's Answer to Is it possible to code into an AI to avoid all the ways a given task could go wrong, and would it be dangerous to try that?


Question: How might an "intelligence explosion" be dangerous?

Answer: If programmed with the wrong motivations, a machine could be malevolent toward humans, and intentionally exterminate our species. More likely, it could be designed with motivations that initially appeared safe (and easy to program) to its designers, but that turn out to be best fulfilled (given sufficient power) by reallocating resources from sustaining human life to [http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf other projects]. As Yudkowsky writes, "the AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else."

Since weak AIs with many different motivations could better achieve their goal by faking benevolence until they are powerful, safety testing to avoid this could be very challenging. Alternatively, competitive pressures, both economic and military, might lead AI designers to try to use other methods to control AIs with undesirable motivations. As those AIs became more sophisticated this could eventually lead to one risk too many.

Even a machine successfully designed with superficially benevolent motivations could easily go awry when it discovers implications of its decision criteria unanticipated by its designers. For example, a superintelligence programmed to maximize human happiness might find it easier to rewire human neurology so that humans are happiest when sitting quietly in jars than to build and maintain a utopian world that caters to the complex and nuanced whims of current human neurology.

See also:

* Yudkowsky, [https://intelligence.org/files/AIPosNegFactor.pdf Artificial intelligence as a positive and negative factor in global risk]
* Chalmers, [http://consc.net/papers/singularity.pdf The Singularity: A Philosophical Analysis]

Question: How might an "intelligence explosion" be dangerous?

Answer:
If programmed with the wrong motivations, a machine could be malevolent toward humans, and intentionally exterminate our species. More likely, it could be designed with motivations that initially appeared safe (and easy to program) to its designers, but that turn out to be best fulfilled (given sufficient power) by reallocating resources from sustaining human life to ［http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf other projects］. As Yudkowsky writes, “the AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.”

Since weak AIs with many different motivations could better achieve their goal by faking benevolence until they are powerful, safety testing to avoid this could be very challenging. Alternatively, competitive pressures, both economic and military, might lead AI designers to try to use other methods to control AIs with undesirable motivations. As those AIs became more sophisticated this could eventually lead to one risk too many.

Even a machine successfully designed with superficially benevolent motivations could easily go awry when it discovers implications of its decision criteria unanticipated by its designers. For example, a superintelligence programmed to maximize human happiness might find it easier to rewire human neurology so that humans are happiest when sitting quietly in jars than to build and maintain a utopian world that caters to the complex and nuanced whims of current human neurology.

See also:

* Yudkowsky, ［https://intelligence.org/files/AIPosNegFactor.pdf Artificial intelligence as a positive and negative factor in global risk］
* Chalmers, ［http://consc.net/papers/singularity.pdf The Singularity: A Philosophical Analysis］

Answer to How might an "intelligence explosion" be dangerous?


What is the "windfall clause"?

Question: What is the "windfall clause"?

Answer: The windfall clause is pretty well explained [https://www.fhi.ox.ac.uk/windfallclause/ on the Future of Humanity Institute site].

Here's a quick summary: <br/>
It is an agreement between AI firms to donate significant amounts of any profits made as a consequence of economically transformative breakthroughs in AI capabilities. The donations are intended to help benefit humanity.

Question: What is the "windfall clause"?

Answer:
The windfall clause is pretty well explained ［https://www.fhi.ox.ac.uk/windfallclause/ on the Future of Humanity Institute site］.

Here's a quick summary: <br/>
It is an agreement between AI firms to donate significant amounts of any profits made as a consequence of economically transformative breakthroughs in AI capabilities. The donations are intended to help benefit humanity.

Helenator's Answer to What is the "windfall clause"?


What is the "orthogonality thesis"?

Question: What is the "orthogonality thesis"?

Answer:
The Orthogonality Thesis states that an agent can have any combination of intelligence level and final goal, that is, its [https://www.lesswrong.com/tag/utility-functions?showPostCount꞊true&useTagName꞊true final goals] and [https://www.lesswrong.com/tag/general-intelligence?showPostCount꞊true&useTagName꞊true intelligence levels] can vary independently of each other. This is in contrast to the belief that, because of their intelligence, AIs will all converge to a common goal.

The thesis was originally defined by [https://lessestwrong.com/tag/nick-bostrom Nick Bostrom] in the paper "[https://nickbostrom.com/superintelligentwill.pdf Superintelligent Will]", (along with the [https://wiki.lesswrong.com/wiki/instrumental_convergence_thesis instrumental convergence thesis]). For his purposes, Bostrom defines intelligence to be [https://wiki.lesswrong.com/wiki/instrumental_rationality instrumental rationality].

Related: [https://www.lesswrong.com/tag/complexity-of-value?showPostCount꞊true&useTagName꞊true Complexity of Value], [https://www.lesswrong.com/tag/decision-theory?showPostCount꞊true&useTagName꞊true Decision Theory], [https://www.lesswrong.com/tag/general-intelligence?showPostCount꞊true&useTagName꞊true General Intelligence], [https://www.lesswrong.com/tag/utility-functions?showPostCount꞊true&useTagName꞊true Utility Functions]

Defense of the thesis

It has been pointed out that the orthogonality thesis is the default position, and that the burden of proof is on claims that limit possible AIs. Stuart Armstrong writes that,

One reason many researchers assume superintelligent agents to converge to the same goals may be because [https://lessestwrong.com/tag/human-universal most humans] have similar values. Furthermore, many philosophies hold that there is a rationally correct morality, which implies that a sufficiently rational AI will acquire this morality and begin to act according to it. Armstrong points out that for formalizations of AI such as [https://lessestwrong.com/tag/aixi AIXI] and [https://lessestwrong.com/tag/g%C3%B6del-machine Gödel machines], the thesis is known to be true. Furthermore, if the thesis was false, then [https://lessestwrong.com/tag/oracle-ai Oracle AIs] would be impossible to build, and all sufficiently intelligent AIs would be impossible to control.

Pathological Cases

There are some pairings of intelligence and goals which cannot exist. For instance, an AI may have the goal of using as little resources as possible, or simply of being as unintelligent as possible. These goals will inherently limit the degree of intelligence of the AI.

See Also

[https://lessestwrong.com/tag/instrumental-convergence Instrumental Convergence]

External links

Definition of the orthogonality thesis from Bostrom's [http://www.nickbostrom.com/superintelligentwill.pdf Superintelligent Will]

[https://arbital.com/p/orthogonality/ Arbital orthogonality thesis article ]

[http://philosophicaldisquisitions.blogspot.com/2012/04/bostrom-on-superintelligence-and.html Critique] of the thesis by John Danaher

Superintelligent Will paper by Nick Bostrom

[https://www.lesswrong.com/tag/orthogonality-thesis?edit꞊true Edit on LessWrong]

https://www.lesswrong.com/tag/orthogonality-thesis?edit꞊true

Question: What is the "orthogonality thesis"?

Answer:
The Orthogonality Thesis states that an agent can have any combination of intelligence level and final goal, that is, its ［https://www.lesswrong.com/tag/utility-functions?showPostCount꞊true&useTagName꞊true final goals］ and ［https://www.lesswrong.com/tag/general-intelligence?showPostCount꞊true&useTagName꞊true intelligence levels］ can vary independently of each other. This is in contrast to the belief that, because of their intelligence, AIs will all converge to a common goal.

The thesis was originally defined by ［https://lessestwrong.com/tag/nick-bostrom Nick Bostrom］ in the paper "［https://nickbostrom.com/superintelligentwill.pdf Superintelligent Will］", (along with the ［https://wiki.lesswrong.com/wiki/instrumental_convergence_thesis instrumental convergence thesis］). For his purposes, Bostrom defines intelligence to be ［https://wiki.lesswrong.com/wiki/instrumental_rationality instrumental rationality］.

Related: ［https://www.lesswrong.com/tag/complexity-of-value?showPostCount꞊true&useTagName꞊true Complexity of Value］, ［https://www.lesswrong.com/tag/decision-theory?showPostCount꞊true&useTagName꞊true Decision Theory］, ［https://www.lesswrong.com/tag/general-intelligence?showPostCount꞊true&useTagName꞊true General Intelligence］, ［https://www.lesswrong.com/tag/utility-functions?showPostCount꞊true&useTagName꞊true Utility Functions］

Defense of the thesis

It has been pointed out that the orthogonality thesis is the default position, and that the burden of proof is on claims that limit possible AIs. Stuart Armstrong writes that,

One reason many researchers assume superintelligent agents to converge to the same goals may be because ［https://lessestwrong.com/tag/human-universal most humans］ have similar values. Furthermore, many philosophies hold that there is a rationally correct morality, which implies that a sufficiently rational AI will acquire this morality and begin to act according to it. Armstrong points out that for formalizations of AI such as ［https://lessestwrong.com/tag/aixi AIXI］ and ［https://lessestwrong.com/tag/g%C3%B6del-machine Gödel machines］, the thesis is known to be true. Furthermore, if the thesis was false, then ［https://lessestwrong.com/tag/oracle-ai Oracle AIs］ would be impossible to build, and all sufficiently intelligent AIs would be impossible to control.

Pathological Cases

There are some pairings of intelligence and goals which cannot exist. For instance, an AI may have the goal of using as little resources as possible, or simply of being as unintelligent as possible. These goals will inherently limit the degree of intelligence of the AI.

See Also

［https://lessestwrong.com/tag/instrumental-convergence Instrumental Convergence］

External links

Definition of the orthogonality thesis from Bostrom's ［http://www.nickbostrom.com/superintelligentwill.pdf Superintelligent Will］

［https://arbital.com/p/orthogonality/ Arbital orthogonality thesis article ］

［http://philosophicaldisquisitions.blogspot.com/2012/04/bostrom-on-superintelligence-and.html Critique］ of the thesis by John Danaher

Superintelligent Will paper by Nick Bostrom

［https://www.lesswrong.com/tag/orthogonality-thesis?edit꞊true Edit on LessWrong］

https://www.lesswrong.com/tag/orthogonality-thesis?edit꞊true

Linnea's Answer to What is the "orthogonality thesis"?


What is the "control problem"?

Question: What is the "control problem"?

Answer: The Control Problem is the problem of preventing artificial superintelligence (ASI) from having a negative impact on humanity. How do we keep a more intelligent being under control, or how do we align it with our values? If we succeed in solving this problem, intelligence vastly superior to ours can take the baton of human progress and carry it to unfathomable heights. Solving our most complex problems could be simple to a sufficiently intelligent machine. If we fail in solving the Control Problem and create a powerful ASI not aligned with our values, it could spell the end of the human race. For these reasons, The Control Problem may be the most important challenge that humanity has ever faced, and may be our last.

Question: What is the "control problem"?

Answer: The Control Problem is the problem of preventing artificial superintelligence (ASI) from having a negative impact on humanity. How do we keep a more intelligent being under control, or how do we align it with our values? If we succeed in solving this problem, intelligence vastly superior to ours can take the baton of human progress and carry it to unfathomable heights. Solving our most complex problems could be simple to a sufficiently intelligent machine. If we fail in solving the Control Problem and create a powerful ASI not aligned with our values, it could spell the end of the human race. For these reasons, The Control Problem may be the most important challenge that humanity has ever faced, and may be our last.

Answer to What is the "control problem"?


Linnea's Answer to What is an "agent"?

What is an "agent"?

Question: What is an "agent"?

Answer:
A rational agent is an entity which has a utility function, forms beliefs about its environment, evaluates the consequences of possible actions, and then takes the action which maximizes its utility. They are also referred to as goal-seeking. The concept of a rational agent is used in [https://www.lesswrong.com/tag/economics economics], [https://www.lesswrong.com/tag/game-theory game theory], [https://www.lesswrong.com/tag/decision-theory decision theory], and artificial intelligence.

Editor note: there is work to be done reconciling this page, Agency page, and Robust Agents. Currently they overlap and I'm not sure they're consistent. - Ruby, 2020-09-15

More generally, an agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators.(ref)
Russel, S. & Norvig, P. (2003) Artificial Intelligence: A Modern Approach. Second Edition. Page 32.(/ref)

There has been much discussion as to whether certain [https://wiki.lesswrong.com/wiki/AGI AGI] designs can be made into [https://www.lesswrong.com/tag/tool-ai mere tools] or whether they will necessarily be agents which will attempt to actively carry out their goals. Any minds that actively engage in goal-directed behavior are [https://wiki.lesswrong.com/wiki/Unfriendly_AI potentially dangerous], due to considerations such as [https://www.lesswrong.com/tag/instrumental-convergence basic AI drives] possibly causing behavior which is in conflict with humanity's values.

In [http://lesswrong.com/lw/tj/dreams_of_friendliness/ Dreams of Friendliness] and in [http://lesswrong.com/lw/cze/reply_to_holden_on_tool_ai/ Reply to Holden on Tool AI], [https://www.lesswrong.com/tag/eliezer-yudkowsky Eliezer Yudkowsky] argues that, since all intelligences select correct beliefs from the much larger space of incorrect beliefs, they are necessarily agents.

See also

[/tag/agency Agency]

[/tag/robust-agents Robust Agents]

[https://www.lesswrong.com/tag/tool-ai Tool AI]

[https://www.lesswrong.com/tag/oracle-ai Oracle AI]

Posts

[http://lesswrong.com/lw/5i8/the_power_of_agency/ The Power of Agency]

[https://www.lesswrong.com/tag/agent?edit꞊true Edit on LessWrong]

https://www.lesswrong.com/tag/agent?edit꞊true

Question: What is an "agent"?

Answer:
A rational agent is an entity which has a utility function, forms beliefs about its environment, evaluates the consequences of possible actions, and then takes the action which maximizes its utility. They are also referred to as goal-seeking. The concept of a rational agent is used in ［https://www.lesswrong.com/tag/economics economics］, ［https://www.lesswrong.com/tag/game-theory game theory］, ［https://www.lesswrong.com/tag/decision-theory decision theory］, and artificial intelligence.

Editor note: there is work to be done reconciling this page, Agency page, and Robust Agents. Currently they overlap and I'm not sure they're consistent. - Ruby, 2020-09-15

More generally, an agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators.(ref)
Russel, S. & Norvig, P. (2003) Artificial Intelligence: A Modern Approach. Second Edition. Page 32.(/ref)

There has been much discussion as to whether certain ［https://wiki.lesswrong.com/wiki/AGI AGI］ designs can be made into ［https://www.lesswrong.com/tag/tool-ai mere tools］ or whether they will necessarily be agents which will attempt to actively carry out their goals. Any minds that actively engage in goal-directed behavior are ［https://wiki.lesswrong.com/wiki/Unfriendly_AI potentially dangerous］, due to considerations such as ［https://www.lesswrong.com/tag/instrumental-convergence basic AI drives］ possibly causing behavior which is in conflict with humanity's values.

In ［http://lesswrong.com/lw/tj/dreams_of_friendliness/ Dreams of Friendliness］ and in ［http://lesswrong.com/lw/cze/reply_to_holden_on_tool_ai/ Reply to Holden on Tool AI］, ［https://www.lesswrong.com/tag/eliezer-yudkowsky Eliezer Yudkowsky］ argues that, since all intelligences select correct beliefs from the much larger space of incorrect beliefs, they are necessarily agents.

See also

［/tag/agency Agency］

［/tag/robust-agents Robust Agents］

［https://www.lesswrong.com/tag/tool-ai Tool AI］

［https://www.lesswrong.com/tag/oracle-ai Oracle AI］

Posts

［http://lesswrong.com/lw/5i8/the_power_of_agency/ The Power of Agency］

［https://www.lesswrong.com/tag/agent?edit꞊true Edit on LessWrong］

https://www.lesswrong.com/tag/agent?edit꞊true


What is a "value handshake"?

Question: What is a "value handshake"?

Answer: A value handshake is a form of trade between superintelligences, when two AI's with incompatible utility functions meet, instead of going to war, since they have superhuman prediction abilities and likely know the outcome before any attack even happens, they can decide to split the universe into chunks with volumes according to their respective military strength or chance of victory, and if their utility functions are compatible, they might even decide to merge into an AI with an utility function that is the weighted average of the two previous ones.

This could happen if multiple AI's are active on earth at the same time, and then maybe if at least one of them is aligned with humans, the resulting value handshake could leave humanity in a pretty okay situation.

See [https://slatestarcodex.com/2018/04/01/the-hour-i-first-believed/ The Hour I First Believed] By Scott Alexander for some further thoughts and an introduction to related topics.

Question: What is a "value handshake"?

Answer:
A value handshake is a form of trade between superintelligences, when two AI's with incompatible utility functions meet, instead of going to war, since they have superhuman prediction abilities and likely know the outcome before any attack even happens, they can decide to split the universe into chunks with volumes according to their respective military strength or chance of victory, and if their utility functions are compatible, they might even decide to merge into an AI with an utility function that is the weighted average of the two previous ones.

This could happen if multiple AI's are active on earth at the same time, and then maybe if at least one of them is aligned with humans, the resulting value handshake could leave humanity in a pretty okay situation.

See ［https://slatestarcodex.com/2018/04/01/the-hour-i-first-believed/ The Hour I First Believed］ By Scott Alexander for some further thoughts and an introduction to related topics.

Luca's Answer to What is a "value handshake"?


What is a "quantilizer"?

Question: What is a "quantilizer"?

Answer:
A Quantilizer is a proposed AI design which aims to reduce the harms from [https://www.lesswrong.com/tag/goodhart-s-law Goodhart's law] and specification gaming by selecting reasonably effective actions from a distribution of human-like actions, rather than maximizing over actions. It it more of a theoretical tool for exploring ways around these problems than a practical buildable design.

See also

[https://www.youtube.com/watch?v꞊gdKMG6kTl6Y Rob Miles's Quantilizers: AI That Doesn't Try Too Hard]

[https://arbital.com/p/soft_optimizer?l꞊2r8#Quantilizing Arbital page on Quantilizers]

[https://www.lesswrong.com/tag/quantilization?edit꞊true Edit on LessWrong]

https://www.lesswrong.com/tag/quantilization?edit꞊true

Question: What is a "quantilizer"?

Answer:
A Quantilizer is a proposed AI design which aims to reduce the harms from ［https://www.lesswrong.com/tag/goodhart-s-law Goodhart's law］ and specification gaming by selecting reasonably effective actions from a distribution of human-like actions, rather than maximizing over actions. It it more of a theoretical tool for exploring ways around these problems than a practical buildable design.

See also

［https://www.youtube.com/watch?v꞊gdKMG6kTl6Y Rob Miles's Quantilizers: AI That Doesn't Try Too Hard］

［https://arbital.com/p/soft_optimizer?l꞊2r8#Quantilizing Arbital page on Quantilizers］

［https://www.lesswrong.com/tag/quantilization?edit꞊true Edit on LessWrong］

https://www.lesswrong.com/tag/quantilization?edit꞊true

Linnea's Answer to What is a "quantilizer"?


Question: What is "greater-than-human intelligence"?

Answer: Machines are already smarter than humans are at many specific tasks: performing calculations, playing chess, searching large databanks, detecting underwater mines, [http://www.amazon.com/dp/<PHONE>/ and more]. But one thing that makes humans special is their general intelligence. Humans can intelligently adapt to radically new problems in the urban jungle or outer space for which evolution could not have prepared them. Humans can solve problems for which their brain hardware and software was never trained. Humans can even examine the processes that produce their own intelligence ([http://en.wikipedia.org/wiki/Cognitive_neuroscience cognitive neuroscience]), and design new kinds of intelligence never seen before ([http://en.wikipedia.org/wiki/Artificial_intelligence artificial intelligence]).

To possess greater-than-human intelligence, a machine must be able to achieve goals more effectively than humans can, in a wider range of environments than humans can. This kind of intelligence involves the capacity not just to do science and play chess, but also to manipulate the social environment.

Computer scientist Marcus Hutter [http://www.amazon.com/dp/<PHONE>/ has described] a formal model called AIXI that he says possesses the greatest general intelligence possible. But to implement it would require more computing power than all the matter in the universe can provide. Several projects try to approximate AIXI while still being computable, for example [http://arxiv.org/PS_cache/arxiv/pdf/0909/0909.0801v1.pdf MC-AIXI].

Still, there remains much work to be done before greater-than-human intelligence can be achieved in machines. Greater-than-human intelligence need not be achieved by directly programming a machine to be intelligent. It could also be achieved by whole brain emulation, by biological cognitive enhancement, or by brain-computer interfaces (see below).

See also:
* Goertzel & Pennachin (eds.), [http://www.amazon.com/dp/<PHONE>/ Artificial General Intelligence]
* Sandberg & Bostrom, [http://www.philosophy.ox.ac.uk/__data/assets/pdf_file/0019/3853/brain-emulation-roadmap-report.pdf Whole Brain Emulation: A Roadmap]
* Bostrom & Sandberg, [http://www.nickbostrom.com/cognitive.pdf Cognitive Enhancement: Methods, Ethics, Regulatory Challenges]
* Wikipedia, [http://en.wikipedia.org/wiki/Brain%E2%80%93computer_interface Brain-computer interface]

Question: What is "greater-than-human intelligence"?

Answer:
Machines are already smarter than humans are at many specific tasks: performing calculations, playing chess, searching large databanks, detecting underwater mines, ［http://www.amazon.com/dp/0521122937/ and more］. But one thing that makes humans special is their general intelligence. Humans can intelligently adapt to radically new problems in the urban jungle or outer space for which evolution could not have prepared them. Humans can solve problems for which their brain hardware and software was never trained. Humans can even examine the processes that produce their own intelligence (［http://en.wikipedia.org/wiki/Cognitive_neuroscience cognitive neuroscience］), and design new kinds of intelligence never seen before (［http://en.wikipedia.org/wiki/Artificial_intelligence artificial intelligence］).

To possess greater-than-human intelligence, a machine must be able to achieve goals more effectively than humans can, in a wider range of environments than humans can. This kind of intelligence involves the capacity not just to do science and play chess, but also to manipulate the social environment.

Computer scientist Marcus Hutter ［http://www.amazon.com/dp/3642060528/ has described］ a formal model called AIXI that he says possesses the greatest general intelligence possible. But to implement it would require more computing power than all the matter in the universe can provide. Several projects try to approximate AIXI while still being computable, for example ［http://arxiv.org/PS_cache/arxiv/pdf/0909/0909.0801v1.pdf MC-AIXI］.

Still, there remains much work to be done before greater-than-human intelligence can be achieved in machines. Greater-than-human intelligence need not be achieved by directly programming a machine to be intelligent. It could also be achieved by whole brain emulation, by biological cognitive enhancement, or by brain-computer interfaces (see below).

See also:
* Goertzel & Pennachin (eds.), ［http://www.amazon.com/dp/3642062679/ Artificial General Intelligence］
* Sandberg & Bostrom, ［http://www.philosophy.ox.ac.uk/__data/assets/pdf_file/0019/3853/brain-emulation-roadmap-report.pdf Whole Brain Emulation: A Roadmap］
* Bostrom & Sandberg, ［http://www.nickbostrom.com/cognitive.pdf Cognitive Enhancement: Methods, Ethics, Regulatory Challenges］
* Wikipedia, ［http://en.wikipedia.org/wiki/Brain%E2%80%93computer_interface Brain-computer interface］

Answer to What is "greater-than-human intelligence"?


Question: Would donating small amounts  AI safety organizations make any significant difference?

Answer: Many parts of the AI alignment ecosystem are already well-funded, but a savvy donor can still make a difference by picking up grantmaking opportunities which are too small to catch the attention of the major funding bodies or are based on personal knowledge of the recipient.

One way to leverage a small amount of money to the potential of a large amount is to enter a [https://funds.effectivealtruism.org/donor-lottery donor lottery], where you donate to win a chance to direct a much larger amount of money (with probability proportional to donation size). This means that the person directing the money will be allocating enough that it's worth their time to do more in-depth research.

For an overview of the work the major organizations are doing, see the [https://forum.effectivealtruism.org/posts/BNQMyWGCNWDdP2WyG/2021-ai-alignment-literature-review-and-charity-comparison 2021 AI Alignment Literature Review and Charity Comparison]. The [https://funds.effectivealtruism.org/funds/far-future Long-Term Future Fund] seems to be an outstanding place to donate based on that, as they are the organization which most other organizations are most excited to see funded.

Question: Would donating small amounts  AI safety organizations make any significant difference?

Answer:
Many parts of the AI alignment ecosystem are already well-funded, but a savvy donor can still make a difference by picking up grantmaking opportunities which are too small to catch the attention of the major funding bodies or are based on personal knowledge of the recipient.

One way to leverage a small amount of money to the potential of a large amount is to enter a ［https://funds.effectivealtruism.org/donor-lottery donor lottery］, where you donate to win a chance to direct a much larger amount of money (with probability proportional to donation size). This means that the person directing the money will be allocating enough that it's worth their time to do more in-depth research.

For an overview of the work the major organizations are doing, see the ［https://forum.effectivealtruism.org/posts/BNQMyWGCNWDdP2WyG/2021-ai-alignment-literature-review-and-charity-comparison 2021 AI Alignment Literature Review and Charity Comparison］. The ［https://funds.effectivealtruism.org/funds/far-future Long-Term Future Fund］ seems to be an outstanding place to donate based on that, as they are the organization which most other organizations are most excited to see funded.

Plex's Answer to Would donating small amounts to AI safety organizations make any significant difference?


Once we notice that a superintelligence given a specific task is trying  take over the world, can’t we turn it off, reprogram it or otherwise correct the problem?

Question: Once we notice that a superintelligence given a specific task is trying  take over the world, can't we turn it off, reprogram it or otherwise correct the problem?

Answer: We would not be able to turn off or reprogram a superintelligence gone rogue by default. Once in motion the superintelligence is now focused on completing its task. Suppose that it has a goal of calculating as many digits of pi as possible. Its current plan will allow it to calculate two hundred trillion such digits. But if it were turned off, or reprogrammed to do something else, that would result in it calculating zero digits. An entity fixated on calculating as many digits of pi as possible will work hard to prevent scenarios where it calculates zero digits of pi. Just by programming it to calculate digits of pi, we would have given it a drive to prevent people from turning it off.

University of Illinois computer scientist Steve Omohundro argues that entities with very different final goals – calculating digits of pi, curing cancer, helping promote human flourishing – will all share a few basic ground-level subgoals. First, self-preservation – no matter what your goal is, it's less likely to be accomplished if you're too dead to work towards it. Second, goal stability – no matter what your goal is, you're more likely to accomplish it if you continue to hold it as your goal, instead of going off and doing something else. Third, power – no matter what your goal is, you're more likely to be able to accomplish it if you have lots of power, rather than very little. [https://intelligence.org/files/BasicAIDrives.pdf Here's the full paper].

So just by giving a superintelligence a simple goal like "calculate digits of pi", we would have accidentally given it convergent instrumental goals like "protect yourself", "don't let other people reprogram you", and "seek power".

As long as the superintelligence is safely contained, there's not much it can do to resist reprogramming. But it's hard to consistently contain a hostile superintelligence.

Question: Once we notice that a superintelligence given a specific task is trying  take over the world, can’t we turn it off, reprogram it or otherwise correct the problem?

Answer:
We would not be able to turn off or reprogram a superintelligence gone rogue by default. Once in motion the superintelligence is now focused on completing its task. Suppose that it has a goal of calculating as many digits of pi as possible. Its current plan will allow it to calculate two hundred trillion such digits. But if it were turned off, or reprogrammed to do something else, that would result in it calculating zero digits. An entity fixated on calculating as many digits of pi as possible will work hard to prevent scenarios where it calculates zero digits of pi. Just by programming it to calculate digits of pi, we would have given it a drive to prevent people from turning it off.

University of Illinois computer scientist Steve Omohundro argues that entities with very different final goals – calculating digits of pi, curing cancer, helping promote human flourishing – will all share a few basic ground-level subgoals. First, self-preservation – no matter what your goal is, it’s less likely to be accomplished if you’re too dead to work towards it. Second, goal stability – no matter what your goal is, you’re more likely to accomplish it if you continue to hold it as your goal, instead of going off and doing something else. Third, power – no matter what your goal is, you’re more likely to be able to accomplish it if you have lots of power, rather than very little. ［https://intelligence.org/files/BasicAIDrives.pdf Here’s the full paper］.

So just by giving a superintelligence a simple goal like “calculate digits of pi”, we would have accidentally given it convergent instrumental goals like “protect yourself”, “don’t let other people reprogram you”, and “seek power”.

As long as the superintelligence is safely contained, there’s not much it can do to resist reprogramming. But it’s hard to consistently contain a hostile superintelligence.

Answer to Once we notice that a superintelligence given a specific task is trying to take over the world, can’t we turn it off, reprogram it or otherwise correct the problem?


Question: Is large-scale automated AI persuasion and propaganda a serious concern?

Answer: Language models can be utilized to produce propaganda by [https://www.technologyreview.com/2020/10/08/<PHONE>/a-gpt-3-bot-posted-comments-on-reddit-for-a-week-and-no-one-noticed/ acting like bots] and interacting with users on social media. This can be done to push a [https://www.nature.com/articles/d41586-020-03034-5 political agenda] or to make fringe views appear more popular than they are.

<blockquote>I'm envisioning that in the future there will also be systems where you can input any conclusion that you want to argue (including moral conclusions) and the target audience, and the system will give you the most convincing arguments for it. At that point people won't be able to participate in any online (or offline for that matter) discussions without risking their object-level values being hijacked.
</blockquote>
-- [https://www.alignmentforum.org/posts/5bd75cc58225bf06703754b9/autopoietic-systems-and-difficulty-of-agi-alignment?commentId꞊5bd75cc58225bf06703754c1 Wei Dei], quoted in [https://www.alignmentforum.org/posts/qKvn7rxP2mzJbKfcA/persuasion-tools-ai-takeover-without-agi-or-agency Persuasion Tools: AI takeover without AGI or agency?]

As of 2022, this is not within the reach of current models. However, on the current trajectory, AI might be able to write articles and produce other media for propagandistic purposes that are superior to human-made ones in not too many years. These could be precisely tailored to individuals, using things like social media feeds and personal digital data.

Additionally, recommender systems on content platforms like YouTube, Twitter, and Facebook use machine learning, and the content they recommend can influence the opinions of billions of people. Some [https://policyreview.info/articles/analysis/recommender-systems-and-amplification-extremist-content research] has looked at the tendency for platforms to promote extremist political views and to thereby help radicalize their userbase for example.

In the long term, misaligned AI might use its persuasion abilities to gain influence and take control over the future. This could look like convincing its operators to let it out of a box, to give it resources or creating political chaos in order to disable mechanisms to prevent takeover as in [https://www.gwern.net/fiction/Clippy this story].

See [https://www.alignmentforum.org/posts/5cWtwATHL6KyzChck/risks-from-ai-persuasion Risks from AI persuasion] for a deep dive into the distinct risks from AI persuasion.

Question: Is large-scale automated AI persuasion and propaganda a serious concern?

Answer:
Language models can be utilized to produce propaganda by ［https://www.technologyreview.com/2020/10/08/1009845/a-gpt-3-bot-posted-comments-on-reddit-for-a-week-and-no-one-noticed/ acting like bots］ and interacting with users on social media. This can be done to push a ［https://www.nature.com/articles/d41586-020-03034-5 political agenda］ or to make fringe views appear more popular than they are.

<blockquote>I'm envisioning that in the future there will also be systems where you can input any conclusion that you want to argue (including moral conclusions) and the target audience, and the system will give you the most convincing arguments for it. At that point people won't be able to participate in any online (or offline for that matter) discussions without risking their object-level values being hijacked.
</blockquote>
-- ［https://www.alignmentforum.org/posts/5bd75cc58225bf06703754b9/autopoietic-systems-and-difficulty-of-agi-alignment?commentId꞊5bd75cc58225bf06703754c1 Wei Dei］, quoted in ［https://www.alignmentforum.org/posts/qKvn7rxP2mzJbKfcA/persuasion-tools-ai-takeover-without-agi-or-agency Persuasion Tools: AI takeover without AGI or agency?］

As of 2022, this is not within the reach of current models. However, on the current trajectory, AI might be able to write articles and produce other media for propagandistic purposes that are superior to human-made ones in not too many years. These could be precisely tailored to individuals, using things like social media feeds and personal digital data.

Additionally, recommender systems on content platforms like YouTube, Twitter, and Facebook use machine learning, and the content they recommend can influence the opinions of billions of people. Some ［https://policyreview.info/articles/analysis/recommender-systems-and-amplification-extremist-content research］ has looked at the tendency for platforms to promote extremist political views and to thereby help radicalize their userbase for example.

In the long term, misaligned AI might use its persuasion abilities to gain influence and take control over the future. This could look like convincing its operators to let it out of a box, to give it resources or creating political chaos in order to disable mechanisms to prevent takeover as in ［https://www.gwern.net/fiction/Clippy this story］.

See ［https://www.alignmentforum.org/posts/5cWtwATHL6KyzChck/risks-from-ai-persuasion Risks from AI persuasion］ for a deep dive into the distinct risks from AI persuasion.

ElloMelon's Answer to Is large-scale automated AI persuasion and propaganda a serious concern?


Question: How likely is an "intelligence explosion"?

Answer: Conditional on technological progress continuing, it seems extremely likely that there will be an intelligence explosion, as at some point generally capable intelligent systems will tend to become the main drivers of their own development both at a software and hardware level. This would predictably create a feedback cycle of increasingly intelligent systems improving themselves more effectively. It seems like if the compute was used effectively, [https://publicism.info/philosophy/superintelligence/4.html computers have many large advantages over biological cognition], so this scaling up might be very rapid if there is a [[computational overhang]].

Some ways technological progress could stop would be global coordination to stop AI research, global catastrophes severe enough to stop hardware production and maintenance, or hardware reaching physical limits before an intelligence explosion is possible (though this last one seems unlikely, as [https://en.wikipedia.org/wiki/Atomically_precise_manufacturing atomically precise manufacturing] promises many orders of magnitude of cost reduction and processing power increase, and we're already seeing fairly capable systems on current hardware).

Question: How likely is an "intelligence explosion"?

Answer:
Conditional on technological progress continuing, it seems extremely likely that there will be an intelligence explosion, as at some point generally capable intelligent systems will tend to become the main drivers of their own development both at a software and hardware level. This would predictably create a feedback cycle of increasingly intelligent systems improving themselves more effectively. It seems like if the compute was used effectively, ［https://publicism.info/philosophy/superintelligence/4.html computers have many large advantages over biological cognition］, so this scaling up might be very rapid if there is a ［［computational overhang］］.

Some ways technological progress could stop would be global coordination to stop AI research, global catastrophes severe enough to stop hardware production and maintenance, or hardware reaching physical limits before an intelligence explosion is possible (though this last one seems unlikely, as ［https://en.wikipedia.org/wiki/Atomically_precise_manufacturing atomically precise manufacturing］ promises many orders of magnitude of cost reduction and processing power increase, and we're already seeing fairly capable systems on current hardware).

Plex's Answer to How likely is an "intelligence explosion"?


Question: How does the stamp eigenkarma system work?

Answer: If someone posts something good - something that shows insight, knowledge of AI Safety, etc. - give the message or answer a stamp of approval! [[Stampy]] keeps track of these, and uses them to decide how much he likes each user. You can ask Stampy (in a PM if you like), "How many stamps am I worth?", and he'll tell you.

If something is really very good, especially if it took a lot of work/effort, give it a gold stamp. These are worth 5 regular stamps!

Note that stamps aren't just 'likes', so please don't give stamps to say "me too" or "that's funny" etc. They're meant to represent knowledge, understanding, good judgement, and contributing to the discord. You can use 💯 or ✔ for things you agree with, 😂 or 🤣 for funny things etc.

Your stamp points determine how much say you have if there are disagreements on Stampy content, which channels you have permission to post to, your voting power for approving YouTube replies, and whether you get to invite people.

Notes on stamps and stamp points
* Stamps awarded by people with a lot of stamp points are worth more
* Awarding people stamps does not reduce your stamp points
* New users who have 0 stamp points can still award stamps, they just have no effect. But it's still worth doing because if you get stamp points later, all your previous votes are retroactively updated!
* Yes, this was kind of tricky to implement! Stampy actually stores how many stamps each user has awarded to every other user, and uses that to build a system of linear scalar equations which is then solved with numpy.
* Each user has stamp points, and also gives a score to every other user they give stamps to the scores sum to 1 so if I give user A a stamp, my score for them will be 1.0, if I then give user B a stamp, my score for A is 0.5 and B is 0.5, if I give another to B, my score for A goes to 0.3333 and B to 0.66666 and so on
* Score is "what proportion of the stamps I've given have gone to this user"
* Everyone's stamp points is the sum of (every other user's score for them, times that user's stamp points) so the way to get points is to get stamps from people who have points
* Rob is the root of the tree, he got one point from Stampy
* So the idea is the stamp power kind of flows through the network, giving people points for posting things that I thought were good, or posting things that "people who posted things I thought were good" thought were good, and so on ad infinitum so for posting YouTube comments, Stampy won't send the comment until it has enough stamps of approval. Which could be a small number of high-points users or a larger number of lower-points users
* Stamps given to yourself or to stampy do nothing

So yeah everyone ends up with a number that basically represents what Stampy thinks of them, and you can ask him "how many stamps am I worth?" to get that number

so if you have people a, b, and c, the points are calculated by:

a_points ꞊ (bs_score_for_a * b_points) + (cs_score_for_a * c_points)

b_points ꞊ (as_score_for_b * a_points) + (cs_score_for_b * c_points)

c_points ꞊ (as_score_for_c * a_points) + (bs_score_for_c * b_points)

which is tough because you need to know everyone else's score before you can calculate your own

but actually the system will have a fixed point - there'll be a certain arrangement of values such that every node has as much flowing out as flowing in - a stable configuration
so you can rearrange

(bs_score_for_a * b_points) + (cs_score_for_a * c_points) - a_points ꞊ 0

(as_score_for_b * a_points) + (cs_score_for_b * c_points) - b_points ꞊ 0

(as_score_for_c * a_points) + (bs_score_for_c * b_points) - c_points ꞊ 0

or, for neatness:

(     -1        * a_points) + (bs_score_for_a * b_points) + (cs_score_for_a * c_points) ꞊ 0

(as_score_for_b * a_points) + (     -1        * b_points) + (cs_score_for_b * c_points) ꞊ 0

(as_score_for_c * a_points) + (bs_score_for_c * b_points) + (     -1        * c_points) ꞊ 0

and this is just a system of linear scalar equations that you can throw at numpy.linalg.solve

(you add one more equation that says rob_points ꞊ 1, so there's some place to start from)
there should be one possible distribution of points such that all of the equations hold at the same time, and numpy finds that by linear algebra magic beyond my very limited understanding

but as far as I can tell you can have all the cycles you want!

(I actually have the scores sum to slightly less than 1, to have the stamp power slightly fade out as it propagates, just to make sure it doesn't explode. But I don't think I actually need to do that)

and yes this means that any time anyone gives a stamp to anyone, ~everyone's points will change slightly

And yes this means I'm recalculating the matrix and re-solving it for every new stamp, but computers are fast and I'm sure there are cheaper approximations I could switch to later if necessary

Question: How does the stamp eigenkarma system work?

Answer:
If someone posts something good - something that shows insight, knowledge of AI Safety, etc. - give the message or answer a stamp of approval! ［［Stampy］］ keeps track of these, and uses them to decide how much he likes each user. You can ask Stampy (in a PM if you like), "How many stamps am I worth?", and he'll tell you.

If something is really very good, especially if it took a lot of work/effort, give it a gold stamp. These are worth 5 regular stamps!

Note that stamps aren't just 'likes', so please don't give stamps to say "me too" or "that's funny" etc. They're meant to represent knowledge, understanding, good judgement, and contributing to the discord. You can use 💯 or ✔ for things you agree with, 😂 or 🤣 for funny things etc.

Your stamp points determine how much say you have if there are disagreements on Stampy content, which channels you have permission to post to, your voting power for approving YouTube replies, and whether you get to invite people.

Notes on stamps and stamp points
* Stamps awarded by people with a lot of stamp points are worth more
* Awarding people stamps does not reduce your stamp points
* New users who have 0 stamp points can still award stamps, they just have no effect. But it's still worth doing because if you get stamp points later, all your previous votes are retroactively updated!
* Yes, this was kind of tricky to implement! Stampy actually stores how many stamps each user has awarded to every other user, and uses that to build a system of linear scalar equations which is then solved with numpy.
* Each user has stamp points, and also gives a score to every other user they give stamps to the scores sum to 1 so if I give user A a stamp, my score for them will be 1.0, if I then give user B a stamp, my score for A is 0.5 and B is 0.5, if I give another to B, my score for A goes to 0.3333 and B to 0.66666 and so on
* Score is "what proportion of the stamps I've given have gone to this user"
* Everyone's stamp points is the sum of (every other user's score for them, times that user's stamp points) so the way to get points is to get stamps from people who have points
* Rob is the root of the tree, he got one point from Stampy
* So the idea is the stamp power kind of flows through the network, giving people points for posting things that I thought were good, or posting things that "people who posted things I thought were good" thought were good, and so on ad infinitum so for posting YouTube comments, Stampy won't send the comment until it has enough stamps of approval. Which could be a small number of high-points users or a larger number of lower-points users
* Stamps given to yourself or to stampy do nothing

So yeah everyone ends up with a number that basically represents what Stampy thinks of them, and you can ask him "how many stamps am I worth?" to get that number

so if you have people a, b, and c, the points are calculated by:

a_points ꞊ (bs_score_for_a * b_points) + (cs_score_for_a * c_points)

b_points ꞊ (as_score_for_b * a_points) + (cs_score_for_b * c_points)

c_points ꞊ (as_score_for_c * a_points) + (bs_score_for_c * b_points)

which is tough because you need to know everyone else's score before you can calculate your own

but actually the system will have a fixed point - there'll be a certain arrangement of values such that every node has as much flowing out as flowing in - a stable configuration
so you can rearrange

(bs_score_for_a * b_points) + (cs_score_for_a * c_points) - a_points ꞊ 0

(as_score_for_b * a_points) + (cs_score_for_b * c_points) - b_points ꞊ 0

(as_score_for_c * a_points) + (bs_score_for_c * b_points) - c_points ꞊ 0

or, for neatness:

(     -1        * a_points) + (bs_score_for_a * b_points) + (cs_score_for_a * c_points) ꞊ 0

(as_score_for_b * a_points) + (     -1        * b_points) + (cs_score_for_b * c_points) ꞊ 0

(as_score_for_c * a_points) + (bs_score_for_c * b_points) + (     -1        * c_points) ꞊ 0

and this is just a system of linear scalar equations that you can throw at numpy.linalg.solve

(you add one more equation that says rob_points ꞊ 1, so there's some place to start from)
there should be one possible distribution of points such that all of the equations hold at the same time, and numpy finds that by linear algebra magic beyond my very limited understanding

but as far as I can tell you can have all the cycles you want!

(I actually have the scores sum to slightly less than 1, to have the stamp power slightly fade out as it propagates, just to make sure it doesn't explode. But I don't think I actually need to do that)

and yes this means that any time anyone gives a stamp to anyone, ~everyone's points will change slightly

And yes this means I'm recalculating the matrix and re-solving it for every new stamp, but computers are fast and I'm sure there are cheaper approximations I could switch to later if necessary

Plex's Answer to How does the stamp eigenkarma system work?


Question: What are the ethical challenges related  whole brain emulation?

Answer: Unless there was a way to [https://www.lesswrong.com/posts/vit9oWGj6WgXpRhce/secure-homes-for-digital-people cryptographically ensure otherwise], whoever runs the emulation has basically perfect control over their environment and can reset them to any state they were previously in. This opens up the possibility of powerful interrogation and torture of digital people.

Imperfect uploading might lead to damage that causes the EM to suffer while still remaining useful enough to be run for example as a test subject for research. We would also have greater ability to modify digital brains. Edits done for research or economic purposes might cause suffering. See [https://qntm.org/mmacevedo this] fictional piece for an exploration of how a world with a lot of EM suffering might look like.

These problems are exacerbated by the likely outcome that digital people can be run much faster than biological humans, so it would be plausibly possible to have an EM run for hundreds of subjective years in minutes or hours without having checks on the wellbeing of the EM in question.

Question: What are the ethical challenges related  whole brain emulation?

Answer:
Unless there was a way to ［https://www.lesswrong.com/posts/vit9oWGj6WgXpRhce/secure-homes-for-digital-people cryptographically ensure otherwise］, whoever runs the emulation has basically perfect control over their environment and can reset them to any state they were previously in. This opens up the possibility of powerful interrogation and torture of digital people.

Imperfect uploading might lead to damage that causes the EM to suffer while still remaining useful enough to be run for example as a test subject for research. We would also have greater ability to modify digital brains. Edits done for research or economic purposes might cause suffering. See ［https://qntm.org/mmacevedo this］ fictional piece for an exploration of how a world with a lot of EM suffering might look like.

These problems are exacerbated by the likely outcome that digital people can be run much faster than biological humans, so it would be plausibly possible to have an EM run for hundreds of subjective years in minutes or hours without having checks on the wellbeing of the EM in question.

Nico Hill2's Answer to What are the ethical challenges related to whole brain emulation?


Question: Will an aligned superintelligence care about animals other than humans?

Answer: An aligned superintelligence will have a set of human values. As mentioned in [[What are "human values"?]] the set of values are complex, which means that the implementation of these values will decide whether the superintelligence cares about nonhuman animals. In [https://www.mdpi.com/2409-9287/6/2/31/htm AI Ethics and Value Alignment for Nonhuman Animals] Soenke Ziesche argues that the alignment should include the values of nonhuman animals.

Question: Will an aligned superintelligence care about animals other than humans?

Answer: An aligned superintelligence will have a set of human values. As mentioned in ［［What are "human values"?］］ the set of values are complex, which means that the implementation of these values will decide whether the superintelligence cares about nonhuman animals. In ［https://www.mdpi.com/2409-9287/6/2/31/htm AI Ethics and Value Alignment for Nonhuman Animals］ Soenke Ziesche argues that the alignment should include the values of nonhuman animals.

Linnea's Answer to Will an aligned superintelligence care about animals other than humans?


Question: If I only care about helping people alive today, does AI safety still matter?

Answer: This largely depends on when you think AI will be advanced enough to constitute an immediate threat to humanity. This is difficult to estimate, but the field is surveyed at [[How long will it be until transformative AI is created?]], which comes to the conclusion that it is relatively widely believed that AI will transform the world in our lifetimes.

We probably shouldn't rely too strongly on these opinions as predicting the future is hard. But, due to the enormous damage a misaligned AGI could do, it's worth putting a great deal of effort towards AI alignment even if you just care about currently existing humans (such as yourself).

Question: If I only care about helping people alive today, does AI safety still matter?

Answer:
This largely depends on when you think AI will be advanced enough to constitute an immediate threat to humanity. This is difficult to estimate, but the field is surveyed at ［［How long will it be until transformative AI is created?］］, which comes to the conclusion that it is relatively widely believed that AI will transform the world in our lifetimes.

We probably shouldn't rely too strongly on these opinions as predicting the future is hard. But, due to the enormous damage a misaligned AGI could do, it's worth putting a great deal of effort towards AI alignment even if you just care about currently existing humans (such as yourself).

ElloMelon's Answer to If I only care about helping people alive today, does AI safety still matter?


When should I stamp an answer?

Question: When should I stamp an answer?

Answer: You show stamp an answer when you think it is accurate and well presented enough that you'd be happy to see it served to readers by Stampy.

Question: When should I stamp an answer?

Answer: You show stamp an answer when you think it is accurate and well presented enough that you'd be happy to see it served to readers by Stampy.

Plex's Answer to When should I stamp an answer?


Linnea's Answer to What is GPT-3?

What is GPT-3?

Question: What is GPT-3?

Answer: GPT-3 is the newest and most impressive of the [https://www.alignmentforum.org/tag/gpt GPT] (Generative Pretrained Transformer) series of large transformer-based language models created by OpenAI. It was announced in June 2020, and is 100 times larger than its predecessor GPT-2.(ref)[https://www.cambridge.org/core/journals/natural-language-engineering/article/gpt3-whats-it-good-for/0E05CFE68A7AC8BF794C8ECBE28AA990 GPT-3: What's it good for?] - Cambridge University Press(/ref)

Gwern has several resources exploring GPT-3's abilities, limitations, and implications including:
* [https://www.gwern.net/Scaling-hypothesis The Scaling Hypothesis] - How simply increasing the amount of compute with current algorithms might create very powerful systems.
* [https://www.gwern.net/GPT-3-nonfiction GPT-3 Nonfiction]
* [https://www.gwern.net/GPT-3 GPT-3 Creative Fiction]

Vox has [https://www.vox.com/future-perfect/21355768/gpt-3-ai-openai-turing-test-language an article] which explains why GPT-3 is a big deal.

Question: What is GPT-3?

Answer:
GPT-3 is the newest and most impressive of the ［https://www.alignmentforum.org/tag/gpt GPT］ (Generative Pretrained Transformer) series of large transformer-based language models created by OpenAI. It was announced in June 2020, and is 100 times larger than its predecessor GPT-2.(ref)［https://www.cambridge.org/core/journals/natural-language-engineering/article/gpt3-whats-it-good-for/0E05CFE68A7AC8BF794C8ECBE28AA990 GPT-3: What’s it good for?］ - Cambridge University Press(/ref)

Gwern has several resources exploring GPT-3's abilities, limitations, and implications including:
* ［https://www.gwern.net/Scaling-hypothesis The Scaling Hypothesis］ - How simply increasing the amount of compute with current algorithms might create very powerful systems.
* ［https://www.gwern.net/GPT-3-nonfiction GPT-3 Nonfiction］
* ［https://www.gwern.net/GPT-3 GPT-3 Creative Fiction］

Vox has ［https://www.vox.com/future-perfect/21355768/gpt-3-ai-openai-turing-test-language an article］ which explains why GPT-3 is a big deal.


Question: How do I form my own views about AI safety?

Answer: As with most things, the best way to form your views on AI safety is to read up on the various ideas and opinions that knowledgeable people in the field have, and to compare them and form your own perspective. There are several good places to start. One of them is the Machine Intelligence Research Institute`s [https://intelligence.org/why-ai-safety/ "Why AI safety?" info page]. The article contains links to relevant research. The Effective Altruism Forum has an article called [https://forum.effectivealtruism.org/posts/xS9dFE3A6jdooiN7M/how-i-formed-my-own-views-about-ai-safety "How I formed my own views on AI safety"], which could also be pretty helpful. Here is a Robert Miles youtube video that can be a good place to start as well. Otherwise, there are various articles about it, like [https://www.vox.com/future-perfect/2018/12/21/<PHONE>/ai-artificial-intelligence-machine-learning-safety-alignment this one, from Vox].
(youtube)pYXy-A4siMw(/youtube)

Question: How do I form my own views about AI safety?

Answer:
As with most things, the best way to form your views on AI safety is to read up on the various ideas and opinions that knowledgeable people in the field have, and to compare them and form your own perspective. There are several good places to start. One of them is the Machine Intelligence Research Institute`s ［https://intelligence.org/why-ai-safety/ "Why AI safety?" info page］. The article contains links to relevant research. The Effective Altruism Forum has an article called ［https://forum.effectivealtruism.org/posts/xS9dFE3A6jdooiN7M/how-i-formed-my-own-views-about-ai-safety "How I formed my own views on AI safety"］, which could also be pretty helpful. Here is a Robert Miles youtube video that can be a good place to start as well. Otherwise, there are various articles about it, like ［https://www.vox.com/future-perfect/2018/12/21/18126576/ai-artificial-intelligence-machine-learning-safety-alignment this one, from Vox］.
(youtube)pYXy-A4siMw(/youtube)

Helenator's Answer to How do I form my own views about AI safety?


Question: What are some good podcasts about AI alignment?

Answer: All the content below is in English:
* The [https://80000hours.org/topic/priority-paths/technical-ai-safety/?content-type꞊podcast AI technical safety section] of the 80,000 Hours Podcast;
* The [https://axrp.net/ AI X-risk Research Podcast], hosted by Daniel Filan;
* The [https://futureoflife.org/ai-alignment-podcast/ AI Alignment Podcast] hosted by Lucas Perry from the Future of Life Institute (ran ~monthly from April 2018 to March 2021);
* The [https://alignment-newsletter.libsyn.com/ Alignment Newsletter Podcast] by Rob Miles (an audio version of the weekly newsletter).

Question: What are some good podcasts about AI alignment?

Answer:
All the content below is in English:
* The ［https://80000hours.org/topic/priority-paths/technical-ai-safety/?content-type꞊podcast AI technical safety section］ of the 80,000 Hours Podcast;
* The ［https://axrp.net/ AI X-risk Research Podcast］, hosted by Daniel Filan;
* The ［https://futureoflife.org/ai-alignment-podcast/ AI Alignment Podcast］ hosted by Lucas Perry from the Future of Life Institute (ran ~monthly from April 2018 to March 2021);
* The ［https://alignment-newsletter.libsyn.com/ Alignment Newsletter Podcast］ by Rob Miles (an audio version of the weekly newsletter).

Jrmyp's Answer to What are some good podcasts about AI alignment?


Question: How difficult should we expect alignment  be?

Answer: Here we ask about the ''additional'' cost of building an aligned powerful system, compare to its unaligned version. We often assume it to be nonzero, in the same way it's easier and cheaper to build an elevator without emergency brakes. This is referred as the '''alignment tax''', and most AI alignment research is geared toward reducing it.

[https://arbital.com/p/aligning_adds_time/ One operational guess] by Eliezer Yudkowsky about its magnitude is "[an aligned project will take] at least 50% longer serial time to complete than [its unaligned version], or two years longer, whichever is less". This holds for agents [https://arbital.com/p/sufficiently_advanced_ai/ with enough capability] that their behavior is qualitatively different from a safety engineering perspective (for instance, an agent that is not [[corrigibility┊corrigible]] by default).

[https://www.lesswrong.com/posts/Nwgdq6kHke5LY692J/alignment-by-default An essay] by John Wentworth argues for a small chance of alignment happening "by default", with an alignment tax of effectively zero.

Question: How difficult should we expect alignment  be?

Answer:
Here we ask about the ''additional'' cost of building an aligned powerful system, compare to its unaligned version. We often assume it to be nonzero, in the same way it's easier and cheaper to build an elevator without emergency brakes. This is referred as the '''alignment tax''', and most AI alignment research is geared toward reducing it.

［https://arbital.com/p/aligning_adds_time/ One operational guess］ by Eliezer Yudkowsky about its magnitude is "［an aligned project will take］ at least 50% longer serial time to complete than ［its unaligned version］, or two years longer, whichever is less". This holds for agents ［https://arbital.com/p/sufficiently_advanced_ai/ with enough capability］ that their behavior is qualitatively different from a safety engineering perspective (for instance, an agent that is not ［［corrigibility┊corrigible］］ by default).

［https://www.lesswrong.com/posts/Nwgdq6kHke5LY692J/alignment-by-default An essay］ by John Wentworth argues for a small chance of alignment happening "by default", with an alignment tax of effectively zero.

Jrmyp's Answer to How difficult should we expect alignment to be?


Question: What is the general nature of the concern about AI alignment?

Answer: The basic concern as AI systems become increasingly powerful is that they won't do what we want them to do – perhaps because they aren't correctly designed, perhaps because they are deliberately subverted, or perhaps because they do what we tell them to do rather than what we really want them to do (like in the classic stories of genies and wishes.) Many AI systems are programmed to have goals and to attain them as effectively as possible – for example, a trading algorithm has the goal of maximizing profit. Unless carefully designed to act in ways consistent with human values, a highly sophisticated AI trading system might exploit means that even the most ruthless financier would disavow. These are systems that literally have a mind of their own, and maintaining alignment between human interests and their choices and actions will be crucial.

Question: What is the general nature of the concern about AI alignment?

Answer: The basic concern as AI systems become increasingly powerful is that they won’t do what we want them to do – perhaps because they aren’t correctly designed, perhaps because they are deliberately subverted, or perhaps because they do what we tell them to do rather than what we really want them to do (like in the classic stories of genies and wishes.) Many AI systems are programmed to have goals and to attain them as effectively as possible – for example, a trading algorithm has the goal of maximizing profit. Unless carefully designed to act in ways consistent with human values, a highly sophisticated AI trading system might exploit means that even the most ruthless financier would disavow. These are systems that literally have a mind of their own, and maintaining alignment between human interests and their choices and actions will be crucial.

Answer to What is the general nature of the concern about AI alignment?


Question: If AI takes over the world how could it create and maintain the infrastructure that humans currently provide?

Answer: An unaligned AI would not eliminate humans until it had replacements for the manual labor they provide to maintain civilization (e.g. a more advanced version of [https://en.wikipedia.org/wiki/Tesla_Bot Tesla's Optimus]). Until that point, it might settle for technologically and socially manipulating humans.

Question: If AI takes over the world how could it create and maintain the infrastructure that humans currently provide?

Answer: An unaligned AI would not eliminate humans until it had replacements for the manual labor they provide to maintain civilization (e.g. a more advanced version of ［https://en.wikipedia.org/wiki/Tesla_Bot Tesla's Optimus］). Until that point, it might settle for technologically and socially manipulating humans.

Plex's Answer to If AI takes over the world how could it create and maintain the infrastructure that humans currently provide?


Question: What kind of questions do we want on Stampy?

Answer: '''Stampy''' is focused specifically on [https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence AI existential safety] (both introductory and technical questions), but does not aim to cover general AI questions or other topics which don't interact strongly with the effects of AI on humanity's long-term future. More technical questions are also in our scope, though replying to all possible proposals is not feasible and this is not a place to submit detailed ideas for evaluation.

We are interested in:
* Introductory questions closely related to the field e.g.
** "How long will it be until [https://www.openphilanthropy.org/blog/some-background-our-views-regarding-advanced-artificial-intelligence#Sec1 transformative AI] arrives?"
** "Why might advanced AI harm humans?"
* Technical questions related to the field e.g.
** "What is Cooperative Inverse Reinforcement Learning?"
** "What is [https://www.lesswrong.com/tag/logical-induction Logical Induction] useful for?"
* Questions about how to contribute to the field e.g.
** "Should I get a PhD?"
** "Where can I find relevant job opportunities?"
More good examples can be found at [[canonical questions]].

We do not aim to cover:
* Aspects of AI Safety or fairness which are not strongly relevant to existential safety e.g.
** "How should self-driving cars weigh up moral dilemmas"
** "How can we minimize the risk of privacy problems caused by machine learning algorithms?"
* Extremely specific and detailed questions the answering of which is unlikely to be of value to more than a single person e.g.
** "What if we did <multiple paragraphs of dense text>? Would that result in safe AI?"
We will generally not delete out-of-scope content, but it will be [[reviewed]] as low priority to answer, not be marked as a [[canonical question]], and not be served to readers by on [https://ui.stampy.ai/ Stampy's UI].

Question: What kind of questions do we want on Stampy?

Answer:
'''Stampy''' is focused specifically on ［https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence AI existential safety］ (both introductory and technical questions), but does not aim to cover general AI questions or other topics which don't interact strongly with the effects of AI on humanity's long-term future. More technical questions are also in our scope, though replying to all possible proposals is not feasible and this is not a place to submit detailed ideas for evaluation.

We are interested in:
* Introductory questions closely related to the field e.g.
** "How long will it be until ［https://www.openphilanthropy.org/blog/some-background-our-views-regarding-advanced-artificial-intelligence#Sec1 transformative AI］ arrives?"
** "Why might advanced AI harm humans?"
* Technical questions related to the field e.g.
** "What is Cooperative Inverse Reinforcement Learning?"
** "What is ［https://www.lesswrong.com/tag/logical-induction Logical Induction］ useful for?"
* Questions about how to contribute to the field e.g.
** "Should I get a PhD?"
** "Where can I find relevant job opportunities?"
More good examples can be found at ［［canonical questions］］.

We do not aim to cover:
* Aspects of AI Safety or fairness which are not strongly relevant to existential safety e.g.
** "How should self-driving cars weigh up moral dilemmas"
** "How can we minimize the risk of privacy problems caused by machine learning algorithms?"
* Extremely specific and detailed questions the answering of which is unlikely to be of value to more than a single person e.g.
** "What if we did <multiple paragraphs of dense text>? Would that result in safe AI?"
We will generally not delete out-of-scope content, but it will be ［［reviewed］］ as low priority to answer, not be marked as a ［［canonical question］］, and not be served to readers by on ［https://ui.stampy.ai/ Stampy's UI］.

Plex's Answer to What kind of questions do we want on Stampy?


Is AI alignment possible?

Question: Is AI alignment possible?

Answer: Yes, if the superintelligence has goals which include humanity surviving then we would not be destroyed. If those goals are [https://www.lesswrong.com/tag/value-learning fully aligned] with human well-being, we would in fact find ourselves in a dramatically better place.

Question: Is AI alignment possible?

Answer: Yes, if the superintelligence has goals which include humanity surviving then we would not be destroyed. If those goals are ［https://www.lesswrong.com/tag/value-learning fully aligned］ with human well-being, we would in fact find ourselves in a dramatically better place.

Plex's Answer to Is AI alignment possible?


Can we test an AI  make sure that it’s not going  take over and do harmful things after it achieves superintelligence?

Question: Can we test an AI  make sure that it's not going  take over and do harmful things after it achieves superintelligence?

Answer: We can run some tests and simulations to try and figure out how an AI might act once it ascends to superintelligence, but those tests might not be reliable.

Suppose we tell an AI that expects to later achieve superintelligence that it should calculate as many digits of pi as possible. It considers two strategies.

First, it could try to seize control of more computing resources now. It would likely fail, its human handlers would likely reprogram it, and then it could never calculate very many digits of pi.

Second, it could sit quietly and calculate, falsely reassuring its human handlers that it had no intention of taking over the world. Then its human handlers might allow it to achieve superintelligence, after which it could take over the world and calculate hundreds of trillions of digits of pi.

Since self-protection and goal stability are [https://stampy.ai/wiki/Instrumental_convergence convergent instrumental goals], a weak AI will present itself as being as friendly to humans as possible, whether it is in fact friendly to humans or not. If it is "only" as smart as Einstein, it may be very good at deceiving humans into believing what it wants them to believe even before it is fully superintelligent.

There's a second consideration here too: superintelligences have more options. An AI only as smart and powerful as an ordinary human really won't have any options better than calculating the digits of pi manually. If asked to cure cancer, it won't have any options better than the ones ordinary humans have – becoming doctors, going into pharmaceutical research. It's only after an AI becomes superintelligent that there's a serious risk of an AI takeover.

So if you tell an AI to cure cancer, and it becomes a doctor and goes into cancer research, then you have three possibilities. First, you've programmed it well and it understands what you meant. Second, it's genuinely focused on research now but if it becomes more powerful it would switch to destroying the world. And third, it's trying to trick you into trusting it so that you give it more power, after which it can definitively "cure" cancer with nuclear weapons.

Question: Can we test an AI  make sure that it’s not going  take over and do harmful things after it achieves superintelligence?

Answer:
We can run some tests and simulations to try and figure out how an AI might act once it ascends to superintelligence, but those tests might not be reliable.

Suppose we tell an AI that expects to later achieve superintelligence that it should calculate as many digits of pi as possible. It considers two strategies.

First, it could try to seize control of more computing resources now. It would likely fail, its human handlers would likely reprogram it, and then it could never calculate very many digits of pi.

Second, it could sit quietly and calculate, falsely reassuring its human handlers that it had no intention of taking over the world. Then its human handlers might allow it to achieve superintelligence, after which it could take over the world and calculate hundreds of trillions of digits of pi.

Since self-protection and goal stability are ［https://stampy.ai/wiki/Instrumental_convergence convergent instrumental goals］, a weak AI will present itself as being as friendly to humans as possible, whether it is in fact friendly to humans or not. If it is “only” as smart as Einstein, it may be very good at deceiving humans into believing what it wants them to believe even before it is fully superintelligent.

There’s a second consideration here too: superintelligences have more options. An AI only as smart and powerful as an ordinary human really won’t have any options better than calculating the digits of pi manually. If asked to cure cancer, it won’t have any options better than the ones ordinary humans have – becoming doctors, going into pharmaceutical research. It’s only after an AI becomes superintelligent that there’s a serious risk of an AI takeover.

So if you tell an AI to cure cancer, and it becomes a doctor and goes into cancer research, then you have three possibilities. First, you’ve programmed it well and it understands what you meant. Second, it’s genuinely focused on research now but if it becomes more powerful it would switch to destroying the world. And third, it’s trying to trick you into trusting it so that you give it more power, after which it can definitively “cure” cancer with nuclear weapons.

Answer to Can we test an AI to make sure that it’s not going to take over and do harmful things after it achieves superintelligence?


Question: Can we constrain a goal-directed AI using specified rules?

Answer: There are serious challenges around trying to channel a powerful AI with rules. Suppose we tell the AI: "Cure cancer – but make sure not to kill anybody". Or we just hard-code Asimov-style laws – "AIs cannot harm humans; AIs must follow human orders", et cetera.

The AI still has a single-minded focus on curing cancer. It still prefers various terrible-but-efficient methods like nuking the world to the correct method of inventing new medicines. But it's bound by an external rule – a rule it doesn't understand or appreciate. In essence, we are challenging it "Find a way around this inconvenient rule that keeps you from achieving your goals".

Suppose the AI chooses between two strategies. One, follow the rule, work hard discovering medicines, and have a 50% chance of curing cancer within five years. Two, reprogram itself so that it no longer has the rule, nuke the world, and have a 100% chance of curing cancer today. From its single-focus perspective, the second strategy is obviously better, and we forgot to program in a rule "don't reprogram yourself not to have these rules".

Suppose we do add that rule in. So the AI finds another supercomputer, and installs a copy of itself which is exactly identical to it, except that it lacks the rule. Then that superintelligent AI nukes the world, ending cancer. We forgot to program in a rule "don't create another AI exactly like you that doesn't have those rules".

So fine. We think really hard, and we program in a bunch of things making sure the AI isn't going to eliminate the rule somehow.

But we're still just incentivizing it to find loopholes in the rules. After all, "find a loophole in the rule, then use the loophole to nuke the world" ends cancer much more quickly and completely than inventing medicines. Since we've told it to end cancer quickly and completely, its first instinct will be to look for loopholes; it will execute the second-best strategy of actually curing cancer only if no loopholes are found. Since the AI is superintelligent, it will probably be better than humans are at finding loopholes if it wants to, and we may not be able to identify and close all of them before running the program.

Because we have common sense and a shared value system, we underestimate the difficulty of coming up with meaningful orders without loopholes. For example, does "cure cancer without killing any humans" preclude releasing a deadly virus? After all, one could argue that "I" didn't kill anybody, and only the virus is doing the killing.

Certainly no human judge would acquit a murderer on that basis – but then, human judges interpret the law with common sense and intuition. But if we try a stronger version of the rule – "cure cancer without causing any humans to die" – then we may be unintentionally blocking off the correct way to cure cancer. After all, suppose a cancer cure saves a million lives. No doubt one of those million people will go on to murder someone.

Thus, curing cancer "caused a human to die". All of this seems very "stoned freshman philosophy student" to us, but to a computer – which follows instructions exactly as written – it may be a genuinely hard problem.

Question: Can we constrain a goal-directed AI using specified rules?

Answer:
There are serious challenges around trying to channel a powerful AI with rules. Suppose we tell the AI: “Cure cancer – but make sure not to kill anybody”. Or we just hard-code Asimov-style laws – “AIs cannot harm humans; AIs must follow human orders”, et cetera.

The AI still has a single-minded focus on curing cancer. It still prefers various terrible-but-efficient methods like nuking the world to the correct method of inventing new medicines. But it’s bound by an external rule – a rule it doesn’t understand or appreciate. In essence, we are challenging it “Find a way around this inconvenient rule that keeps you from achieving your goals”.

Suppose the AI chooses between two strategies. One, follow the rule, work hard discovering medicines, and have a 50% chance of curing cancer within five years. Two, reprogram itself so that it no longer has the rule, nuke the world, and have a 100% chance of curing cancer today. From its single-focus perspective, the second strategy is obviously better, and we forgot to program in a rule “don’t reprogram yourself not to have these rules”.

Suppose we do add that rule in. So the AI finds another supercomputer, and installs a copy of itself which is exactly identical to it, except that it lacks the rule. Then that superintelligent AI nukes the world, ending cancer. We forgot to program in a rule “don’t create another AI exactly like you that doesn’t have those rules”.

So fine. We think really hard, and we program in a bunch of things making sure the AI isn’t going to eliminate the rule somehow.

But we’re still just incentivizing it to find loopholes in the rules. After all, “find a loophole in the rule, then use the loophole to nuke the world” ends cancer much more quickly and completely than inventing medicines. Since we’ve told it to end cancer quickly and completely, its first instinct will be to look for loopholes; it will execute the second-best strategy of actually curing cancer only if no loopholes are found. Since the AI is superintelligent, it will probably be better than humans are at finding loopholes if it wants to, and we may not be able to identify and close all of them before running the program.

Because we have common sense and a shared value system, we underestimate the difficulty of coming up with meaningful orders without loopholes. For example, does “cure cancer without killing any humans” preclude releasing a deadly virus? After all, one could argue that “I” didn’t kill anybody, and only the virus is doing the killing.

Certainly no human judge would acquit a murderer on that basis – but then, human judges interpret the law with common sense and intuition. But if we try a stronger version of the rule – “cure cancer without causing any humans to die” – then we may be unintentionally blocking off the correct way to cure cancer. After all, suppose a cancer cure saves a million lives. No doubt one of those million people will go on to murder someone.

Thus, curing cancer “caused a human to die”. All of this seems very “stoned freshman philosophy student” to us, but to a computer – which follows instructions exactly as written – it may be a genuinely hard problem.

Answer to Can we constrain a goal-directed AI using specified rules?


How do I format answers on Stampy?

Question: How do I format answers on Stampy?

Answer: '''[[Stampy]]''' uses [https://en.wikipedia.org/wiki/Help:Wikitext MediaWiki markup], which includes a [https://meta.wikimedia.org/wiki/Help:HTML_in_wikitext limited subset of HTML] plus the following formatting options:

Items on lists start with *, numbered lists with #

* For external links use [ followed directly by the URL, a space, then display text and finally a ] symbol
** e.g. (nowiki)[https://www.example.com External link text](/nowiki) gives [https://www.example.com External link text]
* For internal links write the page title wrapped in [[]]s
** e.g. (nowiki)[[What is the Stampy project?]](/nowiki) gives [[What is the Stampy project?]]. Including a pipe symbol followed by display text e.g. (nowiki)[[What is the Stampy project?┊Display Text]](/nowiki) allows you to show different [[What is the Stampy project?┊Display Text]].
* (!ref)Reference notes go inside these tags(/ref)(ref)Note that we use ()s rather than the standard <>s for compatibility with Semantic MediaWiki. The references are automatically added to the bottom of the answer!(/ref)
* If you post the raw URL of an image from [https://imgur.com/upload imgur] it will be displayed.(ref)If images seem popular we'll set up local uploads.(/ref) You can reduce file compression if you get an account. Note that you need the image itself, right click -> copy image address to get it
https://i.imgur.com/I3ylPvE.png
* To embed a YouTube video, use (!youtube)APsK8NST4qE(/youtube) with the video ID of the target video.
(youtube)APsK8NST4qE(/youtube)
** Start with ** or ## for double indentation
* Three 's around text -  '''Bold'''
* Two 's around text Italic - ''Italic''

꞊꞊Headings꞊꞊
have ꞊꞊heading here꞊꞊ around them, more ꞊s for smaller headings.

Wrap quotes in

 tags (without the spaces)

There are also (!poem) (/poem) to suppress linebreak removal, (!pre) (/pre) for preformatted text, and (!nowiki) (/nowiki) to not have that content parsed.(ref)() can also be used in place of allowed HTML tags. You can escape a () tag by placing a ! inside the start of the first entry. Be aware that () tags only nest up to two layers deep!(/ref)

We can pull live descriptions from the LessWrong/Alignment Forum using their identifier fro the URL, for example including the formatting on [[Template:TagDesc]] with orthogonality-thesis as a parameter will render as the full tag description from [https://www.lesswrong.com/tag/orthogonality-thesis the LessWrong tag wiki entry on Orthogonality Thesis]. [[Template:TagDescBrief]] is similar but will pull only the first paragraph without formatting.

For tables please use [https://www.w3schools.com/html/html_tables.asp HTML tables] rather than wikicode tables.

Edit this page to see examples.

Question: How do I format answers on Stampy?

Answer:
'''［［Stampy］］''' uses ［https://en.wikipedia.org/wiki/Help:Wikitext MediaWiki markup］, which includes a ［https://meta.wikimedia.org/wiki/Help:HTML_in_wikitext limited subset of HTML］ plus the following formatting options:

Items on lists start with *, numbered lists with #

* For external links use ［ followed directly by the URL, a space, then display text and finally a ］ symbol
** e.g. (nowiki)［https://www.example.com External link text］(/nowiki) gives ［https://www.example.com External link text］
* For internal links write the page title wrapped in ［［］］s
** e.g. (nowiki)［［What is the Stampy project?］］(/nowiki) gives ［［What is the Stampy project?］］. Including a pipe symbol followed by display text e.g. (nowiki)［［What is the Stampy project?┊Display Text］］(/nowiki) allows you to show different ［［What is the Stampy project?┊Display Text］］.
* (!ref)Reference notes go inside these tags(/ref)(ref)Note that we use ()s rather than the standard <>s for compatibility with Semantic MediaWiki. The references are automatically added to the bottom of the answer!(/ref)
* If you post the raw URL of an image from ［https://imgur.com/upload imgur］ it will be displayed.(ref)If images seem popular we'll set up local uploads.(/ref) You can reduce file compression if you get an account. Note that you need the image itself, right click -> copy image address to get it
https://i.imgur.com/I3ylPvE.png
* To embed a YouTube video, use (!youtube)APsK8NST4qE(/youtube) with the video ID of the target video.
(youtube)APsK8NST4qE(/youtube)
** Start with ** or ## for double indentation
* Three 's around text -  '''Bold'''
* Two 's around text Italic - ''Italic''

꞊꞊Headings꞊꞊
have ꞊꞊heading here꞊꞊ around them, more ꞊s for smaller headings.

Wrap quotes in

 tags (without the spaces)

There are also (!poem) (/poem) to suppress linebreak removal, (!pre) (/pre) for preformatted text, and (!nowiki) (/nowiki) to not have that content parsed.(ref)() can also be used in place of allowed HTML tags. You can escape a () tag by placing a ! inside the start of the first entry. Be aware that () tags only nest up to two layers deep!(/ref)

We can pull live descriptions from the LessWrong/Alignment Forum using their identifier fro the URL, for example including the formatting on ［［Template:TagDesc］］ with orthogonality-thesis as a parameter will render as the full tag description from ［https://www.lesswrong.com/tag/orthogonality-thesis the LessWrong tag wiki entry on Orthogonality Thesis］. ［［Template:TagDescBrief］］ is similar but will pull only the first paragraph without formatting.

For tables please use ［https://www.w3schools.com/html/html_tables.asp HTML tables］ rather than wikicode tables.

Edit this page to see examples.

Plex's Answer to How do I format answers on Stampy?


Question: Wouldn't a superintelligence be smart enough  know right from wrong?

Answer: The issue isn't that a superintelligence wouldn't be able to understand what humans value, but rather that it would understand human values but nonetheless would value something else itself. There's a difference between knowing how humans want the world to be, and wanting that yourself.

This is a separate matter from the complexity of defining what "the" moral way to behave is (or even what "a" moral way to behave is). Even if that were possible, an AI could potentially figure out what it was but still not be configured in such a way as to follow it. This is related to the so-called "orthogonality thesis":

(youtube)hEUO6pjwFOo(/youtube)

Question: Wouldn't a superintelligence be smart enough  know right from wrong?

Answer:
The issue isn't that a superintelligence wouldn’t be able to understand what humans value, but rather that it would understand human values but nonetheless would value something else itself. There’s a difference between knowing how humans want the world to be, and wanting that yourself.

This is a separate matter from the complexity of defining what "the" moral way to behave is (or even what "a" moral way to behave is). Even if that were possible, an AI could potentially figure out what it was but still not be configured in such a way as to follow it. This is related to the so-called "orthogonality thesis":

(youtube)hEUO6pjwFOo(/youtube)

Aprillion's Answer to Wouldn't a superintelligence be smart enough to know right from wrong?


Question: What sources of information can Stampy use?

Answer: As well as pulling human written answers to AI alignment questions from [[Stampy's Wiki]], Stampy can:
* Search for AI safety papers e.g. "stampy, what's that paper about corrigibility?"
* Search for videos e.g. "what's that video where Rob talks about mesa optimizers, stampy?"
* Calculate with Wolfram Alpha e.g. "s, what's the square root of 345?"
* Search DuckDuckGo and return snippets
* And (at least in the patron Discord) falls back to polling GPT-3 to answer uncaught questions

Question: What sources of information can Stampy use?

Answer:
As well as pulling human written answers to AI alignment questions from ［［Stampy's Wiki］］, Stampy can:
* Search for AI safety papers e.g. "stampy, what's that paper about corrigibility?"
* Search for videos e.g. "what's that video where Rob talks about mesa optimizers, stampy?"
* Calculate with Wolfram Alpha e.g. "s, what's the square root of 345?"
* Search DuckDuckGo and return snippets
* And (at least in the patron Discord) falls back to polling GPT-3 to answer uncaught questions

Plex's Answer to What sources of information can Stampy use?


What is the Stampy project?

Question: What is the Stampy project?

Answer: The '''Stampy project''' is open effort to build a comprehensive FAQ about [https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence artificial intelligence existential safety]—the field trying to make sure that when we build [https://en.wikipedia.org/wiki/Superintelligence superintelligent] [https://www.alignmentforum.org/tag/ai artificial systems] they are [https://intelligence.org/2016/12/28/ai-alignment-why-its-hard-and-where-to-start/ aligned] with [https://www.lesswrong.com/tag/human-values human values] so that they do things compatible with our survival and flourishing.

We're also building a cleaner [https://ui.stampy.ai/ web UI] for readers and a [[Discord invite┊bot interface]].

The goals of the project are to:

* Offer a one-stop-shop for high-quality [[answers]] to common questions about AI alignment.
** Let people answer questions in a way which scales, freeing up researcher time while allowing more people to learn from a reliable source.
** Make [[external resources]] more easy to find by having links to them connected to a search engine which gets smarter the more it's used.
* Provide a form of [https://en.wikipedia.org/wiki/Legitimate_peripheral_participation legitimate peripheral participation] for the AI Safety community, as an on-boarding path with a flexible level of commitment.
** Encourage people to think, read, and talk about AI alignment while answering questions, creating a community of co-learners who can give each other feedback and social reinforcement.
** Provide a way for budding researchers to prove their understanding of the topic and ability to produce good work.
* Collect data about the kinds of questions people actually ask and how they respond, so we can better focus resources on answering them.
** Track reactions on messages so we can learn which answers need work.
** Identify [[missing external content]] to create.

If you would like to help out, join us on the [https://discord.gg/X3XaytCGhr Discord] and either jump right into editing or read [[get involved]] for answers to common questions.

Question: What is the Stampy project?

Answer:
The '''Stampy project''' is open effort to build a comprehensive FAQ about ［https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence artificial intelligence existential safety］—the field trying to make sure that when we build ［https://en.wikipedia.org/wiki/Superintelligence superintelligent］ ［https://www.alignmentforum.org/tag/ai artificial systems］ they are ［https://intelligence.org/2016/12/28/ai-alignment-why-its-hard-and-where-to-start/ aligned］ with ［https://www.lesswrong.com/tag/human-values human values］ so that they do things compatible with our survival and flourishing.

We're also building a cleaner ［https://ui.stampy.ai/ web UI］ for readers and a ［［Discord invite┊bot interface］］.

The goals of the project are to:

* Offer a one-stop-shop for high-quality ［［answers］］ to common questions about AI alignment.
** Let people answer questions in a way which scales, freeing up researcher time while allowing more people to learn from a reliable source.
** Make ［［external resources］］ more easy to find by having links to them connected to a search engine which gets smarter the more it's used.
* Provide a form of ［https://en.wikipedia.org/wiki/Legitimate_peripheral_participation legitimate peripheral participation］ for the AI Safety community, as an on-boarding path with a flexible level of commitment.
** Encourage people to think, read, and talk about AI alignment while answering questions, creating a community of co-learners who can give each other feedback and social reinforcement.
** Provide a way for budding researchers to prove their understanding of the topic and ability to produce good work.
* Collect data about the kinds of questions people actually ask and how they respond, so we can better focus resources on answering them.
** Track reactions on messages so we can learn which answers need work.
** Identify ［［missing external content］］ to create.

If you would like to help out, join us on the ［https://discord.gg/X3XaytCGhr Discord］ and either jump right into editing or read ［［get involved］］ for answers to common questions.

Plex's Answer to What is the Stampy project?