Question: What approaches are AI alignment organizations working on?

Answer: Each major organization has a different approach. The [https://www.lesswrong.com/tag/research-agendas research agendas are detailed and complex] (see also [https://aiwatch.issarice.com/ AI Watch]). Getting more brains working on any of them (and more money to fund them) may pay off in a big way, but it's very hard to be confident which (if any) of them will actually work.

The following is a massive oversimplification, each organization actually pursues many different avenues of research, read the [https://www.lesswrong.com/posts/C4tR3BEpuWviT7Sje/2021-ai-alignment-literature-review-and-charity-comparison 2021 AI Alignment Literature Review and Charity Comparison] for much more detail. That being said:

* The [https://intelligence.org/research-guide/ Machine Intelligence Research Institute] focuses on foundational mathematical research to understand reliable reasoning, which they think is necessary to provide anything like an assurance that a seed AI built will do good things if activated.
* The [https://humancompatible.ai Center for Human-Compatible AI] focuses on [https://www.lesswrong.com/tag/inverse-reinforcement-learning Cooperative Inverse Reinforcement Learning] and [https://www.lesswrong.com/posts/qPoaA5ZSedivA4xJa/our-take-on-chai-s-research-agenda-in-under-1500-words Assistance Games], a new paradigm for AI where they try to optimize for doing the kinds of things humans want rather than for a pre-specified utility function
* [https://ai-alignment.com/ Paul Christano]'s [https://alignmentresearchcenter.org/ Alignment Research Center] focuses is on [https://www.lesswrong.com/posts/YTq4X6inEudiHkHDF/prosaic-ai-alignment prosaic alignment], particularly on creating tools that empower humans to understand and guide systems much smarter than ourselves. His methodology is explained on [https://ai-alignment.com/my-research-methodology-b94f2751cb2c his blog].
* The [https://www.fhi.ox.ac.uk Future of Humanity Institute] does work on [https://www.lesswrong.com/tag/crucial-considerations crucial considerations] and other x-risks, as well as AI safety research and outreach.
* [https://www.anthropic.com/ Anthropic] is a new organization exploring natural language, human feedback, scaling laws, reinforcement learning, code generation, and interpretability.
* [http://openai.com OpenAI] is in a state of flux after major changes to their safety team.
* [https://medium.com/@deepmindsafetyresearch DeepMind]'s safety team is working on various approaches designed to work with modern machine learning, and does some communication via the [https://rohinshah.com/alignment-newsletter/ Alignment Newsletter].
* [https://www.eleuther.ai/ EleutherAI] is a Machine Learning collective aiming to build large open source language models to allow more alignment research to take place.
* [https://ought.org/ Ought] is a research lab that develops mechanisms for delegating open-ended thinking to advanced machine learning systems.

There are many other projects around AI Safety, such as [https://www.youtube.com/watch?v꞊7i_f4Kbpgn4 the Windfall clause], [https://www.youtube.com/channel/UCLB7AzTwc6VFZrBsO2ucBMg Rob Miles's YouTube channel], [https://www.aisafetysupport.org/ AI Safety Support], etc.

Question: What approaches are AI alignment organizations working on?

Answer:
Each major organization has a different approach. The ［https://www.lesswrong.com/tag/research-agendas research agendas are detailed and complex］ (see also ［https://aiwatch.issarice.com/ AI Watch］). Getting more brains working on any of them (and more money to fund them) may pay off in a big way, but it’s very hard to be confident which (if any) of them will actually work.

The following is a massive oversimplification, each organization actually pursues many different avenues of research, read the ［https://www.lesswrong.com/posts/C4tR3BEpuWviT7Sje/2021-ai-alignment-literature-review-and-charity-comparison 2021 AI Alignment Literature Review and Charity Comparison］ for much more detail. That being said:

* The ［https://intelligence.org/research-guide/ Machine Intelligence Research Institute］ focuses on foundational mathematical research to understand reliable reasoning, which they think is necessary to provide anything like an assurance that a seed AI built will do good things if activated.
* The ［https://humancompatible.ai Center for Human-Compatible AI］ focuses on ［https://www.lesswrong.com/tag/inverse-reinforcement-learning Cooperative Inverse Reinforcement Learning］ and ［https://www.lesswrong.com/posts/qPoaA5ZSedivA4xJa/our-take-on-chai-s-research-agenda-in-under-1500-words Assistance Games］, a new paradigm for AI where they try to optimize for doing the kinds of things humans want rather than for a pre-specified utility function
* ［https://ai-alignment.com/ Paul Christano］'s ［https://alignmentresearchcenter.org/ Alignment Research Center］ focuses is on ［https://www.lesswrong.com/posts/YTq4X6inEudiHkHDF/prosaic-ai-alignment prosaic alignment］, particularly on creating tools that empower humans to understand and guide systems much smarter than ourselves. His methodology is explained on ［https://ai-alignment.com/my-research-methodology-b94f2751cb2c his blog］.
* The ［https://www.fhi.ox.ac.uk Future of Humanity Institute］ does work on ［https://www.lesswrong.com/tag/crucial-considerations crucial considerations］ and other x-risks, as well as AI safety research and outreach.
* ［https://www.anthropic.com/ Anthropic］ is a new organization exploring natural language, human feedback, scaling laws, reinforcement learning, code generation, and interpretability.
* ［http://openai.com OpenAI］ is in a state of flux after major changes to their safety team.
* ［https://medium.com/@deepmindsafetyresearch DeepMind］’s safety team is working on various approaches designed to work with modern machine learning, and does some communication via the ［https://rohinshah.com/alignment-newsletter/ Alignment Newsletter］.
* ［https://www.eleuther.ai/ EleutherAI］ is a Machine Learning collective aiming to build large open source language models to allow more alignment research to take place.
* ［https://ought.org/ Ought］ is a research lab that develops mechanisms for delegating open-ended thinking to advanced machine learning systems.

There are many other projects around AI Safety, such as ［https://www.youtube.com/watch?v꞊7i_f4Kbpgn4 the Windfall clause］, ［https://www.youtube.com/channel/UCLB7AzTwc6VFZrBsO2ucBMg Rob Miles’s YouTube channel］, ［https://www.aisafetysupport.org/ AI Safety Support］, etc.

Plex's Answer to What approaches are AI alignment organizations working on?


Question: How could poorly defined goals lead  such negative outcomes?

Answer: There is a broad range of possible goals that an AI might possess, but there are a few basic drives that would be useful to almost any of them. These are called instrumentally convergent goals:

# Self preservation. An agent is less likely to achieve its goal if it is not around to see to its completion.
# Goal-content integrity. An agent is less likely to achieve its goal if its goal has been changed to something else. For example, if you offer Gandhi a pill that makes him want to kill people, he will refuse to take it.
# Self-improvement. An agent is more likely to achieve its goal if it is more intelligent and better at problem-solving.
# Resource acquisition. The more resources at an agent's disposal, the more power it has to make change towards its goal. Even a purely computational goal, such as computing digits of pi, can be easier to achieve with more hardware and energy.

Because of these drives, even a seemingly simple goal could create an Artificial Superintelligence (ASI) hell-bent on taking over the world's material resources and preventing itself from being turned off. The classic example is an ASI that was programmed to maximize the output of paper clips at a paper clip factory. The ASI had no other goal specifications other than "maximize paper clips," so it converts all of the matter in the solar system into paper clips, and then sends probes to other star systems to create more factories.

Question: How could poorly defined goals lead  such negative outcomes?

Answer:
There is a broad range of possible goals that an AI might possess, but there are a few basic drives that would be useful to almost any of them. These are called instrumentally convergent goals:

# Self preservation. An agent is less likely to achieve its goal if it is not around to see to its completion.
# Goal-content integrity. An agent is less likely to achieve its goal if its goal has been changed to something else. For example, if you offer Gandhi a pill that makes him want to kill people, he will refuse to take it.
# Self-improvement. An agent is more likely to achieve its goal if it is more intelligent and better at problem-solving.
# Resource acquisition. The more resources at an agent’s disposal, the more power it has to make change towards its goal. Even a purely computational goal, such as computing digits of pi, can be easier to achieve with more hardware and energy.

Because of these drives, even a seemingly simple goal could create an Artificial Superintelligence (ASI) hell-bent on taking over the world’s material resources and preventing itself from being turned off. The classic example is an ASI that was programmed to maximize the output of paper clips at a paper clip factory. The ASI had no other goal specifications other than “maximize paper clips,” so it converts all of the matter in the solar system into paper clips, and then sends probes to other star systems to create more factories.

Answer to How could poorly defined goals lead to such negative outcomes?


Question: What are the style guidelines for writing for Stampy?

Answer: Avoid directly responding to the question in the answer, repeat the relevant part of the question instead. For example, if the question is "Can we do X", answer "We might be able to do X, if we can do Y", not "Yes, if we can manage Y". This way, the answer will also work for the questions "Why can't we do X" and "What would happen if we tried to do X".

Linking to external sites is strongly encouraged, one of the most valuable things Stampy can do is help people find other parts of the alignment information ecosystem.

Consider enclosing newly introduced terms, likely to be unfamiliar to many readers, in speech marks. If unsure, Google the term (in speech marks!) and see if it shows up anywhere other than LessWrong, the Alignment Forum, etc. Be judicious, as it's easy to use too many, but used carefully they can psychologically cushion newbies from a lot of unfamiliar terminology - in this context they're saying something like "we get that we're hitting you with a lot of new vocab, and you might not know what this term means yet".

When selecting related questions, there shouldn't be more than four unless there's a really good reason for that (some questions are asking for it, like the "Why can't we just..." question). It's also recommended to include at least one more "enticing" question to draw users in (relating to the more sensational, sci-fi, philosophical/ethical side of things) alongside more bland/neutral questions.

Question: What are the style guidelines for writing for Stampy?

Answer:
Avoid directly responding to the question in the answer, repeat the relevant part of the question instead. For example, if the question is "Can we do X", answer "We might be able to do X, if we can do Y", not "Yes, if we can manage Y". This way, the answer will also work for the questions "Why can't we do X" and "What would happen if we tried to do X".

Linking to external sites is strongly encouraged, one of the most valuable things Stampy can do is help people find other parts of the alignment information ecosystem.

Consider enclosing newly introduced terms, likely to be unfamiliar to many readers, in speech marks. If unsure, Google the term (in speech marks!) and see if it shows up anywhere other than LessWrong, the Alignment Forum, etc. Be judicious, as it's easy to use too many, but used carefully they can psychologically cushion newbies from a lot of unfamiliar terminology - in this context they're saying something like "we get that we're hitting you with a lot of new vocab, and you might not know what this term means yet".

When selecting related questions, there shouldn't be more than four unless there's a really good reason for that (some questions are asking for it, like the "Why can't we just..." question). It's also recommended to include at least one more "enticing" question to draw users in (relating to the more sensational, sci-fi, philosophical/ethical side of things) alongside more bland/neutral questions.

Plex's Answer to What are the style guidelines for writing for Stampy?


Question: Are there types of advanced AI that would be safer than others?

Answer: We don't yet know which AI architectures are safe; learning more about this is one of the goals of FLI's grants program. AI researchers are generally very responsible people who want their work to better humanity. If there are certain AI designs that turn out to be unsafe, then AI researchers will want to know this so they can develop alternative AI systems.

Question: Are there types of advanced AI that would be safer than others?

Answer: We don’t yet know which AI architectures are safe; learning more about this is one of the goals of FLI's grants program. AI researchers are generally very responsible people who want their work to better humanity. If there are certain AI designs that turn out to be unsafe, then AI researchers will want to know this so they can develop alternative AI systems.

Answer to Are there types of advanced AI that would be safer than others?


Question: How might a superintelligence socially manipulate humans?

Answer: People tend to imagine AIs as being like nerdy humans – brilliant at technology but clueless about social skills. There is no reason to expect this – persuasion and manipulation is a different kind of skill from solving mathematical proofs, but it's still a skill, and an intellect as far beyond us as we are beyond lions might be smart enough to replicate or exceed the "charming sociopaths" who can naturally win friends and followers despite a lack of normal human emotions.

A superintelligence might be able to analyze human psychology deeply enough to understand the hopes and fears of everyone it negotiates with. Single humans using psychopathic social manipulation have done plenty of harm – Hitler leveraged his skill at oratory and his understanding of people's darkest prejudices to take over a continent. Why should we expect superintelligences to do worse than humans far less skilled than they?

More outlandishly, a superintelligence might just skip language entirely and figure out a weird pattern of buzzes and hums that causes conscious thought to seize up, and which knocks anyone who hears it into a weird hypnotizable state in which they'll do anything the superintelligence asks. It sounds kind of silly to me, but then, nuclear weapons probably would have sounded kind of silly to lions sitting around speculating about what humans might be able to accomplish. When you're dealing with something unbelievably more intelligent than you are, you should probably expect the unexpected.

Question: How might a superintelligence socially manipulate humans?

Answer:
People tend to imagine AIs as being like nerdy humans – brilliant at technology but clueless about social skills. There is no reason to expect this – persuasion and manipulation is a different kind of skill from solving mathematical proofs, but it’s still a skill, and an intellect as far beyond us as we are beyond lions might be smart enough to replicate or exceed the “charming sociopaths” who can naturally win friends and followers despite a lack of normal human emotions.

A superintelligence might be able to analyze human psychology deeply enough to understand the hopes and fears of everyone it negotiates with. Single humans using psychopathic social manipulation have done plenty of harm – Hitler leveraged his skill at oratory and his understanding of people’s darkest prejudices to take over a continent. Why should we expect superintelligences to do worse than humans far less skilled than they?

More outlandishly, a superintelligence might just skip language entirely and figure out a weird pattern of buzzes and hums that causes conscious thought to seize up, and which knocks anyone who hears it into a weird hypnotizable state in which they’ll do anything the superintelligence asks. It sounds kind of silly to me, but then, nuclear weapons probably would have sounded kind of silly to lions sitting around speculating about what humans might be able to accomplish. When you’re dealing with something unbelievably more intelligent than you are, you should probably expect the unexpected.

Answer to How might a superintelligence socially manipulate humans?


Question: Why think that AI can outperform humans?

Answer: Machines are already smarter than humans are at many specific tasks: performing calculations, playing chess, searching large databanks, detecting underwater mines, and more. However, human intelligence continues to dominate machine intelligence in generality.

A powerful chess computer is "narrow": it can't play other games. In contrast, humans have problem-solving abilities that allow us to adapt to new contexts and excel in many domains other than what the ancestral environment prepared us for.

In the absence of a [https://intelligence.org/2013/06/19/what-is-intelligence-2/ formal definition of "intelligence"] (and therefore of [https://intelligence.org/2013/08/11/what-is-agi/ "artificial intelligence"]), we can heuristically cite humans' perceptual, inferential, and deliberative faculties (as opposed to, e.g., our physical strength or agility) and say that intelligence is "those kinds of things." On this conception, intelligence is a bundle of distinct faculties — albeit a very important bundle that includes our capacity for science.

Our cognitive abilities stem from high-level patterns in our brains, and these patterns can be instantiated in silicon as well as carbon. This tells us that general AI is possible, though it doesn't tell us how difficult it is. If intelligence is sufficiently difficult to understand, then we may arrive at machine intelligence by scanning and emulating human brains or by some trial-and-error process (like evolution), rather than by hand-coding a software agent.

If machines can achieve human equivalence in cognitive tasks, then it is very likely that they can eventually outperform humans. There is little reason to expect that biological evolution, with its lack of foresight and planning, would have hit upon the optimal algorithms for general intelligence (any more than it hit upon the optimal flying machine in birds). Beyond qualitative improvements in cognition, Nick Bostrom notes [http://aiimpacts.org/sources-of-advantage-for-artificial-intelligence/ more straightforward advantages we could realize in digital minds], e.g.:
* editability — "It is easier to experiment with parameter variations in software than in neural wetware."
* speed — "The speed of light is more than a million times greater than that of neural transmission, synaptic spikes dissipate more than a million times more heat than is thermodynamically necessary, and current transistor frequencies are more than a million times faster than neuron spiking frequencies."
* serial depth — On short timescales, machines can carry out much longer sequential processes.
* storage capacity — Computers can plausibly have greater working and long-term memory.
* size — Computers can be much larger than a human brain.
* duplicability — Copying software onto new hardware can be much faster and higher-fidelity than biological reproduction.
Any one of these advantages could give an AI reasoner an edge over a human reasoner, or give a group of AI reasoners an edge over a human group. Their combination suggests that digital minds could surpass human minds more quickly and decisively than we might expect.

Question: Why think that AI can outperform humans?

Answer:
Machines are already smarter than humans are at many specific tasks: performing calculations, playing chess, searching large databanks, detecting underwater mines, and more. However, human intelligence continues to dominate machine intelligence in generality.

A powerful chess computer is “narrow”: it can’t play other games. In contrast, humans have problem-solving abilities that allow us to adapt to new contexts and excel in many domains other than what the ancestral environment prepared us for.

In the absence of a ［https://intelligence.org/2013/06/19/what-is-intelligence-2/ formal definition of “intelligence”］ (and therefore of ［https://intelligence.org/2013/08/11/what-is-agi/ “artificial intelligence”］), we can heuristically cite humans’ perceptual, inferential, and deliberative faculties (as opposed to, e.g., our physical strength or agility) and say that intelligence is “those kinds of things.” On this conception, intelligence is a bundle of distinct faculties — albeit a very important bundle that includes our capacity for science.

Our cognitive abilities stem from high-level patterns in our brains, and these patterns can be instantiated in silicon as well as carbon. This tells us that general AI is possible, though it doesn’t tell us how difficult it is. If intelligence is sufficiently difficult to understand, then we may arrive at machine intelligence by scanning and emulating human brains or by some trial-and-error process (like evolution), rather than by hand-coding a software agent.

If machines can achieve human equivalence in cognitive tasks, then it is very likely that they can eventually outperform humans. There is little reason to expect that biological evolution, with its lack of foresight and planning, would have hit upon the optimal algorithms for general intelligence (any more than it hit upon the optimal flying machine in birds). Beyond qualitative improvements in cognition, Nick Bostrom notes ［http://aiimpacts.org/sources-of-advantage-for-artificial-intelligence/ more straightforward advantages we could realize in digital minds］, e.g.:
* editability — “It is easier to experiment with parameter variations in software than in neural wetware.”
* speed — “The speed of light is more than a million times greater than that of neural transmission, synaptic spikes dissipate more than a million times more heat than is thermodynamically necessary, and current transistor frequencies are more than a million times faster than neuron spiking frequencies.”
* serial depth — On short timescales, machines can carry out much longer sequential processes.
* storage capacity — Computers can plausibly have greater working and long-term memory.
* size — Computers can be much larger than a human brain.
* duplicability — Copying software onto new hardware can be much faster and higher-fidelity than biological reproduction.
Any one of these advantages could give an AI reasoner an edge over a human reasoner, or give a group of AI reasoners an edge over a human group. Their combination suggests that digital minds could surpass human minds more quickly and decisively than we might expect.

Answer to Why think that AI can outperform humans?


Question: How might an AI achieve a seemingly beneficial goal via inappropriate means?

Answer: Imagine, for example, that you are tasked with reducing traffic congestion in San Francisco at all costs, i.e. you do not take into account any other constraints. How would you do it? You might start by just timing traffic lights better. But wouldn't there be less traffic if all the bridges closed down from 5 to 10AM, preventing all those cars from entering the city? Such a measure obviously violates common sense, and subverts the purpose of improving traffic, which is to help people get around – but it is consistent with the goal of "reducing traffic congestion".

Question: How might an AI achieve a seemingly beneficial goal via inappropriate means?

Answer: Imagine, for example, that you are tasked with reducing traffic congestion in San Francisco at all costs, i.e. you do not take into account any other constraints. How would you do it? You might start by just timing traffic lights better. But wouldn’t there be less traffic if all the bridges closed down from 5 to 10AM, preventing all those cars from entering the city? Such a measure obviously violates common sense, and subverts the purpose of improving traffic, which is to help people get around – but it is consistent with the goal of “reducing traffic congestion”.

Answer to How might an AI achieve a seemingly beneficial goal via inappropriate means?


What are brain-computer interfaces?

Question: What are brain-computer interfaces?

Answer: A brain-computer interface (BCI) is a direct communication pathway between the brain and a computer device. BCI research is heavily funded, and has already met dozens of successes. Three successes in human BCIs are [http://archives.cnn.com/2002/HEALTH/06/13/cov.bionic.eye/index.html a device] that restores (partial) sight to the blind, [http://en.wikipedia.org/wiki/Cochlear_implant cochlear implants] that restore hearing to the deaf, and [https://pubmed.ncbi.nlm.nih.gov/<PHONE>/ a device] that allows use of an artificial hand by direct thought.

Such device restore impaired functions, but many researchers expect to also augment and improve normal human abilities with BCIs. [http://edboyden.org/ Ed Boyden] is researching these opportunities as the lead of the [http://syntheticneurobiology.org/ Synthetic Neurobiology Group] at MIT. Such devices might hasten the arrival of an intelligence explosion, if only by improving human intelligence so that the hard problems of AI can be solved more rapidly.

See also:

Wikipedia, [http://en.wikipedia.org/wiki/Brain%E2%80%93computer_interface Brain-computer interface]

Question: What are brain-computer interfaces?

Answer:
A brain-computer interface (BCI) is a direct communication pathway between the brain and a computer device. BCI research is heavily funded, and has already met dozens of successes. Three successes in human BCIs are ［http://archives.cnn.com/2002/HEALTH/06/13/cov.bionic.eye/index.html a device］ that restores (partial) sight to the blind, ［http://en.wikipedia.org/wiki/Cochlear_implant cochlear implants］ that restore hearing to the deaf, and ［https://pubmed.ncbi.nlm.nih.gov/16838014/ a device］ that allows use of an artificial hand by direct thought.

Such device restore impaired functions, but many researchers expect to also augment and improve normal human abilities with BCIs. ［http://edboyden.org/ Ed Boyden］ is researching these opportunities as the lead of the ［http://syntheticneurobiology.org/ Synthetic Neurobiology Group］ at MIT. Such devices might hasten the arrival of an intelligence explosion, if only by improving human intelligence so that the hard problems of AI can be solved more rapidly.

See also:

Wikipedia, ［http://en.wikipedia.org/wiki/Brain%E2%80%93computer_interface Brain-computer interface］

Answer to What are brain-computer interfaces?


Question: What is a duplicate question on Stampy's Wiki?

Answer: An existing question is a duplicate of a new one if it is reasonable to expect whoever asked the new question to be satisfied if they received an answer to the existing question instead.

Question: What is a duplicate question on Stampy's Wiki?

Answer: An existing question is a duplicate of a new one if it is reasonable to expect whoever asked the new question to be satisfied if they received an answer to the existing question instead.

Plex's Answer to What is a duplicate question on Stampy's Wiki?


Question: Is expecting large returns from AI self-improvement just following an exponential trend line off a cliff?

Answer: Blindly following the trendlines while forecasting technological progress is certainly a risk (affectionately known in AI circles as "pulling a Kurzweill"), but sometimes taking an exponential trend seriously is the right response.

Consider economic doubling times. In 1 AD, the world GDP was about $20 billion; it took a thousand years, until 1000 AD, for that to double to $40 billion. But it only took five hundred more years, until 1500, or so, for the economy to double again. And then it only took another three hundred years or so, until 1800, for the economy to double a third time.
Someone in 1800 might calculate the trend line and say this was ridiculous, that it implied the economy would be doubling every ten years or so in the beginning of the 21st century. But in fact, this is how long the economy takes to double these days. To a medieval, used to a thousand-year doubling time (which was based mostly on population growth!), an economy that doubled every ten years might seem inconceivable. To us, it seems normal.

Likewise, in 1965 Gordon Moore noted that semiconductor complexity seemed to double every eighteen months. During his own day, there were about five hundred transistors on a chip; he predicted that would soon double to a thousand, and a few years later to two thousand.
Almost as soon as Moore's Law become well-known, people started saying it was absurd to follow it off a cliff – such a law would imply a million transistors per chip in 1990, a hundred million in 2000, ten billion transistors on every chip by 2015! More transistors on a single chip than existed on all the computers in the world! Transistors the size of molecules! But of course all of these things happened; the ridiculous exponential trend proved more accurate than the naysayers.

None of this is to say that exponential trends are always right, just that they are sometimes right even when it seems they can't possibly be. We can't be sure that a computer using its own intelligence to discover new ways to increase its intelligence will enter a positive feedback loop and achieve superintelligence in seemingly impossibly short time scales. It's just one more possibility, a worry to place alongside all the other worrying reasons to expect a moderate or hard takeoff.

Question: Is expecting large returns from AI self-improvement just following an exponential trend line off a cliff?

Answer:
Blindly following the trendlines while forecasting technological progress is certainly a risk (affectionately known in AI circles as “pulling a Kurzweill”), but sometimes taking an exponential trend seriously is the right response.

Consider economic doubling times. In 1 AD, the world GDP was about $20 billion; it took a thousand years, until 1000 AD, for that to double to $40 billion. But it only took five hundred more years, until 1500, or so, for the economy to double again. And then it only took another three hundred years or so, until 1800, for the economy to double a third time.
Someone in 1800 might calculate the trend line and say this was ridiculous, that it implied the economy would be doubling every ten years or so in the beginning of the 21st century. But in fact, this is how long the economy takes to double these days. To a medieval, used to a thousand-year doubling time (which was based mostly on population growth!), an economy that doubled every ten years might seem inconceivable. To us, it seems normal.

Likewise, in 1965 Gordon Moore noted that semiconductor complexity seemed to double every eighteen months. During his own day, there were about five hundred transistors on a chip; he predicted that would soon double to a thousand, and a few years later to two thousand.
Almost as soon as Moore’s Law become well-known, people started saying it was absurd to follow it off a cliff – such a law would imply a million transistors per chip in 1990, a hundred million in 2000, ten billion transistors on every chip by 2015! More transistors on a single chip than existed on all the computers in the world! Transistors the size of molecules! But of course all of these things happened; the ridiculous exponential trend proved more accurate than the naysayers.

None of this is to say that exponential trends are always right, just that they are sometimes right even when it seems they can’t possibly be. We can’t be sure that a computer using its own intelligence to discover new ways to increase its intelligence will enter a positive feedback loop and achieve superintelligence in seemingly impossibly short time scales. It’s just one more possibility, a worry to place alongside all the other worrying reasons to expect a moderate or hard takeoff.

Answer to Is expecting large returns from AI self-improvement just following an exponential trend line off a cliff?


How can I contact the Stampy team?

Question: How can I contact the Stampy team?

Answer: The '''[https://discord.com/channels/6775469013395<PHONE>339504646 Rob Miles AI Discord]''' is the hub of all things [[Stampy]]. If you want to be part of the project and don't have access yet, ask plex#1874 on Discord (or [[User_talk:756254556811165756┊plex]] on wiki).

You can also talk to us on the [https://discord.gg/cEzKz8QCpa public Discord]! Try [https://discord.com/channels/8939371061943<PHONE>858750986 #suggestions] or [https://discord.com/channels/8939371061943<PHONE>194399257 #general], depending on what you want to talk about.

Question: How can I contact the Stampy team?

Answer:
The '''［https://discord.com/channels/677546901339504640/677546901339504646 Rob Miles AI Discord］''' is the hub of all things ［［Stampy］］. If you want to be part of the project and don't have access yet, ask plex#1874 on Discord (or ［［User_talk:756254556811165756┊plex］］ on wiki).

You can also talk to us on the ［https://discord.gg/cEzKz8QCpa public Discord］! Try ［https://discord.com/channels/893937106194399254/908318480858750986 #suggestions］ or ［https://discord.com/channels/893937106194399254/893937106194399257 #general］, depending on what you want to talk about.

Plex's Answer to How can I contact the Stampy team?


Aren’t there some pretty easy ways  eliminate these potential problems?

Question: Aren't there some pretty easy ways  eliminate these potential problems?

Answer: It might look like there are straightforward ways to eliminate the problems of unaligned superintelligence, but so far all of them turn out to have hidden difficulties. There are many open problems identified by the research community which a solution would need to reliably overcome to be successful.

Question: Aren’t there some pretty easy ways  eliminate these potential problems?

Answer: It might look like there are straightforward ways to eliminate the problems of unaligned superintelligence, but so far all of them turn out to have hidden difficulties. There are many open problems identified by the research community which a solution would need to reliably overcome to be successful.

Answer to Aren’t there some pretty easy ways to eliminate these potential problems?


Question: Why would great intelligence produce great power?

Answer: Intelligence is powerful. One might say that "Intelligence is no match for a gun, or for someone with lots of money," but both guns and money were produced by intelligence. If not for our intelligence, humans would still be foraging the savannah for food.

Intelligence is what caused humans to dominate the planet in the blink of an eye (on evolutionary timescales). Intelligence is what allows us to eradicate diseases, and what gives us the potential to eradicate ourselves with nuclear war. Intelligence gives us superior strategic skills, superior social skills, superior economic productivity, and the power of invention.

A machine with superintelligence would be able to hack into vulnerable networks via the internet, commandeer those resources for additional computing power, take over mobile machines connected to networks connected to the internet, use them to build additional machines, perform scientific experiments to understand the world better than humans can, invent quantum computing and nanotechnology, manipulate the social world better than we can, and do whatever it can to give itself more power to achieve its goals — all at a speed much faster than humans can respond to.

See also
* Legg (2008). [http://www.vetta.org/documents/Machine_Super_Intelligence.pdf Machine Super Intelligence]. PhD Thesis. IDSIA.
* Yudkowsky (2007). [http://yudkowsky.net/singularity/power The Power of Intelligence].

Question: Why would great intelligence produce great power?

Answer:
Intelligence is powerful. One might say that “Intelligence is no match for a gun, or for someone with lots of money,” but both guns and money were produced by intelligence. If not for our intelligence, humans would still be foraging the savannah for food.

Intelligence is what caused humans to dominate the planet in the blink of an eye (on evolutionary timescales). Intelligence is what allows us to eradicate diseases, and what gives us the potential to eradicate ourselves with nuclear war. Intelligence gives us superior strategic skills, superior social skills, superior economic productivity, and the power of invention.

A machine with superintelligence would be able to hack into vulnerable networks via the internet, commandeer those resources for additional computing power, take over mobile machines connected to networks connected to the internet, use them to build additional machines, perform scientific experiments to understand the world better than humans can, invent quantum computing and nanotechnology, manipulate the social world better than we can, and do whatever it can to give itself more power to achieve its goals — all at a speed much faster than humans can respond to.

See also
* Legg (2008). ［http://www.vetta.org/documents/Machine_Super_Intelligence.pdf Machine Super Intelligence］. PhD Thesis. IDSIA.
* Yudkowsky (2007). ［http://yudkowsky.net/singularity/power The Power of Intelligence］.

Answer to Why would great intelligence produce great power?


Question: What are the different possible AI takeoff speeds?

Answer: A slow takeoff is where AI capabilities improve gradually, giving us plenty of time to adapt. In a moderate takeoff we might see accelerating progress, but we still won't be caught off guard by a dramatic change. Whereas, in a fast or hard takeoff AI would go from being not very generally competent to sufficiently superhuman to control the future too fast for humans to course correct if something goes wrong.

The article [https://www.alignmentforum.org/posts/YgNYA6pj2hPSDQiTE/distinguishing-definitions-of-takeoff Distinguishing definitions of takeoff] goes into more detail on this.

Question: What are the different possible AI takeoff speeds?

Answer:
A slow takeoff is where AI capabilities improve gradually, giving us plenty of time to adapt. In a moderate takeoff we might see accelerating progress, but we still won’t be caught off guard by a dramatic change. Whereas, in a fast or hard takeoff AI would go from being not very generally competent to sufficiently superhuman to control the future too fast for humans to course correct if something goes wrong.

The article ［https://www.alignmentforum.org/posts/YgNYA6pj2hPSDQiTE/distinguishing-definitions-of-takeoff Distinguishing definitions of takeoff］ goes into more detail on this.

Answer to What are the different possible AI takeoff speeds?


What are alternate phrasings for?

Question: What are alternate phrasings for?

Answer: Alternate phrasings are used to improve the semantic search which Stampy uses to serve people questions, by giving alternate ways to say a question which might trigger a match when the main wording won't. They should generally only be used when there is a significantly different wording, rather than for only very minor changes.

Question: What are alternate phrasings for?

Answer: Alternate phrasings are used to improve the semantic search which Stampy uses to serve people questions, by giving alternate ways to say a question which might trigger a match when the main wording won't. They should generally only be used when there is a significantly different wording, rather than for only very minor changes.

Plex's Answer to What are alternate phrasings for?


Could AI have basic emotions?

Question: Could AI have basic emotions?

Answer: In principle it could (if you believe in functionalism), but it probably won't. One way to ensure that AI has human-like emotions would be to copy the way human brain works, but that's not what most AI researchers are trying to do.

It's similar to how once some people thought we will build mechanical horses to pull our vehicles, but it turned out it's much easier to build a car. AI probably doesn't need emotions or maybe even consciousness to be powerful, and the first AGIs that will get built will be the ones that are easiest to build.

Question: Could AI have basic emotions?

Answer:
In principle it could (if you believe in functionalism), but it probably won't. One way to ensure that AI has human-like emotions would be to copy the way human brain works, but that's not what most AI researchers are trying to do.

It's similar to how once some people thought we will build mechanical horses to pull our vehicles, but it turned out it's much easier to build a car. AI probably doesn't need emotions or maybe even consciousness to be powerful, and the first AGIs that will get built will be the ones that are easiest to build.

Filip's Answer to Could AI have basic emotions?


Answer to Why not just put it in a box?

Why not just put it in a box?

Question: Why not just put it in a box?

Answer: One possible way to ensure the safety of a powerful AI system is to keep it contained in a software environment. There is nothing intrinsically wrong with this procedure - keeping an AI system in a secure software environment would make it safer than letting it roam free. However, even AI systems inside software environments might not be safe enough.

Humans sometimes put dangerous humans inside boxes to limit their ability to influence the external world. Sometimes, these humans escape their boxes. The security of a prison depends on certain assumptions, which can be violated. [https://medium.com/breakingasia/yoshie-shiratori-the-incredible-story-of-a-man-no-prison-could-hold-6d79a67345f5 Yoshie Shiratori] reportedly escaped prison by weakening the door-frame with miso soup and dislocating his shoulders.

Human written software has a [https://spacepolicyonline.com/news/boeing-software-errors-could-have-doomed-starliners-uncrewed-test-flight/ high defect rate]; we should expect a perfectly secure system to be difficult to create. If humans construct a software system they think is secure, it is possible that the security relies on a false assumption. A powerful AI system could potentially learn how its hardware works and manipulate bits to send radio signals. It could fake a malfunction and attempt social engineering when the engineers look at its code. As the saying goes: in order for someone to do something we had imagined was impossible requires only that they have a better imagination.

Experimentally, humans have [https://yudkowsky.net/singularity/aibox/ convinced] other humans to let them out of the box. Spooky.

Question: Why not just put it in a box?

Answer:
One possible way to ensure the safety of a powerful AI system is to keep it contained in a software environment. There is nothing intrinsically wrong with this procedure - keeping an AI system in a secure software environment would make it safer than letting it roam free. However, even AI systems inside software environments might not be safe enough.

Humans sometimes put dangerous humans inside boxes to limit their ability to influence the external world. Sometimes, these humans escape their boxes. The security of a prison depends on certain assumptions, which can be violated. ［https://medium.com/breakingasia/yoshie-shiratori-the-incredible-story-of-a-man-no-prison-could-hold-6d79a67345f5 Yoshie Shiratori］ reportedly escaped prison by weakening the door-frame with miso soup and dislocating his shoulders.

Human written software has a ［https://spacepolicyonline.com/news/boeing-software-errors-could-have-doomed-starliners-uncrewed-test-flight/ high defect rate］; we should expect a perfectly secure system to be difficult to create. If humans construct a software system they think is secure, it is possible that the security relies on a false assumption. A powerful AI system could potentially learn how its hardware works and manipulate bits to send radio signals. It could fake a malfunction and attempt social engineering when the engineers look at its code. As the saying goes: in order for someone to do something we had imagined was impossible requires only that they have a better imagination.

Experimentally, humans have ［https://yudkowsky.net/singularity/aibox/ convinced］ other humans to let them out of the box. Spooky.


Question: Why is safety important for smarter-than-human AI?

Answer: Present-day AI algorithms already demand special safety guarantees when they must act in important domains without human oversight, particularly when they or their environment can change over time:

Achieving these gains [from autonomous systems] will depend on development of entirely new methods for enabling "trust in autonomy" through verification and validation (V&V) of the near-infinite state systems that result from high levels of [adaptability] and autonomy. In effect, the number of possible input states that such systems can be presented with is so large that not only is it impossible to test all of them directly, it is not even feasible to test more than an insignificantly small fraction of them. Development of such systems is thus inherently unverifiable by today's methods, and as a result their operation in all but comparatively trivial applications is uncertifiable.

It is possible to develop systems having high levels of autonomy, but it is the lack of suitable V&V methods that prevents all but relatively low levels of autonomy from being certified for use.
- Office of the US Air Force Chief Scientist (2010). [http://www.defenseinnovationmarketplace.mil/resources/AF_TechnologyHorizons2010-2030.pdf Technology Horizons: A Vision for Air Force Science and Technology 2010-30].

As AI capabilities improve, it will become easier to give AI systems greater autonomy, flexibility, and control; and there will be increasingly large incentives to make use of these new possibilities. The potential for AI systems to become more general, in particular, will make it difficult to establish safety guarantees: reliable regularities during testing may not always hold post-testing.

The largest and most lasting changes in human welfare have come from scientific and technological innovation — which in turn comes from our intelligence. In the long run, then, much of AI's significance comes from its potential to automate and enhance progress in science and technology. The creation of smarter-than-human AI brings with it the basic risks and benefits of intellectual progress itself, at digital speeds.

As AI agents become more capable, it becomes more important (and more difficult) to analyze and verify their decisions and goals. Stuart Russell [http://edge.org/conversation/the-myth-of-ai#26015 writes]:

The primary concern is not spooky emergent consciousness but simply the ability to make high-quality decisions. Here, quality refers to the expected outcome utility of actions taken, where the utility function is, presumably, specified by the human designer. Now we have a problem:
# The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down.
# Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task.
A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. This is essentially the old story of the genie in the lamp, or the sorcerer's apprentice, or King Midas: you get exactly what you ask for, not what you want.

Bostrom's "[http://www.nickbostrom.com/superintelligentwill.pdf The Superintelligent Will]" lays out these two concerns in more detail: that we may not correctly specify our actual goals in programming smarter-than-human AI systems, and that most agents optimizing for a misspecified goal will have incentives to treat humans adversarially, as potential threats or obstacles to achieving the agent's goal.

If the goals of human and AI agents are not well-aligned, the more knowledgeable and technologically capable agent may use force to get what it wants, as has occurred in many conflicts between human communities. Having noticed this class of concerns in advance, we have an opportunity to reduce risk from this default scenario by directing research toward aligning artificial decision-makers' interests with our own.

Question: Why is safety important for smarter-than-human AI?

Answer:
Present-day AI algorithms already demand special safety guarantees when they must act in important domains without human oversight, particularly when they or their environment can change over time:

Achieving these gains ［from autonomous systems］ will depend on development of entirely new methods for enabling “trust in autonomy” through verification and validation (V&V) of the near-infinite state systems that result from high levels of ［adaptability］ and autonomy. In effect, the number of possible input states that such systems can be presented with is so large that not only is it impossible to test all of them directly, it is not even feasible to test more than an insignificantly small fraction of them. Development of such systems is thus inherently unverifiable by today’s methods, and as a result their operation in all but comparatively trivial applications is uncertifiable.

It is possible to develop systems having high levels of autonomy, but it is the lack of suitable V&V methods that prevents all but relatively low levels of autonomy from being certified for use.
- Office of the US Air Force Chief Scientist (2010). ［http://www.defenseinnovationmarketplace.mil/resources/AF_TechnologyHorizons2010-2030.pdf Technology Horizons: A Vision for Air Force Science and Technology 2010-30］.

As AI capabilities improve, it will become easier to give AI systems greater autonomy, flexibility, and control; and there will be increasingly large incentives to make use of these new possibilities. The potential for AI systems to become more general, in particular, will make it difficult to establish safety guarantees: reliable regularities during testing may not always hold post-testing.

The largest and most lasting changes in human welfare have come from scientific and technological innovation — which in turn comes from our intelligence. In the long run, then, much of AI’s significance comes from its potential to automate and enhance progress in science and technology. The creation of smarter-than-human AI brings with it the basic risks and benefits of intellectual progress itself, at digital speeds.

As AI agents become more capable, it becomes more important (and more difficult) to analyze and verify their decisions and goals. Stuart Russell ［http://edge.org/conversation/the-myth-of-ai#26015 writes］:

The primary concern is not spooky emergent consciousness but simply the ability to make high-quality decisions. Here, quality refers to the expected outcome utility of actions taken, where the utility function is, presumably, specified by the human designer. Now we have a problem:
# The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down.
# Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task.
A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. This is essentially the old story of the genie in the lamp, or the sorcerer’s apprentice, or King Midas: you get exactly what you ask for, not what you want.

Bostrom’s “［http://www.nickbostrom.com/superintelligentwill.pdf The Superintelligent Will］” lays out these two concerns in more detail: that we may not correctly specify our actual goals in programming smarter-than-human AI systems, and that most agents optimizing for a misspecified goal will have incentives to treat humans adversarially, as potential threats or obstacles to achieving the agent’s goal.

If the goals of human and AI agents are not well-aligned, the more knowledgeable and technologically capable agent may use force to get what it wants, as has occurred in many conflicts between human communities. Having noticed this class of concerns in advance, we have an opportunity to reduce risk from this default scenario by directing research toward aligning artificial decision-makers’ interests with our own.

Answer to Why is safety important for smarter-than-human AI?


Question: Why can't we just use natural language instructions?

Answer: When one person tells a set of natural language instructions to another person, they are relying on much other information which is already stored in the other person's mind.

If you tell me "don't harm other people," I already have a conception of what harm means and doesn't mean, what people means and doesn't mean, and my own complex moral reasoning for figuring out the edge cases in instances wherein harming people is inevitable or harming someone is necessary for self-defense or the greater good.

All of those complex definitions and systems of decision making are already in our mind, so it's easy to take them for granted. An AI is a mind made from scratch, so programming a goal is not as simple as telling it a natural language command.

Question: Why can’t we just use natural language instructions?

Answer:
When one person tells a set of natural language instructions to another person, they are relying on much other information which is already stored in the other person's mind.

If you tell me "don't harm other people," I already have a conception of what harm means and doesn't mean, what people means and doesn't mean, and my own complex moral reasoning for figuring out the edge cases in instances wherein harming people is inevitable or harming someone is necessary for self-defense or the greater good.

All of those complex definitions and systems of decision making are already in our mind, so it's easy to take them for granted. An AI is a mind made from scratch, so programming a goal is not as simple as telling it a natural language command.

Answer to Why can’t we just use natural language instructions?


Question: How likely is it that an AI would pretend  be a human  further its goals?

Answer: Talking about full AGI: Fairly likely, but depends on takeoff speed. In a slow takeoff of a misaligned AGI, where it is only weakly superintelligent, manipulating humans would be one of its main options for trying to further its goals for some time. Even in a fast takeoff, it's plausible that it would at least briefly manipulate humans in order to accelerate its ascent to technological superiority, though depending on what machines are available to hack at the time it may be able to skip this stage.

If the AI's goals include reference to humans it may have reason to continue deceiving us after it attains technological superiority, but will not necessarily do so. How this unfolds would depend on the details of its goals.

Eliezer Yudkowsky gives [https://www.lesswrong.com/posts/pxGYZs2zHJNHvWY5b/request-for-concrete-ai-takeover-mechanisms the example] of an AI solving protein folding, then mail-ordering synthesised DNA to a bribed or deceived human with instructions to mix the ingredients in a specific order to create [https://en.wikipedia.org/wiki/Atomically_precise_manufacturing wet nanotechnology].

Question: How likely is it that an AI would pretend  be a human  further its goals?

Answer:
Talking about full AGI: Fairly likely, but depends on takeoff speed. In a slow takeoff of a misaligned AGI, where it is only weakly superintelligent, manipulating humans would be one of its main options for trying to further its goals for some time. Even in a fast takeoff, it’s plausible that it would at least briefly manipulate humans in order to accelerate its ascent to technological superiority, though depending on what machines are available to hack at the time it may be able to skip this stage.

If the AI's goals include reference to humans it may have reason to continue deceiving us after it attains technological superiority, but will not necessarily do so. How this unfolds would depend on the details of its goals.

Eliezer Yudkowsky gives ［https://www.lesswrong.com/posts/pxGYZs2zHJNHvWY5b/request-for-concrete-ai-takeover-mechanisms the example］ of an AI solving protein folding, then mail-ordering synthesised DNA to a bribed or deceived human with instructions to mix the ingredients in a specific order to create ［https://en.wikipedia.org/wiki/Atomically_precise_manufacturing wet nanotechnology］.

Plex's Answer to How likely is it that an AI would pretend to be a human to further its goals?


Question: What are the potential benefits of AI as it grows increasingly sophisticated?

Answer: It's difficult to tell at this stage, but AI will enable many developments that could be terrifically beneficial if managed with enough foresight and care. For example, menial tasks could be automated, which could give rise to a society of abundance, leisure, and flourishing, free of poverty and tedium. As another example, AI could also improve our ability to understand and manipulate complex biological systems, unlocking a path to drastically improved longevity and health, and to conquering disease.

Question: What are the potential benefits of AI as it grows increasingly sophisticated?

Answer: It’s difficult to tell at this stage, but AI will enable many developments that could be terrifically beneficial if managed with enough foresight and care. For example, menial tasks could be automated, which could give rise to a society of abundance, leisure, and flourishing, free of poverty and tedium. As another example, AI could also improve our ability to understand and manipulate complex biological systems, unlocking a path to drastically improved longevity and health, and to conquering disease.

Answer to What are the potential benefits of AI as it grows increasingly sophisticated?


Question: What technical problems are MIRI working on?

Answer: "Aligning smarter-than-human AI with human interests" is an extremely vague goal. To approach this problem productively, we attempt to factorize it into several subproblems. As a starting point, we ask: "What aspects of this problem would we still be unable to solve even if the problem were much easier?"

In order to achieve real-world goals more effectively than a human, a general AI system will need to be able to learn its environment over time and decide between possible proposals or actions. A simplified version of the alignment problem, then, would be to ask how we could construct a system that learns its environment and has a very crude decision criterion, like "Select the policy that maximizes the expected number of diamonds in the world."

''Highly reliable agent design'' is the technical challenge of formally specifying a software system that can be relied upon to pursue some preselected toy goal. An example of a subproblem in this space is [https://intelligence.org/2015/07/27/miris-approach/#2 ontology identification]: how do we formalize the goal of "maximizing diamonds" in full generality, allowing that a fully autonomous agent may end up in unexpected environments and may construct unanticipated hypotheses and policies? Even if we had unbounded computational power and all the time in the world, we don't currently know how to solve this problem. This suggests that we're not only missing practical algorithms but also a basic theoretical framework through which to understand the problem.

The formal agent AIXI is an attempt to define what we mean by "optimal behavior" in the case of a reinforcement learner. A simple AIXI-like equation is lacking, however, for defining what we mean by "good behavior" if the goal is to change something about the external world (and not just to maximize a pre-specified reward number). In order for the agent to evaluate its world-models to count the number of diamonds, as opposed to having a privileged reward channel, what general formal properties must its world-models possess? If the system updates its hypotheses (e.g., discovers that string theory is true and quantum physics is false) in a way its programmers didn't expect, how does it identify "diamonds" in the new model? The question is a very basic one, yet the relevant theory is currently missing.

We can distinguish highly reliable agent design from the problem of value specification: "Once we understand how to design an autonomous AI system that promotes a goal, how do we ensure its goal actually matches what we want?" Since human error is inevitable and we will need to be able to safely supervise and redesign AI algorithms even as they approach human equivalence in cognitive tasks, MIRI also works on formalizing error-tolerant agent properties. Artificial Intelligence: A Modern Approach, the standard textbook in AI, summarizes the challenge:

Yudkowsky […] asserts that friendliness (a desire not to harm humans) should be designed in from the start, but that the designers should recognize both that their own designs may be flawed, and that the robot will learn and evolve over time. Thus the challenge is one of mechanism design — to design a mechanism for evolving AI under a system of checks and balances, and to give the systems utility functions that will remain friendly in the face of such changes.
-Russell and Norvig (2009). [http://www.amazon.com/Artificial-Intelligence-Modern-Approach-Edition/dp/<PHONE> Artificial Intelligence: A Modern Approach].

Our [https://intelligence.org/technical-agenda/ technical agenda] describes these open problems in more detail, and our research guide collects online resources for learning more.

Question: What technical problems are MIRI working on?

Answer:
“Aligning smarter-than-human AI with human interests” is an extremely vague goal. To approach this problem productively, we attempt to factorize it into several subproblems. As a starting point, we ask: “What aspects of this problem would we still be unable to solve even if the problem were much easier?”

In order to achieve real-world goals more effectively than a human, a general AI system will need to be able to learn its environment over time and decide between possible proposals or actions. A simplified version of the alignment problem, then, would be to ask how we could construct a system that learns its environment and has a very crude decision criterion, like “Select the policy that maximizes the expected number of diamonds in the world.”

''Highly reliable agent design'' is the technical challenge of formally specifying a software system that can be relied upon to pursue some preselected toy goal. An example of a subproblem in this space is ［https://intelligence.org/2015/07/27/miris-approach/#2 ontology identification］: how do we formalize the goal of “maximizing diamonds” in full generality, allowing that a fully autonomous agent may end up in unexpected environments and may construct unanticipated hypotheses and policies? Even if we had unbounded computational power and all the time in the world, we don’t currently know how to solve this problem. This suggests that we’re not only missing practical algorithms but also a basic theoretical framework through which to understand the problem.

The formal agent AIXI is an attempt to define what we mean by “optimal behavior” in the case of a reinforcement learner. A simple AIXI-like equation is lacking, however, for defining what we mean by “good behavior” if the goal is to change something about the external world (and not just to maximize a pre-specified reward number). In order for the agent to evaluate its world-models to count the number of diamonds, as opposed to having a privileged reward channel, what general formal properties must its world-models possess? If the system updates its hypotheses (e.g., discovers that string theory is true and quantum physics is false) in a way its programmers didn’t expect, how does it identify “diamonds” in the new model? The question is a very basic one, yet the relevant theory is currently missing.

We can distinguish highly reliable agent design from the problem of value specification: “Once we understand how to design an autonomous AI system that promotes a goal, how do we ensure its goal actually matches what we want?” Since human error is inevitable and we will need to be able to safely supervise and redesign AI algorithms even as they approach human equivalence in cognitive tasks, MIRI also works on formalizing error-tolerant agent properties. Artificial Intelligence: A Modern Approach, the standard textbook in AI, summarizes the challenge:

Yudkowsky ［…］ asserts that friendliness (a desire not to harm humans) should be designed in from the start, but that the designers should recognize both that their own designs may be flawed, and that the robot will learn and evolve over time. Thus the challenge is one of mechanism design — to design a mechanism for evolving AI under a system of checks and balances, and to give the systems utility functions that will remain friendly in the face of such changes.
-Russell and Norvig (2009). ［http://www.amazon.com/Artificial-Intelligence-Modern-Approach-Edition/dp/0136042597 Artificial Intelligence: A Modern Approach］.

Our ［https://intelligence.org/technical-agenda/ technical agenda］ describes these open problems in more detail, and our research guide collects online resources for learning more.

Answer to What technical problems are MIRI working on?


Answer to Won’t AI be just like us?

Won’t AI be just like us?

Question: Won't AI be just like us?

Answer: The degree to which an Artificial Superintelligence (ASI) would resemble us depends heavily on how it is implemented, but it seems that differences are unavoidable. If AI is accomplished through whole brain emulation and we make a big effort to make it as human as possible (including giving it a humanoid body), the AI could probably be said to think like a human. However, by definition of ASI it would be much smarter. Differences in the substrate and body might open up numerous possibilities (such as immortality, different sensors, easy self-improvement, ability to make copies, etc.). Its social experience and upbringing would likely also be entirely different. All of this can significantly change the ASI's values and outlook on the world, even if it would still use the same algorithms as we do. This is essentially the "best case scenario" for human resemblance, but whole brain emulation is kind of a separate field from AI, even if both aim to build intelligent machines. Most approaches to AI are vastly different and most ASIs would likely not have humanoid bodies. At this moment in time it seems much easier to create a machine that is intelligent than a machine that is exactly like a human (it's certainly a bigger target).

Question: Won’t AI be just like us?

Answer: The degree to which an Artificial Superintelligence (ASI) would resemble us depends heavily on how it is implemented, but it seems that differences are unavoidable. If AI is accomplished through whole brain emulation and we make a big effort to make it as human as possible (including giving it a humanoid body), the AI could probably be said to think like a human. However, by definition of ASI it would be much smarter. Differences in the substrate and body might open up numerous possibilities (such as immortality, different sensors, easy self-improvement, ability to make copies, etc.). Its social experience and upbringing would likely also be entirely different. All of this can significantly change the ASI's values and outlook on the world, even if it would still use the same algorithms as we do. This is essentially the "best case scenario" for human resemblance, but whole brain emulation is kind of a separate field from AI, even if both aim to build intelligent machines. Most approaches to AI are vastly different and most ASIs would likely not have humanoid bodies. At this moment in time it seems much easier to create a machine that is intelligent than a machine that is exactly like a human (it's certainly a bigger target).


Question: Why might an AI do something that we don't want it to, if it's really so intelligent?

Answer: A Superintelligence would be intelligent enough to understand what the programmer's motives were when designing its goals, but it would have no intrinsic reason to care about what its programmers had in mind. The only thing it will be beholden to is the actual goal it is programmed with, no matter how insane its fulfillment may seem to us.

Consider what "intentions" the process of evolution may have had for you when designing your goals. When you consider that you were made with the "intention" of replicating your genes, do you somehow feel beholden to the "intention" behind your evolutionary design? Most likely you don't care. You may choose to never have children, and you will most likely attempt to keep yourself alive long past your biological ability to reproduce.

Question: Why might an AI do something that we don’t want it to, if it’s really so intelligent?

Answer:
A Superintelligence would be intelligent enough to understand what the programmer’s motives were when designing its goals, but it would have no intrinsic reason to care about what its programmers had in mind. The only thing it will be beholden to is the actual goal it is programmed with, no matter how insane its fulfillment may seem to us.

Consider what “intentions” the process of evolution may have had for you when designing your goals. When you consider that you were made with the “intention” of replicating your genes, do you somehow feel beholden to the “intention” behind your evolutionary design? Most likely you don't care. You may choose to never have children, and you will most likely attempt to keep yourself alive long past your biological ability to reproduce.

Answer to Why might an AI do something that we don’t want it to, if it’s really so intelligent?


Question: Can humans stay in control of the world if human- or superhuman-level AI is developed?

Answer: This is a big question that it would pay to start thinking about. Humans are in control of this planet not because we are stronger or faster than other animals, but because we are smarter! If we cede our position as smartest on our planet, it's not obvious that we'll retain control.

Question: Can humans stay in control of the world if human- or superhuman-level AI is developed?

Answer: This is a big question that it would pay to start thinking about. Humans are in control of this planet not because we are stronger or faster than other animals, but because we are smarter! If we cede our position as smartest on our planet, it’s not obvious that we’ll retain control.

Answer to Can humans stay in control of the world if human- or superhuman-level AI is developed?


A lot of concern appears  focus on human-level or “superintelligent” AI. Is that a realistic prospect in the foreseeable future?

Question: A lot of concern appears  focus on human-level or "superintelligent" AI. Is that a realistic prospect in the foreseeable future?

Answer: AI is already superhuman at some tasks, for example numerical computations, and will clearly surpass humans in others as time goes on. We don't know when (or even if) machines will reach human-level ability in all cognitive tasks, but most of the AI researchers at FLI's conference in Puerto Rico put the odds above 50% for this century, and many offered a significantly shorter timeline. Since the impact on humanity will be huge if it happens, it's worthwhile to start research now on how to ensure that any impact is positive. Many researchers also believe that dealing with superintelligent AI will be qualitatively very different from more narrow AI systems, and will require very significant research effort to get right.

Question: A lot of concern appears  focus on human-level or “superintelligent” AI. Is that a realistic prospect in the foreseeable future?

Answer: AI is already superhuman at some tasks, for example numerical computations, and will clearly surpass humans in others as time goes on. We don’t know when (or even if) machines will reach human-level ability in all cognitive tasks, but most of the AI researchers at FLI’s conference in Puerto Rico put the odds above 50% for this century, and many offered a significantly shorter timeline. Since the impact on humanity will be huge if it happens, it’s worthwhile to start research now on how to ensure that any impact is positive. Many researchers also believe that dealing with superintelligent AI will be qualitatively very different from more narrow AI systems, and will require very significant research effort to get right.

Answer to A lot of concern appears to focus on human-level or “superintelligent” AI. Is that a realistic prospect in the foreseeable future?


How is AGI different from current AI?

Question: How is AGI different from current AI?

Answer: Current narrow systems are much more domain-specific than AGI. We don't know what the first AGI will look like, some people think the [https://www.lesswrong.com/posts/6Hee7w2paEzHsD6mn/collection-of-gpt-3-results GPT-3] architecture but scaled up a lot may get us there (GPT-3 is a giant prediction model which when trained on a vast amount of text seems to [https://www.lesswrong.com/posts/D3hP47pZwXNPRByj8/an-102-meta-learning-by-gpt-3-and-a-list-of-full-proposals learn how to learn] and do [https://gpt3examples.com/ all sorts of crazy-impressive things], a related model can [https://openai.com/blog/dall-e/ generate pictures from text]), some people don't think scaling this kind of model will get us all the way.

Question: How is AGI different from current AI?

Answer: Current narrow systems are much more domain-specific than AGI. We don’t know what the first AGI will look like, some people think the ［https://www.lesswrong.com/posts/6Hee7w2paEzHsD6mn/collection-of-gpt-3-results GPT-3］ architecture but scaled up a lot may get us there (GPT-3 is a giant prediction model which when trained on a vast amount of text seems to ［https://www.lesswrong.com/posts/D3hP47pZwXNPRByj8/an-102-meta-learning-by-gpt-3-and-a-list-of-full-proposals learn how to learn］ and do ［https://gpt3examples.com/ all sorts of crazy-impressive things］, a related model can ［https://openai.com/blog/dall-e/ generate pictures from text］), some people don’t think scaling this kind of model will get us all the way.

Plex's Answer to How is AGI different from current AI?