Plex's Answer to Why can’t we just…

Why can’t we just…

Question: Why can't we just…

Answer: There are many approaches that initially look like they can eliminate these problems, but then turn out to have hidden difficulties. It's surprisingly easy to come up with "solutions" which don't actually solve the problem. This can be because…

*…they require you to be smarter than the system. Many solutions only work when the system is relatively weak, but break when they achieve a certain level of capability (for multiple reasons, e.g. [https://www.youtube.com/watch?v꞊IeWljQw3UgQ deceptive alignment]).

*…they rely on appearing to make sense in natural language, but when properly unpacked they're not philosophically clear enough to be usable.

*… despite being philosophically coherent, we have no idea how to turn them into computer code (or if that's even possible).

*…they're things which we can't do.

*…although we can do them, they don't solve the problem.

*…they solve a relatively easy subcomponent of the problem but leave the hard problem untouched.

*…they solve the problem but only as long as we stay "in distribution" with respect to the original training data ([https://en.wikipedia.org/wiki/Domain_adaptation distributional shift] will break them).

*…although they might work eventually, we can't expect them to work on the first try (and we [https://stampy.ai/wiki/Why_would_we_only_get_one_chance_to_align_a_superintelligence%3F only get one try at aligning a superintelligence!)].

Here are some of the proposals which often come up:

Question: Why can’t we just…

Answer:
There are many approaches that initially look like they can eliminate these problems, but then turn out to have hidden difficulties. It’s surprisingly easy to come up with “solutions” which don’t actually solve the problem. This can be because…

*…they require you to be smarter than the system. Many solutions only work when the system is relatively weak, but break when they achieve a certain level of capability (for multiple reasons, e.g. ［https://www.youtube.com/watch?v꞊IeWljQw3UgQ deceptive alignment］).

*…they rely on appearing to make sense in natural language, but when properly unpacked they’re not philosophically clear enough to be usable.

*… despite being philosophically coherent, we have no idea how to turn them into computer code (or if that’s even possible).

*…they’re things which we can’t do.

*…although we can do them, they don’t solve the problem.

*…they solve a relatively easy subcomponent of the problem but leave the hard problem untouched.

*…they solve the problem but only as long as we stay “in distribution” with respect to the original training data (［https://en.wikipedia.org/wiki/Domain_adaptation distributional shift］ will break them).

*…although they might work eventually, we can’t expect them to work on the first try (and we ［https://stampy.ai/wiki/Why_would_we_only_get_one_chance_to_align_a_superintelligence%3F only get one try at aligning a superintelligence!)］.

Here are some of the proposals which often come up:


Couldn’t we keep the AI in a box and never give it the ability  manipulate the external world?

Question: Couldn't we keep the AI in a box and never give it the ability  manipulate the external world?

Answer: That is, if you know an AI is likely to be superintelligent, can't you just disconnect it from the Internet, not give it access to any speakers that can make [https://stampy.ai/wiki/What_do_you_mean_by_superintelligences_manipulating_humans_socially%3F mysterious buzzes and hums], make sure the only people who interact with it are trained in caution, et cetera?. Isn't there some level of security – maybe the level we use for that room in the CDC where people in containment suits hundreds of feet underground analyze the latest superviruses – with which a superintelligence could be safe?

This puts us back in the same situation as lions trying to figure out whether or not nuclear weapons are a things humans can do. But suppose there is such a level of security. You build a superintelligence, and you put it in an airtight chamber deep in a cave with no Internet connection and only carefully-trained security experts to talk to. What now?

Now you have a superintelligence which is possibly safe but definitely useless. The whole point of building superintelligences is that they're smart enough to do useful things like cure cancer. But if you have the monks ask the superintelligence for a cancer cure, and it gives them one, that's a clear security vulnerability. You have a superintelligence locked up in a cave with no way to influence the outside world except that you're going to mass produce a chemical it gives you and inject it into millions of people.

Or maybe none of this happens, and the superintelligence sits inert in its cave. And then another team somewhere else invents a second superintelligence. And then a third team invents a third superintelligence. Remember, it was only about ten years between Deep Blue beating Kasparov, and everybody having Deep Blue – level chess engines on their laptops. And the first twenty teams are responsible and keep their superintelligences locked in caves with carefully-trained experts, and the twenty-first team is a little less responsible, and now we still have to deal with a rogue superintelligence.

Superintelligences are extremely dangerous, and no normal means of controlling them can entirely remove the danger.

Question: Couldn’t we keep the AI in a box and never give it the ability  manipulate the external world?

Answer:
That is, if you know an AI is likely to be superintelligent, can’t you just disconnect it from the Internet, not give it access to any speakers that can make ［https://stampy.ai/wiki/What_do_you_mean_by_superintelligences_manipulating_humans_socially%3F mysterious buzzes and hums］, make sure the only people who interact with it are trained in caution, et cetera?. Isn’t there some level of security – maybe the level we use for that room in the CDC where people in containment suits hundreds of feet underground analyze the latest superviruses – with which a superintelligence could be safe?

This puts us back in the same situation as lions trying to figure out whether or not nuclear weapons are a things humans can do. But suppose there is such a level of security. You build a superintelligence, and you put it in an airtight chamber deep in a cave with no Internet connection and only carefully-trained security experts to talk to. What now?

Now you have a superintelligence which is possibly safe but definitely useless. The whole point of building superintelligences is that they’re smart enough to do useful things like cure cancer. But if you have the monks ask the superintelligence for a cancer cure, and it gives them one, that’s a clear security vulnerability. You have a superintelligence locked up in a cave with no way to influence the outside world except that you’re going to mass produce a chemical it gives you and inject it into millions of people.

Or maybe none of this happens, and the superintelligence sits inert in its cave. And then another team somewhere else invents a second superintelligence. And then a third team invents a third superintelligence. Remember, it was only about ten years between Deep Blue beating Kasparov, and everybody having Deep Blue – level chess engines on their laptops. And the first twenty teams are responsible and keep their superintelligences locked in caves with carefully-trained experts, and the twenty-first team is a little less responsible, and now we still have to deal with a rogue superintelligence.

Superintelligences are extremely dangerous, and no normal means of controlling them can entirely remove the danger.

Answer to Couldn’t we keep the AI in a box and never give it the ability to manipulate the external world?


Question: Where can I learn about interpretability?

Answer: [https://christophm.github.io/interpretable-ml-book/ Christoph Molnar's online book] and [https://distill.pub/ distill.pub] are great sources, as well as [https://www.alignmentforum.org/posts/GEPX7jgLMB8vR2qaK/opinions-on-interpretable-machine-learning-and-70-summaries this overview article] which summarizes 70 interpretability papers.

Question: Where can I learn about interpretability?

Answer: ［https://christophm.github.io/interpretable-ml-book/ Christoph Molnar's online book］ and ［https://distill.pub/ distill.pub］ are great sources, as well as ［https://www.alignmentforum.org/posts/GEPX7jgLMB8vR2qaK/opinions-on-interpretable-machine-learning-and-70-summaries this overview article］ which summarizes 70 interpretability papers.

Plex's Answer to Where can I learn about interpretability?


Question: I want  work on AI alignment. How can I get funding?

Answer: See the [https://www.futurefundinglist.com/ Future Funding List] for up to date information!

The organizations which most regularly give grants to individuals working towards AI alignment are the [https://funds.effectivealtruism.org/funds/far-future Long Term Future Fund], [http://survivalandflourishing.org/ Survival And Flourishing (SAF)], the [https://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/the-open-phil-ai-fellowship OpenPhil AI Fellowship] and [https://www.openphilanthropy.org/focus/other-areas/early-career-funding-individuals-interested-improving-long-term-future early career funding], the [https://grants.futureoflife.org/ Future of Life Institute], the [https://www.fhi.ox.ac.uk/aia-fellowship/ Future of Humanity Institute], and [https://longtermrisk.org/grantmaking/ the Center on Long-Term Risk Fund]. If you're able to relocate to the UK, [https://ceealar.org/ CEEALAR (aka the EA Hotel)] can be a great option as it offers free food and accommodation for up to two years, as well as contact with others who are thinking about these issues. The [https://ftxfuturefund.org/apply/ FTX Future Fund] only accepts direct applications for $100k+ with an emphasis on massively scaleable interventions, but their [https://ftxfuturefund.org/announcing-our-regranting-program/ regranters] can make smaller grants for individuals. There are also opportunities from smaller grantmakers which you might be able to pick up if you get involved.

If you want to work on support or infrastructure rather than directly on research, the [https://funds.effectivealtruism.org/funds/ea-community EA Infrastructure Fund] may be able to help. In general, you can [https://www.lesswrong.com/posts/5AAFoigbbMqgrTpDh/you-can-talk-to-ea-funds-before-applying talk to EA funds before applying].

Each grant source has their own criteria for funding, but in general they are looking for candidates who have evidence that they're keen and able to do good work towards reducing existential risk (for example, by completing an [https://aisafety.camp/ AI Safety Camp] project), though the EA Hotel in particular has less stringent requirements as they're able to support people at very low cost. If you'd like to talk to someone who can offer advice on applying for funding, [https://www.aisafetysupport.org/ AI Safety Support] offers [https://calendly.com/aiss free calls].

Another option is to get hired by an organization which works on AI alignment, see the follow-up question for advice on that.

It's also worth checking the AI Alignment tag on the [https://eafunding.softr.app/ EA funding sources website] for up-to-date suggestions.

Question: I want  work on AI alignment. How can I get funding?

Answer:
See the ［https://www.futurefundinglist.com/ Future Funding List］ for up to date information!

The organizations which most regularly give grants to individuals working towards AI alignment are the ［https://funds.effectivealtruism.org/funds/far-future Long Term Future Fund］, ［http://survivalandflourishing.org/ Survival And Flourishing (SAF)］, the ［https://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/the-open-phil-ai-fellowship OpenPhil AI Fellowship］ and ［https://www.openphilanthropy.org/focus/other-areas/early-career-funding-individuals-interested-improving-long-term-future early career funding］, the ［https://grants.futureoflife.org/ Future of Life Institute］, the ［https://www.fhi.ox.ac.uk/aia-fellowship/ Future of Humanity Institute］, and ［https://longtermrisk.org/grantmaking/ the Center on Long-Term Risk Fund］. If you're able to relocate to the UK, ［https://ceealar.org/ CEEALAR (aka the EA Hotel)］ can be a great option as it offers free food and accommodation for up to two years, as well as contact with others who are thinking about these issues. The ［https://ftxfuturefund.org/apply/ FTX Future Fund］ only accepts direct applications for $100k+ with an emphasis on massively scaleable interventions, but their ［https://ftxfuturefund.org/announcing-our-regranting-program/ regranters］ can make smaller grants for individuals. There are also opportunities from smaller grantmakers which you might be able to pick up if you get involved.

If you want to work on support or infrastructure rather than directly on research, the ［https://funds.effectivealtruism.org/funds/ea-community EA Infrastructure Fund］ may be able to help. In general, you can ［https://www.lesswrong.com/posts/5AAFoigbbMqgrTpDh/you-can-talk-to-ea-funds-before-applying talk to EA funds before applying］.

Each grant source has their own criteria for funding, but in general they are looking for candidates who have evidence that they're keen and able to do good work towards reducing existential risk (for example, by completing an ［https://aisafety.camp/ AI Safety Camp］ project), though the EA Hotel in particular has less stringent requirements as they're able to support people at very low cost. If you'd like to talk to someone who can offer advice on applying for funding, ［https://www.aisafetysupport.org/ AI Safety Support］ offers ［https://calendly.com/aiss free calls］.

Another option is to get hired by an organization which works on AI alignment, see the follow-up question for advice on that.

It's also worth checking the AI Alignment tag on the ［https://eafunding.softr.app/ EA funding sources website］ for up-to-date suggestions.

Plex's Answer to I want to work on AI alignment. How can I get funding?


Question: Why don't we just not build AGI if it's so dangerous?

Answer: It certainly would be very unwise to purposefully create an artificial general intelligence now, before we have found a way to be certain it will act purely in our interests.  But "general intelligence" is more of a description of a system's capabilities, and a vague one at that. We don't know what it takes to build such a system.  This leads to the worrying possibility that our existing, narrow AI systems require only minor tweaks, or even just more computer power, to achieve general intelligence.

The pace of research in the field suggests that there's a lot of low-hanging fruit left to pick, after all, and the results of this research produce better, more effective AI in a landscape of strong competitive pressure to produce as highly competitive systems as we can.  "Just" not building an AGI means ensuring that every organization in the world with lots of computer hardware doesn't build an AGI, either accidentally or mistakenly thinking they have a solution to the alignment problem, forever.  It's simply far safer to also work on solving the alignment problem.

Question: Why don't we just not build AGI if it's so dangerous?

Answer:
It certainly would be very unwise to purposefully create an artificial general intelligence now, before we have found a way to be certain it will act purely in our interests.  But "general intelligence" is more of a description of a system's capabilities, and a vague one at that. We don't know what it takes to build such a system.  This leads to the worrying possibility that our existing, narrow AI systems require only minor tweaks, or even just more computer power, to achieve general intelligence.

The pace of research in the field suggests that there's a lot of low-hanging fruit left to pick, after all, and the results of this research produce better, more effective AI in a landscape of strong competitive pressure to produce as highly competitive systems as we can.  "Just" not building an AGI means ensuring that every organization in the world with lots of computer hardware doesn't build an AGI, either accidentally or mistakenly thinking they have a solution to the alignment problem, forever.  It's simply far safer to also work on solving the alignment problem.

SlimeBunnyBat's Answer to Why don't we just not build AGI if it's so dangerous?


Question: What should I read  learn about decision theory?

Answer: [https://www.lesswrong.com/posts/zcPLNNw4wgBX5k8kQ/decision-theory abramdemski and Scott Garrabrant's post on decision theory] provides a good overview of many aspects of the topic, while [https://arxiv.org/abs/1710.05060 Functional Decision Theory: A New Theory of Instrumental Rationality] seems to be the most up to date source on current thinking.

For a more intuitive dive into one of the core problems, [https://www.lesswrong.com/posts/6ddcsdA2c2XpNpE5x/newcomb-s-problem-and-regret-of-rationality Newcomb's problem and regret of rationality] is good, and [https://www.lesswrong.com/posts/puutBJLWbg2sXpFbu/newcomblike-problems-are-the-norm Newcomblike problems are the norm] is useful for seeing how it applies in the real world.

The [https://www.lesswrong.com/tag/decision-theory LessWrong tag for decision theory] has lots of additional links for people who want to explore further.

Question: What should I read  learn about decision theory?

Answer:
［https://www.lesswrong.com/posts/zcPLNNw4wgBX5k8kQ/decision-theory abramdemski and Scott Garrabrant's post on decision theory］ provides a good overview of many aspects of the topic, while ［https://arxiv.org/abs/1710.05060 Functional Decision Theory: A New Theory of Instrumental Rationality］ seems to be the most up to date source on current thinking.

For a more intuitive dive into one of the core problems, ［https://www.lesswrong.com/posts/6ddcsdA2c2XpNpE5x/newcomb-s-problem-and-regret-of-rationality Newcomb's problem and regret of rationality］ is good, and ［https://www.lesswrong.com/posts/puutBJLWbg2sXpFbu/newcomblike-problems-are-the-norm Newcomblike problems are the norm］ is useful for seeing how it applies in the real world.

The ［https://www.lesswrong.com/tag/decision-theory LessWrong tag for decision theory］ has lots of additional links for people who want to explore further.

Plex's Answer to What should I read to learn about decision theory?


Question: When will an intelligence explosion happen?

Answer: Predicting the future is risky business. There are many philosophical, scientific, technological, and social uncertainties relevant to the arrival of an intelligence explosion. Because of this, experts disagree on when this event might occur. Here are some of their predictions:

* Futurist Ray Kurzweil [http://www.amazon.com/dp/<PHONE>/ predicts] that machines will reach human-level intelligence by 2030 and that we will reach "a profound and disruptive transformation in human capability" by 2045.
* Intel's chief technology officer, Justin Rattner, [http://www.techwatch.co.uk/2008/08/22/intel-predicts-singularity-by-2048/ expects] "a point when human and artificial intelligence merges to create something bigger than itself" by 2048.
* AI researcher Eliezer Yudkowsky [http://commonsenseatheism.com/?p꞊12147 expects] the intelligence explosion by 2060.
* Philosopher David Chalmers has [http://consc.net/papers/singularity.pdf over 1/2 credence] in the intelligence explosion occurring by 2100.
* Quantum computing expert Michael Nielsen [http://michaelnielsen.org/blog/what-should-a-reasonable-person-believe-about-the-singularity/ estimates] that the probability of the intelligence explosion occurring by 2100 is between 0.2% and about 70%.
* In 2009, at the AGI-09 conference, experts were asked when AI might reach superintelligence with massive new funding. The [http://sethbaum.com/ac/2011_AI-Experts.pdf median estimates] were that machine superintelligence could be achieved by 2045 (with 50% confidence) or by 2100 (with 90% confidence). Of course, attendees to this conference were self-selected to think that near-term artificial general intelligence is plausible.
* iRobot CEO [http://itc.conversationsnetwork.org/shows/detail3400.html Rodney Brooks] and cognitive scientist [http://video.google.com/videoplay?docid꞊8832143373632003914 Douglas Hofstadter] allow that the intelligence explosion may occur in the future, but probably not in the 21st century.
* Roboticist Hans Moravec predicts that AI will surpass human intelligence "[http://www.scientificamerican.com/article.cfm?id꞊rise-of-the-robots&print꞊true well before 2050]."
* In a 2005 survey of 26 contributors to a series of reports on emerging technologies, the [http://www.wtec.org/ConvergingTechnologies/3/NBIC3_report.pdf median estimate] for machines reaching human-level intelligence was 2085.
* Participants in a 2011 intelligence conference at Oxford gave a [http://www.fhi.ox.ac.uk/__data/assets/pdf_file/0015/21516/MI_survey.pdf median estimate] of 2050 for when there will be a 50% of human-level machine intelligence, and a median estimate of 2150 for when there will be a 90% chance of human-level machine intelligence.
* On the other hand, 41% of the participants in the AI@50 conference (in 2006) [http://www.engagingexperience.com/ai50/ stated] that machine intelligence would never reach the human level.

See also:

* Baum, Goertzel, & Goertzel, [http://sethbaum.com/ac/2011_AI-Experts.pdfHow Long Until Human-Level AI? Results from an Expert Assessment]

Question: When will an intelligence explosion happen?

Answer:
Predicting the future is risky business. There are many philosophical, scientific, technological, and social uncertainties relevant to the arrival of an intelligence explosion. Because of this, experts disagree on when this event might occur. Here are some of their predictions:

* Futurist Ray Kurzweil ［http://www.amazon.com/dp/0143037889/ predicts］ that machines will reach human-level intelligence by 2030 and that we will reach “a profound and disruptive transformation in human capability” by 2045.
* Intel’s chief technology officer, Justin Rattner, ［http://www.techwatch.co.uk/2008/08/22/intel-predicts-singularity-by-2048/ expects］ “a point when human and artificial intelligence merges to create something bigger than itself” by 2048.
* AI researcher Eliezer Yudkowsky ［http://commonsenseatheism.com/?p꞊12147 expects］ the intelligence explosion by 2060.
* Philosopher David Chalmers has ［http://consc.net/papers/singularity.pdf over 1/2 credence］ in the intelligence explosion occurring by 2100.
* Quantum computing expert Michael Nielsen ［http://michaelnielsen.org/blog/what-should-a-reasonable-person-believe-about-the-singularity/ estimates］ that the probability of the intelligence explosion occurring by 2100 is between 0.2% and about 70%.
* In 2009, at the AGI-09 conference, experts were asked when AI might reach superintelligence with massive new funding. The ［http://sethbaum.com/ac/2011_AI-Experts.pdf median estimates］ were that machine superintelligence could be achieved by 2045 (with 50% confidence) or by 2100 (with 90% confidence). Of course, attendees to this conference were self-selected to think that near-term artificial general intelligence is plausible.
* iRobot CEO ［http://itc.conversationsnetwork.org/shows/detail3400.html Rodney Brooks］ and cognitive scientist ［http://video.google.com/videoplay?docid꞊8832143373632003914 Douglas Hofstadter］ allow that the intelligence explosion may occur in the future, but probably not in the 21st century.
* Roboticist Hans Moravec predicts that AI will surpass human intelligence “［http://www.scientificamerican.com/article.cfm?id꞊rise-of-the-robots&print꞊true well before 2050］.”
* In a 2005 survey of 26 contributors to a series of reports on emerging technologies, the ［http://www.wtec.org/ConvergingTechnologies/3/NBIC3_report.pdf median estimate］ for machines reaching human-level intelligence was 2085.
* Participants in a 2011 intelligence conference at Oxford gave a ［http://www.fhi.ox.ac.uk/__data/assets/pdf_file/0015/21516/MI_survey.pdf median estimate］ of 2050 for when there will be a 50% of human-level machine intelligence, and a median estimate of 2150 for when there will be a 90% chance of human-level machine intelligence.
* On the other hand, 41% of the participants in the AI@50 conference (in 2006) ［http://www.engagingexperience.com/ai50/ stated］ that machine intelligence would never reach the human level.

See also:

* Baum, Goertzel, & Goertzel, ［http://sethbaum.com/ac/2011_AI-Experts.pdfHow Long Until Human-Level AI? Results from an Expert Assessment］

Answer to When will an intelligence explosion happen?


Question: How could an intelligence explosion be useful?

Answer: A machine superintelligence, if programmed with the right motivations, could potentially solve all the problems that humans are trying to solve but haven't had the ingenuity or processing speed to solve yet. A superintelligence might cure disabilities and diseases, achieve world peace, give humans vastly longer and healthier lives, eliminate food and energy shortages, boost scientific discovery and space exploration, and so on.

Furthermore, humanity faces several existential risks in the 21st century, including global nuclear war, bioweapons, superviruses, and [http://www.amazon.com/dp/<PHONE>/ more]. A superintelligent machine would be more capable of solving those problems than humans are.

See also:
* Yudkowsky, [https://intelligence.org/files/AIPosNegFactor.pdf Artificial intelligence as a positive and negative factor in global risk]

Question: How could an intelligence explosion be useful?

Answer:
A machine superintelligence, if programmed with the right motivations, could potentially solve all the problems that humans are trying to solve but haven’t had the ingenuity or processing speed to solve yet. A superintelligence might cure disabilities and diseases, achieve world peace, give humans vastly longer and healthier lives, eliminate food and energy shortages, boost scientific discovery and space exploration, and so on.

Furthermore, humanity faces several existential risks in the 21st century, including global nuclear war, bioweapons, superviruses, and ［http://www.amazon.com/dp/0198570503/ more］. A superintelligent machine would be more capable of solving those problems than humans are.

See also:
* Yudkowsky, ［https://intelligence.org/files/AIPosNegFactor.pdf Artificial intelligence as a positive and negative factor in global risk］

Answer to How could an intelligence explosion be useful?


Question: Why might people try  build AGI rather than stronger and stronger narrow AIs?

Answer: Making a narrow AI for every task would be extremely costly and time-consuming. By making a more general intelligence, you can apply one system to a broader range of tasks, which is economically and strategically attractive.

Of course, for generality to be a good option there are some necessary conditions. You need an architecture which is straightforward enough to scale up, such as the transformer which is used for GPT and follows scaling laws. It's also important that by generalizing you do not lose too much capacity at narrow tasks or require too much extra compute for it to be worthwhile.

Whether or not those conditions actually hold: It seems like many important actors (such as DeepMind and OpenAI) believe that they do, and are therefore focusing on trying to build an AGI in order to influence the future, so we should take actions to make it more likely that AGI will be developed safety.

Additionally, it is possible that even if we tried to build only narrow AIs, given enough time and compute we might accidentally create a more general AI than we intend by training a system on a task which requires a broad world model.

See also:
* [https://slatestarcodex.com/2019/08/27/book-review-reframing-superintelligence/ Reframing Superintelligence] - A model of AI development which proposes that we might mostly build narrow AI systems for some time.

Question: Why might people try  build AGI rather than stronger and stronger narrow AIs?

Answer:
Making a narrow AI for every task would be extremely costly and time-consuming. By making a more general intelligence, you can apply one system to a broader range of tasks, which is economically and strategically attractive.

Of course, for generality to be a good option there are some necessary conditions. You need an architecture which is straightforward enough to scale up, such as the transformer which is used for GPT and follows scaling laws. It's also important that by generalizing you do not lose too much capacity at narrow tasks or require too much extra compute for it to be worthwhile.

Whether or not those conditions actually hold: It seems like many important actors (such as DeepMind and OpenAI) believe that they do, and are therefore focusing on trying to build an AGI in order to influence the future, so we should take actions to make it more likely that AGI will be developed safety.

Additionally, it is possible that even if we tried to build only narrow AIs, given enough time and compute we might accidentally create a more general AI than we intend by training a system on a task which requires a broad world model.

See also:
* ［https://slatestarcodex.com/2019/08/27/book-review-reframing-superintelligence/ Reframing Superintelligence］ - A model of AI development which proposes that we might mostly build narrow AI systems for some time.

^,^'s Answer to Why might people try to build AGI rather than stronger and stronger narrow AIs?


Question: What should be marked as a canonical answer on Stampy's Wiki?

Answer: [[Canonical answers]] may be served to readers by Stampy, so only answers which have a reasonably high stamp score should be marked as canonical. All canonical answers are open to be collaboratively edited and updated, and they should represent a consensus response (written from the Stampy Point Of View) to a question which is within Stampy's scope.

Answers to questions from YouTube comments should not be marked as canonical, and will generally remain as they were when originally written since they have details which are specific to an idiosyncratic question. YouTube answers may be forked into wiki answers, in order to better respond to a particular question, in which case the YouTube question should have its canonical version field set to the new more widely useful question.

Question: What should be marked as a canonical answer on Stampy's Wiki?

Answer:
［［Canonical answers］］ may be served to readers by Stampy, so only answers which have a reasonably high stamp score should be marked as canonical. All canonical answers are open to be collaboratively edited and updated, and they should represent a consensus response (written from the Stampy Point Of View) to a question which is within Stampy's scope.

Answers to questions from YouTube comments should not be marked as canonical, and will generally remain as they were when originally written since they have details which are specific to an idiosyncratic question. YouTube answers may be forked into wiki answers, in order to better respond to a particular question, in which case the YouTube question should have its canonical version field set to the new more widely useful question.

Plex's Answer to What should be marked as a canonical answer on Stampy's Wiki?


How can I collect questions for Stampy?

Question: How can I collect questions for Stampy?

Answer: As well as simply adding your own questions over at [[ask question]], you could also message your friends with something like:

Hi,

I'm working on a project to create a comprehensive FAQ about AI alignment (you can read about it here https://stampy.ai/wiki/Stampy%27s_Wiki if interested). We're looking for questions and I thought you may have some good ones. If you'd be willing to write up a google doc with you top 5-10ish questions we'd be happy to write a personalized FAQ for you. https://stampy.ai/wiki/Scope explains the kinds of questions we're looking for.

Thanks!

and maybe bring the google doc to a Stampy editing session so we can collaborate on answering them or improving your answers to them.

Question: How can I collect questions for Stampy?

Answer:
As well as simply adding your own questions over at ［［ask question］］, you could also message your friends with something like:

Hi,

I'm working on a project to create a comprehensive FAQ about AI alignment (you can read about it here https://stampy.ai/wiki/Stampy%27s_Wiki if interested). We're looking for questions and I thought you may have some good ones. If you'd be willing to write up a google doc with you top 5-10ish questions we'd be happy to write a personalized FAQ for you. https://stampy.ai/wiki/Scope explains the kinds of questions we're looking for.

Thanks!

and maybe bring the google doc to a Stampy editing session so we can collaborate on answering them or improving your answers to them.

Plex's Answer to How can I collect questions for Stampy?


Answer to What is MIRI’s mission?

What is MIRI’s mission?

Question: What is MIRI's mission?

Answer: [https://intelligence.org/ MIRI's] mission statement is to "ensure that the creation of smarter-than-human artificial intelligence has a positive impact." This is an ambitious goal, but they believe that some early progress is possible, and they believe that the goal's importance and difficulty makes it prudent to begin work at an early date.

Their two main research agendas, "[https://intelligence.org/technical-agenda Agent Foundations for Aligning Machine Intelligence with Human Interests]" and "[https://intelligence.org/2016/05/04/announcing-a-new-research-program/ Value Alignment for Advanced Machine Learning Systems]," focus on three groups of technical problems:
* highly reliable agent design — learning how to specify highly autonomous systems that reliably pursue some fixed goal;
* value specification — supplying autonomous systems with the intended goals; and
* error tolerance — making such systems robust to programmer error.
That being said, MIRI recently [https://intelligence.org/2020/12/21/2020-updates-and-strategy/ published an update] stating that they were moving away from research directions in unpublished works that they were [https://intelligence.org/2018/11/22/2018-update-our-new-research-directions/ pursuing since 2017].

They publish new [https://intelligence.org/research mathematical results] (although their work is [https://intelligence.org/2018/11/22/2018-update-our-new-research-directions/#section3 non-disclosed by default]), host [https://intelligence.org/research workshops], attend conferences, and [https://intelligence.org/mirix fund outside researchers] who are interested in investigating these problems. They also host a [https://intelligence.org/blog blog] and an [https://www.alignmentforum.org/ online research forum].

Question: What is MIRI’s mission?

Answer:
［https://intelligence.org/ MIRI's］ mission statement is to “ensure that the creation of smarter-than-human artificial intelligence has a positive impact.” This is an ambitious goal, but they believe that some early progress is possible, and they believe that the goal’s importance and difficulty makes it prudent to begin work at an early date.

Their two main research agendas, “［https://intelligence.org/technical-agenda Agent Foundations for Aligning Machine Intelligence with Human Interests］” and “［https://intelligence.org/2016/05/04/announcing-a-new-research-program/ Value Alignment for Advanced Machine Learning Systems］,” focus on three groups of technical problems:
* highly reliable agent design — learning how to specify highly autonomous systems that reliably pursue some fixed goal;
* value specification — supplying autonomous systems with the intended goals; and
* error tolerance — making such systems robust to programmer error.
That being said, MIRI recently ［https://intelligence.org/2020/12/21/2020-updates-and-strategy/ published an update］ stating that they were moving away from research directions in unpublished works that they were ［https://intelligence.org/2018/11/22/2018-update-our-new-research-directions/ pursuing since 2017］.

They publish new ［https://intelligence.org/research mathematical results］ (although their work is ［https://intelligence.org/2018/11/22/2018-update-our-new-research-directions/#section3 non-disclosed by default］), host ［https://intelligence.org/research workshops］, attend conferences, and ［https://intelligence.org/mirix fund outside researchers］ who are interested in investigating these problems. They also host a ［https://intelligence.org/blog blog］ and an ［https://www.alignmentforum.org/ online research forum］.


Question: Can an AI really be smarter than humans?

Answer: Until a thing has happened, it has never happened. We have been consistently improving both the optimization power and generality of our algorithms over that time period, and have little reason to expect it to suddenly stop. We've gone from coding systems specifically for a certain game (like Chess), to algorithms like [https://deepmind.com/blog/article/muzero-mastering-go-chess-shogi-and-atari-without-rules MuZero] which learn the rules of the game they're playing and how to play at vastly superhuman skill levels purely via self-play across a broad range of games (e.g. Go, chess, shogi and various Atari games).

Human brains are a [https://www.lesswrong.com/posts/NQgWL7tvAPgN2LTLn/spaghetti-towers spaghetti tower] generated by evolution with zero foresight, it would be surprising if they are the peak of physically possible intelligence. The brain doing things in complex ways is not strong evidence that we need to fully replicate those interactions if we can throw sufficient compute at the problem, as explained in [https://www.lesswrong.com/posts/HhWhaSzQr6xmBki8F/birds-brains-planes-and-ai-against-appeals-to-the-complexity Birds, Brains, Planes, and AI: Against Appeals to the Complexity/Mysteriousness/Efficiency of the Brain].

It is, however, plausible that for an AGI we need a lot more compute than we will get in the near future, or that some key insights are missing which we won't get for a while. The [https://www.openphilanthropy.org/brain-computation-report#ExecutiveSummary OpenPhilanthropy report on how much computational power it would take to simulate the brain] is the most careful attempt at reasoning out how far we are from being able to do it, and suggests that by some estimates we already have enough computational resources, and by some estimates moore's law may let us reach it before too long.

It also seems that much of the human brain exists to observe and [https://en.wikipedia.org/wiki/Allostasis regulate our biological body], which a body-less computer wouldn't need. If that's true, then a human-level AI might be possible with considerably less compute than the human brain.

Question: Can an AI really be smarter than humans?

Answer:
Until a thing has happened, it has never happened. We have been consistently improving both the optimization power and generality of our algorithms over that time period, and have little reason to expect it to suddenly stop. We’ve gone from coding systems specifically for a certain game (like Chess), to algorithms like ［https://deepmind.com/blog/article/muzero-mastering-go-chess-shogi-and-atari-without-rules MuZero］ which learn the rules of the game they’re playing and how to play at vastly superhuman skill levels purely via self-play across a broad range of games (e.g. Go, chess, shogi and various Atari games).

Human brains are a ［https://www.lesswrong.com/posts/NQgWL7tvAPgN2LTLn/spaghetti-towers spaghetti tower］ generated by evolution with zero foresight, it would be surprising if they are the peak of physically possible intelligence. The brain doing things in complex ways is not strong evidence that we need to fully replicate those interactions if we can throw sufficient compute at the problem, as explained in ［https://www.lesswrong.com/posts/HhWhaSzQr6xmBki8F/birds-brains-planes-and-ai-against-appeals-to-the-complexity Birds, Brains, Planes, and AI: Against Appeals to the Complexity/Mysteriousness/Efficiency of the Brain］.

It is, however, plausible that for an AGI we need a lot more compute than we will get in the near future, or that some key insights are missing which we won’t get for a while. The ［https://www.openphilanthropy.org/brain-computation-report#ExecutiveSummary OpenPhilanthropy report on how much computational power it would take to simulate the brain］ is the most careful attempt at reasoning out how far we are from being able to do it, and suggests that by some estimates we already have enough computational resources, and by some estimates moore’s law may let us reach it before too long.

It also seems that much of the human brain exists to observe and ［https://en.wikipedia.org/wiki/Allostasis regulate our biological body］, which a body-less computer wouldn't need. If that's true, then a human-level AI might be possible with considerably less compute than the human brain.

Plex's Answer to Can an AI really be smarter than humans?


Why can't we simply stop developing AI?

Question: Why can't we simply stop developing AI?

Answer: We could, but we won't. Each advance in capabilities which brings us closer to an intelligence explosion also brings vast profits for whoever develops them (e.g. smarter digital personal assistants like Siri, more ability to automate cognitive tasks, better recommendation algorithms for Facebook, etc.). The incentives are all wrong. Any actor (nation or corporation) who stops will just get overtaken by more reckless ones, and everyone knows this.

Question: Why can't we simply stop developing AI?

Answer: We could, but we won’t. Each advance in capabilities which brings us closer to an intelligence explosion also brings vast profits for whoever develops them (e.g. smarter digital personal assistants like Siri, more ability to automate cognitive tasks, better recommendation algorithms for Facebook, etc.). The incentives are all wrong. Any actor (nation or corporation) who stops will just get overtaken by more reckless ones, and everyone knows this.

Plex's Answer to Why can't we simply stop developing AI?


Question: Can you stop an advanced AI from upgrading itself?

Answer: It depends on what is meant by advanced. Many AI systems which are very effective and advanced narrow intelligences would not try to upgrade themselves in an unbounded way, but becoming smarter is a [https://www.youtube.com/watch?v꞊ZeecOKBus3Q convergent instrumental goal] so we could expect most AGI designs to attempt it.

The problem is that increasing general problem solving ability is climbing in exactly the direction needed to trigger an intelligence explosion, while generating large economic and strategic payoffs to whoever achieves them. So even though we could, in principle, just not build the kind of systems which would recursively self-improve, in practice we probably will go ahead with constructing them, because they're likely to be the most powerful.

Question: Can you stop an advanced AI from upgrading itself?

Answer:
It depends on what is meant by advanced. Many AI systems which are very effective and advanced narrow intelligences would not try to upgrade themselves in an unbounded way, but becoming smarter is a ［https://www.youtube.com/watch?v꞊ZeecOKBus3Q convergent instrumental goal］ so we could expect most AGI designs to attempt it.

The problem is that increasing general problem solving ability is climbing in exactly the direction needed to trigger an intelligence explosion, while generating large economic and strategic payoffs to whoever achieves them. So even though we could, in principle, just not build the kind of systems which would recursively self-improve, in practice we probably will go ahead with constructing them, because they’re likely to be the most powerful.

Plex's Answer to Can you stop an advanced AI from upgrading itself?


Question: What exactly is AGI and what will it look like?

Answer: AGI is an algorithm with [https://www.lesswrong.com/posts/yLeEPFnnB9wE7KLx2/efficient-cross-domain-optimization general intelligence], running not on evolution's biology like all current general intelligences but on a substrate such as silicon engineered by an intelligence (initially computers designed by humans, later on likely dramatically more advanced hardware designed by earlier AGIs).

AI has so far always been designed and built by humans (i.e. a search process running on biological brains), but once our creations gain the ability to do AI research they will likely [https://www.lesswrong.com/posts/JBadX7rwdcRFzGuju/recursive-self-improvement recursively self-improve] by designing new and better versions of themselves initiating an [https://www.lesswrong.com/posts/8vpf46nLMDYPC6wA4/optimization-and-the-intelligence-explosion intelligence explosion] (i.e. use it's intelligence to improve its own intelligence, creating a feedback loop), and resulting in a superintelligence. There are already [https://arxiv.org/abs/2101.07367 early signs] of AIs being trained to optimize other AIs.

Some authors (notably [https://intelligence.org/ai-foom-debate/ Robin Hanson]) have argued that the intelligence explosion hypothesis is likely false, and in favor of a large number of roughly human level emulated minds operating instead, forming an uplifted economy which doubles every few hours. Eric Drexler's [https://slatestarcodex.com/2019/08/27/book-review-reframing-superintelligence/ Comprehensive AI Services] model of what may happen is another alternate view, where many narrow superintelligent systems exist in parallel rather than there being a general-purpose superintelligent agent.

Going by the model advocated by [https://en.wikipedia.org/wiki/Nick_Bostrom Nick Bostrom], [https://en.wikipedia.org/wiki/Eliezer_Yudkowsky Eliezer Yudkowsky] and many others, a superintelligence will likely gain various [https://publicism.info/philosophy/superintelligence/7.html cognitive superpowers] (table 8 gives a good overview), allowing it to direct the future much more effectively than humanity. Taking control of our resources by manipulation and hacking is a likely early step, followed by developing and deploying advanced technologies like [https://en.wikipedia.org/wiki/Molecular_nanotechnology molecular nanotechnology] to dominate the physical world and achieve its goals.

Question: What exactly is AGI and what will it look like?

Answer:
AGI is an algorithm with ［https://www.lesswrong.com/posts/yLeEPFnnB9wE7KLx2/efficient-cross-domain-optimization general intelligence］, running not on evolution’s biology like all current general intelligences but on a substrate such as silicon engineered by an intelligence (initially computers designed by humans, later on likely dramatically more advanced hardware designed by earlier AGIs).

AI has so far always been designed and built by humans (i.e. a search process running on biological brains), but once our creations gain the ability to do AI research they will likely ［https://www.lesswrong.com/posts/JBadX7rwdcRFzGuju/recursive-self-improvement recursively self-improve］ by designing new and better versions of themselves initiating an ［https://www.lesswrong.com/posts/8vpf46nLMDYPC6wA4/optimization-and-the-intelligence-explosion intelligence explosion］ (i.e. use it’s intelligence to improve its own intelligence, creating a feedback loop), and resulting in a superintelligence. There are already ［https://arxiv.org/abs/2101.07367 early signs］ of AIs being trained to optimize other AIs.

Some authors (notably ［https://intelligence.org/ai-foom-debate/ Robin Hanson］) have argued that the intelligence explosion hypothesis is likely false, and in favor of a large number of roughly human level emulated minds operating instead, forming an uplifted economy which doubles every few hours. Eric Drexler’s ［https://slatestarcodex.com/2019/08/27/book-review-reframing-superintelligence/ Comprehensive AI Services］ model of what may happen is another alternate view, where many narrow superintelligent systems exist in parallel rather than there being a general-purpose superintelligent agent.

Going by the model advocated by ［https://en.wikipedia.org/wiki/Nick_Bostrom Nick Bostrom］, ［https://en.wikipedia.org/wiki/Eliezer_Yudkowsky Eliezer Yudkowsky］ and many others, a superintelligence will likely gain various ［https://publicism.info/philosophy/superintelligence/7.html cognitive superpowers］ (table 8 gives a good overview), allowing it to direct the future much more effectively than humanity. Taking control of our resources by manipulation and hacking is a likely early step, followed by developing and deploying advanced technologies like ［https://en.wikipedia.org/wiki/Molecular_nanotechnology molecular nanotechnology］ to dominate the physical world and achieve its goals.

Plex's Answer to What exactly is AGI and what will it look like?


Question: Why should I worry about superintelligence?

Answer: Intelligence is powerful. Because of superior intelligence, we humans have dominated the Earth. The fate of thousands of species depends on our actions, we occupy nearly every corner of the globe, and we repurpose vast amounts of the world's resources for our own use. Artificial Superintelligence (ASI) has potential to be vastly more intelligent than us, and therefore vastly more powerful. In the same way that we have reshaped the earth to fit our goals, an ASI will find unforeseen, highly efficient ways of reshaping reality to fit its goals.

The impact that an ASI will have on our world depends on what those goals are. We have the advantage of designing those goals, but that task is not as simple as it may first seem. As described by MIRI in their [https://intelligence.org/ie-faq/ Intelligence Explosion FAQ]:

"A superintelligent machine will make decisions based on the mechanisms it is designed with, not the hopes its designers had in mind when they programmed those mechanisms. It will act only on precise specifications of rules and values, and will do so in ways that need not respect the complexity and subtlety of what humans value."

If we do not solve the Control Problem before the first ASI is created, we may not get another chance.

Question: Why should I worry about superintelligence?

Answer:
Intelligence is powerful. Because of superior intelligence, we humans have dominated the Earth. The fate of thousands of species depends on our actions, we occupy nearly every corner of the globe, and we repurpose vast amounts of the world's resources for our own use. Artificial Superintelligence (ASI) has potential to be vastly more intelligent than us, and therefore vastly more powerful. In the same way that we have reshaped the earth to fit our goals, an ASI will find unforeseen, highly efficient ways of reshaping reality to fit its goals.

The impact that an ASI will have on our world depends on what those goals are. We have the advantage of designing those goals, but that task is not as simple as it may first seem. As described by MIRI in their ［https://intelligence.org/ie-faq/ Intelligence Explosion FAQ］:

“A superintelligent machine will make decisions based on the mechanisms it is designed with, not the hopes its designers had in mind when they programmed those mechanisms. It will act only on precise specifications of rules and values, and will do so in ways that need not respect the complexity and subtlety of what humans value.”

If we do not solve the Control Problem before the first ASI is created, we may not get another chance.

Answer to Why should I worry about superintelligence?


When will transformative AI be created?

Question: When will transformative AI be created?

Answer: As is often said, it's difficult to make predictions, especially about the future. This has not stopped many people thinking about when AI will transform the world, but all predictions should come with a warning that it's a hard domain to find anything like certainty.

[https://www.alignmentforum.org/posts/KrJfoZzpSDpnrv9va/draft-report-on-ai-timelines This report] for the Open Philanthropy Project is perhaps the most careful attempt so far (and generates [https://docs.google.com/spreadsheets/d/1TjNQyVHvHlC-sZbcA7CRKcCp0NxV6MkkqBvL408xrJw/edit#gid꞊505210495 these graphs], which peak at 2042), and there's been much discussion including [https://www.alignmentforum.org/posts/rzqACeBGycZtqCfaX/fun-with-12-ooms-of-compute this reply and analysis] which argues that we likely need less compute than the OpenPhil report expects.

There have also been [https://slatestarcodex.com/2017/06/08/ssc-journal-club-ai-timelines/ expert surveys], and many people have [https://www.lesswrong.com/tag/ai-timelines shared various thoughts]. Berkeley AI professor [https://en.wikipedia.org/wiki/Stuart_J._Russell Stuart Russell] has given his best guess as "sometime in our children's lifetimes", and [https://en.wikipedia.org/wiki/Ray_Kurzweil Ray Kurzweil] (Futurist and Google's director of engineering) predicts [https://futurism.com/kurzweil-claims-that-the-singularity-will-happen-by-2045 human level AI by 2029 and the singularity by 2045]. The [https://www.metaculus.com/questions/3479/when-will-the-first-artificial-general-intelligence-system-be-devised-tested-and-publicly-known-of/ Metaculus question on publicly known AGI] has a median of around 2029 (around 10 years sooner than it was before the GPT-3 AI showed [https://gpt3examples.com/ unexpected ability on a broad range of tasks]).

The consensus answer, if there was one, might be something like: "highly uncertain, maybe not for over a hundred years, maybe in less than 15, with around the middle of the century looking fairly plausible".

Question: When will transformative AI be created?

Answer:
As is often said, it's difficult to make predictions, especially about the future. This has not stopped many people thinking about when AI will transform the world, but all predictions should come with a warning that it's a hard domain to find anything like certainty.

［https://www.alignmentforum.org/posts/KrJfoZzpSDpnrv9va/draft-report-on-ai-timelines This report］ for the Open Philanthropy Project is perhaps the most careful attempt so far (and generates ［https://docs.google.com/spreadsheets/d/1TjNQyVHvHlC-sZbcA7CRKcCp0NxV6MkkqBvL408xrJw/edit#gid꞊505210495 these graphs］, which peak at 2042), and there's been much discussion including ［https://www.alignmentforum.org/posts/rzqACeBGycZtqCfaX/fun-with-12-ooms-of-compute this reply and analysis］ which argues that we likely need less compute than the OpenPhil report expects.

There have also been ［https://slatestarcodex.com/2017/06/08/ssc-journal-club-ai-timelines/ expert surveys］, and many people have ［https://www.lesswrong.com/tag/ai-timelines shared various thoughts］. Berkeley AI professor ［https://en.wikipedia.org/wiki/Stuart_J._Russell Stuart Russell］ has given his best guess as “sometime in our children’s lifetimes”, and ［https://en.wikipedia.org/wiki/Ray_Kurzweil Ray Kurzweil］ (Futurist and Google’s director of engineering) predicts ［https://futurism.com/kurzweil-claims-that-the-singularity-will-happen-by-2045 human level AI by 2029 and the singularity by 2045］. The ［https://www.metaculus.com/questions/3479/when-will-the-first-artificial-general-intelligence-system-be-devised-tested-and-publicly-known-of/ Metaculus question on publicly known AGI］ has a median of around 2029 (around 10 years sooner than it was before the GPT-3 AI showed ［https://gpt3examples.com/ unexpected ability on a broad range of tasks］).

The consensus answer, if there was one, might be something like: “highly uncertain, maybe not for over a hundred years, maybe in less than 15, with around the middle of the century looking fairly plausible”.

Plex's Answer to When will transformative AI be created?


Question: Would AI alignment be hard with deep learning?

Answer: Ajeya Cotra has written an excellent article named [https://www.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/ Why AI alignment could be hard with modern deep learning] on this question.

Question: Would AI alignment be hard with deep learning?

Answer: Ajeya Cotra has written an excellent article named ［https://www.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/ Why AI alignment could be hard with modern deep learning］ on this question.

Nico Hill2's Answer to Would AI alignment be hard with deep learning?


How can I contribute  Stampy?

Question: How can I contribute  Stampy?

Answer: If you're not already there,  join the [[Discord invite┊Discord]] where the contributors hang out.

The main ways you can help are to [[answer questions]] or [[add question]]s, or help to [[review questions]], [[review answers]], or [[improve answers]] (instructions for helping out with each of these tasks are on the linked pages). You could also [https://stampy.ai/wiki/How_can_I_join_the_Stampy_dev_team%3F join the dev team] if you have programming skills.

Question: How can I contribute  Stampy?

Answer:
If you're not already there,  join the ［［Discord invite┊Discord］］ where the contributors hang out.

The main ways you can help are to ［［answer questions］］ or ［［add question］］s, or help to ［［review questions］］, ［［review answers］］, or ［［improve answers］］ (instructions for helping out with each of these tasks are on the linked pages). You could also ［https://stampy.ai/wiki/How_can_I_join_the_Stampy_dev_team%3F join the dev team］ if you have programming skills.

Plex's Answer to How can I contribute to Stampy?


Plex's Answer to Why is AGI dangerous?

Why is AGI dangerous?

Question: Why is AGI dangerous?

Answer: # [https://www.youtube.com/watch?v꞊hEUO6pjwFOo The Orthogonality Thesis]: AI could have almost any goal while at the same time having high intelligence (aka ability to succeed at those goals). This means that we could build a very powerful agent which would not necessarily share human-friendly values. For example, the classic [https://www.lesswrong.com/tag/paperclip-maximizer paperclip maximizer] thought experiment explores this with an AI which has a goal of creating as many paperclips as possible, something that humans are (mostly) indifferent to, and as a side effect ends up destroying humanity to make room for more paperclip factories.
# [https://www.lesswrong.com/posts/GNnHHmm8EzePmKzPk/value-is-fragile Complexity of value]: What humans care about is not simple, and the space of all goals is large, so virtually all goals we could program into an AI would lead to worlds not valuable to humans if pursued by a sufficiently powerful agent. If we, for example, did not include our value of diversity of experience, we could end up with a world of endlessly looping simple pleasures, rather than beings living rich lives.
# [https://www.youtube.com/watch?v꞊ZeecOKBus3Q Instrumental Convergence]: For almost any goal an AI has there are shared 'instrumental' steps, such as acquiring resources, preserving itself, and preserving the contents of its goals. This means that a powerful AI with goals that were not explicitly human-friendly would predictably both take actions that lead to the end of humanity (e.g. using resources humans need to live to further its goals, such as replacing our crop fields with vast numbers of solar panels to power its growth, or using the carbon in our bodies to build things) and prevent us from turning it off or altering its goals.

Question: Why is AGI dangerous?

Answer:
# ［https://www.youtube.com/watch?v꞊hEUO6pjwFOo The Orthogonality Thesis］: AI could have almost any goal while at the same time having high intelligence (aka ability to succeed at those goals). This means that we could build a very powerful agent which would not necessarily share human-friendly values. For example, the classic ［https://www.lesswrong.com/tag/paperclip-maximizer paperclip maximizer］ thought experiment explores this with an AI which has a goal of creating as many paperclips as possible, something that humans are (mostly) indifferent to, and as a side effect ends up destroying humanity to make room for more paperclip factories.
# ［https://www.lesswrong.com/posts/GNnHHmm8EzePmKzPk/value-is-fragile Complexity of value］: What humans care about is not simple, and the space of all goals is large, so virtually all goals we could program into an AI would lead to worlds not valuable to humans if pursued by a sufficiently powerful agent. If we, for example, did not include our value of diversity of experience, we could end up with a world of endlessly looping simple pleasures, rather than beings living rich lives.
# ［https://www.youtube.com/watch?v꞊ZeecOKBus3Q Instrumental Convergence］: For almost any goal an AI has there are shared ‘instrumental’ steps, such as acquiring resources, preserving itself, and preserving the contents of its goals. This means that a powerful AI with goals that were not explicitly human-friendly would predictably both take actions that lead to the end of humanity (e.g. using resources humans need to live to further its goals, such as replacing our crop fields with vast numbers of solar panels to power its growth, or using the carbon in our bodies to build things) and prevent us from turning it off or altering its goals.


Question: What are the main sources of AI existential risk?

Answer: A comprehensive list of major contributing factors to AI being a threat to humanity's future is maintained on by Daniel Kokotajlo on the [https://www.alignmentforum.org/posts/WXvt8bxYnwBYpy9oT/the-main-sources-of-ai-risk Alignment Forum].

Question: What are the main sources of AI existential risk?

Answer: A comprehensive list of major contributing factors to AI being a threat to humanity's future is maintained on by Daniel Kokotajlo on the ［https://www.alignmentforum.org/posts/WXvt8bxYnwBYpy9oT/the-main-sources-of-ai-risk Alignment Forum］.

Plex's Answer to What are the main sources of AI existential risk?


Question: What is the DeepMind's safety team working on?

Answer: DeepMind has both a [https://80000hours.org/podcast/episodes/pushmeet-kohli-deepmind-safety-research/#long-term-agi-safety-research ML safety team focused on near-term risks], and an alignment team that is working on risks from AGI. The alignment team is pursuing many different research avenues, and is not best described by a single agenda.

Some of the work they are doing is:

*[https://www.lesswrong.com/posts/qJgz2YapqpFEDTLKn/deepmind-alignment-team-opinions-on-agi-ruin-arguments Engaging with recent MIRI arguments].
*Rohin Shah produces the [https://rohinshah.com/alignment-newsletter/ alignment newsletter].
*Publishing interesting research like the [https://arxiv.org/abs/2105.14111 Goal Misgeneralization paper].
*Geoffrey Irving is working on debate as an alignment strategy: [https://www.lesswrong.com/posts/bLr68nrLSwgzqLpzu/axrp-episode-16-preparing-for-debate-ai-with-geoffrey-irving more detail here].
*[https://www.lesswrong.com/posts/XxX2CAoFskuQNkBDy/discovering-agents Discovering agents], which introduces a causal definition of agents, then introduces an algorithm for finding agents from empirical data.

See [https://www.lesswrong.com/posts/QBAjndPuFbhEXKcCr/my-understanding-of-what-everyone-in-technical-alignment-is?commentId꞊CS9qcdkmDbLHR89s2 Rohin's comment] for more research that they are doing, including description of some that is currently unpublished so far.

Question: What is the DeepMind's safety team working on?

Answer:
DeepMind has both a ［https://80000hours.org/podcast/episodes/pushmeet-kohli-deepmind-safety-research/#long-term-agi-safety-research ML safety team focused on near-term risks］, and an alignment team that is working on risks from AGI. The alignment team is pursuing many different research avenues, and is not best described by a single agenda.

Some of the work they are doing is:

*［https://www.lesswrong.com/posts/qJgz2YapqpFEDTLKn/deepmind-alignment-team-opinions-on-agi-ruin-arguments Engaging with recent MIRI arguments］.
*Rohin Shah produces the ［https://rohinshah.com/alignment-newsletter/ alignment newsletter］.
*Publishing interesting research like the ［https://arxiv.org/abs/2105.14111 Goal Misgeneralization paper］.
*Geoffrey Irving is working on debate as an alignment strategy: ［https://www.lesswrong.com/posts/bLr68nrLSwgzqLpzu/axrp-episode-16-preparing-for-debate-ai-with-geoffrey-irving more detail here］.
*［https://www.lesswrong.com/posts/XxX2CAoFskuQNkBDy/discovering-agents Discovering agents］, which introduces a causal definition of agents, then introduces an algorithm for finding agents from empirical data.

See ［https://www.lesswrong.com/posts/QBAjndPuFbhEXKcCr/my-understanding-of-what-everyone-in-technical-alignment-is?commentId꞊CS9qcdkmDbLHR89s2 Rohin's comment］ for more research that they are doing, including description of some that is currently unpublished so far.

RoseMcClelland's Answer to What is the DeepMind's safety team working on?


Question: How might Shard Theory help with alignment?

Answer: Humans care about things! The reward circuitry in our brain reliably causes us to care about specific things. Let's create a mechanistic model of how the brain aligns humans, and then we can use this to do AI alignment.

One perspective that Shard theory has added is that we shouldn't think of the solution to alignment as:

#Find an outer objective that is fine to optimize arbitrarily strongly
#Find a way of making sure that the inner objective of an ML system equals the outer objective.

Shard theory argues that instead we should focus on finding outer objectives that reliably give certain inner values into system and should be thought of as more of a teacher of the values we want to instill as opposed to the values themselves. [https://www.lesswrong.com/posts/pdaGN6pQyQarFHXF4/reward-is-not-the-optimization-target Reward is not the optimization target] — instead, it is more like that which reinforces. People sometimes refer to inner aligning an RL agent with respect to the reward signal, but this doesn't actually make sense. (As pointed out in the comments this is not a new insight, but it was for me phrased a lot more clearly in terms of Shard theory).

Humans have different values than the reward circuitry in our brain being maximized, but they are still pointed reliably. These underlying values cause us to not wirehead with respect to the outer optimizer of reward.

Shard Theory points at the beginning of a mechanistic story for how inner values are selected for by outer optimization pressures. [https://www.lesswrong.com/posts/ZNXDRGshgoq3cmxhB/the-shard-theory-alignment-scheme The current plan] is to figure out how RL induces inner values into learned agents, and then figure out how to instill human values into powerful AI models (probably chain of thought LLMs, because these are the most intelligent models right now). Then, use these partially aligned models to solve the full alignment problem. Shard theory also proposes a subagent theory of mind.

This has some similarities to Brain-like AGI Safety, and has drawn on some research from this post, such as the mechanics of the human reward circuitry as well as the brain being mostly randomly initialized at birth.

Question: How might Shard Theory help with alignment?

Answer:
Humans care about things! The reward circuitry in our brain reliably causes us to care about specific things. Let's create a mechanistic model of how the brain aligns humans, and then we can use this to do AI alignment.

One perspective that Shard theory has added is that we shouldn't think of the solution to alignment as:

#Find an outer objective that is fine to optimize arbitrarily strongly
#Find a way of making sure that the inner objective of an ML system equals the outer objective.

Shard theory argues that instead we should focus on finding outer objectives that reliably give certain inner values into system and should be thought of as more of a teacher of the values we want to instill as opposed to the values themselves. ［https://www.lesswrong.com/posts/pdaGN6pQyQarFHXF4/reward-is-not-the-optimization-target Reward is not the optimization target］ — instead, it is more like that which reinforces. People sometimes refer to inner aligning an RL agent with respect to the reward signal, but this doesn't actually make sense. (As pointed out in the comments this is not a new insight, but it was for me phrased a lot more clearly in terms of Shard theory).

Humans have different values than the reward circuitry in our brain being maximized, but they are still pointed reliably. These underlying values cause us to not wirehead with respect to the outer optimizer of reward.

Shard Theory points at the beginning of a mechanistic story for how inner values are selected for by outer optimization pressures. ［https://www.lesswrong.com/posts/ZNXDRGshgoq3cmxhB/the-shard-theory-alignment-scheme The current plan］ is to figure out how RL induces inner values into learned agents, and then figure out how to instill human values into powerful AI models (probably chain of thought LLMs, because these are the most intelligent models right now). Then, use these partially aligned models to solve the full alignment problem. Shard theory also proposes a subagent theory of mind.

This has some similarities to Brain-like AGI Safety, and has drawn on some research from this post, such as the mechanics of the human reward circuitry as well as the brain being mostly randomly initialized at birth.

RoseMcClelland's Answer to How might Shard Theory help with alignment?


Question: What is the Center on Long-Term Risk (CLR) focused on?

Answer: CLR is focused primarily on reducing suffering-risk (s-risk), where the future has a large negative value. They do foundational research in game theory / decision theory, primarily aimed at multipolar AI scenarios. One result relevant to this work is that [https://www.cambridge.org/core/journals/journal-of-symbolic-logic/article/abs/parametric-resourcebounded-generalization-of-lobs-theorem-and-a-robust-cooperation-criterion-for-opensource-game-theory/16063EA7BFFEE89438631B141E556E79 transparency can increase cooperation].

[https://www.lesswrong.com/posts/QBAjndPuFbhEXKcCr/my-understanding-of-what-everyone-in-technical-alignment-is?commentId꞊mqiYR6X8bgY5wKdme Update after Jesse Clifton commented]: CLR also works on improving coordination for prosaic AI scenarios, [https://forum.effectivealtruism.org/posts/LpkXtFXdsRd4rG8Kb/reducing-long-term-risks-from-malevolent-actors risks from malevolent actors] and [https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-looks-like AI forecasting]. The [https://www.cooperativeai.com/foundation Cooperative AI Foundation (CAIF)] shares personnel with CLR, but is not formally affiliated with CLR, and does not focus just on s-risks.

Question: What is the Center on Long-Term Risk (CLR) focused on?

Answer:
CLR is focused primarily on reducing suffering-risk (s-risk), where the future has a large negative value. They do foundational research in game theory / decision theory, primarily aimed at multipolar AI scenarios. One result relevant to this work is that ［https://www.cambridge.org/core/journals/journal-of-symbolic-logic/article/abs/parametric-resourcebounded-generalization-of-lobs-theorem-and-a-robust-cooperation-criterion-for-opensource-game-theory/16063EA7BFFEE89438631B141E556E79 transparency can increase cooperation］.

［https://www.lesswrong.com/posts/QBAjndPuFbhEXKcCr/my-understanding-of-what-everyone-in-technical-alignment-is?commentId꞊mqiYR6X8bgY5wKdme Update after Jesse Clifton commented］: CLR also works on improving coordination for prosaic AI scenarios, ［https://forum.effectivealtruism.org/posts/LpkXtFXdsRd4rG8Kb/reducing-long-term-risks-from-malevolent-actors risks from malevolent actors］ and ［https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-looks-like AI forecasting］. The ［https://www.cooperativeai.com/foundation Cooperative AI Foundation (CAIF)］ shares personnel with CLR, but is not formally affiliated with CLR, and does not focus just on s-risks.

RoseMcClelland's Answer to What is the Center on Long-Term Risk (CLR) focused on?


Question: What is Anthropic's approach  LLM alignment?

Answer: Anthropic fine tuned a language model to be more helpful, honest and harmless: [https://arxiv.org/abs/2112.00861 HHH].

Motivation: The point of this is to:
#see if we can "align" a current day LLM, and
#raise awareness about safety in the broader ML community.

'''[[How can we interpret what all the neurons mean?]]'''

Chris Olah, the interpretability legend, is working on looking really hard at all the neurons to see what they all mean. The approach he pioneered is [https://distill.pub/2020/circuits/zoom-in/ circuits]: looking at computational subgraphs of the network, called circuits, and interpreting those. Idea: "decompiling the network into a better representation that is more interpretable". In-context learning via attention heads, and interpretability here seems useful.

One result I heard about recently: a linear softmax unit stretches space and encourages neuron monosemanticity (making a neuron represent only one thing, as opposed to firing on many unrelated concepts). This makes the network easier to interpret.

Motivation: The point of this is to get as many bits of information about what neural networks are doing, to hopefully find better abstractions. This diagram gets posted everywhere, the hope being that networks, in the current regime, will become more interpretable because they will start to use abstractions that are closer to human abstractions.

'''[[How do you figure out model performance scales?]]'''

Question: What is Anthropic's approach  LLM alignment?

Answer:
Anthropic fine tuned a language model to be more helpful, honest and harmless: ［https://arxiv.org/abs/2112.00861 HHH］.

Motivation: The point of this is to:
#see if we can "align" a current day LLM, and
#raise awareness about safety in the broader ML community.

'''［［How can we interpret what all the neurons mean?］］'''

Chris Olah, the interpretability legend, is working on looking really hard at all the neurons to see what they all mean. The approach he pioneered is ［https://distill.pub/2020/circuits/zoom-in/ circuits］: looking at computational subgraphs of the network, called circuits, and interpreting those. Idea: "decompiling the network into a better representation that is more interpretable". In-context learning via attention heads, and interpretability here seems useful.

One result I heard about recently: a linear softmax unit stretches space and encourages neuron monosemanticity (making a neuron represent only one thing, as opposed to firing on many unrelated concepts). This makes the network easier to interpret.

Motivation: The point of this is to get as many bits of information about what neural networks are doing, to hopefully find better abstractions. This diagram gets posted everywhere, the hope being that networks, in the current regime, will become more interpretable because they will start to use abstractions that are closer to human abstractions.

'''［［How do you figure out model performance scales?］］'''

RoseMcClelland's Answer to What is Anthropic's approach to LLM alignment?


Question: How is the Alignment Research Center (ARC) trying  solve Eliciting Latent Knowledge (ELK)?

Answer: ARC is trying to solve [https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit Eliciting Latent Knowledge (ELK)]. Suppose that you are training an AI agent that predicts the state of the world and then performs some actions, called a ''predictor''. This predictor is the AGI that will be acting to accomplish goals in the world. How can you create another model, called a ''reporter'', that tells you what the predictor believes about the world? A key challenge in training this reporter is that training your reporter on human labeled training data, by default, incentivizes the predictor to just model what the human thinks is true, because the human is a simpler model than the AI.

Motivation: At a high level, Paul's plan seems to be to produce a minimal AI that can help to do AI safety research. To do this, preventing [https://www.lesswrong.com/posts/ocWqg2Pf2br4jMmKA/does-sgd-produce-deceptive-alignment deception] and [https://www.lesswrong.com/posts/pL56xPoniLvtMDQ4J/the-inner-alignment-problem inner alignment failure] are on the critical path, and the only known solution paths to this require interpretability (this is how all of Evan's [https://www.lesswrong.com/posts/fRsjBseRuvRhMPPE5/an-overview-of-11-proposals-for-building-safe-advanced-ai 11 proposals] plan to get around this problem).

If ARC can solve ELK, this would be a very strong form of interpretability: our reporter is able to tell us what the predictor believes about the world. Some ways this could end up being useful for aligning the predictor include:

*Using the reporter to find deceptive/misaligned thoughts in the predictor, and then optimizing against those interpreted thoughts. At any given point in time, SGD only updates the weights a small amount. If an AI becomes misaligned, it won't be very misaligned, and the interpretability tools will be able to figure this out and do a gradient step to make it aligned again. In this way, we can prevent deception at any point in training.
*Stopping training if the AI is misaligned.

Question: How is the Alignment Research Center (ARC) trying  solve Eliciting Latent Knowledge (ELK)?

Answer:
ARC is trying to solve ［https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit Eliciting Latent Knowledge (ELK)］. Suppose that you are training an AI agent that predicts the state of the world and then performs some actions, called a ''predictor''. This predictor is the AGI that will be acting to accomplish goals in the world. How can you create another model, called a ''reporter'', that tells you what the predictor believes about the world? A key challenge in training this reporter is that training your reporter on human labeled training data, by default, incentivizes the predictor to just model what the human thinks is true, because the human is a simpler model than the AI.

Motivation: At a high level, Paul's plan seems to be to produce a minimal AI that can help to do AI safety research. To do this, preventing ［https://www.lesswrong.com/posts/ocWqg2Pf2br4jMmKA/does-sgd-produce-deceptive-alignment deception］ and ［https://www.lesswrong.com/posts/pL56xPoniLvtMDQ4J/the-inner-alignment-problem inner alignment failure］ are on the critical path, and the only known solution paths to this require interpretability (this is how all of Evan's ［https://www.lesswrong.com/posts/fRsjBseRuvRhMPPE5/an-overview-of-11-proposals-for-building-safe-advanced-ai 11 proposals］ plan to get around this problem).

If ARC can solve ELK, this would be a very strong form of interpretability: our reporter is able to tell us what the predictor believes about the world. Some ways this could end up being useful for aligning the predictor include:

*Using the reporter to find deceptive/misaligned thoughts in the predictor, and then optimizing against those interpreted thoughts. At any given point in time, SGD only updates the weights a small amount. If an AI becomes misaligned, it won't be very misaligned, and the interpretability tools will be able to figure this out and do a gradient step to make it aligned again. In this way, we can prevent deception at any point in training.
*Stopping training if the AI is misaligned.

RoseMcClelland's Answer to How is the Alignment Research Center (ARC) trying to solve Eliciting Latent Knowledge (ELK)?


What is neural network modularity?

Question: What is neural network modularity?

Answer: If a neural network is ''modular'', that means it consists of clusters (modules) of neurons, such that the neurons within the cluster are strongly connected to each other, but only weakly connected to the rest of the network.

Making networks more modular is useful if the modules represent concepts which are understandable because this helps understand the whole system better.

Relevant papers about modularity are [https://arxiv.org/abs/2003.04881v2 Neural Networks are Surprisingly Modular], [https://arxiv.org/abs/2103.03386 Clusterability in Neural Networks], and [https://openreview.net/forum?id꞊tFQyjbOz34 Detecting Modularity in Deep Neural Networks].

Question: What is neural network modularity?

Answer:
If a neural network is ''modular'', that means it consists of clusters (modules) of neurons, such that the neurons within the cluster are strongly connected to each other, but only weakly connected to the rest of the network.

Making networks more modular is useful if the modules represent concepts which are understandable because this helps understand the whole system better.

Relevant papers about modularity are ［https://arxiv.org/abs/2003.04881v2 Neural Networks are Surprisingly Modular］, ［https://arxiv.org/abs/2103.03386 Clusterability in Neural Networks］, and ［https://openreview.net/forum?id꞊tFQyjbOz34 Detecting Modularity in Deep Neural Networks］.

Magdalena's Answer to What is neural network modularity?


Question: What is Conjecture's epistemology research agenda?

Answer: The alignment problem is really hard to do science on: we are trying to reason about the future, and we only get one shot, meaning that [https://www.lesswrong.com/posts/uhxpJyGYQ5FQRvdjY/abstracting-the-hardness-of-alignment-unbounded-atomic we can't iterate]. Therefore, it seems really useful to have a good understanding of meta-science/epistemology, i.e. reasoning about ways to do useful alignment research.

Question: What is Conjecture's epistemology research agenda?

Answer: The alignment problem is really hard to do science on: we are trying to reason about the future, and we only get one shot, meaning that ［https://www.lesswrong.com/posts/uhxpJyGYQ5FQRvdjY/abstracting-the-hardness-of-alignment-unbounded-atomic we can't iterate］. Therefore, it seems really useful to have a good understanding of meta-science/epistemology, i.e. reasoning about ways to do useful alignment research.

RoseMcClelland's Answer to What is Conjecture's epistemology research agenda?


Question: What is Conjecture's Scalable LLM Interpretability research adgenda?

Answer: I don't know much about their research here, other than that they train their own models, which allow them to work on models that are bigger than the biggest publicly available models, which seems like a difference from Redwood.

Current interpretability methods are very low level (e.g., "what does x neuron do"), which does not help us answer high level questions like "is this AI trying to kill us".

They are trying a bunch of weird approaches, with the goal of scalable mechanistic interpretability, but I do not know what these approaches actually are.

Motivation: Conjecture wants to build towards a better paradigm that will give us a lot more information, primarily from the empirical direction (as distinct from ARC, which is working on interpretability with a theoretical focus).

Question: What is Conjecture's Scalable LLM Interpretability research adgenda?

Answer:
I don't know much about their research here, other than that they train their own models, which allow them to work on models that are bigger than the biggest publicly available models, which seems like a difference from Redwood.

Current interpretability methods are very low level (e.g., "what does x neuron do"), which does not help us answer high level questions like "is this AI trying to kill us".

They are trying a bunch of weird approaches, with the goal of scalable mechanistic interpretability, but I do not know what these approaches actually are.

Motivation: Conjecture wants to build towards a better paradigm that will give us a lot more information, primarily from the empirical direction (as distinct from ARC, which is working on interpretability with a theoretical focus).

RoseMcClelland's Answer to What is Conjecture's Scalable LLM Interpretability research adgenda?


Question: What is Aligned AI / Stuart Armstrong working on?

Answer: One of the key problems in AI safety is that there are many ways for an AI to generalize off-distribution, so it is very likely that an arbitrary generalization will be unaligned. See the [https://www.lesswrong.com/posts/k54rgSg7GcjtXnMHX/model-splintering-moving-from-one-imperfect-model-to-another-1 model splintering post] for more detail.  Stuart's plan to solve this problem is as follows:

#Maintain a set of all possible extrapolations of reward data that are consistent with the training process.
#Pick among these for a safe reward extrapolation.

They are currently working on algorithms to accomplish step 1: see [https://www.lesswrong.com/posts/i8sHdLyGQeBTGwTqq/value-extrapolation-concept-extrapolation-model-splintering Value Extrapolation].

Their initial operationalization of this problem is the lion and husky problem. Basically: if you train an image model on a dataset of images of lions and huskies, the lions are always in the desert, and the huskies are always in the snow. So the problem of learning a classifier is under-defined: should the classifier be classifying based on the background environment (e.g. snow vs sand), or based on the animal in the image?

A good extrapolation algorithm, on this problem, would generate classifiers that extrapolate in all the different ways[4],  and so the 'correct' extrapolation must be in this generated set of classifiers. They have also introduced a new dataset for this, with a similar idea: [https://www.lesswrong.com/posts/DiEWbwrChuzuhJhGr/benchmark-for-successful-concept-extrapolation-avoiding-goal Happy Faces].

Step 2 could be done in different ways. Possibilities for doing this include: [https://www.lesswrong.com/posts/PADPJ3xac5ogjEGwA/defeating-goodhart-and-the-closest-unblocked-strategy conservatism], [https://www.lesswrong.com/posts/BeeirdrMXCPYZwgfj/the-blue-minimising-robot-and-model-splintering generalized deference to humans],  or an automated process for removing some goals.   like wireheading/deception/killing everyone.

Question: What is Aligned AI / Stuart Armstrong working on?

Answer:
One of the key problems in AI safety is that there are many ways for an AI to generalize off-distribution, so it is very likely that an arbitrary generalization will be unaligned. See the ［https://www.lesswrong.com/posts/k54rgSg7GcjtXnMHX/model-splintering-moving-from-one-imperfect-model-to-another-1 model splintering post］ for more detail.  Stuart's plan to solve this problem is as follows:

#Maintain a set of all possible extrapolations of reward data that are consistent with the training process.
#Pick among these for a safe reward extrapolation.

They are currently working on algorithms to accomplish step 1: see ［https://www.lesswrong.com/posts/i8sHdLyGQeBTGwTqq/value-extrapolation-concept-extrapolation-model-splintering Value Extrapolation］.

Their initial operationalization of this problem is the lion and husky problem. Basically: if you train an image model on a dataset of images of lions and huskies, the lions are always in the desert, and the huskies are always in the snow. So the problem of learning a classifier is under-defined: should the classifier be classifying based on the background environment (e.g. snow vs sand), or based on the animal in the image?

A good extrapolation algorithm, on this problem, would generate classifiers that extrapolate in all the different ways［4］,  and so the 'correct' extrapolation must be in this generated set of classifiers. They have also introduced a new dataset for this, with a similar idea: ［https://www.lesswrong.com/posts/DiEWbwrChuzuhJhGr/benchmark-for-successful-concept-extrapolation-avoiding-goal Happy Faces］.

Step 2 could be done in different ways. Possibilities for doing this include: ［https://www.lesswrong.com/posts/PADPJ3xac5ogjEGwA/defeating-goodhart-and-the-closest-unblocked-strategy conservatism］, ［https://www.lesswrong.com/posts/BeeirdrMXCPYZwgfj/the-blue-minimising-robot-and-model-splintering generalized deference to humans］,  or an automated process for removing some goals.   like wireheading/deception/killing everyone.

RoseMcClelland's Answer to What is Aligned AI / Stuart Armstrong working on?


Question: How would you explain the theory of Infra-Bayesianism?

Answer: See Vanessa's [https://www.lesswrong.com/posts/5bd75cc58225bf0670375575/the-learning-theoretic-ai-alignment-research-agenda research agenda] for more detail.

If we don't know how to do something given unbounded compute, we are just confused about the thing. Going from thinking that chess was impossible for machines to understanding [https://www.google.com/url?q꞊https://en.wikipedia.org/wiki/Minimax&sa꞊D&source꞊editors&ust꞊1661633213196096&usg꞊AOvVaw3m8tD5QAEl-XXhvaH4d1v3 minimax] was a really good step forward for designing chess AIs, ''even though minimax is completely intractable''.

Thus, we should seek to figure out how alignment might look in theory, and then try to bridge the theory-practice gap by making our proposal ever more efficient. The first step along this path is to figure out a universal [https://www.alignmentforum.org/tag/reinforcement-learning Reinforcement Learning] setting that we can place our formal agents in, and then prove regret bounds in.

A key problem in doing this is embeddedness. AIs can't have a perfect self model — this would be like imagining your ENTIRE brain, inside your brain. There are finite memory constraints. [https://www.lesswrong.com/s/CmrW8fCmSLK7E25sa Infra-Bayesianism] (IB) is essentially a theory of imprecise probability that lets you specify local / fuzzy things. IB allows agents to have abstract models of themselves, and thus works in an embedded setting.

[https://www.lesswrong.com/posts/gHgs2e2J5azvGFatb/infra-bayesian-physicalism-a-formal-theory-of-naturalized Infra-Bayesian Physicalism] (IBP) is an extension of this to RL. IBP allows us to
*Figure out what agents are running [by evaluating the counterfactual where the computation of the agent would output something different, and see if the physical universe is different].
*Give a program, classify it as an agent or a non agent, and then find its utility function.

Vanessa uses this formalism to describe [https://www.lesswrong.com/posts/dPmmuaz9szk26BkmD/vanessa-kosoy-s-shortform?commentId꞊vKw6DB9crncovPxED#vKw6DB9crncovPxED PreDCA], an alignment proposal based on IBP. This proposal assumes that an agent is an IBP agent, meaning that it is an RL agent with fuzzy probability distributions (along with some other things). The general outline of this proposal is as follows:
#Find all of the agents that preceded the AI
#Discard all of these agents that are powerful / non-human like
#Find all of the utility functions in the remaining agents
#Use combination of all of these utilities as the agent's utility function

Vanessa models an AI as a model based RL system with a WM, a reward function, and a policy derived from the WM + reward. [https://www.lesswrong.com/posts/GNhMPAWcfBCASy8e6/a-central-ai-alignment-problem-capabilities-generalization She claims that this avoids the sharp left turn]. The generalization problems come from the world model, but this is dealt with by having an epistemology that doesn't contain [https://www.lesswrong.com/posts/ethRJh2E7mSSjzCay/building-phenomenological-bridges bridge rules], and so the true world is the simplest explanation for the observed data.

It is open to show that this proposal also solves inner alignment, but there is some chance that it does.

This approach deviates from MIRI's plan, which is to focus on a narrow task to perform the pivotal act, and then add corrigibility. Vanessa instead tries to directly learn the user's preferences, and optimize those.

Question: How would you explain the theory of Infra-Bayesianism?

Answer:
See Vanessa's ［https://www.lesswrong.com/posts/5bd75cc58225bf0670375575/the-learning-theoretic-ai-alignment-research-agenda research agenda］ for more detail.

If we don't know how to do something given unbounded compute, we are just confused about the thing. Going from thinking that chess was impossible for machines to understanding ［https://www.google.com/url?q꞊https://en.wikipedia.org/wiki/Minimax&sa꞊D&source꞊editors&ust꞊1661633213196096&usg꞊AOvVaw3m8tD5QAEl-XXhvaH4d1v3 minimax］ was a really good step forward for designing chess AIs, ''even though minimax is completely intractable''.

Thus, we should seek to figure out how alignment might look in theory, and then try to bridge the theory-practice gap by making our proposal ever more efficient. The first step along this path is to figure out a universal ［https://www.alignmentforum.org/tag/reinforcement-learning Reinforcement Learning］ setting that we can place our formal agents in, and then prove regret bounds in.

A key problem in doing this is embeddedness. AIs can't have a perfect self model — this would be like imagining your ENTIRE brain, inside your brain. There are finite memory constraints. ［https://www.lesswrong.com/s/CmrW8fCmSLK7E25sa Infra-Bayesianism］ (IB) is essentially a theory of imprecise probability that lets you specify local / fuzzy things. IB allows agents to have abstract models of themselves, and thus works in an embedded setting.

［https://www.lesswrong.com/posts/gHgs2e2J5azvGFatb/infra-bayesian-physicalism-a-formal-theory-of-naturalized Infra-Bayesian Physicalism］ (IBP) is an extension of this to RL. IBP allows us to
*Figure out what agents are running ［by evaluating the counterfactual where the computation of the agent would output something different, and see if the physical universe is different］.
*Give a program, classify it as an agent or a non agent, and then find its utility function.

Vanessa uses this formalism to describe ［https://www.lesswrong.com/posts/dPmmuaz9szk26BkmD/vanessa-kosoy-s-shortform?commentId꞊vKw6DB9crncovPxED#vKw6DB9crncovPxED PreDCA］, an alignment proposal based on IBP. This proposal assumes that an agent is an IBP agent, meaning that it is an RL agent with fuzzy probability distributions (along with some other things). The general outline of this proposal is as follows:
#Find all of the agents that preceded the AI
#Discard all of these agents that are powerful / non-human like
#Find all of the utility functions in the remaining agents
#Use combination of all of these utilities as the agent's utility function

Vanessa models an AI as a model based RL system with a WM, a reward function, and a policy derived from the WM + reward. ［https://www.lesswrong.com/posts/GNhMPAWcfBCASy8e6/a-central-ai-alignment-problem-capabilities-generalization She claims that this avoids the sharp left turn］. The generalization problems come from the world model, but this is dealt with by having an epistemology that doesn't contain ［https://www.lesswrong.com/posts/ethRJh2E7mSSjzCay/building-phenomenological-bridges bridge rules］, and so the true world is the simplest explanation for the observed data.

It is open to show that this proposal also solves inner alignment, but there is some chance that it does.

This approach deviates from MIRI's plan, which is to focus on a narrow task to perform the pivotal act, and then add corrigibility. Vanessa instead tries to directly learn the user's preferences, and optimize those.

RoseMcClelland's Answer to How would you explain the theory of Infra-Bayesianism?


Question: How is OpenAI planning  solve the full alignment problem?

Answer: The safety team at OpenAI's plan is to build a [https://aligned.substack.com/p/alignment-mvp MVP aligned AGI] to try and help us solve the full alignment problem.

They want to do this with Reinforcement Learning from Human Feedback (RLHF): get feedback from humans about what is good, i.e. give reward to AI's based on the human feedback. Problem: what if the AI makes gigabrain 5D chess moves that humans don't understand, so can't evaluate. Jan Leike, the director of the safety team, views this ([https://ai-alignment.com/the-informed-oversight-problem-1b51b4f66b35 the informed oversight problem]) as the core difficulty of alignment. Their proposed solution: an AI assisted oversight scheme, with a recursive hierarchy of AIs bottoming out at humans. They are working on experimenting with this approach by trying to get current day AIs to do useful supporting work such as [https://openai.com/blog/summarizing-books/ summarizing books] and [https://openai.com/blog/critiques/ criticizing itself].

OpenAI also published GPT-3, and are continuing to push LLM capabilities, with GPT-4 expected to be released at some point soon.

See also: [https://www.lesswrong.com/posts/3S4nyoNEEuvNsbXt8/common-misconceptions-about-openai Common misconceptions about OpenAI] and [https://openai.com/blog/our-approach-to-alignment-research/ Our approach to alignment research].

Question: How is OpenAI planning  solve the full alignment problem?

Answer:
The safety team at OpenAI's plan is to build a ［https://aligned.substack.com/p/alignment-mvp MVP aligned AGI］ to try and help us solve the full alignment problem.

They want to do this with Reinforcement Learning from Human Feedback (RLHF): get feedback from humans about what is good, i.e. give reward to AI's based on the human feedback. Problem: what if the AI makes gigabrain 5D chess moves that humans don't understand, so can't evaluate. Jan Leike, the director of the safety team, views this (［https://ai-alignment.com/the-informed-oversight-problem-1b51b4f66b35 the informed oversight problem］) as the core difficulty of alignment. Their proposed solution: an AI assisted oversight scheme, with a recursive hierarchy of AIs bottoming out at humans. They are working on experimenting with this approach by trying to get current day AIs to do useful supporting work such as ［https://openai.com/blog/summarizing-books/ summarizing books］ and ［https://openai.com/blog/critiques/ criticizing itself］.

OpenAI also published GPT-3, and are continuing to push LLM capabilities, with GPT-4 expected to be released at some point soon.

See also: ［https://www.lesswrong.com/posts/3S4nyoNEEuvNsbXt8/common-misconceptions-about-openai Common misconceptions about OpenAI］ and ［https://openai.com/blog/our-approach-to-alignment-research/ Our approach to alignment research］.

RoseMcClelland's Answer to How is OpenAI planning to solve the full alignment problem?