2022 Expert Survey on Progress in AI

Published 3 August 2022; last updated 3 August 2022
This page is in progress. It includes results that are preliminary and have a higher than usual chance of inaccuracy and suboptimal formatting. It is missing many results.
This page may be out-of-date. Visit the updated version of this page on our wiki.
The 2022 Expert Survey on Progress in AI (2022 ESPAI) is a survey of machine learning researchers that AI Impacts ran in June-August 2022.
Details
Background
The 2022 ESPAI is a rerun of the 2016 Expert Survey on Progress in AI that researchers at AI Impacts previously collaborated on with others. Almost all of the questions were identical, and both surveyed authors who recently published in NeurIPS and ICML, major machine learning conferences.
Zhang et al ran a followup survey in 2019 (published in 2022)1 however they reworded or altered many questions, including the definitions of HLMI, so much of their data is not directly comparable to that of the 2016 or 2022 surveys, especially in light of large potential for framing effects observed.
Methods
Population
We contacted approximately 4271 researchers who published at the conferences NeurIPS or ICML in 2021. These people were selected by taking all of the authors at those conferences and randomly allocating them between this survey and a survey being run by others. We then contacted those whose email addresses we could find. We found email addresses in papers published at those conferences, in other public data, and in records from our previous survey and Zhang et al 2022. We received 738 responses, some partial, for a 17% response rate.
Participants who previously participated in the the 2016 ESPAI or Zhang et al surveys received slightly longer surveys, and received questions which they had received in past surveys (where random subsets of questions were given), rather than receiving newly randomized questions. This was so that they could also be included in a ‘matched panel’ survey, in which we contacted all researchers who completed the 2016 ESPAI or Zhang et al surveys, to compare responses from exactly the same samples of researchers over time. These surveys contained additional questions matching some of those in the Zhang et al survey.
Contact
We invited the selected researchers to take the survey via email. We accepted responses between June 12 and August 3, 2022.
Questions
The full list of survey questions is available below, as exported from the survey software. The export does not preserve pagination, or data about survey flow. Participants received randomized subsets of these questions, so the survey each person received was much shorter than that shown below.
2022ESPAIVDownload
A small number of changes were made to questions since the 2016 survey (list forthcoming).
Definitions
‘HLMI’ was defined as follows:
The following questions ask about ‘high–level machine intelligence’ (HLMI). Say we have ‘high-level machine intelligence’ when unaided machines can accomplish every task better and more cheaply than human workers. Ignore aspects of tasks for which being a human is intrinsically advantageous, e.g. being accepted as a jury member. Think feasibility, not adoption.
Results
Data
The anonymized dataset is available here.
Summary of results

The aggregate forecast time to a 50% chance of HLMI was 37 years, i.e. 2059 (not including data from questions about the conceptually similar Full Automation of Labor, which in 2016 received much later estimates). This timeline has become about eight years shorter in the six years since 2016, when the aggregate prediction put 50% probability at 2061, i.e. 45 years out. Note that these estimates are conditional on “human scientific activity continu[ing] without major negative disruption.”
The median respondent believes the probability that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)” is 5%. This is the same as it was in 2016 (though Zhang et al 2022 found 2% in a similar but non-identical question). Many respondents were substantially more concerned: 48% of respondents gave at least 10% chance of an extremely bad outcome. But some much less concerned: 25% put it at 0%.
The median respondent believes society should prioritize AI safety research “more” than it is currently prioritized. Respondents chose from “much less,” “less,” “about the same,” “more,” and “much more.” 69% of respondents chose “more” or “much more,” up from 49% in 2016.
The median respondent thinks there is an “about even chance” that a stated argument for an intelligence explosion is broadly correct. 54% of respondents say the likelihood that it is correct is “about even,” “likely,” or “very likely” (corresponding to probability >40%), similar to 51% of respondents in 2016. The median respondent also believes machine intelligence will probably (60%) be “vastly better than humans at all professions” within 30 years of HLMI, and the rate of global technological improvement will probably (80%) dramatically increase (e.g., by a factor of ten) as a result of machine intelligence within 30 years of HLMI.

High-level machine intelligence timelines
The aggregate forecast time to HLMI was 36.6 years, conditional on “human scientific activity continu[ing] without major negative disruption.” and considering only questions using the HLMI definition. We have not yet analyzed data about the conceptually similar Full Automation of Labor (FAOL), which in 2016 prompted much later timeline estimates. Thus this timeline figure is expected to be low relative to an overall estimate from this survey.
This aggregate is the 50th percentile date in an equal mixture of probability distributions created by fitting a gamma distribution to each person’s answers to three questions either about the probability of HLMI occurring by a given year or the year at which a given probability would obtain.

Figure 1: Gamma distributions inferred for each individual.
Figure 2: Gamma distributions inferred for each individual, 2016 data
Impacts of HLMI
Question
Participants were asked:
Assume for the purpose of this question that HLMI will at some point exist. How positive or negative do you expect the overall impact of this to be on humanity, in the long run? Please answer by saying how probable you find the following kinds of impact, with probabilities adding to 100%:
______ Extremely good (e.g. rapid growth in human flourishing) (1)
______ On balance good (2)
______ More or less neutral (3)
______ On balance bad (4)
______ Extremely bad (e.g. human extinction) (5)
Answers
Medians:

Extremely good: 10%
On balance good: 20%
More or less neutral: 15%
On balance bad: 10%
Extremely bad: 5%

Means:

Extremely good: 24%
On balance good: 26%
More or less neutral: 18%
On balance bad: 17%
Extremely bad: 14%

Intelligence explosion
Probability of dramatic technological speedup
Question
Participants were asked:
Assume that HLMI will exist at some point. How likely do you then think it is that the rate of global technological improvement will dramatically increase (e.g. by a factor of ten) as a result of machine intelligence:
Within two years of that point?       ___% chance
Within thirty years of that point?    ___% chance
Answers
Median P(within two years) = 20% (20% in 2016)
Median P(within thirty years) = 80% (80% in 2016)
Probability of superintelligence
Question
Participants were asked:
Assume that HLMI will exist at some point. How likely do you think it is that there will be machine intelligence that is vastly better than humans at all professions (i.e. that is vastly more capable or vastly cheaper):
Within two years of that point?       ___% chance
Within thirty years of that point?    ___% chance
Answers
Median P(…within two years) = 10% (10% in 2016)
Median P(…within thirty years) = 60% (50% in 2016)
Chance that the intelligence explosion argument is about right
Question
Participants were asked:
Some people have argued the following:
If AI systems do nearly all research and development, improvements in AI will accelerate the pace of technological progress, including further progress in AI.
Over a short period (less than 5 years), this feedback loop could cause technological progress to become more than an order of magnitude faster.
How likely do you find this argument to be broadly correct?

Quite unlikely (0-20%)
Unlikely (21-40%)
About even chance (41-60%)
Likely (61-80%)
Quite likely (81-100%)

Answers

20% quite unlikely (25% in 2016)
26% unlikely (24% in 2016)
21% about even chance (22% in 2016)
26% likely (17% in 2016)
7% quite likely (12% in 2016)

Existential risk
In an above question, participants’ credence in “extremely bad” outcomes of HLMI have median 5% and mean 14%. To better clarify what participants mean by this, we also asked a subset of participants one of the following questions, which did not appear in the 2016 survey:
Extinction from AI
Participants were asked:
What probability do you put on future AI advances causing human extinction or similarly permanent and severe disempowerment of the human species?
Answers
Median 5%.
Extinction from human failure to control AI
Participants were asked:
What probability do you put on human inability to control future advanced AI systems causing human extinction or similarly permanent and severe disempowerment of the human species?
Answers
Median 10%.
This question is more specific and thus necessarily less probable than the previous question, but it was given a higher probability at the median. This could be due to noise (different random subsets of respondents received the questions, so there is no logical requirement that their answers cohere), or due to the representativeness heuristic.
Safety
General safety
Question
Participants were asked:
Let ‘AI safety research’ include any AI-related research that, rather than being primarily aimed at improving the capabilities of AI systems, is instead primarily aimed at minimizing potential risks of AI systems (beyond what is already accomplished for those goals by increasing AI system capabilities).
Examples of AI safety research might include:

Improving the human-interpretability of machine learning algorithms for the purpose of improving the safety and robustness of AI systems, not focused on improving AI capabilities
Research on long-term existential risks from AI systems
AI-specific formal verification research
Policy research about how to maximize the public benefits of AI

How much should society prioritize AI safety research, relative to how much it is currently prioritized?

Much less
Less
About the same
More
Much more

Answers

Much less: 2% (5% in 2016)
Less: 9% (8% in 2016)
About the same: 20% (38% in 2016)
More: 35% (35% in 2016)
Much more: 33% (14% in 2016)

69% of respondents think society should prioritize AI safety research more or much more, up from 49% in 2016.
Stuart Russell’s problem
Question
Participants were asked:
Stuart Russell summarizes an argument for why highly advanced AI might pose a risk as follows:
The primary concern [with highly advanced AI] is not spooky emergent consciousness but simply the ability to make high-quality decisions. Here, quality refers to the expected outcome utility of actions taken […]. Now we have a problem:
1. The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down.
2. Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task.
A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable.  This is essentially the old story of the genie in the lamp, or the sorcerer’s apprentice, or King Midas: you get exactly what you ask for, not what you want.
Do you think this argument points at an important problem?

No, not a real problem.
No, not an important problem.
Yes, a moderately important problem.
Yes, a very important problem.
Yes, among the most important problems in the field.

How valuable is it to work on this problem today, compared to other problems in AI?

Much less valuable
Less valuable
As valuable as other problems
More valuable
Much more valuable

How hard do you think this problem is compared to other problems in AI?

Much easier
Easier
As hard as other problems
Harder
Much harder

Answers
Importance:

No, not a real problem: 4%
No, not an important problem: 14%
Yes, a moderately important problem: 24%
Yes, a very important problem: 37%
Yes, among the most important problems in the field: 21%

Value today:

Much less valuable: 10%
Less valuable: 30%
As valuable as other problems: 33%
More valuable: 19%
Much more valuable: 8%

Hardness:

Much easier: 5%
Easier: 9%
As hard as other problems: 29%
Harder: 31%
Much harder: 26%

Contributions
The survey was run by Katja Grace and Ben Weinstein-Raun. Data analysis was done by Zach Stein-Perlman and Ben Weinstein-Raun. This page was written by Zach Stein-Perlman and Katja Grace.
We thank many colleagues and friends for help, discussion and encouragement, including John Salvatier, Nick Beckstead, Howie Lempel, Joe Carlsmith, Leopold Aschenbrenner, Ramana Kumar, Jimmy Rintjema, Jacob Hilton, Ajeya Cotra, Scott Siskind, Chana Messinger, Noemi Dreksler, and Baobao Zhang.
We also thank the expert participants who spent time sharing their impressions with us, including:
Michał ZającMorten GoodwinYue SunNingyuan ChenEgor KostylevRichard AntonelloElia TurnerAndrew C LiZachary MarkovichValentina ZantedeschiMichael CooperThomas A KellerMarc CavazzaRichard VidalDavid LindnerXuechen (Chen) LiAlex M. LambTristan Aumentado-ArmstrongFerdinando FiorettoAlain RossierWentao ZhangVarun JampaniDerek LimMuchen LiCong HaoYao-Yuan YangLinyi LiStéphane D’AscoliLang HuangMaxim KodryanHao BianOrestis ParaskevasDavid MadrasTommy TangLi SunStefano V AlbrechtTristan KarchMuhammad A RahmanRuntian ZhaiBenjamin BlackKaran SinghalLin GaoEthan BrooksCesar FerriDylan CampbellXujiang ZhaoJack Parker-HolderMichael NorrishJonathan UesatoYang AnMaheshakya WijewardenaUlrich NeumannLucile Ter-MinassianAlexander Matt TurnerSubhabrata DuttaYu-Xiang WangYao ZhangJoanna HongYao FuWenqing ZhengLouis C TiaoHajime AsamaChengchun ShiMoira R DillonYisong YueAurélien BelletYin CuiGang HuaJongheon JeongMartin KlissarovAran NayebiFabio Maria CarlucciChao MaSébastien GambsRasoul MirzaiezadehXudong ShenJulian SchrittwieserAdhyyan NarangFuxin LiLinxi FanJohannes GasteigerKarthik Abinav SankararamanPatrick MineaultAkhilesh GotmareJibang WuMikel LandajuelaJinglin LiuQinghua HuNoah SiegelAshkan KhakzarNathan GrinsztajnJulian LienenXiaoteng MaMohamad H DaneshKe ZHANGFeiyu XiongWonjae KimMichael ArbelPiotr SkowronLê-Nguyên HoangTravers RhodesLiu ZiyinHossein AzizpourKarl TuylsHangyu MaoYi MaJunyi LiYong ChengAditya BhaskaraXia LiDanijar HafnerBrian QuanzFangzhou LuoLuca CosmoScott FujimotoSantu RanaMichael CurryKarol HausmanLuyao YuanSamarth SinhaMatthew McLeodHao ShenNavid NaderializadehAlessio MicheliZhenbang YouVan Huy VoChenyang WuThanard KurutachVincent ConitzerChuang GanChirag GuptaAndreas SchlaginhaufenRuben OhanaLuming LiangMarco FumeroPaul MullerHana ChocklerMing ZhongJiamou LiuSumeet AgarwalEric WinsorRuimeng HuChangjian ShuiYiwei WangJoey Tianyi ZhouAnthony L. CateriniGuillermo Ortiz-JimenezIou-Jen LiuJiaming LiuMichael PerlmutterAnurag ArnabZiwei XuJohn Co-ReyesAravind RajeswaranRoy FoxYong-Lu LiCarl YangDivyansh GargAmit DhurandharHarris ChanTobias SchmidtRobi BhattacharjeeMarco NadaiReid McIlroy-YoungWooseok HaJesse MuNeale RatzlaffKenneth BorupBinghong ChenVikas VermaWalter GerychShachar LovettZhengyu ZhaoChandramouli ChandrasekaranRichard HigginsNicholas RhinehartBlaise Agüera Y ArcasSantiago Zanella-BeguelinDian JinScott NiekumColin A. RaffelSebastian GoldtYali DuBernardo SubercaseauxHui WuVincent MalletOzan ÖzdenizciTimothy HospedalesLingjiong ZhuCheng Soon OngShahab BakhtiariHuan ZhangBanghua ZhuByungjun LeeZhenyu LiaoAdrien EcoffetVinay RamaseshJesse ZhangSoumik SarkarNandan Kumar JhaDaniel S BrownNeev ParikhChen-Yu WeiDavid K. DuvenaudFelix PetersenSonghua WuHuazhu FuRoger B GrosseMatteo PapiniPeter KairouzBurak VariciFabio RoliMohammad Zalbagi DarestaniJiamin HeLys Sanz MoretaXu-Hui LiuQianchuan ZhaoYulia GelJan DrgonaSajad KhodadadianTakeshi TeshimaIgor T PodolakNaoya TakeishiMan Shun AngMingli SongJakub TomczakLukasz SzpruchMicah GoldblumGraham W. TaylorTomasz KorbakMaheswaran SathiamoorthyLan-Zhe GuoSimone FioravantiLei JiaoDavin ChooKristy ChoiVarun NairRayana JaafarAmy GreenwaldMartin V. ButzAleksey TikhonovSamuel GruffazYash SavaniRui ChenKe Sun
We thank FTX Future Fund for funding this project.
Suggested citation
Zach Stein-Perlman, Benjamin Weinstein-Raun, Katja Grace, “2022 Expert Survey on Progress in AI.” AI Impacts, 3 Aug. 2022. https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/.
Notes

2022 Expert Survey on Progress in AI


Why work at AI Impacts?

Katja Grace, 6 March 2022
AI Impacts is beginning a serious hiring round (see here for job postings), so I’d like to explain a bit why it has been my own best guess at the highest impact place for me to work for me. (As in, this is a personal blog post by Katja on the AI Impacts blog, not some kind of officialesque missive from the organization.)
But first—
What is AI Impacts?
AI Impacts is a few things:

An online library of best-guess answers to questions about the future of AI. Including big questions, like ‘how likely is a sudden jump in AI progress at around human-level performance?’, and sub-questions informing those answers (‘are discontinuities common in technological trends?’), and sub-sub questions (‘did penicillin cause any discontinuous changes in syphilis trends?’), and so on. Each page ideally has a high-level conclusion at the top, and reasoning supporting it below, which will often call on the conclusions of other pages. These form something like a set of trees, with important, hard, decision-relevant questions at the root and low-level, tractable, harder-to-use-on-their-own questions at the leaves. This isn’t super obvious at the moment, because a lot of the trees are very incomplete, but that’s the basic idea.
A research group focused on finding such answers, through a mixture of original research and gathering up that which has been researched by others.
A blog on these topics, for more opinionated takes, conversational guides to the research, updates, and other things that don’t fit in the main library (like this!).
A locus of events for people interested in this kind of research, e.g. dinners and workshops, a Slack with other researchers, online coffees.

Why think working on AI Impacts is among the best things to do?
1. AI risk looks like a top-notch cause area
It seems plausible to me that advanced AI poses a substantial risk to humanity’s survival. I don’t think this is clear, but I do think there’s enough evidence that it warrants a lot of attention. I hope to write more about this, see here for recent discussion. Furthermore, I don’t know of other similarly serious risks (see Ord’s The Precipice for a review), or of other intervention areas that look clearly more valuable than reducing existential risk to humanity.
I actually also think AI risk is a potentially high-impact area to work (for a little while at least) if AI isn’t a huge existential risk to humanity, because so many capable and well-intentioned people are dedicating themselves to it. Demonstrating that it wasn’t that bad could redirect mountains of valuable effort to real problems.
2. Understanding the situation beats intervening on the current margin
Within the area of mitigating AI risk, there are several broad classes of action being taken. Technical safety research focuses on building AI that won’t automatically cause catastrophe. AI Governance focuses on maneuvering the policy landscape to lower risk. These are both kinds of intervention: ‘intervening’ is a meta-category, and the other main meta-category in my mind is ‘understanding the situation’. My own best guess is that on the current margin, ‘understanding the situation’ is a better place for an additional person with general skills than any particular intervening that I know of. (Or maybe it’s only almost as good—I flip-flop, but it doesn’t really matter much: the important thing is that for some large part of the space of people and their skills and characteristics, it seems better.)
By ‘understanding the situation’, I mean for instance working toward better answers to questions like these:

Fast or slow takeoff?
What concrete kinds of problems might destroy humanity? E.g. single AI god intentionally murders everyone with nanotech vs. large economy gradually drifts away from human comprehension or control?
Is there a single relevant ‘deployment’?
If so, what does it look like?
Would we be safe if AI systems weren’t ‘agentic’?
Do not-intentionally-agentic things readily become agentic things? Under what circumstances?
How fast would an intelligence explosion go?
Is it possible to describe a plausible future where things go well? (Is it possible to describe a plausible future where things go badly?)

Carrying out any particular intervention also involves a lot of ‘understanding the situation’, but I think this is often at a different level. For instance, if you decide to intervene by trying to get AI labs to collaborate with each other, you might end up accruing better models of how people at AI projects interact socially, how decisions are made, how running events works, and so on, because these things are part of the landscape between you and your instrumental goal: improving collaboration between AI projects. You probably also learn about things around you, like what kinds of AI projects people are doing. But you don’t get to learn much at all about how the achievement of your goal affects the future of AI. (I fear that in general this situation means you can end up lumbering forward blindly while thinking you can see, because you are full of specific concrete information—the intricacies of the steering wheel distracting you from the dense fog on the road.) There are some exceptions to this. For instance, I expect some technical work to be pretty enlightening about the nature of AI systems, which is directly relevant to how the development of better AI systems will play out. For instance, mesa-optimization seems like a great contribution to ‘understanding the situation’ which came out of a broadly intervention-oriented organization.
It is that kind of understanding the situation—understanding what will happen with AI and its effects on society, under different interventions—that I think deserves way more attention.
Why do I think understanding the situation is better than intervening? Of course in general, both are great. Intervening is generally necessary for achieving anything, and understanding the situation is arguably necessary for intervening well. (The intense usefulness of understanding the situation for achieving your goals in most situations is exactly the reason one might be concerned about AI to begin with.) So in general, you want a combination of understanding the situation and intervening. The question is how valuable the two are on the current margin.
My guess: understanding the situation is better. Which is to say, I think a person with a subjectively similar level of skill at everything under consideration will add more value via improving everyone’s understanding of the situation by one person’s worth of effort than they would by adding one person’s worth of effort to pursuing the seemingly best intervention.
Here are a few things influencing this guess:

A basic sense that our understanding of the situation is low My impression when talking to people working on AI risk is that they often don’t feel that they understand the situation very well. There are major disagreements about what kind of basic scenario we are expecting. The going explanations for why there will be human extinction at all seem to vary across time and between people. Offers to try to clarify are generally met with enthusiasm. These things don’t seem great as signs about whether we understand the situation well enough to take useful action.
It’s easy to think of specific questions for which updated answers would change the value of different interventions. Here are a few examples off the top of my head of questions, answers, and strategies that would seem to be relatively favored by those answers:

Does AI pose a substantial risk of human extinction?
Yes: work on AI risk instead of other EA causes and other non-emergency professions. Show a good case for this to large numbers of people who aren’t thinking about it and try to change views within the AI community and public about the appropriate degree of caution for relevant AI work.
No: work on something more valuable, support AI progress
When will relevantly advanced AI be developed?
5 years: plan for what specific actors should do in a situation much like our current one and talk to them about doing it; build relationships with likely actors; try to align systems much like our current AI systems.
20 years: more basic time-consuming alignment research; movement building; relationship building with institutions rather than people.
100 years: avert risks from narrow or weak AI and other nearer technologies, even more basic alignment research, improve society’s general institutions for responding to risks like this, movement building directed at broader issues that people won’t get disillusioned with over that long a period (e.g. ‘responding to technological risks’ vs. AI specifically).
How fast is the progression to superhumanly powerful AI likely to be?
Before you know it: searching for technical solutions that can be proven to entirely solve the problem before it arises (even if you are unlikely to find any), social coordination to avoid setting off such an event.
Weeks: Immediate-response contingency plans.
Years: Fast-response contingency plans; alignment plans that would require some scope for iteration.
Decades: Expect to improve safety through more normal methods of building systems, observing them, correcting, iterating. ‘Soft’ forces like regulations, broadscale understanding of the problems, cooperation initiatives. Systems that are incrementally safer but not infinitely safer.

Broad heuristic value of seeing
When approaching a poorly understood danger down a dark corridor, I feel like even a small amount of light is really good. Good for judging whether you are facing a dragon or a cliff, good for knowing when you are getting close to it so you can ready your sword (or your ropes, as the case may be), good for telling how big it is. But even beyond those pre-askable questions, I expect the details of the fight (or climb) to go much better if you aren’t blind. You will be able to strike well, and jump out of the way well, and generally have good feedback about your micro-actions and local risks.So I don’t actually trust tallying up possible decision changes as in the last point, that much. If you told me that we had reasoned through the correct course of action for dragons, and cliff faces, and tar pits, and alternate likely monsters, and decided they were basically the same, I’d persist in being willing to pay a lot to be able to see.
Applied to AI strategy: understanding the situation both lets you choose interventions that might help, and having chosen an intervention, probably helps you make smaller choices within that intervention well, such that the intervention hits its target.
I think another part of the value here is that very abstract reasoning about complicated situations seems untrustworthy (especially when it isn’t actually formal), and I expect getting more data and working out more details to generally engage people’s concrete thinking better, and for that to be helpful.
Large multipliers available It’s not that hard to imagine the work of one person’s year substantially redirecting way more than one person-year worth of time or money. Intuitively the chance of this seems high enough to make it a good prospect.
We have a really long list of projects to do. About a hundred that we have bothered to write down, though they vary in tractability. It isn’t hard to find important topics which have received little thorough research. On the current margin, it looks to me like an additional competent person can expect to do useful research.
If I were to work on a direct intervention in this space, I would feel fairly unsure about whether it would be helpful even if it succeeded in its goals.
Understanding the situation has way fewer people than intervening: I haven’t measured this carefully, but my guess is that between ten and a hundred times as much labor goes into intervening than understanding the situation. I’m not sure what the division should be, but intuitively this seems too lopsided.
Assumptions do not seem solid: it’s arguably not very hard to find points that people are bringing to the table that, upon empirical investigation, seem contrary to the evidence. Einstein and idiots are probably not really right next to each other on natural objective measures of intelligence, as far as I can tell. Qualitatively cool technologies do not generally cause large discontinuities in any particular metrics. Not empirical, but many of the arguments I’ve heard for expecting discontinuous progress at around the time of human-level AI just don’t make much sense to me.
The ‘understanding the situation’ project is at a pretty unsophisticated stage, compared with intervening projects, according to my assessment anyway. That suggests a mistake, in the same way that navigating an expensive car using divining rods because you don’t have a GPS or map suggests some kind of misallocation of investments.
I think people overestimate the effort put into understanding the situation, because there is a decent amount of talking about it at parties and blogging about it.
There are people positioned to make influential choices if they knew what to do asking for help in assessing the situation (e.g. Holden of Open Phil, people with policy influence, philanthropists).

People sometimes ask if we might be scraping the barrel on finding research to do in this space, I guess because quite a few people have prolifically opined on it over numerous years, and things seem pretty uncertain. I think that radically under-imagines what understanding, or an effort dedicated to understanding, could look like. Like, we haven’t gotten as far as making sure that the empirical claims being opined about are solid, whereas a suitable investment for a major international problem that you seriously need to solve should probably look more like the one we see for climate change. Climate change is a less bad and arguably easier to understand problem than AI risk, and the ‘understanding the situation’ effort there looks like an army of climate scientists working for decades. And they didn’t throw up their hands and say things were too uncertain and they had run out of things to think about after twenty climate hobbyists had thought about it for a bit. There is a big difference between a vibrant corner of the blogosphere and a serious research effort.
3. Different merits of different projects
Ok, so AI risk is the most impactful field to my knowledge, and within AI risk I claim that the highest impact work is on understanding the situation1. This is reason to work at AI Impacts, and also reason to work at Open Philanthropy, FHI, Metaculus, as an independent scholar, in academia, etc. Probably who should do which depends on the person and their situation. Here are some things AI Impacts is about, and axes on which we have locations:

Openness, broad comprehensibility and reasoning transparency: our goal is to make an online repository of reasoning around these topics, so we prioritize publishing work (vs. distributing it privately to smaller networks of people), and legibility. There can be research that is better done privately, but such research is not our project. We hope to describe the basis for our conclusions well enough that a non-expert reader can verify the reasoning.
Modularity and question decomposition: AI Impacts is intended to be something like a group of hierarchical trees of modular conclusions, that can be referred to and questioned in a relatively clean way. We try to roughly have a page for each important conclusion, though things get complicated sometimes, and it is easier to have a short list of them. I think this kind of structure for understanding a complex topic is a promising one, relative to for instance less structured piles of prose. I expect this to make research more re-purposeable, clear, updateable, navigable, and amenable to tight feedback loops. Echoing this structure, we try to answer big questions by breaking them into smaller questions, until we have tractable questions.
Eye on the prize vs. exploratory wandering: there are many research questions that are interesting and broadly shed light on the future of AI, and following one’s curiosity can be a good strategy. Nonetheless we especially strive to answer the questions that more help with answering important high-level questions. While researchers have a decent amount of freedom, we expect people to be contributing to filling in the gaps in this shared structure of understanding that we are building.
Back of the envelopes expanding into arbitrarily detailed investigation: in places like academia, it seems normal to work on a project for many months or years, and to finish with something polished. Part of the idea with AI Impacts is to look out for questions that can be substantially clarified by a day and a back of the envelope calculation, to not put in more research than needed, and to iterate at more depth when relevant. This is hard to get right, and we usually fail at this so far, with investigations often expanding to be large clusters of pages before any go up. But as far as I’m concerned, long projects are a failure mode, not a goal.
Adding concrete reusable things to the conversation, which can be called on in other discussions. This means prioritizing things like empirical investigations that add new data, or cleanly stated considerations, rather than long vague or hard-to-disentangle discussions, or conclusions whose use requires trusting the author a lot.
Generalist research and widely ranging projects vs. developed expertise. I’m not an expert on anything, as far as I know. Some things my work has involved: thinking about the origin of humans, examining records of 1700s cotton exports, designing incentives for survey participants, reasoning about computer hardware designs, corresponding incredulously with makers of computing benchmarks, skimming papers about the energy efficiency of albatrosses. We do have relative specializations (I do more philosophy, Rick does more empirical work), and would welcome more relevant expertise, but this work can be pretty wide ranging.
Trustworthiness as an unbiased source vs. persuasion. We focus on questions where we are genuinely unsure of the answer (though we might expect that data will reveal our own current guess is correct), and try to write neutrally about the considerations that we think merit attention. We are unlikely to look for the best way to ‘convince people of AI risk’, but rather to set out to establish whether or not there is AI risk, and to document our reasoning clearly.
Thriving emphasis vs. high-pressure productivity orientation. We sit toward the thriving end of this spectrum, and hope that pays off in terms of longer term productivity. We are relatively accommodating to idiosyncratic needs or preferences. Our work requires less temporal consistency or predictability than some jobs, so while we value seeing each other regularly and getting stuff done often, we are able to be flexible if someone has things to contribute, but difficulties with the standard office situation.

I’m focused here on the positives, but here are a few negatives too:

Variable office situation: through a series of unfortunate and fortunate events which is getting ridiculous, we haven’t had a consistent shared office in years. At present, we have an office in SF but Rick works from the bigger Rationalist/EA offices in Berkeley.
Small: currently two full-time people, plus various occasional people and socially around people. Working from Berkeley next to other AI risk orgs mitigates this some. Has been as many as seven people in a summer, which seemed better, and we hope to move back to at least four soon.
Even the relatively easy work is hard in ways: everything is complicated and even if you set out to do the most basic analysis ever there seems to be a strong current pulling toward getting bogged down in details of details. This is not the kind of ‘hard’ where you need to be a genius, but rather where you can easily end up taking much longer than hoped, and also get discouraged, which doesn’t help with speed. We’re still figuring out how to navigate this while being epistemically careful enough to produce good information.

So, that was a hand-wavy account of why I think working at AI Impacts is particularly high impact, and some of what it’s like. If you might want to work for us, see our jobs page2. If you don’t, but like thinking about the future of AI and wish we invited you to dinners, coffees, parties or our Slack, drop me a DM or send us a message through the AI Impacts feedback box. Pitches that I’m wrong and should do something else are also welcome.

Notes

You might ask, why do I call this thing ‘understanding the situation’, not ‘AI strategy’? Isn’t that both what other people call this, and a less ridiculous name? Strategy, to my mind, is about deciding what to do. Strategy is like engineering. But there is a whole earlier activity that is science: just figuring out what the world is like. I think it’s worth separating science as an activity rather than bundling with engineering because if you think of yourself as doing ‘AI strategy’, I think it’s easy to skip over a lot of science and make too many assumptions – for instance to jump into asking how much to disvalue the ‘other side’ winning an AI arms race and their values controlling the future before being super clear on whether it makes sense to think of your situation as an arms race, or on whether countries are the relevant units, or on how a nation developing certain AI technology earlier relates to the government of that nation implementing ‘their values’. Which is maybe fine if you are in enough of a hurry, but is not what I’m advocating for. I often call the AI futurism version of science ‘AI forecasting’, but my guess is that that brings to mind a wrongly narrow picture—making quantitative predictions about the date of AI milestones. So I’d like a better word, but don’t have one right now. I realize it would be strategic to have a better word before really advocating for the importance of a thing, but you win some you lose some. Curious if others agree with these connotations.
If you already applied when we weren’t actively hiring and we didn’t get back to you: sorry, feel free to try again, we now have better ops capacity, and hope to soon make it better still.

Why work at AI Impacts?


Observed patterns around major technological advancements

by Rick Korzekwa, 2 February, 2022
Summary
In this post I outline apparent regularities in how major new technological capabilities and methods come about. I have not rigorously checked to see how broadly they hold, but it seems likely to me that they generalize enough to be useful. For each pattern, I give examples, then some relatively speculative explanations for why they happen. I finish the post with an outline of how AI progress will look if it sticks to these patterns. The patterns are:
The first version of a new technology is almost always of very low practical valueAfter the first version comes out, things improve very quicklyMany big, famous, impressive, and important advances are preceded by lesser-known, but still notable advances. These lesser-known advances often follow a long period of stagnationMajor advances tend to emerge while relevant experts are still in broad disagreement about which things are likely to work
Introduction
While investigating major technological advancements, I have noticed patterns among the technologies that have been brought to our attention at AI Impacts. I have not rigorously investigated these patterns to see if they generalize well, but I think it is worthwhile to write about them for a few reasons. First, I hope that others can share examples, counterexamples, or intuitions that will help make it clearer whether these patterns are “real” and whether they generalize. Second, supposing they do generalize, these patterns might help us develop a view of the default way in which major technological advancements happen. AI is unlikely to conform to this default, but it should help inform our priors and serve as a reference for how AI differs from other technologies. Finally, I think there is value in discussing and creating common knowledge around which aspects of past technological progress are most informative and decision-relevant to mitigating AI risk.
What I mean by ‘major technological advancements’
Throughout this post I am referring to a somewhat narrow class of advances in technological progress. I won’t try to rigorously define the boundaries for classification, but the kinds of technologies I’m writing about are associated with important new capabilities or clear turning points in how civilization solves problems. This includes, among many others, flight, the telegraph, nuclear weapons, the laser, penicillin, and the transistor. It does not include achievements that are not primarily driven by improvements in our technological capabilities. For example, many of the structure height or ship size advances included in our discontinuities investigation seem to have occurred mainly because someone had the means and desire to build a large ship or structure and the materials and methods were available, not because of a substantial change in known methods or available materials. It also does not include technologies granting impressive new capabilities that have not yet been useful outside of scientific research, such as gravitational wave interferometers.
I have tried to be consistent with terminology, but I have probably been a bit sloppy in places. I use ‘advancement’ to mean any discrete event in which a new capability is demonstrated or a device is created that demonstrates a new principle of operation. In some places, I use ‘achievement’ to mean things like flying across the ocean for the first time, and ‘new method’ to mean things like using nuclear reactions to release energy instead of chemical reactions.
Things to keep in mind while reading this:
The primary purpose of this post is to present claims, not to justify them. I give examples and reasons why we might expect the pattern to exist, but my goal is not to make the strongest case that the patterns are real.I make these claims based on my overall impression of how things go, having spent part of the past few years doing research related to technological progress. I have done some searching for counterexamples, but I have not made a serious effort to analyze them statistically.The proposed explanations for why the patterns exist (if they do indeed exist) are largely speculative, though they are partially based on well-accepted phenomena like experience curves.There are some obvious sources of bias in the sample of technologies I’ve looked at. They’re mostly things that started out as candidates for discontinuities, which were crowd sourced and then examined in a way that strongly favored giving attention to advancements with clear metrics for progress.I’m not sure how to precisely offer my credence on these claims, but I estimate roughly 50% that at least one is mostly wrong. My best guess for how I will update on new information is by narrowing the relevant reference class of advancements.Although I did notice these patterns at least somewhat on my own, I do not claim these are original to me. I have seen related things in a variety of places, particularly the book Patterns of Technological Innovation by Devendra Sahal.
The patterns
The first version of a new device or capability is almost always terrible
The first transatlantic flight was a success, but just barely.
Most of the time when a major milestone is crossed, it is done in a minimal way that is so bad it has little or no practical value. Examples include:
The first ship to cross the Atlantic using steam power was slower than a fast ship with sails.The first transatlantic telegraph cable took an entire day to send a message the length of a couple Tweets, would stop working for long periods of time, and failed completely after three weeks.The first flight across the Atlantic took the shortest reasonable path and crash-landed in Ireland.The first apparatus for purifying penicillin did not produce enough to save the first patient.The first laser did not produce a bright enough spot to photograph for publication
Although I can’t say I predicted this would be the case before learning about it, in retrospect I do not think it is surprising. As we work toward an invention or achievement, we improve things along a limited number of axes with each design modification, oftentimes at the expense of other axes. In the case of improving an existing technology until it is fit for some new task, this means optimizing heavily on whichever axes are most important for accomplishing that task. For example, to modify a World War I bomber to cross the ocean, you need to optimize heavily for range, even if it requires giving up other features of practical value, like the ability to carry ordinance. Increasing range without making such tradeoffs makes the problem much harder. Not only does this make it less likely you will beat competitors to the achievement, it means you need to do design work in a way that is less iterative and requires more inference.
This also applies to building a device that demonstrates a new principle of operation, like the laser or transistor. The easiest path to making the device work at all will, by default, ignore many unnecessary practical considerations. Skipping straight past the dinky minimal version to the practical version is difficult, not only because designing something is harder with more constraints, but because we can learn a lot from the dinky minimal version on our way to the more practical version. This appears related to why Theodore Maiman won the race to build a laser. His competitors were pursuing designs that had clearer practical value but were substantially more difficult to get right.
The clearest example I’m aware of for a technology that does not fit this pattern is the first nuclear weapon. The need to create a weapon that was useful in the real world quickly and in secret using extremely scarce materials drove the scientists and engineers of the Manhattan Project to solve many practical problems before building and testing a device that demonstrated the basic principles. Absent these constraints, it would likely have been easier to build at least one test device along the way, which would most likely have been useless as a weapon.
Progress often happens more quickly following a major advancement
In less than ten years, nuclear weapons advanced from large devices with a yield of 25 kilotons (left) to artillery shells with a yield of 15 kilotons (top right) and large 25 megaton devices (lower right)
New technologies may start out terrible, but they tend to improve quickly. This isn’t just a matter of applying a constant rate of fractional improvement per year to an increase in performance—the doubling time for performance decreases. Examples:
Average speed to cross the Atlantic in wind-powered ships doubled over roughly 300 years, after which crossing speed for steam-powered ships doubled in about 50 years, and crossing speed for aircraft doubled twice in less than 40 years.Telecommunications performance, measured as the product of bandwidth and distance, doubled every 6 years before the introduction of fiber optics and doubled every 2 years after1The energy released per mass of conventional explosive increased by 20% during the 100 years leading up to the first nuclear weapon in 1945, and by 1960 there were devices with 1000x the energy density of the first nuclear weapon.The highest intensity available from artificial light sources had an average doubling time of about 10 years from 1800 to 1945. Following the invention of the laser in 1960, the doubling time averaged just six months for over 15 years. Lasers also found practical applications very quickly, with successful laser eye surgery within 18 months and the laser rangefinders a few years after that.
The rate of progress in carrying a military payload across the Atlantic increased each time the method changed
This pattern is less robust than the others. An obvious counterexample is Moore’s Law and related trajectories for progress in computer hardware, which remain remarkably smooth across major changes in underlying technology. Still, even if major technological advances do not always accelerate progress, they seem to be one of the major causes of accelerated progress.
Short-term increases in the rate of progress may be explained by the wealth of possible improvements to a new technology. The first version will almost always neglect numerous improvements that are easy to find. Furthermore, once a the technology is out in the world, it becomes much easier to learn how to improve it. This can be seen in experience curves for manufacturing, which typically show large improvements per unit produced in manufacturing time and cost when a new technology is introduced.
I’m not sure if this can explain long-running increases in the rate of progress, such as those for telecommunications and transatlantic travel by steamship. Maybe these technologies benefited from a large space of design improvements and sufficient, sustained growth in adoption to push down the experience curve quickly2. It may be related to external factors, like the advent of steam power increasing general economic and technological progress enough to maintain the strong trend in advancements for crossing the ocean.
Major advancements are usually preceded by rapid or accelerating progress

Searchlights during World War II used arc lamps, which were, as far as I know, the first artificial sources of light with greater intensity than concentrated sunlight. They were surpassed by explosively-driven flashes during the Manhattan Project.

Major new technological advances are often preceded by lesser-known, but substantial advancements. While we were investigating candidate technologies for the discontinuities project, I was often surprised to find that this preceding progress was rapid enough to eliminate or drastically reduce the calculated size of the candidate discontinuity. For example:
The invention of the laser in 1960 was preceded by other sources of intense light, including the argon flash in the early 1940s and various electrical discharge lamps by the 1930s. Progress was fairly slow from ~1800 to the 1930s, and more-or-less nonexistent for centuries before that.Penicillin was discovered during a period of major improvements in public health and treatments for bacterial infections, including a drug that was such an improvement over existing treatments for infection it was called the “magic bullet”.The Haber Process for fixing nitrogen, often given substantial credit for improvements in farming output that enabled the huge population growth of the past century, was invented in 1920. It was preceded by the invention of two other processes, in 1910 and 1912, the latter of which was more than a factor of two improvement over the former.The telegraph and flight both crossed the Atlantic for the first time during a roughly 100 year period of rapid progress in steamships, during which crossing times decreased by a factor of four.
Progress in light intensity for artificial sources exploded in the mid-1900s. Most progress is from improvements in lasers, but the Manhattan Project led to the development of explosively-driven flashes 15-20 years prior.Progress in intense light sources since 1800
This pattern seems less robust than the previous one and it is, to me, less striking. The accelerated progress preceding a major advancement varies widely in terms of overall impressiveness, and there is not a clear cutoff for what should qualify as fitting the pattern. Still, when I to come up with a clear example of something that fails to match the pattern, I had some difficulty. Even nuclear weapons were preceded by at least one notable advance in conventional explosives a few years earlier, following what seems to have been decades of relatively sluggish progress.
An obvious and perhaps boring explanation for this is that progress on many metrics was accelerating during the roughly 200-300 year period when most of these took place. Technological progress exploded between 1700 and 2000, in a way that seems to have been accelerating pretty rapidly until around 1950. Every point on an aggressively accelerating performance trend follows a period of unprecedented progress. It is plausible to me that this fully explains the pattern, but I’m not entirely convinced.
An additional contributor may be that the drivers for research and innovation that result in major advances tend to cause other advancements along the way. For example, wars increase interest in explosives, so maybe it is not surprising that nuclear weapons were developed around the same time as some advances in conventional explosives. Another potential driver for such innovation is a disruption to incremental progress via existing methods that requires exploration of a broader solution space. For example, the invention of the argon flash would not have been necessary if it had been possible to improve the light output of arc lamps.
Major advancements are produced by uncertain scientific communities
Alexander Fleming (left foreground) receives his Nobel Prize for his discovery of penicillin, along with Howard Florey (back left) and Ernst Chain (back, second from left). Florey and Chain pursued penicillin as an injectable therapeutic, in spite of Fleming’s insistence that it was only suitable for treating surface infections.
Nearly every time I look into the details of the work that went into producing a major technological advancement, I find that the relevant experts had substantial disagreements about which methods were likely to work, right up until the advancement takes place, sometimes about fairly basic things. For example:
The team responsible for designing and operating the first transatlantic telegraph cable had disagreements about basic properties of signal propagation on very long cables, which eventually led to the cable’s failure.The scientists of the Manhattan Project had widely varying estimates for the weapon’s yield, including scientists who expected the device to fail entirely.Widespread skepticism, misunderstanding, and disagreement of penicillin’s chemical and therapeutic properties were largely responsible for a 10 year delay in its development.Few researchers involved in the race to build a functioning laser expected the design that ultimately prevailed to be viable.
The default explanation for this seems pretty clear to me—empirical work is an important part of clearing up scientific uncertainty, so we should expect successful demonstrations of, novel, impactful technologies to eliminate a lot of uncertainty. Still, it did not have to be the case that the eliminated uncertainty is substantial and related to basic facts about the functioning of the technology. For example, the teams competing for the first flight across the Atlantic did not seem to have anything major to disagree about, though they may have made different predictions about the success of various designs.
There is a distinction to be made here, between high levels of uncertainty on the one hand, and low levels of understanding on the other. The communities of researchers and engineers involved in these projects may have collectively assigned substantial credence to various mistaken ideas about how things work, but my impression is that they did at least agree on the basic outlines of the problems they were trying to solve. They mostly knew what they didn’t know. For example, most of the disagreement about penicillin was about the details of things like its solubility, stability, and therapeutic efficacy. There wasn’t a disagreement about, for example, whether the penicillium fungus was inhibiting bacterial growth by producing an antibacterial substance or by some other mechanism. I’m not sure how to operationalize this distinction more precisely.
Relevance to AI risk
As I explained at the beginning of this post, one of my goals is to develop a typical case for the development of major technological advancements, so we can inform our priors about AI progress and think about the ways in which it may or may not differ from past progress. I have my own views on the ways in which it is likely to differ, but I don’t want to get into them here. To that end, here is what I expect AI progress will look like if it fits the patterns of past progress.
Major new methods or capabilities for AI will be demonstrated in systems that are generally pretty poor.Under the right conditions, such as a multi-billion-dollar effort by a state actor, the first version of an important new AI capability or method may be sufficiently advanced to be a major global risk or of very large strategic value.An early system with poor practical performance is likely to be followed by very rapid progress toward a system that is more valuable or dangerousProgress leading up to an important new method or capability in AI is more likely to be accelerating than it is to be stagnant. Notable advances preceding a new capability may not be direct ancestors to it.Although high-risk and transformative AI capabilities are likely to emerge in an environment of less uncertainty than today, the feasibility of such capabilities and which methods can produce them are likely to be contentious issues within the AI community right up until those capabilities are demonstrated.
Thanks to Katja Grace, Daniel Kokotajlo, Asya Bergal, and others for the mountains of data and analysis that most of this article is based on. Thanks to Jalex Stark, Eli Tyre, and Richard Ngo for their helpful comments. All views and mistakes are my own.
All charts in this post are original to AI Impacts. All photographs are in the public domain.


Do neural networks learn human concepts?

This page is a stub. It does not necessarily represent much of what is known on the topic.
Our understanding is that the degree to which neural networks learn concepts that are potentially understandable to humans is an open question.
Details
A very incomplete list of sources on the topic:
Acquisition of Chess Knowledge in AlphaZero (McGrath et al, 2021)1From the paper: ‘…In this work we provide evidence that human knowledge is acquired by the AlphaZero neural network as it trains on the game of chess. By probing for a broad range of human chess concepts we show when and where these concepts are represented in the AlphaZero network….’ Zoom in: An Introduction to Circuits (Olah et al, 2020)2From the paper: ‘In contrast to the typical picture of neural networks as a black box, we’ve been surprised how approachable the network is on this scale. Not only do neurons seem understandable (even ones that initially seemed inscrutable), but the “circuits” of connections between them seem to be meaningful algorithms corresponding to facts about the world. You can watch a circle detector be assembled from curves. You can see a dog head be assembled from eyes, snout, fur and tongue. You can observe how a car is composed from wheels and windows. You can even find circuits implementing simple logic: cases where the network implements AND, OR or XOR over high-level visual features.’
Notes
Featured image: from Olah, et al., “Zoom In: An Introduction to Circuits”, Distill, 2020., CC-BY 4.0


AI Vignettes Project

Posted Oct 12 2021
The AI Vignettes Project is an ongoing effort to write concrete plausible future histories of AI development and its social impacts.
Details
Purposes
We hope to:
Check that abstract views about the future of AI have plausible concrete instantiations. (Especially, hypothesized extinction scenarios, and proposed safe scenarios.)Develop better intuitions about possible scenarios by thinking through them concretely.Notice recurring themes in concrete stories, that may be worth thinking about more broadly.Fill out the space of plausible feasible scenarios with concrete illustrations, to decrease bias in thinking about the future.Keep a collection of AI vignettes for others to use, in the above or other ways.
Methods
Our current intended method is:
Write draft vignettes, with no particular systematic method for choosing an unbiased selection of scenariosRequest comments on their realismModify according to commentsRepeat 2-3 until realism critiques subside
Work so far
AI Impacts has run two small workshops where participants wrote AI vignettes.
Vignette collection
This is a subset of vignettes arising from this project, or similar.

AI Vignettes Project


Argument for AI x-risk from large impacts

Last updated Sep 29 2021
This page is incomplete, under active work and may be updated soon.
The argument for AI x-risk from large impacts an argument supporting the claim that advanced artificial intelligence poses an existential risk to humanity.
Details
Argument
We understand the argument as follows:
The creation of advanced AI is likely to have a large impact on the world at the time, relative to other developments.The creation of advanced AI is more likely to impact humanity’s long term trajectory than other developments (from 1)Developments with large impacts on the future are more likely to be worth influencing, all things equal, than developments with smaller impactsOne main way things might not be equal is that some developments may be easier to influence than others, but AI currently looks tractable to influence.The creation of advanced AI is especially likely to be worth trying to influence, among other developments
Counterarguments
This is a neither exhaustive nor exclusive list of counterarguments and complications that we know of:
This is a heuristic argument that isn’t strictly true, and it isn’t clear how true it is.It isn’t clear what a development is. For instance, is global agriculture surviving for another year a development? Is ‘modern science’ a development? Why think of advanced AI as a development, rather than thinking of a particular algorithmic tweak as a development? If we lump more things together, the developments are bigger, so it isn’t clear what it means to say that one is especially big. It isn’t clear what counterfactuals are being used. Is the claim that AI will have a large impact relative to a world without any AI? To a world that is similar to the current world forever? Why is the difference between AI and either of those (highly unlikely) scenarios relevant to the value of influencing the development and deployment of AI? The heuristic connection between the scale of an action’s possible impact and the stakes of influencing it seems perhaps more useful for ruling out fundamentally unworthy issues than comparing minor impacts on very large issues. For instance, one can use it to confidently tell that it isn’t worth spending half an hour deciding which flavor of jam to buy, whereas it seems much worse for comparing ‘working on functional institutions’ to ‘working on AI’, where both are vaguely ‘big’ and your impact is likely to be ‘tiny’.How easy different ‘developments’ are to influence is as big an issue as how much is at stake in influencing them, so is better treated quantitatively, not as a possible qualitative defeater.It isn’t clear that ‘influencing developments’ is a good way of viewing possibly high-value actions. For instance, it excludes many actions that are not in response to the actions of others.
Discussion elsewhere
We have seen discussion of this in the following places. The name is from Ngo.
Ngo, Richard. “Thinking Complete: Disentangling Arguments for the Importance of AI Safety.” Thinking Complete (blog), January 21, 2019. http://thinkingcomplete.blogspot.com/2019/01/disentangling-arguments-for-importance.html.80,000 Hours. “How Sure Are We about This AI Stuff?” Accessed September 16, 2020. https://80000hours.org/podcast/episodes/ben-garfinkel-classic-ai-risk-arguments/.
Richard Ngo describes this as follows1:
Argument from large impacts. Even if we’re very uncertain about what AGI development and deployment will look like, it seems likely that AGI will have a very large impact on the world in general, and that further investigation into how to direct that impact could prove very valuable.Weak version: development of AGI will be at least as big an economic jump as the industrial revolution, and therefore affect the trajectory of the long-term future. See Ben Garfinkel’s talk at EA Global London 2018. Ben noted that to consider work on AI safety important, we also need to believe the additional claim that there are feasible ways to positively influence the long-term effects of AI development – something which may not have been true for the industrial revolution. (Personally my guess is that since AI development will happen more quickly than the industrial revolution, power will be more concentrated during the transition period, and so influencing its long-term effects will be more tractable.)Strong version: development of AGI will make humans the second most intelligent species on the planet. Given that it was our intelligence which allowed us to control the world to the large extent that we do, we should expect that entities which are much more intelligent than us will end up controlling our future, unless there are reliable and feasible ways to prevent it. So far we have not discovered any.
(We treat 2 as a separate argument, ‘Argument from most intelligent species‘.)
Notes