Relevant pre-AGI possibilities

By Daniel Kokotajlo1, 18 June 2020.
Epistemic status: I started this as an AI Impacts research project, but given that it’s fundamentally a fun speculative brainstorm, it worked better as a blog post.
The default, when reasoning about advanced artificial general intelligence (AGI), is to imagine it appearing in a world that is basically like the present. Yet almost everyone agrees the world will likely be importantly different by the time advanced AGI arrives.
One way to address this problem is to reason in abstract, general ways that are hopefully robust to whatever unforeseen developments lie ahead. Another is to brainstorm particular changes that might happen, and check our reasoning against the resulting list.

This is an attempt to begin the second approach.2 I sought things that might happen that seemed both (a) within the realm of plausibility, and (b) probably strategically relevant to AI safety or AI policy.

I collected potential list entries via brainstorming, asking others for ideas, googling, and reading lists that seemed relevant (e.g. Wikipedia’s list of emerging technologies,3 a list of Ray Kurzweil’s predictions4, and DARPA’s list of projects.5)

I then shortened the list based on my guesses about the plausibility and relevance of these possibilities. I did not put much time into evaluating any particular possibility, so my guesses should not be treated as anything more. I erred on the side of inclusion, so the entries in this list vary greatly in plausibility and relevance. I made some attempt to categorize these entries and merge similar ones, but this document is fundamentally a brainstorm, not a taxonomy, so keep your expectations low.

I hope to update this post as new ideas find me and old ideas are refined or refuted. I welcome suggestions and criticisms; email me (gmail kokotajlod) or leave a comment.
Interactive “Generate Future” button
Asya Bergal and I made an interactive button to go with the list. The button randomly generates a possible future according to probabilities that you choose. It is very crude, but it has been fun to play with, and perhaps even slightly useful. For example, once I decided that my credences were probably systematically too high because the futures generated with them were too crazy. Another time I used the alternate method (described below) to recursively generate a detailed future trajectory, written up here. I hope to make more trajectories like this in the future, since I think this method is less biased than the usual method for imagining detailed futures.6
To choose probabilities, scroll down to the list below and fill each box with a number representing how likely you think the entry is to occur in a strategically relevant way prior to the advent of advanced AI. (1 means certainly, 0 means certainly not. The boxes are all 0 by default.) Once you are done, scroll back up and click the button.
A major limitation is that the button doesn’t take correlations between possibilities into account. The user needs to do this themselves, e.g. by redoing any generated future that seems silly, or by flipping a coin to choose between two generated possibilities that seem contradictory, or by choosing between them based on what else was generated.
Here is an alternate way to use this button that mostly avoids this limitation:
Fill all the boxes with probability-of-happening-in-the-next-5-years (instead of happening before advanced AGI, as in the default method)Click the “Generate Future” button and record the results, interpreted as what happens in the next 5 years.Update the probabilities accordingly to represent the upcoming 5-year period, in light of what has happened so far.Repeat steps 2 – 4 until satisfied. I used a random number generator to determine whether AGI arrived each year.
If you don’t want to choose probabilities yourself, click “fill with pre-set values” to populate the fields with my non-expert, hasty guesses.7

GENERATE FUTURE

Fill with pre-set values (default method)

Fill with pre-set values (alternate method)

Key
Letters after list titles indicate that I think the change might be relevant to:
TML: Timelines—how long it takes for advanced AI to be developedTAS: Technical AI safety—how easy it is (on a technical level) to make advanced AI safe, or what sort of technical research needs to be donePOL: Policy—how easy it is to coordinate relevant actors to mitigate risks from AI, and what policies are relevant to this. CHA: Chaos—how chaotic the world is.8MIS: Miscellaneous
Each possibility is followed by some explanation or justification where necessary, and a non-exhaustive list of ways the possibility may be relevant to AI outcomes in particular (which is not guaranteed to cover the most important ones). Possibilities are organized into loose categories created after the list was generated.
List of strategically relevant possibilities
Inputs to AI

1. Advanced science automation and research tools (TML, TAS, CHA, MIS)

Narrow research and development tools might speed up technological progress in general or in specific domains. For example, several of the other technologies on this list might be achieved with the help of narrow research and development tools.

2. Dramatically improved computing hardware (TML, TAS, POL, MIS)

By this I mean computing hardware improves at least as fast as Moore’s Law. Computing hardware has historically become steadily cheaper, though it is unclear whether this trend will continue. Some example pathways by which hardware might improve at least moderately include:
Ordinary scale economies9Improved data locality10Increased specialization for specific AI applications11Optical computing12Neuromorphic chips133D integrated circuits14Wafer-scale chips15Quantum computing16Carbon nanotube field-effect transistors17
Dramatically improved computing hardware may:
Cause any given AI capability to arrive earlierIncrease the probability of hardware overhang. Affect which kinds of AI are developed first (e.g. those which are more compute-intensive.) Affect AI policy, e.g. by changing the relative importance of hardware vs. research talent

3. Stagnation in computing hardware progress (TML, TAS, POL, MIS)

Many forecasters think Moore’s Law will be ending soon (as of 2020).18 In the absence of successful new technologies, computing hardware could progress substantially more slowly than Moore’s Law would predict.
Stagnation in computing hardware progress may:
Cause any given AI capability to arrive laterDecrease the probability of hardware overhang. Affect which kinds of AI are developed first (e.g. those which are less compute-intensive.) Influence the relative strategic importance of hardware compared to researchersMake energy and raw materials a greater part of the cost of computing

4. Manufacturing consolidation (POL)

Chip fabrication has become more specialized and consolidated over time, to the point where all of the hardware relevant to AI research depends on production from a handful of locations.19 Perhaps this trend will continue.
One country (or a small number working together) could control or restrict AI research by controlling the production and distribution of necessary hardware.

5. Advanced additive manufacturing (e.g. 3D printing or nanotechnology) (TML, CHA)

Advanced additive manufacturing could lead to various materials, products and forms of capital being cheaper and more broadly accessible, as well as to new varieties of them becoming feasible and quicker to develop. For example, sufficiently advanced 3D printing could destabilize the world by allowing almost anyone to secretly produce terror weapons. If nanotechnology advances rapidly, so that nanofactories can be created, the consequences could be dramatic:20
Greatly reduced cost of most manufactured productsGreatly faster growth of capital formationLower energy costsNew kinds of materials, such as stronger, lighter spaceship hullsMedical nanorobotsNew kinds of weaponry and other disruptive technologies

6. Massive resource glut (TML, TAS, POL, CHA)

By “glut” I don’t necessarily mean that there is too much of a resource. Rather, I mean that the real price falls dramatically. Rapid decreases in the price of important resources have happened before.21 It could happen again via:
Cheap energy (e.g. fusion power, He-3 extracted from lunar regolith,22 methane hydrate extracted from the seafloor,23 cheap solar energy24)A source of abundant cheap raw materials (e.g. asteroid mining,25 undersea mining26)Automation of relevant human labor. Where human labor is an important part of the cost of manufacturing, resource extraction, or energy production, automating labor might substantially increase economic growth, which might result in a greater amount of resources devoted to strategically relevant things (such as AI research) which is relevantly similar to a price drop even if technically the price doesn’t drop.27 and therefore investment in AI.
My impression is that energy, raw materials, and unskilled labor combined are less than half the cost of computing, so a decrease in the price of one of these (and possibly even all three) would probably not have large direct consequences on the price of computing.28 But a resource glut might lead to general economic prosperity, with many subsequent effects on society, and moreover the cost structure of computing may change in the future, creating a situation where a resource glut could dramatically lower the cost of computing.29

7. Hardware overhang (TML, TAS, POL)

Hardware overhang refers to a situation where large quantities of computing hardware can be diverted to running powerful AI systems as soon as the AI software is developed.
If advanced AGI (or some other powerful software) appears during a period of hardware overhang, its capabilities and prominence in the world could grow very quickly.

8. Hardware underhang (TML, TAS, POL)

The opposite of hardware overhang might happen. Researchers may understand how to build advanced AGI at a time when the requisite hardware is not yet available.  For example, perhaps the relevant AI research will involve expensive chips custom-built for the particular AI architecture being trained.
A successful AI project during a period of hardware underhang would not be able to instantly copy the AI to many other devices, nor would they be able to iterate quickly and make an architecturally improved version.
Technical tools

9. Prediction tools (TML, TAS, POL, CHA, MIS)

Tools may be developed that are dramatically better at predicting some important aspect of the world; for example, technological progress, cultural shifts, or the outcomes of elections, military clashes, or research projects. Such tools could for instance be based on advances in AI or other algorithms, prediction markets, or improved scientific understanding of forecasting (e.g. lessons from the Good Judgment Project).
Such tools might conceivably increase stability via promoting accurate beliefs, reducing surprises, errors or unnecessary conflicts. However they could also conceivably promote instability via conflict encouraged by a powerful new tool being available to a subset of actors. Such tools might also help with forecasting the arrival and effects of advanced AGI, thereby helping guide policy and AI safety work. They might also accelerate timelines, for instance by assisting project management in general and notifying potential investors when advanced AGI is within reach.

10. Persuasion tools (POL, CHA, MIS)

Present technology for influencing a person’s beliefs and behavior is crude and weak, relative to what one can imagine. Tools may be developed that more reliably steer a person’s opinion and are not so vulnerable to the victim’s reasoning and possession of evidence. These could involve:
Advanced understanding of how humans respond to stimuli depending on context, based on massive amounts of dataCoaching for the user on how to convince the target of somethingSoftware that interacts directly with other people, e.g. via text or email
Strong persuasion tools could:
Allow a group in conflict who has them to quickly attract spies and then infiltrate an enemy groupAllow governments to control their populationsAllow corporations to control their employeesLead to a breakdown of collective epistemology30

11. Theorem provers (TAS)

Powerful theorem provers might help with the kinds of AI alignment research that involve proofs or help solve computational choice problems.

12. Narrow AI for natural language processing (TML, TAS, CHA)

Researchers may develop narrow AI that understands human language well, including concepts such as “moral” and “honest.”
Natural language processing tools could help with many kinds of technology, including AI and various  AI safety projects. They could also help enable AI arbitration systems. If researchers develop software that can autocomplete code—much as it currently autocompletes text messages—it could multiply software engineering productivity.

13. AI interpretability tools (TML, TAS, POL)

Tools for understanding what a given AI system is thinking, what it wants, and what it is planning would be useful for AI safety.31

14. Credible commitment mechanisms (POL, CHA)

There are significant restrictions on which contracts governments are willing and able to enforce–for example, they can’t enforce a contract to try hard to achieve a goal, and won’t enforce a contract to commit a crime. Perhaps some technology (e.g. lie detectors, narrow AI, or blockchain) could significantly expand the space of possible credible commitments for some relevant actors: corporations, decentralized autonomous organizations, crowds of ordinary people using assurance contracts, terrorist cells, rogue AGIs, or even individuals.
This might destabilize the world by making threats of various kinds more credible, for various actors. It might stabilize the world in other ways, e.g. by making it easier for some parties to enforce agreements.

15. Better coordination tools (POL, CHA, MIS)

Technology for allowing groups of people to coordinate effectively could improve, potentially avoiding losses from collective choice problems, helping existing large groups (e.g. nations and companies) to make choices in their own interests, and producing new forms of coordinated social behavior (e.g. the 2010’s saw the rise of the Facebook group)).  Dominant assurance contracts,32 improved voting systems,33 AI arbitration systems, lie detectors, and similar things not yet imagined might significantly improve the effectiveness of some groups of people.
If only a few groups use this technology, they might have outsized influence. If most groups do, there could be a general reduction in conflict and increase in good judgment.
Human effectiveness

16. Deterioration of collective epistemology (TML, TAS, POL, CHA, MIS)

Society has mechanisms and processes that allow it to identify new problems, discuss them, and arrive at the truth and/or coordinate a solution. These processes might deteriorate. Some examples of things which might contribute to this:
Increased investment in online propaganda by more powerful actors, perhaps assisted by chatbots, deepfakes and persuasion toolsEcho chambers, filter bubbles, and online polarization, perhaps driven in part by recommendation algorithmsMemetic evolution in general might intensify, increasing the spreadability of ideas/topics at the expense of their truth/importance34Trends towards political polarization and radicalization might exist and continueTrends towards general institutional dysfunction might exist and continue
This could cause chaos in the world in general, and lead to many hard-to-predict effects. It would likely make the market for influencing the course of AI development less efficient (see section on “Landscape of…” below) and present epistemic hazards for anyone trying to participate effectively.

17. New and powerful forms of addiction (TML, POL, CHA, MIS)

Technology that wastes time and ruins lives could become more effective. The average person spends 144 minutes per day on social media, and there is a clear upward trend in this metric.35 The average time spent watching TV is even greater.36 Perhaps this time is not wasted but rather serves some important recuperative, educational, or other function. Or perhaps not; perhaps instead the effect of social media on society is like the effect of a new addictive drug — opium, heroin, cocaine, etc. — which causes serious damage until society adapts. Maybe there will be more things like this: extremely addictive video games, or newly invented drugs, or wireheading (directly stimulating the reward circuitry of the brain).37
This could lead to economic and scientific slowdown. It could also concentrate power and influence in fewer people—those who for whatever reason remain relatively unaffected by the various productivity-draining technologies. Depending on how these practices spread, they might affect some communities more or sooner than others.

18. Medicine or education to boost human mental abilities (TML, CHA, MIS)

To my knowledge, existing “study drugs” such as modafinil don’t seem to have substantially sped up the rate of scientific progress in any field. However, new drugs (or other treatments) might be more effective. Moreover, in some fields, researchers typically do their best work at a certain age. Medicine which extends this period of peak mental ability might have a similar effect.
Separately, there may be substantial room for improvement in education due to big data, online classes, and tutor software.38
This could speed up the rate of scientific progress in some fields, among other effects.

19. Genetic engineering, human cloning, iterated embryo selection (TML, POL, CHA, MIS)

Changes in human capabilities or other human traits via genetic interventions39 could affect many areas of life. If the changes were dramatic, they might have a large impact even if only a small fraction of humanity were altered by them.
Changes in human capabilities or other human traits via genetic interventions might:
Accelerate research in generalDifferentially accelerate research projects that depend more on “genius” and less on money or experienceInfluence politics and ideologyCause social upheavalIncrease the number of people capable of causing great harmHave a huge variety of effects not considered here, given the ubiquitous relevance of human nature to eventsShift the landscape of effective strategies for influencing AI development (see below)

20. Landscape of effective strategies for influencing AI development changes substantially (CHA, MIS)

For a person at a time, there is a landscape of strategies for influencing the world, and in particular for influencing AI development and the effects of advanced AGI. The landscape could change such that the most effective strategies for influencing AI development are:
More or less reliably helpful (e.g. working for an hour on a major unsolved technical problem might have a low chance of a very high payoff, and so not be very reliable)More or less “outside the box” (e.g. being an employee, publishing academic papers, and signing petitions are normal strategies, whereas writing Harry Potter fanfiction to illustrate rationality concepts and inspire teenagers to work on AI safety is not)40Easier or harder to find, such that marginal returns to investment in strategy research change
Here is a non-exhaustive list of reasons to think these features might change systematically over time:
As more people devote more effort to achieving some goal, one might expect that effective strategies become common, and it becomes harder to find novel strategies that perform better than common strategies. As advanced AI becomes closer, one might expect more effort to flow into influencing the situation. Currently some ‘markets’ are more efficient than others; in some the orthodox strategies are best or close to the best, whereas in others clever and careful reasoning can find strategies vastly better than what most people do. How efficient a market is depends on how many people are genuinely trying to compete in it, and how accurate their beliefs are. For example, the stock market and the market for political influence are fairly efficient, because many highly-knowledgeable actors are competing.  As more people take interest, the ‘market’ for influencing the course of AI may become more efficient. (This would also decrease the marginal returns to investment in strategy research, by making orthodox strategies closer to optimal.) If there is a deterioration of social epistemology (see below), the market might instead become less efficient.  Currently there are some tasks at which the most skilled people are not much better than the average person  (e.g. manual labor, voting) and others in which the distribution of effectiveness is heavy-tailed, such that a large fraction of the total influence comes from a small fraction of individuals (e.g. theoretical math, donating to politicians). The types of activity that are most useful for influencing the course of AI development may change over time in this regard, which in turn might affect the strategy landscape in all three ways described above.Transformative technologies can lead to new opportunities and windfalls for people who recognize them early. As more people take interest, opportunities for easy success disappear. Perhaps there will be a burst of new technologies prior to advanced AGI, creating opportunities for unorthodox or risky strategies to be very successful.
A shift in the landscape of effective strategies for influencing the course of AI is relevant to anyone who wants to have an effective strategy for influencing the course of AI.41 If it is part of a more general shift in the landscape of effective strategies for other goals — e.g. winning wars, making money, influencing politics — the world could be significantly disrupted in ways that may be hard to predict.

21. Global economic collapse (TML, CHA, MIS)

This might slow down research or precipitate other relevant events, such as war.

22. Scientific stagnation (TML, TAS, POL, CHA, MIS)

There is some evidence that scientific progress in general might be slowing down. For example, the millennia-long trend of decreasing economic doubling time seems to have stopped around 1960.42 Meanwhile, scientific progress has arguably come from increased investment in research. Since research investment has been growing faster than the economy, it might eventually saturate and grow only as fast as the economy.43
This might slow down AI research, making the events on this list (but not the technologies) more likely to happen before advanced AGI.

23. Global catastrophe (TML, POL, CHA)

Here are some examples of potential global catastrophes:
Climate change tail risks, e.g. feedback loop of melting permafrost releasing methane44Major nuclear exchangeGlobal pandemicVolcano eruption that leads to 10% reduction in global agricultural production45Exceptionally bad solar storm knocks out world electrical grid46Geoengineering project backfires or has major negative side-effects47
A global catastrophe might be expected to cause conflict and slowing of projects such as research, though it could also conceivably increase attention on projects that are useful for dealing with the problem. It seems likely to have other hard to predict effects.
Attitudes toward AGI

24. Shift in level of public attention on AGI (TML, POL, CHA, MIS)

The level of attention paid to AGI by the public, governments, and other relevant actors might increase (e.g. due to an impressive demonstration or a bad accident) or decrease (e.g. due to other issues drawing more attention, or evidence that AI is less dangerous or imminent).
Changes in the level of attention could affect the amount of work on AI and AI safety. More attention could also lead to changes in public opinion such as panic or an AI rights movement.
If the level of attention increases but AGI does not arrive soon thereafter, there might be a subsequent period of disillusionment.

25. Change in investment in AGI development (TML, TAS, POL)

There could be a rush for AGI, for instance if major nations begin megaprojects to build it. Or there could be a rush away from AGI, for instance if it comes to be seen as immoral or dangerous like human cloning or nuclear rocketry.
Increased investment in AGI might make advanced AGI happen sooner, with less hardware overhang and potentially less proportional investment in safety. Decreased investment might have the opposite effects.

26. New social movements or ideological shifts (TML, TAS, POL, MIS)

The communities that build and regulate AI could undergo a substantial ideological shift. Historically, entire nations have been swept by radical ideologies within about a decade or so, e.g. Communism, Fascism, the Cultural Revolution, and the First Great Awakening.48 Major ideological shifts within communities smaller than nations (or within nations, but on specific topics) presumably happen more often. There might even appear powerful social movements explicitly focused on AI, for instance in opposition to it  or attempting to secure legal rights and moral status for AI agents.49 Finally, there could be a general rise in extremist movements, for instance due to a symbiotic feedback effect hypothesized by some,50 which might have strategically relevant implications even if mainstream opinions do not change.
Changes in public opinion on AI might change the speed of AI research, change who is doing it, change which types of AI are developed or used, and limit or alter discussion. For example, attempts to limit an AI system’s effects on the world by containing it might be seen as inhumane, as might adversarial and population-based training methods. Broader ideological change or a rise in extremisms might increase the probability of a massive crisis, revolution, civil war, or world war.

27. Harbinger of AGI (ALN, POL, MIS)

Events could occur that provide compelling evidence, to at least a relevant minority of people, that advanced AGI is near.
This could increase the amount of technical AI safety work and AI policy work being done, to the extent that people are sufficiently well-informed and good at forecasting. It could also enable people already doing such work to more efficiently focus their efforts on the true scenario.

28. AI alignment warning shot (ALN, POL)

A convincing real-world example of AI alignment failure could occur.
This could motivate more effort into mitigating AI risk and perhaps also provide useful evidence about some kinds of risks and how to avoid them.
Precursors to AGI

29. Brain scanning (TML, TAS, POL, CHA, MIS)

An accurate way to scan human brains at a very high resolution could be developed.
Combined with a good low-level understanding of the brain (see below) and sufficient computational resources, this might enable brain emulations, a form of AGI in which the AGI is similar, mentally, to some original human. This would change the kind of technical AI safety work that would be relevant, as well as introducing new AI policy questions. It would also likely make AGI timelines easier to predict. It might influence takeoff speeds.

30. Good low-level understanding of the brain (TML, TAS, POL, CHA, MIS)

To my knowledge, as of April 2020, humanity does not understand how neurons work well enough to accurately simulate the behavior of a C. Elegans worm, though all connections between its neurons have been mapped51 Ongoing progress in modeling individual neurons could change this, and perhaps ultimately allow accurate simulation of entire human brains.
Combined with brain scanning (see above) and sufficient computational resources, this may enable brain emulations, a form of AGI in which the AI system is similar, mentally, to some original human. This would change the kind of AI safety work that would be relevant, as well as introducing new AI policy questions. It would also likely make the time until AGI is developed more predictable. It might influence takeoff speeds. Even if brain scanning is not possible, a good low-level understanding of the brain might speed AI development, especially of systems that are more similar to human brains.

31. Brain-machine interfaces (TML, TAS, POL, CHA, MIS)

Better, safer, and cheaper methods to control computers directly with our brains may be developed. At least one project is explicitly working towards this goal.52
Strong brain-machine interfaces might:
Accelerate research, including on AI and AI safety53Accelerate in vitro brain technologyAccelerate mind-reading, lie detection, and persuasion toolsDeteriorate collective epistemology (e.g. by contributing to wireheading or short attention spans)Improve collective epistemology (e.g. by improving communication abilities)Increase inequality in influence among people

32. In vitro brains (TML, TAS, POL, CHA)

Neural tissue can be grown in a dish (or in an animal and transplanted) and connected to computers, sensors, and even actuators.54 If this tissue can be trained to perform important tasks, and the technology develops enough, it might function as a sort of artificial intelligence. Its components would not be faster than humans, but it might be cheaper or more intelligent. Meanwhile, this technology might also allow fresh neural tissue to be grafted onto existing humans, potentially serving as a cognitive enhancer.55
This might change the sorts of systems AI safety efforts should focus on. It might also automate much human labor, inspire changes in public opinion about AI research (e.g. promoting concern about the rights of AI systems), and have other effects which are hard to predict.

33. Weak AGI (TML, TAS, POL, CHA, MIS)

Researchers may develop something which is a true artificial general intelligence—able to learn and perform competently all the tasks humans do—but just isn’t very good at them, at least, not as good as a skilled human.
If weak AGI is faster or cheaper than humans, it might still replace humans in many jobs, potentially speeding economic or technological progress. Separately, weak AGI might provide testing opportunities for technical AI safety research. It might also change public opinion about AI, for instance inspiring a “robot rights” movement, or an anti-AI movement.

34. Expensive AGI (TML, TAS, POL, CHA, MIS)

Researchers may develop something which is a true artificial general intelligence, and moreover is qualitatively more intelligent than any human, but is vastly more expensive, so that there is some substantial period of time before cheap AGI is developed.
An expensive AGI might contribute to endeavors that are sufficiently valuable, such as some science and technology, and so may have a large effect on society. It might also prompt increased effort on AI or AI safety, or inspire public thought about AI that produces changes in public opinion and thus policy, e.g. regarding the rights of machines. It might also allow opportunities for trialing AI safety plans prior to very widespread use.

35. Slow AGI (TML, TAS, POL, CHA, MIS)

Researchers may develop something which is a true artificial general intelligence, and moreover is qualitatively as intelligent as the smartest humans, but takes a lot longer to train and learn than today’s AI systems.
Slow AGI might be easier to understand and control than other kinds of AGI, because it would train and learn more slowly, giving humans more time to react and understand it. It might produce changes in public opinion about AI.

36. Automation of human labor (TML, TAS, POL, CHA, MIS)

If the pace of automation substantially increases prior to advanced AGI, there could be social upheaval and also dramatic economic growth. This might affect investment in AI.
Shifts in the balance of power

37. Major leak of AI research (TML, TAS, POL, CHA)

Edward Snowden defected from the NSA and made public a vast trove of information. Perhaps something similar could happen to a leading tech company or AI project.
In a world where much AI progress is hoarded, such an event could accelerate timelines and make the political situation more multipolar and chaotic.

38. Shift in favor of espionage (POL, CHA, MIS)

Espionage techniques might become more effective relative to counterespionage techniques. In particular:
Quantum computing could break current encryption protocols.56Automated vulnerability detection57 could turn out to have an advantage over automated cyberdefense systems, at least in the years leading up to advanced AGI.
More successful espionage techniques might make it impossible for any AI project to maintain a lead over other projects for any substantial period of time. Other disruptions may become more likely, such as hacking into nuclear launch facilities, or large scale cyberwarfare.

39. Shift in favor of counterespionage (POL, CHA, MIS)

Counterespionage techniques might become more effective relative to espionage techniques than they are now. In particular:
Post-quantum encryption might be secure against attack by quantum computers.58Automated cyberdefense systems could turn out to have an advantage over automated vulnerability detection. Ben Garfinkel and Allan Dafoe59 give reason to think the balance will ultimately shift to favor defense.
Stronger counterespionage techniques might make it easier for an AI project to maintain a technological lead over the rest of the world. Cyber wars and other disruptive events could become less likely.

40. Broader or more sophisticated surveillance (POL, CHA, MIS)

More extensive or more sophisticated surveillance could allow strong and selective policing of technological development. It would also have other social effects, such as making totalitarianism easier and making terrorism harder.

41. Autonomous weapons (POL, CHA)

Autonomous weapons could shift the balance of power between nations, or shift the offense-defense balances resulting in more or fewer wars or terrorist attacks, or help to make totalitarian governments more stable. As a potentially early, visible and controversial use of AI, they may also especially influence public opinion on AI more broadly, e.g. prompting anti-AI sentiment.

42. Shift in importance of governments, corporations, and other groups in AI development (POL, CHA)

Currently both governments and corporations are strategically relevant actors in determining the course of AI development. Perhaps governments will become more important, e.g. by nationalizing and merging AI companies. Or perhaps governments will become less important, e.g. by not paying attention to AI issues at all, or by becoming less powerful and competent generally. Perhaps some third kind of actor (such as religion, insurgency, organized crime, or special individual) will become more important, e.g. due to persuasion tools, countermeasures to surveillance, or new weapons of guerilla warfare.60
This influences AI policy by affecting which actors are relevant to how AI is developed and deployed.

43. Catastrophe in strategically important location (TML, POL, CHA, MIS)

Perhaps some strategically important location (e.g. tech hub, seat of government, or chip fab) will be suddenly destroyed. Here is a non-exhaustive list of ways this could happen:
Terrorist attack with weapon of mass destructionMajor earthquake, flood, tsunami, etc. (e.g. this research claims a 2% chance of magnitude 8.0 or greater earthquake in San Francisco by 2044.)61
If it happens, it might be strategically disruptive, causing e.g. the dissolution and diaspora of the front-runner AI project, or making it more likely that some government makes a radical move of some sort.

44. Change in national AI research loci (POL, CHA)

For instance, a new major national hub of AI research could arise, rivalling the USA and China in research output. Or either the USA or China could cease to be relevant to AI research.
This might make coordinating AI policy more difficult. It might make a rush for AGI more or less likely.

45. Large war (TML, POL, CHA, MIS)

This might cause short-term, militarily relevant AI capabilities research to be prioritized over AI safety and foundational research. It could also make global coordination on AI policy difficult.

46. Civil war or regime change in major relevant countries (POL, CHA, MIS)

This might be very dangerous for people living in those countries. It might change who the strategically relevant actors are for shaping AI development. It might result in increased instability, or cause a new social movement or ideological shift.

47. Formation of a world government (POL, CHA)

This would make coordinating AI policy easier in some ways (e.g. there would be no need for multiple governing bodies to coordinate their policy at the highest level), however it might be harder in others (e.g. there might be a more complicated regulatory system overall).
Notes

Relevant pre-AGI possibilities


Description vs simulated prediction

By Rick Korzekwa, 22 April 2020
During our investigation into discontinuous progress1, we had some discussions about what exactly it is that we’re trying to do when we analyze discontinuities in technological progress. There was some disagreement and (at least on my part) some confusion about what we were trying to learn from all of this. I think this comes from two separate, more general questions that can come up when forecasting future progress. These questions are closely related and both can be answered in part by analyzing discontinuities. First I will describe these questions generically, then I will explain how they can become confused in the context of analyzing discontinuous progress.
Question 1: How did tech progress happen in the past?
Knowing something about how historical progress happened is crucial to making good forecasts now, both from the standpoint of understanding the underlying mechanisms and establishing base rates. Merely being able to describe what happened and why can help us make sense of what’s happening now and should enable us to make better predictions. Such descriptions vary in scope and detail. For example:
The number of transistors per chip doubled roughly every two years from 1970 to 2020Wartime funding during WWII led to the rapid development and scaling of penicillin production, which contributed to a 90% decrease in US syphilis mortality from 1945 to 1967Large jumps in metrics for technological progress were not always accompanied by fundamental scientific breakthroughs
This sort of analysis may be used for other work, in addition to forecasting rates of progress. In the context of our work on discontinuities, answering this question mostly consists of describing quantitatively how metrics for progress evolve over time.
Question 2: How would we have fared making predictions in the past?
This is actually a family of questions aimed at developing and calibrating prediction methods based on historical data. These are questions like:
If, in the past, we’d predicted that the current trend would hold, how often and by how much would we have been wrong?Are there domains in which we’d have fared better than others?Are there heuristics we can use to make better predictions?Which methods for characterizing trends in progress would have performed the best?How often would we have seen hints that a discontinuity or change in rate of progress was about to happen?
These questions often, but not always, require the same approach
In the context of our work on discontinuous progress, these questions converge on the same methods most of the time. For many of our metrics, there was a clear trend leading up to a discontinuity, and describing what happened is essentially the same as attempting to (naively) predict what would have happened if the discontinuity had not happened. But there are times when they differ. In particular, this can happen when we have different information now than we would have had at the time the discontinuity happened, or when the naive approach is clearly missing something important. Three cases of this that come to mind are:
The trend leading up to the discontinuity was ambiguous, but later data made it less ambiguous. For example, advances in steamships improved times for crossing the Atlantic, but it was not clear whether this progress was exponential or linear at the time that flight or telecommunications were invented. But if we look at progress that occurred for transatlantic ship voyages after flight, we can see that the overall trend was linear. If we want to answer the question “What happened?”, we might say that progress in steamships was linear, so that it would have taken 500 years at the rate of advancement for steamships to bring crossing time down to that of the first transatlantic flight. If we want to answer the question “How much would this discontinuity have affected our forecasts at the time?”, we might say that it looked exponential, so that our forecasts would have been wrong by a substantially shorter amount of time.
We now have access to information from before the discontinuity that nobody (or no one person) had access to at the time. In the past, the world was much less connected, and it is not clear who knew about what at the time. For example, building heights, altitude records, bridge spans, and military capabilities all showed progress across different parts of the world, and it seems likely that nobody had access to all the information that we have now, so that forecasting may have been much harder or yielded different results. Information that is actively kept secret may have made this problem worse. It seems plausible that nobody knew the state of both the Manhattan Project and the German nuclear weapons program at the time that the first nuclear weapon was tested in 1945.
The inside view overwhelms the outside view. For example, the second transatlantic telegraph cable was much, much better than the first. Using our methodology, it was nearly as large an advancement over the first cable as the first cable was over mailing by ship. But we lose a lot by viewing these advances only in terms of their deviation from the previous trend. The first cable had extremely poor performance, while the second performed about as well as a typical high performance telegraph cable did at the time. If we were trying to predict future progress at the time, we’d focus on questions like “How long do we think it will take to get a normal cable working?” or “How soon will it be until someone is willing to fund the next cable laying expedition?”, not “If we draw a line through the points on this graph, where does that take us?” (Though that outside view may still be worth consideration.) However, if we’re just trying to describe how the metric evolved over time, then the correct thing to do is to just draw a line through the points as best we can and calculate how far off-trend the new advancement is.
Reasons for focusing more on the descriptive approach for now
Both of these questions are important, and we can’t really answer one while totally ignoring the other. But for now, we have focused more on describing what happened (that is, answering question 1).
There are several reasons for this, but first I’ll describe some advantages to focusing on simulating historical predictions:
It mitigates what may be some misleading results from the descriptive approach. See, for example, the description of the transatlantic telegraph above.We’re trying to do forecasting (or enable others to do it), and really good answers to these questions might be more valuable.
But, for now, I think the advantages to focusing on description are greater:
The results are more readily reusable for other projects, either by us or by others. For example, answering a question like “How much of an improvement is a typical major advancement over previous technology?”It does not require us to model a hypothetical forecaster. It’s hard to predict what we (or someone else) would have predicted if asked about future progress in weapons technology just before the invention of nuclear weapons. To me, this process feels like it has a lot of moving parts or at least a lot of subjectivity, which leaves room for error, and makes it harder for other people to evaluate our methods.It is easier to build from question 1 to question 2 than the other way around. A description of what happened is a pretty reasonable starting point for figuring out which forecasting methods would have worked.It is easier to compare across technologies using question 1. Question 2 requires taking a more inside view, which makes comparisons harder.

Notes

Description vs simulated prediction


Surveys on fractional progress towards HLAI

Given simplistic assumptions, extrapolating fractional progress estimates suggests a median time from 2020 to human-level AI of:
372 years (2392), based on responses collected in Robin Hanson’s informal 2012-2017 survey.36 years (2056), based on all responses collected in the 2016 Expert Survey on Progress in AI.142 years (2162), based on the subset of responses to the 2016 Expert Survey on Progress in AI who had been in their subfield for at least 20 years.32 years (2052), based on the subset of responses to the 2016 Expert Survey on Progress in AI about progress in deep learning or machine learning as a whole rather than narrow subfields.
67% of respondents of the 2016 expert survey on AI and 44% of respondents who answered from Hanson’s informal survey said that progress was accelerating.
Details
One way of estimating how many years something will take is to estimate what fraction of progress toward it has been made over a fixed number of years, then to extrapolate the number of years needed for full progress. As suggested by Robin Hanson,1 this method can provide an estimate for when human-level AI will be developed, if we have data on what fraction of progress toward human-level AI has been made and whether it is proceeding at a constant rate.
We know of two surveys that ask about fractional progress and acceleration in specific AI subfields: an informal survey conducted by Robin Hanson in 2012 – 2017, and our 2016 Expert Survey on Progress in AI. We use them to extrapolate progress to human-level AI, assuming that:
AI progresses at the average rate that people have observed so far.Human-level AI will be achieved when the median subfield reaches human-level.
Assumptions
AI progresses at the average rate that people have observed so far
The naive extrapolation method described above assumes that AI progresses at the average rate that people have observed so far, but some respondents perceived acceleration or deceleration. If we guess that this change in the rate of the progress continues into the future, this suggests that a truer extrapolation of each person’s observations would place human-level performance in their subfield either before or after the naively extrapolated date.
Human-level AI will be achieved when the median subfield reaches human-level
Both surveys asked respondents about fractional progress in their subfields. Extrapolating out these estimates to get to human-level performance gives some evidence for when AGI may come, but is not a perfect proxy. It may turn out that we get human-level performance in a small number of subfields much earlier than others, such that we count the resulting AI as ‘AGI’, or it may be the case that certain subfields important to AGI do not exist yet.
Hanson AI Expert Survey
Hanson’s survey informally asked ~15 AI experts to estimate how far we’ve come in their own subfield of AI research in the last twenty years, compared to how far we have to go to reach human level abilities. The subfields represented were analogical reasoning, knowledge representation, computer-assisted training, natural language processing, constraint satisfaction, robotic grasping manipulation, early-human vision processing, constraint reasoning, and “no particular subfield”. Three respondents said the rate of progress was staying the same, four said it was getting faster, two said it was slowing down, and six did not answer (or may not have been asked).
The naive extrapolations2 of the answers from Hanson’s survey give a median time from 2020 to human-level AI (HLAI) of 372 years (2392). See the survey data and our calculations here.
2016 Expert Survey on Progress in AI
The 2016 Expert Survey on Progress in AI (2016 ESPAI) asked machine learning researchers which subfield they were in, how long they had been in their subfield, and what fraction of the remaining path to human-level performance (in their subfield) they thought had been traversed in that time.3 107 out of 111 responses were used in our calculation.4 42 subfields were reported, including “Machine learning”, “Graphical models”, “Speech recognition”, “Optimization”, “Bayesian Learning”, and “Robotics”.5 Notably, Hanson’s survey included subfields that weren’t represented in 2016 ESPAI, including analogic reasoning and knowledge representation. Since 2016 ESPAI was restricted to machine learning researchers, it may exclude non-machine-learning subfields that turn out to be important to fully human-level capabilities.
Acceleration
67% of all respondents said progress in their subfield was accelerating (see Figure 1). Most respondents said progress in their subfield was accelerating in each of the subsets we look at below (ML vs narrow subfield, and time in field).
Figure 1: Number of responses that progress was faster in the first half of the time in the field worked by respondents, the second half, or was about the same in both halves.
Most respondents think progress is accelerating. If this acceleration continues, our naively extrapolated estimates below may be overestimates for time to human-level performance.
Time to HLAI
We calculated estimated years from 2020 until human-level subfield performance by naively extrapolating the reported fractions of the subfield already traversed.6 Figure 2 below shows the implied estimates for time until human-level performance for all respondents’ answers. These estimates give a median time from 2020 until HLAI of 36 years (2056).
Figure 2: Extrapolated estimated time until human-level subfield performance for each respondent, arranged by length of time. The last four responses are above 1000 but have been cut off.
Machine learning vs subfield progress
Some respondents reported broad ‘subfields’, which encompassed all of machine learning, in particular “Machine learning” or “Deep learning”, while others reported narrow subfields, e.g. “Natural language processing” or “Robotics”. We split the survey data based on this subfield narrowness, guessing that progress on machine learning overall may be a better proxy for AGI overall. Among the 69 respondents who gave answers corresponding to the entire field of machine learning, the median implied time was 32 years (2052). Among the 70 respondents who gave narrow answers, the median implied time was 44 years (2064). Figures 3 and 4 show these estimates.
Figure 3: Implied estimates for human-level performance based on respondents who specified broad answers, e.g. “Machine learning” when asked about their subfield. The last three responses are above 1000 but have been cut off.
Figure 3: Implied estimates for human-level performance based on respondents who specified broad answers, e.g. “Machine learning” when asked about their subfield. The last three responses are above 1000 but have been cut off.
Figure 4: Implied estimates for human-level performance based on respondents who specified narrow answers, e.g. “Natural language processing” when asked about their subfield. The last response is above 1000 but has been cut off.
The median implied estimate until human-level performance for machine learning broadly was 12 years sooner than the median estimate for specific subfields. This is counter to what we might expect, if human-level performance in machine learning broadly implies human-level performance on each individual subfield.
Time spent in field
Robin Hanson has suggested that his survey may get longer implied forecasts than 2016 ESPAI because he asks exclusively people who have spent at least 20 years in their field.7 Filtering for people who have spent at least 20 years in their field, we have eight responses, and get a median implied time until HLAI of 142 years from 2020 (2162). Filtering for people who have spent at least 10 years in their field, we have 38 responses, and get a median implied time of 86 years (2106). Filtering for people who have spent less than 10 years in their field, we have 69 responses, and get a median implied time of 24 years (2044). Figures 5, 6 and 7 show estimates for each respondent, for each of these classes of time in field.
Figure 5: Implied estimates for human-level performance based on respondents who were working on their subfield for at least 20 years. The last response is above 1000 but has been cut off.
Figure 6: Implied estimates for human-level performance based on respondents who were working on their subfield for at least 10 years. The last three responses are above 1000 but have been cut off.
Figure 7: Implied estimates for human-level performance based on respondents who were working on their subfield for less than 10 years. The last response is above 1000 but has been cut off.
Comparison of the two surveys
The median implied estimate from 2020 until human-level performance suggested by responses from 2016 ESPAI (36 years) is an order of magnitude smaller than the one suggested by the Hanson survey (372 years). This appears to be at least partly explained by more experienced researchers giving responses that imply longer estimates. Hanson asks exclusively people who have spent at least 20 years in their subfield, whereas the 2016 survey does not filter based on experience. If we filter 2016 survey respondents for researchers who have spent at least 20 years in their subfield we instead get a median estimate of 142 years.
More experienced researchers may generate longer implied estimates because the majority of progress has happened recently– many people think progress accelerated, which is some evidence of this. It could also be that less-experienced researchers feel that progress is more significant than it actually is.
If AI research is accelerating and is going to continue accelerating until we get to human-level AI, the time to HLAI may be sooner than these estimates. If AI research is accelerating now but is not representative of what progress will look like in the future, longer naive estimates by more experienced researchers may be more appropriate.
Comparison to estimates reached by other survey methods
2016 ESPAI also asked people to estimate time until human-level machine intelligence (HLMI) by asking them how many years they would give until a 50% chance of HLMI. The median answer for this question in 2016 was 40 years, or 36 years from 2020 (2056), exactly the same as the median answer of 36 years implied by extrapolating fractional progress. The survey also asked about time to HLMI in other ways, which yielded less consistent answers.
Primary author: Asya Bergal
Notes


2019 recent trends in Geekbench score per CPU price

From 2006 – 2020, Geekbench score per CPU price has grown by around 16% a year, for rates that would yield an order of magnitude over roughly 16 years.
Details
We looked at Geekbench 5,1 a benchmark for CPU performance. We combined Geekbench’s multi-core scores on its ‘Processor Benchmarks’ page2 with release dates and prices that we scraped from Wikichip and Wikipedia.3 All our data and plots can be found here.4 We then calculated score per dollar and adjusted for inflation using the consumer price index.5 For every year, we calculated the 95th percentile score per dollar. We then fit linear and exponential trendlines to those scores.
Figure 1 shows all our data for Geekbench score per CPU price.
Figure 1: Geekbench scores per CPU price, in 2019 dollars. Red dots denote the 95th percentile values in each year from 2006 – 2019 (we start at 2006 since we have <= 2 data points a year prior to then). The exponential trendline through the 95th percentiles is marked in red, while the linear trendline is marked in green. The vertical axis is log-scale.
The data is well-described by a linear or an exponential trendline. Assuming an exponential trend,6 Geekbench score per CPU price grew by around 16% per year between 2006 and 2020, a rate that would yield a factor of ten every 16 years.7
This is a markedly slower growth rate than those observed for CPU price performance trends in the past, however since it is for a different performance metric to any used earlier, it is unclear how similar one should expect them to be– from 1940 to 2008, Sandberg and Bostrom found that CPU price performance grew by a factor of ten every 5.6 years when measured in MIPS per dollar, and by a factor of ten every 7.7 years when measured in FLOPS per dollar.8
Primary author: Asya Bergal
Notes


Precedents for economic n-year doubling before 4n-year doubling

The only times gross world product appears to have doubled in n years without having doubled previously in 4n years were between 4,000 BC and 3,000 BC, and most likely between 10,000 BC and 4,000 BC.
Details
Background
A key open question regarding AI risk is how quickly advanced artificial intelligence will ‘take off’, which is to say something like ‘go from being a small source of influence in the world to an overwhelming one’.
In Superintelligence1, Nick Bostrom defines the following answers, seemingly in line with common usage:
Slow takeoff takes decades or centuriesModerate takeoff takes months or years Fast takeoff takes minutes to days
However the specific criteria for takeoff having occurred are generally ambiguous.
Paul Christiano has suggested2 operationalizing ‘slow takeoff’ as,
There will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles. (Similarly, we’ll see an 8 year doubling before a 2 year doubling, etc.)
Historic precedents
We were interested in whether anything faster than a ‘slow takeoff’ by this definition would be historically unprecedented. That is, we wanted to know whether whenever the economy has doubled in n years, it has always completed a doubling in 4n years or less before the beginning of the n year doubling.
We took historic gross world product (GWP) estimates from Wikipedia3 and checked at each date how long it had taken for the economy to double, and whether it had always at some point doubled in as few as four times as many years prior to the start of that doubling.4
We found two apparent examples of faster takeoffs, so defined:
Between 4,000 BC and 3,000 BC, GWP doubled in 1,000 years, yet it had never before doubled in as few as 4000 yearsBetween 10,000 BC and 4,000 BC, GWP doubled in 6,000 years, yet there is no record of it doubling earlier in as few as 24,000 years. The records at that point are fairly sparse, so this is less clear, but it seems unlikely that there was a doubling in 24,000 years.5 This appears to coincide with the beginning of agriculture, in around 9000BC.6
The 300 year period immediately after 1300 saw a doubling of GWP growth, and the 1200 years beforehand did not see a doubling, however there was an earlier doubling within the 1200 years ending at 1200AD. So this is not technically an instance, but was a case of briefly accelerating growth. GWP between 1100 and 1300 actually declined though, so this is perhaps a different kind of case to the ones we are interested in.
Corresponding author: Daniel Kokotajlo
Notes


Resolutions of mathematical conjectures over time

Conditioned on being remembered as a notable conjecture, the time-to-proof for a mathematical problem appears to be exponentially distributed with a half-life of about 100 years. However, these observations are likely to be distorted by various biases.
Support
In 2014, we found conjectures referenced on Wikipedia, and recorded the dates that they were proposed and resolved, if they were resolved. We updated this list of conjectures in 2020, marking any whose status had changed. We then used a Kaplan-Meier estimator1 to approximate the survivorship function.2
The results of this exercise are recorded here.3 Figure 1 below shows the survivorship function for the mathematical conjectures we found. The data is fit closely by an exponential function with a half-life of 117 years.4
Figure 1: Survivorship function of mathematical conjectures over time, also known as the fraction of mathematical conjectures unresolved at time t after being posed.
Biases
We are using resolution times for remembered conjectures as a proxy for resolution times for all conjectures. Resolution time for remembered conjectures might be biased in several ways: old conjectures are perhaps more likely to be remembered if they are solved than if they are not, very recently solved conjectures are probably more likely to be remembered (though this only matters because the rate of conjecture posing has probably changed over time), and conjectures that were especially hard to solve might also be more notable. The latter hundred years contains few data points, which makes it particularly easy for it to be inaccurate.
Relevance
To the extent that open theoretical problems in AI are similar to math problems, time to solve math problems may be informative for forming a prior on time to solve AI problems.
Corresponding author: Asya Bergal
Notes


Trends in DRAM price per gigabyte

The price of a gigabyte of DRAM has fallen by about a factor of ten every 5 years from 1957 to 2020. Since 2010, the price has fallen much more slowly, at a rate that would yield an order of magnitude over roughly 14 years.
Details
Background
DRAM, “dynamic random-access memory”, is a type of semiconductor memory. It is used as the main memory in modern computers and graphic cards.1
Data
We found two sources for historic pricing of DRAM. One was a dataset of DRAM prices and sizes from 1957 to 2018 collected by technologist and retired Computer Science professor2 John C. McCallum.3 The other dataset was extracted from a graph generated by Objective Analysis,4 a group that sells “third-party independent market research and data” to investors in the semiconductor industry.5 We have not checked where their data comes from and don’t have evidence about whether they are a trustworthy source.
Figure 1 shows McCallum’s data.6
Figure 1: Price per gigabyte of DRAM from 1957 to 2018 from John McCallum’s dataset, which we converted to 2020 dollars using the Consumer Price Index.7
Figure 2 shows the average price per gigabyte of DRAM from 1991 to 2019, according to the Objective Analysis graph.8
Figure 2: Average $ / GB of DRAM from 1991 to 2019 according to Objective Analysis. Dollars are 2020 dollars.
The two datasets appear to line up (see Figure 3 below),9 though we don’t know where the data in the Objective Analysis report came from– it could itself be referencing the McCallum dataset, or both could share data sources.
Figure 3: $ / GB of DRAM from 1957 to 2018, with McCallum’s dataset in blue and the Objective Analysis dataset in red. Dollars are 2020 dollars.
Analysis
For both sources, the data appears to follow an exponential trendline. In the McCallum dataset, we calculate that the price / GB of DRAM has fallen at around 36% per year, for a factor of ten every 5.1 years and a doubling time of 1.5 years on average. The Objective Analysis data is similar, with the price / GB of DRAM falling around 33% per year, for a factor of ten every 5.8 years and a doubling time of 1.7 years.
The 1.5 and 1.7 year doubling times are close to the rate at which Moore’s law observed that transistors in an integrated circuit double.10 It seems possible to us that cheaper and denser transistors following this law are what enabled the cheaper prices of DRAM, though we haven’t investigated this theory.11
Both datasets show slower progress in recent years. From 2010 onwards, the McCallum dataset falls in price by only 15% a year, for a rate that would yield a factor of ten every 14 years, and the Objective Analysis dataset falls by 12% a year, for a rate that would yield a factor of ten every 18.5 years.
Primary author: Asya Bergal
Notes

Trends in DRAM price per gigabyte