AI Timeline Surveys

Published 10 January 2015
This page is incomplete, under active work and may be updated soon.
We know of twelve surveys on the predicted timing of human-level AI. If we collapse a few slightly different meanings of 'human-level AI', then:

Median estimates for when there will be a 10% chance of human-level AI are all in the 2020s (from seven surveys), except for the 2016 ESPAI, which found median estimates ranging from 2013 to long after 2066, depending on question framing.
Median estimates for when there will be a 50% chance of human-level AI range between 2035 and 2050 (from seven surveys), except for the 2016 ESPAI, which found median estimates ranging from 2056 to at least 2106, depending on question framing.
Of three surveys in recent decades asking for predictions but not probabilities, two produced median estimates of when human-level AI will arrive in the 2050s, and one in 2085.

Participants appear to mostly be experts in AI or related areas, but with a large contingent of others. Several groups of survey participants seem likely over-represent people who are especially optimistic about human-level AI being achieved soon.
Details
List of surveys
These are the surveys that we know of on timelines to human-level AI:

Michie (1972)
Bainbridge (2005)
AI@50 (2006)
Klein (2007)
AGI-09 (2009)
FHI Winter Intelligence (2011)
Kruel (2011-12)
Hanson (2012 onwards)
Müller and Bostrom: AGI-12, TOP100, EETN, PTAI (2012-2013)

Results
Results summary

Year
Survey
#
 10%
 50%
 90%
 Other key 'Predictions'
Participants
Response rate
Link to original document

1972
 Michie
67

Median 50y (2022) (vs 20 or >50)
AI, CS
–
link

2005
 Bainbridge
26

 Median 2085
Tech
–
 link

2006
 AI@50

median >50y (2056)
AI conf
–
link

2007
 Klein
888

median 2030-2050
Futurism?
–
link and link

2009
 AGI-09
 21
 2020
 2040
 2075

AGI conf; AI
–
link

2011
 FHI Winter Intelligence
35
 2028
2050
 2150

AGI impacts conf; 44% related technical
41%
link

2011-2012
 Kruel interviews
37
 2025
 2035
 2070

AGI, AI
–
link

2012
 FHI: AGI-12
72
 2022
 2040
 2065

AGI & AGI impacts conf; AGI, technical work
65%
link

2012
 FHI:PT-AI
43
 2023
 2048
 2080

Philosophy & theory of AI conf; not technical AI
49%
link

2012-?
 Hanson
~10

 ≤ 10% progress to human level in past 20y
AI
–
link

2013
 FHI: TOP100
29
2022
 2040
 2075

Top AI
29%
link

2013
 FHI:EETN
26
 2020
 2050
 2093

Greek assoc. for AI; AI
10%

link

Time to a 10% chance and a 50% chance of human-level AI
The FHI Winter Intelligence, Müller and Bostrom, AGI-09, Kruel, and 2016 ESPAI surveys asked for years when participants expected 10%, 50% and 90% probabilities of human-level AI (or a similar concept). All of these surveys were taken between 2009 and 2012, except the 2016 ESPAI.
Survey participants' median estimates for when there will be a 10% chance of human-level AI are all in the 2020s or 2030s. Until the 2016 ESPAI survey, median estimates for when there will be a 50% chance of human-level AI ranged between 2035 and 2050. The 2016 ESPAI asked about human-level AI using both very similar questions to previous surveys, and a different style of question based on automation of specific human occupations. The former questions found median dates of at least 2056, and the latter question prompted median dates of at least 2106.
Non-probabilistic predictions
Three surveys (Bainbridge, Klein, and AI@50) asked about predictions, rather than confidence levels. These produced median predictions of  >2056 (AI@50), 2030-50 (Klein), and 2085 (Bainbridge). It is unclear how participants interpret the request to estimate when a thing will happen; these responses may mean the same as the 50% confidence estimate discussed above. These surveys together appear to contain a high density of people who don't work in AI, compared to the other surveys.
Michie survey
Michie's survey is unusual in being much earlier than the others (1972). In it, less than a third of participants expected human-level AI by 1992, another almost third estimated 2022, and the rest expected it later. Note that the participants' median expectation (50 years away) was further from their present time than those of contemporary survey participants. This point conflicts with a common perception that early AI predictions were shockingly optimistic, and quickly undermined.
Hanson survey
Hanson's survey is unusual in its methodology. Hanson informally asked some AI experts what fraction of the way to human-level capabilities we had come in 20 years, in their subfield. He also asked about apparent acceleration. Around half of answers were in the 5-10% range, and all except one which hadn't passed human-level already were less than 10%. Of six who reported on acceleration, only one saw positive acceleration.
These estimates suggest human-level capabilities in most fields will take more than 200 years, if progress proceeds as it has (i.e. if we progress at 10% per twenty years, it will take 200 years to get to 100%). This estimate is quite different from those obtained from most of the other surveys.
The 2016 ESPAI attempted to replicate this methodology, and did not appear to find similarly long implied timelines, however little attention has been paid to analyzing that data.
This methodology is discussed more in the methods section below.
Methods
Survey participants
In assessing the quality of predictions, we are interested in the expertise of the participants, the potential for biases in selecting them, and the degree to which a group of well-selected experts generally tend to make good predictions. We will leave the third issue to be addressed elsewhere, and here describe the participants' expertise and the surveys' biases. We will see that the participants have much expertise relevant to AI, but – relatedly – their views are probably biased toward optimism because of selection effects as well as normal human optimism about projects.
Summary of participant backgrounds
The FHI (2011), AGI-09, and one of the four FHI collection surveys are from AGI (artificial general intelligence) conferences, so will tend to include a lot of people who work directly on trying to create human-level intelligence, and others who are enthusiastic or concerned about that project. At least two of the aforementioned surveys draw some participants from the 'impacts' section of the AGI conference, which is likely to select for people who think the effects of human-level intelligence are worth thinking about now.
Kruel's participants are not from the AGI conferences, but around half work in AGI. Klein's participants are not known, except they are acquaintances of a person who is enthusiastic about AGI (his site is called 'AGI-world'). Thus many participants either do AGI research, or think about the topic a lot.
Many more participants are AI researchers from outside AGI. Hanson's participants are experts in narrow AI fields. Michie's participants are computer scientists working close to AI. Müller and Bostrom's surveys of the top 100 artificial intelligence researchers, and Members of the Greek Association for Artificial Intelligence, would be almost entirely AI researchers, and there is little reason to expect them to be in AGI. AI@50 seems to include a variety of academics interested in AI rather than those in the narrow field of AGI, though also includes others, such as several dozen graduate and post-doctoral students. 2016 ESPAI is everyone publishing in two top machine learning conferences, so largely machine learning researchers.
The remaining participants appear to be mostly highly educated people from academia and other intellectual areas. The attendees at the 2011 Conference on Philosophy and Theory of AI appear to be a mixture of philosophers, AI researchers, and academics from related fields such as brain sciences. Bainbridge's participants are contributors to 'converging technology' reports, on topics of nanotechnology, biotechnology, information technology, and cognitive science. From looking at what appears to be one of these reports, these seem to be mostly experts from government and national laboratories, academia, and the private sector. Few work in AI in particular. An arbitrary sample includes the Director of the Division of Behavioral and Cognitive Sciences at NSF, a person from the Defense Threat Reduction Agency, and a person from HP laboratories.
AGI researchers
As noted above, many survey participants work in AGI – the project to create general intelligent agents, as opposed to narrow AI applications. In general, we might expect people working on a given project to be unusually optimistic about its success, for two reasons. First, those who are most optimistic initially will more likely find the project worth investing in. Secondly, people are generally observed to be especially optimistic about the time needed for their own projects to succeed. So we might expect AGI researchers to be biased toward optimism, for these reasons.
On the other hand, AGI researchers are working on projects most closely related to human-level AI, so probably have the most relevant expertise.
Other AI researchers
Just as AGI researchers work on topics closer to human-level AI than other AI researchers – and so may be more biased but also more knowledgeable – AI researchers work on more relevant topics than everyone else. Similarly, we might expect them to both be more accurate due to their additional expertise, but more biased due to selection effects and optimism about personal projects.
Hanson's participants are experts in narrow AI fields, but are also reporting on progress in their own fields of narrow AI (rather than on general intelligence), so we might expect them to be more like the AGI researchers – especially expert and especially biased. On the other hand, Hanson asks about past progress rather than future expectations, which should diminish both the selection effect and the effect from the planning fallacy, so we might expect the bias to be weaker.
Definitions of human-level AI
A few different definitions of human-level AI are combined in this analysis.
The AGI-09 survey asked about four benchmarks; the one reported here is the Turing-test capable AI. Note that 'Turing test capable' seems to sometimes be interpreted as merely capable of holding a normal human discussion. It isn't clear that the participants had the same definition in mind.
Kruel only asked that the AI be as good as humans at science, mathematics, engineering and programming, and asks conditional on favorable conditions continuing (e.g. no global catastrophes). This might be expected prior to fully human-level AI.
Even where people talk about 'human-level' AI, they can mean a variety of different things. For instance, it is not clear whether a machine must operate at human cost to be 'human-level', or to what extent it must resemble a human.
At least three surveys use the acronym 'HLMI', but it can stand for either 'human-level machine intelligence' or 'high level machine intelligence' and is defined differently in different surveys.
Here is a full list of exact descriptions of something like 'human-level' used in the surveys:

Michie: 'computing system exhibiting intelligence at adult human level'
Bainbridge: 'The computing power and scientific knowledge will exist to buildmachines that are functionally equivalent to the human brain'
Klein: 'When will AI surpass human-level intelligence?'
AI@50: 'When will computers be able to simulate every aspect of human intelligence?'
FHI 2011: 'Assuming no global catastrophe halts progress, by what year would you assign a 10%/50%/90% chance of the development of human-level machine intelligence? Feel free to answer 'never' if you believe such a milestone will never be reached.'
Müller and Bostrom: '[machine intelligence] that can carry out most human professions at least as well as a typical human'
Hanson: 'human level abilities' in a subfield (wording is probably not consistent, given the long term and informal nature of the poll)
AGI-09: 'Passing the Turing test'
Kruel: Variants on, 'Assuming beneficial political and economic development and that no global catastrophe halts progress, by what year would you assign a 10%/50%/90% chance of the development of artificial intelligence that is roughly as good as humans (or better, perhaps unevenly) at science, mathematics, engineering and programming?'
2016 ESPAI (our emboldening):

Say we have 'high level machine intelligence' when unaided machines can accomplish every task better and more cheaply than human workers. Ignore aspects of tasks for which being a human is intrinsically advantageous, e.g. being accepted as a jury member. Think feasibility, not adoption.
Say an occupation becomes fully automatable when unaided machines can accomplish it better and more cheaply than human workers. Ignore aspects of occupations for which being a human is intrinsically advantageous, e.g. being accepted as a jury member. Think feasibility, not adoption.
Say we have reached 'full automation of labor' "when all occupations are fully automatable. That is, when for any occupation, machines could be built to carry out the task better and more cheaply than human workers."

Inside vs. outside view methods
Hanson's survey was unusual in that it asked participants for their impressions of past rates of progress, from which extrapolation could be made (an 'outside view' estimate), rather than asking directly about expected future rates of progress (an 'inside view' estimate). It also produced much later median dates for human-level AI, suggesting that this outside view methodology in general produces much later estimates (rather than for instance, Hanson's low sample size and casual format just producing a noisy or biased estimate that happened to be late).
If so, this would be important because outside view estimates in general are often informative.
However the 2016 ESPAI included a set of questions similar to Hanson's, and did not at a glance find similarly long implied timelines, though the data has not been carefully analyzed. This is some evidence against the outside view style methodology systematically producing longer timelines, though arguably not enough to overturn the hypothesis.
We might expect Hanson's outside view method to be especially useful in AI forecasting because a key merit is that asking people about the past means asking questions more closely related to their expertise, and the future of AI is arguably especially far from anyone's expertise (relative to say asking a dam designer how long it will take for their dam to be constructed) . On the other hand, AI researchers' expertise may include a lot of information about AI other than how far we have come, and translating what they have seen into what fraction of the way we have come may be difficult and thus introduce additional error.

AI Timeline Surveys


Michie Survey

In a 1972 poll of sixty-seven AI and computer science experts, respondents were roughly divided between expecting human-level intelligence in 20 years, in 50 years and in more than 50 years. They were also roughly divided between considering a 'takeover' by AI a negligible and a substantial risk – with 'overwhelming risk' far less popular.
Details
Methods
Donald Michie reported on the poll in Machines and the Theory of Intelligence (pdf download). The participants were sixty-seven British and American computer scientists working in or close to machine intelligence. The paper does not say much more about the methodology used. It is unclear whether Michie ran the poll.
Findings
Michie presents Figure 4 below, and Firschein and Coles present the table in Figure 2, which appears to be the same data. Michie's interesting findings include:

'Most considered that attainment of the goals of machine intelligence would cause human intellectual and cultural processes to be enhanced rather than to atrophy.'
'Of those replying to a question on the risk of ultimate 'takeover' of human affairs by intelligent machines, about half regarded it as 'negligible', and most of the remainder as 'substantial' with a few voting for 'overwhelming'.'
Almost all participants predicted human level computing systems would not emerge for over twenty years. They were roughly divided between 20, 50, and more. See figure 4 below (from p512).

Figure 2: Firschein and Coles 1973 present this table, which appears to report on the same survey.

Michie Survey


AI and the Big Nuclear Discontinuity

As we've discussed before, the advent of nuclear weapons was a striking technological discontinuity in the effectiveness of explosives. In 1940, no one had ever made an explosive twice as effective as TNT. By 1945 the best explosive was 4500 times more potent, and by 1960 the ratio was 5 million.
Progress in nuclear weapons is sometimes offered as an analogy for possible rapid progress in AI (e.g. by Eliezer Yudkowsky here, and here). It's worth clarifying the details of this analogy, which has nothing to do with the discontinuous progress in weapon effectiveness. It's about a completely different discontinuity: a single nuclear pile's quick transition from essentially inert to extremely reactive.
As you add more fissile material to a nuclear pile, little happens until it reaches a critical mass. After reaching critical mass, the chain reaction proceeds much faster than the human actions that assembled it. By analogy, perhaps as you add intelligence to a pile of intelligence, little will happen until it reaches a critical level which initiates a chain reaction of improvements ('recursive self-improvement') which proceeds much faster than the human actions that assembled it.
This discontinuity in individual nuclear explosions is not straightforwardly related to the technological discontinuity caused by their introduction. Older explosives were also based on chain reactions. The big jump seems to be a move from chemical chain reactions to nuclear chain reactions, two naturally occurring sources of energy with very different characteristic scales–and with no alternatives in between them. This jump has no obvious analog in AI.
One might wonder if the technological discontinuity was nevertheless connected to the discontinuous dynamics of individual nuclear piles. Perhaps the density and volume of fissile uranium required for any explosive was the reason that we did not see small, feeble nuclear weapons in between chemical weapons and powerful nuclear weapons. This doesn't match the history however. Nobody knew that concentrating fissile uranium was important until after fission was discovered in 1938, less than seven years before the first nuclear detonation. Even if nuclear weapons had grown in strength gradually over this period, this would still be around one thousand years of progress at the historical rate per year. The dynamics of individual piles can only explain a miniscule part of the discontinuity.
There may be an important analogy between AI progress and nuclear weapons. And the development of nuclear weapons was in some sense a staggeringly abrupt technological development. But we probably shouldn't conclude that the development of AI is much more likely to be comparably abrupt.
 If you vaguely remember that AI progress and nuclear weapons are analogous, and that nuclear weapons were a staggeringly abrupt development in explosive technology, try not to infer from this that AI is especially likely to be a staggeringly abrupt development.
(Image: Trinity test after ten seconds, taken from Atomic Bomb Test Site Photographs, courtesy of U.S. Army White Sands Missile Range Public Affairs Office)

AI and the Big Nuclear Discontinuity


The Biggest Technological Leaps

Over thousands of years, humans became better at producing explosions. A weight of explosive that would have blown up a tree stump in the year 800 could have blown up more than three tree stumps in the 1930s. Then suddenly, a decade later, the figure became more like nine thousand tree stumps. The first nuclear weapons represented a massive leap – something like 6000 years of progress in one step*.
Though such jumps have been historical exceptions, some some observers think a massive jump in AI capability is likely. Progress may be fast due to the apparent amenability of software to groundbreaking insights, the possibility of rapid applications (to deploy a new algorithm, you don't have to build any factories), the plausibility of simple conceptual ingredients underlying intelligent behavior, and the potential for 'recursive self-improvement' to speed software development to rates characteristic of superhumanly-programmed computers rather than that of humans.
We think the question, 'will AI progress be discontinuous?' is a good one to investigate. Not just because advance notice of abrupt world-changing developments is sure to come in handy somehow–nor because of the exciting degree of disagreement it elicits. What makes this a particularly good topic to study now is that it it helps us know what other information is most relevant to understanding AI progress.
One might hope to make predictions about how soon AI will reach human-level by extrapolating from how fast we are moving and how far we have to go, for instance.**  Or we could monitor the rate at which automation replaces workers, or at which the performance of AI systems improves. These all provide valuable information if you think AI will be reached gradually, by the continuation of existing processes. However if you expect progress to be abrupt and uneven, these indicators are much less informative.
So whether AI will be reached abruptly or incrementally is an important question. But is it a tractable one to make progress on? My guess is yes. Plenty of evidence bears on this question: the historical patterns of progress in other technologies, instances of abnormally uneven progress, arguments suggesting abnormal degrees of abnormality in the AI case, theories explaining past continuity and discontinuities, cases that look relevantly analogous to AI…
We know some examples of very fast technological progress; simply understanding those cases better is likely to be an informative start.
So we've have started a list of cases here. Each case appears to involve abrupt technological progress. We looked into each one a little, usually just enough to check it really involved abrupt progress and to get approximate rates of progress before and during the discontinuity. We intend to do a more thorough job later for the cases seem particularly important or interesting.
This list will hopefully help us understand what fast progress looks like historically (How fast is it? How far is it? How unexpected is it?), and when it happens (Does it usually flow from a huge intellectual insight? The discovery of a new natural phenomenon? Overcoming a large upfront investment?).
So far, we have a couple of really big jumps, a couple of smaller jumps, a bunch of potentially interesting but uncertain cases, and a rich assortment of purported discontinuities that we are yet to investigate.
After nuclear weapons, the second most interesting case we've found is high temperature superconductivity. The maximum temperature of superconduction appears to have made something like 150 years of progress in one jump in 1986,  after the discovery of a new class of materials with maximum temperatures for superconducting behavior above what was thought possible.
Do you have thoughts on this line of research? Do you have ideas for how to investigate cases? Do you know of historical cases of abrupt technological progress? Do you want to see our list?

* Measured in doublings; you would get a much more extreme estimate if you expected linear progress. Relative effectiveness (RE) had doubled less than twice in 1100 years, then it doubled more than eleven times when the first nuclear weapons emerged. (For more on nuclear weapons, see our page on them).
** Interestingly, asking AI researchers about rates of progress gives much more pessimistic estimates than asking them about when human-level AI will arrive, based on some very preliminary research. This may mean that AI researchers expect human-level AI to arrive following abnormally fast progress, though the discrepancy could be explained in many other ways. It seems worth of looking in to.
(Image: The first nuclear chain reaction. Painting by Gary Sheehan (Atomic Energy Commission).

The Biggest Technological Leaps


The AI Impacts Blog

Jan 09 2015
Welcome to the AI Impacts blog.
AI Impacts is premised on two ideas (at least!):

The details of the arrival of human-level artificial intelligence matter
Seven years to prepare is very different from seventy years to prepare. A weeklong transition is very different from a decade-long transition. Brain emulations require different preparations than do synthetic AI minds. Etc.
Available data and reasoning can substantially educate our guesses about these details
We can track progress in AI subfields. We can estimate the hardware represented by the human brain. We can detect the effect of additional labor on software progress. Etc.

Our goal is to assemble relevant evidence and considerations, and to synthesize reasonable views on questions such as when AI will surpass human-level capabilities, how rapid development will be at that point, what advance notice we might expect, and what kinds of AI are likely to reach human-level capabilities first.
We are doing this recursively, first addressing much smaller questions, like:

Is AI likely to surpass human level in a discontinuous spurt, or through incremental progress?
Does AI software undergo discontinuous progress often?
Is technological progress of any sort discontinuous often?
When is technological progress discontinuous?
Why did explosives undergo discontinuous progress in the form of nuclear weapons?

In this way, we hope to inform decisions about how to prepare for advanced AI, and about whether it is worth prioritizing over other pressing issues in the world. Researchers, funders, and other thinkers and doers are choosing how to spend their efforts on the future impacts of AI, and we want to help them choose well.
AI impacts is currently something like a (brief) encyclopedia of semi-original AI forecasting research. That is, it is a growing collection of pages addressing particular questions or bodies of evidence relating to the future of AI. We intend to revise these in an ongoing fashion, according to new investigations and debates.
At the same time as producing reasonable views, we are interested in exploring and bettering humanity's machinery for producing reasonable views. To this end, we have chosen this unusual – but we think promising – format, and may experiment with novel methods of organizing information and resolving questions and disagreements.
If you want to know more about the project overall, see About, or peruse our research pages and see it firsthand.
This blog exists to show you the most interesting findings of the AI Impacts project as we find them, and before they get lost in what we hope becomes a dense network of research pages. We might also write about other things, such as our thoughts on methodology, speculative opinions, news about the project itself, and anything else that seems like a good idea at the time.
If you like the sound of any of these things, consider signing up for one of our RSS feeds (blog, articles). If you don't, or if you think you could (cheaply) like it more, we welcome your thoughts or suggestions.
AI Impacts is currently authored by Paul Christiano and Katja Grace.

The AI Impacts Blog


Cases of Discontinuous Technological Progress

We know of ten events which produced a robust discontinuity in progress equivalent to more than one hundred years at previous rates in some interesting metric. We know of 53 other events which produced smaller or less robust discontinuities.
Background
These cases were researched as part of our discontinuous progress investigation.
List of cases
Events causing large, robust discontinuities

The Pyramid of Djoser, 2650BC (discontinuity in structure height trends)
The SS Great Eastern, 1858 (discontinuity in ship size trends)
The first telegraph, 1858 (discontinuity in speed of sending a 140 character message across the Atlantic Ocean)
The second telegraph, 1866 (discontinuity in speed of sending a 140 character message across the Atlantic Ocean)
The Paris Gun, 1918 (discontinuity in altitude reached by man-made means)
The first non-stop transatlantic flight, in a modified WWI bomber, 1919 (discontinuity in both speed of passenger travel across the Atlantic Ocean and speed of military payload travel across the Atlantic Ocean)
The George Washington Bridge, 1931 (discontinuity in longest bridge span)
The first nuclear weapons, 1945 (discontinuity in relative effectiveness of explosives)
The first ICBM, 1958 (discontinuity in average speed of military payload crossing the Atlantic Ocean)
YBa2Cu3O7 as a superconductor, 1987 (discontinuity in warmest temperature of superconduction)

Events causing moderate, robust discontinuities

HMS Warrior, 1860 (discontinuity in both Royal Navy ship tonnage and Royal Navy ship displacement)
Eiffel Tower, 1889 (discontinuity in tallest existing freestanding structure height, and in other height trends non-robustly)
Fairey Delta 2, 1956 (discontinuity in airspeed)
Pellets shot into space, 1957, measured after one day of travel (discontinuity in altitude achieved by man-made means)1
Burj Khalifa, 2009 (discontinuity in height of tallest building ever)

Non-robust discontinuities
This spreadsheet details all discontinuities found, as of April 2020.


Effect of nuclear weapons on historic trends in explosives

Nuclear weapons constituted a ~7 thousand year discontinuity in relative effectiveness factor (TNT equivalent per kg of explosive).
Nuclear weapons do not appear to have clearly represented progress in the cost-effectiveness of explosives, though the evidence there is weak.
Details
This case study is part of AI Impacts' discontinuous progress investigation.
Background
The development of nuclear weapons is often referenced informally as an example of discontinuous technological progress. Discontinuities are sometimes considered especially plausible in this case because of the involvement of a threshold phenomenon in nuclear chain reactions.
21-kiloton underwater nuclear explosion (Bikini Atoll, 1946)1
Trends
Relative effectiveness factor
The "relative effectiveness factor" (RE Factor) of an explosive measures the mass of TNT required for an equivalent explosion.2
Data
We collected data on explosive effectiveness from an online timeline of explosives and a comparison of RE factors on Wikipedia.3 These estimates modestly understate the impact of nuclear weapons, since the measured mass of the nuclear weapons includes the rest of the bomb while the conventional explosives are just for the explosive itself.
Figures 1-3 below show the data we collected, which can also be found in this spreadsheet. Our data below is incomplete– we elide many improvements between 800 and 1942 that would not affect the size of the discontinuity from "Fat man". We have verified that there are no explosives with higher RE factor than Hexanitrobenzene before "Fat man" (see the 'Relative effectiveness data' in this spreadsheet for this verification).
Figure 1: Approximate relative effectiveness factor for selected explosives over time, prior to nuclear weapons.
Figure 2: Approximate relative effectiveness factor for selected explosives, up to early nuclear bomb (note change to log scale)
Discontinuity Measurement
To compare nuclear weapons to past rates of progress, we treat progress as exponential.4 With this assumption, the first nuclear weapon, "Fat man", represented a around seven thousand years of discontinuity in the RE factor of explosives at previous rates.5 In addition to the size of this discontinuity in years, we have tabulated a number of other potentially relevant metrics here.6
We checked if "Fat Man" constituted a discontinuity, but did not look for other discontinuities, because we have not thoroughly searched for data on earlier developments. Even though we're missing data, since gunpowder is the earliest known explosive and Hexanitrobenzene is the explosive before "Fat man" with the highest RE factor, the missing data should not affect discontinuity calculations for "Fat man" unless it suggests we should be predicting using a different trend. This seems unlikely given that early explosives all have an RE factor close to that of our existing data points, around 1 – 3 (see table here)7, so are not vastly inconsistent with our exponential. If we instead assumed a linear trend, or an exponential ignoring the early gunpowder datapoint, we still get answers of over three thousand years (see spreadsheet for calculations).
Discussion of causes
Interestingly, at face value this discontinuous jump does not seem to be directly linked to the chain reaction that characterizes nuclear explosions, but rather to the massive gap between the energies involved in chemical interactions and nuclear interactions. It seems likely that similar results would obtain in other settings; for example, the accessible energy in nuclear fuel enormously exceeds the energy stored in chemical fuels, and so at some far future time we might expect a dramatic jump in the density with which we can store energy (though arguably not in the cost-effectiveness).8
Cost-effectiveness of explosives
Another important measure of progress in explosives is cost-effectiveness. Cost-effectiveness is particularly important to understand, because some plausible theories of continuous progress would predict continuous improvements in cost-effectiveness much more strongly than they would predict continuous improvements in explosive density.
Data
Cost-effectiveness of nuclear weapons
Assessing the cost of nuclear weapons is not straightforward empirically, and depends on the measurement of cost. The development of nuclear weapons incurred a substantial upfront cost, and so for some time the average cost of nuclear weapons significantly exceeded their marginal cost. We provide estimates for the marginal costs of nuclear weapons, as well as for the "average" cost of all nuclear explosives produced by a certain date.
We focus our attention on WWII and the immediately following period, to understand the extent to which the development of nuclear weapons represented a discontinuous change in cost-effectiveness.
See our spreadsheet for a summary of the data explained below. According to the Brookings Institute, nuclear weapons were by 1950 considered to be especially cost-effective (though not obviously in terms of explosive power per dollar), and adopted for this reason. However, Brookings notes that this has never been validated, and appears to distrust it.9 This disagreement weakly suggests that nuclear weapons are at least not radically more or less cost-effective than other weapons.
According to Wikipedia, the cost of the Manhattan project was about $26 billion (in 2014 dollars), 90% of which "was for building factories and producing the fissile materials."10 The Brookings U.S. Nuclear Weapons Cost Study Project estimates the price as $20 billion 2014 dollars, resulting in similar conclusions.11 This post claims that 9 bombs were produced through the end of "Operation Crossroads" in 1946, citing Chuck Hansen's Swords of Armageddon.12 The explosive power of these bombs was likely to be about 20kT, suggesting a total explosive capacity of 180kT.  Anecdotes suggest that the cost to actually produce a bomb were about $25M,13 or about $335M in 2014 dollars. This would make the marginal cost around $16.8k per ton of TNT equivalent ($335M/20kT = $16.75k/T), and the average cost around $111k/T.
In 2013 the US apparently planned to build 3,000 nuclear weapons for $60B.14 However it appears that at least some of these may be refurbishments rather than building from scratch, and the B61-12 design at least appears to be designed to be less powerful than it could be, since it is less powerful than the bombs it is replacing15 and much less powerful than a nuclear weapon such as the Tsar Bomba, with a yield of 50mT.16 The B61-12 is a 50kT weapon. These estimates give us $400/T ($60B/3,000*50kT). They are very approximate, for reasons given. However have not found better estimates. Note that they are for comparison, and not integral to our conclusions.
These estimates could likely be improved by a more careful survey, and extended to later nuclear weapons; the book Atomic Audit seems likely to contain useful resources.17
Year Description of explosive Cost per ton TNT equivalent 1920 Ammonium nitrate $5.6k 1920 TNT $10.5k 1946 9 (Mark 1 and Mark 3's) x 20kt (marginal) $16.8k (marginal Mark 3) 1946 9 (Mark 1 and Mark 3's) x 20kt (average) $111k (average Mark 3)  3,000 weapons in the 3+2 plan $400
Table 2: Total, average and marginal costs associated with different weapons arsenals
Figure 4: Cost-effectiveness of nuclear weapons
Cost-effectiveness of non-nuclear weapons
We have found little information about the cost of pre-nuclear bombs in the early 20th Century. However what we have (explained below) suggests they cost a comparable amount to nuclear weapons, for a certain amount of explosive energy.
Ammonium nitrate and TNT appear to be large components of many high explosives used in WWII. For instance, blockbuster bombs were apparently filled with amatol, which is a mixture of TNT and ammonium nitrate.18
An appropriations bill from 1920 (p289) suggests that the 1920 price of ammonium nitrate was about $0.10-0.16 per pound, which is about $1.18 per pound in 2014.19 It suggests TNT was $0.44 per pound, or around $5.20 per pound in 2014. These estimates are consistent with that of a Quora commenter.20
This puts TNT at $10.4k/ton: very close to the $16.8k/ton marginal cost of an equivalent energy from Mark 3 nuclear weapons, and well below the average cost of Mark 3 nuclear weapons produced by the end of Operation Crossroads.
Ammonium nitrate is about half as energy dense as TNT, suggesting a price of about $5.6k/T.21 22 This is substantially lower than the marginal cost of the Mark 3.
Note that these figures are for explosive material only, whereas the costs of nuclear weapons used here are more inclusive. Ammonium nitrate may be far from the most expensive component of amatol-based explosives, and so what we have may be a very substantial underestimate for the price of conventional explosives. There is also some error from synergy between the components of amatol.23
Discontinuity Measurement
Without a longer-run price trend in explosives, we do not have enough pre-discontinuity data to measure a discontinuity.24 However, from the evidence we have here, it is unclear that nuclear weapons represent any development at all in cost-effectiveness, in terms of explosive power per dollar. Thus it seems unlikely that nuclear weapons were surprisingly cost-effective, at least on that metric.
Notes


AGI-09 Survey

Baum et al. surveyed 21 attendees of the AGI-09 conference, on AGI timelines with and without extra funding. They also asked about other details of AGI development such as social impacts, and promising approaches.
Their findings include the following:

The median dates when participants believe there is a 10% , 50% and 90% probability that AI will pass a Turing test are 2020, 2040, and 2075 respectively.
Predictions changed by only a few years when participants were asked to imagine $100 billion (or sometimes $1 billion, due to a typo) in funding.
There was apparently little agreement on the ordering of milestones ('turing test', 'third grade', 'Nobel science', 'super human'), except that 'super human' AI would not come before the other milestones.
A strong majority of participants believed 'integrative designs' were more likely to contribute critically to creation of human-level AGI than narrow technical approaches.

Details
Detailed results
Median confidence levels for different milestones
Table 1 shows median dates given for different confidence levels of AI reaching four benchmarks: able to pass an online third grade test, able to pass a Turing test, able to produce science that would win a Nobel prize, and 'super human'.

Best guess times for various milestones
Figure 2 shows the distribution of participants' best guesses – probably usually interpreted as 50 percent confidence points – for the timing of these benchmarks, given status quo levels of funding.

Individual confidence intervals for each milestone
Figure 4 shows all participants' confidence intervals for all benchmarks. Participant 17 appears to be interpreting 'best guess' as something other than fiftieth percentile of probability, though the other responses appear to be consistent with this interpretation.

Expected social impacts
Figure 6 illustrates responses to three questions about social impact. The participants were asked about the probability of negative social impact, if the first AGI that can pass the Turing test is created by an open source project, by the United States military, or by a private company focused on commercial profit. The paper summarises that the experts lacked consensus.
'Fig. 6. Probability of a negative-to-humanity outcome for different development scenarios. The three development scenarios are if the first AGI that can pass the Turing test is created by an open source project (x's), the United States military (squares), or a private company focused on commercial profit (triangles). Participants are displayed in the same order as in figure 4, such that Participant 1 in figure 6 is the same person as Participant 1 in figure 4.'
Methodological details
The survey contained a set of standardized questions, plus individualized followup questions. It can be downloaded from here.
It included questions on:

when AI would meet certain benchmarks (passing third grade, turing test, Nobel quality research, superhuman), with and without billions of dollars of additional funding. Participants were asked for confidence intervals (10%, 25%, 75%, 90%) and 'best estimates' (interpreted above as 50% confidence levels).
Embodiment of the first AGIs (physical, virtual, minimal)
What AI software paradigm the first AGIs would be based on (formal neural networks, probability theory, uncertain logic, evolutionary learning, a large hand-coded knowledge-base, mathematical theory, nonlinear dynamical systems, or an integrative design combining multiple paradigms)
Probability of strongly negative-to-humanity outcome if the first AGIs were created by different parties (an open-source project, the US military, or a private for-profit software company)
If quantum computing or hypercomputing would be required for AGI.
Whether brain emulations would be conscious
The experts' area of expertise

Participants
Most of the participants were actively involved in AI research. The paper describes them:
Study participants have a broad range of backgrounds and experience, all with significant prior thinking about AGI. Eleven are in academia, including six Ph.D. students, four faculty members, and one visiting scholar, all in AI or allied fields. Three lead research at independent AI research organizations and three do the same at information technology organizations. Two are researchers at major corporations. One holds a high-level administrative position at a relevant non-profit organization. One is a patent attorney. All but four participants reported being actively engaged in conducting AI research.
According to the website, the AGI-09 conference gathers "leading academic and industry researchers involved in serious scientific and engineering work aimed directly toward the goal of artificial general intelligence". While these people are expert in the field, they are also probably highly selected for being optimistic about the timing of human-level AI. This seems likely to produce some bias.
Meaning of 'Turing test'
Several meanings of 'Turing test' are prevalent, and it is unclear what distribution of them is being used by participants. The authors note that some participants asked about this ambiguity, and were encouraged verbally to consider the 'one hour version' instead of the 'five minute version', because the shorter one might be gamed by chat-bots (p6). The authors also write, 'Using human cognitive development as a model, one might think that being able to do Nobel level science would take much longer than being able to conduct a social conversation, as in the Turing Test' (p8). Both of these points suggest that the authors at least were thinking of a Turing test as a test of normal social conversation rather than a general test of human capabilities as they can be observed via a written communication channel.

AGI-09 Survey


Bainbridge Survey

A survey of twenty-six technology experts in 2005 produced a median of 2085 as the year in which artificial intelligence would be able to functionally replicate a human brain (p344). They rated this application 5.6/10 in beneficialness to humanity.
Details
In 2005 William Bainbridge reported on a survey of 26 contributors to Converging Technologies reports. The contributors were asked when a large number of applications would be developed, and how beneficial they would be (see Appendix 1). The survey produced 2085 as the median year in which "the computing power and scientific knowledge will exist to build machines that are functionally equivalent to the human brain" (p344). The participants rated this development 5.6 out of 10 in beneficialness.
Participants
Bainbridge's participants are contributors to 'converging technology' reports, which are on topics of nanotechnology, biotechnology, information technology, and cognitive science. From looking at what appears to be one of these reports, these seem to be mostly experts from government and national laboratories, academia, and the private sector. Few work in AI in particular. For instance, an arbitrary sample includes the Director of the Division of Behavioral and Cognitive Sciences at the National Science Foundation, a person from the Defense Threat Reduction Agency, and a person from HP laboratories.

Bainbridge Survey


AI@50 Survey

A seemingly informal seven-question poll was taken of participants at the AI@50 conference in 2006. 41% of respondents said it would take more than 50 years for AI to simulate every aspect of human intelligence, and 41% said it would never happen.
Details
AI timelines question
One question was "when will computers be able to simulate every aspect of human intelligence?" 41% of respondents said "More than 50 years" and 41% said "Never".
Participants and interpretation
We do not know how many people participated in the conference or in the poll.
Luke Muehlhauser points out that many of the respondents were probably college students, rather than experts, and that the question may have been interpreted as asking in part about the possibility of machine consciousness.
Records of the poll
Information about the poll was available at http://web.archive.org/web/20110710193831/http://www.engagingexperience.com/ai50/ when we put up this page, but it has since become inaccessible. Secondary descriptions of parts of it exist at https://intelligence.org/2013/05/15/when-will-ai-be-created/#footnote_2_10199 and http://sethbaum.com/ac/2011_AI-Experts.pdf

AI@50 Survey


Early Views of AI

This is an incomplete list of early works we have found discussing AI or AI related problems.
List
1. Claude Shannon (1950), in Programming a Computer for Playing Chess, offers the following list of "possible developments in the immediate future,"

Machines for designing filters, equalizers, etc
Machines for designing relay and switching circuits
Machines which will handle routing of telephone calls based on the individual circumstances rather than by fixed patterns
Machines for performing symbolic (non-numerical) mathematical operations
Machines capable of translating from one language to another
Machines for making strategic decisions in simplified military operations
Machines capable of orchestrating a melody
Machines capable of logical deduction

2. The proposal for Dartmouth conference on AI offers the following "aspects of the artificial intelligence project":

Automatic computers. This appears to be an application rather than an aspect of the problem; if you can describe how to do a task precisely, it can be automated.
How Can a Computer be Programmed to Use a Language
How can a set of (hypothetical) neurons be arranged so as to form concepts
Theory of the size of a calculation
Self-improvement
Abstractions. "A direct attempt to classify these and to describe machine methods of forming abstractions from sensory and other data would seem worthwhile."
Randomness and creativity

Early Views of AI


FHI Winter Intelligence Survey

The Future of Humanity Institute administered a survey in 2011 at their Winter Intelligence AGI impacts conference. Participants' median estimate for a 50% chance of human-level AI was 2050.
Details
AI timelines question
The survey included the question: "Assuming no global catastrophe halts progress, by what year would you assign a 10%/50%/90% chance of the development of human-level machine intelligence? Feel free to answer 'never' if you believe such a milestone will never be reached."
The first quartile / second quartile / third quartile responses to each of these three questions were as follows:
10% chance: 2015 / 2028 / 2030
50% chance: 2040 / 2050 / 2080
90% chance: 2100 / 2150 / 2250
Participants and selection effects
Survey participants probably expect AI sooner than comparably expert groups, by virtue of being selected from participants at the Winter Intelligence conference. The conference is described as focussing on "artificial intelligence and the impacts it will have on the world," a topic of disproportionately great natural interest to researchers who believe that AI will substantially impact the world soon. The response rate to the survey was 41% (35 respondents), limiting response bias.
When asked "Prior to this conference, how much have you thought about these issues?" the respondents were roughly evenly divided between "Significant interest," "Minor research focus / sustained study," and "major research focus."
When asked to describe their field, of the 35 respondents, 22% indicated an area that the survey administrators considered to be "AI and Robotics" as their field, 22% indicated a field considered to be "computer science and engineering," and the remainder indicated a variety of fields with less direct relevant to AI progress (excepting perhaps cognitive science and neuroscience, whose prevalence the authors do not report). The administrators of the survey write:
"There were no significant (as per ANOVA) inter-group differences in regards to who would develop AI, the outcomes, type of AI, expertise, or likelihood of Watson winning. Merging the AI and computer science group and the philosophy and general academia group did not change anything: participant views did not link strongly to their formal background. " (p. 10).

FHI Winter Intelligence Survey


Hanson AI Expert Survey

In a small informal survey running since 2012, AI researchers generally estimated that their subfields have moved less than ten percent of the way to human-level intelligence. Only one (in the slowest moving subfield) observed acceleration.
This suggests on a simple extrapolation that reaching human-level capability across subfields will take over a century (in contrast with many other predictions).
Details
Robin Hanson has asked experts in various social contexts to estimate how far we've come in their own subfield of AI research in the last twenty years, compared to how far we have to go to reach human level abilities. His results are listed in Table 1. He points out that on an outside view calculation, this suggests at least a century until human-level AI.

Year added to list
 Person
Subfield
Distance in 20y
 Acceleration

 2012
 A few UAI attendees

 5-10%
 ~0

 2012
 Melanie Mitchell
 Analogical reasoning
 5%
 ~0

 2012
 Murray Shanahan
 Knowledge representation
 10%
 ~0

 2013
 Wendy Hall
 Computer-assisted training
 1%

 2013
 Claire Cardie (and Peter Norvig agrees in '14)
 Natural language processing
 20%

 2013
 Boi Faltings (and Peter Norvig agrees in '14)
 Constraint satisfaction
 Past human-level 20 years ago

 2014
 Aaron Dollar
 robotic grasping manipulation
 <1%
 positive

 2014
 Peter Norvig
*

 2014
 Timothy Meese
 early human vision processing
 5%
 negative

 2015
 Francesca Rossi
constraint reasoning
 10%
 negative

 2015
 Margret Boden
 no particular subfield
 5%

 2015
 David Kelley
 big data analysis
 5%
 positive

 2016
 Henry Kautz
 constraint satisfaction
 >100%

 2016
 Henry Kautz
 language
 10%
 positive

 2016
 Jeff Legault
 robotics
 5%
 positive

 2017
 Thore Husfeldt
 human-understandable explanation
 <0.5%

Table 1 : Results from Robin Hanson's informal survey
*Hanson's summary of Peter Norvig's response seems hard to fit into this framework:
After coming to a talk of mine, Peter Norvig told me that he agrees with both Claire Cardie and Boi Faltings, that on speech recognition and machine translation we've gone from not usable to usable in 20 years, though we still have far to go on deeper question answering, and for retrieving a fact or page that is relevant to a search query we've far surpassed human ability in recall and do pretty well on precision.

Hanson AI Expert Survey


Müller and Bostrom AI Progress Poll

Vincent Müller and Nick Bostrom of FHI conducted a poll of four groups of AI experts in 2012-13. Combined, the median date by which they gave a 10% chance of human-level AI was 2022, and the median date by which they gave a 50% chance of human-level AI was 2040.
Details
According to Bostrom, the participants were asked when they expect "human-level machine intelligence" to be developed, defined as "one that can carry out most human professions at least as well as a typical human". The results were as follows. The groups surveyed are described below.

 Response rate
10%
50%
90%

 PT-AI
 43%
2023
2048
2080

 AGI
 65%
2022
2040
2065

 EETN
 10%
2020
2050
2093

 TOP100
 29%
2022
2040
2075

 Combined
 31%
2022
2040
2075

Figure 1: Median dates for different confidence levels for human-level AI, given by different groups of surveyed experts (from Bostrom, 2014).
Surveyed groups:
PT-AI: Participants at the 2011 Philosophy and Theory of AI conference (88 total). By the list of speakers, this appears to have contained a fairly even mixture of philosophers, computer scientists and others (e.g. cognitive scientists). According to the paper, they tend to be interested in theory, to not do technical AI work, and to be skeptical of AI progress being easy.
AGI: Participants at the 2012 AGI-12 and AGI Impacts conferences (111 total). These people mostly do technical work.
EETN: Members of the Greek Association for Artificial Intelligence, which only accepts published AI researchers (250 total).
TOP100: The 100 top authors in artificial intelligence, by citation, in all years, according to Microsoft Academic Search in May 2013. These people mostly do technical AI work, and tend to be relatively old and based in the US.

Müller and Bostrom AI Progress Poll


Kruel AI Interviews

Alexander Kruel interviewed 37 experts on areas related to AI, starting in 2011 and probably ending in 2012. Of those answering the question in a full quantitative way, median estimates for human-level AI (assuming business as usual) were 2025, 2035 and 2070 for 10%, 50% and 90% probabilities respectively. It appears that most respondents found human extinction as a result of human-level AI implausible.
Details
AI timelines question
Kruel asked each interviewee something similar to "Assuming beneficial political and economic development and that no global catastrophe halts progress, by what year would you assign a 10%/50%/90% chance of the development of artificial intelligence that is roughly as good as humans (or better, perhaps unevenly) at science, mathematics, engineering and programming?" Twenty respondents gave full quantitative answers. For those, the median estimates were 2025, 2035 and 2070 for 10%, 50% and 90% respectively, according to this spreadsheet (belonging to Luke Muehlhauser).
AI risk question
Alexander asked each interviewee something like:
'What probability do you assign to the possibility of human extinction as a result of badly done AI?
Explanatory remark to Q2:
P(human extinction | badly done AI) = ?
(Where 'badly done' = AGI capable of self-modification that is not provably non-dangerous.)
An arbitrary selection of (abridged) responses; parts that answer the question relatively directly are emboldened:

Brandon Rohrer: <1%
Tim Finin: .001
Pat Hayes: Zero. The whole idea is ludicrous.
Pei Wang: I don't think it makes much sense to talk about "probability" here, except to drop all of its mathematical meaning…
J. Storrs Hall: …unlikely but not inconcievable. If it happens…it will be because the AI was part of a doomsday device probably built by some military for "mutual assured destruction", and some other military tried to call their bluff. …
Paul Cohen: From where I sit today, near zero….
William Uther: …Personally, I don't think 'Terminator' style machines run amok is a very likely scenario….
Kevin Korb: …we have every prospect of building an AI that behaves reasonably vis-a-vis humans, should we be able to build one at all…
The ability of humans to speed up their own extinction will, I expect, not be matched any time soon by machine, again not in my lifetime
Michael G. Dyer: Loss of human dominance is a foregone conclusion (100% for loss of dominance)…As to extinction, we will only not go extinct if our robot masters decide to keep some of us around…
Peter Gacs: …near 1%…

Interviewees
The MIRI dataset (to be linked soon) contains all of the 'full' predictions mentioned above, and seven more from the Kruel interviews that had sufficient detail for its purposes. Of those 27 participants, we class 10 as AGI researchers, 13 as other AI researchers, 1 as a futurist, and 3 as none of the above.

Kruel AI Interviews


Klein AGI Survey

Futurist Bruce Klein ran an informal online survey in 2007, asking 'When will AI surpass human-level intelligence?". He got 888 responses, from 'friends' of unspecified nature.
Details
The results are shown below, taken from Baum et al, p4. Roughly 50% of respondents gave answers before 2050.

Klein AGI Survey


Similarity Between Historical and Contemporary AI Predictions

AI predictions from public statements made before and after 2000 form similar distributions. Such predictions from before 1980 appear to be more optimistic, though predictions from a larger early survey are not.
Discussion
Similarity of predictions over time
MIRI dataset
We compared early and late predictions using the MIRI dataset. We find the correlation between the date of a prediction and number of years until AI is predicted from that time is 0.13. Most predictions are in the last decade or two however, so this does not tell us much about long run trends (see Figure 1).
Figure 1: Years until AI is predicted (minPY) over time in MIRI AI predictions dataset.
The six predictions prior to 1980 were all below the median 30 years, which would have less than 2% chance if they were really drawn from the same distribution.
The predictions made before and after 2000 form very similar distributions however (see figure 2). The largest difference between the fraction of pre-2000 and since-2000 people who predict AI by any given distance in the future is about 15%. A difference this large is fairly likely by chance, according to our unpublished calculations. See the MIRI dataset page for further details.
Figure 2: Cumulative probability of prediction falling less than X years from date of writing (minPY; from MIRI AI predictions dataset)
Survey data
The surveys we know of provide some evidence against early predictions being more optimistic. Michie's survey is the only survey we know of made more than ten years ago. Ten surveys have been made since 2005 that give median predictions or median fiftieth percentile dates. The median date in Michie's survey is the third furthest out in the set of eleven: fifty years, compared to common twenty to forty year medians now. Michie's survey does not appear to have involved options between twenty and fifty years however, making this result less informative. However it suggests the researchers in the survey were not substantially more optimistic than researchers in modern surveys. They were also apparently more pessimistic than the six early statements from the MIRI dataset discussed above, though some difference should be expected from comparing a survey with public statements. Michie's survey had sixty-three respondents, compared to the MIRI dataset's six, making this substantial evidence.
Figure 3: Survey results, as shown in Michie's paper.
Armstrong and Sotala on failed past predictions
Armstrong and Sotala compare failed predictions of the past to all predictions, in an older version of the same dataset. This is not the same as comparing historical to contemporary predictions, but related. In particular, they use a subset of past predictions to conclude that contemporary predictions are likely to be similarly error-prone.
They find that predictions which are old enough to be proved wrong form a similar distribution to the entire set of predictions (see Figures 5-6). They infer that recent predictions are likely to be flawed in similar ways to the predictions we know to be wrong.
This inference appears to us to be wrong, due to a selection bias. Figure 4 illustrates how such reasoning fails with a simple scenario where the failure is clear. In this scenario, for thirty years people have divided their predictions between ten, twenty and thirty years out. In 2010 (the fictional present), many of these predictions are known to have been wrong (shown in pink). In order for the total distribution of predictions to match that of the past failed predictions, the modern predictions have to form a very distribution from the historical predictions (e.g. one such distribution is shown in the '2010' row).
Figure 4: a hypothetical distribution of predictions.
In general, failed predictions are disproportionately short early predictions, and also disproportionately short bad predictions. If the distributions of failed and total predictions look the same, this suggests that the distribution of early and late predictions is not the same – the later predictions must include fewer longer predictions, to make up for the longer unfalsified predictions inherited from the earlier dates, as well as the longer predictions effectively missing from the earlier dates.
If the characteristic lengths of the predictions was small relative to the time between different predictions, these biases would be small. In the example in figure 4, if the distance between the groups had been one hundred years, there would be no problem. However in the MIRI dataset, both are around twenty years.
In sum, the fact that failed predictions look like all predictions suggests that historical predictions came from a different distribution to present predictions. Which would seem to be good news about present predictions, if past predictions were bad. However, we have other evidence from comparing predictions from before and after 2000 directly that those are fairly similar (see above), so if earlier methods were unsuccessful, this is some mark against current methods. On the other hand, public predictions from before 1980 appear to be systematically more optimistic.
Figure 5: Figures from Armstrong and Sotala
Figure 6: Figure from Armstrong and Sotala
Implications
Accuracy of AI predictions: if people make fairly similar predictions over time, this is some evidence that they are not making their predictions based on information about their environment, which has changed over the decades (at a minimum, time has passed). For instance, some suspect that people make their predictions a fixed number of years into the future, to maximize their personal benefits from making exciting but hard to verify predictions. Evidence for this particular hypothesis seems weak to us, however the general point stands. This evidence against accuracy is not as strong as it may seem however, since there appear to be reasonable prior distributions over the date of AI which look the same after seeing time pass.


Human-Level AI

Published 23 January 2014, last updated Aug 7 2022
'Human-level AI' refers to AI which can reproduce everything a human can do, approximately. Several variants of this concept are worth distinguishing.
Details
Variations in the meaning of 'human-level AI'
Considerations in specifying 'human-level AI' more precisely:

Do we mean to imply anything about running costs? Is an AI that reproduces human behavior for ten billion dollars per year 'human-level', or does it need to be human-priced? See 'human-level at any cost vs. human-level at human cost' below for more details.
What characteristics of a human need to be reproduced? Usually we do not mean that the AI should be indistinguishable from a human. For instance, we usually do not care whether it looks like a human. A common requirement is that the AI have the economically valuable skills of a human. We sometimes also talk about AI being 'human-level' in a narrower set of relevant characteristics, such as in its ability to do further AI research.
What does it mean to reproduce human behavior? If AI replaces all hairdressers in society, but uniformly produces a slightly worse haircut in some dimensions (but so cheaply!), does that count as 'human-level'? If not, then all humans may be replaced even though AI is not 'human-level'. On the other hand, if this does count, then where is the line? Can an AI 'reproduce human behavior' by merely producing anything a buyer would prefer to have than what a human produces? Many machines already do this, and this is not what we mean.
How much of human behavior needs to be reproduced? If AI cannot entirely compete with humans for the job of waiter, merely because some small population prefers human waiters, this will not make a large difference to anything, so a requirement that human-level AI replace humans in all economically useful skills is too high a bar for what we are intuitively interested in. There is a further question of what metric one might use when specifying the bar.
What conditions is the AI available under? Does it matter if it has actually been built? Need it be available in some particular marketplace, or quantity, or price range?

Related definitions
High Level Machine Intelligence (HLMI)
The 2016 and 2022 Expert Surveys on Progress in AI use the following definition:
Say we have 'high-level machine intelligence' when unaided machines can accomplish every task better and more cheaply than human workers. Ignore aspects of tasks for which being a human is intrinsically advantageous, e.g. being accepted as a jury member. Think feasibility, not adoption.
Superhuman AI
A 'superhuman' system is meaningfully more capable than a human-level system. In practice the first human-level system is likely to be superhuman.
Key issues
Human-level at any cost vs. human-level at human cost
In common usage, 'human-level' AI can mean either AI which can reproduce a human at any cost and speed, or AI which can replace a human (i.e. is as cheap as a human, and can be used in the same situations). Both are relevant for different issues. For instance, the 'at any cost' meaning is important when considering how people will respond to human-level artificial intelligence, or whether a human-level artificial intelligence will use illicit means to acquire resources and cause destruction. Human-level at human cost is the relevant concept when thinking about AI replacing humans in the labor market, the economy growing very fast, or legitimate AI development ramping up into an intelligence explosion.
Today few applications are more than an order of magnitude more expensive to run than a human, suggesting a short time before an AI project came down in price to the cost of a human. However some applications are more expensive, and even if an early AI project were only a few orders of magnitude more expensive than a human per time, it may be much slower. Thus it is hard to make useful inferences about the potential time delay between an arbitrarily expensive human-level AI and an AI which might replace a human, even if we assume hardware continues to fall in price regularly.
'Human-level' is superhuman
As explained at the Superintelligence Reading Group:
Another thing to be aware of is the diversity of mental skills. If by 'human-level' we mean a machine that is at least as good as a human at each of these skills, then in practice the first 'human-level' machine will be much better than a human on many of those skills. It may not seem 'human-level' so much as 'very super-human'.
We could instead think of human-level as closer to 'competitive with a human' – where the machine has some super-human talents and lacks some skills humans have. This is not usually used, I think because it is hard to define in a meaningful way. There are already machines for which a company is willing to pay more than a human: in this sense a microscope might be 'super-human'. There is no reason for a machine which is equal in value to a human to have the traits we are interested in talking about here, such as agency, superior cognitive abilities or the tendency to drive humans out of work and shape the future. Thus we talk about AI which is at least as good as a human, but you should beware that the predictions made about such an entity may apply before the entity is technically 'human-level'.
Example of how the first 'human-level' AI may surpass humans in many ways.

Human-Level AI