List of multipolar research projects

This list currently consists of research projects suggested at the Multipolar AI workshop we held on January 26 2015.
Relatively concrete projects are marked [concrete]. These are more likely to already include specific questions to answer and feasible methods to answer them with. Other 'projects' are more like open questions, or broad directions for inquiry.
Projects are divided into three sections:

Paths to multipolar scenarios
What would happen in a multipolar scenario?
Safety in a multipolar scenario

Order is not otherwise relevant. The list is an inclusive collection of the topics suggested at the workshop, rather than a prioritized selection from a larger list.
Luke Muehlhauser's list of 'superintelligence strategy' research questions contains further suggestions.
List
Paths to multipolar scenarios
1.1 If we assume that AI software is similar to other software, what can we infer from observing contemporary software development? [concrete] For instance, is progress in software performance generally smooth or jumpy? What is the distribution? What are typical degrees of concentration among developers? What are typical modes of competition? How far ahead does the leading team tend to be to their competitors? How often does the lead change? How much does a lead in a subsystem produce a lead overall? How much do non-software factors influence who has the lead? How likely is a large player like Google—with its pre-existing infrastructure—to be the frontrunner in a random new area that they decide to compete in?
A large part of this project would be collecting what is known about contemporary software development. This information would provide one view on how AI progress might plausibly unfold. Combined with several such views, this might inform predictions on issues like abruptness, competition and involved players.
1.2 If the military is involved in AI development, how would that affect our predictions? [concrete] This is a variation on 1.1, and would similarly involve a large component of reviewing the nature of contemporary military projects.
1.3 If industry were to be largely responsible for AI development, how would that affect our predictions? [concrete] This is a variation on 1.2, and would similarly involve a large component of reviewing the nature of contemporary industrial projects.
1.4 If academia were to be largely responsible for AI development, how would that affect our predictions? [concrete] This is a variation on 1.2, and would similarly involve a large component of reviewing the nature of contemporary academic projects.
1.5 Survey AI experts on the likelihood of AI emerging in the military, business or academia, and on the likely size of a successful AI project.  [concrete]
1.6 Identify considerations that might tip us between multipolar and unipolar scenarios.
1.7 To what extent will AGI progress be driven by developing significantly new ideas? 1.1 may bear on this. It could be approached in other ways, for instance asking AI researchers what they expect.
1.8 Run prediction markets on near-term questions, such as rates of AI progress, which inform our long-run expectations. [concrete]
1.9 Collect past records of 'lumpiness' of AI success. [concrete] That is, variation in progress over time. This would inform expectations of future lumpiness, and thus potential for single projects to gain a substantial advantage.
What would happen in a multipolar scenario?
2.1 To what extent do values prevalent in the near-term affect the long run, in a competitive scenario? One could consider the role of values over history so far, or examine the ways in which the role of values may change in the future. One could consider the degree of instrumental convergence between actors (e.g. firms) today, and ask how that affects long-term outcomes. One might also consider whether non-values mental features might become locked in in a way that produces similar outcomes to particular values being influential. e.g. priors or epistemological methods that make a particular religion more likely
2.2 What other factors in an initial scenario are likely to have long-lasting effects? For instance social institutions, standards, and locations for cities.
2.3 What would AI's value in a multipolar scenario? We can consider a range of factors that might influence AI values:

The nature of the transition to AI
Prevailing institutions
The extentto which AI values become static, as compared to changing human values
What values do humans want AI's to have
Competitive dynamics

There is a common view that a multipolar scenario would be better in the long run than a hegemonic 'unfriendly AI'. This project would inform that comparison.
2.4 What are the prospects for human capital-holders? In a simple model, humans who own capital might become very wealthy during a transition to AI. On a classical economic picture, this would be a critical way for humans to influence the future. Is this picture plausible? Evaluate the considerations.

What are the implications of capital holders doing no intellectual work themselves?
[concrete] What does the existing literature on principal-agent problems suggest about multipolar AI scenarios?
[concrete] Could humans maintain investments for significant periods of their lives, if during that time aeons of subjective time passes for faster moving populations? (i.e. is it plausible to expect to hold assets through millions of years of human history?) Investigate this via data on past expropriations

2.5 Identify risks distinctive to a multipolar scenario, or which are much more serious in a multipolar scenario.
For instance:

Evolutionary dynamics bring an outcome that nobody desired initially
The AIs are not well integrated into human society, and consequently cause or allow destruction to human society
The AIs—integrated or not—have different values, and most of the resources end up being devoted to those values

2.6 Choose a specific multipolar scenario and try to predict its features in detail. [concrete] Base this on the basic changes we know would occur (e.g. minds could be copied like software), and our best understanding of social science.
Specific instances:

Brain emulations (Robin Hanson is working on this in an upcoming book)
Brain emulations, without the assumption that software minds are opaque
One can buy maximally efficient software for anything you want; everything else is the same
AI is much like contemporary software (see 1.1).

2.7 How would multipolar AI change the nature and severity of violent conflict? For instance, conflict between states.
2.8 Investigate the potential for AI-enforced rights. Think about how to enforce property rights in a multipolar scenario, given advanced artificial intelligence to do it with, and the opportunity to prepare ahead of time. Can you create programs that just enforce deals between two parties, but do nothing else? If you create AI with this stable motivational structure, possessed by many parties, how does this change the way that agents that interact? How could such a system be designed?
2.9 What is the future of democracy in such a scenario? In a world where resources can rapidly and cheaply be turned into agents, the existing assignment of a vote per person may be destructive and unstable.
2.10 How does the lumpiness of economic outcomes vary as a function of the lumpiness of origins? For instance, if one team creates brain emulations years before others, would that group have and retain extreme influence?
2.11 What externalities can we foresee, in computer security? That is, will people invest less (or more) in security than is socially optimal?
2.12 What externalities can we foresee in AI safety generally?
2.13 To what extent can artificial agents make more effective commitments, or more effectively monitor commitments, than humans? How does this change competitive dynamics? What proofs of properties of one's source code may be available in the future?
Safety in a multipolar scenario
3.1 Assess the applicability of general AI safety insights to multipolar scenarios. [concrete] How useful are capability control methods, such as boxing, stunting, incentives, or tripwires in a multi-polar scenario? How useful are motivation selection methods, such as direct specification, domesticity, indirect normatively, augmentation in a multipolar scenario?
3.2 Would selective pressures strongly favor the existence of goal-directed agents, in a multipolar scenario where a variety of AI designs are feasible?
3.3 Develop a good model for the existing computer security phenomenon where nobody builds secure systems, though they can. [concrete] Model the long-run costs of secure and insecure systems, given distributions of attacker sophistication and possibility for incremental system improvement. Determine the likely situation various future scenarios, especially where computer security is particularly important.
3.4 Do paradigms developed for nuclear security and biological weapons apply to AI in a multi-polar scenario? [concrete] For instance, could similar control and detection systems be used?
3.5 What do the features of computer security systems tell us about how multipolar agents might compete?
3.8 What policies could help create more secure computer systems? For instance, the onus being on owners of systems to secure them, rather than on potential attackers to avoid attacking.
3.9 What innovations (either in AI or coinciding technologies) might reduce principal-agent problems?
3.10 Apply 'reliability theory' to the problem of manufacturing trustworthy hardware.
3.11 How can we transition in an economically viable way to hardware that we can trust is uncorrupted? At present, we must assume that the hardware is uncorrupted upon purchase, but this may not be sufficient in the long run.

List of multipolar research projects


How AI timelines are estimated

A natural approach to informing oneself about when human-level AI will arrive is to check what experts who have already investigated the question say about it. So we made this list of analyses that we could find.
It's a short list, though the bar for 'analysis' was low. Blame for the brevity should probably be divided between our neglect of worthy entries and the world's neglect of worthy research. Nonetheless we can say interesting things about the list.
About half of the estimates are based on extrapolating hardware, usually to something like 'human-equivalence'. A stylized estimate along these lines might run as follows:

Calculate how much computation the brain does.
Extrapolate the future costs for computing hardware (it goes downward, fast)
Find the point in the computing hardware cost trajectory where brain-equivalent hardware (1) becomes pretty cheap, for some value of 'pretty cheap'.
Guestimate how long software will take once we have enough hardware; add this to the date produced in (3).
The date produced in (4) is your estimate for human-level AI.

How solid is this kind of estimate? Let us consider it in a bit of detail.
How much computation is a brain worth?
It is not trivial to estimate how much computation a brain does. A basic philosophical problem is that we don't actually know what the brain is doing much, so it's not obvious what part of its behavior is contributing to computation in any particular way. For instance (implausibly) if some of the neuron firing was doing computation, and the rest was just keeping the neurons prepared, we wouldn't know. We don't know how much detail of the neurons and their contents and surrounds is relevant to the information processing we are interested in.
Moravec (2009) estimates how much computation the brain does by extrapolation from the retina. He estimates how much computing hardware would be needed for a computer to achieve the basic image processing that parts of the retina do, then multiplies this by how much heavier the brain is than parts of the retina. As he admits, this is a coarse estimate. I don't actually have much idea how accurate you would expect this to be. Some obvious possible sources of inaccuracy are the retina being unrepresentative of the brain (as it appears to be for multiple reasons), the retina being capable of more than the processing being replicated by a computer, and mass being poorly correlated with capacity for computation (especially across tissue which is in different parts of the body).
One might straightforwardly improve upon this estimate by extrapolating from other parts of the brain in a similar way, or from calculating how much information could feasibly be communicated in patterns of neuron firing (assuming these were the main components contributing to relevant computation).
The relationship between hardware and software
Suppose that you have accurately estimated how much computation the brain does. The argument above treats this as a lower bound on when human-level intelligence will arrive. This appears to rest on a model in which there is a certain level of hardware and a certain level of software that you need, and when you have them both you will have human-level AI.
In reality, the same behavior can often be achieved with different combinations of hardware and software. For instance (as shown in figure 1), you can achieve the same Elo in Go using top of the range software (MoGo) and not much hardware (enough for 64 simulations) or weak software (FatMan) and much more hardware (enough for 1024 simulations, which probably take less hardware each than those used for the sophisticated program). The horizontal axis is doublings of hardware, but FatMan begins with much more hardware.
Figure 1: Performance of strong Mogo and weak FatMan with successive doublings in hardware. Mogo starts out doing 64 simulations, and FatMan 1024. The horizontal axis is doublings of hardware, and the vertical axis is performance measured in Elo.
In Go we thus have a picture of indifference curves something like this:

Go is not much like general intelligence, but the claim is that software in general has this character. If this is true, it suggests that the first human-level AI designed by human engineers might easily use much more or much less hardware than the human brain. This is illustrated in figure 3. Our trajectory of software and hardware progress could run into the frontier of human-level ability above or below human-level. If our software engineering is more sophisticated than that of evolution at the point where we hit the frontier, we would reach human-level AI with much less than 'human-equivalent' hardware.
Figure 3: two possible trajectories for human hardware and software progress, which achieve human-level intelligence (the curved line) with far more and far less hardware than humans require.
As an aside, if we view the human brain as a chess playing machine, an analog to the argument outlined earlier in the post suggests that we should achieve human-level chess playing at human-equivalent hardware. We in fact achieved it much earlier, because indeed humans can program a chess player more efficiently than evolution did when it programmed humans. This is obviously in part because the human brain was not designed to play chess, and is mostly for other things. However, it's not obvious that the human brain was largely designed for artificial intelligence research either, suggesting economic dynamite such as this might also arrive without 'human-level' hardware.
I don't really know how good current human software engineering is compared to evolution, when they set their minds to the same tasks. I don't think I have particularly strong reason to think they are about the same. Consequently, it seems I don't seem to have strong reason to expect hardware equivalent to the brain is a particularly important benchmark (though if I'm truly indifferent between expecting human engineers to be better or worse, human-level hardware is indeed my median estimate).
Human equivalent hardware might be more important however: I said nothing about how hardware trades off against software. If the frontier of human-level hardware/software combinations is more like that in figure 4 below than figure 3, a very large range of software sophistication corresponds to human-level AI occurring at roughly similar levels of hardware, which means at roughly similar times. If this is so, then the advent of human-level hardware is a good estimate for when AI will arrive, because AI would arrive around then for a large range of  levels of software sophistication.
Figure 4: If it takes a lot of software sophistication to replace a small amount of hardware, the amount of hardware that is equivalent to a human brain may be roughly as much as is needed for many plausible designs.
The curve could also look opposite however, with the level of software sophistication being much more important than available hardware. I don't know of strong evidence either way, so for now the probability of human-level AI at around the time we hit human-level hardware only seems moderately elevated.
The shape of the hardware/software frontier we have been discussing could be straightforwardly examined for a variety of software, using similar data to that presented for Go above. Or we might find that this 'human-level frontier' picture is not a useful model. The general nature of such frontiers seem highly informative about the frontier for advanced AI. I have not seen such data for anything other than parts of it for chess and Go. If anyone else is aware of such a thing, I would be interested to see it.
Costs
Will there be human-level AI when sufficient hardware (and software) is available at the cost of a supercomputer? At the cost of a human? At the cost of a laptop?
Price estimates used in this kind of calculation often seem to be chosen to be conservative—low prices so that the audience can be confident that an AI would surely be built if it were that cheap. For instance, when will human-level hardware be available for $1,000? While this is helpful for establishing an upper bound on the date, it does not seem plausible as a middling estimate. If human-level AI could be built for a million dollars instead of a thousand, it would still be done in a flash, and this corresponds to a difference of around fifteen years.
This is perhaps the easiest part of such an estimate to improve, with a review of how much organizations are generally willing to spend on large, valuable, risky projects.
***
In sum, this line of reasoning seems to be a reasonable start. As it stands, it probably produces wildly inaccurate estimates, and appears to be misguided in its implicit model of how hardware and software relate to each other. However it is a good beginning that could be incrementally improved into a fairly well informed estimate, with relatively modest research efforts. Which is just the kind of thing one wants to find among previous efforts to answer one's question.

How AI timelines are estimated


At-least-human-level-at-human-cost AI

Often, when people are asked 'when will human-level AI arrive?' they suggest that it is a meaningless or misleading term. I think they have a point. Or several, though probably not as many as they think they have.
One problem is that if the skills of an AI are developing independently at different rates, then at the point that an AI eventually has the full kit of human skills, they also have a bunch of skills that are way past human-level. For instance, if a 'human-level' AI were developed now, it would be much better than human-level at arithmetic.
Thus the term 'human-level' is misleading because it invites an image of an AI which competes on even footing with the humans, rather than one that is at least as skilled as a human in every way, and thus what we would usually think of as extremely superhuman.
Another problem is that the term is used to mean multiple things, which then get confused with each other. One such thing is a machine which replicates human cognitive behavior, at any cost. Another is a machine which replicates human cognitive behavior at the price of a human. The former could plausibly be built years before the latter, and should arguably not be nearly as economically exciting. Yet often people imagine the two events coinciding, seemingly for lack of conceptual distinction.
Should we use different terms for human level at human cost, and human level at any cost? Should we have a different term altogether, which evokes at-least-human capabilities? I'll leave these questions to you. For now we just made a disambiguation page.
An illustration I made for the Superintelligence Reading Group

At-least-human-level-at-human-cost AI


Penicillin and syphilis

Penicillin was a hugely important discovery. But was it a discontinuity in the normal progression of research, or just an excellent discovery which followed a slightly less excellent discovery, and so on?
There are several senses in which penicillin might have represented a discontinuity. Perhaps its discovery was a huge leap in effectiveness, saving lives at a rate nobody thought possible. Or it might have been dramatically cheaper than expected. Or it could have heralded a step up in life expectancy, or a step down in disease prevalence.
We investigated this, to add to our list of cases of discontinuous technological progress, or alternatively to our new list of things we have investigated to potentially add to that list, but didn't.
To make life easier, we focused on penicillin as used to remedy syphilis in particular. Our only conscious reason for choosing syphilis was that penicillin is used to cure it. It is also a disease against which penicillin was considered an important step forward.
First, was penicillin a huge step forward in effectiveness, compared to the usual progress? In the field of syphilis prevention, the recent competition was actually tough. A treatment for syphilis developed thirty years earlier was literally nicknamed 'magic bullet' and spawned a nobel prize as well as a frightening looking movie. More quantitatively, existing treatments at the time of penicillin's introduction in the early 40s were apparently 'successful' for about 90% of patients who took them, and it doesn't look like penicillin did better initially. So probably penicillin didn't jump ahead on effectiveness.
However a big difference appears to be hidden within 'patients who took them'. The 'magic bullet' was an arsenic compound, unstable in air, needing frequent injections for weeks, and producing risk to life and limb. According to the only paper I found with figures on this, about three quarter of patients 'defected' before receiving a 'minimum curative dose' of an arsenic and bismuth treatment, which suggests that while earlier cures were not obviously worse than dying of syphilis, they were also not obviously better (or else perhaps people dropped out when the severity of their side effects prohibited them from continuing). Penicillin allowed virtually everyone to get a minimum curative dose. So it's possible that penicillin represented a discontinuity in this more inclusive success measure, but alas we don't have the data to check. If penicillin really treated four times as many patients as recent precursors, and both cured most of them, and many would die otherwise, and little progress had happened on this measure since salvarsan, then penicillin would be worth at least thirty years of progress, and perhaps much more.
(I hear you wondering, if this magic bullet was so much better than its precursors, what on earth were the precursors? Breathing mercury was one. From before the 16th century until the discovery of salvarsan, 'mercury treatments' were common. Early mercury treatments included rubbing mercury into one's body, and inhaling the mercury fumes, while later ones involved more sophisticated injections of mercury compounds).
I should point out that I'm confused by these apparently high defection rates. Syphilis has an untreated mortality rate of 8-58%, so defection from treatment would appear to be living pretty dangerously. This suggests I'm wrong about something, just so you know. Nonetheless, being right about everything doesn't seem cost-effective here.
So far it seems like penicillin's main advantage was in being less costly (including costs such as suffering and complications). Was this an abrupt change? It is hard to get figures for the inclusive costs of penicillin and its precursors, let alone a longer term trend to compare to. We can say that penicillin took around eight days at the very start, while other treatments took more than twenty. And penicillin seems to have been a lot less dangerous. However qualitatively speaking, salvarsan also sounds a lot less dangerous than mercury treatments. There were also further improvements in safety prior to penicillin, such as in neosalvarsan. So guessing qualitatively, penicillin might have been a few decades worth of previous progress in reducing costs, but probably not much more.
Even at the height of syphilis, the disease was not common enough that resolving all of it in one year would produce a visible change to life expectancy, so we shan't look at that.
Did syphilis rates decline abruptly? Both having syphilis and dying from it became much less common during the 1940s, which is probably due in large part to antibiotics (see figures 1 and 2). Deaths from syphilis declined by 98%. However, as you can see, these were not abrupt changes. Things got modestly better every year for decades.
Figure 1: Syphilis infection in the US
Figure 2: syphilis deaths declined massively in the middle of last century.
In sum, it seems unlikely that there was abrupt progress on drug effectiveness conditional on completing the treatment, or on how many people had or died from syphilis. There may have been abrupt progress in overall costs or in drug effectiveness conditional on being offered the treatment, but these have been too hard to evaluate here.
So for now, we add it to the list of things we checked, but not the list of things that were abrupt. The list of things we checked is part of a larger page about this project, in case you are curious. We also looked into the Haber process recently, but didn't think it involved much discontinuity. We might blog about it at some point.
Some other interesting facts about syphilis and penicillin I learned:

A pre-penicillin treatment for neurosyphilis was to give the patient malaria, because malaria mitigated the syphilis, and was considered more treatable than syphilis.
Early on, penicillin was so valuable that doctors recycled what hadn't been metabolized by extracting it from patients' urine.
There is a whole wikipedia page of previous occasions in history where people observed that mould prevented bacterial infection, more or less, but apparently didn't follow up that much.

Penicillin and syphilis


Discontinuous progress investigation

Published Feb 2, 2015; last substantially updated April 12 2020
We have collected cases of discontinuous technological progress to inform our understanding of whether artificial intelligence performance is likely to undergo such a discontinuity. This page details our investigation.
We know of ten events that produced a robust discontinuity in progress equivalent to more than a century at previous rates in at least one interesting metric and 53 events that produced smaller or less robust discontinuities.
Details
Motivations
We are interested in learning whether artificial intelligence is likely to see discontinuous progress in the lead-up to human-level capabilities, or to produce discontinuous change in any other socially important metrics (e.g. percent of global wealth possessed by a single entity, economic value of hardware). We are interested because we think this informs us about the plausibility of different future scenarios and about which research and other interventions are best now, and also because it is a source of disagreement, and so perhaps fruitful for resolution.1
We seek to answer this question by investigating the prevalence and nature of discontinuities in other technological progress trends. The prevalence can then act as a baseline for our expectations about AI, which can be updated with any further AI-specific evidence, including that which comes from looking at the nature of other discontinuities (for instance, whether they arise in circumstances that are predicted by the arguments that are made for predicting discontinuous progress in AI).
In particular, we want to know:
How common are large discontinuities in metrics related to technological progress?Do any factors predict where such discontinuities will arise? (For instance, is it true that progress in a conceptual endeavor is more likely to proceed discontinuously? If there have been discontinuities in progress on a metric in the past, are further discontinuities more likely?)
As a secondary goal, we are interested in learning about the circumstances that have surrounded discontinuous technological change in the past, insofar as it may inform our expectations about the consequences of discontinuous progress in AI, should it happen.
Methods
Main article: methodology for discontinuous progress investigation.
To learn about the prevalence and nature of discontinuities in technological progress, we:
Searched for potential examples of discontinuous progress (e.g. 'Eli Whitney's cotton gin') via our own understanding, online search, and suggestions from others.2Chose specific metrics related to these potential examples (e.g. 'cotton ginned per person per day', 'value of cotton ginned per cost') and found historic data on progress on those metrics (usually in conjunction with choosing metrics, since metrics for which we can find data are much preferred). Some datasets we found already formed in one place, while others we collected ourselves from secondary sources.Defined a 'rate of past progress' throughout each historic dataset (e.g. if the trend is broadly flat then gets steeper, we decide whether to call this exponential progress, or two periods of linear growth.)Measured the discontinuity at each datapoint in each trend by comparing the progress at the point to the expected progress at that point based on the last datapoint and the rate of past progress (e.g. if the last datapoint five years ago was 600 units, and progress had been going at two units per year, and now a development took it to 800 units, we would calculate 800 units – 600 units = 200 units of progress = 100 years of progress in 5 years, for a 95 year discontinuity.)Noted any discontinuities of more than ten years ('moderate discontinuities'), and more than one hundred years ('large discontinuities')Judged subjectively whether the discontinuity was a clear divergence from the past trend (i.e. the past trend was well-formed enough that the new point actually seemed well outside of plausible continuations of it).3Noted anything interesting about the circumstances of each discontinuity (e.g. the type of metric it was in, the events that appeared to lead to the discontinuity, the patterns of progress around it.)
Note that this is not an attempt to rigorously estimate the frequency of discontinuities in arbitrary trends, since we have not attempted to select arbitrary trends. We have instead selected trends we think might contain large discontinuities. Given this, it may be used as a loose upper bound on the frequency of discontinuities in similar technological trends.
It is likely that there are many minor errors in this collection of data and analysis, based on the rate at which we have found and corrected them, and the unreliability of sources used.
Definitions
Throughout, we use:
Discontinuity: abrupt progress far above what one would have expected by extrapolation, measured in terms of how many years early the progress appeared relative to its expected date.Moderate discontinuity: 10-100 years of progress at previous rates occurred on one occasionLarge discontinuity: at least 100 years of progress at previous rates occurred on one occasionSubstantial discontinuity: a moderate or large discontinuityRobust discontinuity: a discontinuity judged to involve a clear divergence from the past trend
Summary figures
We collected 21 case studies of potentially discontinuous technological progress (see Case studies below) and investigated 38 trends associated with them.20 trends had a substantial discontinuity, and 15 had a large discontinuity.4 We found 88 substantial discontinuities, 39 of them large.These discontinuities were produced by 63 distinct eventsTen events produced robust large discontinuities in at least one metric.
Case studies
This is a list of areas of technological progress which we have tentatively determined to either involve discontinuous technological progress, or not. Note that we largely investigate cases that looked likely to be discontinuous.
Ship size
Main article: Historic trends in ship size
Trends for ship tonnage (builder's old measurement) and ship displacement for Royal Navy first rate line-of-battle ships saw eleven and six discontinuities of between ten and one hundred years respectively during the period <PHONE>, if progress is treated as linear or exponential as usual. There is a hyperbolic extrapolation of progress such that neither measurement sees any discontinuities of more than ten years.
We do not have long term data for ship size in general, however the SS Great Eastern seems to have produced around 400 years of discontinuity in both tonnage (BOM) and displacement if we use Royal Navy ship of the line size as a proxy, and exponential progress is expected, or 11 or 13 in the hyperbolic trend.
Figure 1a: Record tonnages for Royal Navy ships of the line
Figure 1b: Ship weight (displacement) over time, Royal Navy ships of the line and the Great Eastern, a discontinuously large civilian ship. The largest ship in the world three years prior to the Great Eastern was around 4% larger than the Ship of the Line of that time in this figure, so we know that the overall largest ship trend cannot have been much steeper than the Royal Navy ship of the line trend shown.
Image recognition
Main article: Effect of AlexNet on historic trends in image recognition
AlexNet did not represent a greater than 10-year discontinuity in fraction of images labeled incorrectly, or log or inverse of this error rate, relative to progress in the past two years of competition data.
Figure 2: Error rate (%) of ImageNet competitors from 2010 – 2012
Transatlantic passenger travel
Main article: Historic trends in transatlantic passenger travel
The speed of human travel across the Atlantic Ocean has seen at least seven discontinuities of more than ten years' progress at past rates, two of which represented more than one hundred years' progress at past rates: Columbus' second journey, and the first non-stop transatlantic flight.
Figure 3a: Historical fastest passenger travel across the Atlantic (speeds averaged over each transatlantic voyage)
 Figure 3b: Previous figure, shown since 1730
Transatlantic message speed
Main article: Historic trends in transatlantic message speed
The speed of delivering a short message across the Atlantic Ocean saw at least three discontinuities of more than ten years before 1929, all of which also were more than one thousand years: a 1465-year discontinuity from Columbus' second voyage in 1493, a 2085-year discontinuity from the first telegraph cable in 1858, and then a 1335-year discontinuity from the second telegraph cable in 1866.
 Figure 4: Average speed for message transmission across the Atlantic.
Long range military payload delivery
Main article: Historic trends in long range military payload delivery
The speed at which a military payload could cross the Atlantic ocean contained six greater than 10-year discontinuities in 1493 and between 1841 and 1957:
DateMode of transportKnotsDiscontinuity size(years of progress at past rate)1493Columbus' second voyage5.814651884Oregon18.6101919WWI Bomber (first non-stop transatlantic flight)1063511938Focke-Wulf Fw 200 Condor174191945Lockheed Constellation288251957R-7 (ICBM)~10,000~500
Figure 5: Historic speeds of sending hypothetical military payloads across the Atlantic Ocean
Bridge spans
Main article: Historic trends in bridge span length
We measure eight discontinuities of over ten years in the history of longest bridge spans, four of them of over one hundred years, five of them robust as to slight changes in trend extrapolation.
Figure 6: Record bridge span lengths for five bridge types since 1800
Light intensity
Main article: Historic trends in light intensity
Maximum light intensity of artificial light sources has discontinuously increased once that we know of: argon flashes represented roughly 1000 years of progress at past rates.
Figure 7: Light intensity trend since 1800 (longer trend available)
Book production
Main article: Historic trends in book production
The number of books produced in the previous hundred years, sampled every hundred or fifty years between 600AD to 1800AD contains five greater than 10-year discontinuities, four of them greater than 100 years.  The last two follow the invention of the printing press in 1492.
The real price of books dropped precipitously following the invention of the printing press, but the longer term trend is sufficiently ambiguous that this may not represent a substantial discontinuity.
The rate of progress of book production changed shortly after the invention of the printing press, from a doubling time of 104 years to 43 years.
Figure 8a: Total book production in Western Europe
Figure 8b: Real price of books in England
Telecommunications performance
Main article: Historic trends in telecommunications performance
There do not appear to have been any greater than 10-year discontinuities in telecommunications performance, measured as:
bandwidth-distance product for all technologies 1840-2015bandwidth-distance product for optical fiber 1975-2000total bandwidth across the Atlantic <PHONE>
Radio does not seem likely to have represented a discontinuity in message speed.
Figure 9a: Growth in bandwidth-distance product across all telecommunications during <PHONE> from Agrawal, 20165
Figure 9b:  Bandwidth-distance product in fiber optics alone, from Agrawal, 20166 (Note: 1 Gb = 10^9 bits)
Figure 9c: Transatlantic cable bandwidth of all types. Pre-1980 cables were copper, post-1980 cables were optical fiber.
Cotton gins
Main article: Effect of Eli Whitney's cotton gin on historic trends in cotton ginning
We estimate that Eli Whitney's cotton gin represented a 10 to 25 year discontinuity in pounds of cotton ginned per person per day, in 1793. Two innovations in 1747 and 1788 look like discontinuities of over a thousand years each on this metric, but these could easily stem from our ignorance of such early developments. We tentatively doubt that Whitney's gin represented a large discontinuity in the cost per value of cotton ginned, though it may have represented a moderate one.
Figure 10: Claimed cotton gin productivity figures, 1720 to modern day, coded by credibility and being records. The last credible best point before the modern day is an improved version of Whitney's gin, two years after the original (the original features in the two high non-credible claims slightly earlier).
Altitude
Main article: Historic trends in altitude
Altitude of objects attained by man-made means has seen six discontinuities of more than ten years of progress at previous rates since 1783, shown below.
YearHeight (m)Discontinuity (years)Entity178440001032Balloon180372801693Balloon191842,300227Paris gun194285,000120V-2 Rocket1944174,60011V-2 Rocket1957864,000,00035Pellets (after one day)
Figure 11: Post-1750 altitudes of various objects, including many non-records. Whether we collected data for non-records is inconsistent, so this is not a complete picture of progress within object types. See image in detail here.
Slow light
Main article: Historic trends in slow light technology
Group index of light appears to have seen discontinuities of 22 years in 1995 from Coherent Population Trapping (CPT) and 37 years in 1999 from EIT (condensate). Pulse delay of light over a short distance may have had a large discontinuity in 1994 but our data is not good enough to judge. After 1994, pulse delay does not appear to have seen discontinuities of more than ten years.

Figure 12: Progress in pulse delay and group index. "Human speed" shows the rough scale of motion familiar to humans.
Particle accelerators
Main article: Historic trends in particle accelerator performance
None of particle energy, center-of-mass energy nor Lorentz factor achievable by particle accelerators appears to have undergone a discontinuity of more than ten years of progress at previous rates.
Figure 13a: Particle energy in eV over time
Figure 13b: Center-of-mass energy in eV over time
Figure 13c: Lorentz factor (gamma) over time.
Penicillin on syphilis
Main article: Penicillin and historic syphilis trends
Penicillin did not precipitate a discontinuity of more than ten years in deaths from syphilis in the US. Nor were there other discontinuities in that trend between 1916 and 2015.
The number of syphilis cases in the US also saw steep decline but no substantial discontinuity between 1941 and 2008.
On brief investigation, the effectiveness of syphilis treatment and inclusive costs of syphilis treatment do not appear to have seen large discontinuities with penicillin, but we have not investigated either thoroughly enough to be confident.

Figure 14a: Syphilis—Reported Cases by Stage of Infection, United States, 1941–2009, according to the CDC7
Figure 14b: Syphilis and AIDS mortality rates in the US during the 20th century.8
Nuclear weapons
Main article: Effect of nuclear weapons on historic trends in explosives
Nuclear weapons constituted a ~7 thousand year discontinuity in energy released per weight of explosive (relative effectiveness).
Nuclear weapons do not appear to have clearly represented progress in the cost-effectiveness of explosives, though the evidence there is weak.
Figure 15: Relative effectiveness of explosives, up to early nuclear bomb (note change to log scale)
High temperature superconductors
Main article: Historic trends in the maximum superconducting temperature
The maximum superconducting temperature of any material up to 1993 contained four greater than 10-year discontinuities: A 14-year discontinuity with NbN in 1941, a 26-year discontinuity with LaBaCuO4 in 1986, a 140-year discontinuity with YBa2Cu3O7 in 1987, and a 10-year discontinuity with BiCaSrCu2O9 in 1987.
YBa2Cu3O7 superconductors seem to correspond to a marked change in the rate of progress of maximum superconducting temperature, from a rate of progress of .41 Kelvin per year to a rate of 5.7 Kelvin per year.
Figure 16: Maximum superconducting temperate by material over time through 2015
Land speed records
Main article: historic trends in land speed records
Land speed records did not see any greater-than-10-year discontinuities relative to linear progress across all records. Considered as several distinct linear trends it saw discontinuities of 12, 13, 25, and 13 years, the first two corresponding to early (but not first) jet-propelled vehicles.
The first jet-propelled vehicle just predated a marked change in the rate of progress of land speed records, from a recent 1.8 mph / year to 164 mph / year.
Figure 17: Historic land speed records in mph over time. Speeds on the left are an average of the record set in mph over 1 km and over 1 mile. The red dot represents the first record in a cluster that was from a jet propelled vehicle. The discontinuities of more than ten years are the third and fourth turbojet points, and the last two points.
Chess AI
Main article: Historic trends in chess AI
The Elo rating of the best chess program measured by the Swedish Chess Computer Association did not contain any greater than 10-year discontinuities between 1984 and 2018.
 Figure 18: Elo ratings of the best program on SSDF at the end of each year.
Flight airspeed
Main article: Historic trends in flight airspeed records
Flight airspeed records between 1903 and 1976 contained one greater than 10-year discontinuity: a 19-year discontinuity corresponding to the Fairey Delta 2 flight in 1956.
The average annual growth in flight airspeed markedly increased with the Fairey Delta 2, from 16mph/year to 129mph/year.
Figure 19: Flight airspeed records over time
Structure heights
Main article: Historic trends in structure heights
Trends for tallest ever structure heights, tallest ever freestanding structure heights, tallest existing freestanding structure heights, and tallest ever building heights have each seen 5-8 discontinuities of more than ten years. These are:
Djoser and Meidum pyramids (~2600BC, >1000 year discontinuities in all structure trends)Three cathedrals that were shorter than the all-time record (Beauvais Cathedral in 1569, St Nikolai in 1874, and Rouen Cathedral in 1876, all >100 year discontinuities in current freestanding structure trend)Washington Monument (1884, >100 year discontinuity in both tallest ever structure trends, but not a notable discontinuity in existing structure trend)Eiffel Tower (1889, ~10,000 year discontinuity in both tallest ever structure trends, 54 year discontinuity in existing structure trend)Two early skyscrapers: the Singer Building and the Metropolitan Life Tower (1908 and 1909, each >300 year discontinuities in building height only)Empire State Building (1931, 19 years in all structure trends, 10 years in buildings trend)KVLY-TV mast (1963, 20 year discontinuity in tallest ever structure trend)Taipei 101 (2004, 13 year discontinuity in building height only)Burj Khalifa (2009, ~30 year discontinuity in both freestanding structure trends, 90 year discontinuity in building height trend)
Figure 20a: All-time record structure heights, long term history
Figure 20b: All-time record structure heights, recent history
Figure 20c: All-time record freestanding structure heights, long term history
Figure 20d: All-time record freestanding structure heights, recent history
Figure 20e: At-the-time record freestanding structure heights, long term history
Figure 20f: At-the-time record freestanding structure heights, recent history
Figure 20g: All-time record building heights, longer term history
Figure 20h: All-time record building heights, longer term history
Breech loading rifles
Main article: Effects of breech loading rifles on historic trends in firearm progress
Breech loading rifles do not appear to have represented a discontinuity in firing rate of guns, since it appears that other guns had a similar firing rate already. It remains possible that breech loading rifles represent a discontinuity in another related metric.
Incomplete case studies
This is a list of cases we have partially investigated, but insufficiently to include in this page.
Extended observations
This spreadsheet contains summary data and statistics about the entire set of case studies, including all calculations for findings that follow.
Prevalence of discontinuities

We investigated 38 trends in around 21 broad areas9
Of the 38 trends that we investigated, we found 20 to contain at least one substantial discontinuity, and 15 to contain at least one large discontinuity. (Note that our trends were selected for being especially likely to contain discontinuities, so this is something like an upper bound on their frequency in trends in general. However some trends we investigated for fairly limited periods, so these may have contained more discontinuities than we found.)
Trends we investigated had in expectation 2.3 discontinuities each, including 1 large discontinuity each, and 0.37 large robust discontinuities each (that we found–we did not necessarily investigate trends for the entirety of their history).
We found 88 substantial discontinuities, 20 of them robust, 14 of them large and robust.
These discontinuities were produced by 63 distinct events, 29 of them producing large discontinuities.
The robust large discontinuities were produced by 10 events
32% of trends we investigated saw at least one large, robust discontinuity (though note that trends were selected for being discontinuous, and were a very non-uniform collection of topics, so this could at best inform an upper bound on how likely an arbitrary trend is to have a large, robust discontinuity somewhere in a chunk of its history)
53% of trends saw any discontinuity (including smaller and non-robust ones), and in expectation a trend saw more than two of these discontinuities.
On average, each trend had 0.001 large robust discontinuities per year, or 0.002 for those trends with at least one at some point10
On average 1.4% of new data points in a trend make for large robust discontinuities, or 4.9% for trends which have one.
On average 14% of total progress in a trend came from large robust discontinuities (or 16% of logarithmic progress), or 38% among trends which have at least one.
Across all years of any metric we considered, the rate of discontinuities/year was around 0.02% (though note that this is heavily influenced by how often you consider thousands of years with poor data at the start).

Some fuller related data, from spreadsheet:

All discontinuities
Large
Robust
Robust large

Metrics checked
38
38
38
38

Discontinuity count
88
39
20
14

Trends exhibiting that type of discontinuity
20
15
16
12

Trends with 2+  discontinuities of that type
14
10
4
2

P(discontinuity|trend)
0.53
0.39
0.42
0.32

E(discontinuities per trend)
2.3
1.0
0.5
0.4

P(multiple discontinuities|trend)
0.37
0.26
0.11
0.05

P(multiple discontinuities|trend with at least one)
0.70
0.67
0.25
0.17

P(multiple discontinuities|trend with at least one, and enough search to find more)
0.78
0.77
0.29
0.20

Nature of discontinuous metrics
We categorized each metric as one of:

'technical': to do with basic physical parameters (e.g. light intensity, particle energy in particle accelerators)
'product': to do with usable goods or services (e.g. cotton ginned per person per day, size of largest ships, height of tallest structures)
'industry': to do with an entire industry rather than individual items (e.g. total production of books)
'societal': to do with society at large (e.g. syphilis mortality)

We also categorized each metric as one of:

'feature': a characteristic that is good, but not close to encompassing the purpose of most related efforts (e.g. ship size, light intensity)
'performance proxy': approximates the purpose of the endeavor (e.g. cotton ginned per person per day, effectiveness of syphilis treatment)
'value proxy': approximates the all-things-considered value of the endeavor (e.g. real price of books, cost-effectiveness of explosives)

Most metrics fell into 'product feature' (16) 'technical feature' (8) or 'product performance proxy' (6), with the rest (8) spread across the categories.
Here is what these trends are like (from this spreadsheet):

product featuretechnical featureproduct performance proxyrare categoriesAll discontinuitiesnumber of discontinuities73825number of trends16868number of trends with discontinuities13421discontinuities per trend4.61.00.30.6fraction of trends with discontinuity0.810.500.330.13Large discontinuitiesnumber of large discontinuities32304number of trends16868number of trends with large discontinuities11301large discontinuities per trend2.00.40.00.5fraction of trends with large discontinuity0.690.380.000.13

Primary authors: Katja Grace, Rick Korzekwa, Asya Bergal, Daniel Kokotajlo.
Thanks to many other researchers whose work contributed to this project.
Thanks to Stephen Jordan, Jesko Zimmermann, Bren Worth, Finan Adamson, and others for suggesting potential discontinuities for this project in response to our 2015 bounty, and to many others for suggesting potential discontinuities since, especially notably Nuño Sempere, who conducted a detailed independent investigation into discontinuities in ship size and time to circumnavigate the world11.
Notes

Discontinuous progress investigation


List of Analyses of Time to Human-Level AI

This is a list of most of the substantial analyses of AI timelines that we know of. It also covers most of the arguments and opinions of which we are aware.
Details
The list below contains substantial publically available analyses of when human-level AI will appear. To qualify for the list, an item must provide both a claim about when human-level artificial intelligence (or a similar technology) will exist, and substantial reasoning to support it. 'Substantial' is subjective, but a fairly low bar with some emphasis on detail, novelty, and expertise. We exclude arguments that AI is impossible, though they are technically about AI timelines.
List

Good, Some future social repercussions of computers (1970) predicts 1993 give or take a decade, based roughly on the availability of sufficiently cheap, fast, and well-organized electronic components, or on a good understanding of the nature of language, and on the number of neurons in the brain.
Moravec, Today's Computers, Intelligent Machines and Our Future (1978) projects that ten years later hardware equivalent to a human brain would be cheaply available, and that if software development 'kept pace' then machines able to think as well as a human would begin to appear then.
Solomonoff, The Time Scale of Artificial Intelligence: Reflections on Social Effects (1985) estimates one to fifty years to a general theory of intelligence, then ten or fifteen years to a machine with general problem solving capacity near that of a human, in some technical professions.
Waltz, The Prospect for Building Truly Intelligent Machines (1988) predicts human-level hardware in 2017 and says the development of human-level AI might take another twenty years.
Vinge, The Coming Technological Singularity: How to Survive in the post-Human Era (1993) argues for less than thirty years from 1993, largely based on hardware extrapolation.
Eder, Re: The Singularity (1993) argues for 2035 based on two lines of reasoning: hardware extrapolation to computation equivalent to the human brain, and hyperbolic human population growth pointing to a singularity at that time.
Yudkowsky, Staring Into the Singularity 1.2.5 (1996) presents calculation suggesting a singularity will occur in 2021, based on hardware extrapolation and a simple model of recursive hardware improvement.
Bostrom, How Long Before Superintelligence?(1997) argues that it is plausible to expect superintelligence in the first third of the 21st Century. In 2008 he added that he did not think the probability of this was more than half.
Bostrom, When Machines Outsmart Humans(2000) argues that we should take seriously the prospect of human-level AI before 2050, based on hardware trends and feasibility of uploading or software based on understanding the brain.
Kurzweil, The Singularity is Near (pdf) (2005) predicts 2029, based mostly on hardware extrapolation and the belief that understanding necessary for software is growing exponentially. He also made a bet with Mitchell Kapor, which he explains along with the bet and here. Mitchell also explains his reasoning alongside the bet, though it nonspecific about timing to the extent that it isn't clear whether he thinks AI will ever occur, which is why he isn't included in this list.
Peter Voss, Increased Intelligence, Improved Life (video) (2007) predicts less than ten years and probably less than five, based on the perception that other researchers pursue unnecessarily difficult routes, and that shortcuts probably exist.
Moravec, The Rise of the Robots (2009) predicts AI rivalling human intelligence well before 2050, based on progress in hardware, estimating how much hardware is equivalent to a human brain, and comparison with animals whose brains appear to be equivalent to present-day computers. Moravec made similar predictions in the 1988 book Mind Children.
Legg, Tick, Tock, Tick Tock Bing (2009) predicts 2028 in expectation, based on details of progress and what remains to be done in neuroscience and AI. He agreed with this prediction in 2012.
Allen, The Singularity Isn't Near (2011) criticizes Kurzweil's prediction of a singularity around 2045, based mostly on disagreeing with Kurzweil on rates of brain science and AI progress.
Hutter, Can Intelligence Explode (2012) uses a prediction of not much later than the 2030s, based on hardware extrapolation, and the belief that software will not lag far behind.
Chalmers (2010) guesses that human-level AI is more likely than not this century. He points to several early estimates, but expresses skepticism about hardware extrapolation, based on the apparent algorithmic difficulty of AI. He argues that AI should be feasible within centuries (conservatively) based on the possibility of brain emulation, and the past success of evolution.
Fallenstein and Mennen (2013) suggest using a Pareto distribution to model time until we get a clear sign that human-level AI is imminent. They get a median estimate of about 60 years, depending on the exact distribution (based on an estimate of 60 years since the beginning of the field).
Drum, Welcome, Robot Overlords. Please Don't Fire Us? (2013) argues for around 2040, based on hardware extrapolation.
Muehlhauser, When will AI be Created? (2013) argues for uncertainty, based on surveys being unreliable, hardware trends being insufficient without software, and software being potentially jumpy.
Bostrom, Superintelligence (2014) concludes that '…it may be reasonable to believe that human-level machine intelligence has a fairly sizeable chance of being developed by mid-century and that it has a non-trivial chance of being developed considerably sooner or much later…', based on expert surveys and interviews, such as these.
Sutton, Creating Human Level AI: How and When? (2015) places a 50% chance on human-level AI by 2040, based largely on hardware extrapolation and the view that software has a 1/2 chance of following within a decade of sufficient hardware.


The slow traversal of 'human-level'

Once you have normal-human-level AI, how long does it take to get Einstein-level AI? We have seen that a common argument for 'not long at all' based on brain size does not work in a straightforward way, though a more nuanced assessment of the evidence might. Before we get to that though, let's look at some more straightforward evidence (from our new page on the range of human intelligence).
In particular, let's look at chess. AI can play superhuman-level chess, so we can see how it got there. And how it got there is via about four decades of beating increasingly good players, starting at beginners and eventually passing Kasparov (note that a beginner is something like level F or below, which doesn't make it onto this graph):
Figure 1: Chess AI progress compared to human performance, from Coles 2002. The original article was apparently written before 1993, so note that the right of the graph (after 'now') is imagined, though it appears to be approximately correct.
Something similar is true in Go (where -20 on this graph is a good beginner score, and go bots are not yet superhuman, but getting close):
From Grace 2013.
Backgammon and poker AI's seems to have progressed similarly, though backgammon took about 2 rather than 4 decades (we will soon post more detailed descriptions of progress in board games).
Go, chess, poker, and backgammon are all played using different algorithms. But the underlying problems are sufficiently similar that they could easily all be exceptions.
Other domains are harder to measure, but seem basically consistent with gradual progress. Machine translation seems to be gradually moving through the range of human expertise, as does automatic driving. There are fewer clear cases where AI abilities took years rather than decades to move from subhuman to superhuman, and the most salient cases are particularly easy or narrow problems (such as arithmetic, narrow perceptual tasks, or easy board games).
If narrow AI generally traverses the relevant human range slowly, this suggests that general AI will take some time to go from minimum minimum wage competency to—well, at least to AI researcher competency. If you combine many narrow skills, each progressing gradually through the human spectrum at different times, you probably wouldn't end up with a much more rapid change in general performance. And it isn't clear that a more general method should tend to progress faster than narrow AI.
However, we can point to ways that general AI might be different from board game AI.
Perhaps progress in chess and go has mostly been driven by hardware progress, while progress in general AI will be driven by algorithmic improvements or acquiring more training data.
Perhaps the kinds of algorithms people really use to think scale much better than chess algorithms. Chess algorithms only become 30-60 Elo points stronger with each doubling of hardware, whereas a very rough calculation suggests human brains become more like 300 Elo points better per doubling in size.
In humans, brain size has roughly a 1/3 correlation with intelligence. Given that the standard deviation of brain size is about 10% of the size of the brain (p. 39), this suggests that a doubling of brain size leads to a relatively large change in chess-playing ability. On a log scale, a doubling is a 7 standard deviation change in brain size, which would suggest a ~2 standard deviation change in intelligence. It's hard to know how this relates to chess performance, but in Genius in Chess Levitt gives an unjustified estimate of 300 Elo points. This is what we would expect if intelligence were responsible for half of variation in performance (neglecting the lower variance of chess player intelligence), since a standard deviation of chess performance is about 2000/7 ~ 300 Elo. Each of these correlations is problematic but nevertheless suggestive.
If human intelligence in general scales much better with hardware than existing algorithms, and hardware is important relative to software, then AI based on an understanding of human intelligence may scale from sub-human to superhuman more quickly than the narrow systems we have seen. However these are both open questions.
(Image: The Chess Players, by Honoré Daumier)

The slow traversal of ‘human-level’


Making or breaking a thinking machine

Here is a superficially plausible argument: the brains of the slowest humans are almost identical to those of the smartest humans. And thus—in the great space of possible intelligence—the 'human-level' band must be very narrow. Since all humans are basically identical in design—since you can move from the least intelligent human to the sharpest human with imperceptible changes—then artificial intelligence development will probably cross this band of human capability in a blink. It won't stop on the way to spend years being employable but cognitively limited, or proficient but not promotion material. It will be superhuman before you notice it's nearly human. And from our anthropomorphic viewpoint, from which the hop separating village idiot and Einstein looks like most of the spectrum, this might seem like shockingly sudden progress.
This whole line of reasoning is wrong.
It is true that human brains are very similar. However, this implies very little about the design difficulty of moving from the intelligence of one to the intelligence of the other artificially. The basic problem is that the smartest humans need not be better-designed — they could be better instantiations of the same design.
What's the difference? Consider an analogy. Suppose you have a yard full of rocket cars. They all look basically the same, but you notice that their peak speeds are very different. Some of the cars can drive at a few hundred miles per hour, while others can barely accelerate above a crawl. You are excited to see this wide range of speeds, because you are a motor enthusiast and have been building your own vehicle. Your car is not quite up to the pace of the slowest cars in your yard yet, but you figure that since all those cars are so similar, once you get it to two miles per hour, it will soon be rocketing along.
If a car is slow because it is a rocket car with a broken fuel tank, that car will be radically simpler to improve than the first car you build that can go over 2 miles per hour. The difference is something like an afternoon of tinkering vs. two centuries. This is intuitively because the broken rocket car already contains almost all of the design effort in making a fast rocket car. It's not being used, but you know it's there and how to use it.
Similarly, if you have a population of humans, and some of them are severely cognitively impaired, you shouldn't get too excited about the prospects for your severely cognitively impaired robot.
Another way to see there must be something wrong with the argument is to note that humans can actually be arbitrarily cognitively impaired. Some of them are even dead. And the brain of a dead person can closely resemble the brain of a live person. Yet while these brains are again very similar in design, AI passed dead-human-level years ago, and this did not suggest that it was about to zip on past live-human-level.
Here is a different way to think about the issue. Recall that we were trying to infer from the range of human intelligence that AI progress would be rapid across that range. However, we can predict that human intelligence has a good probability of varying significantly, using only evolutionary considerations that are orthogonal to the ease of AI development.
In particular, if much of the variation in intelligence is from deleterious mutations, then the distribution of intelligence is more or less set by the equilibrium between selection pressure for intelligence and the appearance of new mutations. Regardless of how hard it was to design improvements to humans, we would always see this spectrum of cognitive capacities, so this spectrum cannot tell us about how hard it is to improve intelligence by design. (Though this would be different if the harm inflicted by a single mutation was likely to be closely related to the difficulty of designing an incrementally more intelligent human).
If we knew more about the sources of the variation in human intelligence, we might be able to draw a stronger conclusion. And if we entertain several possible explanations for the variation in human intelligence, we can still infer something; but the strength of our inference is limited by the prior probability that deleterious mutations on their own can lead to significant variation in intelligence. Without learning more, this probability shouldn't be very low.
In sum, while the brain of an idiot is designed much like that of a genius, this does not imply that designing a genius is about as easy as designing an idiot.
We are still thinking about this, so now is a good time to tell us if you disagree. I even turned on commenting, to make it easier for you. It should work on all of the blog posts now.
Rocket car, photographed by Jon 'ShakataGaNai' Davis.
(Top image: One of the first cars, 1769)

Making or breaking a thinking machine


The range of human intelligence

The range of human intelligence seems large relative to the space below it, as measured by performance on tasks we care about—despite the fact that human brains are extremely similar to each other.
Without knowing more about the sources of variation in human performance, however, we cannot conclude much at all about the likely pace of progress in AI: we are likely to observe significant variation regardless of any underlying facts about the nature of intelligence.
Details
Measures of interest
Performance
IQ is one measure of cognitive performance. Chess ELO is a narrower one. We do not have a general measure that is meaningful across the space of possible minds. However when people speak of 'superhuman intelligence' and the intelligence of animals they imagine that these can be meaningfully placed on some rough spectrum. When we say 'performance' we mean this kind of intuitive spectrum.
Development effort
We are especially interested in measuring intelligence by the difficulty of building a machine which exhibits that level of intelligence. We will not use a formal unit to measure this distance, but are interested in comparing the range between humans to distances between other milestones, such as that between a mouse and a human, or a rock and a mouse.
Variation in cognitive performance
It is sometimes argued that humans occupy a very narrow band in the spectrum of cognitive performance. For instance, Eliezer Yudkowsky defends this rough schemata1—

—over these, which he attributes to others:

Such arguments sometimes go further, to suggest that AI development effort needed to traverse the distance from the 'village idiot' to Einstein is also small, and so given that it seems so large to us, AI progress at around human level will seem very fast.
The landscape of performance is not easy to parameterize well, as there are many cognitive tasks and dimensions of cognitive ability, and no good global metric for comparison across different organisms. Nonetheless, we offer several pieces of evidence to suggest that the human range is substantial, relative to the space below it. We do not approach the topic here of how far above human level the space of possible intelligence reaches.
Low human performance on specific tasks
For most tasks, human performance reaches all of the way to the bottom of the possible spectrum. At the extreme, some comatose humans will fail at almost any cognitive task. Our impression is that people who are completely unable to perform a task are not usually isolated outliers, but that there is a distribution of people spread across the range from completely incapacitated to world-champion level. That is, for a task like 'recognize a cat', there are people who can only do slightly better than if they were comatose.
For our purposes we are more interested in where normal human cognitive performance fall relative to the worst and best possible performance, and the best human performance.
Mediocre human performance relative to high human performance
On many tasks, it seems likely that the best humans are many times better than mediocre humans, using relatively objective measures.2
Shockley (1957) found that in science, the productivity of the top researchers in a laboratory was often at least ten times as great as the least productive (and most numerous) researchers. Programmers purportedly vary by an order of magnitude in productivity, though this is debated. A third of people scored nothing in this Putnam competition, while someone scored 100. Some people have to work ten times harder to pass their high school classes than others.
Note that these differences are among people skilled enough to actually be in the relevant field, which in most cases suggests they are above average. Our impression is that something similar is true in other areas such as sales, entrepreneurship, crafts, and writing, but we have not seen data on them.
These large multipliers on performance at cognitive tasks suggest that the range between mediocre cognitive ability and genius is many times larger than the range below mediocre cognitive ability. However it is not clear that such differences are common, or to what extent they are due to differences in underlying general cognitive ability, rather than learning or non-cognitive skills, or a range of different cognitive skills that aren't well correlated.
Human performance spans a wide range in other areas
In qualities other than intelligence, humans appear to span a fairly wide range below their peak levels. For instance, the fastest human runners are multiple times faster than mediocre runners (twice as fast at a 100m sprint, four times as fast for a mile). Humans can vary in height by a factor of about four, and commonly do by a factor of about 1.5. The most accurate painters are hard to distinguish from photographs, while some painters are arguably hard to distinguish from monkeys, which are very easy to distinguish from photographs. These observations weakly suggest that the default expectation should be for humans to span a wide absolute range in cognitive performance also.
AI performance on human tasks
In domains where we have observed human-level performance in machines, we have seen rather gradual improvement across the range of human abilities. Here are five relevant cases that we know of:
1. Chess: human chess Elo ratings conservatively range from around 800 (beginner) to 2800 (world champion). The following figure illustrates how it took chess AI roughly forty years to move incrementally from 1300 to 2800.
Figure 1: Chess AI progress compared to human performance, from Coles 2002. The original article was apparently written before 1993, so note that the right of the graph (after 'now') is imagined, though it appears to be approximately correct.
2. Go: Human go ratings range from 30-20 kyu (beginner) to at least 9p (10p is a special title). Note that the numbers go downwards through kyu levels, then upward through dan levels, then upward through p(rofessional dan) levels. The following figure suggests that it took around 25 years for AI to cover most of this space (the top ratings seem to be closer together than the lower ones, though there are apparently multiple systems which vary).
Figure 2. From Grace 2013.
3. Checkers: According to Wikipedia's timeline of AI, a program was written in 1952 that could challenge a respectable amateur. In 1994 Chinook beat the second highest rated player ever. (In 2007 checkers was solved.) Thus it took around forty years to pass from amateur to world-class checkers-playing. We know nothing about whether intermediate progress was incremental however.
4. Physical manipulation: we have not investigated this much, but our impression is that robots are somewhere in the the fumbling and slow part of the human spectrum on some tasks, and that nobody expects them to reach the 'normal human abilities' part any time soon (Aaron Dollar estimates robotic grasping manipulation in general is less than one percent of the way to human level from where it was 20 years ago).
5. Jeopardy: AI appears to have taken two or three years to move from lower 'champion' level to surpassing world champion level (see figure 9; Watson beat Ken Jennings in 2011). We don't know how far 'champion' level is from the level of a beginner, but would be surprised if it were less than four times the distance traversed here, given the situation in other games, suggesting a minimum of a decade for crossing the human spectrum.
In all of these narrow skills, moving AI from low-level human performance to top-level human performance appears to take on the order of decades. This further undermines the claim that the range of human abilities constitutes a narrow band within the range of possible AI capabilities, though we may expect general intelligence to behave differently, for example due to smaller training effects.
On the other hand, most of the examples here—and in particular the ones that we know the most about—are board games, so this phenomenon may be less usual elsewhere. We have not investigated areas such as Texas hold 'em, arithmetic or constraint satisfaction sufficiently to add them to this list.
What can we infer from human variation?
The brains of humans are nearly identical, by comparison to the brains of other animals or to other possible brains that could exist. This might suggest that the engineering effort required to move across the human range of intelligences is quite small, compared to the engineering effort required to move from very sub-human to human-level intelligence (e.g. see p21 and 29, p70). The similarity of human brains also suggest that the range of human intelligence is smaller than it seems, and its apparent breadth is due to anthropocentrism (see the same sources). According to these views, board games are an exceptional case–for most problems, it will not take AI very long to close the gap between "mediocre human" and "excellent human."
However, we should not be surprised to find meaningful variation in the cognitive performance regardless of the difficulty of improving the human brain. This makes it difficult to infer much from the observed variations.
Why should we not be surprised? De novo deleterious mutations are introduced into the genome with each generation, and the prevalence of such mutations is determined by the balance of mutation rates and negative selection. If de novo mutations significantly impact cognitive performance, then there must necessarily be significant selection for higher intelligence–and hence behaviorally relevant differences in intelligence. This balance is determined entirely by the mutation rate, the strength of selection for intelligence, and the negative impact of the average mutation.
You can often make a machine worse by breaking a random piece, but this does not mean that the machine was easy to design or that you can make the machine better by adding a random piece. Similarly, levels of variation of cognitive performance in humans may tell us very little about the difficulty of making a human-level intelligence smarter.
In the extreme case, we can observe that brain-dead humans often have very similar cognitive architectures. But this does not mean that it is easy to start from an AI at the level of a dead human and reach one at the level of a living human.
Because we should not be surprised to see significant variation–independent of the underlying facts about intelligence–we cannot infer very much from this variation. The strength of our conclusions are limited by the extent of our possible surprise.
By better understanding the sources of variation in human performance we may be able to make stronger conclusions. For example, if human intelligence is improving rapidly due to the introduction of  new architectural improvements to the brain, this suggests that discovering architectural improvements is not too difficult.  If we discover that spending more energy on thinking makes humans substantially smarter, this suggests that scaling up intelligences leads to large performance changes. And so on. Existing research in biology addresses the role of deleterious mutations, and depending on the results this literature could be used to draw meaningful inferences.
These considerations also suggest that brain similarity can't tell us much about the "true" range of human performance. This isn't too surprising, in light of the analogy with other domains. For example, although the bodies of different runners have nearly identical designs, the worst runners are not nearly as good as the best.
xxx[This background rate of human-range crossing is less informative about the future in scenarios where the increasing machine performance of interest is coming about in a substantially different way from how it came about in the past. For instance, it is sometimes hypothesized that major performance improvements will come from fast 'recursive self improvement', in which case the characteristic time scale might be much faster. However the scale of the human performance range (and time to cross it) relative to the area below the human range should still be informative.]

The range of human intelligence


Are AI surveys seeing the inside view?

An interesting thing about the survey data on timelines to human-level AI is the apparent incongruity between answers to 'when will human-level AI arrive?' and answers to 'how much of the way to human-level AI have we come recently?'
In particular, human-level AI will apparently arrive in thirty or forty years, while in the past twenty years most specific AI subfields have apparently moved only five or ten percent of the remaining distance to human-level AI, with little sign of acceleration.
Some possible explanations:

The question about how far we have come has hardly been asked, and the small sample size has hit slow subfields, or hard-to-impress researchers, perhaps due to a different sampling of events.
Hanson (the only person who asked how far we have come) somehow inspires modesty or agreement in his audience. His survey methodology is conversational, and the answers do agree with his own views.
The 'inside view' is overoptimistic: if you ask a person directly when their project will be done, they tend to badly underestimate. Taking the 'outside view' – extrapolating from similar past situations – helps to resolve these problems, and is more accurate. The first question invites the inside view, while the second invites the outside view.
Different people are willing to answer the different questions.
Estimating 'how much of the way between where we were twenty years ago and human-level capabilities' is hopelessly difficult, and the answers are meaningless.
Estimating 'when will we have human-level AI?' is hopelessly difficult, and the answers are meaningless.
When people answer the 'how far have we come in the last twenty years?…' question, they use a different scale to when they answer the '…and are we accelerating?' question, for instance thinking of where we are as a fraction of what is left to do in the first case, and expecting steady exponential growth in that fraction, but not thinking of steady exponential growth as 'acceleration'.
AI researchers expect a small number of fast-growing subfields to produce AI with the full range of human-level skills, rather than for it to combine contributions from many subfields.
Researchers have further information not captured in the past progress and acceleration estimates. In particular, they have reason to expect acceleration.

Since the two questions have so far yielded very different answers, it would be nice to check whether the different answers come from the different kinds of questions (rather than e.g. the small and casual nature of the Hanson survey), and to get a better idea of which kind of answer is more reliable. This might substantially change the message we get from looking at the opinions of AI researchers.
Luke Muehlhauser and I have written before about how to conduct a larger survey like Hanson's. One might also find or conduct experiments comparing these different styles of elicitation on similar predictions that can be sooner verified. There appears to be some contention over which method should be more reliable, so we could also start by having that discussion.

Are AI surveys seeing the inside view?


Event: Multipolar AI workshop with Robin Hanson

On Monday 26 January we will be holding a discussion on promising research projects relating to 'multipolar' AI scenarios. That is, future scenarios where society persists in containing a large number of similarly influential agents, rather than a single winner who takes all. The event will be run in collaboration with Robin Hanson, a leading researcher on the social consequences of whole brain emulation.
The goal of the meeting will be to identify promising concrete research projects.
We will consider projects under various headings, for example:
Causal origins and probability of multipolar scenarios

Collect past records of lumpiness of AI success
Survey military, business or academic projects which were particularly analogous to successful emulation or AI projects, to learn about the situations in which emulations or AI might appear.
Survey AI experts on the likelihood of AI emerging in the military, business or academia, and on the likely size of a successful AI project.
…

Consequences of multipolar scenarios

Hanson's project of detailing a default whole brain emulation scenario
How does the lumpiness of economic outcomes vary as a function of the lumpiness of origins?
Are there initial social institutions which might substantially influence longer term outcomes?
…

Applicability of broader AI safety insights to multipolar outcomes

How useful are capability control methods, such as boxing, stunting, incentives, or tripwires in a multi-polar scenario?
How useful are motivation selection methods, such as direct specification, domesticity, indirect normatively, augmentation in a multipolar scenario?
Would selective pressures strongly favor the existence of goal-directed agents, in a multipolar scenario where a variety of AI designs are feasible?
…

Details in brief
Time:
2pm until 2-3 hours later
There will be an evening social event at 7pm in the same location, which workshop attendees are welcome to stay or return for. Some participants will go to dinner at a nearby restaurant in between.
Date: Monday 26 January 2015
Location: Private Berkeley residence. Detail available upon RSVP.
RSVP: to <EMAIL>.
This is helpful, but not required (if you can deduce the location).
Other things: Tea, coffee and snacks provided.


Michie and overoptimism

We recently wrote about Donald Michie's survey on timelines to human-level AI. Michie's survey is especially interesting because it was taken in 1972, which is three decades earlier than any other surveys we know of that ask about human-level AI.
Early AI predictions are renowned for being absurdly optimistic. And while the scientists in Michie's survey had had a good decade and a half since the Dartmouth Conference to observe the lack of AI, they were still pretty early. Yet they don't seem especially optimistic. Their predictions were so far in the future that almost three quarters of them still haven't been demonstrated wrong.
And you might think computer scientists in the early 70s would be a bit more optimistic than contemporary forecasters, given that they hadn't seen so many decades of research not produce human-level AI. But the median estimate they gave for how many years until AI was further out than those given in almost all surveys since (fifty years vs. thirty to forty). The survey doesn't look like it was very granular – there appear to be only five options – so maybe a bunch of people would have said thirty-five years, and rounded up to fifty. Still, their median expectations don't look like they were substantially more optimistic (in terms of time to AI) than present-day ones.
In terms of absolute dates, the Michie participants' median fifty-year choice of 2022 is still considered at least 10% likely to give rise to human-level AI by recent survey participants, some four decades later.
It's not like anyone said that all early predictions were embarrassingly optimistic though. Maybe Michie's computer scientists were outliers? Perhaps, but if everyone else whose predictions we know of disagreed with them, it would be those others who were the outliers: Michie's survey has sixty-three respondents, whereas the MIRI dataset contains only eleven other predictions made before 1980 (it looks like twelve, but the other interesting looking survey attributed to Firschein and Coles appears to be an accidental duplicate of the Michie survey, which Firschein and Coles mention in their paper). Sixty-three is more than all the predictions in the MIRI dataset until the 2005 Bainbridge survey.
AI researchers may have been extremely optimistic in the very early days (the MIRI dataset suggests this, though it consists of public statements, which tend to be more optimistic anyway). However, it doesn't seem to have taken AI researchers long to move to something like contemporary views. It looks like they didn't just predict 'twenty years' every year since Dartmouth.
Survey results, as shown in Michie's paper (pdf download).
Thanks to Luke Muehlhauser for suggesting this line of inquiry.
(Image: "Donald Michie teaching" by Petermowforth)

Michie and overoptimism


Were nuclear weapons cost-effective explosives?

Nuclear weapons were radically more powerful per pound than any previous bomb. Their appearance was a massive discontinuity in the long-run path of explosive progress, that we have lately discussed.
But why do we measure energy per mass in particular? Energy per dollar may be a better measure of 'progress', if the goal was to explode things cheaply, rather than lightly. So we looked into this too. The first thing we found was that information about the costs of pre-WWII explosives is surprisingly sparse. However from what we can gather, nuclear weapons did not immediately improve the cost-effectiveness of explosives much at all.
The marginal cost of an early 200kt nuclear weapon was about $25M, which is quite similar to the $21M price of the equivalent amount of TNT. Early nuclear explosives did not deliver radically more bang-for-the-buck than their conventional counterparts (even setting aside the considerable upfront development costs); their immediate significance was their greater energy density.
This is not a precise comparison. For one thing, conventional bombs have costs beyond the explosive material, which were not included in the figures here. For another, conventional bombs contain explosives other than TNT (alternatives gave somewhat more explosive power per dollar). Nonetheless, the ballpark costs of the two explosives seem comparable.
This is interesting for a few reasons.
Continuity from continuously falling costs
It tells us something about why and when technological progress is continuous. One possible explanation for continuous progress is that people do things when they first become reasonably cost-effective, at which point they are unlikely to be radically cost-effective. Imagine there are a range of possible projects, with different costs and payoffs. Every year, they all get a little cheaper. If a project is a great deal this year, then it would have been done last year, when it was already a good deal. So all of the available deals would be about as good as each other. On this model, new technologies might be suddenly very good, but only if the advance was also very expensive. Nothing would be suddenly very cost-effective.
On their face, nuclear weapons appear to support this theory. They made abrupt progress, but were incredibly expensive to develop and build. But this story doesn't stand up to closer inspection. A few years before they were deployed, not even physicists generally thought them possible; no one considered and then rejected an investment in nuclear weapons. As soon as the possibility was seriously considered, it was considered to merit a significant fraction of GDP.
Continuity from incremental improvements
It's also plausible that progress is continuous on metrics that people care most about, because if they care they search hard for improvements, and there are in fact plenty of small improvements to find. Cost-effectiveness of explosives is a thing we care about, whereas we care less (though still quite a bit) about explosive power per weight. The nuclear case provides some support for this explanation.
Continuity in something
There are many ways to measure progress. Finding measures that don't change abruptly can help avoid surprises, by allowing us to forecast the trends that are most likely to advance predictably. The nuclear example may help shed light on what kinds of measures tend to be continuous.
~
There are even more relevant metrics than this literal meaning of "bang for the buck."  Destruction per dollar might be closer, but harder to measure; the effect on P(winning the war) would be closer still, and the effect on national interests more broadly would be even better. It was hard enough to find cost-effectiveness of in terms of energy, so we won't be investigating these any time soon.
In general, I would guess that progress becomes more continuous as we move closer to measures that people really care about. The reverse may be true, however, for nuclear weapons: the costs of deploying weapons might see more abrupt progress than the narrower measure we have considered. A nuclear weapon requires one plane, whereas a rain of small bombs requires many, and these planes and crews appear to be much more expensive than the bombs.

This is a small part of an ongoing investigation into discontinuous technological change. More blog posts are here and here. The most relevant pages about this so far are Cases of Discontinuous Technological Progress and Discontinuity from Nuclear Weapons.
(Image: EOD teams detonate expired ordnance in the Kuwaiti desert on July 12, 2002.)


A summary of AI surveys

If you want to know when human-level AI will be developed, a natural approach is to ask someone who works on developing AI. You might however be put off by such predictions being regularly criticized as inaccurate and biased. While they do seem overwhelmingly likely to be inaccurate and biased, I claim they would have to be very inaccurate and biased before they were worth ignoring, especially in the absence of many other sources of quality information. The bar for ridicule is well before the bar for being uninformative.
So on that note, we made a big summary of all of the surveys we know of on timelines to human-level AI. And also a bunch of summary pages on specific human-level AI surveys. We hope they are a useful reference, and also help avert selection bias selection bias from people only knowing about surveys that support their particular views on selection bias.
It's interesting to note the consistency between the surveys that asked participants to place confidence intervals. They all predict there is a ten percent chance of human-level AI sometime in the 2020s, and almost all place a fifty percent chance of human-level AI between 2040 and 2050. They are even pretty consistent on the 90% date, with more than half in 2070-2080. This is probably mostly evidence that people talk to each other and hear about similar famous predictions. However it is some evidence of accuracy, since if each survey produced radically different estimates we must conclude that surveys are fairly inaccurate.
If you know of more surveys on human-level AI timelines, do send them our way.
Here's a summary of our summary:

Year
Survey
#
 10%
 50%
 90%
 Other key 'Predictions'
Participants
Response rate
Link to original document

1972
 Michie
67

Median 50y (2022) (vs 20 or >50)
AI, CS
–
link

2005
 Bainbridge
26

 Median 2085
Tech
–
 link

2006
 AI@50

median >50y (2056)
AI conf
–
link

2007
 Klein
888

median 2030-2050
Futurism?
–
link and link

2009
 AGI-09

 2020
 2040
 2075

AGI conf; AI
–
link

2011
 FHI Winter Intelligence
35
 2028
2050
 2150

AGI impacts conf; 44% related technical
41%
link

2011-2012
 Kruel interviews
37
 2025
 2035
 2070

AGI, AI
–
link

2012
 FHI: AGI
72
 2022
 2040
 2065

AGI & AGI impacts conf; AGI, technical work
65%
link

2012
 FHI:PT-AI
43
 2023
 2048
 2080

Philosophy & theory of AI conf; not technical AI
49%
link

2012-present
 Hanson
~10

 ≤ 10% progress to human level in past 20y
AI
–
link

2013
 FHI: TOP100
29
2022
 2040
 2075

Top AI
29%
link

2013
 FHI:EETN
26
 2020
 2050
 2093

Greek assoc. for AI; AI
10%
link

(Image: AGI-09 participants, by jeriaska)

A summary of AI surveys