Similarity Between Historical and Contemporary AI Predictions

AI predictions from public statements made before and after 2000 form similar distributions. Such predictions from before 1980 appear to be more optimistic, though predictions from a larger early survey are not.

Discussion
----------

### Similarity of predictions over time

#### MIRI dataset

We compared early and late predictions using the [MIRI dataset](http://aiimpacts.org/miri-ai-predictions-dataset/ "MIRI AI Predictions Dataset"). We find the correlation between the date of a prediction and number of years until AI is predicted from that time is 0.13. Most predictions are in the last decade or two however, so this does not tell us much about long run trends (see Figure 1).

[![xxx](http://aiimpacts.org/wp-content/uploads/2014/12/Time-to-AI-more-likely-than-not-1-1024x761.png)](http://aiimpacts.org/wp-content/uploads/2014/12/Time-to-AI-more-likely-than-not-1.png)**Figure 1:** Years until AI is predicted ([minPY](http://aiimpacts.org/miri-ai-predictions-dataset/ "MIRI AI Predictions Dataset")) over time in [MIRI AI predictions dataset](http://aiimpacts.org/miri-ai-predictions-dataset/ "MIRI AI Predictions Dataset").
The six predictions prior to 1980 were all below the median 30 years, which would have less than 2% chance if they were really drawn from the same distribution.

The predictions made before and after 2000 form very similar distributions however (see figure 2). The largest difference between the fraction of pre-2000 and since-2000 people who predict AI by any given distance in the future is about 15%. A difference this large is fairly likely by chance, according to our unpublished calculations. See the [MIRI dataset page](https://sites.google.com/site/aiimpactslibrary/ai-timelines/predictions-of-human-level-ai-dates/miri-ai-predictions-dataset) for further details.

[![Late and early predictions](http://aiimpacts.org/wp-content/uploads/2014/12/time-to-predictions-2-11.png)](http://aiimpacts.org/wp-content/uploads/2014/12/time-to-predictions-2-11.png)**Figure 2:** Cumulative probability of prediction falling less than X years from date of writing ([minPY; from MIRI AI predictions dataset](http://aiimpacts.org/miri-ai-predictions-dataset/ "MIRI AI Predictions Dataset"))
#### Survey data

[The surveys we know of](http://aiimpacts.org/ai-timeline-surveys/) provide some evidence against early predictions being more optimistic. [Michie’s survey](http://aiimpacts.org/michie-survey/) is the only survey we know of made more than ten years ago. Ten surveys have been made since 2005 that give median predictions or median fiftieth percentile dates. The median date in Michie’s survey is the third furthest out in the set of eleven: fifty years, compared to common twenty to forty year medians now. Michie’s survey does not appear to have involved options between twenty and fifty years however, making this result less informative. However it suggests the researchers in the survey were not substantially more optimistic than researchers in modern surveys. They were also apparently more pessimistic than the six early statements from the MIRI dataset discussed above, though some difference [should be expected](http://aiimpacts.org/ai-timeline-predictions-in-surveys-and-statements/) from comparing a survey with public statements. Michie’s survey had sixty-three respondents, compared to the MIRI dataset’s six, making this substantial evidence.

[![Survey results, as shown in Michie's paper.](http://aiimpacts.org/wp-content/uploads/2015/01/michietimelines-copy-1.jpg)](http://aiimpacts.org/wp-content/uploads/2015/01/michietimelines-copy-1.jpg)**Figure 3:** Survey results, as shown in Michie’s paper.
#### Armstrong and Sotala on failed past predictions

[Armstrong and Sotala](http://intelligence.org/files/PredictingAI.pdf) compare failed predictions of the past to all predictions, in an older version of the same dataset. This is not the same as comparing historical to contemporary predictions, but related. In particular, they use a subset of past predictions to conclude that contemporary predictions are likely to be similarly error-prone.

They find that predictions which are old enough to be proved wrong form a similar distribution to the entire set of predictions (see Figures 5-6). They infer that recent predictions are likely to be flawed in similar ways to the predictions we know to be wrong.

This inference appears to us to be wrong, due to a selection bias. Figure 4 illustrates how such reasoning fails with a simple scenario where the failure is clear. In this scenario, for thirty years people have divided their predictions between ten, twenty and thirty years out. In 2010 (the fictional present), many of these predictions are known to have been wrong (shown in pink). In order for the total distribution of predictions to match that of the past failed predictions, the modern predictions have to form a very distribution from the historical predictions (e.g. one such distribution is shown in the ‘2010’ row).

[![xxx](http://aiimpacts.org/wp-content/uploads/2014/12/by-error-or-time-copy-300x192.jpg)](http://aiimpacts.org/wp-content/uploads/2014/12/by-error-or-time-copy.jpg)**Figure** **4:** a hypothetical distribution of predictions.
In general, failed predictions are disproportionately short early predictions, and also disproportionately short bad predictions. If the distributions of failed and total predictions look the same, this suggests that the distribution of early and late predictions is *not* the same – the later predictions must include fewer longer predictions, to make up for the longer unfalsified predictions inherited from the earlier dates, as well as the longer predictions effectively missing from the earlier dates.

If the characteristic lengths of the predictions was small relative to the time between different predictions, these biases would be small. In the example in figure 4, if the distance between the groups had been one hundred years, there would be no problem. However in the MIRI dataset, both are around twenty years.

In sum, the fact that failed predictions look like all predictions suggests that historical predictions came from a different distribution to present predictions. Which would seem to be good news about present predictions, if past predictions were bad. However, we have other evidence from comparing predictions from before and after 2000 directly that those are fairly similar (see above), so if earlier methods were unsuccessful, this is some mark against current methods. On the other hand, public predictions from before 1980 appear to be systematically more optimistic.

[![xxx](http://aiimpacts.org/wp-content/uploads/2014/12/expert-lay-a-and-s-copy.png)](http://aiimpacts.org/wp-content/uploads/2014/12/expert-lay-a-and-s-copy.png)**Figure 5:** Figures from [Armstrong and Sotala](http://intelligence.org/files/PredictingAI.pdf)
[![xxx](http://aiimpacts.org/wp-content/uploads/2014/12/failed-ai-predictions-a-and-s-copy.jpg)](http://aiimpacts.org/wp-content/uploads/2014/12/failed-ai-predictions-a-and-s-copy.jpg)**Figure 6:** Figure from [Armstrong and Sotala](http://intelligence.org/files/PredictingAI.pdf)
### Implications

**Accuracy of AI predictions:** if people make fairly similar predictions over time, this is some evidence that they are not making their predictions based on information about their environment, which has changed over the decades (at a minimum, time has passed). For instance, some suspect that people make their predictions a fixed number of years into the future, to maximize their personal benefits from making exciting but hard to verify predictions. Evidence for this particular hypothesis seems weak to us, however the general point stands. This evidence against accuracy is not as strong as it may seem however, since there appear to be reasonable prior distributions over the date of AI which look the same after seeing time pass.


Human-Level AI

*Published 23 January 2014, last updated Aug 7 2022*

‘*Human-level AI’* refers to AI which can reproduce everything a human can do, approximately. Several variants of this concept are worth distinguishing.

Details
-------

### Variations in the meaning of ‘human-level AI’

Considerations in specifying ‘human-level AI’ more precisely:

* **Do we mean to imply anything about running costs?** Is an AI that reproduces human behavior for ten billion dollars per year ‘human-level’, or does it need to be human-priced? See *‘human-level at any cost vs. human-level at human cost’* below for more details.
* **What characteristics of a human need to be reproduced?** Usually we do not mean that the AI should be indistinguishable from a human. For instance, we usually do not care whether it looks like a human. A common requirement is that the AI have the economically valuable skills of a human. We sometimes also talk about AI being ‘human-level’ in a narrower set of relevant characteristics, such as in its ability to do further AI research.
* **What does it mean to reproduce human behavior?** If AI replaces all hairdressers in society, but uniformly produces a slightly worse haircut in some dimensions (but so cheaply!), does that count as ‘human-level’? If not, then all humans may be replaced even though AI is not ‘human-level’. On the other hand, if this does count, then where is the line? Can an AI ‘reproduce human behavior’ by merely producing anything a buyer would prefer to have than what a human produces? Many machines already do this, and this is not what we mean.
* **How much of human behavior needs to be reproduced?** If AI cannot entirely compete with humans for the job of waiter, merely because some small population prefers human waiters, this will not make a large difference to anything, so a requirement that human-level AI replace humans in all economically useful skills is too high a bar for what we are intuitively interested in. There is a further question of what metric one might use when specifying the bar.
* **What conditions is the AI available under?** Does it matter if it has actually been built? Need it be available in some particular marketplace, or quantity, or price range?

### Related definitions

#### High Level Machine Intelligence (HLMI)

The 2016 and 2022 Expert Surveys on Progress in AI use the following definition:

> Say we have ‘high-level machine intelligence’ when unaided machines can accomplish every task better and more cheaply than human workers. Ignore aspects of tasks for which being a human is intrinsically advantageous, e.g. being accepted as a jury member. Think feasibility, not adoption.
>
>

#### Superhuman AI

A ‘superhuman’ system is meaningfully more capable than a human-level system. In practice the first human-level system [is likely](https://aiimpacts.org/human-level-ai/#Human-level_is_superhuman) to be superhuman.

### Key issues

#### Human-level at any cost vs. human-level at human cost

In common usage, ‘human-level’ AI can mean either AI which can reproduce a human at any cost and speed, or AI which can replace a human (i.e. is as cheap as a human, and can be used in the same situations). Both are relevant for different issues. For instance, the ‘at any cost’ meaning is important when considering how people will respond to human-level artificial intelligence, or whether a human-level artificial intelligence will use illicit means to acquire resources and cause destruction. Human-level at human cost is the relevant concept when thinking about AI replacing humans in the labor market, the economy growing very fast, or legitimate AI development ramping up into an intelligence explosion.

Today few applications are more than an order of magnitude more expensive to run than a human, suggesting a short time before an AI project came down in price to the cost of a human. However some applications are more expensive, and even if an early AI project were only a few orders of magnitude more expensive than a human per time, it may be much slower. Thus it is hard to make useful inferences about the potential time delay between an arbitrarily expensive human-level AI and an AI which might replace a human, even if we assume hardware continues to fall in price regularly.

#### ‘Human-level’ is superhuman

As explained at the [Superintelligence Reading Group](http://lesswrong.com/lw/ku6/superintelligence_reading_group_section_1_past/):

> Another thing to be aware of is the diversity of mental skills. If by ‘human-level’ we mean a machine that is at least as good as a human at each of these skills, then in practice the first ‘human-level’ machine will be much better than a human on many of those skills. It may not seem ‘human-level’ so much as ‘very super-human’.
>
>
> We could instead think of human-level as closer to ‘competitive with a human’ – where the machine has some super-human talents and lacks some skills humans have. This is not usually used, I think because it is hard to define in a meaningful way. There are already machines for which a company is willing to pay more than a human: in this sense a microscope might be ‘super-human’. There is no reason for a machine which is equal in value to a human to have the traits we are interested in talking about here, such as agency, superior cognitive abilities or the tendency to drive humans out of work and shape the future. Thus we talk about AI which is at least as good as a human, but you should beware that the predictions made about such an entity may apply before the entity is technically ‘human-level’.
>
>
> [![From Superintelligence Reading Group](http://aiimpacts.wpengine.com/wp-content/uploads/2014/12/human-level-AI-300x230.png)](http://aiimpacts.wpengine.com/wp-content/uploads/2014/12/human-level-AI.png)Example of how the first ‘human-level’ AI may surpass humans in many ways.

Human-Level AI