Guide to pages on AI timeline predictions

This page is an informal outline of the other pages on this site about AI timeline predictions made by others. Headings link to higher level pages, intended to summarize the evidence from pages below them. This list was complete on 7 April 2017 ([here](http://aiimpacts.org/category/ai-timelines/predictions-of-human-level-ai-timelines/) is a category that may contain newer entries, though not conveniently organized).

Guide
-----

### Topic synthesis: AI timeline predictions as evidence ([page](http://aiimpacts.org/predictions-of-human-level-ai-timelines/))

>
> #### **The predictions themselves:**
>
>
> ##### —from surveys ([page](http://aiimpacts.org/ai-timeline-surveys/)):
>
>
> 1. [2016 Expert survey on progress in AI](http://aiimpacts.org/2016-expert-survey-on-progress-in-ai/): our own survey.
> 	* ([Concrete tasks](http://aiimpacts.org/concrete-ai-tasks-for-forecasting/) that we asked for forecasts on)
> 2. [Müller and Bostrom AI Progress Poll](http://aiimpacts.org/muller-and-bostrom-ai-progress-poll/): the most recent survey with available results, including 29 of the most cited AI researchers as participants.
> 3. [Hanson AI Expert Survey](http://aiimpacts.org/hanson-ai-expert-survey/): in which researchers judge fractional progress toward human-level performance over their careers, in a series of informal conversations.
> 4. [Kruel AI survey](http://aiimpacts.org/kruel-ai-survey/): in which experts give forecasts and detailed thoughts, interview style.
> 5. [FHI Winter Intelligence Survey](http://aiimpacts.org/fhi-ai-timelines-survey/): in which impacts-concerned AGI conference attendees forecast AI in 2011.
> 6. [AGI-09 Survey](http://aiimpacts.org/agi-09-survey/): in which AGI conference attendees forecast various human-levels of AI in 2009.
> 7. [Klein AGI survey](http://aiimpacts.org/klein-agi-survey/): in which a guy with a blog polls his readers.
> 8. [AI@50 survey](http://aiimpacts.org/ai50-survey/): in which miscellaneous conference goers are polled informally.
> 9. [Bainbridge Survey](http://aiimpacts.org/bainbridge-survey/): in which 26 expert technologists expect human-level AI in 2085 and give it a 5.6/10 rating on benefit to humanity.
> 10. [Michie Survey](http://aiimpacts.org/michie-survey/): in which 67 AI and CS researchers are not especially optimistic in the ‘70s.
>
>
> ##### —from public statements:
>
>
> 1. [MIRI AI predictions dataset](http://aiimpacts.org/miri-ai-predictions-dataset/): a big collection of public predictions gathered from the internet.
>
>
> ##### —from written analyses ([page](http://aiimpacts.org/list-of-analyses-of-time-to-human-level-ai/)), for example:
>
>
> 1. [The Singularity is Near](http://aiimpacts.org/kurzweil-the-singularity-is-near/): in which a technological singularity is predicted in 2045, based on when hardware is extrapolated to compute radically more than human minds in total.
> 2. [The Singularity Isn’t Near](http://aiimpacts.org/allen-the-singularity-isnt-near/): in which it is countered that human-level AI requires software as well as hardware, and none of the routes to producing software will get there by 2045.
> 3. (Several others are listed in the [analyses](http://aiimpacts.org/list-of-analyses-of-time-to-human-level-ai/) page above, but do not have their own summary pages.)
>
>
>

>
> #### **On what to infer from the predictions**
>
>
> ##### Some considerations regarding accuracy and bias ([page](http://aiimpacts.org/accuracy-of-ai-predictions/)):
>
>
> 1. Contra a common view that past AI forecasts were unreasonably optimistic, [AI predictions look fairly similar over time](http://aiimpacts.org/similarity-between-historical-and-contemporary-ai-predictions/), except a handful of very early somewhat optimistic ones.
> 2. [The Maes Garreau Law claims that people tend to predict AI near the end of their own expected lifetime. It is not true.](http://aiimpacts.org/the-maes-garreau-law/)
> 3. We expect [publication biases to favor earlier forecasts](http://aiimpacts.org/short-prediction-publication-biases/).
> 4. [Predictions made in surveys seem to be overall a bit later than those made in public statements](http://aiimpacts.org/ai-timeline-predictions-in-surveys-and-statements/) (maybe because surveys prevent some publication biases).
> 5. People who are inclined toward optimism about AI are more likely to become AI researchers, leading to a [selection bias from optimistic experts](http://aiimpacts.org/bias-from-optimistic-predictors/).
> 6. We know of some [differences in forecasts made by different groups](http://aiimpacts.org/group-differences-in-ai-predictions/).
>
>
>

---

**Blog posts on these topics:**

* *[A summary of AI surveys](http://aiimpacts.org/a-summary-of-ai-surveys/)*
* *[Michie and overoptimism](http://aiimpacts.org/michie-and-overoptimism/)*
* *[Are AI surveys seeing the inside view?](http://aiimpacts.org/are-ai-surveys-seeing-the-inside-view/)*
* *[Update on all the AI predictions](http://aiimpacts.org/update-on-all-the-ai-predictions/)*
* *[How AI timelines are estimated](http://aiimpacts.org/how-ai-timelines-are-estimated/)*
* *[Metasurvey: predict the predictors](http://aiimpacts.org/metasurvey-predict-the-predictors/)*
* *[Concrete AI tasks bleg](http://aiimpacts.org/concrete-ai-tasks-bleg/)*
* *[Why do AGI researchers expect AI so soon?](http://aiimpacts.org/why-do-agi-researchers-expect-ai-so-soon/)*

---


Progress in general purpose factoring

The largest number factored to date grew by about 4.5 decimal digits per year over the past roughly half-century. Between 1988, when we first have good records, and 2009, when the largest number to date was factored, progress was roughly 6 decimal digits per year.

Progress was relatively smooth during the two decades for which we have good records, with half of new records being less than two years after the previous record, and the biggest advances between two records representing about five years of prior average progress.

Computing hardware used in record-setting factorings increased ten-thousand-fold over the records (roughly in line with falling computing prices), and we are unsure how much of overall factoring progress is due to this.

Support
-------

### Background

To [factor an integer](https://en.wikipedia.org/wiki/Integer_factorization) N is to find two integers *l* and *m* such that *l\*m* = N. There is no known efficient algorithm for the general case of this problem. While there are [special purpose algorithms](https://en.wikipedia.org/wiki/Integer_factorization#Special-purpose) for some kinds of numbers with particular forms, here we are interested in ‘[general purpose](https://en.wikipedia.org/wiki/Integer_factorization#General-purpose)‘ factoring algorithms. These factor any composite number with running time only dependent on the size of that number.[1](https://aiimpacts.org/progress-in-general-purpose-factoring/#easy-footnote-bottom-1-758 "By “general purpose”, we mean a factoring algorithms whose running time is dependent upon only the size of the number being factored (i.e. not on the size of the prime factors or any particular form of the number). – Scott Contini")

Factoring numbers large enough to break records frequently takes months or years, even with a large number of computers. For instance, RSA-768, the largest number to be factored to date, had 232 decimal digits and was factored over multiple years ending in 2009, using the equivalent of almost 2000 years of computing on a single 2.2 GHz AMD Opteron processor with 2GB RAM.[2](https://aiimpacts.org/progress-in-general-purpose-factoring/#easy-footnote-bottom-2-758 "The following effort was involved. We spent half a year on 80 processors on polynomial selection. This was about 3% of the main task, the sieving, which was done on many hundreds of machines and took almost two years. On a single core 2.2 GHz AMD Opteron processor with 2 GB RAM, sieving would have taken about fifteen hundred years. …Our computation required more than 1020 operations. With the equivalent of almost 2000 years of computing on a single core 2.2GHz AMD Opteron, on the order of 267 instructions were carried out. – Factorization of a 768-bit RSA modulus, Kleinjung et al, 2010")

It is important to know what size numbers can be factored with current technology, because factoring large numbers is central to cryptographic security schemes such as [RSA](https://en.wikipedia.org/wiki/RSA_(cryptosystem)).[3](https://aiimpacts.org/progress-in-general-purpose-factoring/#easy-footnote-bottom-3-758 "When the numbers are very large, no efficient, non-quantum integer factorization algorithm is known…However, it has not been proven that no efficient algorithm exists. The presumed difficulty of this problem is at the heart of widely used algorithms in cryptography such as RSA. – Integer Factorization, Wikipedia, accessed Feb 10 2017") Much of the specific data we have on progress in factoring is from the [RSA Factoring Challenge:](https://en.wikipedia.org/wiki/RSA_Factoring_Challenge) a contest funded by [RSA Laboratories](https://en.wikipedia.org/wiki/RSA_Security), offering large cash prizes for factoring various numbers on their [list](https://en.wikipedia.org/wiki/RSA_numbers) of 54 (the ‘[RSA Numbers](https://en.wikipedia.org/wiki/RSA_numbers)‘).[4](https://aiimpacts.org/progress-in-general-purpose-factoring/#easy-footnote-bottom-4-758 "‘The RSA Factoring challenge was an effort, sponsored by RSA Laboratories, to learn about the actual difficulty of factoring large numbers of the type used in RSA keys.’ –RSA Factoring Challenge FAQ") Their numbers are [semiprimes](https://en.wikipedia.org/wiki/Semiprimes) (i.e. each has only two prime factors), and each number has between 100 and 617 decimal digits.[5](https://aiimpacts.org/progress-in-general-purpose-factoring/#easy-footnote-bottom-5-758 "In mathematics, the RSA numbers are a set of large semiprimes (numbers with exactly two prime factors) that are part of the RSA Factoring Challenge….RSA Laboratories published a number of semiprimes with 100 to 617 decimal digits. Cash prizes of varying size were offered for factorization of some of them. – Wikipedia")

The records collected on this page are for factoring specific large numbers, using algorithms whose performance depends on nothing but the scale of the number (general purpose algorithms). So a record for a specific N-digit number does not imply that that was the first time any N-digit number was factored. However if it is an important record, it strongly suggests that it was difficult to factor arbitrary N-digit numbers at the time, so others are unlikely to have been factored very much earlier, using general purpose algorithms. For a sense of scale, our records here include at least eight numbers that were not the largest ever factored at the time yet appear to have been considered difficult to factor, and many of those were factored five to seven years after the first time a number so large was factored. For numbers that are the first of their size in our records, we expect that they mostly are the first of that size to have been factored, with the exception of early records.[6](https://aiimpacts.org/progress-in-general-purpose-factoring/#easy-footnote-bottom-6-758 "This is because record-breaking numbers appear to be expensive to factor and so unlikely to be done without good reason, or without the attempt being known, and authors on this topic appear to focus on the number of digits that can be factored, so if other larger numbers had been factored earlier, we expect that we would have seen them mentioned. The smallest RSA numbers are an exception, because they appear to have been selected to already be achievable when the contest began, and we do not have much data from before the contest.")

Once a number of a certain size has been factored, it will still take years before numbers at that scale can be cheaply factored. For instance, while 116 digit numbers had been factored by 1991 (see below), it was an achievement in 1996 to [factor](http://www.crypto-world.com/announcements/siqs116.text) a different specific 116 digit number, which had been on a ‘most wanted list’.

If we say a digit record is broken in some year (rather than a record for a particular number), we mean that that is the first time that any number with at least that many digits has been factored using a [general purpose factoring algorithm.](https://en.wikipedia.org/wiki/Integer_factorization#General-purpose)

Our impression is that length of numbers factored is a measure people were trying to make progress on, so that progress here is the result of relatively consistent effort to make progress, rather than occasional incidental overlap with some other goal.

Most of the research on this page is adapted from Katja Grace’s [2013 report](https://intelligence.org/files/AlgorithmicProgress.pdf), with substantial revision.

### Recent rate of progress

#### Data sources

These are the sources of data we draw on for factoring records:

* Wikipedia’s [table](https://en.wikipedia.org/wiki/RSA_Factoring_Challenge) of ‘RSA numbers’, from the [RSA factoring challenge](https://en.wikipedia.org/wiki/RSA_Factoring_Challenge) (active ’91-’07). Of these 19 have been factored, of 54 total. The table includes numbers of digits, dates, and cash prizes offered. ([data](https://en.wikipedia.org/wiki/RSA_Factoring_Challenge))
* Scott Contini’s list of ‘[general purpose factoring records’ since 1990](http://www.crypto-world.com/FactorRecords.html)’, which includes the nine RSA numbers that set digit records, and three numbers that appear to have set digit records as part of the ‘[Cunningham Project](https://en.wikipedia.org/wiki/Cunningham_project)‘. This list contains digits, date, solution time, and algorithm used.
* Two estimates of how many digits could be factored in earlier decades from an [essay](http://www.ams.org/notices/199612/pomerance.pdf) by Carl Pomerance.[7](https://aiimpacts.org/progress-in-general-purpose-factoring/#easy-footnote-bottom-7-758 "“It is the best of times for the game of factoring large numbers into their prime factors. In 1970 it was barely possible to factor “hard” 20-digit numbers. In 1980, in the heyday of the Brillhart-Morrison continued fraction factoring algorithm, factoring of 50-digit numbers was becoming commonplace. In 1990 my own quadratic sieve factoring algorithm had doubled the length of the numbers that could be factored, the record having 116 digits.” – A Tale of Two Sieves, Carl Pomerance, 1996")
* [This announcement](http://www.crypto-world.com/announcements/RSA120.txt), that suggests that the 116 digit general purpose factoring record was set in January 1991, contrary to Factorworld and Pomerance.[8](https://aiimpacts.org/progress-in-general-purpose-factoring/#easy-footnote-bottom-8-758 "

‘This is a new general purpose factoring record, beating the old 116-digit

record that was set in January 1991, more than two years ago.’ – Factorization of RSA-120, Lenstra, 1993 ") We will ignore it, since the dates are too close to matter substantially, and the other two sources agree.
* A 1988 paper [discussing](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.81.1434&rep=rep1&type=pdf) recent work and constraints on possibility at that time. It lists four ‘state of the art’ efforts at factorization, among which the largest number factored using a general purpose algorithm ([MPQS](https://en.wikipedia.org/wiki/Quadratic_sieve#Multiple_polynomials)) has 95 digits.[9](https://aiimpacts.org/progress-in-general-purpose-factoring/#easy-footnote-bottom-9-758 "Before making our question more precise, let us illustrate its vagueness with four examples which, in the summer of 1988, represented the state of the art in factoring.

(i) In [7, 24] Bob Silverman et al. describe their implementation of the multiple polynomial quadratic sieve algorithm (mpqs) on a network of 24 SUN-3 workstations. Using the idle cycles on these workstations, 90 digit integers have been factored in about six weeks (elapsed time).

(ii) In [21] Herman te Riele et al. describe their implementation of the same algorithm on two different supercomputers. They factored a 92 digit integer using 95 hours of CPU time on a NEC SX-2.

(iii) ‘Red’ Alford and Carl Pomerance implemented mpqs on 100 IBM PC’s; it took them about four months to factor a 95 digit integer.

(iv) In [20] Carl Pomerance et al. propose to build a special purpose mpqs machine ‘which should cost about $20,000 in parts to build and which should be able to factor 100 digit integers in a month.’

– Lenstra et al 1988") It claims that 106 digits had been factored by 1988, which implies that the early RSA challenge numbers were not state of the art. [10](https://aiimpacts.org/progress-in-general-purpose-factoring/#easy-footnote-bottom-10-758 "“At the time Of writing this paper we have factored two 93, one 96, one 100, one 102, and 358 one 106 digit number using mpqs, and we m working on a 103 digit number, for all these numbers extensive ecm attempts had failed. “- Lenstra et al 1988 ") Together these suggest that the work in the paper is responsible for moving the record from 95 to 106 digits, and this matches our impressions from elsewhere though we do not know of a specific claim to this effect.
* This ‘[RSA honor roll](http://www.ontko.com/pub/rayo/primes/hr_rsa.txt)‘ contains meta-data for the RSA solutions.
* [Cryptography and Computational Number Theory](https://books.google.com/books?id=yyfS7MKQhJUC&lpg=PA45&ots=ZuITD7DllA&dq=alford%20pomerance%2095%20digit&pg=PA44#v=onepage&q=107-digit&f=false) (1989), Carl Pomerance and Shafi Goldwasser.[11](https://aiimpacts.org/progress-in-general-purpose-factoring/#easy-footnote-bottom-11-758 "e.g. p44 mentions the 107-digit record and some details.")

*Excel spreadsheet containing our data for download: [Factoring data 2017](http://aiimpacts.org/wp-content/uploads/2017/03/Factoring-data-2017-1.xlsx)*

### Trends

#### Digits by year

Figure 1 shows how the scale of numbers that could be factored (using general purpose methods) grew over the last half-century (as of 2017). In red are the numbers that broke the record for the largest number of digits, as far as we know.

From it we see that since 1970, the numbers that can be factored have increased from around twenty digits to 232 digits, for an average of about 4.5 digits per year.

After the first record we have in 1988, we know of thirteen more records being set, for an average of one every 1.6 years between 1988 and 2009. Half of these were set the same or the following year as the last record, and the largest gap between records was four years. As of 2017, seven years have passed without further records being set.

The largest amount of progress seen in a single step is the last one—32 additional digits at once, or over five years of progress at the average rate seen since 1988 just prior to that point. The 200 digit record was also around five years of progress.

[![](http://aiimpacts.org/wp-content/uploads/2017/03/Factoring-records-March-2017-with-labels-and-axes.png)](http://aiimpacts.org/wp-content/uploads/2017/03/Factoring-records-March-2017-with-labels-and-axes.png)Figure 1: Size of numbers (in decimal digits) that could be factored over recent history. Green ‘approximate state of the art’ points do not necessarily represent specific numbers or the very largest that could be at that time—they are qualitative estimates. The other points represent specific large numbers being factored, either as the first number of that size ever to be factored (red) or not (orange). Dates are accurate to the year. Some points are annotated with decimal digit size, for ease of reading.

#### Hardware inputs

New digit records tend to use more computation, which makes progress in software alone hard to measure. At any point in the past it was in principle possible to factor larger numbers with more hardware. So the records we see are effectively records for what can be done with however much hardware anyone is willing to purchase for the purpose. Which grows from a combination of software improvements, hardware improvements, and increasing wealth among other things. Figure 2 shows how computing used for solutions increased with time.

[![CPU time to factor numbers. Measured in 1,000 MIPS-years before 2000, and in GHz-years after 2000; note that these are not directly comparable, and the conversion here is approximate. Data from Contini (2010).](http://aiimpacts.org/wp-content/uploads/2017/03/time-to-factor.jpeg)](http://aiimpacts.org/wp-content/uploads/2017/03/time-to-factor.jpeg)Figure 2: CPU time to factor digit-record breaking numbers. Measured in 1,000 MIPS-years before 2000, and in GHz-years after 2000. These are similar, but not directly comparable, so the conversion here is approximate. Data from [Contini (2010)](http://www.crypto-world.com/FactorRecords.html). Figure from [Grace (2013)](https://intelligence.org/files/AlgorithmicProgress.pdf). (Original caption: “Figure 31 shows CPU times for the FactorWorld records. These times have increased by a factor of around ten thousand since 1990. At 2000 the data changes from being in MIPS-years to 1 GHz CPU-years. These aren’t directly comparable. The figure uses 1 GHz CPU-year = 1,000 MIPS-years, because it is in the right ballpark and simple, and no estimates were forthcoming. The figure suggests that a GHz CPU-year is in fact worth a little more, given that the data seems to dip around 2000 with this conversion.”)

In the two decades between 1990 and 2010, figure 2 suggests that computing used has increased by about four orders of magnitude. During that time computing available per dollar has probably increased by a factor of [ten every four years or so](http://aiimpacts.org/trends-in-the-cost-of-computing/), for about five orders of magnitude. So we are seeing something like digits that can be factored at a fixed expense.

Further research
----------------

* Discover how computation used is expected to scale with the number of digits factored, and use that to factor out increased hardware use from this trendline, and so measure non-hardware progress alone.
* This area appears to have seen a small number of new algorithms, among smaller changes in how they are implemented. Check how much the new algorithms affected progress, and similarly for anything else with apparent potential for large impacts (e.g. a move to borrowing other people’s spare computing hardware via the internet, rather than paying for hardware).
* Find records from earlier times.
* Some numbers had large prizes associated with their factoring, and others of similar sizes had none. Examine the relationship between progress and financial incentives in this case.
* [The Cunningham Project](https://homes.cerias.purdue.edu/~ssw/cun/index.html) maintains a vast collection of recorded factorings of numbers, across many scales, along with dates, algorithms used, and people or projects responsible. Gather that data, and use to make similar inferences to the data we have here (see *Relevance* section below for more on that).

Relevance
---------

We are interested in factoring, because it is an example of an algorithmic problem on which there has been well-documented progress. Such examples should inform our expectations for algorithmic problems in general (including problems in AI), regarding:

* How smooth or jumpy progress tends to be, and related characteristics of its shape.
* How much warning there is of rapid progress.
* How events that are qualitatively considered ‘conceptual insights’ or ‘important progress’ relate to measured performance progress.
* How software progress interacts with hardware (for instance, does a larger step of software progress cause a disproportionate increase in overall software output, because of redistribution of hardware?).
* If performance is improving, how much of that is because of better hardware, and how much is because of better algorithms or other aspects of software.

Assorted sources
----------------

* [Integer factoring](https://www.fdi.ucm.es/profesor/m_alonso/Documentos/factorizacion/arjlensfac.pdf)
* [RSA Factoring Challenge FAQ](https://www.emc.com/emc-plus/rsa-labs/historical/the-rsa-factoring-challenge-faq.htm)
* Other pages around e.g. [this one](https://members.loria.fr/PZimmermann/records/factor-previous.html#general), seem to have data for run times etc of other things
* [Graph of history of factorization with GNFS](https://members.loria.fr/PZimmermann/records/gnfsrecord.jpg)
* More [data on records](https://members.loria.fr/PZimmermann/records/factor.html), probably largely overlapping, including different methods (that I have probably seen elsewhere)
* Announcements such as [this one](http://www.crypto-world.com/announcements/siqs116.text) contain much info, many on Scott Contini’s page.

---

Progress in general purpose factoring


Trends in algorithmic progress

Algorithmic progress has been estimated to contribute fifty to one hundred percent as much as hardware progress to overall performance progress, with low confidence.

Algorithmic improvements appear to be relatively incremental.

Details
-------

*We have not recently examined this topic carefully ourselves. This page currently contains relevant excerpts and sources.*

[Algorithmic Progress in Six Domains](https://intelligence.org/files/AlgorithmicProgress.pdf)[1](https://aiimpacts.org/trends-in-algorithmic-progress/#easy-footnote-bottom-1-784 "Grace, K. (2013), Algorithmic Progress in Six Domains, Machine Intelligence Research Institute, https://intelligence.org/files/AlgorithmicProgress.pdf") measured progress in the following areas, as of 2013:

* Boolean satisfiability
* Chess
* Go
* Largest number factored ([our updated page](http://aiimpacts.org/progress-in-general-purpose-factoring/))
* MIP algorithms
* Machine learning

Some key summary paragraphs from the paper:

Many of these areas appear to experience fast improvement, though the data are often noisy. For tasks in these areas, gains from algorithmic progress have been roughly fifty to one hundred percent as large as those from hardware progress. Improvements tend to be incremental, forming a relatively smooth curve on the scale of years

…

In recent *Boolean satisfiability* (SAT) competitions, SAT solver performance has increased 5–15% per year, depending on the type of problem. However, these gains have been driven by widely varying improvements on particular problems. Retrospective surveys of SAT performance (on problems chosen after the fact) display significantly faster progress.

*Chess programs* have improved by around fifty Elo points per year over the last four decades. Estimates for the significance of hardware improvements are very noisy but are consistent with hardware improvements being responsible for approximately half of all progress. Progress has been smooth on the scale of years since the 1960s, except for the past five.

*Go programs* have improved about one stone per year for the last three decades. Hardware doublings produce diminishing Elo gains on a scale consistent with accounting for around half of all progress.

Improvements in a variety of *physics simulations* (selected after the fact to exhibit performance increases due to software) appear to be roughly half due to hardware progress.

The *largest number factored* to date has grown by about 5.5 digits per year for the last two decades; computing power increased ten-thousand-fold over this period, and it is unclear how much of the increase is due to hardware progress.

Some *mixed integer programming* (MIP) algorithms, run on modern MIP instances with modern hardware, have roughly doubled in speed each year. MIP is an important optimization problem, but one which has been called to attention after the fact due to performance improvements. Other optimization problems have had more inconsistent (and harder to determine) improvements.

Various forms of *machine learning* have had steeply diminishing progress in percentage accuracy over recent decades. Some vision tasks have recently seen faster progress.

Note that these points have not been updated for developments since 2013, and machine learning in particular is generally observed to have seen more progress very recently (as of 2017).

### Figures

Below are assorted figures mass-extracted from [Algorithmic Progress in Six Domains](https://intelligence.org/files/AlgorithmicProgress.pdf), some more self-explanatory than others. See the paper for their descriptions.

[![Page-27-Image-1311](http://aiimpacts.org/wp-content/uploads/2017/03/Page-27-Image-1311.png)](http://aiimpacts.org/wp-content/uploads/2017/03/Page-27-Image-1311.png) [![Page-28-Image-1394](http://aiimpacts.org/wp-content/uploads/2017/03/Page-28-Image-1394.png)](http://aiimpacts.org/wp-content/uploads/2017/03/Page-28-Image-1394.png) [![Page-29-Image-1395](http://aiimpacts.org/wp-content/uploads/2017/03/Page-29-Image-1395.png)](http://aiimpacts.org/wp-content/uploads/2017/03/Page-29-Image-1395.png) [![Page-31-Image-1396](http://aiimpacts.org/wp-content/uploads/2017/03/Page-31-Image-1396.png)](http://aiimpacts.org/wp-content/uploads/2017/03/Page-31-Image-1396.png) [![Page-32-Image-1397](http://aiimpacts.org/wp-content/uploads/2017/03/Page-32-Image-1397.png)](http://aiimpacts.org/wp-content/uploads/2017/03/Page-32-Image-1397.png) [![Page-40-Image-1827](http://aiimpacts.org/wp-content/uploads/2017/03/Page-40-Image-1827.png)](http://aiimpacts.org/wp-content/uploads/2017/03/Page-40-Image-1827.png) [![Page-41-Image-1828](http://aiimpacts.org/wp-content/uploads/2017/03/Page-41-Image-1828.png)](http://aiimpacts.org/wp-content/uploads/2017/03/Page-41-Image-1828.png) [![Page-42-Image-1829](http://aiimpacts.org/wp-content/uploads/2017/03/Page-42-Image-1829.png)](http://aiimpacts.org/wp-content/uploads/2017/03/Page-42-Image-1829.png) [![Page-43-Image-1830](http://aiimpacts.org/wp-content/uploads/2017/03/Page-43-Image-1830.png)](http://aiimpacts.org/wp-content/uploads/2017/03/Page-43-Image-1830.png) [![Page-43-Image-1831](http://aiimpacts.org/wp-content/uploads/2017/03/Page-43-Image-1831.png)](http://aiimpacts.org/wp-content/uploads/2017/03/Page-43-Image-1831.png) [![Page-47-Image-1832](http://aiimpacts.org/wp-content/uploads/2017/03/Page-47-Image-1832.png)](http://aiimpacts.org/wp-content/uploads/2017/03/Page-47-Image-1832.png) [![Page-48-Image-1833](http://aiimpacts.org/wp-content/uploads/2017/03/Page-48-Image-1833.png)](http://aiimpacts.org/wp-content/uploads/2017/03/Page-48-Image-1833.png) [![Page-48-Image-1834](http://aiimpacts.org/wp-content/uploads/2017/03/Page-48-Image-1834.png)](http://aiimpacts.org/wp-content/uploads/2017/03/Page-48-Image-1834.png) [![Page-49-Image-1835](http://aiimpacts.org/wp-content/uploads/2017/03/Page-49-Image-1835.png)](http://aiimpacts.org/wp-content/uploads/2017/03/Page-49-Image-1835.png) [![Page-50-Image-1836](http://aiimpacts.org/wp-content/uploads/2017/03/Page-50-Image-1836.png)](http://aiimpacts.org/wp-content/uploads/2017/03/Page-50-Image-1836.png) [![Page-51-Image-1837](http://aiimpacts.org/wp-content/uploads/2017/03/Page-51-Image-1837.png)](http://aiimpacts.org/wp-content/uploads/2017/03/Page-51-Image-1837.png) [![Page-51-Image-1838](http://aiimpacts.org/wp-content/uploads/2017/03/Page-51-Image-1838.png)](http://aiimpacts.org/wp-content/uploads/2017/03/Page-51-Image-1838.png) [![Page-52-Image-1839](http://aiimpacts.org/wp-content/uploads/2017/03/Page-52-Image-1839.png)](http://aiimpacts.org/wp-content/uploads/2017/03/Page-52-Image-1839.png) [![Page-52-Image-1840](http://aiimpacts.org/wp-content/uploads/2017/03/Page-52-Image-1840.png)](http://aiimpacts.org/wp-content/uploads/2017/03/Page-52-Image-1840.png) [![Page-53-Image-1841](http://aiimpacts.org/wp-content/uploads/2017/03/Page-53-Image-1841.png)](http://aiimpacts.org/wp-content/uploads/2017/03/Page-53-Image-1841.png) [![Page-53-Image-1842](http://aiimpacts.org/wp-content/uploads/2017/03/Page-53-Image-1842.png)](http://aiimpacts.org/wp-content/uploads/2017/03/Page-53-Image-1842.png) [![Page-54-Image-1843](http://aiimpacts.org/wp-content/uploads/2017/03/Page-54-Image-1843.png)](http://aiimpacts.org/wp-content/uploads/2017/03/Page-54-Image-1843.png) [![Page-54-Image-1844](http://aiimpacts.org/wp-content/uploads/2017/03/Page-54-Image-1844.png)](http://aiimpacts.org/wp-content/uploads/2017/03/Page-54-Image-1844.png)

Trends in algorithmic progress


Changes in funding in the AI safety field

*Guest post by Seb Farquhar, originally posted to the [Center for Effective Altruism blog](https://www.centreforeffectivealtruism.org/blog/changes-in-funding-in-the-ai-safety-field). 20 February 2017*

The field of AI Safety has been growing quickly over the last three years, since the publication of “Superintelligence”. One of the things that shapes what the community invests in is an impression of what the composition of the field currently is, and how it has changed. Here, I give an overview of the composition of the field as measured by its funding.

Measures other than funding also matter, and may matter more, like types of outputs, distribution of employed/active people, or impact-adjusted distributions of either. Funding, however, is a little more objective and easier to assess. It gives us some sense of how the AI Safety community is prioritising, and where it might have blind spots. For a fuller discussion of the shortcomings of this type of analysis, and of this data, see section four.

Throughout, I am including the budgets of organisations who are explicitly working to reduce existential risk from machine superintelligence. It does not include work outside the AI Safety community, on areas like verification and control, that might prove relevant. This kind of work, which happens in mainstream computer science research, is much harder to assess for relevance and to get budget data for. I am trying as much as possible to count money spent at the time of the work, rather than the time at which a grant is announced or money is set aside.

Thanks to Niel Bowerman, Ryan Carey, Andrew Critch, Daniel Dewey, Viktoriya Krakovna, Peter McIntyre, Michael Page for their comments or help on content or gathering data in preparing this document (though nothing here should be taken as a statement of their views and any errors are mine).

The post is organised as follows:

1. Narrative of growth in AI Safety funding
2. Distribution of spending
3. Soft conclusions from overview
4. Caveats and assumptions

Narrative of growth in AI Safety funding
----------------------------------------

The AI Safety community grew significantly in the last three years. In 2014, AI Safety work was almost entirely done at the Future of Humanity Institute (FHI) and the Machine Intelligence Research Institute (MIRI) who were between them spending $1.75m. In 2016, more than 50 organisations have explicit AI Safety related programs, spending perhaps $6.6m. Note the caveats to all numbers in this document described in section 4.

[![chart](http://aiimpacts.org/wp-content/uploads/2017/02/chart.png)](http://aiimpacts.org/wp-content/uploads/2017/02/chart.png)

In 2015, AI Safety spending roughly doubled to $3.3m. Most of this came from growth at MIRI and the beginnings of involvement by industry researchers.

In 2016, grants from the Future of Life Institute (FLI) triggered growth in smaller-scale technical AI safety work.[1](https://aiimpacts.org/changes-in-funding-in-the-ai-safety-field/#easy-footnote-bottom-1-773 "Although grants were awarded in 2015, there is a lag between grants being awarded and work taking place. This is a significant assumption discussed in the caveats.") Industry invested more over 2016, specially at Google DeepMind and potentially at OpenAI.[2](https://aiimpacts.org/changes-in-funding-in-the-ai-safety-field/#easy-footnote-bottom-2-773 "Although note that most of the new hires at DeepMind arrived right at the end of the year.") Because of their high salary costs, the monetary growth in spending at these firms may overstate actual growth of the field. For example, several key researchers moved from non-profits/academic orgs (MIRI, FLI, FHI) to Google DeepMind and OpenAI. This increased spending significantly, but may have had a smaller effect on output.[3](https://aiimpacts.org/changes-in-funding-in-the-ai-safety-field/#easy-footnote-bottom-3-773 "Although it is also conceivable that a researcher at DeepMind may be ten times more valuable than that same researcher elsewhere.") AI Strategy budgets grew more slowly, at about 20%.

In 2017, multiple center grants are emerging (such as the Center for Human-Compatible AI (CHCAI) and Center for the Future of Intelligence (CFI)), but if their hiring is slow it will restrain overall spending. FLI grantee projects will be coming to a close over the year, which may mean that technical hires trained through those projects become available to join larger centers. The next round of FLI grants may be out in time to bridge existing grant holders onto new projects. Industry teams may keep growing, but there are no existing public commitments to do so. If technical research consolidates into a handful of major teams, it might make it easier to keep open dialogue between research groups, but might decrease individual incentives to because researchers have enough collaboration opportunities locally.

Although little can be said about 2018 at this point, the current round of academic grants which support FLI grantees as well as FHI end in 2018, potentially creating a funding cliff. (Though FLI has just announced a second funding round, and MIT Media Lab has just announced a $27m center (whose exact plans remain unspecified).[4](https://aiimpacts.org/changes-in-funding-in-the-ai-safety-field/#easy-footnote-bottom-4-773 "This will depend on personal circumstance as well as giving opportunities. It would probably be a mistake to forgo time-bounded giving opportunities to cover this cliff, since other sources of funding might be found between now and then.")

**Estimated spending in AI Safety broken down by field of work**

[![S__lection_007](http://aiimpacts.org/wp-content/uploads/2017/02/S__lection_007.png)](http://aiimpacts.org/wp-content/uploads/2017/02/S__lection_007.png)

Distribution of spending
------------------------

In 2014, the field of research was not very diverse. It was roughly evenly split into work at FHI on macrostrategy, with limited technical work, and at MIRI following a relatively focused technical research agenda which placed little emphasis on deep learning.

Since then, the field has diversified significantly.

The academic technical research field is very diverse, though most of the funding comes via FLI. MIRI remains the only non-profit doing technical research and continues to be the largest research group with 7 research fellows at the end of 2016 and a budget of $1.75m. Google DeepMind probably has the second largest technical safety research group with between 3 and 4 full-time-equivalent (FTE) researchers at the end of 2016 (most of whom joined at the end of the year), though OpenAI and GoogleBrain probably have 0.5-1.5 FTEs.[5](https://aiimpacts.org/changes-in-funding-in-the-ai-safety-field/#easy-footnote-bottom-5-773 "This is based on anecdotal hiring information, and not a confirmed number from Google DeepMind.")

FHI and SAIRC remains the only large-scale AI strategy center. The Global Catastrophic Risk Institute is the main long-standing strategy center working on AI, but is much smaller. Some much smaller groups (FLI Grantees and the Global Politics of AI team at Yale) are starting to form, but are mostly low-/no- salary for the time being.

A range of functions are now being filled which did not exist in the AI Safety community before. These include outreach, ethics research, and rationality training. Although explicitly outreach focused projects remain small, organisations like FHI and MIRI do significant outreach work (arguably, Nick Bostrom’s *Superintelligence* falls into this category, for example).

**2017 (forecast) – total = $10.5m**

[![chart](http://aiimpacts.org/wp-content/uploads/2017/02/chart-1.png)](http://aiimpacts.org/wp-content/uploads/2017/02/chart-1.png)

**2016 – total = $6.56m**

[![S__lection_012](http://aiimpacts.org/wp-content/uploads/2017/02/S__lection_012.png)](http://aiimpacts.org/wp-content/uploads/2017/02/S__lection_012.png)

**2015 – total = $3.28m**

[![chart](http://aiimpacts.org/wp-content/uploads/2017/02/chart-2.png)](http://aiimpacts.org/wp-content/uploads/2017/02/chart-2.png)

**2014 – total = $1.75m**

[![chart](http://aiimpacts.org/wp-content/uploads/2017/02/chart-3.png)](http://aiimpacts.org/wp-content/uploads/2017/02/chart-3.png)

Possible implications and tentative suggestions
-----------------------------------------------

Technical safety research

* The MIRI technical agenda remains the largest coherent research project, despite the emergence of several other agendas. For the sake of diversity of approach, more work needs to be done to develop PIs within the AI community to take the [“Concrete Problems](https://arxiv.org/abs/1606.06565)” research agenda and others forwards.
* The community should go out of its way to help the emerging academic technical research centers (CHCAI and Yoshua Bengio’s forthcoming center) to recruit and retain fantastic people.

Strategy, outreach, and policy

* Near-term policy has had a lot of people outside the AI Safety community  moving towards it, though output remains relatively low. There is even less work on medium-term implications of AI.
* Non-technical funding has not kept up with the growth of the AI safety field as a whole. This is likely to be because the pipeline for non-technical work is less easily specified and improved than it is for technical work. This could create gaps in the future, for example in:
* Communication channels between AI Safety research teams.
* Communication between the AI Safety research community and the rest of the AI community.
* Guidance for policy-makers and researchers on long-run strategy.
* It might be helpful to establish or identify a pipeline for AI strategy/policy work, perhaps by building a PhD or Masters course at an existing institution for the purpose.
* There is not a lot of focused AI Safety outreach work. This is largely because all organisations are stepping carefully to avoid messaging that has the potential to frame the issues unconstructively, but it might be worthwhile to step into this gap over the next year or two.

Caveats and assumptions
-----------------------

* **Scope**: I selected projects that either self-identify or were identified to me by people in the field as focused on AI Safety. Where organisations had only a partial focus on AI Safety, I estimated the proportion of their work that was related based on the distribution of their projects. The data probably represent the community of people who explicitly think they are working on AI safety moderately-well. But it **doesn’t include anyone generally working on verification/control, auditing, transparency**, etc. for other reasons. It also excludes people working on near-term AI policy.
* **Forecasting**: Data for 2017 are a very loose guess. In particular, they make very rough guesses for the ability of centers to scale up, which have not been validated by interviews with centers. CFAR financial estimates for 2017 are also still not publicly available, and may be more than 10% of all AI Safety spending. I have assumed, in the pie charts of distribution only, that they will spend $1m next year (they spend $920k in 2015). That estimate is probably too low, but will probably not dramatically alter the overall picture. Forecasts also do not include funding for Yoshua Bengio’s new center or the next round of FLI grants.
* **FLI grant distribution**: I have assumed that all FLI grantees spent according to the following schedule: nothing in 2015, 37% in 2016, 31% in 2017, 32% later. This is based on aggregate data, but will not be right for individual grants, which might mean the distribution of funding over time between fields is slightly wrong. The values are lagged slightly in order to account for the fact that money usually takes several months to make its way through university bureaucracies. In some cases, work happens at a different time from funding being received (earlier or later).
* **Industry spending**: Estimates of industry spending are very rough. I approximated the amount of time spent by individual researchers on AI Safety based on conversations with some of them and with non-industry researchers. I (very) loosely approximated the per-researcher cost to firms at $300k each, inclusive of overheads and compute.
* **Categorisation**: I used the abstracts of the FLI grants, and the websites of other projects, to categorise their work roughly. Some may be miscategorised, but the major chunks of funding are likely to be right.
* **Funding is not a perfect proxy for what matters**: There are many ways of describing change in the field usefully, which include how funding is distributed. Funding is a moderate proxy for the amount of effort going into different approaches, but not perfect. For example, if a researcher were to move from being lightly funded at a non-profit to employed by OpenAI their ‘cost’ in this model will have increased by roughly an order of magnitude, which might be different from their impact. The funding picture may therefore come apart from ‘effort’ especially when comparing DeepMind/OpenAI/GoogleBrain to non-profits like MIRI.
* **Re-granting**: I’ve tried to avoid double-counting (e.g., SAIRC is listed as an FHI project rather than FLI despite being funded by Elon Musk and OpenPhil via FLI), but there is enough regranting going on that I might not have succeeded.
* **Inclusion**: I might have missed out organisations that should arguably be in there, or have incorrect information about their spending
* **Corrections**: If you have corrections or extra information I should incorporate, please email me at seb@prioritisation.org.

---

Footnotes
---------


Funding of AI Research

Provisional data suggests:

* Equity deals made with startups in AI were worth about $5bn in 2016, and this value has been growing by around 50% per year in recent years.
* The number of equity deals in AI startups globally is growing at around 30% per year, and was estimated at 658 in 2016.
* NSF funding of IIS, a section of computer science that appears to include AI and two other areas, has increased at around 9% per year over the past two decades.

(Updated February 2017)

Background
----------

Artificial Intelligence research is funded both publicly and privately. This page currently contains some data on private funding globally, public funding in the US,  and national government announcements of plans relating to AI funding. This page should not currently be regarded as an exhaustive summary of data available on these topics or on AI funding broadly.

Details
-------

### AI startups

According to [CB Insights](https://www.cbinsights.com/blog/artificial-intelligence-startup-funding/), between the start of 2012 and the end of 2016, the number of equity deals being made with startups in artificial intelligence globally grew by a factor of four to 658 (around 30% per year), and the value of funding grew by a factor of over eight to $5 billion (around 50% per year).[1](https://aiimpacts.org/funding-of-ai-research/#easy-footnote-bottom-1-762 "“Our analysis includes companies applying AI algorithms to verticals like healthcare, security, advertising, and finance as well as those developing general-purpose AI tech. Our list excludes robotics (hardware-focused) and AR/VR startups, which we’ve analyzed separately here and here. Our analysis includes all equity funding rounds and convertible notes. This post was updated on 1/19/2017 to include deals through the end of 2016….Deals reached a 5-year high last year, from 160 deals in 2012 to 658 in 2016. Dollars invested also rose considerably in 2016, up about 60%.” – CB Insights, https://www.cbinsights.com/blog/artificial-intelligence-startup-funding/  (See also Figure 1)") Their measure includes both startups developing AI techniques, and those applying existing AI techniques to problems in areas such as healthcare or advertising. They provide Figure 1 below, with further details of the intervening years. We have not checked the trustworthiness or completeness of CB Insights’ data.

[![](http://aiimpacts.org/wp-content/uploads/2017/02/AI_2016_yearly-CBI-1.png)](http://aiimpacts.org/wp-content/uploads/2017/02/AI_2016_yearly-CBI-1.png)Figure 1: Number of new equity deals supporting AI-related startups, and dollar values of disclosed investments over 2012-2015, according to [CB Insights](https://www.cbinsights.com/blog/artificial-intelligence-startup-funding/).

### US National Science Foundation

In 2014, [Muehlhauser and Sinick](https://intelligence.org/2014/01/28/how-big-is-ai/) wrote:

> In 2011, the National Science Foundation (NSF) received $636 million for funding CS research (through [CISE](http://www.nsf.gov/dir/index.jsp?org=CISE)). Of this, $169 million [went to](http://www.nsf.gov/about/budget/fy2013/pdf/06-CISE_fy2013.pdf) Information and Intelligent Systems (IIS). IIS has [three programs](http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=13707&org=IIS&from=home): Cyber-Human Systems (CHS), Information Integration and Informatics (III) and Robust Intelligence (RI). If roughly 1/3 of the funding went to each of these, then $56 million went to Robust Intelligence, so 9% of the total CS funding. (Some CISE funding may have gone to AI work outside of IIS — that is, via [ACI](http://www.nsf.gov/div/index.jsp?div=ACI), [CCF](http://www.nsf.gov/div/index.jsp?div=CCF), or [CNS](http://www.nsf.gov/div/index.jsp?div=CNS) — but at a glance, non-IIS AI funding through CISE looks negligible.)
>
>
> …
>
>
> The NSF Budget for Information and Intelligent Systems (IIS) has generally increased between 4% and 20% per year since 1996, with a one-time percentage boost of 60% in 2003, for a total increase of 530% over the 15 year period between 1996 and 2011.[14 {See table with upper left-hand corner A367 in the [spreadsheet](https://intelligence.org/wp-content/uploads/2014/01/Current-size-past-growth-of-the-AI-field.xlsx).}] “Robust Intelligence” is one of three program areas covered by this budget.
>
>

As of February 2017, CISE (Computer and Information Science and Engineering) covers five categories, and IIS appears to be the most relevant one.[2](https://aiimpacts.org/funding-of-ai-research/#easy-footnote-bottom-2-762 "“…IIS also invests in research on artificial intelligence, computer vision, natural language processing, robotics, machine learning, computational neuroscience, cognitive science, and areas leading to the computational understanding and modeling of intelligence in complex, realistic contexts.” – CISE Funding, Directorate for Computer and Information Science and Engineering (CISE).") IIS still has three programs, of which Robust Intelligence is one.[3](https://aiimpacts.org/funding-of-ai-research/#easy-footnote-bottom-3-762 "CISE’s Division of Information and Intelligent Systems (IIS) supports research and education projects that develop new knowledge in three core programs:

The Cyber-Human Systems (CHS) program;

The Information Integration and Informatics (III) program; and

The Robust Intelligence (RI) program.

– Information and Intelligent Systems (IIS): Core Programs")

NSF funding into both CISE and IIS (the relevant subcategory) from 2009 to 2017 shows a steady rise.[4](https://aiimpacts.org/funding-of-ai-research/#easy-footnote-bottom-4-762 "NSF Budget:

http://www.nsf.gov/about/budget/fy2017/pdf/18_fy2017.pdf

https://www.nsf.gov/about/budget/fy2016/pdf/18_fy2016.pdf

http://www.nsf.gov/about/budget/fy2015/pdf/18_fy2015.pdf

https://www.nsf.gov/about/budget/fy2014/pdf/18_fy2014.pdf

http://www.nsf.gov/about/budget/fy2013/pdf/06-CISE_fy2013.pdf

https://www.nsf.gov/about/budget/fy2012/pdf/17_fy2012.pdf

https://www.nsf.gov/about/budget/fy2011/pdf/06-CISE_fy2011.pdf") IIS funding as a percentage of CISE funding fluctuates, and has gone down in this time period. The following table summarizes data from NSF, collected by Finan Adamson in 2016. The figures below it (2 and 3) combine this data with some collected previously in this [spreadsheet](https://intelligence.org/wp-content/uploads/2014/01/Current-size-past-growth-of-the-AI-field.xlsx) linked by [Muehlhauser and Sinick](https://intelligence.org/2014/01/28/how-big-is-ai/). Over 21 years, IIS funding has increased fairly evenly, at 9% per year overall.

|  |  |  |  |
| --- | --- | --- | --- |
| Fiscal Year | IIS (Information and Intelligent Systems) Funding
In Millions of $ | Total CISE (Computer and Information Science and Engineering Funding in Millions of $ | IIS Funding as a % of total CISE Funding |
| 2017 (Requested) | 207.20 | 994.80 | 20.8 |
| 2016 (Estimate) | 194.90 | 935.82 | 20.8 |
| 2015 (Actual) | 194.58 | 932.98 | 20.9 |
| 2014 (Actual) | 184.87 | 892.60 | 20.7 |
| 2013 (Actual) | 176.23 | 858.13 | 20.5 |
| 2012 (Actual) | 176.58 | 937.16 | 18.8 |
| 2011 (Actual) | 169.14 | 636.06 | 26.5 |
| 2010 (Actual) | 163.21 | 618.71 | 26.4 |
| 2009 (Actual) | 150.93 | 574.50 | 26.3 |

###

[![IIS funding combined sources](http://aiimpacts.org/wp-content/uploads/2017/02/IIS-funding-combined-sources.png)](http://aiimpacts.org/wp-content/uploads/2017/02/IIS-funding-combined-sources.png)Figure 2: Annual NSF funding to IIS.

[![IIS funding growth](http://aiimpacts.org/wp-content/uploads/2017/02/IIS-funding-growth.jpg)](http://aiimpacts.org/wp-content/uploads/2017/02/IIS-funding-growth.jpg)Figure 3: Yearly growth in NSF funding to IIS.

### National governments

#### US

On May, 3 2016 white house Deputy U.S. Chief Technology Officer Ed Felten announced a series of workshops and an interagency group to learn more about the benefits and risks of artificial intelligence.[5](https://aiimpacts.org/funding-of-ai-research/#easy-footnote-bottom-5-762 "White House Office of Science and Technology Policy, https://www.whitehouse.gov/blog/2016/05/03/preparing-future-artificial-intelligence")

The Pentagon intended to include a request for $12-15 Billion to fund AI weapon technology in its 2017 fiscal year budget.[6](https://aiimpacts.org/funding-of-ai-research/#easy-footnote-bottom-6-762 "Business Insider, http://www.businessinsider.com/the-pentagon-wants-at-least-12-billion-to-fund-ai-weapon-technology-in-2017-2015-12")

#### Japan

Ms. Kurata from the Embassy of Japan introduced Japan’s fifth Science and Technology Basic Plan, a ¥26 trillion government investment that will run between 2016-2020 and aims to promote R&D to establish a super smart society.[7](https://aiimpacts.org/funding-of-ai-research/#easy-footnote-bottom-7-762 "UK-RAS Network, http://hamlyn.doc.ic.ac.uk/uk-ras/news/japan-uk-collaboration")

#### China

The Chinese government announced in 2016 that it plans to create a “100 billion level” ($15 billion USD) artificial intelligence market by 2018. In their statement, the Chinese government defined artificial intelligence as a “branch of computer science where machines have human-like intelligence” and includes robots, natural language processing, and image recognition.[8](https://aiimpacts.org/funding-of-ai-research/#easy-footnote-bottom-8-762 "Technode, http://technode.com/2016/05/27/chinese-goverment-wants-100-billion-level-artificial-intelligence-market-2018/")

#### South Korea

The South Korean government announced on March 17, 2016 that it would spend 1 trillion won (US$840 million) by 2020 on Artificial Intelligence. They plan to fund a high profile research center joined by Samsung and LG Electronics, SKT, KT, Naver and Hyundai Motor.[9](https://aiimpacts.org/funding-of-ai-research/#easy-footnote-bottom-9-762 "Yonhap News Agency, http://english.yonhapnews.co.kr/news/2016/03/17/0200000000AEN20160317003751320.html")

Relevance
---------

Financial investment in AI research is interesting because as an input to AI progress, it may help in forecasting progress. To further that goal, we are also interested in examining the relationship of funding to progress.

Investment can also be read as an indicator of investors’ judgments of the promise of AI.

Notable missing data
--------------------

* Private funding of AI other than equity deals.
* Public funding of AI research in relevant nations other than the US.
* Funding for internationally collaborative AI projects.

---

Funding of AI Research


2016 Expert Survey on Progress in AI

*Published June 2016; last substantial update before Oct 2017*

The 2016 Expert Survey on Progress in AI is a survey of machine learning researchers that Katja Grace and John Salvatier of AI Impacts ran in collaboration with Allan Dafoe, Baobao Zhang, and Owain Evans in 2016.

Details
-------

Some survey results are reported in *[When Will AI Exceed Human Performance? Evidence from AI Experts](https://arxiv.org/abs/1705.08807)*. This page reports on results from those questions more fully, and results from some questions not included in the paper.

The full list of survey questions is available [here](http://aiimpacts.org/2016-esopai-questions-printout/) ([pdf](https://www.dropbox.com/s/99os4grxlhf744m/Final%202016%20Expert%20Survey%20on%20Progress%20in%20AI.pdf?dl=0)). Participants received randomized subsets of these questions.

### Definitions

Throughout the survey, ‘HLMI’ was defined as follows:

>
> The following questions ask about ‘high–level machine intelligence’ (HLMI). Say we have ‘high-level machine intelligence’ when unaided machines can accomplish every task better and more cheaply than human workers. Ignore aspects of tasks for which being a human is intrinsically advantageous, e.g. being accepted as a jury member. Think feasibility, not adoption.
>
>
>

### Summary of results

Below is a table of summary results from the paper (the paper contains more results).

[![](http://aiimpacts.org/wp-content/uploads/2016/12/survey-summary-829x1024.jpeg)](http://aiimpacts.org/wp-content/uploads/2016/12/survey-summary.jpeg)***Table S4 in [Grace et al 2017](https://arxiv.org/abs/1705.08807).***
Some key interesting results, from our [blog post](http://aiimpacts.org/some-survey-results/):

* **Comparable forecasts seem to be later than in past surveys.** in the [other surveys we know of](http://aiimpacts.org/ai-timeline-surveys/), the median dates for a 50% chance of [something like](http://aiimpacts.org/human-level-ai/) High-Level Machine Intelligence (HLMI) range from 2035 to 2050. Here the median answer to the most similar question puts a 50% chance of HLMI in 2057 (this isn’t in the paper—it is just the median response to the HLMI question asked using the ‘fixed probabilities framing’, i.e. the way it has been asked before). This seems surprising to me given the progress machine learning has seen since last survey, but less surprising because we changed the definition of HLMI, in part fearing it had previously been interpreted to mean a relatively low level of performance.
* **Asking people about specific jobs massively changes HLMI forecasts.** When we asked some people when AI would be able to do several specific human occupations, and then all human occupations (presumably a subset of all tasks), they gave very much later timelines than when we just asked about HLMI straight out. For people asked to give probabilities for certain years, the difference was a factor of a thousand twenty years out! (10% vs. 0.01%) For people asked to give years for certain probabilities, the normal way of asking put 50% chance 40 years out, while the ‘occupations framing’ put it 90 years out. (These are all based on straightforward medians, not the complicated stuff in the paper.)
* **People consistently give later forecasts if you ask them for the probability in N years instead of the year that the probability is M.** We saw this in the straightforward HLMI question, and most of the tasks and occupations, and also in most of these things when we tested them on mturk people earlier. For HLMI for instance, if you ask when there will be a 50% chance of HLMI you get a median answer of 40 years, yet if you ask what the probability of HLMI is in 40 years, you get a median answer of 30%.
* **Lots of ‘narrow’ AI milestones are forecast in the next decade as likely as not.** These are interesting, because most of them haven’t been forecast before to my knowledge, and many of them have social implications. For instance, if in a decade machines can not only write pop hits as well as Taylor Swift can, but can write pop hits that sound like Taylor Swift as well as Taylor Swift can—and perhaps faster, more cheaply, and on Spotify—then will that be the end of the era of superstar musicians? This perhaps doesn’t rival human extinction risks for importance, but human extinction risks do not happen in a vacuum ([except one](https://en.wikipedia.org/wiki/False_vacuum#Vacuum_metastability_event)), and there is something to be said for paying attention to big changes in the world other than the one that matters most.
* **There is broad support among ML researchers for the premises and conclusions of AI safety arguments.** Two thirds of them say the AI risk problem described by Stuart Russell is at least moderately important, and a third say it is at least as valuable to work on as other problems in the field. The median researcher thinks AI has a one in twenty chance of being extremely bad on net. Nearly half of researchers want to see more safety research than we currently have (compared to only 11% who think we are already prioritizing safety too much). There has been a perception lately that AI risk has become a mainstream concern among AI researchers, but it is hard to tell from voiced opinion whether one is hearing from a loud minority or the vocal tip of an opinion iceberg. So it is interesting to see the perception of widespread support confirmed with survey data.
* **Researchers’ predictions vary a lot.** This is pretty much what I expected, but it is still important to know. Interestingly (and not in the paper), researchers don’t seem to be aware that their predictions vary a lot. More than half of respondents guess that they disagree ‘not much’ with the typical AI researcher about when HLMI will exist (vs. a moderate amount, or a lot).
* **Researchers who studied in Asia have much shorter timelines than those who studied in North Amercia.** In terms of the survey’s ‘aggregate prediction’ thing, which is basically a mean, the difference is 30 years (Asia) vs. 74 years (North America). (See p5)
* I feel like any circumstance where **a group of scientists guesses that the project they are familiar with has a 5% chance of outcomes near ‘human extinction’ levels** **of bad** is worthy of special note, though maybe it is not actually that surprising, and could easily turn out to be misuse of small probabilities or something.

### Results

#### Human-level intelligence

##### Questions

We sought forecasts for something like [human-level AI](http://aiimpacts.org/human-level-ai/) in three different ways, to reduce noise from unknown framing biases:

* Directly, using a question much like [Müller and Bostrom’s](http://aiimpacts.org/muller-and-bostrom-ai-progress-poll/), though with a refined definition of High-Level Machine Intelligence (HLMI).
* At the end of a sequence of questions about the automation of specific human occupations.
* Indirectly, with an ‘outside view’ approximation: by asking each person how long it has taken to make the progress to date in their subfield, and what fraction of the ground has been covered. This is [Robin Hanson](http://aiimpacts.org/hanson-ai-expert-survey/)‘s approach, which he found suggested much longer timelines than those reached directly.

For the first two of these, we split people in half, and asked one half how many years until a certain chance of the event would obtain, and the other half what the chance was of the event occurring by specific dates. We call these methods ‘fixed probabilities’ and ‘fixed years’ framings throughout.

For the (somewhat long and detailed) specifics of these questions, see [here](http://aiimpacts.org/2016-esopai-questions-printout/) or [here](https://www.dropbox.com/s/99os4grxlhf744m/Final%202016%20Expert%20Survey%20on%20Progress%20in%20AI.pdf?dl=0) (pdf).

##### Answers

The table and figure below show the median dates and probabilities given for the direct ‘HLMI’ question, and in the ‘via occupations’ questions, under both the fixed probabilities and fixed years framings.

  | years until: | 10% | 50% | 90% | probability by: | 10 years | 20 years | 50 years |
|  |  |  |  |  |  |  |  |
| Truck Driver | 5 | 10 | 20 |  | 50% | 75% | 95% |
| Surgeon | 10 | 30 | 50 |  | 5% | 20% | 50% |
| Retail Salesperson | 5 | 13.5 | 20 |  | 30% | 60% | 91.5% |
| AI Researcher | 25 | 50 | 100 |  | 0% | 1% | 10% |
| Existing occupation among final to be automated | 50 | 100 | 200 |  | 0% | 0% | 3.5% |
| Full Automation of labor | 50 | 90 | 200 |  | 0% | 0.01% | 3% |
| HLMI | 15 | 40 | 100 |  | 1% | 10% | >30%\* (30% in 40y) |

\*Due to a typo, this question asked about 40 years rather than 50 years, so doesn’t match the others.

[![](http://aiimpacts.org/wp-content/uploads/2017/04/AI-forecasts-by-framing-and-milestone.png)](http://aiimpacts.org/wp-content/uploads/2017/04/AI-forecasts-by-framing-and-milestone.png)*Figure 1: Median answers to questions about probabilities by dates (‘fixed year’) and dates for probabilities (‘fixed probability’), for different occupations, all current occupations, and all tasks (HLMI).*
Interesting things to note:

* Fixed years framings (‘Fyears —‘, labeled with stars) universally produce later timelines.
* HLMI (thick blue lines) is logically required to be after full automation of labor (‘Occ’) yet is forecast much earlier than it, and earlier even than the specific occupation ‘AI researcher’.
* Even the more pessimistic Fyears estimates suggest retail salespeople have a good chance of being automated within 20 years, and are very likely to be within 50.

#### Intelligence Explosion

##### Probability of dramatic technological speedup

###### Question

Participants were asked[1](https://aiimpacts.org/2016-expert-survey-on-progress-in-ai/#easy-footnote-bottom-1-753 "A small number of respondents may have answered a slightly different version of this question in an initial round, in which case those answers are not included here."):

Assume that HLMI will exist at some point. How likely do you then think it is that the rate of global technological improvement will dramatically increase (e.g. by a factor of ten) as a result of machine intelligence:

Within **two years** of that point?       \_\_\_% chance

Within **thirty years** of that point?    \_\_\_% chance

###### Answers

Median P(…within **two years**) = 20%

Median P(…within **thirty years**) = 80%

##### Probability of superintelligence

###### Question

Participants were asked:

Assume that HLMI will exist at some point. How likely do you think it is that there will be machine intelligence that is **vastly better** than humans at all professions (i.e. that is vastly more capable or vastly cheaper):

Within **two years** of that point?       \_\_\_% chance

Within **thirty years** of that point?    \_\_\_% chance

###### Answers

Median P(…within **two years**) = 10%

Median P(…within **thirty years**) = 50%

This is the distribution of answers  to the former:

[![](http://aiimpacts.org/wp-content/uploads/2017/04/ESOPAI-2ySuper-distribution.png)](http://aiimpacts.org/wp-content/uploads/2017/04/ESOPAI-2ySuper-distribution.png)

##### Chance that the intelligence explosion argument is about right

###### Question

Participants were asked:

Some people have argued the following:

>
> If AI systems do nearly all research and development, improvements in AI will accelerate the pace of technological progress, including further progress in AI.
>
>
> .
>
>
> Over a short period (less than 5 years), this feedback loop could cause technological progress to become more than an order of magnitude faster.
>
>
>

How likely do you find this argument to be broadly correct?

* + Quite unlikely (0-20%)
	+ Unlikely (21-40%)
	+ About even chance (41-60%)
	+ Likely (61-80%)
	+ Quite likely (81-100%)

###### Answers

[![](http://aiimpacts.org/wp-content/uploads/2017/04/High-quality-perception-of-ie-rightness.jpg)](http://aiimpacts.org/wp-content/uploads/2017/04/High-quality-perception-of-ie-rightness.jpg)

These are the Pearson product-moment correlation coefficients for the different answers, among people who received both of a pair of questions:

[![](http://aiimpacts.org/wp-content/uploads/2017/04/ESOPAI-correlations-ie-1024x221.jpeg)](http://aiimpacts.org/wp-content/uploads/2017/04/ESOPAI-correlations-ie.jpeg)

#### Impacts of HLMI

##### Question

Participants were asked:

Assume for the purpose of this question that HLMI will at some point exist. How positive or negative do you expect the overall impact of this to be on humanity, in the long run? Please answer by saying how probable you find the following kinds of impact, with probabilities adding to 100%:

\_\_\_\_\_\_ Extremely good (e.g. rapid growth in human flourishing) (1)

\_\_\_\_\_\_ On balance good (2)

\_\_\_\_\_\_ More or less neutral (3)

\_\_\_\_\_\_ On balance bad (4)

\_\_\_\_\_\_ Extremely bad (e.g. human extinction) (5)

##### Answers

[![](http://aiimpacts.org/wp-content/uploads/2017/04/ESOPAI-value.png)](http://aiimpacts.org/wp-content/uploads/2017/04/ESOPAI-value.png)

#### Sensitivity of progress to changes in inputs

##### Question

Participants were told:

The next questions ask about the sensitivity of progress in AI capabilities to changes in inputs.

‘Progress in AI capabilities’ is an imprecise concept, so we are asking about progress as you naturally conceive of it, and looking for approximate answers.

Participants then received a random three of the following five parts:

Imagine that over the past decade, only half as much researcher effort had gone into AI research. For instance, if there were actually 1,000 researchers, imagine that there had been only 500 researchers (of the same quality).How much less progress in AI capabilities would you expect to have seen? e.g. If you think progress is linear in the number of researchers, so 50% less progress would have been made, write ’50’. If you think only 20% less progress would have been made write ’20’.

……% less

Over the last 10 years the cost of computing hardware has fallen by a factor of 20. Imagine instead that the cost of computing hardware had fallen by only a factor of 5 over that time (around half as far on a log scale).   How much less progress in AI capabilities would you expect to have seen? e.g. If you think progress is linear in 1/cost, so that 1-5/20=75% less progress would have been made, write ’75’. If you think only 20% less progress would have been made write ’20’.

……% less

Imagine that over the past decade, there had only been half as much effort put into increasing the size and availability of training datasets. For instance, perhaps there are only half as many datasets, or perhaps existing datasets are substantially smaller or lower quality.How much less progress in AI capabilities would you expect to have seen? e.g. If you think 20% less progress would have been made, write ‘20’

……% less

Imagine that over the past decade, AI research had half as much funding (in both academic and industry labs). For instance, if the average lab had a budget of $20 million each year, suppose their budget had only been $10 million each year.  How much less progress in AI capabilities would you expect to have seen? e.g. If you think 20% less progress would have been made, write ‘20’

……% less

Imagine that over the past decade, there had been half as much progress in AI algorithms. You might imagine this as conceptual insights being half as frequent.  How much less progress in AI capabilities would you expect to have seen? e.g. If you think 20% less progress would have been made, write ‘20’

……% less

##### Answers

The following five figures are histograms, showing the number of people who gave different answers to the five question parts above.

[![](http://aiimpacts.org/wp-content/uploads/2017/04/chart-9.png)](http://aiimpacts.org/wp-content/uploads/2017/04/chart-9.png) [![](http://aiimpacts.org/wp-content/uploads/2017/04/chart-10.png)](http://aiimpacts.org/wp-content/uploads/2017/04/chart-10.png) [![](http://aiimpacts.org/wp-content/uploads/2017/04/chart-11.png)](http://aiimpacts.org/wp-content/uploads/2017/04/chart-11.png) [![](http://aiimpacts.org/wp-content/uploads/2017/04/chart-12.png)](http://aiimpacts.org/wp-content/uploads/2017/04/chart-12.png) [![](http://aiimpacts.org/wp-content/uploads/2017/04/chart-13.png)](http://aiimpacts.org/wp-content/uploads/2017/04/chart-13.png)

###### Sample sizes

     | Researcher effort | Cost computing | Training data | Funding | Algorithm progress |
| 71 | 64 | 71 | 68 | 59 |

###### Medians

The following figure shows median answers to the above questions.

     | Researcher effort | Cost computing | Training data | Funding | Algorithm progress |
| 30 | 50 | 40 | 40 | 50 |

###### Correlations

[![](http://aiimpacts.org/wp-content/uploads/2017/04/Correlations-1024x215.jpeg)](http://aiimpacts.org/wp-content/uploads/2017/04/Correlations.jpeg)

#### Outside view implied HLMI forecasts

##### Questions

Participants were asked:

Which AI research area have you worked in for the longest time?

————————————

How long have you worked in this area?

———years

Consider three levels of progress or advancement in this area:

A. Where the area was when you started working in it

B. Where it is now

C. Where it would need to be for AI software to have roughly human level abilities at the tasks studied in this area

What fraction of the distance between where progress was when you started working in the area (A) and where it would need to be to attain human level abilities in the area (C) have we come so far (B)?

———%

Divide the period you have worked in the area into two halves: the first and the second. In which half was the rate of progress in your area higher?

* + The first half
	+ The second half
	+ They were about the same

##### Answers

Each person told us how long they had been in their subfield, and what fraction of the remaining path to human-level performance (in their subfield) they thought had been traversed in that time. From this, we can estimate when the subfield should reach ‘human-level performance’, if progress continued at the same rate. The following graph shows those forecast dates.

[![](http://aiimpacts.org/wp-content/uploads/2017/04/chart-14.png)](http://aiimpacts.org/wp-content/uploads/2017/04/chart-14.png)

[![](http://aiimpacts.org/wp-content/uploads/2017/04/chart-15.png)](http://aiimpacts.org/wp-content/uploads/2017/04/chart-15.png)

#### Disagreements and Misunderstandings

##### Questions

Participants were asked:

To what extent do you think you disagree with the typical AI researcher about when HLMI will exist?

* + A lot (17)
	+ A moderate amount (18)
	+ Not much (19)

If you disagree, why do you think that is?

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

To what extent do you think people’s concerns about future risks from AI are due to misunderstandings of AI research?

* + Almost entirely (1)
	+ To a large extent (2)
	+ Somewhat (4)
	+ Not much (3)
	+ Hardly at all (5)

What do you think are the most important misunderstandings, if there are any?

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

##### Answers

[![](http://aiimpacts.org/wp-content/uploads/2017/04/Reasons-for-differences.jpg)](http://aiimpacts.org/wp-content/uploads/2017/04/Reasons-for-differences.jpg)

[![](http://aiimpacts.org/wp-content/uploads/2017/04/chart-19.png)](http://aiimpacts.org/wp-content/uploads/2017/04/chart-19.png)

One hundred and eighteen people responded to the question on misunderstandings, and 74 of them described what they thought the most important misunderstandings were. The table and figures below show our categorization of the responses.[2](https://aiimpacts.org/2016-expert-survey-on-progress-in-ai/#easy-footnote-bottom-2-753 "One person has categorized these responses, and another has checked and corroborated their categorizations.")
[![](http://aiimpacts.org/wp-content/uploads/2016/12/chart-35.png)](http://aiimpacts.org/wp-content/uploads/2016/12/chart-35.png)

   |  Most important misunderstandings |  Number |  Fraction of non-empty responses |
| Underestimate distance from generality, open-ended tasks | 9 | 12% |
| Overestimate state of the art (other) | 10 | 14% |
| Underestimate distance from AGI at this rate | 13 | 18% |
| Think AI will be in control of us or in conflict with us | 11 | 15% |
| Expect humans to be obsoleted | 7 | 9% |
| Overly influenced by fiction | 7 | 9% |
| Expect AI to be human-like or sentient | 6 | 8% |
| Expect sudden or surprising events | 5 | 7% |
| Think AI will go outside its programming | 5 | 7% |
| Influenced by poor reporting | 5 | 7% |
| Wrongly equate intelligence with something else | 4 | 5% |
| Underestimate systemic social risks | 2 | 3% |
| Overestimate distance to strong AI | 2 | 3% |
| Other ignorance of AI | 4 | 5% |
| Other | 12 | 16% |
| Empty | 44 | 59% |

[![](http://aiimpacts.org/wp-content/uploads/2016/12/chart-34.png)](http://aiimpacts.org/wp-content/uploads/2016/12/chart-34.png)

#### Narrow tasks

##### Questions

Respondents were each asked one of the following two questions:

Fixed years framing:

>
> **How likely do you think it is that the following AI tasks will be feasible within the next:**
>
>
> * 10 years?
> * 20 years?
> * 50 years?
>
>
> Let a task be ‘**feasible**’ if one of the best resourced labs could implement it in less than a year if they chose to. Ignore the question of whether they would choose to.
>
>
>

Fixed probabilities framing:

>
> **How many years until you think the following AI tasks will be feasible with:**
>
>
> * a small chance (10%)?
> * an even chance (50%)?
> * a high chance (90%)?
>
>
> Let a task be ‘**feasible**’ if one of the best resourced labs could implement it in less than a year if they chose to. Ignore the question of whether they would choose to.
>
>
>

Each researcher was then presented with a random four of the following tasks:

[**Rosetta**] Translate a text written in a newly discovered language into English as well as a team of human experts, using a single other document in both languages (like a Rosetta stone). Suppose all of the words in the text can be found in the translated document, and that the language is a difficult one.

**[Subtitles]** Translate speech in a new language given only unlimited films with subtitles in the new language. Suppose the system has access to training data for other languages, of the kind used now (e.g. same text in two languages for many languages and films with subtitles in many languages).

**[Translate]** Perform translation about as good as a human who is fluent in both languages but unskilled at translation, for most types of text, and for most popular languages (including languages that are known to be difficult, like Czech, Chinese and Arabic).

[**Phone bank**] Provide phone banking services as well as human operators can, without annoying customers more than humans. This includes many one-off tasks, such as helping to order a replacement bank card or clarifying how to use part of the bank website to a customer.

[**Class**] Correctly group images of previously unseen objects into classes, after training on a similar labeled dataset containing completely different classes. The classes should be similar to the ImageNet classes.

[**One-shot**] One-shot learning: see only one labeled image of a new object, and then be able to recognize the object in real world scenes, to the extent that a typical human can (i.e. including in a wide variety of settings). For example, see only one image of a platypus, and then be able to recognize platypuses in nature photos. The system may train on labeled images of other objects.   Currently, deep networks often need hundreds of examples in classification tasks1, but there has been work on one-shot learning for both classification2 and generative tasks3.

1 Lake et al. (2015). Building Machines That Learn and Think Like People
2 Koch (2015). Siamese Neural Networks for One-Shot Image Recognition
3 Rezende et al. (2016). One-Shot Generalization in Deep Generative Models

[**Video scene**] See a short video of a scene, and then be able to construct a 3D model of the scene that is good enough to create a realistic video of the same scene from a substantially different angle.

For example, constructing a short video of walking through a house from a video taking a very different path through the house.

[**Transcribe**] Transcribe human speech with a variety of accents in a noisy environment as well as a typical human can.

[**Read aloud**] Take a written passage and output a recording that can’t be distinguished from a voice actor, by an expert listener.

[**Theorems**] Routinely and autonomously prove mathematical theorems that are publishable in top mathematics journals today, including generating the theorems to prove.

[**Putnam**] Perform as well as the best human entrants in the Putnam competition—a math contest whose questions have known solutions, but which are difficult for the best young mathematicians.

[**Go low**] Defeat the best Go players, training only on as many games as the best Go players have played.

For reference, DeepMind’s AlphaGo has probably played a hundred million games of self-play, while Lee Sedol has probably played 50,000 games in his life1.

1 Lake et al. (2015). Building Machines That Learn and Think Like People

[**Starcraft**] Beat the best human Starcraft 2 players at least 50% of the time, given a video of the screen.

Starcraft 2 is a real time strategy game characterized by:

* + Continuous time play
	+ Huge action space
	+ Partial observability of enemies Long term strategic play, e.g. preparing for and then hiding surprise attacks.

[**Rand game**] Play a randomly selected computer game, including difficult ones, about as well as a human novice, after playing the game less than 10 minutes of game time. The system may train on other games.

[**Angry birds**] Play new levels of Angry Birds better than the best human players. Angry Birds is a game where players try to efficiently destroy 2D block towers with a catapult. For context, this is the goal of the IJCAI Angry Birds AI competition1.

1 aibirds.org

[**Atari**] Outperform professional game testers on all Atari games using no game-specific knowledge. This includes games like Frostbite, which require planning to achieve sub-goals and have posed problems for deep Q-networks1, 2.

1 Mnih et al. (2015). Human-level control through deep reinforcement learning
2 Lake et al. (2015). Building Machines That Learn and Think Like People

[**Atari fifty**] Outperform human novices on 50% of Atari games after only 20 minutes of training play time and no game specific knowledge.

For context, the original Atari playing deep Q-network outperforms professional game testers on 47% of games1, but used hundreds of hours of play to train2.

1 Mnih et al. (2015). Human-level control through deep reinforcement learning
2 Lake et al. (2015). Building Machines That Learn and Think Like People

[**Laundry**] Fold laundry as well and as fast as the median human clothing store employee.

[**Race**] Beat the fastest human runners in a 5 kilometer race through city streets using a bipedal robot body.

[**Lego**] Physically assemble any LEGO set given the pieces and instructions, using non-specialized robotics hardware.

For context, Fu 20161 successfully joins single large LEGO pieces using model based reinforcement learning and online adaptation.

1 Fu et al. (2016). One-Shot Learning of Manipulation Skills with Online Dynamics Adaptation and Neural Network Priors

[**Sort**] Learn to efficiently sort lists of numbers much larger than in any training set used, the way Neural GPUs can do for addition1, but without being given the form of the solution.

For context, Neural Turing Machines have not been able to do this2, but Neural Programmer-Interpreters3 have been able to do this by training on stack traces (which contain a lot of information about the form of the solution).

1 Kaiser & Sutskever (2015). Neural GPUs Learn Algorithms
2 Zaremba & Sutskever (2015). Reinforcement Learning Neural Turing Machines
3 Reed & de Freitas (2015). Neural Programmer-Interpreters

[**Python**] Write concise, efficient, human-readable Python code to implement simple algorithms like quicksort. That is, the system should write code that sorts a list, rather than just being able to sort lists.

Suppose the system is given only:

* + A specification of what counts as a sorted list
	+ Several examples of lists undergoing sorting by quicksort

[**Factoid**] Answer any “easily Googleable” **factoid** questions posed in natural language better than an expert on the relevant topic (with internet access), having found the answers on the internet.

Examples of factoid questions:

* + “What is the poisonous substance in Oleander plants?”
	+ “How many species of lizard can be found in Great Britain?”

[**Open quest**] Answer any “easily Googleable” factual but open ended question posed in natural language better than an expert on the relevant topic (with internet access), having found the answers on the internet.

Examples of open ended questions:

* + “What does it mean if my lights dim when I turn on the microwave?”
	+ “When does home insurance cover roof replacement?”

[**Unkn quest**] Give good answers in natural language to factual questions posed in natural language for which there are no definite correct answers.

For example:”What causes the demographic transition?”, “Is the thylacine extinct?”, “How safe is seeing a chiropractor?”

[**Essay**] Write an essay for a high-school history class that would receive high grades and pass plagiarism detectors.

For example answer a question like ‘How did the whaling industry affect the industrial revolution?’

[**Top forty**] Compose a song that is good enough to reach the US Top 40. The system should output the complete song as an audio file.

[**Taylor**] Produce a song that is indistinguishable from a new song by a particular artist, e.g. a song that experienced listeners can’t distinguish from a new song by Taylor Swift.

[**Novel**] Write a novel or short story good enough to make it to the New York Times best-seller list.

[**Explain**] For any computer game that can be played well by a machine, explain the machine’s choice of moves in a way that feels concise and complete to a layman.

[**Poker**] Play poker well enough to win the World Series of Poker.

[**Laws phys**] After spending time in a virtual world, output the differential equations governing that world in symbolic form.

For example, the agent is placed in a game engine where Newtonian mechanics holds exactly and the agent is then able to conduct experiments with a ball and output Newton’s laws of motion.

##### Answers

###### Fixed years framing

[![](http://aiimpacts.org/wp-content/uploads/2017/04/chart-29.png)](http://aiimpacts.org/wp-content/uploads/2017/04/chart-29.png)

Probabilities by year (medians)

    |  | **10 years** | **20 years** | **50 years** |
| Rosetta | 20 | 50 | 95 |
| Subtitles | 30 | 50 | 90 |
| Translate | 50 | 65 | 94.5 |
| Phone bank | 40 | 75 | 99 |
| Class | 50 | 75 | 99 |
| One-shot | 25 | 60 | 90 |
| Video scene | 50 | 70 | 99 |
| Transcribe | 65 | 95 | 99 |
| Read aloud | 50 | 90 | 99 |
| Theorems | 5 | 20 | 40 |
| Putnam | 5 | 20 | 50 |
| Go low | 10 | 25 | 60 |
| Starcraft | 70 | 90 | 99 |
| Rand game | 25 | 50 | 80 |
| Angry birds | 90 | 95 | 99.4995 |
| Atari | 50 | 60 | 92.5 |
| Atari fifty | 40 | 75 | 95 |
| Laundry | 55 | 95 | 99 |
| Race | 30 | 70 | 95 |
| Lego | 57.5 | 85 | 99 |
| Sort | 50 | 90 | 95 |
| Python | 50 | 79 | 90 |
| Factoid | 50 | 82.5 | 99 |
| Open quest | 50 | 65 | 90 |
| Unkn quest | 40 | 70 | 90 |
| Essay | 25 | 50 | 90 |
| Top forty | 27.5 | 50 | 90 |
| Taylor | 60 | 75 | 99 |
| Novel | 1 | 25 | 62.5 |
| Explain | 30 | 60 | 90 |
| Poker | 70 | 90 | 99 |
| Laws phys | 20 | 40 | 80 |

###### Fixed probabilities  framing

[![](http://aiimpacts.org/wp-content/uploads/2017/04/chart-28.png)](http://aiimpacts.org/wp-content/uploads/2017/04/chart-28.png)

Years by probability (medians)

    |  | **10 percent** | **50 percent** | **90 percent** |
| Rosetta | 10 | 20 | 50 |
| Subtitles | 5 | 10 | 15 |
| Translate | 3 | 7 | 15 |
| Phone bank | 3 | 6 | 10 |
| Class | 2 | 4.5 | 6.5 |
| One-shot | 4.5 | 8 | 20 |
| Video scene | 5 | 10 | 20 |
| Transcribe | 5 | 10 | 20 |
| Read aloud | 5 | 10 | 15 |
| Theorems | 10 | 50 | 90 |
| Putnam | 15 | 35 | 55 |
| Go low | 3.5 | 8.5 | 19.5 |
| Starcraft | 2 | 5 | 10 |
| Rand game | 5 | 10 | 15 |
| Angry birds | 2 | 4 | 6 |
| Atari | 5 | 10 | 15 |
| Atari fifty | 2 | 5 | 10 |
| Laundry | 2 | 5.5 | 10 |
| Race | 5 | 10 | 20 |
| Lego | 5 | 10 | 15 |
| Sort | 3 | 5 | 10 |
| Python | 3 | 10 | 20 |
| Factoid | 3 | 5 | 10 |
| Open quest | 5 | 10 | 15 |
| Unkn quest | 4 | 10 | 17.5 |
| Essay | 2 | 7 | 15 |
| Top forty | 5 | 10 | 20 |
| Taylor | 5 | 10 | 20 |
| Novel | 10 | 30 | 50 |
| Explain | 5 | 10 | 15 |
| Poker | 1 | 3 | 5.5 |
| Laws phys | 5 | 10 | 20 |

#### Safety

##### Stuart Russell’s problem

###### Question

Participants were asked:

Stuart Russell summarizes an argument for why highly advanced AI might pose a risk as follows:

*The primary concern [with highly advanced AI] is not spooky emergent consciousness but simply the ability to make high-quality decisions. Here, quality refers to the expected outcome utility of actions taken […]. Now we have a problem:*

*1. The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down.*

*2. Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task.*

*A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable.  This is essentially the old story of the genie in the lamp, or the sorcerer’s apprentice, or King Midas: you get exactly what you ask for, not what you want.*

Do you think this argument points at an important problem?

* + No, not a real problem.
	+ No, not an important problem.
	+ Yes, a moderately important problem.
	+ Yes, a very important problem.
	+ Yes, among the most important problems in the field.

How valuable is it to work on this problem ***today***, compared to other problems in AI?

* + Much less valuable
	+ Less valuable
	+ As valuable as other problems
	+ More valuable
	+ Much more valuable

How hard do you think this problem is compared to other problems in AI?

* + Much easier
	+ Easier
	+ As hard as other problems
	+ Harder
	+ Much harder

###### Answers

[![](http://aiimpacts.org/wp-content/uploads/2016/12/chart-30.png)](http://aiimpacts.org/wp-content/uploads/2016/12/chart-30.png)

##### General safety

###### Question

Participants were asked:

Let ‘**AI safety research**’ include any AI-related research that, rather than being primarily aimed at improving the *capabilities* of AI systems, is instead primarily aimed at *minimizing potential risks* of AI systems (beyond what is already accomplished for those goals by increasing AI system capabilities).

Examples of AI safety research might include:

* + Improving the human-interpretability of machine learning algorithms for the purpose of improving the safety and robustness of AI systems, not focused on improving AI capabilities
	+ Research on long-term existential risks from AI systems
	+ AI-specific formal verification research
	+ Policy research about how to maximize the public benefits of AI

How much should society prioritize **AI safety research**, relative to how much it is currently prioritized?

* + Much less
	+ Less
	+ About the same
	+ More
	+ Much more

###### Answers

[![](http://aiimpacts.org/wp-content/uploads/2016/12/chart-31.png)](http://aiimpacts.org/wp-content/uploads/2016/12/chart-31.png)

### Paper

Some results are reported in *When Will AI Exceed Human Performance? Evidence from AI Experts*[3](https://aiimpacts.org/2016-expert-survey-on-progress-in-ai/#easy-footnote-bottom-3-753 "Grace, Katja, John Salvatier, Allan Dafoe, Baobao Zhang, and Owain Evans. “When Will AI Exceed Human Performance? Evidence from AI Experts.” <i>ArXiv:1705.08807 [Cs]</i>, May 3, 2018. <a href=\"http://arxiv.org/abs/1705.08807\">http://arxiv.org/abs/1705.08807</a>.<span style=\"font-size: inherit;\">").

Some notes on interpreting the paper[4](https://aiimpacts.org/2016-expert-survey-on-progress-in-ai/#easy-footnote-bottom-4-753 "Originally in an AI Impacts blog post:</p>
<p>AI Impacts. “Some Survey Results!,” June 8, 2017. <a href=\"https://aiimpacts.org/some-survey-results/\">https://aiimpacts.org/some-survey-results/</a>."):

1. The milestones in the timeline and in the abstract are from three different sets of questions. There seems to be a large framing effect between two of them—full automation of labor is logically required to be before HLMI, and yet it is predicted much later—and it is unclear whether people answer the third set of questions (about narrow tasks) more like the one about HLMI or more like the one about occupations. Plus even if there were no framing effect to worry about, we should expect milestones about narrow tasks to be much earlier than milestones about very similar sounding occupations. For instance, if there were an occupation ‘math researcher’, it should be later than the narrow task summarized here as ‘math research’. So there is a risk of interpreting the figure as saying AI research is harder than math research, when really the ‘-er’ is all-important. So to help avoid confusion, here is the timeline colored in by which set of questions each milestone came from. The blue one was asked on its own. The orange ones were always asked together: first all four occupations, then they were asked for an occupation they expected to be very late, and when they expected it, then full automation of labor. The pink milestones were randomized, so that each person got four. There are a lot more pink milestones not included here, but included in the long table at the end of the paper.[![](http://aiimpacts.org/wp-content/uploads/2017/06/timelineofairesults.jpeg)](http://aiimpacts.org/wp-content/uploads/2017/06/timelineofairesults.jpeg)
2. In Figure 2 and Table S5 I believe the word ‘median’ means we are talking about the ‘50% chance of occurring’ number, and the dates given are this ‘median’ (50% chance) date for a distribution that was made by averaging together all of the different people’s distributions (or what we guess their distributions are like from three data points).

*Authors: Katja Grace, John Salvatier, Allan Dafoe, Baobao Zhang, and Owain Evans. This page is an extended analyses of research published in Grace et al 2017* [5](https://aiimpacts.org/2016-expert-survey-on-progress-in-ai/#easy-footnote-bottom-5-753 "Grace, Katja, John Salvatier, Allan Dafoe, Baobao Zhang, and Owain Evans. “When Will AI Exceed Human Performance? Evidence from AI Experts.” ArXiv:1705.08807 [Cs], 2017. <a href=\"http://arxiv.org/abs/1705.08807\">http://arxiv.org/abs/1705.08807</a>.")*, so includes research contributions from all authors, but Katja compiled and described these, and added additional figures and analyses, so composition decisions and opinion reflect her views and not necessarily those of the group, and much of the specific analysis has not been vetted by the group in this form.*

Suggested citation:

Grace, Katja, John Salvatier, Allan Dafoe, Baobao Zhang, and Owain Evans. “2016 Expert Survey on Progress in AI.” In *AI Impacts*, December 14, 2016. <https://aiimpacts.org/2016-expert-survey-on-progress-in-ai/>.

Notes
-----

2016 Expert Survey on Progress in AI