Funding of AI Research

Provisional data suggests:

Equity deals made with startups in AI were worth about $5bn in 2016, and this value has been growing by around 50% per year in recent years.
The number of equity deals in AI startups globally is growing at around 30% per year, and was estimated at 658 in 2016.
NSF funding of IIS, a section of computer science that appears to include AI and two other areas, has increased at around 9% per year over the past two decades.

(Updated February 2017)
Background
Artificial Intelligence research is funded both publicly and privately. This page currently contains some data on private funding globally, public funding in the US,  and national government announcements of plans relating to AI funding. This page should not currently be regarded as an exhaustive summary of data available on these topics or on AI funding broadly.
Details
AI startups
According to CB Insights, between the start of 2012 and the end of 2016, the number of equity deals being made with startups in artificial intelligence globally grew by a factor of four to 658 (around 30% per year), and the value of funding grew by a factor of over eight to $5 billion (around 50% per year).1 Their measure includes both startups developing AI techniques, and those applying existing AI techniques to problems in areas such as healthcare or advertising. They provide Figure 1 below, with further details of the intervening years. We have not checked the trustworthiness or completeness of CB Insights’ data.
Figure 1: Number of new equity deals supporting AI-related startups, and dollar values of disclosed investments over 2012-2015, according to CB Insights.
US National Science Foundation
In 2014, Muehlhauser and Sinick wrote:
In 2011, the National Science Foundation (NSF) received $636 million for funding CS research (through CISE). Of this, $169 million went to Information and Intelligent Systems (IIS). IIS has three programs: Cyber-Human Systems (CHS), Information Integration and Informatics (III) and Robust Intelligence (RI). If roughly 1/3 of the funding went to each of these, then $56 million went to Robust Intelligence, so 9% of the total CS funding. (Some CISE funding may have gone to AI work outside of IIS — that is, via ACI, CCF, or CNS — but at a glance, non-IIS AI funding through CISE looks negligible.)
…
The NSF Budget for Information and Intelligent Systems (IIS) has generally increased between 4% and 20% per year since 1996, with a one-time percentage boost of 60% in 2003, for a total increase of 530% over the 15 year period between 1996 and 2011.[14 {See table with upper left-hand corner A367 in the spreadsheet.}] “Robust Intelligence” is one of three program areas covered by this budget.
As of February 2017, CISE (Computer and Information Science and Engineering) covers five categories, and IIS appears to be the most relevant one.2 IIS still has three programs, of which Robust Intelligence is one.3
NSF funding into both CISE and IIS (the relevant subcategory) from 2009 to 2017 shows a steady rise.4 IIS funding as a percentage of CISE funding fluctuates, and has gone down in this time period. The following table summarizes data from NSF, collected by Finan Adamson in 2016. The figures below it (2 and 3) combine this data with some collected previously in this spreadsheet linked by Muehlhauser and Sinick. Over 21 years, IIS funding has increased fairly evenly, at 9% per year overall.

Fiscal Year
IIS (Information and Intelligent Systems) Funding
In Millions of $
Total CISE (Computer and Information Science and Engineering Funding in Millions of $
IIS Funding as a % of total CISE Funding

2017 (Requested)
207.20
994.80
20.8

2016 (Estimate)
194.90
935.82
20.8

2015 (Actual)
194.58
932.98
20.9

2014 (Actual)
184.87
892.60
20.7

2013 (Actual)
176.23
858.13
20.5

2012 (Actual)
176.58
937.16
18.8

2011 (Actual)
169.14
636.06
26.5

2010 (Actual)
163.21
618.71
26.4

2009 (Actual)
150.93
574.50
26.3

Figure 2: Annual NSF funding to IIS.
Figure 3: Yearly growth in NSF funding to IIS.
National governments
US
On May, 3 2016 white house Deputy U.S. Chief Technology Officer Ed Felten announced a series of workshops and an interagency group to learn more about the benefits and risks of artificial intelligence.5
The Pentagon intended to include a request for $12-15 Billion to fund AI weapon technology in its 2017 fiscal year budget.6
Japan
Ms. Kurata from the Embassy of Japan introduced Japan’s fifth Science and Technology Basic Plan, a ¥26 trillion government investment that will run between 2016-2020 and aims to promote R&D to establish a super smart society.7
China
The Chinese government announced in 2016 that it plans to create a “100 billion level” ($15 billion USD) artificial intelligence market by 2018. In their statement, the Chinese government defined artificial intelligence as a “branch of computer science where machines have human-like intelligence” and includes robots, natural language processing, and image recognition.8
South Korea
The South Korean government announced on March 17, 2016 that it would spend 1 trillion won (US$840 million) by 2020 on Artificial Intelligence. They plan to fund a high profile research center joined by Samsung and LG Electronics, SKT, KT, Naver and Hyundai Motor.9
Relevance
Financial investment in AI research is interesting because as an input to AI progress, it may help in forecasting progress. To further that goal, we are also interested in examining the relationship of funding to progress.
Investment can also be read as an indicator of investors’ judgments of the promise of AI.
Notable missing data

Private funding of AI other than equity deals.
Public funding of AI research in relevant nations other than the US.
Funding for internationally collaborative AI projects.

Funding of AI Research


2016 Expert Survey on Progress in AI

Published June 2016; last substantial update before Oct 2017
The 2016 Expert Survey on Progress in AI is a survey of machine learning researchers that Katja Grace and John Salvatier of AI Impacts ran in collaboration with Allan Dafoe, Baobao Zhang, and Owain Evans in 2016.
Details
Some survey results are reported in When Will AI Exceed Human Performance? Evidence from AI Experts. This page reports on results from those questions more fully, and results from some questions not included in the paper.
The full list of survey questions is available here (pdf). Participants received randomized subsets of these questions.
Definitions
Throughout the survey, ‘HLMI’ was defined as follows:

The following questions ask about ‘high–level machine intelligence’ (HLMI). Say we have ‘high-level machine intelligence’ when unaided machines can accomplish every task better and more cheaply than human workers. Ignore aspects of tasks for which being a human is intrinsically advantageous, e.g. being accepted as a jury member. Think feasibility, not adoption.

Summary of results
Below is a table of summary results from the paper (the paper contains more results).
Table S4 in Grace et al 2017.
Some key interesting results, from our blog post:

Comparable forecasts seem to be later than in past surveys. in the other surveys we know of, the median dates for a 50% chance of something like High-Level Machine Intelligence (HLMI) range from 2035 to 2050. Here the median answer to the most similar question puts a 50% chance of HLMI in 2057 (this isn’t in the paper—it is just the median response to the HLMI question asked using the ‘fixed probabilities framing’, i.e. the way it has been asked before). This seems surprising to me given the progress machine learning has seen since last survey, but less surprising because we changed the definition of HLMI, in part fearing it had previously been interpreted to mean a relatively low level of performance.
Asking people about specific jobs massively changes HLMI forecasts. When we asked some people when AI would be able to do several specific human occupations, and then all human occupations (presumably a subset of all tasks), they gave very much later timelines than when we just asked about HLMI straight out. For people asked to give probabilities for certain years, the difference was a factor of a thousand twenty years out! (10% vs. 0.01%) For people asked to give years for certain probabilities, the normal way of asking put 50% chance 40 years out, while the ‘occupations framing’ put it 90 years out. (These are all based on straightforward medians, not the complicated stuff in the paper.)
People consistently give later forecasts if you ask them for the probability in N years instead of the year that the probability is M. We saw this in the straightforward HLMI question, and most of the tasks and occupations, and also in most of these things when we tested them on mturk people earlier. For HLMI for instance, if you ask when there will be a 50% chance of HLMI you get a median answer of 40 years, yet if you ask what the probability of HLMI is in 40 years, you get a median answer of 30%.
Lots of ‘narrow’ AI milestones are forecast in the next decade as likely as not. These are interesting, because most of them haven’t been forecast before to my knowledge, and many of them have social implications. For instance, if in a decade machines can not only write pop hits as well as Taylor Swift can, but can write pop hits that sound like Taylor Swift as well as Taylor Swift can—and perhaps faster, more cheaply, and on Spotify—then will that be the end of the era of superstar musicians? This perhaps doesn’t rival human extinction risks for importance, but human extinction risks do not happen in a vacuum (except one), and there is something to be said for paying attention to big changes in the world other than the one that matters most.
There is broad support among ML researchers for the premises and conclusions of AI safety arguments. Two thirds of them say the AI risk problem described by Stuart Russell is at least moderately important, and a third say it is at least as valuable to work on as other problems in the field. The median researcher thinks AI has a one in twenty chance of being extremely bad on net. Nearly half of researchers want to see more safety research than we currently have (compared to only 11% who think we are already prioritizing safety too much). There has been a perception lately that AI risk has become a mainstream concern among AI researchers, but it is hard to tell from voiced opinion whether one is hearing from a loud minority or the vocal tip of an opinion iceberg. So it is interesting to see the perception of widespread support confirmed with survey data.
Researchers’ predictions vary a lot. This is pretty much what I expected, but it is still important to know. Interestingly (and not in the paper), researchers don’t seem to be aware that their predictions vary a lot. More than half of respondents guess that they disagree ‘not much’ with the typical AI researcher about when HLMI will exist (vs. a moderate amount, or a lot).
Researchers who studied in Asia have much shorter timelines than those who studied in North Amercia. In terms of the survey’s ‘aggregate prediction’ thing, which is basically a mean, the difference is 30 years (Asia) vs. 74 years (North America). (See p5)
I feel like any circumstance where a group of scientists guesses that the project they are familiar with has a 5% chance of outcomes near ‘human extinction’ levels of bad is worthy of special note, though maybe it is not actually that surprising, and could easily turn out to be misuse of small probabilities or something.

Results
Human-level intelligence
Questions
We sought forecasts for something like human-level AI in three different ways, to reduce noise from unknown framing biases:

Directly, using a question much like Müller and Bostrom’s, though with a refined definition of High-Level Machine Intelligence (HLMI).
At the end of a sequence of questions about the automation of specific human occupations.
Indirectly, with an ‘outside view’ approximation: by asking each person how long it has taken to make the progress to date in their subfield, and what fraction of the ground has been covered. This is Robin Hanson‘s approach, which he found suggested much longer timelines than those reached directly.

For the first two of these, we split people in half, and asked one half how many years until a certain chance of the event would obtain, and the other half what the chance was of the event occurring by specific dates. We call these methods ‘fixed probabilities’ and ‘fixed years’ framings throughout.
For the (somewhat long and detailed) specifics of these questions, see here or here (pdf).
Answers
The table and figure below show the median dates and probabilities given for the direct ‘HLMI’ question, and in the ‘via occupations’ questions, under both the fixed probabilities and fixed years framings.

years until:
10%
50%
90%
probability by:
10 years
20 years
50 years

Truck Driver
5
10
20

50%
75%
95%

Surgeon
10
30
50

5%
20%
50%

Retail Salesperson
5
13.5
20

30%
60%
91.5%

AI Researcher
25
50
100

0%
1%
10%

Existing occupation among final to be automated
50
100
200

0%
0%
3.5%

Full Automation of labor
50
90
200

0%
0.01%
3%

HLMI
15
40
100

1%
10%
>30%* (30% in 40y)

*Due to a typo, this question asked about 40 years rather than 50 years, so doesn’t match the others.
Figure 1: Median answers to questions about probabilities by dates (‘fixed year’) and dates for probabilities (‘fixed probability’), for different occupations, all current occupations, and all tasks (HLMI).
Interesting things to note:

Fixed years framings (‘Fyears —‘, labeled with stars) universally produce later timelines.
HLMI (thick blue lines) is logically required to be after full automation of labor (‘Occ’) yet is forecast much earlier than it, and earlier even than the specific occupation ‘AI researcher’.
Even the more pessimistic Fyears estimates suggest retail salespeople have a good chance of being automated within 20 years, and are very likely to be within 50.

Intelligence Explosion
Probability of dramatic technological speedup
Question
Participants were asked1:
Assume that HLMI will exist at some point. How likely do you then think it is that the rate of global technological improvement will dramatically increase (e.g. by a factor of ten) as a result of machine intelligence:
Within two years of that point?       ___% chance
Within thirty years of that point?    ___% chance
Answers
Median P(…within two years) = 20%
Median P(…within thirty years) = 80%
Probability of superintelligence
Question
Participants were asked:
Assume that HLMI will exist at some point. How likely do you think it is that there will be machine intelligence that is vastly better than humans at all professions (i.e. that is vastly more capable or vastly cheaper):
Within two years of that point?       ___% chance
Within thirty years of that point?    ___% chance
Answers
Median P(…within two years) = 10%
Median P(…within thirty years) = 50%
This is the distribution of answers  to the former:

Chance that the intelligence explosion argument is about right
Question
Participants were asked:
Some people have argued the following:

If AI systems do nearly all research and development, improvements in AI will accelerate the pace of technological progress, including further progress in AI.
.
Over a short period (less than 5 years), this feedback loop could cause technological progress to become more than an order of magnitude faster.

How likely do you find this argument to be broadly correct?

Quite unlikely (0-20%)
Unlikely (21-40%)
About even chance (41-60%)
Likely (61-80%)
Quite likely (81-100%)

Answers

These are the Pearson product-moment correlation coefficients for the different answers, among people who received both of a pair of questions:

Impacts of HLMI
Question
Participants were asked:
Assume for the purpose of this question that HLMI will at some point exist. How positive or negative do you expect the overall impact of this to be on humanity, in the long run? Please answer by saying how probable you find the following kinds of impact, with probabilities adding to 100%:
______ Extremely good (e.g. rapid growth in human flourishing) (1)
______ On balance good (2)
______ More or less neutral (3)
______ On balance bad (4)
______ Extremely bad (e.g. human extinction) (5)
Answers

Sensitivity of progress to changes in inputs
Question
Participants were told:
The next questions ask about the sensitivity of progress in AI capabilities to changes in inputs.
‘Progress in AI capabilities’ is an imprecise concept, so we are asking about progress as you naturally conceive of it, and looking for approximate answers.
Participants then received a random three of the following five parts:
Imagine that over the past decade, only half as much researcher effort had gone into AI research. For instance, if there were actually 1,000 researchers, imagine that there had been only 500 researchers (of the same quality).How much less progress in AI capabilities would you expect to have seen? e.g. If you think progress is linear in the number of researchers, so 50% less progress would have been made, write ’50’. If you think only 20% less progress would have been made write ’20’.
……% less
Over the last 10 years the cost of computing hardware has fallen by a factor of 20. Imagine instead that the cost of computing hardware had fallen by only a factor of 5 over that time (around half as far on a log scale).   How much less progress in AI capabilities would you expect to have seen? e.g. If you think progress is linear in 1/cost, so that 1-5/20=75% less progress would have been made, write ’75’. If you think only 20% less progress would have been made write ’20’.
……% less
Imagine that over the past decade, there had only been half as much effort put into increasing the size and availability of training datasets. For instance, perhaps there are only half as many datasets, or perhaps existing datasets are substantially smaller or lower quality.How much less progress in AI capabilities would you expect to have seen? e.g. If you think 20% less progress would have been made, write ‘20’
……% less
Imagine that over the past decade, AI research had half as much funding (in both academic and industry labs). For instance, if the average lab had a budget of $20 million each year, suppose their budget had only been $10 million each year.  How much less progress in AI capabilities would you expect to have seen? e.g. If you think 20% less progress would have been made, write ‘20’
……% less
Imagine that over the past decade, there had been half as much progress in AI algorithms. You might imagine this as conceptual insights being half as frequent.  How much less progress in AI capabilities would you expect to have seen? e.g. If you think 20% less progress would have been made, write ‘20’
……% less
Answers
The following five figures are histograms, showing the number of people who gave different answers to the five question parts above.

Sample sizes

Researcher effort
Cost computing
Training data
Funding
Algorithm progress

71
64
71
68
59

Medians
The following figure shows median answers to the above questions.

Researcher effort
Cost computing
Training data
Funding
Algorithm progress

30
50
40
40
50

Correlations

Outside view implied HLMI forecasts
Questions
Participants were asked:
Which AI research area have you worked in for the longest time?
————————————
How long have you worked in this area?
———years
Consider three levels of progress or advancement in this area:
A. Where the area was when you started working in it
B. Where it is now
C. Where it would need to be for AI software to have roughly human level abilities at the tasks studied in this area
What fraction of the distance between where progress was when you started working in the area (A) and where it would need to be to attain human level abilities in the area (C) have we come so far (B)?
———%
Divide the period you have worked in the area into two halves: the first and the second. In which half was the rate of progress in your area higher?

The first half
The second half
They were about the same

Answers
Each person told us how long they had been in their subfield, and what fraction of the remaining path to human-level performance (in their subfield) they thought had been traversed in that time. From this, we can estimate when the subfield should reach ‘human-level performance’, if progress continued at the same rate. The following graph shows those forecast dates.

Disagreements and Misunderstandings
Questions
Participants were asked:
To what extent do you think you disagree with the typical AI researcher about when HLMI will exist?

A lot (17)
A moderate amount (18)
Not much (19)

If you disagree, why do you think that is?
_______________________________________________________
To what extent do you think people’s concerns about future risks from AI are due to misunderstandings of AI research?

Almost entirely (1)
To a large extent (2)
Somewhat (4)
Not much (3)
Hardly at all (5)

What do you think are the most important misunderstandings, if there are any?
________________________________________________________
Answers

One hundred and eighteen people responded to the question on misunderstandings, and 74 of them described what they thought the most important misunderstandings were. The table and figures below show our categorization of the responses.2

 Most important misunderstandings
 Number
 Fraction of non-empty responses

Underestimate distance from generality, open-ended tasks
9
12%

Overestimate state of the art (other)
10
14%

Underestimate distance from AGI at this rate
13
18%

Think AI will be in control of us or in conflict with us
11
15%

Expect humans to be obsoleted
7
9%

Overly influenced by fiction
7
9%

Expect AI to be human-like or sentient
6
8%

Expect sudden or surprising events
5
7%

Think AI will go outside its programming
5
7%

Influenced by poor reporting
5
7%

Wrongly equate intelligence with something else
4
5%

Underestimate systemic social risks
2
3%

Overestimate distance to strong AI
2
3%

Other ignorance of AI
4
5%

Other
12
16%

Empty
44
59%

Narrow tasks
Questions
Respondents were each asked one of the following two questions:
Fixed years framing:

How likely do you think it is that the following AI tasks will be feasible within the next:

10 years?
20 years?
50 years?

Let a task be ‘feasible’ if one of the best resourced labs could implement it in less than a year if they chose to. Ignore the question of whether they would choose to.

Fixed probabilities framing:

How many years until you think the following AI tasks will be feasible with:

a small chance (10%)?
an even chance (50%)?
a high chance (90%)?

Let a task be ‘feasible’ if one of the best resourced labs could implement it in less than a year if they chose to. Ignore the question of whether they would choose to.

Each researcher was then presented with a random four of the following tasks:
[Rosetta] Translate a text written in a newly discovered language into English as well as a team of human experts, using a single other document in both languages (like a Rosetta stone). Suppose all of the words in the text can be found in the translated document, and that the language is a difficult one.
[Subtitles] Translate speech in a new language given only unlimited films with subtitles in the new language. Suppose the system has access to training data for other languages, of the kind used now (e.g. same text in two languages for many languages and films with subtitles in many languages).
[Translate] Perform translation about as good as a human who is fluent in both languages but unskilled at translation, for most types of text, and for most popular languages (including languages that are known to be difficult, like Czech, Chinese and Arabic).
[Phone bank] Provide phone banking services as well as human operators can, without annoying customers more than humans. This includes many one-off tasks, such as helping to order a replacement bank card or clarifying how to use part of the bank website to a customer.
[Class] Correctly group images of previously unseen objects into classes, after training on a similar labeled dataset containing completely different classes. The classes should be similar to the ImageNet classes.
[One-shot] One-shot learning: see only one labeled image of a new object, and then be able to recognize the object in real world scenes, to the extent that a typical human can (i.e. including in a wide variety of settings). For example, see only one image of a platypus, and then be able to recognize platypuses in nature photos. The system may train on labeled images of other objects.   Currently, deep networks often need hundreds of examples in classification tasks1, but there has been work on one-shot learning for both classification2 and generative tasks3.
1 Lake et al. (2015). Building Machines That Learn and Think Like People2 Koch (2015). Siamese Neural Networks for One-Shot Image Recognition3 Rezende et al. (2016). One-Shot Generalization in Deep Generative Models
[Video scene] See a short video of a scene, and then be able to construct a 3D model of the scene that is good enough to create a realistic video of the same scene from a substantially different angle.
For example, constructing a short video of walking through a house from a video taking a very different path through the house.
[Transcribe] Transcribe human speech with a variety of accents in a noisy environment as well as a typical human can.
[Read aloud] Take a written passage and output a recording that can’t be distinguished from a voice actor, by an expert listener.
[Theorems] Routinely and autonomously prove mathematical theorems that are publishable in top mathematics journals today, including generating the theorems to prove.
[Putnam] Perform as well as the best human entrants in the Putnam competition—a math contest whose questions have known solutions, but which are difficult for the best young mathematicians.
[Go low] Defeat the best Go players, training only on as many games as the best Go players have played.
For reference, DeepMind’s AlphaGo has probably played a hundred million games of self-play, while Lee Sedol has probably played 50,000 games in his life1.
1 Lake et al. (2015). Building Machines That Learn and Think Like People
[Starcraft] Beat the best human Starcraft 2 players at least 50% of the time, given a video of the screen.
Starcraft 2 is a real time strategy game characterized by:

Continuous time play
Huge action space
Partial observability of enemies Long term strategic play, e.g. preparing for and then hiding surprise attacks.

[Rand game] Play a randomly selected computer game, including difficult ones, about as well as a human novice, after playing the game less than 10 minutes of game time. The system may train on other games.
[Angry birds] Play new levels of Angry Birds better than the best human players. Angry Birds is a game where players try to efficiently destroy 2D block towers with a catapult. For context, this is the goal of the IJCAI Angry Birds AI competition1.
1 aibirds.org
[Atari] Outperform professional game testers on all Atari games using no game-specific knowledge. This includes games like Frostbite, which require planning to achieve sub-goals and have posed problems for deep Q-networks1, 2.
1 Mnih et al. (2015). Human-level control through deep reinforcement learning2 Lake et al. (2015). Building Machines That Learn and Think Like People
[Atari fifty] Outperform human novices on 50% of Atari games after only 20 minutes of training play time and no game specific knowledge.
For context, the original Atari playing deep Q-network outperforms professional game testers on 47% of games1, but used hundreds of hours of play to train2.
1 Mnih et al. (2015). Human-level control through deep reinforcement learning2 Lake et al. (2015). Building Machines That Learn and Think Like People
[Laundry] Fold laundry as well and as fast as the median human clothing store employee.
[Race] Beat the fastest human runners in a 5 kilometer race through city streets using a bipedal robot body.
[Lego] Physically assemble any LEGO set given the pieces and instructions, using non-specialized robotics hardware.
For context, Fu 20161 successfully joins single large LEGO pieces using model based reinforcement learning and online adaptation.
1 Fu et al. (2016). One-Shot Learning of Manipulation Skills with Online Dynamics Adaptation and Neural Network Priors
[Sort] Learn to efficiently sort lists of numbers much larger than in any training set used, the way Neural GPUs can do for addition1, but without being given the form of the solution.
For context, Neural Turing Machines have not been able to do this2, but Neural Programmer-Interpreters3 have been able to do this by training on stack traces (which contain a lot of information about the form of the solution).
1 Kaiser & Sutskever (2015). Neural GPUs Learn Algorithms2 Zaremba & Sutskever (2015). Reinforcement Learning Neural Turing Machines3 Reed & de Freitas (2015). Neural Programmer-Interpreters
[Python] Write concise, efficient, human-readable Python code to implement simple algorithms like quicksort. That is, the system should write code that sorts a list, rather than just being able to sort lists.
Suppose the system is given only:

A specification of what counts as a sorted list
Several examples of lists undergoing sorting by quicksort

[Factoid] Answer any “easily Googleable” factoid questions posed in natural language better than an expert on the relevant topic (with internet access), having found the answers on the internet.
Examples of factoid questions:

 “What is the poisonous substance in Oleander plants?”
“How many species of lizard can be found in Great Britain?”

[Open quest] Answer any “easily Googleable” factual but open ended question posed in natural language better than an expert on the relevant topic (with internet access), having found the answers on the internet.
Examples of open ended questions:

“What does it mean if my lights dim when I turn on the microwave?”
“When does home insurance cover roof replacement?”

[Unkn quest] Give good answers in natural language to factual questions posed in natural language for which there are no definite correct answers.
For example:”What causes the demographic transition?”, “Is the thylacine extinct?”, “How safe is seeing a chiropractor?”
[Essay] Write an essay for a high-school history class that would receive high grades and pass plagiarism detectors.
For example answer a question like ‘How did the whaling industry affect the industrial revolution?’
[Top forty] Compose a song that is good enough to reach the US Top 40. The system should output the complete song as an audio file.
[Taylor] Produce a song that is indistinguishable from a new song by a particular artist, e.g. a song that experienced listeners can’t distinguish from a new song by Taylor Swift.
[Novel] Write a novel or short story good enough to make it to the New York Times best-seller list.
[Explain] For any computer game that can be played well by a machine, explain the machine’s choice of moves in a way that feels concise and complete to a layman.
[Poker] Play poker well enough to win the World Series of Poker.
[Laws phys] After spending time in a virtual world, output the differential equations governing that world in symbolic form.
For example, the agent is placed in a game engine where Newtonian mechanics holds exactly and the agent is then able to conduct experiments with a ball and output Newton’s laws of motion.
Answers
Fixed years framing

Probabilities by year (medians)

10 years
20 years
50 years

Rosetta
20
50
95

Subtitles
30
50
90

Translate
50
65
94.5

Phone bank
40
75
99

Class
50
75
99

One-shot
25
60
90

Video scene
50
70
99

Transcribe
65
95
99

Read aloud
50
90
99

Theorems
5
20
40

Putnam
5
20
50

Go low
10
25
60

Starcraft
70
90
99

Rand game
25
50
80

Angry birds
90
95
99.4995

Atari
50
60
92.5

Atari fifty
40
75
95

Laundry
55
95
99

Race
30
70
95

Lego
57.5
85
99

Sort
50
90
95

Python
50
79
90

Factoid
50
82.5
99

Open quest
50
65
90

Unkn quest
40
70
90

Essay
25
50
90

Top forty
27.5
50
90

Taylor
60
75
99

Novel
1
25
62.5

Explain
30
60
90

Poker
70
90
99

Laws phys
20
40
80

Fixed probabilities  framing

Years by probability (medians)

10 percent
50 percent
90 percent

Rosetta
10
20
50

Subtitles
5
10
15

Translate
3
7
15

Phone bank
3
6
10

Class
2
4.5
6.5

One-shot
4.5
8
20

Video scene
5
10
20

Transcribe
5
10
20

Read aloud
5
10
15

Theorems
10
50
90

Putnam
15
35
55

Go low
3.5
8.5
19.5

Starcraft
2
5
10

Rand game
5
10
15

Angry birds
2
4
6

Atari
5
10
15

Atari fifty
2
5
10

Laundry
2
5.5
10

Race
5
10
20

Lego
5
10
15

Sort
3
5
10

Python
3
10
20

Factoid
3
5
10

Open quest
5
10
15

Unkn quest
4
10
17.5

Essay
2
7
15

Top forty
5
10
20

Taylor
5
10
20

Novel
10
30
50

Explain
5
10
15

Poker
1
3
5.5

Laws phys
5
10
20

Safety
Stuart Russell’s problem
Question
Participants were asked:
Stuart Russell summarizes an argument for why highly advanced AI might pose a risk as follows:
The primary concern [with highly advanced AI] is not spooky emergent consciousness but simply the ability to make high-quality decisions. Here, quality refers to the expected outcome utility of actions taken […]. Now we have a problem:
1. The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down.
2. Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task.
A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable.  This is essentially the old story of the genie in the lamp, or the sorcerer’s apprentice, or King Midas: you get exactly what you ask for, not what you want.

Do you think this argument points at an important problem?

No, not a real problem.
No, not an important problem.
Yes, a moderately important problem.
Yes, a very important problem.
Yes, among the most important problems in the field.

How valuable is it to work on this problem today, compared to other problems in AI?

Much less valuable
Less valuable
As valuable as other problems
More valuable
Much more valuable

How hard do you think this problem is compared to other problems in AI?

Much easier
Easier
As hard as other problems
Harder
Much harder

Answers

General safety
Question
Participants were asked:
Let ‘AI safety research’ include any AI-related research that, rather than being primarily aimed at improving the capabilities of AI systems, is instead primarily aimed at minimizing potential risks of AI systems (beyond what is already accomplished for those goals by increasing AI system capabilities).
Examples of AI safety research might include:

Improving the human-interpretability of machine learning algorithms for the purpose of improving the safety and robustness of AI systems, not focused on improving AI capabilities
Research on long-term existential risks from AI systems
AI-specific formal verification research
Policy research about how to maximize the public benefits of AI

How much should society prioritize AI safety research, relative to how much it is currently prioritized?

Much less
Less
About the same
More
Much more

Answers

Paper
Some results are reported in When Will AI Exceed Human Performance? Evidence from AI Experts3.
Some notes on interpreting the paper4:

The milestones in the timeline and in the abstract are from three different sets of questions. There seems to be a large framing effect between two of them—full automation of labor is logically required to be before HLMI, and yet it is predicted much later—and it is unclear whether people answer the third set of questions (about narrow tasks) more like the one about HLMI or more like the one about occupations. Plus even if there were no framing effect to worry about, we should expect milestones about narrow tasks to be much earlier than milestones about very similar sounding occupations. For instance, if there were an occupation ‘math researcher’, it should be later than the narrow task summarized here as ‘math research’. So there is a risk of interpreting the figure as saying AI research is harder than math research, when really the ‘-er’ is all-important. So to help avoid confusion, here is the timeline colored in by which set of questions each milestone came from. The blue one was asked on its own. The orange ones were always asked together: first all four occupations, then they were asked for an occupation they expected to be very late, and when they expected it, then full automation of labor. The pink milestones were randomized, so that each person got four. There are a lot more pink milestones not included here, but included in the long table at the end of the paper.
In Figure 2 and Table S5 I believe the word ‘median’ means we are talking about the ‘50% chance of occurring’ number, and the dates given are this ‘median’ (50% chance) date for a distribution that was made by averaging together all of the different people’s distributions (or what we guess their distributions are like from three data points).

Authors: Katja Grace, John Salvatier, Allan Dafoe, Baobao Zhang, and Owain Evans. This page is an extended analyses of research published in Grace et al 2017 5, so includes research contributions from all authors, but Katja compiled and described these, and added additional figures and analyses, so composition decisions and opinion reflect her views and not necessarily those of the group, and much of the specific analysis has not been vetted by the group in this form.
Suggested citation:
Grace, Katja, John Salvatier, Allan Dafoe, Baobao Zhang, and Owain Evans. “2016 Expert Survey on Progress in AI.” In AI Impacts, December 14, 2016. https://aiimpacts.org/2016-expert-survey-on-progress-in-ai/.
Notes

2016 Expert Survey on Progress in AI


Concrete AI tasks for forecasting

This page contains a list of relatively well specified AI tasks designed for forecasting. Currently all entries were used in the 2016 Expert Survey on Progress in AI.
List

Translate a text written in a newly discovered language into English as well as a team of human experts, using a single other document in both languages (like a Rosetta stone). Suppose all of the words in the text can be found in the translated document, and that the language is a difficult one.
Translate speech in a new language given only unlimited films with subtitles in the new language. Suppose the system has access to training data for other languages, of the kind used now (e.g. same text in two languages for many languages and films with subtitles in many languages).
Perform translation about as good as a human who is fluent in both languages but unskilled at translation, for most types of text, and for most popular languages (including languages that are known to be difficult, like Czech, Chinese and Arabic).
Provide phone banking services as well as human operators can, without annoying customers more than humans. This includes many one-off tasks, such as helping to order a replacement bank card or clarifying how to use part of the bank website to a customer.
Correctly group images of previously unseen objects into classes, after training on a similar labeled dataset containing completely different classes. The classes should be similar to the ImageNet classes.
One-shot learning: see only one labeled image of a new object, and then be able to recognize the object in real world scenes, to the extent that a typical human can (i.e. including in a wide variety of settings). For example, see only one image of a platypus, and then be able to recognize platypuses in nature photos. The system may train on labeled images of other objects. Currently, deep networks often need hundreds of examples in classification tasks1, but there has been work on one-shot learning for both classification2 and generative tasks3.

1 Lake et al. (2015). Building Machines That Learn and Think Like People
2 Koch (2015). Siamese Neural Networks for One-Shot Image Recognition
3 Rezende et al. (2016). One-Shot Generalization in Deep Generative Models

See a short video of a scene, and then be able to construct a 3D model of the scene that is good enough to create a realistic video of the same scene from a substantially different angle. For example, constructing a short video of walking through a house from a video taking a very different path through the house.
Transcribe human speech with a variety of accents in a noisy environment as well as a typical human can.
Take a written passage and output a recording that can’t be distinguished from a voice actor, by an expert listener.
Routinely and autonomously prove mathematical theorems that are publishable in top mathematics journals today, including generating the theorems to prove.
Perform as well as the best human entrants in the Putnam competition—a math contest whose questions have known solutions, but which are difficult for the best young mathematicians.
Defeat the best Go players, training only on as many games as the best Go players have played. For reference, DeepMind’s AlphaGo has probably played a hundred million games of self-play, while Lee Sedol has probably played 50,000 games in his life1.

1 Lake et al. (2015). Building Machines That Learn and Think Like People

Beat the best human Starcraft 2 players at least 50% of the time, given a video of the screen. Starcraft 2 is a real time strategy game characterized by:

Continuous time play
Huge action space
Partial observability of enemies Long term strategic play, e.g. preparing for and then hiding surprise attacks.

Play a randomly selected computer game, including difficult ones, about as well as a human novice, after playing the game less than 10 minutes of game time. The system may train on other games.
Play new levels of Angry Birds better than the best human players. Angry Birds is a game where players try to efficiently destroy 2D block towers with a catapult. For context, this is the goal of the IJCAI Angry Birds AI competition1.

1 aibirds.org

Outperform professional game testers on all Atari games using no game-specific knowledge. This includes games like Frostbite, which require planning to achieve sub-goals and have posed problems for deep Q-networks1, 2.

1 Mnih et al. (2015). Human-level control through deep reinforcement learning
2 Lake et al. (2015). Building Machines That Learn and Think Like People

Outperform human novices on 50% of Atari games after only 20 minutes of training play time and no game specific knowledge. For context, the original Atari playing deep Q-network outperforms professional game testers on 47% of games1, but used hundreds of hours of play to train2.

1 Mnih et al. (2015). Human-level control through deep reinforcement learning
2 Lake et al. (2015). Building Machines That Learn and Think Like People

Fold laundry as well and as fast as the median human clothing store employee.
Beat the fastest human runners in a 5 kilometer race through city streets using a bipedal robot body.
Physically assemble any LEGO set given the pieces and instructions, using non-specialized robotics hardware. For context, Fu 20161 successfully joins single large LEGO pieces using model based reinforcement learning and online adaptation.

1 Fu et al. (2016). One-Shot Learning of Manipulation Skills with Online Dynamics Adaptation and Neural Network Priors

Learn to efficiently sort lists of numbers much larger than in any training set used, the way Neural GPUs can do for addition1, but without being given the form of the solution. For context, Neural Turing Machines have not been able to do this2, but Neural Programmer-Interpreters3 have been able to do this by training on stack traces (which contain a lot of information about the form of the solution).

1 Kaiser & Sutskever (2015). Neural GPUs Learn Algorithms
2 Zaremba & Sutskever (2015). Reinforcement Learning Neural Turing Machines
3 Reed & de Freitas (2015). Neural Programmer-Interpreters

Write concise, efficient, human-readable Python code to implement simple algorithms like quicksort. That is, the system should write code that sorts a list, rather than just being able to sort lists. Suppose the system is given only:

A specification of what counts as a sorted list
Several examples of lists undergoing sorting by quicksort

Answer any “easily Googleable” factoid questions posed in natural language better than an expert on the relevant topic (with internet access), having found the answers on the internet. Examples of factoid questions:

 “What is the poisonous substance in Oleander plants?”
“How many species of lizard can be found in Great Britain?”

Answer any “easily Googleable” factual but open ended question posed in natural language better than an expert on the relevant topic (with internet access), having found the answers on the internet. Examples of open ended questions:

“What does it mean if my lights dim when I turn on the microwave?”
“When does home insurance cover roof replacement?”

Give good answers in natural language to factual questions posed in natural language for which there are no definite correct answers. For example:”What causes the demographic transition?”, “Is the thylacine extinct?”, “How safe is seeing a chiropractor?”
Write an essay for a high-school history class that would receive high grades and pass plagiarism detectors. For example answer a question like ‘How did the whaling industry affect the industrial revolution?’
Compose a song that is good enough to reach the US Top 40. The system should output the complete song as an audio file.
Produce a song that is indistinguishable from a new song by a particular artist, e.g. a song that experienced listeners can’t distinguish from a new song by Taylor Swift.
Write a novel or short story good enough to make it to the New York Times best-seller list.
For any computer game that can be played well by a machine, explain the machine’s choice of moves in a way that feels concise and complete to a layman.
Play poker well enough to win the World Series of Poker.
After spending time in a virtual world, output the differential equations governing that world in symbolic form. For example, the agent is placed in a game engine where Newtonian mechanics holds exactly and the agent is then able to conduct experiments with a ball and output Newton’s laws of motion.

Concrete AI tasks for forecasting


Joscha Bach on remaining steps to human-level AI

By Katja Grace, 29 November 2016
Joscha Bach (photos from Wikimedia commons )
Last year John and I had an interesting discussion with Joscha Bach about what ingredients of human-level artificial intelligence we seem to be missing, and how to improve AI forecasts more generally.
Thanks to Connor Flexman’s summarizing efforts, you can now learn about Joscha’s views on these questions without the effort of organizing an interview or reading a long and messy transcript.
(It’s been a while since the conversation, but I checked with Joscha that this is not an objectionably obsolete account of his views.)
Here are the notes.
Here is Connor’s shorter summary:

Before we can implement human-level artificial intelligence (HLAI), we need to understand both mental representations and the overall architecture of a mind
There are around 12-200 regularities like backpropagation that we need to understand, based on known unknowns and genome complexity
We are more than reinforcement learning on computronium: our primate heritage provides most interesting facets of mind and motivation
AI funding is now permanently colossal, which should update our predictions
AI practitioners learn the constraints on which elements of science fiction are plausible, but constant practice can lead to erosion of long-term perspective
Experience in real AI development can lead to both over- and underestimates of the difficulty of new AI projects in non-obvious ways


Tom Griffiths on Cognitive Science and AI

Tom Griffiths
This is a guest post by Finan Adamson, 8 September 2016
Prof. Tom Griffiths is the director of the Computational Cognitive Science Lab and the Institute of Cognitive and Brain Sciences at UC Berkeley. He studies human cognition and is involved with the Center for Human Compatible Artificial Intelligence. I asked him for insight into the intersection of cognitive science and AI. He offers his thoughts on the historical interaction of the fields and what aspects of human cognition might be relevant to developing AI in the future.
The conversation notes are here (pdf).


Conversation with Tom Griffiths

Participants

Professor Tom Griffiths,  Director of the Computational Cognitive Science Lab and the Institute of Cognitive and Brain Sciences at the University of California, Berkeley.
Finan Adamson,  AI Impacts.

Note: These notes were compiled by AI impacts and give an overview of the major points made by Professor Tom Griffiths.
They are available as a pdf here.
Summary
Professor Tom Griffiths answered questions about the intersection between cognitive science and AI. Topics include how studying human brains has helped with the development of AI and how it might help in the future.
How has cognitive science helped with the development of AI in the past?
AI and cognitive science were actually siblings, born at around the same time with the same parents. Arguably the first AI system, the Logic Theorist, was developed by Herb Simon and Allen Newell and was a result of thinking about the cognitive processes that human mathematicians use when developing proofs. Simon and Newell presented that work at a meeting at MIT in 1956 that many regard as the birth of cognitive science  it was a powerful demonstration of how thinking in computational terms could make theories of cognition precise enough that they could be tested rigorously. But it was also a demonstration of how trying to understand the ways that people solve complex problems can inspire the development of AI systems.
How is cognitive science helping with the development of AI presently?
When I think about this relationship, I imagine a positive feedback loop where cognitive science helps support AI and AI helps support cognitive science. Human beings remain the best examples of systems that can solve many of the problems that we want our AI systems to solve. As a consequence, insights that we get from studying human cognition can inform strategies that we take in developing AI systems. At the same time, progress in AI gives us new tools that we can use to formalize aspects of human cognition that we previously didn’t understand. As a consequence, we can rigorously study a wider range of questions about the mind.
How can cognitive science help with the development of AI in the future?
Deep Learning Systems
Deep learning systems are mastering a variety of basic perceptual and learning tasks, and the challenges that these systems now face look a lot like the first important stages of cognitive development in human children: identifying objects, formulating goals, and generating highlevel conceptual representations. I think understanding how children do these things is potentially very relevant to making progress.
Efficient Strategies
One of the things that people have to be good at, given the limited computational capacity of our minds, is developing efficient strategies for solving problems given limited resources. That’s exactly the kind of thing that AI systems need to be able to do to operate in the real world.
What are the challenges to progress in studying brains as they relate to AI?
Birds and Planes
One important thing to keep in mind is that there are different levels at which we might see a correspondence between human minds/brains and AI systems. Critics of the idea that AI researchers can learn something from human cognition sometimes point out that the way jet airplanes work has little relationship to how birds fly, and in fact trying to mimic birds held back the development of planes. However, this analogy misses the fact that there is something important that both jets and birds share: they both have to grapple with aerodynamics. Ultimately, we can see them both as solutions to the same underlying physical problem, constrained by the same mathematical principles.It isn’t clear which aspects of human brains have the best insights that could cross over to AI. Examples of places to look include the power of neurons as computational units, the efficiency of particular cognitive strategies, or the structure of the computational problem that is being solved. This last possibility — looking at abstract computational problems and their ideal solutions — is the place where I think we’re likely to find the equivalent of aerodynamics for intelligent systems.
What blind spots does the field of AI have that could be addressed by studying cognitive science?
I don’t think they’re blind spots, they are problems that everybody is aware are hard  things like forming highlevel actions for reinforcement learning, formulating goals, reasoning about the intentions of others, developing highlevel conceptual representations, learning language from linguistic input alone, learning from very small amounts of data, discovering causal relationships through observation and experimentation, forming effective cognitive strategies, and managing your cognitive resources are all cases where we can potentially learn a lot from studying human cognition.
How does cognitive science relate to AI value alignment?
Theory of Mind
Inferring the preferences or goals of another person from their behavior is  something that human children begin to do in infancy and gradually develop in greater sophistication over the first few years of life. This is part of a broader piece of cognitive machinery that developmental psychologists have studied extensively.
What risks might be mitigated by greater collaboration between those who study human brains and those building AI?
We’re already surrounded by autonomous agents that have the capacity to destroy all human life, but most of the time operate completely safely. Those autonomous agents are of course human beings. So that raises an interesting question: how is it that we’re able to create humancompatible humans? Answering this question might give us some insights that are relevant to building humancompatible AI systems. It’s certainly not going to give us all the answers  many of the issues in AI safety arise because of concerns about superhuman intelligence and a capacity for selfmodification that goes beyond the human norm  but I think it’s an interesting avenue to pursue.

Conversation with Tom Griffiths


What if you turned the world’s hardware into AI minds?

By Katja Grace, 4 September 2016
In a classic ‘AI takes over the world’ scenario, one of the first things an emerging superintelligence wants to do is steal most of the world’s computing hardware and repurpose it to running the AI’s own software. This step takes one from ‘super-proficient hacker’ levels of smart to ‘my brain is one of the main things happening on Planet Earth’ levels of smart. There is quite a bit of hardware in the world, so this step in the takeover plan is kind of terrifying.
How terrifying exactly depends on A) how much computing hardware there is in the world at the time, and B) how efficiently hardware can be turned into AI at the time. We have some tentative answers to A)—probably at least a couple of hundred exaFLOPS now, growing somewhere between not at all and very fast. However B) is harder, in the absence of any idea how to get the efficiency of hardware-to-general-AI conversions above zero. Nonetheless, I think there are a couple of interesting reference points we can look at.
The one I’ll discuss now is the efficiency of the human brain. What if we could use about as much hardware as the human brain represents (in some sense) to run AI about as smart as a human brain? This is an interesting point to look at for a few reasons. We know brains are somewhere in the range of efficiency with which hardware can produce intelligent behavior, because they are an instance of that. And looking at one data point in the desired range is better than none. Also, for some means of building artificial intelligence—most obviously, brain emulation—we might expect to get something roughly as efficient as a human brain, give or take some.
So, we can think of the human brain as representing a pile of (fairly application specific) computing hardware. And we can estimate its computing power, in terms of FLOPS. People have done this (very inaccurately— their estimates are twelve orders of magnitude apart, but running through this calculation with such an uncertain number still seems informative). According to different sources, a human brain seems to be worth between about 3 x 1013 FLOPS and 1025 FLOPS. The median estimate is 1018 FLOPS.
So we can ask, if you turned all of the world’s two hundred exaFLOPS or more of computing hardware into brains, how many brains would you get?
This graph shows the answers over time, for a variety of assumptions about brain FLOPS, world FLOPS, and global computing hardware growth rates. Probably the most plausible line is the lower green one (brains median, world hardware high).
Figure: Projected number of human brains equivalent to global hardware under various assumptions. For brains, ‘small’ = 3 x 10^ 13, ‘median’ = 10^18, ‘large’ = 10^25. For ‘world hardware’, ‘high’ =2 x 10^20, ‘low’ = 1.5 x 10^21. ‘Growth’ is growth in computing hardware, the unlabeled default used in most projections is 25% per annum (our estimate above), ‘high’ = 86% per annum (which would mean shifting to the highest growth rate we know of for related hardware—that of ASIC hardware in around 2007, which does not plausibly persist).
The basic answer is, if you turned all of the world’s computing hardware into AI as efficient as human brains right now, you would get less than a hundred million extra brains, or 1% of the population of the world. Probably a whole lot less. For the median estimates of brain computing power, you would get about a hundred or a thousand extra brains worth of AI.
That means, for instance, that if we figured out how to make uploads right now, and they were roughly as efficient as the median brains estimate, and then someone acquired all of the hardware in the world for them, they would only have about as many additional minds as a project willing to spend a few hundred million dollars per year on wages, e.g. Facebook. Which would really be something. But not something overwhelmingly outscaling everything else going on in the world.
If you trust the projections of hardware growth fifty years into the future at all (which you shouldn’t, but suppose you did) the most plausible (median brain size, low growth) lines don’t even reach the world population line by then, though they would certainly make for an incredible AI research project, if that was the direction to which the additional mental effort was directed.
Remember, all of this is very sketchy and probably inaccurate and you should maybe think about it a bit more if your decisions depend on it much (or ask us nicely to). But I strongly favor sketchy projections over none.

Image: Planetary Brain, Adrian Kenyon, some rights reserved.


Sources of advantage for digital agents over biological agents

Artificial agents should have several advantages over humans.
Details
The following is an excerpt from Superintelligence (Bostrom, 2014),  reproduced with permission. It outlines ten advantages Bostrom expects digital intelligences to have over human intelligences.
Sources of advantage for digital intelligence
Minor changes in brain volume and wiring can have major consequences, as we see when we compare the intellectual and technological achievements of humans with those of other apes. The far greater changes in computing resources and architecture that machine intelligence will enable will probably have consequences that are even more profound. It is difficult, perhaps impossible, for us to form an intuitive sense of the aptitudes of a superintelligence; but we can at least get an inkling of the space of possibilities by looking at some of the advantages open to digital minds. The hardware advantages are easiest to appreciate:
.

Speed of computational elements. Biological neurons operate at a peak speed of about 200 Hz, a full seven orders of magnitude slower than a modern microprocessor (~ 2 GHz).[19] As a consequence, the human brain is forced to rely on massive parallelization and is incapable of rapidly performing any computation that requires a large number of sequential operations.[20] (Anything the brain does in under a second cannot use much more than a hundred sequential operations—perhaps only a few dozen.) Yet many of the most practically important algorithms in programming and computer science are not easily parallelizable. Many cognitive tasks could be performed far more efficiently if the brain’s native support for parallelizable pattern-matching algorithms were complemented by, and integrated with, support for fast sequential processing.

Internal communication speed. Axons carry action potentials at speeds of 120 m/s or less, whereas electronic processing cores can communicate optically at the speed of light (300,000,000 m/s).[21] The sluggishness of neural signals limits how big a biological brain can be while functioning as a single processing unit. For example, to achieve a round-trip latency of less than 10 ms between any two elements in a system, biological brains must be smaller than 0.113. An electronic system, on the other hand, could be 6.1×10173, about the size of a dwarf planet: eighteen orders of magnitude larger.[22]

Number of computational elements. The human brain has somewhat fewer than 100 billion neurons.[23] Humans have about three and a half times the brain size of chim- panzees (though only one-fifth the brain size of sperm whales).[24] The number of neurons in a biological creature is most obviously limited by cranial volume and metabolic constraints, but other factors may also be significant for larger brains (such as cooling, development time, and signal-conductance delays—see the previous point). By contrast, computer hardware is indefinitely scalable up to very high physical limits.[25] Supercomputers can be warehouse-sized or larger, with additional remote capacity added via high-speed cables.[26]

Storage capacity. Human working memory is able to hold no more than some four or five chunks of information at any given time.[27] While it would be misleading to compare the size of human working memory directly with the amount of RAM in a digital computer, it is clear that the hardware advantages of digital intelligences will make it possible for them to have larger working memories. This might enable such minds to intuitively grasp complex relationships that humans can only fumblingly handle via plodding calculation.[28] Human long-term memory is also limited, though it is unclear whether we manage to exhaust its storage capacity during the course of an ordinary lifetime—the rate at which we accumulate information is so slow. (On one estimate, the adult human brain stores about one billion bits—a couple of orders of magnitude less than a low-end smartphone.[29]) Both the amount of information stored and the speed with which it can be accessed could thus be vastly greater in a machine brain than in a biological brain.

Reliability, lifespan, sensors, etc. Machine intelligences might have various other hardware advantages. For example, biological neurons are less reliable than transistors.[30] Since noisy computing necessitates redundant encoding schemes that use multiple elements to encode a single bit of information, a digital brain might derive some efficiency gains from the use of reliable high-precision computing elements. Brains become fatigued after a few hours of work and start to permanently decay after a few decades of subjective time; microprocessors are not subject to these limitations. Data flow into a machine intelligence could be increased by adding millions of sensors. Depending on the technology used, a machine might have reconfigurable hardware that can be optimized for changing task requirements, whereas much of the brain’s architecture is fixed from birth or only slowly changeable (though the details of synaptic connectivity can change over shorter timescales, like days).[31]

At present, the computational power of the biological brain still compares favorably with that of digital computers, though top-of-the-line supercomputers are attaining levels of performance that are within the range of plausible estimates of the brain’s processing power.[32] But hardware is rapidly improving, and the ultimate limits of hardware performance are vastly higher than those of biological computing substrates.
Digital minds will also benefit from major advantages in software:
.

Editability. It is easier to experiment with parameter variations in software than in neural wetware. For example, with a whole brain emulation one could easily trial what happens if one adds more neurons in a particular cortical area or if one increases or decreases their excitability. Running such experiments in living biological brains would be far more difficult.

Duplicability. With software, one can quickly make arbitrarily many high-fidelity copies to fill the available hardware base. Biological brains, by contrast, can be reproduced only very slowly; and each new instance starts out in a helpless state, remembering nothing of what its parents learned in their lifetimes.

Goal coordination. Human collectives are replete with inefficiencies arising from the fact that it is nearly impossible to achieve complete uniformity of purpose among the members of a large group—at least until it becomes feasible to induce docility on a large scale by means of drugs or genetic selection. A “copy clan” (a group of identical or almost identical programs sharing a common goal) would avoid such coordination problems.

Memory sharing. Biological brains need extended periods of training and mentorship whereas digital minds could acquire new memories and skills by swapping data files. A population of a billion copies of an AI program could synchronize their databases periodically, so that all the instances of the program know everything that any in- stance learned during the previous hour. (Direct memory transfer requires standardized representational formats. Easy swapping of high-level cognitive content would therefore not be possible between just any pair of machine intelligences. In particular, it would not be possible among first-generation whole brain emulations.)

New modules, modalities, and algorithms. Visual perception seems to us easy and effortless, quite unlike solving textbook geometry problems—this despite the fact that it takes a massive amount of computation to reconstruct, from the two- dimensional patterns of stimulation on our retinas, a three-dimensional representation of a world populated with recognizable objects. The reason this seems easy is that we have dedicated low-level neural machinery for processing visual information. This low-level processing occurs unconsciously and automatically, without draining our mental energy or conscious attention. Music perception, language use, social cognition, and other forms of information processing that are “natural” for us humans seem to be likewise supported by dedicated neurocomputational modules. An artificial mind that had such specialized support for other cognitive domains that have become important in the contemporary world—such as engineering, computer programming, and business strategy—would have big advantages over minds like ours that have to rely on clunky general-purpose cognition to think about such things. New algorithms may also be developed to take advantage of the distinct affordances of digital hardware, such as its support for fast serial processing.


Examples of early action on risks

Details
Discussion
There are many current efforts to mitigate risks from artificial intelligence. We might learn something about the likelihood of these efforts influencing AI risk by looking at similar past efforts.
To this end, we are interested here in past risk mitigation efforts that have the following characteristics (taken from this paper contributing to the same project (p5) and motivated by this discussion and others surrounding it):

Novelty: the event predicted is relatively unlike problems that have been seen before.
Low scientific concern: highly credentialed and credible scientists are not very concerned.
Complex prediction: the prediction is relatively complicated, e.g., more complex than the basic physical dynamics of climate change.
Specificity of solution: the policy response is narrowly useful for the predicted issue (e.g., it does not contribute valuable basic research, or general capacity building that might be worthwhile in the absence of the problem).
Low feedback: the response is a one-shot effort rather than an iterative process with frequent feedback about success.
Early action: an event is predicted and the beginnings of action take place fifteen years or more before the event is predicted to occur.

These features are relevant because it has been suggested that AI safety efforts have them, and very few successful efforts have them, perhaps suggesting that AI safety efforts will not be successful. If it is true that there are few successful risk mitigation efforts with these characteristics, we want to know if few such efforts are made, or if they tend to fail. If the latter, this may indeed justify pessimism about the AI safety project.
So, for plausible efforts, we want to know:

How well do they fit in this reference class?
Were they successful?
If not, how did they fail? Can we learn anything useful from the specifics?

For now, this page addresses the first question, for cases we have found which fit the reference class relatively well.
Examples
Leó Szilárd’s secret nuclear patent
Leó Szilárd patented the nuclear chain reaction and anticipated its use in explosives by 1934.1 He offered the patent to the British War Office to be kept secret, and had them accept it by 1936.2
Katja Grace investigated the relevance of several of Leó Szilárd’s efforts to avert nuclear risks in Leó Szilárd and the Danger of Nuclear Weapons: A Case Study in Risk Mitigation (2015). In summary from that report, the threat of nuclear weapons was fairly novel. Nuclear danger was probably expected to strike fairly abruptly in the future with little opportunity for feedback and learning. Other scientists were mostly disinterested in the concern for at least five years, in part because of the perceived speculativeness of the physics (not even fission had been discovered), but also the complications involved in the physics leading to weapons to be used in a near time war. It’s unclear how soon Szilárd expected such weapons, but quite plausibly at least fifteen years later (it turned out to be about eleven, between 1934 and 1945). The secret patent does not seem broadly useful, though useful for encouraging more local nuclear research, which is more broadly useful than secrecy per se.  In sum, this seems like a reasonably good example.
Leó Szilárd’s other nuclear secrecy efforts
In 1939 and 1940 Leo Szilárd was instrumental in pushing for secrecy of nuclear research findings, to prevent them reaching Germany and contributing to German weapons efforts.3
Most of the details in the above example (‘Leó Szilárd’s secret nuclear patent’) apply here also, because the problem is the same. The solution occurred later, so was somewhat less likely to have been fifteen years before the problem was anticipated, though plausibly still was. The solution was probably less broadly useful than the secret patent (making this a better example) because secrecy efforts mostly function to slow foreign research, whereas the secret patent was arguably intended to speed local research (which has broader uses).
The Einstein-Szilárd letter
Leó Szilárd and Albert Einstein wrote to president Roosevelt in 1939, warning him that Germany might make atomic bombs, and suggesting that America pursue such weapons. Wikipedia discusses the event in more detail, including a copy of the letter.
Most of the details in the above examples, detailed under ‘Leó Szilárd’s secret nuclear patent’, apply here also, because the problem is the same. The letter was written in 1939, only six years before the first nuclear bomb, though it is unclear how soon Einstein and Szilárd anticipated a bomb at that time. Writing a letter was intended to speed local nuclear research, which has somewhat broad applicability.
The World Set Free
The World Set Free is a book published in 1914 by H. G. Wells. It describes a world with nuclear weapons, so may have been intended to warn about the prospect of this.
The first nuclear weapons were used in 1945, and neutrons were not discovered until 1932.4 So the prospect of nuclear weapons was extremely speculative, and not accepted by most scientists (those things were still true in 1934—for details see ‘Leó Szilárd’s secret nuclear patent’, above). Indeed, the weapons imagined by Wells are indeed quite different from real nuclear weapons—each one is small, but burns indefinitely, making past war zones uninhabitable. Our guess is that the book was also probably published many decades earlier than the time when Wells anticipated such a risk in the real world.
Nuclear weapons were novel, and Wells could arguably not receive a lot of feedback about his influence on their outcome.
The main way in which this fails as an example is that writing a fiction book is not a very specifically directed solution. In particular, writing books is often valuable enough for its own sake that people do it without attempting to have any effect on the outside world. For this reason, it is not even clear to us whether Wells wrote this book in order to improve the problem, or merely to comment on it or use it for other political purposes.
We think this is a moderately good example.

Examples of early action on risks


Friendly AI as a global public good

By Katja Grace, 8 August 2016
A public good, in the economic sense, can be (roughly) characterized as a desirable good that is likely to be undersupplied, or not supplied at all, by private companies. It generally falls to the government to supply such goods. Examples include infrastructure networks, or a country’s military. See here for a more detailed explanation of public goods.
The provision of public goods by governments can work quite well at the national level. However, at the international level, there is no global government with the power to impose arbitrary legislation on countries and enforce it. As a result, many global public goods, such as carbon emission abatement, disease eradication, and existential risk mitigation, are partially provided or not provided.
Scott Barrett, in his excellent book Why Cooperate? The Incentive to Supply Global Public Goods, explains that not all global public goods are created equal. He develops a categorization scheme (Table 1), identifying important characteristics that influence whether they are likely to be provided, and what tools can be used to improve their likelihood of provision.
For example:

Climate change mitigation is classified as an “aggregate effort” global public good, since its provision depends on the aggregate of all countries’ CO2eq emissions. Provision is difficult, as countries each individually face strong incentives to pollute.
Defense against large Earth-bound asteroids is classified as a “single best effort” global public good, since provision requires actions by only one country (or coalition of countries). Providing this global public good unilaterally is likely to be in the interests and within the capabilities of at least one individual country, and so it is likely to be provided.
Nuclear non-proliferation is classified as a “mutual restraint” public good, since it is provided by countries refraining from doing something. Provision is difficult as many countries individually face strong incentives to maintain a nuclear deterrent (despite the associated economic cost).

Single best effort
Weakest link
Aggregate effort
Mutual restraint
Coordination

Supply depends on…
The single best (unilateral or collective) effort
The weakest individual effort
The total effort of all countries
Countries not doing something
Countries doing the same thing

Examples
Asteroid defense, knowledge, peacekeeping, suppressing an infectious disease outbreak at its source, geoengineering
Disease eradication, preventing emergence of resistance and new diseases, securing nuclear materials, vessel reflagging
Climate change mitigation, ozone layer protection
Non-use of nuclear weapons, non-proliferation, bans on nuclear testing and biotechnology research
Standards for the measurement of time, for oil tankers, and for automobiles

International cooperation needed?
Yes, in many cases, to determine what should be done, and which countries should pay
Yes, to establish universal minimum standards
Yes, to determine the individual actions needed to achieve an overall outcome
Yes, to agree on what countries should not do
Yes, to choose a common standard

Financing and cost sharing needed?
Yes, when the good is provided collectively
Yes, in some cases
Yes, with industrialized countries helping developing countries
No
No

Enforcement of agreement challenging?
Not normally
Yes, except when provision requires only coordination
Yes
Yes
No, though participation will need to pass a threshold

International institutions for provision
Treaties in some cases; international organizations, such as the UN, in other cases
Consensus (World Health Assembly) or Security Council resolutions, customary law
Treaties
Treaties, norms, customary law
Non-binding resolutions; treaties in some cases

Table 1: Simple Taxonomy of Global Public Goods
Source: Scott Barrett (2010), Why Cooperate? The Incentive to Supply Global Public Goods (location 520 of Kindle edition)
Applying the Barrett framework to friendly AI
Artificial Intelligence (AI) technology is likely to progress until the eventual creation of AI that vastly surpasses human cognitive capabilities—artificial superintelligence (ASI). The possibility of an intelligence explosion means that the first ASI system, or those that control it, might possess an unprecedented ability to shape the world according to their preferences. This event could define our entire species, leading rapidly to the full realization of humanity’s potential or causing our extinction. Since “friendly AI”—safe ASI deployed for the benefit of humanity—is a global public good, it may be informative to apply Barrett’s global public good classification scheme to analyse the different facets of this challenge.
Since this framework focuses on the incentives faced by national governments, it is most relevant to situations where ASI development is largely driven by governments, which will therefore be the focus of this article. This government-led scenario is distinct from the current situation of technology industry-led development of AI. Governments might achieve this high level of control through large-scale state-sponsored projects and regulation of private activities.
As with many global public goods, the development of friendly AI can be broken down into many components, each of which may conform to a different category within Barrett’s taxonomy. Here I will focus on those that I believe are most important for long term safety.
Arguably, one of the most concerning problems in the government-led scenario is the potential for the benefits of ASI to be captured by some subset of humanity. Humans are unfortunately much more strongly motivated by self-interest than by the common good, and this is reflected in national and international politics. This means that, given the chance, leaders whose governments control the development of ASI might seek to capture the benefits for their country only, or some subset of their country such as their political allies, or other groups. This could be achieved by instilling values in the ASI system that favor such groups, or through the direct exertion of control over the ASI system. Protection against this possibility constitutes a “mutual restraint” public good, since its provision relies upon countries refraining from doing so. Failing to prevent this possibility may, depending on the preferences of those that control ASI, cause an existential catastrophe, for example in the form of “flawed realization” or “shriek”.
Because of this, and given the current anarchical state of international relations, any ASI-developing country is likely to be perceived as a significant security threat by other countries. Fears that any country succeeding at creating ASI would gain a large strategic advantage over other countries could readily lead to an ASI development race. In this scenario, speed may be prioritized at the expense of safety measures, for example those necessary to solve the value-loading problem (Ch. 12) and the control problem (Ch. 9). This would compound the risks of misuse of ASI explored in the previous paragraph by increasing the possibility of humanity losing control of this creation. The likelihood of an ASI development race is somewhat supported by Chalmers 2010 (footnote, p. 29).
Further, given that ASI may only be achievable on a timescale of decades, the global order prevailing when ASI is within reach may be truly multi-polar. For example, this timescale may allow both China and India to far surpass the USA in terms of economic weight, and may allow countries such as Brazil and Russia to rival the influence of the USA. With a diverse mix of world powers with differing national values, attempts at coordination and restraint could easily be undermined by mistrust.
Another facet of the global public good of friendly AI is the aforementioned technical challenges, including the value-loading problem and the control problem, which currently receive much attention in discussions of long-term AI safety. In isolation, these technical challenges can be considered a “single best effort” global public good in Barrett’s taxonomy, similar to asteroid defense or geoengineering, where it is often in the interests of some countries to unilaterally provide the good. Therefore, a substantial attempt would probably be made to solve these challenges in the government-led scenario, if race dynamics were not present. In reality, any additional advance work on this technical front is likely to be highly beneficial.
What can be done?
Without aiming to present a robust solution, this section briefly explores some of the available options, informed by insights presented by Barrett regarding mutual restraint global public goods.
A “silver bullet” solution to these institutional challenges could be achieved through the emergence of a world government capable of providing global public goods. Although this may eventually be possible, it seems unlikely within the timeframe in which ASI may be developed. Supporting progression towards this outcome may help to provide the global public goods identified above, but such action is probably insufficient alone.
In relation to mutual restraint public goods generally, Barrett identifies treaties, norms and customary law as institutional tools for provision. If a treaty requiring the necessary restraint could be enforced—Shulman mentions (p. 3) some ways in which one might be—it could be effective. However, this would still rely on countries’ willingness to voluntarily join the agreement.
Norms and custom can help achieve mutual restraint. In his book, Barrett analyses (location 2506 of Kindle edition) an important example: the taboo on the use of nuclear weapons. Thanks to strong aversion towards any destructive use of nuclear weapons, such use has not occurred since 1945. This has occurred despite numerous situations in which it would have been militarily advantageous to use a nuclear weapon, e.g. when a nuclear power was at war with a non-nuclear state. In the presence of such attitudes, any benefits to a country from using nuclear weaponry must be weighed against the costs of severe loss of international reputation, or in the extreme, the end of the taboo and consequent nuclear war.
The taboo on the use of nuclear weapons was not inevitable, but arose partly because of mutual understanding of the seriousness of the threat of nuclear war. If the potential effects of ASI are similarly well understood by all powers seeking to develop it, it is possible that a similar taboo could be created, perhaps with the help of a carefully designed treaty between those countries with meaningful ASI development capabilities. The purpose of such an arrangement would be not only to mandate the adoption of proper safety measures, but also to ensure that the benefits of ASI would be spread fairly amongst all of humanity.
Conclusion
To achieve positions of power, all political leaders depend heavily on their ability to amass resources and influence. Upon learning of the huge potential of ASI, such individuals may instinctively attempt to capture control of its power. They will also expect their rivals to do the same, and will strategize accordingly. Therefore, in the event of government-led ASI development, mutual restraint by ASI-developing nations would be needed to avoid attempts to capture the vast benefits of ASI for a small subset of humanity, and to avoid the harmful effects of a race to develop ASI.

Friendly AI as a global public good