Costs of extinction risk mitigation

We very roughly estimate that the annual cost of reducing the probability of human extinction by 0.01% is within the range of $1.1 billion to $3.5 trillion.
Introduction
This article is intended to be usable in a Cost-Benefit Analysis (CBA) analysis of extinction risk mitigation. It explores the costs of such efforts. A corresponding article (forthcoming) will explore the benefits. The focus on extinction risk, rather than the broader concept of existential risk, is due to the approach taken by the benefits article.
Ultimately the purpose of this exercise is to analyse the extinction risk arising from artificial superintelligence (ASI). However, the analysis does not take an ASI-specific approach. Although certain parts of the exercise will focus on this particular risk, the analysis is also more generally applicable to other sources of extinction risk.
Motivation
Although estimates of the probability of human extinction are highly subjective and uncertain, the probability of an existential disaster—a broader category—occurring before 2100 is typically estimated (p. 15) to be 10–20%. An earlier survey focusing only on extinction risk estimated a probability of 19%. Even if these estimates are highly mistaken, a probability of even 1% might be well worth mitigating.
CBA can inform decisions of whether to mitigate extinction risk, and provide estimates of how much expenditure can be justified. CBA is a widely used framework for assessing projects, policies, and decisions. By reducing complex issues to aggregable monetary figures, CBA "translates" difficult questions into a form useful to decision makers who may only possess surface knowledge of the relevant subject matter. This article may be useful to such policymakers who are faced with decisions about mitigating extinction risk.
Approach
It is hard to apply CBA to the risk of extinction, because it is a speculative and long-term issue. Of course, precision cannot be achieved. However, extinction risk mitigation efforts may offer benefits that exceed their costs by many orders of magnitude. This is due mainly to the literally astronomical future potential of humanity that would be destroyed in the event of proximate extinction. If benefits do radically exceed costs, then CBA can likely provide firm conclusions despite the huge uncertainty involved.
We investigate the costs of mitigating extinction risks from asteroids and climate change, as a first approximation of costs of extinction risk mitigation in general. We chose these two examples based on the availability of information. We use them to estimate the cost of reducing the probability of extinction by 0.01%.
We discuss numerous limitations to this approach later in the article.
Analysis: planetary defense against asteroid or comet impact
Perhaps one of the best-understood sources of extinction risk is the possibility of the Earth colliding with a sufficiently large Near-Earth Object (NEO)—an asteroid or a comet whose orbit brings it into proximity with Earth. Based on Table 1, small, relatively harmless NEOs are expected to impact Earth relatively frequently. However, a large (>10km diameter) NEO colliding with Earth would probably cause human extinction. This has an estimated  0.0001% chance of occurring during any given 100 year period. In response to this threat, many NEOs have been identified (p. 3), and efforts are underway to test our ability to deflect dangerous asteroids.

Type of Event
Characteristic Diameter
of Impacting Object (m)
Approximate Impact
Energy (MT)
Approximate Average Impact
Interval (years)

Airburst
25
1
200

Local scale
50
10
2,000

Regional scale
140
300
30,000

Continental scale
300
2,000
100,000

Below global catastrophe threshold
600
20,000
200,000

Possible  global catastrophe
1,000
100,000
700,000

Above global catastrophe threshold
5,000
10,000,000
30,000,000

Mass extinction
10,000
100,000,000
100,000,000

Table 1: Scale and frequency of NEO impact
Notes: In the above table, "mass extinction" includes human extinction, based on comments on p. 23 of the below source. The level of danger of any particular impacting object depends crucially on its size, but additionally on other factors, notably its composition, speed of impact, and location of impact.
Source: Defending Planet Earth: Near-Earth Object Surveys and Hazard Mitigation Strategies, 2010 (p. 19).
One estimate for planetary defense costs was given in the 2010 report Defending Planet Earth (pp. 97–99). The indicated cost of monitoring 90% of near-Earth objects of 140 meters in diameter or greater, and launching a test program to deflect an asteroid, was $250m per year. This level of expenditure was expected to be required for somewhat under a decade, and would reduce after the completion of the deflection test.
Thus for CBA purposes we might assume an indefinite cost of $250m per year. This could be combined with an assumption that these efforts would be 50% effective against NEOs threatening human extinction (as per Matheny 2007, p. 1340). The resulting cost per 0.01% of NEO-originating extinction probability eliminated is given in the final section of this article, alongside that from other sources.
Of course there are significant uncertainties in the above cost estimate. Monitoring costs may be greater or lower than expected, and it is not known whether or on how many occasions a NEO would need to be deflected during any time period. Further, it has been noted (p. 849, see footnote to table) that, for those NEOs large enough to cause human extinction, deflection may not be possible with current technology.
One source of overestimation in the above cost estimate is that successful planetary defense against NEOs does not only protect against extinction; it also averts lesser catastrophes and costs. Thus, in an ideal CBA analysis of extinction risk mitigation, not all associated expenses would be counted as costs of averting extinction. In particular, most expenditure is likely to be driven by the desire to avoid less destructive events that are much more frequent and visible.
Analysis: Climate change
The issue of human activity-induced climate change has received abundant global attention in recent years. Unfortunately, authoritative attention is most strongly focused (pp. 45–46) on measurements of the centre point of the range of possibilities, without considering the likelihood and potential impact of more extreme temperature increases. Perhaps because of this, the author has found little research on the risk of extinction posed by climate change.
Wagner and Weitzman, in their book Climate Shock, perform their own analysis (pp. 53–54) of the probability of global temperatures rising by at least 6°C, a threshold beyond which they believe disaster is nearly certain, of which the full implications cannot be known (although they estimate that the costs of allowing this threshold to be crossed may be equivalent to 10%-30% of global GDP). They estimate that the probability of reaching this threshold is 11% if carbon dioxide equivalent concentration (CO2eq) in the atmosphere is 700 parts per million (ppm), and 1.2% if the concentration is 500 ppm.
"Disaster" is not extinction. However, insofar as the 6°C-or-higher scenario is representative of extreme climate scenarios in general, it may be helpful for CBA purposes to estimate the cost of achieving a reduction of its probability from 11% to 1.2%. In this vein, Figure 1, taken from the Intergovernmental Panel on Climate Change's (IPCC) Fifth Assessment in 2014, summarises estimates of economic impacts (costs) of five CO2eq concentration scenarios. The medians of the various cost estimates for the 650–720 ppm and 480–530 ppm scenarios could be taken as indicative of the cost of reducing CO2eq from 700 ppm to 500 ppm. This suggests that this cost may be somewhere in the vicinity of 1–2% of the Net Present Value (NPV) of global GDP and consumption during the period 2015–2100. Thus, an approximate NPV of US $31.9tr–$63.8tr of GDP, or US $24.3tr–$48.5tr of consumption, may be foregone globally to reduce the probability of global temperatures rising by at least 6°C from 11% to 1.2%.1. For comparability purposes, annual costs can be estimated by assuming that these costs are evenly spread in real terms from 2015–2100. This produces estimates of annual costs of approximately US $1.5tr–$3.1tr of GDP, or approximately US $1.2tr–$2.3tr of consumption.

Figure 1: Mitigation costs (NPV 2015–2100, 5% discount rate)
Notes: Multiple models are used to produce Figure 1, and each model is used to run one or more scenarios. Each scenario result is represented by a dot, with the number of scenarios included in the boxplots indicated at the bottom of the panels. Costs are expressed as a fraction of economic output in the baseline, or in the case of consumption losses, consumption in the baseline. The number of scenarios outside the figure range is noted at the top.
Source: Mitigation of Climate Change: Working Group III contribution to the Fifth Assessment Report of the IPCC, Chapter 6, p. 450.
However, it should be borne in mind that a temperature increase of 6°C does not correspond to certain human extinction, and there may even be a small chance of extinction arising from below-6°C temperatures. Overall, the reduction in extinction risk is likely to be somewhat lower than 9.8%. For example, a collaborative effort led by the Global Challenges Foundation established (p. 142) highly approximate estimates of the probability of "infinite impact" (a concept closer to, but not the same as, extinction) from a range of sources. Their estimate of the probability of infinite impact from climate change was 0.01% over the next 200 years; much lower than the 11% probability above. If, by reducing CO2eq concentration from 700 ppm to 500 ppm, the 0.01% probability were reduced by the same proportion as the probability of a 6°C or higher temperature increase, then this would reduce the probability of infinite impact by 0.0088% (i.e. to 0.0012%). This assumption of proportionality is highly approximate.
Similarly to costs incurred in defending against NEOs, the cost of combating climate change is not wholly attributable to the mitigation of extinction risk, since it also mitigates many lesser risks. In particular as noted above, public dialogue and authoritative attention focuses on the more moderate temperature increases that are unlikely to cause human extinction. Since mitigation expenditure may be driven by this focus on lesser risks, a lesser portion of such costs would be included in an ideal CBA analysis of extinction risk mitigation.
This issue can be partially avoided by focusing on the technique of geoengineering (climate engineering) as a way to reduce climate-related risk. Geoengineering may align more closely to extreme climate-related risks, since many geoengineering methods are viewed (pp. 44–45) as a last resort given their problems and dangers. Climate Shock (p. 99) estimates that using geoengineering as a solution to climate change would cost $1b–$10b per year, a much smaller price to pay for removing (most of) the aforementioned 0.01% risk of infinite impact.
Although geoengineering is more likely to be used in the direst circumstances, its use may still be primarily motivated by problems less severe than extinction. Thus the issue of separating extinction mitigation costs from other costs is only reduced, rather than eliminated. Further, the extent to which geoengineering actually reduces the overall risk of human extinction is uncertain, since this technique carries (pp. 58–62) its own risks, particularly if it is executed without adequate quality of governance or before its implications are fully understood.
Sources of uncertainty in extrapolating this analysis to ASI-originating extinction risk
The approach taken in this article, while useful, is highly approximate. The approach is one of extrapolation, where the costs of the mitigation of one extinction risk are used as a proxy for the cost of mitigating ASI-originating extinction risk. This is based on the idea that the challenges of mitigating climate change- and NEO-originating extinction risk may share some characteristics with ASI-originating extinction risk, such as being unprecedented tasks involving sophisticated technology, often at a global scale. However, different sources of extinction risk are of course highly heterogeneous, and thus the monetary cost per unit of probability reduction for one source may not be indicative for another.
In particular, not all actions that mitigate extinction risk are costly from the perspective of society as a whole. For example, the mitigation of the risk of nuclear war through successful nuclear disarmament is actually associated with a reduction in expenditure on nuclear weapons. Similarly, there are likely to be aspects of ASI-originating extinction risk that can also be mitigated through mutual agreements to refrain from certain activities, such as racing to be the first to develop ASI regardless of the adequacy of safety precautions. However, there are also aspects of ASI risk that can indeed be mitigated through expenditure, such as research into and creation of safety mechanisms. It is to the latter type of mitigation effort that CBA is more readily applicable.
The selection of climate change mitigation and planetary defense for analysis in this article was influenced positively by the availability of data. This may lead to assessing relatively cheap risk mitigation measures, because information will be more available for measures that are being taken, which may be those that seem more worthwhile.
Another source of inaccuracy is the phenomenon of diminishing returns, a pattern whereby additional expenditure produces greater gains when total past expenditure is lower. A form of this is often observed in research, and can serve as a model for progress in solving problems of unknown difficulty. This implies that, for a source of extinction risk where relatively little effort or money has been expended such as ASI, a small amount of additional expenditure is likely to be able to achieve relatively large benefits. However, in this article, the estimated cost of reducing extinction risk through planetary defense and climate change mitigation are based on assessments of complete solutions, which include estimates of both the highly beneficial initial expenditure and the later, less beneficial expenditure. They may therefore overestimate the cost of achieving reductions through small increments to expenditure on mitigation of ASI-originating extinction risk.
Despite these numerous and large uncertainties, the overall CBA may still be capable of drawing reasonably firm conclusions, given that there is potential for the benefits of mitigation efforts to exceed their costs by many orders of magnitude.
Summary of results
In Table 2, the three potential mitigation activities considered in this article are compared, with the last column examining their cost effectiveness in reducing the probability of extinction by 0.01%. Only annual costs are presented in Table 2, since the calculation of total costs requires the use of a discount rate, the choice of which is contentious and will receive attention in a related article (forthcoming).

Mitigation activity
Cost per year, 2015–2100
Reduction in probability of extinction, 2015–2100
Scaled cost of achieving probability reduction of 0.01%^

NEOs: detect & deflect
$250m
0.000043%*
$58.1b

Climate change: CO2eq emission abatement
$1.2tr–$3.1tr**
0.008800%***
$1.3tr–$3.5tr**

Climate change: geoengineering
$1b–$10b
0.008800%***
$1.1b–$11.4b

Table 2: Estimates of annual costs of reducing the probability of human extinction
Notes: All cost figures in US dollars at constant prices. ^ Hypothetical, for comparison purposes. * Annual probability of one in 10^8, summed over the 2015–2100 period, combined with 50% success rate of deflection. ** Total of GDP and consumption ranges reported. *** For simplicity, it is assumed that all of the impact on the probability of infinite impact is attributable to costs borne in the period 2015–2100.
Please note the very approximate nature of these estimates.
Source: Author's calculations and estimates as set out in this article.
Table 2 hints at an interesting line of research that could be pursued; the use of a CBA framework to investigate the most cost-effective way to reduce the risk of human extinction. However, Table 2 is unfortunately too approximate to support firm conclusions on such questions, and the topic is outside the scope of this article.
The analysis in this article establishes very rough guidance as to the order of magnitude of the cost component of this CBA exercise. Specifically, we estimate that the annual cost of reducing the probability of human extinction by 0.01% is within the range of $1.1 billion to $3.5 trillion, with the endpoints of this range being highly approximate. In a related article (forthcoming) the benefit component is quantified and compared to these results.
Contributions
Research and writing were done by Michael Wulfsohn. Katja Grace did review and editing.
Footnotes

Costs of extinction risk mitigation


Returns to scale in research

When universities or university departments produce research outputs—such as published papers—they sometimes experience increasing returns to scale, sometimes constant returns to scale, and sometimes decreasing returns to scale. At the level of nations however, R&D tends to see increasing returns to scale. These results are preliminary.
Background
"Returns to scale" refers to the responsiveness of a process' outputs when all inputs (e.g. researcher hours, equipment) are increased by a certain proportion. If all outputs (e.g. published papers, citations, patents) increase by that same proportion, the process is said to exhibit constant returns to scale. Increasing returns to scale and decreasing returns to scale refer to situations where outputs still increase, but by a higher or lower proportion, respectively.
Assessing returns to scale in research may be useful in predicting certain aspects of the development of artificial intelligence, in particular the dynamics of an intelligence explosion.
Results
The conclusions in this article are drawn from an incomplete review of academic literature assessing research efficiency, presented in Table 1. These papers assess research in terms of its direct outputs such as published papers, citations, and patents. The broader effects of the research are not considered.
Most of the papers listed below use the Data Envelopment Analysis (DEA) technique, which is a quantitative technique commonly used to assess the efficiency of universities and research activities. It is capable of isolating the scale efficiency of the individual departments, universities or countries being studied.

Paper
Level of comparison
Activities assessed
Results pertaining to returns to scale

Wang & Huang 2007
Countries' overall R&D activities
Research
Increasing returns to scale in research are exhibited by more than two-thirds of the sample

Kocher, Luptacik & Sutter 2006
Countries' R&D in economics
Research
Increasing returns to scale are found in all countries in the sample except the US

Cherchye & Abeele 2005
Dutch universities' research in Economics and Business Management
Research
Returns to scale vary between decreasing, constant and increasing depending on each university's specialization

Johnes & Johnes 1993
UK universities' research in economics
Research
Constant returns to scale are found in the sample as a whole

Avkiran 2001
Australian universities
Research, education
Constant returns to scale found in most sampled universities

Ahn 1988
US universities
Research, education
Decreasing returns to scale on average

Johnes 2006
English universities
Research, education
Close to constant returns to scale exhibited by most universities sampled

Kao & Hung 2008
Departments of a Taiwanese university
Research, education
Increasing returns to scale exhibited by the five most scale-inefficient departments. However, no aggregate measure of returns to scale within the sample is presented.

Table 1: Sample of studies of research efficiency that assess returns to scale
Note: This table only identifies increasing/constant/decreasing returns to scale, rather than the size of this effect. Although DEA can measure the relative size of the effect for individual departments/universities/countries within a sample, such results cannot be readily compared between samples/studies.
Discussion of results
Of the studies listed in Table 1, the first four are the most relevant to this article, since they focus solely on research inputs and outputs. While the remaining four include educational inputs and outputs, they can still yield worthwhile insights.
Table 1 implies a difference between country-level and university-level returns to scale in research.

The two studies assessing R&D efficiency at the country level, Wang & Huang (2007) and Kocher, Luptacik & Sutter (2006), both identify increasing returns to scale.
The two university-level studies that assessed the scale efficiency of research alone found mixed results. Concretely, Johnes & Johnes (1993) concluded that returns to scale are constant among UK universities, and Cherchye & Abeele (2005) concluded that returns to scale vary among Dutch universities. This ambiguity is echoed by the remainder of the studies listed above, which assess education and research simultaneously and which find evidence of constant, decreasing and increasing returns to scale in different contexts.

Such differences are consistent with the possibility that scale efficiency may be influenced by scale (size) itself. In this framework, as an organisation increases its size, it may experience increasing returns to scale initially, resulting in increased efficiency. However, the efficiency gains from growth may not continue indefinitely; after passing a certain threshold the organisation may experience decreasing returns to scale. The threshold would represent the point of scale efficiency, at which returns to scale are constant and efficiency is maximized with respect to size.
Under this framework, size will influence whether increasing, constant or decreasing returns to scale are experienced. Applying this to research activities, the observation of different returns to scale between country-level and university-level research may mean that the size of a country's overall research effort and the typical size of its universities are not determined by similar factors. For example, if increasing returns to scale at the country level and decreasing returns to scale at the university level are observed, this may indicate that the overall number of universities is smaller than needed to achieve scale efficiency, but that most of these universities are individually too large to be scale efficient.
Other factors may also contribute to the differences between university-level and country-level observations.

The country level studies use relatively aggregated data, capturing some of the non-university research and development activities in the countries sampled.
Country level research effort is not necessarily subject to some of the constraints which may cause decreasing returns to scale in large universities, such as excessive bureaucracy.
Results may be arbitrarily influenced by differences in the available input and output metrics at the university versus country level.

Limitations to conclusions drawn
One limitation of this article is the small scope of the literature review. A more comprehensive review may reveal a different range of conclusions.
Another limitation is that the research outputs studied—published papers, citations, and patents, inter alia—cannot be assumed to correspond directly to incremental knowledge or productivity. This point is expanded upon under "Topics for further investigation" below.
Further limitations arise due to the DEA technique used by most of the studies in Table 1.

DEA is sensitive to the choice of inputs and outputs, and to measurement errors.
Statistical hypothesis tests are difficult within the DEA framework, making it more difficult to separate signal from noise in interpreting results.
DEA identifies relative efficiency (composed of scale efficiency and also "pure technical efficiency") within the sample, meaning that at least one country, university, or department is always identified as fully efficient (including exhibiting full scale efficiency or constant returns to scale). Of course, in practice, no university, organisation or production process is perfectly efficient. Therefore, conclusions drawn from DEA analysis are likely to be more informative for countries, universities, or departments that are not identified as fully efficient.
It may be questionable whether such a framework—where an optimal scale of production exists, past which decreasing returns to scale are experienced—is a good reflection of the dynamics of research activities. However, the frequent use of the DEA framework in assessing research activities would suggest that it is appropriate.

Topics for further investigation
The scope of this article is limited to direct research outputs (such as published papers, citations, and patents). While this is valuable, stronger conclusions could be drawn if this analysis were combined with further work investigating the following:

The impact of other sources of new knowledge apart from universities or official R&D expenditure. For example, innovations in company management discovered through "learning by doing" rather than through formal research may be an important source of improvement in economic productivity.
The translation of research outputs (such as published papers, citations, and patents) into incremental knowledge, and the translation of incremental knowledge into extra productive capacity. Assessment of this may be achievable through consideration of the economic returns to research, or of the value of patents generated by research.

Implications for AI
The scope for an intelligence explosion is likely to be greater if the returns to scale in research are greater. In particular, an AI system capable of conducting research into the improvement of AI may be able to be scaled up faster and more cheaply than the training of human researchers, for example through deployment on additional hardware. In addition, in the period before any intelligence explosion, a scaling-up of AI research may be observed, especially if the resultant technology were seen to have commercial applications.
This review is one component of a larger project to quantitatively model an intelligence explosion. This project, in addition to drawing upon the conclusions in this article, will also consider inter alia the effect of intelligence on research productivity, and actual increases in artificial intelligence that are plausible from research efforts.

Returns to scale in research


Selected Citations

This page is a non-exhaustive collection of places where AI Impacts' work has been cited.
AI Timelines

Muehlhauser, Luke. 2015. "What Do We Know about AI Timelines?" Open Philanthropy Project. (archive)
Muehlhauser, Luke. 2015. "What should we learn from past AI forecasts?" Open Philanthropy Project. (archive)
Brundage, Miles. 2016. "Modeling Progress in AI". arXiv:1512.05849. arXiv.
McClusky, Peter. 2016. "AGI Timelines". Bayesian Investor Blog. (archive)

Computing Power and the Brain

Hsu, Jeremy. 2016. "Estimate: Human Brain 30 Times Faster than Best Supercomputers". IEEE Spectrum. (archive)

Cost of Computing

Hoffman, Ben. 2016. "Paths to singleton: a hierarchical conceptual framework". Ben Models the World. (archive)

Mathematical Conjectures

Future of Life Institute. 2015. "A survey of research questions for robust and beneficial AI". The Future of Life Institute.

Expert Predictions

Hanson, Robin. 2016. "Age of Em". Oxford University Press.

Discontinuous Technological Progress Bounties

Hsu, Jeremy. 2016. "Making Sure AI's Rapid Rise Is No Surprise". Discover Blogs: Lovesick Cyborg. (archive)

Selected Citations


Error in Armstrong and Sotala 2012

Can AI researchers say anything useful about when strong AI will arrive?
Back in 2012, Stuart Armstrong and Kaj Sotala weighed in on this question in a paper called 'How We're Predicting AI—or Failing To'. They looked at a dataset of predictions about AI timelines, and concluded that predictions made by AI experts were indistinguishable from those of non-experts. (Which might suggest that AI researchers don't have additional information).
As far as I can tell—and Armstrong and Sotala agree—this finding is based on an error. Not a fundamental philosophical error, but a spreadsheet construction and interpretation error.
The main clue that there has been a mistake is that their finding is about experts and non-experts, and their public dataset does not contain any division of people into experts and non-experts. (Hooray for publishing data!)
As far as we can tell, the column that was interpreted as 'is this person an expert?' was one of eight tracking 'by what process did this person arrive at a prediction?' The possible answers are 'outside view', 'noncausal model', 'causal model', 'philosophical argument', 'expert authority', 'non-expert authority', 'restatement' and 'unclear'.
Based on comments and context, 'expert authority' appears to mean here that either the person who made the prediction is an expert who consulted their own intuition on something without providing further justification, or that the predictor is a non-expert who used expert judgments to inform their opinion. So the predictions not labeled 'expert authority' are a mixture of predictions made by experts using something other than their intuition—e.g. models and arguments—and predictions made by non-experts which are based on anything other than reference to experts. Plus restatements and unclarity that don't involve any known expert intuition.
The reasons to think that the 'expert authority' column was misintepreted as an 'expert' column are A) that there doesn't seem to be any other plausible expert column, B) that the number of predictions labeled with 'expert authority' is 62, the same as the number of experts Armstrong and Sotala claimed to have compared (and the rest of the set is 33, the number of non-experts they report), and C) Sotala suggests this is what must have happened.
How bad a problem is this? How badly does using unexplained expert opinion as a basis for prediction align with actually being an expert?
Even without knowing exactly what an expert is, we can tell the two aren't all that well aligned because Armstrong and Sotala's dataset contains many duplicates: multiple records of the same person making predictions in different places. All of these people appear at least twice, at least once relying 'expert authority' and at least once not: Rodney Brooks, Ray Kurzweil, Jürgen Schmidhuber, I. J. Good, Hans Moravec. It is less surprising that experts and non-experts have similar predictions when they are literally the same people! But multiple entries of the same people listed as experts and non-experts only accounts for a little over 10% of their data, so this is not the main thing going on.
I haven't checked the data carefully and assessed people's expertise, but here are other names that look to me like they fall in the wrong buckets if we intend 'expert' to mean something like 'works/ed in the field of artificial intelligence': Ben Goertzel (not 'expert authority'), Marcus Hutter (not 'expert authority'), Nick Bostrom ('expert authority'), Kevin Warwick (not 'expert authority'), Brad Darrach ('expert authority').
Expertise and 'expert authority' seem to be fairly related (there are only about 10 obviously dubious entries, out of 95—though 30 are dubious for other reasons), but not enough to take the erroneous result as much of a sign about experts I think.
On the other hand, it seems Armstrong and Sotala have a result they did not intend: predictions based on expert authority look much like those not based on expert authority. Which sounds interesting, though given the context is probably not surprising: whether someone cites reasons with their prediction is probably fairly random, as indicated by several people basing their predictions on expert authority half of the time. e.g. Whether Kurzweil mentions hardware extrapolation on a given occasion doesn't vary his prediction much. A worse problem is that the actual categorization is 'most non-experts' + 'experts who give reasons for their judgments' vs. 'experts who don't mention reasons' + 'non-experts who listen to experts', which is pretty random, and so hard to draw useful conclusions from.
We don't have time right now to repeat this analysis after actually classifying people as experts or not, even though it looks straightforward. We delayed some in the hope of doing that, but it looks like we won't get to it soon, and it seems best to publish this post sooner to avoid anyone relying on the erroneous finding.
In the meantime, here is our graph again of predictions from AI researchers, AGI researchers, futurists and other people—the best proxy we have of 'expert vs. non-expert'. We think they look fairly different, though they are from the same dataset that Armstrong and Sotala used (though an edited version).
Predictions made by different groups since 2000 from the MIRI AI predictions dataset.

Stuart Armstrong adds the following analysis, in the style of the graphs on figure 18 of their paper that it could replace:

Also please forgive the colour hideousness of the following graph:
.

.
Here I did a bar chart of "time to AI after" for the four groups (and for all of them together), in 5-year bar increments (the last bar has all the 75 year+ predictions, not just 75-80). The data is incredibly sparse, but a few patterns do emerge: AGI are optimistic (and pretty similar to futurists), Others are pessimistic.
.
However, to within the limits of the data, I'd say that all groups (apart from "other") still have a clear tendency to predict 10-25 years in the future more often than other dates. Here's the % predictions in 10-25 years, and over 75 years:
.

%10-25
%>75

54%
8%
 AGI

27%
23%
 AI

47%
20%
 Futurist

13%
50%
 Other

36%
22%

Error in Armstrong and Sotala 2012


Metasurvey: predict the predictors

As I mentioned earlier, we've been making a survey for AI researchers.
The survey asks when AI will be able to do things like build a lego kit according to the instructions, be a surgeon, or radically accelerate global technological development. It also asks about things like intelligence explosions, safety research, how hardware hastens AI progress, and what kinds of disagreement AI researchers have with each other about timelines.
We wanted to tell you more about the project before actually surveying people, to make criticism more fruitful. However it turned out that we wanted to start sending out the survey soon even more than that, so we did. We did get an abundance of private feedback, including from readers of this blog, for which we are grateful.
We have some responses so far, and still have about a thousand people to ask. Before anyone (else) sees the results though, I thought it might be amusing to guess what they look like. That way, you can know whether you should be surprised when you see the results, and we can know more about whether running surveys like this might actually change anyone's beliefs about anything.
So we made a second copy of the survey to act as metasurvey, in which you can informally register your predictions.
If you want to play, here is how it works:

Go to the survey here.
Instead of answering the questions as they are posed, guess what the median answer given by our respondents is for each question.
If you want to guess something other than the median given by our other respondents, do so, then write what you are predicting in the box for comments at the end. (e.g. maybe you want to predict the mode, or the interquartile range, or what the subset of respondents who are actually AI researchers say).
If you want your predictions to be identifiable to you, give us your name and email at the end.  This will for instance let us alert you if we notice that you are surprisingly excellent at predicting. We won't make names or emails public.
At the end, you should be redirected to a printout of your answers, which you can save somewhere if you want to be able to demonstrate later how right you were about stuff. There is a tiny pdf export button in the top right corner.
You will only get a random subset of questions to predict, because that's how the survey works. If you want to make more predictions, the printout has all of the questions.
We might publish the data or summaries of it, other than names and email addresses, in what we think is an unidentifiable form.

Some facts about the respondents, to help predict them:

They are NIPS 2015/ICML 2015 authors (so a decent fraction are not AI researchers)
There are about 1600 of them, before we exclude people who don't have real email addresses etc.

John Salvatier points out to me that the Philpapers survey did something like this (I think more formally). It appears to have been interesting—they find that 'philosophers have substantially inaccurate sociological beliefs about the views of their peers', and that 'In four cases [of thirty], the community gets the leading view wrong…In three cases, the community predicts a fairly close result when in fact a large majority supports the leading view'. If it turned out that people thinking about the future of AI were that wrong about the AI community's views, I think that would be good to know about.

Featured image: By DeFacto (Own work) [CC BY-SA 4.0], via Wikimedia Commons

Metasurvey: predict the predictors


Concrete AI tasks bleg

We're making a survey. I hope to write soon about our general methods and plans, so anyone kind enough to criticize them has the chance. Before that though, we have a different request: we want a list of concrete tasks that AI can't do yet, but may achieve sometime between now and surpassing humans at everything. For instance, 'beat a top human Go player in a five game match' would have been a good example until recently. We are going to ask AI researchers to predict a subset of these tasks, to better chart the murky path ahead.
We hope to:

Include tasks from across the range of AI subfields
Include tasks from across the range of time (i.e. some things we can nearly do, some things that are really hard)
Have the tasks relate relatively closely to narrowish AI projects, to make them easier to think about (e.g. winning a 5k bipedal race is fairly close to existing projects, whereas winning an interpretive dance-off would require a broader mixture of skills, so is less good for our purposes)
Have the tasks relate to specific hard technical problems (e.g. one-shot learning or hierarchical planning)
Have the tasks relate to large changes in the world (e.g. replacing all drivers would viscerally change things)

Here are some that we have:

Win a 5km race over rough terrain against the best human 5k runner.
Physically assemble any LEGO set given the pieces and instructions.
Be capable of winning an International Mathematics Olympiad Gold Medal (ignoring entry requirements). That is, solve mathematics problems with known solutions that are hard for the best high school students in the world, better than those students can solve them.
Watch a human play any computer game a small number of times (say 5), then perform as well as human novices at the game without training more on the game. (The system can train on other games).
Beat the best human players at Starcraft, with a human-like limit on moves per second.
Translate a new language using unlimited films with subtitles in the new language, but the kind of training data we have now for other languages (e.g. same text in two languages for many languages and films with subtitles in many languages).
Be about as good as unskilled human translation for most popular languages (including difficult languages like Czech, Chinese and Arabic).
Answer tech support questions as well as humans can.
Train to do image classification on half a dataset (say, ImageNet) then take the other half of the images, containing previously unseen objects, and separate them into the correct groupings (without the correct labels of course).
See a small number of examples of a new object (say 10), then be able to recognize it in novel scenes as well as humans can.
Reconstruct a 3d scene from a 2d image as reliably as a human can.
Transcribe human speech with a variety of accents in a quiet environment as well as humans can.
Routinely and autonomously prove mathematical theorems that are publishable in mathematics journals today.

Can you think of any interesting ones?

Concrete AI tasks bleg


Mysteries of global hardware

This blog post summarizes recent research on our Global Computing Capacity page. See that page for full citations and detailed reasoning.
We recently investigated this intriguing puzzle:

FLOPS (then) apparently performed by all of the world's computing hardware: 3 x 1022 – 3 x 1024

(Support: Vipul Naik estimated 1022 – 1024 IPS by February 2014, and reports a long term growth rate of 85% per year for application-specific computing hardware, which made up 97% of hardware by 2007, suggesting total hardware should have tripled by now. FLOPS are roughly equivalent to IPS).

Price of FLOPS: $3 x 10-9

(Support: See our page on it)

Implied value of global hardware: $<PHONE>

(Support: 3 x 1022 to 3 x 1024 * $3 x 10-9 = $<PHONE> )

Estimated total global wealth: $2.5 * 1014

(Support: see for instance Credit Suisse)

Implication: 40%-4,000% of global wealth is in the form of computing hardware.

Question: What went wrong?

Clues
Could most hardware be in large-scale, unusually cheap, projects? Probably not – our hardware price figures include supercomputing prices. Also, Titan is a supercomputer made from GPUs and CPUs, and doesn't seem to be cheaper per computation than the component GPUs and CPUs.
Could the global wealth figure be off? We get roughly the same anomaly when comparing global GDP figures and the value of computation used annually.
Our solution
We think the estimate of global hardware is the source of the anomaly. We think this because the amount that people apparently spend on hardware each year doesn't seem like it would buy nearly this much hardware.
Annual hardware revenue seems to be around $300bn-$1,500bn recently.1 Based on the prices of FLOPS, (and making some assumptions, e.g. about how long hardware lasts) this suggests the total global stock of hardware can perform around 7.5 x 1019 – 1.5 x 1021 FLOPS/year.2 However the lower end of this range is below a relatively detailed estimate of global hardware made in 2007. It seems unlikely that the hardware base actually shrunk in recent years, so we push our estimate up to 2 x 1020 – 1.5 x 1021 FLOPS/year.
This is about 0.3%-1.9% of global GDP—a more plausible number, we think—so resolves the original problem. But a big reason Naik gave such high estimates for global hardware was that the last time someone measured it—between 1986 and 2007—computing hardware was growing very fast. General purpose computing was growing at 61% per year, and the application specific computers studied (such as GPUs) were growing at 86% per year. Application specific computers made up the vast majority too, so we might expect growth to progress at close to 86% per year.
However if global hardware is as low as we estimate, the growth rate of total computing hardware since 2007 has been 25% or less, much lower than in the previous 21 years. Which would present us with another puzzle: what happened?
We aren't sure, but this is still our best guess for the solution to the original puzzle. Hopefully we will have time to look into this puzzle too, but for now I'll leave interested readers to speculate.

Added March 11 2016: Assuming the 2007 hardware figures are right, how much of the world's wealth was in hardware in 2007? Back then, GWP was probably about $66T (in 2007 dollars). According to Hilbert & Lopez, the world could then perform 2 x 1020 IPS, which is  2 x 1014 MIPS. According to Muehlhauser & Rieber, hardware cost roughly $5 x 10-3/MIPS in 2007. Thus the total value of hardware would have been around $5 x 10-3/MIPS x 2 x 1014 MIPS = $1012 (a trillion dollars), or 1.5% of GWP.

Titan Supercomputer. By an employee of the Oak Ridge National Laboratory.

Mysteries of global hardware


Global computing capacity

[This page is out of date and its contents may have been inaccurate in 2015, in light of new information that we are yet to integrate.]
Computing capacity worldwide was probably around 2 x 1020 – 1.5 x 1021 FLOPS, at around the end of 2015.
Support
We are not aware of recent, plausible estimates for hardware capacity.
Vipul Naik estimated global hardware capacity in February 2014, based on Hilbert & Lopez's estimates for <PHONE>. He calculated that if all computers ran at full capacity, they would perform 10-1000 zettaFLOPS, i.e. 1022 – 1024 FLOPS.1 We think these are substantial overestimates, because producing so much computing hardware would cost more than 10% of gross world product (GWP), which is implausibly high. The most cost-efficient computing hardware we are aware of today are GPUs, which still cost about $3/GFLOPS, or $1/GFLOPSyear if we assume hardware is used for around three years. This means maintaining hardware capable of 1022 – 1024 FLOPS would cost at least $1013 – $1015  per year. Yet gross world product (GWP) is only around $8 x 1013, so this would imply hardware spending constitutes more than 13% – 1300% of GWP. Even the lower end of this range seems implausible.2
One way to estimate global hardware capacity ourselves is based on annual hardware spending. This is slightly complicated because hardware lasts for several years. So to calculate how much hardware exists in 2016, we would ideally like to know how much was bought in every preceding year, and also how much of each annual hardware purchase has already been discarded. To simplify matters, we will instead assume that hardware lasts for around three years.
It appears that very roughly $300bn-$1,500bn was spent on hardware in 2015.3 We previously estimated that the cheapest available hardware (in April 2015) was around $3/GFLOPS. So if humanity spent $300bn-$1,500bn on hardware in 2015, and it was mostly the cheapest hardware, then the hardware we bought should perform around 1020 – 5 x 1020 FLOPS. If we multiply this by three to account for the previous two years' hardware purchases still being around, we have about  3 x 1020 – 1.5 x 1021 FLOPS.
This estimate is rough, and could be improved in several ways. Most likely, more hardware is being bought each year than the previous year. So approximating last years' hardware purchase to this years' will yield too much hardware. In particular, the faster global hardware is growing, the closer the total is to whatever humanity bought this year (that is, counterintuitively, if you think hardware is growing faster, you should suppose that there is less of it by this particular method of estimation). Furthermore, perhaps a lot of hardware is not the cheapest for various reasons. This too suggests there is less hardware than we estimated.
On the other hand, hardware may often last for more than three years (we don't have a strong basis for our assumption there). And our prices are from early 2015, so hardware is likely somewhat cheaper now (in early 2016). Our guess is that overall these considerations mean our estimate should be lower, but probably by less than a factor of four in total. This suggests 7.5 x 1019 – 1.5 x 1021 FLOPS of hardware.
However Hilbert & Lopez (2012) estimated that in 2007 the world's computing capacity was around 2 x 1020 IPS (similar to FLOPS) already, after constructing a detailed inventory of technologies.4 Their estimate does not appear to conflict with data about the global economy at the time.5 Growth is unlikely to have been negative since 2007, though Hilbert & Lopez may have overestimated. So we revise our estimate to 2 x 1020 – 1.5 x 1021 FLOPS for the end of 2015.
This still suggests that in the last nine years, the world's hardware has grown by a factor of 1-7.5, implying a growth rate of 0%-25%. Even 25% would be quite low compared to growth rates between 1986 and 2007 according to Hilbert & Lopez (2012), which were 61% for general purpose computing and 86% for the set of ASICs they studied (which in 2007 accounted for 32 times as much computing as general purpose computers).6 However if we are to distrust estimates which imply hardware is a large fraction of GWP, then we must expect hardware growth has slowed substantially in recent years. For comparison, our estimates are around 2-15% of Naik's lower bound, and suggest that hardware constitutes around 0.3%-1.9% of GWP.
Such large changes in the long run growth rate are surprising to us, and—if they are real—we are unsure what produced them. One possibility is that hardware prices have stopped falling so fast (i.e. Moore's Law is ending for the price of computation). Another is that spending on hardware decreased for some reason, for instance because people stopped enjoying large returns from additional hardware. We think this question deserves further research.
Implications
Global computing capacity in terms of human brains
According to different estimates, the human brain performs the equivalent of between 3 x 1013 and 1025 FLOPS. The median estimate we know of is 1018 FLOPS. According to that median estimate and our estimate of global computing hardware, if the world's entire computing capacity could be directed at running minds around as efficient as those of humans, we would have the equivalent of <PHONE> extra human minds.7 That is, turning all of the world's hardware into human-efficiency minds at present would increase the world's population of minds by at most about 0.00002%. If we select the most favorable set of estimates for producing large numbers, turning all of the world's computing hardware into minds as efficient as humans' would produce around 50 million extra minds, increasing the world's effective population by about 1%.8
Figure: Projected number of human brains equivalent to global hardware under various assumptions. For brains, 'small' = 3 x 10^ 13, 'median' = 10^18, 'large' = 10^25. For 'world hardware', 'high' =2 x 10^20, 'low' = 1.5 x 10^21. 'Growth' is growth in computing hardware, the unlabeled default used in most projections is 25% per annum (our estimate above), 'high' = 86% per annum (the apparent growth rate in ASIC hardware in around 2007).

Global computing capacity


Coordinated human action as example of superhuman intelligence

Collections of humans organized into groups and institutions provide many historical examples of the creation and attempted control of intelligences that routinely outperform individual humans. A preliminary look at the available evidence suggests that individuals are often cognitively outperformed in head-to-head competition with groups of similar average intelligence. This article surveys considerations relevant to the topic and lays out what a plausible research agenda in this area might look like.
Background
Humans are often organized into groups in order to perform tasks beyond the abilities of any single human in the group. Many such groups perform cognitive tasks. The history of forming such groups is long and varied, and provides some evidence about what new forms of superhuman intelligence might be like.
Some examples of humans cooperating on a cognitive task that no one member could perform include:

Ten therapists can see ten times as many patients as one therapist can.
A hospital can perform many more kinds of medical procedure and treat many more kinds of illness than any one person in the hospital.
A team of friends on trivia night might be able to answer more questions than any one of them individually might.

How such institutions are formed, and the sensitivity of their behavior to starting conditions, may help us predict the behavior of similarly constituted AIs or systems of AIs. This information would be especially useful if control or value alignment problems have been solved in some cases, or to the extent that existing human institutions resemble superintelligences or constitute an intelligence explosion.
There are several reasons these kinds of groups may present only a limited analogy to digital artificial intelligence. For instance, humans have no software-hardware distinction, so physical measures such as fences that can control the spread of humans are not likely to be as reliable at controlling the spread of digital intelligences. An individual human cannot easily be separated into different cognitive modules, which limits the design flexibility of intelligences constructed from humans. More generally, AIs may be programmed in ways very different from the heuristics and algorithms executed by the human brain, so while human organizations may be a kind of superhuman intelligence, they are not necessarily representative of the broader space of possible superintelligences.
Questions for further investigation:

Do any human organizations have the characteristics of superintelligences that some AI researchers and futurists expect to cause an intelligence explosion with catastrophic consequences? If so, do we expect catastrophe from human organizations? If not, what distinguishes them from other, potential artificial intelligences?
How similar is the problem of controlling institutional behavior to the value alignment problem with respect to powerful digital AIs? Are the expected consequences similar?
Do control mechanisms require limiting the cognitive performance of groups, or are there control mechanisms that do not appear to degrade in effectiveness as the intelligence of the group increases?
How relevant are the differences between human collective intelligence and digital AI?

Group vs individual performance
Institutions are mainly relevant as an example of constructed intelligence if their intelligence is higher than that of humans, in some sense. This section examines reasons to believe this might be the case.
Mechanisms for cognitive superiority of groups
We can think of several mechanisms by which a group might outperform individual humans on cognitive tasks, although this list is not comprehensive:

Aggregation – A large number of people can often perform cognitive tasks at a higher rate than a single person performing the same tasks. For example, a large accounting firm ought to be able to perform more audits, or prepare more tax returns, than a single accountant. In practice, there are often impediments to work scaling linearly with the number of people involved, as noted in observations such as Parkinson's Law.
Cognitive economies of scale

It is often less costly to teach someone how to perform a task than for them to figure it out on their own. Knowledge transfer between members of a group may therefore accelerate the learning process.
Individuals with different skills can cooperate to produce things or quantities of things that no one person could have produced, through specialization and gains from trade. For example, I, Pencil describes the large number of processes, each requiring a very different set of skills and procedures that it would take a long time to learn, to produce a single pencil.

Model combination and adjustment

In groups solving problems, people can make different suggestions and identify one another's incorrect suggestions, which may help the group avoid wasting time on blind alleys or adopting premature, incorrect solutions.
The average of the individual estimates from a group of people is typically more reliably accurate than the estimate of any individual in the group, because random errors tend to cancel each other out. This is often called the "wisdom of crowds".
Groups of people can also coordinate by comparing predictions and accepting the claim the group finds most credible. Trivia teams typically use this strategy. Groups of people have also been pitted against individuals in chess games.
Markets can be used to combine information from many individuals.

Further investigation on this topic could include:

Generating a more comprehensive list of potential mechanisms by which institutions and groups may have a cognitive advantage, by examining the historical record, arguments, and experimental and case studies of individual vs group performance.
Assessing which mechanisms can be shown to work, and how much group intelligence can exceed individual intelligence, by evaluating historical examples, case studies, and experimental studies.
Assessing in which aspects of intelligence, if any, groups have not outperformed individuals.

Evidence of cognitive superiority of groups
An incomplete survey of literature on collective intelligence found several measures where group performance, distinct from individual performance, has been explicitly evaluated:

Wooley et al. 2010 examined the performance of groups on tasks such as solving visual puzzles, brainstorming, making collective moral judgments, negotiating over limited resources, and playing checkers against a standardized computer opponent. The study found correlation between performance on different tasks, related more to the ability of members to coordinate than to the average or maximum intelligence of group members.
Shaw 1932 compared the timed performance of individuals and four-person groups on simple spatial and logical reasoning problems, and verbal tasks (arranging a set of words to form the end of some text). The study found that on problems where anyone was able to solve them, groups substantially outperformed individuals, mostly by succeeding much more often than individuals did. No one was able to solve the last two problems, but the study did find that on those problems, suggestions rejected during the process of group problem-solving were predominantly incorrect suggestions rejected by someone other than the person who proposed them, which shows error-correction to be potentially an important part of the advantage of group cognition.
Thorndike 1938 compared group and individual performance on vocabulary completion, limerick completion, and solving and making cross-word puzzle tests. Groups outperformed individuals on everything except making crossword puzzles.
Taylor and Faust 1952 tested the ability of individuals, groups of two, and groups of four, to solve "twenty questions" style problems. Groups outperformed individuals, but larger groups did not outperform smaller groups.
Gurnee 1936 compared individual and group performance at maze learning. Groups completed mazes faster and with fewer false moves.
Gordon 1924 compared individual estimates of an object's weight with the average of members of a group. The study found that group averages outperformed individual estimates, and that larger groups performed better than smaller groups.
McHaney et al. 2015 compared the performance of individuals, ad hoc groups, and groups with a prior history of working together, at detecting deception. The study found that groups with a prior history of working together outperform ad hoc groups, and refers to earlier literature that found no difference between the performance of individuals and that of ad hoc groups.

Mostly these studies appear to show groups outperforming individuals. We also found review articles referencing tens of other studies. We may follow up with a more comprehensive review of the evidence in this area in the future.
Questions for further investigation:

Which of the possible mechanisms for cognitive superiority of groups do human institutions demonstrate in practice? Do they have important advantages other than the ones enumerated?
In what contexts has the difference between group and individual performance been measured? Are there measures on which large organizations do much better than a single human? On what kinds of tasks does group performance most exceed that of individuals? How are these groups constituted?
Are there measures on which large organizations cannot be arbitrarily better than a single human? (These might still be things that an AI could do much better, and so where organizations are not a good analogue.) Are there measures for which large organizations have not yet even reached human level intelligence? (It is deprecatory to say something was "written by a committee.")


Recently at AI Impacts

We've been working on a few longer term projects lately, so here's an update in the absence of regular page additions.
New researchers
Stephanie Zolayvar and John Salvatier have recently joined us, to try out research here.
Stephanie recently moved to Berkeley from Seattle, where she was a software engineer at Google. She is making sense of a recent spate of interviews with AI researchers (more below), and investigating purported instances of discontinuous progress. She also just made this glossary of AI risk terminology.
John also recently moved to Berkeley from Seattle, where he was a software engineer at Amazon. He has been interviewing AI researchers with me, helping to design a new survey on AI progress, and evaluating different research avenues.
I've also been working on several smaller scale collaborative projects with other researchers.
AI progress survey
We are making a survey, to help us ask AI researchers about AI progress and timelines. We hope to get answers that are less ambiguous and more current than past timelines surveys. We also hope to learn about the landscape of progress in more detail than we have, to help guide our research.
AI researcher interviews
We have been having in-depth conversations with AI researchers about AI progress and predictions of the future. This is partly to inform the survey, but mostly because there are lots of questions where we want elaborate answers from at least one person, instead of hearing everybody's one word answers to potentially misunderstood questions. We plan to put up notes on these conversations soon.
Bounty submissions
Ten people have submitted many more entries to our bounty experiment. We are investigating these, but are yet to verify that any of them deserve a bounty. Our request was for examples of discontinuous progress, or very early action on a risk. So far the more lucrative former question has been substantially more popular.
Glossary
We just put up a glossary of AI safety terms. Having words for things often helps in thinking about them, so we hope to help in the establishment of words for things. If you notice important words without entries, or concepts without words, please send them our way.

Recently at AI Impacts


Glossary of AI Risk Terminology and common AI terms

Terms
A
AI timeline
An expectation about how much time will lapse before important AI events, especially the advent of human-level AI or a similar milestone. The term can also refer to the actual periods of time (which are not yet known), rather than an expectation about them.
Artificial General Intelligence (also, AGI)
Skill at performing intellectual tasks across at least the range of variety that a human being is capable of. As opposed to skill at certain specific tasks ('narrow' AI).  That is, synonymous with the more ambiguous Human-Level AI for some meanings of the latter.
Artificial Intelligence (also, AI)
Behavior characteristic of human minds exhibited by man-made machines, and also the area of research focused on developing machines with such behavior. Sometimes used informally to refer to human-level AI or another strong form of AI not yet developed.
Associative value accretion
A hypothesized approach to value learning in which the AI acquires values using some machinery for synthesizing appropriate new values as it interacts with its environment, inspired by the way humans appear to acquire values (Bostrom 2014, p189-190)1.
Anthropic capture
A hypothesized control method in which the AI thinks it might be in a simulation, and so tries to behave in ways that will be rewarded by its simulators (Bostrom 2014 p134).
Anthropic reasoning
Reaching beliefs (posterior probabilities) over states of the world and your location in it, from priors over possible physical worlds (without your location specified) and evidence about your own situation. For an example where this is controversial, see The Sleeping Beauty Problem. For more on the topic and its relation to AI, see here.
Augmentation
An approach to obtaining a superintelligence with desirable motives that consists of beginning with a creature with desirable motives (eg, a human), then making it smarter, instead of designing good motives from scratch (Bostrom 2014, p142).
B
Backpropagation
A fast method of computing the derivative of cost with respect to different parameters in a network, allowing for training neural nets through gradient descent. See Neural Networks and Deep Learning2 for a full explanation.
Boxing
A control method that consists of constructing the AI's environment so as to minimize interaction between the AI and the outside world. (Bostrom 2014, p129).
C
Capability control methods
Strategies for avoiding undesirable outcomes by limiting what an AI can do (Bostrom 2014, p129).
Cognitive enhancement
Improvements to an agent's mental abilities.
Collective superintelligence
"A system composed of a large number of smaller intellects such that the system's overall performance across many very general domains vastly outstrips that of any current cognitive system" (Bostrom 2014, p54).
Computation
A sequence of mechanical operations intended to shed light on something other than this mechanical process itself, through an established relationship between the process and the object of interest.
The common good principle
"Superintelligence should be developed only for the benefit of all of humanity and in the service of widely shared ethical ideals" (Bostrom 2014, p254).
Crucial consideration
An idea with the potential to change our views substantially, such as by reversing the sign of the desirability of important interventions.
D
Decisive strategic advantage
Strategic superiority (by technology or other means) sufficient to enable an agent to unilaterally control most of the resources of the universe.
Direct specification
An approach to the control problem in which the programmers figure out what humans value, and code it into the AI (Bostrom 2014, p139-40).
Domesticity
An approach to the control problem in which the AI is given goals that limit the range of things it wants to interfere with (Bostrom 2014, p140-1).
E
Emulation modulation
Starting with brain emulations with approximately normal human motivations (see 'Augmentation'), and modifying their motivations using drugs or digital drug analogs.
Evolutionary selection approach to value learning
A hypothesized approach to the value learning problem which obtains an AI with desirable values by iterative selection, the same way evolutionary selection produced humans  (Bostrom 2014, p187-8).
Existential risk
Risk of an adverse outcome that would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential (Bostrom 2002)
F
Feature
A dimension in the vector space of activations in a single layer of a neural network (i.e. a neuron activation or linear combination of activations of different neurons)
First principal-agent problem
The well-known problem faced by a sponsor wanting an employee to fulfill their wishes (usually called 'the principal agent problem').
G
Genie
An AI that carries out a high level command, then waits for another (Bostrom 2014, p148).
H
Hardware overhang
A situation where large amounts of hardware being used for other purposes become available for AI, usually posited to occur when AI reaches human-level capabilities.
Human-level AI
An AI that matches human capabilities in virtually every domain of interest.  Note that this term is used ambiguously; see our page on human-level AI.
Human-level hardware
Hardware that matches the information-processing ability of the human brain.
Human-level software
Software that matches the algorithmic efficiency of the human brain, for doing the tasks the human brain does.
I
Impersonal perspective
The view that one should act in the best interests of everyone, including those who may be brought into existence by one's choices (see Person-affecting perspective).
Incentive methods
Strategies for controlling an AI that consist of setting up the AI's environment such that it is in the AI's interest to cooperate. e.g. a social environment with punishment or social repercussions often achieves this for contemporary agents (Bostrom 2014, p131).
Incentive wrapping
Provisions in the goals given to an AI that allocate extra rewards to those who helped bring the AI about  (Bostrom 2014, p222-3).
Indirect normativity
An approach to the control problem in which we specify a way to specify what we value, instead of specifying what we value directly (Bostrom, p141-2).
Instrumental convergence thesis
We can identify 'convergent instrumental values'. That is, subgoals that are useful for a wide range of more fundamental goals, and in a wide range of situations (Bostrom 2014, p109).
Intelligence explosion
A hypothesized event in which an AI rapidly improves from 'relatively modest' to superhuman level (usually imagined to be as a result of recursive self-improvement).
M
Macrostructural development accelerator
An imagined lever used in thought experiments which slows the large scale features of history (e.g. technological change, geopolitical dynamics) while leaving the small scale features the same.
Mind crime
The mistreatment of morally relevant computations.
Moore's Law
Any of several different consistent, many-decade patterns of exponential improvement that have been observed in digital technologies. The classic version concerns the number of transistors in a dense integrated circuit, which was observed to be doubling around every year when the 'law' was formulated in 1965. Price-Performance Moore's Law is often relevant to AI forecasting.
Moral rightness (MR) AI
An AI which seeks to do what is morally right.
Motivational scaffolding
A hypothesized approach to value learning in which the seed AI is given simple goals, and these goals are replaced with more complex ones once it has developed sufficiently sophisticated representational structure (Bostrom 2014, p191-192).
Multipolar outcome
A situation after the arrival of superintelligence in which no single agent controls most of the resources.
O
Optimization power
The strength of a process's ability to improve systems.
Oracle
An AI that only answers questions (Bostrom 2014, p145).
Orthogonality thesis
Intelligence and final goals are orthogonal: more or less any level of intelligence could in principle be combined with more or less any final goal.
P
Person-affecting perspective
The view that one should act in the best interests of everyone who already exists, or who will exist independent of one's choices (see Impersonal perspective).
Perverse instantiation
A solution to a posed goal (eg, make humans smile) that is destructive in unforeseen ways (eg, paralyzing face muscles in the smiling position).
Price-Performance Moore's Law
The observed pattern of relatively consistent, long term, exponential price decline for computation.
Principle of differential technological development
"Retard the development of dangerous and harmful technologies, especially the ones that raise the level of existential risk; and accelerate the development of beneficial technologies, especially those that reduce the existential risk posed by nature or by other technologies" (Bostrom 2014, p230).
Principle of epistemic deference
"A future superintelligence occupies an epistemically superior vantage point: it's beliefs are (probably, on most topics) more likely than our to be true.  We should therefore defer to the superintelligence's position whenever feasible" (Bostrom 2014, p226).
Q
Quality superintelligence
"A system that is at least as fast as a human mind and vastly qualitatively smarter" (Bostrom 2014, p56).
R
Recalcitrance
How difficult a system is to improve.
Recursive self-improvement
The envisaged process of AI (perhaps a seed AI) iteratively improving itself.
Reinforcement learning approach to value learning
A hypothesized approach to value learning in which the AI is rewarded for behaviors that more closely approximate human values (Bostrom 2014, p188-9).
S
Second principal-agent problem
The emerging problem of a developer wanting their AI to fulfill their wishes.
Seed AI
A modest AI which can bootstrap into an impressive AI by improving its own architecture.
Singleton
An agent that is internally coordinated and has no opponents.
Sovereign
An AI that acts autonomously in the world, in pursuit of potentially long range objectives (Bostrom 2014, p148).
Speed superintelligence
"A system that can do all that a human intellect can do, but much faster" (Bostrom 2014, p53).
State risk
A risk that comes from being in a certain state, such that the amount of risk is a function of the time spent there. For example, the state of not having the technology to defend from asteroid impacts carries risk proportional to the time we spend in it.
Step risk
A risk that comes from making a transition. Here the amount of risk is not a simple function of how long the transition takes.  For example, traversing a minefield is not safer if done more quickly.
Stunting
A control method that consists of limiting the AI's capabilities, for instance as by limiting the AI's access to information (Bostrom 2014, p135).
Superintelligence
Any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest (Bostrom 2014, p22).
T
Takeoff
The event of the emergence of a superintelligence, often characterized by its speed: 'slow takeoff' takes decades or centuries, 'moderate takeoff' takes months or years and 'fast takeoff' takes minutes to days.
Technological completion conjecture
If scientific and technological development efforts do not cease, then all important basic capabilities that could be obtained through some possible technology will be obtained (Bostrom 2014, p127).
Technology coupling
A predictable timing relationship between two technologies, such that hastening of the first technology will hasten the second, either because the second is a precursor or because it is a natural consequence (Bostrom 2014, p236-8) e.g. brain emulation is plausibly coupled to 'neuromorphic' AI, because the understanding required to emulate a brain might allow one to more quickly create an AI on similar principles.
Tool AI
An AI that is not 'like an agent', but like a more flexible and capable version of contemporary software. Most notably perhaps, it is not goal-directed (Bostrom 2014, p151).
U
Utility function
A mapping from states of the world to real numbers ('utilities'), describing an entity's degree of preference for different states of the world. Given the choice between two lotteries, the entity prefers the lottery with the highest 'expected utility', which is to say, sum of utilities of possible states weighted by the probability of those states occurring.
V
Value learning
An approach to the value loading problem in which the AI learns the values that humans want it to pursue (Bostrom 2014, p207).
Value loading problem
The problem of causing the AI to pursue human values (Bostrom 2014, p185).
W
Wise-Singleton Sustainability Threshold
A capability set exceeds the wise-singleton threshold if and only if a patient and existential risk-savvy system with that capability set would, if it face no intelligent opposition or competition, be able to colonize and re-engineer a large part of the accessible universe (Bostrom 2014, p100).
Whole-brain emulation
Machine intelligence created by copying the computational structure of the human brain.
Word embedding
A mapping of words to high-dimensional vectors that has been trained to be useful in a word task such that the arrangement of words in the vector space is meaningful. For instance, words near one other in the vector-space are related, and similar relationships between different pairs of words correspond to similar vectors between them, so that e.g. if E(x) is the vector for the word 'x', then E(king) – E(queen) ≈ E(woman) – E(man). Word embeddings are explained in more detail here.
Notes


AI timelines and strategies

AI Impacts sometimes invites guest posts from fellow thinkers on the future of AI. These are not intended to relate closely to our current research, nor to necessarily reflect our views. However we think they are worthy contributions to the discussion of AI forecasting and strategy.
This is a guest post by Sarah Constantin.
One frame of looking at AI risk is the "geopolitical" stance. Who are the major players who might create risky strong AI? How could they be influenced or prevented from producing existential risks? How could safety-minded institutions gain power or influence over the future of AI? What is the correct strategy for reducing AI risk?
The correct strategy depends sharply on the timeline for when strong AI is likely to be developed. Will it be in 10 years, 50 years, 100 years or more? This has implications for AI safety research. If a basic research program on AI safety takes 10-20 years to complete and strong AI is coming in 10 years, then research is relatively pointless. If basic research takes 10-20 years and strong AI is coming more than 100 years from now (if at all), then research can wait. If basic research takes 10-20 years and strong AI is coming in around 50 years, then research is a good idea.
Another relevant issue for AI timelines and strategies is the boom-and-bust cycle in AI. Funding for AI research and progress on AI has historically fluctuated since the 1960s, with roughly 15 years between "booms." The timeline between booms may change in the future, but fluctuation in investment, research funding, and popular attention seems to be a constant in scientific/technical fields.
Each AI boom has typically focused on a handful of techniques (GOFAI in the 1970's, neural nets and expert systems in the 1980's) which promised to deliver strong AI but eventually ran into limits and faced a collapse of funding and investment. The current AI boom is primarily focused on massively parallel processing and machine learning, particularly deep neural nets.
This is relevant because institutional and human capital is lost between booms. While leading universities can survive for centuries, innovative companies are usually only at their peak for a decade or so. It is unlikely that the tech companies doing the most innovation in AI during one boom will be the ones leading subsequent booms. (We don't usually look to 1980's expert systems companies for guidance on AI today.) If there were to be a Pax Googleiana lasting 50 years, it might make sense for people concerned with AI safety to just do research and development within Google. But the history of the tech industry suggests that's not likely. Which means that any attempt to influence long-term AI risk will need to survive the collapse of current companies and the end of the current wave of popularity of AI.
The "extremely short-term AI risk scenario" (of strong AI arising within a decade) is not a popular view among experts; most contemporary surveys of AI researchers predict that strong AI will arise sometime in the mid-to-late 21st century. If we take the view that strong AI in the 2020's is vanishingly unlikely (which is more "conservative" than the results of most AI surveys, but may be more representative of the mainstream computer science view), then this has various implications for AI risk strategy that seem to be rarely considered explicitly.
In the "long-term AI risk scenario", there will be at least one "AI winter" before strong AI is developed. We can expect a period (or multiple periods) in the future where AI will be poorly funded and popularly discredited. We can expect that there are one or more jumps in innovation that will need to occur before human-level AI will be possible. And, given the typical life cycle of corporations, we can expect that if strong AI is developed, it will probably be developed by an institution that does not exist yet.
In the "long-term AI risk scenario", there will probably be time to develop at least some theory of AI safety and the behavior of superintelligent agents. Basic research in computer science (and perhaps neuroscience) may well be beneficial in general from an AI risk perspective. If research on safety can progress during "AI winters" while progress on AI in general halts, then winters are particularly good news for safety. In this long-term scenario, there is no short-term imperative to cease progress on "narrow AI", because contemporary narrow AI is almost certainly not risky.
In the "long-term AI risk scenario", another important goal besides basic research is to send a message to the future. Today's leading tech CEOs will not be facing decisions about strong AI; the critical decisionmakers may be people who haven't been born yet, or people who are currently young and just starting their careers. Institutional cultures are rarely built to last decades. What can we do today to ensure that AI safety will be a priority decades from now, long after the current wave of interest in AI has come to seem faddish and misguided?
The mid- or late 21st century may be a significantly different place than the early 21st century. Economic and political situations fluctuate. The US may no longer be the world's largest economy. Corporations and universities may look very different. Imagine someone speculating about artificial intelligence in 1965 and trying to influence the world of 2015. Trying to pass laws or influence policy at leading corporations in 1965 might not have had a lasting effect (this would be a useful historical topic to investigate in more detail.)
And what if the next fifty years looks more like the cataclysmic first half of the 20th century than the comparatively stable second half of the 20th century? How could a speculative thinker of 1895 hope to influence the world of 1945?
Educational and cultural goals, broadly speaking, seem relevant in this scenario. It will be important to have a lasting influence on the intellectual culture of future generations.
For instance: if fields of theoretical computer science relevant for AI risk are developed and included in mainstream textbooks, then the CS majors of 2050 who might grow up to build strong AI will know about the concerns being raised today as more than a forgotten historical curiosity. Of course, they might not be CS majors, and perhaps they won't even be college students. We have to think about robust transmission of information.
In the "long-term AI risk scenario", the important task is preparing future generations of AI researchers and developers to avoid dangerous strong AI. This means performing and disseminating and teaching basic research in new theoretical fields necessary for understanding the behavior of superintelligent agents.
A "geopolitical" approach is extremely difficult if we don't know who the players will be. We'd like the future institutions that will eventually develop strong AI to be run and staffed by people who will incorporate AI safety into their plans. This means that a theory of AI safety needs to be developed and disseminated widely.
Ultimately, long-term AI strategy bifurcates, depending on whether the future of AI is more "centralized" or "decentralized."
In a "centralized" future, a small number of individuals, perhaps researchers themselves, contribute most innovation in AI, and the important mission is to influence them to pursue research in helpful rather than harmful directions.
In a "decentralized" future, progress in AI is spread over a broad population of institutions, and the important mission is to develop something like "industry best practices" — identifying which engineering practices are dangerous and instituting broadly shared standards that avoid them. This may involve producing new institutions focused on safety.
Basic research is an important prerequisite for both the "centralized" and "decentralized" strategies, because currently we do not know what kinds of progress in AI (if any) are dangerous.
The "centralized" strategy means promoting something like an intellectual culture, or philosophy, among the strongest researchers of the future; it is something like an educational mission. We would like future generations of AI researchers to have certain habits of mind: in particular, the ability to reason about the dramatic practical consequences of abstract concepts. The discoverers of quantum mechanics were able to understand that the development of the atomic bomb would have serious consequences for humanity, and to make decisions accordingly. We would like the future discoverers of major advances in AI to understand the same. This means that today, we will need to communicate (through books, schools, and other cultural institutions, traditional and new) certain intellectual and moral virtues, particularly to the brightest young people.
The "decentralized" strategy will involve taking the theoretical insights from basic AI research and making them broadly implementable. Are some types of "narrow AI" particularly likely to lead to strong AI? Are there some precautions which, on the margin, make harmful strong AI less likely? Which kinds of precautions are least costly in immediate terms and most compatible with the profit and performance needs of the tech industry? To the extent that AI progress is decentralized and incremental, the goal is to ensure that it is difficult to go very far in the wrong direction. Once we know what we mean by a "wrong direction", this is a matter of building long-term institutions and incentives that shape AI progress towards beneficial directions.
The assumption that strong AI is a long-term rather than a short-term risk affects strategy significantly. Influencing current leading players is not particularly important; promoting basic research is very important; disseminating information and transmitting culture to future generations, as well as building new institutions, is the most effective way to prepare for AI advances decades from now.
In the event that AI never becomes a serious risk, developing institutions and intellectual cultures that can successfully reason about AI is still societally valuable. The skill (in institutions and individuals) of taking theoretical considerations seriously and translating them into practical actions for the benefit of humanity is useful for civilizational stability in general. What's important is recognizing that this is a long-term strategy — i.e. thinking more than ten years ahead. Planning for future decades looks different from taking advantage of the current boom in funding and attention for AI and locally hill-climbing.
Sarah Constantin blogs at Otium. She recently graduated from Yale with a PhD in mathematics.

AI timelines and strategies


Introducing research bounties

By Katja Grace, 7 August 2015
Sometimes we like to experiment with novel research methods and formats. Today we are introducing 'AI Impacts Research Bounties', in which you get money if you send us inputs to some of our research.
To start, we have two bounties: one for showing us instances of abrupt technological progress, and one for pointing us to instances of people acting to avert risks decades ahead of time. Rewards currently range from $20 to $500, and anyone can enter. We may add more bounties, or adjust prices, according to responses. We welcome feedback on any aspect of this experiment.
Thanks to John Salvatier for ongoing collaboration on this project.

Introducing research bounties


AI Impacts research bounties

We are offering rewards for several inputs to our research, described below. These offers have no specific deadline except where noted. We may modify them or take them down, but will give at least one week's notice here unless there is strong reason not to. To submit an entry, email <EMAIL>. There is currently a large backlog of entries to check, so new entries will not receive a rapid response.
1. An example of discontinuous technological progress ($50-$500)
This bounty offer is no longer available after 3 November 2016.
We are interested in finding more examples of large discontinuous technological progress to add to our collection. We're offering a bounty of around $50-500 per good example.
We currently know of two good examples (and one moderate example):

Nuclear weapons discontinuously increased the relative effectiveness of explosives.
High temperature superconductors led to a dramatic increase in the highest temperature at which superconducting was possible.

To assess discontinuity, we've been using "number of years worth of progress at past rates", as measured by any relevant metric of technological progress. For example, the discovery of nuclear weapons was equal to about 6,000 years worth of previous progress in the relative effectiveness of explosives. However, we are also interested in examples that seem intuitively discontinuous, even if they don't exactly fit the criteria of being a large number of year's progress in one go.
Things that make examples better:

Size: Better examples represent larger changes. More than 20 times normal annual progress is ideal.
Sharpness: Better examples happened over shorter periods. Over less than a year is ideal.
Breadth: Metrics that measure larger categories of things are better. For example, fast adoption curves for highly specific categories (say a particular version of some software) is much less interesting than fast adoption curves for much broader categories (say a whole category of software).
Rarity: As we receive more examples, the interestingness of each one will tend to decline.

AI Impacts is willing to pay more for better examples. Basically we will judge how interesting your example is and then reward you based on that. We will accept examples that violate our stated preferences but satisfy the spirit of the bounty. Our guess is that we would pay about $500 for another example as good nuclear weapons.
How to enter: all that is necessary to submit an example is to email us a paragraph describing the example, along with sources to verify your claims (such sources are likely to involve at least one time series of success on a particular metric). Note that an example should be of the form 'A caused abrupt progress in metric B'. For instance, 'The boliolicopter caused abrupt progress in the maximum rate of fermblangling at sub-freezing temperatures'.
2. An example of early action on a risk ($20-$100)
This bounty offer is no longer available after 3 November 2016.
We want: a one sentence description of a case where at least one person acted to avert a risk that was least fifteen years away, along with a link or citation supporting the claim that the action preceded the risk by at least fifteen years.
We will give: up $100, with higher sums for examples that are better according to our judgment (see criteria for betterness below), and which we don't already know about. We might go over $100 for exceptionally good examples.
Further details
Examples are better if:

The risk is more novel: relatively similar problems have not arisen before, and would probably not arise sooner than fifteen years in the future. e.g. Poverty in retirement is a risk people often prepare for more than fifteen year before it befalls them, however it is not very novel because other people already face an essentially identical risk, and have done many times before.
The solution is more specific: the action taken would not be nearly as useful if the risk disappeared. e.g. Saving money to escape is a reasonable solution to expecting your country to face civil war soon. However saving money is fairly useful in any case, so this solution is not very specific.
We haven't received a lot of examples: as we collect more examples, the value of each one will tend to decline.

Some examples:

Leo Szilard's secret nuclear patent: the threat of nuclear weapons was quite novel. It's unclear when Szilard expected such weapons, but quite plausibly at least fifteen years later in 1934. The secret patent does not seem broadly useful, though useful for encouraging more local nuclear research, which is somewhat more broadly useful than secrecy per se. More details in this report. This is a reasonably good example.
The Asilomar Conference on recombinant DNA: the risk of was arguably quite novel (genetically engineered pandemics), and the solution was reasonably specific (safety rules for dealing with recombinant DNA). However the risks people were concerned about were immediate, rather than decades hence. More details here. This is not a good example.

Evidence that the example is better in the above ways is also welcome, though we reserve the right not to explore it fully.

AI Impacts research bounties


Time flies when robots rule the earth

This week Robin Hanson is finishing off his much anticipated book, The Age of Em: Work, Love and Life When Robots Rule the Earth. He recently told me that it would be helpful to include rough numbers for the brain's memory and computing capacity in the book, so I agreed to prioritize finding the ones AI Impacts didn't already have. Consequently we just put up new pages about information storage in the brain. We also made a summary of related 'human-level' hardware pages, and an index of pages about hardware in general.
Robin's intended use for these numbers is interesting. The premise of his book is that one day (perhaps in the far future) human minds might be emulated on computers, and that this would produce a society that is somewhere between recognizably human and predictably alien. Robin's project is a detailed account of life in such a society, as far as it can be discerned by peering through social science and engineering theory.
One prediction Robin makes is that these emulated minds ('ems') will run at a wide variety of speeds, depending on their purposes and the incentives. So some ems will have whole lifetimes while others are getting started on a single thought. Robin wanted to know how slow the very slowest ems would run. And for this,  he wanted to know much memory the brain uses, and how much computing it does.
His reasoning is as follows. The main costs of running emulations are computing hardware and memory. If Anna is running twice as fast as Ben, then Anna needs about twice as much computing power to run, which will cost about twice as much. However Anna still uses around as much memory as Ben to store the contents of her brain over time. So if most of the cost of an emulation is in computation, then halving the speed would halve the cost, and would often be worth it. But once an emulation is moving so slowly that memory becomes the main cost, slowing down by half makes little difference, and soon stops being worth it. So the slowest emulations should run at around the speed at which computing hardware and memory contribute similarly to cost.
Hardware and memory costs have been falling at roughly similar rates in the past, so if this continues, then the ratio between their costs now is a reasonable (if noisy) predictor of their ratio in several decades time. Given our numbers, Robin estimates that the slowest emulations will operate at between a one hundred trillionth of human speed and one millionth of human speed, with a middle estimate of one tenth of a billionth of human speed.
At these rates, immortality looks a lot like dying. If you had been experiencing the world at these speeds since the beginning of the universe, somewhere between an hour and a thousand years would seem to have passed, with a middle estimate of a year. Even if the em economy somehow lasts for a thousand years, running this slowly would mean immediately jumping into whatever comes next.
These things are rough of course, but it seems pretty cool to me that we can make reasonable guesses at all about such exotic future scenarios, using clues from our everyday world, like the prevailing prices of hard disks and memory.
If you are near Berkeley, CA and want to think more about this kind of stuff, or in this kind of style, remember you can come and meet Robin and partake in more economic futurism at our event this Thursday. We expect a good number of attendees already, but could squeeze in a few more.
Image: Robin Hanson in a field, taken by Katja Grace

Time flies when robots rule the earth