The cost of TEPS

A billion [Traversed Edges Per Second](http://en.wikipedia.org/wiki/Traversed_edges_per_second) (a [G](http://en.wikipedia.org/wiki/Giga-)TEPS) can be bought for around $0.26/hour via a powerful supercomputer, including hardware and energy costs only. We do not know if GTEPS can be bought more cheaply elsewhere.

We estimate that available TEPS/$ grows by a factor of ten every four years, based the relationship between TEPS and FLOPS. TEPS have not been measured enough to see long-term trends directly.

Background
----------

Traversed edges per second ([TEPS](http://en.wikipedia.org/wiki/Traversed_edges_per_second)) is a measure of computer performance, similar to [FLOPS](http://en.wikipedia.org/wiki/FLOPS) or [MIPS](http://en.wikipedia.org/wiki/Instructions_per_second#Millions_of_instructions_per_second).  Relative to these other metrics, TEPS emphasizes the communication capabilities of machines: the ability to move data around inside the computer. Communication is especially important in very large machines, such as supercomputers, so TEPS is particularly useful in evaluating these machines.

The [Graph 500](http://www.graph500.org/results_nov_2013) is a list of computers which have been evaluated according to this metric. It is [intended](http://www.graph500.org/) to complement the [Top 500](http://www.top500.org/lists/2014/11/), which is a list of  the most powerful 500 computers, measured in FLOPS. The Graph 500 began in 2010, and so far has measured 183 machines, though many of these are not supercomputers, and would presumably not rank among the best 500 TEPS scores if more supercomputers computers were measured.

The TEPS benchmark is defined as the number of graph edges traversed per second during a breadth-first search of a very large graph. The scale of the graph is tuned to grow with the size of the hardware. See the [Graph500 benchmarks page](http://www.graph500.org/specifications) for further details.

### The brain in TEPS

We are interested in TEPS in part because we would like to estimate the brain’s capacity in terms of TEPS, as an input to forecasting AI timelines. One virtue of this is that it will be a relatively independent measure of how much hardware the human brain is equivalent to, which we can then compare to other estimates. It is also easier to measure information transfer in the brain than computation, making this a more accurate estimate. We also expect that at the scale of the brain, communication is a significant bottleneck (much as it is for a supercomputer), making TEPS a particularly relevant benchmark. The brain’s contents support this theory: much of its mass and energy appears to be used on moving information around.

Current TEPS available per dollar
---------------------------------

We estimate that a TEPS can currently be produced for around $0.26 per hour in a supercomputer.

### Our estimate

Table 1 shows our calculation, and sources for price figures.

We recorded the TEPS scores for the top eight computers in the [Graph 500](http://www.graph500.org/results_nov_2014) (i.e. the best TEPS-producing computers known). We searched for price estimates for these computers, and found five of them. We assume these prices are for hardware alone, though this was not generally specified. The prices are generally from second-hand sources, and so we doubt they are particularly reliable.

#### Energy costs

We took energy use figures for the five remaining computers from the [Top 500](http://www.top500.org/list/2014/11/) list. Energy use on the Graph 500 and Top 500 benchmarks are probably somewhat different, especially because computers are often scaled down for the Graph 500 benchmark. See ‘Bias from scaling down’ below for discussion of this problem. There is a Green Graph 500 list, which gives energy figures for some of the supercomputers doing similar problems to those in the Graph 500, but the computers are run at different scales there to in the Graph 500 (presumably to get better energy ratings), so the energy figures given there are also not directly applicable.

The cost of electricity varies by location. We are interested in how cheaply one can produce TEPS, so we suppose computation is located somewhere where power is cheap, charged at industrial rates. Prevailing energy prices in the US [are around](http://www.eia.gov/electricity/monthly/epm_table_grapher.cfm?t=epmt_5_6_a) $0.20 / kilowatt hour, but in some parts of Canada [it seems](http://en.wikipedia.org/wiki/Electricity_sector_in_Canada#Rates) industrial users pay less than $0.05 / kilowatt hour. This is also low relative to [industrial energy prices](http://www.statista.com/statistics/263262/industrial-sector-electricity-prices-in-selected-european-countries/) in various European nations (though these nations too may have small localities with cheaper power). Thus we take $0.05 to be a cheap but feasible price for energy.

#### Bias from scaling down

Note that our method likely overestimates necessary hardware and energy costs, as many computers [do not use all of their cores](http://spectrum.ieee.org/computing/hardware/better-benchmarking-for-supercomputers) in the Graph 500 benchmark (this can be verified by comparing to cores used in the Top 500 list compiled at the same time). This means that one could get better TEPS/$ prices by just not building parts of existing computers. It also means that the energy used in the Graph 500 benchmarking (not listed) was probably less than that used in the Top 500 benchmarking.

We correct for this by scaling down prices according to cores used. This is probably not a perfect adjustment: the costs of building and running a supercomputer are unlikely to be linear in the number of cores it has. However this seems a reasonable approximation, and better than making no adjustment.

This change makes the data more consistent. The apparently more expensive sources of TEPS were using smaller fractions of their cores (if we assume they used all cores in the Graph 500), and the very expensive Tianhe-2 was using only 6% of its cores. Scaled according to the fraction of cores used in Graph 500, Tianhe-2 produces TEPShours at a similar price to Sequoia. The two apparently cheapest sources of TEPShours (Sequoia and Mira) appear to have been using all of their cores. Figure 1 shows the costs of TEPShours on the different supercomputers, next to the costs when scaled down according to the fraction of cores that were used in the Graph 500 benchmark.

[![](http://aiimpacts.org/wp-content/uploads/2015/03/image-7.png)](http://aiimpacts.org/wp-content/uploads/2015/03/image-7.png)**Figure 1**: Cost of TEPShours using five supercomputers, and cost naively adjusted for fraction of cores used in the benchmark test.

#### Other costs

Supercomputers have many costs besides hardware and energy, such as property, staff and software. Figures for these are hard to find. [This presentation](http://www.efiscal.eu/files/presentations/amsterdam/Snell_IS360_TCO_presentation.pdf) suggests the total cost of a large supercomputerover several years can be more than five times the upfront hardware cost. However these figures seem surprisingly high, and we suspect they are not applicable to the problem we are interested in: running AI. High property costs are probably because supercomputers tend to be built in college campuses. Strong AI software is presumably more expensive than what is presently bought, but we do not want to price this into the estimate. Because the figures in the presentation are the only ones we have found, and appear to be inaccurate, we will not further investigate the more inclusive costs of producing TEPShours here, and focus on upfront hardware costs and ongoing energy costs.

#### Supercomputer lifespans

We assume a supercomputer lasts for five years. This was the age of [Roadrunner](http://en.wikipedia.org/wiki/IBM_Roadrunner) when decommissioned in 2013, and is consistent with the ages of the computers whose prices we are calculating here — they were all built between 2011 and 2013. [ASCI Red](http://en.wikipedia.org/wiki/ASCI_Red) lasted for nine years, but was apparently considered ‘[supercomputing’s high-water mark in longevity](http://www.upi.com/Science_News/2006/06/29/Worlds-first-supercomputer-decommissioned/UPI-60321151628137/)‘. We did not find other examples of large decommissioned supercomputers with known lifespans.

#### Calculation

From all of this, we calculate the price of a GTEPShour in each of these systems, as shown in table 1.

| Name | GTeps | Estimated Price (million) | Hardware cost/hour (5 year life) | Energy (kW) | Hourly energy cost (at 5c/kWh) | Total $/hour
(including hardware and energy) | $/GTEPShours
(including hardware and energy) | $/GTEPShours scaled by cores used | Cost sources |
| DOE/NNSA/LLNL Sequoia (IBM – BlueGene/Q, Power BQC 16C 1.60 GHz) | 23751 | $250 | $5,704 | 7,890.00 | $394.50 | 6,098.36 | $0.26 |  $0.26 | [1](https://aiimpacts.org/cost-of-teps/#easy-footnote-bottom-1-457 "“Livermore told us it spent roughly $250 million on Sequoia.” http://arstechnica.com/information-technology/2012/06/with-16-petaflops-and-1-6m-cores-doe-supercomputer-is-worlds-fastest/") |
| K computer (Fujitsu – Custom supercomputer) | 19585.2 | $1,000 | $22,815 | 12,659.89 | $632.99 | 23,448.42 | $1.20 |  $1.13 | [2](https://aiimpacts.org/cost-of-teps/#easy-footnote-bottom-2-457 "“The K Computer in Japan, for example, cost more than $1 billion to build and $10 million to operate each year.” http://arstechnica.com/information-technology/2012/06/with-16-petaflops-and-1-6m-cores-doe-supercomputer-is-worlds-fastest/ (note that our estimated energy expenses come to around $5M, which seems consistent with this).") |
| DOE/SC/Argonne National Laboratory Mira (IBM – BlueGene/Q, Power BQC 16C 1.60 GHz) | 14982 | $50 | $1,141 | 3,945.00 | $197.25 | 1,338.02 | $0.09 | $0.09 | [3](https://aiimpacts.org/cost-of-teps/#easy-footnote-bottom-3-457 "“Mira is expected to cost roughly $50 million, according to reports.” https://www.alcf.anl.gov/articles/mira-worlds-fastest-supercomputer”IBM did not reveal the price for Mira, though it did say Argonne had purchased it with funds from a US$180 million grant.” http://www.pcworld.com/article/218951/us_commissions_beefy_ibm_supercomputer.html,") |
| Tianhe-2 (MilkyWay-2) (National University of Defense Technology – MPP) | 2061.48 | $390 | $8,898 | 17,808.00 | $890.40 | 9,788.42 | $4.75 | $0.30 | [4](https://aiimpacts.org/cost-of-teps/#easy-footnote-bottom-4-457 "“Cost: 2.4 billion Yuan or 3 billion Hong Kong dollars (390 million US Dollars)” http://www.crizmo.com/worlds-top-10-supercomputers-with-their-cost-speed-and-usage.html ") |
| Blue Joule (IBM – BlueGene/Q, Power BQC 16C 1.60 GHz) | 1427 | $55.3 | $1,262 | 657.00 | $32.85 | 1,294.54 | $0.91 |  $0.46 | [5](https://aiimpacts.org/cost-of-teps/#easy-footnote-bottom-5-457 "“Blue Joule…The cost of this system appears to be 10 times (£37.5 million) the above mentioned grant to develop the Emerald GPU supercomputer.” http://hexus.net/business/news/enterprise/41937-uks-powerful-gpu-supercomputer-booted/ Note that £37.5M = $55.3M ") |

***Table 1**: Calculation of costs of a TEPS over one hour in five supercomputers.*

#### Sequoia as representative of cheap TEPShours

Mira and then Sequoia produce the cheapest TEPShours of the supercomputers investigated here, and are also the only ones which used all of their cores in the benchmark, making their costs less ambiguous. Mira’s costs are ambiguous nonetheless, because the $50M price estimate we have was projected by an unknown source, ahead of time. Mira is also known to have been bought using some part of a $180M grant. If Mira cost most of that, it would be more expensive than Sequoia. Sequoia’s price was given by the laboratory that bought it, after the fact, so is more likely to be reliable.

Thus while Sequoia does not appear to be the cheapest source of TEPS, it does appear to be the second cheapest, and its estimate seems substantially more reliable. Sequoia is also a likely candidate to be especially cheap, since it is ranked first in the Graph 500, and is the largest of the IBM [Blue Gene/Q](http://en.wikipedia.org/wiki/Blue_Gene)s, which dominate the top of the Graph 500 list. This somewhat supports the validity of its apparent good price performance here.

Sequoia is also not much cheaper than the more expensive supercomputers in our list, once they are scaled down according to the number of cores they used on the benchmark (see Table 1), further supporting this price estimate.

Thus we estimate that GTEPShours can be produced for around $0.26 on current supercomputers. This corresponds to around $11,000/GTEP to buy the hardware alone.

#### Price of TEPShours in lower performance computing

We have only looked at the price of TEPS in top supercomputers. While these produce the most TEPS, they might not be the part of the range which produces TEPS most cheaply. However because we are interested in the application to AI, and thus to systems roughly as large as the brain, price performance near the top of the range is particularly relevant to us. Even if a laptop could produce a TEPS more cheaply than Sequoia, it produces too few of them to run a brain efficiently. Nonetheless, we plan to investigate TEPS/$ in lower performing computers in future.

For now, we checked the efficiency of an iPad 3, since one was listed near the bottom of the Graph 500. These are sold for [$349.99](http://www.amazon.com/Apple-MC705LL-Wi-Fi-Black-Generation/dp/B00746LVOM/ref=sr_1_1?ie=UTF8&qid=1426895358&sr=8-1&keywords=3rd+generation+ipad), and apparently produce 0.0304 GTEPS. Over five years, this comes out at exactly the same price as the Sequoia: $0.26/GTEPShour. This suggests both that cheaper computers may be more efficient than large supercomputers (the iPad is not known for its cheap computing power) and that the differences in price are probably not large across the performance spectrum.

Trends in TEPS available per dollar
-----------------------------------

The long-term trend of TEPS is not well known, as the benchmark is new. This makes it hard to calculate a TEPS/$ trend. Figure 2 is from a powerpoint *[Announcing the 9th Graph500 List!](http://www.graph500.org/sites/default/files/files/bof/Graph500-BoF-SC14-v1.pdf)* from the [Top 500 website](http://www.graph500.org/bof). One thing it shows is top performance in the Graph 500 list since the list began in 2010. Top performance grew very fast (3.5 orders of magnitude in two years), before completely flattening, then growing slowly. The powerpoint attributes this pattern to ‘maturation of the benchmark’, suggesting that the steep slope was probably not reflective of real progress.

One reason to expect this pattern is that during the period of fast growth, pre-existing high performance computers were being tested for the first time. This appears to account for some of it. However we note that in June 2012, Sequoia (which tops the list at present) and Mira (#3) had both already been tested, and merely had lower performance than they do now, suggesting at least one other factor is at play. One possibility is that in the early years of using the benchmark, people develop good software for the problem, or in other ways adjust how they use particular computers on the benchmark.

[![teps trend top 500 copy](http://aiimpacts.org/wp-content/uploads/2015/03/teps-trend-top-500-copy-1024x772.png)](http://aiimpacts.org/wp-content/uploads/2015/03/teps-trend-top-500-copy.png)**Figure 2**: Performance of the top supercomputer on Graph 500 each year since it has existed (along with the 8th best, and an unspecified sum).

### Relationship between TEPS and FLOPS

The top eight computers in the Graph 500 are also in the [Top 500](http://en.wikipedia.org/wiki/TOP500), so we can compare their TEPS and FLOPS ratings. Because many computers did not use all of their cores in the Graph 500, we scale down the FLOPS measured in the Top 500 by the fraction of cores used in the Graph 500 relative to the Top 500 (this is discussed further in ‘Bias from scaling down’ above). We have not checked thoroughly whether FLOPS scales linearly with cores, but this appears to be a reasonable approximation, based on the first page of the Top 500 list.

The supercomputers measured here consistently achieve around 1-2 GTEPS per scaled TFLOPS (see Figure 3). The median ratio is 1.9 GTEPS/TFLOP, the mean is 1.7 GTEPS/TFLOP, and the variance 0.14 GTEPS/TFLOP. Figure 4 shows GTEPS and TFLOPS plotted against one another.

The ratio of GTEPS to TFLOPS may vary across the range of computing power. Our figures may may also be slightly biased by selecting machines from the top of the Graph 500 to check against the Top 500. However the current comparison gives us a rough sense, and the figures are consistent.

[This presentation](http://on-demand.gputechconf.com/gtc/2013/presentations/S3089-Breadth-First-Search-Multiple-GPUs.pdf) (slide 23) reports that a Kepler GPU produces 109 TEPS, as compared to 1012 FLOPS reported [here](http://en.community.dell.com/techcenter/high-performance-computing/b/weblog/archive/2013/11/25/accelerating-high-performance-linpack-hpl-with-kepler-k20x-gpus.aspx) (assuming that both are top end models), suggesting a similar ratio holds for less powerful computers.

[![Figure xxx: GTEPS/scaled TFLOPS, based on Graph 500 and Top 500.](http://aiimpacts.org/wp-content/uploads/2015/03/image-10.png)](http://aiimpacts.org/wp-content/uploads/2015/03/image-10.png)Figure 3: GTEPS/scaled TFLOPS, based on Graph 500 and Top 500.

[![Figure xxx: GTEPS and scaled TFLOPS achieved by the top 8 machines on Graph 500. See text for scaling description. ](http://aiimpacts.org/wp-content/uploads/2015/03/image-9.png)](http://aiimpacts.org/wp-content/uploads/2015/03/image-9.png)Figure 4: GTEPS and scaled TFLOPS achieved by the top 8 machines on Graph 500. See text for scaling description.

####

#### Projecting TEPS based on FLOPS

Since the conversion rate between FLOPS and TEPS is approximately consistent, we can project growth in TEPS/$ based on the better understood growth of FLOPS/$. In the last quarter of a century, FLOPS/$ [has grown](http://aiimpacts.org/trends-in-the-cost-of-computing/ "Trends in the cost of computing") by a factor of ten roughly every four years. This suggests that TEPS/$ also grows by a factor of ten every four years.

---

The cost of TEPS


Allen, The Singularity Isn’t Near

[The Singularity Isn’t Near](http://www.technologyreview.com/view/425733/paul-allen-the-singularity-isnt-near/) is an article in [MIT Technology Review](http://www.technologyreview.com/) by [Paul Allen](http://en.wikipedia.org/wiki/Paul_Allen) which argues that a singularity brought about by super-human-level AI will not arrive by 2045 (as is [predicted](https://sites.google.com/site/aiimpactslibrary/ai-timelines/predictions-of-human-level-ai-dates/published-analyses-of-time-to-human-level-ai/kurzweil-the-singularity-is-near) by Kurzweil).

The summarized argument
-----------------------

We will not have human-level AI by 2045:

1. To reach human-level AI, we need software as well as hardware.

2. To get this software, we need one of the following:

* a detailed scientific understanding of the brain
* a way to ‘duplicate’ brains
* creation of something equivalent to a brain from scratch

3. A detailed scientific understanding of the brain is unlikely by 2045:

1. To have enough understanding by 2045, we would need a massive acceleration of scientific progress:
	1. We are just scraping the surface of understanding the foundations of human cognition.
2. A massive acceleration of progress in brain science is unlikely
	1. Science progresses irregularly:
		1. e.g. The discovery of long-term potentiation, the columnar organization of cortical areas, neuroplasticity.
	2. Science doesn’t seem to be exponentially accelerating
	3. There is a ‘complexity break’: the more we understand, the more complicated the next level to understand is

4. ‘Duplicating’ brains is unlikely by 2045:

1. Even if we have good scans of brains, we need good understanding of how the parts behave to complete the model
2. We have little such understanding
3. Such understanding is not exponentially increasing

5. Creation of something equivalent to a brain from scratch is unlikely by 2045:

1. Artificial intelligence research appears to be far from providing this
2. Artificial intelligence research is unlikely to improve fast:
	1. Artificial intelligence research does not appear to be exponentially improving
	2. The ‘complexity break’ (see above) also operates here
	3. This is the kind of area where progress is not a reliable exponential

Comments
--------

The controversial parts of this argument appear to be the parallel claims that progress is insufficiently fast (or accelerating) to reach an adequate understanding of the brain or of artificial intelligence algorithms by 2045. Allen’s argument does not present enough support to evaluate them from this alone. Others with at least as much expertise disagree with these claims, so they appear to be open questions.

To evaluate them, it appears we would need more comparable measures of accomplishments and rates of progress in brain science and AI. With only the qualitative style of Allen’s claims, it is hard to know whether progress being slow, and needing to go far, implies that it won’t get to a specific place by a specific date.

Allen, The Singularity Isn’t Near


Kurzweil, The Singularity is Near

[The Singularity Is Near](http://en.wikipedia.org/wiki/The_Singularity_Is_Near) is a book by [Ray Kurzweil](http://en.wikipedia.org/wiki/Ray_Kurzweil). It argues that a [technological singularity](http://en.wikipedia.org/wiki/Technological_singularity) will occur in around 2045. This appears to be largely based on extrapolation from hardware in combination with a guess for how much machine computation is needed to produce a large disruption to human society. The book relatedly claims that a machine will be able to pass a Turing test by 2029.

Details
-------

### Calculation of the date of the singularity

The following is our reconstruction of an argument Kurzweil makes in the book, for expecting the Singularity in 2045.

1. In the early 2030s one thousand dollars’ worth of computation will buy about 10¹⁷ computations per second (p119)
2. Today we spend more than $10¹¹/year on computation, which will conservatively rise to $10¹²/year by 2030 (p119-20).
3. Therefore in the early 2030s we will be producing about 10²⁶-10²⁹ computations per second of nonbiological computation per year, and by the mid 2040s, we will produce 10²⁶ cps with $1000 (p120)
4. The sum of all living biological human intelligence operates at around 10²⁶ computations per second (p113)
5. Thus in the early 2030s we will produce new computing power roughly equivalent to the capacity of all living biological human intelligence, every year. In the mid 2040s the total computing capacity we produce each year will be a billion times more powerful than all of human intelligence today. (p120)
6. Non-biological intelligence will be better than our own brains because machines have some added advantages, such as accuracy and ability to run at peak capacity. (p120)
7. The early 2030s will not be a singularity, because its events do not yet correspond to a sufficiently profound expansion of our intelligence. (p120)
8. In the 1940s, when the computing capacity we produce each year is a billion times more powerful than all human intelligence today, these events will represent a profound and disruptive transformation in human capability, i.e. a singularity. (p120)

### Relevance of software

While he doesn’t mention it in the prediction explained above, Kurzweil appears elsewhere to agree that substantial software progress is needed alongside hardware progress for human-level intelligence. He says, “The hardware computational capacity is necessary but not sufficient. Understanding the organization and content of these resources—the software of intelligence—is even more critical and is the objective of the brain reverse-engineering undertaking.” (p126).

His argument that the necessary understanding for producing human-level software will come in time with the hardware appears to be as follows:

1. Understanding of the  brain is reasonably good; researchers rapidly turn data from studies into effective working models (p147)
2. Understanding of the brain is growing exponentially:
	1. Our ability to observe the brain is growing exponentially: ‘Scanning and sensing tools are doubling their overall spatial and temporal resolution each year’. (p163)
	2. ‘Databases of brain-scanning information and model building are also doubling in size about once per year.’ (p163)
	3. Our ability to model the brain follows closely behind our acquisition of the requisite tools and data (p163) and so is also growing exponentially in some sense.

### Human-level AI

According to Kurzweil, ‘With both the hardware and software needed to fully emulate human intelligence, we can expect computers to pass the Turing test, indicating intelligence indistinguishable from that of biological humans, by the end of the 2020s.’

The claims that hardware and software will be human-level by 2029 appear to share their justification with the above claims about the timing of the Singularity.

Kurzweil [bet](http://longbets.org/1/) that by 2029 a computer would pass the turing test, and wrote an article explaining his optimism about the bet [here](https://web.archive.org/web/20110720061136/http://www.kurzweilai.net/a-wager-on-the-turing-test-why-i-think-i-will-win).

Comments
--------

If the ‘singularity’ is meant to refer to some particular event, it is unclear why this event would occur when the hardware produced is a billion times more powerful than all human intelligence today. This number might make some sense as an upper bound on when something disruptive should have happened. However it is unclear why the events predicted in the early 2030s would not cause a profound and disruptive transformation, while those in the mid 2040s would.

Kurzweil’s calculation of the date of the Singularity appears to have other minor gaps:

1. The argument is about flows of hardware, where it wants to make a conclusion in terms of stocks of hardware. Kurzweil wants to compare total biological and non-biological computation. However he calculates the computing hardware produced per year, instead of the total available that year, or the computation done in that year. These numbers are probably fairly similar in practice, if we suppose that hardware lasts a small number of years.
2. That non-biological machines appear to have some advantages over humans does not imply that some given non-biological machines have advantages overall.
3. The argument suggests software will develop ‘fast’ in some sense, but this isn’t actually compared to hardware progress or measured in years, so it is unclear whether it would be developed in time.

A key disagreement with other commentators appears to be over the rate of progress of understanding relevant to producing software. In particular, Kurzweil believes that such understanding is growing exponentially, and that it will be be sufficient for producing machines as intelligent as humans in line with the hardware. [Allen](http://aiimpacts.org/allen-the-singularity-isnt-near/ "Allen, The Singularity Isn’t Near"), for instance, has argued with this. Resolving this disagreement would require better measures of neuroscience progress, as well as a better understanding of its relevance.

Kurzweil, The Singularity is Near


Wikipedia history of GFLOPS costs

This is a list from [Wikipedia](http://en.wikipedia.org/wiki/FLOPS#Hardware_costs), showing hardware configurations that authors claim perform efficiently, along with their prices per GFLOPS at different times in recent history.

In it, prices generally fall at around an order of magnitude every five years, and have continued to do so recently.

Notes
-----

This list is from November 5 2017 ([archive version](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/FLOPS)). It is not necessarily credible. We had trouble verifying at least one datapoint, of the few we tried. Performance numbers appear to be a mixture of theoretical peak performance and empirical performance. It is not clear to what extent one should expect the included systems to be especially cost-effective, or why these particular systems were chosen.

The last point is in October 2017, and appears to be roughly in line with the rest of the trend. The last order of magnitude  took around 4.5 years. The overall rate in the figure appears to be very roughly an order of magnitude every five years.

List
----

| Date | Approximate cost per GFLOPS | Approximate cost per GFLOPS inflation adjusted to 2013 US dollars[[54]](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/FLOPS#cite_note-54) | Platform providing the lowest cost per GFLOPS | Comments |
| --- | --- | --- | --- | --- |
| 1961 | US$18,672,000,000 ($18.7 billion) | US$145.5 billion | About 2400 [IBM 7030 Stretch](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/IBM_7030_Stretch "IBM 7030 Stretch") supercomputers costing $7.78 million each | The [IBM 7030 Stretch](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/IBM_7030_Stretch "IBM 7030 Stretch") performs one floating-point multiply every 2.4 microseconds.[[55]](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/FLOPS#cite_note-55) |
| 1984 | $18,750,000 | $42,780,000 | [Cray X-MP](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/Cray_X-MP "Cray X-MP")/48 | $15,000,000 / 0.8 GFLOPS |
| 1997 | $30,000 | $42,000 | Two 16-processor [Beowulf](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/Beowulf_(computing) "Beowulf (computing)")clusters with [Pentium Pro](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/Pentium_Pro "Pentium Pro")microprocessors[[56]](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/FLOPS#cite_note-56) |  |
| April 2000 | $1,000 | $1,300 | [Bunyip Beowulf cluster](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/Beowulf_(computing) "Beowulf (computing)") | Bunyip was the first sub-US$1/MFLOPS computing technology. It won the Gordon Bell Prize in 2000. |
| May 2000 | $640 | $836 | [KLAT2](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/Kentucky_Linux_Athlon_Testbed "Kentucky Linux Athlon Testbed") | KLAT2 was the first computing technology which scaled to large applications while staying under US-$1/MFLOPS.[[57]](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/FLOPS#cite_note-57) |
| August 2003 | $82 | $100 | KASY0 | KASY0 was the first sub-US$100/GFLOPS computing technology.[[58]](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/FLOPS#cite_note-58) |
| August 2007 | $48 | $52 | Microwulf | As of August 2007, this 26.25 GFLOPS “personal” Beowulf cluster can be built for $1256.[[59]](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/FLOPS#cite_note-59) |
| March 2011 | $1.80 | $1.80 | HPU4Science | This $30,000 cluster was built using only commercially available “gamer” grade hardware.[[60]](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/FLOPS#cite_note-60) |
| August 2012 | $0.75 | $0.73 | Quad AMD Radeon 7970 GHz System | A quad [AMD](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/AMD "AMD") [Radeon 7970](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/Radeon_HD_7000_Series "Radeon HD 7000 Series") desktop computer reaching 16 TFLOPS of single-precision, 4 TFLOPS of double-precision computing performance. Total system cost was $3000; Built using only commercially available hardware.[[61]](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/FLOPS#cite_note-61) |
| June 2013 | $0.22 | $0.22 | Sony PlayStation 4 | The Sony [PlayStation 4](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/PlayStation_4 "PlayStation 4") is listed as having a peak performance of 1.84 TFLOPS, at a price of $400[[62]](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/FLOPS#cite_note-62) |
| November 2013 | $0.16 | $0.16 | AMD Sempron 145 & GeForce GTX 760 System | Built using commercially available parts, a system using one AMD [Sempron](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/Sempron "Sempron") 145 and three [Nvidia](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/Nvidia "Nvidia") [GeForce GTX 760](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/GeForce_700_series "GeForce 700 series") reaches a total of 6.771 TFLOPS for a total cost of $1090.66.[[63]](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/FLOPS#cite_note-63) |
| December 2013 | $0.12 | $0.12 | Pentium G550 & Radeon R9 290 System | Built using commercially available parts. [Intel](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/Intel "Intel") [Pentium G550](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/Sandy_Bridge "Sandy Bridge") and AMD [Radeon R9 290](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/AMD_Radeon_Rx_200_series "AMD Radeon Rx 200 series") tops out at 4.848 TFLOPS grand total of US$681.84.[[64]](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/FLOPS#cite_note-64) |
| January 2015 | $0.08 | $0.08 | Celeron G1830 & Radeon R9 295X2 System | Built using commercially available parts. Intel [Celeron G1830](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/Haswell_(microarchitecture) "Haswell (microarchitecture)") and AMD [Radeon R9 295X2](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/AMD_Radeon_Rx_200_series "AMD Radeon Rx 200 series")tops out at over 11.5 TFLOPS at a grand total of US$902.57.[[65]](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/FLOPS#cite_note-65)[[66]](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/FLOPS#cite_note-66) |
| June 2017 | $0.06 | $0.06 | AMD Ryzen 7 1700 & AMD Radeon Vega Frontier Edition | Built using commercially available parts. AMD Ryzen 7 1700 CPU combined with AMD Radeon Vega FE cards in CrossFire tops out at over 50 TFLOPS at just under US$3,000for the complete system.[[67]](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/FLOPS#cite_note-67) |
| October 2017 | $0.03 | $0.03 | Intel Celeron G3930 & AMD RX Vega 64 | Built using commercially available parts. Three [AMD RX Vega 64](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/AMD_RX_Vega_series "AMD RX Vega series") graphics cards provide just over 75 TFLOPS half precision (38 TFLOPS SP or 2.6 TFLOPS DP when combined with the CPU) at ~$2,050 for the complete system.[[68]](https://web.archive.org/web/20171105094854/https://en.wikipedia.org/wiki/FLOPS#cite_note-68) |

The following is a figure we made, of the above list.

Further discussion
------------------

*[Trends in the cost of computing](http://aiimpacts.org/trends-in-the-cost-of-computing/ "Trends in the cost of computing")*

Wikipedia history of GFLOPS costs


Trends in the cost of computing

*Posted 10 Mar 2015*

Computing power available per dollar has probably increased by a factor of ten roughly every four years over the last quarter of a century (measured in FLOPS or MIPS).

Over the past 6-8 years, the rate has been slower: around an order of magnitude every 10-16 years, measured in single precision theoretical peak FLOPS or Passmark’s benchmark scores.

Since the 1940s, MIPS/$ have grown by a factor of ten roughly every five years, and FLOPS/$ roughly every 7.7 years.

Evidence
--------

### Nordhaus

[Nordhaus (2001)](https://web.archive.org/web/20160222082744/http://www.econ.yale.edu/~nordhaus/homepage/prog_083001a.pdf) analyzes the cost of computing over the past century and a half, and produces Figure 1 (though the scale on the vertical axis appears to be off by many orders of magnitude). Much of his data comes from Moravec’s *[Mind Children](http://books.google.com/books/about/Mind_Children.html?id=56mb7XuSx3QC)*(an updated version of the data is [here](https://web.archive.org/web/20161112110101/http://www.transhumanist.com:80/volume1/moravec.htm)). He converts all data points to ‘million standard operations per second’ (MSOPS), where a standard operation is a weighted mixture of multiplications and additions. He says it is approximately equivalent to 1 MIPS under the Dhrystone metric.

He calculates that performance improved at an average rate of 55% per year since 1940. That is, an order of magnitude roughly every five years. However he finds that the average growth rate in different decades differed markedly, with growth since 1980 (until writing in 2001) at around 80% per year, and growth in the 60s and 70s at less than 30% (see figure 2). This would correspond to improving by an order of magnitude every four years in the 80s and 90s.

[!["The progress of computing measured in cost per million standardized operations per second (MSOPS) deflated by the consumer price index." (From Figure 1, Nordhaus, 2001)](http://aiimpacts.org/wp-content/uploads/2015/03/nordhauscomp-copy.png)](http://aiimpacts.org/wp-content/uploads/2015/03/nordhauscomp-copy.png)**Figure 1:** “The progress of computing measured in cost per million standardized operations per second (MSOPS) deflated by the consumer price index.” **Note that the vertical axis appears to be mislabeled—the scale is around seven orders of magnitude different from other sources, such as [Moravec](https://web.archive.org/web/20161112110101/http://www.transhumanist.com:80/volume1/moravec.htm).** (From Figure 1, [Nordhaus, 2001](https://web.archive.org/web/20160222082744/http://www.econ.yale.edu/~nordhaus/homepage/prog_083001a.pdf), p38)

[![Figure xxx: ](http://aiimpacts.org/wp-content/uploads/2015/03/nordhausdecades-copy-1024x859.png)](http://aiimpacts.org/wp-content/uploads/2015/03/nordhausdecades-copy.png)**Figure 2:** From Nordhaus p42,”Rate of Growth of Computer Power by Epoch…Real computer power is the inverse of the decline of real computation costs…”

### Sandberg and Bostrom

[Sandberg and Bostrom (2008)](http://www.fhi.ox.ac.uk/brain-emulation-roadmap-report.pdf) investigate hardware performance trends in their Whole Brain Emulation Roadmap (Appendix B). They plot price performance in MIPS/$ and FLOPS/$, as shown in Figures 3 and 4. They find MIPS/$ grows by a factor of ten every 5.6 years (with a [bootstrap](http://en.wikipedia.org/wiki/Bootstrapping_%28statistics%29) 95% confidence interval of 5.3-5.9), and FLOPs/$ grows by a factor of ten every 7.7 years (with a bootstrap confidence interval of 6.5‐9.2 years).

They find that growth in MIPS/$ slowed in the 70s and 80s, then accelerated again (most recently gaining an order of magnitude every 3.5 years), which is close to what Nordhaus found.

Sandberg and Bostrom’s data is from John McCallum’s [CPU price performance dataset,](http://www.jcmit.com/cpu-performance.htm) which does not appear to draw directly from Moravec’s data.

[![Figure xxx:](http://aiimpacts.org/wp-content/uploads/2015/03/wberm-mips-copy.jpg)](http://aiimpacts.org/wp-content/uploads/2015/03/wberm-mips-copy.jpg)**Figure 3:** Processing power available per dollar over time, measured in MIPS and 2007 US dollars.

[![Figure xxx: ](http://aiimpacts.org/wp-content/uploads/2015/03/wberm-flops-copy-1024x746.jpg)](http://aiimpacts.org/wp-content/uploads/2015/03/wberm-flops-copy.jpg)**Figure 4:** Processing power available per dollar over time, measured in FLOPS using the LINPACK benchmark and in 2007 US dollars

### Rieber and Muehlhauser

Muehlhauser and Rieber (2014) [extended](https://intelligence.org/2014/05/12/exponential-and-non-exponential/#footnote_7_11027) [Koh and Magee’s](http://web.mit.edu/cmagee/www/documents/15-koh_magee-tfsc_functional_approach_studying_technological_progress_vol73p1061-1083_2006.pdf) data on MIPS available per dollar to 2014 (data [not currently] available [here](https://docs.google.com/spreadsheets/d/1qPBpgqxHsqQgcLLXJ5H-4yto9SPQinR4H0f9p5Dh4g4/edit#gid=952780094)). Koh and Magee’s data largely comes from Moravec (like Nordhaus’ above), though they too extended it some. Muehlhauser and Rieber produced Figure 5.

In this data, performance since 1940 appears to be growing by a factor of ten roughly every 5 years (14.2 orders of magnitude in 74 years). In the first fourteen years of this century, log(MIPS/$) grew from roughly -0.7 to 2.8, which corresponds to one order of magnitude every four years (or 77% growth per year).

[![](http://aiimpacts.org/wp-content/uploads/2015/03/image-5.png)](http://aiimpacts.org/wp-content/uploads/2015/03/image-5.png)**Figure 5:** Rieber and Muehlhauser’s MIPS/$ data (modified to fix typo).

### Wikipedia

[Wikipedia](http://en.wikipedia.org/wiki/FLOPS#Hardware_costs) has a small list of hardware configurations that authors claim produce gigaFLOPS efficiently, along with their prices at different times in recent history. Their data does not appear to cite other sources mentioned above.

[Here](http://aiimpacts.org/wikipedia-history-of-gflops-costs/) is their table, as of March 2 2015. Figure 6 shows inflation adjusted costs of gigaFLOPS over time, taken from the table. The examples in the table were apparently selected as follows:

> The “cost per GFLOPS” is the cost for a set of hardware that would theoretically operate at one billion floating-point operations per second. During the era when no single computing platform was able to achieve one GFLOPS, this table lists the total cost for multiple instances of a fast computing platform which speed sums to one GFLOPS. Otherwise, the least expensive computing platform able to achieve one GFLOPS is listed.
>
>

We find this table dubious. It lacks many citations, and the citations it has frequently lack detail. For instance, the claims that the collections of hardware specified produce a GFLOPS are often unsubstantiated. We spent around thirty minutes trying to substantiate the 2015 figure, to no avail. The figure is more than an order of magnitude cheaper than [current FLOPS prices](http://aiimpacts.org/current-flops-prices/ "Current FLOPS prices") we found.

In this data, the price of a gigaFLOPS falls by an order of magnitude roughly every four years (14 orders of magnitude in 54 years is 3.9 years per order of magnitude). Since 1997, each order of magnitude only took three years (5.7 orders of magnitude in 18 years). Note that there is very little data before 1997.

[![Price of GFLOPS in different years, adjusted to 2013 US dollars.](http://aiimpacts.org/wp-content/uploads/2014/12/price-of-gflops.png)](http://aiimpacts.org/wp-content/uploads/2014/12/price-of-gflops.png)**Figure 6:** Price of GFLOPS in different years according to Wikipedia, adjusted to 2013 US dollars.

### Short term trends

*Main article:* [***Recent trends in the cost of computing***](http://aiimpacts.org/recent-trend-in-the-cost-of-computing/)

The cheapest hardware prices (for single precision FLOPS/$) are on track to fall by around an order of magnitude every 10-16 years, based on data from around 2011-2017. There was no particular sign of slowing between 2011 and 2017.

Summary
-------

We have looked at four efforts to measure long term hardware price performance trajectories. Two of them are based on Moravec’s earlier effort, while the other two appear to be more independent (though we suspect still draw on similar sources). Two investigations measured (G)FLOPS, two measured MIPS, and one measured MSOPS.

Results seem fairly consistent in recent decades, and for MIPS/$ in the longer run. There is insufficient data on FLOPS in the long run to check consistency. All four estimates of growth later than the 1990s produce 3.5-4 years as the time for price performance to to grow an order of magnitude (we did not include an estimate for recent years from Sandberg and Bostrom’s FLOPS data, since they did not make one and it was not straightforward to make one ourselves).[1](https://aiimpacts.org/trends-in-the-cost-of-computing/#easy-footnote-bottom-1-448 "This is consistent with Sandberg and Bostrom’s estimate of the relationship between FLOPS and MIPS: ‘Fitting a relationship suggests that FLOPS scales as MIPS to the power of 0.89, i.e. slightly slower than unity’ (p89).") Though note that these measures are from different spans within that period, and use different benchmarks (two were MIPS, one FLOPS, one MSOPS). Only Rieber and Muehlhauser and Wikipedia have data after 2002. Though they give similar recent growth figures, it is not clear how consistent they are: Rieber and Muehlhauser’s data appears to decline sharply in the last few years, and appears to only use CPUs, while the Wikipedia data is fairly even, and moves to GPUs in later years.

If we take an MSOPS to be more or less equivalent to a MIPS (as Nordhaus claims), then growth in MIPS since the 1940s is fairly consistent across studies, gaining an order of magnitude roughly every 5 years (Nordhaus), 5 years (Rieber and Muehlhauser) or 5.6 years (Sandberg and Bostrom). Note that the former two draw on similar data.

Our two estimates of long run growth in FLOPS/$ differ substantially: we have gained an order of magnitude either every 4 years or every 7.7 years. However the four year estimate comes from Wikipedia, which only has two entries prior to 1990, while Sandberg and Bostrom have on the order of hundreds of entries from that period. Thus we rely on Sanberg and Bostrom here, and estimate FLOPS grow by an order of magnitude every 7.7 years.

Prior to the 1940s, growth appears to be ambiguous and small. It looks like 2.4 orders of magnitude over forty eight years in Rieber and Muehlhauser’s figure, for an order of magnitude every 20 years. Nordhaus measures it as negative.

Further work
------------

Further work on this subject might:

* Check Moravec’s data, as it appears to be widely cited and reused (perhaps just check consistency between the fraction of data from Moravec and that added later from another source in existing datasets).
* Separate different types of computers (e.g. treat desktop CPUs, supercomputers, and GPUs separately)
* Find other datasets and analyses
* Combine all of the datasets into one
* Produce more relevant data
* Construct and measure a more relevant benchmark

Trends in the cost of computing


What’s up with nuclear weapons?

*By Katja Grace, 27 February 2015*

When nuclear weapons were first built, the explosive power you could extract from a tonne of explosive [skyrocketed](http://aiimpacts.org/discontinuity-from-nuclear-weapons/ "Discontinuity from Nuclear Weapons"). But why?

Here’s a guess. Until nuclear weapons, explosives were based on chemical reactions. Whereas nuclear weapons are based on nuclear reactions. As you can see from the below table of specific energies and energy densities I got (and innocuously shortened) from [Wikipedia](http://en.wikipedia.org/wiki/Energy_density#Energy_densities_of_common_energy_storage_materials), the characteristic scale of nuclear energy stored in things is about a hundred thousand times higher than that of chemical energy stored in things (by mass). And in particular, there are an empty three orders of magnitude between the most chemical energy packed into a thing and the least nuclear energy packed into a thing. This is perhaps to do with the [fact](http://environ.andrew.cmu.edu/m3/s3/06forces.shtml) that chemical reactions exploit the electromagnetic force, while nuclear reactions exploit the strong fundamental force.

| Storage material | Energy type | Specific energy (MJ/kg) | Energy density (MJ/L) | Direct uses |
| --- | --- | --- | --- | --- |
| **[Uranium](http://en.wikipedia.org/wiki/Uranium "Uranium") (in [breeder](http://en.wikipedia.org/wiki/Breeder_reactor "Breeder reactor"))** | [Nuclear](http://en.wikipedia.org/wiki/Nuclear_power "Nuclear power") fission | 80,620,000[[2]](http://en.wikipedia.org/wiki/Energy_density#cite_note-whatisnuclear-2) | 1,539,842,000 | Electric power plants (nuclear reactors), industrial process heat (to drive chemical reactions, water desalination, etc.) |
| **[Thorium](http://en.wikipedia.org/wiki/Thorium-based_nuclear_power "Thorium-based nuclear power") (in [breeder](http://en.wikipedia.org/wiki/Breeder_reactor "Breeder reactor"))** | [Nuclear](http://en.wikipedia.org/wiki/Nuclear_power "Nuclear power") fission | 79,420,000[[2]](http://en.wikipedia.org/wiki/Energy_density#cite_note-whatisnuclear-2) | 929,214,000 | Electric power plants (nuclear reactors), industrial process heat |
| **[Tritium](http://en.wikipedia.org/wiki/Tritium#Decay "Tritium")** | [Nuclear](http://en.wikipedia.org/wiki/Nuclear_power "Nuclear power") decay | 583,529 |  ? | Electric power plants (nuclear reactors), industrial process heat |
| **[Hydrogen (compressed)](http://en.wikipedia.org/wiki/Compressed_hydrogen "Compressed hydrogen")** | [Chemical](http://en.wikipedia.org/wiki/Chemical_energy#Chemical_energy "Chemical energy") | 142 | 5.6 | Rocket engines, automotive engines, grid storage & conversion |
| **[methane](http://en.wikipedia.org/wiki/Methane "Methane") or [natural gas](http://en.wikipedia.org/wiki/Natural_gas "Natural gas")** | [Chemical](http://en.wikipedia.org/wiki/Chemical_energy#Chemical_energy "Chemical energy") | 55.5 | 0.0364 | Cooking, home heating, automotive engines, lighter fluid |
| **[Diesel](http://en.wikipedia.org/wiki/Diesel_fuel "Diesel fuel") / [Fuel oil](http://en.wikipedia.org/wiki/Fuel_oil "Fuel oil")** | Chemical | 48 | 35.8 | Automotive engines, power plants[[3]](http://en.wikipedia.org/wiki/Energy_density#cite_note-AFDC-3) |
| **[LPG](http://en.wikipedia.org/wiki/Liquefied_petroleum_gas "Liquefied petroleum gas") (including [Propane](http://en.wikipedia.org/wiki/Propane "Propane")/ [Butane](http://en.wikipedia.org/wiki/Butane "Butane"))** | Chemical | 46.4 | 26 | Cooking, home heating, automotive engines, lighter fluid |
| **[Jet fuel](http://en.wikipedia.org/wiki/Jet_fuel "Jet fuel")** | Chemical | 46 | 37.4 | Aircraft |
| **[Gasoline](http://en.wikipedia.org/wiki/Gasoline "Gasoline") (petrol)** | Chemical | 44.4 | 32.4 | Automotive engines, power plants |
| **[Fat](http://en.wikipedia.org/wiki/Fat "Fat") (animal/vegetable)** | Chemical | 37 | 34 | Human/animal nutrition |
| **[Ethanol fuel](http://en.wikipedia.org/wiki/Ethanol_fuel "Ethanol fuel")** (E100) | Chemical | 26.4 | 20.9 | Flex-fuel, racing, stoves, lighting |
| **[Coal](http://en.wikipedia.org/wiki/Coal "Coal")** | Chemical | 24 |  | Electric power plants, home heating |
| **[Methanol fuel](http://en.wikipedia.org/wiki/Methanol_fuel "Methanol fuel")** (M100) | Chemical | 19.7 | 15.6 | Racing, model engines, safety |
| **[Carbohydrates](http://en.wikipedia.org/wiki/Carbohydrate "Carbohydrate")(including sugars)** | Chemical | 17 |  | Human/animal nutrition |
| **[Protein](http://en.wikipedia.org/wiki/Protein_in_nutrition "Protein in nutrition")** | Chemical | 16.8 |  | Human/animal nutrition |
| **[Wood](http://en.wikipedia.org/wiki/Wood_fuel "Wood fuel")** | Chemical | 16.2 |  | Heating, outdoor cooking |
| **[TNT](http://en.wikipedia.org/wiki/Trinitrotoluene "Trinitrotoluene")** | Chemical | 4.6 |  | Explosives |
| **[Gunpowder](http://en.wikipedia.org/wiki/Gunpowder "Gunpowder")** | Chemical | 3 |  | Explosives |
|  |  |  |  |  |

Thus it seems very natural that the first, lousiest, nuclear weapons that anyone could invent would be much more explosive than any chemical weapon ever known. The power of explosives is mostly a matter of physics, and physics contains discontinuities, for some reason.

But this doesn’t quite explain it. Consider cars. Turbojet propelled cars seem just fundamentally capable of greater speeds than internal combustion engine propelled cars. But the first [turbojet cars](https://aiimpacts.org/feed/ed_record#1963.E2.80.93present_.28jet_and_rocket_propulsion.29) that were faster than internal combustion cars were not much faster—it looks like they just had a steeper trajectory, which passed other cars and kept climbing. I’m not sure what caused this pattern in the car case specifically, but I hear it’s common. Maybe people basically know what current technology is capable of, and introduce new things as soon as they can be done at all, rather than as soon as they can be done well.

Anyway, we could imagine the same thing happening with nuclear weapons: even if nuclear power was fundamentally very powerful, the first nukes could have made use of it very badly, exploding like a weak chemical explosive the first times, but being quickly improved.

But that isn’t how nuclear weapons work. For a nuclear weapon to be less explosive per mass it would need to contain less fissile material, be smaller (so the outside casing is more of the mass, and so that fewer neutrons hit other atoms), or be less well contained (so fewer neutrons hit other atoms). But to get a nuclear explosion going at all, you need to get enough neutrons to hit other atoms that the chain reaction starts. Nuclear weapons have a ‘[critical mass](http://en.wikipedia.org/wiki/Critical_mass)‘. I’m not sure how much less powerful the first nuclear weapons could easily have been than they were, but measly inexplosive nuclear weapons were basically out.

So the first nuclear weapons had to be much more explosive than the chemical explosives they replaced, because they were based on much more powerful reactions, and primitive nuclear weapons weren’t an option.

So nuclear weapons were basically guaranteed to revolutionize explosives in a single hop: even if humanity had known about nuclear reactions for hundreds of years, and put a tiny amount of effort into nuclear weapons research each year, humanity would never have seen feeble, not-much-better-than-TNT type nuclear weapons. There would just have been no nuclear weapons, and then at some point there would have been powerful nuclear weapons.

It is somewhat interesting that this is not what happened. Physicists mostly came to believe nuclear weapons were plausible from about 1939, and within a few years America spent nominal [$19Bn](http://blog.nuclearsecrecy.com/2013/05/17/the-price-of-the-manhattan-project/) (roughly 1% of [1943 GDP](http://useconomy.about.com/od/GDP-by-Year/a/US-GDP-History.htm), but spread over a few years) on nuclear weapons, and built some. So our story is that progress in explosives was very slow, and then America spent a huge pile of money on it, and then it was very fast, but the progress was independent of the massive influx of funding.

That sounds surprising. But perhaps the influx of funding was *because of* the large natural discontinuity visible in the distance? Why would you ever spend small amounts of money every year, if it was clear at the outset that you had to spend a gajillion dollars to get anywhere? If there wasn’t much requirement for serial activities, probably you would just save it up and spend it in one go. America didn’t save it up though—they tried to build nuclear weapons basically as soon as they realized it was feasible at all. So it looks like nuclear weapons were just discovered after it was cost-effective to build them.

But if it was immediately cost-effective to build nuclear weapons thousands of times more powerful than other bombs, then isn’t the requirement that nuclear weapons be fairly powerful irrelevant to the spending? If it was worth building powerful bombs immediately, then what does it matter if it is possible to build lesser weapons? Not really, because cost-effectiveness is relative. If is only possible to buy toothpaste in a large bucket, you will probably pay for it, and it will have been a good deal. However if it’s also available in small tubes, then the same bucket is probably a bad deal.

Similarly, if nuclear weapons must be powerful, then there’s a decent chance that as soon as they are discovered it will be cost-effective to spend a lot on them and make them so. However if they can come in many lower levels of quality, the same large amount of spending may not be cost effective, because it will often be better to spend an intermediate amount.

So a requirement that nuclear weapons be very explosive when they are first built could at least partly explain the huge amount of spending. And the inherently large amounts of energy available from nuclear reactions still seems relevant: any given amount of development will be cost-effective when it is more costly, if it is more effective compared to the alternative.

This also appears to fit in with an explanation of the further coincidence that there happened to be a huge war at the time. That is, the war made all military technologies more cost-effective, and thus made it more likely that when nuclear weapons became feasible to develop, they would already be cost-effective. However the war also makes it more likely that high quality weapons would already be cost-effective compared to cheaper counterparts, thus partly undermining the proposal that the large expenditure was due in part to nuclear weapons requiring a minimal level of quality.

Here’s another plausible explanation for the large expense: because of their extreme explosiveness, nuclear weapons were very cost-effective at the time they were first considered. That is, they could have been produced a lot more cheaply than they were. However, due to the war, America was willing to pay a lot to make them come faster. In particular, America was willing to keep paying to make them come faster up until the point when they were roughly as cost-effective as older weapons, taking into account the upfront cost of making them come faster. This would explain the large amount of spending, and perhaps also why it aligned so well with what America could barely afford. It also explains why nuclear weapons appear to have been very roughly as cost-effective as older weapons. However on its own, it seems to leave the large amount of spending and the large amount of progress as coincidences.

In other ways, this story is in line with what I know about the development of nuclear weapons. For instance, that enriching uranium via several different methods in parallel was [around half](http://blog.nuclearsecrecy.com/2013/05/17/the-price-of-the-manhattan-project/) of the cost of the Manhattan project, and that the project was a lot more expensive than other countries’ later nuclear weapons projects.

Perhaps the inherent explosiveness of nuclear weapons made them very cost-effective, and thus able to be sped up a lot and still be cost-effective? (Thus connecting the expense with the explosiveness) But if nuclear weapons had been too expensive already to speed up much, it seems we would have seen a similar amount of spending (or more) over a somewhat longer time. So on this story it seems the heavy spending didn’t cause the high explosiveness, and the high explosiveness (and thus cheapness) didn’t seem to cause the steep spending.

It seems there was probably one coincidence however: a physics discovery leading to weapons of unprecedented power was made just before the largest war in history, and it’s hard to see how the war and the discovery were related, unless history was choreographed to make Leo Szilard’s life interesting. Perhaps the weapons and the war are related because nuclear weapons caused us to think of WWII as a large war by averting later wars? But the World Wars [really were quite large](http://en.wikipedia.org/wiki/List_of_wars_and_anthropogenic_disasters_by_death_toll) compared to wars in slightly earlier history, rather than just the last in a trend of growing conflicts. If there is at least one coincidence anyway, perhaps it doesn’t matter whether the massive expense is explained by the unique qualities of nuclear weapons or merely by the war inspiring haste.

In sum, my guesses: nuclear weapons represented abrupt progress in explosiveness because of a discontinuity inherent in physics, and because ineffective nuclear weapons weren’t feasible.  Coincidentally, at the same time as nuclear weapons were discovered, there was a large war. America spent a lot on nuclear weapons for a combination of reasons. Nuclear explosions were inherently unusually powerful, and so could be cost-effective while still being very expensive. They also required investment on a large scale, so were probably invested in at an unusually large scale. America probably also spent a lot more on nuclear weapons to get them quickly, because they were so cost-effective under the circumstances.

My guesses are pretty speculative however, and I’m not an expert here. Further speculation, or well-grounded theorizing, is welcome.

(image: *[Oak Ridge Y-12 Alpha Track](http://commons.wikimedia.org/wiki/File:Oak_Ridge_Y-12_Alpha_Track.jpg)*)

What’s up with nuclear weapons?


Possible Empirical Investigations

In the course of our work, we have noticed a number of empirical questions which bear on our forecasts and might be (relatively) cheap to resolve. In the future we hope to address some of these.

* [Our partial list of investigations into forecasting AI timelines](http://aiimpacts.org/research-topic-hardware-software-and-ai/ "Research topic: Hardware, software and AI")
* [Our list of investigations that bear on ‘multipolar’ AI scenarios](http://aiimpacts.org/multipolar-research-projects/ "List of multipolar research projects")
* Look at the work of ancient or enlightenment mathematicians and control for possible selection effects in [this analysis](https://sites.google.com/site/aiimpactslibrary/resolutions-of-mathematical-conjectures) of historical mathematical conjectures.
* Look for historical characterizations of the AI problem, and try to obtain unbiased (though uninformed) breakdowns of the problem which could be used to gauge progress.
* Identify previous examples of technological projects with clear long-term goals, and then produce estimates of the time required to achieve those goals to varying degrees.
* Analyze the performance of different versions of software for benchmark problems, like SAT solving or chess, and determine the extent to which hardware and software progress facilitated improvement.
* Obtain a clearer picture of the extent to which historical developments in neuroscience have played a meaningful role in historical progress in AI. Our impression is that this influence has been minimal, but this judgment might be attributable to hindsight bias.
* In the field of AI, estimate the ratio of spending on hardware to spending on researchers.
* Estimate the change in inputs in mathematicians, scientists, or engineers, as a complement to estimates for rates of progress in those fields.
* Estimate the historical and present size of the AI field, ideally with plausible adjustments for quality (for example performing in-depth investigations for a small number of random samples, perhaps invoking expert opinion) and using these as a basis for quality-adjustments.
* [Luke Muehlhauser](http://lukemuehlhauser.com/some-studies-which-could-improve-our-strategic-picture-of-superintelligence/) and the [Future of Life Institute](http://futureoflife.org/static/data/documents/research_survey.pdf)‘s section on forecasting both list further projects.

Unfortunately this is an incomplete list (even of the ideas which have struck as promising during this project). We are beginning to flesh it out further in our aforementioned [list of projects bearing on AI timelines](http://aiimpacts.org/research-topic-hardware-software-and-ai/ "Research topic: Hardware, software and AI").

Possible Empirical Investigations


Research topic: Hardware, software and AI

This is the first in a sequence of articles outlining research which could help forecast AI development.

---

Interpretation
--------------

Concrete research projects are in boxes. ∑5 ∆8  means we guess the project will take (very) roughly five hours, and we rate its value (very) roughly 8/10.

Most projects could be done to very different degrees of depth, or at very different scales. Our time cost estimates correspond to a size that we would be likely to intend if we were to do the project. Value estimates are merely ordinal indicators of worth, based on our intuitive sense, and unworthy of being taken very seriously.

---

1. How does AI progress depend on hardware and software?
--------------------------------------------------------

At a high level, AI improves when people make better software, when they can run it on better hardware, when they gather bigger, better training sets, etc. This makes present-day hardware and software progress a natural place to look for evidence about when advanced AI will arrive. In order to interpret any such data however, it is important to know how these pieces fit together. For instance, is the progress we see now mostly driven by hardware progress, or software progress? Can the same level of performance usually be achieved by widely varying mixtures of hardware and software? Does progress on software depend on progress on hardware?

It is important to understand the relationship between hardware, software and AI for several reasons. If hardware progress is the main driver of AI progress, then quite different evidence would tell us about AI timelines than if software is the main driver. Thus different research is valuable, and different timelines are likely. Many people base their AI predictions on hardware progress, while others decline to, so it would be broadly useful to know whether one should. We also expect understanding here to be generally useful.

So we think research in this direction seems valuable. We also think several projects seem tractable. Yet little appears to have been done in this direction. Thus this topic seems a high priority.

### 1.1 How does AI progress depend qualitatively on hardware and software progress?

For instance, will human-level AI appear when we have both a certain amount of hardware, and certain developments in software? Or can hardware and software substitute for one another? Substitution seems a natural model of the relationship between hardware and software, since anecdotally many tasks can be done by low quality software and lots of hardware, or by high quality software and less hardware. However the extent of this is unclear. This kind of model is also [not commonly used](http://aiimpacts.org/how-ai-timelines-are-estimated/ "How AI timelines are estimated") in estimating [AI timelines](http://aiimpacts.org/list-of-analyses-of-time-to-human-level-ai/ "List of Analyses of Time to Human-Level AI"), so judging whether it should be might be a useful contribution. Having a good model would also bear on the priority of other research directions. As far as we know, this issue has received almost no attention. It seems moderately tractable.

> **1.1.A Evaluate qualitative models of the relationships between hardware, software and AI** ∑30 ∆5
>
> One way to approach the question of qualitative relationships is to assume some model, and work on projects such as those in 1.2 that measure quantitative details of the model, then revise the model if the measurements don’t make sense in it. Before that step, we might spend a short time detailing plausible models, and examining empirical and theoretical evidence we might already have, or could cheaply find. If we were going to follow up with empirical research, we would think about what evidence we would expect the research to reveal, given alternative models.
>
> ~
>
> For instance, we find the hardware-software [indifference curve](http://en.wikipedia.org/wiki/Indifference_curve) model described briefly above (and outlined better in a [blog post](http://aiimpacts.org/how-ai-timelines-are-estimated/ "How AI timelines are estimated")) plausible. Here are some ways it might be inadequate, that we might consider in evaluating it:
>
>
> * ‘Hardware’ and ‘software’ are not sufficiently measurable entities for a ‘level’ of each in some domain to produce a stable level of performance.
> * Performance depends strongly on other factors, e.g. exactly what kind of hardware and software progress you make, unique details of the software being developed, training data available.
> * Different problem types, and different performance metrics on them have different kinds of behavior
> * There are ‘indifference curves’ in a sense but they are not sufficiently consistent to be worth reasoning about.
> * Humanity’s technological progress is not well characterized by an expanding rectangle of feasible hardware and software levels, but more as a complicated region of feasible combinations.
>
>
>

### 1.2 How much do marginal hardware and software improvements alter AI performance?

As mentioned above, this question is key to determining which other investigations are worthwhile. Naturally, it could also change our timelines substantially. Thus this question seems thus important to resolve. We think the projects here are particularly tractable, though not particularly cheap. For all of these projects, we would probably choose a specific set of benchmarks on particular problems to focus on. We might do multiple of these projects on the same set of benchmarks, to trace a more complete picture.

> **1.2.A Search for natural experiments combining modern hardware and early software approaches or vice versa.**∑80 ∆7
>
> For instance, we might find early projects with very large hardware budgets, or recent projects with [intentionally restricted hardware](http://1kchess.an3.es/). Where these were tested on commonly used benchmarks, we can use them to map out the broad contributions of hardware and software to progress. For instance, if very small chess programs today run better than old chess programs which used similar (but then normal) amounts of hardware, then the difference between them can be attributed to improving software, roughly.
>
>

> **1.2.B Apply a modern understanding of software to early hardware** ∑2,000 ∆9
>
> Choose a benchmark problem that people worked on in the past, e.g. in the 1980s. Use a modern understanding of AI to solve the problem again, still using 1980’s hardware. Compare this to how researchers did in the 1980’s. This project requires substantial time from at least one AI researcher. Ideally they would spend a similar amount of effort as the past researchers did, so it may be worth choosing a problem where it is known that an achievable level of effort was applied in the past.
>
>

> **1.2.C Apply early software understanding to modern hardware** ∑2,000 ∆8
>
> Using contemporary hardware and a 1970’s or 1980’s understanding of connectionism, observe the extent to which a modern AI researcher (or student) could replicate contemporary performance on benchmark AI problems. This project is relatively expensive, among those we are describing. It requires substantial time from collaborators with a historically accurate minimal understanding of AI. Students may satisfy this role well, if their education is incomplete in the right ways. One might compare to the work of similar students who had also learned about modern methods.
>
>

> **1.2.D Measure marginal effects of hardware and software in existing performance trends** ∑100 ∆8
>
> Often the same software can be used with modest changes in hardware, so changes in performance from hardware over small margins can be measured. Improved software is also often written to be run on the same hardware as earlier software, so changes in performance from software alone can be measured over moderate margins. Thus we can often estimate these marginal changes from looking at existing performance measurements.
>
> ~
>
> We can also look at overall progress over time on some applications, and factor out what we know about hardware or software change, assuming it is close to the marginal values measured by the above methods. For instance, we can see how much individual Go programs improve with more hardware, and then we can look at longer term improvements in computer Go, and guess how much of that improvement came from hardware, given our earlier estimate of marginal improvement from hardware. In general these estimates will be less valid over larger distances, as the impact of hardware or software diverge from their marginal impact, and because arbitrary of hardware and software can’t generally be combined without designing the software to make use of the hardware. [Grace 2013](http://intelligence.org/files/AlgorithmicProgress.pdf) includes some work this project.
>
>

> **1.2.E Interview AI researchers on the relative importance of hardware and software in driving the progress they have seen.** ∑20 ∆7
>
> AI researchers likely have firsthand experience regarding how hardware and software contribute to overall progress within the vicinity of their own work. This project will probably give relatively noisy estimates, but is very cheap compared to others described here. One could just ask for views on this question, and supporting anecdotes, or devise a more structured questionnaire beforehand.
>
>

### 1.3 How do hardware and software progress interact?

Do hardware and software progress relatively independently, or for instance do advances in hardware encourage advances in software? This might change how we generally expect software progress to proceed, and what combinations of hardware and software we expect to first produce human-level AI. We are likely to get some information about this from other projects looking at historical performance data e.g. 1.2.D. For instance, if overall progress is generally proportional to hardware progress, even as hardware progress varies, then this would be suggestive. Below are further possibilities.

> **1.3.A Find natural experiments**∑80 ∆4
>
> Search for performance data from cases where hardware being used for an application was largely constant then shifted upward at some point. Such cases are probably hard to find, and hard to interpret when found. However, a short search for them may be worthwhile.
>
>

> **1.3.B Interview researchers** ∑20 ∆7
>
> If hardware tends to affect software research, it is likely that researchers notice this, and can talk about it. This seems a cheap and effective method of learning qualitatively about the topic. This project should probably be combined with 1.2.E.
>
>

> **1.3.C Consider plausible models** ∑10 ∆5
>
> This is a short theoretical project that would benefit from being done in concert with 1.3.B (interview researchers), since researchers probably have a relatively good understanding of which models are plausible, and we are likely to ask better questions of them if we have thought about the topic. This project should probably be combined with 1.1.A.
>
>


Multipolar research questions

*By Katja Grace, 11 February 2015*

The [Multipolar AI workshop](http://aiimpacts.org/event-multipolar-ai-workshop-with-robin-hanson/ "Event: Multipolar AI workshop with Robin Hanson") we ran a fortnight ago went well, and we just put up a [list of research projects](http://aiimpacts.org/multipolar-research-projects/ "List of multipolar research projects") from it. I hope this is helpful inspiration to those of you thinking about applying to the new [FLI grants](http://futureoflife.org/grants/large/initial) in the coming weeks.

Thanks to the many participants who contributed ideas!

*(Image by [Evan Amos](http://en.wikipedia.org/wiki/Colored_pencil#/media/File:Colored-Pencils.jpg))*

Multipolar research questions


List of multipolar research projects

This list currently consists of research projects suggested at the [Multipolar AI workshop](http://aiimpacts.org/event-multipolar-ai-workshop-with-robin-hanson/ "Event: Multipolar AI workshop with Robin Hanson") we held on January 26 2015.

Relatively concrete projects are marked [concrete]. These are more likely to already include specific questions to answer and feasible methods to answer them with. Other ‘projects’ are more like open questions, or broad directions for inquiry.

Projects are divided into three sections:

1. Paths to multipolar scenarios
2. What would happen in a multipolar scenario?
3. Safety in a multipolar scenario

Order is not otherwise relevant. The list is an inclusive collection of the topics suggested at the workshop, rather than a prioritized selection from a larger list.

Luke Muehlhauser’s [list of ‘superintelligence strategy’ research questions](http://lukemuehlhauser.com/some-studies-which-could-improve-our-strategic-picture-of-superintelligence/) contains further suggestions.

List
----

### Paths to multipolar scenarios

**1.1 If we assume that AI software is similar to other software, what can we infer from observing contemporary software development? [concrete]**For instance, is progress in software performance generally smooth or jumpy? What is the distribution? What are typical degrees of concentration among developers? What are typical modes of competition? How far ahead does the leading team tend to be to their competitors? How often does the lead change? How much does a lead in a subsystem produce a lead overall? How much do non-software factors influence who has the lead? How likely is a large player like Google—with its pre-existing infrastructure—to be the frontrunner in a random new area that they decide to compete in?

A large part of this project would be collecting what is known about contemporary software development. This information would provide one view on how AI progress might plausibly unfold. Combined with several such views, this might inform predictions on issues like abruptness, competition and involved players.

**1.2 If the military is involved in AI development, how would that affect our predictions? [concrete]**This is a variation on 1.1, and would similarly involve a large component of reviewing the nature of contemporary military projects.

**1.3 If industry were to be largely responsible for AI development, how would that affect our predictions? [concrete]**This is a variation on 1.2, and would similarly involve a large component of reviewing the nature of contemporary industrial projects.

**1.4** **If academia were to be largely responsible for AI development, how would that affect our predictions? [concrete]**This is a variation on 1.2, and would similarly involve a large component of reviewing the nature of contemporary academic projects.

**1.5 Survey AI experts on the likelihood of AI emerging in the military, business or academia, and on the likely size of a successful AI project.  [concrete]**

**1.6 Identify considerations that might tip us between multipolar and unipolar scenarios.**

**1.7 To what extent will AGI progress be driven by developing significantly new ideas?**1.1 may bear on this. It could be approached in other ways, for instance asking AI researchers what they expect.

**1.8 Run prediction markets on near-term questions, such as rates of AI progress, which inform our long-run expectations. **[concrete]****

**1.9 Collect past records of ‘lumpiness’ of AI success. [concrete]** That is, variation in progress over time. This would inform expectations of future lumpiness, and thus potential for single projects to gain a substantial advantage.

### What would happen in a multipolar scenario?

**2.1 To what extent do values prevalent in the near-term affect the long run, in a competitive scenario?**One could consider the role of values over history so far, or examine the ways in which the role of values may change in the future. One could consider the degree of instrumental convergence between actors (e.g. firms) today, and ask how that affects long-term outcomes. One might also consider whether non-values mental features might become locked in in a way that produces similar outcomes to particular values being influential. e.g. priors or epistemological methods that make a particular religion more likely

**2.2 What other factors in an initial scenario are likely to have long-lasting effects?** For instance social institutions, standards, and locations for cities.

**2.3 What would AI’s value in a multipolar scenario?**We can consider a range of factors that might influence AI values:

1. The nature of the transition to AI
2. Prevailing institutions
3. The extentto which AI values become static, as compared to changing human values
4. What values do humans want AI’s to have
5. Competitive dynamics

There is a common view that a multipolar scenario would be better in the long run than a hegemonic ‘unfriendly AI’. This project would inform that comparison.

**2.4 What are the prospects for human capital-holders?**In a simple model, humans who own capital might become very wealthy during a transition to AI. On a classical economic picture, this would be a critical way for humans to influence the future. Is this picture plausible? Evaluate the considerations.

1. What are the implications of capital holders doing no intellectual work themselves?
2. **[concrete]** What does the existing literature on principal-agent problems suggest about multipolar AI scenarios?
3. **[concrete]** Could humans maintain investments for significant periods of their lives, if during that time aeons of subjective time passes for faster moving populations? (i.e. is it plausible to expect to hold assets through millions of years of human history?) Investigate this via data on past expropriations

**2.5 Identify risks distinctive to a multipolar scenario, or which are much more serious in a multipolar scenario.**

For instance:

* Evolutionary dynamics bring an outcome that nobody desired initially
* The AIs are not well integrated into human society, and consequently cause or allow destruction to human society
* The AIs—integrated or not—have different values, and most of the resources end up being devoted to those values

**2.6 Choose a specific multipolar scenario and try to predict its features in detail. [concrete]**Base this on the basic changes we know would occur (e.g. minds could be copied like software), and our best understanding of social science.

Specific instances:

1. Brain emulations (Robin Hanson is working on this in an upcoming book)
2. Brain emulations, without the assumption that software minds are opaque
3. One can buy maximally efficient software for anything you want; everything else is the same
4. AI is much like contemporary software (see 1.1).

**2.7 How would multipolar AI change the nature and severity of violent conflict?** For instance, conflict between states.

**2.8 Investigate the potential for AI-enforced rights.**Think about how to enforce property rights in a multipolar scenario, given advanced artificial intelligence to do it with, and the opportunity to prepare ahead of time. Can you create programs that just enforce deals between two parties, but do nothing else? If you create AI with this stable motivational structure, possessed by many parties, how does this change the way that agents that interact? How could such a system be designed?

**2.9 What is the future of democracy in such a scenario?**In a world where resources can rapidly and cheaply be turned into agents, the existing assignment of a vote per person may be destructive and unstable.

**2.10 How does the lumpiness of economic outcomes vary as a function of the lumpiness of origins?** For instance, if one team creates brain emulations years before others, would that group have and retain extreme influence?

**2.11 What externalities can we foresee, in computer security?**That is, will people invest less (or more) in security than is socially optimal?

**2.12 What externalities can we foresee in AI safety generally?**

**2.13 To what extent can artificial agents make more effective commitments, or more effectively monitor commitments, than humans?** How does this change competitive dynamics? What proofs of properties of one’s source code may be available in the future?

### Safety in a multipolar scenario

**3.1 Assess the applicability of general AI safety insights to multipolar scenarios. [concrete]** How useful are capability control methods, such as boxing, stunting, incentives, or tripwires in a multi-polar scenario? How useful are motivation selection methods, such as direct specification, domesticity, indirect normatively, augmentation in a multipolar scenario?

**3.2 Would selective pressures strongly favor the existence of goal-directed agents, in a multipolar scenario where a variety of AI designs are feasible?**

**3.3 Develop a good model for the existing computer security phenomenon where nobody builds secure systems, though they can.** **[concrete]** Model the long-run costs of secure and insecure systems, given distributions of attacker sophistication and possibility for incremental system improvement. Determine the likely situation various future scenarios, especially where computer security is particularly important.

**3.4 Do paradigms developed for nuclear security and biological weapons apply to AI in a multi-polar scenario? [concrete]** For instance, could similar control and detection systems be used?

**3.5 What do the features of computer security systems tell us about how multipolar agents might compete?**

**3.8 What policies could help create more secure computer systems?**For instance, the onus being on owners of systems to secure them, rather than on potential attackers to avoid attacking.

**3.9 What innovations (either in AI or coinciding technologies) might reduce principal-agent problems?**

**3.10 Apply ‘reliability theory’ to the problem of manufacturing trustworthy hardware.**

**3.11 How can we transition in an economically viable way to hardware that we can trust is uncorrupted?**At present, we must assume that the hardware is uncorrupted upon purchase, but this may not be sufficient in the long run.

List of multipolar research projects


How AI timelines are estimated

*By Katja Grace, 9 February 2015*

A natural approach to informing oneself about when [human-level AI](http://aiimpacts.org/human-level-ai/ "Human-Level AI") will arrive is to check what experts who have already investigated the question say about it. So we made [this list](http://aiimpacts.wpengine.com/list-of-analyses-of-time-to-human-level-ai/ "List of Analyses of Time to Human-Level AI") of analyses that we could find.

It’s a short list, though the bar for ‘analysis’ was low. Blame for the brevity should probably be divided between our neglect of worthy entries and the world’s neglect of worthy research. Nonetheless we can say interesting things about the list.

About half of the estimates are based on extrapolating hardware, usually to something like ‘human-equivalence’. A stylized estimate along these lines might run as follows:

1. Calculate how much computation the brain does.
2. Extrapolate the future costs for computing hardware (it goes downward, fast)
3. Find the point in the computing hardware cost trajectory where brain-equivalent hardware (1) becomes pretty cheap, for some value of ‘pretty cheap’.
4. Guestimate how long software will take once we have enough hardware; add this to the date produced in (3).
5. The date produced in (4) is your estimate for human-level AI.

How solid is this kind of estimate? Let us consider it in a bit of detail.

#### How much computation is a brain worth?

It is not trivial to estimate how much computation a brain does. A basic philosophical problem is that we don’t actually know what the brain is doing much, so it’s not obvious what part of its behavior is contributing to computation in any particular way. For instance (implausibly) if some of the neuron firing was doing computation, and the rest was just keeping the neurons prepared, we wouldn’t know. We don’t know how much detail of the neurons and their contents and surrounds is relevant to the information processing we are interested in.

[Moravec (2009)](http://www.scientificamerican.com/article/rise-of-the-robots/) estimates how much computation the brain does by extrapolation from the retina. He estimates how much computing hardware would be needed for a computer to achieve the basic image processing that parts of the retina do, then multiplies this by how much heavier the brain is than parts of the retina. As he admits, this is a coarse estimate. I don’t actually have much idea how accurate you would expect this to be. Some obvious possible sources of inaccuracy are the retina being unrepresentative of the brain (as it appears to be for multiple reasons), the retina being capable of more than the processing being replicated by a computer, and mass being poorly correlated with capacity for computation (especially across tissue which is in different parts of the body).

One might straightforwardly improve upon this estimate by extrapolating from other parts of the brain in a similar way, or from calculating how much information could feasibly be communicated in patterns of neuron firing (assuming these were the main components contributing to relevant computation).

#### The relationship between hardware and software

Suppose that you have accurately estimated how much computation the brain does. The argument above treats this as a lower bound on when human-level intelligence will arrive. This appears to rest on a model in which there is a certain level of hardware and a certain level of software that you need, and when you have them both you will have human-level AI.

In reality, the same behavior can often be achieved with different combinations of hardware and software. For instance (as shown in figure 1), you can achieve the same Elo in Go using top of the range software (MoGo) and not much hardware (enough for 64 simulations) or weak software (FatMan) and much more hardware (enough for 1024 simulations, which probably take less hardware each than those used for the sophisticated program). The horizontal axis is doublings of hardware, but FatMan begins with much more hardware.

[![Figure 1: Performance of strong Mogo and weak FatMan with successive doublings in hardware. Mogo starts out doing 64 simulations, and FatMan 1024.](http://aiimpacts.org/wp-content/uploads/2015/02/go-hardware-software.jpg)](http://aiimpacts.org/wp-content/uploads/2015/02/go-hardware-software.jpg)Figure 1: [Performance](http://cgos.boardspace.net/study/) of strong Mogo and weak FatMan with successive doublings in hardware. Mogo starts out doing 64 simulations, and FatMan 1024. The horizontal axis is doublings of hardware, and the vertical axis is performance measured in Elo.
In Go we thus have a picture of indifference curves something like this:

[![](http://aiimpacts.org/wp-content/uploads/2015/02/hardware-software-indifference-copy.jpg)](http://aiimpacts.org/wp-content/uploads/2015/02/hardware-software-indifference-copy.jpg)

Go is not much like general intelligence, but the claim is that software in general has this character. If this is true, it suggests that the first human-level AI designed by human engineers might easily use much more or much less hardware than the human brain. This is illustrated in figure 3. Our trajectory of software and hardware progress could run into the frontier of human-level ability above or below human-level. If our software engineering is more sophisticated than that of evolution at the point where we hit the frontier, we would reach human-level AI with much less than ‘human-equivalent’ hardware.

[![hardware software indifference 2](http://aiimpacts.org/wp-content/uploads/2015/02/hardware-software-indifference-2.jpg)](http://aiimpacts.org/wp-content/uploads/2015/02/hardware-software-indifference-2.jpg)Figure 3: two possible trajectories for human hardware and software progress, which achieve human-level intelligence (the curved line) with far more and far less hardware than humans require.
As an aside, if we view the human brain as a chess playing machine, an analog to the argument outlined earlier in the post suggests that we should achieve human-level chess playing at human-equivalent hardware. We in fact achieved it much earlier, because indeed humans can program a chess player more efficiently than evolution did when it programmed humans. This is obviously in part because the human brain was not designed to play chess, and is mostly for other things. However, it’s not obvious that the human brain was largely designed for artificial intelligence research either, suggesting economic dynamite such as this might also arrive without ‘human-level’ hardware.

I don’t really know how good current human software engineering is compared to evolution, when they set their minds to the same tasks. I don’t think I have particularly strong reason to think they are about the same. Consequently, it seems I don’t seem to have strong reason to expect hardware equivalent to the brain is a particularly important benchmark (though if I’m truly indifferent between expecting human engineers to be better or worse, human-level hardware is indeed my median estimate).

Human equivalent hardware might be more important however: I said nothing about how hardware trades off against software. If the frontier of human-level hardware/software combinations is more like that in figure 4 below than figure 3, a very large range of software sophistication corresponds to human-level AI occurring at roughly similar levels of hardware, which means at roughly similar times. If this is so, then the advent of human-level hardware is a good estimate for when AI will arrive, because AI would arrive around then for a large range of  levels of software sophistication.

[![Figure 4: If it takes a lot of software sophistication to replace a small amount of hardware, the amount of hardware that is equivalent to a human brain may be roughly as much as is needed for many plausible designs.](http://aiimpacts.org/wp-content/uploads/2015/02/hardware-software-indifference-21.jpg)](http://aiimpacts.org/wp-content/uploads/2015/02/hardware-software-indifference-21.jpg)Figure 4: If it takes a lot of software sophistication to replace a small amount of hardware, the amount of hardware that is equivalent to a human brain may be roughly as much as is needed for many plausible designs.
The curve could also look opposite however, with the level of software sophistication being much more important than available hardware. I don’t know of strong evidence either way, so for now the probability of human-level AI at around the time we hit human-level hardware only seems moderately elevated.

The shape of the hardware/software frontier we have been discussing could be straightforwardly examined for a variety of software, using similar data to that presented for Go above. Or we might find that this ‘human-level frontier’ picture is not a useful model. The general nature of such frontiers seem highly informative about the frontier for advanced AI. I have not seen such data for anything other than parts of it for chess and Go. If anyone else is aware of such a thing, I would be interested to see it.

#### Costs

Will there be human-level AI when sufficient hardware (and software) is available at the cost of a supercomputer? At the cost of a human? At the cost of a laptop?

Price estimates used in this kind of calculation often seem to be chosen to be conservative—low prices so that the audience can be confident that an AI would surely be built if it were that cheap. For instance, [when](http://futureoflife.org/PDF/rich_sutton.pdf) will human-level hardware be available for $1,000? While this is helpful for establishing an upper bound on the date, it does not seem plausible as a middling estimate. If human-level AI could be built for a million dollars instead of a thousand, it would still be done in a flash, and this corresponds to a difference of around [fifteen years](http://futureoflife.org/PDF/rich_sutton.pdf).

This is perhaps the easiest part of such an estimate to improve, with a review of how much organizations are generally willing to spend on large, valuable, risky projects.

\*\*\*

In sum, this line of reasoning seems to be a reasonable start. As it stands, it probably produces wildly inaccurate estimates, and appears to be misguided in its implicit model of how hardware and software relate to each other. However it is a good beginning that could be incrementally improved into a fairly well informed estimate, with relatively modest research efforts. Which is just the kind of thing one wants to find among previous efforts to answer one’s question.

How AI timelines are estimated


At-least-human-level-at-human-cost AI

*By Katja Grace, 7 February 2015*

[Often](http://wiki.lesswrong.com/wiki/Interview_series_on_risks_from_AI), when people are asked ‘when will human-level AI arrive?’ they suggest that it is a meaningless or misleading term. I think they have a point. Or several, though probably not as many as they think they have.

One problem is that if the skills of an AI are developing independently at different rates, then at the point that an AI eventually has the full kit of human skills, they also have a bunch of skills that are way past human-level. For instance, if a ‘human-level’ AI were developed now, it would be much better than human-level at arithmetic.

Thus the term ‘human-level’ is misleading because it invites an image of an AI which competes on even footing with the humans, rather than one that is *at least* as skilled as a human in every way, and thus what we would usually think of as *extremely superhuman*.

Another problem is that the term is used to mean multiple things, which then get confused with each other. One such thing is a machine which replicates human cognitive behavior, at any cost. Another is a machine which replicates human cognitive behavior at the price of a human. The former could plausibly be built years before the latter, and should arguably not be nearly as economically exciting. Yet often people imagine the two events coinciding, seemingly for lack of conceptual distinction.

Should we use different terms for human level at human cost, and human level at any cost? Should we have a different term altogether, which evokes at-least-human capabilities? I’ll leave these questions to you. For now we just made a [disambiguation page](http://aiimpacts.org/human-level-ai/ "Human-Level AI").

[![](http://aiimpacts.org/wp-content/uploads/2015/02/t3_ku6_1-300x230.png)](http://aiimpacts.org/wp-content/uploads/2015/02/t3_ku6_1.png)An illustration I made for the [Superintelligence Reading Group](http://lesswrong.com/lw/ku6/superintelligence_reading_group_section_1_past/)

At-least-human-level-at-human-cost AI