Glial Signaling

The presence of glial cells may increase the capacity for signaling in the brain by a small factor, but is unlikely to qualitatively change the nature or extent of signaling in the brain.
Support
Number of glial cells
Azevado et al. physically count the number of cells in a human brain and find about 10¹¹ each of neurons and glial cells, suggesting that the number of glia is quite similar to the number of neurons.
References to much larger numbers of glial cells appear to be common, but we could not track down the empirical research supporting these claims. For example the Wikipedia article on neuroglia states “In general, neuroglial cells are smaller than neurons and outnumber them by five to ten times,” and an article about glia in Scientific American opens “Nearly 90 percent of the brain is composed of glial cells, not neurons.” An informal blog post suggests that the factor of ten figure may be a popular myth, although that post also draws on Azevado et al. so should not be considered independent support.
Nature of glial signaling
Sandberg and Bostrom write: “…the time constants for glial calcium dynamics is generally far slower than the dynamics of action potentials (on the order of seconds or more), suggesting that the time resolution would not have to be as fine” (p. 36). This suggests that the computational role of glial cells is not too great.
Newman and Zahs 1998 mechanically stimulate glial cells in a rat retina, and find that this stimulation results in slow-moving waves of increased calcium concentration.1 These calcium waves had an effect on neuron activity (see figure 4 in their paper, which also provides some indication concerning the characteristic timescale). For reference, these speeds are about a million times slower than action potential propagation (neuron firing). These figures support Sandberg and Bostrom’s claims, and as far as we are aware they are consistent with the broader literature on calcium dynamics.
Astrocytes—a type of glial cell—take in information from action potentials (from neurons).2  There is some evidence that a small fraction of glia can generate action potentials, though such cells are “estimated to represent 5–10% of the cells” and so unlikely to substantially change calculations based on neurons.
It seems possible that further study or a more comprehensive survey of the literature would reveal other high-bandwidth signaling between glial cells, or that timescale-based estimates for the bandwidth of calcium signaling are too low, but at the moment we have little reason to suspect this.
Energy of glial signaling
If glia were performing substantially more computation than neurons, we would weakly expect them to consume more (or at least comparable) energy for a number of reasons:

The energy demands of the brain are very significant. If glia could perform comparable computation with much lower energy, we would expect them to predominate in terms of volume, whereas this does not seem to be the case.
It would be surprising if different computational elements in the brain exhibited radically different efficiency.

However, the majority of energy in the brain is used to maintain resting potentials and propagate action potentials, for example a popularization in Scientific American summarizes “two thirds of the brain’s energy budget is used to help neurons or nerve cells “fire” or send signals.”
Although we can imagine many possible designs on which glia would perform most of the information transfer in the brain while neurons provided particular kinds of special-purpose communication at great expense, this does not seem likely given our current understanding. This provides further mild evidence that the computational role of glial cells is unlikely to substantially exceed the role of neurons.

Glial Signaling


Scale of the Human Brain

The brain has about 10¹¹ neurons and 1.8-3.2 x 10¹⁴ synapses. These probably account for the majority of computationally interesting behavior.
Support
Number of neurons in the brain
The number of neurons in the brain is about 10¹¹. For instance, Azevado et al physically counted them and found 0.6-1 * 10¹¹. Eric Chudler has collected estimates from a range of textbooks, which estimate 1-2 x 10¹⁰ of these (10%-30%) are in the cerebral cortex.1
Number of synapses in the brain
The number of synapses in the brain is known much less precisely, but is probably about 10¹⁴. For instance Human-memory.net reports 10¹⁴-10¹⁵ (100 – 1000 trillion) synapses in the brain, with no citation or explanation. Wikipedia says the brain contains 100 billion neurons, with 7,000 synaptic connections each, for 7 x 10¹⁴ synapses in total, but this seems possibly in error.2
Number of synapses in the neocortex
One way to estimate of the number of synapses in the brain is to extrapolate from the number in the neocortex. According to stereologic studies that we have not investigated, there are around 1.4 x 10¹⁴ synapses in the neocortex.3 This is roughly consistent with Eric Chudler’s summary of textbooks, which gives estimates between 0.6-2.4 x 10¹⁴ for the number of synapses in the cerebral cortex.4
We are not aware of convincing estimates for synaptic density outside of the cerebral cortex, and our impression is that widely reported estimates of 10¹⁴ are derived from the assumption that the neocortex contains the great bulk of synapses in the brain. This seems plausible given the large volume of the neocortex, despite the fact that it contains a minority of the brain’s neurons. By volume, around 80% of the human brain is neocortex.5 The neocortex also consumes around 44% of the brain’s total energy, which may be another reasonable indicator of the fraction of synapses in contains.6 So our guess is that the number of synapses in the entire brain is somewhere between 1.3 and 2.3 times the number in the cerebral cortex. From above, the cerebral cortex contains around 1.4 x 10¹⁴ synapses, so this gives us 1.8-3.2 x 10¹⁴ total synapses.
Number of synapses per neuron
The number of synapses per neuron varies considerably. According to Wikipedia, the majority of neurons are cerebellum granule cells, which have only a handful of synapses, while the statistics above suggest that the average neuron has around 1,000 synapses. Purkinje cells have up to 200,000 synapses.7
Number of glial cells in the brain
Main article: Glial signaling
Azevado et al aforementioned investigation finds about 10¹¹ glial cells (the same as the number of neurons).
Relevance of cells other than neurons to computations in the brain
Main article: Glial signaling
It seems that the timescales of glial dynamics are substantially longer than for neuron dynamics. Sandberg and Bostrom write: “However, the time constants for glial calcium dynamics is generally far slower than the dynamics of action potentials (on the order of seconds or more), suggesting that the time resolution would not have to be as fine” (p. 36). This suggests that the computational role of glial cells is not too great. References to much larger numbers of glial cells appear to be common, but we were unable to track down any empirical research supporting these claims. An informal blog post suggests that a common claim that there are ten times as many glial cells as neurons may be a popular myth.
We are not aware of serious suggestions that cells other than neurons or glia play a computationally significant role in the functioning of the brain.

Scale of the Human Brain


Neuron firing rates in humans

Our best guess is that an average neuron in the human brain transmits a spike about 0.1-2 times per second.
Support
Bias from neurons with sparse activity
When researchers measure neural activity, they can fail to see neurons which rarely fire during the experiment (those with ‘sparse’ activity).1 Preferentially recording more active neurons means overestimating average rates of firing. The size of the bias seems to be around a factor of ten: it appears that around 90% of neurons are ‘silent’, so unlikely to be detected in these kinds of experiments. This suggests that many estimates should be scaled down by around a factor of around ten.
Assorted estimates
Informal estimates
Informal websites and articles commonly report neurons as firing between <1 and 200 times per second.2 These sources lack references and are not very consistent, so we do not put much stock in them.
Estimates of rate of firing in human neocortex
Based on the energy budget of the brain, it appears that the average cortical neuron fires around 0.16 times per second. It seems unlikely that the average cortical neuron spikes much more than once per second.
The neocortex is a large part of the brain. It accounts for around 80% of the brain’s volume3, and uses 44% of its energy4. It appears to hold at least a third of the brain’s synapses if not many more5. Thus we might use rates of firing of cortical neurons as a reasonable proxy for normal rates of neuron firing in the brain. We can also do a finer calculation.
We might roughly expect energy used by the brain to scale in proportion both to the spiking rate of neurons and to volume. This is because the energy required for every neuron to experience a spike scales up in proportion to the surface area of the neurons involved6, which we expect to be roughly proportional to volume.
So we can calculate:
energy(cortex) = volume(cortex) * spike_rate(cortex) * c
energy(brain) = volume(brain) * spike_rate(brain) * c
For c a constant.
Thus,
energy(cortex)/energy(brain) = volume(cortex) * spike_rate(cortex)/volume(brain) * spike_rate(brain)
From figures given above then, we can estimate:
0.44 = 0.8 * 0.16/spike_rate(brain)
spike_rate(brain) = 0.8 * 0.16 /0.44 = 0.29
Or for a high estimate:
0.44 = 0.8 * 1/spike_rate(brain)
spike_rate(brain) = 0.8 * 1 /0.44 = 1.82
So based on this rough extrapolation from neocortical firing rates, we expect average firing rates across the brain to be around 0.29 per second, and probably less than 1.82 per second. This has been a very rough calculation however, and we do not have great confidence in these numbers.
Estimates of rate of firing in non-human visual cortex
A study of macaque and cat visual cortex found rates of neural firing averaging 3-4 spikes per second for cats in different conditions, and 14-18 spikes per second for macaques. A past study found 9 spikes per second for cats.7 It is hard to know how these estimates depend on the region being imaged and on the animal being studied, which significantly complicates extracting conclusions from these results. Furthermore, these studies appear to be subject to the bias discussed above, from only sampling visually responsive cells. Thus they probably overestimate overall neural activity by something like a factor of ten. This suggests figures in the 0.3-1.8 range, consistent with estimates from the neocortex. Note that the visual cortex is part of the neocortex, so this increases our confidence in our estimates for that, without reducing our uncertainty about the rest of the brain.
Maximum neural firing rates
The ‘refractory period’ for a neuron is the time after it fires during which it either can’t fire again (‘absolute refractory period’) or needs an especially large stimulus to fire again (‘relative refractory period’). According to physiologyweb.com, absolute refractory periods tend to be 1-2ms and relative refractory periods tend to be 3-4ms.8 This implies than neurons are generally not capable of firing at more than 250-1000 Hz. This is suggestive, however the site does not say anything about the distribution of maximum firing rates for different types of neurons, so the mean firing rate could in principle be much higher.
Conclusions
Informal estimates place neural firing rates in the <1-200Hz range. Estimates from energy use in the neocortex suggests a firing rate of 0.16Hz in the neocortex, which suggests around 0.29Hz in the entire brain, and probably less than 1.8Hz, though we are not very confident in our estimation methodology here. We saw animal visual cortex firing rates in the 3-18Hz range, but these are probably an order of magnitude too high due to bias from recording active neurons, suggesting real figures of 0.3-1.8 Hz, which is consistent with the estimates from the neocortex previously discussed. Neuron refractory periods (recovery times) suggest 1000Hz is around as fast as a normal neuron can possibly fire. Combined with the observation that 90% of neurons rarely fire, this suggests 100Hz as a high upper bound on the average firing rate. However this does not tell us about unusual neurons, of which there might be many.
So we have two relatively weak lines of reasoning suggesting average firing rates of around 0.1Hz-2Hz. These estimates are low compared to the range of informal claims. However the informal claims appear to be unreliable, especially given that two are higher than our upper bound on neural firing rates (though these are also unreliable). 0.1-2Hz is also low compared to these upper bounds, as it should be. Thus our best guess is that neurons fire at 0.1-2Hz on average.
Notes

Neuron firing rates in humans


Metabolic Estimates of Rate of Cortical Firing

Cortical neurons are estimated to spike around 0.16 times per second, based on the amount of energy consumed by the human neocortex.1 They seem unlikely to spike much more than once per second on average, based on this analysis.
Support
Energy spent on spiking
Lennie 2003 estimates the rate of neuron firing in the cortex based on estimates for energy spent on Na/K ion pumps during spikes, and the energy required by Na/K ion pumps per spike.
Lennie produces estimates for energy consumed in three parts:

Estimates for adenosine triphosphate (ATP) molecules consumed by the neocortex: According to brain scans, glucose is metabolized at a rate of about 0.40 micro mol/g/min. Each glucose molecule yields around 30 molecules of ATP. This suggests that the entire cortex consumes 3.4 * 10²¹ molecules of ATP per minute.2 Note that ATP’s function is as energy source, so this is a measure of how much energy the neocortex uses.
Estimates for the fraction of this ATP used to maintain ion balances: If you inactivate Na/K ion pumps with the drug ouabain, this reduces energy consumption by 50%, suggesting that these ion pumps use about half of the cortex’s energy. 3 This gives us 1.7 * 10²¹ molecules of ATP per minute being used to maintain ion balances.
Estimates for the fraction of ion balancing ATP used in spikes: Maintaining resting potentials (not part of spiking) in all neurons costs 1.3 x 10²¹ ATP molecules per minute. This leaves 3.9 * 10²⁰ ATP per minute for spiking.4

However, other authors report higher fractions of cortical energy are spent on spiking. Laughlin 2001 writes that spiking accounts for 80% of total energy consumption in mammalian cortex.5 Other work by Laughlin and Attwell, which is a primary source for Lennie’s estimates, reports that spiking consumes around 47% of energy.6
Our understanding is that the difference can be attributed to differences between the rodent brain and the human brain, and the scaling estimates from one to the other. We are not particularly confident in this methodology.
Energy per spike
According to Lenny, each spike consumes around 2.4 * 10⁹ molecules of ATP.7 This estimate is produced by scaling up estimates for the rat brain.8 The estimates for the rat brain were inferred from ‘anatomic and physiologic data’, which we have not scrutinized.9  We are not particularly confident in this scaling methodology. These estimates appear to be produced by counting ion channels and applying detailed knowledge of the mechanics of ion channels (which consume a roughly fixed amount of ATP per transported molecule).
Spikes per neuron per second
We saw above that the cortex uses 3.9 * 10²⁰ ATP/minute for spiking, and that each spike consumes around 2.4 * 10⁹ molecules of ATP. So the cortex overall has around 2.7 * 10⁹ spikes per second.  There are 1.9 *10¹⁰ neurons in the cortex, so together we can calculate that these neurons produce around 0.16 spikes per second on average.10
Even assuming that essentially all of the energy in the brain is spent on signaling, this would introduce a bias of only a factor of 8 in Lennie’s estimates. On page S1 Lennie presents an analysis of other possible sources of error, and overall it seems unlikely to us that the estimate is too low by more than an order of magnitude or so.


Preliminary prices for human-level hardware

By Katja Grace, 4 April 2015
Computer hardware has been getting cheap now for about seventy five years. Relatedly, large computing projects can afford to be increasingly large. If you think the human brain is something like a really impressive computer1, then a natural question is ‘what happens when big projects can afford to use as much computation as the brain?’
One possibility is that we get something like human-level AI then and there. Maybe the brain just uses easy algorithmic ideas, and as soon as someone gets enough hardware together and tries a bit on the brain-like software, the creation will be about as good as a human brain.
Another possibility is that sufficient hardware to simulate a brain is basically irrelevant. On this story, the brain has big algorithmic secrets we don’t know about, and it would take unimaginable warehouses of hardware to replace these insights with brute force computation.
Most possibilities lie somewhere between these two, where hardware and software are both somewhat helpful. In these cases, knowing when we will have enough hardware to run a brain isn’t everything, but is informative. For more discussion of how to model this situation, see our older post.
There doesn’t seem to be consensus on how important hardware is. But given the range of possibilities, whenever ‘human-level hardware’ arrives seems like a disproportionately likely time for human-level AI to arrive.
For this reason, we want to know when this human-level hardware point is. Which means we want to know the price of hardware, how fast that price is falling, how much hardware you need to do what the human brain does, and how much anyone is likely to pay for that. Ideally, we would like a few relatively independent estimates of several of these things.
We’ve just been checking the prices, and price trends, so this seems like a good time to pause and see what they imply when combined with some existing estimates of the brain’s computational requirements. Later we will explore these estimates in more detail, and hopefully add at least one based on measuring TEPS.
Moravec estimates that the brain performs around 100 million MIPS.2 MIPS are not directly comparable to MFLOPS (millions of FLOPS), and have deficiencies as a measure, but the empirical relationship in computers is something like MFLOPS = 2.3 x MIPS0.89, according to Sandberg and Bostrom (2008).3 This suggests Moravec’s estimate coincides with around 3.0 x 10¹³ FLOPS. Given that an order of magnitude increase in computing power per dollar corresponds to about four years of time, knowing that MFLOPS and MIPS are roughly comparable is plenty of precision.
We estimated FLOPShours cost around $10-13 each. Thus if we want to run a brain, putting these figures together, it would cost us around $3/hour!
That is, if Moravec’s estimate was right, and my conversion of it to FLOPS was basically right, and our prices were right, and hardware mattered a lot more than software, we would already be in the robot revolution.4 How informative!
Our prices are pretty consistent with the latest extrapolation in Moravec’s 1997 graph (see Figure 1).5 However his graph doesn’t reach human-equivalence until around 2020, and he predicts such computers appear around then.6 This difference from our reading of his figures is because his graph is of what can be bought with $1,000, whereas we expect someone would build an AI by the time it cost around $1M at the latest.7 So our threshold of ‘affordable’ is three orders of magnitude more expensive than his, and thus a bit over a decade earlier. This seems to be a disagreement about at what price it becomes economically viable to replace a human, but we do not understand his position8 well. A ‘brain’ that costs $1000, and runs for a few years, lets say doing useful work for only 40h/week, is costing $0.16/hour! Yet it can earn at least several hundred times more than that. There are other costs to laboring than having a virtual brain, but not that many.
Figure 1: The growth of cheap MIPS, from Moravec.
Incidentally, by 2009, Moravec lengthened his prediction to 20-30 years to ‘close the gap’, seemingly due to cost reductions abating since 1990.9 We don’t know of evidence of this slowing however, and as mentioned, it seems recent hardware prices are in line with his earlier estimates.
Moravec’s is not the only estimate of computation done by the brain. Sandberg and Bostrom (2008) project the processing required to emulate a human brain at different levels of detail.10 For the three levels that their workshop participants considered most plausible, their estimates are 10¹⁸, 10²², and 10²⁵ FLOPS. These would presently cost around $100K/hour, $1bn/hour and $1T/hour.
We estimated that FLOPS are coming down in price by roughly an order of magnitude roughly every four years, so if that is about right and continues, these figures suggest that an AI might compete with a $100/hour human in 12 years, 28 years or 40 years respectively: between 2027 and 2055.
Incidentally, Sandberg and Bostrom predicted a $1M project could purchase enough hardware for a brain between 2019 and 2044 or between 2042 and 2087, depending on whether one is extrapolating retail computing or supercomputing price trends. So our extrapolation of their figures based on recent data is in basically line with theirs. Figure 2 shows roughly where our 2015 FLOPS/$ figures fall on their graph of MFLOPS.
Figure 2: Growth in MFLOPS/$ according to Sandberg and Bostrom (p88), annotated with our recent estimate of cheap FLOPS in 2015.
Another estimate of human-level hardware comes from Kurzweil. In The Singularity is Near, he claimed that a human brain required 10¹⁶ calculations per second, which appears to be roughly equivalent to 10¹⁶ FLOPS.11 This would put the current price of human-level hardware at $1000/hour, beginning to be competitive with some highly expensive people, and reaching the $100/hour people in about four years.12
So it seems human-level hardware presently costs between $3/hour and $1T/hour. And the date when we will reach it lies somewhere in a half-century long haze. We have already stepped into the haze: of the five dates we saw, we have passed one, and are reaching another. This is important, because the era of maybe-human-level hardware has a heightened chance of bringing human-level AI.
It is also important to dispel the haze. If we knew there was a substantially heightened chance of human-level AI in the next ten years, we might act differently from how we would if we knew hardware would be lacking for forty years. Also, if were sure we had uneventfully passed human-level hardware, then we could exclude the possibility that hardware was overwhelmingly important, and start on excluding nearby possibilities (e.g. that hardware is pretty important). Knowing better how much difference hardware makes would change our other expectations about the arrival of human-level AI. Though human-level hardware coming sooner would tend to shorten our AI timelines, if we knew that we had already thoroughly passed it we might expect longer timelines.
This has been a preview of our investigation into these things. The next steps will hopefully resolve some ambiguity around how much computation a brain does, or at least add some more to be resolved later.
***
(Featured image: Pyramidal hippocampal neuron 40x by MethoxyRoxy)


Current FLOPS prices

In November 2017, we estimate the price for one GFLOPS to be between $0.03 and $3 for single or double precision performance, using GPUs (therefore excluding some applications). Amortized over three years, this is $1.1 x 10-5 -$1.1 x 10-7 /GFLOPShour.
Background
We have written about long term trends and short term trends in the costs of computing hardware. We are interested in evaluating the current prices more thoroughly, both to validate the trend data, and because current hardware prices are particularly important to know about.
Details
We separately investigated CPUs, GPUs, computing as a service, and supercomputers. We used somewhat different methods to estimate the price in these categories, based on the data available. We did not find any definitive source on the most cost-effective in any category, or in general, so our examples are probably not the very cheapest. Nevertheless, these figures give a crude sense for the cost of computation in the contemporary market. Our data is here.
Included costs
For CPUs and GPUs, we include only the original recommended retail price of the CPU or GPU, and not other computer components (i.e. we do not even include the cost of CPUs in the price of GPUs). In 2015 we compared prices between one complete rack server and the set of four processors inside it, and found the complete server was around 36% more expensive ($30,000 vs. $22,000). We expect this is representative at this scale, but diminishes with scale.
For computing services, we list the cheapest price for renting the instance for a long period, with no additional features. We do not include spot prices.
For supercomputers, we list costs cited where we could find them, which don’t tend to come with elaboration. We expect that they only include upfront costs, and that most of the costs are for hardware.
We have not included the costs of energy or other ongoing expenses in any prices. Non-energy costs are hard to find, and we suspect a relatively small and consistent fraction of costs. In 2015 we estimated energy costs to be around 10% of hardware costs.1
FLOPS measurements
We are interested in empirical performance figures from benchmark tests, but often the only data we could find was for theoretical maximums. We try to use figures for LINPACK and sometimes for DGEMM benchmarks, depending on which are available. LINPACK relies heavily on DGEMM, suggesting DGEMM is fairly comparable.2
Prices
Graphics processing units (GPUs) and Xeon Phi machines
We collected performance and price figures from Wikipedia3, which are available here (see ‘Wikipedia GeForce, Radeon, Phi simplified’). These are theoretical performance figures, which we understand to generally be between somewhat optimistic and ten times too high. So this data suggests real prices of around $0.03-$0.3/GFLOPS. We collected both single and double precision figures, but the cheapest were similar.
Note that GPUs are typically significantly restricted in the kinds of applications they can run efficiently; this performance is achieved for highly regular computations that can be carried out in parallel throughout a GPU (of the sort that are required for rendering scenes, but which have also proved useful in scientific computing). Xeon Phi units are similar to GPUs, and have broader application,4 but in this dataset were not among the cheapest machines.
Central processing units (CPUs)
We looked at a small number of popular CPUs on Geekbench from the past five years, and found the cheapest to be around $0.71/GFLOPS.5 However there appear to be 5x disparities between different versions of Geekbench, so we do not trust these numbers a great deal (these figures are from the version we have seen to give relatively high performance figures, and thus low implied prices).
We did not investigate these numbers in great depth, or search far for cheaper CPUs, because CPUs seem to be expensive relative to GPUs, and this minimal investigation, plus our previous investigation in 2015, support this.
Computing as service
Another way to purchase FLOPS is via virtual computers.
Amazon Elastic Cloud Compute (EC2) is a major seller of virtual computing. Based on their current pricing, as of October 5th, 2017, renting a c4.8xlarge instance costs $0.621 per hour (if you purchase it for three years, and pay upfront).
According to a Geekbench report  from 2015, a c4.8xlarge instance delivers around 97.5 GFLOPS.6 We do not know if ‘c4.8xlarge’ referred to the same computing hardware in 2015, and we do know that the current version of Geekbench gives substantially different answers to the one in use here. However we estimate that the hardware should be less than twice as good as it was, and Geekbench seems unlikely to underestimate performance by more than an order of magnitude.
This implies that a GFLOPShour costs $6.3 x 10-3 , or optimistically as little as $3.2 x 10-4 . This is much higher than a GPU, at $3.4 x 10-6 for a GFLOPShour, if we suppose the hardware is used over around three years. Amazon is probably not the cheapest provider of cloud computing, however the difference seems to be something like a factor of two,7 which is not enough to make cloud computing competitive with GPUs.
In sum, virtual computing appears to cost two to three orders of magnitude more than GPUs. This high price is presumably partly because there are non-hardware costs which we have not accounted for in the prices of buying hardware, but are naturally included in the cost of renting it. However it is unlikely that these additional costs make up a factor of one hundred to one thousand, so cloud computing does not seem competitive.
Supercomputing
A top supercomputer can perform a GFLOPS for around $3, in 2017. (See Price performance trend in top supercomputers)
Tensor processing units (TPUs)
Tensor processing units appear to perform a GFLOPS for around $1, in February 2018. However it is unclear how this GFLOPS is measured, which makes it somewhat harder to compare (e.g. whether it is single precision or double precision). Such a high price is also at odds with rumors we have heard that TPUs are an especially cheap source of computing, so possibly TPUs are more efficient for a particular set of applications other than the ones where most of these machines have been measured.
Further considerations
In 2015, we estimated GPUs to cost around $3/GFLOPS, i.e. 10-100 times more than we would currently estimate. We do not believe that there has been nearly that much improvement in the past two years, so this discrepancy must be due to error and noise. We remain uncertain about the source of all of the difference, so until we resolve that question, it is plausible that our current GPU estimate errs. If so, the price should still be no higher than $3/GFLOPS (our previous estimate, and our current estimate for supercomputer prices).
Summary
The lowest estimated GFLOPS prices we know of are $0.03-$3/GFLOPS, for GPUs and TPUs.
This is a summary of all of the prices we found:

Type of computerSourceType of performanceCurrent price ($/GFLOPS)Comments

GPUs and Xeon Phi (single precision)WikipediaTheoretical peak.03-0.3$0.03/GFLOPS is given, but is underestimate

GPUs and Xeon Phi (double precision)WikipediaTheoretical peak0.3-0.8Upward sloping; probably not optimized for (in GPUs)

CloudAmazon EC2 and GeekbenchEmpirical158Expensive so less relevant; shallow investigation

SupercomputingTop500 and misc pricesEmpirical2.94Expensive, so less relevant; shallow investigation

CPUsGeekbench and misc pricesEmpirical0.71Unreliable, 5x disagreements between Geekbench versions

TPUsGoogle Cloud Platform BlogUnclear0.95

Notes

Current FLOPS prices


The cost of TEPS

A billion Traversed Edges Per Second (a GTEPS) can be bought for around $0.26/hour via a powerful supercomputer, including hardware and energy costs only. We do not know if GTEPS can be bought more cheaply elsewhere.
We estimate that available TEPS/$ grows by a factor of ten every four years, based the relationship between TEPS and FLOPS. TEPS have not been measured enough to see long-term trends directly.
Background
Traversed edges per second (TEPS) is a measure of computer performance, similar to FLOPS or MIPS.  Relative to these other metrics, TEPS emphasizes the communication capabilities of machines: the ability to move data around inside the computer. Communication is especially important in very large machines, such as supercomputers, so TEPS is particularly useful in evaluating these machines.
The Graph 500 is a list of computers which have been evaluated according to this metric. It is intended to complement the Top 500, which is a list of  the most powerful 500 computers, measured in FLOPS. The Graph 500 began in 2010, and so far has measured 183 machines, though many of these are not supercomputers, and would presumably not rank among the best 500 TEPS scores if more supercomputers computers were measured.
The TEPS benchmark is defined as the number of graph edges traversed per second during a breadth-first search of a very large graph. The scale of the graph is tuned to grow with the size of the hardware. See the Graph500 benchmarks page for further details.
The brain in TEPS
We are interested in TEPS in part because we would like to estimate the brain’s capacity in terms of TEPS, as an input to forecasting AI timelines. One virtue of this is that it will be a relatively independent measure of how much hardware the human brain is equivalent to, which we can then compare to other estimates. It is also easier to measure information transfer in the brain than computation, making this a more accurate estimate. We also expect that at the scale of the brain, communication is a significant bottleneck (much as it is for a supercomputer), making TEPS a particularly relevant benchmark. The brain’s contents support this theory: much of its mass and energy appears to be used on moving information around.
Current TEPS available per dollar
We estimate that a TEPS can currently be produced for around $0.26 per hour in a supercomputer.
Our estimate
Table 1 shows our calculation, and sources for price figures.
We recorded the TEPS scores for the top eight computers in the Graph 500 (i.e. the best TEPS-producing computers known). We searched for price estimates for these computers, and found five of them. We assume these prices are for hardware alone, though this was not generally specified. The prices are generally from second-hand sources, and so we doubt they are particularly reliable.
Energy costs
We took energy use figures for the five remaining computers from the Top 500 list. Energy use on the Graph 500 and Top 500 benchmarks are probably somewhat different, especially because computers are often scaled down for the Graph 500 benchmark. See ‘Bias from scaling down’ below for discussion of this problem. There is a Green Graph 500 list, which gives energy figures for some of the supercomputers doing similar problems to those in the Graph 500, but the computers are run at different scales there to in the Graph 500 (presumably to get better energy ratings), so the energy figures given there are also not directly applicable.
The cost of electricity varies by location. We are interested in how cheaply one can produce TEPS, so we suppose computation is located somewhere where power is cheap, charged at industrial rates. Prevailing energy prices in the US are around $0.20 / kilowatt hour, but in some parts of Canada it seems industrial users pay less than $0.05 / kilowatt hour. This is also low relative to industrial energy prices in various European nations (though these nations too may have small localities with cheaper power). Thus we take $0.05 to be a cheap but feasible price for energy.
Bias from scaling down
Note that our method likely overestimates necessary hardware and energy costs, as many computers do not use all of their cores in the Graph 500 benchmark (this can be verified by comparing to cores used in the Top 500 list compiled at the same time). This means that one could get better TEPS/$ prices by just not building parts of existing computers. It also means that the energy used in the Graph 500 benchmarking (not listed) was probably less than that used in the Top 500 benchmarking.
We correct for this by scaling down prices according to cores used. This is probably not a perfect adjustment: the costs of building and running a supercomputer are unlikely to be linear in the number of cores it has. However this seems a reasonable approximation, and better than making no adjustment.
This change makes the data more consistent. The apparently more expensive sources of TEPS were using smaller fractions of their cores (if we assume they used all cores in the Graph 500), and the very expensive Tianhe-2 was using only 6% of its cores. Scaled according to the fraction of cores used in Graph 500, Tianhe-2 produces TEPShours at a similar price to Sequoia. The two apparently cheapest sources of TEPShours (Sequoia and Mira) appear to have been using all of their cores. Figure 1 shows the costs of TEPShours on the different supercomputers, next to the costs when scaled down according to the fraction of cores that were used in the Graph 500 benchmark.
Figure 1: Cost of TEPShours using five supercomputers, and cost naively adjusted for fraction of cores used in the benchmark test.
Other costs
Supercomputers have many costs besides hardware and energy, such as property, staff and software. Figures for these are hard to find. This presentation suggests the total cost of a large supercomputerover several years can be more than five times the upfront hardware cost. However these figures seem surprisingly high, and we suspect they are not applicable to the problem we are interested in: running AI. High property costs are probably because supercomputers tend to be built in college campuses. Strong AI software is presumably more expensive than what is presently bought, but we do not want to price this into the estimate. Because the figures in the presentation are the only ones we have found, and appear to be inaccurate, we will not further investigate the more inclusive costs of producing TEPShours here, and focus on upfront hardware costs and ongoing energy costs.
Supercomputer lifespans
We assume a supercomputer lasts for five years. This was the age of Roadrunner when decommissioned in 2013, and is consistent with the ages of the computers whose prices we are calculating here — they were all built between 2011 and 2013. ASCI Red lasted for nine years, but was apparently considered ‘supercomputing’s high-water mark in longevity‘. We did not find other examples of large decommissioned supercomputers with known lifespans.
Calculation
From all of this, we calculate the price of a GTEPShour in each of these systems, as shown in table 1.

Name
GTeps
Estimated Price (million)
Hardware cost/hour (5 year life)
Energy (kW)
Hourly energy cost (at 5c/kWh)
Total $/hour
(including hardware and energy)
$/GTEPShours
(including hardware and energy)
$/GTEPShours scaled by cores used
Cost sources

DOE/NNSA/LLNL Sequoia (IBM – BlueGene/Q, Power BQC 16C 1.60 GHz)
23751
$250
$5,704
7,890.00
$394.50
6,098.36
$0.26
 $0.26
1

K computer (Fujitsu – Custom supercomputer)
19585.2
$1,000
$22,815
12,659.89
$632.99
23,448.42
$1.20
 $1.13
2

DOE/SC/Argonne National Laboratory Mira (IBM – BlueGene/Q, Power BQC 16C 1.60 GHz)
14982
$50
$1,141
3,945.00
$197.25
1,338.02
$0.09
$0.09
3

Tianhe-2 (MilkyWay-2) (National University of Defense Technology – MPP)
2061.48
$390
$8,898
17,808.00
$890.40
9,788.42
$4.75
$0.30
4

Blue Joule (IBM – BlueGene/Q, Power BQC 16C 1.60 GHz)
1427
$55.3
$1,262
657.00
$32.85
1,294.54
$0.91
 $0.46
5

Table 1: Calculation of costs of a TEPS over one hour in five supercomputers.
Sequoia as representative of cheap TEPShours
Mira and then Sequoia produce the cheapest TEPShours of the supercomputers investigated here, and are also the only ones which used all of their cores in the benchmark, making their costs less ambiguous. Mira’s costs are ambiguous nonetheless, because the $50M price estimate we have was projected by an unknown source, ahead of time. Mira is also known to have been bought using some part of a $180M grant. If Mira cost most of that, it would be more expensive than Sequoia. Sequoia’s price was given by the laboratory that bought it, after the fact, so is more likely to be reliable.
Thus while Sequoia does not appear to be the cheapest source of TEPS, it does appear to be the second cheapest, and its estimate seems substantially more reliable. Sequoia is also a likely candidate to be especially cheap, since it is ranked first in the Graph 500, and is the largest of the IBM Blue Gene/Qs, which dominate the top of the Graph 500 list. This somewhat supports the validity of its apparent good price performance here.
Sequoia is also not much cheaper than the more expensive supercomputers in our list, once they are scaled down according to the number of cores they used on the benchmark (see Table 1), further supporting this price estimate.
Thus we estimate that GTEPShours can be produced for around $0.26 on current supercomputers. This corresponds to around $11,000/GTEP to buy the hardware alone.
Price of TEPShours in lower performance computing
We have only looked at the price of TEPS in top supercomputers. While these produce the most TEPS, they might not be the part of the range which produces TEPS most cheaply. However because we are interested in the application to AI, and thus to systems roughly as large as the brain, price performance near the top of the range is particularly relevant to us. Even if a laptop could produce a TEPS more cheaply than Sequoia, it produces too few of them to run a brain efficiently. Nonetheless, we plan to investigate TEPS/$ in lower performing computers in future.
For now, we checked the efficiency of an iPad 3, since one was listed near the bottom of the Graph 500. These are sold for $349.99, and apparently produce 0.0304 GTEPS. Over five years, this comes out at exactly the same price as the Sequoia: $0.26/GTEPShour. This suggests both that cheaper computers may be more efficient than large supercomputers (the iPad is not known for its cheap computing power) and that the differences in price are probably not large across the performance spectrum.
Trends in TEPS available per dollar
The long-term trend of TEPS is not well known, as the benchmark is new. This makes it hard to calculate a TEPS/$ trend. Figure 2 is from a powerpoint Announcing the 9th Graph500 List! from the Top 500 website. One thing it shows is top performance in the Graph 500 list since the list began in 2010. Top performance grew very fast (3.5 orders of magnitude in two years), before completely flattening, then growing slowly. The powerpoint attributes this pattern to ‘maturation of the benchmark’, suggesting that the steep slope was probably not reflective of real progress.
One reason to expect this pattern is that during the period of fast growth, pre-existing high performance computers were being tested for the first time. This appears to account for some of it. However we note that in June 2012, Sequoia (which tops the list at present) and Mira (#3) had both already been tested, and merely had lower performance than they do now, suggesting at least one other factor is at play. One possibility is that in the early years of using the benchmark, people develop good software for the problem, or in other ways adjust how they use particular computers on the benchmark.
Figure 2: Performance of the top supercomputer on Graph 500 each year since it has existed (along with the 8th best, and an unspecified sum).

Relationship between TEPS and FLOPS
The top eight computers in the Graph 500 are also in the Top 500, so we can compare their TEPS and FLOPS ratings. Because many computers did not use all of their cores in the Graph 500, we scale down the FLOPS measured in the Top 500 by the fraction of cores used in the Graph 500 relative to the Top 500 (this is discussed further in ‘Bias from scaling down’ above). We have not checked thoroughly whether FLOPS scales linearly with cores, but this appears to be a reasonable approximation, based on the first page of the Top 500 list.
The supercomputers measured here consistently achieve around 1-2 GTEPS per scaled TFLOPS (see Figure 3). The median ratio is 1.9 GTEPS/TFLOP, the mean is 1.7 GTEPS/TFLOP, and the variance 0.14 GTEPS/TFLOP. Figure 4 shows GTEPS and TFLOPS plotted against one another.
The ratio of GTEPS to TFLOPS may vary across the range of computing power. Our figures may may also be slightly biased by selecting machines from the top of the Graph 500 to check against the Top 500. However the current comparison gives us a rough sense, and the figures are consistent.
This presentation (slide 23) reports that a Kepler GPU produces 109 TEPS, as compared to 1012 FLOPS reported here (assuming that both are top end models), suggesting a similar ratio holds for less powerful computers.
Figure 3: GTEPS/scaled TFLOPS, based on Graph 500 and Top 500.
Figure 4: GTEPS and scaled TFLOPS achieved by the top 8 machines on Graph 500. See text for scaling description.

Projecting TEPS based on FLOPS
Since the conversion rate between FLOPS and TEPS is approximately consistent, we can project growth in TEPS/$ based on the better understood growth of FLOPS/$. In the last quarter of a century, FLOPS/$ has grown by a factor of ten roughly every four years. This suggests that TEPS/$ also grows by a factor of ten every four years.

The cost of TEPS


Allen, The Singularity Isn’t Near

The Singularity Isn’t Near is an article in MIT Technology Review by Paul Allen which argues that a singularity brought about by super-human-level AI will not arrive by 2045 (as is predicted by Kurzweil).
The summarized argument
We will not have human-level AI by 2045:
1. To reach human-level AI, we need software as well as hardware.
2. To get this software, we need one of the following:

a detailed scientific understanding of the brain
a way to ‘duplicate’ brains
creation of something equivalent to a brain from scratch

3. A detailed scientific understanding of the brain is unlikely by 2045:

To have enough understanding by 2045, we would need a massive acceleration of scientific progress:

We are just scraping the surface of understanding the foundations of human cognition.

A massive acceleration of progress in brain science is unlikely

Science progresses irregularly:

e.g. The discovery of long-term potentiation, the columnar organization of cortical areas, neuroplasticity.

Science doesn’t seem to be exponentially accelerating
There is a ‘complexity break’: the more we understand, the more complicated the next level to understand is

4. ‘Duplicating’ brains is unlikely by 2045:

Even if we have good scans of brains, we need good understanding of how the parts behave to complete the model
We have little such understanding
Such understanding is not exponentially increasing

5. Creation of something equivalent to a brain from scratch is unlikely by 2045:

Artificial intelligence research appears to be far from providing this
Artificial intelligence research is unlikely to improve fast:

Artificial intelligence research does not appear to be exponentially improving
The ‘complexity break’ (see above) also operates here
This is the kind of area where progress is not a reliable exponential

Comments
The controversial parts of this argument appear to be the parallel claims that progress is insufficiently fast (or accelerating) to reach an adequate understanding of the brain or of artificial intelligence algorithms by 2045. Allen’s argument does not present enough support to evaluate them from this alone. Others with at least as much expertise disagree with these claims, so they appear to be open questions.
To evaluate them, it appears we would need more comparable measures of accomplishments and rates of progress in brain science and AI. With only the qualitative style of Allen’s claims, it is hard to know whether progress being slow, and needing to go far, implies that it won’t get to a specific place by a specific date.

Allen, The Singularity Isn’t Near


Kurzweil, The Singularity is Near

The Singularity Is Near is a book by Ray Kurzweil. It argues that a technological singularity will occur in around 2045. This appears to be largely based on extrapolation from hardware in combination with a guess for how much machine computation is needed to produce a large disruption to human society. The book relatedly claims that a machine will be able to pass a Turing test by 2029.
Details
Calculation of the date of the singularity
The following is our reconstruction of an argument Kurzweil makes in the book, for expecting the Singularity in 2045.

In the early 2030s one thousand dollars’ worth of computation will buy about 10¹⁷ computations per second (p119)
Today we spend more than $10¹¹/year on computation, which will conservatively rise to $10¹²/year by 2030 (p119-20).
Therefore in the early 2030s we will be producing about 10²⁶-10²⁹ computations per second of nonbiological computation per year, and by the mid 2040s, we will produce 10²⁶ cps with $1000 (p120)
The sum of all living biological human intelligence operates at around 10²⁶ computations per second (p113)
Thus in the early 2030s we will produce new computing power roughly equivalent to the capacity of all living biological human intelligence, every year. In the mid 2040s the total computing capacity we produce each year will be a billion times more powerful than all of human intelligence today. (p120)
Non-biological intelligence will be better than our own brains because machines have some added advantages, such as accuracy and ability to run at peak capacity. (p120)
The early 2030s will not be a singularity, because its events do not yet correspond to a sufficiently profound expansion of our intelligence. (p120)
In the 1940s, when the computing capacity we produce each year is a billion times more powerful than all human intelligence today, these events will represent a profound and disruptive transformation in human capability, i.e. a singularity. (p120)

Relevance of software
While he doesn’t mention it in the prediction explained above, Kurzweil appears elsewhere to agree that substantial software progress is needed alongside hardware progress for human-level intelligence. He says, “The hardware computational capacity is necessary but not sufficient. Understanding the organization and content of these resources—the software of intelligence—is even more critical and is the objective of the brain reverse-engineering undertaking.” (p126).
His argument that the necessary understanding for producing human-level software will come in time with the hardware appears to be as follows:

Understanding of the  brain is reasonably good; researchers rapidly turn data from studies into effective working models (p147)
Understanding of the brain is growing exponentially:

Our ability to observe the brain is growing exponentially: ‘Scanning and sensing tools are doubling their overall spatial and temporal resolution each year’. (p163)
‘Databases of brain-scanning information and model building are also doubling in size about once per year.’ (p163)
Our ability to model the brain follows closely behind our acquisition of the requisite tools and data (p163) and so is also growing exponentially in some sense.

Human-level AI
According to Kurzweil, ‘With both the hardware and software needed to fully emulate human intelligence, we can expect computers to pass the Turing test, indicating intelligence indistinguishable from that of biological humans, by the end of the 2020s.’
The claims that hardware and software will be human-level by 2029 appear to share their justification with the above claims about the timing of the Singularity.
Kurzweil bet that by 2029 a computer would pass the turing test, and wrote an article explaining his optimism about the bet here.
Comments
If the ‘singularity’ is meant to refer to some particular event, it is unclear why this event would occur when the hardware produced is a billion times more powerful than all human intelligence today. This number might make some sense as an upper bound on when something disruptive should have happened. However it is unclear why the events predicted in the early 2030s would not cause a profound and disruptive transformation, while those in the mid 2040s would.
Kurzweil’s calculation of the date of the Singularity appears to have other minor gaps:

The argument is about flows of hardware, where it wants to make a conclusion in terms of stocks of hardware. Kurzweil wants to compare total biological and non-biological computation. However he calculates the computing hardware produced per year, instead of the total available that year, or the computation done in that year. These numbers are probably fairly similar in practice, if we suppose that hardware lasts a small number of years.
That non-biological machines appear to have some advantages over humans does not imply that some given non-biological machines have advantages overall.
The argument suggests software will develop ‘fast’ in some sense, but this isn’t actually compared to hardware progress or measured in years, so it is unclear whether it would be developed in time.

A key disagreement with other commentators appears to be over the rate of progress of understanding relevant to producing software. In particular, Kurzweil believes that such understanding is growing exponentially, and that it will be be sufficient for producing machines as intelligent as humans in line with the hardware. Allen, for instance, has argued with this. Resolving this disagreement would require better measures of neuroscience progress, as well as a better understanding of its relevance.

Kurzweil, The Singularity is Near


Wikipedia history of GFLOPS costs

This is a list from Wikipedia, showing hardware configurations that authors claim perform efficiently, along with their prices per GFLOPS at different times in recent history.
In it, prices generally fall at around an order of magnitude every five years, and have continued to do so recently.
Notes
This list is from November 5 2017 (archive version). It is not necessarily credible. We had trouble verifying at least one datapoint, of the few we tried. Performance numbers appear to be a mixture of theoretical peak performance and empirical performance. It is not clear to what extent one should expect the included systems to be especially cost-effective, or why these particular systems were chosen.
The last point is in October 2017, and appears to be roughly in line with the rest of the trend. The last order of magnitude  took around 4.5 years. The overall rate in the figure appears to be very roughly an order of magnitude every five years.
List

Date
Approximate cost per GFLOPS
Approximate cost per GFLOPS inflation adjusted to 2013 US dollars[54]
Platform providing the lowest cost per GFLOPS
Comments

1961
US$18,672,000,000 ($18.7 billion)
US$145.5 billion
About 2400 IBM 7030 Stretch supercomputers costing $7.78 million each
The IBM 7030 Stretch performs one floating-point multiply every 2.4 microseconds.[55]

1984
$18,750,000
$42,780,000
Cray X-MP/48
$15,000,000 / 0.8 GFLOPS

1997
$30,000
$42,000
Two 16-processor Beowulfclusters with Pentium Promicroprocessors[56]

April 2000
$1,000
$1,300
Bunyip Beowulf cluster
Bunyip was the first sub-US$1/MFLOPS computing technology. It won the Gordon Bell Prize in 2000.

May 2000
$640
$836
KLAT2
KLAT2 was the first computing technology which scaled to large applications while staying under US-$1/MFLOPS.[57]

August 2003
$82
$100
KASY0
KASY0 was the first sub-US$100/GFLOPS computing technology.[58]

August 2007
$48
$52
Microwulf
As of August 2007, this 26.25 GFLOPS “personal” Beowulf cluster can be built for $1256.[59]

March 2011
$1.80
$1.80
HPU4Science
This $30,000 cluster was built using only commercially available “gamer” grade hardware.[60]

August 2012
$0.75
$0.73
Quad AMD Radeon 7970 GHz System
A quad AMD Radeon 7970 desktop computer reaching 16 TFLOPS of single-precision, 4 TFLOPS of double-precision computing performance. Total system cost was $3000; Built using only commercially available hardware.[61]

June 2013
$0.22
$0.22
Sony PlayStation 4
The Sony PlayStation 4 is listed as having a peak performance of 1.84 TFLOPS, at a price of $400[62]

November 2013
$0.16
$0.16
AMD Sempron 145 & GeForce GTX 760 System
Built using commercially available parts, a system using one AMD Sempron 145 and three Nvidia GeForce GTX 760 reaches a total of 6.771 TFLOPS for a total cost of $1090.66.[63]

December 2013
$0.12
$0.12
Pentium G550 & Radeon R9 290 System
Built using commercially available parts. Intel Pentium G550 and AMD Radeon R9 290 tops out at 4.848 TFLOPS grand total of US$681.84.[64]

January 2015
$0.08
$0.08
Celeron G1830 & Radeon R9 295X2 System
Built using commercially available parts. Intel Celeron G1830 and AMD Radeon R9 295X2tops out at over 11.5 TFLOPS at a grand total of US$902.57.[65][66]

June 2017
$0.06
$0.06
AMD Ryzen 7 1700 & AMD Radeon Vega Frontier Edition
Built using commercially available parts. AMD Ryzen 7 1700 CPU combined with AMD Radeon Vega FE cards in CrossFire tops out at over 50 TFLOPS at just under US$3,000for the complete system.[67]

October 2017
$0.03
$0.03
Intel Celeron G3930 & AMD RX Vega 64
Built using commercially available parts. Three AMD RX Vega 64 graphics cards provide just over 75 TFLOPS half precision (38 TFLOPS SP or 2.6 TFLOPS DP when combined with the CPU) at ~$2,050 for the complete system.[68]

The following is a figure we made, of the above list.

Further discussion
Trends in the cost of computing

Wikipedia history of GFLOPS costs


Trends in the cost of computing

Posted 10 Mar 2015
Computing power available per dollar has probably increased by a factor of ten roughly every four years over the last quarter of a century (measured in FLOPS or MIPS).
Over the past 6-8 years, the rate has been slower: around an order of magnitude every 10-16 years, measured in single precision theoretical peak FLOPS or Passmark’s benchmark scores.
Since the 1940s, MIPS/$ have grown by a factor of ten roughly every five years, and FLOPS/$ roughly every 7.7 years.
Evidence
Nordhaus
Nordhaus (2001) analyzes the cost of computing over the past century and a half, and produces Figure 1 (though the scale on the vertical axis appears to be off by many orders of magnitude). Much of his data comes from Moravec’s Mind Children (an updated version of the data is here). He converts all data points to ‘million standard operations per second’ (MSOPS), where a standard operation is a weighted mixture of multiplications and additions. He says it is approximately equivalent to 1 MIPS under the Dhrystone metric.
He calculates that performance improved at an average rate of 55% per year since 1940. That is, an order of magnitude roughly every five years. However he finds that the average growth rate in different decades differed markedly, with growth since 1980 (until writing in 2001) at around 80% per year, and growth in the 60s and 70s at less than 30% (see figure 2). This would correspond to improving by an order of magnitude every four years in the 80s and 90s.
Figure 1: “The progress of computing measured in cost per million standardized operations per second (MSOPS) deflated by the consumer price index.” Note that the vertical axis appears to be mislabeled—the scale is around seven orders of magnitude different from other sources, such as Moravec. (From Figure 1, Nordhaus, 2001, p38)
Figure 2: From Nordhaus p42,”Rate of Growth of Computer Power by Epoch…Real computer power is the inverse of the decline of real computation costs…”
Sandberg and Bostrom
Sandberg and Bostrom (2008) investigate hardware performance trends in their Whole Brain Emulation Roadmap (Appendix B). They plot price performance in MIPS/$ and FLOPS/$, as shown in Figures 3 and 4. They find MIPS/$ grows by a factor of ten every 5.6 years (with a bootstrap 95% confidence interval of 5.3-5.9), and FLOPs/$ grows by a factor of ten every 7.7 years (with a bootstrap confidence interval of 6.5‐9.2 years).
They find that growth in MIPS/$ slowed in the 70s and 80s, then accelerated again (most recently gaining an order of magnitude every 3.5 years), which is close to what Nordhaus found.
Sandberg and Bostrom’s data is from John McCallum’s CPU price performance dataset, which does not appear to draw directly from Moravec’s data.
Figure 3: Processing power available per dollar over time, measured in MIPS and 2007 US dollars.
Figure 4: Processing power available per dollar over time, measured in FLOPS using the LINPACK benchmark and in 2007 US dollars
Rieber and Muehlhauser
Muehlhauser and Rieber (2014) extended Koh and Magee’s data on MIPS available per dollar to 2014 (data [not currently] available here). Koh and Magee’s data largely comes from Moravec (like Nordhaus’ above), though they too extended it some. Muehlhauser and Rieber produced Figure 5.
In this data, performance since 1940 appears to be growing by a factor of ten roughly every 5 years (14.2 orders of magnitude in 74 years). In the first fourteen years of this century, log(MIPS/$) grew from roughly -0.7 to 2.8, which corresponds to one order of magnitude every four years (or 77% growth per year).
Figure 5: Rieber and Muehlhauser’s MIPS/$ data (modified to fix typo).
Wikipedia
Wikipedia has a small list of hardware configurations that authors claim produce gigaFLOPS efficiently, along with their prices at different times in recent history. Their data does not appear to cite other sources mentioned above.
Here is their table, as of March 2 2015. Figure 6 shows inflation adjusted costs of gigaFLOPS over time, taken from the table. The examples in the table were apparently selected as follows:
The “cost per GFLOPS” is the cost for a set of hardware that would theoretically operate at one billion floating-point operations per second. During the era when no single computing platform was able to achieve one GFLOPS, this table lists the total cost for multiple instances of a fast computing platform which speed sums to one GFLOPS. Otherwise, the least expensive computing platform able to achieve one GFLOPS is listed.
We find this table dubious. It lacks many citations, and the citations it has frequently lack detail. For instance, the claims that the collections of hardware specified produce a GFLOPS are often unsubstantiated. We spent around thirty minutes trying to substantiate the 2015 figure, to no avail. The figure is more than an order of magnitude cheaper than current FLOPS prices we found.
In this data, the price of a gigaFLOPS falls by an order of magnitude roughly every four years (14 orders of magnitude in 54 years is 3.9 years per order of magnitude). Since 1997, each order of magnitude only took three years (5.7 orders of magnitude in 18 years). Note that there is very little data before 1997.
Figure 6: Price of GFLOPS in different years according to Wikipedia, adjusted to 2013 US dollars.
Short term trends
Main article: Recent trends in the cost of computing
The cheapest hardware prices (for single precision FLOPS/$) are on track to fall by around an order of magnitude every 10-16 years, based on data from around 2011-2017. There was no particular sign of slowing between 2011 and 2017.
Summary
We have looked at four efforts to measure long term hardware price performance trajectories. Two of them are based on Moravec’s earlier effort, while the other two appear to be more independent (though we suspect still draw on similar sources). Two investigations measured (G)FLOPS, two measured MIPS, and one measured MSOPS.
Results seem fairly consistent in recent decades, and for MIPS/$ in the longer run. There is insufficient data on FLOPS in the long run to check consistency. All four estimates of growth later than the 1990s produce 3.5-4 years as the time for price performance to to grow an order of magnitude (we did not include an estimate for recent years from Sandberg and Bostrom’s FLOPS data, since they did not make one and it was not straightforward to make one ourselves).1 Though note that these measures are from different spans within that period, and use different benchmarks (two were MIPS, one FLOPS, one MSOPS). Only Rieber and Muehlhauser and Wikipedia have data after 2002. Though they give similar recent growth figures, it is not clear how consistent they are: Rieber and Muehlhauser’s data appears to decline sharply in the last few years, and appears to only use CPUs, while the Wikipedia data is fairly even, and moves to GPUs in later years.
If we take an MSOPS to be more or less equivalent to a MIPS (as Nordhaus claims), then growth in MIPS since the 1940s is fairly consistent across studies, gaining an order of magnitude roughly every 5 years (Nordhaus), 5 years (Rieber and Muehlhauser) or 5.6 years (Sandberg and Bostrom). Note that the former two draw on similar data.
Our two estimates of long run growth in FLOPS/$ differ substantially: we have gained an order of magnitude either every 4 years or every 7.7 years. However the four year estimate comes from Wikipedia, which only has two entries prior to 1990, while Sandberg and Bostrom have on the order of hundreds of entries from that period. Thus we rely on Sanberg and Bostrom here, and estimate FLOPS grow by an order of magnitude every 7.7 years.
Prior to the 1940s, growth appears to be ambiguous and small. It looks like 2.4 orders of magnitude over forty eight years in Rieber and Muehlhauser’s figure, for an order of magnitude every 20 years. Nordhaus measures it as negative.
Further work
Further work on this subject might:

Check Moravec’s data, as it appears to be widely cited and reused (perhaps just check consistency between the fraction of data from Moravec and that added later from another source in existing datasets).
Separate different types of computers (e.g. treat desktop CPUs, supercomputers, and GPUs separately)
Find other datasets and analyses
Combine all of the datasets into one
Produce more relevant data
Construct and measure a more relevant benchmark

Trends in the cost of computing


What’s up with nuclear weapons?

By Katja Grace, 27 February 2015
When nuclear weapons were first built, the explosive power you could extract from a tonne of explosive skyrocketed. But why?
Here’s a guess. Until nuclear weapons, explosives were based on chemical reactions. Whereas nuclear weapons are based on nuclear reactions. As you can see from the below table of specific energies and energy densities I got (and innocuously shortened) from Wikipedia, the characteristic scale of nuclear energy stored in things is about a hundred thousand times higher than that of chemical energy stored in things (by mass). And in particular, there are an empty three orders of magnitude between the most chemical energy packed into a thing and the least nuclear energy packed into a thing. This is perhaps to do with the fact that chemical reactions exploit the electromagnetic force, while nuclear reactions exploit the strong fundamental force.

Storage material
Energy type
Specific energy (MJ/kg)
Energy density (MJ/L)
Direct uses

Uranium (in breeder)
Nuclear fission
80,620,000[2]
1,539,842,000
Electric power plants (nuclear reactors), industrial process heat (to drive chemical reactions, water desalination, etc.)

Thorium (in breeder)
Nuclear fission
79,420,000[2]
929,214,000
Electric power plants (nuclear reactors), industrial process heat

Tritium
Nuclear decay
583,529
 ?
Electric power plants (nuclear reactors), industrial process heat

Hydrogen (compressed)
Chemical
142
5.6
Rocket engines, automotive engines, grid storage & conversion

methane or natural gas
Chemical
55.5
0.0364
Cooking, home heating, automotive engines, lighter fluid

Diesel / Fuel oil
Chemical
48
35.8
Automotive engines, power plants[3]

LPG (including Propane/ Butane)
Chemical
46.4
26
Cooking, home heating, automotive engines, lighter fluid

Jet fuel
Chemical
46
37.4
Aircraft

Gasoline (petrol)
Chemical
44.4
32.4
Automotive engines, power plants

Fat (animal/vegetable)
Chemical
37
34
Human/animal nutrition

Ethanol fuel (E100)
Chemical
26.4
20.9
Flex-fuel, racing, stoves, lighting

Coal
Chemical
24

Electric power plants, home heating

Methanol fuel (M100)
Chemical
19.7
15.6
Racing, model engines, safety

Carbohydrates(including sugars)
Chemical
17

Human/animal nutrition

Protein
Chemical
16.8

Human/animal nutrition

Wood
Chemical
16.2

Heating, outdoor cooking

TNT
Chemical
4.6

Explosives

Gunpowder
Chemical
3

Explosives

Thus it seems very natural that the first, lousiest, nuclear weapons that anyone could invent would be much more explosive than any chemical weapon ever known. The power of explosives is mostly a matter of physics, and physics contains discontinuities, for some reason.
But this doesn’t quite explain it. Consider cars. Turbojet propelled cars seem just fundamentally capable of greater speeds than internal combustion engine propelled cars. But the first turbojet cars that were faster than internal combustion cars were not much faster—it looks like they just had a steeper trajectory, which passed other cars and kept climbing. I’m not sure what caused this pattern in the car case specifically, but I hear it’s common. Maybe people basically know what current technology is capable of, and introduce new things as soon as they can be done at all, rather than as soon as they can be done well.
Anyway, we could imagine the same thing happening with nuclear weapons: even if nuclear power was fundamentally very powerful, the first nukes could have made use of it very badly, exploding like a weak chemical explosive the first times, but being quickly improved.
But that isn’t how nuclear weapons work. For a nuclear weapon to be less explosive per mass it would need to contain less fissile material, be smaller (so the outside casing is more of the mass, and so that fewer neutrons hit other atoms), or be less well contained (so fewer neutrons hit other atoms). But to get a nuclear explosion going at all, you need to get enough neutrons to hit other atoms that the chain reaction starts. Nuclear weapons have a ‘critical mass‘. I’m not sure how much less powerful the first nuclear weapons could easily have been than they were, but measly inexplosive nuclear weapons were basically out.
So the first nuclear weapons had to be much more explosive than the chemical explosives they replaced, because they were based on much more powerful reactions, and primitive nuclear weapons weren’t an option.
So nuclear weapons were basically guaranteed to revolutionize explosives in a single hop: even if humanity had known about nuclear reactions for hundreds of years, and put a tiny amount of effort into nuclear weapons research each year, humanity would never have seen feeble, not-much-better-than-TNT type nuclear weapons. There would just have been no nuclear weapons, and then at some point there would have been powerful nuclear weapons.
It is somewhat interesting that this is not what happened. Physicists mostly came to believe nuclear weapons were plausible from about 1939, and within a few years America spent nominal $19Bn (roughly 1% of 1943 GDP, but spread over a few years) on nuclear weapons, and built some. So our story is that progress in explosives was very slow, and then America spent a huge pile of money on it, and then it was very fast, but the progress was independent of the massive influx of funding.
That sounds surprising. But perhaps the influx of funding was because of the large natural discontinuity visible in the distance? Why would you ever spend small amounts of money every year, if it was clear at the outset that you had to spend a gajillion dollars to get anywhere? If there wasn’t much requirement for serial activities, probably you would just save it up and spend it in one go. America didn’t save it up though—they tried to build nuclear weapons basically as soon as they realized it was feasible at all. So it looks like nuclear weapons were just discovered after it was cost-effective to build them.
But if it was immediately cost-effective to build nuclear weapons thousands of times more powerful than other bombs, then isn’t the requirement that nuclear weapons be fairly powerful irrelevant to the spending? If it was worth building powerful bombs immediately, then what does it matter if it is possible to build lesser weapons? Not really, because cost-effectiveness is relative. If is only possible to buy toothpaste in a large bucket, you will probably pay for it, and it will have been a good deal. However if it’s also available in small tubes, then the same bucket is probably a bad deal.
Similarly, if nuclear weapons must be powerful, then there’s a decent chance that as soon as they are discovered it will be cost-effective to spend a lot on them and make them so. However if they can come in many lower levels of quality, the same large amount of spending may not be cost effective, because it will often be better to spend an intermediate amount.
So a requirement that nuclear weapons be very explosive when they are first built could at least partly explain the huge amount of spending. And the inherently large amounts of energy available from nuclear reactions still seems relevant: any given amount of development will be cost-effective when it is more costly, if it is more effective compared to the alternative.
This also appears to fit in with an explanation of the further coincidence that there happened to be a huge war at the time. That is, the war made all military technologies more cost-effective, and thus made it more likely that when nuclear weapons became feasible to develop, they would already be cost-effective. However the war also makes it more likely that high quality weapons would already be cost-effective compared to cheaper counterparts, thus partly undermining the proposal that the large expenditure was due in part to nuclear weapons requiring a minimal level of quality.
Here’s another plausible explanation for the large expense: because of their extreme explosiveness, nuclear weapons were very cost-effective at the time they were first considered. That is, they could have been produced a lot more cheaply than they were. However, due to the war, America was willing to pay a lot to make them come faster. In particular, America was willing to keep paying to make them come faster up until the point when they were roughly as cost-effective as older weapons, taking into account the upfront cost of making them come faster. This would explain the large amount of spending, and perhaps also why it aligned so well with what America could barely afford. It also explains why nuclear weapons appear to have been very roughly as cost-effective as older weapons. However on its own, it seems to leave the large amount of spending and the large amount of progress as coincidences.
In other ways, this story is in line with what I know about the development of nuclear weapons. For instance, that enriching uranium via several different methods in parallel was around half of the cost of the Manhattan project, and that the project was a lot more expensive than other countries’ later nuclear weapons projects.
Perhaps the inherent explosiveness of nuclear weapons made them very cost-effective, and thus able to be sped up a lot and still be cost-effective? (Thus connecting the expense with the explosiveness) But if nuclear weapons had been too expensive already to speed up much, it seems we would have seen a similar amount of spending (or more) over a somewhat longer time. So on this story it seems the heavy spending didn’t cause the high explosiveness, and the high explosiveness (and thus cheapness) didn’t seem to cause the steep spending.
It seems there was probably one coincidence however: a physics discovery leading to weapons of unprecedented power was made just before the largest war in history, and it’s hard to see how the war and the discovery were related, unless history was choreographed to make Leo Szilard’s life interesting. Perhaps the weapons and the war are related because nuclear weapons caused us to think of WWII as a large war by averting later wars? But the World Wars really were quite large compared to wars in slightly earlier history, rather than just the last in a trend of growing conflicts. If there is at least one coincidence anyway, perhaps it doesn’t matter whether the massive expense is explained by the unique qualities of nuclear weapons or merely by the war inspiring haste.
In sum, my guesses: nuclear weapons represented abrupt progress in explosiveness because of a discontinuity inherent in physics, and because ineffective nuclear weapons weren’t feasible.  Coincidentally, at the same time as nuclear weapons were discovered, there was a large war. America spent a lot on nuclear weapons for a combination of reasons. Nuclear explosions were inherently unusually powerful, and so could be cost-effective while still being very expensive. They also required investment on a large scale, so were probably invested in at an unusually large scale. America probably also spent a lot more on nuclear weapons to get them quickly, because they were so cost-effective under the circumstances.
My guesses are pretty speculative however, and I’m not an expert here. Further speculation, or well-grounded theorizing, is welcome.
(image: Oak Ridge Y-12 Alpha Track)

What’s up with nuclear weapons?


Possible Empirical Investigations

In the course of our work, we have noticed a number of empirical questions which bear on our forecasts and might be (relatively) cheap to resolve. In the future we hope to address some of these.

Our partial list of investigations into forecasting AI timelines
Our list of investigations that bear on ‘multipolar’ AI scenarios
Look at the work of ancient or enlightenment mathematicians and control for possible selection effects in this analysis of historical mathematical conjectures.
Look for historical characterizations of the AI problem, and try to obtain unbiased (though uninformed) breakdowns of the problem which could be used to gauge progress.
Identify previous examples of technological projects with clear long-term goals, and then produce estimates of the time required to achieve those goals to varying degrees.
Analyze the performance of different versions of software for benchmark problems, like SAT solving or chess, and determine the extent to which hardware and software progress facilitated improvement.
Obtain a clearer picture of the extent to which historical developments in neuroscience have played a meaningful role in historical progress in AI. Our impression is that this influence has been minimal, but this judgment might be attributable to hindsight bias.
In the field of AI, estimate the ratio of spending on hardware to spending on researchers.
Estimate the change in inputs in mathematicians, scientists, or engineers, as a complement to estimates for rates of progress in those fields.
Estimate the historical and present size of the AI field, ideally with plausible adjustments for quality (for example performing in-depth investigations for a small number of random samples, perhaps invoking expert opinion) and using these as a basis for quality-adjustments.
Luke Muehlhauser and the Future of Life Institute‘s section on forecasting both list further projects.

Unfortunately this is an incomplete list (even of the ideas which have struck as promising during this project). We are beginning to flesh it out further in our aforementioned list of projects bearing on AI timelines.

Possible Empirical Investigations


Research topic: Hardware, software and AI

This is the first in a sequence of articles outlining research which could help forecast AI development.

Interpretation
Concrete research projects are in boxes. ∑5 ∆8  means we guess the project will take (very) roughly five hours, and we rate its value (very) roughly 8/10.
Most projects could be done to very different degrees of depth, or at very different scales. Our time cost estimates correspond to a size that we would be likely to intend if we were to do the project. Value estimates are merely ordinal indicators of worth, based on our intuitive sense, and unworthy of being taken very seriously.

1. How does AI progress depend on hardware and software?
At a high level, AI improves when people make better software, when they can run it on better hardware, when they gather bigger, better training sets, etc. This makes present-day hardware and software progress a natural place to look for evidence about when advanced AI will arrive. In order to interpret any such data however, it is important to know how these pieces fit together. For instance, is the progress we see now mostly driven by hardware progress, or software progress? Can the same level of performance usually be achieved by widely varying mixtures of hardware and software? Does progress on software depend on progress on hardware?
It is important to understand the relationship between hardware, software and AI for several reasons. If hardware progress is the main driver of AI progress, then quite different evidence would tell us about AI timelines than if software is the main driver. Thus different research is valuable, and different timelines are likely. Many people base their AI predictions on hardware progress, while others decline to, so it would be broadly useful to know whether one should. We also expect understanding here to be generally useful.
So we think research in this direction seems valuable. We also think several projects seem tractable. Yet little appears to have been done in this direction. Thus this topic seems a high priority.
1.1 How does AI progress depend qualitatively on hardware and software progress?
For instance, will human-level AI appear when we have both a certain amount of hardware, and certain developments in software? Or can hardware and software substitute for one another? Substitution seems a natural model of the relationship between hardware and software, since anecdotally many tasks can be done by low quality software and lots of hardware, or by high quality software and less hardware. However the extent of this is unclear. This kind of model is also not commonly used in estimating AI timelines, so judging whether it should be might be a useful contribution. Having a good model would also bear on the priority of other research directions. As far as we know, this issue has received almost no attention. It seems moderately tractable.
1.1.A Evaluate qualitative models of the relationships between hardware, software and AI ∑30 ∆5
One way to approach the question of qualitative relationships is to assume some model, and work on projects such as those in 1.2 that measure quantitative details of the model, then revise the model if the measurements don’t make sense in it. Before that step, we might spend a short time detailing plausible models, and examining empirical and theoretical evidence we might already have, or could cheaply find. If we were going to follow up with empirical research, we would think about what evidence we would expect the research to reveal, given alternative models.
~
For instance, we find the hardware-software indifference curve model described briefly above (and outlined better in a blog post) plausible. Here are some ways it might be inadequate, that we might consider in evaluating it:

‘Hardware’ and ‘software’ are not sufficiently measurable entities for a ‘level’ of each in some domain to produce a stable level of performance.
Performance depends strongly on other factors, e.g. exactly what kind of hardware and software progress you make, unique details of the software being developed, training data available.
Different problem types, and different performance metrics on them have different kinds of behavior
There are ‘indifference curves’ in a sense but they are not sufficiently consistent to be worth reasoning about.
Humanity’s technological progress is not well characterized by an expanding rectangle of feasible hardware and software levels, but more as a complicated region of feasible combinations.

1.2 How much do marginal hardware and software improvements alter AI performance?
As mentioned above, this question is key to determining which other investigations are worthwhile. Naturally, it could also change our timelines substantially. Thus this question seems thus important to resolve. We think the projects here are particularly tractable, though not particularly cheap. For all of these projects, we would probably choose a specific set of benchmarks on particular problems to focus on. We might do multiple of these projects on the same set of benchmarks, to trace a more complete picture.
1.2.A Search for natural experiments combining modern hardware and early software approaches or vice versa. ∑80 ∆7
For instance, we might find early projects with very large hardware budgets, or recent projects with intentionally restricted hardware. Where these were tested on commonly used benchmarks, we can use them to map out the broad contributions of hardware and software to progress. For instance, if very small chess programs today run better than old chess programs which used similar (but then normal) amounts of hardware, then the difference between them can be attributed to improving software, roughly.
1.2.B Apply a modern understanding of software to early hardware ∑2,000 ∆9
Choose a benchmark problem that people worked on in the past, e.g. in the 1980s. Use a modern understanding of AI to solve the problem again, still using 1980’s hardware. Compare this to how researchers did in the 1980’s. This project requires substantial time from at least one AI researcher. Ideally they would spend a similar amount of effort as the past researchers did, so it may be worth choosing a problem where it is known that an achievable level of effort was applied in the past.
1.2.C Apply early software understanding to modern hardware ∑2,000 ∆8
Using contemporary hardware and a 1970’s or 1980’s understanding of connectionism, observe the extent to which a modern AI researcher (or student) could replicate contemporary performance on benchmark AI problems. This project is relatively expensive, among those we are describing. It requires substantial time from collaborators with a historically accurate minimal understanding of AI. Students may satisfy this role well, if their education is incomplete in the right ways. One might compare to the work of similar students who had also learned about modern methods.
1.2.D Measure marginal effects of hardware and software in existing performance trends  ∑100 ∆8
Often the same software can be used with modest changes in hardware, so changes in performance from hardware over small margins can be measured. Improved software is also often written to be run on the same hardware as earlier software, so changes in performance from software alone can be measured over moderate margins. Thus we can often estimate these marginal changes from looking at existing performance measurements.
~
We can also look at overall progress over time on some applications, and factor out what we know about hardware or software change, assuming it is close to the marginal values measured by the above methods. For instance, we can see how much individual Go programs improve with more hardware, and then we can look at longer term improvements in computer Go, and guess how much of that improvement came from hardware, given our earlier estimate of marginal improvement from hardware. In general these estimates will be less valid over larger distances, as the impact of hardware or software diverge from their marginal impact, and because arbitrary of hardware and software can’t generally be combined without designing the software to make use of the hardware. Grace 2013 includes some work this project.
1.2.E Interview AI researchers on the relative importance of hardware and software in driving the progress they have seen. ∑20 ∆7
AI researchers likely have firsthand experience regarding how hardware and software contribute to overall progress within the vicinity of their own work. This project will probably give relatively noisy estimates, but is very cheap compared to others described here. One could just ask for views on this question, and supporting anecdotes, or devise a more structured questionnaire beforehand.
1.3 How do hardware and software progress interact?
Do hardware and software progress relatively independently, or for instance do advances in hardware encourage advances in software? This might change how we generally expect software progress to proceed, and what combinations of hardware and software we expect to first produce human-level AI. We are likely to get some information about this from other projects looking at historical performance data e.g. 1.2.D. For instance, if overall progress is generally proportional to hardware progress, even as hardware progress varies, then this would be suggestive. Below are further possibilities.
1.3.A Find natural experiments ∑80 ∆4
Search for performance data from cases where hardware being used for an application was largely constant then shifted upward at some point. Such cases are probably hard to find, and hard to interpret when found. However, a short search for them may be worthwhile.
1.3.B Interview researchers ∑20 ∆7
If hardware tends to affect software research, it is likely that researchers notice this, and can talk about it. This seems a cheap and effective method of learning qualitatively about the topic. This project should probably be combined with 1.2.E.
1.3.C Consider plausible models ∑10 ∆5
This is a short theoretical project that would benefit from being done in concert with 1.3.B (interview researchers), since researchers probably have a relatively good understanding of which models are plausible, and we are likely to ask better questions of them if we have thought about the topic. This project should probably be combined with 1.1.A.


Multipolar research questions

By Katja Grace, 11 February 2015
The Multipolar AI workshop we ran a fortnight ago went well, and we just put up a list of research projects from it. I hope this is helpful inspiration to those of you thinking about applying to the new FLI grants in the coming weeks.
Thanks to the many participants who contributed ideas!
 (Image by Evan Amos)

Multipolar research questions