Brain performance in TEPS

Traversed Edges Per Second (TEPS) is a benchmark for measuring a computer’s ability to communicate information internally. Given several assumptions, we can also estimate the human brain’s communication performance in terms of TEPS, and use this to meaningfully compare brains to computers. We estimate that (given these assumptions) the human brain performs around  0.18 – 6.4 \* 1014 TEPS. This is within an order of magnitude more than existing supercomputers.

At current prices for TEPS, we estimate that it costs around $4,700 – $170,000/hour to perform at the level of the brain. Our best guess is that ‘human-level’ TEPS performance will cost less than $100/hour in seven to fourteen years, though this is highly uncertain.

Motivation: why measure the brain in TEPS?
------------------------------------------

### **Why measure communication?**

Performance benchmarks such as floating point operations per second (FLOPS) and millions of instructions per second (MIPS) mostly measure how fast a computer can perform individual operations. However a computer also needs to move information around between the various components performing operations.[1](https://aiimpacts.org/brain-performance-in-teps/#easy-footnote-bottom-1-510 "“According to Richard Murphy, a Principal Member of the Technical Staff at Sandia, “The Graph500’s goal is to promote awareness of complex data problems.” He goes on to explain, “Traditional HPC benchmarks – HPL being the preeminent – focus more on compute performance. Current technology trends have led to tremendous imbalance between the computer’s ability to calculate and to move data around, and in some sense produced a less powerful system as a result. Because “big data” problems tend to be more data movement and less computation oriented, the benchmark was created to draw awareness to the problem.”…And yet another perspective comes from Intel’s John Gustafson, a Director at Intel Labs in Santa Clara, CA, “The answer is simple: Graph 500 stresses the performance bottleneck for modern supercomputers. The Top 500 stresses double precision floating-point, which vendors have made so fast that it has become almost completely irrelevant at predicting performance for the full range of applications. Graph 500 is communication-intensive, which is exactly what we need to improve the most. Make it a benchmark to win, and vendors will work harder at relieving the bottleneck of communication.”” – Marvyn, The Case for the Graph 500 – Really Fast or Really Productive? Pick One") This communication takes time, space and wiring, and so can substantially affect overall performance of a computer, especially on data intensive applications. Consequently when comparing computers it is useful to have performance metrics that emphasize communication as well as ones that emphasize computation. When comparing computers to the brain, there are further reasons to be interested in communication performance, as we shall see below.

#### Communication is a plausible bottleneck for the brain

In modern high performance computing, communication between and within processors and memory is often a significant cost.[2](https://aiimpacts.org/brain-performance-in-teps/#easy-footnote-bottom-2-510 "“Unfortunately, due to a lack of locality, graph applications are often memory-bound on shared-memory systems or communication-bound on clusters.” – Beamer et al, Graph Algorithm Platform") [3](https://aiimpacts.org/brain-performance-in-teps/#easy-footnote-bottom-3-510 "“While traditional performance benchmarks for high-performance computers measure the speed of arithmetic operations, memory access time is a more useful performance gauge for many large problems today. The Graph 500 benchmark has been developed to measure a computer’s performance in memory retrieval…Results are explained in detail in terms of the machine architecture, which demonstrates that the Graph 500 benchmark indeed provides a measure of memory access as the chief bottleneck for many applications.” Angel et al (2012), The Graph 500 Benchmark on a Medium-Size Distributed-Memory Cluster with High-Performance Interconnect") [4](https://aiimpacts.org/brain-performance-in-teps/#easy-footnote-bottom-4-510 "“The Graph 500 was created to chart how well the world’s largest computers handle such data intensive workloads…In a nutshell, the Graph 500 benchmark looks at “how fast [a system] can trace through random memory addresses,” Bader said. With data intensive workloads, “the bottleneck in the machine is often your memory bandwidth rather than your peak floating point processing rate,” he added.” Jackson (2012) World’s most powerful big data machines charted on Graph 500") [5](https://aiimpacts.org/brain-performance-in-teps/#easy-footnote-bottom-5-510 "“Making transistors — the tiny on-off switches of silicon chips — smaller and smaller has enabled the computer revolution and the $1 trillion-plus electronics industry. But if some smart scientist doesn’t figure out how to make copper wires better, progress could grind to a halt. In fact, the copper interconnection between transistors on a chip is now a bigger challenge than making the transistors smaller.” Takahashi (2012) Copper wires might be the bottleneck in the way of Moore’s Law") Our impression is that in many applications it is more expensive than performing individual bit operations, making operations per second a less relevant measure of computing performance.

We should expect computers to become increasingly bottlenecked on communication as they grow larger, for theoretical reasons. If you scale up a computer, it requires linearly more processors, but superlinearly more connections for those processors to communicate with one another quickly. And empirically, this is what happens: the computers which prompted the creation of the TEPS benchmark were large supercomputers.

It’s hard to estimate the relative importance of computation and communication in the brain. But there are some indications that communication is an important expense for the human brain as well. A substantial part of the brain’s energy is used to transmit action potentials along axons rather than to do non-trivial computation.[6](https://aiimpacts.org/brain-performance-in-teps/#easy-footnote-bottom-6-510 "See Lennie (2003), table 1. Spikes and resting potentials appear to make up around 40% of energy use in the brain. Around 30% of energy in spikes is spent on axons, and we suspect more of the energy on resting potentials is spent on axons. Thus we estimate that at least 10% of energy in the brain is used on communication. We don’t know a lot about the other components of energy use in this chart, so the fraction could be much higher.") Our impression is also that the parts of the brain responsible for communication (e.g. axons) comprise a substantial fraction of the brain’s mass. That substantial resources are spent on communication suggests that communication is high value on the margin for the brain. Otherwise, resources would likely have been directed elsewhere during our evolutionary history.

Today, our impression is that networks are typically implemented on single machines because communication between processors is otherwise very expensive. But the power of individual processors is not increasing as rapidly as costs are falling, and even today it would be economical to use thousands of machines if doing so could yield human-level AI. So it seems quite plausible that communication will become a very large bottleneck as neural networks scale further.

In sum, we suspect communication is a bottleneck for the brain for three reasons: the brain is a large computer, similar computing tasks tend to be bottlenecked in this way, and the brain uses substantial resources on communication.

If communication is a bottleneck for the brain, this suggests that it will also be a bottleneck for computers with similar performance to the brain. It does not strongly imply this: a different kind of architecture might be bottlenecked by different factors.

#### Cost-effectiveness of measuring communication costs

It is much easier to estimate communication within the brain than to estimate computation. This is because action potentials seem to be responsible for most of the long-distance communication[7](https://aiimpacts.org/brain-performance-in-teps/#easy-footnote-bottom-7-510 "“To achieve long distance, rapid communication, neurons have evolved special abilities for sending electrical signals (action potentials) along axons. This mechanism, called conduction, is how the cell body of a neuron communicates with its own terminals via the axon. Communication between neurons is achieved at synapses by the process of neurotransmission.” – Stufflebeam (2008), Neurons, Synapses, Action Potentials and Neurotransmission"), and their information content is relatively easy to quantify. It is much less clear how many ‘operations’ are being done in the brain, because we don’t know in detail how the brain represents the computations it is doing.

Another issue that makes computing performance relatively hard to evaluate is the potential for custom hardware. If someone wants to do a lot of similar computations, it is possible to design custom hardware which computes much faster than a generic computer. This could happen with AI, making timing estimates based on generic computers too late. Communication may also be improved by appropriate hardware, but we expect the performance gains to be substantially smaller. We have not investigated this question.

Measuring the brain in terms of communication is especially valuable because it is a relatively independent complement to estimates of the brain’s performance based on computation. [Moravec](http://www.scientificamerican.com/article/rise-of-the-robots/), [Kurzweil](http://en.wikipedia.org/wiki/The_Singularity_Is_Near) and [Sandberg and Bostrom](http://www.fhi.ox.ac.uk/brain-emulation-roadmap-report.pdf) have all estimated the brain’s computing performance, and used this to deduce AI timelines. We don’t know of estimates of the total communication within the brain, or the cost of programs with similar communication requirements on modern computers. These an important and complementary aspect of the cost of ‘human-level’ computing hardware.

### **TEPS**

[Traversed edges per second](http://en.wikipedia.org/wiki/Traversed_edges_per_second) (TEPS) is a metric that was recently developed to measure communication costs, which were seen as neglected in high performance computing.[8](https://aiimpacts.org/brain-performance-in-teps/#easy-footnote-bottom-8-510 "“According to Richard Murphy, a Principal Member of the Technical Staff at Sandia, “The Graph500’s goal is to promote awareness of complex data problems.” He goes on to explain, “Traditional HPC benchmarks – HPL being the preeminent – focus more on compute performance. Current technology trends have led to tremendous imbalance between the computer’s ability to calculate and to move data around, and in some sense produced a less powerful system as a result. Because “big data” problems tend to be more data movement and less computation oriented, the benchmark was created to draw awareness to the problem.”- Marvyn, The Case for the Graph 500 – Really Fast or Really Productive? Pick One

“The Graph 500 was created to chart how well the world’s largest computers handle such data intensive workloads…In a nutshell, the Graph 500 benchmark looks at “how fast [a system] can trace through random memory addresses,” Bader said. With data intensive workloads, “the bottleneck in the machine is often your memory bandwidth rather than your peak floating point processing rate,” he added.” Jackson (2012) World’s most powerful big data machines charted on Graph 500

“While traditional performance benchmarks for high-performance computers measure the speed of arithmetic operations, memory access time is a more useful performance gauge for many large problems today. The Graph 500 benchmark has been developed to measure a computer’s performance in memory retrieval…Results are explained in detail in terms of the machine architecture, which demonstrates that the Graph 500 benchmark indeed provides a measure of memory access as the chief bottleneck for many applications.” Angel et al (2012), The Graph 500 Benchmark on a Medium-Size Distributed-Memory Cluster with High-Performance Interconnect") The TEPS benchmark measures the time required to perform a [breadth-first search](http://en.wikipedia.org/wiki/Breadth-first_search) on a large random graph, requiring propagating information across every edge of the graph (either by accessing memory locations associated with different nodes, or communicating between different processors associated with different nodes).[9](https://aiimpacts.org/brain-performance-in-teps/#easy-footnote-bottom-9-510 "From Graph 500 specifications page:

The benchmark performs the following steps:

Generate the edge list.

Construct a graph from the edge list (timed, kernel 1).

Randomly sample 64 unique search keys with degree at least one, not counting self-loops.

For each search key:

Compute the parent array (timed, kernel 2).

Validate that the parent array is a correct BFS [breadth first search] search tree for the given search tree.

Compute and output performance information.

")  You can read about the benchmark in more detail at the [Graph 500 site](http://www.graph500.org/specifications).

### **TEPS as a meaningful way to compare brains and computers**

#### Basic outline of how to measure a brain in TEPS

Though a brain cannot run the TEPS benchmark, we can roughly assess the brain’s communication ability in terms of TEPS. The brain is a large network of neurons, so we can ask how many edges between the neurons (synapses) are traversed (transmit signals) every second. This is equivalent to TEPS performance in a computer in the sense that the brain is sending messages along edges in a graph. However it differs in other senses. For instance, a computer with a certain TEPS performance can represent many different graphs and transmit signals in them, whereas we at least do not know how to use the brain so flexibly. This calculation also makes various assumptions, to be discussed shortly.

One important interpretation of the brain’s TEPS performance calculated in this way is as a lower bound on communication ability needed to simulate a brain on a computer to a level of detail that included neural connections and firing. The computer running the simulation would need to be traversing this many edges per second in the graph that represented the brain’s network of neurons.

#### Assumptions

##### Most relevant communication is between neurons

The brain could be simulated at many levels of detail. For instance, in the brain, there is both communication between neurons and communication within neurons. We are considering only communication between neurons. This means we might underestimate communication taking place in the brain.

Our impression is that essentially all long-distance communication in the brain takes place between neurons, and that such long-distance communication is a substantial fraction of the brain’s communication. The reasons for expecting communication to be a bottleneck—that the brain spends much matter and energy on it; that it is a large cost in large computers; and that algorithms which seem similar to the brain tend to suffer greatly from communication costs—also suggest that long distance communication alone is a substantial bottleneck.

##### Traversing an edge is relevantly similar to spiking

We are assuming that a computer traversing an edge in a graph (as in the TEPS benchmark) is sufficient to functionally replicate a neuron spiking. This might not be true, for instance if the neuron spike sends more information than the edge traversal. This might happen if there were more perceptibly different times each second at which the neuron could send a signal. We could usefully refine the current estimate by measuring the information contained in neuron spikes and traversed edges.[10](https://aiimpacts.org/brain-performance-in-teps/#easy-footnote-bottom-10-510 "One author personally expects this to make a difference of less than about a factor of two. He would be surprised if action potentials transferred a lot more information than edge traversals in the TEPS benchmark. Also, in general, increasing time resolution only increases the information contained in a signal logarithmically. That is, if neurons can send signals at twice as many different times, this only adds one bit of information to their message. However we have not investigated this topic.")

##### Distributions of edges traversed don’t make a material difference

The distribution of edges traversed in the brain is presumably quite different from the one used in the TEPS benchmark. We are ignoring this, assuming that it doesn’t make a large difference to the number of edges that can be traversed. This might not be true, if for instance the ‘short’ connections in the brain are used more often. We know of no particular reason to expect this, but it would be a good thing to check in future.

##### Graph characteristics are relevantly similar

Graphs vary in how many nodes they contain, how many connections exist between nodes, and how the connections are distributed. If these parameters are quite different for the brain and the computers tested on the TEPS benchmark, we should be more wary interpreting computer TEPS performance as equivalent to what the brain does. For instance, if the brain consisted of a very large number of nodes with very few connections, and computers could perform at a certain level on much smaller graphs with many connections, then even if the computer could traverse as many edges per second, it may not be able to carry out the edge traversals that the brain is doing.

However graphs with different numbers of nodes are more comparable than they might seem. Ten connected nodes with ten links each can be treated as one node with around ninety links. The links connecting the ten nodes are a small fraction of those acting as outgoing links, so whether the central ‘node’ is really ten connected nodes should make little difference to a computer’s ability to deal with the graph. The most important parameters are the number of edges and the number of times they are traversed.

We can compare the characteristics of brains and graphs in the TEPS benchmark. The TEPS benchmark uses graphs with up to 2 \* 1012nodes,[11](https://aiimpacts.org/brain-performance-in-teps/#easy-footnote-bottom-11-510 "According to the Graph 500, 2014 list sorted by problem scale, ‘Problem scale’ refers to base two logarithm of the number of graph vertices, and the largest problem scale is 41 (for Sequoia). 241 = 2.2 * 1012") while the human brain has around [1011 nodes (neurons)](http://aiimpacts.org/scale-of-the-human-brain/ "Scale of the Human Brain"). Thus the human brain is around twenty times smaller (in terms of nodes) than the largest graphs used in the TEPS benchmark.

The brain contains many more links than the TEPS benchmark graphs. TEPS graphs appear to have average degree 32 (that is, each node has 32 links on average),[12](https://aiimpacts.org/brain-performance-in-teps/#easy-footnote-bottom-12-510 "This page (section 3.4) at the Graph 500 site suggests that ‘edgefactor’ is 16 for the parameter settings they use, and that ‘edgefactor’ is half of degree. Note that our count for the ‘degree’ of a neuron also reflects both incoming and outgoing synapses.") while the brain apparently has average degree around 3,600 – 6,400.[13](https://aiimpacts.org/brain-performance-in-teps/#easy-footnote-bottom-13-510 "The brain has 1.8-3.2 x 10¹⁴ synapses and 1011 neurons, implying each neuron is connected to an average of 1.8-3.2 x 10¹⁴ * 2/ 1011 synapses, which is 3,600 – 6,400")

The distribution of connections in the brain and the TEPS benchmark are probably different. Both are [small-world](http://en.wikipedia.org/wiki/Small-world_network) distributions, with some highly connected nodes and some sparsely connected nodes, however we haven’t compared them in depth. The TEPS graphs are produced randomly, which should be a particularly difficult case for traversing edges in them (according to our understanding). If the brain has more local connections, traversing edges in it should be somewhat easier.

We expect the distribution of connections to make a small difference. In general, the time required to do a [breadth first search](http://en.wikipedia.org/wiki/Breadth-first_search) depends linearly on the number of edges, and doesn’t depend on degree. The TEPS benchmark is essentially a breadth first search, so we should expect it basically have this character. However in a physical computer, degree probably matters somewhat. We expect that in practice that the cost scales with edges \* log(edges), because the difficulty of traversing each edge should scale with log(edges) as edges become more complex to specify. A graph with more local connections and fewer long-distance connections is much like a smaller graph, so that too should not change difficulty much.

How many TEPS does the brain perform?
-------------------------------------

We can calculate TEPS performed by the brain as follows:

**TEPS = synapse-spikes/second in the brain**

**= Number of synapses in the brain \* Average spikes/second in synapses**

≈ **Number of synapses in the brain \* Average spikes/second in neurons**

**= [1.8-3.2 x 10^14](http://aiimpacts.org/scale-of-the-human-brain/ "Scale of the Human Brain")  \*  [0.1-2](http://aiimpacts.org/rate-of-neuron-firing/ "Neuron firing rates in humans")**

**= 0.18 – 6.4 \* 10^14**

That is, the brain performs at around 18-640 trillion TEPS.

Note that the average firing rate of neurons is not necessarily equal to the average firing rate in synapses, even though each spike involves both a neuron and synapses. Neurons have many synapses, so if neurons that fire faster tend to have more or less synapses than slower neurons, the average rates will diverge. We are assuming here that average rates are similar. This could be investigated further.

For comparison, the highest TEPS performance by a computer is 2.3 \* 10^13 TEPS (23 trillion TEPS)[14](https://aiimpacts.org/brain-performance-in-teps/#easy-footnote-bottom-14-510 "According to the Graph 500 November 2014 rankings, Sequoia at Lawrence Livermore National Laboratory can perform at 23,751 GTEPS."), which according to the above figures is within the plausible range of brains (at the very lower end of the range).

Implications
------------

That the brain performs at around 18-640 trillion TEPS means that if communication is in fact a major bottleneck for brains, and also for computer hardware functionally replicating brains, then existing hardware can probably already perform at the level of a brain, or at least at one thirtieth of that level.

### **Cost of ‘human-level’ TEPS performance**

We can also calculate the price of a machine equivalent to a brain in TEPS performance, given current prices for TEPS:

**Price of brain-equivalence = TEPS performance of brain \* price of TEPS**

= **TEPS performance of brain/billion \* price of GTEPS**

**= 0.18 – 6.4 \* 10^14/10^9 \* [$0.26/hour](http://aiimpacts.org/cost-of-teps/ "The cost of TEPS")**

**= $0.047 – 1.7 \* 10^5/hour**

**= $4,700 – $170,000/hour**

For comparison, supercomputers seem to cost around $2,000-40,000/hour to run, if we amortize their costs across three years.[15](https://aiimpacts.org/brain-performance-in-teps/#easy-footnote-bottom-15-510 "“The K Computer in Japan, for example, cost more than $1 billion to build and $10 million to operate each year. Livermore told us it spent roughly $250 million on Sequoia.” – Ars Technica, 2012. This makes the K computer over $38,000/hour.

“In other UK supercomputer news today Daresbury Laboratory in Cheshire has become home to the UK’s most powerful supercomputer…The cost of this system appears to be 10 times (£37.5 million) the above mentioned grant to develop the Emerald GPU supercomputer.” – Hexus, 2012. This places Blue Joule at around $2,100/hour to run. We evaluated the costs of several other supercomputers, and they fell roughly in this range.") So the lower end of this range is within what people pay for computing applications (naturally, since the brain appears to be around as powerful as the largest supercomputers, in terms of TEPS). The lower end of the range is still about 1.5 orders of magnitude more than what people regularly pay for labor. Though the highest paid CEOs appear to make at least $12k/hour.[16](https://aiimpacts.org/brain-performance-in-teps/#easy-footnote-bottom-16-510 "According to Forbes, seven CEOs earn more than $50M per year. If we assume they work 80 hour weeks and take no holidays, this is around $12k/hour ")

### **Timespan for ‘human-level’ TEPS to arrive**

[Our best guess](http://aiimpacts.org/cost-of-teps/ "The cost of TEPS") is that TEPS/$ grows by a factor of ten every four years, roughly. Thus for computer hardware to compete on TEPS with a human who costs $100/hour should take about seven to thirteen years.[17](https://aiimpacts.org/brain-performance-in-teps/#easy-footnote-bottom-17-510 "4*log(47) – 4*log(1,700)") We are [fairly unsure](http://aiimpacts.org/cost-of-teps/ "The cost of TEPS") of the growth rate of TEPS however.

---

Brain performance in TEPS


Glial Signaling

The presence of glial cells may increase the capacity for signaling in the brain by a small factor, but is unlikely to qualitatively change the nature or extent of signaling in the brain.

Support
-------

### Number of glial cells

[Azevado et al.](http://www.google.com/url?q=http%3A%2F%2Fwww.ncbi.nlm.nih.gov%2Fpubmed%2F19226510&sa=D&sntz=1&usg=AFrqEzcqwJNvttCpOXugbm4aGXwFzs1lvQ) physically count the number of cells in a human brain and find about 10¹¹ each of neurons and glial cells, suggesting that the number of glia is quite similar to the number of neurons.

References to much larger numbers of glial cells appear to be common, but we could not track down the empirical research supporting these claims. For example the Wikipedia article on neuroglia [states](http://en.wikipedia.org/wiki/Neuroglia) “In general, neuroglial cells are smaller than neurons and outnumber them by five to ten times,” and an article about glia in Scientific American [opens](http://www.scientificamerican.com/article/the-root-of-thought-what/) “Nearly 90 percent of the brain is composed of glial cells, not neurons.” An [informal blog post](http://www.google.com/url?q=http%3A%2F%2Fneurocritic.blogspot.com%2F2009%2F09%2Ffact-or-fiction-there-ten-times-more.html&sa=D&sntz=1&usg=AFrqEzcOOa6NqYQuGpT_HmadO5ZAbFT9Mw) suggests that the factor of ten figure may be a popular myth, although that post also draws on Azevado et al. so should not be considered independent support.

### Nature of glial signaling

[Sandberg and Bostrom](http://www.google.com/url?q=http%3A%2F%2Fwww.fhi.ox.ac.uk%2Fbrain-emulation-roadmap-report.pdf&sa=D&sntz=1&usg=AFrqEzdz0Nu_-YYgvpIUCCdkCpuWTPVRMw) write: “…the time constants for glial calcium dynamics is generally far slower than the dynamics of action potentials (on the order of seconds or more), suggesting that the time resolution would not have to be as fine” (p. 36). This suggests that the computational role of glial cells is not too great.

[Newman and Zahs 1998](http://www.jneurosci.org/content/18/11/4022.full.pdf) mechanically stimulate glial cells in a rat retina, and find that this stimulation results in slow-moving waves of increased calcium concentration.[1](https://aiimpacts.org/glial-signaling/#easy-footnote-bottom-1-145 "“The resulting Ca2+ waves, traveling through astrocytes and Muller cells, were similar to those observed previously in the isolated retina (Newman and Zahs, 1997), although the propagation velocities were somewhat slower: 13.8 + 0.4 micrometers/sec (57) compared with 23.1 micrometers/sec in the isolated retina (where the bathing solution was supplemented with glutamate and ATP). In the eyecup, the largest Ca2+ waves attained a diameter of about 400 micrometers.” – Newman and Zahs 1998 ") These calcium waves had an effect on neuron activity (see figure 4 in their paper, which also provides some indication concerning the characteristic timescale). For reference, these speeds are about a million times slower than [action potential propagation](http://en.wikipedia.org/wiki/Conduction_velocity) (neuron firing). These figures support Sandberg and Bostrom’s claims, and as far as we are aware they are consistent with the broader literature on calcium dynamics.

[Astrocytes](http://en.wikipedia.org/wiki/Astrocyte)—a type of glial cell—take in information from action potentials (from neurons).[2](https://aiimpacts.org/glial-signaling/#easy-footnote-bottom-2-145 "“Instead of integrating membrane depolarization and hyperpolarization into action potential output, like neurons do, astrocytes sense and integrate information mainly through the generation of intracellular calcium (Ca2+) signals (Figure 1). It is now well-established that astrocytes are able to sense transmitters released by neurons and other glial cells (either astrocytes or microglia)” – Min, Santello, and Nevian, 2012")  There is [some evidence](http://www.nature.com/neuro/journal/v11/n4/full/nn0408-379.html) that a small fraction of glia can generate action potentials, though such cells are “estimated to represent 5–10% of the cells” and so unlikely to substantially change calculations based on neurons.

It seems possible that further study or a more comprehensive survey of the literature would reveal other high-bandwidth signaling between glial cells, or that timescale-based estimates for the bandwidth of calcium signaling are too low, but at the moment we have little reason to suspect this.

### Energy of glial signaling

If glia were performing substantially more computation than neurons, we would weakly expect them to consume more (or at least comparable) energy for a number of reasons:

* The energy demands of the brain are very significant. If glia could perform comparable computation with much lower energy, we would expect them to predominate in terms of volume, whereas this does not seem to be the case.
* It would be surprising if different computational elements in the brain exhibited radically different efficiency.

However, the majority of energy in the brain is used to maintain resting potentials and propagate action potentials, for example a popularization in Scientific American [summarizes](http://www.scientificamerican.com/article/why-does-the-brain-need-s/) “two thirds of the brain’s energy budget is used to help neurons or nerve cells “fire” or send signals.”

Although we can imagine many possible designs on which glia would perform most of the information transfer in the brain while neurons provided particular kinds of special-purpose communication at great expense, this does not seem likely given our current understanding. This provides further mild evidence that the computational role of glial cells is unlikely to substantially exceed the role of neurons.

Glial Signaling


Scale of the Human Brain

The brain has about 10¹¹ neurons and 1.8-3.2 x 10¹⁴ synapses. These probably account for the majority of computationally interesting behavior.

Support
-------

#### Number of neurons in the brain

The number of neurons in the brain is about 10¹¹. For instance, [Azevado et al](http://www.ncbi.nlm.nih.gov/pubmed/19226510) physically counted them and found 0.6-1 \* 10¹¹. [Eric Chudler](http://faculty.washington.edu/chudler/facts.html) has collected estimates from a range of textbooks, which estimate 1-2 x 10¹⁰ of these (10%-30%) are in the cerebral cortex.[1](https://aiimpacts.org/scale-of-the-human-brain/#easy-footnote-bottom-1-143 "Total number of neurons in cerebral cortex = 10 billion (from G.M. Shepherd, The Synaptic Organization of the Brain, 1998, p. 6). However, C. Koch lists the total number of neurons in the cerebral cortex at 20 billion (Biophysics of Computation. Information Processing in Single Neurons, New York: Oxford Univ. Press, 1999, page 87). ")

#### Number of synapses in the brain

The number of synapses in the brain is known much less precisely, but is probably about 10¹⁴. For instance [Human-memory.net](http://www.human-memory.net/brain_neurons.html) reports 10¹⁴-10¹⁵ (100 – 1000 trillion) synapses in the brain, with no citation or explanation. Wikipedia says the brain contains 100 billion neurons, with 7,000 synaptic connections each, for 7 x 10¹⁴ synapses in total, but this seems possibly in error.[2](https://aiimpacts.org/scale-of-the-human-brain/#easy-footnote-bottom-2-143 "“The human brain has a huge number of synapses. Each of the 1011 (one hundred billion) neurons has on average 7,000 synaptic connections to other neurons. It has been estimated that the brain of a three-year-old child has about 10¹⁵ synapses (1 quadrillion). This number declines with age, stabilizing by adulthood. Estimates vary for an adult, ranging from 10¹⁴ to 5 x 10¹⁴ synapses (100 to 500 trillion).” Wikipedia accessed April 13 ’15, citing “Do we have brain to spare?“. Neurology 64 (12): 2004–5. We have not accessed most of the Drachman paper, but it does at least say “Within the liter and a half of human brain, stereologic studies estimate that there are approximately 20 billion neocortical neurons, with an average of 7,000 synaptic connections each”. This suggests that the Wikipedia page errs in attributing the 7,000 synaptic connections per neuron to the brain at large instead of the neocortex.")

##### Number of synapses in the neocortex

One way to estimate of the number of synapses in the brain is to extrapolate from the number in the neocortex. According to stereologic studies that we have not investigated, there are around 1.4 x 10¹⁴ synapses in the neocortex.[3](https://aiimpacts.org/scale-of-the-human-brain/#easy-footnote-bottom-3-143 "“Within the liter and a half of human brain, stereologic studies estimate that there are approximately 20 billion neocortical neurons, with an average of 7,000 synaptic connections each”.”Do we have brain to spare?“. Neurology 64 (12): 2004–5. ") This is roughly consistent with [Eric Chudler’s summary of textbooks](http://faculty.washington.edu/chudler/facts.html), which gives estimates between 0.6-2.4 x 10¹⁴ for the number of synapses in the cerebral cortex.[4](https://aiimpacts.org/scale-of-the-human-brain/#easy-footnote-bottom-4-143 "“Number of synapses in cortex = 0.15 quadrillion (Pakkenberg et al., 1997; 2003)… [the ‘cortex’ probably refers either the cerebral cortex or the neocortex, which is part of and thus should be smaller than the cerebral cortex.]

…Total number of synapses in cerebral cortex = 60 trillion (yes, trillion) (from G.M. Shepherd, The Synaptic Organization of the Brain, 1998, p. 6). However, C. Koch lists the total synapses in the cerebral cortex at 240 trillion (Biophysics of Computation. Information Processing in Single Neurons, New York: Oxford Univ. Press, 1999, page 87).” – Chudler, Brain facts and figures")

We are not aware of convincing estimates for synaptic density outside of the cerebral cortex, and our impression is that widely reported estimates of 10¹⁴ are derived from the assumption that the neocortex contains the great bulk of synapses in the brain. This seems plausible given the large volume of the neocortex, despite the fact that it contains a minority of the brain’s neurons. By volume, around 80% of the human brain is neocortex.[5](https://aiimpacts.org/scale-of-the-human-brain/#easy-footnote-bottom-5-143 "Dunbar references anatomical measurements from 1981 and writes “With a neocortical volume of 1006.5 cc and a total brain volume of 1251.8 cc (Stephan et al. 1981), the neocortex ratio for humans is CR = 4.1.” (p. 682).") The neocortex also consumes around 44% of the brain’s total energy, which may be another reasonable indicator of the fraction of synapses in contains.[6](https://aiimpacts.org/scale-of-the-human-brain/#easy-footnote-bottom-6-143 "“Thus, neocortex accounts for 44% of the brain’s overall consumption.” Lennie, 2003 (p. 495)") So our guess is that the number of synapses in the entire brain is somewhere between 1.3 and 2.3 times the number in the cerebral cortex. From above, the cerebral cortex contains around 1.4 x 10¹⁴ synapses, so this gives us 1.8-3.2 x 10¹⁴ total synapses.

##### Number of synapses per neuron

The number of synapses per neuron varies considerably. [According to Wikipedia](http://en.wikipedia.org/wiki/Cerebellum_granule_cell#cite_note-SOB-1), the majority of neurons are cerebellum granule cells, which have only a handful of synapses, while the statistics above suggest that the average neuron has around 1,000 synapses. Purkinje cells have up to 200,000 synapses.[7](https://aiimpacts.org/scale-of-the-human-brain/#easy-footnote-bottom-7-143 "“Number of synapses made on a Purkinje cell = up to 200,000” – Chudler, Brain facts and figures")

#### Number of glial cells in the brain

*Main article: [Glial signaling](http://aiimpacts.org/glial-signaling/ "Glial Signaling")*

[Azevado et al](http://www.ncbi.nlm.nih.gov/pubmed/19226510) aforementioned investigation finds about 10¹¹ glial cells (the same as the number of neurons).

#### Relevance of cells other than neurons to computations in the brain

*Main article: [Glial signaling](http://aiimpacts.org/glial-signaling/ "Glial Signaling")*

It seems that the timescales of glial dynamics are substantially longer than for neuron dynamics. [Sandberg and Bostrom](http://www.fhi.ox.ac.uk/brain-emulation-roadmap-report.pdf) write: “However, the time constants for glial calcium dynamics is generally far slower than the dynamics of action potentials (on the order of seconds or more), suggesting that the time resolution would not have to be as fine” (p. 36). This suggests that the computational role of glial cells is not too great. References to much larger numbers of glial cells appear to be common, but we were unable to track down any empirical research supporting these claims. An [informal blog post](http://neurocritic.blogspot.com/2009/09/fact-or-fiction-there-ten-times-more.html) suggests that a common claim that there are ten times as many glial cells as neurons may be a popular myth.

We are not aware of serious suggestions that cells other than neurons or glia play a computationally significant role in the functioning of the brain.

---

Scale of the Human Brain


Neuron firing rates in humans

Our best guess is that an average neuron in the human brain transmits a spike about 0.1-2 times per second.

Support
-------

### Bias from neurons with sparse activity

When researchers measure neural activity, they can fail to see neurons which rarely fire during the experiment (those with ‘sparse’ activity).[1](https://aiimpacts.org/rate-of-neuron-firing/#easy-footnote-bottom-1-142 "Table 1 of Shoham et al. reports on a variety of investigations of sparsity in neural behavior, most of which suggest that more than 90% of neurons are sufficiently silent that they are not easily detectable. Summarizing their own results, they say “Table 1 suggests that such proportions may vary widely among different brain regions and preparations, a notion which is consistent with hierarchical, increasingly sparse neural coding schemes. Conservative estimates may, however, be possible by considering those parameters of the neuron–electrode interface that affect the detection of unit signals…suggesting a silent fraction of at least 90%.” (p. 782).Experimenters recording from a rat cortex find “Both electrical and optical recordings consistently revealed that individual neurons as well as populations of neurons display sparse spontaneous activity. Single neurons displayed low AP rates of <0.1 Hz, in agreement with previous in vivo studies.” (Kerr et al 2005)") Preferentially recording more active neurons means overestimating average rates of firing. The size of the bias seems to be around a factor of ten: it appears that around 90% of neurons are ‘silent’, so unlikely to be detected in these kinds of experiments. This suggests that many estimates should be scaled down by around a factor of around ten.

### **Assorted estimates**

#### **Informal estimates**

Informal websites and articles commonly report neurons as firing between <1 and 200 times per second.[2](https://aiimpacts.org/rate-of-neuron-firing/#easy-footnote-bottom-2-142 "‘But generally, the range for a “typical” neuron is probably from <1 Hz (1 spike per second) to ~200 Hz (200 spikes per second).’ -‘Astra Bryant, Ask a neuroscientist! – what is the synaptic firing rate of the human brain?’

“A typical neuron fires 5 – 50 times every second.” – www.human-memory.net

“The brain can’t handle neurons firing all the time. Neurons fire around 10x per second and already the brain is consuming 20% of the body’s energy at 2% of the body’s weight.” – Paul King, computational neuroscientist, on Quora”Modern computer chips handle data at the mind-blowing rate of some 10^13 bits per second. Neurons, by comparison, fire at a rate of around 100 times per second or so. And yet the brain outperforms the best computers in numerous tasks.” – MIT Technology Review") These sources lack references and are not very consistent, so we do not put much stock in them.

#### **Estimates of rate of firing in human neocortex**

Based on the energy budget of the brain, it [appears](http://aiimpacts.org/metabolic-estimates-of-rate-of-cortical-firing/) that the average cortical neuron fires around 0.16 times per second. It seems unlikely that the average cortical neuron spikes much more than once per second.

The neocortex is a large part of the brain. It accounts for around 80% of the brain’s volume[3](https://aiimpacts.org/rate-of-neuron-firing/#easy-footnote-bottom-3-142 "Dunbar references anatomical measurements from 1981 and writes “With a neocortical volume of 1006.5 cc and a total brain volume of 1251.8 cc (Stephan et al. 1981), the neocortex ratio for humans is CR = 4.1.” (p. 682)."), and uses 44% of its energy[4](https://aiimpacts.org/rate-of-neuron-firing/#easy-footnote-bottom-4-142 "“Using the best estimate, in the normal awake state, cortex accounts for 44% of whole brain energy consumption in 200 ms, the brain’s normal energy consumption supports a strong (solid horizontal line, intercept on ordinate).” – Lennie 2003"). It appears to hold at least a third of the brain’s synapses if not many more[5](https://aiimpacts.org/rate-of-neuron-firing/#easy-footnote-bottom-5-142 "

“The average total number of synapses in the neocortex of five young male brains was 164 x 10(12) (CV = 0.17).” Tang et al, 2001

“Number of synapses in cortex = 0.15 quadrillion (Pakkenberg et al., 1997; 2003)” – Eric Chudler

“The human brain has a huge number of synapses. Each of the 10^11 (one hundred billion) neurons has on average 7,000 synaptic connections to other neurons. It has been estimated that the brain of a three-year-old child has about 10^15 synapses (1 quadrillion). This number declines with age, stabilizing by adulthood. Estimates vary for an adult, ranging from 10^14 to 5 x 10^14 synapses (100 to 500 trillion).” Wikipedia accessed April 13 ’15, citing Drachman, D (2005). “Do we have brain to spare?”. Neurology 64 (12): 2004–5. We have not accessed most of the Drachman paper, but it does at least say “Within the liter and a half of human brain, stereologic studies estimate that there are approximately 20 billion neocortical neurons, with an average of 7,000 synaptic connections each”. It seems improbable that the average number of synapses per neuron in the brain is the same as that in the neocortex, weakly suggesting the Wikipedia contributor made an error.

These figures suggest that the neocortex accounts for between a third and most of synapses."). Thus we might use rates of firing of cortical neurons as a reasonable proxy for normal rates of neuron firing in the brain. We can also do a finer calculation.

We might roughly expect energy used by the brain to scale in proportion both to the spiking rate of neurons and to volume. This is because the energy required for every neuron to experience a spike scales up in proportion to the surface area of the neurons involved[6](https://aiimpacts.org/rate-of-neuron-firing/#easy-footnote-bottom-6-142 "“The cost of propagating an action potential in an unmyelinated axon is proportional to its surface area.” – Lennie, 2003"), which we expect to be roughly proportional to volume.

So we can calculate:

**energy(cortex) = volume(cortex) \* spike\_rate(cortex) \* c**

**energy(brain) = volume(brain) \* spike\_rate(brain) \* c**

For *c* a constant.

Thus,

**energy(cortex)/energy(brain) = volume(cortex) \* spike\_rate(cortex)/volume(brain) \* spike\_rate(brain)**

From figures given above then, we can estimate:

**0.44 = 0.8 \* 0.16/spike\_rate(brain)**

**spike\_rate(brain) = 0.8 \* 0.16 /0.44 = 0.29**

Or for a high estimate:

**0.44 = 0.8 \* 1/spike\_rate(brain)**

**spike\_rate(brain) = 0.8 \* 1 /0.44 = 1.82**

So based on this rough extrapolation from neocortical firing rates, we expect average firing rates across the brain to be around 0.29 per second, and probably less than 1.82 per second. This has been a very rough calculation however, and we do not have great confidence in these numbers.

#### **Estimates of rate of firing in non-human visual cortex**

[A study](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1688734/pdf/9447735.pdf) of macaque and cat visual cortex found rates of neural firing averaging 3-4 spikes per second for cats in different conditions, and 14-18 spikes per second for macaques. A past study found 9 spikes per second for cats.[7](https://aiimpacts.org/rate-of-neuron-firing/#easy-footnote-bottom-7-142 "“spikes were recorded while a given video sequence representative of natural scenes was played. Data were collected from three cats, and two macaques. The cats were anaesthetized and the macaques were awake and free viewing. Only visually responsive cells were used… For V1 of the anaesthetized cats, the firing rates for the video-stimulated neurons were low (mean = 3.96Hz, s.d. = 3.61Hz). This was lower than has been previously reported (Legendy & Salcman 1985) for the unanaesthetized cat (mean = 8.9 Hz, s.d.  = 7.0 Hz), but was significantly higher than when the cells were stimulated with high contrast white noise (mean = 2.45Hz, s.d. = 2.18 Hz). It is proposed that the low average rates were partly due to the effect of the anaesthetic (which could be tested by systematically varying its level). For the macaque IT cells, generally in the upper bank of the superior temporal sulcus at sites similar to those in (Rolls & Tovee 1995), the average rate was higher for both video stimulation (mean = ˆ18 Hz, s.d.ˆ =10.3 Hz), and blank screen viewing (meanˆ = 14 Hz, s.d.ˆ= 8.3 Hz.)” Baddeley et al 1997 (p. 1776)") It is hard to know how these estimates depend on the region being imaged and on the animal being studied, which significantly complicates extracting conclusions from these results. Furthermore, these studies appear to be subject to the bias discussed above, from only sampling visually responsive cells. Thus they probably overestimate overall neural activity by something like a factor of ten. This suggests figures in the 0.3-1.8 range, consistent with estimates from the neocortex. Note that the visual cortex [is part of](http://en.wikipedia.org/wiki/Neocortex) the neocortex, so this increases our confidence in our estimates for that, without reducing our uncertainty about the rest of the brain.

### **Maximum neural firing rates**

The *‘refractory period’* for a neuron is the time after it fires during which it either can’t fire again (*‘absolute refractory period’*) or needs an especially large stimulus to fire again (*‘relative refractory period’*). According to [physiologyweb.com](http://www.physiologyweb.com/lecture_notes/neuronal_action_potential/neuronal_action_potential_refractory_periods.html), absolute refractory periods tend to be 1-2ms and relative refractory periods tend to be 3-4ms.[8](https://aiimpacts.org/rate-of-neuron-firing/#easy-footnote-bottom-8-142 "Therefore, it takes about 3-4 ms for all Na+ channels to come out of inactivation in order to be ready for activation (opening) again. The period from the initiation of the action potential to immediately after the peak is referred to as the absolute refractory period (ARP) (see Figs. 1 and 2). This is the time during which another stimulus given to the neuron (no matter how strong) will not lead to a second action potential. Thus, because Na+ channels are inactivated during this time, additional depolarizing stimuli do not lead to new action potentials. The absolute refractory period takes about 1-2 ms…

…During the absolute refractory period, a second stimulus (no matter how strong) will not excite the neuron. During the relative refractory period, a stronger than normal stimulus is needed to elicit neuronal excitation.After the absolute refractory period, Na+channels begin to recover from inactivation and if strong enough stimuli are given to the neuron, it may respond again by generating action potentials. However, during this time, the stimuli given must be stronger than was originally needed when the neuron was at rest. This situation will continue until all Na+ channels have come out of inactivation. The period during which a stronger than normal stimulus is needed in order to elicit an action potential is referred to as the relative refractory period (RRP).") This implies than neurons are generally not capable of firing at more than 250-1000 Hz. This is suggestive, however the site does not say anything about the distribution of maximum firing rates for different types of neurons, so the mean firing rate could in principle be much higher.

**Conclusions**
---------------

Informal estimates place neural firing rates in the <1-200Hz range. Estimates from energy use in the neocortex suggests a firing rate of 0.16Hz in the neocortex, which suggests around 0.29Hz in the entire brain, and probably less than 1.8Hz, though we are not very confident in our estimation methodology here. We saw animal visual cortex firing rates in the 3-18Hz range, but these are probably an order of magnitude too high due to bias from recording active neurons, suggesting real figures of 0.3-1.8 Hz, which is consistent with the estimates from the neocortex previously discussed. Neuron refractory periods (recovery times) suggest 1000Hz is around as fast as a normal neuron can possibly fire. Combined with the observation that 90% of neurons rarely fire, this suggests 100Hz as a high upper bound on the average firing rate. However this does not tell us about unusual neurons, of which there might be many.

So we have two relatively weak lines of reasoning suggesting average firing rates of around 0.1Hz-2Hz. These estimates are low compared to the range of informal claims. However the informal claims appear to be unreliable, especially given that two are higher than our upper bound on neural firing rates (though these are also unreliable). 0.1-2Hz is also low compared to these upper bounds, as it should be. Thus our best guess is that neurons fire at 0.1-2Hz on average.

Notes
-----

Neuron firing rates in humans


Metabolic Estimates of Rate of Cortical Firing

Cortical neurons are estimated to spike around 0.16 times per second, based on the amount of energy consumed by the human neocortex.[1](https://aiimpacts.org/metabolic-estimates-of-rate-of-cortical-firing/#easy-footnote-bottom-1-144 "The article alternates between ‘cortex’ and ‘neocortex’ in a way that suggests they refer to the same, though we are not sure that this is common usage. For instance, on page 494 the article refers to a table entitled ‘Basic statistics of the human neocortex’ to learn about the ‘cortex’.") They seem unlikely to spike much more than once per second on average, based on this analysis.

Support
-------

### Energy spent on spiking

[Lennie 2003](http://www.bcs.rochester.edu/people/plennie/pdfs/Lennie03a.pdf) estimates the rate of neuron firing in the cortex based on estimates for energy spent on Na/K ion pumps during spikes, and the energy required by Na/K ion pumps per spike.

Lennie produces estimates for energy consumed in three parts:

* **Estimates for adenosine triphosphate (ATP) molecules consumed by the neocortex**: According to brain scans, glucose is metabolized at a rate of about 0.40 micro mol/g/min. Each glucose molecule yields around 30 molecules of ATP. This suggests that the entire cortex consumes 3.4 \* 10²¹ molecules of ATP per minute.[2](https://aiimpacts.org/metabolic-estimates-of-rate-of-cortical-firing/#easy-footnote-bottom-2-144 "“Positron emission tomography (PET) and Magnetic Resonance Spectroscopy (MRS) measurements of glucose metabolism in human cortex show overall resting consumption of about 0.40 micro mol/g/min [10–12]. Assuming a yield of 30 ATP per molecule of glucose [13], this would give rise to 12 micro mol ATP/g/min. With 1 cm3 of cortex weighing 1 g [14], from Table 1, the cortical mass is 475 g, resulting in a gross consumption of 3.4 * 10²¹ molecules of ATP per minute.” – Lennie 2003 (p494) ") Note that ATP’s function is as energy source, so this is a measure of how much energy the neocortex uses.
* **Estimates for the fraction of this ATP used to maintain ion balances**: If you inactivate Na/K ion pumps with the drug ouabain, this reduces energy consumption by 50%, suggesting that these ion pumps use about half of the cortex’s energy. [3](https://aiimpacts.org/metabolic-estimates-of-rate-of-cortical-firing/#easy-footnote-bottom-3-144 "“The principal cost of restoring and maintaining ionic balances can be estimated from the decrease in energy consumption brought about by inactivating the Na/K pump with ouabain or an equivalent agent. Doing this reduces overall energy consumption by about 50%.”- Lennie 2003") This gives us 1.7 \* 10²¹ molecules of ATP per minute being used to maintain ion balances.
* **Estimates for the fraction of ion balancing ATP used in spikes**: Maintaining resting potentials (not part of spiking) in all neurons costs 1.3 x 10²¹ ATP molecules per minute. This leaves 3.9 \* 10²⁰ ATP per minute for spiking.[4](https://aiimpacts.org/metabolic-estimates-of-rate-of-cortical-firing/#easy-footnote-bottom-4-144 "“the cost of maintaining resting potentials in all neurons and glia is 1.3 * 10²¹ ATP molecules per minute, leaving 3.9 * 10²⁰ per minute to support ionic movements associated with spikes,” i.e. that about 23% of energy is used for spiking.- Lennie 2003")

However, other authors report higher fractions of cortical energy are spent on spiking. [Laughlin 2001](http://uploads.tombertalan.com/13fall2013/501Aneu501A/hw/hw6/others/Laughlin-2001-CurrOpinNeuro.pdf) writes that spiking accounts for 80% of total energy consumption in mammalian cortex.[5](https://aiimpacts.org/metabolic-estimates-of-rate-of-cortical-firing/#easy-footnote-bottom-5-144 "“For a mammalian brain… recent studies in NMR spectroscopy, which associate energy usage with neural function by following the turnover of identified metabolites and neurotransmitters, suggest that signaling accounts for 80% of the total consumption in cortex…Maintaining resting potentials and counteracting leakage from organelles accounts for less than 15% of the total consumption.”(p. 475)") Other work by Laughlin and Attwell, which is a primary source for Lennie’s estimates, [reports](http://apps.webofknowledge.com/InboundService.do?UT=000171432300001&IsProductCode=Yes&mode=FullRecord&SID=2BGqCCCtS2EraP7OEQa&product=WOS&smartRedirect=yes&SrcApp=literatum&DestFail=http%3A%2F%2Fwww.webofknowledge.com%3FDestApp%3DCEL%26DestParams%3D%253Faction%253Dretrieve%2526mode%253DFullRecord%2526product%253DCEL%2526UT%253D000171432300001%2526customersID%253Datyponcel%26e%3D3WVyiuw2sBSOd7GLNSrxGn0O4OxdNDA5mYe7CY7nD5EW1pzAsO6Um4lIk4sEfDmD%26SrcApp%3Dliteratum%26SrcAuth%3Datyponcel&Init=Yes&action=retrieve&customersID=atyponcel&Func=Frame&SrcAuth=atyponcel) that spiking consumes around 47% of energy.[6](https://aiimpacts.org/metabolic-estimates-of-rate-of-cortical-firing/#easy-footnote-bottom-6-144 "“Action potentials and postsynaptic effects of glutamate are predicted to consume much of the energy (47% and 34%, respectively).”")

Our understanding is that the difference can be attributed to differences between the rodent brain and the human brain, and the scaling estimates from one to the other. We are not particularly confident in this methodology.

### Energy per spike

According to Lenny, each spike consumes around 2.4 \* 10⁹ molecules of ATP.[7](https://aiimpacts.org/metabolic-estimates-of-rate-of-cortical-firing/#easy-footnote-bottom-7-144 "“A spike consumes 2.4 109 molecules of ATP.” – Lennie 2003 (p494)") This estimate is produced by scaling up estimates for the rat brain.[8](https://aiimpacts.org/metabolic-estimates-of-rate-of-cortical-firing/#easy-footnote-bottom-8-144 "“Neurons in human neocortex are larger than those in rat and receive and make more synapses, but they are not otherwise known to differ in their basic structure or organization. Thus, with appropriate scaling of parameters for the larger neurons, Attwell and Laughlin’s analysis can be used to estimate the energy consumed by a pyramidal neuron in human neocortex.”") The estimates for the rat brain were inferred from ‘anatomic and physiologic data’, which we have not scrutinized.[9](https://aiimpacts.org/metabolic-estimates-of-rate-of-cortical-firing/#easy-footnote-bottom-9-144 "“Anatomic and physiologic data are used to analyze the energy expenditure on different components of excitatory signaling in the grey matter of rodent brain…

…Thus, the minimum Na+ influx to initiate the action potential and propagate it is 2.88 × 10⁸ Na+ (if dendrite depolarization were due to entry of Ca2 + instead of Na+ , with each Ca2 + extruded in exchange for 3 Na+ , this figure would increase by 6.8%). A realistic estimate of the Na+ entry needed is obtained by quadrupling this to take account of simultaneous activation of Na+ and K+ channels (Hodgkin, 1975), resulting in 11.5 × 10⁸ Na+ which have to be pumped out again, requir- ing 3.84 × 10⁸ ATP molecules to be hydrolyzed (Figs. 1B, 2, and 3). This 4-fold increase is validated by calculations by A. Roth and M. Hausser (as in Vetter et al., 2001), based on cell morphology and ionic current properties, which give ATP values of 3.3 × 10⁸ for a cortical pyramidal cell with a myelinated axon, and 5.4 × 10⁸ for a hippocampal pyramidal cell with an unmyelinated axon, similar to the estimate made above.” Attwell and Laughlin 2001 (pdf download)")  We are not particularly confident in this scaling methodology. These estimates appear to be produced by counting ion channels and applying detailed knowledge of the mechanics of ion channels (which consume a roughly fixed amount of ATP per transported molecule).

### Spikes per neuron per second

We saw above that the cortex uses 3.9 \* 10²⁰ ATP/minute for spiking, and that each spike consumes around 2.4 \* 10⁹ molecules of ATP. So the cortex overall has around 2.7 \* 10⁹ spikes per second.  There are 1.9 \*10¹⁰ neurons in the cortex, so together we can calculate that these neurons produce around 0.16 spikes per second on average.[10](https://aiimpacts.org/metabolic-estimates-of-rate-of-cortical-firing/#easy-footnote-bottom-10-144 "“From the previous section, reversing the Na+ and K+ fluxes moved by a single spike consumes 2.2 * 10⁹ ATP molecules. Given this, and 1.9 * 10¹⁰cortical neurons, the ATP available for the Na/K pump would support an average discharge rate of 0.16 spikes/s/neuron.” – Lennie 2003 (p494). That is, 3.9*10²⁰ ATP/minute/(60 seconds/ minute * 2.2* 10⁹  ATP/spike * 1.9 * 10¹⁰ neurons = 0.16 spikes/neuronsecond). Note that Lenny uses 2.2* 10⁹ ATP/spike here though he earlier said 2.4* 10⁹ ATP/spike. This inconsistency appears to be an error, but a small one).")

Even assuming that essentially all of the energy in the brain is spent on signaling, this would introduce a bias of only a factor of 8 in Lennie’s estimates. On page S1 Lennie presents an analysis of other possible sources of error, and overall it seems unlikely to us that the estimate is too low by more than an order of magnitude or so.

---


Preliminary prices for human-level hardware

*By Katja Grace, 4 April 2015*

Computer hardware has been getting cheap now for about [seventy five years](http://aiimpacts.org/trends-in-the-cost-of-computing/ "Trends in the cost of computing"). Relatedly, large computing projects can afford to be [increasingly large](http://en.wikipedia.org/wiki/TOP500#/media/File:Supercomputers-history.svg). If you think the human brain is something like a really impressive computer[1](https://aiimpacts.org/preliminary-prices-for-human-level-hardware/#easy-footnote-bottom-1-490 "The extent and nature of the brain’s resemblance to a really impressive computer is an important question that we hope to discuss another time. For now, we shall see what this assumption implies, and whether it is worth resolving more nebulous problems for."), then a natural question is ‘what happens when big projects can afford to use as much computation as the brain?’

One possibility is that we get something like human-level AI then and there. Maybe the brain just uses easy algorithmic ideas, and as soon as someone gets enough hardware together and tries a bit on the brain-like software, the creation will be about as good as a human brain.

Another possibility is that sufficient hardware to simulate a brain is basically irrelevant. On this story, the brain has big algorithmic secrets we don’t know about, and it would take unimaginable warehouses of hardware to replace these insights with brute force computation.

Most possibilities lie somewhere between these two, where hardware and software are both somewhat helpful. In these cases, knowing when we will have enough hardware to run a brain isn’t everything, but is informative. For more discussion of how to model this situation, see our [older post](http://aiimpacts.org/how-ai-timelines-are-estimated/ "How AI timelines are estimated").

There doesn’t seem to be consensus on how important hardware is. But given the range of possibilities, whenever ‘human-level hardware’ arrives seems like a disproportionately likely time for human-level AI to arrive.

For this reason, we want to know when this human-level hardware point is. Which means we want to know the price of hardware, how fast that price is falling, how much hardware you need to do what the human brain does, and how much anyone is likely to pay for that. Ideally, we would like a few relatively independent estimates of several of these things.

We’ve just been checking [the](http://aiimpacts.org/cost-of-teps/ "The cost of TEPS") [prices](http://aiimpacts.org/current-flops-prices/ "Current FLOPS prices"), and [price trends](http://aiimpacts.org/trends-in-the-cost-of-computing/ "Trends in the cost of computing"), so this seems like a good time to pause and see what they imply when combined with some existing estimates of the brain’s computational requirements. Later we will explore these estimates in more detail, and hopefully add at least one based on measuring [TEPS](http://aiimpacts.org/cost-of-teps/ "The cost of TEPS").

[Moravec](http://www.scientificamerican.com/article/rise-of-the-robots/) estimates that the brain performs around 100 million MIPS.[2](https://aiimpacts.org/preliminary-prices-for-human-level-hardware/#easy-footnote-bottom-2-490 "” it would take, in round numbers, 100 million MIPS (100 trillion instructions per second) to emulate the 1,500-gram human brain.” – Moravec, 2009") MIPS are not directly comparable to MFLOPS (millions of FLOPS), and [have deficiencies](http://www.econ.yale.edu/~nordhaus/homepage/prog_083001a.pdf) as a measure, but the empirical relationship in computers is something like MFLOPS = 2.3 x MIPS0.89, according to [Sandberg and Bostrom](http://www.fhi.ox.ac.uk/brain-emulation-roadmap-report.pdf) (2008).[3](https://aiimpacts.org/preliminary-prices-for-human-level-hardware/#easy-footnote-bottom-3-490 "See p89. It actually says FLOPS not MFLOPS, but this appears to be an error, given the graph.") This suggests Moravec’s estimate coincides with around 3.0 x 10¹³ FLOPS. Given that an order of magnitude increase in computing power per dollar [corresponds to](http://aiimpacts.org/trends-in-the-cost-of-computing/) about four years of time, knowing that MFLOPS and MIPS are roughly comparable is plenty of precision.

We [estimated](http://aiimpacts.org/current-flops-prices/ "Current FLOPS prices") FLOPShours cost around $10-13 each. Thus if we want to run a brain, putting these figures together, it would cost us around $3/hour!

That is, if Moravec’s estimate was right, and my conversion of it to FLOPS was basically right, and [our prices](http://aiimpacts.org/current-flops-prices/ "Current FLOPS prices") were right, and hardware mattered a lot more than software, we would already be in the robot revolution.[4](https://aiimpacts.org/preliminary-prices-for-human-level-hardware/#easy-footnote-bottom-4-490 "Humans who cost $100/hour would have become more expensive than the equivalent hardware in about 2009: an hour is currently factor of 30 cheaper than $100, or 1.5 factors of ten, which corresponds to about six years of progress in hardware price performance.") How informative!

Our prices are pretty consistent with the latest extrapolation in Moravec’s [1997 graph](http://www.transhumanist.com/volume1/moravec.htm) (see Figure 1).[5](https://aiimpacts.org/preliminary-prices-for-human-level-hardware/#easy-footnote-bottom-5-490 "In 2015, it appears to be at around 2.3 million MIPS/$1000, which is around 1M MFLOPS/$1000 according to Sandberg and Bostrom’s conversion, which is $1 per GFLOP. Our estimate was $3/GFLOP.") However his graph doesn’t reach human-equivalence until around 2020, and he predicts such computers appear around then.[6](https://aiimpacts.org/preliminary-prices-for-human-level-hardware/#easy-footnote-bottom-6-490 "‘At the present rate, computers suitable for humanlike robots will appear in the 2020s.’ –Moravec") This difference from our reading of his figures is because his graph is of what can be bought with $1,000, whereas we expect someone would build an AI by the time it cost around $1M at the latest.[7](https://aiimpacts.org/preliminary-prices-for-human-level-hardware/#easy-footnote-bottom-7-490 "A computer that lasts for three years and runs a human replacement who can work for around $50/hour should be worth around $1M.") So our threshold of ‘affordable’ is three orders of magnitude more expensive than his, and thus a bit over a decade earlier. This seems to be a disagreement about at what price it becomes economically viable to replace a human, but we do not understand his position[8](https://aiimpacts.org/preliminary-prices-for-human-level-hardware/#easy-footnote-bottom-8-490 "‘By our estimate, today’s very biggest supercomputers are within a factor of a hundred of having the power to mimic a human mind. Their successors a decade hence will be more than powerful enough. Yet, it is unlikely that machines costing tens of millions of dollars will be wasted doing what any human can do, when they could instead be solving urgent physical and mathematical problems nothing else can touch. Machines with human-like performance will make economic sense only when they cost less than humans, say when their “brains” cost about $1,000. When will that day arrive?’ – Moravec") well. A ‘brain’ that costs $1000, and runs for a few years, lets say doing useful work for only 40h/week, is costing $0.16/hour! Yet it can earn at least several hundred times more than that. There are other costs to laboring than having a virtual brain, but not that many.

[![xxx](http://aiimpacts.org/wp-content/uploads/2015/04/moravecevolutionofcpc-copy.png)](http://aiimpacts.org/wp-content/uploads/2015/04/moravecevolutionofcpc-copy.png)**Figure 1**: The growth of cheap MIPS, from [Moravec](http://www.transhumanist.com/volume1/moravec.htm).
Incidentally, by 2009, [Moravec](http://www.scientificamerican.com/article/rise-of-the-robots/) lengthened his prediction to 20-30 years to ‘close the gap’, seemingly due to cost reductions abating since 1990.[9](https://aiimpacts.org/preliminary-prices-for-human-level-hardware/#easy-footnote-bottom-9-490 "“Since 1990 cost and size reductions have abated, but power has risen to about 10,000 MIPS for a home computer. At the present pace, only about 20 or 30 years will be needed to close the gap…It suggests that robot intelligence will surpass our own well before 2050.”- Moravec, 2009") We don’t know of evidence of this slowing however, and as mentioned, it seems recent hardware prices are in line with his earlier estimates.

Moravec’s is not the only estimate of computation done by the brain. [Sandberg and Bostrom](http://www.fhi.ox.ac.uk/brain-emulation-roadmap-report.pdf) (2008) project the processing required to emulate a human brain at different levels of detail.[10](https://aiimpacts.org/preliminary-prices-for-human-level-hardware/#easy-footnote-bottom-10-490 "From Sandberg and Bostrom, table 9: Processing demands (emulation only, human brain)(p80):

spiking neural network: 10¹⁸ FLOPS (Earliest year, $1 million: commodity computer estimate: 2042, supercomputer estimate: 2019)

electrophysiology: 10²² FLOPS (Earliest year, $1 million: commodity computer estimate: 2068, supercomputer estimate: 2033)

metabolome: 10²⁵ FLOPS (Earliest year, $1 million: commodity computer estimate: 2087, supercomputer estimate: 2044)

") For the three levels that their workshop participants considered most plausible, their estimates are 10¹⁸, 10²², and 10²⁵ FLOPS. These would presently cost around $100K/hour, $1bn/hour and $1T/hour.

We [estimated](http://aiimpacts.org/trends-in-the-cost-of-computing/ "Trends in the cost of computing") that FLOPS are coming down in price by roughly an order of magnitude roughly every four years, so if that is about right and continues, these figures suggest that an AI might compete with a $100/hour human in 12 years, 28 years or 40 years respectively: between 2027 and 2055.

Incidentally, Sandberg and Bostrom predicted a $1M project could purchase enough hardware for a brain between 2019 and 2044 or between 2042 and 2087, depending on whether one is extrapolating retail computing or supercomputing price trends. So our extrapolation of their figures based on recent data is in basically line with theirs. Figure 2 shows roughly where our [2015 FLOPS/$](http://aiimpacts.org/current-flops-prices/ "Current FLOPS prices") figures fall on their graph of MFLOPS.

[![](http://aiimpacts.org/wp-content/uploads/2015/04/wberm-flops-copy.jpg)](http://aiimpacts.org/wp-content/uploads/2015/04/wberm-flops-copy.jpg)**Figure 2**: Growth in MFLOPS/$ according to [Sandberg and Bostrom](http://www.fhi.ox.ac.uk/brain-emulation-roadmap-report.pdf) (p88), annotated with our [recent estimate](http://aiimpacts.org/current-flops-prices/ "Current FLOPS prices") of cheap FLOPS in 2015.
Another estimate of human-level hardware comes from Kurzweil. In [The Singularity is Near](http://en.wikipedia.org/wiki/The_Singularity_Is_Near), he claimed that a human brain required 10¹⁶ calculations per second, which appears to be roughly equivalent to 10¹⁶ FLOPS.[11](https://aiimpacts.org/preliminary-prices-for-human-level-hardware/#easy-footnote-bottom-11-490 "‘If we use the figure of 10¹⁶ cps that I believe will be sufficient for functional emulation of human intelligence…’ – Kurzweil, The Singularity is Near, p121") This would put the current price of human-level hardware at $1000/hour, beginning to be competitive with some highly expensive people, and reaching the $100/hour people in about four years.[12](https://aiimpacts.org/preliminary-prices-for-human-level-hardware/#easy-footnote-bottom-12-490 " 10¹⁶ x $10-¹³/hour = $1000/hour")
So it seems human-level hardware presently costs between $3/hour and $1T/hour. And the date when we will reach it lies somewhere in a half-century long haze. We have already stepped into the haze: of the five dates we saw, we have passed one, and are reaching another. This is important, because the era of maybe-human-level hardware has a heightened chance of bringing human-level AI.

It is also important to dispel the haze. If we knew there was a substantially heightened chance of human-level AI in the next ten years, we might act differently from how we would if we knew hardware would be lacking for forty years. Also, if were sure we had uneventfully passed human-level hardware, then we could exclude the possibility that hardware was overwhelmingly important, and start on excluding nearby possibilities (e.g. that hardware is pretty important). Knowing better how much difference hardware makes would change our other expectations about the arrival of human-level AI. Though human-level hardware coming sooner would tend to shorten our AI timelines, if we knew that we had already thoroughly passed it we might expect longer timelines.

This has been a preview of our investigation into these things. The next steps will hopefully resolve some ambiguity around how much computation a brain does, or at least add some more to be resolved later.

\*\*\*

(Featured image: *Pyramidal hippocampal neuron 40x* by [MethoxyRoxy](http://commons.wikimedia.org/wiki/File:Pyramidal_hippocampal_neuron_40x.jpg#/media/File:Pyramidal_hippocampal_neuron_40x.jpg))


Current FLOPS prices

In November 2017, we estimate the price for one GFLOPS to be between $0.03 and $3 for single or double precision performance, using GPUs (therefore excluding some applications). Amortized over three years, this is $1.1 x 10-5 -$1.1 x 10-7 /GFLOPShour.

Background
----------

We have written about [long term trends](http://aiimpacts.org/trends-in-the-cost-of-computing/ "Trends in the cost of computing") and [short term trends](http://aiimpacts.org/recent-trend-in-the-cost-of-computing/) in the costs of computing hardware. We are interested in evaluating the current prices more thoroughly, both to validate the trend data, and because current hardware prices are particularly important to know about.

Details
-------

We separately investigated CPUs, GPUs, computing as a service, and supercomputers. We used somewhat different methods to estimate the price in these categories, based on the data available. We did not find any definitive source on the most cost-effective in any category, or in general, so our examples are probably not the very cheapest. Nevertheless, these figures give a crude sense for the cost of computation in the contemporary market. Our data is [here](https://docs.google.com/spreadsheets/d/1yqX2cENwkOxC26wV_sBOvV0NxHzzfmL6tU7StzrFXRc/edit?usp=sharing).

### Included costs

For CPUs and GPUs, we include only the original recommended retail price of the CPU or GPU, and not other computer components (i.e. we do not even include the cost of CPUs in the price of GPUs). In 2015 we compared prices between [one complete rack server](https://drive.google.com/file/d/1xTA4LoooCuHhCWLhkJZ2ZbOWDxso3a0UILO2JxLn5CweadJVJUziux7CMiumtITXTOjXfttY5zNHzCee/view?usp=sharing) and the set of four [processors](http://www.ebay.com/itm/like/351337480917?lpid=82&chn=ps) inside it, and found the complete server was around 36% more expensive ($30,000 vs. $22,000). We expect this is representative at this scale, but diminishes with scale.

For computing services, we list the cheapest price for renting the instance for a long period, with no additional features. We do not include spot prices.

For supercomputers, we list costs cited where we could find them, which don’t tend to come with elaboration. We expect that they only include upfront costs, and that most of the costs are for hardware.

We have not included the costs of energy or other ongoing expenses in any prices. Non-energy costs are hard to find, and we suspect a relatively small and consistent fraction of costs. In 2015 we estimated energy costs to be around 10% of hardware costs.[1](https://aiimpacts.org/current-flops-prices/#easy-footnote-bottom-1-477 "The Intel Xeon E5-2699 uses 527.8 watts and costs $5,190. The processor can be bought here for $5,190 as of April 1 2015. Its energy consumption is 527.8 watts under load, or 90.9 watts idle. Over three years, with $0.05/kWh this is $694, or 13% of the hardware cost.

Titan also uses 13% of its hardware costs in energy over three years. Titan cost about $4000 dollars per hour amortized over 3 years, and consumes about 10M watts, at a cost of $500 per hour (assuming $0.05 per kWh), which is also 13% of its hardware cost.")

### FLOPS measurements

We are interested in empirical performance figures from benchmark tests, but often the only data we could find was for theoretical maximums. We try to use figures for [LINPACK](http://en.wikipedia.org/wiki/LINPACK_benchmarks) and sometimes for [DGEMM](http://matthewrocklin.com/blog/work/2012/10/29/Matrix-Computations/) benchmarks, depending on which are available. [LINPACK relies heavily on DGEMM](http://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms), suggesting DGEMM is fairly comparable.[2](https://aiimpacts.org/current-flops-prices/#easy-footnote-bottom-2-477 "For instance, this presentation (page ‘Results on a single node’) reports Linpack performance of 95% and 89% of DGEMM performance for their hardware in two tests.")

Prices
------

### Graphics processing units (GPUs) and Xeon Phi machines

We collected performance and price figures from Wikipedia[3](https://aiimpacts.org/current-flops-prices/#easy-footnote-bottom-3-477 "Wikipedia pages: Xeon Phi, List of Nvidia Graphics Processing Units, List of AMD Graphics Processing Units

Other sources are visible in the last column of our dataset (see ‘Wikipedia GeForce, Radeon, Phi simplified’ sheet)"), which are available [here](https://docs.google.com/spreadsheets/d/1yqX2cENwkOxC26wV_sBOvV0NxHzzfmL6tU7StzrFXRc/edit?usp=sharing) (see ‘Wikipedia GeForce, Radeon, Phi simplified’). These are theoretical performance figures, which we understand to generally be between somewhat optimistic and ten times too high. So this data suggests real prices of around $0.03-$0.3/GFLOPS. We collected both single and double precision figures, but the cheapest were similar.

Note that GPUs are typically significantly restricted in the kinds of applications they can run efficiently; this performance is achieved for highly regular computations that can be carried out in parallel throughout a GPU (of the sort that are required for rendering scenes, but which have also proved useful in scientific computing). Xeon Phi units are similar to GPUs, and have broader application,[4](https://aiimpacts.org/current-flops-prices/#easy-footnote-bottom-4-477 "“Since it was originally based on an earlier GPU design by Intel, it shares application areas with GPUs.[citation needed] The main difference between Xeon Phi and a GPGPU like Nvidia Tesla is that Xeon Phi, with an x86-compatible core, can, with less modification, run software that was originally targeted at a standard x86 CPU.” – Wikipedia") but in this dataset were not among the cheapest machines.

### Central processing units (CPUs)

We looked at a small number of popular CPUs on Geekbench from the past five years, and found the cheapest to be around $0.71/GFLOPS.[5](https://aiimpacts.org/current-flops-prices/#easy-footnote-bottom-5-477 "See ‘Geekbench 4 History’ tab") However there appear to be 5x disparities between different versions of Geekbench, so we do not trust these numbers a great deal (these figures are from the version we have seen to give relatively high performance figures, and thus low implied prices).

We did not investigate these numbers in great depth, or search far for cheaper CPUs, because CPUs seem to be expensive relative to GPUs, and this minimal investigation, plus our previous investigation in 2015, support this.

### Computing as service

Another way to purchase FLOPS is via virtual computers.

Amazon [Elastic Cloud Compute](http://en.wikipedia.org/wiki/Amazon_Elastic_Compute_Cloud) (EC2) is a major seller of virtual computing. Based on their [current pricing](https://aws.amazon.com/ec2/pricing/), as of October 5th, 2017, renting a [c4.8xlarge](https://aws.amazon.com/ec2/instance-types/) instance [costs](https://aws.amazon.com/ec2/pricing/) $0.621 per hour (if you purchase it for three years, and pay upfront).

According to [a Geekbench report](http://browser.primatelabs.com/geekbench3/1694602)  from 2015, a c4.8xlarge instance [delivers](http://browser.primatelabs.com/geekbench3/1694602) around 97.5 GFLOPS.[6](https://aiimpacts.org/current-flops-prices/#easy-footnote-bottom-6-477 " Geekbench Browser allows users to measure performance in FLOPS using a variety of tasks. 97.5 is the multi-core DGEMM score a user reported for c4.8xlarge. We use a multi-core score because the cost cited is for purchasing all of the cores. On other tasks, Geekbench reports scores from 46 to 199 GFLOPS.") We do not know if ‘c4.8xlarge’ referred to the same computing hardware in 2015, and we do know that the current version of Geekbench gives substantially different answers to the one in use here. However we estimate that the hardware should be less than twice as good as it was, and Geekbench seems unlikely to underestimate performance by more than an order of magnitude.

This implies that a GFLOPShour costs $6.3 x 10-3 , or optimistically as little as $3.2 x 10-4 . This is much higher than a GPU, at $3.4 x 10-6 for a GFLOPShour, if we suppose the hardware is used over around three years. Amazon is probably not the cheapest provider of cloud computing, however the difference seems to be something like a factor of two,[7](https://aiimpacts.org/current-flops-prices/#easy-footnote-bottom-7-477 "We wrote in 2015: “Other sources of virtual computing seem to be similarly priced. An informal comparison of computing providers suggests that on a set of “real-world java benchmarks” three providers are quite closely comparable, with all between just above Amazon’s price and just under half Amazon’s price for completing the benchmarks, across different instance sizes. This analysis also suggests Amazon is a relatively costly provider…”") which is not enough to make cloud computing competitive with GPUs.

In sum, virtual computing appears to cost two to three orders of magnitude more than GPUs. This high price is presumably partly because there are non-hardware costs which we have not accounted for in the prices of buying hardware, but are naturally included in the cost of renting it. However it is unlikely that these additional costs make up a factor of one hundred to one thousand, so cloud computing does not seem competitive.

### Supercomputing

A top supercomputer can perform a GFLOPS for around $3, in 2017. (See *[Price performance trend in top supercomputers](http://aiimpacts.org/price-performance-trend-in-top-supercomputers/))*

### Tensor processing units (TPUs)

[Tensor processing units](https://en.wikipedia.org/wiki/Tensor_processing_unit) appear to perform a GFLOPS for around $1, in February 2018. However it is unclear how this GFLOPS is measured, which makes it somewhat harder to compare (e.g. whether it is single precision or double precision). Such a high price is also at odds with rumors we have heard that TPUs are an especially cheap source of computing, so possibly TPUs are more efficient for a particular set of applications other than the ones where most of these machines have been measured.

Further considerations
----------------------

In 2015, we estimated GPUs to cost around $3/GFLOPS, i.e. 10-100 times more than we would currently estimate. We do not believe that there has been nearly that much improvement in the past two years, so this discrepancy must be due to error and noise. We remain uncertain about the source of all of the difference, so until we resolve that question, it is plausible that our current GPU estimate errs. If so, the price should still be no higher than $3/GFLOPS (our previous estimate, and our current estimate for supercomputer prices).

Summary
-------

The lowest estimated GFLOPS prices we know of are $0.03-$3/GFLOPS, for GPUs and TPUs.

This is a summary of all of the prices we found:

| Type of computer | Source | Type of performance | Current price ($/GFLOPS) | Comments |
| --- | --- | --- | --- | --- |
| GPUs and Xeon Phi (single precision) | Wikipedia | Theoretical peak | .03-0.3 | $0.03/GFLOPS is given, but is underestimate |
| GPUs and Xeon Phi (double precision) | Wikipedia | Theoretical peak | 0.3-0.8 | Upward sloping; probably not optimized for (in GPUs) |
| Cloud | Amazon EC2 and Geekbench | Empirical | 158 | Expensive so less relevant; shallow investigation |
| Supercomputing | Top500 and misc prices | Empirical | 2.94 | Expensive, so less relevant; shallow investigation |
| CPUs | Geekbench and misc prices | Empirical | 0.71 | Unreliable, 5x disagreements between Geekbench versions |
| TPUs | Google Cloud Platform Blog | Unclear | 0.95 |  |

Notes
-----

Current FLOPS prices