Evidence on good forecasting practices from the Good Judgment Project: an accompanying blog post

Figure 0: The "four main determinants of forecasting accuracy." 1
Experience and data from the Good Judgment Project (GJP) provide important evidence about how to make accurate predictions. For a concise summary of the evidence and what we learn from it, see this page. For a review of Superforecasting, the popular book written on the subject, see this blog.
This post explores the evidence in more detail, drawing from the book, the academic literature, the older Expert Political Judgment book, and an interview with a superforecaster. Readers are welcome to skip around to parts that interest them:
1. The experiment
IARPA ran a forecasting tournament from 2011 to 2015, in which five teams plus a control group gave probabilistic answers to hundreds of questions. The questions were generally about potential geopolitical events more than a month but less than a year in the future, e.g. "Will there be a violent incident in the South China Sea in 2013 that kills at least one person?" The questions were carefully chosen so that a reasonable answer would be somewhere between 10% and 90%.2 The forecasts were scored using the original Brier score—more on that in Section 2.3
The winning team was the GJP, run by Philip Tetlock & Barbara Mellers. They recruited thousands of online volunteers to answer IARPA's questions. These volunteers tended to be males (83%) and US citizens (74%). Their average age was forty. 64% of respondents held a bachelor's degree, and 57% had postgraduate training.4
GJP made their official predictions by aggregating and extremizing the predictions of their volunteers.5 They identified the top 2% of predictors in their pool of volunteers each year, dubbing them "superforecasters," and put them on teams in the next year so they could collaborate on special forums. They also experimented with a prediction market, and they did a RCT to test the effect of a one-hour training module on forecasting ability. The module included content about probabilistic reasoning, using the outside view, avoiding biases, and more. Attempts were made to find out which parts of the training were most helpful—see Section 4.
2. The results & their intuitive meaning
Here are some of the key results:
"In year 1 GJP beat the official control group by 60%. In year 2, we beat the control group by 78%. GJP also beat its university-affiliated competitors, including the University of Michigan and MIT, by hefty margins, from 30% to 70%."6
"The Good Judgment Project outperformed a prediction market inside the intelligence community, which was populated with professional analysts who had classified information, by 25 or 30 percent, which was about the margin by which the superforecasters were outperforming our own prediction market in the external world."7
"Teams of ordinary forecasters beat the wisdom of the crowd by about 10%. Prediction markets beat ordinary teams by about 20%. And [teams of superforecasters] beat prediction markets by 15% to 30%."8 "On average, teams were 23% more accurate than individuals."9
What does Tetlock mean when he says that one group did X% better than another? By examining Table 4 (in Section 4)  it seems that he means X% lower Brier score. What is the Brier score? For more details, see the Wikipedia article; basically, it measures the average squared distance from the truth. This is why it's better to have a lower Brier score—it means you were on average closer to the truth.10
Here is a bar graph of all the forecasters in Year 2, sorted by Brier score:11

For this set of questions, guessing randomly (assigning even odds to all possibilities) would yield a Brier score of 0.53. So most forecasters did significantly better than that. Some people—the people on the far left of this chart, the superforecasters—did much better than the average. For example, in year 2, the superforecaster Doug Lorch did best with 0.14. This was more than 60% better than the control group.12 Importantly, being a superforecaster in one year correlated strongly with being a superforecaster the next year; there was some regression to the mean but roughly 70% of the superforecasters maintained their status from one year to the next.13
OK, but what does all this mean, in intuitive terms? Here are three ways to get a sense of how good these scores really are:
Way One: Let's calculate some examples of prediction patterns that would give you Brier scores like those mentioned above. Suppose you make a bunch of predictions with 80% confidence and you are correct 80% of the time. Then your Brier score would be 0.32, roughly middle of the pack in this tournament. If instead it was 93% confidence correct 93% of the time, your Brier score would be 0.132, very close to the best superforecasters and to GJP's aggregated forecasts.14 In these examples, you are perfectly calibrated, which helps your score—more realistically you would be imperfectly calibrated and thus would need to be right even more often to get those scores.
Way Two: "An alternative measure of forecast accuracy is the proportion of days on which forecasters' estimates were on the correct side of 50%. … For all questions in the sample, a chance score was 47%. The mean proportion of days with correct estimates was 75%…"15 According to this chart, the superforecasters were on the right side of 50% almost all the time:16

Way Three: "Across all four years of the tournament, superforecasters looking out three hundred days were more accurate than regular forecasters looking out one hundred days."17 (Bear in mind, this wouldn't necessarily hold for a different genre of questions. For example, information about the weather decays in days, while information about the climate lasts for decades or more.)
3. Correlates of good judgment
The data from this tournament is useful in two ways: It helps us decide whose predictions to trust, and it helps us make better predictions ourselves. This section will focus on which kinds of people and practices best correlate with success—information which is relevant to both goals. Section 4 will cover the training experiment, which helps to address causation vs. correlation worries.
Feast your eyes on this:18

This shows the correlations between various things.19 The leftmost column is the most important; it shows how each variable correlates with (standardized) Brier score. (Recall that Brier scores measure inaccuracy, so negative correlations are good.)
It's worth mentioning that while intelligence correlated with accuracy, it didn't steal the show.20 The same goes for time spent deliberating.21 The authors summarize the results as follows: "The best forecasters scored higher on both intelligence and political knowledge than the already well-above-average group of forecasters. The best forecasters had more open-minded cognitive styles. They benefited from better working environments with probability training and collaborative teams. And while making predictions, they spent more time deliberating and updating their forecasts."22
That big chart depicts all the correlations individually. Can we use them to construct a model to take in all of these variables and spit out a prediction for what your Brier score will be? Yes we can:
Figure 3. Structural equation model with standardized coefficients.
This model has a multiple correlation of 0.64.23 Earlier, we noted that superforecasters typically remained superforecasters (i.e. in the top 2%), proving that their success wasn't mostly due to luck. Across all the forecasters, the correlation between performance in one year and performance in the next year is 0.65.24 So we have two good ways to predict how accurate someone will be: Look at their past performance, and look at how well they score on the structural model above.
I speculate that these correlations underestimate the true predictability of accuracy, because the forecasters were all unpaid online volunteers, and many of them presumably had random things come up in their life that got in the way of making good predictions—perhaps they have a kid, or get sick, or move to a new job and so stop reading the news for a month, and their accuracy declines.25 Yet still 70% of the superforecasters in one year remained superforecasters in the next.
Finally, what about superforecasters in particular? Is there anything to say about what it takes to be in the top 2%?

Tetlock devotes much of his book to this. It is hard to tell how much his recommendations come from data analysis and how much are just his own synthesis of the interviews he's conducted with superforecasters. Here is his "Portrait of the modal superforecaster."26
Philosophic outlook:

Cautious: Nothing is certain.
Humble: Reality is infinitely complex.
Nondeterministic: Whatever happens is not meant to be and does not have to happen.

Abilities & thinking styles:

Actively open-minded: Beliefs are hypotheses to be tested, not treasures to be protected.
Intelligent and knowledgeable, with a "Need for Cognition": Intellectually curious, enjoy puzzles and mental challenges.
Reflective: Introspective and self-critical.
Numerate: Comfortable with numbers.

Methods of forecasting:

Pragmatic: Not wedded to any idea or agenda.
Analytical: Capable of stepping back from the tip-of-your-nose perspective and considering other views.
Dragonfly-eyed: Value diverse views and synthesize them into their own.
Probabilistic: Judge using many grades of maybe.
Thoughtful updaters: When facts change, they change their minds.
Good intuitive psychologists: Aware of the value of checking thinking for cognitive and emotional biases.

Work ethic:

Growth mindset: Believe it's possible to get better.
Grit: Determined to keep at it however long it takes.

Additionally, there is experimental evidence that superforecasters are less prone to standard cognitive science biases than ordinary people.27 This is particularly exciting because—we can hope—the same sorts of training that help people become superforecasters might also help overcome biases.
Finally, Tetlock says that "The strongest predictor of rising into the ranks of superforecasters is perpetual beta, the degree to which one is committed to belief updating and self-improvement. It is roughly three times as powerful a predictor as its closest rival, intelligence."28 Unfortunately, I couldn't find any sources or data on this, nor an operational definition of "perpetual beta," so we don't know how he measured it.29
4. The training and Tetlock's commandments
This section discusses the surprising effect of the training module on accuracy, and finishes with Tetlock's training-module-based recommendations for how to become a better forecaster.30
The training module, which was randomly given to some participants but not others, took about an hour to read.31 The authors describe the content as follows:
"Training in year 1 consisted of two different modules: probabilistic reasoning training and scenario training. Scenario-training was a four-step process: 1) developing coherent and logical probabilities under the probability sum rule; 2) exploring and challenging assumptions; 3) identifying the key causal drivers; 4) considering the best and worst case scenarios and developing a sensible 95% confidence interval of possible outcomes; and 5) avoid over-correction biases. … Probabilistic reasoning training consisted of lessons that detailed the difference between calibration and resolution, using comparison classes and base rates (Kahneman & Tversky, 1973; Tversky & Kahneman, 1981), averaging and using crowd wisdom principles (Surowiecki, 2005), finding and utilizing predictive mathematical and statistical models (Arkes, 1981; Kahneman & Tversky, 1982), cautiously using time-series and historical data, and being self-aware of the typical cognitive biases common throughout the population."32
In later years, they merged the two modules into one and updated it based on their observations of the best forecasters. The updated training module is organized around an acronym:33

Impressively, this training had a lasting positive effect on accuracy in all four years:

One might worry that training improves accuracy by motivating the trainees to take their jobs more seriously. Indeed it seems that the trained forecasters made more predictions per question than the control group, though they didn't make more predictions overall. Nevertheless it seems that the training also had a direct effect on accuracy as well as this indirect effect.34
Moving on, let's talk about the advice Tetlock gives to his audience in Superforecasting, advice which is based on, though not identical to, the CHAMPS-KNOW training. The book has a few paragraphs of explanation for each commandment, a transcript of which is here; in this post I'll give my own abbreviated explanations:
TEN COMMANDMENTS FOR ASPIRING SUPERFORECASTERS
(1) Triage: Don't waste time on questions that are "clocklike" where a rule of thumb can get you pretty close to the correct answer, or "cloudlike" where even fancy models can't beat a dart-throwing chimp.
(2) Break seemingly intractable problems into tractable sub-problems: This is how Fermi estimation works. One related piece of advice is "be wary of accidentally substituting an easy question for a hard one," e.g. substituting "Would Israel be willing to assassinate Yasser Arafat?" for "Will at least one of the tests for polonium in Arafat's body turn up positive?"
(3) Strike the right balance between inside and outside views: In particular, first anchor with the outside view and then adjust using the inside view. (More on this in Section 5)
(4) Strike the right balance between under- and overreacting to evidence: "Superforecasters aren't perfect Bayesian predictors but they are much better than most of us."35 Usually do many small updates, but occasionally do big updates when the situation calls for it. Take care not to fall for things that seem like good evidence but aren't; remember to think about P(E|H)/P(E|~H); remember to avoid the base-rate fallacy.
(5) Look for the clashing causal forces at work in each problem: This is the "dragonfly eye perspective," which is where you attempt to do a sort of mental wisdom of the crowds: Have tons of different causal models and aggregate their judgments. Use "Devil's advocate" reasoning. If you think that P, try hard to convince yourself that not-P. You should find yourself saying "On the one hand… on the other hand… on the third hand…" a lot.
(6) Strive to distinguish as many degrees of doubt as the problem permits but no more: Some people criticize the use of exact probabilities (67%! 21%!) as merely a way to pretend you know more than you do. There might be another post on the subject of why credences are better than hedge words like "maybe" and "probably" and "significant chance;" for now, I'll simply mention that when the authors rounded the superforecaster's forecasts to the nearest 0.05, their accuracy dropped.36 Superforecasters really were making use of all 101 numbers from 0.00 to 1.00! (EDIT: I am told this may be wrong; the number should be 0.1, not 0.05. See the discussion here and here.)
(7) Strike the right balance between under- and overconfidence, between prudence and decisiveness.
(8) Look for the errors behind your mistakes but beware of rearview-mirror hindsight biases.
(9) Bring out the best in others and let others bring out the best in you: The book spent a whole chapter on this, using the Wehrmacht as an extended case study on good team organization.37 One pervasive guiding principle is "Don't tell people how to do things; tell them what you want accomplished, and they'll surprise you with their ingenuity in doing it." The other pervasive guiding principle is "Cultivate a culture in which people—even subordinates—are encouraged to dissent and give counterarguments."38
(10) Master the error-balancing bicycle: This one should have been called practice, practice, practice. Tetlock says that reading the news and generating probabilities isn't enough; you need to actually score your predictions so that you know how wrong you were.
(11) Don't treat commandments as commandments: Tetlock's point here is simply that you should use your judgment about whether to follow a commandment or not; sometimes they should be overridden.
It's worth mentioning at this point that the advice is given at the end of the book, as a sort of summary, and may make less sense to someone who hasn't read the book. In particular, Chapter 5 gives a less formal but more helpful recipe for making predictions, with accompanying examples. See the end of this blog post for a summary of this recipe.
5. On the Outside View & Lessons for AI Impacts
The previous section summarized Tetlock's advice for how to make better forecasts; my own summary of the lessons I think we should learn is more concise and comprehensive and can be found at this page. This section goes into detail about one particular, more controversial matter: The importance of the "outside view," also known as reference class forecasting. This research provides us with strong evidence in favor of this method of making predictions; however, the situation is complicated by Tetlock's insistence that other methods are useful as well. This section discusses the evidence and attempts to interpret it.
The GJP asked people who took the training to self-report which of the CHAMPS-KNOW principles they were using when they explained why they made a forecast; 69% of forecast explanations received tags this way. The only principle significantly positively correlated with successful forecasts was C: Comparison classes.39 The authors take this as evidence that the outside view is particularly important. Anecdotally, the superforecaster I interviewed agreed that reference class forecasting was perhaps the most important piece of the training. (He also credited the training in general with helping him reach the ranks of the superforecasters.)
Moreover, Tetlock did an earlier, much smaller forecasting tournament from <PHONE>, in which experts of various kinds made the forecasts.40 The results were astounding: Many of the experts did worse than random chance, and all of them did worse than simple algorithms:

Figure 3.2, pulled from Expert Political Judgment, is a gorgeous depiction of some of the main results.41 Tetlock used something very much like a Brier score in this tournament, but he broke it into two components: "Discrimination" and "Calibration." This graph plots the various experts and algorithms on the axes of discrimination and calibration. Notice in the top right corner the "Formal models" box. I don't know much about the model used but apparently it was significantly better than all of the humans. This, combined with the fact that simple case-specific trend extrapolations also beat all the humans, is strong evidence for the importance of the outside view.
So we should always use the outside view, right? Well, it's a bit more complicated than that. Tetlock's advice is to start with the outside view, and then adjust using the inside view.42 He even goes so far as to say that hedgehoggery and storytelling can be valuable when used properly.
First, what is hedgehoggery? Recall how the human experts fall on a rough spectrum in Figure 3.2, with "hedgehogs" getting the lowest scores and "foxes" getting the highest scores. What makes someone a hedgehog or a fox? Their answers to these questions.43 Tetlock characterizes the distinction as follows:
Low scorers look like hedgehogs: thinkers who "know one big thing," aggressively extend the explanatory reach of that one big thing into new domains, display bristly impatience with those who "do not get it," and express considerable confidence that they are already pretty proficient forecasters, at least in the long term. High scorers look like foxes: thinkers who know many small things (tricks of their trade), are skeptical of grand schemes, see explanation and prediction not as deductive exercises but rather as exercises in flexible "ad hocery" that require stitching together diverse sources of information, and are rather diffident about their own forecasting prowess, and … rather dubious that the cloudlike subject of politics can be the object of a clocklike science.44
Next, what is storytelling? Using your domain knowledge, you think through a detailed scenario of how the future might go, and you tweak it to make it more plausible, and then you assign a credence based on how plausible it seems. By itself this method is unpromising.45
Despite this, Tetlock thinks that storytelling and hedgehoggery are valuable if handled correctly. On hedgehogs, Tetlock says that hedgehogs provide a valuable service by doing the deep thinking necessary to build detailed causal models and raise interesting questions; these models and questions can then be slurped up by foxy superforecasters, evaluated, and aggregated to make good predictions.46 The superforecaster Bill Flack is quoted in agreement.47 As for storytelling, see these slides from Tetlock's edge.org seminar:

As the second slide indicates, the idea is that we can sometimes "fight fire with fire" by using some stories to counter other stories. In particular, Tetlock says there has been success using stories about the past—about ways that the world could have gone, but didn't—to "reconnect us to our past states of ignorance."48 The superforecaster I interviewed said that it is common practice now on superforecaster forums to have a designated "red team" with the explicit mission of finding counter-arguments to whatever the consensus seems to be. This, I take it, is an example of motivated reasoning being put to good use.
Moreover, arguably the outside view simply isn't useful for some questions.49 People say this about lots of things—e.g. "The world is changing so fast, so the current situation in Syria is unprecedented and historical averages will be useless!"—and are proven wrong; for example, this research seems to indicate that the outside view is far more useful in geopolitics than people think. Nevertheless, maybe it is true for some of the things we wish to predict about advanced AI. After all, a major limitation of this data is that the questions were mainly on geopolitical events only a few years in the future at most. (Geopolitical events seem to be somewhat predictable up to two years out but much more difficult to predict five, ten, twenty years out.)50 So this research does not directly tell us anything about the predictability of the events AI Impacts is interested in, nor about the usefulness of reference-class forecasting for those domains.51
That said, the forecasting best practices discovered by this research seem like general truth-finding skills rather than cheap hacks only useful in geopolitics or only useful for near-term predictions. After all, geopolitical questions are themselves a fairly diverse bunch, yet accuracy on some was highly correlated with accuracy on others.52 So despite these limitations I think we should do our best to imitate these best-practices, and that means using the outside view far more than we would naturally be inclined.
One final thing worth saying is that, remember, the GJP's aggregated judgments did at least as well as the best superforecasters.53 Presumably at least one of the forecasters in the tournament was using the outside view a lot; after all, half of them were trained in reference-class forecasting.  So I think we can conclude that straightforwardly using the outside view as often as possible wouldn't get you better scores than the GJP, though it might get you close for all we know. Anecdotally, it seems that when the superforecasters use the outside view they often aggregate between different reference-class forecasts.54 The wisdom of the crowds is powerful; this is consistent with the wider literature on the cognitive superiority of groups, and the literature on ensemble methods in AI.55
Tetlock describes how superforecasters go about making their predictions.56 Here is an attempt at a summary:

Sometimes a question can be answered more rigorously if it is first "Fermi-ized," i.e. broken down into sub-questions for which more rigorous methods can be applied.
Next, use the outside view on the sub-questions (and/or the main question, if possible). You may then adjust your estimates using other considerations ('the inside view'), but do this cautiously.
Seek out other perspectives, both on the sub-questions and on how to Fermi-ize the main question. You can also generate other perspectives yourself.
Repeat steps 1 – 3 until you hit diminishing returns.
Your final prediction should be based on an aggregation of various models, reference classes, other experts, etc.

Footnotes


Evidence on good forecasting practices from the Good Judgment Project

According to experience and data from the Good Judgment Project, the following are associated with successful forecasting, in rough decreasing order of combined importance and confidence:

Past performance in the same broad domain
Making more predictions on the same question
Deliberation time
Collaboration on teams
Intelligence
Domain expertise
Having taken a one-hour training module on these topics
'Cognitive reflection' test scores
'Active open-mindedness'
Aggregation of individual judgments
Use of precise probabilistic predictions
Use of 'the outside view'
'Fermi-izing'
'Bayesian reasoning'
Practice

Details
1. 1. Process
The Good Judgment Project (GJP) was the winning team in IARPA's 2011-2015 forecasting tournament. In the tournament, six teams assigned probabilistic answers to hundreds of questions about geopolitical events months to a year in the future. Each competing team used a different method for coming up with their guesses, so the tournament helps us to evaluate different forecasting methods.
The GJP team, led by Philip Tetlock and Barbara Mellers, gathered thousands of online volunteers and had them answer the tournament questions. They then made their official forecasts by aggregating these answers. In the process, the team collected data about the patterns of performance in their volunteers, and experimented with aggregation methods and improvement interventions. For example, they ran an RCT to test the effect of a short training program on forecasting accuracy. They especially focused on identifying and making use of the most successful two percent of forecasters, dubbed 'superforecasters'.
Tetlock's book Superforecasting describes this process and Tetlock's resulting understanding of how to forecast well.

1.2. Correlates of successful forecasting
1.2.1. Past performance
Roughly 70% of the superforecasters maintained their status from one year to the next 1. Across all the forecasters, the correlation between performance in one year and performance in the next year was 0.65 2. These high correlations are particularly impressive because the forecasters were online volunteers; presumably substantial variance year-to-year came from forecasters throttling down their engagement due to fatigue or changing life circumstances 3.

1.2.2. Behavioral and dispositional variables
Table 2  depicts the correlations between measured variables amongst GJP's volunteers in the first two years of the tournament 4.  Each is described in more detail below.

The first column shows the relationship between each variable and standardized Brier score, which is a measure of inaccuracy: higher Brier scores mean less accuracy, so negative correlations are good. "Ravens" is an IQ test; "Del time" is deliberation time, and "teams" is whether or not the forecaster was assigned to a team. "Actively open-minded thinking" is an attempt to measure "the tendency to evaluate arguments and evidence without undue bias from one's own prior beliefs—and with recognition of the fallibility of one's judgment." 5
The authors conducted various statistical analyses to explore the relationships between these variables. They computed a structural equation model to predict a forecaster's accuracy:

Yellow ovals are latent dispositional variables, yellow rectangles are observed dispositional variables, pink rectangles are experimentally manipulated situational variables, and green rectangles are observed behavioral variables. This model has a multiple correlation of 0.64.6
As these data indicate, domain knowledge, intelligence, active open-mindedness, and working in teams each contribute substantially to accuracy. We can also conclude that effort helps, because deliberation time and number of predictions made per question ("belief updating") both improved accuracy. Finally, training also helps. This is especially surprising because the training module lasted only an hour and its effects persisted for at least a year. The module included content about probabilistic reasoning, using the outside view, avoiding biases, and more.

1.3. Aggregation algorithms
GJP made their official predictions by aggregating and extremizing the predictions of their volunteers. The aggregation algorithm was elitist, meaning that it weighted more heavily people who were better on various metrics. 7 The extremizing step pushes the aggregated judgment closer to 1 or 0, to make it more confident. The degree to which they extremize depends on how diverse and sophisticated the pool of forecasters is. 8 Whether extremizing is a good idea is still controversial.  9
GJP beat all of the other teams. They consistently beat the control group—which was a forecast made by averaging ordinary forecasters—by more than 60%. 10 They  also beat a prediction market inside the intelligence community—populated by professional analysts with access to classified information—by 25-30%. 11
That said, individual superforecasters did almost as well, so the elitism of the algorithm may account for a lot of its success.12

1.4. Outside View
The forecasters who received training were asked to record, for each prediction, which parts of the training they used to make it. Some parts of the training—e.g. "Post-mortem analysis"—were correlated with inaccuracy, but others—most notably "Comparison classes"—were correlated with accuracy. 13  'Comparison classes' is another term for reference-class forecasting, also known as 'the outside view'. It is the method of assigning a probability by straightforward extrapolation from similar past situations and their outcomes.

1.5. Tetlock's "Portrait of the modal superforecaster"
This subsection and those that follow will lay out some more qualitative results, things that Tetlock recommends on the basis of his research and interviews with superforecasters. Here is Tetlock's "portrait of the modal superforecaster:" 14
Philosophic outlook:

Cautious: Nothing is certain.
Humble: Reality is infinitely complex.
Nondeterministic: Whatever happens is not meant to be and does not have to happen.

Abilities & thinking styles:

Actively open-minded: Beliefs are hypotheses to be tested, not treasures to be protected.
Intelligent and knowledgeable, with a "Need for Cognition": Intellectually curious, enjoy puzzles and mental challenges.
Reflective: Introspective and self-critical
Numerate: Comfortable with numbers

Methods of forecasting:

Pragmatic: Not wedded to any idea or agenda
Analytical: Capable of stepping back from the tip-of-your-nose perspective and considering other views
Dragonfly-eyed: Value diverse views and synthesize them into their own
Probabilistic: Judge using many grades of maybe
Thoughtful updaters: When facts change, they change their minds
Good intuitive psychologists: Aware of the value of checking thinking for cognitive and emotional biases 15

Work ethic:

Growth mindset: Believe it's possible to get better
Grit: Determined to keep at it however long it takes

1.6. Tetlock's "Ten Commandments for Aspiring Superforecasters:"
This advice is given at the end of the book, and may make less sense to someone who hasn't read the book. A full transcript of these commandments can be found here; this is a summary:
(1) Triage: Don't waste time on questions that are "clocklike" where a rule of thumb can get you pretty close to the correct answer, or "cloudlike" where even fancy models can't beat a dart-throwing chimp.
(2) Break seemingly intractable problems into tractable sub-problems: This is how Fermi estimation works. One related piece of advice is "be wary of accidentally substituting an easy question for a hard one," e.g. substituting "Would Israel be willing to assassinate Yasser Arafat?" for "Will at least one of the tests for polonium in Arafat's body turn up positive?"
(3) Strike the right balance between inside and outside views: In particular, first anchor with the outside view and then adjust using the inside view.
(4) Strike the right balance between under- and overreacting to evidence: Usually do many small updates, but occasionally do big updates when the situation calls for it. Remember to think about P(E|H)/P(E|~H); remember to avoid the base-rate fallacy. "Superforecasters aren't perfect Bayesian predictors but they are much better than most of us." 16
(5) Look for the clashing causal forces at work in each problem: This is the "dragonfly eye perspective," which is where you attempt to do a sort of mental wisdom of the crowds: Have tons of different causal models and aggregate their judgments. Use "Devil's advocate" reasoning. If you think that P, try hard to convince yourself that not-P. You should find yourself saying "On the one hand… on the other hand… on the third hand…" a lot.
(6) Strive to distinguish as many degrees of doubt as the problem permits but no more.
(7) Strike the right balance between under- and overconfidence, between prudence and decisiveness.
(8) Look for the errors behind your mistakes but beware of rearview-mirror hindsight biases.
(9) Bring out the best in others and let others bring out the best in you. The book spent a whole chapter on this, using the Wehrmacht as an extended case study on good team organization. One pervasive guiding principle is "Don't tell people how to do things; tell them what you want accomplished, and they'll surprise you with their ingenuity in doing it." The other pervasive guiding principle is "Cultivate a culture in which people—even subordinates—are encouraged to dissent and give counterarguments." 17
(10) Master the error-balancing bicycle: This one should have been called practice, practice, practice. Tetlock says that reading the news and generating probabilities isn't enough; you need to actually score your predictions so that you know how wrong you were.
(11) Don't treat commandments as commandments: Tetlock's point here is simply that you should use your judgment about whether to follow a commandment or not; sometimes they should be overridden.

1.7. Recipe for Making Predictions
Tetlock describes how superforecasters go about making their predictions. 18 Here is an attempt at a summary:

Sometimes a question can be answered more rigorously if it is first "Fermi-ized," i.e. broken down into sub-questions for which more rigorous methods can be applied.
Next, use the outside view on the sub-questions (and/or the main question, if possible). You may then adjust your estimates using other considerations ('the inside view'), but do this cautiously.
Seek out other perspectives, both on the sub-questions and on how to Fermi-ize the main question. You can also generate other perspectives yourself.
Repeat steps 1 – 3 until you hit diminishing returns.
Your final prediction should be based on an aggregation of various models, reference classes, other experts, etc.

1.8. Bayesian reasoning & precise probabilistic forecasts
Humans normally express uncertainty with terms like "maybe" and "almost certainly" and "a significant chance." Tetlock advocates for thinking and speaking in probabilities instead. He recounts many anecdotes of misunderstandings that might have been avoided this way. For example:
In 1961, when the CIA was planning to topple the Castro government by landing a small army of Cuban expatriates at the Bay of Pigs, President John F. Kennedy turned to the military for an unbiased assessment. The Joint Chiefs of Staff concluded that the plan had a "fair chance" of success. The man who wrote the words "fair chance" later said he had in mind odds of 3 to 1 against success. But Kennedy was never told precisely what "fair chance" meant and, not unreasonably, he took it to be a much more positive assessment. 19
This example hints at another advantage of probabilistic judgments: It's harder to weasel out of them afterwards, and therefore easier to keep score. Keeping score is crucial for getting feedback from reality, which is crucial for building up expertise.
A standard criticism of using probabilities is that they merely conceal uncertainty rather than quantify it—after all, the numbers you pick are themselves guesses. This may be true for people who haven't practiced much, but it isn't true for superforecasters, who are impressively well-calibrated and whose accuracy scores decrease when you round their predictions to the nearest 0.05. (EDIT: This should be 0.1)20
Bayesian reasoning is a natural next step once you are thinking and talking probabilities—it is the theoretical ideal in several important ways 21 —and Tetlock's experience and interviews with superforecasters seems to bear this out. Superforecasters seem to do many small updates, with occasional big updates, just as Bayesianism would predict. They recommend thinking in the Bayesian way, and often explicitly make Bayesian calculations. They are good at breaking down difficult questions into more manageable parts and chaining the probabilities together properly.

2. Discussion: Relevance to AI Forecasting
2.1. Limitations
A major limitation is that the forecasts were mainly on geopolitical events only a few years in the future at most. (Uncertain geopolitical events seem to be somewhat predictable up to two years out but much more difficult to predict five years out.) 22 So evidence from the GJP may not generalize to forecasting other types of events (e.g. technological progress and social  consequences) or events further in the future.
That said, the forecasting best practices discovered by this research are not overtly specific to geopolitics or near-term events.  Also, geopolitical questions are diverse and accuracy on some was highly correlated with accuracy on others. 23
Tetlock has ideas for how to handle longer-term, nebulous questions. He calls it "Bayesian Question Clustering." (Superforecasting 263) The idea is to take the question you really want to answer and look for more precise questions that are evidentially relevant to the question you care about. Tetlock intends to test the effectiveness of this idea in future research.

2.2 Value
The benefits of following these best practices (including identifying and aggregating the best forecasters) appear to be substantial: Superforecasters predicting events 300 days in the future were more accurate than regular forecasters predicting events 100 days in the future, and the GJP did even better. 24 If these benefits generalize beyond the short-term and beyond geopolitics—e.g. to long-term technological and societal development—then this research is highly useful to almost everyone. Even if the benefits do not generalize beyond the near-term, these best practices may still be well worth adopting. For example, it would be extremely useful to have 300 days of warning before strategically important AI milestones are reached, rather than 100.

3. Contributions
Research, analysis, and writing were done by Daniel Kokotajlo. Katja Grace and Justis Mills contributed feedback and editing. Tegan McCaslin, Carl Shulman, and Jacob Lagerros contributed feedback.

4. Footnotes


Reinterpreting "AI and Compute"

This is a guest post by Ben Garfinkel. We revised it slightly, at his request, on February 9, 2019.
A recent OpenAI blog post, "AI and Compute," showed that the amount of computing power consumed by the most computationally intensive machine learning projects has been doubling every three months. The post presents this trend as a reason to better prepare for "systems far outside today's capabilities." Greg Brockman, the CTO of OpenAI, has also used the trend to argue for the plausibility of "near-term AGI." Overall, it seems pretty common to interpret the OpenAI data as evidence that we should expect extremely capable systems sooner than we otherwise would.
However, I think it's important to note that the data can also easily be interpreted in the opposite direction. A more pessimistic interpretation goes like this:

If we were previously underestimating the rate at which computing power was increasing, this means we were overestimating the returns on it.
In addition, if we were previously underestimating the rate at which computing power was increasing, this means that we were overestimating how sustainable its growth is.1
Let's suppose, as the original post does, that increasing computing power is currently one of the main drivers of progress in creating more capable systems. Then — barring any major changes to the status quo — it seems like we should expect progress to slow down pretty soon and we should expect to be underwhelmed by how far along we are when the slowdown hits.

I actually think of this more pessimistic interpretation as something like the default one. There are many other scientific fields where R&D spending and other inputs are increasing rapidly, and, so far as I'm aware, these trends are nearly always interpreted as reasons for pessimism and concern about future research progress.2 If we are going to treat the field of artificial intelligence differently, then we should want clearly articulated and compelling reasons for doing so.

These reasons certainly might exist.3 Still, whatever the case may be, I think we should not be too quick to interpret the OpenAI data as evidence for dramatically more capable systems coming soon.4
Thank you to Danny Hernandez and Ryan Carey for comments on a draft of this post.

Reinterpreting “AI and Compute”


Time for AI to cross the human performance range in diabetic retinopathy

In diabetic retinopathy, automated systems started out just below expert human level performance, and took around ten years to reach expert human level performance.
Details
Diabetic retinopathy is a complication of diabetes in which the back of the eye is damaged by high blood sugar levels.1 It is the most common cause of blindness among working-age adults.2 The disease is diagnosed by examining images of the back of the eye. The gold standard used for diabetic retinopathy diagnosis is typically some sort of pooling mechanism over several expert opinions. Thus, in the papers below, each time expert sensitivity/specificity (Se/Sp) is considered, it is the Se/Sp of individual experts graded against aggregate expert agreement.
As a rough benchmark for expert-level performance we'll take the average Se/Sp of ophthalmologists from a few studies. Based on Google Brain's work (detailed below), this paper 3, and this paper 4 , the average specificity of 14 opthamologists, which indicates expert human-level performance, is 95% and the average sensitivity is 82%.
As far as we can tell, 1996 is when the first algorithm automatically detecting diabetic retinopathy was developed. When compared to opthamologists' ratings, the algorithm achieved 88.4% sensitivity and 83.5% specificity.
In late 2016 Google algorithms were on par with eight opthamologist diagnoses of diabetic retinopathy. See Figure 1.5 The high-sensitivity operating point (labelled on the graph) achieved 97.5/93.4 Se/Sp.
Figure 1: Performance comparison of a late 2016 Google algorithm, and eight opthalmologists, from here. The black curve represents the algorithm and the eight colored dots are opthamologists.
Many other papers were published in between 1996 and 2016. However, none of them achieved better than expert human-level performance on both specificity and sensitivity. For instance 86/77 Se/Sp was achieved in 2007, 97/59 in 2013, and 94/72 by another team in 2016. 6
Thus it took about ten years to go from just below expert human level performance to slightly superhuman performance.
Contributions
Aysja Johnson researched and wrote this page. Justis Mills and Katja Grace contributed feedback.
Footnotes


AGI-11 survey

The AGI-11 survey was a survey of 60 participants at the AGI-11 conference. In it:

Nearly half of respondents believed that AGI would appear before 2030.
Nearly 90% of respondents believed that AGI would appear before 2100.
About 85% of respondents believed that AGI would be beneficial for humankind.

Details
James Barrat and Ben Goertzel surveyed participants at the AGI-11 conference on AGI timelines. The survey had two questions, administered over email after the conference. The results were fairly similar to those from the more complex AGI-09 survey, for which Ben Goertzel was also an author.
Sixty people total responded to the survey, out of over 200 conference registrations. Nobody skipped either of the two questions.
The data in this post, and the results tables, are taken from the write up on this survey in h+ magazine.
Question One
Question one was: "I believe that AGI (however I define it) will be effectively implemented in the following timeframe:"
The answer choices were:

Before 2030
2030-2049
2050-2099
After 2100
Never

The results were:

Question Two
Question two was: "I believe that AGI (however I define it) will be a net positive event for humankind:"
The answer choices were:

True
False

The results were:

Comments
As well as the two survey questions, there was also a form where respondents could submit comments. These are recorded in the h+ magazine write up. Many of the comments expressed concern with the survey structure, suggesting that there could have been more or different options.
Contributions
Justis Mills researched for and wrote this post. Katja Grace provided feedback on style and content. Thanks to James Barrat for linking us to the h+ magazine write up when we reached out to him about his survey.

AGI-11 survey


On the (in)applicability of corporate rights cases to digital minds

This is a guest cross-post by Cullen O'Keefe.
High-Level Takeaway
The extension of rights to corporations likely does not provide useful analogy to potential extension of rights to digital minds.
Introduction
Examining how law can protect the welfare of possible future digital minds is part of my research agenda. I expect that study of historical efforts to secure legal protections ("rights") for previously unprotected classes (e.g., formerly enslaved persons, nonhuman animals, young children) will be crucial to this line of research.
I recently read We the Corporations: How American Businesses Won Their Civil Rights by UCLA constitutional law professor Adam Winkler. The book chronicles how business corporations gradually won various constitutional and statutory civil rights, culminating in the (in)famous recent Citizens United and Hobby Lobby cases.
A key insight from Winkler's book is that, contrary to some popular portrayals of corporate rights cases, these cases usually do not rely primarily on corporate personhood: "While the Supreme Court has on occasion said that corporations are people, the justices have more often relied upon a very different conception of the corporation, one that views it as an association capable of asserting the rights of its members." Id. at xx. The Court, in other words, "pierced the corporate veil" to give corporations rights properly belonging to its members. See id. at 54–55.
The Supreme Court's opinion in Citizens United is illustrative. In determining that the First Amendment's free speech protections applied to corporations, the Court wrote: "[Under the challenged campaign finance statute,] certain disfavored associations of citizens—those that have taken on the corporate form—are penalized for engaging in [otherwise-protected] political speech." 558 U.S. at 356. The Court held that this was impermissible: the shareholders' right to free speech imbued the corporation—which it viewed as merely an association of rights-bearing shareholders—with those same rights. See id. at 365.
Earlier cases that, Winkler argues, exhibit this same pattern include:

Bank of U.S. v. Deveaux, holding that federal jurisdiction over corporations depends on jurisdiction over the individuals comprising the corporation;
Trustees of Dartmouth Coll. v. Woodward, holding that corporate charters gave trustees private rights therein, which were protected against state alteration by the Constitution;
NAACP v. Alabama ex rel. Patterson, holding that non-profit corporation could assert First Amendment rights of its members;
Bains LLC v. Arco Prod. Co., Div. of Atl. Richfield Co., holding that a corporation had standing to bring racial discrimination claim for racial discrimination against its employees.

Implications
I believe that this understanding of the corporate civil rights "struggle" has small-but-nontrivial implications for a potential future strategy to secure legal protections for digital minds. Specifically, I think Winkler's thesis suggests that the extension of rights to corporations is not a useful historical or legal analogy for the potential extension of rights to digital minds. This is because Winkler's book demonstrates that corporations gained rights primarily because their constitutive members (i.e., shareholders) already had rights. In the case of digital minds generally, I see no obvious analogy to shareholders: digital minds as such are not mere extensions or associations of entities already bearing rights.
More concretely, this suggests that securing legal personhood for digital minds for instrumental reasons is not likely, on its own, to increase the likelihood of legal protections for them.
Carrick Flynn suggested to me (and I now agree) that nonhuman animal protections probably provide the best analog for future digital mind protections. To the extent that it rules out another possible method of approaching the question, this post supports that thesis.
This work was financially supported by the Berkeley Existential Risk Initiative.


Hardware overhang

Hardware overhang refers to a situation where large quantities of computing hardware can be diverted to running powerful AI systems as soon as the software is developed.
Details
Definition
In the context of AI forecasting, hardware overhang refers to a situation where enough computing hardware to run many powerful AI systems already exists by the time the software to run such systems is developed. If such hardware is repurposed to AI, this would mean that as soon as one powerful AI system exists, probably a large number of them do. This might amplify the impact of the arrival of human-level AI.
The alternative to a hardware overhang is for software sufficient for powerful AI to be developed before the hardware to run it has become cheap. In that case, we might instead expect to see the first powerful AI be built when it is very expensive, and therefore continues to be rare.

Hardware overhang


Historic trends in structure heights

Trends for tallest ever structure heights, tallest ever freestanding structure heights, tallest existing freestanding structure heights, and tallest ever building heights have each seen 5-8 discontinuities of more than ten years. These are:
Djoser and Meidum pyramids (~2600BC, >1000 year discontinuities in all structure trends)Three cathedrals that were shorter than the all-time record (Beauvais Cathedral in 1569, St Nikolai in 1874, and Rouen Cathedral in 1876, all >100 year discontinuities in current freestanding structure trend)Washington Monument (1884, >100 year discontinuity in both tallest ever structure trends, but not a notable discontinuity in existing structure trend)Eiffel Tower (1889, ~10,000 year discontinuity in both tallest ever structure trends, 54 year discontinuity in existing structure trend)Two early skyscrapers: the Singer Building and the Metropolitan Life Tower (1908 and 1909, each >300 year discontinuities in building height only)Empire State Building (1931, 19 years in all structure trends, 10 years in buildings trend)KVLY-TV mast (1963, 20 year discontinuity in tallest ever structure trend)Taipei 101 (2004, 13 year discontinuity in building height only)Burj Khalifa (2009, ~30 year discontinuity in both freestanding structure trends, 90 year discontinuity in building height trend)
Details
Background
Over human history, the tallest man-made structures have included mounds, pyramids, churches, towers, a monument, skyscrapers, and radio and TV masts.
Height records often distinguish between structures and buildings, where a building is 'regularly inhabited or occupied' according to Wikipedia, or is 'designed for residential, business or manufacturing purposes' and 'has floors' according to the Council on Tall Buildings and Urban Habitat.1 Figure 1a is an illustration from Wikipedia showing the historic relationship between the heights of buildings and structures.2

Figure 1a: Recent history of tall structures by type.3
Height records also distinguish 'freestanding' structures from other structures. According to Wikipedia, "To be freestanding a structure must not be supported by guy wires, the sea or other types of support. It therefore does not include guyed masts, partially guyed towers and drilling platforms but does include towers, skyscrapers (pinnacle height) and chimneys."4 Definitions vary, for instance Guinness World Records apparently treats underwater structures as 'freestanding'.5 We ignore underwater height in general, excluding underwater structures from 'freestanding' records and and 'all structures' records. The heights of buildings in particular are commonly measured in terms of 'architectural height' or 'height to tip', which both start at the lowest, significant, open-air, pedestrian entrance, but differ in that 'to tip' includes 'functional-technical equipment' like antennae, signage or flag poles, while architectural height does not6 Our understanding is that 'pinnacle height' is the same as 'height to tip'. There are also several less common measures in use.
Height records must also distinguish between the tallest structure standing at a given time, and the tallest structure to have ever existed, at that time. The tallest building or structure at a particular time is sometimes not the tallest ever, when the tallest is damaged without anything taller being built. For instance, the tallest structures in the 1700s were shorter than earlier records, because those were church spires which became damaged without replacement (see Figure 1b).

Figure 1b: An illustration of structure heights over time by location from Wikipedia. 7 (Click to enlarge)
Trends
We collected data for several combinations of measurement possibilities mentioned:
Tallest ever structures on land (i.e. freestanding or not, but not underwater), measured to tipTallest ever freestanding structures, measured to tipTallest existing freestanding structures, measured to tipTallest ever buildings, measured to architectural height
Tallest ever structure heights
Data
We collected height records from numerous Wikipedia lists of tall buildings and structures.8 We have not extensively verified these sources, though we made minor adjustments and additions from elsewhere online where sources were inconsistent or records incomplete. Our data is in this spreadsheet, sheet 'Structures collection'. Figure 2 shows this data.
Figure 2: Our collection of height records for man-made above ground structures, from a variety of online sources (excluding two earlier records). Note that some records are repeated in slightly different versions or are for the same structure being extended, or becoming a record again after the destruction of another structure. The collection is constructed to contain the tallest structures, but the subset of non-tallest structures included is arbitrary.
We constructed a timeline of tallest ever structures by pinnacle height from the tallest ever records in this dataset (see sheet 'Structures (all time, pinnacle)' in the spreadsheet). This is shown in figures 3a and 3b below.
Figure 3a: Recent history of tallest structures ever built on land (not necessarily freestanding). The record may be taller than any structure standing at a given time.
Figure 3b: Longer term history of tallest structures ever built on land (not necessarily freestanding), on a log scale. The record may be taller than any structure standing at a given time.

Discontinuity measurement
We treat this data as exponential initially followed by three linear trends. Using these trends as the previous rate to compare to, we calculated for each record how many years ahead of the trend it was.9 The series contained six unambiguous greater-than-ten-year discontinuities, shown in Table 1 below.
The Bent Pyramid appears to represent a 12 year discontinuity, but we ignore this because its date of construction seems uncertain relative to the small discontinuity.10
While our early records are presumably incomplete, we do not avoid measuring early discontinuities for this reason, because the large discontinuities that we find before the 19th Century seem unlikely to depend substantially on the exact set of earlier records.
Table 1: discontinuities in tallest ever structure heights
YearHeight (m)Discontinuity (years)Structure2650 BC62.5~9000Pyramid of Djoser2610 BC91.65~1000Meidum Pyramid1884169.3106Washington Monument1889300~10,000Eiffel Tower193138119Empire State Building1963628.820KVLY-TV mast
A number of other potentially relevant metrics are tabulated here.
Tallest ever freestanding structure heights
Data
This data is another subset of the 'structures collection' described above, this time including only records for 'freestanding' structures. This excludes structures supported by guy ropes, such as radio masts. Guyed masts were the tallest structures on land overall between 1954 and 2008, so this dataset differs from the 'tallest ever structure heights' dataset above between those years.
This dataset can be found in this spreadsheet, sheet 'Freestanding structures (all time, pinnacle)'. Figures 4-5 below illustrate it.
Figure 4: Recent history of tallest freestanding structures ever built.
Figure 5: Longer term history of tallest freestanding structures ever built, on a log scale.
Discontinuity measurement
We treat this data as exponential initially followed by three linear trends. Using these trends as the previous rate to compare to, we calculated for each record how many years ahead of the trend it was.11 The series contained six unambiguous greater-than-ten-year discontinuities. The first five are the same as those in the previous dataset, since the series do not diverge until later (see Tallest ever structure heights section above for further details). The last discontinuity is a 32 year jump in 2009 from the Burj Khalifa.
We tabulated a number of other potentially relevant metrics here.
Tallest existing freestanding structure heights
We constructed a dataset of tallest freestanding structures over time largely from Wikipedia's Timeline of world's tallest freestanding structures12, with some modifications. This is available in our spreadsheet, sheet 'Freestanding structures (current, pinnacle)', and is shown in Figures 6-7 below.
Figure 6: Recent history of tallest freestanding structures standing. New records are sometimes shorter than old records.
Figure 7: Longer term history of tallest freestanding structures standing, on a log scale. New records are sometimes shorter than old records.
Discontinuity measurement
We treat this data as exponential initially, followed by four linear trends. Using these trends as the 'previous rate' to compare to,13 the data contained eight unambiguous greater than ten year discontinuities, shown in Table 2 below.14
This series differs from that of all-time tallest freestanding structures above by the insertion of a series of records between Lincoln Cathedral in 1311 and the Washington Monument in 1889. This change made the Washington Monument unexceptional rather than a 100 year discontinuity, and the Eiffel Tower a fifty-year discontinuity rather than a ten-thousand year one. Later discontinuities from the Empire State Building and Burj Khalifa are very similar.
Table 2: discontinuities in tallest existing freestanding structures
YearHeight (m)Discontinuity (years)Structure2650 BC62.5~9000Pyramid of Djoser2610 BC91.65~1000Meidum Pyramid1569153138Beauvais Cathedral1874147.3224St Nikolai1876151307Rouen Cathedral188930054Eiffel Tower193138119Empire State Building2009829.835Burj Khalifa
We have tabulated a number of other potentially relevant metrics here.15
Tallest ever building heights
Data
We collected data on the tallest ever buildings from Wikipedia's History of the world's tallest buildings,16 and added it to this spreadsheet (sheet 'Buildings (all time, architectural)'). We have not thoroughly verified it, but have made minor modifications (noted in the spreadsheet). Figure 8 shows this data.
Figure 8: Height of tallest buildings ever built, measured using 'architectural height', which excludes some additions such as antennae.
Figure 9: Close up of Figure 8
Discontinuity measurement
We treated this data as an exponential trend followed by a linear trend.17 Compared to previous rates within these trends, tallest buildings over time contained five greater than ten year discontinuities, shown in Table 3 below.18
Table 3: discontinuities in tallest ever building heights
YearHeight (m)Discontinuity (years)Building1908186.57383Singer Building1909213.36320Metropolitan Life Tower193138110Empire State Building2004509.213Taipei 101201082890Burj Khalifa
We have tabulated a number of other potentially relevant metrics here.19
Figure 10: Burj Khalifa, current record holder for every listed metric, and discontinuously tall freestanding structure and building.
Notes

Historic trends in structure heights


Interpreting AI compute trends

This is a guest post by Ryan Carey.
Over the last few years, we know that AI experiments have used much more computation than previously. But just last month, an investigation by OpenAI made some initial estimates of just how fast this growth has been. Comparing AlphaGo Zero to AlexNet, they found that the largest experiment now is 300,000-fold larger than the largest experiment six years ago. In the intervening time, the largest experiment in each year has been growing exponentially, with a doubling time of 3.5 months.
The rate of growth of experiments according to this AI-Compute trend is astoundingly fast, and this deserves some analysis. In this piece, I explore two issues. The first is that if experiments keep growing so fast, they will quickly become unaffordable, and so the trend will have to draw to a close. Unless the economy is drastically reshaped, this trend can be sustained for at most 3.5-10 years, depending on spending levels and how the cost of compute evolves over time. The second issue is that if this trend is sustained for even 3.5 more years, the amount of compute used in an AI experiment will have passed some interesting milestones. Specifically, the compute used by an experiment will have passed the amount required to simulate, using spiking neurons, a human mind thinking for eighteen years. Very roughly speaking, we could say that the trend would surpass the level required to reach the level of intelligence of an adult human, given an equally efficient algorithm. In sections (1) and (2), I will explore these issues in turn, and then in section (3), I will discuss the limitations of this analysis and weigh how this work might bear on AGI forecasts.
1. How long can the AI-Compute trend be sustained?
To figure out how long the AI-Compute trend can be economically sustained, we need to know three things: the rate of growth of the cost of experiments, the cost of current experiments, and the maximum amount that can be spent on an experiment in the future.
The size of the largest experiments is increasing with a doubling time of 3.5 months, (about an order of magnitude per year)1, while the cost per unit of computation is decreasing by an order of magnitude every 4-12 years (the long-run trend has improved costs by 10x every 4 years, whereas recent trends have improved costs by 10x every 12 years)2. So the cost of the largest experiments is increasing by an order of magnitude every 1.1 – 1.4 years.3
The largest current experiment, AlphaGo Zero, probably cost about $10M.4
The largest that experiments can get depends who is performing them. The richest actor is probably the US government. Previously, the US spent 1% of annual GDP5 on the Manhattan Project, and ~0.5% of annual GDP on NASA during the Apollo program.6 So let's suppose they could similarly spend at most 1% of GDP, or $200B, on one AI experiment. Given the growth of one order of magnitude per 1.1-1.4 years, and the initial experiment size of $10M, the AI-Compute trend predicts that we would see a $200B experiment in 5-6 years.7 So given a broadly similar economic situation to the present one, that would have to mark an end to the AI-Compute trend.
We can also consider how long the trend can last if government is not involved. Due to their smaller size, economic barriers hit a little sooner for private actors. The largest among these are tech companies: Amazon and Google have current research and development budgets of about ~20B/yr each8, so we can suppose that the largest individual experiment outside of government is $20B. Then the private sector can keep pace with the AI-Compute trend for around ¾ as long as government, or ~3.5-4.5 years.910
On the other hand, the development of specialized hardware could cheapen computation, and thereby cause the trend to be sustainable for a longer period. If some new hardware cheapened compute by 1000x over and above price-performance Moore's Law, then the economic barriers bite a little later– after an extra 3-4 years.11
In order for the AI-Compute trend to be maintained for a really long time (more than about a decade), economic output would have to start growing by an order of magnitude or more per year. This is a really extreme scenario, but the main thing that would make it possible would presumably be some massive economic gains from some extremely powerful AI technology, that would also serve to justify the massive ongoing AI investment.
Of course, it's important to be clear that these figures are upper bounds, and they do not preclude the possibility that the AI-Compute trend may halt sooner (e.g. if AI research proves less economically useful than expected) either in a sudden or more gradual fashion.
So we have shown one kind of conclusion from a rapid trend — that it cannot continue for very long, specifically, beyond 3.5-10 years.
2. When will the AI-Compute trend pass potentially AGI-relevant milestones?
The second conclusion that we can draw is that if the AI-Compute trend continues at its current rapid pace, it will pass some interesting milestones. If the AI-Compute trend continues for 3.5-10 more years, then the size of the largest experiment is projected to reach 107-5×1013 Petaflop/s-days, and so the question is which milestones arrive below that level.12 Which milestones might allow the development of AGI is a controversial topic, but three candidates are:

The amount of compute required to simulate a human brain for the duration of a human childhood
The amount of compute required to simulate a human brain to play the number of Go games Alphago Zero required to become superhuman
The amount of compute required to simulate the evolution of the human brain

Human-childhood milestone
One natural guess for the amount of computation required to create artificial intelligence is the amount of computation used by the human brain. Suppose an AI had (compared to a human):

a similarly efficient algorithm for learning to perform diverse tasks (with respect to both both compute and data),
similar knowledge built in to its architecture,
similar data, and
enough computation to simulate a human brain running for eighteen years, at sufficient resolution to capture the intellectual performance of that brain.

Then, this AI should be able to learn to solve a similarly wide range of problems as an eighteen year-old can solve.13
There is a range of estimates for how many floating point operations per second are required to simulate a human brain for one second. Those collected by AI Impacts have a median of 1018 FLOPS (corresponding roughly to a whole-brain simulation using Hodgkin-Huxley neurons14), and ranging from 3×1013FLOPS (Moravec's estimate) to 1×1025FLOPS (simulating the metabolome). Running such simulations for eighteen years would correspond to a median of 7 million Petaflop/s-days (range 200 – 7×1013 Petaflop/s-days).15
So for the shortest estimates, such as the Moravec estimate, we have already reached enough compute to pass the human-childhood milestone. For the median estimate, and the Hodgkin-Huxley estimates, we will have reached the milestone within 3.5 years. For the metabolome estimates, the required amount of compute cannot be reached within the coming ten year window before the AI-Compute trend is halted by economic barriers. After the AI-Compute trend is halted, it's worth noting that Moore's Law could come back to the fore, and cause the size of experiments to continue to slowly grow. But on Moore's Law, milestones like the metabolome estimate are still likely decades away.
AlphaGo Zero-games milestone
One objection to the human-childhood milestone is that AI systems presently are "slower-learners" than humans. AlphaGo Zero used 2.5 million Go games to become superhuman16, which if each game took at hour, would correspond to 300 years of Go games17. We might ask how long it would take to run something as complex as the human brain, for 300 years, rather than just eighteen. In order for this milestone to be reached, the trend would have to continue for another 14 months longer than the human-childhood milestone18.
Brain-evolution milestone
A more conservative milestone is the amount of compute required to simulate all neural evolution. One approach, described by Shulman and Bostrom 2012, is to look at the cost of simulating the evolution of nervous systems. This entails simulating 1025 neurons for one billion years.19 Shulman and Bostrom estimate the cost simulating a neuron for one second at 1-1010 floating point operations,20 and so the total cost for simulating evolution is 3×1021-3×1031 Petaflop/s-days21. This figure would not be reached until far beyond the time when the current AI-Compute trend must end. So the AI-Compute trend does not change the conclusion of Shulman and Bostrom that simulation of brain evolution on Earth is far away — even with a rapid increase in spending, this compute milestone would take many decades of advancement of Moore's Law to be reached22.
Overall, we can see that although the brain-evolution milestone is well beyond the AI-Compute trend, the others are not necessarily. For some estimates — especially metabolome estimates — the human-childhood and AlphaGo Zero-games milestones cannot be reached either. But some of the human-childhood and AlphaGo Zero-games milestones will be reached if the AI Compute trend continues for the next few years.
3. Discussion and Limitations
In light of this analysis, a reasonable question to ask is: for the purpose of predicting AGI, which milestone should we care most about? This is very uncertain, but I would guess that building AGI is easier than the brain-evolution milestone would suggest, but that AGI could arrive either before, or after the AlphaGo Zero-games milestone is reached.
The first claim is because the brain-evolution milestone assumes that the process of algorithm discovery must be performed by the AI itself. It seems more likely to me that the appropriate algorithm is provided (or mostly provided) by the human designers at no computational cost (or at hardly any cost compared to simulating evolution).
The second matter — evaluating the difficulty of AGI relative to the AlphaGo Zero-games milestone — is more complex. One reason for thinking that the AlphaGo Zero-games milestone makes AGI look too easy is that more training examples ought to be required to teach general intelligence, than are required to learn the game of Go.23 In order to perform a wider range of tasks, it will be necessary to consider a larger range of dependencies and to learn a more intricate mapping from actions to utilities. This matter could be explored further by comparing the sample efficiency of various solved AI problems and extrapolating the sample efficiency of AGI based on how much more complicated general intelligence seems. However, there are also reasons the AlphaGo Zero-games milestone might make things look too hard. Firstly, AlphaGo Zero does not use any pre-existing knowledge, whereas AGI systems might. If we had looked instead at the original AlphaGo, this would have required an order of magnitude fewer games relative to AlphaGo Zero24, and further efficiency gains might be possible for more general learning tasks. Secondly, there might be one or more orders of magnitude of conservatism built-in to the approach of using simulations of the human brain. Simulating the human brain on current hardware may be a rather inefficient way to capture its computing function: that is, the human brain might only be using some fraction of the computation that is needed to simulate it. So it's hard to judge whether the AlphaGo Zero-games milestone is too late or too soon for AGI.
There is another reason for some more assurance that AGI is more than six years away. We can simply look at the AI-Compute trend and ask ourselves: is AGI as close to AlphaGo Zero as AlphaGo Zero is to AlexNet? If we think that the difference (in terms of some combination of capabilities, compute, or AI research) between the first pair is larger than the second, then we should think that AGI is more than six years away.
In conclusion, we can see that the AI-Compute trend is an extraordinarily fast trend that economic forces (absent large increases in GDP) cannot sustain beyond 3.5-10 more years. Yet the trend is also fast enough that if it is sustained for even a few years from now, it will sweep past some compute milestones that could plausibly correspond to the requirements for AGI, including the amount of compute required to simulate a human brain thinking for eighteen years, using Hodgkin Huxley neurons. However, other milestones will not be reached before economic factors halt the AI-Compute trend. For example, this analysis shows that we will not have enough compute to simulate the evolution of the human brain for (at least) decades.
Thanks Jack Gallagher, Danny Hernandez, Jan Leike, and Carl Shulman for discussions that helped with this post.

Interpreting AI compute trends


Occasional update July 5 2018

Before I get to substantive points, there has been some confusion over the distinction between blog posts and pages on AI Impacts. To make it clearer, this blog post shall proceed in a way that is silly, to distinguish it from the very serious and authoritative reference pages that comprise the bulk of AI Impacts.
Now for a picture of a duck, to remind you that this is silly, and also that we are all fragile biological organisms that evolved because apparently that's what happens if you just leave a bunch of wet mud on a space rock for long enough, alone and vulnerable in a hostile and uncharted world.

And now to the exciting facts on the ground, as we try to marginally rectify that situation.
People
Tegan McCaslin is now working at AI Impacts, as far as I can tell about five hundred hours a week. It's going well, except that the rate at which she sends me extensive, carefully researched articles about neuroanatomy and genetics and such to review is in slight tension with my preferred lifestyle.
We also welcome Carl Shulman as occasional consultant on everything, and reviewer of things (especially articles about neuroanatomy and genetics and such…)
Justis Mills joined us last year, to work on miscellany. He usually does one of those software related things, but in his spare time he has been making illustrative timelines of near-term AI predictions and checking whether everything on AI Impacts isn't obviously false, and fixing bits of it, and such.
We mostly-farewell Michael Wulfsohn—an Australian economist, called to us from the Central Bank of Lesotho by WaitButWhy—who is winding up his assessment of how great avoiding human extinction might be (having already estimated how how much of a bother it might be). He has gone to get a PhD, the better to save the world.
Places
Our implicit office has moved from a spare room in my house to Tegan's house. This is good, because she has an absurdly nice rug, and an excellent snack drawer, and it is a minor ambition of mine to head an organization which has Bay Area start-up quality snack areas.
We have also been trying out co-working with other save-the-world-something-something-AI related folks around Berkeley, which seems promising. We have also been trying out co-working with Oxford, which seems promising, but not as Bay Area convenient as we would like.
Things we want
We want to hire more people. Relatedly, we would like money. We think these things would nicely complement our many brilliant and tractable research ideas and our ambition. We also want to have our own T-shirts, but that is on the back-burner.
Things we got
$100,000 from The Open Philanthropy Project, for the next two years.
$39,000 from another donor to support several specific research projects from our list of promising research projects.
Projects
You can mostly see what we are up to by watching various parts of our front page, so I shan't go into it all, except to say that I for one am especially enjoying my investigation into reasons to (or not to) expect AI discontinuities. If you too are fascinated by this topic, and want to give especially many pointed comments on it, you can do so on this doc version.
Our survey became the 16th most discussed journal article of 2017, so that was neat. If I recall, I was at least relatively in favor of not writing a paper about it, so I was probably wrong there. (Probably good job, everyone else!) I suspect this success is related to the journalists who have been writing to me endlessly, and me being invited to give talks, and go to Chile and be on the radio and that kind of thing. Which has all been an unusual experience.
How you can get involved
If you want to do this kind of work, consider applying for a job with us, or just doing one of these projects anyway, and sending it to us. If you want to chat about this kind of research, or spy on it, or help a tiny bit noncommittally, ask us nicely and we might add you to our fairly open Slack. If you want to help in some other way, we especially welcome money and any good researchers you have hanging around, but are open to other ideas.

Occasional update July 5 2018


Trend in compute used in training for headline AI results

Compute used in the largest AI training runs appears to have roughly doubled every 3.5 months between 2012 and 2018.
Details
According to Amodei and Hernandez, on the OpenAI Blog:
…since 2012, the amount of compute used in the largest AI training runs has been increasing exponentially with a 3.5 month-doubling time (by comparison, Moore's Law had an 18-month doubling period). Since 2012, this metric has grown by more than 300,000x (an 18-month doubling period would yield only a 12x increase)…
They give the following figure, and some of their calculations. We have not verified their calculations, or looked for other reports on this issue.
Figure 1: Originally captioned: The chart shows the total amount of compute, in petaflop/s-days, that was used to train selected results that are relatively well known, used a lot of compute for their time, and gave enough information to estimate the compute used. A petaflop/s-day (pfs-day) consists of performing 1015 neural net operations per second for one day, or a total of about 1020 operations. The compute-time product serves as a mental convenience, similar to kW-hr for energy. We don't measure peak theoretical FLOPS of the hardware but instead try to estimate the number of actual operations performed. We count adds and multiplies as separate operations, we count any add or multiply as a single operation regardless of numerical precision (making "FLOP" a slight misnomer), and we ignore ensemble models. Example calculations that went into this graph are provided in this appendix. Doubling time for line of best fit shown is 3.43 months.


The tyranny of the god scenario

Michael Wulfsohn is an AI Impacts researcher/contributor.
I was convinced. An intelligence explosion would result in the sudden arrival of a superintelligent machine. Its abilities would far exceed those of humans in ways we can't imagine or counter. It would likely arrive within a few decades, and would wield complete power over humanity. Our species' most important challenge would be to solve the value alignment problem. The impending singularity would lead either to our salvation, our extinction, or worse.
Intellectually, I knew that it was not certain that this "god scenario" would come to pass. If asked, I would even have assigned it a relatively low probability, certainly much less than 50%. Nevertheless, it dominated my thinking. Other possibilities felt much less real: that humans might achieve direct control over their superintelligent invention, that reaching human-level intelligence might take hundreds of years, that there might be a slow progression from human-level intelligence to superintelligence, and many others. I paid lip service to these alternatives, but I didn't want them to be valid, and I didn't think about them much. My mind would always drift back to the god scenario.
I don't know how likely the god scenario really is. With currently available information, nobody can know for sure. But whether or not it's likely, the idea definitely has powerful intuitive appeal. For example, it led me to change my beliefs about the world more quickly and radically than I ever had before. I doubt that I'm the only one.
Why did I find the god scenario so captivating? I like science fiction, and the idea of an intelligence explosion certainly has science-fictional appeal. I was able to relate to the scenario easily, and perhaps better think through the implications. But the transition from science fiction to reality in my mind wasn't immediate. I remember repeatedly thinking "nahhh, surely this can't be right!" My mind was trying to put the scenario in its science-fictional place. But each time the thought occurred, I remember being surprised at the scenario's plausibility, and at my inability to rule out any of its key components.
I also tend to place high value on intelligence itself. I don't mean that I've assessed various qualities against some measure of value and concluded that intelligence ranks highly. I mean it in a personal-values sense. For example, the level of intelligence I have is a big factor in my level of self-esteem. This is probably more emotional than logical.
This emotional effect was an important part of the god scenario's impact on me. At first, it terrified me. I felt like my whole view of the world had been upset, and almost everything people do day to day seemed to no longer matter. I would see a funny video of a dog barking at its reflection, and instead of enjoying it, I'd notice the grim analogy of the intellectual powerlessness humanity might one day experience. But apart from the fear, I was also tremendously excited by the thought of something so sublimely intelligent. Having not previously thought much about the limits of intelligence itself, the concept was both consuming and eye-opening, and the possibilities were inspiring. The notion of a superintelligent being appealed to me similarly to the way Superman's abilities have enthralled audiences.
Other factors included that I was influenced by highly engaging prose, since I first learned about superintelligence by reading this excellent waitbutwhy.com blog post. Another was my professional background; I was accustomed to worrying about improbable but significant threats, and to arguments based on expected value. The concern of prominent people—Bill Gates, Elon Musk, and Stephen Hawking—helped. Also, I get a lot of satisfaction from working on whatever I think is humanity's most important problem, so I really couldn't ignore the idea.
But there were also countervailing effects in my mind, leading away from the god scenario. The strongest was the outlandishness of it all. I had always been dismissive of ideas that seem like doomsday theories, so I wasn't automatically comfortable giving the god scenario credence in my mind. I was hesitant to introduce the idea to people who I thought might draw negative conclusions about my judgement.
I still believe the god scenario is a real possibility. We should assiduously prepare for it and proceed with caution. However, I believe I have gradually escaped its intuitive capture. I can now consider other possibilities without my mind constantly drifting back to the god scenario.
I believe a major factor behind my shift in mindset was my research interest in analyzing AI safety as a global public good. Such research led me to think concretely about other scenarios, which increased their prominence in my mind. Relatedly, I began to think I might be better equipped to contribute to outcomes in those other scenarios. This led me to want to believe that the other scenarios were more likely, a desire compounded by the danger of the god scenario. My personal desires may or may not have influenced my objective opinion of the probabilities. But they definitely helped counteract the emotional and intuitive appeal of the god scenario.
Exposure to mainstream views on the subject also moderated my thinking. In one instance, reading an Economist special report on artificial intelligence helped counteract the effects I've described, despite that I actually disagreed with most of their arguments against the importance of existential risk from AI.
Exposure to work done by the Effective Altruism community on different future possibilities also helped, as did my discussions with Katja Grace, Robin Hanson, and others during my work for AI Impacts. The exposure and discussions increased my knowledge and the sophistication of my views such that I could better imagine the range of AI scenarios. Similarly, listening to Elon Musk's views of the importance of developing brain-computer interfaces, and seeing OpenAI pursue goals that may not squarely confront the god scenario, also helped. They gave me a choice: decide without further ado that Elon Musk and OpenAI are misguided, or think more carefully about other potential scenarios.
Relevance to the cause of AI safety
I believe the AI safety community probably includes many people who experience the god scenario's strong intuitive appeal, or have previously experienced it. This tendency may be having some effects on the field.
Starting with the obvious, such a systemic effect could cause pervasive errors in decision-making. However, I want to make clear that I have no basis to conclude that it has done so among the Effective Altruism community. For me, the influence of the god scenario was subtle, and driven by its emotional facet. I could override it when asked for a rational assessment of probabilities. But its influence was pervasive, affecting the thoughts to which my mind would gravitate, the topics on which I would tend to generate ideas, and what I would feel like doing with my time. It shaped my thought processes when I wasn't looking.
Preoccupation with the god scenario may also entail a public relations risk. Since the god scenario's strong appeal is not universal, it may polarize public opinion, as it can seem bizarre or off-putting to many. At worst, a rift may develop between the AI safety community and the rest of society. This matters. For example, policymakers throughout the world have the ability to promote the cause of AI safety through funding and regulation. Their involvement is probably an essential component of efforts to prevent an AI arms race through international coordination. But it is easier for them to support a cause that resonates with the public.
Conversely, the enthusiasm created by the intuitive appeal of the god scenario can be quite positive, since it attracts attention to related issues in AI safety and existential risk. For example, others' enthusiasm and work in these areas led me to get involved.
I hope readers will share their own experience of the intuitive appeal of the god scenario, or lack thereof, in the comments. A few more data points and insights might help to shed light.

The tyranny of the god scenario


Promising research projects

This is an incomplete list of concrete projects that we think are tractable and important. We may do any of them ourselves, but many also seem feasible to work on independently. Those we consider especially well suited to this are marked Ψ. More potential projects are listed here.
Project
Review the literature on forecasting (in progress) Ψ
Summarize what is known about procedures that produce good forecasts, and measures that are relatively easier to forecast. This may involve reading secondary sources, or collecting past forecasts and investigating what made some of them successful.
This would be an input to improving our own forecasting practices, and to knowing which other forecasting efforts to trust.
We have reviewed some literature associated with the Good Judgment Project in particular.
Review considerations regarding the chance of local, fast takeoff Ψ
We have a list of considerations here. If you find local, fast take-off likely, check if the considerations that lead you to this view are represented. Alternately, interview someone else with a strong position about the considerations they find important. If there are any arguments or counterarguments that you think are missing, write a short page explaining the case.
Collecting arguments on this topic is helpful because opinion among well-informed thinkers on the topic seems to diverge from what would be expected given the arguments that we know about. This suggests that we are missing some important considerations, that we would need to well assess the chance of local, fast takeoff.
Quantitatively model an intelligence explosion Ψ
An intelligence explosion (or 'recursive self-improvement') consists of a feedback loop where researcher efforts produce scientific progress, which produces improved AI performance, which produces more efficient researcher efforts. This forms a loop, because the researchers involved are artificial themselves.
Though this loop does not yet exist, relatively close analogues to all of the parts of it already occur: for instance, researcher efforts do lead to scientific progress; scientific progress does lead to better AI; better AI does lead to more capacity at the kinds of tasks that AI can do.
Collect empirical measurements of proxies like these, for different parts of the hypothesized loop (each part of this could be a stand-alone project). Model the speed of the resulting loop if they were put together, under different background conditions.
This would give us a very rough estimate of the contribution of intelligence explosion dynamics to the speed of intelligence growth in a transition to an AI-based economy. Also, a more detailed model may inform our understanding of available strategies to improve outcomes.
Interview AI researchers on topics of interest Ψ
Find an AI researcher with views on matters of interest (e.g. AI risk, timelines, the relevance of neuroscience to AI progress) and interview them. Write a summary, or transcript (with their permission). Some examples here, here, here. (If you do not expect to run an interview well enough to make a good impression on the interviewee, consider practicing elsewhere first, so as not to discourage interacting with similar researchers in the future.)
Talking to AI researchers about their views can be informative about the nature of AI research (e.g. What problems are people trying to solve? How much does it seem like hardware matters?), and provide an empirically informed take on questions and considerations of interest to us (e.g. Current techniques seem really far from general). They also tell us about state of opinion within the AI research community, which may be relevant in itself.
Review what is known about the relative intelligence of humans, chimps, and other animals (in progress)
Review efforts to measure animal and human intelligence on a single scale, and efforts to quantify narrower cognitive skills across a range of animals.
Humans are radically more successful than other animals, in some sense. This is taken as reason to expect that small modifications to brain design (for instance whatever evolution did between the similar brains of chimps and humans) can produce outsized gains in some form of mental performance, and thus that AI researchers may see similar astonishing progress near human-level AI.
However without defining or quantifying the mental skills of any relevant animals, it is unclear a) whether individual intelligence in particular accounts for humans' success (rather than e.g. ability to accrue culture and technology), b) whether the gap in capabilities between chimps and humans is larger than expected (maybe chimps are also astonishingly smarter than smaller mammals), or c) whether the success stems from something that evolution was 'intentionally' progressing on. These things are all relevant to the strength of an argument for AI 'fast take-off' based on human success over chimps (see here).
Review explanations for humans' radical success over apes (in progress)
Investigate what is known about the likely causes of human success, relative to that of other similar animals. In particular, we are interested in how likely improvement in individual cognitive ability is to account for this (as opposed to say communication and group memory abilities).
This would help resolve the same issues described in the last section ('Review what is known about the relative intelligence of humans, chimps, and other animals').
Collect data on time to cross the human range on intellectual skills where machines have surpassed us (in progress) Ψ
For intellectual skills where machines have surpassed humans, find out how long it took to go from the worst performance to average human skill, and from average human skill to superhuman skill.
This would contribute to this project.
Measure the importance of hardware progress in a specific narrow AI trajectory Ψ
Take an area of AI progress, and assess how much of annual improvement can be attributed to hardware improvements vs. software improvements, or what the more detailed relationship between the two is.
Understanding the overall importance of hardware progress and software progress (and other factors) in overall AI progress lets us know to what extent our future expectations should be a function of expected hardware developments, versus software developments. This both alters what our timelines look like (e.g. see here), and tells us what we should be researching to better understand AI timelines.

Promising research projects


Brain wiring: The long and short of it

When I took on the task of counting up all the brain's fibers and figuratively laying them end-to-end, I had a sense that it would be relatively easy–do a bit of strategic Googling, check out a few neuroscience references, and you've got yourself a relaxing Sunday afternoon project. By that afternoon project's 40th hour, I had begun to question my faith in Paul Christiano's project length estimates.
It was actually pretty surprising how thin on the ground numbers and quantities about the metrics I was after seemed to be. Even somewhat simple questions, like "how many of these neuron things does the brain even have", proved not to have the most straightforward answer. According to one author, the widely-cited, rarely-caveated figure in textbooks of 100 billion neurons couldn't be sourced in any primary literature published before the late 2000s, and this echo chamber-derived estimate was subsequently denounced for being off by tens of billions in either direction (depending on who you ask). But hey, what's a few tens of billions between friends?
The question of why these numbers are so hard to find is an interesting one. One answer is that it's genuinely difficult to study populations of cells at the required level of detail. Another is that perhaps neuroscientists are too busy studying silly topics like "how the brain works" or "clinically relevant things" to get down to the real meat of science, which is anal-retentively cataloging every quantity that could plausibly be measured. Perhaps the simplest explanation is just that questions like "how long is the entire dendritic arbor of a Purkinje cell" didn't have a great argument for why they might be useful, prior to now.
Or maybe its "fuck it"-quotient was too high.
Which brings us rather neatly to the point of why an AI forecasting organization might care about the length of all the wires in the brain, even when the field of neuroscience seems not to. At a broad level, it's probably the case that neuroscientists care about very different aspects of the brain than AI folks do, because neuroscientists mostly aren't trying to solve an engineering problem (at least, not the engineering problem of "build a brain out of bits of metal and plastic"). The particular facet of that engineering problem we were interested in here was: how much of a hurdle is hauling information around going to be, once computation is taken care of?
Our length estimates don't provide an exhaustive answer to that question, and to be honest they can't really tell you anything on their own. But, as is the case with AI Impacts' 2015 article on brain performance in TEPS, learning these facts about the brain moves us incrementally closer to understanding how promising our current models of hardware architectures are, and where we should expect to encounter trouble.
Some interesting takeaways: long-range fibers–that is, myelinated ones–probably account for about 20% of the total length of brain wires. Also, the neocortex is huge, but not because it has lots of neurons.
Scroll to the bottom of our article if you're a cheater who just wants to see a table full of summary statistics, but read the whole thing if you want those numbers to have some context. And please contact me if you spot anything wrong, or think I missed something, or if you're Wen or Chklovskii of Wen and Chklovskii 2004 and you want to explain your use of tildes in full detail.
By Tegan McCaslin

Brain wiring: The long and short of it