I really liked this report: it’s extremely thorough and anticipates and responds to a large number of potential reactions. I’ve made my own timelines estimate using the provided spreadsheet, and have adopted the resulting graph (with a few modifications) as my TAI timeline (which ends up with a median of ~2055). This is saying quite a lot: it’s pretty rare that a quantitative model is compelling enough that I’m inclined to only slightly edit its output, as opposed to simply using the quantitative model to inform my intuitions.

Here are the main ways in which my model is different from the one in the report:

**1. Ignoring the genome anchor**

I ignore the genome anchor because I don’t buy the model: even if researchers did create a very parameter-efficient model class (which seems unlikely), I would not expect the same scaling laws to apply to that model class. The report mentions that you could also interpret the genome anchor as simply providing a constraint on how many data points are needed to train long-horizon behaviors (since that’s what evolution was optimizing), but I prefer to take this as (fairly weak) evidence that informs what weights to place on short vs. medium vs. long horizons for neural nets.

**2. Placing more weight on short and medium horizons relative to long horizons**

I place 30% on short horizons, 40% on medium horizons, and 10% on long horizons. The report already names several reasons why we might expect the long horizon assumption to be too conservative. I agree with all of those, and have one more of my own:

If meta-learning turns out to require a huge amount of compute, we can instead directly train on some transformative task with a lower horizon. Even some of the hardest tasks like scientific R&D shouldn’t have a huge horizon: even if we assume that it takes human scientists a year to produce the equivalent of a single data point, at 40 hours a week that comes out to a horizon of 2000 subjective hours, or 7e6 seconds. This is near the beginning of the long horizon realm of 1e6-1e9 seconds and seems like a very conservative overestimate to me.

(Note that in practice I’d guess we will train something like a meta-learner, because I suspect the skill of meta-learning will not require such large average effective horizons.)

**3. Reduced willingness to spend**

My willingness to spend forecasts are somewhat lower: the predictions and reasoning in this report feel closer to upper bounds on how much people might spend rather than predictions of how much they will spend. Assuming we reduce the ratio of all-in project costs to final training run costs to 10x, spending $1B on a training run by 2025 would imply all-in project costs of $10B, which is ~40% of Google’s yearly R&D budget of $26B, or 10% of the budget for a 4-year project. Possibly this wouldn’t be classified as R&D, but it would also be _2% of all expenditures over 4 years_. This feels remarkably high to me for something that’s supposed to happen within 5 years; while I wouldn’t rule it out, it wouldn’t be my median prediction.

**4. Accounting for challenges**

While the report does talk about challenges in e.g. getting the right data and environments by the right time, I think there are a bunch of other challenges as well: for example, you need to ensure that your model is aligned, robust, and reliable (at least if you want to deploy it and get economic value from it). I do expect that these challenges will be easier than they are today, partly because more research will have been done, and partly because the models themselves will be more capable.

Another example of a challenge would be PR concerns: it seems very plausible to me that there will be a backlash against transformative AI systems resulting in those systems being deployed later than we’d expect them to be according to this model.

To be more concrete, if we ignore points 1-3 and assume this is my only disagreement, then for the median of 2052, rather than assuming that reasons for optimism and pessimism approximately cancel out to yield 2050 as the median for TAI, I’d be inclined to shade upwards to 2055 or 2060 as my median for TAI.

Draft report on AI timelines

Once again, we have a piece of work so large and detailed that I need a whole newsletter to summarize it! This time, it is a quantitative model for forecasting when transformative AI will happen. Note that since this is still a draft report, the numbers are in flux; the numbers below were taken when I read it and may no longer be up to date.

**The overall framework**

The key assumption behind this model is that if we train a neural net or other ML model that uses about as much computation as a human brain, that will likely result in transformative AI (TAI) (defined as AI that has an impact comparable to that of the industrial revolution). In other words, we _anchor_ our estimate of the ML model's inference computation to that of the human brain. This assumption allows us to estimate how much compute will be required to train such a model _using 2020 algorithms_. By incorporating a trend extrapolation of how algorithmic progress will reduce the required amount of compute, we can get a prediction of how much compute would be required for the final training run of a transformative model in any given year.

We can also get a prediction of how much compute will be _available_ by predicting the cost of compute in a given year (which we have a decent amount of past evidence about), and predicting the maximum amount of money an actor would be willing to spend on a single training run. The probability that we can train a transformative model in year Y is then just the probability that the compute _requirement_ for year Y is less than the compute _available_ in year Y.

The vast majority of the report is focused on estimating the amount of compute required to train a transformative model using 2020 algorithms (where most of our uncertainty would come from); the remaining factors are estimated relatively quickly without too much detail. I'll start with those so that you can have them as background knowledge before we delve into the real meat of the report. These are usually modeled as logistic curves in log space: that is, they are modeled as improving at some constant rate, but will level off and saturate at some maximum value after which they won't improve.

**Algorithmic progress**

First off, we have the impact of _algorithmic progress_. <@AI and Efficiency@> estimates that algorithms improve enough to cut compute times in half every 16 months. However, this was measured on ImageNet, where researchers are directly optimizing for reduced computation costs. It seems less likely that researchers are doing as good a job at reducing computation costs for "training a transformative model", and so the author increases the **halving time to 2-3 years**, with a maximum of **somewhere between 1-5 orders of magnitude** (with the assumption that the higher the "technical difficulty" of the problem, the more algorithmic progress is possible).

**Cost of compute**

Second, we need to estimate a trend for compute costs. There has been some prior work on this (summarized in [AN #97](https://mailchi.mp/a2b5efbcd3a7/an-97-are-there-historical-examples-of-large-robust-discontinuities)). The report has some similar analyses, and ends up estimating **a doubling time of 2.5 years**, and a (very unstable) maximum of improvement by **a factor of 2 million by 2100**.

**Willingness to spend**

Third, we would like to know the maximum amount (in 2020 dollars) any actor might spend on a single training run. Note that we are estimating the money spent on a _final training run_, which doesn't include the cost of initial experiments or the cost of researcher time. Currently, the author estimates that all-in project costs are 10-100x larger than the final training run cost, but this will likely go down to something like 2-10x, as the incentive for reducing this ratio becomes much larger.

The author estimates that the most expensive run _in a published paper_ was the final <@AlphaStar@>(@AlphaStar: Mastering the Real-Time Strategy Game StarCraft II@) training run, at ~1e23 FLOP and $1M cost. However, there have probably been unpublished results that are slightly more expensive, maybe $2-8M. In line with <@AI and Compute@>, this will probably increase dramatically to about **$1B in 2025**.

Given that AI companies each have around $100B cash on hand, and could potentially borrow additional several hundreds of billions of dollars (given their current market caps and likely growth in the worlds where AI still looks promising), it seems likely that low hundreds of billions of dollars could be spent on a single run by 2040, corresponding to a doubling time (from $1B in 2025) of about 2 years.

To estimate the maximum here, we can compare to megaprojects like the Manhattan Project or the Apollo program, which suggests that a government could spend around 0.75% of GDP for ~4 years. Since transformative AI will likely be more valuable economically and strategically than these previous programs, we can shade that upwards to 1% of GDP for 5 years. Assuming all-in costs are 5x that of the final training run, this suggests the maximum willingness to spend should be 1% of GDP of the largest country, which we assume grows at ~3% every year.

**Strategy for estimating training compute for a transformative model**

In addition to the three factors of algorithmic progress, cost of compute, and willingness to spend, we need an estimate of how much computation would be needed to train a transformative model using 2020 algorithms (which I'll discuss next). Then, at year Y, the compute required is given by computation needed with 2020 algorithms * improvement factor from algorithmic progress, which (in this report) is a probability distribution. At year Y, the compute available is given by FLOP per dollar (aka compute cost) * money that can be spent, which (in this report) is a point estimate. We can then simply read off the probability that the compute required is greater than the compute available.

Okay, so the last thing we need is a distribution over the amount of computation that would be needed to train a transformative model using 2020 algorithms, which is the main focus of this report. There is a lot of detail here that I'm going to elide over, especially in talking about the _distribution_ as a whole (whereas I will focus primarily on the median case for simplicity). As I mentioned early on, the key hypothesis is that we will need to train a neural net or other ML model that uses about as much compute as a human brain. So the strategy will be to first translate from "compute of human brain" to "inference compute of neural net", and then to translate from "inference compute of neural net" to "training compute of neural net".

**How much inference compute would a transformative model use?**

We can talk about the rate at which synapses fire in the human brain. How can we convert this to FLOP? The author proposes the following hypothetical: suppose we redo evolutionary history, but in every animal we replace each neuron with N [floating-point units](https://en.wikipedia.org/wiki/Floating-point_unit) that each perform 1 FLOP per second. For what value of N do we still get roughly human-level intelligence over a similar evolutionary timescale? The author then does some calculations about simulating synapses with FLOPs, drawing heavily on the <@recent report on brain computation@>(@How Much Computational Power It Takes to Match the Human Brain@), to estimate that N would be around 1-10,000, which after some more calculations suggests that the human brain is doing the equivalent of 1e13 - 1e16 FLOP per second, with **a median of 1e15 FLOP per second**, and a long tail to the right.

Does this mean we can say that a transformative model will use 1e15 FLOP per second during inference? Such a model would have a clear flaw: even though we are assuming that algorithmic progress reduces compute costs over time, if we did the same analysis in e.g. 1980, we'd get the _same_ estimate for the compute cost of a transformative model, which would imply that there was no algorithmic progress between 1980 and 2020! The problem is that we'd always estimate the brain as using 1e15 FLOP per second (or around there), but for our ML models there is a difference between FLOP per second _using 2020 algorithms_ and FLOP per second _using 1980 algorithms_. So how do we convert form "brain FLOP per second" to "inference FLOP per second for 2020 ML algorithms"?

One approach is to look at how other machines we have designed compare to the corresponding machines that evolution has designed. An [analysis](https://docs.google.com/document/d/1HUtUBpRbNnnWBxiO2bz3LumEsQcaZioAPZDNcsWPnos/edit) by Paul Christiano concluded that human-designed artifacts tend to be 2-3 orders of magnitude worse than those designed by evolution, when considering energy usage. Presumably a similar analysis done in the past would have resulted in higher numbers and thus wouldn't fall prey to the problem above. Another approach is to compare existing ML models to animals with a similar amount of computation, and see which one is subjectively "more impressive". For example, the AlphaStar model uses about as much computation as a bee brain, and large language models use somewhat more; the author finds it reasonable to say that AlphaStar is "about as sophisticated" as a bee, or that <@GPT-3@>(@Language Models are Few-Shot Learners@) is "more sophisticated" than a bee.

We can also look at some abstract considerations. Natural selection had _a lot_ of time to optimize brains, and natural artifacts are usually quite impressive. On the other hand, human designers have the benefit of intelligent design and can copy the patterns that natural selection has come up with. Overall, these considerations roughly balance each other out. Another important consideration is that we're only predicting what would be needed for a model that was good at most tasks that a human would currently be good at (think a virtual personal assistant), whereas evolution optimized for a whole bunch of other skills that were needed in the ancestral environment. The author subjectively guesses that this should reduce our estimate of compute costs by about an order of magnitude.

Overall, putting all these considerations together, the author intuitively guesses that to convert from "brain FLOP per second" to "inference FLOP per second for 2020 ML algorithms", we should add an order of magnitude to the median, and add another two orders of magnitude to the standard deviation to account for our large uncertainty. This results in a median of **1e16 FLOP per second** for the inference-time compute of a transformative model.

**Training compute for a transformative model**

We might expect a transformative model to run a forward pass **0.1 - 10 times per second** (which on the high end would match human reaction time of 100ms), and for each parameter of the neural net to contribute **1-100 FLOP per forward pass**, which implies that if the inference-time compute is 1e16 FLOP per second then the model should have **1e13 - 1e17 parameters**, with a median of **3e14 parameters**.

We now need to estimate how much compute it takes to train a transformative model with 3e14 parameters. We assume this is dominated by the number of times you have to run the model during training, or equivalently, the number of data points you train on times the number of times you train on each data point. (In particular, this assumes that the cost of acquiring data is negligible in comparison. The report argues for this assumption; for the sake of brevity I won't summarize it here.)

For this, we need a relationship between parameters and data points, which we'll assume will follow a power law KP^α, where P is the number of parameters and K and α are constants. A large number of ML theory results imply that the number of data points needed to reach a specified level of accuracy grows linearly with the number of parameters (i.e. α=1), which we can take as a weak prior. We can then update this with empirical evidence from papers. <@Scaling Laws for Neural Language Models@> suggests that for language models, data requirements scale as α=0.37 or as α=0.74, depending on what measure you look at. Meanwhile, [Deep Learning Scaling is Predictable, Empirically](https://arxiv.org/abs/1712.00409) suggests that α=1.39 for a wide variety of supervised learning problems (including language modeling). However, the former paper studies a more relevant setting: it includes regularization, and asks about the number of data points needed to reach a target accuracy, whereas the latter paper ignores regularization and asks about the minimum number of data points that the model _cannot_ overfit to. So overall the author puts more weight on the former paper and estimates a median of α=0.8, though with substantial uncertainty.

We also need to estimate how many epochs will be needed, i.e. how many times we train on any given data point. The author decides not to explicitly model this factor since it will likely be close to 1, and instead lumps in the uncertainty over the number of epochs with the uncertainty over the constant factor in the scaling law above. We can then look at language model runs to estimate a scaling law for them, for which the median scaling law predicts that we would need 1e13 data points for our 3e14 parameter model.

However, this has all been for supervised learning. It seems plausible that a transformative task would have to be trained using RL, where the model acts over a sequence of timesteps, and then receives (non-differentiable) feedback at the end of those timesteps. How would scaling laws apply in this setting? One simple assumption is to say that each rollout over the _effective horizon_ counts as one piece of "meaningful feedback" and so should count as a single data point. Here, the effective horizon is the minimum of the actual horizon and 1/(1-γ), where γ is the discount factor. We assume that the scaling law stays the same; if we instead try to estimate it from recent RL runs, it can change the results by about one order of magnitude.

So we now know we need to train a 3e14 parameter model with 1e13 data points for a transformative task. This gets us nearly all the way to the compute required with 2020 algorithms: we have a ~3e14 parameter model that takes ~1e16 FLOP per forward pass, that is trained on ~1e13 data points with each data point taking H timesteps, for a total of H * 1e29 FLOP. The author's distributions are instead centered at H * 1e30 FLOP; I suspect this is simply because the author was computing with distributions whereas I've been directly manipulating medians in this summary.

The last and most uncertain piece of information is the effective horizon of a transformative task. We could imagine something as low as 1 subjective second (for something like language modeling), or something as high as 1e9 subjective seconds (i.e. 32 subjective years), if we were to redo evolution, or train on a task like "do effective scientific R&D". The author splits this up into short, medium and long horizon neural net paths (corresponding to horizons of 1e0-1e3, 1e3-1e6, and 1e6-1e9 respectively), and invites readers to place their own weights on each of the possible paths.

There are many important considerations here: for example, if you think that the dominating cost will be generative modeling (GPT-3 style, but maybe also for images, video etc), then you would place more weight on short horizons. Conversely, if you think the hard challenge is to gain meta learning abilities, and that we probably need "data points" comparable to the time between generations in human evolution, then you would place more weight on longer horizons.

**Adding three more potential anchors**

We can now combine all these ingredients to get a forecast for when compute will be available to develop a transformative model! But not yet: we'll first add a few more possible "anchors" for the amount of computation needed for a transformative model. (All of the modeling so far has "anchored" the _inference time computation of a transformative model_ to the _inference time computation of the human brain_.)

First, we can anchor _parameter count of a transformative model_ to the _parameter count of the human genome_, which has far fewer "parameters" than the human brain. Specifically, we assume that all the scaling laws remain the same, but that a transformative model will only require 7.5e8 parameters (the amount of information in the human genome) rather than our previous estimate of ~1e15 parameters. This drastically reduces the amount of computation required, though it is still slightly above that of the short-horizon neural net, because the author assumed that the horizon for this path was somewhere between 1 and 32 years.

Second, we can anchor _training compute for a transformative model_ to the _compute used by the human brain over a lifetime_. As you might imagine, this leads to a much smaller estimate: the brain uses ~1e24 FLOP over 32 years of life, which is only 10x the amount used for AlphaStar, and even after adjusting upwards to account for man-made artifacts being worse than those made by evolution, the resulting model predicts a significant probability that we would already have been able to build a transformative model.

Finally, we can anchor _training compute for a transformative model_ to the _compute used by all animal brains over the course of evolution_. The basic assumption here is that our optimization algorithms and architectures are not much better than simply "redoing" natural selection from a very primitive starting point. This leads to an estimate of ~1e41 FLOP to train a transformative model, which is more than the long horizon neural net path (though not hugely more).

**Putting it all together**

So we now have six different paths: the three neural net anchors (short, medium and long horizon), the genome anchor, the lifetime anchor, and the evolution anchor. We can now assign weights to each of these paths, where each weight can be interpreted as the probability that that path is the _cheapest_ way to get a transformative model, as well as a final weight that describes the chance that none of the paths work out.

The long horizon neural net path can be thought of as a conservative "default" view: it could work out simply by training directly on examples of a long horizon task where each data point takes around a subjective year to generate. However, there are several reasons to think that researchers will be able to do better than this. As a result, the author assigns 20% to the short horizon neural net, 30% to the medium horizon neural net, and 15% to the long horizon neural net.

The lifetime anchor would suggest that we either already could get TAI, or are very close, which seems very unlikely given the lack of major economic applications of neural nets so far, and so gets assigned only 5%. The genome path gets 10%, the evolution anchor gets 10%, and the remaining 10% is assigned to none of the paths working out.

This predicts a **median of 2052** for the year in which some actor would be willing and able to train a single transformative model, with the full graphs shown below:

<GRAPHS HERE>

**How does this relate to TAI?**

Note that what we've modeled so far is the probability that by year Y we will have enough compute for the final training run of a transformative model. This is not the same thing as the probability of developing TAI. There are several reasons that TAI could be developed _later_ than the given prediction:

1. Compute isn't the only input required: we also need data, environments, human feedback, etc. While the author expects that these will not be the bottleneck, this is far from a certainty.
2. When thinking about any particular path and making it more concrete, a host of problems tends to show up that will need to be solved and may add extra time. Some examples include robustness, reliability, possible breakdown of the scaling laws, the need to generate lots of different kinds of data, etc.
3. AI research could stall, whether because of regulation, a global catastrophe, an AI winter, or something else.

However, there are also compelling reasons to expect TAI to arrive _earlier_:

1. We may develop TAI through some other cheaper route, such as a <@services model@>(@Reframing Superintelligence: Comprehensive AI Services as General Intelligence@).
2. Our forecasts apply to a "balanced" model that has a similar profile of abilities as a human. In practice, it will likely be easier and cheaper to build an "unbalanced" model that is superhuman in some domains and subhuman in others, that is nonetheless transformative.
3. The curves for several factors assume some maximum after which progress is not possible; in reality it is more likely that progress slows to some lower but non-zero growth rate.

In the near future, it seems likely that it would be harder to find cheaper routes (since there is less time to do the research), so we should probably assume that the probabilities are overestimates, and for similar reasons for later years the probabilities should be treated as underestimates.

For the median of 2052, the author guesses that these considerations roughly cancel out, and so rounds the median for development of TAI to **2050**. A sensitivity analysis concludes that 2040 is the "most aggressive plausible median", while the "most conservative plausible median" is 2080.


I found this paper very interesting and was quite surprised by its results. That said, I remain confused about what conclusions I should draw from it. The power law trend does seem to fit historical data very well, but the past 70 years are fit quite well by an [exponential trend](https://ourworldindata.org/grapher/world-gdp-over-the-last-two-millennia?yScale=log&time=1950..). Which one is relevant for predicting the future, if either, is quite unclear to me.

The theoretical model proposed makes more sense to me. If technology is responsible for the growth rate, then reinvesting production in technology will cause the growth rate to be faster. I'd be curious to see data on what fraction of GWP gets reinvested in improved technology and how that lines up with the other trends.

**Rohin’s opinion:** I enjoyed this post; it gave me a visceral sense for what hyperbolic models with noise look like (see the blog post for this, the summary doesn’t capture it). Overall, I think my takeaway is that the picture used in AI risk of explosive growth is in fact plausible, despite how crazy it initially sounds. Of course, it won’t literally diverge to infinity -- we will eventually hit some sort of limit on growth, even with “just” exponential growth -- but this limit could be quite far beyond what we have achieved so far. See also [this related post](https://www.lesswrong.com/posts/66FKFkWAugS8diydF/modelling-continuous-progress).

Modeling the Human Trajectory

This post analyzes the human trajectory from 10,000 BCE to the present and considers its implications for the future. The metric used for this is Gross World Product (GWP), the sum total of goods and services produced in the world over the course of a year.

Looking at GWP over this long stretch leads to a few interesting conclusions. First, until 1800, most people lived near subsistence levels. This means that growth in GWP was primarily driven by growth in population. Since then population growth has slowed and GWP per capita has increased, leading to our vastly improved quality of life today. Second, an exponential function does not fit the data well at all. In an exponential function, the time for GWP to double would be constant. Instead, GWP seems to be doubling faster, which is better fit by a power law. However, the conclusion of extrapolating this relationship forward is extremely rapid economic growth, approaching infinite GWP as we near the year 2047.

Next, Roodman creates a stochastic model in order to analyze not just the modal prediction, but also get the full distribution over how likely particular outcomes are. By fitting this to only past data, he analyzes how surprising each period of GWP was. This finds that the industrial revolution and the period after it was above the 90th percentile of the model's distribution, corresponding to surprisingly fast economic growth. Analogously, the past 30 years have seen anomalously lower growth, around the 25th percentile. This suggests that the model's stochasticity does not appropriately capture the real world -- while a good model can certainly be "surprised" by high or low growth during one period, it should probably not be _consistently_ surprised in the same direction, as happens here.

In addition to looking at the data empirically, he provides a theoretical model for how this accelerating growth can occur by generalizing a standard economic model. Typically, the economic model assumes technology is a fixed input or has a fixed rate of growth and does not allow for production to be reinvested in technological improvements. Once reinvestment is incorporated into the model, then the economic growth rate accelerates similarly to the historical data.


This measurement and forecasting work seems great; it constrains how we should expect future AI systems to look, and also improves our understanding of the impacts of AI, which probably helps us develop plans for deployment.

I was not very convinced by the reasoning about economic impact. I would believe that the economic impact of neural nets has grown exponentially, but it seems like we should be analyzing trends in machine learning (ML) overall, not just neural nets, and it seems much less likely to me that we see an exponential growth in that. Any time you see a new, better version of a previous technology (as with neural nets in relation to ML), you’re going to see an exponential trend as the new technology is adopted; this doesn’t mean that the exponential will keep going on and lead to transformative impact.

Danny Hernandez on forecasting and the drivers of AI progress

This podcast is a great introduction to the practice of forecasting and measurement in AI, and why it is important. I won't summarize everything in the podcast, but here are some of the points made.

Danny talks about the <@AI and Compute@> and <@AI and Efficiency@> work that he did at OpenAI. The former shows that the compute devoted to the largest-scale experiments has increased by a factor of 300,000 from 2012 to 2018, and the latter suggests that algorithms have been able to achieve similar performance with 25x less compute over the same time period (later updated to 44x from 2012 to 2019).

One thing I didn't realize earlier was that the 25x / 44x factor should be thought of as a loose lower bound: in other areas such as language modeling, the factor looks higher. But more importantly, the methodology used doesn't allow us to model the effects of an algorithm allowing us to do something we couldn't do before (which we could interpret as something we could do, but with way more compute). Possibly this algorithmic progress should be thought of as a 100x or even 1000x improvement in efficiency. Overall, Danny sees both algorithmic progress and increase in compute as pretty big factors in predicting how AI will go in the future.

Unfortunately, it's hard to draw strong implications from these measurements for the downstream things we care about -- should we think that AI progress is "slow" or "fast", or "linear" or "exponential", based on these results? It's important to be specific about the units you're using when thinking about such a question. Danny thinks the economic impact of AI is an important lens here. It seems to him that neural nets were having very little impact back in (say) 2008, but since then they have been having a lot more impact, e.g. by making ~15% of Google searches better (by using a new language model). To his eye, this trend looks exponential.

In any case, Danny thinks that this sort of rigorous measurement and forecasting work is important, because it provides concrete inputs that can allow decision makers to perform their job better. This is at least one reason why OpenAI's communication policy involves blog posts that deliberately target a wide audience: any decision maker can read these posts and get value out of them (unlike e.g. research papers).

This work is part of the broader work done by the Foresight team at OpenAI (which is hiring for research engineers): other work includes <@Scaling Laws for Neural Language Models@> and <@How AI Training Scales@>.

Danny thinks work in AI hardware is promising and under-explored by the community: it seems like it will be a particularly important field in the future, as it will drive some of the progress in increased compute, and as a result having some influence in the area could be quite helpful. For example, perhaps one could advocate for a <@windfall clause@>(@The Windfall Clause: Distributing the Benefits of AI@) at AI hardware companies.


The ideas I’ve listed in this summary seem quite compatible to me; I believe all of them to at least some degree (though perhaps not in the same way as the authors). I am not sure if either author would strongly disagree with any of the claims on this list. (Of course, this does not mean that they agree -- presumably there are some other claims that have not yet been made explicit on which they disagree.)

Discussion of "Takeoff Speeds"

This post focuses on the question of whether we should expect AI progress to look discontinuous or not. It seemed to me that the two participants were mostly talking past each other, and so I'll summarize their views separately and not discuss the parts where they were attempting to address each other's views.

Some ideas behind the "discontinuous" view:
1. When things are made up of a bunch of parts, you only get impact once all of the parts are working. So, if you have, say, 19 out of 20 parts done, there still won't be much impact, and then once you get the 20th part, then there is a huge impact, which looks like a discontinuity.
2. A continuous change in inputs can lead to a discontinuous change in outputs or impact. Continuously increasing the amount of fissile material leads to a discontinuous change from "inert-looking lump" to "nuclear explosion". Continuously scaling up a language model from GPT-2 to GPT-3 leads to many new capabilities, such as few-shot learning. A misaligned AI that is only capable of concealing 95% of its deceptive activities will not perform any such activities; it will only strike once it is scaled up to be capable of concealing 100% of its activities.
3. Fundamentally new approaches to a problem will often have prototypes which didn't have much impact. The difference is that they will scale much better, and so once they start having an impact this will look like a discontinuity in the rate of improvement on the problem.
4. The evolution from chimps to humans tells us that there is, within the space of possible mind designs, an area in which you can get from shallow, non-widely-generalizing cognition to deep, much-more-generalizing cognition, with only relatively small changes.
5. Our civilization tends to prevent people from doing things via bureaucracy and regulatory constraints, so even if there are productivity gains to be had from applications of non-scary AI, we probably won't see them; as a result we probably do not see GWP growth before the point where an AI can ignore bureaucracy and regulatory constraints, which makes it look discontinuous.

Some ideas behind the "continuous" view:
1. When people are optimizing hard in pursuit of a metric, then the metric tends to grow smoothly. While individual groups may find new ideas that improve the metric, those new ideas are unlikely to change the metric drastically more than previously observed changes in the metric.
2. A good heuristic for forecasting is to estimate (1) the returns to performance from additional effort, using historical data, and (2) the amount of effort currently being applied. These can then be combined to give a forecast.
3. How smooth and predictable the improvement is depends on how much effort is being put in. In terms of effort put in currently, coding assistants < machine translation < semiconductors, as a result we should expect semiconductor improvement to be smoother than machine translation improvement, which in turn will be smoother than coding assistant improvement.
4. In AI we will probably have crappy versions of economically useful systems before we have good versions of those systems. By the time we have good versions, people will be throwing lots of effort at the problem. For example, Codex is a crappy version of a coding assistant; such assistants will now improve over time in a somewhat smooth way.

There's further discussion on the differences between these views in a [subsequent post](https://www.alignmentforum.org/posts/7MCqRnZzvszsxgtJi/christiano-cotra-and-yudkowsky-on-ai-progress).


I'm surprised that mathematical conjectures take so long to be resolved, I would have expected a smaller half-life than _117 years_. I'm not sure if I should update strongly though -- it's possible that we only remember conjectures that took a long time to be resolved (though it's somewhat surprising then how well the data fits an exponential).

Various trends relevant to AI alignment

AI Impacts has published a few analyses of trends relevant to AI alignment (see links below).

Will we see a continuous or discontinuous takeoff? [Takeoff speeds](https://sideways-view.com/2018/02/24/takeoff-speeds/) operationalizes continuous takeoff (there called "slow takeoff") as: There will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles. AI impacts searched for [precedents for economic n-year doubling before 4n-year doubling](https://aiimpacts.org/precedents-for-economic-n-year-doubling-before-4n-year-doubling/), and found that this happened between 4,000 and 3,000 BC, and probably also between 10,000 and 4,000 BC. (Note this implies there was a 6000-year doubling before the 1000-year doubling, even though there wasn't a 4000-year doubling.)

How hard will it be to solve a crisply-stated problem of alignment? One way to get an outside view on the matter is to look at [resolutions of mathematical conjectures over time](https://aiimpacts.org/resolutions-of-mathematical-conjectures-over-time/). While there is obvious sampling bias in which conjectures are remembered as being important, the results could nonetheless be informative. They find that "the data is fit closely by an exponential function with a half-life of 117 years".

Since AI progress seems to be driven at least partially by <@compute@>(@AI and Compute@), forecasting trends in compute seems important to forecasting AI progress. [DRAM price per gigabyte](https://aiimpacts.org/trends-in-dram-price-per-gigabyte/) has fallen by about an order of magnitude every 5 years from 1957 to 2020, although since 2010, the data suggests more like 14 years for a drop by an order of magnitude. [Geekbench score per CPU price](https://aiimpacts.org/2019-recent-trends-in-geekbench-score-per-cpu-price/) has grown by around 16% a year from 2006-2020, which would yield an order of magnitude over 16 years. This is slower than other [CPU growth trends](https://aiimpacts.org/trends-in-the-cost-of-computing/), but this could be because Geekbench score is a markedly different metric.


I enjoyed this post a lot; the question of openness in AI research is an important one, that depends both on the scientific community and industry practice. The scientific community is extremely open, and the second paper especially seems to capture well the reason why. In contrast industry is often more secret (plausibly due to <@patents@>(@Who owns artificial intelligence? A preliminary analysis of corporate intellectual property strategies and why they matter@)). To the extent that we would like to change one community in the direction of the other, a good first step is to understand their incentives so that we can try to then change those incentives.

Openness Norms in AGI Development

This post summarizes two papers that provide models of why scientific research tends to be so open, and then applies it to the development of powerful AI systems. The [first](http://www.strevens.org/research/scistruc/Communicans.pdf) models science as a series of discoveries, in which the first academic group to reach a discovery gets all the credit for it. It shows that for a few different models of info-sharing, info-sharing helps everyone reach the discovery sooner, but doesn't change the probabilities for who makes the discovery first (called _race-clinching probabilities_): as a result, sharing all information is a better strategy than sharing none (and is easier to coordinate on than the possibly-better strategy of sharing just some information).

However, this theorem doesn't apply when info sharing compresses the discovery probabilities _unequally_ across actors: in this case, the race-clinching probabilities _do_ change, and the group whose probability would go down is instead incentivized to keep information secret (which then causes everyone else to keep their information secret). This could be good news: it suggests that actors are incentivized to share safety research (which probably doesn't affect race-clinching probabilities) while keeping capabilities research secret (thereby leading to longer timelines).

The [second paper](http://philsci-archive.pitt.edu/13452/1/Heesen%202017%20Communism%20and%20the%20Incentive%20to%20Share%20in%20Science%20preprint.pdf) assumes that scientists are competing to complete a k-stage project, and whenever they publish, they get credit for all the stages they completed that were not yet published by anyone else. It also assumes that earlier stages have a higher credit-to-difficulty ratio (where difficulty can be different across scientists). It finds that under this setting scientists are incentivized to publish whenever possible. For AI development, this seems not to be too relevant: we should expect that with powerful AI systems, most of the "credit" (profit) comes from the last few stages, where it is possible to deploy the AI system to earn money.


2019 trends in GPU price per FLOPS

This post analyzes the the trends in cost per FLOP for GPUs. There are a bunch of details in how to do this analysis, but they end up finding that this cost goes down by an order of magnitude over 17 years for single-precision FLOPS (halving time: 5 years), 10 years for half-precision FLOPS (halving time: 3 years), and 5 years for half-precision fused multiply-add FLOPS (halving time: 1.5 years). However, the latter two categories have become more popular in recent years with the rise of deep learning, so their low halving times might be because some of the single-precision hardware was converted to half-precision hardware, rather than fundamental technological improvements.


I found this post useful for clarifying exactly which axes of takeoff people disagree about, and also for introducing me to some notions of takeoff I hadn't seen before (though I haven't summarized them here).

Distinguishing definitions of takeoff

This post lists and explains several different "types" of AI takeoff that people talk about. Rather than summarize all the definitions (which would only be slightly shorter than the post itself), I'll try to name the main axes that definitions vary on (but as a result this is less of a summary and more of an analysis):

1. _Locality_. It could be the case that a single AI project far outpaces the rest of the world (e.g. via recursive self-improvement), or that there will never be extreme variations amongst AI projects across all tasks, in which case the "cognitive effort" will be distributed across multiple actors. This roughly corresponds to the Yudkowsky-Hanson FOOM debate, and the latter position also seems to be that taken by <@CAIS@>(@Reframing Superintelligence: Comprehensive AI Services as General Intelligence@).
2. _Wall clock time_. In [Superintelligence](https://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/<PHONE>), takeoffs are defined based on how long it takes for a human-level AI system to become strongly superintelligent, with "slow" being decades to centuries, and "fast" being minutes to days.
3. _GDP trend extrapolation_. Here, a continuation of an exponential trend would mean there is no takeoff (even if we some day get superintelligent AI), a hyperbolic trend where the doubling time of GDP decreases in a relatively continuous / gradual manner counts as continuous / gradual / slow takeoff, and a curve which shows a discontinuity would be a discontinuous / hard takeoff.


I’m looking forward to reading AI Impacts’ conclusions after completing these case studies. My impression from reading through these is that discontinuities happen, but rarely, and small discontinuities are more common than larger ones. However, I remain uncertain of a) how relevant each of these examples is to AI progress, and b) if I missed any key ways in which the examples differ from each other.

Incomplete case studies of discontinuous progress

Historic trends in technological progress

One key question in thinking about AGI deployment and which safety problems to focus on is whether technological progress will be *continuous* or *discontinuous*.  AI Impacts has researched the frequency of discontinuities in a number of case studies, that were selected on the possibility of having discontinuities. An example of a discontinuity in flight speed records would be the Fairey Delta 2 flight in 1956 which represented 19 years of progress at the previous trend. On the other hand, penicillin did not create a discontinuity of more than ten years in the number of deaths from syphilis in the US.  This post summarizes a number of those case studies.  As it is already a summary, I will just refer you to the post for more information.


This is a cool idea, and seems worth doing to me. I especially like that the survey would simply reveal problems by collecting two sources of information from people and checking their consistency with each other: there isn't any particular argument being made; you are simply showing inconsistency in people's own beliefs to them, if and only if such inconsistency exists. In practice, I'm sure there will be complications -- for example, perhaps the set of researchers taking the survey is different from the set of "others" whose actions and beliefs they are predicting -- but it still seems worth at least trying out.

Coordination Surveys: why we should survey to organize responsibilities, not just predictions

This post suggests that when surveying researchers about the future impact of their technology, we should specifically ask them about their beliefs about what actions other people will take, and what they personally are going to do, rather than just predicting total impact. (For example, we could ask how many people will invest in safety.) Then, by aggregating across survey respondents, we can see whether or not the researchers beliefs about what others will do match the empirical distribution of what researchers are planning to do. This can help mitigate the effect where everyone thinks that everyone else will deal with a problem, and the effect where everyone tries to solve a problem because they all think no one else is planning to solve it. Critch has offered to provide suggestions on including this methodology in any upcoming surveys; see the post for details.


As the post notes, it is important to remember that this is one of many plausible accounts for human success, but I find it reasonably compelling. It moves me closer to the camp of "there will likely be discontinuities in AI development", but not by much.

I'm more interested in what predictions about AI development we can make based on this model. I actually don't think that this suggests that AI development will need both social and asocial learning: it seems to me that in this model, the need for social learning arises because of the constraints on brain size and the limited lifetimes. Neither of these constraints apply to AI -- costs grow linearly with "brain size" (model capacity, maybe also training time) as opposed to superlinearly for human brains, and the AI need not age and die. So, with AI I expect that it would be better to optimize just for asocial learning, since you don't need to mimic the transmission across lifetimes that was needed for humans.

Musings on Cumulative Cultural Evolution and AI

A [recent paper](https://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.pcbi.<PHONE>&type=printable) develops a conceptual model that retrodicts human social learning. They assume that asocial learning allows you adapt to the current environment, while social learning allows you to copy the adaptations that other agents have learned. Both can be increased by making larger brains, at the cost of increased resource requirements. What conditions lead to very good social learning?

First, we need high transmission fidelity, so that social learning is effective. Second, we need some asocial learning, in order to bootstrap -- mimicking doesn't help if the people you're mimicking haven't learned anything in the first place. Third, to incentivize larger brains, the environment needs to be rich enough that additional knowledge is actually useful. Finally, we need low _reproductive skew_, that is, individuals that are more adapted to the environment should have only a slight advantage over those who are less adapted. (High reproductive skew would select too strongly for high asocial learning.) This predicts pair bonding rather than a polygynous mating structure.

This story cuts against the arguments in [Will AI See Sudden Progress?](https://www.lesswrong.com/posts/AJtfNyBsum6ZzWxKR/will-ai-see-sudden-progress) and [Takeoff speeds](https://sideways-view.com/2018/02/24/takeoff-speeds/): it seems like evolution "stumbled upon" high asocial and social learning and got a discontinuity in reproductive fitness of species. We should potentially also expect discontinuities in AI development.

We can also forecast the future of AI based on this story. Perhaps we need to be watching for the perfect combination of asocial and social learning techniques for AI, and once these components are in place, AI intelligence will develop very quickly and autonomously.


Signup form for AI Metaculus

Recently, forecasting platform Metaculus launched a new instance dedicated specifically to AI in order to get good answers for empirical questions (such as AGI timelines) that can help avoid situations like [info-cascades](https://www.lesswrong.com/posts/2uDBJWCksvzhDzHGf/understanding-information-cascades). While most questions don't have that many predictions, the current set of beta-users were invited based on forecasting track-record and AI domain-expertise, so the signal of the average forecast should be high.

Some interesting predictions include:
    - By end of 2019, will there be an agent at least as good as AlphaStar using non-controversial, human-like APM restrictions? _[mean: 58%, median: 66%, n = 26]_
    - When will there be a superhuman Starcraft II agent with no domain-specific hardcoded knowledge, trained using <=$10,000 of publicly available compute? _[50%: 2021 to 2037, with median 2026, n = 35]_

This forecast is supported by a [Guesstimate model](https://www.getguesstimate.com/models/12709), which estimates current and future sample efficiency of Starcraft II algorithms, based on current performance, algorithmic progress, and the generalization of Moore's law. For algorithmic progress, they look at the improvement in sample efficiency on Atari, and find a doubling time of roughly a year, via DQN → DDQN → Dueling DDQN → Prioritized DDQN → PPO → Rainbow → IMPALA.

Overall, there are 50+ questions, including on malicious use of AI, publishing norms, conference attendance, MIRI's research progress, the max compute doubling trend, OpenAI LP, nationalisation of AI labs, whether financial markets expect AGI, and more. You can sign-up to join [here](https://docs.google.com/forms/d/e/1FAIpQLSduBjn3W_MpHHjsKUEhzV6Krkup78ujE5-8bpNJ5HDE7GGnmA/viewform?usp=sf_link).


This quick read provides a natural, accessible analysis stemming from recent results, while staying self-aware (and informing readers) of potential improvements. The comments section too includes some interesting debates, e.g. about the Goodhart-ability of the Perplexity metric.

I personally felt these estimates were broadly in line with my own intuitions. I would go so far as to say that with the confluence of improved generation capabilities across text, speech/audio, video, as well as multimodal consistency and integration, virtually any kind of content we see ~10 years from now will be algorithmically generated and indistinguishable from the work of human professionals.

**Rohin's opinion:** I would generally adopt forecasts produced by this sort of method as my own, perhaps making them a bit longer as I expect the quickly growing compute trend to slow down. Note however that this is a forecast for human-level language models, not transformative AI; I would expect these to be quite different and would predict that transformative AI comes significantly later.

Forecasting progress in language models

This post aims to forecast when a "human-level language model" may be created. To build up to this, the author swiftly covers basic concepts from information theory and natural language processing such as entropy, N-gram models, modern LMs, and perplexity. Data for perplexity achieved from recent state-of-the-art models is collected and used to estimate - by linear regression - when we can expect to see future models score below certain entropy levels, approaching the hypothesised entropy for the English language.

These predictions range across the next 15 years, depending on which dataset, method, and entropy level is being solved for; there's an attached [python notebook](https://github.com/MatthewJBarnett/LanguageModelExtrapolation/blob/master/Human-level%20language%20models.ipynb) with these details for curious readers to further investigate. Preemptively disjunctive, the author concludes "either current trends will break down soon, or human-level language models will likely arrive in the next decade or two."


AXRP Episode 10 - AI’s Future and Impacts

This podcast goes over various strands of research from [AI Impacts](https://aiimpacts.org/), including lots of work that I either haven't covered or have covered only briefly in this newsletter:

**AI Impacts' methodology.** AI Impacts aims to advance the state of knowledge about AI and AI risk by recursively decomposing important high-level questions and claims into subquestions and subclaims, until reaching a question that can be relatively easily answered by gathering data. They generally aim to provide new facts or arguments that people haven't considered before, rather than arguing about how existing arguments should be interpreted or weighted.

**Timelines.** AI Impacts is perhaps most famous for its [survey of AI experts](https://arxiv.org/abs/1705.08807) on timelines till high-level machine intelligence (HLMI). The author's main takeaway is that people give very inconsistent answers and there are huge effects based on how you frame the question. For example:

1. If you estimate timelines by asking questions like "when will there be a 50% chance of HLMI", you'll get timelines a decade earlier than if you estimate by asking questions like "what is the chance of HLMI in 2030".
2. If you ask about when AI will outperform humans at all tasks, you get an estimate of ~2061, but if you ask when all occupations will be automated, you get an estimate of ~2136.
3. People whose undergraduate studies were in Asia estimated ~2046, while those in North America estimated ~2090.

The survey also found that the median probability of outcomes approximately as bad as extinction was 5%, which the author found surprisingly high for people working in the field.

**Takeoff speeds.** A common disagreement in the AI alignment community is whether there will be a discontinuous "jump" in capabilities at some point. AI Impacts has three lines of work investigating this topic:

1. Checking how long it typically takes to go from "amateur human" to "expert human". For example, it took about [3 years](https://aiimpacts.org/time-for-ai-to-cross-the-human-performance-range-in-imagenet-image-classification/) for image classification on ImageNet, [38 years](https://aiimpacts.org/time-for-ai-to-cross-the-human-range-in-english-draughts/) on checkers, [21 years](https://aiimpacts.org/time-for-ai-to-cross-the-human-range-in-starcraft/) for StarCraft, [30 years](https://aiimpacts.org/time-for-ai-to-cross-the-human-performance-range-in-go/) for Go, [30 years](https://aiimpacts.org/time-for-ai-to-cross-the-human-performance-range-in-chess/) for chess, and ~3000 years for clock stability (how well you can measure the passage of time).
2. Checking <@how often particular technologies have undergone discontinuities in the past@>(@Discontinuous progress in history: an update@). A (still uncertain) takeaway would be that discontinuities are the kind of thing that legitimately happen sometimes, but they don't happen so frequently that you should expect them, and you should have a pretty low prior on a discontinuity happening at some specific level of progress.
3. Detailing [arguments](https://aiimpacts.org/likelihood-of-discontinuous-progress-around-the-development-of-agi/) for and against discontinuous progress in AI.

**Arguments for AI risk, and counterarguments.** The author has also spent some time thinking about how strong the arguments for AI risk are, and has focused on a few areas:

1. Will superhuman AI systems actually be able to far outpace humans, such that they could take over the world? In particular, it seems like humans can use non-agentic tools to help keep up.
2. Maybe the AI systems we build won't have goals, and so the argument from [instrumental subgoals](https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf) won't apply.
3. Even if the AI systems do have goals, they may have human-compatible goals (especially since people will be explicitly trying to do this).
4. The AI systems may not destroy everything: for example, they might instead simply trade with humans, and use their own resources to pursue their goals while leaving humans alone.


AXRP Episode 7.5 - Forecasting Transformative AI from Biological Anchors

This podcast goes over the <@biological anchors framework@>(@Draft report on AI timelines@), as well as [three](https://arxiv.org/abs/1705.08807) <@other@>(@Modeling the Human Trajectory@) <@approaches@>(@Semi-informative priors over AI timelines@) to forecasting AI timelines and a post on <@aligning narrowly superhuman models@>(@The case for aligning narrowly superhuman models@). Since I've summarized these works before, I won't summarize this podcast; I recommend reading the past summaries instead. This podcast can help contextualize all of the work, adding in details that you wouldn't naturally see if you just read the reports or my summaries of them.

For example, I learned that there is a distinction between noise and effective horizon length. To the extent that your gradients are noisy, you can simply fix the problem by increasing your batch size (which can be done in parallel). However, the effective horizon length is measuring how many _sequential_ steps you have to take before you get feedback on how well you're doing. The two are separated in the bio anchors work because the author wanted to impose specific beliefs on the effective horizon length, but was happy to continue extrapolating from current examples for noise.


This is one of the few datasets that poses a challenge to today's Transformers, which makes it noteworthy. Despite its difficulty, accuracy is nontrivial and reliably increasing. In the appendix of a recent work, I and others show that performance on LogiQA is around [50%](https://arxiv.org/pdf/2103.03874.pdf#page=11&zoom=100,90,136) for an 11 billion parameter Transformer model. (Note that 50% is the models OOD generalization accuracy or transfer accuracy. The model was fine-tuned to answer some types of multiple-choice questions, but it did not fine-tune on LogiQA-style questions at all.) Consequently current models are already attaining nontrivial performance. Having done some LogiQA questions myself, I am surprised accuracy is already this high. Moreover, LogiQA accuracy is reliably increasing, as accuracy is increasing by about 15% for every order of magnitude increase in model size. If trends continue, a 1 trillion parameter model (10x the size of GPT-3) should be able to "solve" LogiQA.

I think LogiQA provides clear evidence that off-the-shelf Transformers are starting to acquire many "System 2" reasoning skills and can perform more than just snap judgments.

LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning

Jian Liu*, Leyang Cui*, Hanmeng Liu, Dandan Huang, Yile Wang, Yue Zhang

LogiQA is a benchmark that attempts to track models' understanding of logic and reason.
It consists of translated questions from the Civil Servants Examination of China, designed to test civil servant candidates.
The questions often require thought and deliberation. Two examples are as follows,

David knows Mr. Zhang's friend Jack, and Jack knows David's friend Ms. Lin. Everyone of them who knows Jack has a master's degree, and everyone of them who knows Ms. Lin is from Shanghai.
Who is from Shanghai and has a master's degree?
A. David.
B. Jack.
C. Mr. Zhang.
D. Ms. Lin.

Last night, Mark either went to play in the gym or visited his teacher Tony. If Mark drove last night, he didn't go to play in the gym. Mark would go visit his teacher Tony only if he and his teacher had an appointment. In fact, Mark had no appointment with his teacher Tony in advance.
Which is true based on the above statements?
A. Mark went to the gym with his teacher Tony last night.
B. Mark visited his teacher Tony last night.
C. Mark didn't drive last night.
D. Mark didn't go to the gym last night.

See Figure 2 of the paper for the answers to these two questions (I don't want to spoil the answers). In the paper, the authors show that RoBERTa models obtain around 36% accuracy, whereas human-level accuracy is around 86%.


This seems like a good quantification of what the outside view suggests for AI timelines. Unfortunately, I have never really spent much time figuring out how best to combine outside view and inside view evidence, because research generally requires you to think about a detailed, gearsy, inside-view model, and so outside views feel pretty irrelevant to me. (They’re obviously relevant to Open Phil, who have to make funding decisions based on AI timelines, and so really do benefit from having the better estimates of timelines.) So I will probably continue to act based on the <@bio anchors framework@>(@Draft report on AI timelines@).

This is also why I haven’t highlighted this particular piece, despite the content being excellent. I generally highlight things that would be valuable for technical alignment researchers to read; my guess is that timelines are actually _not_ that important for researchers to have good beliefs about (though inside-view models that predict timelines are important).

Some feedback on the report takes issue with the use of Laplace’s Rule because it models each “attempt” to make AGI as independent, which is obviously false. I’m not too worried about this; while the model might be obviously wrong, I doubt that a more sophisticated model would give very different results; most of the “oomph” is coming from the reference classes.

Semi-informative priors over AI timelines

This report aims to analyze outside view evidence for AI timelines. In this setting, "outside view" roughly means that we take into account when AI research started, and how its inputs (data, compute, researcher time) have changed over time, but nothing else. The report considers four potential reference classes from which an outside view can be formed.

For each reference class, we're going to use it to estimate how hard we would have thought AGI would be before we had tried to build AGI at all, and then we're going to update that probability based on the observation that we've tried for some amount of calendar time / researcher time / compute, and haven't yet gotten AGI. The report uses a simple generalization of Laplace's Rule to actually synthesize it all together; I'm not going to go into that here.

I found the reference classes most interesting and will summarize them here. Note that the author says that the main contribution is in the framework, and that the individual reference classes are much less well done (there are several suggestions on other reference classes to investigate in the future). With that caveat, in order of the weight assigned to each, the four references classes are:

1. **STEM goal:** AGI is a highly ambitious but feasible technology that a serious STEM field is explicitly trying to develop. Looking at other such examples, the author suggests putting between 5% and 50% on developing AGI in 50 years.
2. **Transformative technology:** AGI is a technological development that would have a transformative effect on the nature of work and society. While these have been incredibly rare, we might expect that their probability increases with more technological development, making it more likely to occur now. Based on this, the author favors an upper bound of 1% per year on AGI.
3. **Futurism goal:** AGI is a high-impact technology that a serious STEM field is trying to build in 2020. There are a lot of such technologies, but we probably shouldn't expect too many high-impact technologies to work out. The author suggests this should put it at below 1% per year.
4. **Math conjecture:** AGI is kinda sorta like a notable math conjecture. AI Impacts [investigated](https://aiimpacts.org/resolutions-of-mathematical-conjectures-over-time/) ([AN #97](https://mailchi.mp/a2b5efbcd3a7/an-97-are-there-historical-examples-of-large-robust-discontinuities)) the rate at which notable math conjectures are resolved, and their results imply 1/170 chance per year of a conjecture being resolved.

Aggregating these all together, the author favors assigning 0.1% - 1% per year at the beginning of AI research in 1956, with a point estimate of 0.3%. After updating on the fact that we don't yet have AGI, the framework gives 1.5% - 9% for AGI by 2036 and 7% - 33% for AGI by 2100.

We can also run the same analysis where you get a new "chance" to develop AGI every time you increase the researcher pool by a constant fraction. (This is almost like having a log uniform prior on how many researcher hours are needed to get AGI.) Since there have been a few large booms in AI, this gives somewhat higher probabilities than the previous method, getting to 2% - 15% for AGI by 2036. Doing the same thing for compute gets 2% - 22% for AGI by 2036.

A weighted aggregation of all of the methods together (with weights set by intuition) gives 1% - 18% for AGI by 2036, and 5% - 35% for AGI by 2100.


While reading this paper, I kept stopping to do the math problems because, well, I’m just easily distracted by math problems. But it did demonstrate one thing -- when the model gets it right, it can be really impressively right (at least in this one presumably cherry picked example). In one example from the paper (search for “ab5”), the ground-truth solution is horribly hacky, my housemate and I each separately got significantly more elegant solutions, but the model-generated solution was more elegant than either of our solutions. It’s a good example of how AI capabilities can be really lopsided -- no human would generate this good of an explanation if they were getting 6% accuracy overall.

Measuring Mathematical Problem Solving With the MATH Dataset

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, Jacob Steinhardt

We've seen <@GPT-3@>(@Language Models are Few-Shot Learners@) perform well on lots of downstream tasks. What about challenging high school math problems that require intuition to solve? The authors create the MATH dataset and demonstrate that this is in fact challenging for models: models currently get around 5-7%, even when pretraining on a dataset of math-relevant text and finetuning on the MATH training dataset. Note that the models have to get the answer exactly right: there is no partial credit.

Not only are current models not very good at the task, but also they scale poorly -- while there isn't much data to extrapolate from yet, a simple extrapolation suggests that models would need 10^35 parameters to achieve just 40% accuracy. (This is in contrast to easier tasks, which <@might be solved with some more scaling@>(@Extrapolating GPT-N performance@).) In contrast, in a simple study with university students, performance ranged between 40% and 90%, with the best human only making minor arithmetic errors. This suggests we'll need additional algorithmic improvements for better performance.

The authors also consider allowing language models to have "scratch space" to work on the problem: the models are prompted to generate a solution where they explain their work. They find that this actually _decreases_ accuracy, presumably because the poor generations at the beginning end up confusing the model.


I really liked this post: the problem is important, the approach to tackle it makes sense, and most importantly it’s very easy to follow the reasoning. I don’t think that directly substituting in the 1,000 number into the timelines calculation is the right approach; I think there are a few reasons (explained [here](https://www.alignmentforum.org/posts/yW3Tct2iyBMzYhTw7/how-does-bee-learning-compare-with-machine-learning?commentId=rcJuytMfdQNMb82rR), some of which were mentioned in the post) to think that the comparison was biased in favor of the ML models. I would instead wildly guess that this comparison suggests that a transformative model would use 20x less compute than a human, which still shortens timelines, probably to 2045 or so. (This is before incorporating uncertainty about the conclusions of the report as a whole.)

How does bee learning compare with machine learning?

The <@biological anchors approach@>(@Draft report on AI timelines@) to forecasting AI timelines estimates the compute needed for transformative AI based on the compute used by animals. One important parameter of the framework is needed to "bridge" between the two: if we find that an animal can do a specific task using X amount of compute, then what should we estimate as the amount of compute needed for an ML model to do the same task? This post aims to better estimate this parameter, by comparing few-shot image classification in bees to the same task in ML models. I won't go through the details here, but the upshot is that (after various approximations and judgment calls) ML models can reach the same performance as bees on few-shot image classification using 1,000 times less compute.

If we plug this parameter into the biological anchors framework (without changing any of the other parameters), the median year for transformative AI according to the model changes from 2050 to 2035, though the author advises only updating to (say) 2045 since the results of the investigation are so uncertain. The author also sees this as generally validating the biological anchors approach to forecasting timelines.


Following in the footsteps of <@AI and Efficiency@>(@AI and Efficiency@), here we have a paper showing exponential gains in sample efficiency in particular. I'm really glad someone did this analysis-- I think I'm surprised by how fast progress is, though as the paper notes it's unclear exactly how to relate historic improvements on fixed task performance to a sense of overall improvement in continuous control (though several of the main contributors listed in the appendix seem fairly general). I also really appreciate how thorough the full paper is in listing limitations to this work.

Since these papers are coming up in the same newsletter, I'll note the contrast between the data-unlimited domains explored in the scaling laws paper and the severely data-limited domain of real-world robotics emphasized in this paper. In robotics, it seems we are definitely still constrained by algorithmic progress that lets us train on fewer samples (or do better <@transfer from simulations@>(@Let's Discuss OpenAI's Rubik's Cube Result@)). Of course, maybe progress in data-unlimited domains will ultimately result in AIs that make algorithmic progress in data-limited domains faster than humans ever could.

Measuring Progress in Deep Reinforcement Learning Sample Efficiency

This paper measures historic increases in sample efficiency by looking at the number of samples needed to reach some fixed performance level on Atari games and virtual continuous control tasks. The authors find exponential progress in sample efficiency, with estimated doubling times of 10 to 18 months on Atari, 5 to 24 months on state-based continuous control, and 4 to 9 months on pixel-based continuous control, depending on the specific task and performance level. They find that these gains were mainly driven by improvements in off-policy and model-based deep RL learning approaches, as well as the use of auxiliary learning objectives to speed up representation learning, and not by model size improvements. The authors stress that their study is limited in studying only the published training curves for only three tasks, not accounting for the extent to which hyperparameter tuning may have been responsible for historic gains.


The examples of the questions are pretty interesting, and show that this really is a hard challenge: while experts in each subject would probably get very high scores, if we tested me on all of these subjects I don't think I would do very well. I like this method of evaluation because it gets a bit closer to what we care about: whether our models can capture enough domain knowledge that they can then be used widely for automation. Depending on your beliefs about how AI will progress, there might be too much of a focus on this generality -- maybe our models only need to understand “general reasoning” and then we can finetune them for specific domains.

Measuring Massive Multitask Language Understanding

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt

With the advent of large language models, there has been a shift to evaluating these models based on the knowledge they have acquired, i.e. evaluating their "common sense". However, with <@GPT-3@>(@Language Models are Few-Shot Learners@) models have reached approximately human performance even on these benchmarks. What should be next?

We've <@previously seen@>(@Aligning AI With Shared Human Values@) a benchmark that evaluates models based on their knowledge of ethics. This benchmark (with many of the same authors) goes further by testing models with multiple choice questions on a variety of subjects that humans need to learn. These are not easy: their 57 subjects include advanced topics like Professional Medicine, College Mathematics, and International Law.

All but the largest of the GPT-3 models do about as well as random chance (25%). However, the largest 175 billion parameter model does significantly better, reaching an average score of 43.9%. This performance is very lopsided: on US Foreign Policy it gets almost 70%, while on College Chemistry and Moral Scenarios it gets about 25% (i.e. still random chance). The authors note that GPT-3 tends to do worse on subjects that require calculations and thus speculate that it is harder for GPT-3 to acquire procedural knowledge compared to declarative knowledge. The authors also find that GPT-3 is very uncalibrated about its answers in the zero-shot setting, and becomes more calibrated (though still not very good) in the few-shot setting.

It isn't _necessary_ to have huge models in order to do better than chance: in fact, you can do better with a smaller model that is finetuned for question answering. In particular, the UnifiedQA system has an order of magnitude fewer parameters than GPT-3, but outperforms it with a score of 48.9% accuracy. This system was trained on other question answering datasets (but notably was not trained on the questions in this dataset, as this dataset is meant for evaluation rather than training). A small UnifiedQA model with only 60 million parameters (over 3 orders of magnitude smaller than GPT-3) can still do better than chance, achieving 29.3% on the dataset.


I really liked this post, although I haven't gotten a chance to get through the entire full-length report. I found the reasoning extremely legible and transparent, and there's no place where I disagree with Joseph's estimates or conclusions. See also [Import AI's summary](https://jack-clark.net/2020/09/14/import-ai-214-nvidias-40bn-arm-deal-a-new-57-subject-nlp-test-ai-for-plant-disease-detection/).

Full Report: How Much Computational Power Does It Take to Match the Human Brain?

How Much Computational Power It Takes to Match the Human Brain

In this blog post, Joseph Carlsmith gives a summary of his longer report estimating the number of floating point operations per second (FLOP/s) which would be _sufficient_ to perform any cognitive task that the human brain can perform. He considers four different methods of estimation.

Using *the mechanistic method*, he estimates the FLOP/s required to model the brain's low-level mechanisms at a level of detail adequate to replicate human task-performance. He does this by estimating that ~1e13 - 1e17 FLOP/s is enough to replicate what he calls "standard neuron signaling" — neurons signaling to each other via using electrical impulses (at chemical synapses) — and learning in the brain, and arguing that including the brain's other signaling processes would not meaningfully increase these numbers. He also suggests that various considerations point weakly to the adequacy of smaller budgets.

Using *the functional method*, he identifies a portion of the brain whose function we can approximate with computers, and then scales up to FLOP/s estimates for the entire brain. One way to do this is by scaling up models of the human retina: Hans Moravec's estimates for the FLOP/s of the human retina imply 1e12 - 1e15 FLOP/s for the entire brain, while recent deep neural networks that predict retina cell firing patterns imply 1e16 - 1e20 FLOP/s.

Another way to use the functional method is to assume that current image classification networks with known FLOP/s requirements do some fraction of the computation of the human visual cortex, adjusting for the increase in FLOP/s necessary to reach robust human-level classification performance. Assuming somewhat arbitrarily that 0.3% to 10% of what the visual cortex does is image classification, and that the EfficientNet-B2 image classifier would require a 10x to 1000x increase in frequency to reach fully human-level image classification, he gets 1e13 - 3e17 implied FLOP/s to run the entire brain. Joseph holds the estimates from this method very lightly, though he thinks that they weakly suggest that the 1e13 - 1e17 FLOP/s estimates from the mechanistic method are not radically too low.

Using *the limit method*, Joseph uses the brain's energy budget, together with physical limits set by Landauer's principle, which specifies the minimum energy cost of erasing bits, to upper-bound required FLOP/s to ~7e21. He notes that this relies on arguments about how many bits the brain erases per FLOP, which he and various experts agree is very likely to be > 1 based on arguments about algorithmic bit erasures and the brain's energy dissipation.

Lastly, Joseph briefly describes *the communication method*, which uses the communication bandwidth in the brain as evidence about its computational capacity. Joseph thinks this method faces a number of issues, but some extremely preliminary estimates suggest 1e14 FLOP/s based on comparing the brain to a V100 GPU, and 1e16 - 3e17 FLOP/s based on estimating the communication capabilities of brains in traversed edges per second (TEPS), a metric normally used for computers, and then converting to FLOP/s using the TEPS to FLOP/s ratio in supercomputers.

Overall, Joseph thinks it is more likely than not that 1e15 FLOP/s is enough to perform tasks as well as the human brain (given the right software, which may be very hard to create). And he thinks it's unlikely (<10%) that more than 1e21 FLOP/s is required. For reference, an NVIDIA V100 GPU performs up to 1e14 FLOP/s (although FLOP/s is not the only metric which differentiates two computational systems.)


I’m glad to know that the empirical support for the HGH seems mostly limited to the modern era, and may be weakly disconfirmed by data from the pre-modern era. I’m not entirely sure how I should update -- it seems that both hypotheses would be consistent with future accelerating growth, though HGH predicts it more strongly. It also seems plausible to me that we should still assign more credence to HGH because of its theoretical support and relative simplicity -- it doesn’t seem like there is strong evidence suggesting that HGH is false, just that the empirical evidence for it is weaker than we might have thought. See also [Paul Christiano’s response](https://forum.effectivealtruism.org/posts/CWFn9qAKsRibpCGq8/does-economic-history-point-toward-a-singularity?commentId=j9BymthAthZQ6dnGp).

Does Economic History Point Toward a Singularity?

One important question for the long-term future is whether we can expect accelerating growth in total GDP in the near future (see e.g. this <@recent report@>(@Modeling the Human Trajectory@)). For AI alignment in particular, the answer to this question could have a significant impact on AI timelines: if some arguments suggested that it would be very unlikely for us to have accelerating growth soon, we should probably be more skeptical that we will develop transformative AI soon.

So far, the case for accelerating growth relies on one main argument that the author calls the _Hyperbolic Growth Hypothesis_ (HGH). This hypothesis posits that the growth _rate_ rises in tandem with the population size (intuitively, a higher population means more ideas for technological progress which means higher growth rates). This document explores the _empirical_ support for this hypothesis.

I'll skip the messy empirical details and jump straight to the conclusion: while the author agrees that growth rates of total GDP (rather than GDP per capita) have been increasing since the Industrial Revolution up till a few decades ago, he does not see much support for the HGH prior to the modern era. The data seems very noisy and hard to interpret, and even when using this noisy data it seems that models with constant growth rates fit the pre-modern era better than hyperbolic models. Thus, we should be uncertain between the HGH and the hypothesis that the industrial revolution triggered a one-off transition to increasing growth rates that have now stabilized.


Forecasting Thread: AI Timelines

This post collects forecasts of timelines until human-level AGI. One of the authors reflects on lessons learned [here](https://www.lesswrong.com/posts/6LJjzTo5xEBui8PqE/reflections-on-ai-timelines-forecasting-thread).


I highly recommend this paper as a starting point for anyone who wants to get started on AI forecasting research. Identifying an interesting research question is typically one of the parts of the research process where expert feedback and mentorship helps the most, and the expert suggestions aggregated here seem quite valuable for that.

I also agree with the recommendation for literature reviews. In order for AI safety research to have its desired impact, it eventually needs to be communicated to decision makers, including researchers, company executives, and government leaders. Literature reviews are a valuable academic method for doing this, but I am also excited by more creative ways to communicate these research topics like this newsletter or [these](https://www.youtube.com/channel/UCLB7AzTwc6VFZrBsO2ucBMg) videos.

Forecasting AI Progress: A Research Agenda

Ross Gruetzemacher, Florian Dorner, Niko Bernaola-Alvarez, Charlie Giattino, David Manheim

This paper develops a research agenda using the Delphi Process. The Delphi process consists of 4 steps:
1.  Ask experts a series of open-ended questions to identify interesting research questions and methods.
2.  Authors summarize and aggregate results and send back to experts.
3.  The experts comment on and discuss the results.
4.  The experts score the research questions and methods on importance and feasibility.

This process yields a large list of questions and methods. A few that I am personally interested in are:
-   What are the most useful indicators (e.g. compute, talent, economic impact) of AI progress?
-   How effective is long-term technological forecasting and how can we best validate near- and mid-term forecasts?
-   How do we utilize forecasts to inform decision makers and develop interventions?
-   What are the most likely scenarios for the development of TAI?
There is already an existing body of work on many of these questions, so their strongest recommendation for future work is for literature reviews.


The updated posterior seems roughly right to me -- looking at the reasoning of the prize-winning comment, it seems like a $1 trillion training run in 2030 would be about 10^11 petaflop/s-days, which seems like the far end of the spectrum. The posterior assigns about 20% to it being even larger than this, which seems too high to me, but the numbers above do assume a “business-as-usual” world, and if you assign a significant probability to getting AGI before 2030, then you probably should have a non-trivial probability assigned to extreme outcomes.

Amplified forecasting: What will Buck's informed prediction of compute used in the largest ML training run before 2030 be?

[Ought](https://ought.org/) has recently run experiments on how to amplify expert reasoning, to produce better answers than a time-limited expert could produce themselves. This experiment centers on the question of how much compute will be used in the largest ML training run before 2030. Rather than predict the actual answer, participants provided evidence and predicted what Buck's posterior would be after reading through the comments and evidence.

Buck's quick [prior](https://elicit.ought.org/builder/aFElCFp8E) was an extrapolation of the trend identified in <@AI and Compute@>, and suggested a median of around 10^13 petaflop/s-days. Commenters pointed out that the existing trend relied on a huge growth rate in the amount of money spent on compute, that seemed to lead to implausible amounts of money by 2030 (a point previously made <@here@>(@Interpreting AI Compute Trends@)). Buck's updated [posterior](https://elicit.ought.org/builder/2yV4pA-Wc) has a median of around 10^9 petaflop/s-days, with a mode of around 10^8 petaflop/s-days (estimated to be 3,600 times larger than AlphaStar).


Competition: Amplify Rohin’s Prediction on AGI researchers & Safety Concerns

Ought ran a second competition to amplify my forecast on a question of my choosing. I ended up asking "When will a majority of top AGI researchers agree with safety concerns?", specified in more detail in the post. Notably, I require the researchers to understand the concerns that I think the AI safety community has converged on, as opposed to simply saying that they are concerned about safety. I chose the question because it seems like any plan to mitigate AI risk probably requires consensus amongst at least AI researchers that AI risk is a real concern. (More details in [this comment](https://www.alignmentforum.org/posts/Azqmzp5JoXJihMcr4/competition-amplify-rohin-s-prediction-on-agi-researchers?commentId=AQvjXqk9KwCoh8Y6X).)

My model is that this will be caused primarily by compelling demonstrations of risk (e.g. warning shots), and these will be easier to do as AI systems become more capable. So it depends a lot on models of progress; I used a median of 20 years until "human-level reasoning". Given that we'll probably get compelling demonstrations before then, but also it can take time for consensus to build, I also estimated a median of around 20 years for consensus on safety concerns, and then made a vaguely lognormal [prior](https://elicit.ought.org/builder/YYPXqX_eC) with that median. (I also estimated a 25% chance that it never happens, e.g. due to a global catastrophe that prevents more AI research, or because we build an AGI and see it isn't risky, etc.)

Most of the commenters were more optimistic than I was, thinking that we might already have consensus (given that I restricted it to AGI researchers), which led to several small updates towards optimism. One commenter pointed out that in practice, concern about AI risk tends to be concentrated amongst RL researchers, which are a tiny fraction of all AI researchers, and probably a tiny fraction of AGI researchers as well (given that natural language processing and representation learning seem likely to be relevant to AGI). This led to a single medium-sized update towards pessimism. Overall these washed out, and my [posterior](https://elicit.ought.org/builder/rBxYYzM-f) was a bit more optimistic than my prior, and was higher entropy (i.e. more uncertain).


The general point of this post seems clearly correct and worth pointing out. I’m looking forward to the work we’ll see in the future figuring out how to apply these broad and general methods to real tasks in a reliable way.

Alignment As A Bottleneck To Usefulness Of GPT-3

Currently, [many people](https://twitter.com/xuenay/status/1283312640199196673) are trying to figure out how to prompt GPT-3 into doing what they want -- in other words, how to align GPT-3 with their desires. GPT-3 may be capable of the task, but that doesn't mean it will do it ([potential example](https://www.lesswrong.com/posts/H9knnv8BWGKj6dZim/usd1000-bounty-for-openai-to-show-whether-gpt3-was)). This suggests that alignment will soon be a bottleneck on our ability to get value from large language models.

Certainly GPT-3 isn't perfectly capable yet. The author thinks that in the immediate future the major bottleneck will still be its capability, but we have a clear story for how to improve its capabilities: just scale up the model and data even more. Alignment on the other hand is much harder: we don't know how to <@translate@>(@Alignment as Translation@) the tasks we want into a format that will cause GPT-3 to "try" to accomplish that task.

As a result, in the future we might expect a lot more work to go into prompt design (or whatever becomes the next way to direct language models at specific tasks). In addition, once GPT is better than humans (at least in some domains), alignment in those domains will be particularly difficult, as it is unclear how you would get a system trained to mimic humans <@to do better than humans@>(@The easy goal inference problem is still hard@).


I think this is an important question for AI timelines, and the plausibility of “hard paths” is one of the central reasons that my timelines are longer than others who work on deep learning-based AGI. However, <@GPT-3@>(@Language Models are Few-Shot Learners@) demonstrates quite a lot of generality, so recently I’ve started putting more weight on “actually, designing the environment won’t be too hard”, which has correspondingly shortened my timelines.

Environments as a bottleneck in AGI development

Models built using deep learning are a function of the learning algorithm, the architecture, and the task / environment / dataset. While a lot of effort is spent on analyzing learning algorithms and architectures, not much is spent on the environment. This post asks how important it is to design a good environment in order to build AGI.

It considers two possibilities: the "easy paths hypothesis" that many environments would incentivize AGI, and the "hard paths hypothesis" that such environments are rare. (Note that "hard paths" can be true even if an AGI would be optimal for most environments: if AGI would be optimal, but there is no path in the loss landscape to AGI that is steeper than other paths in the loss landscape, then we probably wouldn't find AGI in that environment.)

The main argument for "hard paths" is to look at the history of AI research, where we often trained agents on tasks that were "hallmarks of intelligence" (like chess) and then found that the resulting systems were narrowly good at the particular task, but were not generally intelligent. You might think that it can't be too hard, since our environment led to the creation of general intelligence (us), but this is subject to anthropic bias: only worlds with general intelligence would ask whether environments incentivize general intelligence, so they will always observe that their environment is an example that incentivizes general intelligence. It can serve as a proof of existence, but not as an indicator that it is particularly likely.


While there is a lot of uncertainty over timelines, I don’t think under 5% chance of AGI in the next 20 years is very plausible. Claims of the form “neural nets are fundamentally incapable of X” are almost always false: recurrent neural nets are Turing-complete, and so can encode arbitrary computation. Thus, the real question is whether we can _find_ the parameterization that would correspond to e.g. causal reasoning.

I’m quite sympathetic to the claim that this would be very hard to do: neural nets find the simplest way of doing the task, which usually does not involve general reasoning. Nonetheless, it seems like by having more and more complex and diverse tasks, you can get closer to general reasoning, with <@GPT-3@>(@Language Models are Few-Shot Learners@) being the latest example in this trend. Of course, even then it may be hard to reach AGI due to limits on compute. I’m not claiming that we already have general reasoning, nor that we necessarily will get it soon: just that it seems like we can’t rule out the possibility that general reasoning does happen soon, at least not without a relatively sophisticated analysis of how much compute we can expect in the future and some lower bound on how much we would need for AGI-via-diversity-of-tasks.

Reasons you might think human level AI soon is unlikely

There is a lot of disagreement about AI timelines, that can be quite decision-relevant. In particular, if we were convinced that there was a < 5% chance of AGI in the next 20 years, that could change the field's overall strategy significantly: for example, we might focus more on movement building, less on empirical research, and more on MIRI's agent foundations research. This talk doesn't decisively answer this question, but discusses three different sources of evidence one might have for this position: the results of expert surveys, trends in compute, and arguments that current methods are insufficient for AGI.

Expert surveys usually suggest a significantly higher than 5% chance of AGI in 20 years, but this is quite sensitive to the specific framing of the question, and so it's not clear how informative this is. If we instead ask experts what percentage of their field has been solved during their tenure and extrapolate to 100%, the extrapolations for junior researchers tend to be optimistic (decades), whereas those of senior researchers are pessimistic (centuries).

Meanwhile, the <@amount spent on compute@>(@AI and Compute@) has been increasing rapidly. At the estimated trend, it would hit $200 billion in 2022, which is within reach of large governments, but would presumably have to slow down at that point, potentially causing overall AI progress to slow. Better price performance (how many flops you can buy per dollar) might compensate for this, but hasn't been growing at comparable rates historically.

Another argument is that most of our effort is now going into deep learning, and methods that depend primarily on deep learning are insufficient for AGI, e.g. because they can't use human priors, or can't do causal reasoning, etc. Asya doesn't try to evaluate these arguments, and so doesn't have a specific takeaway.


Surveys on fractional progress towards HLAI

One way to predict AGI timelines is to ask experts to estimate what fraction of progress has been made over a fixed number of years, then to extrapolate to the full 100% of progress. Doing this with the [2016 expert survey](https://aiimpacts.org/2016-expert-survey-on-progress-in-ai/) yields an estimate of 2056 (36 years from now), while doing this with Robin Hanson's informal ~15-expert survey gives 2392 (372 years from now). Part of the reason for the discrepancy is that Hanson only asked experts who had been in their field for at least 20 years; restricting to just these respondents in the 2016 survey yields an estimate of 2162 (142 years from now).


Atari early

With DeepMind's Agent57 (summarized below), it seems that it is feasible to outperform professional game testers on all Atari games using no game-specific knowledge. Interestingly, in a 2016 survey, the median response put a small chance (10%) on this being feasible by 2021, and a medium chance (50%) of being feasible by 2026.


While I expected that the public would expect HLMI sooner than AI experts, I was surprised that AI and robotics experts agreed so closely -- I would have thought that robotics experts would have longer timelines.

Walsh 2017 Survey

In this survey, AI experts, robotics experts, and the public estimated a 50% chance of high-level machine intelligence (HLMI) by 2061, 2065, and 2039 respectively. The post presents other similar data from the survey.


While I haven’t watched enough of the games to have a strong opinion on whether AlphaStar is empirically reacting to its opponents' strategies, I agree with Paul Christiano’s [comment](https://www.lesswrong.com/posts/SvhzEQkwFGNTy6CsN/alphastar-impressive-for-rl-progress-not-for-agi-progress#kRpwqPPjcGEbEhXHA) that in principle causal reasoning is just one type of computation that should be learnable.

This discussion also highlights the need for interpretability tools for deep RL so that we can have more informed discussions on exactly how and why strategies are decided on.

AlphaStar: Impressive for RL progress, not for AGI progress

This post argues that while it is impressive that AlphaStar can build up concepts complex enough to win at StarCraft, it is not actually developing reactive strategies. Rather than scouting what the opponent is doing and developing a new strategy based on that, AlphaStar just executes one of a predetermined set of strategies. This is because AlphaStar does not use causal reasoning, and that keeps it from beating any of the top players.