The updated posterior seems roughly right to me -- looking at the reasoning of the prize-winning comment, it seems like a $1 trillion training run in 2030 would be about 10^11 petaflop/s-days, which seems like the far end of the spectrum. The posterior assigns about 20% to it being even larger than this, which seems too high to me, but the numbers above do assume a “business-as-usual” world, and if you assign a significant probability to getting AGI before 2030, then you probably should have a non-trivial probability assigned to extreme outcomes.

Amplified forecasting: What will Buck's informed prediction of compute used in the largest ML training run before 2030 be?

[Ought](https://ought.org/) has recently run experiments on how to amplify expert reasoning, to produce better answers than a time-limited expert could produce themselves. This experiment centers on the question of how much compute will be used in the largest ML training run before 2030. Rather than predict the actual answer, participants provided evidence and predicted what Buck’s posterior would be after reading through the comments and evidence.

Buck’s quick [prior](https://elicit.ought.org/builder/aFElCFp8E) was an extrapolation of the trend identified in <@AI and Compute@>, and suggested a median of around 10^13 petaflop/s-days. Commenters pointed out that the existing trend relied on a huge growth rate in the amount of money spent on compute, that seemed to lead to implausible amounts of money by 2030 (a point previously made <@here@>(@Interpreting AI Compute Trends@)). Buck’s updated [posterior](https://elicit.ought.org/builder/2yV4pA-Wc) has a median of around 10^9 petaflop/s-days, with a mode of around 10^8 petaflop/s-days (estimated to be 3,600 times larger than AlphaStar).


Competition: Amplify Rohin’s Prediction on AGI researchers & Safety Concerns

Ought ran a second competition to amplify my forecast on a question of my choosing. I ended up asking “When will a majority of top AGI researchers agree with safety concerns?”, specified in more detail in the post. Notably, I require the researchers to understand the concerns that I think the AI safety community has converged on, as opposed to simply saying that they are concerned about safety. I chose the question because it seems like any plan to mitigate AI risk probably requires consensus amongst at least AI researchers that AI risk is a real concern. (More details in [this comment](https://www.alignmentforum.org/posts/Azqmzp5JoXJihMcr4/competition-amplify-rohin-s-prediction-on-agi-researchers?commentId=AQvjXqk9KwCoh8Y6X).)

My model is that this will be caused primarily by compelling demonstrations of risk (e.g. warning shots), and these will be easier to do as AI systems become more capable. So it depends a lot on models of progress; I used a median of 20 years until “human-level reasoning”. Given that we’ll probably get compelling demonstrations before then, but also it can take time for consensus to build, I also estimated a median of around 20 years for consensus on safety concerns, and then made a vaguely lognormal [prior](https://elicit.ought.org/builder/YYPXqX_eC) with that median. (I also estimated a 25% chance that it never happens, e.g. due to a global catastrophe that prevents more AI research, or because we build an AGI and see it isn’t risky, etc.)

Most of the commenters were more optimistic than I was, thinking that we might already have consensus (given that I restricted it to AGI researchers), which led to several small updates towards optimism. One commenter pointed out that in practice, concern about AI risk tends to be concentrated amongst RL researchers, which are a tiny fraction of all AI researchers, and probably a tiny fraction of AGI researchers as well (given that natural language processing and representation learning seem likely to be relevant to AGI). This led to a single medium-sized update towards pessimism. Overall these washed out, and my [posterior](https://elicit.ought.org/builder/rBxYYzM-f) was a bit more optimistic than my prior, and was higher entropy (i.e. more uncertain).


The general point of this post seems clearly correct and worth pointing out. I’m looking forward to the work we’ll see in the future figuring out how to apply these broad and general methods to real tasks in a reliable way.

Alignment As A Bottleneck To Usefulness Of GPT-3

Currently, [many people](https://twitter.com/xuenay/status/1283312640199196673) are trying to figure out how to prompt GPT-3 into doing what they want -- in other words, how to align GPT-3 with their desires. GPT-3 may be capable of the task, but that doesn’t mean it will do it ([potential example](https://www.lesswrong.com/posts/H9knnv8BWGKj6dZim/usd1000-bounty-for-openai-to-show-whether-gpt3-was)). This suggests that alignment will soon be a bottleneck on our ability to get value from large language models.

Certainly GPT-3 isn’t perfectly capable yet. The author thinks that in the immediate future the major bottleneck will still be its capability, but we have a clear story for how to improve its capabilities: just scale up the model and data even more. Alignment on the other hand is much harder: we don’t know how to <@translate@>(@Alignment as Translation@) the tasks we want into a format that will cause GPT-3 to “try” to accomplish that task.

As a result, in the future we might expect a lot more work to go into prompt design (or whatever becomes the next way to direct language models at specific tasks). In addition, once GPT is better than humans (at least in some domains), alignment in those domains will be particularly difficult, as it is unclear how you would get a system trained to mimic humans <@to do better than humans@>(@The easy goal inference problem is still hard@).


I think this is an important question for AI timelines, and the plausibility of “hard paths” is one of the central reasons that my timelines are longer than others who work on deep learning-based AGI. However, <@GPT-3@>(@Language Models are Few-Shot Learners@) demonstrates quite a lot of generality, so recently I’ve started putting more weight on “actually, designing the environment won’t be too hard”, which has correspondingly shortened my timelines.

Environments as a bottleneck in AGI development

Models built using deep learning are a function of the learning algorithm, the architecture, and the task / environment / dataset. While a lot of effort is spent on analyzing learning algorithms and architectures, not much is spent on the environment. This post asks how important it is to design a good environment in order to build AGI.

It considers two possibilities: the “easy paths hypothesis" that many environments would incentivize AGI, and the “hard paths hypothesis” that such environments are rare. (Note that “hard paths” can be true even if an AGI would be optimal for most environments: if AGI would be optimal, but there is no path in the loss landscape to AGI that is steeper than other paths in the loss landscape, then we probably wouldn’t find AGI in that environment.)

The main argument for “hard paths” is to look at the history of AI research, where we often trained agents on tasks that were “hallmarks of intelligence” (like chess) and then found that the resulting systems were narrowly good at the particular task, but were not generally intelligent. You might think that it can’t be too hard, since our environment led to the creation of general intelligence (us), but this is subject to anthropic bias: only worlds with general intelligence would ask whether environments incentivize general intelligence, so they will always observe that their environment is an example that incentivizes general intelligence. It can serve as a proof of existence, but not as an indicator that it is particularly likely.


While there is a lot of uncertainty over timelines, I don’t think under 5% chance of AGI in the next 20 years is very plausible. Claims of the form “neural nets are fundamentally incapable of X” are almost always false: recurrent neural nets are Turing-complete, and so can encode arbitrary computation. Thus, the real question is whether we can _find_ the parameterization that would correspond to e.g. causal reasoning.

I’m quite sympathetic to the claim that this would be very hard to do: neural nets find the simplest way of doing the task, which usually does not involve general reasoning. Nonetheless, it seems like by having more and more complex and diverse tasks, you can get closer to general reasoning, with <@GPT-3@>(@Language Models are Few-Shot Learners@) being the latest example in this trend. Of course, even then it may be hard to reach AGI due to limits on compute. I’m not claiming that we already have general reasoning, nor that we necessarily will get it soon: just that it seems like we can’t rule out the possibility that general reasoning does happen soon, at least not without a relatively sophisticated analysis of how much compute we can expect in the future and some lower bound on how much we would need for AGI-via-diversity-of-tasks.

Reasons you might think human level AI soon is unlikely

There is a lot of disagreement about AI timelines, that can be quite decision-relevant. In particular, if we were convinced that there was a < 5% chance of AGI in the next 20 years, that could change the field’s overall strategy significantly: for example, we might focus more on movement building, less on empirical research, and more on MIRI’s agent foundations research. This talk doesn't decisively answer this question, but discusses three different sources of evidence one might have for this position: the results of expert surveys, trends in compute, and arguments that current methods are insufficient for AGI.

Expert surveys usually suggest a significantly higher than 5% chance of AGI in 20 years, but this is quite sensitive to the specific framing of the question, and so it’s not clear how informative this is. If we instead ask experts what percentage of their field has been solved during their tenure and extrapolate to 100%, the extrapolations for junior researchers tend to be optimistic (decades), whereas those of senior researchers are pessimistic (centuries).

Meanwhile, the <@amount spent on compute@>(@AI and Compute@) has been increasing rapidly. At the estimated trend, it would hit $200 billion in 2022, which is within reach of large governments, but would presumably have to slow down at that point, potentially causing overall AI progress to slow. Better price performance (how many flops you can buy per dollar) might compensate for this, but hasn't been growing at comparable rates historically.

Another argument is that most of our effort is now going into deep learning, and methods that depend primarily on deep learning are insufficient for AGI, e.g. because they can’t use human priors, or can’t do causal reasoning, etc. Asya doesn’t try to evaluate these arguments, and so doesn’t have a specific takeaway.


FLI Podcast: On Superforecasting


Surveys on fractional progress towards HLAI

One way to predict AGI timelines is to ask experts to estimate what fraction of progress has been made over a fixed number of years, then to extrapolate to the full 100% of progress. Doing this with the [2016 expert survey](https://aiimpacts.org/2016-expert-survey-on-progress-in-ai/) yields an estimate of 2056 (36 years from now), while doing this with Robin Hanson's informal ~15-expert survey gives 2392 (372 years from now). Part of the reason for the discrepancy is that Hanson only asked experts who had been in their field for at least 20 years; restricting to just these respondents in the 2016 survey yields an estimate of 2162 (142 years from now).


Atari early

With DeepMind's Agent57 (summarized below), it seems that it is feasible to outperform professional game testers on all Atari games using no game-specific knowledge. Interestingly, in a 2016 survey, the median response put a small chance (10%) on this being feasible by 2021, and a medium chance (50%) of being feasible by 2026.


While I expected that the public would expect HLMI sooner than AI experts, I was surprised that AI and robotics experts agreed so closely -- I would have thought that robotics experts would have longer timelines.

Walsh 2017 Survey

In this survey, AI experts, robotics experts, and the public estimated a 50% chance of high-level machine intelligence (HLMI) by 2061, 2065, and 2039 respectively. The post presents other similar data from the survey.


While I haven’t watched enough of the games to have a strong opinion on whether AlphaStar is empirically reacting to its opponents' strategies, I agree with Paul Christiano’s [comment](https://www.lesswrong.com/posts/SvhzEQkwFGNTy6CsN/alphastar-impressive-for-rl-progress-not-for-agi-progress#kRpwqPPjcGEbEhXHA) that in principle causal reasoning is just one type of computation that should be learnable.

This discussion also highlights the need for interpretability tools for deep RL so that we can have more informed discussions on exactly how and why strategies are decided on.

AlphaStar: Impressive for RL progress, not for AGI progress

This post argues that while it is impressive that AlphaStar can build up concepts complex enough to win at StarCraft, it is not actually developing reactive strategies. Rather than scouting what the opponent is doing and developing a new strategy based on that, AlphaStar just executes one of a predetermined set of strategies. This is because AlphaStar does not use causal reasoning, and that keeps it from beating any of the top players.


Note that the post is measuring compute used to _train_ models, which was less important in past AI research (e.g. it doesn't include Deep Blue), so it's not too surprising that we don't see the impact of AI winters.

Addendum to AI and Compute

Girish Sastry, Jack Clark, Greg Brockman, Ilya Sutskever

Last year, OpenAI <@wrote@>(@AI and Compute@) that since 2012, the amount of compute used in the largest-scale experiments has been doubling every 3.5 months. This addendum to that post analyzes data from 1959-2012, and finds that during that period the trend was a 2-year doubling time, approximately in line with Moore's Law, and not demonstrating any impact of previous "AI winters".


I liked this post and am glad it highlighted the distinction between learning ability and competence that seems to often be ignored in debates about AI progress. I would be excited to see some further exploration of the "broken parts" model and its implication about differing variances in cognitive abilities between humans and arbitrary intelligences.

Two explanations for variation in human abilities

How quickly might AI exceed human capabilities? One piece of evidence is the variation of intelligence within humans: if there isn’t much variation, we might expect AI not to stay at human level intelligence for long. It has been argued that variation in human cognitive abilities is small compared to such variation for arbitrary agents. However, the variation of human ability in games like chess seems to be quite pronounced, and it took chess computers more than forty years to transition from beginner level to beating the best humans. The blog post presents two arguments to reconcile these perspectives:

First, **similar minds could have large variation in learning ability**: If we break a random part of a complex machine, it might perform worse or stop working altogether, even if the broken machine is very similar to the unbroken one. Variation in human learning ability might be mostly explainable by lots of small "broken parts" like harmful mutations.

Second, **small variation in learning ability** can be consistent with **large variation in competence**, if the latter is explained by variation in another factor like practice time. For example, a chess match is not very useful to determine who's smarter, if one of the players has played a lot more games than the other. This perspective also reframes AlphaGo's superhumanity: the version that beat Lee Sedol had played around 2000 times as many games as him.


I agree with the conceptual argument, and I think it does hold in practice, quite strongly. I don't really agree that the field's response to critics implies that they are biased towards short timelines -- see [these](https://www.lesswrong.com/posts/KnQs55tjxWopCzKsk/the-ai-timelines-scam#LjaBHuCcnfqjBcqJa) [comments](https://www.lesswrong.com/posts/KnQs55tjxWopCzKsk/the-ai-timelines-scam#JaPQwejnFP95Q8nWk). Nonetheless, I'm going to do exactly what this post critiques, and say that I put significant probability on short timelines, but not explain my reasons (because they're complicated and I don't think I can convey them, and certainly can't convey them in a small number of words).

The AI Timelines Scam

This post argues that AI researchers and AI organizations have an incentive to predict that AGI will come soon, since that leads to more funding, and so we should expect timeline estimates to be systematically too short. Besides the conceptual argument, we can also see this in the field's response to critics: both historically and now, criticism is often met with counterarguments based on "style" rather than engaging with the technical meat of the criticism.


AI conference attendance

This post presents data on attendance numbers at AI conferences. The main result: "total large conference participation has grown by a factor 3.76 between 2011 and 2019, which is equivalent to a factor of 1.21 per year during that period". Looking at the graph, it seems to me that the exponential growth started in 2013, which would mean a slightly higher factor of around 1.3 per year. This would also make sense given that the current boom is often attributed to the publication of AlexNet in 2012.


Upon reading the experimental setup I didn't really know which way the answer was going to turn out, so I'm quite happy about now having another data point with which to understand learning dynamics. Of course, it's not clear how data about evolution will generalize to AI systems. For example, architectural improvements probably require some hard-to-find insight which make them hard to find via random search (imagine how hard it would be to invent CNNs by randomly trying stuff), while scaling up model size is easy, and so we might expect AI researchers to be differentially better at finding architectural improvements relative to scaling up model size (as compared to evolution).

Investigation into the relationship between neuron count and intelligence across differing cortical architectures

Primates vs birds: Is one brain architecture better than the other?

Progress in AI can be driven by both larger models as well as architectural improvements (given sufficient data and compute), but which of these is more important? One source of evidence comes from animals: different species that are closely related will have similar neural architectures, but potentially quite different brain sizes. This post compares intelligence across birds and primates: while primates (and mammals more generally) have a neocortex (often used to explain human intelligence), birds have a different, independently-evolved type of cortex. Using a survey over non-expert participants about how intelligent different bird and primate behavior is, it finds that there is not much difference in intelligence ratings between birds and primates, but that species with larger brains are rated as more intelligent than those with smaller brains. This only suggests that there are at least two neural architectures that work -- it could still be a hard problem to find them in the vast space of possible architectures. Still, it is some evidence that at least in the case of evolution, you get more intelligence through more neurons, and architectural improvements are relatively less important.


Evidence on good forecasting practices from the Good Judgment Project

This post lists some of the key traits which are associated with successful forecasting, based on work from the Good Judgement Project (who won IARPA's forecasting tournament by a wide margin). The top 5: past performance in the same broad domain; making more predictions on the same question; deliberation time; collaboration on teams; and intelligence.  The authors also summarise various other ideas from the Superforecasting book.


Forecasting Transformative AI: An Expert Survey

Ross Gruetzemacher, David Paradice, Kang Bok Lee


I think the evidence in the first part suggesting an abundance of compute is mostly explained by the fact that academics expect that we need ideas and algorithmic breakthroughs rather than simply scaling up existing algorithms, so you should update on that fact rather than this evidence which is a downstream effect. If we _condition_ on AGI requiring new ideas or algorithms, I think it is uncontroversial that we do not require huge amounts of compute to test out these new ideas.

The "we are bottlenecked on compute" argument should be taken as a statement about how to advance the state of the art in big unsolved problems in a sufficiently general way (that is, without encoding too much domain knowledge). Note that ImageNet is basically solved, so it does not fall in this category. At this point, it is a "small" problem and it's reasonable to say that it has an overabundance of compute, since it [requires](https://blog.openai.com/ai-and-compute/) _four orders of magnitude_ less compute than AlphaGo (and probably Dota). For the unsolved general problems, I do expect that researchers do use efficient training tricks where they can find them, and they probably optimize hyperparameters in some smarter way. For example, AlphaGo's hyperparameters were [trained via Bayesian optimization](https://arxiv.org/abs/1812.06855).

More details in [this comment](https://www.alignmentforum.org/posts/hSw4MNTc3gAwZWdx9/reasons-compute-may-not-drive-ai-capabilities-growth#AqaL4wutwNbocyR2u). I don't know much about trends in hardware so I won't comment on the second part.

Reasons compute may not drive AI capabilities growth

A common narrative (for example, [at OpenAI](https://blog.openai.com/ai-and-compute/)) is that AI progress will be driven by improvements in compute, but there are a few reasons we may not expect this to be the case. First, there are many known techniques to train faster that only require some engineering effort, that researchers often don't use. Second, researchers still use grid searches to optimize hyperparameters rather than [more efficient methods](https://www.automl.org/blog_bohb/). These two points suggest that researchers spend compute in order to avoid engineering effort, and so compute must not be the bottleneck.

In addition, the trends that have previously powered increasing levels of compute may be slowing down. For example, we had one-time gains by moving to GPUs and then to custom accelerators like TPUs, which probably will not happen again. In addition, many RL experiments require simulations on CPUs, and CPU improvements appear to be slowing down. GPU memory is often a bottleneck as well, though this could start increasing now that there is demand for larger memories, or we could get faster hardware interconnects that allow you to split models across multiple chips.


Time for AI to cross the human performance range in diabetic retinopathy


How rapidly are GPUs improving in price performance?


I do think that the probability of AGI in ten years is larger than 1-2%. I suspect my main disagreement is with the conception of what counts as groundbreaking progress. Tobias gives the example of transfer from one board game to many other board games; I think that AGI wouldn't be able to solve this problem from scratch, and humans are only capable of this because of [good priors](https://rach0012.github.io/humanRL_website/) from all the other learning we've done throughout life, _especially_ since games are designed to be human-understandable. If you make a sufficiently large neural net and give it a complex enough environment, some simple unsupervised learning rewards, and the opportunity to collect as much data as a human gets throughout life, maybe that does result in AGI. (I'd guess not, because it does seem like we have some good priors from birth, but I'm not very confident in that.)

Thoughts on short timelines

This post argues that the probability of AGI in the next ten years is very low, perhaps 1-2%. The primary argument is that to get AGI that quickly, we would need to be seeing research breakthroughs frequently, and empirically this is not the case. This might not be true if we expect that progress will accelerate in the future, but there's no reason to expect this -- we won't get recursive self-improvement before AGI and there won't be a huge increase in resources devoted to AI (since there is already so much excitement). We might also say that we are so clueless that we should assign at least 10% to AGI in ten years, but it doesn't seem we are that ignorant, and in any case it's not obvious that a prior should assign 10% to this outcome. Expert surveys estimate non-negligible probability on AGI in ten years, but in practice it seems the predominant opinion is to confidently dismiss a short timelines scenario.


Distinguishing AI takeover scenarios

This post summarizes several AI takeover scenarios that have been proposed and categorizes them according to three main variables. **Speed** refers to the question of whether there is a sudden jump in AI capabilities. **Uni/multipolarity** asks whether a single AI system takes over, or many. **Alignment** asks what goals the AI systems pursue, and if they are misaligned, further asks whether they are outer or inner misaligned. They also analyze other properties of the scenarios, such as how agentic, general and/or homogenous the AI systems are, and whether AI systems coordinate with each other or not. A [followup post](https://www.alignmentforum.org/posts/zkF9PNSyDKusoyLkP/investigating-ai-takeover-scenarios) investigates social, economic, and technological characteristics of these scenarios. It also generates new scenarios by varying some of these factors.

Since these posts are themselves summaries and comparisons of previously proposed scenarios that we’ve covered in this newsletter, I won’t summarize them here, but I do recommend them for an overview of AI takeover scenarios.


Updates and Lessons from AI Forecasting

This post provides an update on a project obtaining professional forecasts about progress in AI. I’m not going to summarize the full post here and instead list a few high-level takeaways:

1. The author found two of the forecasts surprising, while the other four were more in line with his expectations. The surprising forecasts suggested faster progress than he would have expected, and he has updated accordingly.
2. The forecasts imply confidence that AGI won’t arrive before 2025, but at the same time there will be clear and impressive progress in ML by then.
3. If you want to use forecasting, one particularly valuable approach is to put in the necessary work to define a good forecasting target. In this case, the author’s research group did this by creating the <@MATH@>(@Measuring Mathematical Problem Solving With the MATH Dataset@) and <@Multitask@>(@Measuring Massive Multitask Language Understanding@) datasets.


What 2026 looks like

This post describes the author’s median expectations around AI from now until 2026. It is part I of an attempt to write a detailed plausible future trajectory in chronological order, i.e. incrementally adding years to the story rather than writing a story with the end in mind. The hope is to produce a nice complement to the more abstract discussions about timelines and takeoff that usually occur. For example, there are discussions about how AI tools are used by nations for persuasion, propaganda and censorship.


This seems like a pretty clear reductio to me, though it is possible to argue that this argument doesn’t apply because compute isn’t the bottleneck, i.e. even with infinite compute we wouldn’t know how to make AGI. (That being said, I mostly do think we could build AGI if only we had enough compute; see also <@last week’s highlight on the scaling hypothesis@>(@The Scaling Hypothesis@).)

Fractional progress estimates for AI timelines and implied resource requirements

One [methodology](https://www.overcomingbias.com/2012/08/ai-progress-estimate.html) for forecasting AI timelines is to ask experts how much progress they have made to human-level AI within their subfield over the last T years. You can then extrapolate linearly to see when 100% of the problem will be solved. The post linked above collects such estimates, with a typical estimate being 5% of a problem being solved in the twenty year period between 1992 and 2012. Overall these estimates imply a timeline of [372 years](https://aiimpacts.org/surveys-on-fractional-progress-towards-hlai/).

This post provides a reductio argument against this pair of methodology and estimate. The core argument is that if you linearly extrapolate, then you are effectively saying “assume that business continues as usual: then how long does it take”? But “business as usual” in the case of the last 20 years involves an increase in the amount of compute used by AI researchers by a factor of ~1000, so this effectively says that we’ll get to human-level AI after a 1000^{372/20} = 10^56 increase in the amount of available compute. (The authors do a somewhat more careful calculation that breaks apart improvements in price and growth of GDP, and get 10^53.)

This is a stupendously large amount of compute: it far dwarfs the amount of compute used by evolution, and even dwarfs the maximum amount of irreversible computing we could have done with all the energy that has ever hit the Earth over its lifetime (the bound comes from [Landauer’s principle](https://en.wikipedia.org/wiki/Landauer%27s_principle)).

Given that evolution _did_ produce intelligence (us), we should reject the argument. But what should we make of the expert estimates then? One interpretation is that “proportion of the problem solved” behaves more like an exponential, because the inputs are growing exponentially, and so the time taken to do the last 90% can be much less than 9x the time taken for the first 10%.


You can see my thoughts on the trends in model size in [this comment](https://www.alignmentforum.org/posts/GzoWcYibWYwJva8aL/parameter-counts-in-machine-learning?commentId=sFFBeva2fDgsoynDC).

Parameter counts in Machine Learning

Jaime Sevilla, Pablo Villalobos, Juan Felipe Cerón

This post presents a dataset of the parameter counts of 139 ML models from 1952 to 2021. The resulting graph is fairly noisy and hard to interpret, but suggests that:

1. There was no discontinuity in model size in 2012 (the year that AlexNet was published, generally acknowledged as the start of the deep learning revolution).
2. There was a discontinuity in model size for language in particular some time between 2016-18.


Deep limitations? Examining expert disagreement over deep learning

This paper reports on the results of a qualitative survey of 25 experts, conducted in 2019 and early 2020, on the possibility of deep learning leading to high-level machine intelligence (HLMI), defined here as an “algorithmic system that performs like average adults on cognitive tests that evaluate the cognitive abilities required to perform economically relevant tasks”. Experts disagreed strongly on whether deep learning could lead to HLMI. Optimists tended to focus on the importance of scale, while pessimists tended to emphasize the need for additional insights.

Based on the interviews, the paper gives a list of 40 limitations of deep learning that some expert pointed to, and a more specific list of five areas that both optimists and pessimists pointed to as in support of their views (and thus would likely be promising areas to resolve disagreements). The five areas are (1) abstraction; (2) generalization; (3) explanatory, causal models; (4) emergence of planning; and (5) intervention.


I generally agree with the post and its conclusions. I'm not sure how much to care about the compute milestones -- it still seems likely to me that we are bottlenecked on algorithms for _general_ intelligence, but I don't think about this very much.

Interpreting AI Compute Trends

A previous [OpenAI post](https://blog.openai.com/ai-and-compute/) showed that the amount of compute used in the most expensive AI experiments has been growing exponentially for six years, with a doubling time of 3.5 _months_. This is extraordinarily fast, and can be thought of as a combination of growth in the amount spent on an experiment, and a decrease in the cost of computation. Such a trend can only continue for a few more years, before the cost of the experiment exceeds the budget of even the richest actors (such as the US government). However, this might still be enough to reach some important milestones for compute, such as "enough compute to simulate a human brain for 18 years", which is plausibly enough to get to AGI. (This would not happen for some of the larger estimates of the amount of computation in the human brain, but would happen for some of the smaller estimates.) It is still an open question which milestone we should care about.


I broadly agree that (1) and (3) don’t seem to be discussed much during forecasting, despite being quite important. (Though see e.g. [value of the long tail](https://www.lesswrong.com/posts/Nbcs5Fe2cxQuzje4K/value-of-the-long-tail).) I disagree with (2): while it is obviously possible that people are overestimating recent progress, or are overconfident about how useful scaling will be, there has at least been a lot of thought put into that particular question -- it seems like one of the central questions tackled by <@bio anchors@>(@Draft report on AI timelines@). See more discussion in this [comment thread](https://www.alignmentforum.org/posts/Z5gPrKTR2oDmm6fqJ/three-reasons-to-expect-long-ai-timelines?commentId=F7FNee8Bpa8hemQkd).

Three reasons to expect long AI timelines

This post outlines and argues for three reasons to expect long AI timelines that the author expects are not taken into account in current forecasting efforts:

1. **Technological deployment lag:** Most technologies take decades between when they're first developed and when they become widely impactful.
2. **Overestimating the generality of AI technology:** Just as people in the 1950s and 1960s overestimated the impact of solving chess, it seems likely that current people are overestimating the impact of recent progress, and how far it can scale in the future.
3. **Regulation will slow things down,** as with [nuclear energy](https://rootsofprogress.org/devanney-on-the-nuclear-flop), for example.

You might argue that the first and third points don’t matter, since what we care about is when AGI is _developed_, as opposed to when it becomes widely deployed. However, it seems that we continue to have the opportunity to intervene until the technology becomes widely impactful, and that seems to be the relevant quantity for decision-making. You could have some specific argument like “the AI goes FOOM and very quickly achieves all of its goals” that then implies that the development time is the right thing to forecast, but none of these seem all that obvious.


These kinds of surveys seem important to conduct, even if they don't always deliver very surprising results. That said, I was surprised by the large amount of small AGI projects (for which I expect the chances of success to be tiny) and the overall small amount of Chinese AGI projects.

2020 Survey of Artificial General Intelligence Projects for Ethics, Risk, and Policy

McKenna Fitzgerald, Aaron Boddy, Seth D. Baum

This is a survey of AGI research and development (R&D) projects, based on public information like publications and websites. The survey finds 72 such projects active in 2020 compared to 70 projects active in 2017. This corresponds to 15 new projects and 13 projects that shut down since 2017. Almost half of the projects are US-based (and this is fewer than in 2017!), and most of the rest is based in US-allied countries. Around half of the projects publish open-source code. Many projects are interconnected via shared personnel or joint projects and only a few have identifiable military connections (fewer than in 2017). All of these factors might facilitate cooperation around safety.

The projects form three major clusters: 1) corporate projects active on AGI safety 2) academic projects not active on AGI safety and 3) small corporations not active on AGI safety. Most of the projects are rather small and project size varies a lot, with the largest projects having more than 100 times as many employees as the smallest ones. While the share of projects with a humanitarian focus has increased to more than half, only a small but growing number is active on safety. Compared to 2017, the share of corporate projects has increased, and there are fewer academic projects. While academic projects are more likely to focus on knowledge expansion rather than humanitarian goals, corporate projects seem more likely to prioritize profit over public interest and safety. Consequently, corporate governance might be especially important.


This outline seems pretty reasonable to me. There are a lot of specific points to nitpick with; for example, I am not convinced that we can just use cloud compute. It seems plausible that manipulation tasks require quick, iterative feedback, where the latency of cloud compute would be unacceptable. (Indeed, the quick, iterative feedback of touch is exactly why it is such a valuable sensor.) Nonetheless, I broadly like the outlined plan and it feels like these sorts of nitpicks are things that we will be able to solve as we work on the problem.

I am more skeptical of the cost estimate, which seems pretty optimistic to me. The author basically took existing numbers and then multiplied them by some factor for the increased hardness; I think that those factors are too low (for the AI aspects, idk about the robot hardware aspects), and I think that there are probably lots of other significant “invisible” costs that aren’t being counted here.

How The Hell Do We Create General-Purpose Robots?

A **general-purpose robot** (GPR) is one that can execute simple commands like “unload the dishwasher” or “paint the wall”. This post outlines an approach to get to such robots, and estimates how much it would cost to get there.

On the hardware side, we need to have hardware for the body, sensors, and brain. The body is ready; the Spot robot from Boston Dynamics seems like a reasonable candidate. On sensors, we have vision, hearing and lidar covered; however, we don’t have great sensors for touch yet. That being said, it seems possible to get by with bad sensors for touch, and compensate with vision. Finally, for the brain, even if we can’t put enough chips on the robot itself, we can use more compute via the cloud.

For software, in principle a large enough neural network should suffice; all of the skills involved in GPRs have already been demonstrated by neural nets, just not as well as would be necessary. (In particular, we don’t need to posit AGI.) The big issue is that we don’t know how to train such a network. (We can’t train in the real world, as that is way too slow.)

With a big enough investment, it seems plausible that we could build a simulator in which the robot could learn. The simulator would have to be physically realistic and diverse, which is quite a challenge. But we don’t have to write down physically accurate models of all objects: instead, we can _virtualize_ objects. Specifically, we interact with an object for a couple of minutes, and then use the resulting data to build a model of the object in our simulation. (You could imagine an AlphaFold-like system that does this very well.)

The author then runs some Fermi estimates and concludes that it might cost around $42 billion for the R&D in such a project (though it may not succeed), and concludes that this would clearly be worth it given the huge economic benefits.


This all seems generally right to me, and is part of the reason I like the <@biological anchors approach@>(@Draft report on AI timelines@) to forecasting transformative AI.

Birds, Brains, Planes, and AI: Against Appeals to the Complexity/Mysteriousness/Efficiency of the Brain

This post argues against a particular class of arguments about AI timelines. These arguments have the form: “The brain has property X, but we don’t know how to make AIs with property X. Since it took evolution a long time to make brains with property X, we should expect it will take us a long time as well”. The reason these are not compelling is because humans often use different approaches to solve problems than evolution did, and so humans might solve the overall problem without ever needing to have property X. To make these arguments more convincing, you need to argue 1) why property X really is _necessary_ and 2) why property X won’t follow quickly once everything else is in place.

This is illustrated with a hypothetical example of someone trying to predict when humans would achieve heavier-than-air flight: in practice, you could have made decent predictions just by looking at the power to weight ratios of engines vs. birds. Someone who argued that we were far away because “we don't even know how birds stay up for so long without flapping their wings” would have made incorrect predictions.


How energy efficient are human-engineered flight designs relative to natural ones?

When forecasting AI timelines from <@biological anchors@>(@Draft report on AI timelines@), one important subquestion is how well we expect human-made artifacts to compare to natural artifacts (i.e. artifacts made by evolution). This post gathers empirical data for flight, by comparing the Monarch butterfly and the Wandering Albatross to various types of planes. The albatross is the most efficient, with a score of 2.2 kg-m per Joule (that is, a ~7 kg albatross spends ~3 Joules for every meter it travels). This is 2-8x better than the most efficient manmade plane that the authors considered, the Boeing 747-400, which in turn is better than the Monarch butterfly. (The authors also looked at distance per Joule without considering mass, in which case unsurprisingly the butterfly wins by miles; it is about 3 orders of magnitude better than the albatross, which is in turn better than all the manmade solutions.)


This seems like a pretty interesting direction, though I’m not totally clear on how it relates to AI alignment (assuming it is supposed to). It does seem quite related to iterated amplification, which also relies on this sort of decomposition of questions.

I found the video mock ups and demos in the blog post to be particularly interesting, both in the original post and in the [call for beta testers](https://www.lesswrong.com/posts/WBgT4jAqTrPN7qh3Z/beta-test-gpt-3-based-research-assistant); I think they were much better at showcasing the potential value than the abstract description, and I recommend you all watch them.

Automating reasoning about the future at Ought

Roughly speaking, we can think of an axis of reasoning, spanning from high-confidence statistical reasoning with lots of data to general quantitative reasoning to very qualitative reasoning. Ought is now building Elicit to help with _judgmental forecasting_, which is near but not at the end of the spectrum. In judgmental forecasting, we might take a complicated question such as “Will Roe v. Wade be overturned if Trump nominates a new justice”, decompose it into subquestions, estimate probabilities for those subquestions, and then combine them to get to a final forecast. Crucially, this requires relying on people’s judgment: we cannot just look at the historical rate at which landmark Supreme Court decisions are overturned, since the situation has rarely arisen before.

Currently, Elicit has several features that help with the quantitative aspects of judgmental forecasting, for example by enabling users to input complex distributions and visualizing these distributions. However, in the long-term, the hope is to also help with the more qualitative aspects of judgmental forecasting as well, for example by proposing an important subquestion for answering the current question under consideration, or recommending a source of data that can answer the question at hand.

Ought is now working on adding these sorts of features by using large language models (GPT-3 in particular). They are currently [looking for beta testers](https://www.lesswrong.com/posts/WBgT4jAqTrPN7qh3Z/beta-test-gpt-3-based-research-assistant) for these features!


AGI Predictions

A collection of interesting questions relevant to AI safety, as well as aggregated predictions from readers of the post.


1st International Workshop on Evaluating Progress in Artificial Intelligence

I like the methodology proposed in this paper, and I found the list of named limitations of deep learning interesting and informative. I’m not sure that I personally agree with the particular canaries identified in the example (which the authors emphasize is just one possible causal graph). It seems plausible to me that both flexible memory and symbol-like representations would be an emergent property of any deep learning system with a sufficiently rich training dataset, curriculum, compute available, etc. and the real milestones to watch would be advances in those inputs.

Canaries in Technology Mines: Warning Signs of Transformative Progress in AI

In this paper, Cremer et al. propose a methodology for identifying early warning signs ('canaries') for transformative AI progress. The methodology consists of identifying key milestones using expert elicitation, arranging those milestones into causal graphs where any given milestone may make another milestone more likely, and then using the causal graph representation to identify canaries-- nodes which have a significant number of outgoing nodes.

As an example, they give a partial implementation of using this methodology to identify canaries for high-level machine intelligence. Cremer et al. interview 25 experts in a variety of fields about the limitations of deep learning, then collate the named limitations and translate them into 'milestones'. Interviewees name 34 (potentially overlapping) milestones in total, including causal reasoning, meta-learning, hierarchical decomposition, (abstract) representation, flexible memory, common sense, architecture search, and navigating brittle environments.

Cremer et al. then construct one possible causal graph for these milestones, and identify two that may act as canaries: _Symbol-like representations_, i.e. the ability to construct abstract, discrete, and disentangled representations of inputs, could underly grammar, mathematical reasoning, concept formation, and flexible memory. _Flexible memory_, the ability to store, recognize, and re-use knowledge, could unlock the ability to learn from dynamic data, the ability to do continuous learning, and the ability to learn how to learn.


Roadmap to a Roadmap: How Could We Tell When AGI is a ‘Manhattan Project’ Away?

The key hypothesis of this paper is that once there is a clear “roadmap” or “runway” to AGI, it is likely that state actors could invest a large number of resources into achieving it, comparably to the Manhattan project. The fact that we do not see signs of such investment now does not imply that it won’t happen in the future: currently, there is so little “surface area” on the problem of AGI that throwing vast amounts of money at the problem is unlikely to help much.

If this were true, then once such a runway is visible, incentives could change quite sharply: in particular, the current norms of openness may quickly change to norms of secrecy, as nations compete (or perceive themselves to be competing) with other nations to build AGI first. As a result, it would be good to have a good measure of whether we have reached the point where such a runway exists.


My personal opinion on timelines is that I think it is much more informative to draw out the full CDF/PDF of when we will get to AGI instead of percentages by different years. It isn’t included in the post, but you can find Alex’s [here](https://elicit.ought.org/builder/BsNkKzJoc). I end up placing higher likelihood on AGI happening sooner than Alex does, but I largely agree with his reasoning.

More uncertainty than the original prediction seems warranted to me; the original prediction had a very high likelihood of AGI between 2045-2050 that I didn’t understand. Of the rest of the arguments, I agree most strongly with the section on tooling providing a speedup. I’d even push the point farther to say that there are many inputs into current ML systems, and all of them seem to be improving at a rapid clip. Hardware, software tools, data, and the number of ML researchers all seem to be on track to improve significantly over the next decade.

My AI Timelines Have Sped Up

Alex Irpan updates his predictions of AGI sooner to:

10% chance by 2035 (previously 2045)
50% chance by 2045 (previously 2050)
90% chance by 2070

The main reasons why are:

- Alex is now more uncertain because research pace over the past five years have been more surprising than expected, faster in some domains, but slower than others.
- Accounting for improvements in tooling. New libraries like TensorFlow and PyTorch have accelerated progress. Even CNNs can be used as a “tool” that provides features for downstream tasks like robotic control.
- He previously thought that labeled data might be a bottleneck, based on scaling laws showing that data needs might increase faster than compute; however, semi- and unsupervised learning have improved significantly, GPT-3 being the latest example of this.
- Alex now believes that compute will play a larger role and that compute can scale faster than algorithms because there is large worldwide consumer demand.

The post ends with a hypothetical description of how AGI may happen soon that I will leave out of the summary but recommend reading.


I think this sort of reasoning about the future, where you are forced into a scenario and have to reason what must have happened and draw implications, seems particularly good for ensuring that you don’t get too locked in to your own beliefs about the future, which will likely be too narrow.

Relevant pre-AGI possibilities

This page lists 47 things that could plausibly happen before the development of AGI, that could matter for AI safety or AI policy. You can also use the web page to generate a very simple trajectory for the future, as done in this [scenario](https://docs.google.com/document/d/1gd6qQx-SP6rfAVQE5rzfPH0zXYOOHXBa13JSUd2zROQ/edit) that Daniel wrote up.


More on disambiguating "discontinuity"

This post considers three different kinds of “discontinuity” that we might imagine with AI development. First, there could be a sharp change in progress or the rate of progress that breaks with the previous trendline (this is the sort of thing <@examined@>(@Discontinuous progress in history: an update@) by AI Impacts). Second, the rate of progress could either be slow or fast, regardless of whether there is a discontinuity in it. Finally, the calendar time could either be short or long, regardless of the rate of progress.

The post then applies these categories to three questions. Will we see AGI coming before it arrives? Will we be able to “course correct” if there are problems? Is it likely that a single actor obtains a decisive strategic advantage?


While I agree that the debate about takeoff speeds is primarily about the path by which we get to powerful AI systems, that seems like a pretty important question to me with <@many ramifications@>(@Clarifying some key hypotheses in AI alignment@).

Will AI undergo discontinuous progress?

This post argues that the debate over takeoff speeds is over a smaller issue than you might otherwise think: people seem to be arguing for either discontinuous progress, or continuous but fast progress. Both camps agree that once AI reaches human-level intelligence, progress will be extremely rapid; the disagreement is primarily about whether there is already quite a lot of progress _before_ that point. As a result, these differences don't constitute a "shift in arguments on AI safety", as some have claimed.

The post also goes through some of the arguments and claims that people have made in the past, which I'm not going to summarize here.


It seems very hard to actually make a principled distinction between these two facets of intelligence, because culture has such an influence over our "raw innovative capability" in the sense of our ability to make original discoveries / learn new things. While feral children might be less intelligent than animals (I wouldn't know), the appropriate comparison would be against "feral animals" that also didn't get opportunities to explore their environment and learn from their parents, and even so I'm not sure how much I'd trust results from such a "weird" (evolutionarily off-distribution) setup.

Might humans not be the most intelligent animals?

We can roughly separate intelligence into two categories: _raw innovative capability_ (the ability to figure things out from scratch, without the benefit of those who came before you), and _culture processing_ (the ability to learn from accumulated human knowledge). It's not clear that humans have the highest raw innovative capability; we may just have much better culture. For example, feral children raised outside of human society look very "unintelligent", [The Secret of Our Success](https://www.amazon.com/dp/B00WY4OXAS/ref=dp-kindle-redirect?_encoding=UTF8&btkr=1) documents cases where culture trumped innovative capability, and humans actually _don't_ have the most neurons, or the most neurons in the forebrain.

(Why is this relevant to AI alignment? Matthew claims that it has implications on AI takeoff speeds, though he doesn't argue for that claim in the post.)


Etzioni 2016 survey

Oren Etzioni surveyed 193 AAAI fellows in 2016 and found that 67.5% of them expected that ‘we will achieve Superintelligence’ someday, but in more than 25 years. Only 7.5% thought we would achieve it sooner than that.


AI Forecasting Dictionary

One big challenge with forecasting the future is operationalizing key terms unambiguously, so that a question can be resolved when the future actually arrives. Since we'll probably need to forecast many different questions, it's crucial that we make it as easy as possible to create and answer well-operationalized questions. To that end, the authors have created and open-sourced an AI Forecasting Dictionary, which gives precise meanings for important terms, along with examples and non-examples to clarify further.


AI Forecasting Resolution Council

Even if you operationalize forecasting questions well, often the outcome is determined primarily by factors other than the one you are interested in. For example, progress on a benchmark might be determined more by the number of researchers who try to beat the benchmark than by improvements in AI capabilities, even though you were trying to measure the latter. To deal with this problem, an AI Forecasting Resolution Council has been set up: now, forecasters can predict what the resolution council will say at some particular time in the future. This allows for questions that get at what we want: in the previous case, we could now forecast how the resolution council will answer the question "would current methods be able to beat this benchmark" in 2021.


How to write good AI forecasting questions + Question Database

As discussed above, operationalization of forecasting questions is hard. This post collects some of the common failure modes, and introduces a database of 76 questions about AI progress that have detailed resolution criteria that will hopefully avoid any pitfalls of operationalization.


Historical economic growth trends

Data on historical economic growth "suggest that (proportional) rates of economic and population growth increase roughly linearly with the size of the world economy and population", at least from around 0 CE to 1950. However, this trend has not held since 1950 - in fact, growth rates have fallen since then.


Note there's a strong selection effect, as AGI is a conference specifically aimed at general intelligence.

AGI-11 Survey

A survey of participants in the AGI-11 participants (with 60 respondents out of over 200 registrations) found that 43% thought AGI would appear before 2030, 88% thought it would appear before 2100, and 85% believed it would be beneficial for humankind.


Why Is the Human Brain So Efficient?

Overall point for this audience is that, despite how slow and imprecise neuron signals are, the human brain beats computers because of how massively parallel it is.


Jeff Hawkins on neuromorphic AGI within 20 years


[Will AI See Sudden Progress?](https://www.lesserwrong.com/posts/AJtfNyBsum6ZzWxKR/will-ai-see-sudden-progress), [Takeoff speeds](https://sideways-view.com/2018/02/24/takeoff-speeds/) and [The Hanson-Yudkowsky AI-Foom Debate](https://intelligence.org/ai-foom-debate/)

Double Cruxing the AI Foom debate


I really like this paper, it's a great concrete example of how systems of agents can have very different behavior than any one individual agent _even if_ each of the agents have similar goals. The idea makes intuitive sense and I think the model captures its salient aspects. There are definitely many quibbles you could make with the model (though perhaps it is the standard model, I don't know this field), but I don't think they're important. My perspective is that the model is a particularly clear and precise way of communicating the effect that the authors are describing, as opposed to something that is supposed to track reality closely.

Legible Normativity for AI Alignment: The Value of Silly Rules

Dylan Hadfield-Menell, McKane Andrus, Gillian K. Hadfield

One issue we might have with value learning is that our AI system might look at "silly rules" and infer that we care about them deeply. For example, we often enforce dress codes through social punishments. Given that dress codes do not have much functional purpose and yet we enforce them, should an AI system infer that we care about dress codes as much as we care about (say) property rights? This paper claims that these "silly rules" should be interpreted as a coordination mechanism that allows group members to learn whether or not the group rules will be enforced by neutral third parties. For example, if I violate the dress code, no one is significantly harmed but I would be punished anyway -- and this can give everyone confidence that if I were to break an important rule, such as stealing someone's wallet, _bystanders_ would punish me by reporting me to the police, even though they are not affected by my actions and it is a cost to them to report me.

They formalize this using a model with a pool of agents that can choose to be part of a group. Agents in the group play "important" games and "silly" games. In any game, there is a scofflaw, a victim, and a bystander. In an important game, if the bystander would punish any rule violations, then the scofflaw follows the rule and the victim gets +1 utility, but if the bystander would not punish the violation, the scofflaw breaks the rule and the victim gets -1 utility. Note that in order to signal that they would punish, bystanders must pay a cost of c. A silly game works the same way, except the victim always gets 0 utility. Given a set of important rules, the main quantity of interest is how many silly rules to add. The authors quantify this by considering the _proportion_ of all games that are silly games, which they call the density. Since we are imagining _adding_ silly rules, all outcomes are measured with respect to the number of _important_ games. We can think of this as a proxy for time, and indeed the authors call the expected number of games till an important game a _timestep_.

Now, for important games the expected utility to the victim is positive if the probability that the bystander is a punisher is greater than 0.5. So, each of the agents cares about estimating this probability in order to decide whether or not to stay in the group. Now, if we only had important games, we would have a single game per timestep, and we would only learn whether one particular agent is a punisher. As we add more silly games, we get more games per timestep, and so we can learn much more quickly the proportion of punishers, which leads to more stable groups. However, the silly rules are not free. The authors prove that if they _are_ free, then we keep adding silly rules and the density would approach 1. (More precisely, they show that as density goes to 1, the value of being told the true probability of punishment goes to 0, meaning that the agent already knows everything.)

They then show experimental results showing a few things. When the agents are relatively certain of the probability of an agent being a punisher, then silly rules are not very useful and the group is more likely to collapse (since the cost of enforcing the silly rules starts to be important). Second, as long as c is low (so it is easy to signal that you will enforce rules), then groups with more silly rules will be more resilient to shocks in individual's beliefs about the proportion of punishers, since they will very quickly converge to the right belief. If there aren't any silly rules it can take more time and your estimate might be incorrectly low enough that you decide to leave the group even though group membership is still net positive. Finally, if the proportion of punishers drops below 0.5, making group membership net negative, agents in groups with high density will learn this faster, and their groups will disband much sooner.


A major challenge for an AI would be to figure out how to aggregate preferences across humans. This is not necessarily a problem that we have to solve immediately -- it's possible that we build an AI that mimics what we do initially and then develops its own theory of voting. However, if you are doing any kind of reward learning (such as inverse reinforcement learning), you will have to confront this problem head on. This article points out a lot of problems that we would have to be aware of in this case. The solutions seem less likely to transfer, because they are optimized for a different scenario (such as presidential elections).
A few months ago I was looking into social choice theory for this reason, and it took me a few hours to get to an understanding of maybe a quarter of the content in these articles, so I really recommend not doing that and instead just reading these articles.

Will MacAskill's PhD thesis, [Normative Uncertainty](http://commonsenseatheism.com/wp-content/uploads/2014/03/MacAskill-Normative-Uncertainty.pdf), considers how to combine different moral theories that have different preferences over world states.

[A voting theory primer for rationalists](https://www.lesswrong.com/posts/D6trAzh6DApKPhbv4/a-voting-theory-primer-for-rationalists) and [5 voting pathologies: lesser names of Moloch](https://www.lesswrong.com/posts/4vEFX6EPpdQZfqnnS/5-voting-pathologies-lesser-names-of-moloch)

Voting theory, or social choice theory, studies voting methods, which take a set of preferences over outcomes from voters, and decides which outcome should occur. The field is littered with impossibility results and difficult problems that are hard to resolve, but there are voting methods that do reasonably well in practice. The second post elaborates more on the 5 problems that are hard to resolve.


You could imagine two different philosophies on how to build AI systems -- the first option is to train them on the actual task of interest (for Overcooked, training agents to play against humans or human models), while the second option is to train a more robust agent on some more general task that hopefully includes the actual task within it (the approach in this paper). Besides Overcooked, another example would be supervised learning on some natural language task (the first philosophy), as compared to pretraining on the Internet GPT-style and then prompting the model to solve your task of interest (the second philosophy). In some sense the quest for a single unified AGI system is itself a bet on the second philosophy -- first you build your AGI that can do all tasks, and then you point it at the specific task you want to do now.

Historically, I think AI has focused primarily on the first philosophy, but recent years have shown the power of the second philosophy. However, I don’t think the question is settled yet: one issue with the second philosophy is that it is often difficult to fully “aim” your system at the true task of interest, and as a result it doesn’t perform as well as it “could have”. In Overcooked, the FCP agents will not learn specific quirks of human gameplay that could be exploited to improve efficiency (which the human-aware agent could do, at least in theory). In natural language, even if you prompt GPT-3 appropriately, there’s still some chance it ends up rambling about something else entirely, or neglects to mention some information that it “knows” but that a human on the Internet would not have said. (See also <@this post@>(@The case for aligning narrowly superhuman models@).)

I should note that you can also have a hybrid approach, where you start by training a large model with the second philosophy, and then you finetune it on your task of interest as in the first philosophy, gaining the benefits of both.

I’m generally interested in which approach will build more useful agents, as this seems quite relevant to forecasting the future of AI (which in turn affects lots of things including AI alignment plans).

Related paper: Maximum Entropy Population Based Training for Zero-Shot Human-AI Coordination

Collaborating with Humans without Human Data

DJ Strouse, Kevin R. McKee, Matt Botvinick, Edward Hughes, Richard Everett

We’ve previously seen that if you want to collaborate with humans in the video game Overcooked, <@it helps to train a deep RL agent against a human model@>(@Collaborating with Humans Requires Understanding Them@), so that the agent “expects” to be playing against humans (rather than e.g. copies of itself, as in self-play). We might call this a “human-aware” model. However, since a human-aware model must be trained against a model that imitates human gameplay, we need to collect human gameplay data for training. Could we instead train an agent that is robust enough to play with lots of different agents, including humans as a special case?

This paper shows that this can be done with **Fictitious Co-Play** (FCP), in which we train our final agent against a population of self-play agents and their past checkpoints taken throughout training. Such agents get significantly higher rewards when collaborating with humans in Overcooked (relative to the human-aware approach in the previously linked paper).

In their ablations, the authors find that it is particularly important to include past checkpoints in the population against which you train. They also test whether it helps to have the self-play agents have a variety of architectures, and find that it mostly does not make a difference (as long as you are using past checkpoints as well).


I think there are three main sources of impact of this agenda from an x-risk perspective:
1. If an AI system has better agent-agnostic cooperative intelligence, it should be less likely to fall into “traps” of multiagent situations, such as <@conflict, bargaining failures@>(@Cooperation, Conflict, and Transformative Artificial Intelligence: A Research Agenda@), or <@commitment races@>(@The "Commitment Races" problem@).
2. If an AI system has better human-specific cooperative intelligence, it should be easier to (a) align that system with a human principal and (b) have that system cooperate with other humans (besides its principal).
3. If an AI system promotes cooperation between other agents (including humans), that seems to help with a variety of other major global problems, such as biorisk, nuclear war, and climate change.

I agree that all three of these things, if achieved, would make the world better in expectation (though see <@here@>(@Thoughts on Human Models@) for some arguments against). I feel best about (3), because it seems to have the largest surface area for improvement. In fact, (1) and (2) are almost special cases of (3), in which the AI system improves cooperation by being one of the agents that is cooperating with others (presumably on behalf of some human principal who wouldn't have done as good a job).

I am more uncertain about how best to achieve the goals laid out in (1) - (3). The article promotes multiagent reinforcement learning (MARL) for goal (1), which I think is plausible but not obvious: I could imagine that it would be better to (say) pretrain a large language model and then finetune it to be cooperative using human judgments. Within the Cooperative AI paradigm, I’m most excited about figuring out the best research bets to make in order to achieve the three goals above. The authors are very interested in finding others to help with this.

Cooperative AI: machines must learn to find common ground

Allan Dafoe, Yoram Bachrach, Gillian Hadfield, Eric Horvitz, Kate Larson, Thore Graepel

This short piece argues that rather than building autonomous AI systems (which typically involves a non-social environment), we should instead work on building AI systems that are able to promote mutually beneficial joint action, that is, we should work on <@Cooperative AI@>(@Open Problems in Cooperative AI@). This can be separated into three main categories:

1. AI-AI cooperation: Here, two AI systems must cooperate with each other. Think for example of games like Hanabi or Diplomacy.
2. AI-human cooperation: This setting involves an AI system that must understand and work with a human. <@Assistance games@>(@Human Compatible: Artificial Intelligence and the Problem of Control@) are a central example. When there are multiple humans, it becomes important for our AI system to understand norms and institutions as well.
3. Human-human cooperation: Here, AI systems are used to enhance cooperation between humans. For example, machine translation helps people who speak different languages cooperate with each other.

There is now a new nonprofit, the [Cooperative AI Foundation](https://www.cooperativeai.com/foundation), that supports research on these topics.


I believe that humanity could be a lot better off if we were better at cooperating instead of wasting resources on competition over positional goods, so I am excited about more work exploring how AI might help with facilitating cooperation. Given that current RL often requires specialized algorithms to learn cooperation, even when agents are trained jointly, it seems likely that failures to cooperate will become even more problematic if we start to deploy more and more RL agents without first making progress in cooperative AI. While it has been argued that a shift towards more attention to learning cooperation is bound to happen because of economic incentives, I am still glad to see more steps in that direction.

**Rohin’s opinion:** Amongst possible technical approaches that could help with AI governance, cooperative AI seems like a good contender (though there can be downsides, as the paper notes). I’d be especially excited to get more clarity on when and where cooperative skills are important, perhaps through “deployment stories” of AI systems in which cooperative skills play a critical role in ensuring good outcomes.

Open Problems in Cooperative AI

Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R. McKee, Joel Z. Leibo, Kate Larson, Thore Graepel

Cooperation can often lead to better outcomes for everyone involved and some progress in AI, like improved machine translation or smart contracts, can make cooperation easier. On the other hand, as AI agents will likely become more and more important, it is crucial they can cooperate with each other and with humans. The term **cooperative AI** combines these perspectives and refers to "AI research trying to help individuals, humans and machines, to find ways to improve their joint welfare". While previous AI research has focused on individual intelligence of artificial agents, more work on social intelligence and cooperative capabilities is needed. The paper provides an overview of research relevant to cooperative AI from a wide range of disciplines and aims to facilitate more interdisciplinary conversations about cooperative AI by providing a common framework and vocabulary.

Research on cooperative opportunities, situations in which gains from cooperation are possible, can differ along four major dimensions:
1) How much do interests overlap or conflict?
2) What kind of agents are involved? (Humans, machines, organizations)
3) What perspective is taken: the perspective of an individual trying to cooperate or a social planner trying to incentivize cooperation?
4) What is the scope and how interdisciplinary should research be?

Four types of capabilities can be crucial for cooperation:
1) Understanding: In cases where cooperation is optimal for all parties, with no incentive to deviate, but agents lack understanding of either the environment or one another, they may still fail to reach a cooperative equilibrium. For example, one agent might have false beliefs about the others' beliefs and preferences and thus their incentive for defection. This is particularly hard to get around, because (1) preferences might not be defined explicitly or might even be incoherent; (2) there might be incentives to misrepresent preferences; and (3) the recursive nature of beliefs about other agents' beliefs may be challenging to handle.
2) Communication: In other cases, like the Stag Hunt game, there are multiple equilibria but agents' incentives are mostly aligned. Still, a lack of communicative abilities or common ground to interpret each other's messages can lead to agents converging to a suboptimal equilibrium. This is complicated further by constraints like limited bandwidth, high latency, or compatibility with human forms of communication.
3) Commitment: While communication helps when incentives are mostly aligned, in games like Chicken where some equilibria clearly favour one agent, communication alone is insufficient as agents have incentive to lie. In some cases, these problems can be circumvented using costly signals. However, often some form of credible commitment device to ensure cooperation, or at least truth-telling, is needed. Such commitment devices can enable unconditional ("I won't defect") or conditional ("I won't defect if you won't") and unilateral or multilateral commitments that require multiple actors to consent but bind them all. While unilateral unconditional commitments are most accessible, other forms can be a lot more powerful enablers of cooperation. Mechanisms that could be useful for these more powerful commitments include reputation, delegation to a trusted third party and (smart) contracts.
4) Institutions: In some games like the Prisoner's Dilemma, defection is a dominant strategy for all players, even if they would all be better off with mutual cooperation. In such situations, changing the rules of the game to align incentives and facilitate cooperation can make everyone better off, for example by linking different games (as done with the Iterated Prisoner's Dilemma) or introducing institutions. Institutions can be decentralized and entail (sometimes implicit) enforcement by players (norms, conventions, trust and reputation) or involve a centralized enforcing authority. The study of centralized institutions can draw on the literature in social choice theory, fairness, and mechanism design.

Lastly, the authors list potential downsides that better cooperative capabilities could have: While they increase the welfare of the cooperating agents, this might be at the cost of other agents. For example, better cooperation between criminals would likely be bad for society. Similarly, cooperation can undermine prosocial competition at the expense of society, as seen in the example of cartels. On the other hand, a better understanding of the world and others' preferences makes it easier to threaten others efficiently and coercive capabilities greatly benefit from credible conditional commitments to carry out a threat. Furthermore, coercive capabilities might be important for stabilizing cooperation as in the case of punishing defectors. Lastly, powerful bad actors can use more efficient institutions to serve their own antisocial goals.


I agree that we should put more emphasis on the safety of multi-agent systems. We already have <@evidence@>(@Emergent Tool Use from Multi-Agent Interaction@) that complex behavior can arise from simple objectives in current systems, and this seems only more likely as systems become more powerful. Two-agent paradigms such as GANs, self-play, and debate, are already quite common in ML. Lastly, humans evolved complex behavior from the simple process of evolution so we have at least one example of this working. I also think this is an interesting area where there is lots to learn from other fields, such as game theory and evolutionary biology,

For any empirically-minded readers of this newsletter, I think this sequence opens up a lot of potential for research. The development of safety benchmarks for multi-agent systems and then the evaluation of these approaches seems like it would make many of the considerations discussed here more concrete. I personally would find them much more convincing with empirical evidence to back up that they work with current ML.

**Rohin's opinion:** The AGI model here in which powerful AI systems arise through multiagent interaction is an important and plausible one, and I'm excited to see some initial thoughts about it. I don't particularly expect any of these ideas to be substantially useful, but I'm also not confident that they won't be useful, and given the huge amount of uncertainty about how multiagent interaction shapes agents, that may be the best we can hope for currently. I'd be excited to see empirical results testing some of these ideas out, as well as more conceptual posts suggesting more ideas to try.

Shaping Safer Goals

Much of safety research focuses on a single agent that is directly incentivized by a loss/reward function to take particular actions. This sequence instead considers safety in the case of multi-agent systems interacting in complex environments. In this situation, even simple reward functions can yield complex and highly intelligent behaviors that are only indirectly related. For example, evolution led to humans who can learn to play chess, despite the fact that the ancestral environment did not contain chess games. In these situations, the problem is not how to construct an aligned reward function, the problem is how to shape the experience that the agent gets at training time such that the final agent policy optimizes for the goals that we want. This sequence lays out some considerations and research directions for safety in such situations.

One approach is to teach agents the generalizable skill of obedience. To accomplish this, one could design the environment to incentivize specialization. For instance, if an agent A is more powerful than agent B, but can see less of the environment than B, A might be incentivized to obey B’s instructions if they share a goal. Similarly we can increase the ease and value of coordination through enabling access to a shared permanent record or designing tasks that require large-scale coordination.

A second approach is to move agents to simpler and safer training regimes as they develop more intelligence. The key assumption here is that we may require complex regimes such as competitive multi-agent environments to jumpstart intelligent behavior, but may be able to continue training in a simpler regime such as single-task RL later. This is similar to current approaches for training a language model via supervised learning and then finetuning with RL, but going in the opposite direction to increase safety rather than capabilities.

A third approach is specific to a collective AGI: an AGI that is composed of a number of separate general agents trained on different objectives that learn to cooperatively solve harder tasks. This is similar to how human civilization is able to accomplish much more than any individual human. In this regime, the AGI can be effectively sandboxed by either reducing the population size or by limiting communication channels between the agents. One advantage of this approach to sandboxing is that it allows us to change the effective intelligence of the system at test-time, without going through a potentially expensive retraining phase.


I am generally excited about more work on demonstrating safety challenges; I think it helps to seed and grow the field in concrete directions. I am particularly excited about the possibility for TanksWorld to demonstrate multi-agent safety problems with agents in direct competition. I feel unsure about whether TanksWorld will be a good demonstration of general problems with human-machine interaction-- intuitively, that seems to me like it would be very difficult to capture and require more complex real-world modeling.

TanksWorld: A Multi-Agent Environment for AI Safety Research

Corban G. Rivera, Olivia Lyons, Arielle Summitt, Ayman Fatima, Ji Pak, William Shao, Robert Chalmers, Aryeh Englander, Edward W. Staley, I-Jeng Wang, Ashley J. Llorens

This paper presents TanksWorld, a simulation environment that attempts to illustrate three important aspects of real-world AI safety challenges: competing performance objectives, human-machine learning, and multi-agent competition. TanksWorld consists of two teams of N vs. N tanks. Tanks move and shoot while navigating in a closed arena with obstacles. Tanks are rewarded for killing opponent tanks and penalized for killing neutral and allied tanks according to a specified reward function. Each tank is controlled by either its own AI or a special policy meant to mimic a 'human' teammate. Each individual tank can only see a small portion of its environment, and must communicate with other teammates to gain more information. The following parameters can be varied to emphasize different research challenges:
- The communication range between tanks -- meant to represent environmental uncertainty.
- The number of neutral tanks and obstacles -- meant to represent the extent to which tanks must care about 'safety', i.e. avoid collateral damage.
- The control policies of teammates -- meant to represent the variability of human-machine teams.


I wrote a short [blog post](https://www.alignmentforum.org/posts/dBMC63hjkc5wPqTC7/human-ai-collaboration) talking about the implications of the work. Briefly, there are three potential impacts. First, it seems generically useful to understand how to coordinate with an unknown agent. Second, it is specifically useful for scaling up <@assistance games@>(@Cooperative Inverse Reinforcement Learning@), which are intractable to solve optimally. Finally, it can lead to more ML researchers focusing on solving problems with real humans, which may lead to us finding and solving other problems that will need to be solved in order to build aligned AI systems.

Paper: On the Utility of Learning about Humans for Human-AI Coordination

Collaborating with Humans Requires Understanding Them

Micah Carroll, Rohin Shah, Mark K. Ho, Thomas L. Griffiths, Sanjit A. Seshia, Pieter Abbeel, Anca Dragan

_Note: I am second author on this paper._ Self-play agents (like those used to play <@Dota@>(@OpenAI Five@) and <@Starcraft@>(@AlphaStar: Mastering the Real-Time Strategy Game StarCraft II@)) are very good at coordinating with _themselves_, but not with other agents. They "expect" their partners to be similar to them; they are unable to predict what human partners would do.  In competitive games, this is fine: if the human deviates from optimal play, even if you don't predict it you will still beat them. (Another way of saying this: the minimax theorem guarantees a minimum reward _regardless_ of the opponent.) However, in cooperative settings, things are not so nice: a failure to anticipate your partner's plan can lead to arbitrarily bad outcomes. We demonstrate this with a simple environment that requires strong coordination based on the popular game Overcooked. We show that agents specifically trained to play alongside humans perform much better than self-play or population-based training when paired with humans, both in simulation and with a real user study.


Humans don't play equilibrium policies, since they are often suboptimal. For example, in Overcooked, any equilibrium policy will zip around the layout, rarely waiting, which humans are not capable of doing. However, when you have a very limited dataset of human behavior, the bias provided by the assumption of an equilibrium policy probably does help the agent generalize better than a vanilla imitation learning model, and so this technique might do better when there is not much data.

Learning Existing Social Conventions via Observationally Augmented Self-Play

This paper starts from the same key insight about self-play not working when it needs to generalize to out-of-distribution agents, but then does something different. They assume that the test-time agents are playing an **equilibrium policy**, that is, each agent plays a best response policy assuming all the other policies are fixed. They train their agent using a combination of imitation learning and self-play: the self-play gets them to learn an equilibrium behavior, while the imitation learning pushes them towards the equilibrium that the test-time agents use. They outperform both vanilla self-play and vanilla imitation learning.


PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings

Nicholas Rhinehart, Rowan McAllister, Kris Kitani, Sergey Levine

This paper models a multi-agent self driving car scenario by developing a model of future states conditional on both its own action and the action of multiple humans, and picking the latent-space action that balances between the desiderata of reaching its goal and preferring trajectories seen in the expert multi-agent trajectories its shown (where, e.g., two human agents rarely crash into one another).


Collaborative game specification: arriving at common models in bargaining

A major challenge in cooperative AI is when agents aren’t even playing the same game: perhaps they have <@very different priors@>(@Equilibrium and prior selection problems in multipolar deployment@), or before communication is established they make <@conflicting commitments@>(@The "Commitment Races" problem@), or their opponent’s strategy is [unidentifiable](https://longtermrisk.org/weak-identifiability-and-its-consequences-in-strategic-settings/) (previous summary). This sort of misspecification can lead to arbitrarily bad outcomes.

This post proposes a simple solution called collaborative game specification (CGS). In CGS, we simply add an initial phase in which the agents talk to each other and determine a shared model of the game being played. The agents then act according to the equilibrium in that game. (Choosing an appropriate equilibrium notion can be part of the talking phase.)

There is of course an incentive for each agent to lie about their model of the game, in order to get an outcome more favorable to them. In order to combat this incentive, agents must also refuse to continue with CGS if the other player’s model is too different from their own (which is some evidence that they are lying in order to get a better outcome for themselves).


I see this paper as a well-organized literature review of the many reasons why you _don’t_ want to handle AI alignment by finding the “true human utility function” or the “aggregated preferences of humanity” and then encoding them into the AI: there’s a myriad of challenges in even finding such an object. (A separate objection, out of scope for this paper, is that even if we did have such an object, we don’t know how to encode that goal into an AI system.)

You might then reasonably ask what we should be doing instead. I see the goal of AI _alignment_ as figuring out how, given a fuzzy but relatively well-specified task, to build an AI system that is reliably pursuing that task, in the way that we intended it to, but at a capability level beyond that of humans. This does not give you the ability to leave the future in the AI’s hands, but it would defuse the central (to me) argument for AI risk: that an AI system might be adversarially optimizing against you. (Though to be clear, there are still <@other risks@>(@The Main Sources of AI Risk?@) to consider.)

Social choice ethics in artificial intelligence

If we want to program ethics into an AI system, should we do so by aggregating the ethical views of existing humans? This is often justified on procedural grounds: “everyone gets to affect the outcome”, or by abstention: “AI designers don’t have to think about ethics; the AI will deal with that”. (There is also a wisdom of the crowds justification, though this presupposes that there is some notion of “better” ethics independent of humans; which is out of scope for the paper.)

However, actually implementing an aggregative procedure requires three major design decisions: 1) _standing_, that is, whose views should be aggregated, 2) _measurement_, that is, how we determine what their ethical views are, and 3) _aggregation_, that is, how the views are put together into a whole. All of these are challenging.

For standing, we have to determine whom to include. Should we include children, psychopaths, non-human animals, ecosystems, future generations, and other AI systems? We must determine this ahead of time, since once we have decided on a social choice system, that system will then determine whose preferences are counted -- we can’t just modify it later.

For measurement, we have to back out human values somehow, which is quite a challenge given that humans have all sorts of cognitive biases and give different answers depending on the context. (See also <@ambitious value learning@>(@What is ambitious value learning?@) and subsequent posts in the sequence.)

For aggregation, the problems are well known and studied in the field of social choice theory. Some famous impossibility results include [Arrow’s theorem](https://en.wikipedia.org/wiki/Arrow%27s_impossibility_theorem) and the [Gibbard-Satterthwaite theorem](https://en.wikipedia.org/wiki/Gibbard%E2%80%93Satterthwaite_theorem).


I really liked this podcast: I definitely hadn’t understood Critch’s full reasons for doing this work. I didn’t include all the points in the summary, so I recommend you listen to it in addition to this summary.

AXRP 3: Negotiable Reinforcement Learning

This podcast centers on <@negotiable RL@>(@Negotiable Reinforcement Learning for Pareto Optimal Sequential Decision-Making@), which studies how to aggregate preferences of multiple principals (humans) who have _different beliefs_. In the case where the principals have the same beliefs, Harsanyi’s utilitarianism theorem tells us that any reasonable method of aggregating preferences will end up optimizing some linear combination of the principals’ utility functions. In the case of differing beliefs, this paper proves that every Pareto optimal policy must be optimizing some linear combination of the principals’ utility functions, except that over time the weights are modified based on how well the principals’ beliefs model reality. Intuitively, the principals are both agreeing to the contract “the AI will optimize more for the person whose beliefs are more correct”; since each principal believes their own beliefs, they are both happy with this contract.

Most of the podcast is about the motivation and reason for writing this paper. Critch envisions a world in which people and AI systems must cooperate rather than fight, and this paper can be thought of as a study in how people can maximize cooperation. Unfortunately, it turns out that the cooperation-maximizing approach ends up being _unfair_: people whose beliefs are incorrect end up getting penalized (in terms of actual outcomes, rather than what they believe will happen).

More broadly, Critch hopes that this will spur more research into how parties with different beliefs can share control of AI systems: this seems important for AI to go well in the future.


What counts as defection?

We often talk about cooperating and defecting in general-sum games. This post proposes that we say that a player P has defected against a coalition C (that includes P) currently playing a strategy S when P deviates from the strategy S in a way that increases his or her own personal utility, but decreases the (weighted) average utility of the coalition. It shows that this definition has several nice intuitive properties: it implies that defection cannot exist in common-payoff games, uniformly weighted constant-sum games, or arbitrary games with a Nash equilibrium strategy. A Pareto improvement can also never be defection. It then goes on to show the opportunity for defection can exist in the Prisoner’s dilemma, Stag hunt, and Chicken (whether it exists depends on the specific payoff matrices).


Multi-Agent Imitation Learning for Driving Simulation


Multiagent GAIL


In practice, I expect users are unlikely to want this, for similar reasons to why we don't bet on every belief difference we have with other people, even though it's net positive in expectation. But we do need to handle this somehow. An interesting scenario to consider is when a regular person has different beliefs from an expert. Does the regular person have to defer exactly to the judgments of the expert to avoid being exploited? What if there's a regular person and _two_ experts who disagree with each other?

Negotiable Reinforcement Learning for Pareto Optimal Sequential Decision-Making

Nishant Desai, Andrew Critch, and Stuart Russell

As AI systems are deployed in the real world, they will have to navigate value differences between different users. Regardless of our beliefs on how they should trade off between users, we at least want our AI systems to be Pareto optimal -- if there is some way to help one user without harming any other user, it should do it. A classic result from social choice theory, Harsanyi's theorem, states that a Pareto optimal agent must be optimizing a weighted linear combination of the users' utility functions. (The weights determine how to trade off between users.) However, this assumes that the users all have the same _beliefs_. When users have different beliefs, we can get even better outcomes if the users bet on their beliefs. If I'm sure that our robot will make a chocolate chip cookie, while you're sure its going to be a brownie, rather than splitting whatever the robot brings, we can agree that if it's a cookie I get it, whereas if it's a brownie you get it, and this makes both of us better off by our own beliefs. This paper shows that a Pareto optimal agent helping users with different beliefs must be optimizing a weighted linear combination of the user's utility functions, where the weights are proportional to how accurate the user's beliefs turn out to be. It's as if the users bet with each other about their beliefs, with the stakes being the assistance of the agent.


These are indeed pretty hard problems in any non-competitive game. While this post takes the framing of considering optimal principals and/or agents (and so considers Bayesian strategies in which only the prior and choice of equilibrium are free variables), I prefer the framing taken in <@our paper@>(@Collaborating with Humans Requires Understanding Them@): the issue is primarily that the optimal thing for you to do depends strongly on who your partner is, but you may not have a good understanding of who your partner is, and if you're wrong you can do arbitrarily poorly.

Equilibrium and prior selection problems in multipolar deployment

Consider the scenario in which two principals with different terminal goals will separately develop and deploy learning agents, that will then act on their behalf. Let us call this a _learning game_, in which the "players" are the principals, and the actions are the agents developed.

One strategy for this game is for the principals to first agree on a "fair" joint welfare function, such that they and their agents are then licensed to punish the other agent if they take actions that deviate from this welfare function. Ideally, this would lead to the agents jointly optimizing the welfare function (while being on the lookout for defection).

There still remain two coordination problems. First, there is an _equilibrium selection problem_: if the two deployed learning agents are Nash strategies from _different_ equilibria, payoffs can be arbitrarily bad. Second, there is a _prior selection problem_: given that there are many reasonable priors that the learning agents could have, if they end up with different priors from each other, outcomes can again be quite bad, especially in the context of <@threats@>(@Cooperation, Conflict, and Transformative Artificial Intelligence: A Research Agenda@).


Scalable Centralized Deep Multi-Agent Reinforcement Learning via Policy Gradients

Arbaaz Khan, Clark Zhang, Daniel D. Lee, Vijay Kumar, Alejandro Ribeiro


Learning to Teach in Cooperative Multiagent Reinforcement Learning

Shayegan Omidshafiei, Dong-Ki Kim, Miao Liu, Gerald Tesauro, Matthew Riemer, Christopher Amato, Murray Campbell, Jonathan P. How