If multi-agent learning is the answer, what is the question?

Introduction The topic of learning in multi-agent systems, or multi-agent learning (MAL henceforth), has a long history in game theory, almost as long as the history of game theory itself.  2  As early as 1951, fictitious play  [10]  was proposed as a learning algorithm for computing equilibria in games and there have been proposals for how to evaluate the success of learning rules going back to  [23]  and  [5] . Since that time hundreds, if not thousands, of articles have been published on the topic, and at least two books (  [20]  and  [54] ). In Artificial Intelligence (AI) the history of single-agent learning is as rich if not richer, with thousands of articles, many books, and some very compelling applications in a variety of fields (for some examples see  [29, 40] , or  [50] ). While it is only in recent years that AI has branched into the multi-agent aspects of learning, it has done so with something of a vengeance. If in 2003 one could describe the AI literature on MAL by enumerating the relevant articles, today this is no longer possible. The leading conferences routinely feature articles on MAL, as do the journals.  3  While the AI literature maintains a certain flavor that distinguishes it from the game theoretic literature, the commonalities are greater than the differences. Indeed, alongside the area of mechanism design, and perhaps the computational questions surrounding solution concepts such as the Nash equilibrium, MAL is today arguably one of the most fertile interaction grounds between computer science and game theory. The MAL research in both fields has produced some inspiring results. We will not repeat them here, since we cannot be comprehensive in this article, but nothing we say subsequently should be interpreted as belittling the achievements in the area. Yet alongside these successes there are some indications that it could be useful to take a step back and ask a few basic questions about the area of MAL. One surface indication is the presence of quite a number of frustrating dead ends. For example, the AI literature attempting to extend Bellman-style single-agent reinforcement learning techniques (in particular, Q-learning  [53] ) to the multi-agent setting, has fared well in zero-sum repeated games (e.g.,  [36]  and  [38] ) as well as common-payoff (or 'team') repeated games (e.g.,  [14, 31, 52] ), but less well in general-sum stochastic games (e.g.,  [21, 26, 37] ) (for the reader unfamiliar with this line of work, we cover it briefly in Section 4). Indeed, upon close examination, it becomes clear that the very foundations of MAL could benefit from explicit discussion. What exact question or questions is MAL addressing? What are the yardsticks by which to measure answers to these questions? The present article focuses on these foundational questions. To start with the punch line, following an extensive look at the literature we have reached two conclusions: • There are several different agendas being pursued in the MAL literature. They are often left implicit and conflated; the result is that it is hard to evaluate and compare results. • We ourselves can identify and make sense of five distinct research agendas. Not all work in the field falls into one of the five agendas we identify. This is not necessarily a critique of work that doesn't; it simply means that one must identify yet other well-motivated and well defined problems addressed by that work. We expect that as a result of our throwing down the gauntlet additional such problems will be defined, but also that some past work will be re-evaluated and reconstructed. Certainly we hope that future work will always be conducted and evaluated against well-defined criteria, guided by this article and the discussion engendered by it among our colleagues in AI and game theory. In general we view this article not as a final statement but as the start of a discussion. In order to get to the punch line outlined above, we proceed as follows. In the next section we define the formal setting on which we focus. In Section 3 we illustrate why the question of learning in multi-agent settings is inherently more complex than in the single-agent setting, and why it places a stress on basic game theoretic notions. In Section 4 we provide some concrete examples of MAL approaches from both game theory and AI. This is anything but a comprehensive coverage of the area, and the selection is not a value judgment. Our intention is to anchor the discussion in something concrete for the benefit of the reader who is not familiar with the area, and-within the formal confines we discuss in Section 2-the examples span the space of MAL reasonably well. In Section 5 we identify five different agendas that we see (usually) implicit in the literature, and which we argue should be made explicit and teased apart. We end in Section 6 with a summary of the main points made in this article. A final remark is in order. The reader may find some of the material in the next three sections basic or obvious; different readers will probably find different parts so. We don't mean to insult anyone's intelligence, but we err on the side of explicitness for two reasons. First, this article is addressed to at least two different communities with somewhat different backgrounds. Second, our goal is to contribute to the clarification of foundational issues; we don't want to be guilty of vagueness ourselves.
 The formal setting We will couch our discussion in the formal setting of stochastic games (aka Markov games). Most of the MAL literature adopts this setting, and indeed most of it focuses on the even more narrow class of repeated games. Furthermore, stochastic games also generalize Markov Decision Problems (MDPs), the setting from which much of the relevant learning literature in AI originates. These are defined as follows. A stochastic game can be represented as a tuple: (N, S, A, R, T ). N is a set of agents indexed 1, . . . , n. S is a set of n-agent stage games. A = A 1 , . . . , A n , with A i the set of actions (or pure strategies) of agent i (note that we assume the agent has the same strategy space in all games; this is a notational convenience, but not a substantive restriction). R = R 1 , . . . , R n , with R i : S × A → R giving the immediate reward function of agent i for stage game S. T : S × A → Π(S) is a stochastic transition function, specifying the probability of the next stage game to be played based on the game just played and the actions taken in it. We also need to define a way for each agent to aggregate the set of immediate rewards received in each state. For finitely repeated games we can simply use the sum or average, while for infinite games the most common approaches are to use either the limit average or the sum of discounted awards ∞ t=1 δ t r t , where r t is the reward received at time t. A repeated game is a stochastic game with only one stage game, while an MDP is a stochastic game with only one agent. While most of the MAL literature lives happily in this setting, we would be remiss not to acknowledge the literature that does not. Certainly one could discuss learning in the context of extensive-form games of incomplete and/or imperfect information (cf.  [28] ). We don't dwell on those since it would distract from the main discussion, and since the lessons we draw from our setting will apply there as well. Although we will not specifically include them, we also intend our comments to apply at a general level to large population games and evolutionary models, and particularly replicator dynamics (RD)  [47]  and evolutionary stable strategies (ESS)  [49] . These are defined as follows. The replicator dynamic model assumes a population of homogeneous agents each of which continuously plays a two-player game against every other agent. Formally the setting can be expressed as a tuple (A, P 0 , R). A is the set of possible pure strategies/actions for the agents indexed 1, . . . , m. P 0 is the initial distribution of agents across possible strategies, m i=1 P 0 (i) = 1. R : A × A → R is the immediate reward function for each agent with R(a, a ) giving the reward for an agent playing strategy a against another agent playing strategy a . The population then changes proportions according to how the reward for each strategy compares to the average reward: dt (P t (a)) = P t (a)[u t (a) − u * t ], where u t (a) = a P t (a )R(a, a ) and u * t = a P t (a)u t (a). A strategy a is then defined to be an evolutionary stable strategy if and only if for some > 0 and for all other strategies a , R(a, (1 − )a + a ) > R(a , (1 − )a + a ). As the names suggest, one way to interpret these settings is as building on population genetics, that is, as representing a large population undergoing frequent pairwise interactions. An alternative interpretation however is as a repeated game between two agents, with the distribution of strategies in the population representing the agent's mixed strategy (in the homogeneous definition above the two agents have the same mixed strategy, but there exist more general definitions with more than two agents and with non-identical strategies). The second interpretation reduces the setting to the one we discuss. The first bears more discussion, and we do it briefly in Section 4. And so we stay with the framework of stochastic games. What is there to learn in these games? Here we need to be explicit about some aspects of stochastic games that were glossed over so far. Do the agents know the stochastic game, including the stage games and the transition probabilities? If not, do they at least know the specific game being played at each stage, or only the actions available to them? What do they see after each stage game has been played-only their own rewards, or also the actions played by the other agent(s)? Do they perhaps magically see the other agent(s)' mixed strategy in the stage game? And so on. In general, games may be known or not, play may be observable or not, and so on. We will focus on known, fully observable games, where the other agent's strategy (or agents' strategies) is not known a priori (though in some case there is a prior distribution over it). In our restricted setting there two possible things to learn. First, the agent can learn the opponent's (or opponents') strategy (or strategies), so that the agent can then devise a best (or at least a good) response. Alternatively, the agent can learn a strategy of his own that does well against the opponents, without explicitly learning the opponent's strategy. The first is sometimes called model-based learning, and the second modelfree learning. In broader settings there is more to learn. In particular, with unknown games, one can learn the game itself. Some will argue the restricted setting is not a true learning setting, but (a) much of the current work on MAL, particularly in game theory, takes place in this setting, and (b) the foundational issues we wish to tackle surface already here. In particular, our comments are intended to also apply to the work in the AI literature on games with unknown payoffs, work which builds on the success of learning in unknown MDPs. We will have more to say about the nature of 'learning' in the setting of stochastic games in the following sections.
 On some special characteristics of multi-agent learning Before launching into specifics, we wish to highlight the special nature of MAL. There are two messages we would like to get across, one aimed at AI researchers specifically and one more broadly. Both lessons can be gleaned from simple and well-known examples. Consider the game described in Fig.  1 . In this game the row player has a strictly dominant strategy, Down, and so seemingly there is not much more to say about this game. But now imagine a repeated version of this game. If the row player indeed repeatedly plays Down, assuming the column player is paying any attention, he (the column player) will start responding with Left, and the two will end up with a repeated (Down, Left) play. If, however, the row player starts repeatedly playing Up, and again assuming the column player is awake, he may instead start responding by playing Right, and the two players will end up with a repeated (Up, Right) play. The lesson from this is simple yet profound: In a multi-agent setting one cannot separate learning from teaching. In this example, by playing his dominated strategy, the row player taught the column player to play in a way that benefits both. Indeed, for this reason it might be more appropriate to speak more neutrally about multi-agent adaptation rather than learning. We will not fight this linguistic battle, but the point remains important, especially for computer scientists who are less accustomed to thinking about interactive considerations than game theorists. In particular, it follows there is no a priori reason to expect that machine learning techniques that have proved successful in AI for single-agent settings will also prove relevant in the multi-agent setting. The second lesson we draw from the well-known game of Rochambeau, or Rock-Paper-Scissors, given in Fig.  2 . As is well known, this zero-sum game has a unique Nash equilibrium in which each player randomizes uniformly among the three strategies. One could conclude that there is not much more to say about the game. But suppose you entered a Rochambeau tournament. Would you simply adopt the equilibrium strategy? If you did, you would not win the competition. This is no idle speculation; such competitions take place routinely. For example, starting in 2002, the World Rock Papers Scissors Society (WRPS) standardized a set of rules for international play and has overseen annual International World Championships as well as many regional and national events throughout the year. These championships have been attended by players from around the world and have attracted widespread international media attention. The winners are never equilibrium players. For example, on October 25th, 2005, 495 people entered the competition in Toronto from countries as diverse as Norway, Northern Ireland, the Cayman Islands, Australia, New Zealand and the UK. The winner was Toronto Lawyer Andrew Bergel, who beat These tournaments of course are not a perfect match with the formal model of repeated games. However, we include the example not only for entertainment value. The rules of the RPS tournaments call for 'matches' between players, each match consisting of several 'games', where a game is a single play of RPS. The early matches adopted a "best of three of three" format, meaning that the player who wins a best of three set garners one point and requires two points to take the match. The Semi-Finals and the Final Match used the "best of three of five" format, meaning that the player who wins a best of three set garners one point and requires three points to take the match. And so the competition really consisted of a series of repeated games, some of them longer than others.  4  Entertainment aside, what do we learn from this? We believe that this is a cautioning tale regarding the predictive or prescriptive role of equilibria in complex games, and in particular in repeated games. There are many examples of games with complex strategy spaces, in which equilibrium analysis plays little or no role-including familiar parlor games, or the Trading Agent Competition (TAC), a computerized trading competition.  5  The strategy space in a repeated game (or more generally a stochastic game) is immense-all mappings from past history to mixed strategies in the stage game. In such complex games it is not reasonable to expect that players contemplate the entire strategy space-their own or that of the opponent(s). Thus, (e.g., Nash) equilibria don't play here as great a predictive or prescriptive role. Our cautioning words should be viewed as countering the default blind adoption of equilibria as the driving concept in complex games, but not as a sweeping statement against the relevance of equilibria in some cases. The simpler the stage game, and the longer its repetition, the more instructive are the equilibria. Indeed, despite our example above, we do believe that if only two players play a repeated RPS game for long enough, they will tend to converge to the equilibrium strategy (this is particularly true of computer programs, that don't share the human difficulty with throwing a mental die). Even in more complex games there are examples where computer calculation of approximate equilibria within a restricted strategy space provided valuable guidance in constructing effective strategies. This includes the game of Poker (  [33]  and  [4] ), and even, as an exception to the general rule we mentioned, one program that competed in the Trading Agent Competition  [13] . Our point has only been that in the context of complex games, so-called "bounded rationality", or the deviation from the ideal behavior of omniscient agents, is not an esoteric phenomenon to be brushed aside.
 A (very partial) sample of MAL work To make the discussion concrete, it is useful to look at MAL work over the years. The selection that follows is representative but very partial; no value judgment or other bias are intended by this selection. The reader familiar with the literature may wish to skip to Section 4.3, where we make some general subjective comments. Unless we indicate otherwise, our examples are drawn from the special case of repeated, two-person games (as opposed to stochastic, n-player games). We do this both for ease of exposition, and because the bulk of the literature indeed focuses on this special case. We divide the coverage into three parts: techniques, results, and commentary.
 Some MAL techniques We will discuss three classes of techniques-one representative of work in game theory, one more typical of work in AI, and one that seems to have drawn equal attention from both communities. entertaining reading; of note is the restriction of the strategy space to Rock, Paper and Scissors, and explicitly ruling out others: "Any use of Dynamite, Bird, Well, Spock, Water, Match, Fire, God, Lightning, Bomb, Texas Longhorn, or other non-sanctioned throws, will result in automatic disqualification". The overview of the RPS society and its tournaments is adapted from the inimitable Wikipedia, the collaborative online encyclopedia, as available on January 2, 2006. Wikipedia goes on to list the champions since 2002; we note without comment that they are all male Torontonians. The results of the specific competition cited are drawn from the online edition of the Boise Weekly dated November 2, 2005. The Boise Weekly starts the piece with "If it weren't true, we wouldn't report on it".  5  http://tac.eecs.umich.edu.
 Model-based approaches The first approach to learning we discuss, which is common in the game theory literature, is the model-based one. It adopts the following general scheme: 1. Start with some model of the opponent's strategy. 2. Compute and play the best response. 3. Observe the opponent's play and update your model of her strategy. 4. Goto step 2. Among the earliest, and probably the best-known, instance of this scheme is fictitious play  [10] . The model is simply a count of the plays by the opponent in the past. The opponent is assumed to be playing a stationary strategy, and the observed frequencies are taken to represent the opponent's mixed strategy. Thus after five repetitions of the Rochambeau game in which the opponent played (R, S, P , R, P ), the current model of her mixed strategy is (R = 0.4, P = 0.4, S = 0.2). There exist many variants of the general scheme, for example those in which one does not play the exact best response in step 2. This is typically accomplished by assigning a probability of playing each pure strategy, assigning the best response the highest probability, but allowing some chance of playing any of the strategies. A number of proposals have been made of different ways to assign these probabilities such as smooth fictitious play  [18]  and exponential fictitious play  [19] . A more sophisticated version of the same scheme is seen in rational learning  [30] . The model is a distribution over the repeated-game strategies. One starts with some prior distribution; for example, in a repeated Rochambeau game, the prior could state that with probability 0.5 the opponent repeatedly plays the equilibrium strategy of the stage game, and, for all k > 1, with probability 2 −k she plays R k times and then reverts to the repeated equilibrium strategy. After each play, the model is updated to be the posterior obtained by Bayesian conditioning of the previous model. For instance, in our example, after the first non-R play of the opponent, the posterior places probability 1 on the repeated equilibrium play.
 Model-free approaches An entirely different approach that has been commonly pursued in the AI literature  [29] , is the model-free one, which avoids building an explicit model of the opponent's strategy. Instead, over time one learns how well one's own various possible actions fare. This work takes place under the general heading of reinforcement learning,  6  and most approaches have their roots in the Bellman equations  [3] . The basic algorithm for solving for the best policy in a known MDP starts by initializing a value function, V 0 : S → R, with a value for each state in the MDP. The value function can then be iteratively updated using the Bellman equation: V k+1 ← R(s) + γ max a s T (s, a, s )V k (s ) The optimal policy can then be obtained by selecting the action, a, at each state, s, that maximizes the expected value: s T (s, a, s )V k (s ). Much of the work in AI has focused strategies for rapid convergence, on very large MDPs, and in particular on unknown and partially observable MDPs. While this is not our focus, we do briefly discuss the unknown case, since this is where the literature leading to many of the current approaches for stochastic games originated. For MDPs with unknown reward and transition functions, the Q-learning algorithm  [53]  can be used to compute an optimal policy. Q(s, a) ← (1 − α t )Q(s, a) + α t R(s, a) + γ V (s ) V (s) ← max a∈A Q(s, a) As is well known, with certain assumptions about the way in which actions are selected at each state over time and constraints on the learning rate schedule, α t , Q-learning can be shown to converge to the optimal value function V * . The Q-learning algorithm can be extended to the multi-agent stochastic game setting by having each agent simply ignore the other agents and pretend that the environment is passive: Q i (s, a i ) ← (1 − α t )Q i (s, a i ) + α t R i (s, a) + γ V i (s ) V i (s) ← max a i ∈A i Q i (s, a i ) Several authors have tested variations of the basic Q-learning algorithm for MAL (e.g.,  [48] ). However, this approach ignores the multi-agent nature of the setting entirely. The Q-values are updated without regard for the actions selected by the other agents. While this can be justified when the opponents' distributions of actions are stationary, it can fail when an opponent may adapt its choice of actions based on the past history of the game. A first step in addressing this problem is to define the Q-values as a function of all the agents' actions: Q i (s, a) ← (1 − α)Q i (s, a) + α R i (s, a) + γ V i (s ) We are however left with the question of how to update V , given the more complex nature of the Q-values. For (by definition, two-player) zero-sum SGs, Littman suggests the minimax-Q learning algorithm, in which V is updated with the minimax of the Q values  [36] : V 1 (s) ← max P 1 ∈Π(A 1 ) min a 2 ∈A 2 a 1 ∈A 1 P 1 (a 1 )Q 1 s, (a 1 , a 2 ) Later work (such as the joint-action learners in  [14]  and the Friend-or-Foe Q algorithm in  [37] ) proposed other update rules for the Q and V functions focusing on the special case of common-payoff (or 'team') games. A stage game is common-payoff if at each outcome all agents receive the same payoff. The payoff is in general different in different outcomes, and thus the agents' problem is that of coordination; indeed these are also called games of pure coordination. The work on zero-sum and common-payoff games continues to be refined and extended (e.g.,  [8, 32, 34, 52] ). Much of this work has concentrated on provably optimal tradeoffs between exploration and exploitation in unknown, zerosum games; this is a fascinating topic, but not germane to our focus. More relevant are the most recent efforts in this line of research to extend the "Bellman heritage" to general-sum games (e.g., Nash-Q by  [25]  and CE-Q by  [21] ). We do not cover these for two reasons: The description is more involved, and the results have been less satisfactory; more on the latter below.
 Regret minimization approaches Our third and final example of prior work in MAL is no-regret learning. It is an interesting example for two reasons. First, it has some unique properties that distinguish it from the work above. Second, both the AI and game theory communities appear to have converged on it independently. The basic idea goes back to early work on how to evaluate the success of learning rules  [5, 23] , and has since been extended and rediscovered numerous times over the years under the names of universal consistency, no-regret learning, and the Bayes envelope (see  [16]  for an overview of this history). We will describe the algorithm proposed in  [24]  as a representative of this body of work. We start by defining the regret, r t i (a j , s i ) of agent i for playing the sequence of actions s i instead of playing action a j , given that the opponents played the sequence s −i . r t i (a j , s i |s −i ) = t k=1 R a j , s k −i − R s k i , s k −i The agent then selects each of its actions with probability proportional to max(r t i (a j , s i ), 0) at each time step t + 1. Recently, these ideas have also been adopted by researchers in the computer science community (e.g.,  [17, 27, 55] ). Note that the application of approaches based on regret minimization has been restricted to the case of repeated games. The difficulties of extending this concept to stochastic games are discussed in  [39] .
 Some typical results One sees at least three kinds of results in the literature regarding the learning algorithms presented above, and others like them. These are: 1. Convergence of the strategy profile to an (e.g., Nash) equilibrium of the stage game in self play (that is, when all agents adopt the learning procedure under consideration). 2. Successful learning of an opponent's strategy (or opponents' strategies). 3. Obtaining payoffs that exceed a specified threshold. Each of these types comes in many flavors; here are some examples. The first type is perhaps the most common in the literature, in both game theory and AI. For example, while fictitious play does not in general converge to a Nash equilibrium of the stage game, the distribution of its play can be shown to converge to an equilibrium in zero-sum games  [46] , 2 × 2 games with generic payoffs  [41] , or games that can be solved by iterated elimination of strictly dominated strategies  [42] . Similarly in AI, in  [38]  minimax-Q learning is proven to converge in the limit to the correct Q-values for any zero-sum game, guaranteeing convergence to a Nash equilibrium in self-play. This result makes the standard assumptions of infinite exploration and the conditions on learning rates used in proofs of convergence for single-agent Q-learning. Claus and Boutilier  [14]  conjecture that both single-agent Q-learners and the belief-based joint action learners they proposed converge to an equilibrium in common payoff games under the conditions of self-play and decreasing exploration, but do not offer a formal proof. Friend-or-Foe Q and Nash-Q were both shown to converge to a Nash equilibrium in a set of games that are a slight generalization on the set of zero-sum and common payoff games. Rational learning exemplifies results of the second type. The convergence shown is to correct beliefs about the opponent's repeated game strategy; thus it follows that, since each agent adopts a best response to their beliefs about the other agent, in the limit the agents will converge to a Nash equilibrium of the repeated game. This is an impressive result, but it is limited by two factors; the convergence depends on a very strong assumption of absolute continuity, and the beliefs converged to are only correct with respect to the aspects of history that are observable given the strategies of the agents. This is an involved topic, and the reader is referred to the literature for more details. The literature on no-regret learning provides an example of the third type of result, and has perhaps been the most explicit about criteria for evaluating learning rules. For example, in  [19]  two criteria are suggested. The first is that the learning rule be 'safe', which is defined as the requirement that the learning rule guarantee at least the minimax payoff of the game. (The minimax payoff is the maximum expected value a player can guarantee against any possible opponent.) The second criterion is that the rule should be 'consistent'. In order to be 'consistent', the learning rule must guarantee that it does at least as well as the best response to the empirical distribution of play when playing against an opponent whose play is governed by independent draws from a fixed distribution. They then define 'universal consistency' as the requirement that a learning rule do at least as well as the best response to the empirical distribution regardless of the actual strategy the opponent is employing (this implies both safety and consistency) and show that a modification of the fictitious play algorithm achieves this requirement. In  [20]  they strengthen their requirement by requiring that the learning rule also adapt to simple patterns in the play of its opponent. The requirement of 'universal consistency' is in fact equivalent to requiring that an algorithm exhibit no-regret, generally defined as follows, against all opponents. ∀ > 0, lim t→inf 1 t max a j ∈A i r t i (a j , s i |s −i ) < In both game theory and artificial intelligence, a large number of algorithms have been show to satisfy universal consistency or no-regret requirements. In addition, recent work  [6]  has tried to combine these criteria resulting in GIGA-WoLF, a no-regret algorithm that provably achieves convergence to a Nash equilibrium in self-play for games with two players and two actions per player. Meanwhile, the regret matching algorithm  [24]  described earlier guarantees that the empirical distributions of play converge to the set of correlated equilibria of the game. Other recent work by  [2]  has addressed concerns with only requiring guarantees about the behavior in the limit. Their algorithm is guaranteed to achieve -no-regret payoff guarantees with small polynomial bounds on time and uses only the agent's ability to observe what payoff it receives for each action.
 Some observations and questions We have so far described the work without comment; here we take a step back and ask some questions about this representative work. Our first comment concerns the settings in which the results are presented. While the learning procedures apply broadly, the results for the most part focus on self play (that is, when all agents adopt the learning procedure under consideration). They also tend to focus on games with only two agents. Why does most of the work have this particular focus? Is it technical convenience, or is learning among more than two agents, or among agents using different learning procedures, less relevant for some reason? Our second comment pertains to the nature of the results. With the exception of the work on no-regret learning, the results we described investigate convergence to equilibrium play of the stage game (albeit with various twists). Is this the pertinent yardstick? If the process (say of self play between two agents) does not converge to equilibrium play, should we be disturbed? More generally, and again with the exception of no-regret learning, the work focuses on the play to which the agents converge, not on the payoffs they obtain. Which is the right focus? No-regret learning is distinguished by its starting with criteria for successful learning, rather than a learning procedure. The question one might ask is whether the particular criteria are adequate. In particular, the requirement of consistency ignores the basic lesson regarding learning-vs.-teaching discussed in Section 3. By measuring the performance only against stationary opponents, we do not allow for the possibility of teaching opponents. Thus, for example, in an infinitely repeated Prisoners' Dilemma game, no-regret dictates the strategy of always defecting, precluding the possibility of cooperation (for example, by the mutually reinforcing Tit-For-Tat strategies). Our goal here is not to critique the existing work, but rather to shine a spotlight on the assumptions that have been made, and ask some questions that get at the basic issues addressed, questions which we feel have not been discussed as clearly and as explicitly as they deserve. In the next section we propose an organized way of thinking about these questions.
 Five distinct agendas in multi-agent learning After examining the MAL literature-the work surveyed here and much else-we have reached the conclusion that there are several distinct agendas at play, which are often left implicit and conflated. We believe that a prerequisite for success in the field is to be very explicit about the problem being addressed. We ourselves can identify five distinct possible goals of MAL research. There may well be others, but these are the ones we can identify. They each have a clear motivation and a success criterion that will allow researchers to evaluate new contributions, even if people's judgments may diverge regarding their relative importance or success to date. They can be caricatured as follows: 1. Computational 2. Descriptive 3. Normative 4. Prescriptive, cooperative 5. Prescriptive, non-cooperative We can now consider each of the five in turn. The first agenda is computational in nature. It views learning algorithms as an iterative way to compute properties of the game, such as solution concepts. As an example, fictitious play was originally proposed as a way of computing a sample Nash equilibrium for zero-sum games  [10] , and replicator dynamics has been proposed for computing a sample Nash equilibrium in symmetric games. Other adaptive procedures have been proposed more recently for computing other solution concepts (for example, computing equilibria in local-effect games  [35] ). These tend not to be the most efficient computation methods, but they do sometimes constitute quick-and-dirty methods that can easily be understood and implemented. The second agenda is descriptive-it asks how natural agents learn in the context of other learners. The goal here is to investigate formal models of learning that agree with people's behavior (typically, in laboratory experiments), or possibly with the behaviors of other agents (for example, animals or organizations). This same agenda could also be taken to apply to large-population models, if those are indeed interpreted as representing populations. This problem is clearly an important one, and when taken seriously calls for strong justification of the learning dynamics being studied. One approach is to apply the experimental methodology of the social sciences. There are several good examples of this approach in economics and game theory, for example  [15]  and  [11] . There could be other supports for studying a given learning process. For example, to the extent that one accepts the Bayesian model as at least an idealized model of human decision making, one could justify Kalai and Lehrer's model of rational learning.  7  However, it seems to us that sometimes there is a rush to investigate the convergence properties, motivated by the wish to anchor the central notion of game theory in some process, at the expense of motivating that process rigorously.  8  The centrality of equilibria in game theory underlies the third agenda we identify in MAL, which for lack of a better term we called normative, and which focuses on determining which sets of learning rules are in equilibrium with each other. More precisely, we ask which repeated-game strategies are in equilibrium; it just so happens that in repeated games, most strategies embody a learning rule of some sort. For example, we can ask whether fictitious play and Q-learning, appropriately initialized, are in equilibrium with each other in a repeated Prisoner's Dilemma game. Although one might expect that game theory purists might flock to this approach, there are very few examples of it. In fact, the only example we know originates in AI rather than game theory  [9] , and it is explicitly rejected by at least some game theorists  [18] . We consider it a legitimate normative theory. Its practicality depends on the complexity of the stage game being played and the length of play; in this connection see our discussion of the problematic role of equilibria in Section 3. The last two agendas are prescriptive; they ask how agents should learn. The first of these involves distributed control in dynamic systems. There is sometimes a need or desire to decentralize the control of a system operating in a dynamic environment, and in this case the local controllers must adapt to each other's choices. This direction, which is most naturally modeled as a repeated or stochastic common-payoff (or 'team') game, has attracted much attention in AI in recent years. Proposed approaches can be evaluated based on the value achieved by the joint policy and the resources required, whether in terms of computation, communication, or time required to learn the policy. In this case there is rarely a role for equilibrium analysis; the agents have no freedom to deviate from the prescribed algorithm. Examples of this work include  [12, 14, 22]  to name a small sample. Researchers interested in this agenda have access to a large body of existing work both within AI and other fields such as control theory and distributed computing. In our final agenda, termed 'prescriptive, non-cooperative', we ask how an agent should act to obtain high reward in the repeated (and more generally, stochastic) game. It thus retains the design stance of AI, asking how to design an optimal (or at least effective) agent for a given environment. It just so happens that this environment is characterized by the types of agents inhabiting it, agents who may do some learning of their own. The objective of this agenda is to identify effective strategies for environments of interest. An effective strategy is one that achieves a high reward in its environment, where one of the main characteristics of this environment is the selected class of possible opponents. This class of opponents should itself be motivated as being reasonable and containing opponents of interest. Convergence to an equilibrium is not a goal in and of itself. There are various possible instantiations of the term 'high reward'. One example is the no-regret line of work which we discussed. It clearly defines what it means for a reward to be high enough (namely, to exhibit no regret); we also discussed the limitations of this criterion. A more recent example, this one from AI, is  [7] . This work puts forward two criteria for any learning algorithm in a multi-agent setting:  (1)  The learning should always converge to a stationary policy, and (2) if the opponent converges to a stationary policy, the algorithm must converge to a best response. There are possible critiques of these precise criteria. They can be too weak since in many cases of interest the opponents will not converge on a stationary strategy. And they can be too strong since attaining a precise best response, without a constraint on the opponent's strategy, is not feasible. But this work, to our knowledge, marks the first time a formal criterion was put forward in AI. A third example of the last agenda is our own work in recent years. In  [45]  we define a criterion parameterized by a class of 'target opponents'; with this parameter we make three requirements of any learning algorithm: (1) (Targeted optimality) The algorithm must achieve an -optimal payoff against any 'target opponent', (2) (Safety) The algorithm must achieve at least the payoff of the security level strategy minus against any other opponent, and (3) (Autocompatibility) The algorithm must perform well in self-play (the precise technical condition is omitted here). We then demonstrate an algorithm that provably meets these criteria when the target set is the set of stationary opponents in general-sum two-player repeated games. More recent work has extended these results to handle opponents whose play is conditional on the recent history of the game  [44]  and settings with more than two players  [51] .
 Summary In this article we have made the following points: 1. Learning in MAS is conceptually, not only technically, challenging. 2. One needs to be crystal clear about the problem being addressed and the associated evaluation criteria. 3. For the field to advance one cannot simply define arbitrary learning strategies, and analyze whether the resulting dynamics converge in certain cases to a Nash equilibrium or some other solution concept of the stage game. This in and of itself is not well motivated. 4. We have identified five coherent agendas. 5. Not all work in the field falls into one of these buckets. This means that either we need more buckets, or some work needs to be revisited or reconstructed so as to be well grounded. There is one last point we would like to make, which didn't have a natural home in the previous sections, but which in our view is important. It regards evaluation methodology. We have focused throughout the article on formal criteria, and indeed believe these to be essential. However, as is well known in computer science, many algorithms that meet formal criteria fail in practice, and vice versa. And so we advocate complementing the formal evaluation with an experimental one. We ourselves have always included a comprehensive bake-off between our proposed algorithms and the other leading contenders across a broad range of games. The algorithms we coded ourselves; the games were drawn from GAMUT, an existing testbed (see  [43]  and http://gamut.stanford.edu). GAMUT is available to the community at large. It would be useful to have a learning-algorithm repository as well. To conclude, we re-emphasize the statement made at the beginning: This article is meant to be the beginning of a discussion in the field, not its end. Fig. 1 .Fig. 2 . 12 Fig. 1. Stackelberg stage game: The payoff for the row player is given first in each cell, with the payoff for the column player following.
			 We acknowledge a simplification of history here. There is definitely MAL work in AI that predates the last few years, though the relative deluge is indeed recent. Similarly, we focus on AI since this is where most of the action is these days, but there are also other areas in computer science that feature MAL material; we mean to include that literature here as well.
			 We do acknowledge some degree of humor in the example. The detailed rules in http://www.rpschamps.com/rules.html make for additional
			 We note that the term is used somewhat differently in the game theory literature.
			 Although this is beyond the scope of this article, we note that the question of whether one can justify the Bayesian approach in an interactive setting goes beyond the familiar contravening experimental data; even the axiomatic justification of the expected-utility approach does not extend naturally to the multi-agent case. 8  It has been noted that game theory is somewhat unusual in having the notion of an equilibrium without associated dynamics that give rise to the equilibrium [1] .

Abstract: The area of learning in multi-agent systems is today one of the most fertile grounds for interaction between game theory and artificial intelligence. We focus on the foundational questions in this interdisciplinary area, and identify several distinct agendas that ought to, we argue, be separated. The goal of this article is to start a discussion in the research community that will result in firmer foundations for the area. 1


How long until human-level AI? Results from an expert assessment

Introduction The field of Artificial Intelligence was founded in the mid 1950s with the aim of constructing "thinking machines"computer systems with human-like general intelligence, and capability equaling and eventually exceeding that of human beings. After a few initial successes, early AI researchers predicted that human-level AI would soon be achieved. In 1965, Herbert Simon predicted that "machines will be capable, within twenty years, of doing any work that a man can do"  [1, p.108] . Two years later, Marvin Minsky predicted that "within a generation… few compartments of intellect will remain outside the machine's realm"  [1, p.109 ]; see also  [2] . But the task proved recalcitrant and the failure of their optimistic timing projections was discouraging. Subsequent timing predictions became far more muted and AI research shifted focus towards producing AI systems demonstrating intelligence on specific tasks in relatively narrow domains. Much of this "narrow AI" research has been dramatically successful, with applications in many areas of science and industry. But one lesson learned through this work is that the magnitude of the difference between narrow, task-specific AI capability and more general, human-like intelligence (referred to throughout this paper as artificial general intelligence, or AGI) including the ability to generalize across divergent contexts and to maintain a sense of self. For most recent and contemporary AI researchers, building AI with human-like general intelligence is simply not a concrete research goal. Over the past few years, however, there has been a resurgence of research interest in the original goals of AI  [3] [4] [5] [6] [7] . This resurgence is based on the assessment that advances in computer hardware, computer science, cognitive psychology, neuroscience, and domainspecific AI put contemporary researchers in a far better position to approach these goals than were the founders of AI. Recent years have seen a growing number of special sessions, workshops, and conferences devoted specifically to topics such as human-level AI and AGI. This includes the annual Biologically Inspired Cognitive Architectures AAAI Symposium, a series of workshops on integrative cognitive architectures; the most recent is the AAAI-10: Special Track on Integrated Intelligence (II)  [8] . Another major AGI event is the international AGI conference series (AGI-08, AGI-09, AGI-10, etc.)  [9] . Some recent convocations of leading AI researchers, such as the AI@50 conference [10], the 50th Anniversary Summit of Artificial Intelligence  [11] , and the annual Singularity Summits at Stanford University, San Francisco, and New York City  [12, 13]  have also focused largely on the AGI problem. A report from AI@50 notes that these days "much of the original optimism is back, driven by rapid progress in artificial intelligence technologies"  [14] . Meanwhile, MIT has recently launched a 5-year initiative aimed at exploring novel approaches to creating advanced thinking machines  [15] . And noted futurist and AI inventor Ray Kurzweil has widely publicized his view that human-level AGI will be achieved within the next few decades  [16, 17]  a view recently supported by Intel Chief Technology Officer Justin Rattner  [18] . We summarize these recent trends as a resurgence in "AI optimism", whereby "optimism" in this context we mean specifically "timing optimism". Timing optimism is of course not the same as optimism about the beneficial effects of human-level AGI. One can be optimistic that AGI will be achieved soon, but pessimistic that the consequences of AGI will be harmful. Likewise, one can be pessimistic that AGI will not be achieved soon, but optimistic that the consequences of AGI will be beneficial, whenever AGI is achieved. Of course, many in the field are less optimistic about the timing of human-level AGI. Not all experts believe that the time is ripe for a return to the original goals. Craig Silverstein of Google (a company that carries out a huge amount of narrow-AI research under the supervision of AI guru Peter Norvig) recently told a reporter that such computers are "hundreds of years away"  [19] . Mark Andreesen, the founder of Netscape, said that "we are no closer to a computer that thinks like a human than we were fifty years ago"  [19] . Some AI researchers, such as Selmer Bringsjord (chair of RPI's Department of Cognitive Science) even doubt that computers will ever display humanlike intelligence  [20] . And the standard university AI curriculum continues to focus almost entirely on narrow AI, with only passing reference to the field's original grand goals. While AI experts with less optimistic views on timing rarely take the time to elaborate the reasons for their views, several relevant perspectives are well articulated in the literature. Hubert Dreyfus's classic critique of AI  [21, 22]  and John Searle's perspective of "biological naturalism"  [23]  both argue that achieving human-like intelligence would require human-like physical embodiment and social context. Philosopher William Hurlbut  [24]  argues that (very roughly speaking) human mind emerges from human brain according to patterns and processes too complex for humans to fully understand anytime in the foreseeable future. Roger Penrose  [25]  and Stuart Hameroff  [26]  believe that human intelligence relies on quantum or quantum gravity phenomena that don't exist in ordinary digital computers. Finally, some experts may believe that there is no fundamental barrier to creating human-level AGI on digital computers, but that it's simply a very hard problem which would require one or more radical breakthroughs so far beyond current understanding that they will probably take the AI community many centuries to solve. Anecdotally, based on conversations with survey participants and others at AGI-09, our impression is that many AI researchers who are timing pessimists don't have any highly specific objection to AGI optimism in mind, but rather feel that there is a large amount of uncertainty involved so that there are probably many large "hidden rocks" along the path to human-level AGI, which might or might not prove insurmountable. If the path to AGI is laden with unforeseen obstacles, then there may be no way to predict when or if the requisite scientific breakthrough might occur. On the other hand, AGI timing optimists generally believe that human-level intelligence is a straightforward although complex "mechanical" process that can be understood by extending today's scientific ideas. If this is correct, then it is reasonable to consider detailed projections regarding how much time will pass until human-level AGI is built. One issue on which there is minimal disagreement is that achievement of AGI would have a radically transformative effect on multiple aspects of science, industry, commerce, government, and arts. In the most extreme case, an intelligence explosion would ensue in which initial AGIs would design and build ever-smarter AGIs, leaving humans in the dust. In this eventuality, superintelligent AGIs might use their vast intelligence to either solve a great many of humanity's problems or to destroy (perhaps inadvertently) their human creators  [27] . But whether the results are good or bad, the achievement of human-level AGI would be among the most important events in human history. It is thus of considerable importance and interest to make the best estimates possible of when and how AGI milestones may occur. Unfortunately, the database for making such estimates is quite limited. We can extrapolate hardware trends such as Moore's Law into the future, but there is no guarantee that these trends will persist, or that the requisite software will accompany them. Nor is it certain that hardware equivalent to the wetware in the human brain will be necessary or sufficient for human-level AI. There may be software paradigms that will permit human-level AGI on sub-human-level hardware, or perhaps more advanced hardware will be required for some reason. Given the limitations of hardware and software data, expert opinion can be an important information source for our understanding of AGI. Expert opinion refers simply to the views and estimates of people who have exceptional knowledge about some topic, in this case AGI. Expert opinion, if carefully elicited, can yield meaningful insights on what is and isn't known about likely developments in a range of phenomena, including AGI. Expert assessment is a particularly insightful methodology whenas is the case with AGIexperts lack consensus on important parameters. We are aware of only two previous studies explicitly exploring expert opinion on the future of AGI. In 2006, a seven-question poll was taken of participants at the AI@50 conference  [28] . Four of the seven questions are particularly relevant. Asked "when will computers be able to simulate every aspect of human intelligence?", 41% said "More than 50 years" and 41% said "Never". Thus approximately 80% of the participants at the conference were AI "timing pessimists" and 20% were "timing optimists". Seventynine percent said that an accurate model of thinking is impossible without further discoveries about brain functioning. Sixty percent of the participants strongly agreed that "AI should take a multidisciplinary approach, incorporating stats, machine learning, linguistics, computer science, cognitive psychology, philosophy, and biology". Finally, 71% said that "statistical/ probabilistic methods are most accurate in representing how the brain works". The other survey, taken in 2007 by futurist entrepreneur Bruce Klein  [29] , was an online survey that garnered 888 responses, asking one question: "When will AI surpass human-level intelligence?" Most of those who chose to respond to this survey were AI "optimists" who believed that human-level artificial intelligence would be achieved during the next half century. The distribution of responses is reproduced in Fig.  1 . While the results show that a substantial number of optimists exist, Klein's study has several important limitations. First, the sample population is broad, containing many non-experts. Second, the study only asks for point estimates for when superhuman AI will occur, neglecting any uncertainty in study participants' beliefs. Finally, the study only asked one question, despite the plethora of other important aspects of AGI. The AI@50 and Klein studies are interesting because they show that significant numbers of experts and interested persons believe that AGI with intelligence at least equaling the human level will exist within upcoming decades. The current study probes more deeply into the thinking of people with substantial AGI expertise. Many (though not all) of these experts can be classified as timing optimists. While their views are not representative of the larger population of AI experts, they are nonetheless worth considering. In many cases, these experts are working seriously on AGI because they believe there is a strong likelihood of achieving success with the fundamental theoretical and hardware tools at hand or likely to be developed in the next few decades. This study is based on an expert assessment survey distributed to attendees of the Artificial General Intelligence 2009 (AGI-09) conference  [30] . This was a gathering of AI researchers and other individuals interested specifically in pursuing theoretical or practical work on AGI. The survey covered the timing of AGI milestones, the effectiveness of various technical approaches to achieving AI, the potential for AGI to help or harm humanity, and the possibility of an AI being conscious.
 Methods Expert assessments have been used for several decades to provide timely information and aid decision making. Expert assessments have been used in the highest levels of study, including the United Nations-sponsored Intergovernmental Panel on Climate Change  [31] . A range of assessment methods exist  [32] [33] [34] . There is no single optimal methodology for assessing a group of experts. Each assessment should be customized to meet the questions of the research, the abilities of the experts, and any other relevant factors. Our study draws heavily on recent expert elicitation methodology that emphasizes highlighting the diversity of expert opinion instead of reaching consensus  [34, p.146-154] . While this expert elicitation has been used primarily for estimation of static parameter values, it has also been used for several technological forecasting study  [35] [36] [37] [38] . Our study used a two-part survey. The first part is a standardized survey instrument distributed at the AGI-09 conference. This survey is adapted from previous expert elicitation surveys, in particular  [39] . It features primarily quantitative questions. Many expert elicitation studies only use a survey along these lines. However, we felt that our quantitative survey results would be insightfully complemented by qualitative follow-up questions. So, the second part of our survey consists of follow-up questions customized for each survey participant and distributed to participants via email. Unlike follow-up questions used in Delphi studies, ours were focused on clarifying and expanding original survey responses instead of working towards consensus. These follow-up questions give us a richer understanding of the meanings and reasons behind the quantitative responses given in the initial survey.
 Initial survey The initial survey was in a single standardized form printed across 18 pages for each of the survey participants. Participants reported spending an average of 28 min on the survey. All questions were quantitative, closed-ended questions, asking for numbers or binary true/false or yes/no answers. However, participants were encouraged to write any clarifying or other relevant remarks anywhere on the survey instrument; most participants took advantage of this option. The full survey instrument that we used for our investigation is available online  [40] . Here we briefly summarize the most important questions that were posed. Extensive empirical research shows that experts and non-experts alike tend to be overconfident in their probability estimates  [34] . In rough terms, this means that for any given parameter, individuals tend to underestimate the probability that the actual value of the parameter is quite different from their best guess estimate. For example, someone might estimate that there is a 90% chance of the parameter values being within some range when that the actual values are within the range only 50% of the time. Our survey took several steps to reduce this overconfidence bias. Before any technical questions were asked, the survey explained overconfidence so as to sensitize participants to be cautious about it when answering questions. Additionally, some questions were worded so as to reduce overconfidence. The first set of questions elicited experts' beliefs about when AI would reach each of four milestones: passing the Turing test, 1 performing Nobel quality work, passing third grade, and becoming superhuman. These specific milestones were selected to span a variety of levels of advanced general intelligence. For each milestone, two question versions were askedwith and without additional fundingmaking for eight total milestone questions. These two versions explore the possibility that when the milestones are reached depends on the resources available to researchers. The amount of additional funding listed, $100 billion per year, is obviously more than could be used for AGI research; the intent with this figure is to ensure that money would not be a scarce resource in this hypothetical AGI development scenario. For each of the eight milestone questions, we asked the respondents to give us estimates representing 10%, 25%, 75%, and 90% confidence intervals, as well as their best estimate dates. The 10% point represents the date in which the expert estimates that there is a 10% chance that the milestone will have been achieved, and so on. We asked experts to estimate the 10% and 90% intervals first in order to reduce overconfidence. Collectively, the five points correspond at least roughly with points on participants' probability density functions for the uncertain milestone achievement dates. The five point approach represents a trade-off between gaining deeper insight into experts' beliefs and demanding more detail from the experts. Subsequent questions requested only single point estimates instead of multiple points on a probability density function. This choice was made to keep the overall survey instrument relatively short. The subsequent questions covered four topics. Three questions asked what embodiment the first AGIs would have: physical or virtual robot bodies or a minimal text-or voice-only embodiment. Eight questions asked what AI software paradigm the first AGIs would be based on: formal neural networks, probability theory, uncertain logic, evolutionary learning, a large hand-coded knowledge-base, mathematical theory, nonlinear dynamical systems, or an integrative design combining multiple paradigms. Three questions asked the likelihood of a strongly negative-to-humanity outcome if the first AGIs were created by: an open-source project, the US military, or a private for-profit software company. Two true/false questions asked if quantum computing or hypercomputing would be required for AGI. Two yes/ no questions asked if AGIs emulating the human brain conceptually or near-exactly would be conscious in the sense that humans are. Finally, 14 questions asked experts to evaluate their own expertise in several subjects: cognitive science, neural networks, probability theory, uncertain logic, expert systems, theories of ethics, evolutionary learning, quantum theory, quantum gravity theory, robotics, virtual worlds, software engineering, computer hardware design, and cognitive neuroscience.
 Follow-up questions After analyzing results of the initial survey, we wrote and distributed follow-up questions customized for each participant. Whereas the initial survey consisted entirely of quantitative, closed-ended questions, the follow-up questions were all qualitative, open-ended questions, asking for prose explanations and clarifications of their initial survey responses. Participants received between two and six questions. While the initial survey responses raised a large number of issues we were interested in following up on, we chose only the most important questions so as to minimize the burden on survey participants' time.
 Study participants Participants were recruited at the AGI-09 conference. Due to the highly specialized nature of the AGI-09 conference, no conference attendees were excluded. Participants thus have a range of levels of expertise, from graduate students to senior researchers. This inclusive approach risked eliciting low-quality responses from non-experts who happen to be at the conference. On the other hand, more exclusive approaches risk missing important ideas or information and also risk biasing results in certain directions. All of the participants in our study displayed a depth of thinking demonstrative of significant AGI expertise. Twenty-one people participated in the study. Nineteen people completed the initial survey at the AGI-09 conference; two completed it later, submitting it via postal mail. Also, 18 people (including both of those who submitted via postal mail) completed the follow-up questions. Such levels of participation are common for expert elicitationone recent study  [38]  had just 7 experts; another study  [41]  had just 12 experts. While a larger number of participants may bring additional information, our participant group is very much adequate to yield meaningful insights. It should be emphasized here that the purpose of our study is ultimately to gain insight about AGI itself, not about AGI experts. Thus our sample of AGI experts must provide rich insights about AGI, but it does not need to be in some way representative of the broader population of AGI experts. After all, there is no guarantee that an accurate assessment of the opinions of the broader on AGI population correlates with the realities of AGI. It is entirely possible that the deepest insights about AGI come from outliers from the broader population of experts. Indeed, the very existence of disagreement among experts is grounds for reflection on the state of our knowledge. As our results make clear, our sample succeeds quite well at providing rich insight about AGI. While we believe our sample to be at least somewhat representative of the broader AGI expert population, this is of secondary importance. Study participants have a broad range of backgrounds and experience, all with significant prior thinking about AGI. Eleven are in academia, including six Ph.D. students, four faculty members, and one visiting scholar, all in AI or allied fields. Three lead research at independent AI research organizations and three do the same at information technology organizations. Two are researchers at major corporations. One holds a high-level administrative position at a relevant non-profit organization. One is a patent attorney. All but four participants reported being actively engaged in conducting AI research. We are not listing the names of participating experts for several reasons. First, prior expert assessments vary in their approach to anonymity. Many studies do not list experts' names at all (e.g.  [42, 43] ). Many other studies list experts' names but do not link specific names to specific responses (e.g.  [35] [36] [37] [38] [39] 41] ). Second, due to the sensitive nature of AGI, several of our experts requested that their names not be listed. The others participated on the condition that their names not be linked to their responses. Listing experts' names and discussing results in this way could enable some readers to link specific names to specific responses.
 Results Although the majority of the experts who participated in our study were clearly AGI timing optimists, there was a significant minority of timing pessimists. While we don't know how the views of study participants compare to those of the rest of the conference attendees, it seems unsurprising to find an optimist majority at an AGI conference. Most AGI pessimists may see little reason to bother attending an AGI conference, although this is not universalsome known AGI pessimists do regularly attend AGI conferences. Finally, all experts in our study, including those who could be classified as timing pessimists, gave at least a 10% chance of some AI milestones being achieved within a few decades. Additionally, the amount of agreement between the experts varied considerably from one question to another. On several questions, experts disagreed stronglymore so than is typically found in expert assessmentsindicating a lack of professional consensus and the presence of multiple competing mental models of the future of AGI. Meanwhile, on other questions, much more agreement existed, though rarely full consensus.
 Milestone timing without additional funding Much of the initial survey and follow-up questions focused on the timing of four AI milestones: passing the Turing test, performing Nobel-quality research, passing the third grade, and gaining dramatically superhuman capabilities. The range of the best guess estimates given for these milestones without massive additional funding is summarized in Fig.  2 . This figure clearly shows the dichotomy between timing optimists and timing pessimists. It also shows that a substantial number of AGI experts expect that key milestones will be reached within upcoming decades. The overlap in the best guess estimates for the milestones was surprising. Using human cognitive development as a model, one might think that being able to do Nobel level science would take much longer than being able to conduct a social conversation, as in the Turing test. It is true that many complex problems can be solved effectively by artificial expert systems. But performing Nobel level science requires more than just complex problem solving: it also requires creativity in the formulation of new research questions and ideas, something that expert systems cannot achieve. Thus we might expect that performing Nobel level science would be more difficult than passing the Turing test. However, some experts thought it would be quite different for an AGI. One expert observed that "making an AGI capable of doing powerful and creative thinking is probably easier than making one that imitates the many, complex behaviors of a human mindmany of which would actually be hindrances when it comes to creating Nobel-quality science". This expert observed that "humans tend to have minds that bore easily, wonder [sic] away from a given mental task, and that care about things such as sexual attraction, all [of] which would probably impede scientific ability, rather that promote it". To successfully emulate a human, a computer might have to disguise many of its abilities, masquerading as being less intelligent, in certain ways, than it actually was. This is one reason that the Turing test milestone might be reached later than the Nobel-quality science milestone. We computed median estimates for each milestone (Table  1 ). The medians would be only slightly lower if these less optimistic respondents were excluded from the analysis. Medians were used, instead of means, because these are less susceptible to distortion via statistical outliers.  2
 Milestone estimates with massive new funding Another interesting result concerns the role of fundingthe predicted impact on the AGI field of a hypothetical $100 billion per year funding infusion. The range of the best guess estimates given for these milestones with massive additional funding is summarized in Fig.  3 . Most experts estimated that a massive funding increase of this nature would cause the AI milestones to be reached sooner. However, for many of these experts, the difference in milestone timing with and without massive funding was quite smalljust a few years. Furthermore, several experts estimated that massive funding would actually cause the AI milestones to be reached later. One reason given for this is that with so much funding, "many scholars would focus on making money and administration" instead of on research. Another reason given is that "massive funding increases corruption in a field and its oppression of dissenting views in the long term". Of those who thought that funding would make little difference, a common reason was that AGI progress requires theoretical breakthroughs from just a few dedicated, capable researchers, something that does not depend on massive funding. Another common reason was that the funding would not be wisely targeted. Several noted that funding could be distributed better if there was a better understanding of what paradigms could produce AGI, but such an understanding is either not presently available or not likely to be understood by those who would distribute funds. Because of the lack of agreement on a single paradigm, several experts recommended that modest amounts of funding should be distributed to a variety of groups following different approaches, instead of large amounts of funding being given to a "Manhattan Project" type crash program following one approach. Only one expert recommended concentrating funds in a fewer number of groups. Similarly, several observed that well-funded efforts guided by a single paradigm had failed in the past, including the Japanese Fifth Generation Computer Systems project. While most experts did not address the question of how to distribute funds across groups, our initial impression is that there would be some consensus for distributing funds to a larger variety of groups. This result could have major implications for how available funds are to be allocated. Several experts gave technical reasons for why funding would not have a significant impact. On this, one said, "AGI requires more theoretical study than real investment." Another said, "I believe the development of AGI's to be more of a tool and evolutionary problem than simply a funding problem. AGI's will be built upon tools that have been developed from previous tools. This evolution in tools will take time. Even with a crash project and massive funding, these tools will still need time to develop and mature." Given that these experts are precisely those who would benefit most from increased funding, their skeptical views of the impact of hypothetical massive funding are very likely sincere.
 Milestone timing: estimates of certainty Since we do not know for sure when milestones will be reached, it is helpful to characterize the uncertainty about this. Some insights can be gained from considering the distributions of best guess estimates of experts (as in Figs.  2 and 3 ) or of broader populations (as in Fig.  1 ). However, each individual has uncertainty about his/her own estimates. These uncertainties are not found from presentations of best guess estimates, but they can be found from the estimates across the range of confidence intervals elicited in our survey. Participants showed a broad range of certainty levels in their timing estimates. Several participants showed little uncertainty, indicating that they believed it was very likely that milestones would be achieved within a narrow range of dates. The participants giving the narrowest ranges were all strong timing optimists. Meanwhile, many of the participants showed much uncertainty. Some of these individuals gave substantial probabilities for both early and late milestone achievement dates. For these individuals, the timing optimist/pessimist dichotomy does not readily apply. Other individuals showing much uncertainty could be classified as timing pessimists in that they considered it most likely that milestones would be achieved much later. The fact that these individuals are not certain about their pessimism could explain why they thought that attending a conference on AGI would be worthwhile. One expert estimated that there was at least a 25% chance that the Turing test would never be achieved, but he also thought that there was a 10% chance that it would be achieved by the year 2020. Another estimated that there was a 25% chance it would not be achieved until the year 3000, but that there was a 10% chance it would happen by 2050. The stronger optimists at the conference were much more certain of their opinions. One estimated that there was a 90% chance that the Turing test would be achieved by 2016 and a 10% chance that it would be achieved as soon as 2010. The range of variation in the estimates for the four milestones without massive additional funding is shown in Fig.  4 . Dates after 2100 are removed to improve resolution in the rest of the plots. There was substantial agreement among the more optimistic participants in the study. Most thought that there was a substantial likelihood that the various milestones would be passed sometime between 2020 and 2040. Sixteen experts gave a date before 2050 as their best estimate for when the Turing test would be passed. Thirteen experts gave a date between 2020 and 2060 as their best estimate for when superhuman AI would be achieved. (The others all estimated later dates, ranging from 2100 to never.) The opinions of the optimists were similar to or perhaps somewhat more optimistic than Kurzweil's well-known projections. As noted above, optimism in this sense is about the timing of AI milestones. It does not imply a belief that achieving AGI would be a good thing. To the contrary, one can be optimistic that AGI will happen soon yet believe that AGI would have negative outcomes. Indeed, several experts reported this pair of beliefs. Results about the likelihood of negative outcomes are discussed further below.
 Milestone order There was little agreement among experts on the order in which the four milestones (Turing test; third grade; Nobel; and superhuman) would be achieved. The only area of consensus was that the superhuman milestone would be achieved either last or at the same time as other milestones. Meanwhile, there was significant divergence regarding the order of the other three milestones. One expert argued that the Nobel milestone would be easier precisely because it is more sophisticated: to pass the Turing test, an AI must "skillfully hide such mental superiorities". Another argued that a Turing test-passing AI needs the same types of intelligence as a Nobel AI "but additionally needs to fake a lot of human idiosyncrasies (irrationality, imperfection, emotions)". Finally, one expert noted that the third grade AI might come first because passing a third grade exam might be achieved "by advances in natural language processing, without actually creating an AI as intelligent as a third-grade child". This diversity of views on milestone order suggests a rich, multidimensional understanding of intelligence. It may be that a range of milestone orderings are possible, depending on how AI development proceeds. The milestone order results highlight the fact that many experts do not consider it likely that the first human-level AGI systems will closely mimic human intelligence. Analogy to human intelligence would suggest that achieving an AGI capable of Nobel level science would take much longer than achieving an AGI capable of conducting a social conversation. However, as discussed above, an AGI would not necessarily mimic human intelligence. This could enable it to achieve the intelligence milestones in other orders.
 Technical approaches Our initial survey asked about eight technical approaches used in constructing AI systems: formal neural networks, probability theory, uncertain logic, evolutionary learning, a large hand-coded knowledge-base, mathematical theory, nonlinear dynamical systems, and integrative designs combining multiple paradigms. For each, it requested a point estimate of the odds that the approach would be critical in the creation of human-level AGI. They survey also asked about the odds that physical robotics, virtual agent control, or minimal text-or voice-based embodiments would play a critical role.  The most striking finding in this section was a strong majority opinion in favor of integrative designs. Of the 18 respondents who answered this question, 13 gave a probability of 0.5 or greater; the mean response across all 18 was 0.63.  3  None of the specific technical approaches mentioned received strong support from any more than a small plurality of respondents. Probability theory received the strongest support, with a mean probability of 0.35 and with 7 out of 20 respondents answering the question giving a probability of 0.5 or greater. The question about robotics vs. other forms of embodiment received a range of responses (Fig.  5 ). There were responses of 0.9, 0.94, 0.89, and 0.60 for physical robot embodiment, but the mean response was only 0.28. The handful of participants who felt robotics was crucial were all relatively optimistic, as seen by their mean date of 2034 for the Turing test without massive additional funding. The virtual agents option received a similar but considerably less strong response, with top responses of 0.80, 0.65, and 0.60 and a mean response of 0.34. While some of the virtual worlds enthusiasts were optimistic about timing, this was much less of a strong pattern than in the robotics case. The preliminary impression one obtains is that a few researchers are highly bullish on robotics as the correct path to AGI in the relatively near term, whereas the rest feel robotics is probably not necessary for AGI. Meanwhile, a larger number of researchers are moderately optimistic about the role of virtual worlds for AGI, but with milder opinions about the importance of the approach.
 Impacts of AGI In science fiction, intelligent computers frequently become dangerous competitors to humanity, sometimes even seeking to exterminate humanity as an inferior life form. Several researchers have recently expressed similar concerns  [27, 44] . Motivated by these concerns, we asked experts to estimate the probability of a negative-to-humanity outcome occurring if an AGI passes the Turing test. Our question was broken into three parts, for each of three possible development scenarios: if the first AGI that can pass the Turing test is created by an open source project, by the United States military, or by a private company focused on commercial profit. This set of questions marked another instance in which the experts lacked consensus (see Fig.  6 ). Five experts estimated a less than 20% chance of a negative outcome, regardless of the development scenario. Four experts estimated a greater than 60% chance of a negative outcome, regardless of the development scenario. Only four experts gave the same estimate for all three development scenarios. Finally, several experts were more concerned about the risk from AGI itself, whereas others were more concerned that AGI could be misused by humans who controlled it. Interesting insights can be found in the experts' orderings of the riskiness of the development scenarios. Of the 11 experts who gave different estimates for each of the three scenarios, 10 gave the private company scenario the middle value. Of these 10, 6 gave the US military scenario the highest value and 4 gave it the lowest value. Thus the open source scenario and the US military scenario tend to perceived as relatively safe or relatively dangerous, but experts are divided on which is which. Experts who estimated that the US military scenario is relatively safe noted that the US military faces strong moral constraints, has experience handling security issues, and is very reluctant to develop technologies that may backfire (such as biological weapons), whereas open source development lacks these features and cannot easily prevent the "deliberate insertion of malicious code". In contrast,  3  In this section the survey did not require that the probabilities all add up to one, because the technical approaches are not necessarily mutually exclusive. When the numbers are normalized such that each participant's set of probabilities add up to one, then only 2 experts give integrated design a probability of 0.5 or greater, but 12 still rank it the most likely, including 1 who gave the highest probability to "none of the above". After normalization, the average for integrated design is 0.32; all other approaches ranged 6% (mathematical theory) and 12% (probability theory). Several experts noted potential impacts of AGI other than the catastrophic. One predicted that "in thirty years, it is likely that virtually all the intellectual work that is done by trained human beings such as doctors, lawyers, scientists, or programmers, can be done by computers for pennies an hour. It is also likely that with AGI the cost of capable robots will drop, drastically decreasing the value of physical labor. Thus, AGI is likely to eliminate almost all of today's decently paying jobs." This would be disruptive, but not necessarily bad. Another expert thought that, "societies could accept and promote the idea that AGI is mankind's greatest invention, providing great wealth, great health, and early access to a long and pleasant retirement for everyone." Indeed, the experts' comments suggested that the potential for this sort of positive outcome is a core motivator for much AGI research.
 Conclusion Despite decades of setbacks, the development of AI with the general intelligence of humans remains an important possibility both for the AI research community and for society at large. Given the paucity of hardware and software data enabling detailed extrapolations regarding the future of such AI, expert opinion is an important source of information. In this article, we have presented the most careful assessment of expert opinion on AGI to date. Given the long history of AI researchers making erroneous predictions, our results must be interpreted cautiously, but they nonetheless add much to our understanding of the future of AGI. In the broadest of terms, our results on AGI timing concur with those of the two previous studies mentioned in the Introduction  [28, 29] . All three studies suggest that significant numbers of interested, informed individuals believe it is likely that AGI at the human level or beyond will occur around the middle of this century, and plausibly even sooner. Due to the greater depth of the questions, our survey also revealed some interesting additional information, such as the disagreement among experts over the likely order of AGI milestones and the relative safety of different AGI development scenarios. The experts' suggestions regarding funding (in particular, skepticism about the importance of funding and a preference for distributing funds among a broad range of research groups) are also potentially valuable. Our results reinforce the idea that there is a significant likelihood of AGI being achieved within upcoming decades. This idea has profound societal implications given the strong potential for AGI to act as a transformative agent, either to massively help humanity or to destroy it. The experts in our study, like others who have commented on the issue, were divided on whether the impacts of AGI would be positive or negative for humanity. Our results add depth to our understanding of AGI impacts by exploring multiple development scenarios. However, the jury remains out regarding how to achieve positive outcomes and avoid negative outcomes. This is a crucial area for future research. Currently, AGI experts hold a rich diversity of views, a situation which challenges our ability to make confident predictions about the future of AGI, but is not surprising given the state of development of the field. It would be interesting to carry out similar studies in future and see how expert views evolve and converge or diverge as AI technology advances. Participants are displayed in the same order as in Fig.  4 , such that Participant 1 in Fig.  6  is the same person as Participant 1 in Fig.  4 . Fig. 1 . 1 Fig.1. Results from the survey taken by Klein [29]  in 2007 asking the question "When will AI surpass human-level intelligence?".
 Fig. 2 . 2 Fig. 2. Milestone best estimate guesses without massive additional funding. Estimates are for when AI would achieve four milestones: the Turing test (horizontal lines), third grade (white), Nobel-quality work (black), and superhuman capability (grey).
 Fig. 3 . 3 Fig. 3. Milestone best estimate guesses with massive additional funding. Estimates are for when AI would achieve four milestones: the Turing test (horizontal lines), third grade (white), Nobel-quality work (black), and superhuman capability (grey).
 Fig. 4 . 4 Fig. 4. Estimates for when AI would achieve four milestones without massive additional funding. Estimates are given at five confidence intervals. Asterisks represent 10% and 90% confidence bounds. Horizontal lines represent 25% and 75% confidence bounds. Diamonds represent best guess estimates. In each plot, participants are ordered by their Turing test best guess estimates.
 Fig. 5 . 5 Fig.5. Number of respondents for different embodiment approaches. Respondents are grouped into probability intervals for each of three different embodiment approaches: physical robot (black), virtual agent (white), and minimal embodiment (grey). Intervals are labeled on the horizontal axis by their lower bound. For example, the interval labeled "0.1" represents the interval 0.1 through 0.199.
 Fig. 6 . 6 Fig. 6. Probability of a negative-to-humanity outcome for different development scenarios. The three development scenarios are if the first AGI that can pass the Turing test is created by an open source project (x's), the United States military (squares), or a private company focused on commercial profit (triangles).Participants are displayed in the same order as in Fig.4, such that Participant 1 in Fig.6is the same person as Participant 1 in Fig.4.
 Table 1 1 Median estimates for each AGI milestone at each level of confidence. Milestone 10% 25% 50% 75% 90% Turing test 2020 2030 2040 2050 2075 Nobel science 2020 2030 2045 2080 2100 Third grade 2020 2025 2030 2045 2075 Superhuman 2025 2035 2045 2080 2100
			 Participants expressed concern that the Turing test milestone is ambiguous, due to the numerous variations of the test. In response to this, several participants specified the Turing test variant their responses are based on. At the time of survey distribution, a verbal suggestion was given to consider the "one hour" rather than the "five minute" Turing test as some potential participants felt the latter could too easily be "gamed" by narrow-AI chatbots without significant general intelligence.
			 In this case, the sample size is so small that it's hard to rigorously speak about "outliers," but it's clear qualitatively that the very long estimates given by some experts, including those who thought that AGI might never be achieved, would have a disproportionate effect upon the means.

Abstract: The development of human-level AI has been a core goal of the AI field since its inception, though at present it occupies only a fraction of the field's efforts. To help understand the viability of this goal, this article presents an assessment of expert opinions regarding humanlevel AI research conducted at AGI-09, a conference for this AI specialty. We found that various experts strongly disagree with each other on certain matters, such as timing and ordering of key milestones. However, we did find that most experts expect human-level AI to be reached within upcoming decades, and all experts give at least some chance that some milestones will be reached within this time. Furthermore, a majority of experts surveyed favor an integrative approach to human-level AI rather than an approach centered on a particular technique. Finally, experts are skeptical about the impact of massive research funding, especially if it is concentrated in relatively few approaches. These results suggest that the possibility of achieving human-level AI in the near term should be given serious consideration.