Karl Tuyls
Julien Pérolat
Marc Lanctot
Georg Ostrovski
Rahul Savani
Joel Z Leibo
Toby Ord
Thore Graepel
Shane Legg

Symmetric Decomposition of Asymmetric Games

can be a very useful tool to analyse the Nash structure and dynamics of several interacting agents in a multiagent system. However, when dealing with asymmetric games the analysis quickly becomes tedious, as in this case we have a coupled system of replicator equations, and changes in the behaviour of one agent immediately change the dynamics in the linked replicator equation describing the behaviour of the other agent, and vice versa. This paper sheds new light on asymmetric games, and reveals a number of theorems, previously unknown, that allow for a more elegant analysis of asymmetric multiagent games. The major innovation is that we decouple asymmetric games in their symmetric counterparts, which can be studied in a symmetric fashion using symmetric replicator dynamics. The Nash equilibria of these symmetric counterparts are formally related to the Nash equilibria of the original asymmetric game, and as such provide us with a means to analyse the asymmetric game using its symmetric counterparts. Note that we do not consider asymmetric replicator dynamics in which both intra-species (within a population) and inter-species interactions (between different populations) take place  19  , but we only consider inter-species interactions in which two different roles interact, i.e., truly asymmetric games  20  . One of our main findings is that the x strategies (player 1) and the y strategies (player 2) of a mixed Nash equilibrium of full support in the original asymmetric game, also constitute Nash equilibria in the symmetric counterpart games. The symmetric counterpart of player 1 (x) is defined on the payoff of player 2 and vice versa. We prove that for full support strategies, Nash equilibria of the asymmetric game are pairwise combinations of Nash equilibria of the two symmetric counterparts. Then, we show that this property stands without the assumption of full support as well. Though this analysis does not allow us to visualise the evolutionary dynamics of the asymmetric game itself, it does allow us to identify its Nash equilibria by investigating the evolutionary dynamics of the counterparts. As such we can easily distinguish Nash equilibria from other restpoints in the asymmetric game and get an understanding of its underlying Nash structure. The paper is structured as follows: we first describe related work, then we continue with introducing essential game theoretic concepts. Subsequently, we present the main contributions and we illustrate the strengths of the theory by carrying out an evolutionary analysis on four canonical examples. Finally, we discuss the implications and provide a deeper understanding of the theoretical results.
 Related Work The most straightforward and classical approach to asymmetric games is to treat agents as evolving separately: one population per player, where each agent in a population interacts by playing against agent(s) from the other population(s), i.e. co-evolution  21  . This assumes that players of these games are always fundamentally attached to one role and never need to know/understand how to play as the other player. In many cases, though, a player may want to know how to play as either player. For example, a good chess player should know how to play as white or black. This reasoning inspired the role-based symmetrization of asymmetric games  22  . The role-based symmetrization of an arbitrary bimatrix game defines a new (extensive-form) game where before choosing actions the role of the two players are decided by uniform random chance. If two roles are available, an agent is assigned one specific role with probability  1 2  . Then, the agent plays the game under that role and collects the role-specific payoff appropriately. A new strategy space is defined, which is the product of both players' strategy spaces, and a new payoff matrix computing (expected) payoffs for each combination of pure strategies that could arise under the different roles. There are relationships between the sets of evolutionarily stable strategies and rest points of the replicator dynamics between the original and symmetrized game  19,23 .  This single-population model forces the players to be general: able to devise a strategy for each role, which may unnecessarily complicate algorithms that compute strategies for such players. In general, the payoff matrix in the resulting role-based symmetrization is n! (n being the number of agents) times larger due to the number of permutations of player role assignments. There are two-population variants that formulate the problem slightly differently: a new matrix that encapsulates both players' utilities assigns 0 utility to combinations of roles that are not in one-to-one correspondence with players  24  . This too, however, results in an unnecessarily larger (albeit sparse) matrix. Lastly, there are approaches that have structured asymmetry, that arises due to ecological constraints such as locality in a network and genotype/genetic relationships between population members  25  . Similarly here, replicator dynamics and their properties are derived by transforming the payoff matrix into a larger symmetric matrix. Our primary motivation is to enable analysis techniques for asymmetric games. However, we do this by introducing new symmetric counterpart dynamics rather than using standard dynamics on a symmetrised game. Therefore, the traditional role interpretation as well as any method that enlarges the game for the purpose of obtaining symmetry is unnecessarily complex for our purposes. Consequently, we consider the original co-evolutionary interpretation, and derive new (lower-dimensional) strategy space mappings.
 Preliminaries and Methods In this section we concisely outline (evolutionary) game theoretic concepts necessary to understand the remainder of the paper  23, 26, 27  . We briefly specify definitions of Normal Form Games and solution concepts such as Nash Equilibrium in a single population game and in a two-population game. Furthermore, we introduce the Replicator Dynamics (RD) equations for single and two population games and briefly discuss the concept of Evolutionary Stable Strategies (ESS) introduced by Smith and Price in 1973  [28] [29] [30]  . Normal Form Games and Nash Equilibrium. Definition. A two-player Normal Form Game (NFG) G is a 4-tuple G = (S 1 , S 2 , A, B), with pure strategy sets S 1 and S 2 for player 1, respectively player 2, and corresponding payoff tables A and B. Both players choose their pure strategies (also called actions) simultaneously. The payoffs for both players are represented by a bimatrix (A, B), which gives the payoff for the row player in A, and the column player in B (see Table  1  for a two strategy example). Specifically, the row player chooses one of the two rows, the column player chooses one of the columns, and the outcome of their joint strategy determines the payoff to both. In case S 1 = S 2 and A = B T the players are interchangeable and we call the game symmetric. In case at least one of these conditions is not met we have an asymmetric game. In classical game theory the players are considered to be individually rational, in the sense that each player is perfectly logical trying to maximise their own payoff, assuming the others are doing likewise. Under this assumption, the Nash equilibrium (NE) solution concept can be used to study what players will reasonably choose to do. We denote a strategy profile of the two players by the tuple (x, y) ∈ ΔS 1 × ΔS 2 , where ΔS 1 , ΔS 2 are the sets of mixed strategies, that is, distributions over the pure strategy sets or action sets. The strategy x (respectively y) is represented as a vector in  | | S 1 (respectively  | | S 2 ) where each entry is the probability of playing the corresponding action. The payoff associated with player 1 is x T Ay and x T By is the payoff associated with player 2. A strategy profile (x,y) now forms a NE if no single player can do better by unilaterally switching to a different strategy. In other words, each strategy in a NE is a best response against all other strategies in that equilibrium. Formally we have, Definition. A strategy profile (x,y) is a Nash equilibrium, iff the following holds: ∆ ∆ ∀ ′ ∈ ≥ ′ ∀ ′ ∈ ≥ ′ x S x Ay x Ay and y S x By x By , , T T T T 1 2 In the following, we will write NE(A, B) for the set of Nash equilibria of the game G = (S 1 , S 2 , A, B). Furthermore, a Nash equilibrium is said to be pure if only one strategy of the strategy set is played and we will say that it is completely mixed if all pure strategies are played with a non-zero probability. In evolutionary game theory, games are often considered with a single population. In other words, a player is playing against itself and only a single payoff table A is necessary to define the game (note that this definition only makes sense when |S 1 | = |S 2 | = n). In this case, the payoff received by the player is x T Ax and the following definition describes the Nash equilibrium: Definition. In a single population game, a strategy x is a Nash equilibrium, iff the following holds: ∀ ′ ≥ ′ x x Ax x Ax , T T In this single population case, we will write that x ∈ NE(A). Replicator Dynamics. Replicator Dynamics in essence are a system of differential equations that describe how a population of pure strategies, or replicators, evolve through time  26, 32  . In their most basic form they correspond to the biological selection principle, i.e. survival of the fittest. More specifically the selection replicator dynamic mechanism is expressed as follows: = − dx dt x Ax x Ax [( ) ] ( 1 ) i i i T Each replicator represents one (pure) strategy i. This strategy is inherited by all the offspring of the replicator. x i represents the density of strategy i in the population, A is the payoff matrix which describes the different payoff values each individual replicator receives when interacting with other replicators in the population. The state of the population x can be described as a probability vector x = (x 1 , x 2 , ..., x n ) which expresses the different densities of all the different types of replicators in the population. Hence (Ax) i is the payoff which replicator i receives in a population with state x and x T Ax describes the average payoff in the population. The support I x of a strategy is the set of actions (or pure strategies) that are played with a non-zero probability I x = {i |x i > 0}. In essence this equation compares the payoff a strategy receives with the average payoff of the entire population. If the strategy scores better than average it will be able to replicate offspring, if it scores lower than average its presence in the population will diminish and potentially approach extinction. The population remains in the simplex (∑ i x i = 1) since ∑ i (dx i )/(dt) = 0.
 Evolutionary Stable Strategies. Originally, an Evolutionary Stable Strategy was introduced in the context of a symmetric single population game  28, 32  (as introduced in the previous section), though this can be extended to multi-population games as well as defined in the next section  23, 33  . Imagine a population of simple agents playing the same strategy. Assume that this population is invaded by a different strategy, which is initially played by a small proportion of the total population. If the reproductive success of the new strategy is smaller than the original one, it will not overrule the original strategy and will eventually disappear. In this case we say that the strategy is evolutionary stable (ESS) against this newly appearing strategy. In general, we say a strategy is ESS if it is robust Asymmetric Replicator Dynamics. We have assumed replicators come from a single population, which makes the model only applicable to symmetric games. One can now wonder how the previous introduced equations extend to asymmetric games. Symmetry assumes that strategy sets and corresponding payoffs are the same for all players in the interaction. An example of an asymmetric game is the famous Battle of the Sexes (BoS) game illustrated in Table  2 . In this game both players do have the same strategy set, i.e., go to the opera or go to the movies, however, the corresponding payoffs for each are different, expressing the difference in preferences that both players have in their respective roles. If we would like to carry out a similar evolutionary analysis as before we will now need two populations, one for each player over its respective strategy set, and we need to use the asymmetric or coupled version of the replicator dynamics, i.e., Definition. = − = − dx dt x Ay x Ay and dy dt y x B x By [( ) ] [( ) ] (2) i i i T i i T i T with payoff tables A and B, respectively for player 1 and 2. In case A = B T the equations reduce to the single population model.
 Symmetric Counterpart Replicator Dynamics. We now introduce a new concept, the symmetric counterpart replicator dynamics (SCRD) of asymmetric replicator equations. We consider the two payoff tables A and B as two independent games that are no longer coupled, and in which both players participate. In the first counterpart game all players choose their strategy according to distribution y, the original strategy or replicator distribution for the 2nd population, or player 2, and in the second counterpart game all players choose their strategy according to distribution x, the original strategy or replicator distribution for the 1st population, or player 1. This gives us the following two sets of replicator equations: = − dy dt y Ay y Ay [( ) ] ( 3 ) i i i T and = − dx dt x x B x Bx [( ) ] ( 4 ) i i T i T In the results Section we will introduce some remarkable relationships between the equilibria of asymmetric replicator equations and the equilibria of their symmetric counterpart equations, which facilitates, and substantially simplifies, the analysis of the Nash structure of asymmetric games.
 Visualising evolutionary dynamics. One can visualise the replicator dynamics in a directional field and trajectory plot, which provides useful information about the equilibria, flow of dynamics and basins of attraction. As long as we stay in the realm of 2-player 2-action games this can be achieved relatively easily by plotting the probability with which player 1 plays its first action on the x-axis, and the probability with which player 2 plays its first action on the y-axis. Since there are only 2 actions for each player, this immediately gives a complete image of the dynamics over all strategies, since the probability for the second action a 2 to be chosen is one minus the first. By means of example we show a directional field plot here for the famous Prisoner's dilemma game (game illustrated in Table  3 ). The directional field plot, and corresponding trajectories, are shown in Fig.  1 . For both players the axis represents the probability with which they play Defect (D). As can be observed all dynamics are absorbed by the pure Nash equilibrium (D, D) in which both players defect.   Unfortunately, we cannot use the same type of plot illustrating the dynamics when we consider more than two strategies. However, if we move to single population games we can easily rely on a simplex plot. In the case of a two population game the situation become tedious as we will discuss later. Specifically, the set of probability distributions over n elements can be represented by the set of vectors (x 1 , ..., x n )  ∈ n , satisfying x 1 , ..., x n ≥ 0 and ∑ i x i = 1. This can be seen to correspond to an n − 1-dimensional structure called a simplex Σ n (or simply Σ, when n is clear from the context). In many of the figures throughout the paper we use Σ 3 , projected as an equilateral triangle. For example, consider the single population Rock-Paper-Scissors game, described by the payoff matrix shown in Fig.  2a . O M O 3, 2 0, 0 M 0, 0 2, 3 The game has one completely mixed Nash equilibrium, being ( ) , , 1 3 1 3 1 3 . In Fig.  2b  we have plotted the replicator equations Σ 3 trajectory plot for this game. Each of the corners of the simplex corresponds to one of the pure strategies, i.e., {Rock, Paper, Scissors}. For three strategies in the strategy simplex we then plot a trajectory illustrating the flow of the replicator dynamics. As can be observed from the plot, trajectories of the dynamics cycle around the mixed Nash equilibrium, which is not ESS and not asymptotically stable. In fact, three categories of rest points can be discerned in single population replicator dynamics (see  Figs 3, 4 and 5) . Figure  3  displays a stable Nash equilibrium called an Evolutionary Stable Strategy (ESS). An ESS is an attractor of the RD dynamical system defined in the previous section and has been one of the main foci of evolutionary game theory. The second type of rest points are the ones that are Nash but not ESS (Fig.  4 ). These rest points are not an attractor of the RD but they have a specific form. Specifically, if a strategy is a Nash equilibrium, all the actions that are not part of the support are dominated, i.e., the support is invariant under the RD, which means that the fraction of a strategy cannot become non-zero if it is zero at some point. The third category that can occur is illustrated in Fig.  5 . Those rest points are not Nash and thus there is an action outside of the support that is dominant. Thus, the flow will leave from points in the close vicinity of the rest point, which is called a source.
 Results In the following, we first present our main findings, formally relating Nash equilibria in asymmetric 2-player games with the Nash equilibria that can be found in the corresponding counterpart games. We also examine the stability properties of the corresponding rest points of the replicator dynamics in these games. Then we experimentally illustrate these findings in some canonical examples. Theoretical Findings. In this section, we prove the following result: if (x, y) ∈ NE(A, B) (where x and y have the same support), then x ∈ NE(B Τ ) and y ∈ NE(A). In addition, we prove that the reverse is true: if x ∈ NE(B Τ ) and y ∈ NE(A) (where x and y have the same support) then (x,y) ∈ NE(A,B). We will prove this result in two steps (Theorem 1 and its generalization Theorem 2). The theorems introduced apply to games where both players can play the same number of actions (i.e. square games). This condition can be weakened by adding dominated strategies to the player having the smallest number of actions (see the extended Battle of the Sexes example in the experimental section). Thus, without loss of generality, the theory will focus on square games. To begin, we state an important well-known property of Nash equilibria, that has been given different names; Gintis calls it fundamental theorem of Nash equilibria  32  . For sake of completeness, we provide a proof. Property 1. Let the strategy profile (x, y) be a Nash equilibrium of an asymmetric normal form game (A, B), and denote I z = {i | z i > 0} the support of a strategy z. Then, = ⊂ Τ Τ z Ay x Ay for all z such that I I and , (  5 ) z x = ⊂. Τ Τ x Bz x By for all z such that I I (6) z y Proof. This result is widely known. We provide it as it is a basis of our theoretical results and for the sake of completeness. If x and y constitute a Nash equilibrium then, by definition z Τ Ay ≤ x Τ Ay,∀z. Let us suppose that there exists a z with I z ⊂ I x such that z Τ Ay < x Τ Ay. Then there is a i ∈ I z ⊂ I x satisfying (Ay) i < x Τ Ay, and we get = ∑ < ∑ = Τ ∈ ∈ Τ Τ x Ay x Ay xx Ay x Ay ( ) i I i i i I i x x , which is a contradiction, proving the first claim. The claim for B follows analogously. ◽ Property 2. Let the strategy x be a Nash equilibrium of a single population game A. Then, = ⊂ .
 Τ Τ z Ax x Ax for all z such that I I (7) z x Proof. The proof is similar to the proof of Property 1. ◽ This property will be useful in the steps of the proofs that follow. We now present our first main result: a correspondence between the Nash equilibria of full support in the asymmetric game with those of full support in the counterpart games. Theorem 2 subsumes this result and we introduce this simpler version first for the sake of readability. Theorem 1. If strategies x and y constitute a Nash equilibrium of an asymmetric normal form game G = (S 1 , S 2 , A, B), with both x i > 0 and y j > 0 for all i, j (full support), and |S 1 | = |S 2 | = n, then it holds that x is a Nash equilibrium of the single population game B T and y is a Nash equilibrium of the single population game A. The reverse is also true. Proof . This result follows naturally from Property 1 and is implied by Theorem 2. We start by assuming that x and y constitute a full support Nash equilibrium of the asymmetric game (A, B). By Property 1 and since x and y have full support, we know that:  T i n i T i n T i {1, , } { 1, , } From this we also know that y T Ay = (Ay) i (since the (Ay) i are equal for all I's in the vector Ay, so multiplying Ay with y T will yield the same number Ay max ( ) i i ), and similarly (x T B) i = x T Bx (and thus (B T x) i = x T B T x), implying that: ∀ ′ = ′ ∀ ′ = ′ y y Ay y Ay x x B x x B x , a nd, , T T T T T T This concludes the proof. ◽ For the first counterpart game this means that the players will use the y part of the Nash equilibrium of player 2 of the original asymmetric game, in the symmetric counterpart game determined by payoff table A. And similarly, for the second counterpart game this means that players will play according to the x part of the Nash equilibrium of player 1 of the original asymmetric game, in the symmetric game determined by payoff table B. As such both players consider a symmetric version of the asymmetric game, for which this y component and x component constitute a Nash equilibrium in the two new respective symmetric games. In essence, these two symmetric counterpart games can be considered as a decomposition of the original asymmetric game, which gives us a means to illustrate in a smaller strategy space where the mixed and pure equilibria are located. A direct consequence of Theorem 1 is the following corollary that gives insights on the geometrical structure of Nash equilibrium, Corollary 1. Combinations of Nash equilibria of full support of the games corresponding to the symmetrical counterparts of the original asymmetric game also form Nash equilibria of full support in this asymmetric game. Proof. This is a direct consequence of Theorem 1. ◽ The next theorem explores the case where the equilibrium is not of full support. We prove that the theorem stands if the strategies of both players have the same support. Indeed, the first theorem requires that both players play all actions with a positive probability, here we will only require that they play the actions with the same index with a positive probability. We say that x and y have the same support if the set of played actions Proof. We start by assuming that x and y constitute a Nash equilibrium of same support (I x = I y ) of the asymmetric game (A, B). By Property 1, and since x and y have the same support, we know that: I x = {i | x i > 0} and I y = {i | y i > 0} are equal. = ⊂ Τ Τ z Ay x Ay z I I for all such that , and z x ′ = ′ ⊂ . Τ Τ ′ (8) x Bz x By z I I for all such that  (9)  z y Implying that y Τ Ay = x Τ Ay and x Τ Bx = x Τ By (by setting z = y and z′ = x). Then, from the Nash equilibrium condition we can write:  ∆ ∆ ∀ ′ ∈ ≥ ′ ∀ ′ ∈ ≥ ′ x S x T 2 T T 1 T ∆ ∆ ∀ ′ ∈ ≥ ′ ∀ ′ ∈ ≥ ′ ′ y S y Ay y Ay x S x B x x B , a nd , T T 2 T 1 T T T which implies that y is a Nash equilibrium of B Τ and x is a Nash equilibrium of A. The proof of the other direction follows similar mechanics and uses Property 2. Let us now assume that y is a Nash equilibrium of B Τ and x is a Nash equilibrium of A with I x = I y . Then, from Property 2 we have: = ⊂ Τ Τ z Ay y Ay z I I for all such that , and z y ′ = ′ ⊂ . Τ Τ Τ Τ ′ z B x x B x z I I for all such that (11) z x (10) In particular we get y Τ Ay = x Τ Ay and x Τ Bx = x Τ By (by setting z = x and z′ = y). From the Nash equilibrium condition of the single population games we can write:  ∆ ∆ ∀ ′ ∈ ≥ ′ ∀ ′ ∈ ≥ ′ y S y Ay y Ay x S x B x x B x , a nd , T 2 T 1 T T T T ∆ ∆ ∀ ′ ∈ ≥ ′ ∀ ′ ∈ ≥ ′ y S y Ay y Ay x S x Bx x Bx , a nd , T T 2 T 1 T ∆ ∆ ∀ ′ ∈ ≥ ′ ∀ ′ ∈ ≥ ′ x S x i i i T and B, = − = dx dt x x B x Bx (( ) ) 0 ( 13 ) i i T i T Proof. This is a direct consequence of Theorem 2. ◽ The theorems can only be used for equilibria in the counterpart games with matching supports (I x = I y ) from both players. One can work around this condition though by simply permuting the actions of one player in matrix A and B to study all configurations of supports of the same cardinality. To be precise, we need to analyze all the counterpart games defined by A Σ = AΣ and = Σ Σ B B ( ) T T for all permutation matrices Σ. This technique is sufficient to study non-degenerate games, as in a non-degenerate game all Nash equilibria have a support of same size (in a non-degenerate game if (x, y) is a Nash equilibrium then |I x | = |I y |  34  ).
 Stability Analysis. We can now examine the stability of the pure Nash equilibria discussed in the previously derived theorems. Corollary 3. Strategy y is a strict Nash equilibrium of the first counterpart game defined by A and strategy x is a strict Nash equilibrium of the second counterpart game defined by B, if and only if, (x, y) is a locally asymptotically stable equilibrium and a two-species ESS of the asymmetric normal form game G = (S 1 , S 2 , A, B) with support on the strategy with the same index in their respective strategy sets S 1 and S 2 . Proof. This a direct consequence of Corollary 2. More specifically, from Corollary 2 we know that (x, y) is a strict Nash equilibrium of G. It has been shown that (x, y) is a strict Nash equilibrium of G iff it is a two-species ESS  19, 20, 27  . ◽
 Experimental illustration We will now illustrate how the theoretical links between asymmetric games and their counterpart symmetric replicator dynamics facilitate analysis of asymmetric multiagent games, and provide a convenient tool to get insight into their equilibrium landscape. We do this for several examples. The first example concerns the Battle of the Sexes game to illustrate the intuition behind the results. The second example extends the Battle of the Sexes game with one strategy for one of the players, illustrating the permutation argument of the theorems and how to apply the results in case of a non-square game. The third example is a bimatrix game generated in the context of a multiagent learning algorithm called PSRO (Policy Space Response Oracles 9 ) and concerns Leduc Poker. This algorithm produces normal-form "empirical games" which each correspond to an extensive-form game with a reduced strategy space, using incremental best response learning. Finally, the last asymmetric game illustrates the theorems for a single mixed equilibrium of full support, while its counterpart games have many more equilibria. A fundamental complexity arises when using the evolutionary dynamics of a 2-player asymmetric game to analyse its equilibrium structure, as the dynamics for the two players is intrinsically coupled and high-dimensional. While one could fix a player's strategy and consider the induced dynamics for the other player in its respective strategy simplex, a static trajectory plot of this would not faithfully represent the complexity of the full 2-player dynamics. To gain a somewhat more complete intuitive picture, one can represent this dynamics as a movie, showing the change in induced dynamics for one player, as one varies the (fixed) strategy for the other (we will illustrate this in the PSRO-produced game on Leduc Poker). The theorems introduced in the previous section help to overcome this problem, and allow to analyse the evolutionary dynamics of the symmetric counterpart games instead of the asymmetric game itself, revealing the landscape of Nash equilibria, which seriously simplifies the analysis. Battle of the Sexes. Symmetry assumes that strategy sets and corresponding payoffs are the same for all players in the interaction. An example of an asymmetric game is the Battle of the Sexes (BoS) game illustrated in Table  2 . In this game both players do have the same strategy set, i.e., go to the opera or go to the movies, however, the corresponding payoffs for each are different, expressing the difference in preferences that both players have over their choices. The Battle of the Sexes has two pure Nash equilibria, which are ESS as well (located at coordinates (0, 0) and (1, 1)), and one unstable completely mixed Nash equilibrium in which the players play respectively x = ( ) , . Figure  6  illustrates the two-player evolutionary dynamics using the replicator equations, in which the x-axis corresponds to the probability with which player 1 plays O (Opera), and the y-axis corresponds to the probability with which the 2nd player plays O (Opera). The blue arrows show the vector field and the black lines are the corresponding trajectories. Note that it is still possible here to capture all of the dynamics in a static plot for the case of 2-player 2-action games, but is generally not possible in games with more than two actions. We now use this game to illustrate Theorem 1. If we apply Theorem 1 we know that the first and second counterpart symmetric games can be described by the payoff tables shown in Table  4 . The first counterpart game has ( ) ( ) ( ) , , , 2 5 3 5 2 5 3 5 as a mixed Nash equilibrium, and the second counterpart game has ( ) ( ) ( ) , , , 3 5 2 5 3 5 2 5 as a mixed Nash equilibrium. In Fig.  7 (b) and (c) we show the evolutionary dynamics of both counterpart games, from which the respective equilibria can be observed, as predicted by Theorem 1. Additionally, we also know that the reverse holds, i.e., if we were given the symmetric counterpart games, we would know that ( ) ( ) ( ) , , , 3 5 2 5 2 5 3 5 would also be a mixed Nash equilibrium of the original asymmetric BoS. In this case we can combine the mixed Nash equilibria of both counterpart games into the mixed Nash equilibrium of the original asymmetric game, as prescribed by Theorem 1. Specifically, as y = ( ) , which is a mixed Nash equilibrium of full support of the asymmetric Battle of the Sexes game. If we now apply Theorem 2 to the Battle of the Sexes game, then we find that pure strategy Nash equilibria x = (1, 0) (and y = (1, 0) for the second counterpart) and x = (0, 1) (and y = (0, 1) for the second counterpart), which are both ESS, are also Nash equilibria in the counterpart games shown in Table  4 . Also here the reverse holds, i.e., if we know the counterpart games, and we observe that x = (1, 0) and x = (0, 1) (y = (1, 0) and y = (0, 1) for the other counterpart of the game) are Nash in both games, we know that x = y = (1, 0) and x = y = (0, 1) are also Nash in the original asymmetric game. This can also be observed in Fig.  7(a),(b) and (c ). Specifically, the pure Nash equilibria are situated at coordinates (0, 0) and (1, 1) in Fig.  7(b) and (c ). Furthermore, it is important to understand that the counterpart dynamics are visualised only on the diagonal from coordinates (0, 0) to (1, 1), as that is where both players play with the same strategy distribution over their respective actions. Extended Battle of the Sexes game. In order to illustrate the theorems in a game that is non-square, including permutation of strategies, we extend the Battle of the Sexes game with a third strategy. Specifically, we give the second player a third strategy R in which she can choose to listen to a concert on the radio instead of going to the opera or movies with her partner. This game is illustrated in Table  5 . If we would like to carry out a similar evolutionary analysis as before we need two populations for the asymmetric replicator equations. Note that in this case the strategy sets of both players are different. Using the asymmetric replicator dynamics to plot the evolutionary dynamics quickly becomes complicated since the full dynamical picture is high-dimensional and not faithfully represented by projections to the respective player's individual strategy simplices. In other words, a static plot of the dynamics for one player does not immediately allow conclusions about equilibria, as it only describes a player's strategy evolution assuming a fixed (rather than  dynamically evolving) strategy of the other player. Again we can apply the counterpart RD theorems here to remedy this problem and consequently analyse the equilibrium structure in the symmetric counterpart games instead, yielding insight into the equilibrium landscape of the asymmetric game. In Tables  6 and 7  we show the counterpart games A and B. Note that we introduce a dummy action D for the first player, in order to make sure that both players have the same number of actions in their strategy set (a requirement to apply the theorems) by just adding −1 for both players playing this strategy, which makes D completely dominated and thus redundant.    The three Nash equilibria of interest of this asymmetric game are the following, {(x = (0.6, 0.4, 0),y = (0.4, 0, 0.6)),(x = (0, 1, 0),y = (0, 0, 1)),(x = (1, 0, 0),y = (1, 0, 0)))} (we use the online banach solver http://banach.lse. ac.uk/ to check that the Nash equilibria we find are correct  31  ). We now look for the y and x parts of these equilibria in the counterpart games. In Fig.  8  we show the evolutionary dynamics of the first counterpart game and in Fig.  9  the evolutionary dynamics of the second counterpart game. In the first counterpart we only need to consider the 1-face formed by strategies O and M as the third strategy is our dummy strategy. In this game there are two Nash equilibria, i.e., (1, 0, 0) (stable, yellow oval) and (0, 1, 0) (unstable, orange oval), so either playing O or M. The second counterpart game also has two Nash equilibria, i.e., (1, 0, 0) and (0, 0, 1) playing either O or M as well. Note there are also two rest points at the faces formed by O and R and O and M, which are not Nash (see Fig.  5  for an explanation). There is no mixed equilibrium of full support, so we cannot apply Theorem 1 here. If we apply Theorem 2 we know that ((1, 0, 0), (1, 0, 0)) must also    be a pure Nash equilibrium in the original asymmetric game, and we can remove the dummy strategy for player 1. At this stage we are left with equilibria (x = (0.6, 0.4, 0),y = (0.4, 0, 0.6)) and (x = (0, 1, 0),y = (0, 0, 1)) in the asymmetric game for which we did not find a symmetric counterpart at this stage. Now the permutation of the counterpart games, explained earlier in the findings section, comes into play. Recall that in order to study all configurations of supports of the same cardinal for both players one needs to simply permute the actions of one player in matrix A and B. Let's have a look at such a permutation, specifically, let's permute the 2nd and 3rd action for player 2, resulting in Tables  8 and 9 . Again we can analyse these counterpart games. Specifically, we find Nash equilibria (1, 0, 0), (0.4, 0.6, 0), and (0, 1, 0) for permuted counterpart game 1 (Table  8 ), and Nash equilibria (0, 0, 1), (0.6, 0.4, 0), (0, 1, 0), and (1,0,0) for permuted counterpart game 2 (Table  9 ), which are illustrated in Figs  10 and 11 . From these identified Nash equilibria in both counterpart games we can combine the remaining Nash equilibria for the asymmetric game. Specifically, by applying Theorem 2 we find (x = (0.6, 0.4, 0),y = (0.4, 0.6, 0)), which translates into (x = (0.6, 0.4, 0),y = (0.4, 0, 0.6)) for the asymmetric game as we permuted actions 2 and 3 for the second player and we need to swap these again. Additionally, we also find (x = (0, 1, 0),y = (0, 1, 0)), which translates into equilibrium (x = (0, 1, 0),y = (0, 0, 1)) for the asymmetric game as we permuted action 2 and 3 for the second player. Now we have found all Nash equilibria of the original asymmetric game. So, also in this case, i.e., when the game is not square and strategies need to be permuted, the theorems are still applicable and allow for analysis of the original asymmetric game. Poker generated asymmetric games. Policy Space Response Oracles (PSRO) is a multiagent reinforcement learning process that reduces the strategy space of large extensive-form games via iterative best response computation. PSRO can be seen as a generalized form of fictitious play that produces approximate best responses, with arbitrary distributions over generated responses computed by meta-strategy solvers. PSRO was applied to a commonly-used benchmark problem in artificial intelligence research known as Leduc poker  35  . Leduc poker has a deck of 6 cards (jack, queen, king in two suits). Each player receives an initial private card, can bet a fixed amount of 2 chips in the first round, 4 chips in the second round (with a maxium of two raises in each round). Before the second round starts, a public card is revealed.   In Table  10  we present such an asymmetric 3 × 3 2-player PSRO generated game, playing Leduc Poker. In the game illustrated here, each player has three strategies that, for ease of the exposition, we call {A, B, C} for player 1, and {D, E, F} for player 2. Each one of these strategies represents a larger strategy in the full extensive-form game of Leduc poker, specifically an approximate best response to a distribution over previous opponent strategies. The game produced here then is truly asymmetric, in the sense that the strategy spaces in the original game are inherently asymmetric since player 1 always starts each round, the strategy spaces are defined by different (mostly unique) betting sequences, and even under perfect equilibrium play there is a slight advantage to player 2  9  . So, both players have significantly different strategy sets. In Tables  11 and 12  we show the two symmetric counterpart games of the empirical game produced by PSRO on Leduc poker. O M R O 3 0 0.5 M 0 2 0.5 D −1 −1 −1 Again we can now analyse the landscape of equilibria of this game using the introduced theorems. Since the Leduc poker empirical game is asymmetric we need two populations for the asymmetric replicator equations. As mentioned before, analysing and plotting the evolutionary asymmetric replicator dynamics now quickly becomes very tedious as we deal with two simplices, one for each player. More precisely, if we consider a strategy for one player in its corresponding simplex, and that player is adjusting its strategy, will immediately cause the trajectory in the second simplex to change, and vice versa. Consequently, it is not straightforward anymore to analyse the     In order to facilitate the process of analysing this game we can apply the counterpart RD theorems here to remedy the problem, and consequently analyse the game in the far simpler symmetric counterpart games that will shed light onto the equilibrium landscape of the Leduc Poker empirical game. In Figs 12 and 13 we show the evolutionary dynamics of the counterpart games. As can be observed in Fig.  12  the first counterpart game has only one equilibrium, i.e., a mixed Nash equilibrium at the face formed by A and C, which absorbs the entire strategy space. Looking at Fig.  13  we see the situation is a bit more complex in the second counterpart game, here we observe three Nash equilibria: one pure at strategy D, one pure at strategy F, and one unstable mixed equilibrium at the 1-face formed by strategies D and F. Note there is also a rest point at the face formed by strategies D and E, which is not Nash. Given that there are no mixed equilibria with full support in both games we cannot apply Theorem 1. Using Theorem 2 we now know that we only maintain the two mixed equilibria, i.e. (0.32, 0, 0.68) (CP1) and (0.83, 0, 0.17) (CP2), forming the mixed Nash equilibrium (x = (0.83, 0, 0.17),y = (0.32, 0, 0.68)) of the asymmetric Leduc poker empirical game. The other equilibria in the second  counterpart game can be discarded as candidates for Nash equilibria in the Leduc poker empirical game since they also do not appear for player 1 when we permute the strategies for player 1 (not shown here). Mixed equilibrium of full support. As a final example to illustrate the introduced theory, we examine an asymmetric game, that has one completely mixed equilibrium and several equilibria in its counterpart games. The bimatrix game (A,B) is illustrated in Table  13  and its symmetric counterparts are shown in Tables  14 and 15 . The asymmetric game has a unique completely mixed Nash equilibrium with different mixtures for the two players, i.e., (x = = = ( )  , , ;  , , , (p 1 = (0,0,1), ( ) ( ) x y p 2 = (1,0,0)), (p 1 = = = ( ) ( ) ( ) p p 0, , , , , 0 1 1 2 1 2 2 1 2 1 2 }. Note that there are also two rest points, which are not Nash, at the faces formed by A and B and B and C. From these seven equilibria only (a), (b) and (c) are of interest since these are symmetric equilibria in which both players play with the same strategy (or support). Also counterpart game 2 has seven equilibria, i.e., = = { ( ) ( ) ( ) d p p ( ) , , , , , 1 1 3 1 3 1 3 2 1 3 1 3 1 3 , = = ( ) ( ) ( ) p p , 0, , 0, , 1 1 2 1 2 2 1 2 1 2 , = = e p p (( ) (0, 0, 1), (0, 0, 1)) 1 2 , = = ( ) ( ) ( ) p p 0, , , , 0, 1 1 2 1 2 2 1 2 1 2 , = = ( ) ( ) ( ) f p p ( ) , , 0 , , , 0 1 1 2 1 2 2 1 2 1 2 , = = p p ( (1, 0, 0), (0, 1, 0)) 1 2 , = = p p ( (0, 1, 0), (1, 0, 0) } 1 2 of which only (d), (e) and (f) are of interest. We obser ve that only the completely mixed equilibrium of the asymmetric game, i.e., = = ( )  , , ;  , , ( ) ( ) x y 1 3 1 3 2 7 3 7 13 , has its counterpart in the symmetric games. To apply the theorems we only need to have a look at equilibria (a), (b) and (c) in counterpart game 1, and (d), (e) and (f) in counterpart game 2. These equilibria can also be observed in the directional field plots, respectively, trajectory plots, illustrating the evolutionary dynamics of both counterpart games in Figs 14, 15, 16 and 17. Figure  14  visualises the three remaining equilibria (a), (b) and (c), with (a) indicated as a yellow oval, and (b) and (c) both indicated as green ovals. As can be observed, (a) is an unstable mixed equilibrium, (b) is a stable pure equilibrium, and (c) is a partly mixed equilibrium at the 2-face formed by strategies A and C. We can make the same observation for the second counterpart game, and see that (d), (e) and (f) are equilibria in Fig.  16 . Equilibrium (d), indicated by a yellow oval, is completely mixed, equilibrium (e) is a pure equilibrium in corner F (green oval), and (f) is a partly mixed equilibrium on the 2-face formed by strategies D and E (green ovals as well). If we now apply Theorem 1 we know that we can combine the mixed equilibria of full support of both counterpart games into the mixed equilibrium of the original asymmetric game, in which the mixed equilibrium of counterpart game 1, i.e.  ( )  , , , becomes the part of the mixed equilibrium in the asymmetric game of player 2, and the mixed equilibrium of counterpart game 2, i.e. ( ) , , 1 3 1 3 1 3 , becomes the part of the mixed equilibrium in     , , ;  , , . Both equilibria are unstable in the counterpart games and also form an unstable mixed equilibrium in the asymmetric game. D E F A 1, 0 0, 1 2, 0 B 0, 1 2, 0 0, 0 C 2, 0 0, 0 1, 1
 Discussion Replicator Dynamics have proved to be an excellent tool to analyse the Nash landscape of multiagent interactions and distributed learning in both abstract games and complex systems 1,2,4,6 . The predominant approach has been the use of symmetric replicator equations, allowing for a relatively straightforward analysis in symmetric games. Many interesting real-world settings though involve roles or player-types for the different agents that take part in an interaction, and as such are asymmetric in nature. So far, most research has avoided to carry out RD analysis in this type of interactions, by either constructing a new symmetric game, in which the various actions of the different roles are joined together in one population  23, 24  , or by considering the various roles and strategies as heuristics, grouped in one population as well  2, 3, 8  . In the latter approach the payoffs due to different player-types are averaged over many samples of the player type resulting in a single average payoff to each player for each entry in the payoff table. The work presented in this paper takes a different stance by decomposing an asymmetric game into its symmetric counterparts. This method proves to be mathematically simple and elegant, and allows for a straightforward analysis of asymmetric games, without the need for turning the strategy spaces into one simplex or population, but instead allows to keep separate simplices for the involved populations of strategies. Furthermore, the counterpart games allow to get insight in the type and form of interaction of the asymmetric game under study, identifying its equilibrium structure and as such enabling analysis of abstract and empirical games  A deeper counter-intuitive understanding of the theoretical results of this paper is that when identifying Nash equilibria in the counterpart games with matching support (including permutations of strategies for one of the players), it turns out that also the combination of those equilibria form a Nash equilibrium in the corresponding asymmetric game. In general, the vector field for the evolutionary dynamics of one player is a function of the other player's strategy, and hence a vector field in one player's simplex doesn't carry much information as any equilibria you observe in it are changing with time as the other player is moving too. However, if you position the second player at a Nash equilibrium, it turns out that player one becomes indifferent between his different strategies, and remains stationary under the RD. This gives the unique situation in which the vector field plot for the second player's simplex is actually meaningful, because the assumption of player one being stationary actually holds (and vice versa). This is what we end up using when establishing the correspondence of the Nash Equilibria in asymmetric and counterpart games, and why the single-simplex plots for the counterpart games are actually meaningful for the asymmetric game -but this is also why they only describe the Nash Equilibria faithfully, but fail to be a valid decomposition of the full asymmetric game away from equilibrium. These findings shed new light on asymmetric interactions between multiple agents and provide new insights that facilitate a thorough and convenient analysis of asymmetric games. As pointed out by Veller and Hayward 36 , many real-world situations, in which one aims to study evolutionary or learning dynamics of several interacting agents, are better modelled by asymmetric games. As such these theoretical findings can facilitate deeper analysis  of equilibrium structures in evolutionary asymmetric games relevant to various topics including economic theory, evolutionary biology, empirical game theory, the evolution of cooperation, evolutionary language games and artificial intelligence  11, 12, [37] [38] [39] [40]  . Finally, the results of this paper also nicely underpin what is said in H. Gintis' book on the evolutionary dynamics of asymmetric games, i.e., 'although the static game pits the row player against the column player, the evolutionary dynamic pits row players against themselves and column players against themselves' 32 (chapter 12, p.292). He also indicates that this aspect of an evolutionary dynamic is often misunderstood. The use of our counterpart dynamics supports and illustrates this statement very clearly, showing that in the counterpart games species play games within a population and as such show an intra-species survival of the fittest, which is then combined into an equilibrium of the asymmetric game. Figure 1 . 1 Figure 1. Directional field plot of the Prisoner's Dilemma game.
 Figure 2 . 2 Figure 2. (a) Payoff matrix for the Rock-Paper-Scissors game. Strategies R, S and P correspond to playing respectively Rock, Scissors, Paper. (b) Σ 3 Trajectory plot of the Rock-Paper-Scissors game. The Nash equilibrium is marked with a full yellow dot.
 Figure 3 . 3 Figure 3. ESS.
 Figure 4 . 4 Figure 4. NE but not ESS.
 Figure 5 . 5 Figure 5. Rest point but not NE.
 Theorem 2 . 2 Strategies x and y constitute a Nash equilibrium of an asymmetric game G = (S 1 , S 2 , A, B) with the same support (i.e. I x = I y ) if and only if x is a Nash equilibrium of the single population game B T , y is a Nash equilibrium of the single population game A and I x = I y .
 the Nash equilibrium in the first counterpart game and x = ( ) counterpart game, we can combine them into (x = ( )
 Figure 6 . 6 Figure 6. Directional field plot of the Battle of the Sexes game.
 Figure 7 . 7 Figure 7. This plot shows a visual representation of how the mixed Nash equilibrium is decomposed into Nash equilibria in both counterpart games. (a) shows the directional field plot of the Battle of the Sexes game. (b) illustrates how the y-component of the asymmetric Nash equilibrium becomes a Nash equilibrium in the first counterpart game, and (c) shows how the x-component of the asymmetric Nash equilibrium becomes a Nash equilibrium in the first counterpart game.
 Figure 8 . 8 Figure 8. Directional field plot Σ 3 of the first counterpart game of the extended Battle of the Sexes game.
 Figure 9 . 9 Figure 9. Directional field plot Σ 3 of the second counterpart game of the extended Battle of the Sexes game.
 Table 9 . 9 Permuted payoff matrix for the 2nd counterpart game of the Extended BoS game.
 Figure 10 . 10 Figure 10. Directional field plot Σ 3 of the first counterpart game of the permuted extended Battle of the Sexes game.
 Figure 11 . 11 Figure 11. Directional field plot Σ 3 of the second counterpart game of the permuted extended Battle of the Sexes game.
 dynamics and equilibrium landscape for both players, as any trajectory in one simplex causes the other simplex to change. A movie illustrates what is meant: we show how the dynamics of player 2 changes in function of player 1. We overlay the simplex of the second player with the simplex of the first player; the yellow dots indicate what the strategy of the first player is. The movie then shows how the dynamics of the second player changes when the yellow dot changes, see https://youtu.be/10m0f3iBECc.
 Figure 12 . 12 Figure 12. Directional field plot Σ 3 of the first counterpart game of the Leduc poker empirical game under study.
 Figure 13 . 13 Figure 13. Directional field plot Σ 3 of the second counterpart game of the Leduc poker empirical game under study.
 .,(p 1 = 1 The two symmetric counterpart game each have seven equilibria. Counterpart game 1 (Table14), has the following set of Nash equilibria: {(a)(p 1 (1,0,0), p 2 = (0,0,1)), (b)(p 1 = (0,1,0), p 2 = (0,1,0)),((c)
 Figure 14 . 14 Figure 14. Directional field plot Σ 3 of the first counterpart game of the mixed equilibrium asymmetric game.
 Figure 15 . 15 Figure 15. Trajectory plot Σ 3 of the first counterpart game of the mixed equilibrium asymmetric game.
 Figure 16 . 16 Figure 16. Directional field plot Σ 3 of the second counterpart game of the mixed equilibrium asymmetric game.
 Figure 17 . 17 Figure 17. Trajectory plot Σ 3 of the second counterpart game of the mixed equilibrium asymmetric game.
 Table 1 . 1 General payoff bimatrix (A, B) for a two-player two-action normal form game, where player 1 can choose between actions A 1 and A 2 , and player 2 can choose between actions B 1 and B 2 . Player 2 B 1 B 2 Player 1 A 1 A 2 a 11 , b 11 b 21, a 21 a 12 , b 12 a 22 , b 22
 Table 2 . 2 Payoff bimatrix for the Battle of the Sexes game. Strategies O and M correspond to going to the Opera and going to the Movies respectively.
 Table 3 . 3 Payoff matrix for the Prisoner's Dilemma game. Strategies D and C correspond to the actions Defect and Cooperate.
 Table 4 . 4 Counterpart matrix game 1 and 2 for the Battle of the Sexes game.
 Table 5 . 5 Extended Battle of the Sexes game.
 Table 6 . 6 Payoff matrix for the 1st counterpart game of the Extended BoS game. Strategy D is added to make the matrix completely square.
 Table 7 . 7 Payoff matrix for the 2nd counterpart game of the Extended BoS game. Strategy D is added to make the matrix completely square.
 Table 8 . 8 Permuted payoff matrix for the 1st counterpart game of the Extended BoS game.
 Table 10 . 10 Payoff matrix of an asymmetric empirical game produced by PSRO applied to Leduc poker. A B C A −0.16 0.05 0.08 B 0.64 −0.17 −0.31 C 0.79 1.06 −0.37
 Table 11 . 11 First counterpart game of the Leduc poker empirical game. D E F D 0.16 −0.05 −0.08 E −0.64 0.17 0.31 F −0.79 −1.06 0.37
 Table 12 . 12 Second counterpart game of the Leduc poker empirical game.
 Table 13 . 13 Payoff matrix of an asymmetric game with mixed equilibrium of full support. A B C A 1 0 2 B 0 2 0 C 2 0 1
 Table 14 . 14 First counterpart game of the asymmetric game. D E F D 0 1 0 E 1 0 0 F 0 0 1
 Table 15 . 15 Second counterpart game of the asymmetric game. the asymmetric game of player 1, leading to = ( ( = ) ( ) ) x y
			 Scientific REPORTS | (2018) 8:1015 | DOI:10.1038/s41598-018-19194-4

Abstract: We introduce new theoretical insights into two-population asymmetric games allowing for an elegant symmetric decomposition into two single population symmetric games. Specifically, we show how an asymmetric bimatrix game (A,B) can be decomposed into its symmetric counterparts by envisioning and investigating the payoff tables (A and B) that constitute the asymmetric game, as two independent, single population, symmetric games. We reveal several surprising formal relationships between an asymmetric two-population game and its symmetric single population counterparts, which facilitate a convenient analysis of the original asymmetric game due to the dimensionality reduction of the decomposition. The main finding reveals that if (x,y) is a Nash equilibrium of an asymmetric game (A,B), this implies that y is a Nash equilibrium of the symmetric counterpart game determined by payoff table A, and x is a Nash equilibrium of the symmetric counterpart game determined by payoff table B. Also the reverse holds and combinations of Nash equilibria of the counterpart games form Nash equilibria of the asymmetric game. We illustrate how these formal relationships aid in identifying and analysing the Nash structure of asymmetric games, by examining the evolutionary dynamics of the simpler counterpart games in several canonical examples. We are interested in analysing the Nash structure and evolutionary dynamics of strategic interactions in multi-agent systems. Traditionally, such interactions have been studied using single population replicator dynamics models, which are limited to symmetric situations, i.e., players have access to the same set of strategies and the payoff structure is symmetric as well 1 . For instance, Walsh et al. introduce an empirical game theory methodology (also referred to as heuristic payoff table method) that allows for analysing multiagent interactions in complex multiagent games 2,3 . This method has been extended by others and been applied e.g. in continuous double auctions, variants of poker and multi-robot systems 1,4-9 . Similar evolutionary methods have been applied to the modelling of human cooperation, language, and complex social dilemma's  [10] [11] [12] [13] [14] [15] [16]  . Though these evolutionary methods have been very useful in providing insights into the type and form of interactions in such systems, the underlying Nash structure, and evolutionary dynamics, the analysis is limited to symmetric situations, i.e., players or agents can be interchanged and have access to the same strategy set, in other words there are no different roles for the various agents involved in the interactions (e.g. a seller vs a buyer in an auction). As such this method is not directly applicable to asymmetric situations in which the players can choose strategies from different sets of actions, with asymmetric payoff structures. Many interesting multiagent scenarios involve asymmetric interactions though, examples include simple games from game theory such as e.g. the Ultimatum Game or the Battle of the Sexes and more complex board games that can involve various roles such as Scotland Yard, but also trading on the internet for instance can be considered asymmetric. There exist approaches that deal with asymmetry in multiagent interactions, but they usually propose to transform the asymmetric game into a symmetric game, with new strategy sets and payoff structure, which then can be analysed again in the context of symmetric games. This is indeed a feasible approach, but not easily scalable to the complex interactions mentioned before, nor is it practical or intuitive to construct a new symmetric game before the asymmetric one can be analysed in full. The approach we take in this paper does not require constructing a new game and is theoretically underpinned, revealing some new interesting insights in the relation between the Nash structure of symmetric and asymmetric games. Analysing multiagent interactions using evolutionary dynamics, or replicator dynamics, provides not only valuable insights into the (Nash) equilibria and their stability properties, but also sheds light on the behaviour trajectories of the involved agents and the basins of attraction of the equilibrium landscape  1, 4, 15, 17, 18  . As such it


Occam's razor is insufficient to infer the preferences of irrational agents

Introduction In today's reinforcement learning systems, a simple reward function is often hand-crafted, and still sometimes leads to undesired behaviors on the part of RL agent, as the reward function is not well aligned with the operator's true goals 4 . As AI systems become more powerful and autonomous, these failures will become more frequent and grave as RL agents exceed human performance, operate at time-scales that forbid constant oversight, and are given increasingly complex tasks -from driving cars to planning cities to eventually evaluating policies or helping run companies. Ensuring that the agents behave in alignment with human values is known, appropriately, as the value alignment problem  [Amodei et al., 2016 , Hadfield-Menell et al., 2016 , Russell et al., 2015 , Bostrom, 2014 , Leike et al., 2017 . One way of resolving this problem is to infer the correct reward function by observing human behaviour. This is known as Inverse reinforcement learning (IRL)  [Ng and Russell, 2000 , Abbeel and Ng, 2004 , Ziebart et al., 2008 . Often, learning a reward function is preferred over imitating a policy: when the agent must outperform humans, transfer to new environments, or be interpretable. The reward function is also usually a (much) more succinct and robust task representation than the policy, especially in planning tasks  [Abbeel and Ng, 2004] . Moreover, supervised learning of long-range and goal-directed behavior is often difficult without the reward function  [Ratliff et al., 2006] . Usually, the reward function is inferred based on the assumption that human behavior is optimal or noisily optimal. However, it is well-known that humans deviate from rationality in systematic, non-random ways  [Tversky and Kahneman, 1975] . This can be due to specific biases such as timeinconsistency, loss aversion and hundreds of others, but also limited cognitive capacity, which leads to forgetfulness, limited planning and false beliefs. This limits the use of IRL methods for tasks that humans don't find trivial. Some IRL approaches address specific biases  [Evans et al., 2015b,a] , and others assume noisy rationality  [Ziebart et al., 2008 , Boularias et al., 2011 . But a general framework for inferring the reward function from suboptimal behavior does not exist to our knowledge. Such a framework needs to infer two unobserved variables simultaneously: the human reward function and their planning algorithm  5  which connects the reward function with behaviour, henceforth called a planner. The task of observing human behaviour (or the human policy) and inferring from it the human reward function and planner will be termed decomposing the human policy. This paper will show there is a No Free Lunch theorem in this area: it is impossible to get a unique decomposition of human policy and hence get a unique human reward function. Indeed, any reward function is possible. And hence, if an IRL agent acts on what it believes is the human policy, the potential regret is near-maximal. This is another form of unidentifiability of the reward function, beyond the well-known ones  Russell, 2000, Amin and Singh, 2016] . The main result of this paper is that, unlike other No Free Lunch theorems, this unidentifiability does not disappear when regularising with a general simplicity prior that formalizes Occam's razor  [Vitanyi and Li, 1997] . This result will be shown in two steps: first, that the simplest decompositions include degenerate ones, and secondly, that the most 'reasonable' decompositions according to human judgement are of high complexity. So, although current IRL methods can perform well on many well-specified problems, they are fundamentally and philosophically incapable of establishing a 'reasonable' reward function for the human, no matter how powerful they become. In order to do this, they will need to build in 'normative assumptions': key assumptions about the reward function and/or planner, that cannot be deduced from observations, and allow the algorithm to focus on good ways of decomposing the human policy. Future work will sketch out some potential normative assumptions that can be used in this area, making use of the fact that humans assess each other to be irrational, and often these assessments agree. In view of the No Free Lunch result, this shows that humans must share normative assumptions. One of these 'normative assumption' approaches is briefly illustrated in an appendix, while another appendix demonstrates how to use the planner-reward formalism to define when an agent might be manipulating or overriding human preferences. This happens when the agent pushes the human towards situations where their policy is very suboptimal according to their reward function.
 Related Work In the first IRL papers from  Ng and Russell [2000]  and  Abbeel and Ng [2004]  a max-margin algorithm was used to find the reward function under which the observed policy most outperforms other policies. Suboptimal behavior was first addressed explicitly by  Ratliff et al. [2006]  who added slack variables to allow for suboptimal behavior. This finds reward functions such that the observed policy outperforms most other policies and the biggest margin by which another policy outperforms it is minimal, i.e. the observed policy has low regret.  Shiarlis et al. [2017]  introduce a modern max-margin technique with an approximate planner in the optimisation. However, the max-margin approach has mostly been replaced by the max entropy IRL  [Ziebart et al., 2008] . Here, the assumption is that observed actions or trajectories are chosen with probability proportional to the exponent of their value. This assumes a specific suboptimal planning algorithm which is noisily rational (also known as Boltzmann-rational). Noisy rationality explains human behavior on various data sets better  [Hula et al., 2015] . However,  Evans et al. [2015b]  and  Evans et al. [2015a]  showed that this can fail since humans deviate from rationality in systematic, non-random ways. If noisy rationality is assumed, repeated suboptimal actions throw off the inference. Literature on inferring the reasoning capabilities of an agent is scarce.  Evans et al. [2015b]  and  Evans et al. [2015a]  use Bayesian inference to identify specific planning biases such as myopic planning and hyperbolic time-discounting. They simultaneously infer the agent's preferences.  Cundy and Filan [2018]  adds bias resulting from hierarchical planning.  Hula et al. [2015]  similarly let agents infer features of their opponent's reasoning such as planning depth and impulsivity in simple economic games. Recent work learns the planning algorithm with two assumptions: being close to noisily rational in a high-dimensional planner space and supervised planner-learning  [Anonymous, 2019] . The related ideas of meta-reasoning  [Russell, 2016] , computational rationality  [Lewis et al., 2014]  and resource rationality  [Griffiths et al., 2015]  may create the possibility to redefine irrational behavior as rational in an 'ancestral' distribution of environments where the agent optimises its rewards by choosing among the limited computations it is able to perform or jointly minimising the cost of computation and maximising reward. This could in theory redefine many biases as computationally optimal in some distribution of environments and provide priors on human planning algorithms. Unfortunately the problem of doing this in practice seems to be extremely difficult -and it assumes that human goals are roughly the same as evolution's goals, which is certainly not the case.
 Problem setup and background A human will be performing a series of actions, and from these, an agent will attempt to estimate both the human's reward function and their planning algorithm. The environment M in which the human operates is an MDP/R, a Markov Decision Process without reward function (a world-model ). An MDP/R is defined as a tuple, S, A, T, ŝ consisting of a discrete state space S, a finite action space A, a fixed starting state ŝ, and a probabilistic transition function T : S × A × S → [0, 1] to the next state (also called the dynamics). At each step, the human is in a certain state s, takes a certain action a, and ends up in a new state s as given by T (s | s, a). Let R = {R : S × A → [−1, 1]} = [−1, 1] S×A be the space of candidate reward functions; a given R will map any state-reward pair to a reward value in the interval [−1, 1]. Let Π be the space of deterministic, Markovian policies. So Π is the space of functions S → A. The human will be following the policy π ∈ Π. The results of this paper apply to both discounted rewards and episodic environments settings 6 .
 Planners and reward functions: decomposing the policy The human has their reward function, and then follows a policy that presumably attempts to maximise it. Therefore there is something that bridges between the reward function and the policy: a piece of greater or lesser rationality that transforms knowledge of the reward function into a plan of action. This bridge will be modeled as a planner p : R → Π, a function that takes a reward and outputs a policy. This planner encodes all the rationality, irrationality, and biases of the human. Let P be the set of planners. The human is therefore defined by a planner-reward pair (p, R) ∈ P × R. Similarly, (p, R) with p(R) = π is a decomposition of the policy π. The task of the agent is to find a 'good' decomposition of the human policy π.
 Compatible pairs and evidence The agent can observe the human's behaviour and infer their policy from that. In order to simplify the problem and separate out the effect of the agent's learning, we will assume the agent has perfect knowledge of the human policy π and of the environment M . At this point, the agent cannot learn anything by observing the human's actions, as it can already perfectly predict these. Then a pair (p, R) is defined to be compatible with π, if p(R) = π -thus that pair is a possible candidate for decomposing the human policy into the human's planner and reward function.
 Irrationality-based unidentifiability Unidentifiability of the reward is a well-known problem in IRL  [Ng and Russell, 2000] .  Amin and Singh [2016]  categorise the problem into representational and experimental unidentifiability. The former means that adding a constant to a reward function or multiplying it with a positive scalar does not change what is optimal behavior. This is unproblematic as rescaling the reward function doesn't change the preference ordering. The latter can be resolved by observing optimal policies in a whole class of MDPs which contains all possible transition dynamics. We complete this framework with a third kind of identifiability, which arises when we observe suboptimal agents. This kind of unidentifiability is worse as it cannot necessarily be resolved by observing the agent in many tasks. In fact, it can lead to almost arbitrary regret.
 Weak No Free Lunch: unidentifiable reward function and half-maximal regret The results in this section show that without assumptions about the rationality of the human, all attempts to optimise their reward function are essentially futile.  work in a similar setting as we do: in their case, a corrupted version of the reward function is observed. The problem our case is that a 'corrupted' version π of an optimal policy π * Ṙ is observed and used as information to optimise for the ideal reward Ṙ. A No Free Lunch result analogous to theirs applies in our case; both resemble the No Free Lunch theorems for optimisation  [Wolpert and Macready, 1997] . More philosophically, this result is as an instance of the well-known is-ought problem from metaethics.  Hume [1888]  argued that what ought to be (here, the human's reward function) can never be concluded from what is (here, behavior) without extra assumptions. Equivalently, the human reward function cannot be inferred from behavior without assumptions about the planning algorithm p. In probabilistic terms, the likelihood P (π|R) = p∈P P (π | R, p)P (p) is undefined without P (p). As shown in Section 5 and Section 5.2, even a simplicity prior on p and R will not help.
 Unidentifiable reward functions Firstly, we note that compatibility (p(R) = π), puts no restriction on R, and few restrictions on p: Theorem 1. For all π ∈ Π and R ∈ R, there exists a p ∈ P such that p(R) = π. For all p ∈ P and π ∈ Π in the image of p, there exists an R such that p(R) = π. Proof. Trivial proof: define the planner 7 p as mapping all of R to π; then p(R) = π. The second statement is even more trivial, as π is in the image of p, so there must exist R with p(R) = π.
 Half-maximal regret The above shows that the reward function cannot be constrained by observation of the human, but what about the expected long-term value? Suppose that an agent is unsure what the actual human reward function is; if the agent itself is acting in an MDP/R, can it follow a policy that minimises the possible downside of its ignorance? This is prevented by a recent No Free Lunch theorem. Being ignorant of the reward function one should maximise is equivalent of having a corrupted reward channel with arbitrary corruption. In that case,  demonstrated that whatever policy π the agent follows, there is a R ∈ R for which π is half as bad as the worst policy the agent could have followed. Specifically, let V π R (s) be the expected return of reward function R from state s, given that the agent follows policy π. If π was the optimal policy for R, then this can be written as V * R (s). The regret of π for R at s is given by the difference:  demonstrates that for any π, Reg(π, R)(s) = V * R (s) − V π R (s). Then max R∈R Reg(π, R)(s) ≥ 1 2 max π ∈Π,R∈R Reg(π , R)(s) . So for any compatible (p, R) = π, we cannot rule out that maximizing R leads to at least half of the worst-case regret.
 Simplicity of degenerate decompositions Like many No Free Lunch theorems, the result of the previous section is not surprising given there are no assumptions about the planning algorithm. No Free Lunch results are generally avoided by placing a simplicity prior on the algorithm, dataset, function class or other object  [Everitt et al., 2014] . This amounts to saying algorithms can benefit from regularisation. This section is dedicated to showing that, surprisingly, simplicity does not solve the No Free Lunch result. Our simplicity measure is minimum description length of an object, defined as Kolmogorov complexity  [Kolmogorov, 1965] , the length of the shortest program that outputs a string describing the object. This is the most general formalization of Occam's razor we know of  [Vitanyi and Li, 1997] . Appendix A explores how the results extend to other measures of complexity, such as those that include computation time. We start with informal versions of our main results. Theorem 2 (Informal simplicity theorem). Let ( ṗ, Ṙ) be a 'reasonable' planner-reward pair that captures our judgements about the biases and rationality of a human with policy π = ṗ( Ṙ). Then there are degenerate planner-reward pairs, compatible with π, of lower complexity than ( ṗ, Ṙ), and a pair ( ṗ , − Ṙ) of similar complexity to ( ṗ, Ṙ), but with opposite reward function. There are a few issues with this theorem as it stands. Firstly, simplicity in algorithmic information theory is relative to the computer language (or equivalently Universal Turing Machine) L used  Vitányi, 2014, Calude, 2002] , and there exists languages in which the theorem is clearly false: one could choose a degenerate language in which ( ṗ, Ṙ) is encoded by the string '0', for example, and all other planner-reward pairs are of extremely long length. What constitutes a 'reasonable' language is a long-standing open problem, see  Leike et al. [2017]  and  Müller [2010] . For any pair of languages, complexities differ only by a constant, the amount required for one language to describe the other, but this constant can be arbitrarily large. Nevertheless, this section will provide grounds for the following two semi-formal results: Proposition 3. If π is a human policy, and L is a 'reasonable' computer language, then there exists degenerate planner-reward pairs amongst the pairs of lowest complexity compatible with π. Proposition 4. If π is a human policy, and L is a 'reasonable' computer language with ( ṗ, Ṙ) a compatible planner-reward pair, then there exist a pair ( ṗ , − Ṙ) of comparable complexity to ( ṗ, Ṙ), but opposite reward function. The last part of Theorem 2, the fact that any 'reasonable' ( ṗ, Ṙ) is expected to be of higher complexity, will be addressed in Section 6.
 Simple degenerate pairs The argument in this subsection will be that 1) the complexity of π is close to a lower bound on any pair compatible with it and 2) degenerate decompositions are themselves close to this bound. The first statement follows because for any decomposition (p, R) compatible with π, the map (p, R) → p(R) = π will be a simple one, adding little complexity. And if a compatible pair (p , R ) can be from π with little extra complexity, then it too will have a complexity close to the minimal complexity of any other pair compatible with it. Therefore we will first produce three degenerate pairs that can be simply constructed from π.
 The degenerate pairs We can define the trivial constant reward function 0, and the greedy planner p g . The greedy planner p g acts by taking the action that maximises the immediate reward in the current state and the next action. Thus 8 p g (R)(s) = argmax a R(s, a). We can also define the anti-greedy planner −p g , with −p g (R)(s) = argmin a R(s, a). In general, it will be useful to define the negative of a planner: Definition 5. If p : R → Π is a planner, the planner −p is defined by −p(R) = p(−R). For any given policy π, we can define the indifferent planner p π , which maps any reward function to π. We can also define the reward function R π , so that R π (s, a) = 1 if π(s) = a, and R π (s, a) = 0 otherwise. The reward function −R π is defined to be the negative of R π . Then: Lemma 6. The pairs (p π , 0), (p g , R π ), and (−p g , −R π ) are all compatible with π. Proof. Since the image p π is π, p π (0) = π. Now, R π (s, a) > 0 iff π(s) = a, hence for all s: p g (R π )(s) = argmax a R π (s, a) = π(s), so p g (R π ) = π. Then −p g (−R π ) = p g (−(−R π )) = p g (R π ) = π, by Definition 5.
 Complexity of basic operations We will look the operations that build the degenerate planner-reward pairs from any compatible pair: 1. For any planner p, f 1 (p) = (p, 0) as a planner-reward pair. 2. For any reward function R, f 2 (R) = (p g , R). 3. For any planner-reward pair (p, R), f 3 (p, R) = p(R). 4. For any planner-reward pair (p, R), f 4 (p, R) = (−p, −R). 5. For any policy π, f 5 (π) = p π . 6. For any policy π, f 6 (π) = R π . These will be called the basic operations, and there are strong arguments that reasonable computer languages should be able to express them with short programs. The operation f 1 , for instance, is simply appending the flat trivial 0, f 2 appends a planner defined by the simple 9 search operator argmax, f 3 applies a planner to the object -a reward function -that the planner naturally acts on, f 4 is a double negation, while f 5 and f 6 are simply described in subsubsection 5.1.1. From these basic operations, we can define three composite operations that map any compatible planner-reward pair to one of the degenerate pairs (the element F 4 = f 4 is useful for later definitions). Thus define F = {F 1 = f 1 • f 5 • f 3 , F 2 = f 2 • f 6 • f 3 , F 3 = f 4 • f 2 • f 6 • f 3 , F 4 = f 4 }. For any π-compatible pair (p, R) we have F 1 (p, R) = (p π , 0), F 2 (p, R) = (p g , R π ), and F 3 (p, R) = (−p g , −R π ) (see the proof of Proposition 7). Let K L denote Kolmogorov complexity in the language L: the shortest algorithm in L that generates a particular object. We define the F -complexity of L as max (p,R),Fi∈F K L (F i (p, R)) − K L (p, R). Thus the F -complexity of L is how much the F i potentially increase 10 the complexity of pairs. For a constant c ≥ 0, this allows us to formalise what we mean by L being a c-reasonable language for F : that the F -complexity of L is at most c. A reasonable language is a c-reasonable language for a c that we feel is intuitively low enough.
 Low complexity of degenerate planner-reward pairs To formalise the concepts 'of lowest complexity', and 'of comparable complexity', choose a constant c ≥ 0, then (p, R) and (p , R ) are of 'comparable complexity' if ||K L (p, R) − K L (p , R )|| ≤ c. For a set S ⊂ P × R, the pair (p, R) ∈ S is amongst the lowest complexity in S if ||K L (p, R) − min (p ,R )∈S K L (p , R )|| ≤ c, thus K L is within distance c of the minimum complexity element of S. Now formalize Proposition 3: Proposition 7. If π is the human policy, c defines a reasonable measure of comparable complexity, and L is a c-reasonable language for F , then the degenerate planner-reward pairs (p π , 0), (p g , R π ), and (−p g , −R π ) are amongst the pairs of lowest complexity among the pairs compatible with π. Proof. By Lemma 6, (p π , 0), (p g , R π ), and (−p g , −R π ) are compatible with π. By the definitions of the f i and F i , for(p, R) compatible with π, f 3 ((p, R)) = p(R) = π and hence F 1 (p, R) = f 1 • f 5 ( π) = f 1 (p π ) = (p π , 0), F 2 (p, R) = f 2 • f 6 ( π) = f 2 (R π ) = (p g , R π ), F 3 (p, R) = f 4 • F 2 (p, R) = (−p g , −R π ). Now pick (p, R) to be the simplest pair compatible with π. Since L is c-reasonable for F , K L (p π , 0) ≤ c + K L (p, R). Hence (p π , 0) is of lowest complexity among the pairs compatible with π; the same argument applies for the other two degenerate pairs.
 Negative reward If ( ṗ, Ṙ) is compatible with π, then so is (− ṗ, − Ṙ) = f 4 ( ṗ, Ṙ) = F 4 ( ṗ, Ṙ ). This immediately implies the formalisation of Proposition 4: Proposition 8. If π is a human policy, c defines a reasonable measure of comparable complexity, L is a c-reasonable language for F , and ( ṗ, Ṙ) is compatible with π, then (− ṗ, − Ṙ) is of comparable complexity to ( ṗ, Ṙ). So complexity fails to distinguish between a reasonable human reward function and its negative. 6 The high complexity of the genuine human reward function Section 5 demonstrated that there are degenerate planner-reward pairs close to the minimum complexity among all pairs compatible with π. This section will argue that any reasonable pair ( ṗ, Ṙ) is unlikely to be close to this minimum, and is therefore of higher complexity than the degenerate pairs. Unlike simplicity, reasonable decomposition cannot easily be formalised. Indeed, a formalization would likely already solve the problem, yielding an algorithm to maximize it. Therefore, the arguments in this section are mostly qualitative. We use reasonable to mean 'compatible with human judgements about rationality'. Since we do not have direct access to such a decomposition, the complexity argument will be about showing the complexity of these human judgements. This argument will proceed in three stages: 1. Any reasonable ( ṗ, Ṙ) is of high complexity, higher than it may intuitively seem to us. 2. Even given π, any reasonable ( ṗ, Ṙ) involves a high number of contingent choices. Hence any given ( ṗ, Ṙ) has high information (and thus high complexity), even given π. 3. Past failures to find a simple ( ṗ, Ṙ) derived from π are evidence that this is tricky.
 The complexity of human (ir)rationality Humans make noisy and biased decisions all the time. Though noise is important  [Kahneman et al., 2016] , many biases, such as anchoring bias, overconfidence, planning fallacies, and so on, affect humans in a highly systematic way; see  Kahneman and Egan [2011]  for many examples. Many people may feel that they have a good understanding of rationality, and therefore assume that assessing the (ir)rationality of any particular decision is not a complicated process. But an intuition for bias does not translate into a process for establishing a ( ṗ, Ṙ). Consider the anchoring bias defined in  Ariely et al. [2004] , where irrelevant information -the last digits of social security numbers -changed how much people were willing to pay for goods. When defining a reasonable ( ṗ, Ṙ), it does not suffice to be aware of the existence of anchoring bias 11 , but one has to precisely quantify the extent of the bias -why does anchoring bias seem to be stronger for chocolate than for wine, for instance? And why these precise percentages and correlations, and not others? And can people's judgment tell which people are more or less susceptible to anchoring bias? And can one quantify the bias for a single individual, rather than over a sample? Any given ( ṗ, Ṙ) can quantify the form and extent of these biases by computing objects like the regret function Reg( ṗ, Ṙ)(s ) := Reg( ṗ( Ṙ), Ṙ)(s) = V * Ṙ (s) − V ṗ( Ṙ) Ṙ (s), which measures the divergence between the expected value of the actual and optimal human policies 12 . Thus any given ( ṗ, Ṙ) -which contains the information to compute quantities like Reg( ṗ, Ṙ)(s) or similar measures of bias 13 , in every state -carries a high amount of numerical information about bias, and hence a high complexity. Since humans do not easily have access to this information, this implies that human judgement of irrationality is subject to Moravec's paradox  [Moravec, 1988] . It is similar to, for example, social skills: though it seems intuitively simple to us, it is highly complex to define in algorithmic terms. Other authors have argued directly for the complexity of human values, from fields as diverse as computer science, philosophy, neuroscience, and economics  [Minsky, 1984 , Bostrom, 2014 , Glimcher et al., 2009 , Muehlhauser and Helm, 2012 , Yudkowsky, 2011 .
 The contingency of human judgement The previous section showed that reasonable ( ṗ, Ṙ) carry large amounts of information/complexity, but the key question is whether it requires information additional to that in π. This section will show that even when π is known, there are many contingent choices that need to be made to define any specific reasonable ( ṗ, Ṙ). Hence any given ( ṗ, Ṙ) contains a large amount of information beyond that in π, and hence is of higher complexity. Reasons to believe that human judgement about reasonable ( ṗ, Ṙ) contains many contingent choices: • There is a variability of human judgement between cultures. When Miller  [1984]  compared American and Indian assessments of the same behaviours, they found systematically different explanations for them  14  Basic intuitions about rationality also vary between cultures  [Nisbett et al., 2001 , Brück, 1999 ]. • There is a variability of human judgement within a single culture. When  Slovic and Tversky [1974]  analysed the "Allais Paradox", they found that different people gave different answers as to what the rational behaviour was in their experiments. • There is evidence of variability of human judgement within the same person.  Slovic and Tversky [1974]  further attempted to argue for the rationality of one of the answers. This sometimes resulted in the participant sometimes changing their minds, and contradicting their previous assessment of rationality. • There is a variability of human judgement for the same person assessing their own values, caused by differences as trivial as question ordering  [Schuman and Ludwig, 1983] . So human meta-judgement, of own values and rationality, is also contingent and variable. • People have partial bias blind spots around their own biases  [Scopelliti et al., 2015] . Thus if a human is following policy π, a decomposition ( ṗ, Ṙ) would provide additional information about the cultural background of the decomposer, their personality within their culture, and even about the past history of the decomposer and how the issue is being presented to them. Those last pieces prevents us from 'simply' using the human's own assessment of their own rationality, as that assessment is subject to change and re-interpretation depending on their possible histories. 12 To exactly quantify the anchoring bias above, we could use a regret function that contrasts π with the same policy, but where the decision is optimal for one turn only (rather than for all turns, as in standard regret).  13  In constrast, regret for the degenerate planner-reward pairs is trivial. Reg(p π , 0) and Reg(pg, R π ) are identically zero -in the second case, since pg(R π ) is actually optimal for R π , getting the maximal possible reward -while (−pg, −R π ) has a regret that is identically −1 at each step. 14 "Results show that there were cross-cultural and developmental differences related to contrasting cultural conceptions of the person [...] rather than from cognitive, experiential, and informational differences [...]."
 The search for human rationality models One final argument that there is no simple algorithm for going from π to ( ṗ, Ṙ): many have tried and failed to find such an algorithm. Since the subject of human rationality has been a major one for several thousands of years, the ongoing failure is indicative -though not a proof -of the difficulties involved. There have been many suggested philosophical avenues for finding such a reward (such as reflective equilibrium  [Rawls, 1971] ), but all have been underdefined and disputed. The economic concept of revealed preferences  [Samuelson, 1948]  is the most explicit, using the assumption of rational behaviour to derive human preferences. This is an often acceptable approximation, but can be taken too far: failure to take achieve an achievable goal does not imply that failure was desired. Even within the confines of economics, it has been criticised by behavioural economics approaches, such as prospect theory  [Kahneman and Tversky, 2013]  -and there are counter-criticisms to these. Using machine learning to deduce the intentions and preferences of humans is in its infancy, but we can see non-trivial real-world examples, even in settings as simple as car-driving  [Lazar et al., 2018] . Thus to date, neither humans nor machine learning have been able to find simple ways of going from π to ( ṗ, Ṙ), nor any simple and explicit theory for how such a decomposition could be achieved. This suggests that ( ṗ, Ṙ) is a complicated object, even if π is known. In conclusion: Conjecture 9 (Informal complexity proposition). If π is a human policy, and L is a 'reasonable' computer language with ( ṗ, Ṙ) a 'reasonable' compatible planner-reward pair, then the complexity of ( ṗ, Ṙ) is not close to minimal amongst the pairs compatible with π.
 Conclusion We have shown that some degenerate planner-reward decompositions of a human policy have nearminimal description length and argued that decompositions we would endorse do not. Hence, under the Kolmogorov-complexity simplicity prior, a formalization of Occam's Razor, the posterior would endorse degenerate solutions. Previous work has shown that noisy rationality is too strong an assumption as it does not account for bias; we tried the weaker assumption of simplicity, strong enough to avoid typical No Free Lunch results, but it is insufficient here. This is no reason for despair: there is a large space to explore between these two extremes. Our hope is that with some minimal assumptions about planner and reward we can infer the rest with enough data. Staying close to agnostic is desirable in some settings: for example, a misspecified model of the human reward function can lead to disastrous decisions with high confidence .  Anonymous [2019]  makes a promising first try -a high-dimensional parametric planner is initialized to noisy rationality and then adapts to fit the behavior of a systematically irrational agent. How can we reconcile our results with the fact that humans routinely make judgments about the preferences and irrationality of others? And, that these judgments are often correlated from human to human? After all, No Free Lunch applies to human as well as artificial agents. Our result shows that they must be using shared priors, beyond simplicity, that are not learned from observations. We call these normative assumptions because they encode beliefs about which reward functions are more likely and what constitutes approximately rational behavior. Uncovering minimal normative assumptions would be an ideal way to build on this paper; Appendix C shows one possible approach. 			 Technically we only need to infer the human reward function, but inferring that from behaviour requires some knowledge of the planning algorithm.
			 The setting is only chosen for notational convenience: it also emulates discrete POMDPs, non-Markovianness (eg by encoding the whole history in the state) and pseudo-random policies.
			 This is the 'indifferent' planner pπ of subsubsection 5.1.1.
			 Recall that pg is a planner, pg(R) is a policy, so pg(R) can be applied to states, and pg(R)(s) is an action.
			 In most standard computer languages, argmax just requires a for-loop, a reference to R, a comparison with a previously stored value, and possibly the storage of a new value and the current action.10 F -complexity is non-negative: F4 • F4 is the identity, so that KL(F4(p, R)) − KL(p, R) = −(KL(F4(F4(p, R)) − KL(F4(p, R)), meaning that max (p,R),F 4 KL(F4(p, R)) − KF (p, R) must be nonnegative; this is a reason to include F4 in the definition of F .
			 The fact that many cognitive biases have only been discovered recently argue against people having a good intuitive grasp of bias and rationality, as do people's persistent bias blind spots [Scopelliti et al., 2015] .

Abstract: Inverse reinforcement learning (IRL) attempts to infer human rewards or preferences from observed behavior. Since human planning systematically deviates from rationality, several approaches have been tried to account for specific human shortcomings. However, the general problem of inferring the reward function of an agent of unknown rationality has received little attention. Unlike the well-known ambiguity problems in IRL, this one is practically relevant but cannot be resolved by observing the agent's policy in enough environments. This paper shows (1) that a No Free Lunch result implies it is impossible to uniquely decompose a policy into a planning algorithm and reward function, and (2) that even with a reasonable simplicity prior/Occam's razor on the set of decompositions, we cannot distinguish between the true decomposition and others that lead to high regret. To address this, we need simple 'normative' assumptions, which cannot be deduced exclusively from observations.