Scott Emmons
Caspar Oesterheld
Andrew Critch
Vince Conitzer
Stuart Russell

Symmetry, Equilibria, and Robustness in Common-Payoff Games

INTRODUCTION We consider common-payoff games (also known as identical interest games  [38] ), in which the payoff to all players is always the same.  1  Such games model a wide range of situations involving Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). Appears at the 3rd Games, Agents, and Incentives Workshop (GAIW 2021). Held as part of the Workshops at the 20th International Conference on Autonomous Agents and Multiagent Systems., Aziz, Ceppi, Dickerson, Hosseini, Lev, Mattei, McElfresh, Zick (chairs), May 2021, London, UK. © 2021 Copyright held by the owner/author(s). cooperative action towards a common goal. Under the heading of team theory, they form an important branch of economics  [19, 20] . In AI, the common-payoff assumption holds in Dec-POMDPs  [25] , where multiple agents operate independently according to policies designed centrally to achieve a common objective. Many applications of multiagent reinforcement learning also assume a common payoff  [7, 8, 12] . Finally, in assistance games  [31]  (also known as cooperative inverse reinforcement learning or CIRL games  [13] ), which include at least one human and one or more "robots," it is assumed that the robots' payoffs are exactly the human's payoff, even if the robots do not know what it is. Common-payoff games lead naturally to considerations of symmetry in game structure-for example, the assumption that two players' actions produce the same effect on the common payoff. Indeed, von Neumann and Morgenstern  [41]  and Nash  [23]  introduced fairly general group-theoretic notions of symmetry, which we adopt and explain in Section 2. More recent work has analyzed narrower notions of symmetry  [22, 30, 39] . For example, Daskalakis and Papadimitriou  [5]  study "anonymous games" and show that anonymity substantially reduces the complexity of finding solutions. Finally, Ham  [14]  generalizes the player-based notion of symmetry to include further symmetries revealed by renamings of actions. We conjecture our results extend to this more general case, at some cost in notational complexity, but we leave this to future work. In games exhibiting symmetry, it is then reasonable to consider symmetry in players' strategies. (Section 2 defines this in a precise sense.) For example, in team theory, it is common to develop a strategy that can be implemented by every employee in a given category and leads to high payoff for the company. (Notice that this does not lead to identical behavior, because strategies are statedependent.) In civic contexts, symmetry commonly arises through notions of fairness and justice. In treaty negotiations and legislation that mandates how parties behave, for example, there is often a constraint that all parties be treated equally. In DecPOMDPs, an offline solution search may consider only symmetric strategies for identical agents as a way of reducing the search space. In commonpayoff multiagent reinforcement learning, each agent may collect percepts and rewards independently, but the reinforcement learning updates can be pooled to learn a single parameterized policy that all agents share. Common-payoff and symmetric games have a number of desirable properties that may simplify the search for solutions. For the Table  1 : Three versions of the laundry/washing up game. Solutions are described in the text. purposes of this paper, we consider Nash equilibria-strategy profiles for all players from which no individual player has an incentive to deviate-as a reasonable solution concept. For example, Marschak and Radner  [20]  make the obvious point that a globally optimal (possibly asymmetric) strategy profile-one that achieves the highest common payoff-is necessarily a Nash equilibrium. Moreover, it can be found in time linear in the size of the payoff matrix. Another solution concept often used in multiagent RL and differential (i.e., continuous-action-space) games is that of a locally optimal strategy profile-roughly speaking, a strategy profile from which no player has an incentive to slightly deviate. Obviously, a locally optimal profile may not be a Nash equilibrium, as a player may still have an incentive to deviate to some more distant point in strategy space. Nonetheless, local optima, sometimes called local Nash equilibria-are important. For example, Ratliff et al.  [29]  argue that a local Nash equilibrium may still be stable in a practical sense if agents are computationally unable to find a better strategy. Similarly, gradient-based game solvers and multiagent RL algorithms may converge to local optima. Our first main result, informally stated, is that in a symmetric, common-payoff game, every local optimum in symmetric strategies is a (global) Nash equilibrium. Section 3 states the result more precisely and gives an example illustrating its generality.  2  Despite many decades of research on symmetric, common-payoff games, the result appears to be novel and perhaps useful. There are some echoes of the result in the literature on single-agent decision making  [4, 27, 32] , which can be connected to symmetric solutions of common-payoff games by treating all players jointly as a single agent, but our result appears more general than published results. The proof we give of our result contains elements similar to the proof (of a related but different result) in Taylor  [34] . To gain some intuition for these concepts and claims, let us consider a situation in which two children, Ali and Bo, have to do some housework-specifically, laundry (𝐿) and washing up (𝑊 ). Here, the "common payoff," if any, is to the parents. It is evident that a symmetric strategy profile-both doing the laundry or both doing the washing up-is not ideal, because the other task will not get done. The first version of the game, whose payoffs 𝑈 are shown in Table  1a , is asymmetric: while Ali is competent at both tasks, Bo does not know how to do the laundry properly and will ruin the clothes. Here, as Marschak and Radner pointed out, the strategy profile (𝐿,𝑊 ) is both globally optimal and a Nash equilibrium. If we posit a mixed (randomized) strategy profile in which Ali and Bo have laundry probabilities 𝑝 and 𝑞 respectively, the gradients 𝜕𝑈 /𝜕𝑝 and 𝜕𝑈 /𝜕𝑞 are +1 and −1, driving the solution towards (𝐿,𝑊 ).  2  Complete proofs for all of our results are in the appendices.   1b ). Although the symmetric optimum has lower expected utility than the unrestricted optima, total symmetry of the game implies that the symmetric optimum is a Nash equilibrium; this is a special case of Theorem 3.2. In the second version of the game (Table  1b ), Ali has taught Bo how to do the laundry, and symmetry is restored. The pure profiles (𝐿,𝑊 ) and (𝑊 , 𝐿) are (asymmetric) globally optimal solutions and hence Nash equilibria. Figure  1  shows the entire payoff landscape as a function of 𝑝 and 𝑞: looking just at symmetric strategy profiles, it turns out that there is a local optimum at 𝑝 = 𝑞 = 0.5, i.e., where Ali and Bo toss fair coins to decide what to do. Although the expected payoff of this solution is lower than that of the asymmetric optima, the local optimum is, nonetheless, a Nash equilibrium. All unilateral deviations from the symmetric local optimum result in the same expected payoff because if one child is tossing a coin, the other child can do nothing to improve the final outcome. In the third version of the game (Table  1c ), the parents derive greater payoff from watching their children working happily together on a single task than they do from getting both tasks done. In this case, there is again a Nash equilibrium at 𝑝 = 𝑞 = 0.5, but it is a local minimum rather than a local maximum in symmetric strategy space. Thus, not all symmetric Nash equilibria are symmetric local optima; this is because Nash equilibria depend on unilateral deviations, whereas symmetric local optima depend on joint deviations that maintain symmetry. In the second half of the paper, we turn to the issue of robustness of symmetric solutions. In practice, a variety of factors can lead to modelling errors and approximate solutions, which motivates us to consider perturbations in payoffs and strategy profiles. Making general arguments about Nash equilibria, we show that our first main result is robust in the sense that it degrades linearly under 𝜖-magnitude perturbations into 𝑘𝜖-Nash equilibria (for some gamedependent constant 𝑘). Stability turns out to be a thornier issue. Instability, if not handled carefully, might lead to major coordination failures in practice  [3] . While it is already known that local strict optima in a totally symmetric team game attain one type of stability, the issue is complex because there are several ways of enforcing (or not enforcing) strict symmetries in payoffs and strategies  [22] . Our final results focus on the stability of agents making possibly-asymmetric updates from a symmetric solution. We prove for a non-degenerate class of games that local optima in symmetric strategy space fail to be local optima in asymmetric strategy space if and only if at least one player is mixing, and we experimentally quantify how often mixing occurs for learning algorithms in the GAMUT suite of games  [24] .
 PRELIMINARIES: GAMES AND SYMMETRIES 2.1 Normal-form games Throughout, we consider normal-form games G = (𝑁 , 𝐴, 𝑢) defined by a finite set 𝑁 with |𝑁 | = 𝑛 players, a finite set of action profiles 𝐴 = 𝐴 1 × 𝐴 2 × . . . × 𝐴 𝑛 with 𝐴 𝑖 specifying the actions available to player 𝑖, and the utility function 𝑢 = (𝑢 1 , 𝑢 2 , . . . , 𝑢 𝑛 ) with 𝑢 𝑖 : 𝐴 → R giving the utility for each player 𝑖  [33] . We call G common-payoff if 𝑢 𝑖 (𝑎) = 𝑢 𝑗 (𝑎) for all action profiles 𝑎 ∈ 𝐴 and all players 𝑖, 𝑗. In common-payoff games we may omit the player subscript 𝑖 from utility functions. We model each player as employing a (mixed) strategy 𝑠 𝑖 ∈ Δ(𝐴 𝑖 ), a probability distribution over actions. We denote the support of the probability distribution 𝑠 𝑖 by supp(𝑠 𝑖 ). Given a (mixed) strategy profile 𝑠 = (𝑠 1 , 𝑠 2 , . . . , 𝑠 𝑛 ) that specifies a strategy for each player, player 𝑖's expected utility is 𝐸𝑈 𝑖 (𝑠) = 𝑎 ∈𝐴 𝑢 𝑖 (𝑎) Note that, while we have chosen to use the normal-form game representation for simplicity, normal-form games are highly expressive. Normal-form games can represent mixed strategies in all finite games, including games with sequential actions, stochastic transitions, and partial observation such as imperfect-information extensive form games with perfect recall, Markov games, and Dec-POMDPs. To represent a sequential game in normal form, one simply lets each normal-form action be a complete strategy (contingency plan) accounting for every potential game decision.
 Symmetry in game structure Our notion of symmetry in game structure is built upon von Neumann and Morgenstern  [41] 's and borrows notation from Plan  [28] . The basic building block is a symmetry of a game: Definition 2.1. Call a permutation of player indices 𝜌 : {1, 2, ..., 𝑛} → {1, 2, ..., 𝑛} a symmetry of a game G if, for all strategy profiles (𝑠 1 , 𝑠 2 , ..., 𝑠 𝑛 ), permuting the strategy profile permutes the expected payoffs: 𝐸𝑈 𝜌 (𝑖) ((𝑠 1 , 𝑠 2 , ..., 𝑠 𝑛 )) = 𝐸𝑈 𝑖 ((𝑠 𝜌 (1) , 𝑠 𝜌 (2) , ..., 𝑠 𝜌 (𝑛) )), ∀𝑖. Note that, when we speak of a symmetry of a game, we implicitly assume 𝐴 𝑖 = 𝐴 𝑗 for all 𝑖, 𝑗 with 𝜌 (𝑖) = 𝑗 so that permuting the strategy profile is well-defined.  3  We characterize the symmetric structure of a game by its set of game symmetries: Definition 2.2. Denote the set of all symmetries of a game G by: Γ(G) = {𝜌 : {1, 2, ..., 𝑛} → {1, 2, ..., 𝑛} a symmetry of G}. A spectrum of game symmetries is possible. On one end of the spectrum, the identity permutation might be the only symmetry for a given game. On the other end of the spectrum, all possible permutations might be symmetries for a given game. Following the terminology of von Neumann and Morgenstern  [41] , we call the former case totally unsymmetric and the latter case totally symmetric: Definition 2.3. If Γ(G) = 𝑆 𝑛 , the full symmetric group, we call the game Γ(G) totally symmetric. If Γ(G) contains only the identity permutation, we call the game totally unsymmetric. Let P ⊆ Γ(G) be any subset of the game symmetries. Because Γ(G) is closed under composition, we can repeatedly apply permutations in P to yield a group of game symmetries ⟨P⟩: Definition 2.4. Let P ⊆ Γ(G) be a subset of the game symmetries. The group generated by P, denoted ⟨P⟩, is the set of all permutations that can result from (possibly repeated) composition of permutations in P: ⟨P⟩ = {𝜌 1 • 𝜌 2 • . . . • 𝜌 𝑚 | 𝑚 ∈ N, 𝜌 1 , 𝜌 2 , . . . , 𝜌 𝑚 ∈ P}. Group theory tells us that ⟨P⟩ defines a closed binary operation (permutation composition) including an identity and inverse maps, and ⟨P⟩ is the closure of P under function composition. With a subset of game symmetries P ⊆ Γ(G) in hand, we can use the permutations in P to carry one player index to another. For each player 𝑖, we give a name to the set of player indices to which permutations in P can carry 𝑖: we call it player 𝑖's orbit. Definition 2.5. Let P ⊆ Γ(G) be a subset of the game symmetries Γ(G). The orbit of player 𝑖 under P is the set of all other player indices that ⟨P⟩ can assign to 𝑖: P (𝑖) = {𝜌 (𝑖) | 𝜌 ∈ ⟨P⟩}. In fact, it is a standard result from group theory that the orbits of a group action on a set partition the set's elements, which leads to the following proposition: Proposition 2.6. Let P ⊆ Γ(G). The orbits of P partition the game's players. By Proposition 2.6, each P ⊆ Γ(G) yields an equivalence relation among the players. To gain intuition for this equivalence relation, consider two extreme cases. In a totally unsymmetric game, Γ(G) contains only the identity permutation, in which case each player is in its own orbit of Γ(G); the equivalence relation induced by the orbit partition shows that no players are equivalent. In a totally symmetric game, by contrast, every permutation is an element of Γ(G), i.e., Γ(G) = 𝑆 𝑛 , the full symmetric group; now, all the players share the same orbit of Γ(G), and the equivalence relation induced by the orbit partition shows that all the players are equivalent. We leverage the orbit structure of an arbitrary P ⊆ Γ(G) to define an equivalence relation among players because it adapts to however much or little symmetry is present in the game. Between the extreme cases of no symmetry (𝑛 orbits) and total symmetry (1 orbit) mentioned above, there could be any intermediate number of orbits of P. Furthermore, two players can share an orbit of P even if those two players cannot be arbitrarily swapped. In Example 3.3, all the players can be rotated in a circle, so all the players share an orbit of P = Γ(G) even though the game does not admit arbitrary swapping of players.
 Symmetry in strategy profiles Having formalized a symmetry of a game in the preceding section, we follow Nash  [23]  and define symmetry in strategy profiles with respect to symmetry in game structure: Definition 2.7. Let P ⊆ Γ(G) be a subset of the game symmetries Γ(G). We call a strategy profile 𝑠 = (𝑠 1 , 𝑠 2 , ..., 𝑠 𝑛 ) P-invariant if (𝑠 1 , 𝑠 2 , ..., 𝑠 𝑛 ) = (𝑠 𝜌 (1) , 𝑠 𝜌 (2) , ..., 𝑠 𝜌 (𝑛) ) for all 𝜌 ∈ ⟨P⟩. The equivalence relation among players induced by the orbit structure of P is fundamental to our definition of symmetry in strategy profiles by the following proposition: Proposition 2.8. A strategy profile 𝑠 = (𝑠 1 , 𝑠 2 , ..., 𝑠 𝑛 ) is P-invariant if and only if 𝑠 𝑖 = 𝑠 𝑗 for each pair of players 𝑖 and 𝑗 with P (𝑖) = P ( 𝑗). To state Proposition 2.8 another way, a strategy profile is Pinvariant if all pairs of players 𝑖 and 𝑗 that are equivalent under the orbits of P play the same strategy.
 LOCAL SYMMETRIC OPTIMA ARE (GLOBAL) NASH EQUILIBRIA After the formal definitions of symmetry in the previous section, we are almost ready to formally state the first of our three main results. The only remaining definition is that of a local symmetric optimum: Definition 3.1. Call 𝑠 a locally optimal P-invariant strategy profile of a common-payoff game if: (i) 𝑠 is P-invariant, and (ii) for some 𝜖 > 0, no P-invariant strategy 𝑠 ′ with 𝐸𝑈 (𝑠 ′ ) > 𝐸𝑈 (𝑠) can be formed by adding or subtracting at most 𝜖 to the probability of taking any given action 𝑎 𝑖 ∈ 𝐴 𝑖 . If, furthermore, condition (ii) holds for all 𝜖 > 0, we call 𝑠 a globally optimal P-invariant strategy profile or simply an optimal P-invariant strategy profile. Now we can formally state our first main theorem, that local symmetric optima are (global) Nash equilibria: Theorem 3.2. Let G be a common-payoff normal-form game, and let P ⊆ Γ(G) be a subset of the game symmetries Γ(G). Any locally optimal P-invariant strategy profile is a Nash equilibrium. Proof. We provide a sketch here and full details in Appendix A. Suppose, for the sake of contradiction, that an individual player 𝑖 could beneficially deviate to action 𝑎 𝑖 (if a beneficial deviation exists, then there is one to a pure strategy). Then, consider instead a collective change to a symmetric strategy profile in which all the players in 𝑖's orbit shift very slightly more probability to 𝑎 𝑖 . By making the amount of probability shifted ever smaller, the probability that this change affects exactly one agent's realized action (making it 𝑎 𝑖 when it would not have been before) can be arbitrarily larger than the probability that it affects multiple agents' realized actions. Moreover, if this causes exactly one agent's realized action to change, this must be in expectation beneficial, since the original unilateral deviation was in expectation beneficial. Hence, the original strategy profile cannot have been locally optimal. □
 Example illustrating general symmetry Here, we give an example that shows how Theorem 3.2 is more general than the case of total symmetry. The example illustrates the existence of rotational symmetry without total symmetry, and it illustrates how picking different P ⊆ Γ(G) leads to different optimal P-invariant strategies and thus different P-invariant Nash equilibria by Theorem 3.2. Example 3.3. There are four radio stations positioned in a square. We number these 1,2,3,4 clockwise, such that, e.g., 1 neighbors 4 and 2. There is also a neighborhood of people at each vertex of the square. The people can tune in to the radio station at their vertex of the square and to the radio stations at adjacent vertices of the square, but they cannot tune in to the station at the opposite vertex. The game has each radio station choose what to broadcast. For simplicity, suppose each radio station can broadcast the weather or music. The common payoff of the game is the sum of the utilities of the four neighborhoods. For each neighborhood, if the neighborhood cannot tune in to the weather, the payoff for that neighborhood is 0. If the neighborhood can only tune in to the weather, the payoff is 1, and if the neighborhood can tune in to both weather and music, the neighborhood's payoff is 2. The symmetries of the game Γ(G) include the set of permutations generated by rotating the radio stations once clockwise. In standard notation for permutations, {(1, 2, 3, 4), (2, 3, 4, 1), (3, 4, 1, 2), (4, 1, 2, 3)} ⊂ Γ(G). First, consider applying the theorem to P = Γ(G). In this case, the constraint of P-invariance requires all the radio stations to play the same strategy because all stations are in the same orbit. As we show in Appendix B, the optimal P-invariant strategy then is for each station to broadcast music with probability √ 2−1. Theorem 3.2 tells us that this optimal P-invariant strategy profile is a Nash equilibrium. Appendix B also shows how to verify this without the use of Theorem 3.2. Second, consider applying the theorem to the case where P consists only of the rotation twice clockwise, i.e., the permutation which maps each station onto the station on the opposite vertex of the square. In standard notation for permutations, P = {(3, 4, 1, 2)}. Now, the constraint of P-invariance requires radio stations at opposite vertices of the square to play the same strategy. However, neighboring stations can broadcast different programs. The optimal P-invariant strategy is for one pair of opposite-vertex radio stations, e.g., 1 and 3, to broadcast the weather and for the other pair of radio stations, 2 and 4, to broadcast music. While it turns out to be immediate that this optimal P-invariant strategy is a Nash equilibrium because it achieves the globally optimal outcome, we could have applied Theorem 3.2 to know that this optimal P-invariant strategy profile is a Nash equilibrium even without knowing what the optimal P-invariant strategy was.
 ROBUSTNESS OF THE MAIN RESULT TO PAYOFF AND STRATEGY PERTURBATIONS The first type of robustness we consider is robustness to perturbations in the game's payoff function. Formally, we define an 𝜖perturbation of a game as follows: Definition 4.1. Let G be a normal-form game with utility function 𝜇. For some 𝜖 > 0, we call G ′ an 𝜖-perturbation of G if G ′ has utility function 𝜇 ′ satisfying max 𝑖 ∈𝑁 ,𝑎 ∈𝐴 |𝑢 ′ 𝑖 (𝑎) − 𝑢 𝑖 (𝑎)| ≤ 𝜖. There are a variety of reasons why 𝜖-perturbations might arise in practice. Our game model may contain errors such as the game not being perfectly symmetric; the players' preferences might drift over time; or we might have used function approximation to learn the game's payoffs. With Proposition 4.2, we note a generic observation about Nash equilibria showing that our main result, Theorem 3.2, is robust in the sense of degrading linearly in the payoff perturbation's size: Proposition 4.2. Let G be a common-payoff normal-form game, and let 𝑠 * be a locally-optimal P-invariant strategy profile for some subset of game symmetries P ⊆ Γ(G). Suppose 𝐺 ′ is an 𝜖-perturbation of G. Then 𝑠 * is a 2𝜖-Nash equilibrium in G ′ . The second type of robustness we consider is robustness to symmetric solutions that are only approximate. For example, we might try to find a symmetric local optimum through an approximate optimization method, or the evolutionary dynamics among players' strategies might lead them to approximate local symmetric optima. Again, a generic result about Nash equilibria shows that the guarantee of Theorem 3.2 degrades linearly in this case: Theorem 4.3. Let G be a common-payoff normal-form game, and let 𝑠 * be a locally-optimal P-invariant strategy profile for some subset of game symmetries P ⊆ Γ(G). Suppose 𝑠 is a strategy profile with total variation distance 𝑇𝑉 (𝑠, 𝑠 * ) ≤ 𝛿. Then 𝑠 is an 𝜖-Nash equilibrium with 𝜖 = 4𝛿 max 𝑖 ∈𝑁 ,𝑎 ∈𝐴 |𝑢 𝑖 (𝑎)|. By Theorem 4.3, we have a robustness guarantee in terms of the total variation distance between an approximate local symmetric optimum and a true local symmetric optimum. Without much difficulty, we can also convert this into a robustness guarantee in terms of the Kullback-Leibler divergence: Corollary 4.4. Let G be a common-payoff normal-form game, and let 𝑠 * be a locally-optimal P-invariant strategy profile for some subset of game symmetries P ⊆ Γ(G). Suppose 𝑠 is a strategy profile with Kullback-Leibler divergence satisfying 𝐷 𝐾𝐿 (𝑠 ||𝑠 * ) ≤ 𝜈 or 𝐷 𝐾𝐿 (𝑠 * ||𝑠) ≤ 𝜈. Then 𝑠 is an 𝜖-Nash equilibrium with 𝜖 = 2 √ 2𝜈 max 𝑖 ∈𝑁 ,𝑎 ∈𝐴 |𝑢 𝑖 (𝑎)|. While the results of this section show the robustness of Nash equilibria, we note that Nash equilibria, by definition, consider the possibility of only a single agent deviating; Nash equilibria cannot guarantee stability under dynamics that allow for multiple agents to deviate. In the next section, we investigate when multiple agents might have an incentive to simultaneously deviate by studying the optimality of symmetric strategy profiles in possibly-asymmetric strategy space.
 WHEN ARE LOCAL OPTIMA IN SYMMETRIC STRATEGY SPACE ALSO LOCAL OPTIMA IN POSSIBLY-ASYMMETRIC STRATEGY SPACE? Our first main theoretical result, Theorem 3.2, applies to locally optimal P-invariant, i.e., symmetric, strategy profiles. This still leaves open the question of how well locally optimal symmetric strategy profiles perform when considered in the broader, possiblyasymmetric strategy space. When are locally optimal P-invariant strategy profiles also locally optimal in possibly-asymmetric strategy space? This question is important in machine learning (ML) applications where users of symmetrically optimal ML systems might be motivated to make modifications to the systems, even for purposes of a common payoff. To address this issue more precisely, we formally define a local optimum in possibly-asymmetric strategy space: Definition 5.1. A strategy profile 𝑠 = (𝑠 1 , 𝑠 2 , . . . , 𝑠 𝑛 ) of a commonpayoff normal-form game is locally optimal among possiblyasymmetric strategy profiles, or, equivalently, a local optimum in possibly-asymmetric strategy space, if for some 𝜖 > 0, no strategy profile 𝑠 ′ with 𝐸𝑈 (𝑠 ′ ) > 𝐸𝑈 (𝑠) can be formed by changing 𝑠 in such a way that the probability of taking any given action 𝑎 𝑖 ∈ 𝐴 𝑖 for any player 𝑖 changes by at most 𝜖. Definition 5.1 relates to notions of stability under dynamics, such as those with perturbations or stochasticity, that allow multiple players to make asymmetric deviations. In particular, if 𝑠 is not a local maximum in asymmetric strategy space, this means that there is some set of players 𝐶 and strategy 𝑠 ′ 𝐶 arbitrarily close to 𝑠, such that if players 𝐶 were to play 𝑠 ′ 𝐶 (by mistake or due to stochasticity), some Player 𝑖 ∈ 𝑁 − 𝐶 would develop a strict preference over the support of 𝑠 𝑖 . To illustrate this, we return to the laundry/washing up game of the introduction. Example 5.2. Consider again the game of Table  1b . As Figure  1  illustrates, the symmetric optimum is for both Ali and Bo to randomize uniformly between W and L. While this is a Nash equilibrium, it is not a local optimum in possibly-asymmetric strategy space. If one player deviates from uniformly randomizing, the other player develops a strict preference for either 𝑊 or 𝐿. To understand when the phenomenon of Example 5.2 happens in general, we use the following degeneracy condition: Definition 5.3. Let 𝑠 be a Nash equilibrium of a game G: Intuitively, our definition says that a deterministic Nash equilibrium is non-degenerate when it is strict or almost strict (allowing the exception of at most one player who may be indifferent over available actions). A mixed Nash equilibrium, on the other hand, is non-degenerate when mixing matters. When speaking of a game G, we determine its degeneracy by the degeneracy of its Nash equilibria: Definition 5.4. We call a game G degenerate if it has at least one degenerate Nash equilibrium; otherwise, we call G non-degenerate. • If 𝑠 is deterministic, i.e., We note that "degnerate" is already an established term in the game-theoretical literature where it is often applied only to twoplayer games [see, e.g, 42, Definition 3.2]. While similar to the established notion of degeneracy, our definition of degeneracy is stronger, which makes our statements about non-degenerate games more general. If a two-player game G is non-degenerate in the usual sense from the literature, it is non-degenerate in the sense of Definition 5.3. Moreover, if G is common-payoff, then for each player 𝑖, we can define a two-player game played by 𝑖 and another single player who controls the strategies of 𝑁 − {𝑖}. If for all 𝑖 these two-player games are non-degenerate in the established sense, then G is non-degenerate in the sense of Definition 5.3. In non-degenerate games, our next theorem shows that a local symmetric optimum is a local optimum in possibly-asymmetric strategy space if and only if it is deterministic. Formally: Theorem 5.5. Let G be a non-degenerate common-payoff normalform game, and let P ⊆ Γ(G) be a subset of the game symmetries Γ(G). A locally optimal P-invariant strategy profile is locally optimal among possibly-asymmetric strategy profiles if and only if it is deterministic. To see why the (non-)degeneracy condition is needed in Theorem 5.5, we provide an example of a degenerate game: -10 Here, (𝑎, 𝑎) is the unique global optimum in symmetric strategy space. By Theorem 3.2, it is therefore also a Nash equilibrium. However, it is a degenerate Nash equilibrium and not locally optimal in asymmetric strategic space. The payoff can be improved by, e.g., Player 1 playing 𝑏 with small probability (and 𝑎 otherwise) and Player 2 playing 𝑐 with small probability (and 𝑎 otherwise). The following game illustrates how a global symmetric optimum, even if it is a non-degenerate, deterministic equilibrium, might still not be globally optimal in possibly-asymmetric strategy space. Example 5.7. Consider |𝑁 | = 3 and 𝐴 = {0, 1}, let 𝑘 be the number of players who choose action 1, and let the payoffs be: 0 if 𝑘 = 0, −1 if 𝑘 = 1, 1 if 𝑘 = 2, and −1 if 𝑘 = 3. Then the global symmetric optimum is for everyone to play 0. The global asymmetric optimum, on the other hand, is to coordinate to achieve 𝑘 = 2. Hence, the global symmetric optimum is strictly worse than the global asymmetric optimum. Of course, by Theorem 5.5, (0, 0, 0) is still a local optimum of asymmetric strategy space.
 LEARNING SYMMETRIC STRATEGIES IN GAMUT Theorem 5.5 shows that, in non-degenerate games, a locally optimal symmetric strategy profile is stable in the sense of Section 5 if and only if it is pure. For those concerned about stability, this raises the question: how often are optimal strategies pure, and how often are they mixed? To answer this question, we present an empirical analysis of learning symmetric strategy profiles in the GAMUT suite of game generators  [24] . We are interested both in how centralized optimization algorithms (such as gradient methods) search for symmetric strategies and in how decentralized populations of agents evolve symmetric strategies. To study the former, we run Sequential Least SQuares Programming (SLSQP)  [17, 40] , a local search method for constrained optimization. To study the latter, we simulate the replicator dynamics  [9] , an update rule from evolutionary game theory with connections to reinforcement learning  [2, 36, 37] . (See Appendix E.3 for details.)
 Experimental setup We ran experiments in all three classes of symmetric GAMUT games: RandomGame, CoordinationGame, and CollaborationGame. Intuitively, a RandomGame draws all payoffs uniformly at random, whereas in a CoordinationGame and a CollaborationGame, the highest payoffs are always for outcomes where all players choose the same action. (See Appendix E.1 details.) Because CoordinationGame and CollaborationGame have such similar game structures, our experimental results in the two games are nearly identical. To avoid redundancy, we only include experimental results for Coordina-tionGame in this paper. For each game class, we sweep the parameters of the game from 2 to 5 players and 2 to 5 actions, i.e., with (|𝑁 |, |𝐴 𝑖 |) ∈ {2, 3, 4, 5} × {2, 3, 4, 5}. We sample 100 games at each parameter setting and then attempt to calculate the global symmetric optimum using (i) 10 runs of SLSQP and (ii) 10 runs of the replicator dynamic (each with a different initialization drawn uniformly at random over the simplex), resulting in 10 + 10 = 20 solution attempts per game. Because we do not have ground truth for the globally optimal solution of the game, we instead use the best of our 20 solution attempts, which we call the "best solution. " To apply our previously developed theory to GAMUT games, we observe that RandomGames, CoordinationGames, and Collab-orationGames are (almost surely) non-degenerate in the sense of Definition 5.4: Proposition 6.1. Drawing a degenerate game is a measure-zero event in RandomGames, CoordinationGames, and CollaborationGames.
 What fraction of symmetric optima are local optima in possibly-asymmetric strategy space? Here, we try to get a sense for how often symmetric optima are stable in the sense that they are also local optima in possiblyasymmetric strategy space (see Section 5). In Appendix Table  3b , we show in what fraction of games the best solution of our 20 optimization attempts is mixed; by Theorem 5.5, this is the fraction of games whose symmetric optima are not local optima in possiblyasymmetric strategy space. In CoordinationGames, the symmetric optimum is always (by construction) for all players to choose the same action, leading to stability. By contrast, we see that 36% to 60% of RandomGames are unstable. We conclude that if real-world games do not have the special structure of CoordinationGames, then instability may be common.
 How often do SLSQP and the replicator dynamic find an optimal solution? As sequential least squares programming and the replicator dynamic are not guaranteed to converge to a global optimum, we test empirically how often each run converges to the best solution of our 20 optimization runs. In Appendix
 COMPUTATIONAL COMPLEXITY OF COMPUTING GAME SYMMETRIES AND SYMMETRIC STRATEGIES 7.1 Finding symmetries In some cases, domain knowledge can provide the symmetries of a game. For example, in the laundry game of Table  1b , symmetry arises from a simple observation: it matters only what chores get done, not which children do the chores. In other cases, however, players may face a potentially symmetric common-payoff game and first have to determine what the symmetries of the game are, e.g., by computing a generating set of the group of symmetries. Call this problem the game automorphism (GA) problem. Can it be solved efficiently? The complexity of the GA problem depends on how the game is represented. The simplest representation is to give the full table of payoffs. However, the size is then exponential in the number of players. A simple alternative is to only explicitly represent non-zero entries in the payoff table. This way, some games of many players can be represented succinctly. Calling the latter a sparse representation and the former a non-sparse representation, we obtain the following: Theorem 7.1. On a non-sparse game representation, the GA problem can be solved in polynomial time. On a sparse representation, the GA problem is polynomial-time equivalent to the graph isomorphism problem. For a general introduction to the graph isomorphism problem, see Grohe and Schweitzer  [11] . Notably, the problem is in NP but neither known to be solvable in polynomial time nor known to be NP-hard.
 Finding an optimal symmetric strategy profile Once it is known what the symmetries P of a given game are, what is the complexity of finding an optimal P-invariant strategy profile? In Appendix G.2, we show that the problem of optimizing symmetric strategies is equivalent to the problem of optimizing polynomials on Cartesian products of unit simplices. However, depending on how the polynomials and games are represented, the reductions may increase the problem instance exponentially. Nevertheless, we can import results from the literature on optimizing polynomials to obtain results such as the following: Theorem 7.2. Deciding for a given game G with symmetries P and a given number 𝐾 whether there is a P-invariant profile with expected utility at least 𝐾 is NP-hard, even for 2-player symmetric games.
 CONCLUSION When ML is deployed in the world, it is natural to instantiate multiple agents from the same template. This naturally restricts strategy profiles to symmetric ones, and it puts the focus on finding optimal symmetric strategy profiles. This, in turn, raises questions about the properties of such profiles. Would individual agents (or the users they serve) want to deviate from these profiles? Are they robust to small changes in the game or in the executed strategies? Could there be better asymmetric strategy profiles nearby? Our results yield a mix of good and bad news. Theorems 3.2 and 4.3 are good news, showing that even local optima in symmetric strategy space are (global) Nash equilibria in a robust sense. So, with respect to unilateral deviations among team members, symmetric optima are relatively stable strategies. On the other hand, Theorem 5.5 is perhaps bad news, because it shows that a broad class of symmetric local optima are unstable when considering joint deviations in asymmetric strategy space (Section 5). Furthermore, our empirical results with learning algorithms in GAMUT suggest that these unstable solutions may not be uncommon in practice (Section 6.2).
 A PROOFS OF SECTION 3 RESULTS Theorem 3.2. Let G be a common-payoff normal-form game, and let P ⊆ Γ(G) be a subset of the game symmetries Γ(G). Any locally optimal P-invariant strategy profile is a Nash equilibrium. Proof. We proceed by contradiction. Suppose 𝑠 = (𝑠 1 , 𝑠 2 , . . . , 𝑠 𝑛 ) is locally optimal among P-invariant strategy profiles that is not a Nash equilibrium. We will construct an 𝑠 ′ arbitrarily close to 𝑠 with 𝐸𝑈 (𝑠 ′ ) > 𝐸𝑈 (𝑠). Without loss of generality, suppose 𝑠 1 is not a best response to 𝑠 −1 but that the pure strategy of always playing 𝑎 1 is a best response to 𝑠 −1 . For an arbitrary probability 𝑝 > 0, consider the modified strategy 𝑠 ′ 1 that plays action 𝑎 1 with probability 𝑝 and follows 𝑠 1 with probability 1 −𝑝. Now, construct 𝑠 ′ = (𝑠 ′ 1 , 𝑠 ′ 2 , . . . , 𝑠 ′ 𝑛 ) as follows: 𝑠 ′ 𝑖 = 𝑠 ′ 𝑖 = 𝑠 ′ 1 if 𝑖 ∈ P (1) 𝑠 ′ 𝑖 = 𝑠 𝑖 otherwise. In words, 𝑠 ′ modifies 𝑠 by having the members of player 1's orbit mix in a probability 𝑝 of playing 𝑎 1 . We claim for all sufficiently small 𝑝 that 𝐸𝑈 (𝑠 ′ ) > 𝐸𝑈 (𝑠). To establish this claim, we break up the expected utility of 𝑠 ′ according to cases of how many players in 1's orbit play the action 𝑎 1 because of mixing in 𝑎 1 with probability 𝑝. In particular, we observe 𝐸𝑈 (𝑠 ′ ) = 𝑝)𝐸𝑈 (𝑠) + 𝐵(𝑚=1, 𝑝)𝐸𝑈 ((𝑠 ′ 1 , 𝑠 2 , . . . , 𝑠 𝑛 )) + 𝐵(𝑚>1, 𝑝)𝐸𝑈 (. . .), where 𝐵(𝑚, 𝑝) is the probability of 𝑚 successes for a binomial random variable on 𝑚 independent events that each have success probability 𝑝 and where 𝐸𝑈 (. . .) is arbitrary. Note that the crucial step in writing this expression is grouping the terms with the coefficient 𝐵(𝑚=1, 𝑝). We can do this because for any player 𝑗 ∈ P (1), there exists a symmetry 𝜌 ∈ Γ(G) with 𝜌 ( 𝑗) = 1. Now, to achieve 𝐸𝑈 (𝑠 ′ ) > 𝐸𝑈 (𝑠), we require 𝐸𝑈 (𝑠) < 𝐵(𝑚 = 1, 𝑝) 𝐵(𝑚 > 0, 𝑝) 𝐸𝑈 ((𝑠 ′ 1 , 𝑠 2 , . . . , 𝑠 𝑛 )) + 𝐵(𝑚 > 1, 𝑝) 𝐵(𝑚 > 0, 𝑝) 𝐸𝑈 (...). We know 𝐸𝑈 ((𝑠 ′ 1 , 𝑠 2 , ..., 𝑠 𝑛 )) > 𝐸𝑈 (𝑠), but we must deal with the case when 𝐸𝑈 (...) is arbitrarily negative. Because lim 𝑝→0 𝐵(𝑚 > 1, 𝑝)/𝐵(𝑚 = 1, 𝑝) = 0, by making 𝑝 sufficiently small, 𝐵(𝑚 = 1, 𝑝)/𝐵(𝑚 > 0, 𝑝) can be made greater than 𝐵(𝑚 > 1, 𝑝)/𝐵(𝑚 > 0, 𝑝) by an arbitrarily large ratio. The result follows. □
 B OPTIMAL SYMMETRIC POLICY FOR THE RADIO STATION GAME OF EXAMPLE 3.3 We here calculate the optimal Γ(G)-invariant strategy profile for Example 3.3. Let 𝑝 be the probability of broadcasting the weather forecasts. By symmetry of the game and linearity of expectation, the expected utility given 𝑝 is simply four times the expected utility of any individual neighborhood. The value of an individual neighborhood is 0 with probability (1 − 𝑝)  3  , is 1 with probability 𝑝 3 and is 2 with the remaining probability. Hence, the expected utility of a single neighborhood is 𝑝 3 + (1 − (1 − 𝑝) 3 − 𝑝 3 ) • 2 = 2 − 2(1 − 𝑝) 3 − 𝑝 3 . The maximum of this term (and thus the maximum of the overall utility of all neighborhoods) can be found by any computer algebra system to be 𝑝 = 2 − √ 2, which gives an expected utility of 4( √ 2 − 1) ≈ 1.66. To double-check, we can also calculate the symmetric Nash equilibrium of this game. It's easy to see that that Nash equilibrium must be mixed and therefore must make each player (radio station) indifferent about what to broadcast. So let 𝑝 again be the probability with which everyone else broadcasts the weather. The expected utility of broadcasting the weather relative to broadcasting nothing for any of the three relevant neighborhoods is 2(1 − 𝑝) 2 . (Broadcasting the weather lifts the utility of a neighborhood from 0 to 2 if they do not already get the weather. Otherwise, it is useless to air the weather.) The expected utility of broadcasting music again relative to broadcasting nothing is simply 𝑝 2 . We can find the symmetric Nash equilibrium by setting 2(1 − 𝑝) 2 = 𝑝 2 , which gives us the same solution for 𝑝 as before.
 C PROOFS OF SECTION 4 RESULTS Proposition 4.2. Let G be a common-payoff normal-form game, and let 𝑠 * be a locally-optimal P-invariant strategy profile for some subset of game symmetries P ⊆ Γ(G). Suppose 𝐺 ′ is an 𝜖-perturbation of G. Then 𝑠 * is a 2𝜖-Nash equilibrium in G ′ . Proof. By Theorem 3.2, 𝑠 * is a Nash equilibrium in G. After perturbing G by 𝜖 to form G ′ , payoffs have increased / decreased at most ±𝜖, so the difference between any two actions' expected payoffs has changed by at most 2𝜖. □ Theorem 4.3. Let G be a common-payoff normal-form game, and let 𝑠 * be a locally-optimal P-invariant strategy profile for some subset of game symmetries P ⊆ Γ(G). Suppose 𝑠 is a strategy profile with total variation distance 𝑇𝑉 (𝑠, 𝑠 * ) ≤ 𝛿. Then 𝑠 is an 𝜖-Nash equilibrium with 𝜖 = 4𝛿 max 𝑖 ∈𝑁 ,𝑎 ∈𝐴 |𝑢 𝑖 (𝑎)|. Proof. Consider the perspective of an arbitrary player 𝑖. The difference in expected utility of playing any action 𝑎 𝑖 between the opponent strategy profiles 𝑠 −𝑖 and 𝑠 * −𝑖 is given by: 𝐸𝑈 𝑖 (𝑎 𝑖 , 𝑠 −𝑖 ) − 𝐸𝑈 𝑖 (𝑎 𝑖 , 𝑠 * −𝑖 ) = 𝑎 −𝑖 ∈𝐴 −𝑖 𝑠 −𝑖 (𝑎 −𝑖 )𝑢 𝑖 (𝑎 𝑖 , 𝑎 −𝑖 ) − 𝑎 −𝑖 ∈𝐴 −𝑖 𝑠 * −𝑖 (𝑎 −𝑖 )𝑢 𝑖 (𝑎 𝑖 , 𝑎 −𝑖 ) ≤ 𝑎 −𝑖 ∈𝐴 −𝑖 |𝑢 𝑖 (𝑎 𝑖 , 𝑎 −𝑖 )| 𝑠 −𝑖 (𝑎 −𝑖 ) − 𝑠 * −𝑖 (𝑎 −𝑖 ) ≤ 2𝑇𝑉 (𝑠, 𝑠 * ) max 𝑖 ∈𝑁 ,𝑎 ∈𝐴 |𝑢 𝑖 (𝑎)| ≤ 2𝛿 max 𝑖 ∈𝑁 ,𝑎 ∈𝐴 |𝑢 𝑖 (𝑎)|. In particular, let 𝑎 𝑖 be an action in the support of 𝑠 * 𝑖 , and let 𝑎 ′ 𝑖 be any other action. Then, using the above, we have:  𝐸𝑈 𝑖 (𝑎 ′ 𝑖 , 𝑠 −𝑖 ) − 𝐸𝑈 𝑖 (𝑎 𝑖 , 𝑠 −𝑖 ) = 𝐸𝑈 𝑖 (𝑎 ′ 𝑖 , 𝑠 −𝑖 ) − 𝐸𝑈 𝑖 (𝑎 ′ 𝑖 ,
 D PROOFS OF SECTION 5 RESULTS Theorem 5.5. Let G be a non-degenerate common-payoff normalform game, and let P ⊆ Γ(G) be a subset of the game symmetries Γ(G). A locally optimal P-invariant strategy profile is locally optimal among possibly-asymmetric strategy profiles if and only if it is deterministic. Proof. Let 𝑠 be a locally optimal P-invariant strategy profile. By Theorem 3.2, 𝑠 is a Nash equilibrium. Because G is non-degenerate, so is 𝑠. We prove the claim by proving that (1) if 𝑠 is deterministic, it is locally optimal in asymmetric strategy space; and (2) if 𝑠 is mixed then it is not locally optimal in asymmetric strategy space. (1) The deterministic case: Let 𝑠 be deterministic. Now consider a potentially asymmetric strategy profile 𝑠 ′ . We must show as 𝑠 ′ becomes sufficiently close to 𝑠 that 𝐸𝑈 (𝑠 ′ ) ≤ 𝐸𝑈 (𝑠). Let 𝜖 1 , 𝜖 2 , ..., 𝜖 𝑛 and ŝ1 , ..., ŝ𝑛 be such that for 𝑖 ∈ 𝑁 , 𝑠 ′ 𝑖 can be interpreted as following 𝑠 𝑖 with probability 1 − 𝜖 𝑖 and following ŝ𝑖 with probability 𝜖 𝑖 , where 𝑠 𝑖 ∉ supp( ŝ𝑖 ). Then (similar to the proof of Theorem 3.2), we can write 𝐸𝑈 (𝑠 ′ ) = 𝑖 ∈𝑁 (1 − 𝜖 𝑖 ) 𝐸𝑈 (𝑠) + 𝑗 ∈𝑁 𝜖 𝑗 𝑖 ∈𝑁 −{ 𝑗 } 1 − 𝜖 𝑖 • 𝐸𝑈 ( ŝ𝑗 , 𝑠 −𝑗 ) + 𝑗,𝑙 ∈𝑁 :𝑗≠𝑙 𝜖 𝑗 𝜖 𝑖 𝑖 ∈𝑁 −{ 𝑗,𝑙 } 1 − 𝜖 𝑖 • 𝐸𝑈 ( ŝ𝑗 , ŝ𝑙 , 𝑠 −𝑗−𝑙 ) + ... The second line is the expected value if everyone plays 𝑠, the third line is the sum over the possibilities of one player 𝑗 deviating to ŝ𝑗 , and so forth. We now make two observations. First, because 𝑠 is a Nash equilibrium, the expected utilities 𝐸𝑈 ( ŝ𝑗 , 𝑠 −𝑗 ) in the third line are all at most as big as 𝐸𝑈 (𝑠). Now consider any later term corresponding to the deviation of some set 𝐶, containing at least two players 𝑖, 𝑗. Note that it may be 𝐸𝑈 ( ŝ𝐶 , 𝑠 −𝐶 ) > 𝐸𝑈 (𝑠). However, this term is multiplied by 𝜖 𝑖 𝜖 𝑗 . Thus, as the 𝜖 go to 0, the significance of this term in the average vanishes in comparison to that of both the terms corresponding to the deviation of just 𝑖 and just 𝑗, which are multiplied only by 𝜖 𝑖 and 𝜖 𝑗 , respectively. By non-degeneracy, it is 𝐸𝑈 ( ŝ𝑖 , 𝑠 −𝑖 ) < 𝐸𝑈 (𝑠) or 𝐸𝑈 ( ŝ𝑗 , 𝑠 −𝑗 ) < 𝐸𝑈 (𝑠). Thus, if the 𝜖 𝑖 are small enough, the overall sum is less than 𝐸𝑈 (𝑠). (2) The mixed case: Let 𝑠 be mixed. We proceed by constructing a strategy profile 𝑠 ′ that is arbitrarily close to 𝑠 with 𝐸𝑈 (𝑠 ′ ) > 𝐸𝑈 (𝑠). Let 𝑚 be the largest integer where for all subsets of players 𝐶 ⊆ 𝑁 with |𝐶 | ≤ 𝑚, the expected payoff is constant across all joint deviations to 𝑎 𝑖 ∈ supp(𝑠 𝑖 ) for all 𝑖 ∈ 𝐶, i.e., where 𝐸𝑈 (𝑎 𝐶 , 𝑠 −𝐶 ) = 𝐸𝑈 (𝑠) for all 𝑎 𝐶 ∈ supp(𝑠 𝐶 ). As 𝑠 is a non-degenerate Nash equilibrium, 1 ≤ 𝑚 < 𝑛. By definition of 𝑚, there exists a subset of players 𝐶 ⊂ 𝑁 with |𝐶 | = 𝑚 and choice of actions 𝑎 𝐶 ∈ supp(𝑠 𝐶 ) where 𝐸𝑈 (𝑎 𝑗 , 𝑎 𝐶 , 𝑠 −𝑗−𝐶 ) is not constant across the available actions 𝑎 𝑗 ∈ 𝐴 𝑗 for some player 𝑗 ∈ 𝑁 −𝐶. Denote player 𝑗's best response to the joint deviation 𝑎 𝐶 as 𝑎 * 𝑗 ∈ argmax 𝑎 𝑗 𝐸𝑈 (𝑎 𝑗 , 𝑎 𝐶 , 𝑠 −𝑗−𝐶 ), and note 𝐸𝑈 (𝑎 𝑗 , 𝑎 𝐶 , 𝑠 −𝑗−𝐶 ) > 𝐸𝑈 (𝑎 𝐶 , 𝑠 −𝐶 ) = 𝐸𝑈 (𝑠). To construct 𝑠 ′ , modify 𝑠 by letting player 𝑗 mix according to 𝑠 𝑗 with probability (1 − 𝜖) and play action 𝑎 𝑗 with probability 𝜖. Similarly, let each player 𝑖 ∈ 𝐶 mix according to 𝑠 𝑖 with probability (1 − 𝜖) and play their action 𝑎 𝑖 specified by 𝑎 𝐶 with probability 𝜖. Because we allow 𝜖 > 0 to be arbitrarily small, all we have left to show is 𝐸𝑈 (𝑠 ′ ) > 𝐸𝑈 (𝑠). Observe as before that we can break 𝐸𝑈 (𝑠 ′ ) up into cases based on the number of players who deviate according to the modified   + 𝑘 ∈𝐶∪{ 𝑗 } 𝜖 𝐸𝑈 (𝑎 𝑗 , 𝑎 𝐶 , 𝑠 −𝑗−𝐶 ). By construction, every value in the expected value calculation 𝐸𝑈 (𝑠 ′ ) is equal to 𝐸𝑈 (𝑠) except for the last value 𝐸𝑈 (𝑎 𝑗 , 𝑎 𝐶 , 𝑠 −𝑗−𝐶 ), which is greater than 𝐸𝑈 (𝑠). We conclude 𝐸𝑈 (𝑠 ′ ) > 𝐸𝑈 (𝑠). □
 E GAMUT DETAILS AND ADDITIONAL EXPERIMENTS E.1 GAMUT games In Section 6.1, we analyzed all three classes of symmetric GAMUT games: RandomGame, CoordinationGame, and CollaborationGame. Below, we give a formal definiton of these game classes: Note that these games define payoffs for each unordered action profile because the games are totally symmetric (Definition 2.3). Table  2  gives illustrative examples.
 E.3 Replicator dynamics Consider a game where all players share the same action set, i.e., with 𝐴 𝑖 = 𝐴 𝑗 for all 𝑖, 𝑗, and consider a totally symmetric strategy profile 𝑠 = (𝑠 1 , 𝑠 1 , . . . , 𝑠 1 ). In the replicator dynamic, each action can be viewed as a species, and 𝑠 1 defines the distribution of each individual species (action) in the overall population (of actions). At each iteration of the replicator dynamic, the prevalence of an individual species (action) grows in proportion to its relative fitness in the overall population (of actions). In particular, the replicator dynamic evolves 𝑠 1 (𝑎) over time 𝑡 for each 𝑎 ∈ 𝐴 1 as follows: 𝑑 𝑑𝑡 𝑠 1 (𝑎) = 𝑠 1 (𝑎) [𝐸𝑈 (𝑎, 𝑠 −1 ) − 𝐸𝑈 (𝑠)] . To simulate the replicator dynamic with Euler's method, we need to choose a stepsize and a total number of iterations. Experimentally, we found the fastest convergence with a stepsize of 1, and we found that 100 iterations sufficed for convergence; see Figure  2 . For good measure, we ran 10,000 iterations of the replicator dynamic in all of our experiments. We are interested in the replicator dynamic for two reasons. First, it is a model for how agents in the real world may collectively arrive at a symmetric solution to a game (e.g., through evolutionary pressure). Second, it is a learning algorithm that performs local search in the space of symmetric strategies. In our experiments of Appendix E.5, we find that using the replicator dynamic as an optimization algorithm is competitive with Sequential Least SQuares Programming (SLSQP), a local search method from the constrained optimization literature  [17, 40] . E.4 What fraction of symmetric optima are local optima in possibly-asymmetric strategy space? As discussed in Section 6.2, we would like to get a sense for how often symmetric optima are stable in the sense that they are also local optima in possibly-asymmetric strategy space (see Section 5). Table  3  shows in what fraction of games the best solution we found is unstable.
 E.5 How often do SLSQP and the replicator dynamic find an optimal solution? As discussed in Section 6.3, Table  4  and Table  5  show how often SLSQP finds an optimal solution, while Table  6  and Table  7  show how often the replicator dynamic finds an optimal solution. E.6 How costly is payoff perturbation under the simultaneous best response dynamic? When a game is not stable in the sense of Section 5, we would like to understand how costly the worst-case 𝜖-perturbation of the game can be. (See Definition 4.1 for the definition of an 𝜖-perturbation of a game.) In particular, we study the case when individuals simultaneously update their strategies in possibly-asymmetric ways by defining the following simultaneous best response dynamic: Definition E.4. The simultaneous best response dynamic at 𝑠 updates from strategy profile 𝑠 = (𝑠 1 , 𝑠 2 , . . . , 𝑠 𝑛 ) to strategy profile 𝑠 ′ = (𝑠 ′ 1 , 𝑠 ′ 2 , . . . , 𝑠 ′ 𝑛 ) with every 𝑠 ′ 𝑖 a best response to 𝑠 −𝑖 . For each of the RandomGames in Section 6.2 whose symmetric optimum 𝑠 is not a local optimum in possibly-asymmetric strategy space, we compute the worst-case 𝜖 payoff perturbation for infinitesimal 𝜖. Then, we update each player's strategy according to the simultaneous best response dynamic at 𝑠. This necessarily leads to a decrease in the original common payoff because the players take simultaneous updates on an objective that, after payoff perturbation, is no longer common. Table  8  reports the average percentage decrease in expected utility, which ranges from 55% to 89%. Our results indicate that simultaneous best responses after payoff perturbation in RandomGames can be quite costly.
 F THE COMPUTATIONAL COMPLEXITY OF FINDING THE SYMMETRIES OF A GAME In this section, we analyze the computational complexity of finding the symmetries of a common-payoff game. In general, symmetries as defined in Definition 2.1 can be found in exponential time in the number of players. Therefore, if we represent the game explicitly as a full payoff matrix, then the symmetries can be found in polynomial time in the size of the input. However, if we can represent the game more efficiently by giving only non-zero entries of the payoff matrix, the problem becomes graph isomorphism-complete, i.e., polynomial-time equivalent to the graph isomorphism problem, which is neither known to be solvable in polynomial time nor known to be NP-hard [see 11, for an overview]. We also show (in Section F.3) that if we consider a more general notion of game symmetry that permutes actions in addition to players, the computational problem becomes graph isomorphism-complete on an explicit payoff matrix representation.
 F.1 The hypergraph automorphism problem We here introduce the hypergraph automorphism problem and some existing results about it. In the next section, we will prove our results by relating the game automorphism problem (i.e., the problem of finding the symmetries of a game) to the hypergraph automorphism problem. A hypergraph is a pair (𝑉 , 𝐸), where 𝑉 is a (finite) set of vertices and 𝐸 ⊆ 2 𝑉 is a set of hyperedges. A symmetry or automorphism of a hypergraph is a bijection 𝜌 : 𝑉 → 𝑉 s.t. for each set of vertices 𝑒 ⊆ 𝑉 , it is 𝑒 ∈ 𝐸 if and only if 𝜌 (𝑒) ∈ 𝐸, where 𝜌 (𝑒) {𝜌 (𝑣) | 𝑣 ∈ 𝑒}. In other words: For 𝜌 to be a symmetry it must be the case that any set of vertices 𝑣 1 , ..., 𝑣 𝑘 are connected by a hyperedge if and only if 𝜌 (𝑣 1 ), ..., 𝜌 (𝑣 𝑘 ) are connected by a hyperedge. Definition F.1. The hypergraph automorphism (HA) problem asks for a given hypergraph (𝑉 , 𝐸) to provide a set of symmetries of (𝑉 , 𝐸) that generate the group of all symmetries of (𝑉 , 𝐸). There are two natural ways to represent the edges of a hypergraph. The first is to provide what one would call an adjacency matrix in the case of a regular graph. That is, we give a table of bits that specifies for each 𝑒 ∈ 2 𝑉 whether 𝑒 ∈ 𝐸. That is, for each set of vertices, we specify whether there is a hyperedge connecting that set of vertices. The downside of this representation style is that it always costs 𝑂 (2 |𝑉 | ) bits. An alternative is to explicitly list 𝐸, such that graphs with few edges can be represented in much less space than 𝑂 (2 |𝑉 | ). We call the former notation non-sparse and the latter sparse. Lemma F.2. The HA problem on a sparse hypergraph representation is graph isomorphism-complete. Proof. Mathon  [21]  shows that the problem of giving the generators of the automorphism group of a given graph isomorphism complete. So it is left to show that the automorphism problem on sparse hypergraphs is polynomially equivalent to analogous problem on graphs. Since graphs are hypergraphs, we only need to reduce HA to the graph automorphism problem. This is easy and the main idea has been noted before, e.g., see the introduction of Arvind et al.  [1] . □ Theorem F.3  (Luks, 1999, Theorem 4.2) . The HA problem is solvable in 𝑂 (𝑐 |𝑉 | ) for some constant 𝑐. In particular, it follows immediately that HA on non-sparse representations is solvable in polynomial time. Coordina-tionGame Table  3 : The fraction of games whose symmetric optima are mixed. By Theorem 5.5, these symmetric equilibria are the ones unstable in the sense of Section 5. Numbers in the table were empirically determined from 100 randomly sampled games per GAMUT class.         8 : The average decrease in expected utility that worst-case infinitesimal asymmetric payoff perturbations cause to unstable symmetric optima. To get these numbers, we first perturb payoffs in the 100 RandomGames from Section 6.2 whose symmetric optima 𝑠 are not local optima in possibly-asymmetric strategy space. Then, in each perturbed game, we compute a simultaneous best-response update to 𝑠 and record its decrease in expected utility.
 F.2 Polynomial-time equivalence Recall from the main text that -for the purpose of our paper -a symmetry of an 𝑛-player (common-payoff) game (𝐴, 𝑢) is a permutation 𝜌 : {1, ..., 𝑛} → {1, ..., 𝑛} s.t. for all pure strategy profiles a ∈ 𝐴, it is 𝑢 (𝑎 1 , ..., 𝑎 𝑛 ) = 𝑢 (𝑎 𝜌 (1) , ..., 𝑎 𝜌 (𝑛) ). In particular, we do not consider permutations of the actions. Definition F.4. The (common-payoff) game automorphism (GA) problem asks us to compute for a common-payoff given game a generating set of the symmetries of the game. As with hypergraphs, we distinguish two representations for a game. A sparse representation lists only non-zero payoffs. A nonsparse representation gives the payoff for each 𝐴. As before, the downside of the full payoff table representation is that its size is exponential in the number of players. Theorem F.5. The sparse/non-sparse representation HA problem is polynomial-time equivalent to the sparse/non-sparse representation GA problem. By polynomial-time, we here mean in time bound by a polynomial in the number of players and the size in bits of the given instance. Proof. HA→GA: We first reduce the HA problem to the GA problem, which is the easier direction. We will use the same construction for both the sparse-to-sparse and non-sparse-to-nonsparse case. Take a given hypergraph (𝑉 , 𝐸). WLOG assume 𝑉 = {1, ..., 𝑛}. Then we construct an 𝑛-player game, in which each player has two actions, 𝑎 0 , 𝑎 1 . For any 𝑀 ⊆ {1, ..., 𝑛}, let a 𝑀 be the payoff profile in which the set of players playing 𝑎 1 is exactly 𝑀. Then let the payoff 𝑢 (a 𝑀 ) be 1 if 𝑀 ∈ 𝐸 and 0 otherwise. We now show that this reduction is valid by showing that the game and the graph have the same symmetries. Let 𝜌 be a bijection. Then: 𝜌 is a symmetry of (𝑉 , 𝐸) iff ∀𝑒 ∈ 2 𝑉 : 𝑒 ∈ 𝐸 ⇐⇒ 𝜌 (𝑒) ∈ 𝐸 (2) iff ∀𝑀 ∈ 2 𝑉 : 𝑢 (a 𝑀 ) = 1 ⇐⇒ 𝑢 (a 𝜌 (𝑀) ) = 1 (3) iff ∀𝑀 ∈ 2 𝑉 : 𝑢 (a 𝑀 ) = 𝑢 (a 𝜌 (𝑀) ) (4) iff ∀a ∈ 𝐴 : 𝑢 (𝑎 1 , ..., 𝑎 𝑛 ) = 𝑢 (𝑎 𝜌 (1) , ..., 𝑎 𝜌 (𝑛) ) iff 𝜌 is a symmetry of (𝐴, 𝑢) It is easy to see that this construction can be performed in polynomial (indeed linear) time for both sparse and non-sparse representations. GA→HA: We now reduce in the opposite direction. This is more complicated and we therefore provide only a sketch. Consider an 𝑛-player game (𝐴, 𝑢). We construct the hypergraph as follows. First, for each player 𝑖, we generate a vertex. We also generate log 2 (|𝐴 𝑖 |) vertices that we use to encode 𝐴 𝑖 , player 𝑖's actions and connect them with the vertex representing 𝑖. For players 𝑖, 𝑗 that have the same action label sets, this encoding must be done consistently for 𝑖, 𝑗. We also need to add some kind of structure to ensure that symmetries of the hypergraph can only map the 𝑘th action-encoding vertex of player 𝑖 on the 𝑘th action-encoding vertex of a player 𝑗 that has the same action label set as 𝑖. Next, we represent the payoff function 𝑢. To do so, we introduce ⌈log 2 (|𝑢 (𝐴) − {0}|)⌉ payoff encoding vertices. Note that |𝑢 (𝐴) − {0}| is the number of distinct non-zero payoffs of the game. To encode |𝑢 (𝐴) − {0}| we therefore need ⌈log 2 (|𝑢 (𝐴) − {0}|)⌉ bits. We connect these bits in such a way that any symmetry must map each of them onto itself. We fix some binary encoding of 𝑢 (𝐴) − {0}. For instance, let's say the non-zero payoffs of the game are {−3, −1, 7, 8, 10, 11, 13, 100}. Then we need three bits, and might encode them as −3 ↦ → 000, −1 ↦ → 001, 7 ↦ → 010, 8 ↦ → 011, and so forth. For each a ∈ 𝐴 with 𝑢 (a) ≠ 0, we then add an edge that contains for each Player 𝑖 the action encoding vertices corresponding to 𝑎 𝑖 ; and those bits from the payoff encoding vertices that together represent the payoff. (So, for example, if the payoff is encoded by 011, then the hyperedge contains the two lower payoff encoding vertices. Similarly for the action encoding vertices.) We omit a proof of the correctness of this reduction. It is left to show that the reduction is polynomial-time for both representation styles. For the sparse representation styles, it is trivial because up to some small number of extra vertices and edges, there is a one-to-one correspondence between edges of the hypergraph and action profiles with non-zero payoffs. On to the non-sparse representation. Clearly each entry of the adjacency matrix can be filled in polynomial (perhaps even constant or logarithmic) time. It is left to show that the adjacency matrix is not too large. In particular, we need to show that the size of the adjacency matrix is polynomial in the size of the payoff matrix. To assess the size of the adjacency matrix we need to count the number of vertices in the above construction. First, the number of player vertices is 𝑛 ≤ log 2 (|𝐴 1 |) + ... + log 2 (|𝐴 𝑛 |) = log 2 (|𝐴 1 | • ... • |𝐴 𝑛 |) = log 2 (|𝐴|). (The inequality assumes each player has at least two actions.) Second, the number of action-encoding vertices is ⌈log 2 (|𝐴 1 |)⌉ + ... + ⌈log 2 (|𝐴 𝑛 |)⌉ ≤ 2 log 2 (|𝐴 1 |) + ... + 2 log 2 (|𝐴 𝑛 |) = 2 log 2 (|𝐴|). Finally, the number of payoff encoding vertices is about ⌈log 2 (|𝑢 (𝐴)|)⌉ ≤ 2 log 2 (|𝑢 (𝐴)|) ≤ 2 log 2 (|𝐴|). The overall number of vertices in the above construction is therefore at most 5 log 2 (|𝐴|). Thus, the size (in terms of number of bits) of the adjacency matrix is bound by 2 5(log 2 |𝐴 |) = |𝐴| 5 . Since |𝐴| is a lower bound on the size of the payoff matrix (in bits), this is polynomial in the size of the game's payoff matrix, as required. □ One might wonder: In the non-sparse representation case, why does the reduction to HA not also work if we use a more traditional sense of game symmetries? If it were to work that would show that GI is polynomial-time solvable! But this does not work (with the proof strategy used above). In the current reduction, actions do not get their own vertices. Thus, (even if we dropped the constraint structures that prevent actions from being remapped), a hypergraph automorphism cannot remap, e.g., an action encoded as 11011 to an action encoded as 01010. To express full action relabelings in the hypergraph, it seems that we need to introduce an action per vertex. However, the size of the adjacency matrix then blows up more than polynomially. Combining Lemma F.2 and Theorems F.3 and F.5, we get a characterization of the complexity of the graph automorphism problem. Corollary F.6. GA is solvable in polynomial time on a non-sparse representation and is GI-complete on a sparse representation.
 F.3 An alternative notion of game symmetry As mentioned in the main text, we only consider symmetries that relabel the players and the above is on the computational problem resulting from that notion of symmetry. As noted in footnote 3, this was done in part to keep notation simple and an alternative, slightly more complicated notion allows for the actions to be permuted. A natural question then is what the complexity is of finding this new type of symmetry in a given common-payoff game. In this case, the answer is that finding symmetries is GI-complete regardless of how the game is represented and follows almost immediately from existing ideas from Mathon  [21]  and Gabarró et al.  [10] . A player-action (PA) symmetry of a game is pair of a bijection 𝜌 : {1, ..., 𝑛} → {1, ..., 𝑛} on players and a family of bijections 𝜏 𝑖 : 𝐴 𝜌 (𝑖) → 𝐴 𝑖 s.t. for all pure strategy profiles (𝑎 1 , ..., 𝑎 𝑛 ), 𝑢 (𝑎 1 , ..., 𝑎 𝑛 ) = 𝑢 (𝜏 1 (𝑎 𝜌 (1) ), ..., 𝜏 𝑛 (𝑎 𝜌 (𝑛) )). So the idea in this new definition is that 𝜏 𝑖 translates the action names from those of player 𝜌 (𝑖) to player 𝑖. Define the PAGA problem analogously to the GA problem above, as finding a generating set of the PA symmetries of a given commonpayoff game. This time, the complexity is independent of whether the game is represented sparsely or not. Theorem F.7. The PAGA problem is GI-complete. Proof. PAGA→GI For their proof of GI-completeness of the game isomorphism problem, Gabarró et al.  [10]  sketch how a generalsum game can be represented as a graph in a way that maintains the isomorphisms. In particular, we can therefore represent a single common-payoff game as a graph in a way that maintains PA symmetries. The have thus given a sketch of a polynomial-time reduction from PAGA to the problem of finding the automorphisms of a graph. This latter problem can in turn be reduced in polynomial time to the graph isomorphism problem as was shown by Mathon  [21] . GI→PAGA Second, we show GI-hardness. As shown by Mathon  [21] , it is enough to reduce the graph automorphism problem to the PAGA problem. For this, we can slightly modify the construction of Gabarró et al.  [10, Lemma 5] . Specifically, they reduce the graph isomorphism problem to the 4-player general-sum game isomorphism problem, where actions represent vertices and the graph isomorphisms can be recovered from those PA game isomorphisms where the player isomorphism is 𝜌 = id. Obviously, the same construction can be used to reduce the graph automorphism problem to the general-sum game automorphism problem. The only issue is therefore that their construction uses general-sum games, but we can simply encode their payoff vectors as single numbers. In particular, because their payoffs are binary, we might translate (0, 0, 0, 0) ↦ → 0, (0, 0, 0, 1) ↦ → 1,(0, 0, 1, 0) ↦ → 2, and so forth. It  '  for some 𝑚, where the 𝑐 (𝑒 1 ,...,𝑒 𝑘 ) are some set of real coefficients. The terms 𝑐 (𝑒 1 ,...,𝑒 𝑘 ) 𝑥 𝑒 1 1 ...𝑥 𝑒 𝑘 1 for which 𝑐 (𝑒 1 ,...,𝑒 𝑘 ) ≠ 0 are called the monomials of the polynomial. The maxdegree of a monomial 𝑐 (𝑒 1 ,...,𝑒 𝑘 ) 𝑥 𝑒 1 1 ...𝑥 𝑒 𝑘 1 is max 𝑖=1,...,𝑘 𝑒 𝑖 . The maxdegree of a polynomial is the maximum of the maxdegrees of its monomials. Similarly, the total degree of a monomial 𝑐 (𝑒 1 ,...,𝑒 𝑘 ) 𝑥 𝑒 1 1 ...𝑥 𝑒 𝑘 1 is 𝑘 𝑖=1 𝑒 𝑖 . The total degree of a polynomial is the maximum of the total degrees of its monomials. The degree of a variable 𝑥 𝑖 in a monomial 𝑐 (𝑒 1 ,...,𝑒 𝑘 ) 𝑥 𝑒 1 1 ...𝑥 𝑒 𝑘 1 is 𝑒 𝑖 . The maxdegree of 𝑥 𝑖 in a polynomial is the maximum of the maxdegrees of 𝑥 𝑖 in all of the monomials. We can partition the parameters of a polynomial into vectors and write the polynomial as 𝑓 (x 1 , ..., x 𝑘 ), where x 1 , ..., x 𝑘 are real vectors. We define the degree of x 𝑖 in a monomial as the sum of the degrees of the entries of x 𝑖 in the monomial. We define the maxdegree of x 𝑖 in the polynomial as the maximum of the degrees of x 𝑖 in the polynomial's monomials. In the following we will interpret the set Δ(𝐴 𝑖 ) of probability distributions over 𝐴 𝑖 as the set of |𝐴 𝑖 |-dimensional vectors of nonnegative reals whose entries sum to one. We will index these vectors by 𝐴 𝑖 (rather than numbers 1, ..., |𝐴 𝑖 |). The sets Δ(𝐴 𝑖 ) are also called unit simplices.
 G.2 Optimizing symmetric strategies as maximizing polynomials It is immediately obvious that in a symmetric game, the expected utility as a function of the probabilities that each of the orbits assign to each of the strategies is a polynomial over a Cartesian product of unit simplices. Formally: Proposition G.1. Let G be an 𝑛-player game and P ⊆ Γ(G) be a subset of the game symmetries of G. Let the orbits of P be 𝑀 1 , ..., 𝑀 𝑘 . Further, let the set of actions of orbit 𝑖 be 𝐴 𝑖 . Then the expected utility function over P-invariant strategy profiles of G is a polynomial over Δ(𝐴 1 ) × ... × Δ(𝐴 𝑘 ) with a max degree of (at most) max 𝑖 |𝑀 𝑖 | and a total degree of (at most) 𝑛. This polynomial can be created in polynomial-time in the size of a sparse or non-sparse (as per Appendix F.2) representation of the game. It follows that we can use algorithms for optimizing polynomials to find optimal symmetric strategies and that positive results on optimizing polynomials transfer to finding optimal mixed strategies. Unfortunately, these results are generally somewhat cumbersome to state. This is because the optimum can in general not be represented exactly algebraically, even using 𝑛-th roots, as implied by the Abel-Ruffini theorem. Positive results must therefore be given in terms of approximations of the optimal solution. One striking result from the literature is that, roughly speaking, for a fixed number of variables, the optimal solution can be approximated in polynomial time  [16, Section 6.1] . Translated to our setting, this means that the optimal symmetric strategy can be approximated in polynomial time if we keep constant the number of orbits and the number of actions available to each orbit, but potentially increase the number of players in each orbit. For more discussion of the complexity of optimizing polynomials on unit simplices, see de Klerk  [6] .
 G.3 Expressing polynomials as symmetric games We now show that, conversely, for any polynomial over a Cartesian product of simplices there exists a symmetric game whose expected utility term is exactly that polynomial. However, depending on how we represent polynomials and how we represent games, the size of the game may blow up exponentially. We first show that each polynomial over Δ(𝐴 1 ) × ... × Δ(𝐴 𝑘 ) can be rewritten in such a way that each input x 𝑖 appears in the same degree in all monomials. Lemma G.2. Let 𝑓 (x 1 , ..., x 𝑘 ) be a polynomial on real vectors of dimensions 𝐴 1 , ..., 𝐴 𝑘 . Then there exists a polynomial 𝑔 on the same inputs s.t. for all (x 1 , ..., x 𝑘 ) ∈ Δ(𝐴 1 ) × ... × Δ(𝐴 𝑘 ) 𝑔(x 1 , ..., x 𝑘 ) = 𝑓 (x 1 , ..., x 𝑘 ), and the degree of every x 𝑖 in all monomials of 𝑔 is the maxdegree of x 𝑖 in 𝑓 . Proof. Consider any monomial f of 𝑓 in which x 𝑖 does not have its max degree. Then for all (x 1 , ..., x 𝑘 ) ∈ Δ(𝐴 Notice that this is the sum of |𝐴 𝑖 | monomials in which x 𝑖 occurs in 1 plus the degree in which it occurs in f . We can iterate this transformation until we arrive at the desired f . □ Note, however, that if web take a given polynomial represented as a sum of monomials -e.g., 𝑓 (𝑥 1 , 𝑥 2 ) = 𝑥  4  1 − 3𝑥 2 -and rewriting it as outlined in the Lemma and its proof, the size may blow up exponentially. E.g., 𝑓 (𝑥 1 , 𝑥 2 ) = 𝑥  4  1 − 3𝑥 2 = 𝑥 4 1 − 3(𝑥 1 + 𝑥 2 ) 3 𝑥 2 and (𝑥 1 + 𝑥 2 ) 3 expands into a sum of 2 3 = 8 terms. However, in some table-of-coefficient representations of polynomials the size of the instance does not change at all and the transformation can be performed in polynomial time in the input. For example, this is the case if 𝑘 = 1 and we represent a polynomial as a table of the coefficients of all terms 𝑥 𝑒 1 1 ...𝑥 𝑒 𝑘 𝑘 where 𝑒 1 + ... + 𝑒 𝑘 are at most the polynomial's maxdegree. Once we have a polynomial of the structure described in Lemma G.2, we can transform it into a game: Proposition G.3. Let 𝑓 (x 1 , ..., x 𝑘 ) be a polynomial in which each x 𝑖 appears in the same degree in all monomials. Then we can construct a game G with symmetries P that create 𝑘 orbits, where the number of players in orbit 𝑖 = 1, ..., 𝑘 is the degree of x 𝑖 in 𝑓 and the number of actions for the players in orbit 𝑖 is the number of entries of x 𝑖 . Proof. Consider games Γ with orbits 𝑀 1 , ..., 𝑀 𝑘 of the specified sizes and sets of actions 𝐴 1 , ..., 𝐴 𝑘 also of the specified sizes where specifically the players in each 𝑀 𝑖 are totally symmetric. Then such a game if fully specified as follows. For each family of numbers 𝑛 1,1 , ..., 𝑛 □ Note that if the polynomial is represented as a table of coefficients, then this reduction takes linear time in the size of the input. Similarly, if the polynomial is given as a list of only the monomials with non-zero coefficients -all of which satisfy the degree requirement -the reduction can also be done in polynomial time. This in particular gives us the following negative result, translated from the literature on optimizing polynomials: Corollary G.4. Deciding for a given game G with symmetries P and a given number 𝐾 whether there is a P-invariant profile with expected utility at least 𝐾 is NP-hard, even for 2-player symmetric games. Proof. Follows from Proposition G.3 and the NP-hardness of optimizing quadratic polynomials over the unit simplex [6, Section 3.2]. □ Figure 1 : 1 Figure 1: The strategy profile landscape of the symmetric laundry game (Figure1b). Although the symmetric optimum has lower expected utility than the unrestricted optima, total symmetry of the game implies that the symmetric optimum is a Nash equilibrium; this is a special case of Theorem 3.2.
 Example 5 . 6 . 56 Consider the 3x3 symmetric game with the following payoff matrix: 10 1 + 𝜖 c 1 1 + 𝜖
 probability 𝜖: 𝐸𝑈 (𝑠 ′ ) = 𝑘 ∈𝐶∪{ 𝑗 } (1 − 𝜖) 𝐸𝑈 (𝑠) + 𝑙 ∈𝐶∪{ 𝑗 } 𝜖 𝑘 ∈𝐶∪{ 𝑗 }:𝑘≠𝑙 1 − 𝜖 𝐸𝑈 (𝑎 𝑙 , 𝑠 −𝑙 ) + ...
 Definition E. 1 . 1 A RandomGame with |𝑁 | players and |𝐴| actions assumes 𝐴 𝑖 = 𝐴 𝑗 for all 𝑖, 𝑗 and draws a payoff from Unif (−100, 100) for each unordered action profile 𝑎 ∈ 𝐴. Definition E.2. A CoordinationGame with |𝑁 | players and |𝐴| actions assumes 𝐴 𝑖 = 𝐴 𝑗 for all 𝑖, 𝑗. For each unordered action profile 𝑎 ∈ 𝐴 with 𝑎 𝑖 = 𝑎 𝑗 for all 𝑖, 𝑗, it draws a payoff from Unif (0, 100); for all other unordered action profiles, it draws a payoff from Unif (−100, 0). Definition E.3. A CollaborationGame with |𝑁 | players and |𝐴| actions assumes 𝐴 𝑖 = 𝐴 𝑗 for all 𝑖, 𝑗. For each unordered action profile 𝑎 ∈ 𝐴 with 𝑎 𝑖 = 𝑎 𝑗 for all 𝑖, 𝑗, the payoff is 100; for all other unordered action profiles, it draws a payoff from Unif (−100, 99).
 Figure 2 : 2 Figure 2: The magnitude of the replicator dynamics update step averaged over 10,000 RandomGames 4 with 2 players and 2 actions. Although this plot indicates that the replicator dynamics converge by 100 iterations, we ran 10,000 iterations for good measure in all of our experiments.
 31 ( 31 92 0.81 0.70 0.64 3 0.80 0.69 0.57 0.48 4 0.75 0.57 0.40 0.35 5 0.70 0.45 0.36 0.59 0.50 0.40 0.33 3 0.53 0.38 0.28 0.29 4 0.53 0.37 0.29 0.26 5 0.53 0.36 0.33 0.25 (b) CoordinationGame
 𝑛 𝑗=1 𝑠 𝑗 (𝑎 𝑗 ). If a strategy 𝑠 𝑖 for player 𝑖 maximizes expected utility given the strategies 𝑠 −𝑖 of all the other players, i.e., if 𝑠 𝑖 ∈ argmax 𝑠 ′ 𝑖 ∈Δ(𝐴 𝑖 ) 𝐸𝑈 𝑖 (𝑠 ′ 𝑖 , 𝑠 −𝑖 ), we call 𝑠 𝑖 a best response to 𝑠 −𝑖 . If each strategy 𝑠 𝑖 in a strategy profile 𝑠 is a best response to 𝑠 −𝑖 , we call 𝑠 a Nash equilibrium. A Nash equilibrium 𝑠 is strict if every 𝑠 𝑖 is the unique best response to 𝑠 −𝑖 .
 if every 𝑠 𝑖 is a Dirac delta function on some 𝑎 𝑖 , then 𝑠 is degenerate if at least two players 𝑖 are indifferent between 𝑎 𝑖 and some other 𝑎 ′ 𝑖 ∈ 𝐴 𝑖 − {𝑎 𝑖 }. • Otherwise, if 𝑠 is mixed, then 𝑠 is degenerate if for all players 𝑖 and all 𝑎 −𝑖 ⊆ supp(𝑠 −𝑖 ), the term 𝐸𝑈 𝑖 (𝑎 𝑖 , 𝑎 −𝑖 ) is constant across 𝑎 𝑖 ∈ supp(𝑠 𝑖 ).
 Table 4 / 4 Table 6, we show what fraction of the time any single SLSQP / replicator dynamics run finds the best solution, and in Appendix Table 5 / Table 7, weshow what fraction of the time at least 1 of 10 SLSQP / replicator dynamics runs finds the best solution. First, we note that the tables for SLSQP and the replicator dynamics are quite similar, differing by no more than a few percentage points in all cases. So the replicator dynamics, which are used as a model for how populations evolve strategies, can also be used as an effective optimization algorithm. Second, we see that individual runs of each algorithm are up to 93% likely to find the best solution in small RandomGames, but they are less likely (as little as 24% likely) to find the best solution in larger RandomGames and in CoordinationGames. The best of 10 runs, however, finds the best solution ≥ 87% of the time, indicating that random algorithm restarts benefit symmetric strategy optimization.
 𝐸𝑈 𝑖 (𝑎 𝑖 , 𝑠 * −𝑖 ) + 𝐸𝑈 𝑖 (𝑎 𝑖 , 𝑠 * −𝑖 ) − 𝐸𝑈 𝑖 (𝑎 𝑖 , 𝑠 −𝑖 ) ≤ 𝐸𝑈 𝑖 (𝑎 ′ 𝑖 , 𝑠 −𝑖 ) − 𝐸𝑈 𝑖 (𝑎 ′ 𝑖 , 𝑠 * −𝑖 ) + 𝐸𝑈 𝑖 (𝑎 𝑖 , 𝑠 * −𝑖 ) − 𝐸𝑈 𝑖 (𝑎 𝑖 , 𝑠 −𝑖 ) ≤ 𝐸𝑈 𝑖 (𝑎 ′ 𝑖 , 𝑠 −𝑖 ) − 𝐸𝑈 𝑖 (𝑎 ′ 𝑖 , 𝑠 * −𝑖 ) + 𝐸𝑈 𝑖 (𝑎 𝑖 , 𝑠 −𝑖 ) − 𝐸𝑈 𝑖 (𝑎 𝑖 , 𝑠 * Let G be a common-payoff normal-form game, and let 𝑠 * be a locally-optimal P-invariant strategy profile for some subset of game symmetries P ⊆ Γ(G). Suppose 𝑠 is a strategy profile with Kullback-Leibler divergence satisfying 𝐷 𝐾𝐿 (𝑠 ||𝑠 * ) ≤ 𝜈 or 𝐷 𝐾𝐿 (𝑠 𝑠  *  −𝑖 ) + 𝐸𝑈 𝑖 (𝑎 ′ 𝑖 , 𝑠  *  −𝑖 ) − −𝑖 ) ≤ 4𝛿 max 𝑖 ∈𝑁 ,𝑎 ∈𝐴 |𝑢 𝑖 (𝑎)|, where 𝐸𝑈 𝑖 (𝑎 ′ 𝑖 , 𝑠  *  −𝑖 ) − 𝐸𝑈 𝑖 (𝑎 𝑖 , 𝑠  *  −𝑖 ) ≤ 0 because 𝑠  *  𝑖 is a Nash equilib- rium by Theorem 3.2. □ Corollary 4.4. * ||𝑠) ≤ 𝜈. Then 𝑠 is an 𝜖-Nash equilibrium with 𝜖 = 2 √ 2𝜈 max 𝑖 ∈𝑁 ,𝑎 ∈𝐴 |𝑢 𝑖 (𝑎)|.Proof. By Pinsker's inequality [35] , we have𝑇𝑉 (𝑠, 𝑠 * ) ≤ 1 2𝐷 𝐾𝐿 (𝑠 ||𝑠 * ).As 𝑇𝑉 (𝑠, 𝑠 * ) = 𝑇𝑉 (𝑠 * , 𝑠) and with a similar application of Pinsker's inequality, we have by assumption that 𝑇𝑉 (𝑠, 𝑠 * ) ≤ 𝜈/2. Applying Theorem 4.3 with 𝛿 = 𝜈/2 yields the result. □
 Table 2 : 2 𝛼 𝑢 𝛼𝛼 𝑢 𝛼 𝛽 𝛽 𝑢 𝛼 𝛽 𝑢 𝛽𝛽 A payoff matrix with |𝑁 | = 2 and 𝐴 1 = 𝐴 2 = {𝛼, 𝛽} to illustrate GAMUT games. In a RandomGame, 𝑢 𝛼𝛼 , 𝑢 𝛼 𝛽 , and 𝑢 𝛽𝛽 are i.i.d. draws from Unif (−100, 100). In a Coordi-nationGame, 𝑢 𝛼𝛼 and 𝑢 𝛽𝛽 are i.i.d. draws from Unif (0, 100) while 𝑢 𝛼 𝛽 is a draw from Unif (−100, 0). In a Collabora-tionGame, 𝑢 𝛼𝛼 = 𝑢 𝛽𝛽 = 100, and 𝑢 𝛼 𝛽 is a draw from Unif (−100, 99). Player 2 𝛼 𝛽 Player 1
 Table 4 : 4 The fraction of single SLSQP runs that achieve the best solution found in our 20 total optimization attempts. Numbers in the table were empirically determined from 100 randomly sampled games per GAMUT class. A 2 3 4 5 N 2 1.00 0.99 0.99 0.98 3 1.00 0.99 1.00 0.96 4 1.00 0.96 0.94 0.88 5 0.98 0.90 0.88 0.91 (a) RandomGame
 Table 5 : 5 The fraction of games in which at least 1 of 10 SLSQP runs achieves the best solution found in our 20 total optimization attempts. Numbers in the table were empirically determined from 100 randomly sampled games per GAMUT class. A 2 3 4 5 N 2 0.93 0.81 0.68 0.65 3 0.81 0.70 0.58 0.46 4 0.76 0.58 0.36 0.34 5 0.69 0.43 0.36 0.30
 Table 6 : 6 The fraction of single replicator dynamics runs that achieve the best solution found in our 20 total optimization attempts. Numbers in the table were empirically determined from 100 randomly sampled games per GAMUT class. This is harder to show. Note that a brute force method that tests all |𝑉 |! bijections is super-exponential in |𝑉 | and super-polynomial in (the problem size) 2 |𝑉 | . A 2 3 4 5 A 2 3 4 5 N N 2 1.00 1.00 1.00 1.00 2 1.00 1.00 0.99 0.94 3 0.99 1.00 0.95 0.96 3 1.00 0.97 0.93 0.96 4 1.00 0.98 0.91 0.91 4 0.99 1.00 0.93 0.92 5 0.98 0.97 0.92 0.87 5 1.00 0.98 0.96 0.90 (a) RandomGame (b) CoordinationGame
 Table 7 : 7 The fraction games in which at least 1 of 10 replicator dynamics runs achieves the best solution found in our 20 total optimization attempts. Numbers in the table were empirically determined from 100 randomly sampled games per GAMUT class. A 2 3 4 5 N 2 58.9% 55.9% 61.8% 64.6% 3 73.7% 70.9% 73.4% 73.7% 4 74.1% 77.4% 78.4% 82.5% 5 77.4% 84.9% 89.9% 87.5% (a) RandomGame
 s easy to see that the symmetries with 𝜌 = id remain the same under this transformation. (multivariate) polynomial in 𝑘 variables is a function (𝑥 1 , ..., 𝑥 𝑘 ) ↦ → 𝑒 1 ,...,𝑒 𝑘 ∈ {1,...,𝑚 } 𝑐 (𝑒 1 ,...,𝑒 𝑘 ) 𝑥 𝑒 1 1 ...𝑥 𝑒 𝑘 □ G THE COMPUTATIONAL COMPLEXITY OF FINDING OPTIMAL SYMMETRIC STRATEGIES G.1 Polynomials 1 A
 1 ) × ... × Δ(𝐴 𝑘 ), f (x 1 , ..., x 𝑘 ) = 𝑎 𝑖 ∈𝐴 𝑖 𝑥 𝑖,𝑎 𝑖 f (x 1 , ..., x 𝑘 ) = 𝑎 𝑖 ∈𝐴 𝑖 𝑥 𝑖,𝑎 𝑖 f (x 1 , ..., x 𝑘 ).
 1, |𝐴 1 | , ...., 𝑛 𝑘,1 , ..., 𝑛 𝑘, |𝐴 𝑘 | with 𝑛 𝑖,1 + ... + 𝑛 𝑖, |𝐴 𝑖 | = |𝑀 𝑖 | we need to specify the utility 𝑣 obtained if for all 𝑖, 𝑙, 𝑛 𝑖,𝑗 players in orbit 𝑖 play action number 𝑗 from 𝐴 𝑖 . In the expected utility function of G, each such entry creates a summand 𝑛 𝑖,1 , ..., 𝑛 𝑖, |𝐴 𝑖 | 𝑖,𝑙 𝑝 𝑛 𝑖,𝑙 𝑖,𝑙 , where 𝑝 𝑖,𝑙 is the probability with which players in orbit 𝑖 player action 𝑙 and |𝑀 𝑖 | 𝑛 𝑖,1 ,...,𝑛 𝑖,|𝐴 𝑖 | is a multinomial. By setting 𝑣 appropriately, we can thus obtain any monomial with exponents (𝑛 𝑖,𝑙 ) 𝑖,𝑙 . By setting the values 𝑣 all different sets of (𝑛 𝑖,𝑙 ) 𝑖,𝑙 appropriately, we obtain any polynomial in which each (𝑛 𝑖,𝑙 ) 𝑙 appears with the same degree |𝑀 𝑖 | in all monomials. 𝑣 • 𝑖|𝑀 𝑖 |
			 This condition is relaxed in (weighted) potential games where the players' payoffs need only imply the same ordering of outcomes [33] ; (weighted) potential games are best-response equivalent to common-payoff games [15, 26] .
			 We make this choice to ease notational burden, but we conjecture that our results can be generalized to allow for mappings between actions [14] , which we leave for future work.
			 In this simulation only we rescaled the RandomGames so that each payoff is a draw from Unif (0, 1).

Abstract: Although it has been known since the 1970s that a globally optimal strategy profile in a common-payoff game is a Nash equilibrium, global optimality is a strict requirement that limits the result's applicability. In this work, we show that any locally optimal symmetric strategy profile is also a (global) Nash equilibrium. Applied to machine learning, our result provides a global guarantee for any gradient method that finds a local optimum in symmetric strategy space. Furthermore, we show that this result is robust to perturbations to the common payoff and to the local optimum. While these results indicate stability to unilateral deviation, we nevertheless identify broad classes of games where mixed local optima are unstable under joint, asymmetric deviations. We analyze the prevalence of instability by running learning algorithms in a suite of symmetric games, and we conclude with results on the complexity of computing game symmetries.