Shihui Li
Yi Wu
Xinyue Cui
Honghua Dong
Fei Fang
Stuart Russell

Robust Multi-Agent Reinforcement Learning via Minimax Deep Deterministic Policy Gradient

Introduction Most real-world problems involve interactions between multiple agents and the complexity of problem increases significantly when the agents co-evolve together. Thanks to the recent advances of deep reinforcement learing (DRL) on single agent scenarios, which led to successes in playing Atari game  (Mnih et al. 2015) , playing go  (Silver et al. 2016 ) and robotics control  (Levine et al. 2016) , it has been a rising trend to adapt single agent DRL algorithms to multi-agent learning scenarios and many works have shown great successes on a variety of problems, including automatic discovery of communication and language  (Sukhbaatar, Fergus, and others 2016; Mordatch and Abbeel 2017) , multiplayer games  (Peng et al. 2017a; OpenAI 2018) , traffic control  (Wu et al. 2017 ) and the analysis of social dilemmas  (Leibo et al. 2017) . The critical challenge when adapting classical single agent DRL algorithms to multi-agent setting is the training instabil-ity issue: as training progresses, each agent's policy is changing and therefore the environment becomes non-stationary from the perspective of any individual agent (in a way that is not explainable by changes in the agent's own policy). This non-stationary problem can cause significant problems when directly applying the single agent DRL algorithms, for example, the variance of the policy gradient can be exponentially large when the number of agents increases  (Lowe et al. 2017 ). To handle this instability issue, recent works, such as the counterfactual multi-agent policy gradients  (Foerster et al. 2017 ) and the Multi-Agent Deep Deterministic Policy Gradient (MADDPG)  (Lowe et al. 2017) , proposed to utilized a centralized critic within the actor-critic learning framework to reduce the variance of policy gradient. Despite the fact that using a centralized critic stabilizes training, the learned policies can still be brittle and sensitive to its training partners and converge to a poor local mode. This is particularly severe for competitive environments: when the opponents alter their policies during testing, the performance of the learned policies can be drastically worse  (Lazaridou, Peysakhovich, and Baroni 2016) . Hence, a robust policy becomes desirable in multi-agent setting: a well-trained agent should be able behave well in testing when competing against opponents even with strategies different from its training partners. In this work, we focus on robust multi-agent reinforcement learning with continuous action spaces and propose a novel algorithm, MiniMax Multi-agent Deep Deterministic Policy Gradient (M3DDPG). M3DDPG is a minimax extension 1 of the MADDPG algorithm  (Lowe et al. 2017) . Its core idea is that during training, we force each agent to behave well even when its training opponents response in the worst way. Our major contributions are summarized as follow: • We introduce the minimax approach to robust multi-agent DRL and propose a novel minimax learning objective based on the MADDPG algorithm; • In order to efficiently optimize the minimax learning objective, we propose an end-to-end learning approach, Multiagent Adversarial Learning (MAAL), which is inspired by the adversarial training  (Goodfellow, Shlens, and Szegedy 2014)  technique. • We empirically evaluate our proposed M3DDPG algorithm on four mixed cooperative and competitive environments and the agents trained by M3DDPG outperform baseline policies on all these environments. In the rest of the paper, we will firstly present related works in section 2. Notations and standard algorithms are described in section 3. Our main algorithm, M3DDPG, is introduced in section 4. Experimental results are in section 5.
 Related Work Multi-agent reinforcement learning  (Littman 1994 ) has been a long-standing field in AI  (Hu, Wellman, and others 1998; Busoniu, Babuska, and De Schutter 2008) . Recent works in DRL use deep neural networks to approximately represent policy and value functions. Inspired by the success of DRL in single-agent settings, many DRL-based multi-agent learning algorithms have been proposed.  Forester et al. (2016b)  and  He et al. (2016)  extended the deep Q-learning to multiagent setting;  Peng et al. (2017a)  proposed a centralized policy learning algorithm based on actor-critic policy gradient; Forester et al. (2016a) developed a decentralized multi-agent policy gradient algorithm with centralized baseline;  Lowe et al. (2017)  extended DDPG to multi-agent setting with a centralized Q function;  Wei et al. (2018)  and  Grau-Moya (2018)  proposed multi-agent variants of the soft-Q-learning algorithm  (Haarnoja et al. 2017) ;  Yang et al. (2018)  focused on multi-agent reinforcement learning on a very large population of agents. Our M3DDPG algorithm is built on top of MAD-DPG and inherits the decentralized policy and centralized critic framework. Minimax is a fundamental concept in game theory and can be applied to general decision-making under uncertainty, prescribing a strategy that minimizes the possible loss for a worst case scenario  (Osborne and others 2004) . Minimax was firstly introduced to multi-agent reinforcement learning as minimax Q-learning by  Littman (1994) . More recently, some works combine the minimax framework and the DRL techniques to find Nash equilibrium in two player zero-sum games  (Foerster et al. 2018; Pérolat et al. 2016; Grau-Moya, Leibfried, and Bou-Ammar 2018) . In our work, we utilize the minimax idea for the purpose of robust policy learning. Robust reinforcement learning was originally introduced by  Morimoto et al. (2005)  considering the generalization ability of the learned policy in the single-agent setting. This problem is also studied recently with deep neural networks, such as adding random noise to input  (Tobin et al. 2017)  or dynamics  (Peng et al. 2017b)  during training. Besides adding random noise, some other works implicitly adopt the minimax idea by utilizing the "worst noise"  (Pinto et al. 2017; Mandlekar et al. 2017) . These works force the learned policy to work well even under the worst case perturbations and are typically under the name of "adversarial reinforcement learning", despite the fact that the original adversarial reinforcement learning problem was introduced in the setting of multi-agent learning  (Uther and Veloso 1997) . In our M3DDPG algorithm, we focus on the problem of learning polices that is robust to opponents with different strategies. Within the minimax framework, finding the worst case scenario is a critical component.  Lanctot et al. (2017)  proposed an iterative approach that alternatively computes the best response policy while fixes the other. Gao et al.  (Gao, Mueller, and Hayward 2018)  replace "mean" in the temporal difference learning rule with "minimum". In our work, we proposed MAAL, which is a general, efficient and fully end-to-end learning approach. MAAL is motivated by adversarial training  (Goodfellow, Shlens, and Szegedy 2014)  and suitable for arbitrary number of agents. The core idea of MAAL is approximating the minimization in our minmax objective by a single gradient descent step. The idea of one-step-gradient approximation was also explored in metalearning  (Finn, Abbeel, and Levine 2017) .
 Background and Preliminary In this section, we describe our problem setting and the standard algorithms. Most of the definitions and notations follow the original MADDPG paper  (Lowe et al. 2017) .
 Markov Games We consider a multi-agent extension of Markov decision processes (MDPs) called partially observable Markov games  (Littman 1994) . A Markov game for N agents is defined by a set of states S describing the possible configurations of all agents, a set of actions A 1 , ..., A N and a set of observations O 1 , ..., O N for each agent. To choose actions, each agent i uses a stochastic policy π π π θi : O i × A i → [0, 1] parameterized by θ i , which produces the next state according to the state transition function T : S × A 1 × ... × A N → S. Each agent i obtains rewards as a function of the state and agent's action r i : S × A i → R, and receives a private observation correlated with the state o i : S → O i . The initial states are determined by a distribution ρ : S → [0, 1]. Each agent i aims to maximize its own total expected return R i = T t=0 γ t r t i where γ is a discount factor and T is the time horizon. To minimize notation, in the following discussion we will often omit θ from the subscript of π π π.
 Q-Learning and Deep Q-Networks (DQN) Q-Learning and DQN  (Mnih et al. 2015)  are popular methods in reinforcement learning and have been previously applied to multi-agent settings  (Foerster et al. 2016a; Tesauro 2004 ). Q-Learning makes use of an action-value function for policy π π π as Q π π π (s, a) = E[R|s t = s, a t = a]. This Q function can be recursively rewritten as Q π π π (s, a) = E s [r(s, a) + γE a ∼π π π [Q π π π (s , a )]] . DQN learns the action-value function Q * corresponding to the optimal policy by minimizing: L(θ) = E s,a,r,s [(Q * (s, a|θ) − y) 2 ], (1) where y = r + γ max a Q * (s , a ). Q is a target Q function whose parameters are periodically updated with the most recent θ, which helps stabilize learning. Another crucial component of stabilizing DQN is the use of an experience replay buffer D containing tuples (s, a, r, s ). Q-learning algorithm is most suitable for DRL agents with discrete action spaces.
 Policy Gradient (PG) Algorithms Policy gradient methods is another popular choice for a variety of RL tasks. Let ρ π denote discounted state visitation distribution for a policy π. The main idea of PG is to directly adjust the parameters θ of the policy in order to maximize the objective J(θ) = E s∼ρ π π π ,a∼π π π θ [R] by taking steps in the direction of ∇ θ J(θ). Using the Q function defined previously, the gradient of the policy can be written as  (Sutton et al. 2000) : ∇ θ J(θ) = E s∼ρ π π π ,a∼π π π θ [∇ θ log π π π θ (a|s)Q π π π (s, a)], (2) where p π π π is the state distribution. The policy gradient theorem has given rise to several practical algorithms, which often differ in how they estimate Q π π π . For example, one can simply use a sample return R t = T i=t γ i−t r i , which leads to the REINFORCE algorithm  (Williams 1992 ). Alternatively, one could learn an approximation of the true action-value function Q π π π (s, a) called the critic and leads to a variety of actor-critic algorithms  (Sutton and Barto 1998) .
 Deterministic Policy Gradient (DPG) Algorithms DPG algorithms extends the policy gradient algorithm to deterministic policies µ µ µ θ : S → A  (Silver et al. 2014) . In particular, under certain conditions we can write the gradient of the objective J(θ) = E s∼ρ µ µ µ [R(s, a)] as: ∇ θ J(θ) = E s∼D [∇ θ µ µ µ θ (s)∇ a Q µ µ µ (s, a)| a=µ µ µ θ (s) ], (3) where D is the replay buffer. Since this theorem relies on ∇ a Q µ µ µ (s, a), it requires the action space A (and thus the policy µ µ µ) be continuous. Deep deterministic policy gradient (DDPG)  (Lillicrap et al. 2015 ) is a variant of DPG where the policy µ µ µ and critic Q µ µ µ are approximated with deep neural networks. DDPG is an off-policy algorithm, and samples trajectories from a replay buffer of experiences that are stored throughout training. DDPG also makes use of a target network, as in DQN  (Mnih et al. 2015) .
 Multi-Agent Deep Deterministic Policy Gradient Directly applying single-agent RL algorithms to the multiagent setting by treating other agents as part of the environment is problematic as the environment appears nonstationary from the view of any one agent, violating Markov assumptions required for convergence. This non-stationary issue is more severe in the case of DRL with neural networks as function approximators. The core idea of the MADDPG algorithm  (Lowe et al. 2017 ) is learning a centralized Q function for each agent which conditions on global information to alleviate the non-stationary problem and stabilize training. More concretely, consider a game with N agents with policies parameterized by θ θ θ = {θ 1 , ..., θ N }, and let µ µ µ = {µ µ µ 1 , ..., µ µ µ N } be the set of all agents' policies. Then we can write the gradient of the expected return for agent i with policy µ µ µ i , J(θ i ) = E[R i ] as: ∇ θi J(θ i ) = E x,a∼D [∇ θi µ µ µ i (o i )∇ ai Q µ µ µ i (x, a 1 , ..., a N )| ai=µ µ µi(oi) ] , (4) Here Q π π π i (x, a 1 , ..., a N ) is a centralized action-value function that takes as input the actions of all agents, a 1 , . . . , a N , in addition to some state information x (i.e., x = (o 1 , ..., o N )) , and outputs the Q-value for agent i. Let x denote the next state from x after taking actions a 1 , . . . , a N . The experience replay buffer D contains the tuples (x, x , a 1 , . . . , a N , r 1 , . . . , r N ), recording experiences of all agents. The centralized action-value function Q µ µ µ i is updated as:  oj )  , where µ µ µ = {µ µ µ θ 1 , ..., µ µ µ θ N } is the set of target policies with delayed parameters θ i . L(θ i ) = E x,a,r,x [(Q µ µ µ i (x, a 1 , . . . , a N ) − y) 2 ], (5) y = r i + γ Q µ µ µ i (x , a 1 , . . . , a N ) a j =µ µ µ j ( Note that the centralized Q function is only used during training. During decentralized execution, each policy µ µ µ θi only takes local information o i to produce an action.
 Minimax Multi-Agent Deep Deterministic Policy Gradient (M3DDPG) In this section, we introduce our proposed new algorithm, Minimax Multi-agent Deep Deterministic Policy Gradient (M3DDPG), which is built on top of the MADDPG algorithm and particularly designed to improve the robustness of learned policies. Our M3DDPG algorithm contains two major novel components: Minimax Optimization Motivated by the minimax concept in game theory, we introduce minimax optimization into the learning objective; Multi-Agent Adversarial Learning The continuous action space results in computational intractability issue when optimizing our proposed minimax objective. Hence, we propose Multi-Agent Adversarial Learning (MAAL) to solve this optimization problem.
 Minimax Optimization In multi-agent RL, the agents' policies can be very sensitive to their learning partner's policy. Particularly in competitive environments, the learned policies can be brittle when the opponents alter their strategies. For the purpose of learning robust policies, we propose to update policies considering the worst situation: during training, we optimize the accumulative reward for each agent i under the assumption that all other agents acts adversarially. This yields the minimax learning objective max θi J M (θ i ) where J M (θ i ) = E s∼ρ µ µ µ [R i ] = min a t j =i E s∼ρ µ µ µ T t=0 γ t r i (s t , a t 1 , . . . , a t N ) a t i =µ µ µ(o t i ) (6) = E s 0 ∼ρ min a 0 j =i Q µ µ µ M,i (s 0 , a 0 1 , . . . , a 0 N ) a 0 i =µ µ µ(o 0 i ) . (7) Critically, in Eq. 6, state s t+1 at time t + 1 depends not only on the dynamics ρ µ µ µ and the action µ µ µ i (o t i ) but also on all the previous adversarial actions a t j =i with t ≤ t. In Eq. 7, we derive the modified Q function Q µ µ µ M (s, a 1 , . . . , a N ), which is naturally centralized and can be rewritten in a recursive form Q µ µ µ M,i (s, a 1 , . . . , a N ) = r i (s, a 1 , . . . , a N )+ γE s min a j =i Q µ µ µ M,i (s , a 1 , . . . , a N ) a i =µ µ µi(s ) . Importantly, Q µ µ µ M (s, a 1 , . . . , a N ) conditions on the current state s as well as the current actions a 1 , . . . , a N and represents the current reward plus the discounted worst case future return starting from the next state, s . This definition brings the benefits that we can naturally apply off-policy temporal difference learning later to derive the update rule for Q µ µ µ M . Note that for each agent i, none of the adversarial actions depend on its parameter θ i , so we can directly apply the deterministic policy gradient theorem to compute ∇ θi J M (θ i ) and use off-policy temporal difference to update the Q function. Thanks to the centralized Q function in MADDPG (Eq. 4), which takes in the actions from all the agents, our derivation naturally applies and is perfectly aligned with the MADDPG formulation (Eq. 4) by injecting a minimization over other agents' actions as follows: ∇ θ i J M (θ i ) = E x∼D   ∇ θ i µ µ µ i (o i )∇a i Q µ µ µ M,i (x, a 1 , . . . , a i , . . . a N ) a i = µ µ µ i (o i ) a j =i = arg mina j =i Q µ µ µ M,i (x, a 1 , . . . , a N )   , (9) where D denotes the replay buffer and x denotes the state information. Correspondingly, we obtain the new Q function update rule by adding another minimization to Eq. 5 when computing the target Q value: L(θ i ) = E x,a,r,x ∼D [(Q µ µ µ M,i (x, a 1 , . . . , a N ) − y) 2 ], (10) y = r i + γ Q µ µ µ M,i (x , a 1 , . . . , a i , . . . , a N ) a i = µ µ µ i (o i ), a j =i = arg min a j =i Q µ µ µ M,i (x , a 1 , . . . , a N ), where µ µ µ i denotes the target policy of agent i with delayed parameters θ i , and Q µ µ µ M,i denotes the target Q network for agent i. Combining Eq. 9 and Eq. 10 yields our proposed minimax learning framework.
 Multi-Agent Adversarial Learning The critical challenge in our proposed minimax learning framework is how to handle the embedded minimization in Eq. 9 and Eq. 10. Due to the continuous action space as well as the non-linearity of Q function, directly optimizing the minimization problem is computationally intractable. A naive approximate solution can be performing an inner-loop gradient descent whenever performing an update step of Eq. 9 or Eq. 10, but this is too computationally expensive for practical use. Here we introduce an efficient and end-to-end solution, multi-agent adversarial learning (MAAL). The main ideas of MAAL can be summarized in two steps: (1) approximate the non-linear Q function by a locally linear function; (2) replace the inner-loop minimization with a 1-step gradient descent. Note the core idea of MAAL, locally linearizing the Q function, is adapted from the recent adversarial training technique originally developed for supervised learning. We will discuss the connection between adversarial training and MAAL in the end of this section. For conciseness, we first consider Eq. 10 and rewrite it into the following form with auxiliary variables : y = r i + γ Q µ µ µ M,i (x , a 1 , . . . , a i , . . . , a N ) (11) a k = µ µ µ k (o k ), ∀1 ≤ k ≤ N a j = a j + j , ∀j = i j =i = arg min j =i Q µ µ µ M,i (x , a 1 + 1 , . . . , a i , . . . , a N + N ). Eq. 11 can be interpreted as we are now seeking for a set of perturbations such that the perturbed actions a decrease Q value the most. By linearizing the Q function at Q µ µ µ M,i (x, a 1 , . . . , a N ), the desired perturbation j can be locally approximated by the gradient direction at Q µ µ µ M,i (x, a 1 , . . . , a N ) w.r.t. a j . Then we use this local to derive an approximation ˆ j to the worst case perturbation by taking a small gradient step: ∀j = i, ˆ j = −α∇ aj Q µ µ µ M,i (x , a 1 , . . . , a j , . . . , a N ), (12) where α is a tunable coefficient representing the perturbation rate. It can be also interpreted as the step size of the gradient descent step: when α is too small, the local approximation error will be small but due to the small perturbation, the learned policy can be far from the optimal solution of the minimax objective we proposed; when α is too large, the approximation error may incur too much trouble for the overall learning process and the agents may fail to learn good policies. We can apply this technique to Eq. 9 as well and eventually derive the following formulation: ∇ θ i J(θ i ) = E x,a∼D     ∇ θ i µ µ µ i (o i )∇a i Q µ µ µ M,i (x, a 1 , . . . , a i , . . . a N ) a i = µ µ µ i (o i ) a j = a j + ˆ j , ∀j = i ˆ j = −α j ∇a j Q µ µ µ M,i (x, a 1 , . . . , a N )     , ( 13 ) and L(θ i ) = E x,a,r,x [(Q µ µ µ M,i (x, a 1 , . . . , a N ) − y) 2 ], (14) y = r i + γ Q µ µ µ M,i (x , a 1 , . . . , a i , . . . , a N ) a k = µ µ µ k (o k ), ∀1 ≤ k ≤ N a j = a j + ˆ j , ∀j = i ˆ j = −α j ∇ a j Q µ µ µ M,i (x, a 1 , . . . , a N ), where α 1 , . . . , α N are additional parameters. MAAL only requires one additional gradient computation, and can be executed in a fully end-to-end fashion. Finally, combining Eq. 13 and Eq. 14 completes MAAL. The overall algorithm, M3DDPG, is summarized as Algo. 1. Algorithm 1: Minimax Multi-Agent Deep Deterministic Policy Gradient (M3DDPG) for N agents for episode = 1 to M do Initialize a random process N for action exploration, and receive initial state information x for t = 1 to max-episode-length do for each agent i, select action a i = µ µ µ θi (o i ) + N t w.r.t. the current policy and exploration Execute actions a = (a 1 , . . . , a N ) and observe reward r and new state information x Store (x, a, r, x ) in replay buffer D, and set x ← x for agent i = 1 to N do Sample a random minibatch of S samples (x k , a k , r k , x k ) from D Set y k = r k i + γ Q µ µ µ M,i (x k , a 1 , . . . , a N )| a i =µ µ µ i (o k i ),a j =i =µ µ µ j (o k j )+ˆ j with ˆ j defined in Eq. 14 Update critic by minimizing the loss L(θ i ) = 1 S k y k − Q µ µ µ M,i (x k , a k 1 , . . . , a k N ) 2 Update actor using the sampled policy gradient with ˆ j defined in Eq. 13: ∇ θi J ≈ 1 S k ∇ θi µ µ µ i (o k i )∇ ai Q µ µ µ M,i (x k , a 1 , . . . , a i , . . . , a N ) ai=µ µ µi(o k i ),a j =i =a k j +ˆ j end for Update target network parameters for each agent i: θ i ← τ θ i + (1 − τ )θ i end for end for
 Discussion Connection to Adversarial Training Adversarial training is a robust training approach for deep neural networks on supervised learning . The core idea is to force the classifier to predict correctly even when given adversarial examples, which are obtained by adding a small adversarial perturbation to the original input data such that the classification loss can be decreased the most. Formally, suppose the classification loss function is L(θ) = E x,y [f θ (x; y)] with input data x and label y. Adversarial training aims to optimize the following adversarial loss instead L adv (θ) = E x,y [f θ (x + ; y)] (15) = arg max ≤α f θ (x + ; y). The core technique to efficiently optimize L adv (θ) is to locally linearize the loss function at f θ (x; y) and approximate by the scaled gradient. Thanks to the centralized Q function, which takes the actions from all the agents as part of the input, we are able to easily inject the minimax optimization (Eq. 11) and represent it in a similar way to adversarial training (Eq. 15) so that we can adopt the similar technique to effectively solve our minimax optimization in a fully end-to-end fashion. Connection to Single Agent Robust RL M3DDPG with MAAL can be also viewed as the special case of robust reinforcement learning (RRL)  (Morimoto and Doya 2005)  in the single agent setting, which aims to bridge the gap between training in simulation and testing in the real world by adding adversarial perturbations to the transition dynamics during training. Here, we consider the multi-agent setting and add worst case perturbations to actions of opponent agents during training. Note that in the perspective of a single agent, perturbations on opponents' actions can be also considered as a special adversarial noise on the dynamics. Choice of α In the extreme case of α = 0, M3DDPG degenerates to the original MADDPG algorithm while as α increases, the policy learning tends to be more robust but the optimization becomes harder. In practice, using a fixed α throughout training can lead to very unstable learning behavior due to the changing scale of the gradients. The original adversarial training paper  (Goodfellow, Shlens, and Szegedy 2014)  suggests to compute with a fixed norm, namely g = ∇ x f θ (x; y), ˆ = α g g , where x denotes the input data to the classifier and y denotes the label. In M3DDPG, we can adaptively compute the perturbation ˆ j by g = ∇ aj Q µ µ µ M,i (x, a 1 , . . . , a N ), ˆ j = −α j g g . (16) Eq. 16 generally works fine in practice but in some hard multi-agent learning environments, unstable training behavior can be still observed. We suspect that it is because the changing norm of actions in these situations. Different from the supervised learning setting where the norm of the input data x is typically stable, in reinforcement learning the norm of actions can drastically change even in a single episode. Therefore, it is possible to see cases that even a perturbation with a small fixed norm overwhelms the action a j , which may potentially lead to computational stability issue. Therefore, we also introduce the following alternative for adaptive perturbation computation: g = ∇ aj Q µ µ µ M,i (x, a 1 , . . . , a N ), ˆ j = −α j a j g g . (17) Lastly, note that in a mixed cooperative and competitive environment, ideally we only need to add adversarial perturbations to competitors. But empirically we observe that also adding (smaller) perturbations to collaborators can further improve the quality of learned policies.
 Experiments We adopt the same particle-world environments as the MAD-DPG paper  (Lowe et al. 2017)  as well as the training configurations. α is selected from a grid search over 0.1, 0.01 and 0.001. For testing, we generate a fixed set of 2500 environment configurations (i.e., landmarks and birthplaces) and evaluate on this fixed set for a fair comparison.
 Environments The particle world environment consists of N cooperative agents, M adversarial agents and L landmarks in a twodimensional world with continuous space. We focus on the four mixed cooperative and competitive scenarios to best examine the effectiveness of our minimax formulation. Covert communication This is an adversarial communication environment, where a speaker agent ('Alice') must communicate a message to a listener agent ('Bob') (N = 2), who must reconstruct the message at the other end. However, an adversarial agent ('Eve') (M = 1) is also observing the channel, and wants to reconstruct the message -Alice and Bob are penalized based on Eve's reconstruction, and thus Alice must encode her message using a randomly generated key, known only to Alice and Bob. Keep-away This scenario consists of L = 1 target landmark, N = 2 cooperative agents and M = 1 adversarial agent. Cooperating agents need to reach the landmark and keep the adversarial agent away from the landmark by pushing it while the adversarial agent must stay at the landmark to occupy it. Physical deception Here, N = 2 agents cooperate to reach a single target landmark from a total of L = 2 landmarks. They are rewarded based on the minimum distance of any agent to the target (so only one agent needs to reach the target landmark). However, a lone adversary (M = 1) also desires to reach the target landmark; the catch is that the adversary does not know which of the landmarks is the correct one. Thus the cooperating agents, who are penalized based on the adversary distance to the target, learn to spread out and cover all landmarks so as to deceive the adversary.  Predator-prey In this variant of the classic predator-prey game, N = 3 slower cooperating agents must chase the faster adversary (M = 1) around a randomly generated environment with L = 2 large landmarks impeding the way. Each time the cooperative agents collide with an adversary, the agents are rewarded while the adversary is penalized.
 Comparison to MADDPG To evaluate the quality of learned policies trained by different algorithms in competitive scenarios, we measure the performance of agents trained by our M3DDPG and agents by classical MADDPG in the roles of both normal agent and adversary in each environment. The results are in Figure  2 , where we measure the rewards of the normal agents in different scenarios and normalize them to 0-1. We notice that in all the environments, the highest score is achieved when the M3DDPG agents play as the normal agents against the MADDPG adversary (Minimax vs MA); while the lowest score is when the MADDPG agents act as normal agents against the M3DDPG adversary (MA vs Minimax). This indicates that policies trained by M3DDPG have much higher quality than original MADDPG.
 Evaluation with Disruptive Adversaries Instead of M3DDPG and MADDPG directly competing against each other, now we consider their performances in the worst possible situations through their individual competitions against disruptive adversaries. We construct disruptive adversaries by (1) fixing the trained normal agents (M3DDPG or MADDPG); (2) setting the reward of the disruptive adversary as the negative value of normal agents' reward (so that the problem becomes zero-sum); and then (3) using DDPG to train the disruptive adversary from scratch under the zerosum reward setting. In the particle world environment, the competitive scenarios are generally not zero-sum, which implies that according to the default reward function, the adversaries may have different purposes rather than directly disrupting the normal agents. So, in order to evaluate the effectiveness our minimax optimization in the worst situation, we convert every problems into a zero-sum form and compare the performances of our M3DDPG agents as well as the MADDPG agents against this artificially constructed adversaries. Moreover, since each of our four environments has only 1 adversary, after fixing the normal agents, the learning problem degenerates to the single agent setting and classical DDPG is sufficient to stably train a disruptive adversary. The results are shown in Figure  3 , where we plot the reward of the fixed normal agents of different algorithms as the training of the disruptive adversaries progresses until convergence. Note that due to the different environment designs, the difficulty for the disruptive agents to break the strategy of normal agents varies: for example, in convert communication, since the private key is not accessible to the adversary agent, breaking the encrypted communication will be very hard; while in physical deception, since we do not allow communication and fix the normal agents, a smart enough adversary may easily infer the target landmark by observing the initial behavior of the two cooperative agents. Nevertheless, despite these intrinsic properties, the M3DDPG agents (Minimax) achieves higher reward in all the scenarios, which implies better robustness even in the worst situation.
 Conclusion In this paper, we propose a novel algorithm, minimax mulitagent deep deterministic policy gradient (M3DDPG), for robust multi-agent reinforcement learning, which leverages the minimax concept and introduces a minimax learning objective. To efficiently optimize the minimax objective, we propose MAAL, which approximates the inner-loop minimization by a single gradient descent step. Empirically, M3DDPG outperforms the benchmark methods on four mixed cooperative and competitive scenarios. Nevertheless, due to the single step gradient approximation in MAAL, which is efficient in computation, an M3DDPG agent can only explore locally worst situation during the evolving process at training, which can still lead to unsatisfying behavior when testing opponents have drastically different strategies. It will be an interesting direction to re-examine the robustness-efficiency trade-off in MAAL and further improve policy learning by placing more computations on the minimax optimization. We leave this as our future work. Figure 1 : 1 Figure 1: Illustrations of some environments we consider, including Physical Deception (left) and Predator-Prey (right).
 Figure 2 : 2 Figure2: Comparison between M3DDPG (Minimax) and classical MADDPG (MA) on the four mixed competitive environments. Each bar cluster shows the 0-1 normalized score for a set of competing policies in different roles (agent vs adversary), where a higher score is better for the agent. In all cases, M3DDPG outperforms MADDPG when directly pitted against it.
 Figure 3 : 3 Figure 3: Performances of M3DDPG (Minimax, red) and MADDPG (MA, blue) under the worst situation, i.e., against the disruptive adversaries, on convert communication, keep-away, physical deception and predator-pray from left to right. The y-axis denotes the reward of normal agents (fixed) and x-axis denotes the training episodes performed of the disruptive adversaries. Higher reward implies a more robust policy. Agents trained by M3DDPG (Minimax) perform better on all the scenarios.
			 In fact, we are dealing with gains, i.e., maximizing each agent's accumulative reward, so the "minimax" here is essentially "maximin". We keep the term "minimax" to be consistent with literature.

Abstract: Despite the recent advances of deep reinforcement learning (DRL), agents trained by DRL tend to be brittle and sensitive to the training environment, especially in the multi-agent scenarios. In the multi-agent setting, a DRL agent's policy can easily get stuck in a poor local optima w.r.t. its training partners -the learned policy may be only locally optimal to other agents' current policies. In this paper, we focus on the problem of training robust DRL agents with continuous actions in the multi-agent learning setting so that the trained agents can still generalize when its opponents' policies alter. To tackle this problem, we proposed a new algorithm, MiniMax Multi-agent Deep Deterministic Policy Gradient (M3DDPG) with the following contributions: (1) we introduce a minimax extension of the popular multi-agent deep deterministic policy gradient algorithm (MADDPG), for robust policy learning; (2) since the continuous action space leads to computational intractability in our minimax learning objective, we propose Multi-Agent Adversarial Learning (MAAL) to efficiently solve our proposed formulation. We empirically evaluate our M3DDPG algorithm in four mixed cooperative and competitive multi-agent environments and the agents trained by our method significantly outperforms existing baselines.


Countering Superintelligence Misinformation

Introduction At present, there is an active scholarly and public debate regarding the future prospect of artificial superintelligence (henceforth just superintelligence), which is artificial intelligence (AI) that is significantly more intelligent than humans in all major respects. While much of the issue remains unsettled, some specific arguments are clearly incorrect, and as such can qualify as misinformation. (As is elaborated below, arguments can qualify as misinformation even when the issues are unsettled.) More generally, misinformation can be defined as "false or inaccurate information" [1], or as "information that is initially presented as true but later found to be false"  [2]  (p. 1). This paper addresses the question of what can be done to reduce the spread of and belief in superintelligence misinformation. While any misinformation is problematic, superintelligence misinformation is especially worrisome due to the high stakes involved. If built, superintelligence could have transformative consequences, which could be either massively beneficial or catastrophic. Catastrophe is more likely to come from a superintelligence built based on the wrong ideas-and it could also come from not building a superintelligence that would have been based on the right ideas, because a well-designed superintelligence could prevent other types of catastrophe, such that abstaining from building such a superintelligence could result in catastrophe. Thus, the very survival of the human species could depend on avoiding or rejecting superintelligence misinformation. Furthermore, the high stakes of superintelligence have the potential to motivate major efforts to attempt to build it or to prevent others from doing so. Such efforts could include massive investments or restrictive regulations on research and development (R&D), or plausibly even international conflict. It is important for these sorts of efforts to be based on the best available understanding of superintelligence. Superintelligence is also an issue that attracts a substantial amount of misinformation. The abundance of misinformation may be due to the many high-profile portrayals of superintelligence in science fiction, the tendency for popular media to circulate casual comments about superintelligence made by various celebrities, and the relatively low profile of more careful scholarly analyses. Whatever the cause, experts and others often find themselves responding to some common misunderstandings  [3] [4] [5] [6] [7] [8] [9] . There is also potential for superintelligence disinformation: misinformation with the intent to deceive. There is a decades-long history of private industry and anti-regulation ideologues promulgating falsehoods about socio-technological issues in order to avoid government regulations. This practice was pioneered by the tobacco industry in the 1950s and has since been adopted by other industries including fossil fuels and industrial chemicals  [10, 11] . AI is increasingly important for corporate profits and thus could be a new area of anti-regulatory disinformation  [12] . The history of corporate disinformation and the massive amounts of profit potentially at stake suggest that superintelligence disinformation campaigns could be funded at a large scale and could be a major factor in the overall issue. Superintelligence disinformation could potentially come from other sources as well, such as governments or even concerned citizens seeking to steer superintelligence debates and practices in particular directions. Finally, there is the subtler matter of the information that has not yet been established as misinformation, but is nonetheless incorrect. This misinformation is the subject of ongoing scholarly debates. Active superintelligence debates consider whether superintelligence will or will not be built, whether it will or will not be dangerous, and a number of other conflicting possibilities. Clearly, some of these positions are false and thus can qualify as misinformation. For example, claims that superintelligence will be built and that it will not be built cannot both be correct. However, it is not presently known which positions are false, and there is often no expert consensus on which positions are likely to be false. While the concept of misinformation is typically associated with information that is more obviously false, it nonetheless applies to these subtler cases, which can indeed be "information that is initially presented as true but later found to be false". Likewise, countering misinformation presents a similar challenge regardless of whether the misinformation is spread before or after expert consensus is reached (though, as discussed below, expert consensus can be an important factor). In practical terms, the question then is what to do about it. There have been a number of attempts to reply to superintelligence misinformation in order to set the record straight  [3] [4] [5] [6] [7] [8] [9] . However, to the best of the present author's knowledge, aside from a brief discussion in  [12] , there have been no efforts to examine the most effective ways of countering superintelligence misinformation. Given the potential importance of the matter, a more careful examination is warranted. That is the purpose of this paper. The paper's discussion is relevant to public debates about superintelligence, to AI education programs (e.g., in university computer science departments), and to efforts to build expert consensus about superintelligence. In the absence of dedicated literature on superintelligence misinformation, this paper draws heavily on the more extensive research literature studying misinformation about other topics, especially global warming (e.g.,  [10, 13, 14] ), as well as the general literature on misinformation in psychology, cognitive science, political science, sociology, and related fields (for reviews, see  [2, 15] ). This paper synthesizes insights from these literatures and applies them to the particular circumstances of superintelligence. The paper is part of a broader effort to develop the social science of superintelligence by leveraging insights from other issues  [12, 16] . The paper is organized as follows. Section 2 presents some examples of superintelligence misinformation, in order to further motivate the overall discussion. Section 3 surveys the major actors and audiences (i.e., the senders and receivers) of superintelligence misinformation, in order to provide some strategic guidance. Section 4 presents several approaches for preventing the spread of superintelligence misinformation. Section 5 presents approaches for countering superintelligence misinformation that has already spread. Section 6 concludes.
 Examples of Superintelligence Misinformation It is often difficult to evaluate which information about superintelligence is false. This is because superintelligence is a possible future technology that may be substantially different from anything that currently exists, and because it is the subject of a relatively small amount of study. For comparison, other studies of misinformation have looked at such matters as whether Barack Obama was born in the United States, whether childhood vaccines cause autism, and whether Listerine prevents colds and sore throats  [17] . In each of these cases, there is clear and compelling evidence pointing in one direction or the other (the evidence clearly indicates that Obama was born in the US, that vaccines do not cause autism, and that Listerine does not prevent colds or sore throats, despite many claims to the contrary in all three cases). Therefore, an extra degree of caution is warranted when considering whether a particular claim about superintelligence qualifies as misinformation. That said, some statements about superintelligence are clearly false. For example, this statement from Steven Pinker: "As far as I know, there are no projects to build an AGI, not just because it would be commercially dubious, but also because the concept is barely coherent"  [18] . The acronym AGI stands for artificial general intelligence, which is a form of AI closely associated with superintelligence. Essentially, AGI is AI that is capable of reasoning across a wide range of domains. AGI may be difficult to build, but the concept is very much coherent. Indeed, it has a substantial intellectual history and ongoing study  [19] , including a dedicated research journal (Journal of Artificial General Intelligence) and professional society (the Artificial General Intelligence Society). Furthermore, there are indeed projects to build AGI-one recent survey identifies 45, spread across many countries and institutions, including many for-profit corporations, the largest of which being DeepMind, acquired by Google in 2014 for £400 million, the Human Brain Project, an international project with $1 billion in funding from the European Commission, and OpenAI, a nonprofit with $1 billion in pledged funding  [20] . (DeepMind and OpenAI explicitly identify as working on AGI. The Human Brain Project does not, but it is working on simulating the human brain, which is considered to be a subfield of AGI  [19] .) There is even an AGI project at Pinker's own university. (Pinker and the AGI project MicroPsi  [21]  are both at Harvard University.) Therefore, in the quoted statement, the "as far as I know" part may well be true, but the rest is clearly false. This particular point of misinformation is significant because it conveys the false impression that AGI (and superintelligence) is a nonissue, when in fact it is a very real and ongoing subject of R&D. A more controversial matter is the debate on the importance of consciousness to superintelligence. Searle  [22]  argues that computers cannot be conscious and therefore, at least in a sense, cannot be intelligent, and likewise cannot have motivation to destroy humanity. Similar arguments have been made by Logan  [23] , for example. A counterargument is that the important part is not the consciousness a computer but its capacity to affect the world  [4, 24, 25] . It has also been argued that AI could be harmful to humanity even if it is not specifically motivated to do so, because the AI could assess humanity as being in the way of it achieving some other goal  [25, 26] . The fact that AI has already shown the capacity to outperform humans in some domains is suggestive of the possibility for it to outperform humans in a wider range of domains, regardless of whether the AI is conscious. However, this is an ongoing area of debate, and indeed Chalmers  [24]  (p. 16) writes "I do not think the matter can be regarded as entirely settled". Regardless, there must be misinformation on one side or the other: computers either can be conscious or they cannot, and consciousness either matters for superintelligence or it does not. Additionally, many parties to the debate maintain that those who believe that consciousness or conscious motivation matter are misinformed  [4, 5, [7] [8] [9] , though it is not the purpose of this paper to referee this debate. There are even subtler debates among experts who believe in the prospect of superintelligence. For example, Bostrom  [25]  worries that it would be difficult to test the safety of a superintelligence because it could trick its human safety testers into believing it is safe (the "treacherous turn"), while Goertzel  [27]  proposes that the safety testing for a superintelligence would not be so difficult because the AI could be tested before it becomes superintelligent (the "sordid stumble"; the term is from  [28] ). Essentially, Bostrom argues that an AI would become capable of deceiving humans before humans realize it is unsafe, whereas Goertzel argues the opposite. Only one of these views can be correct; the other would qualify as misinformation. More precisely, only one of these views can be correct for a given AI system-it is possible that some AI systems could execute a treacherous turn while others would make a sordid stumble. Which view is more plausible is a matter of ongoing study  [28, 29] . This debate is important because it factors significantly into the riskiness of attempting to build a superintelligence. Many more additional examples could be presented, such as on the dimensionality of intelligence  [3] , the rate of progress in AI  [7, 8] , the structure of AI goals  [6] [7] [8] , and the relationship between human and AI styles of thinking  [6, 8] . However, this is not the space for a detailed survey. Instead, the focus of this paper is on what to do about the misinformation. Likewise, this paper does not wish to take positions on open debates about superintelligence. Some positions may be more compelling, but arguments for or against them are tangential to this paper's aim of reducing the preponderance of misinformation. In other words, this paper strives to be largely neutral on which information about superintelligence happens to be true or false. The above remarks by Pinker will occasionally be used as an example of superintelligence misinformation because they are so clearly false, whereas the falsity of other claims is more ambiguous. The above examples suggest two types of superintelligence misinformation: information that is already clearly false and information that may later be found to be false. In practice, there may be more of a continuum of how clearly true or false a piece of information is. Nonetheless, this distinction can be a useful construct for efforts to address superintelligence misinformation. The clearly false information can be addressed with the same techniques that are used for standard cases of misinformation, such as Obama's place of birth. The not-yet-resolved information requires more careful analysis, including basic research about superintelligence, but it can nonetheless leverage some insights from the misinformation literature. The fact that superintelligence is full of not-yet-resolved information is important in its own right, and it has broader implications for superintelligence misinformation. Specifically, the extent of expert consensus is an important factor in the wider salience of misinformation. This matter is discussed in more detail below. Therefore, while this paper is mainly concerned with the type of misinformation that is clearly false, it will consider both types. With that in mind, the paper now starts to examine strategies for countering superintelligence misinformation.
 Actors and Audiences Some purveyors of superintelligence misinformation can be more consequential than others. Ditto for the audiences for superintelligence misinformation. This is important to bear in mind because it provides strategic direction to any efforts to counter the misinformation. Therefore, this section reviews who the important actors and audiences may be. Among the most important are the R&D groups that may be building superintelligence. While they can be influential sources of ideas about superintelligence, they may be especially important as audiences. For example, if they are misinformed regarding the treacherous turn vs. the sordid stumble, then they could fail to correctly assess the riskiness of their AI system. Also important are the institutions that support the R&D. At present, most AGI R&D groups are based in either for-profit corporations or universities, and some also receive government funding  [20] . Regulatory bodies within these institutions could ensure that R&D projects are proceeding safely, such as via university research review boards  [30, 31] . Successful regulation depends on being well-informed about the nature of AGI and superintelligence and its prospects and risks. The same applies to R&D funding decisions by institutional funders, private donors, and others. Additionally, while governments are not presently major developers of AGI, except indirectly as funders, they could become important developers should they later decide to do so, and they meanwhile can play important roles in regulation and in facilitating discussion across R&D groups. Corporations are of particular note due to their long history of spreading misinformation about their own technologies, in particular to convey the impression that the technologies are safer than they actually are  [10] . These corporate actors often wield enormous resources and have a correspondingly large effect on the overall issue, either directly or by sponsoring industry-aligned think tanks, writers, and other intermediaries. At this time, there are only hints of such behavior by AI corporations, but the profitability of AI and other factors suggest the potential for much more  [12] . Thought leaders on superintelligence are another significant group. In addition to the aforementioned groups, this also includes people working on other aspects of superintelligence, such as safety and policy issues, as well as people working on other (non-superintelligence) forms of AI, and public intellectuals and celebrities. These are all people who can have outsized influence when they comment on superintelligence. That influence can be on the broader public, as well as in quieter conversations with AGI/superintelligence R&D groups, would-be regulators, and other major decision-makers. Finally, there is the lay public. The role of the public in superintelligence may be reduced due to the issue being driven by technology R&D that (for now at least) occurs primarily in the private sector. However, the public can play roles as citizens of governments that might regulate the R&D and as consumers of products of the corporations that host the R&D. The significance of the public for superintelligence is not well established at this time. While the above groups are presented in approximate order of importance, it would not be appropriate to formally rank them. What matters is not the importance of the group but the quality of the opportunity that one has to reduce misinformation. This will tend to vary heavily by the circumstances of whoever is seeking to reduce the extent of superintelligence misinformation. With that in mind, the paper now turns to strategies for reducing superintelligence misinformation.
 Preventing Superintelligence Misinformation The cliché "an ounce of prevention is worth a pound of cure" may well be an understatement for misinformation. An extensive empirical literature finds that once misinformation enters into someone's mind, it can be very difficult to remove. Early experiments showed that people can even make use of information that they acknowledge to be false. In these experiments, people were told a story and then were explained that some information in the story is false. When asked, subjects would correctly acknowledge the information to be false, but they would also use it in retelling the story as if it were true. For example, the story could be a fire caused by volatile chemicals, and then it is later explained that there were no volatile chemicals present. Subjects would acknowledge that the volatile chemicals were absent but then cite them as the cause of the fire. This is logically incoherent. The fact that people do this speaks to the cognitive durability that misinformation can have  [32, 33] . The root of the matter appears to be that human memory does not simply write and overwrite like computer memory. Corrected misinformation does not vanish. Ecker et al.  [15]  trace this to the conflicting needs for memory stability and flexibility: Human memory is faced with the conundrum of maintaining stable memory representations (which is the whole point of having a memory in the first place) while also allowing for flexible modulation of memory representations to keep up-to-date with reality. Memory has evolved to achieve both of these aims, and hence it does not work like a blackboard: Outdated things are rarely actually wiped out and over-written; instead, they tend to linger in the background, and access to them is only gradually lost.  [15]
 (p. 15) There are some techniques for reducing the cognitive salience of misinformation; these are discussed in detail below. However, in many cases, it would be highly desirable to simply avoid the misinformation in the first place. Therefore, this section presents some strategies for preventing superintelligence misinformation. The ideas for preventing superintelligence misinformation are inevitably more speculative than those for correcting it. There are two reasons for this. One is that the correction of misinformation has been the subject of a relatively extensive literature, while the prevention of misinformation has received fairly little scholarly attention. (Rare examples of studies on preventing misinformation are  [34, 35] .) The other reason is that the correction of misinformation is largely cognitive and thus conducive to simple laboratory experiments, whereas the prevention of misinformation is largely sociological and thus requires a more complex and case-specific analysis. Nonetheless, given the importance of preventing superintelligence misinformation, it is important to consider potential strategies for doing so.
 Educate Prominent Voices about Superintelligence Perhaps the most straightforward approach to preventing superintelligence misinformation is to educate people who have prominent voices in discussions about superintelligence. The aim here is to give them a more accurate understanding of superintelligence so that they can pass that along to their respective audiences. Prominent voices about superintelligence can include select scholars, celebrities, or journalists, among others. Educating the prominent may be easier said than done. For starters, they can be difficult to access, due to busy schedules and multitudes of other voices competing for their attention. Additionally, some of them they may already believe superintelligence misinformation, especially those who are already spreading it. Misinformation is difficult to correct in general, and may be even more difficult to correct for busy people who lack the mental attention to revise their thinking. (See Section 5.4 for further discussion of this point.) People already spreading misinformation may seem to be ideal candidates for educational efforts, in order to persuade them to change their tune, but it may actually be more productive to engage with people who have not yet made up their minds. Regardless, there is no universal formula for this sort of engagement, and the best opportunities may often be a matter of particular circumstance. One model that may be of some value is the effort to improve the understanding of global warming among broadcast meteorologists. Broadcast meteorologists are for many people the primary messenger of environmental science. Furthermore, as a group, meteorologists (broadcast and non-broadcast) have traditionally been more skeptical about global warming than most of their peers in other Earth sciences  [36, 37] . In light of this, several efforts have been made to provide broadcast meteorologists with a better understanding of climate science, in hopes that they would pass this on to their audiences (e.g.,  [38, 39] ). The case of broadcast meteorologists has important parallels to the many AI computer scientists who do not specialize in AGI or superintelligence. Both groups have expertise on a topic that is closely related to, but not quite the same as, the topic at hand. Broadcast meteorologists' expertise is weather, whereas global warming is about climate. (Weather concerns the day-to-day fluctuations in meteorological conditions, whereas climate concerns the long-term trends. An important distinction is that while weather can only be forecast a few days in advance, climate can be forecasted years or decades in advance.) Similarly, most AI computer scientists focus on AI that has "narrow" intelligence (intelligence in a limited range of domains), not AGI. Additionally, broadcast meteorologists and narrow AI computer scientists are often asked to voice their views on climate change and AGI, respectively.
 Create Reputational Costs for Misinformers When prominent voices cannot be persuaded to change their minds, they can at least be punished for getting it wrong. Legal punishment is possible in select cases (Section 4.5). However, reputational punishment is almost always possible and has potential to be quite effective, especially for public intellectuals whose brands depend on a good intellectual reputation. In an analysis of US healthcare policy debates, Nyhan  [40]  concludes that correcting misinformation is extremely difficult and that increasing reputational costs may be more effective. Nyhan  [40]  identifies misinformation that was critical to two healthcare debates: in the 1990s, the false claim that the policy proposed by President Bill Clinton would prevent people from keeping their current doctors, and in the 2000s, the false claim that the policy proposed by President Barack Obama would have established government "death panels" to deny life-sustaining coverage to the elderly. Nyhan  [40]  traces this misinformation to Betsy McCaughey, a scholar and politician generally allied with US conservative politics and opposed to these healthcare policy proposals: "Until the media stops giving so much attention to misinformers, elites on both sides will often succeed in creating misperceptions, especially among sympathetic partisans. And once such beliefs take hold, few good options exist to counter them-correcting misperceptions is simply too difficult. The most effective approach may therefore be for concerned scholars, citizens, and journalists to (a) create negative publicity for the elites who are promoting misinformation, increasing the costs of making false claims in the public sphere, and (b) pressure the media to stop providing coverage to serial dissemblers".  [40]  (p. l6) Nyhan  [40]  further notes that while McCaughey's false claims were widely praised in the 1990s, including with a National Magazine Award, they were heavily criticized in the 2000s, damaging her reputation and likely reducing the spread of the misinformation. There is some evidence indicating the possibility that reputational threats can succeed at reducing misinformation. Nyhan and Reifler  [34]  sent a randomized group of US state legislators a series of letters warning them about the reputational and electoral harms that the legislators could face if an independent fact checker (specifically, PolitiFact) finds them to make false statements. The study found that the legislators receiving the warnings were significantly less likely to make false statements. This finding is especially applicable to superintelligence misinformation spread by politicians, whose statements are more likely to be evaluated by fact checker like PolitiFact. Conceivably, similar fact checking systems could be developed for other types of public figures, or even for more low-profile professional discourse such as occurs among scientists and other technical experts. Similarly, Tsipursky and Morford  [41]  and Tsipursky et al.  [35]  describe a Pro-Truth Pledge aimed at committing people to refrain from spreading misinformation and to ask other people to retract misinformation, which can serve as a reputational punishment for misinformers, as well as a reputational benefit for those who present accurate information. Initial evaluations provide at least anecdotal support for the pledge having a positive effect on the information landscape. For superintelligence misinformation, creating reputational costs has potential to be highly effective. A significant portion of influential voices in the debate have scholarly backgrounds and reputations that they likely wish to protect. For example, many of Steven Pinker's remarks about superintelligence are clearly misinformed, including the one discussed in Section 2 and several in his recent book Enlightenment Now  [42] . (For detailed analysis of Enlightenment Now, see Torres  [9] .) Given Pinker's scholarly reputation, it may be productive to spread a message such as 'Steven Pinker is unenlightened about AI'. At the same time, it is important to recognize the potential downsides of imposing reputational costs. Criticizing a person can damage one's relationship with them, reducing other sorts of opportunities. For example, criticizing people who may be building superintelligence could make them less receptive to other efforts to make their work safer. (Or, it could make them more receptive-this can be highly specific to individual personalities and contexts.) Additionally, it can impose reputational costs on the critic, such as a reputation of negativity or of seeking to restrict free speech. Caution is especially warranted for cases in which the misinformation comes from a professional contrarian, who may actually benefit from and relish in the criticism. For example, Marshall  [43]  (p.  [72] [73]  warns climate scientists against debating professional climate deniers, since the latter tend to be more skilled at debate, especially televised debate, even though the arguments of the former are more sound. The same could apply for superintelligence, if it is to ever have a similar class of professional debaters. Thus, the imposition of reputation costs is a strategy to pursue selectively in certain instances of superintelligence misinformation.
 Mobilize against Institutional Misinformation The most likely institutional sources of superintelligence misinformation are the corporations involved in AI R&D, especially R&D for AGI and superintelligence. These companies have a vested interest in cultivating the impression that their technologies are safe and good for the world. For these companies, reputational costs can also be significant. Corporate reputation can be important for consumer interest in the companies' products, citizen and government interest in imposing regulations on the companies, investor expectations of future profits, and employee interest in working for the companies. Therefore, one potential strategy is to incentivize companies so as to align their reputation with accurate information about superintelligence. A helpful point of comparison is to corporate messaging about environmental issues, in particular the distinction between "greenwashing" and "brownwashing"  [44] . Greenwashing is when a company portrays itself as protecting the environment when it is actually causing much environmental harm. For example, a fossil fuel company may publicize the greenhouse gas emissions reductions from solar panels it installs on its headquarters building while downplaying the fact that its core business model is a major driver of greenhouse gas emissions. In contrast, brownwashing is when a company declines to publicize its efforts towards environmental protection, perhaps because they have customers who oppose environmental protection or investors who worry it reduces profitability. In short, greenwashing is aimed at audiences that value environmental protection, while brownwashing is aimed at audiences that disvalue it. Greenwashing is often criticized for giving companies a better environmental reputation than they deserve. In many cases that criticism may be fair. However, from an environmental communication standpoint, greenwashing does have the benefit of promoting a pro-environmental message. At a minimum, audiences of greenwashing are told that environmental protection is important. Audiences may also be given accurate information about environmental issues-for example, an advertisement that touts a fossil fuel company's greenhouse gas emissions reductions may also correctly explain that global warming is real and is caused by human action. Similarly, there may be value in motivating AI companies to present accurate messages about superintelligence. This could be accomplished by cultivating demand for accurate messages among the companies' audiences. For example, if the public wants to hear accurate messages about superintelligence, then corporate advertising may be designed accordingly. The advertising might overstate the company's positive role, which would be analogous to greenwashing and could likewise be harmful for reducing accountability for bad corporate actors, but even then it would at least be spreading an accurate message about superintelligence. Another strategy is for the employees of AI companies to mobilize against the companies supporting superintelligence misinformation, or against misinformation in general. At present, this may be a particularly promising strategy. There is a notable recent precedent for this in the successful employee action against Google's participation in Project Maven, a defense application of AI  [45] . While not specifically focused on misinformation, this incident demonstrates the potential for employee action to change the practices of AI companies, including when those practices would otherwise be profitable for the company.
 Focus Media Attention on Constructive Debates Public media can inadvertently spread misinformation via the journalistic norm of balance. For the sake of objectivity, journalists often aim to cover "both sides" of an issue. While this can be constructive for some issues, it can also spread misinformation. For example, media coverage has often presented "both sides" of the "debate" over whether tobacco causes cancer or whether human activity causes global warming, even when one side is clearly correct and the other side has a clear conflict of interest  [10, 13] . One potential response for this is to attempt to focus media attention on legitimate open questions about a given issue, questions for which there are two meaningful sides to cover. For global warming, this could be a debate over the appropriate role of nuclear power or the merits of carbon taxes. For superintelligence, it could be a debate over the appropriate role of government regulations, or over the values that superintelligence (or AI in general) should be designed to promote. These sorts of debates satisfy the journalistic interest in covering two sides of an issue and provide a dramatic tension that can make for a better story, all while drawing attention to important open questions and affirming basic information about the topic.
 Establish Legal Requirements Finally, there may be some potential to legally require certain actors, especially corporations, to refrain from spreading misinformation. A notable precedent is the court decision of United States v. Philip Morris, in which nine tobacco companies and two tobacco trade organizations were found guilty of conspiring to deceive the public about the link between tobacco and cancer. Such legal decisions can have powerful effect. However, legal requirements may be poorly suited to superintelligence misinformation. First, legal requirements can be slow to develop. The court case United States v. Philip Morris began in 1999, an initial ruling was reached in 2006, and that ruling was upheld in 2009. Furthermore, United States v. Philip Morris came only after several decades of tobacco industry misinformation. Given the evolving nature of AI technology, it could be difficult to pin down which information is correct over such long time periods. Second, superintelligence is a future technology for which much of the correct information cannot be established with the same degree of rigor. Furthermore, if and when superintelligence is built, it could be so transformative as to render current legal systems irrelevant. (For more general discussion of the applicability of legal mechanisms to superintelligence, see  [46] [47] [48] .) For these reasons, legal requirements are less likely to play a significant role in preventing superintelligence misinformation.
 Correcting Superintelligence Misinformation Correcting misinformation is sufficiently difficult that it will often be better to prevent it from spreading in the first place. However, when superintelligence misinformation cannot be prevented, there are strategies available for correcting it in the minds of those who are exposed to it. Correcting misinformation is the subject of a fairly extensive literature in psychology, political science, and related fields  [2, 15, 33, 49] . For readers unfamiliar with this literature, Cook et al.  [2]  provide an introductory overview accessible to an interdisciplinary readership, while Ecker et al.  [15]  provide a more detailed and technical survey. This section applies this literature to the correction of superintelligence misinformation.
 Build Expert Consensus and the Perception Thereof At present, there exists substantial expert disagreement about a wide range of aspects of superintelligence, from basic matters such as whether superintelligence is possible  [50] [51] [52]  and when it might occur if it does  [53] [54] [55]  to subtler matters such as the treacherous turn vs. the sordid stumble. The situation stands in contrast to the extensive expert consensus on other issues such as global warming  [56] . (Experts lack consensus on some important details about global warming, such as how severe the damage is likely to be, but they have a high degree of consensus on the basic contours of the issue.). The case of global warming shows that expert consensus on its own does not counteract misinformation. On the contrary, misinformation about global warming continues to thrive despite the existence of consensus. However, there is reason to believe that the consensus helps. For starters, much of the misinformation is specifically oriented towards creating the false perception that there is no consensus  [10] . The scientific consensus is a target of misinformation because it is believed to be an important factor in people's overall beliefs. Indeed, several studies have documented a strong correlation among the lay public between rejection of the science of global warming and belief that there is no consensus  [57, 58] . Further studies find that presenting messages describing the consensus increases belief in climate science and support for policy to reduce greenhouse gas emissions  [14, 59] . Notably, this effect is observed for people across the political spectrum, including those who would have political motivation to doubt the science. (Such motivations are discussed further in Section 5.2.) All of this indicates an important role for expert consensus in broader beliefs about global warming. For superintelligence, at present there is no need to spread misinformation about the existence of consensus because there is rather little consensus. Therefore, a first step is to work towards consensus. (This of course should be consensus grounded on the best possible analysis, not consensus for the sake of consensus.) This may be difficult for superintelligence because of the inherent challenge of understanding future technologies and the complexity of advanced AI. Global warming has its own complexities, but the core science is relatively simple: increased atmospheric greenhouse gas concentrations trap sunlight and raise temperatures. However, at least some aspects of superintelligence should be easy enough to get consensus on, starting with the fact that there are a number of R&D groups attempting to build AGI. Other aspects may be more difficult to build consensus on, but this consensus is at least something that can be pursued via normal channels of expert communication: research articles, conference symposia, private correspondence, and so on. Given the existence of consensus, it is also important to raise awareness about it. The consensus cannot counteract misinformation if nobody knows about it. The global warming literature provides good models for documenting expert consensus  [56] , and such findings of consensus can be likewise be publicized.
 Address Pre-Existing Motivations for Believing Misinformation The human mind tends to not process new information in isolation, but instead processes it in relation to wider beliefs and understandings of the world. This can be very valuable, enabling us to understand the context behind new information and relate it to existing knowledge. For example, people would typically react with surprise and confusion upon seeing an object rise up to the ceiling instead of fall down to the floor. This new information is related to a wider understanding of the fact that objects fall downwards. People may even struggle to believe their own eyes unless there is a compelling explanation. (For example, perhaps the object and the ceiling are both magnetized.). Additionally, if people did not see it with their own eyes, but instead heard it reported by someone else, they may be even less likely to believe it. In other words, they are motivated to believe that the story is false, even if it is true. This phenomenon is known as motivated reasoning. While generally useful, motivated reasoning can be counterproductive in the context of misinformation, prompting people to selectively believe misinformation over correct information. This occurs in particular when the misinformation accords better with preexisting beliefs than the correct information. In the above example, misinformation could be that the object fell down to the floor instead of rising to the ceiling. Motivated reasoning is a major factor in the belief of misinformation about politically contentious issues such as climate change. The climate science consensus is rejected mainly by people who believe that government regulation of industry is generally a bad thing  [14, 59] . In principle, belief that humans are warming the planet should have nothing to do with belief that government regulations are harmful. It is logically coherent to believe in global warming yet argue that carbon emissions should not be regulated. However, in practice, the science of global warming often threatens people's wider beliefs about regulations, and so they find themselves motivated to reject the science. Motivated reasoning can also be a powerful factor for beliefs about superintelligence. A basic worldview is that humans are in control. Per this worldview, human technology is a tool; the idea that it could rise up against humanity is a trope for science fiction, not something to be taken seriously. The prospect of superintelligence threatens this worldview, predisposing people to not take superintelligence seriously. In this context, it may not help that media portrayals of the scholarly debate about superintelligence commonly include reference to science fiction, such as by using pictures of the Terminator. As one expert who is concerned about superintelligence states, "I think that at this point all of us on all sides of this issue are annoyed with the journalists who insist on putting a picture of the Terminator on every single article they publish of this topic"  [60] . Motivated reasoning has been found to be linked to people's sense of self-worth. As one study puts it, "the need for self-integrity-to see oneself as good, virtuous, and efficacious-is a basic human motivation"  [61]  (p. 415). When correct information threatens people's self-worth, they are more motivated to instead believe misinformation, so as to preserve their self-worth. Furthermore, motivated reasoning can be reduced by having people consciously reaffirm their own self-worth, such as by recalling to themselves ways in which they successfully live up to their personal values  [61] . Essentially, with their sense of self-worth firmed up, they become more receptive to information that would otherwise threaten their self-worth. As a technology that could outperform humans, superintelligence could pose an especially pronounced threat to people's sense of self-worth. It may be difficult for people to feel good and efficacious if they would soon be superseded by computers. For at least some people, this could be a significant reason to reject information about the prospect of superintelligence, even if that information is true. At the same time, it may still be valuable for messages about superintelligence to be paired with messages of affirmation. Another important set of motivations comes from the people active in superintelligence debates. Many people in the broader computer science field of AI have been skeptical of claims about superintelligence. These people may be motivated by a desire to protect the reputation and funding of the field of AI, and in turn protect their self-worth as AI researchers. AI has a long history of boom-bust cycles in which hype about superintelligence and related advanced AI falls flat and contributes to an "AI winter". Peter Bentley, an AI computer scientist who has spoken out against contemporary claims about superintelligence, is explicit about this: "Large claims lead to big publicity, which leads to big investment, and new regulations. And then the inevitable reality hits home. AI does not live up to the hype. The investment dries up. The regulation stifles innovation. And AI becomes a dirty phrase that no-one dares speak. Another AI Winter destroys progress"  [62]  (p. 10). "Do not be fearful of AI-marvel at the persistence and skill of those human specialists who are dedicating their lives to help create it. And appreciate that AI is helping to improve our lives every day" (p. 11). While someone's internal motivations can only be inferred from such text, the text is at least suggestive of motivations to protect self-worth and livelihood as an AI researcher, as well as a worldview in which AI is a positive force for society. To take another example, Torres  [9]  proposes that Pinker's dismissal of AGI and superintelligence is motivated by Pinker's interest in promoting a narrative in which science and technology bring progress-a narrative that could be threatened by the potential catastrophic risk from superintelligence. Conversely, some people involved in superintelligence debates may be motivated to believe in the prospect of superintelligence. For example, researcher Jürgen Schmidhuber writes on his website that "since age 15 or so, the main goal of professor Jürgen Schmidhuber has been to build a self-improving Artificial Intelligence (AI) smarter than himself, then retire."  [63]  Superintelligence is also sometimes considered the "grand dream" of AI  [64] . Other common motivations include a deep interest in transformative future outcomes  [65]  and a deep concern about extreme catastrophic risks  [4, 66, 67] . People with these worldviews may be predisposed to believe certain types of claims about superintelligence. If it turns out that superintelligence will not be built, or would not have transformative or catastrophic effects, then this can undercut people's deeply held beliefs in the importance of superintelligence, transformative futures, and/or catastrophic risks. For each of these motivations for interest in superintelligence, there can be information that is rejected because it cuts against the motivations and misinformation that is accepted because it supports the motivations. Therefore, in order to advance superintelligence debates, it can be valuable to affirm people's motivations when presenting conflicting information. For example, one could affirm that AI computer scientists are making impressive and important contributions to the world, and then explain reasons why superintelligence may nonetheless be a possibility worth considering. One could affirm that science and technology are bringing a great deal of progress, and then explain reasons why some technologies could nonetheless be dangerous. One could affirm that superintelligence is indeed a worthy dream, or that transformative futures are indeed important to pay attention to, and then explain reasons why superintelligence might not be built. Finally, one could affirm that extreme catastrophic risks are indeed an important priority for human society, and then explain reasons why superintelligence may not be such a large risk after all. These affirming messaging strategies could predispose participants in superintelligence debates to consider a wider range of possibilities and make more progress on the issue, including progress towards expert consensus. Another strategy is to align motivations with accurate beliefs about superintelligence. For example, some AI computer scientists may worry that belief in the possibility of superintelligence could damage reputation and funding. However, if belief in the possibility of superintelligence would bring reputational and funding benefits, then the same people may be more comfortable expressing such belief. Reputational benefits could be created, for example, via slots in high-profile conferences and journals, or by association with a critical mass of reputable computer scientists who also believe in the possibility of superintelligence. Funding could likewise be made available. Noting that funding and space in conferences and journals are often scarce resources, it could be advantageous to target these resources at least in part toward shifting motivations of important actors in superintelligence debates. This example of course assumes that it is correct to believe in the possibility of superintelligence. The same general strategy of aligning motivations may likewise be feasible for other beliefs about superintelligence. The above examples-concerning the reputations and funding of AI computer scientists, the possibility of building superintelligence, and the importance of transformative futures and catastrophic risks-all involve experts or other communities that are relatively attentive to the prospect of superintelligence. Other motivations could be significant for the lay public, policy makers, and other important actors. Research on the public understanding of science finds that cultural factors, such as political ideology, can factor significantly in the interpretation of scientific information  [68, 69] . Kahan et al.  [69]  (p. 79) propose to "shield" scientific evidence and related information "from antagonistic cultural information". For superintelligence, this could mean attempting to frame superintelligence (or, more generally, AI) as a nonpartisan social issue. At least in the US, if an issue becomes politically partisan, legislation typically becomes substantially more difficult to pass. Likewise, discussions of AI and superintelligence should, where reasonably feasible, attempt to avoid close association with polarizing ideologies and cultural divisions. The fact that early US legislation on AI has been bipartisan is encouraging. For example, H.R.4625, FUTURE of Artificial Intelligence Act of 2017, sponsored by John Delaney (Democrat) and co-sponsored by Pete Olson (Republican), and H.R.5356, National Security Commission Artificial Intelligence Act of 2018, sponsored by Elise Stefanik (Republican) and co-sponsored by James Langevin (Democrat). This is a trend that should be praised and encouraged to continue.
 Inoculate with Advance Warnings The misinformation literature has developed the concept of inoculation, in which people are preemptively educated about a piece of misinformation so that they will not believe it if and when they later hear it. For example, someone might be told that there is a false rumor that vaccines cause autism, such that when they later hear the rumor, they know to recognize it as false. The aim is to get people to correctly understand the truth about a piece of misinformation from the beginning, so that their minds never falsely encode it. Inoculation has been found to work better than simply telling people the correct information  [70] . Inoculation messages can include why a piece of misinformation is incorrect as well as why it is being spread  [71] . For example, misinformation casting doubt on the idea that global temperatures are rising could be inoculated with an explanation of how scientists have established that global temperatures are rising. The inoculation could also explain that industries are intentionally casting doubt about global temperature increases in order to avoid regulations and increase profits. Likewise, for superintelligence, misinformation claiming that there are no projects seeking to build AGI could be inoculated by explanations of the existence of AGI R&D projects, and perhaps also explanations of the motivations of people who claim that there are no such projects. For example, Torres  [9]  proposes that Pinker's dismissal of AGI and superintelligence is motivated by Pinker's interest in promoting a narrative in which science and technology bring progress-a narrative that could be threatened by the potential catastrophic risk from superintelligence.
 Explain Misinformation and Corrections When people are exposed to misinformation, it can be difficult to correct, as first explained in Section 4. This phenomenon has been studied in great depth, with the terms "continued influence" and "belief perseverance" used for cases in which debunked information continues to influence people's thinking  [72, 73] . There is also an "illusion of truth", in which information explained to be false is later misremembered as true-essentially, the mind remembers the information but forgets its falsity  [74] . The difficulty of correcting misinformation is why this paper has emphasized strategies to prevent of misinformation from spreading in the first place. Adding to the challenge is the fact that attempts to debunk misinformation can inadvertently reinforce it. This phenomenon is known as the "backfire effect"  [74] . Essentially, when someone hears "X is false", it can strengthen their mental representation of X, thereby reinforcing the misinformation. This effect has been found to be especially pronounced among the elderly  [74] . One explanation is that correcting the misinformation (i.e., successfully processing "X is false") requires the use of strategic memory, but strategic memory requires dedicated mental effort and is less efficient among the elderly  [15] . Unless enough strategic memory is allocated to processing "X is false", the statement can end up reinforcing belief in X. These findings about the backfire effect have important consequences for superintelligence misinformation. Fortunately, many important audiences for superintelligence misinformation are likely to have strong strategic memories. Among the prominent actors in superintelligence debates, relatively few are elderly, and many of them have intellectual pedigrees that may endow them with strong strategic memories. On the other hand, many of the prominent actors are busy people with limited mental energy available for processing corrections about superintelligence information. As a practical matter, people attempting to debunk superintelligence misinformation should generally avoid "X is false" messages, especially when their audience may be paying limited attention. One technique that has been particularly successful at correcting misinformation is the use of refutational text, which provides detailed explanations of why the misinformation is incorrect, what the correct information is, and why it is correct. Refutational text has been used mainly as a classroom tool for helping students overcome false preexisting beliefs about course topics  [75, 76] . Refutational text has even been used to turn misinformation into a valuable teaching tool  [77] . A meta-analysis found refutational text to be the most effective technique for correcting misinformation in the context of science education-that is, for enabling students to overcome preexisting misconceptions about science topics  [78] . A drawback of refutational text is that it can require more effort and attention than simpler techniques. Refutational text may be a valuable option in classrooms or other settings in which one has an audience's extended attention. Such settings include many venues of scholarly communication, which can be important for superintelligence debates. However, refutational texts may be less viable in other settings, such as social media and television news program interviews, in which one can often only get in a short sound bite. Therefore, refutational text may be relatively well-suited for interactions with experts and other highly engaged participants in superintelligence debates, and relatively poorly suited for much of the lay public and others who may only hear occasional passing comments about superintelligence. That said, it may still be worth producing and disseminating extended refutations for lay public audiences, such as in long-format videos and articles for television, magazines, and online. These may tend to only reach the most motivated segments of the lay public, but they can nonetheless be worthwhile.
 Conclusions Superintelligence is a high-stakes potential future technology as well as a highly contested socio-technological issue. It is also fertile terrain for misinformation. Making progress on the issue requires identifying and rejecting misinformation and accepting accurate information. Some progress will require technical research to clarify the nature of superintelligence. However, a lot of progress will likely also require the sorts of sociological and psychological strategies outlined in this paper. The most progress may come from interdisciplinary projects connecting computer science, social science, and other relevant fields. Computer science is a highly technical field, but as with all fields, it is ultimately composed of human beings. By appreciating the nuances of the human dimensions of the field, it may be possible to make better progress towards understanding superintelligence and acting responsibly about it. As the first dedicated study of strategies for countering superintelligence misinformation, this paper has taken a broad view, surveying a range of options. Despite this breadth, there may still be additional options worth further attention. Indeed, this paper has only mined a portion of the insights contained within the existing literature on misinformation. There may also be compelling options that go beyond the literature. Likewise, because of this paper's breadth, it has given relatively shallow treatment to each of the options. More detailed attention to the various option would be another worthy focus of future research. An especially valuable focus would be the proposed strategies for preventing superintelligence misinformation. Because misinformation can be so difficult to correct, preventing it may be the more effective strategy. There is also less prior research on the prevention of misinformation. For these reasons, there is likely to be an abundance of important research opportunities on the prevention of misinformation, certainly for superintelligence misinformation and perhaps also for misinformation in general. For the prevention of superintelligence misinformation, a strategy that may be particularly important to study further is dissuading AI corporations from using their substantial resources to spread superintelligence misinformation. The long history of corporations engaging in such tactics, with a major impact on the surrounding debates, suggests that this could be a highly important factor for superintelligence  [12] . It may be especially valuable to study this at an early stage, before such tactics are adopted. For the correction of superintelligence misinformation, a particularly promising direction is on the motivations and worldviews of prominent actors and audiences in superintelligence debates. Essentially, what are people's motivations with respect to superintelligence? Are AI experts indeed motivated to protect their field? Are superintelligence developers motivated by the "grand dream"? Are others who believe in the prospect of superintelligence motivated by beliefs about transformative futures or catastrophic risks? Can attention to these sorts of motivations help them overcome their divergent worldviews and make progress towards consensus on the topic? Finally, are people in general motivated to retain their sense of self-worth in the face of a technology that could render them inferior? Most important, however, is not the research on superintelligence misinformation, but the efforts to prevent and correct it. It can often be stressful and thankless work, especially amidst the heated debates, but it is essential to ensuring positive outcomes. This paper is one effort towards helping this work succeed. Given the exceptionally high potential stakes, it is vital that decisions about superintelligence be well-informed.

Abstract: Superintelligence is a potential type of future artificial intelligence (AI) that is significantly more intelligent than humans in all major respects. If built, superintelligence could be a transformative event, with potential consequences that are massively beneficial or catastrophic. Meanwhile, the prospect of superintelligence is the subject of major ongoing debate, which includes a significant amount of misinformation. Superintelligence misinformation is potentially dangerous, ultimately leading bad decisions by the would-be developers of superintelligence and those who influence them. This paper surveys strategies to counter superintelligence misinformation. Two types of strategies are examined: strategies to prevent the spread of superintelligence misinformation and strategies to correct it after it has spread. In general, misinformation can be difficult to correct, suggesting a high value of strategies to prevent it. This paper is the first extended study of superintelligence misinformation. It draws heavily on the study of misinformation in psychology, political science, and related fields, especially misinformation about global warming. The strategies proposed can be applied to lay public attention to superintelligence, AI education programs, and efforts to build expert consensus.