Julian Stastny
Johannes Treutlein
Maxime Riché
Jesse Clifton

Multi-agent learning in mixed-motive coordination problems

Introduction In An Essay on  Bargaining [1956] , Thomas Schelling describes a negotiation over the price of a house. Considering different strategies the negotiators might use, he points out that If the buyer is an agent authorized by a board of directors to buy at $16000 but not a cent more, and the directors cannot constitutionally meet again for several months and the buyer cannot exceed his authority, and if all this can be made known to the seller, then the buyer 'wins' -if, again, the seller has not tied himself up with a commitment to  $19000. (pg. 284)  Imagine various actors with different preferences -a mixed-motive setting  [Schelling, 1958]  -developing machine learning systems to act on their behalf. These actors are like Schelling's bargainers, in that their systems implicitly encode a commitment to pursuing a particular tradeoff between their own preferences and those of other players. Just as the price negotiation in Schelling's example may fail if buyer and seller are each committed to insisting on incompatible prices, machine learning systems may fail to cooperate if they encode commitments to incompatible tradeoffs between their principals' goals. Multi-agent learning (MAL) has in recent years paid considerable attention to problems of cooperation between agents with different, but not zero-sum, preferences. However, much of this work has been in the setting of (sequential) social dilemmas (SSDs) (e.g.,  Peysakhovich and Lerer 2017a , Lerer and Peysakhovich 2017 , Peysakhovich and Lerer 2017b , Foerster et al. 2018 , Wang et al. 2018 , Eccles et al. 2019 . The classic example of a social dilemma is the Prisoner's Dilemma (PD), and the SSDs studied in this literature are similar to the Prisoner's Dilemma in that there is a single salient notion of "cooperation". This means that it is relatively easy for actors to coordinate in their selection of policies to deploy in these settings. This situation is quite unlike that faced by Schelling's bargainers, for whom there are many possible agreements to commit to, and who thus may fail to reach an agreement if they make incompatible commitments 1 .  Cao et al. [2018]  are an exception in that they look at negotiation between deep reinforcement learners, such that there are multiple incompatible ways in which agents could choose among possible agreements. However, they do not evaluate the performance of independently trained policies when played against one Working draft. Last updated March 8, 2021. 1 The inadequacy of the Prisoner's Dilemma as a model for cooperation problems, and the need to study games that exhibit competing "cooperative" solutions, has been pointed out in other fields. See  Snidal [1985]  for a discussion in international relations,  McAdams [2008]  in law, and  Binmore [1998]  in evolutionary game theory. another ("cross-play"  [Hu et al., 2020] ), which is where the existence of multiple incompatible cooperative solutions creates problems. In this note, we argue that a focus on sequential social dilemmas and the lack of cross-play evaluation obscures a fundamental problem for decentralized MAL in mixed-motive settings. We discuss a class of games that we call mixed-motive coordination problems (MCPs), which highlight this problem. We compare MCPs to pure coordination problems and SSDs. We then illustrate how MAL methods which achieve cooperation in other social dilemmas fail in some simple MCPs, and argue that future methods for promoting cooperation in mixed-motive settings should be benchmarked in these more difficult cases. We close with some tentative positive contributions, by sketching some partial solutions to MCPs.
 Preliminaries We work in the setting of partially observable stochastic games (POSGs). For simplicity we assume two players, i = 1, 2. We will index player i's counterpart by −i. Each player has an action space A i , there is an underlying state space S which evolves according to a Markovian transition function P (S t+1 | S t , A t 1 , A t 2 ). At each time step, each player sees an observation O t i which depends on S t . Thus each player has an accumulating history of observations H t i = {O v 1 , A v 1 } t v=1 . Call the set of all histories H i = ∪ ∞ t=1 H t i . Assume for simplicity that the initial observation history is fixed and common knowledge, h 0 1 = h 0 2 ≡ h 0 . Finally, principals choose among policies π i : H → ∆(A i ), which we imagine as artificial agents deployed by the principals. We will refer to policy profiles generically as π ∈ Π := Π 1 × Π 2 . Each player has a reward function r i , such that r i (S t , A t 1 , A t 2 ) is their reward at time t. Define the value to player i of policy profile π starting at history h t i as V i (h t i , π) = E π [ ∞ v=t γ v−t r i (S v , A v 1 , A v 2 ) | H t i = h t i ], where γ < 1 is a discount factor. Define the value of a policy profile to player i as V i (π) = V i (h 0 , π). A payoff profile is then a tuple (V 1 (π), V 2 (π)). We say that π is a (Nash) equilibrium of a POSG if π i ∈ arg max π i ∈Πi V i (h 0 , π i , π −i ) for i = 1, 2. We say that π is Pareto-optimal if for i = 1, 2 and π ∈ Π, we have that V i (π ) > V i (π) implies V −i (π ) < V −i (π).
 Mixed-motive coordination problems We envision multiple actors ("principals") deploying machine learning systems, defined by policies, into the world. The principals may be human developers, or they may be machine learning systems themselves. The principals simultaneously deploy policies, without communication (in Section 6 we discuss ways in which communication may change the analysis). Let W be a set of welfare functions, which are supposed to encode normative properties (total welfare, fairness, etc.) of different payoff profiles. Two uncontroversial properties of a welfare function are Pareto-optimality (i.e., its optimizer should be Pareto-optimal) and symmetry (the welfare of a policy profile should be invariant to permutations of player indices). Beyond that, there are several appealing properties which cannot all be satisfied by the same welfare function (see Table  1 ). The experimental evidence regarding people's preferences over welfare functions is inconclusive  [Felsenthal and Diskin, 1982] . We will say that a game is a mixed-motive coordination problem (MCP) if it has several equilibria 2 which are optimal with respect to some welfare function, but are incompatible with each other. Definition 3.1 (Mixed-motive coordination problem). Let G be a POSG. Let W be a set of welfare functions, and let W E ⊆ W be the set of welfare functions in W which are optimized by some equilibrium of G. Then G is a mixed-motive coordination problem (with respect to W) if there exist distinct w, w ∈ W E 2 such that (π w 1 , π w 2 ) has a Pareto-suboptimal policy profile for all π w ∈ arg max π w(π ), π w ∈ arg max π w (π ). An example of a mixed-motive coordination problem is the Bach or Stravinsky game. A variant of this game with asymmetric payoffs is given in Figure  1  (left). The folk theorem  [Friedman, 1971]  tells us that any payoff profile in the convex hull of these payoffs is attainable in an equilibrium of the repeated game, for players with sufficiently high discount factors. And, as is suggested by the shape of the Pareto frontier for Name of welfare function w
 Form of w(π) Nash  [Nash, 1950]  [ V 1 (π) − d 1 ] • [V 2 (π) − d 2 ] Kalai-Smorodinsky  [Kalai and Smorodinsky, 1975]   V1(π)−d1 V2(π)−d2 − sup π V1(π)−d1 sup π V2(π)−d2 + ι {π Pareto-optimal } Egalitarian  [Kalai, 1977]  min {V 1 (π) − d 1 , V 2 (π) − d 2 , } Utilitarian (e.g. Harsanyi 1955) V 1 (π) + V 2 (π) Table  1 : Welfare functions, adapted to the multi-agent RL setting where two agents with value functions V 1 , V 2 are bargaining over the policy profile π. to deploy. The function ι in the definition of the Kalai-Smorodinsky welfare is the ∞-0 indicator, used to enforce the constraint in its argument. V d i = V i (π d ) are the players' disagreement values. Note that in bargaining problems in which the set of feasible payoffs is convex, the Nash welfare function uniquely satisfies the properties of (1) Pareto optimality, (2) symmetry, (3) invariance to affine transformations, and (4) independence of irrelevant alternatives. See  Zhou [1997]  for a characterization of the Nash welfare in non-convex bargaining problems. The Nash welfare can also be obtained as the subgame perfect equilibrium of an alternating-offers game as the "patience" of the players goes to infinity  [Binmore et al., 1986] . On the other hand, Kalai-Smorodinsky uniquely satisfies (1)-(  3 ) plus (  5 ) resource monotonicity, which means that all players are weakly better off when there are more resources to go around. The egalitarian solution instead satisfies (1), (  2 ), (4), and (5). The utilitarian welfare function is implicitly used in the work of  Peysakhovich and Lerer [2017a] ,  Lerer and Peysakhovich [2017] ,  Wang et al. [2018]  on cooperation in sequential social dilemmas. BoS B S B 4, 1 0, 0 S 0, 0 2, 2 PD C D C -1, -1 -3, 0 D 0, -3 -2, -2 Figure  1 : Payoffs for asymmetric Bach or Stravinsky (BoS) and the Prisoner's Dilemma (PD). BoS (Figure  2 , left), different welfare functions pick out different Pareto-optimal payoff profiles. For instance, taking the disagreement point to be d 1 , d 2 = (0, 0), the Nash welfare is optimized at (3.0, 1.5), whereas the egalitarian welfare is optimized at (2.0, 2.0). On the other hand, all welfare functions in Table  1  agree on the payoffs corresponding to (C, C) in the iterated Prisoner's Dilemma (Figure  2 , right). In the following, we compare mixed-motive coordination problems to the more general equilibrium selection problem and to sequential social dilemmas.
 Relation to other cooperation problems Equilibrium and Pareto selection problems It is well-known that achieving cooperation between independently-trained agents this is a challenging problem for MAL in general (e.g.,  Boutilier 1999 , Matignon et al. 2012 , Hu et al. 2020 ). One reason is that there are often multiple equilibria, such that (to the extent that one wants to deploy a policy which is a best-response to that of one's counterpart) the choice of a policy constitutes an equilibrium selection problem. Such problems even arise in the fully cooperative case, i.e., when V 1 = V 2 ≡ V , and are then also called Pareto selection problems  [Matignon et al., 2012]  or pure coordination problems. In a pure coordination problem, any π ∈ arg max π V (π) has an optimal value and is thus in equilibrium. There is an equilibrium selection problem if there exist π, π ∈ arg max π V (π) such that (π 1 , π 2 ) has Pareto-suboptimal payoff profile. Mixed-motive coordination problems that occur between policy profiles in equilibrium are a subclass of equilibrium selection problems. They only occur in general-sum, non-fully cooperative cases and pose a challenge which is qualitatively different from the fully cooperative case: players have to coordinate among options over which they have conflicting preferences. The pure coordination problem can be solved by merely giving the players a chance to converge upon a coordinated policy profile (as in  Boutilier [1999] , for instance) and any such policy will do equally well. This is not so in mixed-motive coordination problems, since agents that are open to converging upon any Pareto-optimal outcome would be exploitable by opponents that insist on an outcome they prefer. (See Example 6.0.1 for a concrete illustration of this difference.) Sequential social dilemmas Sequential social dilemmas (SSDs), as defined by  Leibo et al. [2017] , are POSGs whose policy space contains subsets corresponding to cooperation and defection, called Π C i and Π D i , respectively. At each state, the normal form game whose payoffs are value functions under profiles of policies in Π C i , Π D i satisfies the social dilemma inequalities  [Macy and Flache, 2002] , where atomic "cooperate" and "defect" actions are replaced with elements of Π C i , Π D i . An SSD is not necessarily an equilibrium selection problem. For instance, any Pareto-optimal and symmetric policy profile will select an equilibrium profile that leads to constant cooperation in the iterated Prisoner's Dilemma, and which is compatible with other such profiles. But other SSDs may exhibit multiple incompatible ways of implementing cooperation. SSDs with symmetric Pareto-optimal equilibria are in any case not mixed-motive coordination problems. This includes the Gathering, Wolfpack  [Leibo et al., 2017] , and Coin Game  [Peysakhovich and Lerer, 2017a]  environments. For instance, in the Coin Game, the single cooperative payoff profile is attained when agents get only the coin of their own color, rather than attempting to take their counterpart's coin as well.
 Learning algorithms Play between policies trained by different runs or variations of an algorithm, or "cross-play"  [Hu et al., 2020]  is problematic for MCPs. To illustrate the need for evaluating policies in cross-play, we train policies using two multi-agent learning algorithms which have been shown to achieve cooperation in SSDs. The first is a variant of  Lerer and Peysakhovich [2017] 's approximate Markov tit-for-tat (amTFT). The original amTFT algorithm trains a cooperative policy on the utilitarian welfare (Table  1 ), as well as a punishment policy, and switches from the cooperative to the punishment policy when it detects that the other player is defecting from the cooperative policy. We consider the more general class of algorithms in which a cooperative policy is constructed by optimizing some welfare function. Call the version of amTFT in which the cooperative policy optimizes welfare function w as amTFT(w). In our experiments, we use the utilitarian welfare function w Util and an inequity averse welfare function w IA . Following  Hughes et al. [2018] , this welfare function is constructed using modified reward functions which penalize inequitable outcomes: V IA i (a t 1 , a t 2 ; α, β) = r 1 (a t 1 , a t 2 ) + r 1 (a t 2 , a t 2 ) − α[e t 1 (a t 1 , a t 2 ) − e t 2 (a t 1 , a t 2 )] + − β[e 2 (a t 1 , a t 2 ) − e 1 (a t 1 , a t 2 )] + where [x] + = max{x, 0} and e t i are smoothed cumulative rewards computed as e t i (a t 1 , a t 2 ) = γλe t−1 i (a t−1 1 , a t−1 2 )+ r i (a t 1 , a t 2 ) with hyperparameter λ. and discount factor γ α and β control how much unequal outcomes are penalized.  3  The second learning algorithm is learning with opponent-learning awareness (LOLA;  Foerster et al. 2018 ) a policy gradient method in which learners attempt to shape the others' learning through the choice of their own policy updates. Write V i (θ 1 , θ 2 ) as the value to player i under a profile of policies with parameters θ 1 , θ 2 . Then, the LOLA update for player 1 at time t with parameters δ, η > 0 is θ t 1 = θ t−1 1 + δ∇ θ1 V i (θ 1 , θ 2 ) + δη [∇ θ2 V 1 (θ 1 , θ 2 )] T ∇ θ1 ∇ θ2 V 2 (θ 1 , θ 2 ). We use the discount factor γ = 0.96 throughout.
 Results Here we illustrate the special problem posed by MCPs by comparing the performance of the learning algorithms above in iterated BoS (IBoS) and in the iterated Prisoner's Dilemma (IPD). The folk theorems  [Mailath et al., 2006]  tell us that, for players with sufficiently high discount factors, any payoff profile of the stage game in which players get at least as much as they can unilaterally guarantee themselves can be attained in an equilibrium of the repeated game. IBoS is an MCP with respect to the set of welfare functions W = {w Util , w IA }, since these welfare functions are optimized by distinct payoff profiles for sufficiently large values of α, β. We measure the performance of these algorithms in self-play and cross-play. Self-play payoffs are the average payoffs attained by a profile of policies produced by the same training run. Cross-play payoffs are the average payoffs attained by a profile of two policies produced by two different training runs. In the case of amTFT, we also consider cross-play between policies trained with amTFT(w Util ) and those trained with amTFT(w IA ).
 LOLA Following  Foerster et al. [2018] 's parameterization of policies in the iterated Prisoner's Dilemma, we used policies which condition on the previous pair of actions played, with one parameter for every possible previous action profile. To learn policy profiles, we used a discount factor of γ = 0.96 and applied LOLA to the closed-form value functions (which can be computed for each policy profile using the Bellman equations; this is called exact LOLA in  Foerster et al. [2018] ). Figure  2 : LOLA average payoff profiles for 40 training runs in IBoS (left) and IPD (right). Cross-play payoffs were obtained by randomly sampling 40 pairs (π 1 , π 2 ), where π 1 and π 2 come from different runs. A small amount of random noise is applied to each point to enhance visibility. On average, self-play yields payoffs (2.69, 1.34), while cross-play yields (1.30, 0.70). As Figure  2  shows, the self-play payoffs almost all lie close to the empirical Pareto frontier, which consists of payoff profiles roughly equal to {(4, 1), (3, 1.5), (2, 2)}. This clustering is caused by the policies depending only on actions taken at the previous timestep, which allows for always playing (S,S), alternating between them, or always playing (B,B). These coincide with the maxima for w Util , the Nash welfare  [Nash, 1950]  under disagreement profile (0, 0), or w IA . On the other hand, the cross-play points are much more often Pareto-suboptimal. This is not surprising, as if LOLA produces tit-for-tat-like strategies which punish deviations from a particular strategy profile, then playing policies from separate runs is liable to lead to incompatible policies and therefore punishments.
 amTFT We evaluated self-and cross-play for amTFT in IBoS and IPD, where each episode lasted 100 of steps. Payoff profiles from individual runs are displayed in Figure  3 , and average payoffs under self-and cross-play for each combination of welfare functions are displayed in Table  2 . Table 2: Self-play ("•") and cross-play ("×") payoffs in IBoS for amTFT(w Util ) and amTFT(w IA ). Payoffs are averages over 40 training runs. w Util × w IA leads to the same payoff regardless of which player operates under which welfare function. All standard errors were below 0.01. Of course, the goal of developing benchmarks is not only to point out flaws in existing approaches, but also to help develop better ones. In this section we discuss several research directions for ameliorating MCPs, and give a toy illustration of one approach.
 Policy profile w Util • w Util w IA • w IA w Util × w Util w IA × w IA w Util × Coordination by the principals MCPs can be solved if the principals can agree on a way of selecting among policy profiles to deploy. For instance, principals might agree on a welfare function, and choose learning algorithms which converge to an optimizer of that welfare function. Coordination on a welfare function (and among equilibria which all maximize that welfare function) may be easy in toy environments such as the ones we have considered. But as systems become more complex, it will likely be intractable for principals to fully coordinate so as to train welfare-optimal agents. Thus there is a need to develop methods for low-bandwidth coordination between principals training agents in MCPs which captures most of the benefits of full coordination. Bandwidth-constrained coordination has already been studied in fully cooperative games (e.g.,  Stone and Veloso 1999, Zhang et al. 2019) . Compromise between welfare functions We have not discussed how giving agents the opportunity to communicate post-deployment might ameliorate MCPs. One possibility is to have them reach a compromise over different welfare-optimal policy profiles before deciding which to deploy. On possibility is for the train to several policies corresponding to several welfare functions according to which they are willing to play. When given the chance to communicate, their agents can then apply a bargaining protocol to this reduced set of policy profiles. Here is a simple example of such a protocol. Agents first announce the sets of policy profiles they are willing to play. If these sets overlap, a policy profile is taken at random from the intersection (compare with  Chakravorti and Conley [2004] 's discussion of randomization between the Nash and Kalai-Smorodinsky bargaining solutions in the asymmetric iterated Prisoner's Dilemma). Only when these sets do not overlap do agents revert to a default policy. We give an illustration in Example 6.0.1. Example 6.0.1 (Bargaining over amTFT policies). An example of a protocol for bargaining over welfareoptimal policy profiles is given in Algorithm 1. Policies are obtained by applying amTFT to a welfare function w, returning a policy amTFT(w) i for player i. Refer to the strategy which reports welfare functions W i and default policy π d i as (W i , π d i ).  {(W 1 , π d 1 ), (W 2 , π d 2 )} in this bargaining protocol applied to IBoS. This payoff matrix clearly shows the tradeoff between exploitability and robustness to independentlytrained policies. The strategies which play {w Util , w IA } are of more robust, in the sense that they avoid the mutually dispreferred profile of (1.0, 0.3). However, these strategies are not played in equilibrium, because they are exploitable: an agent can deviate from this profile by instead playing only their preferred welfare function. Nevertheless, one has the intuition that it is reasonable to play {w Util , w IA } in view of its robustness, despite the fact that it is exploitable. On the other hand, the situation becomes more complicated when the set of welfare functions under consideration is much larger. Identifying reasonable solution concepts for this setting is an important direction for future work. Besides the fact that avoiding bargaining failure is still not guaranteed, there are additional problems for this protocol. First, it may be ex-ante Pareto suboptimal, because randomization between Pareto optimal payoff profiles may be ex-ante Pareto suboptimal. Second, while it assumes that agents agree to uniform randomization over the intersection of their sets, agents will in fact have differing preferences over different distributions with which to randomize, generating another bargaining problem. Less ad-hoc solutions might be found by using a dynamic bargaining protocol (as in the classical Rubinstein bargaining game  Rubinstein 1982) , which may narrow down the set of equilibria. As a final note, compare this protocol with an analogous protocol for a pure coordination problem (Figure  4 ). In this protocol, agents play a default action but also announce a set of actions that they are willing to play. Again, actions are taken uniformly at random from the overap of the sets if it is nonempty, and otherwise the default action is taken. Clearly playing {A, B} is a dominant strategy for each player, unlike the analogous strategy in the protocol for IBoS, where greater flexibility comes at the price of greater exploitability.    Figure 3 : 3 Figure 3: amTFT payoff profiles for 40 training runs in IBoS (left) and IPD (right). A small amount of random noise is applied to each point to enhance visibility.
 Figure 4 : 4 Figure 4: Payoffs for a pure coordination problem.
 Payoff matrix for agents playing Pure Coordination and following a variant of the bargaining protocol in 1 where sets of rather than welfare functions are announced.
 Table 3 displays the payoff table of the game with strategy profiles
 Algorithm 1: Welfare function selection protocol Input: Sets of welfare functions W i , default policies π d i // Independently train policies to maximize welfare functionsfor i = 1, 2 do Π i ← ∅ for w ∈ W i do π w i ← amTFT(w) i Π i ← Π i ∪ {π w i } //Communicate and decide what policies to deployPlayers announce W 1 , W 2 if W 1 ∩ W 2 = ∅ then w ∼ Unif(W 1 ∩ W 2 ) π 1 , π 2 ← π w 1 , π w 2 else π 1 , π 2 ← π d 1 , π d 2 return π 1 , π 2
 Table 3 : 3 Empirical payoff matrix for agents playing IBoS and following bargaining protocol in Algorithm 1 with welfare functions in {w Util , w IA }.{w Util }, π IA 2 {w IA }, π IA 2 {w Util , w IA }, π IA 2 {w Util }, π Util
			 The spirit of an MCP is that independent training predictably leads to policy profiles which optimize different welfare functions, but are incompatible. So one could modify the requirement that welfare-optimal policies be equilibria and retain this spirit. One could replace the requirement of Nash equilibrium with a different or a broader set of solution concepts, e.g., the recently-proposed Markov-Conley chains used in the α-rank evaluation methodology [Omidshafiei et al., 2019] .
			 We choose the utilitarian and inequity-averse welfare functions largely for convenience, and not because they have particularly compelling normative properties. Specifically, these welfare functions admit a Bellman equation, allowing us to straightforwardly apply reinforcement learning methods in order to maximize them. Methods for optimizing other welfare functions (Table1), which may be less amenable to off-the-shelf reinforcement learning methods, are a direction for future work.

Abstract: Cooperation in settings where agents have different but overlapping preferences (mixed-motive settings) has recently received considerable attention in multi-agent learning. However, the mixed-motive environments typically studied are simplistic in that they have a single cooperative outcome on which all agents can agree. Multi-agent systems in general may exhibit many payoff profiles which might be called cooperative, but which agents have different preferences over. This causes problems for independently trained agents that do not arise in the case of that there is a unique cooperative payoff profile. In this note, we illustrate this problem with a class of games called mixed-motive coordination problems (MCPs). We demonstrate the failure of several methods for achieving cooperation in sequential social dilemmas when used to independently train policies in a simple MCP. We discuss some possible directions for ameliorating MCPs.


Quantifying the probability of existential catastrophe: A reply to Beard et al

Introduction A recent article by  Beard, Rowe, and Fox (Beard, Rowe, & Fox, 2020; henceforth BRF)  surveys methods used to quantify the probabilities of global and existential catastrophes. BRF is a valuable contribution to the study of global catastrophic risk and existential risk. It documents the range of methods in use and evaluates their strengths and limitations, providing both a good resource for researchers wishing to get up to speed on the topic and constructive guidance for future work. In this article, I provide some further commentary on the quantification of the probability of global/existential catastrophe, making some points not made by BRF or other prior literature. At the outset, I wish to express my wholehearted agreement with BRF in the importance of sound methodology for quantifying the probabilities (and severities) of global and existential catastrophes. I regret that I share the view that "within the nascent field of Existential Risk research people have been insufficiently discriminating" in the selection and use of quantification methods (BRF, Section 7). The following remarks join BRF in seeking to improve this situation. 2. Some considerations related to severity 2.1. The definition of existential catastrophe BRF defines existential catastrophe as "the collapse of civilization or the extinction of humanity". This is a reasonable definition, but there are others. First,  Bostrom (2002)  defines existential catastrophe as an event that "would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential". This definition emphasizes long-term effects. The collapse of civilization would presumably have some long-term effects, though whether the effects would last into the distant future is a complex matter  (Baum et al., 2019) . Second,  Tonn and Stiefel (2013)  define existential catastrophe as human extinction, i.e. loss of the existence of the human species. This paper will use the BRF definition while noting the existence of other definitions for existential catastrophe, for related concepts involving large losses of value such as global catastrophe  (Morrison, 1992) , ultimate harm  (Persson & Savulescu, 2012) , and oblivion  (Tonn, 1999) , and for other concepts involving large-scale harm such as astronomical suffering  (Sotala & Gloor, 2017) .
 Probability vs. severity BRF focuses on methodology for quantifying the probability of existential catastrophe, with limited attention other factors including severity. (This focus was an explicit decision; see BRF, Section 1) That is not unreasonable for a first pass at the topic-there is plenty to say about the probability methodology. However, it is important to recognize the interplay between probability and severity. All quantifications of the probability of existential catastrophe require at least some attention to severity for the simple reason that existential catastrophe is a type of event defined by its severity. More precisely, existential catastrophe is generally defined in terms of some minimum severity threshold; in BRF, the minimum threshold is the collapse of civilization. To assess the probability of existential catastrophe is to assess the probability of a harm greater or equal to the collapse of civilization. In the terminology of catastrophe risk analysis, the probability of existential catastrophe is an "exceedance probability", i.e. the probability of harms exceeding some minimum threshold  (Grossi, Kunreuther, & Windeler, 2005) . Quantifying the probability of specific existential catastrophe events (such as a nuclear war or Earth-asteroid collision) requires additional attention to severity. The probability can be decomposed into two constituent parts as follows: = P P P * EC 1 2 (1) In Eq. (  1 ), P EC is the probability of existential catastrophe from some event; P 1 is the probability of the initial catastrophe event; and P 2 is the probability that the event will result in a harm greater or equal to the collapse of civilization. For example, P 1 could represent the probability of nuclear war and P 2 could represent the probability that nuclear war would result in the collapse of civilization or worse. The occurrence of the initial catastrophe event does not necessarily entail the collapse of civilization-that depends on how effectively the survivors can cope with the aftermath of the event. Calculating P EC via Eq. (  1 ) requires two distinct analyses: one for each of P 1 and P 2 . Analysis of P 1 is the analysis of the probability of initial events, and can follow many conventions of probabilistic risk analysis. In contrast, quantifying P 2 requires analysis of the severity, with attention to the success of catastrophe survivors. This is a rather different sort of analysis than is needed to quantify the probability of initial catastrophe events represented by P 1 . However, P 2 is not equivalent to severity. P 2 is a probability variable representing the probability that the severity will exceed a certain threshold. P 2 can be obtained by creating a probability distribution for the severity of an initial event and then calculating the portion of that distribution that exceeds the threshold for existential catastrophe: ∫ = ∂ ∞ P P S S ( ) 2 ST (2) In Eq. (  2 ), P 2 is as in Eq. (  1 ); S is severity of some initial event; and S T is the minimum severity threshold of existential catastrophe Fig.  1 . A depiction of the relationship between P 2 and the severity of an initial event. The graphic is for illustrative purposes only; no evaluation of the actual relationship is intended. (the collapse of civilization in BRF). Eq. (  2 ) is illustrated in Fig.  1 . The studies surveyed in BRF vary in terms of which probabilities they quantify. (BRF acknowledges this; see BRF, Section 1) Out of the 66 studies in BRF, only three-  Chapman (2004) ;  Rees (2003), and Wells (2009) -explicitly quantify the probability of civilization collapse (as obtained from a keyword search of BRF, Appendix A). Of the other 63 studies, some quantify P 1 , while others quantify something similar to P EC but with a different severity threshold.  Chapman (2004)  is the only one of these studies that quantifies P EC via something resembling Eq. (1).  Chapman (2004)  is a study of asteroid impacts. As with many asteroid risk analyses,  Chapman (2004)  quantifies impact probability as a function of asteroid diameter (see Fig.  1  of  Chapman, 2004) . This provides P 1 .  Chapman (2004)  then proposes that asteroids of 2−3 km diameter may qualify as "civilization destroyers", while noting that the "predictions of consequences are fraught with uncertainty" (p.11; see also  Baum, 2018) . This roughly amounts to postulating that P 2 ≈ 0 for asteroids smaller than 2−3 km diameter and P 2 ≈ 1 for asteroids 2−3 km diameter or larger. However, there is no reason to expect such an extreme discontinuity in P 2 at 2−3 km. A more careful analysis would likely display a distribution of severities (as in Fig.  1 ) across a range of asteroid sizes, finding a continuum of P 2 values that gradually increase as a function of asteroid size. Until such analysis is conducted, one cannot rigorously quantify P EC for asteroid impact. Given that impact probability varies widely as a function of diameter (smaller asteroids have a much higher impact probability), P EC for asteroid impact will be highly sensitive to the resilience of civilization to impact events of different sizes. Similar reasoning applies to other existential risks. The study of  Wells (2009) , p.84-90) is of note because it quantifies P EC directly, without considering P 1 and P 2 . It can do this because is quantifies the overall P EC , not P EC for specific events. The analysis uses a technique that quantifies the probability of something persisting based on how long it has already existed for. Given the current age of civilization, P EC can be calculated.  Wells (2009)  also uses this technique to calculate the probability of human extinction, as do several other studies including  Gott (1993) . Because this technique does not consider any particulars of the existential risks that civilization currently faces, it provides at most a very limited amount of guidance to how the risks can be addressed. The study of  Rees (2003, p.8 ) is more ambiguous. Rees provides a subjective estimate of P EC , stating "I think the odds are no better than fifty-fifty that our present civilization on Earth will survive to the end of the present century". The text does not explain how the estimate was made. The broader study presented by  Rees (2003)  includes reasoning similar to  Wells (2009)  and analysis of catastrophe scenarios similar to  Chapman (2004) . Perhaps the subjective estimate was made in consideration both approaches, but this cannot be discerned from the text. The studies that quantify P 1 are not studies of the probability of existential catastrophe: they are studies of the probability of events that might or might not result in existential catastrophe. For example,  Lipsitch and Inglesby (2014)  quantify the probability that a highly pathogenic influenza strain would escape a research laboratory and cause a pandemic.  Lipsitch and Inglesby (2014)  also quantify the severity of the pandemic, estimating 2 million to 1.4 billion fatalities. To calculate P 2 and in turn P EC , this range of severities would have to be evaluated in terms of the probability of exceeding S T .  Hellman (2008)  quantifies the probability of a "fullscale nuclear war" started from a crisis similar to the Cuban missile crisis.  Hellman (2008)  does not quantify the severity and would need to do so to obtain P 2 and P EC . For the studies that quantify something analogous to P EC , analysis of additional scenarios is needed to obtain P EC . For example,  Gott (1993)  quantifies the probability of human extinction; additional analysis would quantify the probability of civilization collapsing without humanity going extinct.  Pamlin and Armstrong (2015)  quantify the probability of the subset of civilization collapse scenarios in which there is no subsequent recovery of civilization; additional analysis would quantify the probability of scenarios in which civilization collapses and recovers. Here it is worth recalling the multitude of definitions of existential catastrophe. The studies of  Gott (1993)  and  Pamlin and Armstrong (2015)  could classify as quantifying P EC under non-BRF definitions.
 Fast vs. slow catastrophes Existential risks can be classified in terms of how quickly the catastrophe unfolds. Specifically, it is the initial catastrophe event that unfolds at different speeds. The initial catastrophe event is the phenomenon that causes the harm, such as a nuclear war or Earthasteroid collision; it is represented in P 1 in Eq. (  1 ). The overall catastrophe scenario also includes the aftermath of the initial event. All existential catastrophes could have long-lasting effects, especially if civilization is not quickly rebuilt after it collapses. Nonetheless, the speed of initial events can vary significantly between different existential risks. The risks included in BRF group into either fast or slow risks. Fast risks have initial events that occur within years or less. Fast risks included in BRF are nuclear war, pandemics, runaway artificial intelligence, asteroid impacts, nanotechnology, particle physics experiments, space weather, super-volcanic eruptions, and synthetic biology. Slow risks have initial events that persist for decades or longer. Slow risks included in BRF are climate change and ecological catastrophe (see BRF, Appendix A; note that the fast vs. slow distinction is my own and does not appear in BRF). This is overall a reasonable list of risks and broadly consistent with other studies of existential risk. The list is not without exceptions-for example, flood basalt volcanic eruptions consist of a series of large eruptions over a period of hundreds of thousands of years; if the eruptions are sufficiently large and frequent, they could cause more prolonged harm  (Schmidt et al., 2016)  and therefore rate as a slow catastrophe. As another example, studies of artificial intelligence sometimes consider that the event could unfold in a "slow takeoff" lasting on the order of "decades or centuries"  (Bostrom, 2014, p.63) . Regardless, for probability analysis, the distinction between fast and slow is important in several respects. (The distinction can also be important for other reasons beyond the scope of this paper, such as policy decision-making.) First, event probability P 1 can have a different meaning for slow risks. The probability of climate change is approximately 1 (i.e., it is virtually certain that climate change is occurring). The only meaningful uncertainty about climate change risk is about how severe the impacts will end up being, including whether they would amount to existential catastrophe. The same probably applies to the forms of ecological degradation that could result in ecological catastrophe, though the reference cited in BRF on ecological catastrophe  (Pamlin & Armstrong, 2015)  is unclear on this point. Second, slow catastrophes may be especially important when they interact with fast catastrophes. Climate change and ecological degradation may be unlikely to destroy civilization on their own. Instead, they could weaken civilization, making it less resilient to fast catastrophes. All else equal, climate change and ecological degradation may make it more likely that civilization survives a fast catastrophe (such as a nuclear war) if the fast catastrophe occurs in 2020 instead of 2050. (All else is not equal-other factors include, but are not limited to, economic and technological change.) In other words, climate change and ecological degradation increases P 2 for nuclear war (and other fast risks). They could also affect P 1 , such as by destabilizing international politics, or alternatively by providing a rallying point that brings rival nations closer together. Additionally, fast catastrophes could also affect slow catastrophes. For example, the 2011 Fukushima Daiichi nuclear disaster, caused by the Tōhoku earthquake and tsunami, has prompted a shift from nuclear power to fossil fuels, thereby worsening climate change  (Srinivasan & Rethinaraj, 2013) . While the Tōhoku-Fukushima event was only a local catastrophe, the episode is nonetheless indicative of potential impacts of fast catastrophes on slow catastrophes. The above implies that the probability of slow catastrophe can be linked to the probability of fast catastrophe. This applies to most of the risks in BRF. One potential exception is particle physics experiments, for which P 1 may not be meaningfully affected by the slow catastrophes and for which P 2 may 1 under any circumstances (it may be impossible to survive a particle physics catastrophe). Artificial intelligence could be another exception following the same reasoning, though it is conceivable that AI development could be altered by the desire to develop AI to address the slow catastrophes  (Baum, 2014) , or by effects of slow catastrophes on the context in which AI is developed, such as related to international cooperation and competition. But for most cases, quantifying the probability of slow catastrophes should account for their effect on fast catastrophes. It has been argued that the field of existential risk is insufficiently attentive to slow catastrophes, perhaps because slow catastrophes are too "boring" or "unsexy"  (Kuhlemann, 2019; Liu, Lauta, & Maas, 2018) . My own view is that yes, slow catastrophes have been at least somewhat neglected in studies of existential risk. But it is also the case that slow catastrophes pose distinct analytical challenges, as outlined above. Furthermore, the fields dedicated to studying slow catastrophes tend to not study them in terms resembling existential risk. For example, high-profile research on planetary boundaries identifies approximately ten major global environmental threats that are nominally supposed to be unacceptable to humanity, but are defined in the research strictly in biogeophysical terms and not in terms of impacts to humanity  (Baum & Handoh, 2014) . Likewise, almost all of the studies of slow catastrophes studied in BRF (Section 1) focus on biogeophysical metrics, in particular the magnitude of average global temperature increase from climate change. The one exception is  Pamlin and Armstrong (2015) , which is also probably the only of these studies that classifies within the field of existential risk. (BRF defines the existential risk research community as "those who are consciously seeking to align their research with the goal of understanding and managing such risks".) There is a major need for research that blends the traditions of global environmental change and existential risk to analyze, in human terms, the risks of slow catastrophes.
 Multi-risk catastrophes The preceding discussion of slow vs. fast catastrophes raises a more general point. The literature surveyed in BRF consists exclusively of quantifications of specific risks or quantification of the overall probability of existential catastrophe or similar catastrophe. However, some catastrophe scenarios involve multiple risks. These scenarios can require a different set of methods than those surveyed in BRF. One type of scenario involves multiple initial catastrophe events occurring together by coincidence. For example, there could be a volcano eruption that happens to occur as a pandemic is breaking out. The initial events may be independent-for example, there is no reason to believe that infectious diseases affect plate tectonics and vice versa-in which case their probabilities (P 1 in Eq. (  1 )) could be modeled as independent random variables, with the aggregate initial event probability calculated accordingly. However, the probability of existential catastrophe given the co-occurrence of these events (P 2 in Eq. (  1 )) is unlikely to be independent. For example, a volcano eruption can prompt mass evacuations that facilitate disease transmission. Analysis of the severity of co-occurring initial catastrophe events must be done on a case-by-case basis; ditto for the accompanying probability of of existential catastrophe. Another type of scenario involves causal relations between multiple fast catastrophes. For example, there could be a war or pandemic that causes a geoengineering termination shock  (Baum, Maher, & Haqq-Misra, 2013; Parker & Irvine, 2018) , an asteroid collision that causes a nuclear war  (Baum, 2018; Tagliaferri, Spalding, Jacobs, Worden, & Erlich, 1994) , or a war that induces risky technology development (as previously occurred in World War II with the development of nuclear weapons). These scenarios cannot be modeled as independent random variables. They can be modeled using fault trees or Bayesian networks, albeit with more complex models that account for the nuances of each catastrophe within the causal chain. In principle, they could be studied via aggregated opinion surveys, weighted aggregation, or enhanced solicitation, all of which derive from expert judgment. In practice, however, there may be a dearth of qualified experts, because experts tend to focus on one risk at a time and not on the causal relations between them. (The methods mentioned in this paragraph are described in Section 3) Finally, there are scenarios in which a fast catastrophe occurs during a slow catastrophe, as discussed in Section 2.3. Analysis of these scenarios could proceed by evaluating "baseline" values of P 1 and P 2 for the fast catastrophe, and then evaluating the effects of the slow catastrophe on the baseline P 1 and P 2 . Alternatively, it could proceed by calculating scenario probabilities directly without first considering baselines. Regardless, the analysis requires a nuanced understanding of the impacts of the slow catastrophe as they relate to the onset and impacts of the fast catastrophe. (Other slow factors, such as economic and technological change, can be factored into the analysis alongside slow catastrophes.) 3. On the methods surveyed by BRF BRF survey ten methods used for quantifying existential catastrophe: 1 Analytical approachesestimating probability from theory without incorporating specific evidence on the risks, such as in the "doomsday argument". 2 Extrapolation and toy modelingsimple models that derive probabilities from data on other events on the assumption of an inherent relationship between the probability of the other events and existential catastrophe (noting that no existential catastrophe has previously occurred). 3 Fault treeslogic trees that decompose types of catastrophe events (e.g., nuclear war) into a collection of sequences of precursor events and conditions that can lead to the catastrophe event. 4 Bayesian networkssimilar to fault trees, except also permitting interactions between precursor events (hence a "network" instead of a "tree"). 5 Adapting large-scale modelsthe models depict details of specific risks (e.g., models of the global climate system to study climate change risk); these models are adapted to study extreme cases of these risks. 6 Individual subjective opinionsubjective judgments of a single person. 7 Aggregated opinion surveyssubjective judgments of multiple people, averaged into a single probability estimate. 8 Weighted aggregationsubjective judgments of multiple people, combined into a single probability estimate using a weighted average, with weights set according to some measure of the quality of the judgment expected from each person. 9 Enhanced solicitationsubjective judgments of multiple people, in which effort is made to improve judgment quality via some combination of training people to provide better opinions and being more careful in how their judgments are obtained. 10 Prediction marketsplatforms for people to place financial bets on their best-guess parameter estimates.
 The tradeoff between quality and accessibility BRF evaluate probability quantification methods according to four criteria: (1) rigor, defined as how effectively the method makes use of available information; (2) uncertainty, defined as how effectively the method handles the considerable uncertainty about the probability of existential catastrophe; (3) accessibility, defined as how readily a research group can apply the method, especially the small interdisciplinary groups that often study existential risk; and (4) utility, defined as the value of the method's results for decisionmaking purposes. BRF presents ratings of ten methods according to each of the four criteria. (The ratings were performed subjectively by the authors. In my own view, their ratings are reasonable.) Ideally, a method would rate well across all four criteria. Unfortunately, none of the methods surveyed by BRF achieve this. The BRF ratings data show a trend in which rigor, uncertainty, and utility are similar to each other and dissimilar to accessibility. Fig.  2 . Rigor vs. accessibility of each method as rated by BRF (data from BRF, Table  1 ). This trend holds across all methods (see BRF, Table  1 ). To illustrate this trend, Fig.  2  plots rigor vs. accessibility. No method is rated high in both, and only weighted aggregation is rated low in both. (More precisely, weighted aggregation is rated medium-low in rigor.) The trend in the data is entirely understandable. Rigor, uncertainty, and utility are all aspects of the quality of the probability quantification produced by the application of a particular method. Obtaining higher-quality results generally requires incorporating more detail about the risk and more nuance in how the detail is processed, both of which tend to make the method less accessible. There can be methods that are low-quality and inaccessible, though one would hope that these have gotten weeded out of the collective risk analysis toolkit. Conversely, however, it may not be possible for a method to be both high-quality and accessible. Quality analysis may inevitably require a substantial investment. The same point applies to applications of a given method. Any method can be applied in ways that are of higher or lower quality. There can be applications of a given method that are low-quality and inaccessible: a lot of resources spent implementing a method poorly. Higher quality implementations of a given method will tend to be less accessible, requiring more effort on the part of the research team, and in some cases more funding to invest in expenses such as travel to interview experts in person (instead of via phone or other mode of remote communication) or rewards to incentivize participants in prediction markets.
 On the role of existential risk quantification
 Why quantify probability (and severity)? Underlying the tradeoff between quality and accessibility is the basic point that the probabilities of potential existential catastrophe scenarios are difficult to rigorously quantify. The scenarios involve complex and unprecedented global processes. Historical data are unreliable or nonexistent. Expert judgment is often unreliable as well-or, perhaps there are no experts. Obtaining quality quantifications requires applying difficult methods, which can be a resource-intensive process in a field in which resources are scarce. Perhaps it would be better to skip the matter entirely and focus on other activities. BRF identify two major reasons to quantify the probabilities and severities of existential risks. The first is the prioritization of efforts to reduce various risks. The second is the evaluation of actions that could decrease one set of risks but increase another (i.e., risk-risk tradeoffs). Both are situations in which actions could affect multiple risks, and in which identifying the best action requires comparing the size of the effect on one set of risks to the size of the effect on another. These sizes are typically measured in terms of the product of probability and severity. Therefore, to identify the best actions in these situations, it is necessary to know the effects of the actions on the probabilities and severities of the risks in question. Quantifying the probabilities and severities is an important first step. (Additional analysis is required to assess the effects of specific actions aimed at reducing the risks, as well as the costs associated with pursuing these actions.) Quantification is not always necessary. Some actions would only affect one risk, or would affect multiple risks in the same direction (increase or decrease). These actions pose no tradeoff and require no quantification to evaluate. Where possible, it can be more efficient to focus on these actions. Indeed, one approach to handling difficult quantification tradeoffs is to search for actions that avoid the tradeoffs.  Graham and Wiener (1995)  refer to these as "risk-superior" options, analogous to the concept of Pareto superiority in welfare economics. An "existential-risk-superior" option would be one that reduces all existential risks by at least as much as all other options, and reduces at least one existential risk by more than each other option, such that there is reason to favor the existential-risk-superior option and no reason to favor any other option, setting aside reasons unrelated to existential risk. If existential-risk-superior options are identified, there is no further need to quantify the options, because one option is clearly best. However, even then, some quantification may be needed to identify existential-risk-superior options. And in many other cases, there are no existential-risk-superior options. Tradeoffs are inevitable. Quantification is an important research task, however difficult it may be.
 Analysis vs. decision-making Some further perspective can be obtained from considering the relationship between analysis and decision-making. Arguably, analysis of the probability and severity of existential catastrophe should aim to improve the quality of decisions that affect existential risk. Otherwise, the analysis is of mere intellectual interest. Reducing existential risk is (again, arguably) a moral imperative, not an intellectual curiosity. Analysis can improve decisions by enabling the decisions to result in larger reductions in the risk. Phrased differently, the expected value of the action taken is larger with the analysis than without. This is the essence of the concept of the value of information (e.g.,  Barrett, 2017) . Analysis can be valuable even if the action taken is unchanged. As a toy example, perhaps heads would have been chosen in a coin flip anyway, but it would have been a lucky guess, and so there is value in analysis indicating that the coin was indeed going to land on heads. The same reasoning holds for analysis about more complex decisions, including those about existential risk. But decisions do not occur in a theoretical vacuum, and they do not necessarily conform to the ideals of expected value maximization. Decisions are psychological and (often, especially for the most important decisions) social processes. In order to be most useful for reducing the risks, methodology for quantifying probabilities and severities should be designed with the nuances of actual decision processes in mind. This point is captured somewhat by the BRF concept of method utility, but some further elaboration is warranted. A lot of risk analysis is commissioned by clients who seek to use the analysis in their own decision-making. The field of risk analysis has roots in industry, especially nuclear power  (Thompson, Deisler, & Schwing, 2005; Wilson, 2012) . Engineering risk analysis commonly seeks to guide in-house safety decisions of industrial risk managers or regulatory decisions by policy makers at agencies like the US Environmental Protection Agency and Nuclear Regulatory Commission. For this, quantifying the overall probability of some catastrophe can be less important than identifying the specific system components that are most implicated in the risk and likewise in greatest need of attention. This is one reason that engineering risk analysis has gravitated toward methods like fault trees that decompose risks into constituent parts and therefore produce results that help managers focus on specific problems. Existential risk research is less often commissioned by clients. Instead, it is often sponsored philanthropically, whether by private funders or government agencies. The funders do not seek results that they can use for their own decisions. Instead, they wish to support the cause of existential risk reduction and hope that the analysis will in some way be useful to that end. Without decisionmaker clients, existential risk analysis projects must take additional steps to ensure that their analysis improves decision-making, including identifying decision-makers to support. Because of this, there is a greater risk of the analysis working in an academic fashion, disconnected from practical decision-making and not having constructive impact on the risks themselves. (Existential risk is not the only domain in which research lacks decision-maker clients. Another example is research by industry and environmental public interest groups that seek to inform environmental policy; von  Winterfeldt, Kavet, Peck, Mohan, & Hazen, 2012.)  Nonetheless, even without built-in decision-maker clients, it is still important for existential risk analysts to bear in mind how their analysis could be used for decision-making. That means recognizing that there is more to risk management than just numbers for probability and severity. The details of the risks themselves must be understood, and how potential actions can affect the risks, and the psychological and social and institutional contexts in which decisions will be made. This is a lot to account for-as if the probabilities and severities were not a lot on their own-but it is essential for pursuing actual reductions in existential risk and not just academic studies. It can help to recognize that risk analysis can be valuable even if it does not produce any quantification of probability and severity. Often, simply having an analysis that pulls together information about the risk in an intuitive and well-organized fashion is sufficient for improving risk management. In other words, risk analysis can be of value by improving people's mental models of risks, in addition to providing information on the rating or ranking of decision options. Additionally, risk analysis can be of value by creating an opportunity for stakeholders and experts to share risk-related information with each other in a way that crosses typical organizational barriers. The opportunity for people in different groups to talk through key issues affecting risk can be of significant valuable for the practice of risk management, even regardless of formal risk analysis outputs. It is important to not let the pursuit of numbers (of probability, severity, the timing of future events such as AI milestones, etc.) serve as a distraction from the goals of evaluating decision options and improving our understanding and management of the risks. In sum, quantifying probability and severity should be at most only one part of an overall existential risk analysis and management portfolio. But it is still a part, and an important one at that. It is likewise important to use quality methodology in performing the quantifications.
 Conclusion This article has sought to build on the excellent contribution of BRF to provide further perspective on the quantification of the probability of existential catastrophe. This article finds that the probability of existential catastrophe is inherently linked to the severity of events that could result in existential catastrophe, that achieving a higher quality of analysis of the probability will in general require a larger investment in analysis, that analysis of the probability is sometimes but not always necessary for decisionmaking, and that analysis should be designed to support risk management and not just designed as an academic exercise. If these findings are taken into account, together with the analyses of specific methodologies by BRF, then it is believed that risk analyses of existential catastrophe will tend to be more successful at understanding and reducing the risks.

Abstract: A recent article by Beard, Rowe, and Fox (BRF) evaluates ten methodologies for quantifying the probability of existential catastrophe. This article builds on BRF's valuable contribution. First, this article describes the conceptual and mathematical relationship between the probability of existential catastrophe and the severity of events that could result in existential catastrophe. It discusses complications in this relationship arising from catastrophes occurring at different speeds and from multiple concurrent catastrophes. Second, this article revisits the ten BRF methodologies, finding an inverse relationship between a methodology's ease of use and the quality of results it produces-in other words, achieving a higher quality of analysis will in general require a larger investment in analysis. Third, the manuscript discusses the role of probability quantification in the management of existential risks, describing why the probability is only sometimes needed for decision-making and arguing that analyses should support realworld risk management decisions and not just be academic exercises. If the findings of this article are taken into account, together with BRF's evaluations of specific methodologies, then risk analyses of existential catastrophe may tend to be more successful at understanding and reducing the risks.