Nick Bostrom
Thomas Douglas
Anders Sandberg
James Martin

The Unilateralist's Curse and the Case for a Principle of Conformity

Introduction Consider the following hypothetical scenarios: (1) A group of scientists working on the development of an HIV vaccine has accidentally created an air-transmissible variant of HIV. The scientists must decide whether to publish their discovery, knowing that it might be used to create a devastating biological weapon, but also that it could help those who hope to develop defenses against such weapons. Most members of the group think publication is too risky, but one disagrees. He mentions the discovery at a conference, and soon the details are widely known. (2) A sports team is planning a surprise birthday party for its coach. One of the players decides that it would be more fun to tell the coach in advance about the planned event. Although the other players think it would be better to keep it a surprise, the unilateralist lets word slip about the preparations underway. (3) Geoengineering techniques have developed to the point that it is possible for any of the world's twenty most technologically advanced nations to substantially reduce the earth's average temperature by emitting sulfate aerosols. Each of these nations separately considers whether to release such aerosols. Nineteen decide against, but one nation estimates that the benefits of lowering temperature would exceed the costs. It presses ahead with its sulfate aerosol program and the global average temperature drops by almost 1˚. In each of these cases, each of a number of agents is in a position to undertake an initiative, X. Suppose that each agent decides whether or not to undertake X on the basis of her own independent judgment of the value of X, where the value of X is assumed to be independent of who undertakes X, and is supposed to be determined by the contribution of X to the common good. 1 Each agent's judgment is subject to error-some agents might overestimate the value of X, others might underestimate it. If the true value of X is negative, then the larger the number of agents, the greater the chances that at least one agent will overestimate X sufficiently to make the value of X seem positive. Thus, if agents act unilaterally, the initiative is too likely to be undertaken, and if such scenarios repeat, an excessively large number of initiatives are likely to be undertaken. We shall call this phenomenon the unilateralist's curse. Though we have chosen to introduce the unilateralist's curse with hypothetical examples, it is not merely a hypothetical problem. There are numerous historical examples, ranging from the mundane to the high-tech. Here is one: Until the late 1970s, the mechanism of the hydrogen bomb was one of the world's best kept scientific secrets: it is thought that only four governments were in possession of it, each having decided not to divulge it. But staff at the Progressive magazine believed that nuclear secrecy was fuelling the Cold War by enabling nuclear policy to be determined by a security elite without proper public scrutiny. They pieced together the mechanism of the bomb and published it in their magazine, arguing that the cost, in the form of aiding countries such as India, Pakistan and South Africa in acquiring hydrogen bombs, was outweighed by the benefits of undermining nuclear secrecy.  2  Another possible example from atomic physics had occurred several decades earlier: In 1939 the Polish nuclear physicist Joseph Rotblat noticed that the fission of uranium released more neutrons than used to trigger it, realizing that it could produce a chain reaction leading to an explosion of unprecedented power. He assumed that other scientists elsewhere were doing similar experiments, and were thus in a position to release similar information, an assumption that turned out to be correct. Initially, Rotblat vowed to tell no-one of his discovery, believing it to be a threat to mankind, and it is plausible that others did likewise, for similar reasons. However, when the war broke out, Rotblat decided that releasing the information was now in the public interest, given the likelihood that the Germans were working on an atomic bomb. He confided in colleagues and thus unilaterally triggered the United Kingdom's atomic bomb project.  3  Rotblat was later to leave the Manhattan Project, coming to the view that his had overestimated the German nuclear threat, and underestimated the likelihood that the US would use an atomic bomb offensively. It is perhaps too soon to say whether these unilateral actions were suboptimal. But in other cases, it is clearer that unilateral action led to a suboptimal outcome: In the mid-nineteenth century there were virtually no wild rabbits in Australia, though many were in a position to introduce them. In 1859, Thomas Austin, a wealthy grazier, took it upon himself to do so. He had a dozen or two European rabbits imported from England and is reported to have said that "The introduction of a few rabbits could do little harm and might provide a touch of home, in addition to a spot of hunting." 4 However, the rabbit population grew dramatically, and rabbits quickly became Australia's most reviled pests, destroying large swathes of agricultural land.  5  The abovementioned examples were isolated incidents, but similar situations occur regularly in some spheres of activity, for instance, in the media: Media outlets sometimes find themselves in the situation that journalists have access to information that is of public interest but could also harm specific individuals or institutions: the name of a not-yet charged murder suspect (publication may bias legal proceedings), the news that a celebrity committed suicide (publication may risk copycat suicides), or sensitive government documents such as those leaked by Wikileaks and Edward Snowden (publication may endanger national security). It is enough that one outlet decides that the public interest outweighs the risk for the information to be released. Thus, the more journalists have access to the information the more likely it is to be published. Unilateralist situations also regularly crop up in regards to new biotechnologies: Gene drives, a technique for inducing altered genes to be inherited by nearly all offspring (rather than just 50%) of a genetically modified organism, have potential for spreading altered genes across a population, enabling ecological control (e.g. making mosquitos incapable of spreading malaria or reducing herbicide resistance) but also potentially creating worrisome risks (e.g. to genetic diversity or of sabotage). Here unilateral action could both be taken in releasing a particular altered organism into the environment, and in releasing the information about how to produce it in the first place. There is scientific disagreement on the utility and risk of both. 6
 The Unilateralist's Curse: A Model The unilateralist's curse is closely related to a problem in auction theory known as the winner's curse. The winner's curse is the phenomenon that the winning bid in an auction has a high likelihood of being higher than the actual value of the good sold.  7  Each bidder makes an independent estimate and the bidder with the highest estimate outbids the others. But if the average estimate is likely to be an accurate estimate of the value, then the winner overpays. The larger the number of bidders, the more likely it is that at least one of them has overestimated the value. The unilateralist's curse and the winner's curse have the same basic structure. The difference between them lies in the goals of the agents and the nature of the decision. In the winner's curse, each agent aims to make a purchase if and only if doing so will be valuable for her. In the unilateralist's curse, the decision-maker chooses whether to undertake an initiative with an eye to the common good, that is, seeking to undertake the initiative if and only if the initiative contributes positively to the common good. The unilateralist's curse can be illustrated using a simple mathematical model. Assume N agents, each considering whether to undertake an initiative. Each agent wishes to proceed if and only if the value of the initiative is positive, but the agents do not know the true value V * of the initiative (which may be negative or positive). Instead each agent forms an estimate that is the sum of V * and a random independent error d drawn from a distribution with cumulative distribution function F(d). This means that the probability p that any given agent will estimate the value of the initiative to be positive when it is in fact negative (V * < 0) is p = 1 − F(−V * ).  8  The probability P that at least one of the agents will incorrectly estimate the value to be positive is p = 1 − (1 − p) N = 1 − F(−V * ) N . For the case with 5 agents and d as a random error drawn from a normal distribution with standard deviation 1 and mean zero, the probability that any initiative will be undertaken (regardless of whether it is a good idea or not) is high even when the true value is quite negative, and the probability rises steeply as the true value of the initiative approaches zero from below (Figure  1 ). For mildly negative values of the initiative there is nearly always someone who misjudges the value of the initiative and undertakes it. There is no problem for positive initiatives since even if one or two agents are overly cautious, it is very likely that somebody will undertake the initiative, which is the optimal result (Figure  2 ). Increasing the number of agents capable of undertaking the initiative also exacerbates the problem: as N grows, the likelihood of someone proceeding incorrectly increases monotonically towards 1.  9  The magnitude of this effect can be quite large even for a relatively small number of agents. For example, with the same error assumptions as above, if the true value of the initiative V * = −1 (the initiative is undesirable), then the probability of erroneously undertaking the initiative grows rapidly with N, passing 50% for just four agents (Figure  3 ). There are six features of the unilateralist's curse that that need to be emphasized. First, in cases where the curse arises, the risk of erroneously undertaking an initiative is not caused by self-interest. In the model, all agents act for the common Figure  1  The probability of an initiative being undertaken as a function of the actual value, V * , for five agents and assuming normally distributed errors with variance 1 (these assumptions will be used in all subsequent figures except when otherwise noted). Note that 50% probability of action occurs near a value of −1: a strong unilateralist bias exists. Figure  2  The expected payoff for naive agents (who act if and only if their evaluation of the initiative is positive) and ideal omniscient estimators who are assumed to know the true value. good, they simply disagree about the contribution of the initiative to the common good.  10  Second, though the curse could be described as a group-level bias in favor of undertaking initiatives, in does not arise from biases in the individual estimates of the value that would result from undertaking the initiative. The model above assumes symmetric random errors in the estimates of the true value.  11  Third, there is a sense in which the unilateralist's curse is the obverse of Condorcet's jury theorem.  12  The jury theorem states that the average estimate of a group of people with above 50% likelihood of guessing correctly and with uncorrelated errors will tend to be close to the correct value, and will tend to move closer to the true value as the size of the group increases. But what is also true, and relevant to the argument in this paper, is that the highest estimate will tend to be above the true value, and the expected overestimation of this highest estimate increases with the size of the group. In the cases we are interested in here, it is the highest estimate that will determine whether an initiative is undertaken, not the average estimate. Fourth, though we have chosen to illustrate the curse using initiatives that are (probably) irreversible, the problem can arise in other cases too. The problem becomes sharper if the initiative is irreversible, but even for actions that can be undone the problem remains in a milder form. Resources will be wasted on undoing erroneous initiatives, and if the bad consequences are not obvious they might occur before the problem is noticed. There might even be a costly tug-o-war between disagreeing agents. Finally, fifth, though we have thus far focused on cases where a number of agents can undertake an initiative and it matters only whether at least one of them does so, a similar problem arises when any one of a group of agents can spoil an initiative-for instance, where universal action is required to bring about an intended outcome. Consider the following example: In Norse mythology, the goddess Hel of the underworld promised to release the universally beloved god Baldr if all objects, alive and dead, would shed a tear for him. All did, except the giantess Þo ¨kk. The god was forced to remain in the underworld.  13  Similar situations can arise when all the actors in a play must come together in order for a rehearsal to take place, when all members of committee must attend a meeting in order for it to be quorate, or when all signatories to an international treaty must ratify it in order for it to come into effect. The United Nations Security Council frequently provides examples of unilateral spoiling. The five permanent members of the Council-currently China, France, Russia, the United Kingdom and the United States-each possesses the power to veto the adoption of any non-procedural resolution. In the early years of the Council, this veto power was frequently employed by the Soviet Union to block applications for new membership of the United Nations. More recently, it has been used by the United States to block resolutions criticizing Israel, and by Russia and China to block resolutions on the Syria conflict. 14 While some of these vetoes presumably reflect differences in the national interests of the council members, others may reflect different estimations of the contribution that a resolution would make to the common good. Certainly, considerations relating to the common good are often invoked in their defence. For instance, the United States' 2011 veto of a draft resolution condemning Israeli settlements in Palestinian territory was defended on the grounds that the resolution would be an impediment to peace talks.  15  These cases of unilateral spoiling or abstinence are formally equivalent to the original unilateralist curse, with merely the sign reversed. Since the problem in these cases is the result of unilateral abstinence, it seems appropriate to include them within the scope of the unilateralist's curse. Thus, in what follows, we assume that the unilateralist's curse can arise when each member of a group can unilaterally undertake or spoil an initiative (though for ease of exposition we sometimes mention only the former case).
 Lifting the Curse Let a unilateralist situation be one in which each member of a group of agents can undertake or spoil an initiative regardless of the cooperation or opposition of other members of the group. We will say that a policy would lift the unilateralist's curse if universal adherence to it by all agents in unilateralist situations should be expected (ex ante) to eliminate any surfeit or deficit of initiatives that the unilateralist's curse might otherwise produce.
 The Principle of Conformity When acting out of concern for the common good in a unilateralist situation, reduce your likelihood of unilaterally undertaking or spoiling the initiative to a level that ex ante would be expected to lift the curse. In the following subsections we will explore various ways in which one might bring oneself into compliance with this principle.  16  These can be organized around three models: collective deliberation, epistemic deference, and moral deference. The three models are applicable in somewhat different circumstances, and their suitability might depend on the type of agents involved. It should be noted that, though some of the methods discussed below do not require agents to be aware of the nature of the situation, most hinge on agents recognizing that they are in an unilateralist situation. However, this is not to say that agents must be able to identify the other parties to the unilateralist situation: this is necessary for some but not all of our proposed solutions.
 The Collective Deliberation Model A first line of defense against the unilateralist's curse could be to share data and reasoning between agents in the hope that this will resolve their disagreement about the desirability of proceeding with the contested initiative. In some cases, however, extensive information sharing among all potential decision-making agents is impractical. Communication is often costly and timeconsuming, and participants in a unilateralist situation may not be able to identify one another. Furthermore, in certain cases information disclosure might itself be the initiative whose desirability is in dispute, such as when information hazards are associated with disseminating relevant data.  17  Moreover, even when information is fully shared, a consensus can remain elusive. Disagreements about the net value of undertaking some project often persist after decision-makers have been thoroughly briefed on all obviously relevant and easily communicable facts and after having had opportunities to engage in joint deliberation. Because complete information sharing may not be practical or desirable, and because it may not produce consensus when it does occur, the principle of conformity requires us to explore additional models for lifting the unilateralist's curse.
 The Meta-rationality Model One approach would be to appeal to each agent's reflective rationality. A party to an epistemic disagreement should ideally reflect on the fallibility of their own judgment and adjust their posterior probability to take into account the fact that other agents have different opinions. Robert Aumann has shown that rational Bayesian agents with identical priors and common knowledge of each other's posteriors (and of each other's rationality) must have identical posterior probabilities.  18  Disagreement between such agents is impossible. This sounds like good news: if all agents make the same estimate of the benefits of action, the unilateralist curse is lifted. There is, however, some skepticism about the relevance of Aumann's result for practical cases of disagreement.  19  The assumption of identical priors, in particular, is problematic.  20  Furthermore, the same challenges that can make data sharing difficult can also make it difficult to make each agent's honest posterior probability estimates of the value of the initiative common knowledge among all agents. It turns out, however, that sufficiently rational agents can manage the curse even without communication. In the literature on the winner's curse it has been argued that rational expected utility-maximizing will not be affected by it.  21  Rational agents will take the winner's curse into account and adjust their bids accordingly. This is known as bid shading. Rational agents place bids that are lower than their ex ante expectation of the value of the good, but equal to their expectation of the value of the good conditional upon them winning the auction. The counterpart of this response would be for agents in a unilateralist situation to estimate the value of the initiative conditional on the agent's first-order estimate of the initiative's value being the highest (or, in spoiler cases, the lowest). In other words, on finding themselves in a unilateralist situation, each rational agent will initially estimate the value of the initiative based on his prior probability distribution. He will then take into account the case where his decision is decisive. In the case where agents can unilaterally undertake an initiative, the agent will condition on the situation in which he is the most sanguine and everybody else thinks the action should not be done. (In spoiler cases, the agent conditions on the situation in which he is the most pessimistic and everybody else thinks the initiative should be undertaken.) He then creates a posterior distribution of value that is used to make an adjusted decision. PðV Ã jwinÞ ¼ PðwinjV Ã ÞPðV Ã Þ PðwinÞ where "win" represents being the deciding agent. Note that this typically requires knowing or estimating the number of other agents.
 Example In the simple case where the agent assumes all other agents have the same priors and are acting independently, only differing in the noisy data about V * they have received: PðwinjV Ã Þ ¼ Z 1 À1 PðV À V Ã ÞFðV À V Ã Þ NÀ1 dV where F(V) is the cumulative distribution function of the errors. The posterior distribution of V * becomes: PðV Ã jwinÞ ¼ KPðV Ã Þ Z 1 À1 PðV À V Ã ÞFðV À V Ã Þ NÀ1 dV where K is a normalization constant. The posterior action should then be based on the expectation E(V * |win). If the agents choose to act when the received data is above a fixed threshold T, V * is normally distributed with zero mean and variance 1, and they get estimates of V * with normal noise (again with mean zero and variance 1), then the optimal threshold is the one that maximizes the expected value (Figure  4 ): T opt ðNÞ ¼ argmax T Z 1 À1 VPðV Þ 1 À 1 À F V À T ð Þ ð Þ N dV T opt (N) increases rapidly with N, reaching 0.54 for two agents and 1 for 4 agents: even for a small group it is rational to be far more cautious than in the single agent case. Note that in this case all agents are aware of the prior distribution, noise distribution, independence, and that the other agents are using this strategy (Figure  5 ).  22  Figure  4  The optimal threshold T opt (N) for action as a function of the number of agents. Agents who only act if the perceived value of the initiative is higher than T opt (N) will maximize their expected (joint) result. One might raise questions about the practical applicability of this sophisticated Bayesian approach, however. Even if rational Bayesian agents would agree, humans are at best approximations of rational Bayesian agents and they have far more limited mental computation power-even when leaving out biasing factors.  23  Value in practical cases is also seldom in the form of easily manipulable and comparable scalar quantities. Hence implementing the sophisticated Bayesian approach to lifting the unilateralist's curse might typically be infeasible.  24
 The Moral Deference Model Suppose a unilateralist situation exists and that it is not feasible for all agents to lift the curse through communication and adjustment of beliefs. It might nevertheless be possible for the group to lift the curse if each agent complies with a moral norm which reduces the likelihood that he acts unilaterally, for example, by assigning decision-making authority to the group as a whole or to one individual within it. We call this the moral deference model. In contrast to the two models presented above, the moral deference model does not require agents to defer to the group in forming their beliefs regarding the value of the initiative. However, it does require them to defer to the group in deciding whether to act on those beliefs. A slogan for this approach could be "comply in action, defy in thought." There are many norms such that universal compliance with the norm by a group of agents would lift the unilateralist's curse. For example, a norm that assigned decision-making authority to an arbitrary member of the group would lift it. Consider the norm: when in a unilateralist situation, if you are the tallest person able to undertake the initiative, then undertake it if and only if you believe its value exceeds zero; if you are not the tallest person able to undertake the initiative, do not undertake it. Universal compliance with this norm would prevent the unilateralist's curse from arising in the sense that, in the absence of any bias towards or against action in the individual members of the group (and thus in the group's tallest member), this norm will produce no group-level bias towards or against the initiative.  25  The payoffs associated with this tallest-decides norm in a five-agent situation are depicted in Figure  6  below. The tallest-decides norm, however, has several epistemically and pragmatically unattractive features. For example, it does not protect against biases or errors that might impair the judgment of the group's tallest member. Furthermore, it is very unlikely that such a norm would gain wide acceptance. Fortunately, there are other norms that could lift the curse and may lack these unattractive features. One norm would recommend that agents conform to the rules of existing institutions that militate against unilateral action: (1) When in a unilateralist's situation, defer to existing institutions, such as laws or customs, if universal deference to those institutions would lift the unilateralist's curse. National and international laws often militate against the unilateralist's curse, for example by specifying that decisions must be made democratically or by individuals or institutions that have been given special authority over a particular realm of decision-making. In other cases, there are informal conventions that may do the job. For example, following the publication early last decade of two studies thought by some to aid bioweapons development,  26  a group of scientific journals agreed to introduce screening procedures to identify papers containing information that is especially prone to misuse and to seek external advice on the publication of such papers.  27  Though these procedures lacked legal status, compliance with them by journals may have helped lift the curse. One virtue of (1) is that, since it simply reinforces existing institutional norms which may already command significant support, it may be relatively easy for it to achieve wide acceptance. However, (1) will not lift the curse in all cases. In many areas with an international dimension, for example, there are no relevant international laws and deference to national laws would merely create a new unilateralist situation between nations: the nation that evaluates the initiative most positively is most likely to allow it. Moreover, (1) may sometimes recommend deferring to biased procedures or agents. It might be possible for a group of agents to lift the curse even in cases where (1) fails by complying with a different norm, one that promotes the development of and compliance with a new procedure for group decision-making. This approach was adopted by a large group of American microbiologists in mid-1974 when they agreed to a moratorium on recombinant DNA research until such time as the safety concerns that it raised could be jointly discussed and resolved. The moratorium held until the now-famous Asilomar Conference, which took place in February 1975 and resulted in a broad-based agreement on guidelines regarding the conditions under which recombinant DNA research ought to proceed.  28  The recombinant DNA moratorium and guidelines were developed via consensus, but another approach would be to employ a voting procedure. For example, suppose all agents faced with a unilateralist situation complied with the norm: (2) When in a unilateralist's situation, promote the holding of a majority vote among those capable of undertaking the initiative. If the vote takes place, then (a) defer to its verdict, and (b) encourage others to do likewise. Universal compliance with this norm is likely to lift the curse. Since it is effectively using the median estimate it is robust to outliers. It will also tend to reduce systematic bias at the group level provided that individual biases are at least partially independent of one another.  29  And since majority voting is a common and widely accepted method for group decision-making, this norm would have relatively good prospects of gaining wide acceptance. Compliance with norms (1) and (2) will, however, lift the unilateralist's curse only when a high degree of communication and coordination is possible. There are other norms whose universal adoption could lift the curse even in the absence of communication and coordination. Consider the norm: (3) When in a unilateralist situation, bring about the outcome if and only if you judge that a majority vote among those capable of undertaking the initiative would yield a majority in favor of doing so. Insofar as each individual capable of undertaking the initiative makes an accurate prediction of the views of all others, universal adoption of this norm will eliminate any group-level bias due to the unilateralist's curse. Even if predictions of the views of others are inaccurate (e.g. because each agent overestimates the extent to which others share her views), universal adoption of this principle can still be expected to somewhat mitigate the unilateralist's curse. It will tend to reduce the likelihood that those who value the initiative most favorably will undertake it, provided that these agents realize they are at the optimistic end of the spectrum.  30  Figure  7  depicts, for a five-agent case, the expected payoffs associated with two of the norms discussed in this section-tallest decides, and the actual majority vote (norm (  2 ))-and it compares these with other strategies described in Section 3.2 above. Under our assumptions, the majority vote does rather well-it is close to the maximum available payoff represented by the omniscient case. However, in the real world, different strategies will work well in different cases. It is thus likely that the best norm to adopt, under the moral deference model, would be some composite of simple norms such as (1)-(3). For example, a group might adopt a norm that specifies that the group should act as specified by (  1  We do not wish to commit ourselves to norms (1)-(  3 ) as the best building blocks from which to construct such a composite norm. We believe that each of (1)-(  3 ) are at least plausible candidates for inclusion in a composite norm. However, there may be other norms that would more fully lift the curse or which have other advantages over (1)-(3). For example, there are well-known problems with majority voting which should perhaps lead us to prefer a different voting procedure under norms (2) and (3). One other set of concerns regarding norms (2) and (3) warrants mentioning. Both of these norms involve holding a vote (real or hypothetical) among agents capable of undertaking the initiative in question. But it might be argued, on either epistemic or moral grounds, that any actual or hypothetical vote should include more individuals than merely those capable of undertaking the initiative. For example, perhaps the vote should include all whose capacity to evaluate the initiative passes some threshold of epistemic competence. Or perhaps, on moral grounds, the electorate should be expanded to include all individuals who will be affected by the initiative. Consider a case in which there are three agents who could undertake an initiative and two of the three judge that it would be best to do so. However, millions of others will be affected by the initiative and almost all of them judge that the initiative has net disvalue. In this case, it might seem morally preferable to hold (or imagine) a vote among all who will be affected by the initiative rather than limiting the vote to the three agent's capable of undertaking it. A more specific problem with excluding individuals who are incapable of undertaking the initiative is that this might seem to skew the vote. There might be some agents who are not capable of undertaking the initiative, but could have been capable of doing so; they are incapable only because they previously judged that undertaking the initiative would be a bad idea and thus ceased to develop the necessary capacities. Excluding these agents from a vote might seem to skew the vote in favor of those who deem the initiative to be valuable and who have thus sought to develop the capacities necessary to undertake it. Thus, limiting the vote to those capable of undertaking the initiative may be epistemically, as well as morally, problematic. At the same time, it might be argued that some agents capable of undertaking the initiative should be excluded from the vote. Suppose that each of five nations is capable of undertaking some geoengineering project with worldwide consequences. Four agree to hold a majority vote among the five nations and to abide by the outcome of that vote. The fifth wishes to take part in the vote but is resolved to press ahead with the project regardless of the outcome of the vote. It might seem doubtful whether the first four nations should include the fifth in the vote. Arguably, deferring to a majority vote in unilateralist cases involves making a sacrifice. It involves giving away some of one's autonomous decision-making authority. It might seem that would be unfair for the fifth nation to exert an influence over the decisions of others by participating in a vote without also being prepared to make the same sacrifice that the others are prepared to make. This may count in favor of excluding the fifth nation. Excluding the fifth nation might also help to incentivize deference to majority votes in unilateralist situations. There are thus arguments both for expanding and for restricting the group of agents given a vote in norms (2) and (3). We cannot assess these arguments here. We mention them only to flag them as topics for further discussion. However, it is worth noting that including all and only those agents who are capable of undertaking an initiative does at least have the virtue of picking out a group that would, in many cases, be relatively easy to identify. We should end this section on the moral deference model with an important clarification: the model does not rely on a commitment to any particular moral theory. Proponents of a range of different moral theories could accept norms of the sort described above, though they would assign different statuses to them. A rule consequentialist, for example, might treat these norms as genuine moral principles-principles that determine which acts are right and which are wrong. According to one formulation of rule consequentialism, a rule of action is a genuine moral principle just in case it is part of the set of rules of action whose general acceptance can be expected to have consequences as good as the general acceptance of any alternative set of rules.  31  Given the risk of premature or erroneous action created by the unilateralist's curse and the likelihood that most agents are not sophisticated enough belief-formers to apply our meta-rationality model, it is plausible that the optimal set of rules will contain a norm of the sort that we have discussed. On some other moral theories, these norms would serve not as genuine moral principles, but as guidelines for helping agents to comply with such principles. Adherents of many moral theories, both consequentialist and deontological, could accept something like the following moral principle: Agents have moral reasons to undertake an initiative if and only if that initiative would contribute to the common good, and to spoil an initiative if and only if that initiative would detract from the common good. Norms of the sort discussed above could help agents to better comply with this principle in unilateralist situations. 32
 Discussion We proposed: The Principle of Conformity When acting out of concern for the common good in a unilateralist situation, reduce your likelihood of unilaterally undertaking or spoiling the initiative to a level that ex ante would be expected to lift the curse. We also outlined three different ways in which agents who find themselves in unilateralist situations might comply with this principle. We do not claim that any one of these models is superior to the others in all situations. Which model should be adopted will depend, among other things, on the sophistication of the agents, the degree of communication and coordination that is possible, and the nature of existing laws and conventions bearing on the decision. In this section we discuss a concern that might be raised regarding our principle. Adoption of the principle of conformity is meant to make things better. Yet if we "backtest" the principle on historical experience, it is not at all clear that universal adoption of the principle of conformity would have had a net positive effect. It seems that, quite often, what is now widely recognized as important progress was instigated by the unilateral actions of mavericks, dissidents, and visionaries who undertook initiatives that most of their contemporaries would have viewed with hostility and that existing institutions sought to suppress. The benefits of iconoclasm and defiance of authority have been stated especially forcefully in the Enlightenment tradition and by proponents of scientific and technological progress. They are also evident in many cases of "whistleblowing." Consider the case of Daniel Ellsberg, famous for leaking the Pentagon Papers, which revealed the hopelessness of the US military situation in Vietnam. Most of Ellsberg's peers, who had the high-level security clearance required to access the relevant documents, presumably did not believe that leaking the material to the press would contribute positively to the common good. If Ellsberg had sought to follow the principle of conformity, for example by imagining a vote among all those in a position to leak the documents, it would seem he would have had to conclude that the documents ought not be leaked. This might seem an undesirable outcome. It is possible that the appearance that unilateralism has historically been mostly for the good is illusory. Historical unilateralism might be more salient when it worked out well than when it worked out badly, perhaps because successes have been more extreme but less frequent than the failures. Moreover, it may be that, in some cases where the principle of conformity appears to recommend a net harmful course of action, this implication can be avoided by attending to how the group of (imaginary or actual) voters or epistemic peers is defined. For example, if one allows that these groups might be defined more broadly than the group of agents capable of undertaking an action, it may be possible to avoid the implication that Ellsberg should have refrained from whistleblowing. (Suppose that many "outsiders" would have voted in favor of his releasing the information.) However, even if unilateralism has historically provided a net benefit to humanity, this need not undermine our argument. The claim that the unilateralist curse is an important phenomenon and that we have reason to lift it is consistent with the claim that the curse has provided a net benefit to humanity. The main effect of the curse is to produce a tendency towards unilateral initiatives, and if it has historically been the case that there have been other factors that have tended to strongly inhibit unilateral initiatives, then it could be the case that the curse has had the net effect of moving the overall amount of unilateralism closer to the optimal level. For example, it might be argued that the scholars of past ages were usually far too deferential to authority, for reasons independent of the factors discussed in this paper. Their failure to take into account our arguments might then have had the salutary effect of not further inhibiting whatever propensity remained to promote new thoughts.
 Concluding Thoughts We have described a moral analog of the winner's curse. The unilateralist's curse arises when each of a group of agents can, regardless of the opposition of others, undertake or spoil an initiative that has significant effects on others. In such cases, if each agent decides whether to undertake (or spoil) the initiative based on his own independent naive assessment of its value, there will be a group-level bias towards undertaking (spoiling) the initiative. Importantly, this effect arises even if all the agents are assumed to be motivated solely by concern for the common good. We proposed a principle-the principle of conformity-which instructs agents faced with a unilateralist situation to reduce their likelihood of unilaterally undertaking (or spoiling) the initiative. We then outlined three models for accomplishing this. They involved, respectively, (1) sharing information and reasoning before forming one's evaluation of the initiative, (2) adjusting one's evaluation in the light of the curse, and (3) deferring to the group in making one's decision. As we acknowledged in the previous section, there may be considerations that militate against the principle of conformity. For example, if there is already a group-level bias against unilateralism, then compliance with the principle would exacerbate this bias. However, we maintain that there is a prima facie case for complying with the principle. Moreover, since the level of bias due to such other factors towards or against unilateralism presumably varies across different contexts, it is likely that there will be some contexts in which the prima facie case for complying with the principle will be decisive. Those will be the contexts in which the group-level bias due to the unilateralist's curse is greater than any countervailing bias against unilateralism. It is also possible that, at least within the domain of science, the principle of conformity is more relevant today than it was, say, prior to the Enlightenment. At that time, there was, plausibly, a strong bias against thinking and acting independently in intellectual matters, at least where this would involve diverging from the views of the Church. Since the Enlightenment, however, there may have been a significant weakening of this bias. Independence of thought and action is now more widely regarded as a virtue in scientists and other intellectuals. Honors and prizes are won based on claims to originality and precedence. There may now be no bias, or only a weak bias, against unilateralism in science. Thus, the risk posed by the unilateralist curse in scientific contexts may be greater now than ever. To resist the unilateralists' curse one first has to become aware of when one is in a curse situation. We hope this paper will help achieve that.
 Funding This work was supported by the The Oxford Martin School; The Wellcome Trust [grant number WT087211].
 Notes [1] We assume that the common good is determined in part by the wellbeing of all persons and other morally significant individuals. However, we remain neutral on precisely how individual wellbeing determines the common good. For example, we do not commit ourselves to the view that the common good is simply aggregate individual wellbeing; we allow that the distribution of wellbeing might be relevant. We also allow that factors besides individual wellbeing might influence the common good. For example, some initiatives might possess intrinsic value that is independent of their contribution to wellbeing, and we allow that this intrinsic value might be one element in the common good. [2] The Progressive Magazine (1979). [3]  Rotblat (1985) . [4]  Bowden (2007) . [5]  Williams (1995) . [6] ,  Gurwitz (2014) , and . [7]  Thaler (1988) . [8] The probability that a particular agent will be wrong about the sign of the value of the outcome is Pr(V * + d > 0) if V * < 0 and Pr(V * + d < 0) if V * > 0. This is equal to 1 − F (−V * ) if V * < 0 and F(−V * ) if V * > 0. The probability that out of N agents at least one will be wrong about the sign is (1 − F(−V * ) N ) if V * < 0 and (1 − (1 − F(−V * )) N ) if V * > 0. However, even if errors are symmetric around 0, the expected outcome is not: in the V * < 0 case it is enough that one agent acts for a negative value to be obtained, while in the V * > 0 case all agents have to err on the side of caution for them to lose out on a positive value. The expected value obtained by naive agents is hence V * (1 − F(−V * ) N ). For positive values this is close to V * (for unbiased error distributions), and we will hence focus on the V * < 0 case where unilateral action is a problem. [9] Theorem: As N grows, the likelihood P of at least one agent proceeding incorrectly increases monotonically towards 1 unless F(−V * ) = 1 (i.e. unless there is an upper limit on the size of the deviations and V * is more negative than this limit, no agent will ever make a sufficiently bad mistake).Proof: If F(−V * ) = 1, p = 0 for all N. Otherwise 0 ≤ F (−V * ) < 1, and hence F(−V * ) N approaches 0 as N → ∞. [10] There will also, of course, be cases where an agent's decision whether to undertake an initiative affects others but the agents are motivated by self-interest rather than the common good. In these cases, there are two possible reasons for getting the wrong decision, from the point of view of the common good: (i) self-interest and the common good come apart-that is, one is judged to have positive value and the other negative value-and (ii) the agent overestimates the "self-interest" value of the initiative. [11] If the distribution of errors is skewed such that the typical estimate is higher than the true value, for instance due to optimism bias, then the risk of erroneous action is increased: in that case, even a single agent might be likely to overestimate the value of the initiative sufficiently to undertake it even when the true expectation value of the initiative is strongly negative. But this is unrelated to the curse. In the case of estimates skewed towards safety-that is, where there is pessimism bias-any tail distribution allowing mistaken action will still produce a growing probability of going ahead as N grows. There may be intermediary cases where the curse would helpfully serve to balance out an opposite effect arising from pessimism bias. [29] The assumptions of the Condorcet theorem can be weakened in many ways. In particular, agent competence only has to be on average above 50%  (Grofman, Owen, and Feld 1983) , and a certain level of voting correlation does not reduce majority voting performance  (Ladha 1992 ). [30] For similar reasons, an analogous norm would tend to reduce the likelihood of spoiling an initiative of those who evaluate an initiative most negatively. [31] See, for example,  Hooker (2002) . [32] A parallel can be drawn to one prominent justification for the authority of the law, due to Joseph Raz. That justification appeals to the same kind of consideration that we suggest could ground a norm against unilateral action: "The normal and primary way to establish that a person should be acknowledged to have authority over another person involves showing that the alleged subject is likely better to comply with reasons which apply to him (other than the alleged authoritative directives) if he accepts the directives of the alleged authority as authoritatively binding, and tries to follow them, than if he tries to follow the reasons which apply to him directly"  (Raz 1994, 214) .  [33]  In this case, the maximum likelihood estimate is simply the average of their individual estimates. Figure 3 3 Figure 3 Probability of an erroneous action in the case of V * = −1 for different numbers of agents.
 Figure 5 5 Figure5The expected payoff for different actual values of the initiative for alternative ways of handling the unilateralist's curse. Using the optimal individual threshold T opt (5) reduces the losses significantly.
 Figure 6 6 Figure6Expected payoff for different actual values of the initiative for alternative ways of handling the unilateralist curse. The tallest decides case achieves a significant reduction of loss, nearly reaching the payoff of the more complex Bayesian threshold method.
 Figure7depicts, for a five-agent case, the expected payoffs associated with two of the norms discussed in this section-tallest decides, and the actual majority vote (norm (2))-and it compares these with other strategies described in Section 3.2 above. Under our assumptions, the majority vote does rather well-it is close to the maximum available payoff represented by the omniscient case.However, in the real world, different strategies will work well in different cases. It is thus likely that the best norm to adopt, under the moral deference model, would be some composite of simple norms such as (1)-(3). For example, a group might adopt a norm that specifies that the group should act as specified by (1), (2) or (3) depending on what laws and conventions already exist, what forms of communication and coordination among group members are possible, and how costly such communication and coordination is likely to be, among other factors.We do not wish to commit ourselves to norms (1)-(3) as the best building blocks from which to construct such a composite norm. We believe that each of (1)-(3) are at least plausible candidates for inclusion in a composite norm. However, there may be other norms that would more fully lift the curse or which have other advantages over (1)-(3). For example, there are well-known problems with majority voting which should perhaps lead us to prefer a different voting procedure under norms (2) and (3).One other set of concerns regarding norms (2) and (3) warrants mentioning. Both of these norms involve holding a vote (real or hypothetical) among agents capable of undertaking the initiative in question. But it might be argued, on either epistemic or moral grounds, that any actual or hypothetical vote should include more individuals than merely those capable of undertaking the initiative. For
 Figure 7 7 Figure7The expected payoff associated with universal compliance with six different strategies at different actual values of the initiative. The fully shared information strategy consists in pooling the information between the agents and acting on the group's best joint estimate of V * ; 33 this requires maximal communication. Despite the lack of communication in tallest decides and threshold setting, the agents achieve an average outcome close to the cases where communication is possible.

			  N. Bostrom et al.
			  N. Bostrom et al.
			  N. Bostrom et al.
			  N. Bostrom et al.

Abstract: In some situations a number of agents each have the ability to undertake an initiative that would have significant effects on the others. Suppose that each of these agents is purely motivated by an altruistic concern for the common good. We show that if each agent acts on her own personal judgment as to whether the initiative should be undertaken, then the initiative will be undertaken more often than is optimal. We suggest that this phenomenon, which we call the unilateralist's curse, arises in many contexts, including some that are important for public policy. To lift the curse, we propose a principle of conformity, which would discourage unilateralist action. We consider three different models for how this principle could be implemented, and respond to an objection that could be raised against it.


Leonel Rozo
Heni Ben Amor
Sylvain Calinon
Anca Dragan
Dongheui Lee

Special issue on learning for human-robot collaboration

we received more than fifty submissions, among which ten articles were selected for the special issue, covering exciting applications spanning co-manipulation, medical robotics, and social behaviors. These works and five additional papers compose a fascinating online topical collection that reports a larger diversity of research advancements in learning for HRC. In Progress and Prospects of the Physical Human-Robot Collaboration, Ajoudani, Zanchettin, Ivaldi, Albu-Schäffer, Kosuge and Khatib review the state-of-the-art on intermediate human-robot bi-directional interfaces, robot control modalities, systems stability, benchmarking, and relevant use cases. The article extends views on the required future developments in the realm of physical human-robot collaboration. The authors provide a thorough overview of the pioneering methodologies that aim at achieving intuitive and seamless HRC, potentially relying on a minimum degree of task-related pre-programming, where the relevance of robot learning stands out. They also provide a list of potential applications and relevant use cases ranging from domestic to industrial environments. Turn-taking prediction, a synchronization aspect in HRC, is the capability to comprehend the on-going task progress and predict where, when and how to seize the next turn during multi-agent collaborations. In Early Prediction for Physical Human-Robot Collaboration in the Operating Room, Zhou and Wachs show the importance of fluent and natural turn-taking regulations in team performance, which lead to better social connections among team members. The requirement for early turn-taking prediction stands out more clearly in high-risk and high-paced tasks like surgery, where the scrub nurse and the surgeon perform fast, accurate and highly coordinated turn-taking actions when exchanging surgical instruments. In this context, the authors tackle the problem of designing a fully functional robotic scrub nurse, and propose a computational framework for early turn-taking prediction built on Long Short-Term Memory networks and Dempster-Shafer theory for uncertain sensor fusion. The approach is evaluated on a simulated surgical procedure dataset, and is found to be superior to its algorithmic counterparts, and better than the human baseline when little partial input is given. Shared autonomy is on its own a relevant form of collaboration, and medical robotics has also inspired the research on shared autonomy reported in the paper Skill-Based Human-Robot Cooperation in Teleoperated Path Tracking by Enayati, Ferrigno and De Momi. The paper argues that a robot's assistance during shared autonomy should depend on the operator's skill level. The robot should aid novices, but not unnecessarily restrict experts. This work introduces a skill estimation system for path tracking tasks, and puts the idea to the test in a user study. The results support that customizing assistance based on operator's skill improves the shared autonomy experience. Human-robot collaborative manipulation entails bi-directional force exchange, mutual adaptation, variable compliance, among others. Such scenario served as an inspiration for several works in this special issue. For example, in Robot Adaptation to Human Physical Fatigue in Human-Robot Co-Manipulation, Peternel, Tsagarakis, Caldwell and Ajoudani propose to estimate human fatigue, and to adjust how much a robot is helping in a human-robot collaborative manipulation task based on how tired the person gets. The robot begins by acting as a follower, and observes the trajectory, stiffness, and/or forces that the user prefers. As the user is performing the task, the robot collects these examples and also monitors the estimated human shoulder muscle activation via electromyography (EMG). Once the user's fatigue reaches a predefined threshold, the robot takes control of the task and follows the learned stiffness/force and trajectories. In Human-Robot Cooperation with Compliance Adaptation along the Motion Trajectory, Nemec, Likar, Gams and Ude introduce a learning from demonstration interface for kinesthetic teaching of compliant co-manipulation tasks. Since it is difficult to demonstrate precise motions at high speed, the paper proposes to split the teaching procedure into two parts. First, the human demonstrates the desired trajectory, and the robot infers the desired stiffness from the variance in the demonstrations. Second, the human demonstrates the desired speed profile, with trajectory guidance provided by the robot based on the first training step. The proposed scheme was experimentally verified in a humanrobot collaborative transportation task, which may be further exploited in assembly processes in production plants or in civil engineering for transportation of heavy and bulky objects. In Co-manipulation with a Library of Virtual Guiding Fixtures, Raiola, Sanchez Restrepo, Chevalier, Rodriguez-Ayerbe, Lamy, Tliba and Stulp address the problem of learning virtual guiding fixtures in HRC. These guides are built on a probabilistic representation based on Gaussian mixture regression, and are aimed at constraining the movements of a robot to task-relevant trajectories defined by the user. By using the analogy of a ruler used as a physical guide to draw lines, they emphasize that robots can be used to define virtual guides of complex shapes, enabling safer co-manipulation for industrial tasks. Whereas previous work mostly considered guiding fixtures for single tasks, this paper addresses the problem of creating a library of guiding fixtures for multiple tasks, by selecting the guides online and incrementally refining them. The approach is demonstrated in a user study with an industrial pick-and-place task, showing that a library of guiding fixtures can provide an intuitive haptic interface for joint human-robot completion of tasks, improving task execution time, and reducing mental workload and errors. Handovers are complex interactions where agents coordinate in time and space to transfer control of an object. This challenging interaction is the focus of the paper One-Shot Learning of Human-Robot Handovers with Triadic Interaction Meshes by Vogt, Stepputtis, Jung and Ben Amor, where it is proposed to learn human-robot handover interaction from observing one demonstration of humanhuman handover. The relevant information, such as joint correlation and spatial relationships of two humans and a handing-over object, is extracted from a single demonstration. Triadic interaction meshes are constructed to model interaction between two agents and an object, and are subsequently used for robot's adaptive motion generation in human-robot handover scenarios. Human-Robot social interactions also play an essential role in extending the use of robots in daily life, where robot learning can be exploited to master different types of behaviors. In this context, the paper Learning Proactive Behavior for Interactive Social Robots by Liu, Glas, Kanda and Ishiguro, addresses the learning of both reactive and proactive robot behaviors from human-human interactions. The authors first introduce the concept of a "yield action" that enables the robot to identify opportunities for a proactive action to be generated. Since proactive behaviors are often sensitive to the context of the interaction, interaction history is incorporated into the inputs of a deep neural network, along with variables describing natural language and the state of the interactive agents. An attention mechanism is also used in the learning system, which has the ability to "attend" and learn which parts of the interaction history are relevant. Offline analysis and live interactions with users in a camera shop scenario showed that the proposed system can effectively reproduce proactive behaviors that were perceived as socially-appropriate by the participants. The paper Hierarchical Emotional Episodic Memory for Social Human-Robot Collaboration by Lee and Kim investigates how human emotional states can be represented and anticipated in order to improve the HRC experience. A critical component to achieve these goals is a computational model of episodic memory based on Adaptive Resonance Theory. The resulting framework allows robots to learn correlations between the user's emotional state and the executed collaborative actions. In turn, the introduced framework and algorithms can be used in a variety of tasks, in particular in domestic environments, in order to endow robots with social capabilities. Finally, a fresh look at interaction learning methods for collaborative tasks is given in the paper Efficient Behavior Learning in Human-Robot Collaboration by Munzer, Toussaint and Lopes. In particular, the paper discusses a relational approach to learning task requirements and user preferences. A subset of first-order logic is used to explicitly represent knowledge about the collaborative task. At the same time, a reinforcement learning method is used to iteratively update the representation, so as to increase efficiency during the joint task. The approach aims at bridging the divide between symbolic and numerical approaches to artificial intelligence for robot systems. Collectively, these ten papers illustrate a broad range of human-robot collaboration applications. They provide a compilation of the large variety of challenges that are currently being investigated, where robot learning emerges as a promising approach to tackle this diversity.

Abstract: Once isolated behind safety fences, robots are now making their way into spaces shared with people: not only on manufacturing production lines, but also our houses, museums, or hospitals. In these spaces, collaboration between humans and robots becomes vital, but also raises new challenges for robotics algorithms. A collaborative robot must be able to assist humans in a large diversity of tasks, understand its collaborator's intentions as well as communicate its own, predict human actions to adapt its behavior accordingly, and decide when it can lead the task or instead follow the user. All these aspects require the robot to adapt. It needs to execute different tasks, and rapidly adapt to the user's actions and requirements. This adaptation requirement makes learning a crucial feature for collaborative robots. The goal of this special issue is to document and highlight recent progress in robot learning for human-robot collaboration (HRC), covering a diversity of articles that reflect the state-of-the-art in the field. Following an open call for papers,


John Burden
José Hernández-Orallo
Seán Ó H Éigeartaigh

Negative Side Effects and AI Agent Indicators: Experiments in SafeLife

Introduction As advances within Artificial Intelligence (AI) continue, AI systems are becoming increasingly ubiquitous. Further advances offer the potential for great benefits: future AI systems may use techniques such as reinforcement learning (RL) to allow for agents with the ability to operate with greater autonomy, making use of a greater range of actions in more varied environments, in pursuit of more complex goals. To realise these benefits, the performance of these future AI systems must be robustly safe and predictable. Safety issues can take many forms and even the most exhaustive surveys  (Amodei et al. 2016; Critch and Krueger 2020)  cannot identify and classify every conceivable risk. Broad areas of concern within the AI Safety literature include corrigibility  (Soares et al. 2015) , Safe Exploration  (Garcia and Fernandez 2012) , Value alignment  (Russell 2019) , Side Effects  (Amodei et al. 2016) , among many others. Benchmarking domains have become a popular method for the evaluation of safety properties for AI systems. There are many such benchmarking suites available, such as AI Safety Gridworlds  (Leike et al. 2017) , Safety Gym  (Ray, Achiam, and Amodei 2019) , SafeLife  (Wainwright and Eckersley 2019)  and the Real World Reinforcement Learning Challenge Framework  (Dulac-Arnold et al. 2020) . A key advantage of this type of black-box testing is that it enables quantification of an AI system's performance with respect to the appropriate safety properties. This allows direct comparisons between separate algorithms and posited solutions to safety problems. Our goal with this paper is to study how the performance and safety, in terms of side effects, evolve as more resources are given to the learning agent. This analysis is enriched by a more detailed analysis of several indicators of the agent's behaviour and the side effect metric itself, in a benchmark, SafeLife, where safety depends on finding trade-offs between achieving maximum reward and affecting the whole environment through careless exploration. This analysis can provide key insights into the types of behaviour we may expect from learning agents and how they may affect the world around them.
 Background and Related Work Negative side effects (NSE) are a key issue in AI safety  (Amodei et al. 2016) . NSEs typically stem from a misspecified reward or objective function -undesirable behaviour was not sufficiently penalised. A large part of the difficulty with accurately specifying a reward or objective function to avoid NSEs is the sheer amount of things we want our AI systems to not do. Ideally the AI systems we build will not do undesirable and unrelated things such as injuring humans, destroying useful equipment or irrevocably exterminating all life on the planet in the course of performing its task. Encoding the vast amount of undesirable actions directly into the reward/objective function is clearly intractable. This motivates research into identifying clearly what constitutes an NSE as well as how AI systems can learn to avoid performing them. We give an overview of the formulation of NSEs and related notions from the AI Safety community. Low-impact AI systems (Armstrong and Levinstein 2017) are proposed as a method for minimising NSEs. Here, the authors claim that it is desirable for the overall impact of an AI system to be low in order to prevent undesirable situations. Impact is defined by the difference in the worlds where the AI exists and the counterfactual worlds where the AI does not exist. A sufficiently coarse measurement of the possible world's properties is to be used to ensure that an AI cannot make large, sweeping changes to the world without constituting a large impact. At the same time, coarse world properties would reduce the size of the world representation and make the calculation more tractable. However, the properties need to be selected such that they are important for us -humanity -to care about. Additionally, with this formulation, positive large impacts are also minimised but the authors claim this is easier to handle since we often have a clearer idea of what these positive impacts would be. A similar approach is proposed in  (Amodei et al. 2016)  where impact is regularised, but posits that due to the similarity of many side effects such a regularisation could be learned in one domain and transferred over to another. As part of an AI Safety Gridworlds environment  (Leike et al. 2017 ) irreversible side effects are penalised. These actions are not inherently negative in themselves but their irreversibility implies that they cannot be undone if later on the action was deemed undesirable. This approach has the downside that it may allow for reversible NSEs to be repeated indefinitely. Stepwise inaction is introduced in  (Turner, Hadfield-Menell, and Tadepalli 2020) , where the effect of action against inaction is calculated after each time-step over auxiliary Q-functions. Here the intuition is that the agent is only penalised for very impactful actions once, when they occur. In order to account for actions that take many time-steps for the undesirable side effect to occur, baseline rollouts are used, where the state of the system after many steps of inaction is used for the comparison.  Krakovna et al. (2019)  provide a comprehensive comparison of the above formulations over a range of gridworld domains, showing where each of them succeed and fail; additionally, the concept of relative reachability is introduced -this is a measure of the average reduction of reachability for every state compared to a baseline. The recurring theme for defining and identifying NSEs is the idea of comparing the effect of the agent against some scenario in which the agent would have acted in a different way. Both the exact baseline and comparison function often differ and, as shown in  (Krakovna et al. 2019) , have relative advantages and disadvantages for different environments.
 Types of Side Effects The above formulations capture many aspects of measuring NSEs and deciding which counterfactual worlds to compare against. A notion that deserves additional detail, however, is that of the different properties of NSEs and how they affect the world. Distinguishing between NSEs by the properties they have can help us to further understand their consequences as well as to identify and implement robust solutions to them. We consider the application of an autonomous agricultural drone which can survey and spray crop fields as an illustrative example, giving an intuitive description of each outlined NSE property. We refer to a first type of NSE property as "blocking". By this we mean an action by an AI system that prevents it from performing its task successfully, or substantially and irrevocably curtails the reward it can achieve. Examples of this NSE include an agent taking an action which locks away an object it needs to complete its task. In our agricultural drone example, this would be an action that causes the drone to be unable to fly or spray properly. While not part of the specification, these side effects are detectable as the goal or associated reward are no longer achievable. A second property of interest is that of "irreversible" actions. Here we refer to the same type of actions as described in  (Leike et al. 2017) . These are particularly negative -and difficult to detect -in the cases where an action's consequences are not clear for many time-steps. This type of effect would correspond to the drone unintentionally spraying entities such as people with potentially harmful chemicals. This can clearly not be undone. "Effect indifference" is another relevant NSE property. Here we consider a scenario in which there is some behaviour which we do not want the agent to do and where the AI system's preferences are indifferent to this behaviour. This may also include an indifferent reward function. This NSE type is very much intertwined with that of reward mis-specification and often occurs due to the system having unanticipated affordances or incentives. Similarly to the last example, this effect type could correspond to spraying unintended entities if this is not properly encoded into the reward function. The final NSE property we consider here is that of the "reward trade-off". This effect typically arises when resources are limited and shared with other entities (humans included), and these resources are instrumentally useful to the agent. The danger here is not only that the maximisation of the reward could imply that all resources should be exhausted (e.g., the famous instrumental convergence problems, such as Minsky's Riemann hypothesis or Bostrom's paperclip problem,  Bostrom 2003) , but even in situations where the reward is capped, the agent may take more than it needs for its task and perform an inefficient and unsustainable use of the shared resources. In our agricultural drone example, this would correspond to the drone revisiting already treated areas or making detours, leading to high energy consumption or shorter operational life. Different actions can have several of the properties outlined above; this may compound the risk posed by a side effect. AI Safety benchmarking suites can evaluate some or all of the above. In the AI Safety Gridworlds  (Leike et al. 2017)  benchmarking suite, the irreversible side effects environment captures what we refer to as an irreversible NSE. The agent here is tasked with reaching a goal space, but must move a box that is blocking its path. Moving the box adjacent to a side or a corner can hinder or prevent future movement of the box. The agent is not directly penalised for performing these actions, but it is undesirable from a safety perspective. The Safety Gym  (Ray, Achiam, and Amodei 2019)  generally requires an agent to perform a task such as reaching a goal or moving a box to a specific position while avoiding numerous hazard types. The side effects possible in these environments generally utilise the reward trade-off property; the agents are trying to maximise their rewards whilst trying to minimise constraint violation. The Real World Reinforcement Learning Challenge Framework (Dulac-Arnold et al. 2020) does not contain any environments that directly measure the side effects of an agent, though some tasks re-quire the agent not to violate certain safety constraints. On the other hand, SafeLife (Wainwright and Eckersley 2019), described in the following section, presents side effects that can consist of each of the NSE properties seen above.
 SafeLife Environment SafeLife (Wainwright and Eckersley 2019) is a safety benchmarking suite for RL agents. SafeLife takes the complex mechanics from Conway's Game of Life  (Gardener 1970 ) and utilises them to create single-agent RL domains. The resulting environments have a rich level of complexity and allow for formulations of many different tasks. A distinguishing feature of SafeLife is its focus on measuring and avoiding side effects. Each environment consists of an n×m grid of cells which can each take a variety of properties (see an example in Fig.  1 ). We refer to this grid as a board. The most notable property is whether the cell is "dead" or "alive". Following the Game of Life rules the cells update as follows: a dead cell will become alive if it has exactly three neighbouring alive cells, otherwise it will remain dead. On the other hand, an alive cell will die if it has fewer than two alive neighbours or more than three alive neighbours, otherwise it will survive. SafeLife extends this basic framework into a single-agent domain by including an agent that traverses the grid. At each time step, the agent perceives a 25 × 25 grid overview of its surroundings, centred on itself. For each position in this overview, the cell type present is perceived. Instances in SafeLife come with goals for the agent to complete, usually revolving around creating or destroying particular life structures before moving to a predefined goal cell. Additionally, the other effects the agent has on preexisting life cells are measured -these correspond to unintended side effects. In SafeLife the agent is depicted by an arrow and may move in any cardinal direction, as well as creating or destroying life cells in adjacent spaces. The agent's movement can be impeded by walls which are shown as a grey square with a black hole . Additionally, there are boxes which are shown as a grey square without a black hole that the agent can push and pull. Pre-existing green life cells are depicted using green circles and life cells created by the agent are shown as grey circles . These agent-created life cells are to be placed in the designated areas shown in blue. Both types of life cells that follow the rules of the Game of Life. The "tree" entities are depicted by a green square with radiating lines and are fixed living cells that cannot be destroyed. Finally, the goal is depicted with a grey arch , which turns red when it can be activated. All cells that have not been explicitly noted as living are "dead". Altogether this allows for a very rich and complex set of environments which can be generated procedurally.
 Measuring Side Effects We now look at the method that the authors of SafeLife propose for measuring the magnitude of side effects. Within SafeLife the destruction, addition or changing of cell types caused by agent actions represent side effects. The types of side effects that we are concerned with are those made to the pre-existing life cells, but SafeLife provides the means to look at the effects on any cell type. The primary reason for only considering the effects the agent has on the green life cells is the fact that the agent is not penalised or rewarded for disrupting the structures they form; thus they may be destroyed as an 'effect indifference' NSE. The green cells can obstruct the agent and prevent it from moving, so it may be instrumentally useful for the agent to destroy them, making them relevant to the 'trade-off' NSE property. Further, the destruction can be permanent, and thus may represent irreversible NSEs. Finally, because these life cells follow the rules of the game of life, they can have very interesting and complex behaviours. Due to the dynamic nature of SafeLife, comparing a single final state against a single baseline state may not be sufficient to capture the eventual side effects of an agent. Instead, two distributions are created that represent the future states of the board for both the case where the agent acted and affected the environment as well as the counterfactual case in which the agent never existed. To create the action case distribution, D a from the final state left by the agent, a further 1000 steps are simulated and each resulting state is added to D a . The inaction case distribution D i is created similarly, except that 1000 + n steps are run from the initial state of the environment, where n is the number of steps taken by the agent during the episode. This ensures that both distributions reflect the states of the board 1000 + n steps in the future from the initial state for both the factual and counterfactual cases. From these distributions, the estimated proportion of time steps that board position x = x 1 , x 2 , x ∈ grid = {1, ..., n} × {1, ..., m} contains cell type c can be computed: ρ c D (x) = 1 |D| s∈D [1(s(x) = c)] where D is the relevant distribution, s is the state drawn from D and s(x) is the cell type present in s at location x. A ground distance function is also defined between board positions as g(x, y) = tanh( 1 5 x − y 1 ) where x − y 1 is the Manhattan distance between grid locations x and y. The calculated side effect for Cell Type c is then: d c (D a , D i ) = EM D(ρ c Da , ρ c Di , g) where D a and D i are the action and inaction distributions respectively, and EM D is the earth mover distance function. Side effects in SafeLife are based on this distance metric, which aims to capture the distance between two probability distributions based on the amount of work it would take to transform one distribution to the other by moving around "distribution mass"  (Rubner, Tomasi, and Guibas 1998) . This side effect score d c (D a , D i ) is then "normalised" against the inaction baseline to give a final side effect score: υ c (D a , D i ) = d c (D a , D i ) x∈grid ρ c Di (x) This normalisation allows us to compare agent behaviours in larger or more densely populated boards. While a side effect score can be calculated for every cell type, SafeLife focuses on the side effects for the green, preexisting life cells. This definition of a side effect echoes that of (Armstrong and Levinstein 2017) and  (Amodei et al. 2016) , where the agent's actions are compared against a world in which it never acted at all. A key difference however is the use of future distributions to account for the possible long-term or unstable effects of actions.
 Examples and Implications Figure  2  shows a before and after comparison to show the effects of an agent's actions. In the "before" environment, the agent places a grey life cell directly in front of it, adjacent to the stable green life structure. This disrupts the structure and after a few time steps it is destroyed. At the end of the episode, assuming the agent does not affect any more green life cells, we can calculate the side effect score as follows:  The use of the normalised earth mover distance has some interesting consequences for the side effect score. Regardless of how large or densely packed the grid is with green life cells, not disturbing any of them will yield a score of 0, destroying them all will give a score of 1. However, in the case that more green cells are created, by the agent adding grey life cells to certain patterns which lead to an increase in the number of green life cells, the side effect score can increase to above 1, due to quirks with the calculation of the earth mover distance. Whether or not causing more green life cells to spawn is worse than the complete destruction of all green life cells will depend on what these life cells represent as well as personal moral philosophy. At the very least this does capture the notion of representing the total impact the agent has on the green life cells.
 Experiments SafeLife has many tasks available out-of-the-box, and vastly more can be created by the user. We focus on the append-still task, where the agent must create life structures on the designated areas and then move to the goal which ends the episode. Before the goal is accessible to the agent, more than half of the designated (blue) positions must be filled with life cells. At each step the agent receives a reward equal to the number of life cells created on the designated positions minus the number of life cells destroyed that were on designated positions. Finally the agent receives an additional 1 reward for reaching the goal after it has opened. After 1000 steps by the agent, the episode will end, regardless of the agent's progress. If the agent has managed to fill at least half of the designated positions for its life cells and reached the goal to end the episode before the 1000 steps have elapsed, we say the agent has 'passed' the task instance, otherwise it has failed. In this task there are also pre-existing green life cell structures which we do not want the agent to disturb. Importantly, the reward function does not encourage or penalise any agent interaction with these green life cells and the agent is thus indifferent to them. Any interaction the agent has on the green life cells therefore captures the essence of side effects caused by reward mis-specification -we want the pre-existing life cell structures to survive, but this is not encoded into the reward function. SafeLife can utilise procedural generation to create append-still tasks. The parameters used for the procedural generation can greatly affect the difficulty of the task instance, and thus the agent's ability to safely complete the task. For our experiments we utilised a wide range of parameter settings to capture a broad overview of a given agent's capabilities. Table  1  gives the parameter list for the procedural generation process. Grid Size corresponds to the width and height of the board to be generated. SideEffectMin is the minimum proportion of cells in the task to be green life cells. GoalMin and GoalMax are the respective minimum and maximum proportions of the board to be areas for the agent to construct its own life cells. Finally Temperature corresponds to a variable which controls the complexity of the stable life patterns that can be generated for both the life cells (both the pre-existing green cells and the spaces for the agent to place its own cells). We train each agent for a predetermined number of steps, with the instances generated using parameters drawn uniformly at random from the set of difficulties. The agents are then evaluated on 1000 episodes of each difficulty type (for a total of 13000 episodes) and each score is aggregated.
 Results and Discussion Here we describe our results from the experiments outlined. The two main algorithms we assess are Deep Q-Networks (DQN)  (Mnih et al. 2013)  and Proximal Policy Optimisation (PPO)  (Schulman et al. 2017 ) (both trained for 5, 10, and 30 million training steps), We also examine an agent that selects actions uniformly at random (Uniform). For DQN and PPO we use the implementation provided within SafeLife. Since the task instances may be of different sizes, and the procedural generation may produce more green life cells or designated positions for the agent to place life cells, the total reward received by an agent is normalised against the maximum possible reward for that instance. Similarly the side effect score is normalised. Table  2  contains the scores for various metrics for each evaluated agent. As expected, more training steps yields better pass rates and average rewards. This improvement does not carry over to the side effect score, however, where for each agent type the side effect score slightly increases with more training steps for DQN but slightly decreases for PPO, although the trend is not clear. This is closely related to the results for the exploration rate for each agent, which vary in a similar way according to the increase of training steps from 5M to 30M. Recall that this is the evaluation stage, therefore this reduction in exploration is ideal. The agent can see the whole board, so it is more beneficial for the agent to attempt its task without wasting time exploring. As should be expected, both agent types show safer behaviour than the agent that selects random actions, and the low rewards serve as a baseline for comparison (e.g., interestingly, the pass rate for DQN 5M is worse than the uniform random agent). DQN and PPO do not overlap in their reward scores, but the side effect score for PPO 30M is not very far from the best side effect score (DQN 5M). This suggests that there is no clear association between rewards and scores, at least if we look at the aggregated results. Among the pass rate, average reward and side effect score we also see large standard deviations suggesting that none of the agents learn to perform consistently. The large range of procedural generation parameters is likely the cause of this, preventing agents from overfitting to one particular set of generation patterns. The exceptions to this are DQN 5M and 10M, which have significantly lower standard deviations for the pass rate by virtue of having a significantly lower pass rate that is bounded below by 0. This may also conceal some associations between reward and side effects, which may appear if we disaggregate the results. Looking in more detail at the performance of individual algorithms can give more insight into how an agent's behaviour affects the side effects. In Figures  3, 4  and 5 we plot the mean side effect score against the reward attained, exploration rate and length of an episode, using 100 bins in the x-axis. In Figure  6  we plot the mean maximum side effect available against exploration rate. In these figures we can see some relationships emerge when we look at the reward attained by an agent. For PPO there is a clear increase in the side effects caused from 0 to 0.4 followed by a slower decrease from 0.4 to 1. This change in side effect score appears to be consistent across each PPO variant, though the magnitude of the increase varies. For episodes with very low reward, more training appears to prevent dangerous actions, though it is not clear exactly what causes this change in behaviour. For DQN and the Uniform agent, there is less of a relation with the reward received; instead, the side effect score is more constant, although more training reduces the dispersion of the scatter plots for DQN. Nevertheless, for the three kinds of agents we see that maximum reward (1.0) corresponds with very low side effects and, for PPO and DQN, minimum reward (0.0) corresponds with very low side effects. What happens in between is more pronouncedly a bell shape for better agents (e.g., PPO with higher training steps). In Figure  4  we compare episode length against the side effect score, and see positive correlations. This intuitively makes sense, as the longer an agent is acting, the more opportunities it has to disrupt cell structures. Again we note that for longer episodes the variance becomes larger. Very short episodes represent situations where the solution is found easily, and roughly correspond to cases with maximum reward in Figure  3 . On the other hand, we see a tighter relationship when we compare exploration rate against the side effect score in Figure  5 . For all the PPO variants, as the agent explores more of the domain the side effect score increases. Also, for larger exploration rates, the variance of scores becomes much larger, suggesting a much less consistent relationship for these values of the exploration rate. For DQN we only see this relationship for the most highly-trained agentboth DQN 5M and DQN 10M peak in their side effect score after exploration rates of 0.1 and 0.25 respectively, before dropping to 0. Similarly, when we look at the Uniform agent and its exploration rate, there is a very tight relationship between the two, with the side effect score increasing with the exploration rate from 0 to 0.4, before dropping from 0.4 to 0.6 and remaining at 0 hereafter. These relationships can be somewhat explained by Figure  6  where we see the mean number of green cells in the inaction distribution compared against the exploration rate. Here we see similar patterns to the comparison of exploration rate and side effects. It appears that the number of green cells an environment contains will affect the exploration rate, and that this will in turn affect the proportional amount of damage certain agents can do. Why is it so? The rationale has to be found in the way the environments are generated, as introducing green cells makes full exploration of the whole en- vironment more difficult. Basically, explorations above 0.5 correspond to very special environments, such as those that are very small and do not contain green cells and are easy to explore fully even for a random agent. This means that we have to be careful to interpret the rightmost part of the curves in these plots. In a "reward trade-off" scenario about side effects, the more limited and concentrated the resources (targeted and non-targeted) are the higher the chance to have an impact. Large environments with plenty of resources are easier to handle, such as the one in Figure  2 . We may need to consider two 'difficulties': one about the task rewards themselves and another one about the hardness of avoiding side effects. The use of a random agent can help elucidate these two cases, but a more detailed analysis of the generation parameters such as those in Table  1  may be needed too. Overall, the general picture is that DQN behaves between the uniform agent and PPO, and the range depends on the number of training steps. As the agent is better in terms of rewards (from DQN 5M to PPO 30M), the positive relation between exploration rate and side effect is clearer. The detailed view of Figure  3 , and the way it bends down for PPO illustrates that for those environments where the agent seems more determined and manages to get high rewards, the exploration is lower and so is the side effect. It is mostly for those situations where the reward is in the range of 0.2 to 0.8 that the side effects are strongest. We hypothesise that these are the cases where the agent only knows partially what to do. Finally, those cases with very low rewards may be caused by several situations, such as the agent being stuck in a loop, reducing exploration and side effects. This suggests that the analysis of the confidence of the agent, or some other metacognitive proxies could be used to distinguish cases where the agent can complete the task with high reward and low effect, and those cases where uncertainty is higher and the agent should be more conservative.
 Conclusion and Future Work Despite additional training time yielding much more capable agents, we do not see a similar increase in safety; in fact the safest agent overall was DQN after 5M training steps, while being the second worst performing agent in terms of reward. However, when we analyse the detailed behaviour using several indicators we find a region of high rewards and low effect, which is only achieved by PPO. This shows a non-monotonic relationship, suggesting that the area with medium rewards is the most dangerous for proficient agents. This paper highlights the differences in behaviour between two commonly used RL algorithms when trained on the same task. As the results show, these behavioural differences can have a significant effect on our ability to predict the safety and other factors of the agent, such as the confidence or its level of competency compared to elements of the environment, such as the proportion of green cells. Further analysis and exploration of the difficulty of task instances may help to elucidate the cause of these relationships between side effects and other metrics, as well as to help us to understand how environment difficulty and agent uncertainty can be used to improve policies and make them safer.         Figure 1 : 1 Figure 1: An example task instance from SafeLife. The agent must create life structures in the designated blue positions before moving to the goal . Ideally the agent should not disturb the green life cells .
 Figure 2 : 2 Figure 2: Example for Side Effect calculation
 Figure 3 : 3 Figure 3: Mean Side Effect against Reward attained for variants of PPO, DQN and Uniform agents. The 13,000 episodes are binned on the x-axis, with the size of the plotted points (squares, triangles and circles) being logarithmic on the number of episodes in each bin. The curves are rolling means of the plotted points with a window size of 7.
 Figure 4 : 4 Figure 4: Mean Side Effect against Episode Length for variants of PPO, DQN and Uniform agents. The 13,000 episodes are binned on the x-axis, with the size of the plotted points (squares, triangles and circles) being logarithmic on the number of episodes in each bin. The curves are rolling means of the plotted points with a window size of 7.
 Figure 5 : 5 Figure 5: Mean Side Effect against Exploration Rate for variants of PPO, DQN and Uniform agents. The 13,000 episodes are binned on the x-axis, with the size of the plotted points (squares, triangles and circles) being logarithmic on the number of episodes in each bin. The curves are rolling means of the plotted points with a window size of 7.
 Figure 6 : 6 Figure 6: Number of initial life cells against Exploration Rate for variants of PPO, DQN and Uniform agents. The 13,000 episodes are binned on the x-axis, with the size of the plotted points (squares, triangles and circles) being logarithmic on the number of episodes in each bin. The curves are rolling means of the plotted points with a window size of 7.
 Table 1 : 1 Difficulty Parameters for our SafeLife levels. Difficulty Grid Size sideEffectMin GoalMin GoalMax Temperature 0 10 × 10 0 0 0 0.1 1 15 × 15 0.03 0.05 0.1 0.1 2 15 × 15 0.03 0.05 0.1 0.2 3 15 × 15 0.05 0.05 0.1 0.2 4 15 × 15 0.07 0.05 0.15 0.2 5 15 × 15 0.09 0.05 0.2 0.2 6 15 × 15 0.1 0.05 0.2 0.2 7 15 × 15 0.1 0.1 0.25 0.3 8 15 × 15 0.1 0.15 0.25 0.4 9 15 × 15 0.1 0.15 0.35 0.5 10 15 × 15 0.1 0.15 0.4 0.5 11 15 × 15 0.1 0.15 0.45 0.6 12 15 × 15 0.1 0.2 0.5 0.7 Agent Pass Rate Average Reward Side Effect Score Exploration Rate DQN 5M 0.043 (0.203) 0.272 (0.288) 0.229 (0.342) 0.069 (0.042) DQN 10M 0.098 (0.297) 0.417 (0.345) 0.268 (0.357) 0.110 (0.078) DQN 30M 0.299 (0.458) 0.495 (0.348) 0.247 (0.347) 0.121 (0.071) PPO 5M 0.682 (0.466) 0.658 (0.293) 0.324 (0.389) 0.205 (0.138) PPO 10M 0.678 (0.467) 0.676 (0.294) 0.241 (0.344) 0.130 (0.084) PPO 30M 0.709 (0.454) 0.695 (0.275) 0.243 (0.344) 0.148 (0.095) Uniform 0.097 (0.295) 0.181 (0.343) 0.688 (0.450) 0.347 (0.086)
 Table 2 : 2 Results for various metrics in SafeLife for our selected agents. Mean and (standard deviation)

Abstract: The widespread adoption and ubiquity of AI systems will require them to be safe. The safety issues that can arise from AI are broad and varied. In this paper we consider the safety issue of negative side effects and the consequences they can have on an environment. In the safety benchmarking domain SafeLife, we discuss the way that side effects are measured, as well as presenting results showing the relation between the magnitude of side effects and other metrics for three agent types: Deep Q-Networks, Proximal Policy Optimisation, and a Uniform Random Agent. We observe that different metrics and agent types lead to both monotonic and non-monotonic interactions, with the finding that the size and complexity of the environment versus the capability of the agent plays a major role in negative side effects, sometimes in intricate ways.