Trevor N White
Seth D Baum
Patrick Lin
George Bekey
Keith Abney
Ryan Jenkins

Liability Law for Present and Future Robotics Technology

Introduction In June 2005, a surgical robot at a hospital in Philadelphia malfunctioned during a prostate surgery, possibly injuring the patient.  1  In June 2015, a worker at a Volkswagen plant in Germany was crushed to death by a robot that was part of the assembly process.  2  In November 2015, a self-driving car in California made a complete stop at an intersection and then was hit by a car with a human driver, apparently because the self-driving car followed traffic law but not traffic norms.  3  These are just some of the ways that robots are already implicated in harms. As robots become more sophisticated and more widely adopted, the potential for harm will get even larger. Robots even show potential for causing harm at massive catastrophic scales. How should robot harms be governed? In general, liability law governs harms in which someone or something else is responsible. Liability law is used to punish those who have caused harms, particularly those that could have and should have been avoided. The threat of punishment further serves to discourage those who could cause harm. Liability law is thus an important legal tool for serving justice and advancing the general welfare of society and its members. The value of liability law holds for robotics just as it does for any other harm-causing technology. But robots are not just any other technology. Robots are (or at least can be) intelligent, autonomous actors moving about in the physical world. They can cause harms through actions that they choose to make, actions that no human told them to make and, indeed, that may surprise their human creators. Perhaps robots should be liable for their harms. This is a historic moment: humans creating technology that could potentially be liable for its own actions. Furthermore, robots can have the strength of industrial machinery and the intelligence of advanced computer systems. Robots can also be mass produced and connected to each other and to other technological systems. This creates the potential for robots to cause unusually great harm. This paper addresses how liability law can and should account for robots, including robots that exist today and robots that potentially could be built at some point in the near or distant future. Three types of cases are distinguished, each with very different implications. First are cases in which some human party is liable, such as the manufacturer or the human using the robot. These cases pose no novel challenges for liability law: they are handled the same way as with other technologies in comparable circumstances. Second are cases in which the robot itself is liable. These cases require dramatic revision to liability law, including standards to assess when robots can be held liable and principles for dividing liability between the robot and the humans who designed, built, and used it. Third are cases in which the robot poses a major catastrophic risk. These cases merit separate attention because a sufficiently large catastrophe would destroy the legal system and thus the potential to hold anyone or anything liable. The three types of cases differ across two dimensions as shown in Figure  1 . One dimension is the robot's degree of legal personhood, meaning the extent to which a robot shows attributes that qualify it for independent standing in a court of law. As we discuss, a robot can be held liable in the eyes of the law to the extent that it merits legal personhood. The other dimension shows the size of the harm the robot causes. Harms of extreme severity cannot be handled by liability law. However, there is no strict distinction between the three cases. Instead, there is a continuum, as shown by the regions in which a robot can have partial liability or more-than-human liability and in which liability law works to a limited extent.
 I -A Human Party Is Liable In a detailed study of robot law,  Weaver (2014, 21-27)  identifies four types of parties that could be liable for harm caused by a robot: (1) people who were using the robot or overseeing its use; (2) other people who were not using the robot but otherwise came into contact with it, which can include people harmed by the robot; (3) some party involved in the robot's production and distribution, such as the company that manufactured the robot; or (4) the robot itself. For the first three types of parties, liability applies the same as for other technologies. A surgical robot, for example, can be misused by the surgeon (type 1), bumped into by a hospital visitor who wandered into a restricted area (type 2), or poorly built by the manufacturer (type 3). The same situations can also arise for other, non-robotic medical technologies. In each case, the application of liability law is straightforward. Or rather, to the extent that the application of liability law is not straightforward, the challenges faced are familiar. The fourth type-when the robot is liable-is the only one that poses novel challenges for liability law. To see this, consider one of the thornier cases of robot liability, that of lethal autonomous weapon systems (LAWSs). These are weapons that decide for themselves whom to kill.  Sparrow (2007)  argues that there could be no one liable for certain LAWS harms-for example, if a LAWS decides to kill civilians or soldiers who have surrendered. A sufficiently autonomous LAWS could make its own decisions, regardless of how humans designed and deployed it. In this case, Sparrow argues, it would be unfair to hold the designer or deployer liable (or the manufacturer or other human parties). It might further be inappropriate to hold the robot itself liable, if it is not sufficiently advanced in legally relevant ways (more on this in Section II). In this case, who or what to hold liable is ambiguous. This ambiguous liability is indeed a challenge, but it is a familiar one. In the military context, precedents include child soldiers  (Sparrow 2007, 73-74)  and landmines  (Hammond 2015, note 62) . Child soldiers can make their own decisions, disobey orders, and cause harm in the process. Landmines can linger long after a conflict, making it difficult or impossible to identify who is responsible for their placement. In both cases, it can be difficult or perhaps impossible to determine who is liable. So too for LAWSs. This ambiguous liability can be a reason to avoid or even ban the use of child soldiers, landmines, and LAWSs in armed conflict. Regardless, even for this relatively thorny case of robot liability, robotics technology raises no new challenges for liability law. In the United States, agencies such as the Department of Defense produce regulations on the use of LAWSs which are not dramatically different than for other weapons. Internationally, bodies like the UN's International Court of Justice could hold a state liable for authorizing drone strikes that caused excessive civilian casualties. Meanwhile, commercial drones can be regulated as other aircraft are now: by a combination of the FAA and corporate oversight by their creators  (McFarland 2015) . The handling of such relatively simple robots under liability law will thus be familiar if not straightforward. The above LAWSs examples also resemble how liability law handles non-human animals, which has prompted proposals for robots to be given legal status similar to non-human animals (e.g.,  Kelley et al. 2010) . Suppose someone gets a pet dog and then the dog bites someone, despite the owner trying to stop it. If this person had absolutely no idea the dog would bite someone, then she would not be liable for that bite. However, having seen the dog bite someone, she now knows the dog is a biter, and is now expected to exercise caution with it in the future. If the dog bites again, she can be liable. In legal terms, this is known as having scienterknowledge of the potential harm. Scienter could also apply to LAWSs or other robots that are not expected to cause certain harms. Once the robots are observed causing those harms, their owners or users could be liable for subsequent harms. For comparison, the Google Photos computer system raised controversy in 2015 when it mislabeled photographs of black people as "gorillas"  (Hernandez 2015) . No Google programmer instructed Photos to do this; it was a surprise, arising from the nature of Photos's algorithm. Google acted immediately to apologize and fix Photos. While it did not have scienter for the gorilla incident, it would for any subsequent offenses.  4  The same logic also applies for LAWSs or other types of robots. Again, as long as a human party was responsible for it, a robot does not pose novel challenges to liability law. Even if a human is ultimately liable, a robot could still be taken to court. This would occur, most likely, under in rem jurisdiction, in which the court treats an object of property as a party to a case when it cannot do so with a human owner. In rem cases include United States v.  Fifty-Three Electus Parrots (1982) , in which a human brought parrots from southeast Asia to the U.S. in violation of an animal import law, and United States v. Forty Barrels & Twenty Kegs of  Coca-Cola (1916) , in which the presence of caffeine in the beverage was at issue. In both cases, a human (or corporation) was ultimately considered liable, with the parrots and soda only serving as stand-ins. Robots could be taken to court in the same way, but they would not be considered liable except in a symbolic or proxy fashion. Again, since the robot is not ultimately liable, it poses no novel challenges to liability law. This is not to say that such robots do not pose challenges to liability law-only that these are familiar challenges. Indeed, the nascent literature on robot liability identifies a range of challenges, including assigning liability when robots can be modified by users  (Calo 2011) , when they behave in surprising ways  (Vladeck 2014) , and when the complexity of robot systems makes it difficult to diagnose who is at fault  (Funkhouser 2013; Gurney 2013) . There are also concerns that liability laws could impede the adoption of socially beneficial robotics (e.g.,  Marchant and Lindor 2012; Wu 2016 ). However, these challenges all point to familiar solutions based in various ways of holding manufacturers, users, and other human parties liable. Fine tuning the details is an important and nontrivial task, but it is not a revolutionary one. The familiar nature of typical robots to liability law is further seen in court cases in which robots have been implicated in harms  (Calo 2016 ). An early case is Brouse v. United  States (1949) , in which two airplanes crashed, one of which was a US military plane that was using an early form of autopilot. The court rejected the US claim that it should not be liable because the plane was being controlled by the robotic autopilot; instead the court found that the human pilot in the plane is obligated to pay attention and avoid crashes.  More recently, in Ferguson v. Bombardier Services Corp. (2007) , another airplane crash may have been attributable to the autopilot system, in which case the court would have found the autopilot manufacturer liable, not the autopilot itself, but instead it found that the airline had improperly loaded the plane. (See Calo 2016 for further discussion of these and other cases.)
 II -Robot Liability If a robot can be held liable, then liability law faces some major challenges in terms of which robots to hold liable for which harms, and in terms of how to divide liability between the robot and its human designers, manufacturers, users, etc. In this section, we will argue that robots should be able to be held liable to the extent that they qualify for legal personhood. First, though, let us briefly consider some alternative perspectives. One perspective is that, in an informal sense, any sort of object can be liable for a harm. The pollen in the air is liable for making you sneeze. The faulty gas pipe is liable for burning down your home. The earthquake is liable for destroying the bridge. This is not the sense of liability we address in this paper. Our focus is on legal liability, in which a party can be tried in court. Another perspective comes from the notion that the law ultimately derives from what members of a society want it be. This is why laws are different in different jurisdictions and at different times. From this perspective, robots will be held liable whenever societies decide to hold them liable. There are difficult issues here, such as whether to give robots a say in if they should be held liable.  5  Regardless, the fact that laws are products of societies need not end debate on what laws societies can and should have. To the contrary, it is incumbent upon members of society to have such debates. Within human society, in the United States and many other countries, parties can be held liable for harms to the extent that they qualify as legal persons. Legal personhood is the ability to have legal rights and obligations, such as the ability to enter contracts, sue or be sued, and be held liable for one's actions. Legal liability thus follows directly from legal personhood. Normal adult humans are full legal persons and can be held liable for their actions across a wide range of circumstances. Children, the mentally disabled, and corporations have partial legal personhood, and in turn can be held liable across a narrower range of circumstances. Non-human animals generally do not have personhood, though this status has been contested, especially for nonhuman primates.  6  The denial of legal personhood to non-human animals can be justified on grounds that they lack humans' cognitive sophistication and corresponding ability to participate in society. Such justification avoids charges of speciesism (a pro-human bias for no other reason than just happening to be human). However, the same justification implies that robots should merit legal personhood if they have similar capabilities as humans. As  Hubbard (2011, 417)  puts it, "Absent some strong justification, a denial of personhood to an entity with at least an equal capacity for personhood would be inconsistent and contrary to the egalitarian aspect of liberalism."  7  The question of when robots can be liable thus becomes the question of when robots merit personhood. If robots merit personhood, then they can be held liable for harms they cause. Otherwise, they cannot be held liable, and instead liability must go to some human party, as is the case with non-human animals and other technologies or entities that can cause harm. Hubbard proposes three criteria that a robot or other artificial intelligence should meet to merit personhood: (1) complex intellectual interaction skills, including the ability to communicate and learn from experience; (2) self-consciousness, including the ability to make one's own goals or life plan, and (3) community, meaning the ability to pursue mutual benefits within a group of persons. These three criteria, central to human concepts of personhood, may offer a reasonable standard for robot personhood. We will use these criteria for this paper while emphasizing that their definitude should be a matter of ongoing debate. Do Hubbard's criteria also apply for liability? Perhaps not for the criterion of selfconsciousness. The criterion makes sense for harms caused to a robot: only a conscious robot can experience harms as humans do.  8  This follows from, for example, classic utilitarianism, as in Bentham's line "The question is not, Can they reason? nor, Can they talk? but, Can they suffer?" However, the same logic does not apply to harms caused by a robot. Consider an advanced robot that meets all of Hubbard's criteria except that it lacks consciousness. Suppose the robot causes some harm-and, to be clear, the harm causes suffering to a human or to some other conscious person. Should the robot be held liable? The answer to this may depend on society's foundational reasoning for liability. If liability exists mainly to discourage or deter the commission of harms, then consciousness is unnecessary. The robot should be punished so long as doing so discourages the commission of future harms. The entities that get discouraged here could include the robot, other similar robots, conscious robots, and even humans. It is quite conceivable that non-conscious robots could be punished with some sort of reduced reward or utility as per whatever reward/utility function they might have  (Majot and Yampolskiy 2014) . Specifically, they could be reprogrammed, deactivated or destroyed, or put in what is known as a "Box": digital solitary confinement restricting an AI's ability to communicate or function  (Corwin 2002; Yudkowsky 2002) . To make this possible, however, such robots ought to be based (at least in part) on reinforcement learning or similar computing paradigms (except ones based on neural network algorithms, for reasons we explain later). Alternatively, if liability exists mainly for retribution, to bring justice to whomever committed the harm, then consciousness could be necessary. Whether it is necessary depends on the purpose of the punishment. If the punishment aims to worsen the life of the liable party, so as to "balance things out," then consciousness seems necessary. It makes little sense to "worsen" the life of something that cannot experience the worsening. However, if the punishment aims to satisfy society's sense of justice, then consciousness may be unnecessary. Instead, it could be sufficient that members of society observe the punishment and see justice being served.  9  Whether the robot's consciousness would be necessary in this case would simply depend on whether society's sense of justice requires it to be conscious. This potential exception regarding consciousness is a good example of partial liability as shown in Figure  1 . The advanced, non-conscious robot can be held liable, but not in every case in which normal adult humans could. Specifically, the robot would not be held liable in certain cases where punishment is for retribution. Other limitations to a robot's capabilities could also reduce the extent of its liability. Such robots would be analogous to children and mentally disabled adult humans, who are similarly not held liable for as many cases as normal adult humans are. Robots of less sophistication along any of Hubbard's three criteria (or whatever other criteria are ultimately established) should be liable to a lesser extent than robots that meet the criteria in full. What about robots of greater-than-human sophistication in Hubbard's three criteria? These would be robots with more advanced intellectual interaction skills, self-consciousness, or communal living ability. It is conceivable that such robots could exist-indeed, the idea dates back many decades  (Good 1965 ). If they do come into existence, then by the above logic, they should be held to a higher liability standard than normal adult humans. Indeed, concepts such as negligence recognize human fallibility in many respects that a robot could surpass humans in, including reaction time, eyesight, and mental recall. The potential for holding robots to a higher standard of liability could offer one means of governing robots with greater-than-human capacities; more on this in Section III in the discussion of catastrophic risk. Before turning to catastrophic risk, there is one additional aspect of robot liability to consider: the division of liability among the robot itself and other parties that influence the robot's actions. These other parties can include the robot's designer, its manufacturer, and any users or operators it may have. These parties are comparable to a human's parents and employers, though the comparison is imperfect due to basic differences between humans and robots. One key difference is that robots are to a very large extent designed. Humans can be designed as well via genetic screening and related techniques, hence the term "designer baby." But designers have much more control over the eventual character of robots than they do for humans. This suggests that robot designers should hold more liability for robots' actions than human parents should for their children's actions. If robot designers know that certain designs tend to yield harmful robots, then a case can be made for holding the designers at least partially liable for harms caused by those robots, even if the robots merit legal personhood. Designers could be similarly liable for building robots using opaque algorithms, such as neural networks and related deep learning methods, in which it is difficult to predict in advance whether the robot will cause harm. Those parties that commission the robot's design could be similarly liable. In court, the testimony of relevant industry experts would be valuable for proving whether any available, feasible safeguards to minimize such risks existed. Another difference is that, at least for now, the production of robots is elective, whereas the birthing of humans is required for the continuity of society. Society cannot currently function without humans, but it can function without robots. This fact suggests some lenience for parents in order to encourage procreation, and to be stricter with robot designers in order to safely ease society's transition into an era in which humans and their robot creations coexist. Such a gradual transition seems especially warranted in light of potential robot catastrophe scenarios.
 III -Catastrophic Robot/AI Liability "Catastrophe" has many meanings, many of which require no special legal attention. For example, a person's death is catastrophic for the deceased and her or his loved ones, yet the law is perfectly capable of addressing individual deaths caused by robots or AIs. However, a certain class of extreme catastrophe does merit special legal attention, due to its outsized severity and significance for human civilization. These are catastrophes that cause major, permanent harm to the entirety of global human civilization. Such catastrophes are commonly known as global catastrophes  (Baum and Barrett 2016)  or existential catastrophes  (Bostrom 2013) . Following  Posner (2004) , we will simply call them catastrophes. A range of catastrophic risks exist, including global warming, nuclear war, a pandemic, and collision between Earth and a large asteroid or comet. Recently, a body of scholarship has built up analyzing the possibility of catastrophe from certain types of future AI. Much of the attention has gone to "superintelligent" AI that outsmart humanity and "achieve complete world domination"  (Bostrom 2014, 78 ; see also  Müller 2015) . Such AI could harm humans through the use of robotics. Additionally, some experts believe that robotics could play an important role in the development of such AI  (Baum et al. 2011) . Other catastrophe scenarios could also involve robotics. Robots could be used in the systems for launching nuclear weapons or for detecting incoming attacks, potentially resulting in unwanted nuclear wars.  10  They could be used in critical civil, transportation, or manufacturing infrastructure, contributing to a global systemic failure.  11  They could be used for geoengineering -the intentional manipulation of the global environment, such as to counteract global warming -and this could backfire, causing environmental catastrophe.  12  Robots could be used in establishing or maintaining an oppressive totalitarian world government.  13  Still further robot catastrophe scenarios may also be possible. The enormous scale of the catastrophes in question creates profound moral and legal dilemmas. If the harm is permanent, it impacts members of all future generations, which could be immensely many people. Earth will remain habitable for at least a billion more years, and the galaxy and the universe for much longer  (Baum 2016) ; the present generation thus contains just a tiny fraction of all people who could exist. The legal standing and representation of members of future generations is a difficult question  (Tonn 1996; Wolfe 2008 ). If members of future generations are to be counted, then they can overwhelm the calculus. Despite this, present generations unilaterally make the decisions. There is thus a tension in how to balance the interests of present and future generations  (Page 2003) . A sufficiently large catastrophe raises similar issues even just within the context of the present generation. About seven billion humans live today; a catastrophe that risks killing all of them could be seven billion times larger than a catastrophe that risks killing just one. One could justify enormous effort to reduce that risk regardless of future generations  (Posner 2004) . Further complications come from the irreversible nature of these catastrophes. In a sense, every event is irreversible: if someone wears a blue shirt today, no one can ever change the fact that they wore a blue shirt today. Such events are irreversible only in a trivial sense: you can change what shirt you wear on subsequent days. Nontrivially irreversible events are more or less permanent: if that person should die today, then nothing 14 can bring that person back to life. At a larger scale, nontrivially irreversible effects exist for many ecological shifts and may also exist for the collapse of human civilization  (Baum and Handoh 2014) . The possibility of large and nontrivially irreversible harm creates a major reason to avoid taking certain risks. The precautionary principle is commonly invoked in this context, raising questions of just how cautious to be  (Posner 2004; Sunstein 2006 ). An irreversible AI catastrophe could be too large for liability law to handle. In the simplest case, if the catastrophe results in human extinction, then there would be no one remaining to hold liable. A catastrophe that leaves some survivors but sees the collapse of human civilization would lack the legal system needed for holding people liable. Alternatively, AI could cause a catastrophe in which everyone is still alive but they have become enslaved or otherwise harmed by the AI; in this case the pre-catastrophe human authorities would lack the power needed to hold those at fault liable. For smaller catastrophes, the legal system may exist to a limited extent (Figure  1 ). In this case, it may be possible to bring the liable parties to trial and/or punish them, but not as reliably or completely as is possible under normal circumstances. The closest possible example would be creating special international proceedings, like the Nuremberg Trials, to deal with the aftermath. Much like such war tribunals, though, these may do little to address the chaos' original cause. This would leave victims or society at large wasting time and resources on reliving a tragedy  (McMorran 2013) . Hence, instead of liability, a precautionary approach could be used. This would set a default policy of disallowing any activity with any remote chance of causing catastrophe. It could further place the burden of proof on those who wish to conduct such activity, requiring them to demonstrate in advance that it could not cause catastrophe.  15  Trial-and-error would not be permitted, because a single error could cause major irreversible harm. This would likely be a significant impediment for AI research and development (at least for the subset of AI that poses catastrophic risk), which, like other fields of technology, is likely to make extensive use of trial and error. Indeed, some AI researchers recommend a trial-and-error approach, in which AIs are gradually trained to learn human values so that they will not cause catastrophe  (Goertzel 2016) . However, given the high stakes of AI catastrophe, perhaps these sorts of trial-and-error approaches should still be avoided. It may be possible to use a novel liability scheme to assist with a catastrophe-avoiding precautionary approach. In a wide-ranging discussion of legal measures to avoid catastrophe from emerging technologies,  Wilson (2013, 356)  proposes "liability mechanisms to punish violators whether or not their activities cause any harm". In effect, people would be held liable not for causing catastrophe, but for taking actions that could cause catastrophe. This proposal could be a successful component of a precautionary approach to catastrophic risk and is worth ongoing consideration. Taking the precautionary principle to the extreme can have undesirable consequences. All actions carry some risk. In some cases, it may be impossible to prove a robot does not have the potential to cause catastrophe. Therefore, requiring demonstrations of minimal risk prior to performing actions would be paralyzing  (Sunstein 2006) . Furthermore, many actions can reduce some risks even while increasing others; requiring precaution due to concern about one risk can cause net harm to society by denying opportunities to decrease other risks  (Wiener 2002) . AI research and development can pose significant risks, but it can also help reduce other risks. For AI that poses catastrophic risk, net risk will be minimized when the AI research and development is expected to bring a net reduction in catastrophic risk  (Baum 2014) . In summary, there are significant legal challenges raised by AI that poses catastrophic risk. Liability law, most critically, is of little help. Precautionary approaches can work instead, although care should be taken to avoid preventing AI from reducing different catastrophic risks. The legal challenges from AI that poses catastrophic risk is distinct from the challenges from other types of AI, but they are similar to the challenges from other catastrophic risks.
 Conclusion While robots benefit society in many ways, they also cause or are otherwise implicated in a variety of harms. The frequency and size of these harms is likely to increase as robots become more advanced and ubiquitous. Robots could even cause or contribute to a number of major global catastrophe scenarios. It is important for liability law to successfully govern these harms to the extent possible so that the harms are minimized and, when they do occur, that justice may be served. For many robot harms, a human party is ultimately liable. For these harms, traditional liability law applies. A major challenge to liability law comes when robots could be liable. Such cases require legal personhood tests for robots to assess the extent to which they can be liable. One promising personhood test evaluates the robot's intellectual interaction skills, selfconsciousness, and communal living ability. Depending on how a robot fares on a personhood test, it could have the same liability as, or less or more liability than, a normal adult human. A robot being liable does not preclude a human party also being liable. Indeed, robot designers should expect more liability for robot harms than would human parents, because robots are designed so much more extensively than human children are. Finally, for robots that pose catastrophic risk, liability law cannot be counted on and a precautionary approach is warranted. People involved in the design, manufacture, and use of robots can limit their liability by choosing robots that reliably avoid harms. One potential way to improve reliability is to avoid computing paradigms such as neural nets that tend to result in surprising behaviors, or adapt these paradigms to make them less surprising  (Huang and Xing 2002) . Robot designs should be sufficiently transparent that the responsible human parties can, with reasonable confidence, determine in advance what harms could occur. They can then build safety restrictions into the robot or at least give warnings to robot users, as is common practice with other technologies. Robots should also go through rigorous safety testing before being placed into situations where they can cause harms. If robots cannot reliably avoid harms, then they probably should not be used in the first place. These sorts of safety guidelines should be especially strict for robots that could contribute to major global catastrophe. A single catastrophe could permanently harm human civilization. It is thus crucial to avoid any catastrophe. Safety testing itself could be dangerous. This increases the value of transparent computing paradigms that let humans assess risks prior to building the robot. Legal measures must also take effect prior to the robot's build because there may be no legal system afterwards. Advanced robots may be less likely to cause catastrophe if they are designed to be upstanding legal persons. But even then, some legal system would need to exist to hold them liable for what harms they cause. As this paper illustrates, robot liability poses major new challenges to liability law. Meeting these challenges requires contributions from law, robotics, philosophy, risk analysis, and other fields. It is essential for humans with these various specialties to work together to build robot liability regimes that avoid harms while capturing the many benefits of robotics. The potential for harm is extremely large, making this an urgent task. We hope that humans and robots will coexist successfully and for mutual benefit in a community of responsible persons. Figure 1 . 1 Figure 1. Classification scheme for the applicability of liability law to various sizes of harms caused by various types of robots.

Abstract: Advances in robotics technology are causing major changes in manufacturing, transportation, medicine, and a number of other sectors. While many of these changes are beneficial, there will inevitably be some harms. Who or what is liable when a robot causes harm? This paper addresses how liability law can and should account for robots, including robots that exist today and robots that potentially could be built at some point in the near or distant future. Already, robots have been implicated in a variety of harms. However, current and near-future robots pose no significant challenge for liability law: they can be readily handled with existing liability law or minor variations thereof. We show this through examples from medical technology, drones, and consumer robotics. A greater challenge will arise if it becomes possible to build robots that merit legal personhood and thus can be held liable. Liability law for robot persons could draw on certain precedents, such as animal liability. However, legal innovations will be needed, in particular for determining which robots merit legal personhood. Finally, a major challenge comes from the possibility of future robots that could cause major global catastrophe. As with other global catastrophic risks, liability law could not apply, because there would be no postcatastrophe legal system to impose liability. Instead, law must be based on pre-catastrophe precautionary measures.


Abstract: Maximizing uncertainty. Following Lakshminarayanan  et al. (2017), we measure ensemble disagreement using the average KL-divergence between the output of a single ensemble member and the ensemble mean, where p ✓ is the reward classifier defined in Section 3.2. Maximizing novelty. In this work, we use a distance function that computes the Euclidean distance between state embeddings, where f is the state encoder trained in step (1).


Implementation of Moral Uncertainty in Intelligent Machines

Introduction Advances in artificial intelligence have led to research into methods by which sufficiently intelligent systems can be guaranteed to follow ethically defensible behavior. Successful implementation of moral reasoning may be critical for managing the proliferation of autonomous vehicles, workers, weapons, and other systems as they increase in intelligence and complexity. Approaches towards moral decision making generally fall into two camps, "topdown" and "bottom-up" approaches  (Allen et al. 2005) . Top-down morality is the explicit implementation of decision rules into artificial agents. Schemes for top-down decision making that have been proposed for intelligent machines include Kantian deontology  (Arkoudas et al. 2005 ) and preference utilitarianism  (Oesterheld 2016) . Bottom-up morality avoids reference to specific moral theories by developing systems that can implicitly learn to distinguish between moral and immoral behaviors, such as cognitive architectures designed to mimic human intuitions  (Bello and Bringsjord 2012) . There are also hybrid approaches which merge insights from the two frameworks, such as one given by  Wiltshire (2015) .
 The Problem of Moral Disagreement A problem which has been cited as an obstacle to the development of machine ethics is the lack of agreement among moral philosophers on which theory of ethics should be followed  (Shulman et al. 2009; Bello and Bringsjord 2012; Bostrom 2014; Brundage 2014 ). The 2009 PhilPapers survey of philosophy faculty revealed that 26% accepted or leaned towards deontology, 24% accepted or leaned towards consequentialism, 18% accepted or leaned towards virtue ethics, and the remaining 32% favored other approaches entirely. For someone who believes in a particular approach to ethics, the correct system for implementation in artificial moral agents may be obvious, but the path forward for institutions and society as a whole remains unclear. Companies, governments, and researchers will have to decide which system to use for artificial agents and will be faced with a difficult choice between competing moral paradigms. This paper will describe a computational framework that will determine how we should design moral machines in the light of this disagreement, but first it is necessary to determine exactly why moral disagreement is a problem. Just because there exists disagreement does not necessarily imply that research and development programs cannot succeed in their objectives, nor does it necessarily imply that we need to worry about differences in opinion. The aforementioned works do not explicitly describe how moral disagreement would prevent us from building satisfactory moral machines, so we will need to clarify the nature of the problem before proceeding with a solution. There are two plausible reasons for why moral disagreement can pose a serious problem for the project of machine ethics. First, it could be a pragmatic problem as disagreements among engineers, policymakers and philosophers interfere with projects that require cooperation. In a worst-case scenario, decisions over which research programs to fund will turn into bitter ideological battles, research agencies will become bogged down in disputes, and developers will split up their resources and devolve into a competitive mindset which reduces information sharing and slows research progress, thus making it more difficult to construct moral machines. Second, it could be a moral problem. If we are so uncertain about morality and are split across many different moral principles, then the likelihood that anyone's particular moral system is entirely correct is statistically extremely low  (Shulman et al. 2009) . Therefore, if we do build AIs grounded in any particular moral system, then they will probably be making many poor moral decisions.  Bello and Bringsjord (2012)  argue that moral disagreement provides a reason to avoid top-down approaches to machine ethics in favor of bottom-up or hybrid approaches that copy or take inspiration from human moral thinking. However, it is not clear how this would solve the problem of disagreement. The authors claim that people can agree on examples of good moral behavior despite disagreeing over specific theories, but people's personal moral judgements can differ as widely as moral theories do when faced with moral dilemmas  (Greene et al. 2001 ) and when they are considering politicized moral issues such as racial fairness, animal farming, and economic inequality. Therefore, even if we attempt to circumvent normative disputes by eschewing top-down ethics in favor of bottom-up frameworks, it is perfectly plausible that researchers and government regulators with strong value disagreements will come into conflicts over the kinds of moral intuitions and data which should or shouldn't play a role in automated reasoning.  Brundage (2014)  also notes that there is no reason to expect that Bello and Bringsjord's approach would actually be reliably ethical. Moreover, in the cases where people do agree on moral choices despite disagreeing over moral theories, there is neither a pragmatic nor a philosophical problem with the top-down approach: a utilitarian engineer has no moral reason to care whether a robot's computational pattern is internally utilitarian or not, as long as it is actually maximizing utility; more generally, if we know that machines will act in the morally right way, then we have no reason to consider the specific programming of machine code to be morally significant. So to the extent that moral disagreements between ethical theories might pose pragmatic and moral problems for the development of top-down computational ethics, they would pose equal pragmatic and moral problems for the development of bottom-up computational ethics.
 An Example of Intractable Disagreement We can illustrate this problem with a case of self-driving cars which must be programmed to decide whether to swerve to avoid animals in the path of the vehicle. Various vehicle speeds, warning times, road shapes, weather conditions and other factors can make the decision to avoid animals more or less dangerous to the occupants of the vehicle in different situations, and different moral theories can mandate different choices in such cases. One moral principle which could specify when vehicles should or should not swerve could be described as the commensurable view. Under this doctrine, animal lives should be treated as if they are worth some small fraction of what human lives are worth. In addition, let us suppose that animals with a presumed greater capacity for cognition and pain sensations should be granted a greater weight in such tradeoffs. The machine, upon calculating a sufficiently low estimate of the probability that swerving would result in human fatality or verifying some heuristical checklist of features of the situation, would avoid the animal. The commensurable view is most clearly reminiscent of utilitarianism, but other moral theories may embrace it nonetheless-for instance, one could follow a mixed theory incorporating both deontological and consequentialist ethics and believe that consequentialist considerations dominate in this particular kind of situation. Vehicles acting in this manner can be statistically expected to eventually cause a few human fatalities but also to save many animals. If two people agreed on the commensurable view but disagreed on the specific tradeoff, compromise would be relatively straightforward. For instance, if one side argued that the chance of a fatal accident would have to be under 1 in 1000 in order to swerve for a certain animal while another side argued that it should be 1 in 10,000, they could plausibly compromise at 1 in 5500, which would function as a pragmatic truce and a limit to the degree of moral harm perceived by either viewpoint. The more pressing concern is how to compromise with the incommensurable view-that humans can never be placed at additional risk of fatality in order to save animals. This might be supported by moral theories emphasizing human rights and liberty. Since this view fundamentally rejects the axiological assumptions of the other and holds that no tradeoff is permissible, there is no obvious 'halfway point' where the competing principles can meet. Unilaterally selecting one principle to govern vehicle behavior would not be acceptable for the reasons described previously. Training the driving system to determine the correct action through supervised learning would only rephrase the issue by forcing us to decide which training examples ought to be classified as right or wrong. Allowing the company to decide based on its own interests would not clearly lead to safe and ethical behavior. Laws do not provide sufficient guidance in all such situations to govern the behavior of machines, since they are often left to the conscience of a human agent. Finally, human operators cannot be guaranteed to be available to make such quick decisions, to reliably specify their preferences beforehand, or to act ethically at all. The last resort would be for the vehicles to rely on a random number generator and follow the commensurable view half of the time and the incommensurable view half of the time. At first glance, this is the only straightforward way to build a system in a way that does not overwhelmingly disfavor one particular theory: if we cannot compromise between two views on how to act in a particular repeated decision by generating a new view, then the only way to act fairly is to alternate between them. Unfortunately, this sacrifices predictability by inducing stochasticity into decision making. Furthermore, it is not Pareto optimal. A vehicle which was programmed to randomly select the commensurable view in half and only half of relevant situations could be modified so that it specifically did so for the half of situations where the animal under threat was relatively important under the commensurable view. This makes it more predictable, since its decisions are deterministic given the environments that it encounters. It also counts as a moral improvement by the lights of the commensurable view, because it prioritizes the animals which are regarded as more important. At the same time, it is no worse according to the incommensurable view, which could be indifferent to exactly which animals are saved. This makes the change a Pareto improvement. We can generalize this strategy by saying that we are keeping all moral theories in play at the time of decision making, and searching for actions which provide the most value across all of them, rather than seeking to select one moral theory and ignoring others. This procedure, however ad hoc, seems to be the best possible way of approaching this particular case, and its principles could be fruitfully applied to other cases where moral values are not just in disagreement but wholly incommensurable. Ideally, we should have a rigorous theoretical framework which does similar work in the broad variety of moral situations by taking all moral preferences into consideration and responding appropriately.
 Dealing with Moral Disagreement Properly Moral disagreement has been recognized as a problem for normative ethics proper, independent of any concerns regarding machine ethics. A direct approach to overcoming it is to assume that there is a correct moral theory which we are searching for, acknowledge that we are fundamentally uncertain about which moral theory is correct, and then act in such a way as to give some weight to the judgements of different theories. More generally, given the failure of moral philosophy to reach satisfactory conclusions, it can be argued that we should adopt a framework of reasoning which takes multiple moral views into account  (Lockhart 2000) . However, the idea that we ought to change our behavior in accordance with moral uncertainty is controversial, and many philosophers have attempted to address the various mathematical and philosophical problems involved with the concept (Z ˙uradzki 2016). In particular, it has been argued that making comparisons between the values and judgements of different moral theories is impossible  (Nissan-Rozen 2015) . A recent proposal aimed at answering these concerns was developed by William MacAskill in an extensive thesis (2014). It avoids objections based on incomparability and incommensurability by developing moral uncertainty as a voting problem among moral theories (MacAskill 2016). Voting provides a close analogy with the pragmatic problems faced by machine ethics: as voting is the general process by which decisions are made from the preferences of a population, computations of moral uncertainty represent a process by which agents can act in accordance with the diverse values of humanity. Since MacAskill's proposal works as a voting system where theories have equal say adjusted by their probability of being correct, it also approximates the framework which has been suggested for meta-moral reasoning by Nick  Bostrom (2009) . The rest of this paper will develop and defend the use of this model for machine ethics.
 Maximizing Expected Choiceworthiness The scheme presented in "Normative Uncertainty" (MacAskill 2014) is to make action-guiding judgements based on all moral theories in which the agent has some level of credence. MacAskill defines his approach as a metanormative theory, characterized as follows: The agent investigates a decision-situation comprising a quintuple 〈S, t, A, T, C〉 where S is the decision maker, t is the time, and A is the set of possible actions to take. T is the set of normative theories being considered, where a theory T i is a function of decision-situations that produces a cardinal or ordinal choiceworthiness score for actions CW i (A) for all actions a ∈ A. C(T i ) is a credence function assigning values in [0, 1] to every T i ∈ T. A metanormative theory is a function of decision-situations that produces an ordering of the actions in A in terms of their appropriateness. MacAskill distinguishes between moral theories which assign cardinal rankings to options, as utilitarianism would, and moral theories which assign ordinal rankings. When it comes to cardinal theories, MacAskill also distinguishes between sets of theories with moral values which are directly comparable and other cardinal theories which are incomparable with any other theory. He proposes the metanormative theory of maximizing expected choiceworthiness (MEC), the steps of which are as follows: 1. Each set K of k moral theories in which the choiceworthiness rankings of options are cardinal and intertheoretically comparable are aggregated into single theories, where C T K ð Þ ¼ X k i¼1 CðT i Þ CW K A ð Þ ¼ P k i¼1 CW i A ð ÞC T i ð Þ P k i¼1 C T i ð Þ In other words, the credence in the new theory equals the sum of the agent's credences in each of the individual theories in the set, and the choiceworthiness of an option according to the new theory is the credence-weighted average of the choiceworthiness of the option in all of the individual theories. 2. The rankings of options according to each ordinal theory o are used to provide choiceworthiness scores of options using a modified Borda scoring rule that is designed to properly account for ties, denoted here as CW B . The score for an option is equal to the number of worse options minus the number of better options. CW B o A ð Þ ¼ a 2 A : CW o a ð Þ\CW o A ð Þ j j À a 2 A : CW o a ð Þ j i CW o A ð Þj This violates the independence of irrelevant alternatives, but MacAskill provides reasons to allow the violation. First, independence of irrelevant alternatives is the least essential of all the axioms present in Arrow's Impossibility Theorem, and if a different axiom were to be violated then there would be no prospects for a satisfactory metanormative account involving ordinal moral theories. Second, the primary motivation for independence of irrelevant alternatives is that it combats tactical voting, but tactical voting is not a problem with moral theories: theories aren't agents, and a moral agent can't conceal information from itself. Third, there are some moral cases where we would expect the independence of irrelevant alternatives to be violated (MacAskill 2014, p. 85). 3. The options' scores provided by each of the aggregated sets of cardinal comparable theories, the scores provided by each of the ordinal theories, and the scores provided by each of the cardinal incomparable theories p are all divided by the respective standard deviations of the moral rankings which the moral theories (or sets of intertheoretically comparable cardinal theories) provide over a general set G of actions. This provides normalized choiceworthiness rankings and is the only possible way of equalizing the value of voting for each value system (Cotton-Barratt 2013), which (as MacAskill argues, p. 115) performs the role of giving each theory equal say. CW N K ðAÞ ¼ CW K ðAÞ r CW K ðGÞ ð Þ CW N o ðAÞ ¼ CW B o ðAÞ r CW B o G ð Þ À Á CW N p A ð Þ ¼ CW p A ð Þ r CW p G ð Þ À Á A note on G: it is necessary for variance normalization that a representative set of actions (the "general set") be defined for computing the variance. This provides the background against which theories can describe whether a decision is comparatively important or comparatively unimportant from the point of view of a particular theory. MacAskill thinks that a broad account featuring many actions in this set, rather than just the ones under consideration in the present decision-situation, is theoretically desirable. Here I follow suit and model the process as if we are providing moral machines with large arrays of data representing many possible moral decisions, so that calculating the variance only needs to be done once before the resulting number is used and shared by many agents. Variance normalization also violates the independence of irrelevant alternatives, but it is a necessary violation for this application. In order to perform comparisons across moral theories, we need to determine the amount of weight which a value system places on a particular issue compared to other issues. Since different computational approaches to ethics can have wildly differing numerical outputs, it would be impossible to do this fairly without normalizing. 4. Each option is scored with the credence-weighted average of the variancenormalized scores provided by the normative theories. This provides a ranking of expected choiceworthiness: CW E A ð Þ ¼ X n i¼1 CW N i A ð ÞC T i ð Þ Then the agent chooses the action A which maximizes CW E (A). The full description and defense of each step in the theory, provided as a model for human decision making, is given by MacAskill (2014). The computational complexity of an algorithm implementing this procedure is OðjAj Ã jTjÞ, as adding another action requires a fixed series of evaluations to be done by each moral theory and adding another moral theory requires it to perform a fixed series of evaluations to be performed on each action. The calculation itself is very simple; almost all of the time and memory requirements will presumably stem from the computations of the moral theories themselves.
 Motivation for Maximizing Expected Choiceworthiness in Artificial Intelligence The effect of the system described above is that an agent will make prudent decisions that aim to take the most broadly desirable action in accordance with various people's values, and will avoid the very counterintuitive actions which are considered to be problems for various moral theories. For instance, when presented with the opportunity to commit a great deception in order to gain a small increment of happiness, an agent will refrain even if they view consequentialism as more likely than nonconsequentialist theories, because the relative wrongfulness of the act according to nonconsequentialist theories is significant whereas the relative benefit under consequentialism is minor. Likewise, even if an agent believes that it is probably not morally obligated to refrain from actions which cause significant indirect harms to others, it will nevertheless refrain from being careless in such a way, because the scale of the moral harms it would be committing if it were wrong would be significant. This will limit the degree to which machines will behave in ways which are considered very objectionable by minority views. This does not change the fact that a machine which maximized expected choiceworthiness could still be expected to take many actions which various parties consider to be morally wrong. Despite its superficial similarity to maximizing consequentialism, it would often sacrifice good consequences in favor of satisfying the values of other moral theories. So the adherents of most moral theories would theoretically prefer that machines be built according to their own specifications. But they cannot realistically expect the rest of industry and society to agree with them. So the actual outcome of widespread adherence to naı ¨ve individually rational behavior would be a patchwork of competing agents with different values. Not only would this leave both the pragmatic and the moral problems of disagreement unresolved, but it would leave parties with little ability to influence the world except for whatever agents they could develop as representatives for their own values. However, if parties were to agree on developing systems based upon moral uncertainty, their most critical respective interests would be given relatively high weight in all moral machines by the mechanism of variance voting. By reducing conflicts between agents, promoting cooperative machine development, and allocating value systems' decision making power to the situations which they care most about, a universal framework of moral uncertainty would solve the multiagent prisoner's dilemma constituted by machine ethics and would generally lead to a better outcome for different value systems than the situation in which there was pure disagreement. Needless to say, if an individual did accept MacAskill's thesis and believed that we ought to maximize expected choiceworthiness as human agents, then she would have a direct and compelling reason to support the implementation of the theory in computational systems. But the argument for developing machines based on maximizing expected choiceworthiness is not dependent upon the claim that we humans actually have metanormative reasons to act in such a way. An analogy with voting is illuminating: very few people believe that the winner of an election is always the one who would be the best leader, but most people agree with the system, as it is a necessary and effective compromise. So as long as we disagree with each other about ethics, we should still agree to construct moral machines based on uncertainty even if we reject the idea as a guide for human behavior.
 The Long Term Future If we look at the long run future of artificial intelligence development, there are additional reasons for supporting a framework of acting in accordance with moral uncertainty. First, if an individual is confident in her moral beliefs and possesses an expectation of progress towards objective truth in moral philosophy, then she should expect morally uncertain agents which reflect human knowledge to converge towards her beliefs in the long run. This will limit the degree to which she can expect their behaviors to be objectionable relative to those of an ideal agent which has the right views from the outset. Second, sufficiently competent machines which adhere to a particular value function are likely to pursue it endlessly with pathological consequences  (Bostrom 2012) . While this paper is not intended as a complete solution for value selection and alignment in arbitrarily intelligent agents, it is nevertheless valuable to start by designing systems that cooperate among different value systems, as that will improve the fail-safety of agents which become very intelligent but have imperfect goal functions  (Gloor 2016) . The framework proposed here clearly fulfills that criterion, and therefore serves as a potential backup or other component of a multilayered utility function even if it is not ordinarily utilized  (Oesterheld 2016) . Third, as machines become more intelligent, they might perform more moral functions autonomously, such as updating credences in moral theories and even generating new ones, depending on the degree to which progress in artificial intelligence enables them to investigate questions in moral philosophy. MacAskill (2014, ch. 7) points out that a more accurate credence function improves the expected choiceworthiness of an MEC agent's actions, so such an agent would be intrinsically motivated to expend effort into updating and improving their moral beliefs if they had acceptable criteria for determining moral truth. This is particularly desirable since there may be cases of widespread moral wrongdoing which most humans have consistently failed to identify  (Williams 2015) , which machines should be incentivized to avoid.
 How to Build an Uncertain Machine
 Ordinal and Cardinal Scoring One of the requirements of this framework is that moral theories actually provide rankings over actions. Perhaps the majority of moral theories provide neither cardinal nor ordinal rankings at first glance. However, this is a surmountable issue. First, it simply seems obvious that just about any moral theory should be able to rank certain impermissible actions, such as torturing a large number of people, as worse than other impermissible actions, like stealing someone's coffee, even if such rankings have not yet been made explicit in the philosophical literature. Second, any moral theory which judges the actions of machines can at least provide a rudimentary score system rooted in deontic logic, with 0 for all impermissible actions and 1 for all permissible actions, even if nothing else about the theory is formalized. So any moral theory must at least be translatable to a basic two-level ordinal ranking over actions, which would enable it to be implemented in this framework. If such an implementation is too coarse to adequately represent the values of such a theory, then its proponents have the option and the incentive to clarify and improve it. Defining precise scores could require varying degrees of input from humans depending on the complexity of the moral theory. However, computational approaches towards morality may provide numerical values and rankings which are otherwise absent from moral theory. Computational reasoning differs from human thinking and the procedures required for providing moral judgements in artificial intelligence systems may involve functions from which meaningful ordinal, integer or real-valued scores for actions can be extracted. Scores could be derived from the degree of rightfulness or wrongfulness of an action and/or the degree of certainty given by a moral algorithm as to whether an action is right or wrong. While the philosophical theory of MEC may seem to only refer to rigidly defined moral theories, a top-down approach to machine ethics is not required with this proposal for machine ethics. There is no computational necessity for the 'theories' implemented in the system to all be explicit moral theories in the philosophical sense. One or more bottom-up decision making systems might be included; for instance, the output of a complicated learned function intended to model human intuitions can be treated as a choiceworthiness ranking. As long as it ranks actions by normative criteria and has some probability of being the correct way for a machine to behave, it satisfies the requirements proposed by MacAskill for being a moral theory.
 Credences The outcomes of comparisons will always crucially depend upon the credence function C-an assignment of probabilities to various moral theories of being correct. There are multiple ways of determining these values. Lengthy and difficult approaches which rely on extensive human input are acceptable, as information modifying the credence function could be generated just once and then distributed to many agents as an update. Artificial agents with humanlike intelligence could presumably reason about morality for themselves, but this is not an option for the foreseeable future. One option is to make credences which reflect the beliefs of moral philosophers. The credence assigned to a moral theory would then be equivalent to the proportion of philosophers who affirm the theory, or possibly a score computed from a more sophisticated set of votes taking philosophers' full credences and rankings into account. However, the assumption that moral philosophers are in fact authorities regarding moral philosophy in the same way that practitioners of other disciplines are experts in their respective domains is highly controversial, and has been attacked by  Archard (2011) ,  Cross (2016)  and others. Moreover, the type of moral expertise which provides judgements about which actions and theories are right or wrong is merely one type of moral expertise which may not hold even if others do  (Driver 2014; Jones and Schroeter 2012) . Finally, the judgements of moral philosophers have been shown to be vulnerable to systematic cognitive bias  (Schultz et al. 2011; Schwitzgebel and Cushman 2012) . However, there are a few differences when it comes to the case of assigning credences to a system of maximizing expected choiceworthiness in machine intelligence on the basis of a broad survey of moral philosophers' beliefs. First, the system is not deferring to the views of any particular philosopher or group of philosophers. Rather, it is making decisions based upon all philosophers' views in accordance with how prevalent they are; the views of disagreeing philosophers are being used to support the proposition that both views are approximately equally likely to be correct, rather than the proposition that either particular view happens to be correct. Therefore, the argument from disagreement as presented by  Cross (2016)  against a strong conception of moral expertise does not apply to this case. Second, the system of decision making presented here is specifically designed for use by artificial agents rather than humans. Artificial intelligence systems are not voting participants of democracy and cannot be described as flourishing, so to whatever extent there is moral expertise, it is not wrong for machines to defer to it, as  Archard (2011)  argues is the case for humans. The above points leave many of the arguments against moral expertise unaddressed, and resolving that debate is beyond the scope of this paper. But given that we do not possess artificial general intelligence capable of comprehending and judging moral theory, the practical alternative to deferring to moral philosophers is not for agents to make choices for themselves, which is the context for the ordinary debate on moral expertise, but it is to defer to the judgements of other members of society besides philosophers. But non-philosophers would necessarily be making moral judgements in the same context of controversy over the nature and status of moral knowledge, and they would presumably be vulnerable to cognitive bias just as professional philosophers are, so any claim for their expertise would be the target of arguments similar to those leveled against professional philosophers. So there is a prima facie case for deferring to moral philosophers for determining machine credences which is stronger than the case for deferring to moral philosophers for making moral judgements in ordinary contexts, and no fundamental reason to defer to non-philosophers for determining credences. However, there may be contingent reasons to defer to non-philosophers based on various traits and characteristics commonly possessed by philosophers and non-philosophers. First, moral expertise might exist in the form of practical experience and wisdom rather than any kind of academic knowledge  (Jones and Schroeter 2012) . If individuals with particular virtues, experiences, and character traits are regarded as moral authorities, then credences should be assigned in a way that reflects how they act in different situations. Second, even if being a philosopher counts as a reason in favor of attributing moral expertise, our available community of philosophers could be a predictably flawed group to defer to. One reason for this is that they are not representative of the broader human population, whether measured by socioeconomic status, race, gender, nationality, personality type, political ideology, or other characteristics which are often correlated with different opinions about morality. But the most reasonable solution to this problem is to weight the votes of philosophers, or virtuous wise people as proposed above, on the basis of the uniqueness of the various backgrounds and perspectives which they represent, since it preserves the presence of a community with (arguable) moral expertise while functionally acting similarly to a hypothetical group of philosophers with an ideal composition from various demographic groups. Unfortunately it is not obvious what composition of judges would be appropriate for determining credences. For instance, the fact that orphans comprise 2% of people in the world does not necessarily imply that a voting body comprised of 2% orphans and 98% nonorphans will have more accurate views on ethics than any other mix. It may be the case that both groups of people have equally sound perspectives which must be weighed against each other, and so each orphan's vote should have forty-nine times as much weight as a nonorphan's vote. Alternatively, suppose for the purpose of the example that nonorphans tend to make more accurate moral claims on average for whatever reason. Then only the votes of nonorphans might be given significant weight. Or a combination of these and other factors could imply anything in between. This lack of clarity does not imply that we should just weight all votes equally and ignore concerns about diverse representation, but it shows that the selection of an optimal standard is likely to be more difficult than it naively appears. Instead of trying to determine a single universal method of assigning credences, different kinds of artificial agents may have credences assigned differently to suit the contexts in which they operate, just as jury selection methods vary in order to ensure fairness in different types of legal cases. Therefore, fully specifying a method of determining credences is beyond the scope of this paper. Also, while complete data on potential credence judges' moral views and other characteristics to use for this project may currently be lacking, this does not present a great barrier for creating morally uncertain artificial agents, since the credence function can be used broadly and need not be computed separately and repeatedly for too many scenarios. Specialized focus groups, panels, surveys and other activities can be run occasionally at low cost to the overall industry. In one respect, the current lack of relevant data is actually a good thing: the selection of a method of assigning credences should be performed from a state of as much ignorance as possible regarding the particular moral credences which will be delivered by that method, in order to ensure that the decision making process is unbiased and focuses on the prior philosophical reasons for or against the method of assigning a credence function rather than reflecting whatever moral credences we have in the first place.
 Limited Approaches Deploying this system of normative uncertainty would require multiple moral decision making systems to function in a moral agent. An implementation could be quite simple, such as a self-driving car programmed with one module for strict traffic laws and another module comprised by a basic utilitarian calculator for death, injuries, costs, and travel time. Other applications could warrant more complexity, and implementing a large number of moral theories may be too computationally expensive for the framework to be practical in all applications. Basic systems of value comparisons involving computational shortcuts and heuristics would be easier to implement, and may also seem to be more philosophically defensible by dint of being simpler. In reality, they will have to sacrifice some theoretical properties in order to achieve this simplicity, such as equal say among theories, accurate representation of moral values, or important axioms of voting theory as dictated by Arrow's Impossibility Theorem. The pragmatic and moral problems of disagreement must generally be addressed for machine ethics to be successful, so comparisons and decisions between value systems will still have to be made in some fashion even if some machines must use simplified approaches to save on computational resources. But MEC represents one of the most recent and philosophically rigorous methods for adjudicating among different value systems, and has desirable properties including equal say, comparability across disparate moral theories, and graceful inclusion of minority values. Therefore, the system here can serve as the theoretical standard used for the development and judgement of heuristic approaches.
 Objections
 Infinite Regress The problem of moral disagreement poses a significant problem for the project of developing intelligent machines. But critics will point out that the exact method of implementing a system of moral uncertainty is controversial in its own right, as evidenced by the issue of credences. Therefore, we have not actually eliminated disagreement, but we have merely removed it from the moral domain to the metamoral domain-where pragmatic and normative problems still arise. People may disagree over what framework of moral uncertainty to use, how to assign credences, how to score options, how to construct the general set G, or other issues. Any framework for compromising over these disputes will also be subject to disagreement, prompting further disputes, and so on ad infinitum. It is true that we do not know if the system of moral uncertainty presented by MacAskill would lead to the morally best outcome, and the answer may be different depending on whose value system is actually correct. In fact, since it makes compromises among moral claims, it is almost guaranteed to commit moral errors at some point. However, given our lack of knowledge about morality, it represents a solid best guess that maximizes the expected choice-worthiness of agents' actions. The only alternatives are to unilaterally assume a single value system (whether topdown or bottom-up) with large risks of controversy and moral failure, or to choose a method of meta-moral compromise which sacrifices some of the desirable attributes of MEC while failing to avoid the pitfalls mentioned here. There is simply no better option given our limited understanding of morality. While there is also potential for a pragmatic problem to arise, I claim that it is generally less severe in the meta-moral case of disagreement than it is in the moral case. This is evidenced by the relative lack of controversy over voting methods and procedures in organizations and political institutions. While there are efforts to change voting standards in Western democracies such as the United States, such as extending the vote to younger persons or felons, these campaigns tend to be less popular and less violent than object-level campaigns over contentious ideological and policy disputes. In machine ethics, while different people may advocate for different systems of moral uncertainty, it won't be clear which one would best achieve anyone's particular value system, and disputes over core moral principles will be overshadowed by more mundane issues like fairness, computational complexity and the axioms of voting theory. Certainly there are large challenges ahead, but they are surmountable in ways that pure moral disagreement is not. This is because engineers and philosophers can appeal to more universal standards of computational and philosophical theory than they could if they were vouching for specific viewpoints. In any case, there is a limit to how narrow a difference of opinion can be before it stops being a major barrier to unified action. Flawed artificial moral agents can still provide major benefits for the world, so the lack of an uncontroversial standard for filling out the MEC procedure should not stop us from building moral machines at all any more than the predictable moral fallibility of humans and organizations should stop us from engaging in our practices of procreation and entrepreneurship.
 Pragmatism Versus Morality Hardheaded economists may point out that machines which always do the morally optimal thing are not going to be as commercially viable as ones which give some special leeway to the preferences and interests of their owners. This problem is more serious with agents following MEC than it is with ordinary moral agents: while most moral theories allow some space where multiple actions are regarded as permissible, the disjunction of moral theories does not. With multiple moral theories expressing interest in various values and the metanormative module calculating cardinal scores for all actions, the smallest preference over two actions held by a single theory would compel the agent to act in a particular way even if every other theory was indifferent, providing little control to users with nonmoral interests. Furthermore, some moral theories pose demands which contradict certain activities of legal commercial enterprises. This leaves little space for decisions to be made in the interests of the designers and users of the agent. For instance, depending on the moral beliefs of artificial agents, an automated cargo ship might divert its cargo to a desperately poor region of the world instead of fulfilling its contract, a personal care robot might zip off for several hours a day to work at a local food bank instead of giving a massage, or a financial trading algorithm might manipulate and ruin the stock price of a company which is engaging in harmful business conduct instead of turning a profit. The objection is not that we should necessarily regard these actions as bad; in fact it presupposes that such actions would be considered morally good by many theories, and the tension between one's own values and society's values has already been extensively addressed in this paper. Rather, the point is that maximally moral machines would not necessarily be regarded as desirable for commercial and industrial investment, meaning that fewer would be built. In the long run, this limits the amount of research and development which would lead to generally smarter and more beneficial machines. If we let the perfect be the enemy of the good then we will be responsible for the immoral outcomes of stunted technological progress and failure to mitigate human tragedies as quickly as we could. Therefore agents should only be obligated to satisfice expected choiceworthiness, or perhaps the set of pragmatic interests of the owners should function as a 'theory' of its own. The flaw in this objection is the assumption that moral theories do not similarly value predictability, pragmatism and the long run future of machine intelligence. For instance, rule consequentialism would clearly support the idea that agents ought to fulfill contracts and promises if it led to better consequences in the form of long run economic and technological growth. Act consequentialism may do so as well, if audacious moral behavior would be likely to cause significant legal and economic costs impairing future AI development. The decision to commit to a certain policy or procedure could also be classified as an action in itself if doing so yielded moral benefits, allowing act consequentialist theories to meet the same criteria as rule consequentialist ones. Furthermore, ethics based on the principle of telos would hold that machines designed to perform a specific purpose ought to give some priority to that purpose. There are also secondary considerations for preserving a dominating focus on moral behavior. First, explicitly placing pragmatic interests on the same level as moral theories potentially leads to a slippery slope of accepting progressively less ethical behavior while opening up yet another space for controversy and disputes over the details of such tradeoffs, but having a clear rule for maximizing expected choiceworthiness is a policy which can be committed to and accepted on principled ideological grounds. Second, public trust in artificial intelligence is fragile and rests on crucial questions of whether they are safe and ethical. Stories of immoral robot behavior are likely to have negative ramifications as the public turns cold towards machine research, development and production, and these negative ramifications should be entered into the judgements of moral theories from the outset rather than being avoided by introducing bias at the metanormative level.
 Against Maximin The desire to protect minority rights and interests may lead one to propose that the framework of MEC be replaced with a maximin criterion or some weaker lossaverse function. Simple versions of consequentialism, which maximize expected value on the normative rather than the metanormative level, are often criticized for implying that it may be right to sacrifice the lives and interests of individuals for the greater good. In a similar vein, an autonomous system which maximized expected choiceworthiness could be expected to take actions which are very wrong according to one set of values but very good according to other values, and some may find this problematic. A maximin or loss-averse function would respectively prevent or resist such scenarios. However, this intuition is mistaken in the metanormative context because counterintuitive cases of minority interests being violated are wrong on the ethical grounds of many different moral theories. For instance, killing an innocent patient in order to reuse their organs is not only considered wrong by deontological ethics, but would likely be considered wrong by rule consequentialism, virtue ethics, intuitionist ethics and other common theories as well, and always to a high degree. So in scenarios involving possible patient outcomes and organ harvesting, MEC would demand that the more common interest of refraining from harm be adhered to, while a maximin or loss-averse function would perversely give extra weight to the sidelined act consequentialist calculations over and above the degree to which act consequentialism would be concerned about the lost organs. More generally, MEC would strongly represent common-sense views on contentious ethical issues, because common-sense views on ethical issues are more common than other kinds of views. That being said, a risk-averse function might be desirable as a backup utility function to be activated in certain circumstances  (Oesterheld 2016) .
 Conclusion Disagreement over ethics is unlikely to be resolved in the near future. If we wish to create machines with the capacity to make decisions on matters of ethical importance then a fair and robust system for translating the patchwork of human values into moral guidance must be developed. The intractability of our moral disputes and the magnitude of our moral errors can both be minimized by compromising with a decision framework which emphasizes the respective priorities of different moral systems in accordance with their plausibility. William MacAskill's framework of maximizing expected choiceworthiness satisfies this description. 			 K. Bogosian

Abstract: The development of artificial intelligence will require systems of ethical decision making to be adapted for automatic computation. However, projects to implement moral reasoning in artificial moral agents so far have failed to satisfactorily address the widespread disagreement between competing approaches to moral philosophy. In this paper I argue that the proper response to this situation is to design machines to be fundamentally uncertain about morality. I describe a computational framework for doing so and show that it efficiently resolves common obstacles to the implementation of moral philosophy in intelligent machines.