Canaries in Technology Mines: Warning Signs of Transformative Progress in AI

INTRODUCTION Progress in artificial intelligence (AI) research has accelerated in recent years, and applications are already beginning to impact society  [9] [43]. Some researchers warn that continued progress could lead to much more advanced AI systems, with potential to precipitate transformative societal changes  [13]   [19]   [21] [27]  [39] . For example, advanced machine learning systems could be used to optimise management of safety-critical infrastructure  [33] ; advanced language models could be used to corrupt our online information ecosystem  [31] ; and AI systems could even gradually begin to replace a large portion of economically useful work  [17] . We use the term "transformative AI" to describe a range of possible advances in AI with potential to impact society in large and hard-to-reverse ways  [22] . Preparing for the future impacts of AI is challenging given considerable uncertainty about how AI systems will develop. There is substantial expert disagreement around when different advances in AI capabilities should be expected  [11]   [19]   [34] . Policy and regulation will likely struggle to keep up with the fast pace of technological progress  [12]   [15] [42], resulting in either stale, outdated regulation or policy paralysis  [3] . It would therefore be valuable to be able to identify 'early warning signs' of transformative AI progress, to enable more anticipatory governance, as well as better prioritisation of research and allocation of resources. We call these early warning signs 'canaries', based on the colloquial use of the phrase 'canary in a coal mine' to indicate advance warning of an extreme event. Our use of this term takes its inspiration from an article by Etzioni  [16] , in which he stresses the importance of identifying such canaries. We want to take this suggestion seriously and propose a systematic methodology for identifying such canaries. Many types of indicators could be of interest and classed as canaries, including: research progress towards key cognitive faculties (e.g., natural language understanding); overcoming known technical challenges (such as improving the data efficiency of deep learning algorithms); or improved applicability of AI to economically-relevant tasks (e.g. text summarization). What distinguishes canaries from general markers of AI progress (like those discussed in  [32]  or  [36] ) is that they indicate that particularly transformative impacts of AI may be on the horizon. Given that our definition of "transformative AI" is currently very broad, canaries are therefore defined relative to a specific form of transformative AI or impact. For example, we might identify canaries for human-level artificial intelligence; canaries for automation of a specific sector of work; or canaries for specific types of societal risks from AI. From a governance perspective, we are particularly interested in canaries which indicate that faster or discontinuous progress may be on the horizon, since the impacts of rapid progress would be especially difficult to manage once they begin to manifest. These therefore particularly warrant advanced preparation. In what follows, we describe and discuss a methodology for identifying canaries of progress in technological research. We focus on AI progress, but believe this method, once trialled and tested, could be applied to other areas of technological development. We motivate and describe the methodological approach, which combines expert elicitation and causal mapping, before presenting one implementation of this methodology to identify canaries for progress towards high-level machine intelligence (HLMI). After discussing potential canaries for HLMI specifically, we discuss how to make use of canaries in monitoring of AI progress and suggest how the limitations of this methodology might be addressed in future iterations of this work. advanced AI will be achieved  [5][19] [21]  [34] . For example, Baum et al.  [5]  survey experts on when four specific milestones in AI will be achieved: passing the Turing Test, performing Nobel quality work, passing third grade, and becoming superhuman. Both Müller and Bostrom  [34] , and Grace et al.  [19]  focus their survey questions around predicting the arrival of high-level machine intelligence (HLMI), which the latter define as being achieved when "unaided machines can accomplish every task better and more cheaply than human workers".
 METHODOLOGY
 Background and Motivation However, these studies have several limitations  [6]  that should make us cautious about giving their results too much weight. The experts surveyed in these studies have no experience in quantitative forecasting and receive no training before being surveyed, which likely renders their predictions unreliable  [10][41] . Issues of reliability aside, these forecasting studies are also limited in what they can tell us about future AI progress. They have little to say about impactful milestones on the path to advanced AI, let alone early-warning signs. Experts disagree substantially about when capabilities will be achieved  [11] [19] and without knowing who (if anyone) is more accurate in their predictions, these forecasts cannot easily inform decisions and prioritisation around AI today. Quantitative expert elicitations like these also do not tell us why different experts disagree, what kinds of progress might cause them to change their judgements, or what aspects they in fact do agree upon. While broad probability estimates for when advanced AI might be achieved are interesting, they tell us little about the path from where we are now, or what could be done today to shape the future development and impact of AI. At the same time, several research projects have begun to track and measure progress in AI  [7] [23]  [36] . These projects focus on a range of indicators relevant to AI progress, but do not make any systematic attempt to identify which markers of progress are most important for anticipating potentially transformative AI. Given limited time and resources for tracking progress in AI, it is crucial that we find ways to prioritise those measures that are most relevant to ensuring societal benefits and mitigating risks of AI. The approach we propose in this paper aims to address the limitations of both work on AI forecasting and on measuring progress in AI. In a sense, the limitations of these two bodies of work are complementary. The AI forecasting literature focuses on anticipating the most extreme impacts and advanced progress in AI, but is unable to say much about the warning signs or kinds of progress that will be important in the near future. AI measurement initiatives, conversely, monitor near-future progress in AI, but with little systematic prioritisation or reflection on what progress in different areas might mean for society and governance. What is needed, building on work in both these areas, are attempts to identify areas of progress today that may be particularly important to pay attention to, given concerns about the kinds of transformative AI systems that may be possible in future. Progress in these areas would serve as crucial warning signs -canaries, as well call them -suggesting more advance preparation for future AI systems and their impacts is needed. We believe that identifying canaries for transformative AI is a tractable problem and therefore worth investing considerable research effort in today. In both engineering and cognitive development, capabilities are achieved sequentially, meaning that there are often key underlying capabilities which, if attained, unlock progress in many other areas. For example, musical protolanguage is thought to have enabled grammatical competence in the development of language in homo sapiens  [8] . AI progress so far has also arguably seen such amplifiers: the use of multilayered non-linear learning or stochastic gradient descent arguably laid the foundation for unexpectedly fast progress on image recognition, translation and speech recognition  [29] . By mapping out the dependencies between different capabilities or milestones needed to reach some notion of transformative AI, therefore, we should be able to identify milestones which are particularly important for enabling many others -these are our canaries. This is the general idea behind our approach to identifying canaries, outlined in more detail in the following sections.
 Proposed Methodology The proposed methodology can be used to identify 'canaries' for any transformative event. In the case of AI, the focus might be on a transformative technology such as HLMI or AGI, a transformative application such as flexible robotics, or a transformative impact such as the automation of at least 50% of jobs. Given a transformative event, our methodology has three main steps:  (1)  identifying key milestones towards the event;  (2)  identifying dependency relations between these milestones; and (3) identifying milestones which underpin many others as 'canaries'.
 Identifying key milestones using expert elicitation Like other studies of AI progress, we rely on expert elicitation throughout this process. However, the reliability of expert elicitation studies depends on the proper selection and use of expertise. Though there are inevitable limitations of using any form of subjective judgement, no matter how expert, these limitations can be minimised with careful selection of experts and questions. We suggest carefully selecting experts with varied expertise relevant to the chosen question. For example, for identifying milestones towards human-level AI, the cohort should include experts in machine learning and computer science but also cognitive scientists, philosophers, developmental psychologists, evolutionary biologists, and animal cognition experts. This diverse group would bring together expertise on the current capabilities and limitations of AI, with expertise on key milestones in human cognitive development and the order in which cognitive faculties develop. We also encourage careful design and phrasing of questions to enable participants to make best use of their expertise. For example, rather than asking experts to identify specific milestones towards human-level AI, which is a question for which they are not trained, we might ask machine learning researchers about the limitations of the methods they use every day, or ask psychologists what important human capacities they see lacking in machines. There are several different methods available for expert elicitation: including surveys, interviews, workshops and focus groups, each with advantages and disadvantages  [2] . Interviews provide greater opportunity to tailor questions to the specific expert, but can be extremely time-intensive compared to surveys, making it difficult to consult a large number of experts. If possible, some combination of the two may be ideal: using carefully selected semi-structured interviews to elicit initial milestones, followed-up with surveys with a much broader group to validate which milestones are widely accepted as being key.
 Mapping dependencies between milestones using causal graphs The second step of our methodology involves convening experts to identify dependency relations between identified milestones: that is, which milestones may underpin, lead to, or depend on which others. Experts should be guided in generating directed causal graphs to represent perceived causal relations between milestones  [35] . Causal graphs show causal links between elements of a system, represented as nodes (elements) and arrows (causal links). A directed positive arrow from A and B indicates that A has a positive causal influence on B. Such causal maps have been used to support decision-making, structure knowledge, and improve visualisation of complex scenarios  [20] [25][26]  [28]  and are particularly useful for exploring and understanding possible futures, rather than aiming to predict a single future  [26] . They are easily modified and constructed collaboratively, and therefore are well-suited to helping us structure expert knowledge on dependencies between different technological milestones. Fuzzy cognitive maps (FCMs), a specific type of causal graph, may be a particularly useful method for our purposes. FCMs capture all benefits of causal mapping but can be extended into a quantitative model, and thus lend themselves to computerised simulations  [25] . This will not always be necessary, but given that our proposed method is applicable to many contexts, a flexible model is desirable. FCMs are well able to document non-linear interactions and experts' mental models of causal interactions because they can handle imprecise causal links. The variables (nodes) can take any state between 0 and 1 (hence 'fuzzy'), indicating the extent to which the variable is 'present'. When a variable changes its state, it affects all concepts that are causally dependent on it. FCMs have been used successfully in environmental science  [20]   [38] , strategic planning  [30] , and other areas  [25] , and have been recommended for use in futures studies, forecasting, and technology road mapping  [1]   [26] . In a workshop format, experts should be given brief training in causal graph methods or FCMs, and then break into groups to discuss dependencies between milestones. Each group should then collaboratively construct a directed causal graph or FCM to represent these relationships. Groups should be formed so as to maximise the variation of expertise in each group.
 2.2.3 Identifying canaries from causal graphs. Finally, the resulting causal graphs can be aggregated and analysed to identify canaries, by identifying the nodes with the highest number of outgoing arrows. The aggregation process should first focus on identifying commonalities between all graphs which can be shared in the final graph. Substantive disagreements may remain, which can be the subject of mediated discussion, with a voting process to decide on final aspects of the graph. Experts then identify nodes which they agree have significantly more outgoing nodes (some amount of discretion from the experts/conveners will be needed to determine what counts as 'significant'). Since nodes with a high density of outgoing arrows represent milestones which underpin many others, progress on these milestones can act as 'canaries', indicating that we may see further advances in many other areas in the near future. These canaries can therefore act as early warning signs for more rapid and potentially discontinuous progress, as well as for new applications becoming ready for deployment (depending on which exact capabilities they are likely to unlock).
 IMPLEMENTATION: CANDIDATE CANARIES FOR HLMI We describe a partial implementation of the proposed method to identify canaries for achieving high-level machine intelligence (HLMI). We define HLMI here as an AI system (or collection of AI systems) that performs at the level of an average human adult on key cognitive measures required for economically relevant tasks.  2  We interviewed experts about the limitations of current deep learning methods from the perspective of achieving HLMI, and used the findings to construct a causal graph of milestones. This allowed us to identify candidate canary capabilities. The results must be understood as preliminary, because the causal graphs were developed just by the authors, not a cohort of experts, and so have limited validity. However, this initial demonstration and preliminary findings will form the basis for a full study with a broader range of experts in future.
 Expert elicitation to identify milestones To identify key milestones for achieving HLMI, we interviewed 25 experts (using both a non-probabilistic, purposive sampling method and stratified sampling method, as described by  [12]  in chapter six). The sample covered experts in machine learning  (9) , computer science with specialisation in AI (  5 ), cognitive psychology (2), animal cognition (1), philosophy of mind and AI (3), mathematics , neuroscience (1), neuro-informatics (1), engineering (1). Interviewees came from both academia and industry, and were deliberately selected to be at a variety of career stages. We conducted individual, semi-structured interviews, with a set of core questions and themes to guide more open-ended discussion. Semi-structured interviews use an interview guide with core questions and themes to be explored in response to open-ended questions to allow interviewees to explain their position freely  [24] . Initial questions included: what do you believe deep learning will never be able to do? Do you see limitations of deep learning that others seem not to notice? In response to these and similar questions tailored to the interviewee's specific expertise, they were asked to name what they thought were the biggest limitations of current deep learning methods, from the perspective of achieving HLMI. All named limitations were collated, with shortened explanations, and translated into 'milestones', i.e. capabilities experts believe deep learning is yet to achieve on the path to HLMI. Table  1 . shows all milestones based on limitations, named by interviewees. Because we have maintained each interviewee's preferred terminology, several of the milestones listed may turn out to refer to the same or highly similar problems.
 Table 1. Limitations of deep learning as perceived and named by experts Causal reasoning: the ability to detect and generalise from causal relations in data. Common sense: having a set of background beliefs or assumptions which are useful across domains and tasks.
 Meta-learning: the ability to learn how to best learn in each domain. Architecture search: the ability to automatically choose the best architecture of a neural network for a task. Hierarchical decomposition: the ability to decompose tasks and objects into smaller and hierarchical sub-components.
 Cross-domain generalization: the ability to apply learning from one task or domain to another. Representation: the ability to learn abstract representations of the environment for efficient learning and generalisation. Variable binding: the ability to attach symbols to learned representations, enabling generalisation and re-use. Disentanglement: the ability to understand the components and composition of observations, and recombine and recognise them in different contexts. Analogical reasoning: the ability to detect abstract similarity across domains, enabling learning and generalisation. Concept formation: the ability to formulate, manipulate and comprehend abstract concepts. Object permanence: the ability to represent objects as consistently existing even when out of sight. Grammar: the ability to construct and decompose sentences according to correct grammatical rules. Reading comprehension: the ability to detect narratives, semantic context, themes and relations between characters in long texts or stories. Mathematical reasoning: the ability to develop, identify and search mathematical proofs and follow logical deduction in reasoning.
 Visual question answering: the ability to answer open-ended questions about the content and interpretation of an image. Uncertainty estimation: the ability to represent and consider different types of uncertainty. Positing unobservables: the ability to account for unobservable phenomena, particularly in representing and navigating environments. Reinterpretation: the ability to partially re-categorise, re-assign or reinterpret data in light of new information without retraining from scratch.
 Theorising and hypothesising: the ability to propose theories and testable hypotheses, understand the difference between theory and reality, and the impact of data on theories. Flexible memory: the ability to store, recognise and retrieve knowledge so that it can be used in new environments and tasks. Efficient learning: the ability to learn efficiently from small amounts of data. Interpretability: the ability for humans to interpret internal network dynamics so that researchers can manipulate network dynamics. Continual learning: the ability to learn continuously as new data is acquired. Active learning: the ability to learn and explore in self-directed ways. Learning from inaccessible data: the ability to learn in domains where data is missing, difficult or expensive to acquire. Learning from dynamic data: the ability to learn from a continually changing stream of data.
 Navigating brittle environments: the ability to navigate irregular, and complex environments which lack clear reward signals and short feedback loops. Generating valuation functions: the ability to generate new valuation functions immediately from scratch to follow newly-given rules. Scalability: the ability to scale up learning to deal with new features without needing disproportionately more data, model parameters, and computational power. Learning in simulation: the ability to learn all relevant experience from a simulated environment. Metric identification: the ability to identify appropriate metrics of success for complex tasks, such that optimising for the measured quantity accomplishes the task in the way intended. Conscious perception: the ability to experience the world from a firstperson perspective.
 Context-sensitive decision making: the ability to adapt decision-making strategies to the needs and constraints of a given time or context. It is worth noting there are apparent similarities and relationships between many of these milestones. For example, representation: the ability to learn abstract representations of the environment, seems closely related to variable binding: the ability to formulate place-holder concepts. The ability to apply learning from one task to another, cross-domain generalisation, seems closely related to analogical reasoning. Further progress in research will tell which of these are clearly separate milestones or more closely related notions.
 Causal graphs to identify dependencies between milestones Having identified key milestones, we explore dependencies between them using fuzzy cognitive maps (FCM). We focus on how capabilities enable, not inhibit, other capabilities, which means we use only positive influence arrows. FCMs are particularly well-suited to representing the uncertainty inherent in this analysis, as it assumes that each arrow could have a weight to represent varying levels of strength. In this analysis we have not specified the weights on connections, but adding these weights could be trialled with experts in the future. A previous survey  [5]  suggests that this endeavour is a highly uncertain one, finding that many different relationships between AI milestones seem plausible to experts. Our analysis does not claim nor aim to resolve this disagreement, but instead shows only one out of many possible mappings, to illustrate the use and value of FCMs in AI progress monitoring. We use the software VenSim (vensim.com) to illustrate the hypothesised relationships between perceived milestones in Figure  1 . For example, we hypothesise that the ability to formulate, comprehend and manipulate abstract concepts may be an important prerequisite for the ability to account for unobservable phenomena, which is in turn important for reasoning about causality. A positive influence arrow does not mean that achieving one milestone necessarily leads to another, but rather that progress on the first  This map was constructed by the authors, and is therefore far from definitive or the only possible way of representing dependencies between capabilities. However, this initial map does provide an important illustration of the kind of output this methodology should aim to achieve, and generates some initial hypotheses for relationships between milestones.
 Candidate Canary Capabilities Based on this causal map, we can identify two candidates for canary capabilities. The capabilities with the most outgoing arrows are: Symbol-like representations: the ability to construct abstract, discrete and disentangled representations of inputs, to allow for efficiency and variable-binding. We hypothesise that this capability underpins several others, including grammar, mathematical reasoning, concept formation, and flexible memory. Flexible memory: the ability to store, recognise, and re-use knowledge. We hypothesise that this ability would unlock many others, including the ability to learn from dynamic data, the ability to learn in a continual fashion, and the ability to learn how to learn. We therefore tentatively suggest that these are two important capabilities to track progress on from the perspective of anticipating HLMI. We discuss one such capability, flexible memory, in more detail below. Flexible memory, as described by experts in our sample, is the ability to recognize and store reusable information, in a format that is flexible so that it can be retrieved and updated when new knowledge is gained. We explain the reasoning behind the labelled arrows in figure  2 : • (a): compact representations are a prerequisite for flexible memory since storing high-dimensional input in memory requires compressed, efficient and thus abstract representations. • (B): the ability to reinterpret data in light of new information likely requires flexible memory, since it requires the ability to retrieve and alter previously stored information. • (C) and (E): to make use of dynamic and changing data input, and to learn continuously over time, an agent must be able to store, correctly retrieve and modify previous data as new data comes in. • (D): in order to plan and execute strategies in brittle environments with long delays between actions and rewards, an agent must be able to store memories of past actions and rewards, but easily retrieve this information and continually update its best guess about how to obtain rewards in the environment. • (F): analogical reasoning involves comparing abstract representations, which requires forming, recognising, and retrieving representations of earlier observations. Progress in flexible memory therefore seems likely to unlock or enable many other capabilities important for HLMI, especially those crucial for applying AI systems in real environments and more complex tasks. These initial hypotheses should be validated and explored in more depth by a wider range of experts.
 DISCUSSION
 Advantages We believe the proposed method for identifying canaries has many strengths and could be applied to a broad range of important questions about transformative AI systems and impacts. The general methodology of using expert elicitation to identify milestones and then causal mapping to elucidate dependencies between those milestones is extremely flexible, meaning it could be applied beyond AI to other fields of science and technology progress. The method can also be adapted to the preferred level of detail for a given study: causal graphs can be made arbitrarily complex  [18]  and can be analysed both quantitatively and qualitatively. With this method, it is possible to combine different types of expertise relating to milestones: including well-understood technical limitations of current methods, with informed speculation about unknown capabilities that may be important prerequisites to some transformative event. With early warning signs we can track progress towards canary milestones, or directly prepare for the transformative events that follow after it.
 Uses We envision that this methodology could be used to identify warning signs for a number of important potentially transformative events in AI progress, such as foundational research breakthroughs, the use of AI to automate scientific research, or the automation of tasks that affect a wide range of jobs. Once canaries have been identified for some transformative event, there are numerous ways we might use them to improve preparation for its impact, including by: • Automating the collection, tracking and flagging of new publications relevant to canary capabilities, and building a database of relevant publications (perhaps similar to that described by  [40] );
 Limitations and future directions This methodology nonetheless has some limitations which further iterations could seek to improve on. There may be a fundamental trade-off between the benefits of consulting a large, diverse group of experts -enabling more thorough and robust identification of relevant milestones -and the feasibility of reaching agreement upon a single causal map, and therefore agreeing upon canaries. Relatedly, if uncertainty about milestones is too high, it may be difficult for experts to agree on a single causal map or candidates for canaries: finding questions where there is enough uncertainty for this process to be useful, but not so much uncertainty that no agreement can be reached, may be a challenge in some cases. It will also be important to recognise any potential limitations of the specific sample of experts involved in the process: recognising that machine learning researchers may be biased towards emphasising the importance of areas they themselves work on, for example, or that non-computer scientists may often lack a full understanding of what current systems can and cannot do. In using FCMs to generate causal maps, it is not clear what level of detail and quantitative analysis will be most useful. In the implementation described here, we hypothesised relationships at a high level of abstraction and without quantitative analysis, due to the high level at which experts highlighted limitations in the first stage. The higher the level of abstraction, the more uncertain the mapping will be and the less useful it may be to indicate weights. It would be valuable for future work to explore various levels of abstractions, including a more detailed and quantitative analysis using more clearly-defined technical milestones, which could result in more precise forecasts and hypotheses. Finally, it is important to note that attempts to anticipate and understand progress in AI (or any other technology) are not independent of that progress itself. Better understanding of key milestones towards AGI, HLMI, or some other notion of transformative AI, does not just improve our ability to anticipate that progress, but may also improve our ability to make progress towards transformative AI. We must therefore be cautious in identifying 'canary' capabilities, to consider the potential risks of making progress on these capabilities, and to communicate and encourage consideration of these risks to those researchers driving forward AI development. likelihood of progress on other arrows it points to.
 Figure 1 . 1 Figure 1. Cognitive map of dependencies between milestones collected in expert elicitations. Arrows coloured in pink and green indicate capabilities that have significantly more outgoing arrows.
 Figure 2 . 2 Figure 2. Extract of Figure 1, showing one candidate canary capability.
 This work builds on two main bodies of existing literature: research on AI forecasting, and an emerging body of work on measuring AI progress.The AI forecasting literature generally uses expert elicitation to generate probabilistic estimates for when different types of 1st International Workshop on Evaluating Progress in Artificial Intelligence -EPAI 2020In conjunction with the 24th European Conference on Artificial Intelligence -ECAI 2020 Santiago de Compostela, Spain
 Conducting more in-depth research on the societal and governance implications of achieving canary milestones, and preparing governance responses for these milestones ahead of time. • Generating metrics and benchmarks for evaluating progress on canary capabilities; • Using prediction platforms such as Metaculus (ai.metaculus.com) to track and forecast progress on canary capabilities; • Conducting more focused expert elicitation, for example periodically consult experts on their updated forecasts (in the form of cumulative probability estimates) for when different milestones are achieved, or when they are presented with updated progress metrics on canary capabilities; • Conducting more in-depth research to empirically and theoretically investigate hypothesised relationships between milestones: for example, to what extent do improvements in memory structures lead to empirical improvements in performance in brittle environments? •
			 We use this definition, adapted from Grace et al., to highlight that what is key for saying HLMI has been reached is that AI has the cognitive ability to perform every task better than humans workers, not that it is in practice deployed to do so.

Abstract: In this paper we introduce a methodology for identifying early warning signs of transformative progress in AI, to aid anticipatory governance and research prioritisation. We propose using expert elicitation methods to identify milestones in AI progress, followed by collaborative causal mapping to identify key milestones which underpin several others. We call these key milestones 'canaries' based on the colloquial phrase 'canary in a coal mine' to describe advance warning of an extreme event: in this case, advance warning of transformative AI. After describing and motivating our proposed methodology, we present results from an initial implementation to identify canaries for progress towards high-level machine intelligence (HLMI). We conclude by discussing the limitations of this method, possible future improvements, and how we hope it can be used to improve monitoring of future risks from AI progress.


Sebastian Farquhar
Owen Cotton-Barratt
Andrew Snyder-Beattie

Pricing Externalities to Balance Public Risks and Benefits of Research

I n 2012, multiple research groups conducted experiments that produced strains of H5N1 avian influenza that were airborne transmissible between mammals.  1, 2  Given that this disease is often fatal in humans, these particular ''gain-of-function'' experiments sparked wide controversy.* Advocates of the research argued that the experiments were necessary to improve our understanding of the virus, thus enabling better disease surveillance and vaccine production. Opponents of the research argued that a laboratory accident could trigger a global pandemic, or that the research could be used by malicious actors.  3  The debate culminated in a moratorium on US public funding of research ''that may be reasonably anticipated to confer attributes to influenza, MERS, or SARS viruses such that the virus would have enhanced pathogenicity and/or transmissibility in mammals via the respiratory route.''  4  Meanwhile, the US government also commissioned an Sebastian Farquhar, MPhysPhil, is Project Manager; Owen Cotton-Barratt, DPhil, is a Research Fellow; and Andrew Snyder-Beattie, MS, is Director of Research; all are at the University of Oxford, Future of Humanity Institute, Oxford, England. This article reflects only the view of the authors. The ERCEA is not responsible for any use that may be made of the information it contains. ª Sebastian Farquhar et al., 2017; Published by Mary Ann Liebert, Inc. This Open Access article is distributed under the terms of the Creative Commons License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. *Gain-of-function experiments encompass a wide range of research, and only a small subset of such experiments were subject to this controversy. In this article, we focus on gain-of-function research of concern, by which we mean research of the sort affected by the funding moratorium, as a case study of the more general problem of making research funding decisions for dualuse research of concern. Health Security Volume 15, Number 4, 2017 Mary Ann Liebert, Inc. DOI: 10.1089/hs.2016.0118 independent study from Gryphon Scientific to weigh the benefits of the research against the possible biosafety and biosecurity risks.  5  While their research succeeded in producing a quantitative estimate of biosafety risk, it explained the benefits of the research only in qualitative terms. This controversy highlights the difficulty of evaluating research that has the potential to entail significant public risk. Research that is particularly likely to lead to both positive and negative impacts is especially hard to evaluate.  6  These negative consequences might be either the risk of an accident linked directly to the research itself, or risks of deliberate misuse of the resulting technologies. Avoiding research that has any theoretical risk at all would mean missing out on crucial opportunities. But research communities that underestimate the potential for a significant accident, or assume that people will use information gained through research only for good, may be neglecting their duty to society.  7  The H5N1 flu experiments mark a particularly interesting case study that we use to highlight some problems in evaluating the risks and benefits of dual-use research. { In this article, we discuss some of the current difficulties of existing approaches to evaluating risky research, and we propose a general framework that avoids many of the current problems. Within this framework, we outline 2 possible approaches to implementing the framework: a marketled liability insurance approach, and a state-led approach. We conclude by comparing the approaches with each other and with current policy.
 Difficulties of Risk-Benefit Analysis for Scientific Research Risk-benefit analysis, the approach used in the Gryphon Scientific report, is an analytical tool that has been successfully applied to a wide range of domains.  8  Analysts outline a range of scenarios with positive and negative impacts, estimate the probabilities of those scenarios and the net size of the positive impact, and compute the expected value of the proposal. When attempting to do a riskbenefit analysis of research, determining the scenarios and the probabilities of each scenario can be difficult. But as it turns out, estimating biosafety risks is far easier than esti-mating the benefits of research. The reason for this is that one can outline the potential scenarios for biosafety risks fairly well. In order for a laboratory accident to cause a global pandemic, a set of events must occur. The probability of each of these steps can then be estimated using historical data on laboratory accidents and models of disease spread. These estimates are highly uncertain, because of limitations in available data. The data describing exactly what activities labs engage in and their historical failure rates are incomplete, though some records are kept. Moreover, assumptions are required to extrapolate past records into the future. For example, lab safety standards and equipment have improved over the past several decades, but at the same time new labs are being built under the jurisdiction of regulators with less of a track record establishing their biosafety capabilities. Risk estimators must also make some estimate of the probability distribution of key parameters that cannot be empirically tested. For example, what is the probability that a reassortant virus will be sufficiently transmissible to spread to a wide population? Nevertheless, quantitative estimates are possible, though they lead to disagreement.  [9] [10] [11]  Many risks that are in principle very difficult to estimate for similar reasons are nevertheless quantitatively priced by insurers today-such as terrorism,  12  cyberattack,  13  and harm to patients in clinical trials.  14  Conversely, estimating the benefits of research is extremely difficult. Unlike estimating the probability of a laboratory accident scenario, evaluating the benefits of research involves uncertainty over what the possible scenarios even are. Typically, researchers have little information about the potential results of research-which is one of the motivations to do research in the first place. Even the most experienced researchers find it difficult to estimate the exante value of new research projects. The work recreating the genome of the influenza virus responsible for the 1918 pandemic  15  received some criticism for vaguely promising unspecified future benefits  16  but, after some time, has plausibly created tangible benefits including, perhaps, aiding in the management of the 2009 H1N1 pandemic, according to the authors of the original paper.  17  In their review, Gryphon Scientific did not even attempt to assess the benefits of research quantitatively, and it is this part of the risk-benefit calculation that typically poses the greatest difficulty.  5, 18  Evaluating the risks from malicious activity is more difficult than estimating the risk of accidents, but easier than estimating the benefits of research. Ultimately, the Gryphon Scientific report included a quantitative assessment of some biosecurity risks (though not the risks posed by the information discovered by the research). Unlike the accident scenarios, uncertainties around both the intentions and capabilities of malicious actors mean a wider range of scenarios is possible. But despite the difficulties in quantifying such risk, many insurance companies still offer terrorism insurance, and models do exist for putting rough numbers on such risks.  12
 { In the context of atomic technologies, discussions of dual-use technology focused on the ability to rapidly convert civilian technologies for military purposes,  8  and this trend continued for much of the 20th century.  9  Recently, discussions of dual-use research have broadened to include nonmilitary actors and the risk of accidents related to research.  5, 10  It is this last, broadest sense of dual-use research that we focus on in this article, as it captures the full spectrum of risks resulting from research activities. We acknowledge but do not address the fact that the definition of ''dual-use'' remains unsettled,  11  and that one might want to consider dual-use research, technology built on that research, and artefacts implementing that technology separately.  9
 BALANCING RISKS AND BENEFITS OF RESEARCH When science funders evaluate dual-use research, they are faced with risks and benefits that are uncertain in different ways. Most of the risks are quantitatively estimable, but the parameters are hard to pin down. It is our view that, even when highly uncertain, explicit quantitative risk estimates are preferable to implicit ones, because they allow transparent discussion about disagreements and mitigate human biases in judgments around small probabilities.  19  The benefits, however, are so uncertain that even specifying the scenarios is impossible. Fortunately, existing grantmaking approaches are already designed to compare projects with nebulous benefits. So, as long as the quantified risks are accounted for in the cost of a project, these existing systems are well suited for evaluating this research.
 Existing Approaches and the Scientific Grant-Making Process Grant makers already assess the uncertain benefits of research against their costs and the opportunity costs of unfunded research. Although it is hard to judge quantitatively, reviewers assess the potential for scientific excellence in different proposals. However, the costs that reviewers consider are primarily direct financial costs, rather than externalities such as the risks borne by the public from dual-use research. In principle, research funders prefer to fund portfolios of research with maximal benefits of research for society (including positive externalities) and minimal costs (including negative externalities). The concept of scientific merit already includes some sense of positive externalities, and the UK, at least, also considers the social impact of research and its effect on the research ecosystem.  20  Conceptual tools and intuitions on risk externalities are less well developed. Reviewers often implicitly evaluate dual-use risk as a matter of personal or professional ethics  21  or for the sake of reputation of the field. However, for most research, the dual-use stakes are low. In these situations, researchers have no need to also become experts in risk assessment and management. Research institutions rely instead on safety regulations for particular categories of risk in order to make sure research processes are safe.  22  These tools work very well when most of the risk is local, for accidental releases at the community level, or for health risks to lab researchers themselves. Historically, this is where most of the risk in biological research has lain and is therefore an entirely appropriate default. However, in the case of risks that carry a small chance of large catastrophe (eg, research that creates potential pandemic pathogens), these regulations are not enough. Some funders, such as the US government, provide additional decisionprocess guidelines for funding research involving specific pathogens, like H5N1,  23  as well as institutional guidelines for best practice for dual-use research.  24  Others, such as the UK's Biotechnology and Biological Sciences Research Council, the Medical Research Council, and the Wellcome Trust, require grant applicants to consider near-and medium-term risks of misuse but rely heavily on self-governance and a responsible research culture.  25  These frameworks put a high burden on those pursuing dual-use research to become risk assessors. Moreover, public funding guidelines only affect government funding and cooperative private funders. Private dual-use research is less constrained.  26  Some additional judgment can in principle be applied at the stage of publication,  27  but a 2012 survey of 127 life sciences journals found that none reported refusing a submission on biosecurity grounds.  28  Export controls may also be able to restrict some kinds of publication  29  in some jurisdictions. It may be that under the Australia Group's ''no undercut'' principle, in its guidelines for transfers of sensitive chemical or biological weapons, if any participant nation were to block publication using export controls then all would be obliged to (though the mechanisms for applying the guidelines to research results is unclear).  30
 Pricing and Using Externalities in Grant Making
 Absolute Risk as a Cost of a Grant Hard-and-fast rules about what research techniques are too risky, without considering the benefits, may not be responsive enough, because more important research merits more risk.  31  Our principal proposal is that one could assess and price absolute expected risk and explicitly include this cost in grant proposals. This helps resolve the fact that while risks can be comparatively easy to assess, the benefits of research are extremely challenging to quantify. Our proposal allows grant reviewers to use existing frameworks to evaluate grant proposals but weigh potential benefits against a fairer reflection of their social costs. It also follows calls to strengthen the evaluation of risks at pre-funding stages of the research process.  32  For now, we set aside the issues of where an independent estimate of the cost of externalities comes from and where the money to cover this explicit cost is paid. The 2 main approaches to setting estimates, which we consider in detail in the next section, are: 1. To establish clear liability in case of catastrophe and require grant holders to purchase liability insurance as part of the grant. The strength of this is that it is a market-led approach, with insurers incentivized to price externalities correctly. 2. To centrally commission absolute risk assessments to price the externalities, and to require a payment to a state or non-state body to cover the expected cost. The strength of this approach is that it works even if there may be no clear liability after the fact, so it could address biosecurity as well as biosafety risks.
 Benefits of the Approaches Both approaches have 4 main advantages. First, they give decision makers greater incentive to use an accurately priced risk when deciding what research to fund. Currently, the costs of accidental or deliberate harm from gain-offunction research of concern are borne by members of the public in any country as well as lab workers. For example, after a foot-and-mouth disease outbreak in Pirbright in 2007, caused by poorly maintained drainage,  33  prosecutors were unable to seek damages because there was no demonstrable negligence of a specific party.  34  Instead, governments bore the financial cost of compulsory slaughter, farmers bore the costs of damaged and restricted business, and consumers experienced higher food prices. Where the risks of an activity are not borne by those making decisions about the activity, economic theory predicts that more risk will be taken than is socially optimal.  35  So long as the risks are priced correctly and the transaction costs are sufficiently low, research institutions that pay upfront for the risks of research are more likely to engage in socially optimal risktaking. If implemented successfully, this would result in both better grant choices and better use of risk reducing techniques and equipment. Second, these approaches keep decisions about what research gets funded in the hands of researchers. One of the largest difficulties in assessing the social value of research is assessing the intellectual benefits of research. Researchers themselves are best placed to evaluate the potential for success and the intellectual merit of research because they have the deepest knowledge of the area. Third, these approaches incorporate democratic or public input on the significance of risks. Although researchers have expertise in the potential for intellectual success from a grant proposal, they are not necessarily able to speak for the values of society. Researchers have an obligation to incorporate public input on value systems and consider the justice implications of their work.  31  Our proposal creates ways for the public to express their values without creating rules allowing or disallowing certain research techniques or areas of inquiry. In the case of an insurance mechanism, the legislative branch effectively determines the social cost borne were the risk to materialize, via the judiciary. In the case of a central body setting the costs, this body would ultimately be accountable to the electorate. Fourth, these approaches ensure appropriate risk assessment and governance expertise is brought to bear. Researchers submitting grant proposals are often asked to consider potential dual-use and accident risk. Grant makers do so similarly. However, neither group tends to include specialists in risk assessment or biosecurity.  26  If absolute risk is priced separately, the price can be set by expert risk assessors. Throughout, our guiding principle is to ensure the right incentives are in place and that components of the funding decision are broken up such that those best placed to assess them do so.
 Disadvantages of the Approaches Although we believe these approaches might improve the capacity of the research funding system to manage catastrophic risk, they introduce some new concerns. First, research institutions currently receive a subsidy in the form of an implied guarantee covering the costs of imposed risks. If they were asked to pay for this guarantee, it would represent an additional unfunded expense. Since the aim of this proposal is to distribute research effort within, say, the life sciences rather than to change the absolute amount of life science work being done, one might increase funding for life sciences research to compensate. This would appear to increase explicit expenditure in government research budgets, but in fact moves an implicit, random, and large subsidy from the budgets of disaster response agencies, who do not make decisions that determine the risk exposure, to an explicit, predictable, and moderate payment by those who do make such decisions. For example, governments and consumers bear a large part of the cost of escapes of animal pathogens from research labs  36  but have less influence over risk exposure than research funding bodies. Second, penalizing laboratories for reporting accidents and near misses in a timely way harms biosafety and biosecurity in the long run. Increasing reporting would make it easier to use lessons from mistakes to improve lab design and improve accident response. Mechanisms for pricing risk will work best if they avoid creating perverse incentives around reporting. Options for facilitating this include simply not using the information from recent accidents in the case of central analysis, or forbidding a premium hike in the case of liability insurance requirements. Third, these approaches increase bureaucratic overhead. There are several things that might be done to reduce this overhead. One could apply our proposal only to areas with particularly high risk of low-probability but high-impact events. For example, it might be applied only to the gainof-function research that has been covered by the recent US moratorium. Or rather than carrying out a separate risk analysis for each proposal, one could establish broad categories of similar work and set costs for that category as a whole. For some categories, it might be possible to get involvement from intelligence communities to incorporate information not normally available. In the future, scope could be expanded by addressing new areas or by making the cost estimates more granular.
 Making Decisions Under Uncertainty There are things these approaches would not fix. Risk in many cases of dual-use research is very uncertain. No risk management or risk governance process can solve this entirely. It is hoped that in the case of market-led approaches, the profit motive of companies providing insurance as well as an awareness of ''winners' curse'' dynamics will incentivize BALANCING RISKS AND BENEFITS OF RESEARCH appropriate pricing. In the case of a state-led approach, it is hoped that the system will increase the capacity for riskassessment by developing a core of expert assessors. Political pressures or ''short-termism'' might cause prices to be inaccurate. However, this is also a feature of existing approaches to dual-use risk management.
 Approaches to Risk Pricing
 Mandatory Liability Insurance Laboratories conducting experiments of appropriate kinds could be mandated to purchase insurance against liability claims arising from damaging accidents from their research. Insurance companies would then pay out to injured parties in the public in case of an accident. Some proposals in this direction have already been discussed in the UK.  27  In some cases, appropriate use of liability for damages is more effective than regulation in encouraging risk-reducing behavior.  37  In most jurisdictions, negligence leading to catastrophic damage would already establish liability. Sometimes, the complexity of the research process makes it hard to establish negligence of a specific party.  34  Governments might therefore need to legislate to establish clear duty of liability in any damages caused by accident, even if there was no negligence. Liability would give universities strong incentives to minimize the risk. Framers of this legislation would need to be quite specific about which sorts of liability were included (eg, health damage) and which were not (eg, adversely affected tourism, perhaps, or terrorism). Such an approach might be similar to some aspects of the 1957 Price-Anderson Act, which placed a strict liability on operators of potentially risky private nuclear facilities and a mandate to purchase the highest commercially available amount of insurance coverage.  38  A similar approach has been tried and advocated for in environmental risk, though it proved hard to maintain a healthy insurance market.  39, 40  Requiring insurance is important because some risk takers will not be able to compensate victims for catastrophic harms. Many universities currently self-insure against the damage of accidents in their research. Where there is a small chance of catastrophic damage, they do not have enough assets to cover the damages-they are ''judgment-proof.'' Insurance providers, with much greater financial liquidity, are able to bear the full costs even when payouts are very large, which incentivizes them to set premiums appropriately.  41  For the most catastrophic forms of damage, such as a global pandemic, costs would be too big even for a reinsurer to bear. As a result, an insurance approach could involve an explicitly negotiated liability cap, in which case regulatory bodies tasked with managing the most extreme forms of risk are essential. Various amendments to the Price-Anderson Act demonstrate the problems facing liability limits. Indemnifying those creating the risk from all costs above the privately insurable can cause excess risk taking.  42  The most recent version of Price-Anderson requires regulated nuclear energy providers to pool to jointly cover more than $10 billion of costs beyond that covered for each individual party by insurers. While this protects the public by identifying a source of funds beforehand, the fact that the individual contributions do not need to be set aside in advance means that the entire industry carries a large, uninsured, systemic risk. A modern system based on catastrophe bonds might offer a safer way to increase the feasible levels of financing.
 Advantages of the Liability Approach There are 2 main advantages to this market-led approach. First, it is a relatively light intervention, requiring little ongoing work from the state (other than maintaining a healthy insurance market). Second, the insurers would have a profit incentive to accurately estimate the risk, reducing possible politicization of the risk assessment process. In a sufficiently liquid market, competitive pressures should bring premiums down. But insurers are aware of a ''winners' curse'' in the insurance market and would avoid setting premiums too low to cover the long-term risk. If insurers adjust their premiums when safer systems and equipment are adopted, scientists and engineers would be incentivized to devise effective safety protocols to reduce their institutions' insurance costs. This would create a financial driver for applied biosafety and biosecurity research. Imposing liability has improved outcomes in other domains such as occupational safety, medicine, and general risk management in nonprofits and government agencies.  43
 Possible Issues with the Liability Approach Insurers must be willing to insure against biosafety and biosecurity risk. While insurers have shown some interest,  44  there is no functioning market for the same 2 main reasons insurers were unwilling to take on nuclear risk before the 1957 Price-Anderson Act:  38  First, the potential risks are simply too large. A catastrophic global pandemic could kill hundreds of millions of people, and even the largest reinsurers would be unable to absorb this cost without bankrupting themselves (costs above this level will be implicitly backed by the state or the public in any case). It is better to be explicit and cap liability at a specific industry-wide figure. If the cap were sufficiently large, the effect should be more appropriate risk aversion, even if the tail risk for the insurer were not fully internalized. Second, the risks are hard to model and the market would be quite small. Developing models to estimate the risk could be more costly than the expected profit from being in the market. However, insurers do already have models for hard-to-anticipate risks such as terrorism and global pandemics. If need be, the development of appropriate models to facilitate this insurance could be explicitly subsidized. Whether or not insurers are willing to take on the risk, there are challenges to international adoption. Liability laws differ across jurisdictions, which would particularly affect international collaborations. One effective work-around might be to form an agreement among top-tier research journals to require proof of suitable liability insurance for some types of papers. { Liability insurance can also increase moral hazard, by making actors less responsible for the consequences of their actions. This effect could be reduced if the excess on the insurance were large enough that research institutions still had a direct incentive to avoid risks. Finally, using a liability approach to capture biosecurity risks would be difficult. If a terrorist group used information they gleaned from a paper published by a research group, establishing liability would require lengthy legal dispute. Moreover, done carelessly, the uncertainty additional but unclear liability could create might have an inappropriate chilling effect on research.
 Centrally Commissioned Risk Assessments The second approach is to centralize risk assessments. When an area of potential concern is identified, a body commissioned by the state would perform an analysis of the risks involved. This might be similar to the recent Gryphon Scientific analysis, except that it would not attempt to analyze the benefits. This absolute risk analysis would present its outcomes in monetary terms, using value of statistical life figures to convert into a cost. In order to do work of a priced type, laboratories would have to pay the appropriate charge to a central authority. This charge could cover part of the risk of biosafety and biosecurity preparation as well as the cost of risk assessment.
 Advantages of the Centrally Commissioned Risk Assessment Compared to the market-led approach, centralizing the risk assessments has 2 main benefits. First, it can be done by fiat without needing to persuade insurers to enter and stay in the market. Second, it works well even in cases where there will be no clear causal chain establishing liability, such as for biosecurity risks (including risks from the misuse of information produced from such research). Funds are collected based on the central assessment of the size of the risk posed by the research, with no need to work out after the fact which specific group is liable. The Gryphon Scientific report concluded that the biosecurity risks looked at least as large as the biosafety risks, so this is a significant benefit in the case of gain-of-function research of concern.  5  Third, at least in principle, avoiding the need for courts to establish liability might speed up payments.
 Possible Issues with the Central Approach Patchy implementation only in certain jurisdictions might just move research from nations with good risk management processes to those without. In practice, however, many researchers are likely to want to remain at top labs, limiting the risk of research simply moving to jurisdictions that do not assess the externalities of research. There may be a need for special arrangements for research done in partnership with labs outside the collecting jurisdiction, such as applying a requirement for proof of insurance at the stage of publishing. The risk assessment itself would be difficult. The centrally commissioned assessment would be more likely than an insurance-led one to need to assess the magnitude of biosecurity risks, which is particularly hard to model. However, stakeholders might have incentives to put pressure on regulators to make certain decisions. This is more likely to be successful because of the amount of judgment involved in the calculations of risk. Unlike the insurance-led solution, there would be no countervailing profit motive.
 Comparisons and Discussion The 2 approaches discussed would work by aligning the incentives for scientists and for funding bodies more closely with those of society as a whole. This approach keeps assessment of the benefits of scientific research purely in the hands of scientists, while bringing the right skills to bear at the right points and managing risk proportionately. We have explored 2 different ways to achieve this. Each has its advantages and disadvantages. The liability approach is more market based. As a result, the risk assessors have a financial incentive to accurately estimate risk, and political pressures are diminished. It might also be easier to use as a template internationally. Since the risks are global and the potentially risky research is not being pursued in just 1 country, being able to build global solutions is extremely valuable. The main benefit of the centrally commissioned analysis is it could bypass lack of clarity about who is liable for eventual damages in cases of biosecurity. These approaches have clear application for gain-offunction research of concern in the life sciences. However, similar approaches may work in a broader range of fields. The insurance mechanism is best suited for dual-use work where risk is predominantly through accident rather than
 { We are grateful to Piers Millett for this suggestion. BALANCING RISKS AND BENEFITS OF RESEARCH malicious intent. In all cases, pricing the absolute size of the risk and leaving researchers in charge of decisions about how to spend their budgets might offer a more responsive and proportionate approach to the challenges of funding research with potential public risks than existing systems. 			 Volume 15, Number 4, 2017

Abstract: How should scientific funders evaluate research with public health risks? Some risky work is valuable, but accepting too much risk may be ethically neglectful. Recent controversy over H5N1 influenza experiments has highlighted the difficulty of this problem. Advocates of the research claim the work is needed to understand pandemics, while opponents claim that accidents or misuse could release the very pandemic the work is meant to prevent. In an attempt to resolve the debate, the US government sponsored an independent evaluation that successfully produced a quantitative estimate of the risks involved, but only a qualitative estimate of the benefits. Given the difficulties of this ''apples-to-oranges'' risk-benefit analysis, what is the best way forward? Here we outline a general approach for balancing risks and benefits of research with public risks. Instead of directly comparing risks and benefits, our approach requires only an estimate of risk, which is then translated into a financial price. This estimate can be obtained either through a centrally commissioned risk assessment or by mandating liability insurance, which allows private markets to estimate the financial burden of risky research. The resulting price can then be included in the cost of the research, enabling funders to evaluate grants as usual-comparing the scientific merits of a project against its full cost to society. This approach has the advantage of aligning incentives by assigning costs to those responsible for risks. It also keeps scientific funding decisions in the hands of scientists, while involving the public on questions of values and risk experts on risk evaluation.


Introduction 1 There is a growing sense of concern over the development of increasingly autonomous non-human agents-in the public and the media, as well as among policy makers and researchers. Such concerns are not unprecedented, yet there is something different about this next wave of technological innovation and change. While the impacts of the adoption of any technology are in some sense uncertain, and result in many and various unintended consequences, there seems to be something particularly unsettling and deeply uncertain about increasingly autonomous technologies. I believe that this sense of concern stems from the recognition that autonomous systems will not only be unpredictable in terms of their unintended actions and general effects on society, but that they may also be out of control, in the sense that these effects will occur beyond the scope of human responsibility. Previous technological innovations have not raised this concern in quite the same way, or to the same degree, because it was possible to rely upon human morality, as well as law, to regulate the use of new technologies.  2  While there were recognized risks to using technologies like guns, cars, airplanes, steam engines, and many more, those risks were circumscribed by limits on their immediate effects and the proximate responsibility of individual humans and human institutions for those effects. In most cases, these risks were also limited in their complexity and longevity, in the sense that those effects could be anticipated, managed, and reduced over time.  3  As long as there is a human hand on the controls, there is a margin of assurance that the technology cannot go too far out of control before it is reigned in by human agency or regulatory policy. Thus, human control presents itself as a 2 There are some notable exceptions. There have been debates over the morality of research and development of technologies with potentially catastrophic results or applications (and sometimes the dissemination of basic research in these areas) particularly the weaponization of nuclear physics and biological and chemical agents. There have been similar debates over the risks of basic research itself in the case of genetic engineering and nanotechnology, where the experiments themselves might go "out of control" and cause irrevocable or catastrophic harm. Many of these concerns are related to the potential causal "chain reactions" of positive feedback systems and exponential growth.
 3 Notable exceptions here are certain tragedies of the commons and the problem of many hands  (Nissenbaum 1996) , in which proximate responsibility is small and requires many participants or instances for the harm to be realized. Environmental degradation is a good example where the participation of many separate responsible agents leads to a diffusion of perceived responsibility. Such harms are difficult to regulate largely because it is more difficult to establish the causal link to the responsible agents and rule out intervening or contributory causes. "kill switch" to technology going completely out of control, while human responsibility further acts as "policy lever" for policy to enact regulation over material technologies. Non-human agents, including robotics and software agents, and especially those using advanced artificial intelligence (AI), are becoming increasingly autonomous in terms of the complexity of tasks they can perform, their potential casual impacts on the world that are unmitigated by human agents, and the diminishing ability of human agents to understand, predict or control how they operate. This increasing autonomy of non-human agents seriously challenges human control, and thus challenges both the "kill switch" and the "policy lever" functions of human control. Some have argued that one factor or other is more troubling, or that one challenge or the other is more dire. But, I believe that the growing concern over increasingly autonomous agents stems from the combination of these factors and the challenges they impart, and they must be addressed comprehensively and systematically. Central to addressing these challenges will be finding a way to manage the liability problem-how to continue to hold people legally liable for increasingly autonomous agents. This is closely related to the responsibility problem-how to ensure that human agents take moral responsibility for the technologies they make and use as these become increasingly autonomous. How should we regulate artificial computational agents, including AI and robotic systems, as they become increasingly autonomous, and supersede current capabilities? In order to take a systematic approach to this question, we must integrate an analysis of fundamental concepts of autonomy and agency, with a review of current processes of systems design and development, and chart a course for future regulatory policy. This will include investigating 1) the conceptual foundations of autonomy and agency; 2) survey the existing legal and moral theories of liability and responsibility and their applicability to artificial agents; and 3) explore how implementing an enhanced theory of liability and agency for autonomous artificial agents could shape regulatory policy going forward. In this paper I will focus on sketching out 1) the conceptual foundations of autonomy and agency, through an examination of the liability problem. One of the central problems facing the development of autonomous artificial agents as technological capabilities continue to increase, is the uncertain status of liability for the effects caused by artificial agents-the liability problem. Resolving the liability problem will require untangling a set of theoretical and philosophical issues surrounding causation, intention, agency, responsibility, culpability and compensation. The primary objective of the proposed research is to address this problem through a conceptual analysis that is informed by existing legal and moral theories, and grounded in technological capabilities of current and potential artificial agents.
 The Liability Problem for Autonomous Artificial Agents In order for society to enjoy many of the benefits of advanced AI and robotics, it will be necessary to be able to deal with situations that arise in which autonomous artificial agents violate laws or cause harm. If we want to allow AIs and robots to roam the internet and the physical world and take actions that are unsupervised by humansas might be necessary for, e.g. personal shopping assistants, self-driving cars, and host of other applications-we must be able to manage the liability for the harms they might cause to individuals and property. Already, there have been automated shopping software bots that have purchased illegal items on the DarkNet, albeit as an art project  (Kasperkevic 2015) . There have also been serious questions raised as to who could be held accountable for the deaths caused by autonomous weapons in war  (Human Rights Watch 2015) . Traditional approaches to handling liability are inadequate for dealing with autonomous artificial agents due to a combination of two factors-unpredictability, and causal agency without legal agency. First, unlike traditional engineering and design, the actual functioning of an autonomous artificial agent is not necessarily predictable in the same way as most engineered systems. Some artificial agents may be unpredictable in principle, and many will be unpredictable in practice. Predictability is critical to current legal approaches to liability. In traditional product liability, the manufacturer is responsible for the product working as designed, and foreseeing likely problems or harms it may cause. Determining what is "foreseeable" often falls upon courts to decide, but the legal standards used are whether the manufacturer had knowledge of the potential problem, or whether a reasonable person should have foreseen it, or whether there is an industry standard of practice that would have revealed it. While there is a degree of unpredictability in the performance of any engineered product, due to failures or unforeseen circumstances of use, there are shared expectations regarding its performance, testing for the limits of that performance and likelihood of failure, and management of foreseeable risks. In the case of advanced AI, a system that learns from environmental data may act in ways that its designers have no feasible way to foresee  (Asaro 2008) . In a limited sense, this is already the case with current machine learning techniques, which use large sets of training data and produce novel problem solutions. Currently, it is possible to analyze and test a learned function and determine its behavior, as with traditional engineering. But when AI systems are allowed to continue modifying their functions and learn after they are deployed, their behavior will become dependent on novel input data, which designers and users cannot predict or control. As a result, the behavior of the learned functions will, to various degrees, also be unpredictable. A truly robust AI program capable of open-ended learning could learn functions that its manufacturer could not foresee, perhaps in principle. Insofar as autonomous artificial agents utilize learning and open-ended learning, their behavior will also be unpredictable. Unpredictability by itself is not an insurmountable problem for liability, insofar as the agents who introduce that unpredictability could be themselves held liable, or the risks from unpredictability could be managed. However, the second factor challenging traditional approaches to liability is that autonomous artificial agents may "act" on their own, yet are not accountable or liable in a legal sense. With most engineered products, the predictability is bounded by the actions of other agentsconsumers, users, service technicians, etc.-and how they maintain and use a given product.  4  In those cases, there is a clear legal agent or user who may have used the product inappropriately and is thus liable for the consequences of the use in that instance (at least partially, if not completely). Autonomous artificial agents can act in the world independently of their designers or operators. This makes it difficult to identify the user or operator, who would normally be liable. In the case of learning systems, the causal influence resulting in the unpredictability of a system stems from the datasets used for learning, not a legally responsible agent (unless someone deliberately seeks to influence the datasets and learning process). These cases are somewhat analogous to the way that parents may be held liable for the actions of small children, but they are not usually held liable for the actions of their adult children. As adults, the children have learned enough about the world to become their own legal agents. Something similar applies in cases of employer-employee liability. Usually the employer is responsible for any damages their employee causes in the course of doing their job (such as damage from a delivery truck hitting a parked car while making routine deliveries), but if the employee is off on a frolic (and took the delivery truck to visit a friend), then the employer is not responsible for the damages and the employee is  (Asaro 2011) .
 Approaches to the Problem Taken together these two factors result in a number of problems with applying traditional theories of legal liability to autonomous artificial agents. There are two basic liability frameworks in the law, criminal and civil (primarily tort) liability. In both domains, it is difficult to hold the artificial agent legally liable for its actions, as they are not legal persons, and treating them as legally fictitious persons (like corporations) does not really solve the problem  (Asaro 2011) . It is possible in some criminal situations to hold the people who operate the artificial systems liable, provided you can show intent to commit a crime, or foreseeable risk of a harm rising to the level of criminal negligence. To the extent that artificial agents become increasingly complex, those who build or deploy advanced AIs and robotics will not necessarily have intent or foresight of the actions those systems may take. This is especially true for systems that can substantially change their operations through advanced learning techniques, or those future systems that might even become genuinely autonomous in generating their own goals and purposes, or even intentions. At some point in the evolution of autonomous artificial agents, they might become legal and moral agents, and society will be faced with the question of whether to grant them some or all of the legal rights bestowed on persons or corporations. At that point, some or all of current liability law might apply to those AIs and robots that qualify as legal persons, though it might not be clear what the exact boundaries of a particular entity might be, or how to punish it  (Asaro 2011) . For example, whether a particular instantiation of a program is the legal subject, or all copies of a program are part of the same legal subject (as copyright law might suggest). It will also not be clear how to appropriately punish them, or otherwise correct their future actions and provide retributive compensation to those that have been harmed. Despite these challenges, being able to treat autonomous artificial agents as legal persons is likely to be the easier problem. In the near term, the hard liability problem for autonomous artificial agents will lie in devising a system of liability that promotes beneficial innovation while offering just and adequate compensation to those harmed. Justice requires that those who will be held liable should be able to understand the scope and extent of the risks and liability they are assuming in deploying an autonomous artificial agent, and have some means of managing that risk through controls over the system. Adequate compensation means that there needs to be sufficient means to compensate those harmed, monetary or otherwise-whether this entails holding large corporations accountable, or providing proper risk-pooling insurance. For insurance to work, of course, it will be necessary for actuaries to be able to assess the risks posed by various artificial agents. Assessing those risks will require both expertise, and probably means for imposing or assuring predictability in the systems themselves. And even then, there may be catastrophic risks involved, which may not be manageable under these frameworks. Ultimately there will be questions that society must address as to the perceived benefits and risks of such technologies, and who in society should receive those benefits and should shoulder the burden of those risks. There are a few liability policy options already in use in tort law, such as joint and several liability, strict liability, and risk-pooling insurance that might be deployed. But there are reasons to doubt that these will scale adequately or address all the problems posed by autonomous artificial agents. Existing forms of joint and several liability permit those harmed to seek monetary damages from the deepest pockets among those parties sharing some portion of the liability. This works well where there is a large corporation or government that is able to pay damages. While the amount they pay is disproportionate to their contribution to the damages, those harmed are more likely to be adequately compensated for their damages. One result of this is that those manufacturers likely to bear the liability burden would seek to limit the ability of consumers and users to modify, adapt or customize their advanced AI and robotics products in order to retain greater control over how they are used. This would stifle innovation coming from the hacking, open-source and DIY communities. The hacking, open-source and DIY communities, while a powerful source of innovation, will have limited means of compensating those who might be harmed from their products-not just those who chose to use them. Strict liability goes further and designates or stipulates a party, usually the manufacturer or owner, as strictly liable for any damages caused. This model of liability applies to such things as the keeping of wild animals. It is expected that tigers will harm people if they get free, so as the keeper of a tiger you are strictly liable for any and all damages the tiger may cause. We apply normal property liability to domesticated animals, however, which we expect to not harm people under normal circumstances. It has been suggested that we could apply this to robotics  (Schaerer, Kelley, and Nicolescu 2009) , and perhaps designate certain advanced AIs as essentially wild animals, and others as domesticated. But then the question arises: How does one determine whether an advanced AI or robot is appropriately domesticated, and stays that way? A designated liable person may not know the extent to which the autonomous system is capable of changing itself once it is activated. More problematic, however, is that a system of strict liability might result in the slow adoption of beneficial AI technologies, as those who are strictly liable would have a large and uncertain risk, and be less likely to produce or use the technology, or soon go out of business. There are other proposals that aim to design mechanisms of accountability into complex sociotechnical systems (van den Hoven, Robichaud, and Santoni de Sio 2015). But such approaches, much like strict liability, do not really address the fundamental issue of autonomous agency-how to define it or to regulate it. They merely try to manage it in an ad hoc effort to maintain traditional concepts rather than find a simpler and more powerful solution. We thus have a few general options when it comes to regulating autonomous artificial agents. We could preclude or prohibit autonomous artificial agents because of their risks and uncertainties. We might call this the precautionary approach. To the extent that it works, it would also impede the development and deployment of many beneficial advanced AIs and robots because of their problematic autonomy. We could permit the development and deployment of autonomous artificial agents, and accept the risks and costs at a social level, without developing a better framework for regulating autonomy. We might call this the permissive approach. 5 This would allow many beneficial applications of advanced AI and robots, but also many harmful ones, including many harms for which no one might be liable and those harmed would not be compensated. As a secondary effect, there would likely be a general backlash against advanced AI and robotics as the technology comes to be seen as harmful and offering few options for restitution of those harms. Or, we could pursue one of the heavy-handed liability schemes, such as strict liability, that would regulate the industry to some extent, but also limit innovation to those areas where there are sufficient profits to motivate large capital companies to enter the market and accept the risks. We might see this work in applications such as financial trading algorithms, self-driving cars and medical-care robots, but many other beneficial applications of AI may not have such clear profit margins. Even in clearly profitable areas, strict liability is likely to stifle innovation for a range of beneficial applications of autonomous artificial agents. Alternatively, we could seek a better solution to the liability problem than current models afford. We could attempt to reconceptualize how we think about agency, causality, liability responsibility, culpability and autonomy for the new age of artificial autonomous agents. While it is not yet clear how this might work, a clear framing of the problem, as this paper has presented, is the first step in that larger project. The next steps involve clarifying the distinctions between the purposes of algorithms and machines from the purposes of the persons who use them  (Asaro forthcoming) , and distinguishing different types of agency accordingly. 			 For example, a hammer is designed to drive nails, but can also be used as a weapon-with the liability falling on the person who uses it as a weapon. Similarly, the car owner is responsible for changing the brake pads of a car when they are worn, and the mechanic is responsible for doing it properly, while the manufacturer's liability is limited to properly informing car owners of the service needs for the car.
			 This is the approach taken by Chopra and White (2011) .

Abstract: This paper describes and frames a central ethical issue-the liability problem-facing the regulation of artificial computational agents, including artificial intelligence (AI) and robotic systems, as they become increasingly autonomous, and supersede current capabilities. While it frames the issue in legal terms of liability and culpability, these terms are deeply imbued and interconnected with their ethical and moral correlateresponsibility. In order for society to benefit from advances in AI technology, it will be necessary to develop regulatory policies which manage the risk and liability of deploying systems with increasingly autonomous capabilities. However, current approaches to liability have difficulties when it comes to dealing with autonomous artificial agents because their behavior may be unpredictable to those who create and deploy them, and they will not be proper legal or moral agents. This problem is the motivation for a research project that will explore the fundamental concepts of autonomy, agency and liability; clarify the different varieties of agency that artificial systems might realize, including causal, legal and moral; and the illuminate the relationships between these. The paper will frame the problem of liability in autonomous agents, sketch out its relation to fundamental concepts in human legal and moral agencyincluding autonomy, agency, causation, intention, responsibility and culpability-and their applicability or inapplicability to autonomous artificial agents.