Future Progress in Artificial Intelligence: A Survey of Expert Opinion

Introduction Artificial Intelligence began with the "… conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it."  (McCarthy, Minsky, Rochester, & Shannon, 1955, p. 1)  and moved swiftly from this vision to grand promises for general human-level AI within a few decades. This vision of general AI has now become merely a long-term guiding idea for most current AI research, which focuses on specific scientific and engineering problems and maintains a distance to the cognitive sciences. A small minority believe the moment has come to pursue general AI directly as a technical aim with the traditional methods -these typically use the label 'artificial general intelligence' (AGI) (see  Adams et al., 2012) . If general AI were to be achieved, this might also lead to superintelligence: "We can tentatively define a superintelligence as any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest."  (Bostrom, 2014 ch. 2) . One idea how 2. Questionnaire
 Respondents The questionnaire was carried out online by invitation to particular individuals from four different groups for a total of ca. 550 participants (see Appendix 2). Each of the participants got an email with a unique link to our site to fill in an online form (see Appendix 1). If they did not respond within 10 days, a reminder was sent, and another 10 days later, with the note that this is the last reminder. In the case of EETN (see below) we could not obtain the individual email addresses and thus sent the request and reminders to the members' mailing list. Responses were made on a single web page with one 'submit' button that only allowed submissions through these unique links, thus making non-invited responses extremely unlikely. The groups we asked were: 1. PT-AI: Participants of the conference on "Philosophy and Theory of AI", Thessaloniki October 2011, organized by one of us (see  Müller, 2012 Müller, , 2013 . Participants were asked in November 2012, i.e. over a year after the event. The total of 88 participants include a workshop on "The Web and Philosophy" (ca. 15 people), from which a number of non-respondents came. A list of participants is on: http://www.pt-ai.org/2011/registered-participants We reduced the list to living authors, added as many as necessary to get back to 100, searched for professional e-mails on the web and sent notices to these. The questionnaire was sent with our names on it and with an indication that we would use it for this paper and Nick Bostrom's new book on superintelligence  (Bostrom, 2014)  -our request email is in Appendix 1. Given that the respondent groups 1 and 2 attended conferences organized by us, they knew whom they were responding to. In groups 3 and 4 we would assume that the majority of experts would not know us, or even of us. These differences are reflected in the response rates. These groups have different theoretical-ideological backgrounds: The participants of PT-AI are mostly theory-minded, mostly do not do technical work, and often have a critical view on large claims for easy progress in AI (Herbert Dreyfus was a keynote speaker in 2011). The participants of AGI are committed to the view that AI research should now return from technical details to 'artificial general intelligence'thus the name AGI. The vast majority of AGI participants do technical work. The EETN is a professional association in Greece that accepts only published researchers from AI. The TOP100 group also works mostly in technical AI; its members are senior and older than the average academic; the USA is strongly represented. Several individuals are members of more than one of these four sets and they were unlikely to respond to the same questionnaire more than once. So, in these cases, we sent the query only once, but counted a response for each set -i.e. we knew which individuals responded from the individual tokens they received (except in the case of EETN).
 Methodology In this field, it is hard to ask questions that do not require lengthy explanations or generate resistance in certain groups of potential respondents (and thus biased results). It is not clear what constitutes 'intelligence' or 'progress' and whether intelligence can be measured or at least compared as 'more' or 'less' as a single dimension. Furthermore, for our purposes we need a notion of intelligence at a level that may surpass humans or where technical intelligent systems might contribute significantly to research -but 'human-level intelligence' is a rather elusive notion that generates resistance. Finally, we need to avoid using terms that are already in circulation and would thus associate the questionnaire with certain groups or opinions, like "artificial intelligence", "singularity", "artificial general intelligence" or "cognitive system". For these reasons, we settled for a definition that a) is based on behavioral ability, b) avoids the notion of a general 'human-level' and c) uses a newly coined term. We put this definition in the preamble of the questionnaire: "Define a 'high-level machine intelligence' (HLMI) as one that can carry out most human professions at least as well as a typical human." (We still had one expert writing back to us that they could not say what a 'typical human' is -though they could be convinced to respond, after all.) In hindsight, it may have been preferable to specify what we mean by 'most' and whether we think of 'most professions' or of 'the professions most working people do'. One merit of our behavioral question is that having HLMI in our sense very likely implies being able to pass a classic Turing test. To achieve a high response rate, we tried to have few questions with simple choices and eventually settled for four questions, plus three on the respondents. We tried to choose questions that would allow us to compare our results with those of earlier questionnaires -see below. In order to improve on the quality of predictions, we tried to 'prime' respondents into thinking about what is involved in reaching HLMI before asking when they expect this. We also wanted to see whether people with a preference for particular approaches to HLMI would have particular responses to our central questions on prediction (e.g. whether people who think that 'embodied systems' are crucial expect longer than average time to HLMI). For these two purposes, we inserted a first question about contributing research approaches with a list to choose from -the options that were Future Progress in Artificial Intelligence: A Poll Among Experts 5/19 given are an eclectic mix drawn from many sources, but the particular options are not of much significance.
 Prior work A few groups have recently made attempts to gauge opinions. We tried to phrase our questions such that the answers can be compared to these earlier questionnaires. Notable are: 1.  (Michie, 1973, p. 511f) : "an opinion poll taken last year among sixty-seven British and American computer scientists working in, or close to, the machine intelligence field". 2. Questions asked live during the 2006 AI@50 conference at Dartmouth College through a wireless voting device (VCM participated (see  Müller, 2007) ). Despite a short report on the conference in  (Moor, 2006) , the results were not published, but thankfully we were able to acquire them from the organizers James H. Moor and Carey E. Heckman -we publish a selection below. 3.  (Baum, Goertzel, & Goertzel, 2011) : participants of AGI 2009, not anonymous, on paper, 21 respondents, response rate unknown. 2 4.  (Sandberg & Bostrom, 2011) : participants of Winter Intelligence Conference 2011, anonymous, on paper, 35 respondents, 41% response rate. 1. The reference by the famous AI researcher Donald Michie is very brief (all the details he gives are in the above quote) but of great of historical interest: 1972/3 were turning years for AI with the publication of Hubert Dreyfus' "What computers can't do" (Hubert L.  Dreyfus, 1972) , the "Lighthill Debates" on BBC TV (with Michie, McCarthy and R. Gregory) and the influential "Lighthill Report"  (Lighthill, 1973 ). Michie's poll asked for the estimated number of years before "computing exhibiting intelligence at adult human level" and Michie's graph shows 5 data points: Years Percentage 5 0% 10 1% 20 17% 50 19% >50 25% He also asked about "significant industrial spin-off", "contributions to brain studies" and "contributions from brain studies to machine intelligence". Michie adds "Of 2 A further, more informal, survey was conducted in August 2007 by Bruce J Klein (then of Novamente and the Singularity Institute) "… on the time-frame for when we may see greaterthan-human level AI", with a few numerical results and interesting comments, archived on https://web.archive.org/web/20110226225452/http://www.novamente.net/bruce/?p=54. Future Progress in Artificial Intelligence: A Poll Among Experts 6/19 those responding to a question on the risk of ultimate 'takeover' of human affairs by intelligent machines, about half regarded it as 'negligible', and most of the remainder as 'substantial', with a view voting for 'overwhelming'."  (Michie, 1973, p. 512) . 2. AI@50 hosted many prominent AI researchers, including all living participants of the 1956 Dartmouth Conference, a set of DARPA-funded graduate students, plus a few theoreticians. The participants were asked 12 multiple choice questions on day one, 17 on day two and another 10 on day three. We select three results from day one here: 3.) The earliest that machines will be able to simulate learning and every other aspect of human intelligence: Within 10 years 5% Between 11 and 25 years 2% Between 26 and 50 years 11% More than 50 years 41% Never 41% Totals 100% 5.) The earliest we will understand the basic operations (mental steps) of the human brain sufficiently to create machine simulation of human thought is: today (we already understand enough) 6% within the next 10 years 12% within the next 25 years 10% within the next 50 years 21% within the next 100 years or more 29% never (we will never understand enough) 21% Totals 100% 6.) The earliest we will understand the architecture of the brain (how its organizational control is structured) sufficiently to create machine simulation of human thought is: The median estimate of when there will be 50% chance of human-level machine intelligence was 2050. So, despite significant overlap with AGI 2009, the group asked by Sandberg and Bostrom in 2011 was a bit more guarded in their expectations. We think it is worthwhile to make a new attempt because the prior ones asked specific groups and small samples, sometimes have methodological problems, and we also want to see how the answers change over time, or do not change -which is why tried to use similar questions. As explained below, we also think it might be worthwhile to repeat our questionnaire at a later stage, to compare results.  The percentages here are over the total of responses. There were no significant differences between groups here, except that 'Whole brain emulation' got 0% in TOP100, but 46% in AGI. We did also not find relevant correlations between the answers given here and the predictions made in the following questions (of the sort that, for example, people who think 'embodied systems' crucial would predict later onset of HLMI).
 Questions & Responses
 Research Approaches Future Progress in Artificial Intelligence: A Poll Among Experts 9/19
 When HLMI? "2. For the purposes of this question, assume that human scientific activity continues without major negative disruption. By what year would you see a (10% / 50% / 90%) probability for such HLMI to exist?" -For each of these three probabilities, the respondents were asked to select a year  [2012-5000, in   For the 50% mark, the overall median is 2040 (i.e. half of the respondents gave a year earlier than 2040 and half gave a year later than 2040) but the overall mean (average) is 2081. The median is always lower than the mean here because there cannot be outliers towards 'earlier' but there are outliers towards 'later' (the maximum possible selection was 5000, then 'never').
 From HLMI to superintelligence "3. Assume for the purpose of this question that such HLMI will at some point exist. How likely do you then think it is that within (2 years / 30 years) thereafter there will be machine intelligence that greatly surpasses the performance of every human in most professions?" -Respondents were asked to select a probability from a dropdown menu in 1% increments, starting with 0%.  Experts allocate a low probability for a fast takeoff, but a significant probability for superintelligence within 30 years after HLMI.
 The impact of superintelligence "4. Assume for the purpose of this question that such HLMI will at some point exist. How positive or negative would be overall impact on humanity, in the long run? Please indicate a probability for each option. (The sum should be equal to 100%.)" -Respondents had to select a probability for each option (in 1% increments). The addition of the selection was displayed; in green if the sum was 100%, otherwise in red. The five options were: "Extremely good -On balance good -More or less neutral -On balance bad -Extremely bad (existential catastrophe)". Percentages here are means, not medians as in the other tables. There is a notable difference here between the 'theoretical' (PT-AI and AGI) and the 'technical' groups (EETN and TOP100).
 % PT-AI
 Respondents Statistics We then asked the respondents 3 questions about themselves: And we finally invited participants to make a comment, plus a possibility to add their name, if they wished. (We cannot reproduce these here; but they are on our site, see below). A number of comments concerned the difficulty of formulating good questions, much fewer the difficulty of predicting.
 Evaluation
 Selection-bias in the respondents? One concern with the selection of our respondents is that people who think HLMI is unlikely, or a confused idea, are less likely to respond (though we pleaded otherwise in the letter, see below). Here is a characteristic response from a keynote speaker at PT-AI 2011: "I wouldn't think of responding to such a biased questionnaire. … I think any discussion of imminent super-intelligence is misguided. It shows no understanding of the failure of all work in AI. Even just formulating such a questionnaire is biased and is a waste of time." (Hubert Dreyfus, quoted with permission). So, we tried to find out what the non-respondents think. To this end, we made a random selection of non-respondents from two groups (11 for PT-AI and 17 from TOP100) and pressured them via personal email to respond, explaining that this would help us understand bias. The two groups were selected because AGI appears already biased in the opposite direction and EETN appears very similar to TOP100 but for EETN we did not Future Progress in Artificial Intelligence: A Poll Among Experts 14/19 have the data to show us who responded and who did not. We got one additional response from PT-AI and two from TOP100 in this way. For question 2 "… By what year would you see a (10% / 50% / 90%) probability for such HLMI to exist?" we compared the additional responses to the responses we already had from the same respective group (PT-AI and TOP100, respectively). We found the following differences: The one additional respondent from PT-AI expected HLMI earlier than the mean but later than the median, while the two respondents from TOP100 (last row) expected HLMI earlier than mean and median. The very small sample forbids confident judgment, but we found no support for the worry that the non-respondents would have been biased towards a later arrival of HLMI. 10%
 Lessons and outlook We complement this paper with a small site on http://www.pt-ai.org/ai-polls/. On this site, we provide a) the raw data from our results [anonymous unless the participants decided to put their name on their responses], b) the basic results of the questionnaire, c) the comments made, and d) the questionnaire in an online format where anyone can fill it in. We expect that that online questionnaire will give us an interesting view of the 'popular' view of these matters and on how this view changes over time. In the medium run, it be interesting to do a longitudinal study that repeats this exact questionnaire. We leave it to the reader to draw their own detailed conclusions from our results, perhaps after investigating the raw data. Let us stress, however, that the aim was to 'gauge the perception', not to get well-founded predictions. These results should be taken with some grains of salt, but we think it is fair to say that the results reveal a view among experts that AI systems will probably (over 50%) reach overall human ability by 2040-50, and very likely (with 90% probability) by 2075. From reaching human ability, it will move on to superintelligence in 2 years (10%) to 30 years (75%) Future Progress in Artificial Intelligence: A Poll Among Experts 15/19 thereafter. The experts say the probability is 31% that this development turns out to be 'bad' or 'extremely bad' for humanity. So, the experts think that superintelligence is likely to come in a few decades and quite possibly bad for humanity -this should be reason enough to do research into the possible impact of superintelligence before it is too late. We could also put this more modestly and still come to an alarming conclusion: We know of no compelling reason to say that progress in AI will grind to a halt (though deep new insights might be needed) and we know of no compelling reason that superintelligent systems will be good for humanity. So, we should better investigate the future of superintelligence and the risks it poses for humanity.
 Acknowledgements Toby Ord and Anders Sandberg were helpful in the formulation of the questionnaire. The technical work on the website form, sending mails and reminders, database and initial data analysis was done by Ilias Nitsos (under the guidance of VCM). Theo Gantinas provided the emails of the TOP100. Stuart Armstrong made most graphs for presentation. The audience at the PT-AI 2013 conference in Oxford provided helpful feedback. Mark Bishop, Carl Shulman, Miles Brundage and Daniel Dewey made detailed comments on drafts. We are very grateful to all of them. Contribution to this questionnaire is by invitation only. If the questionnaire is filled in without such an invitation, the data will be disregarded. Answers will be anonymized. Results will be made publicly available on the site of the Programme on the Impacts of Future Technology: http://www.futuretech.ox.ac.uk (http://www.futuretech.ox.ac.uk) . Thank you for your time!  " 1 . 1 In your opinion, what are the research approaches that might contribute the most to the development of such HLMI?" [Selection from list, more than one selection possible.] − Algorithmic complexity theory − Algorithms revealed by computational neuroscience − Artificial neural networks − Bayesian nets − Cognitive science − Embodied systems − Evolutionary algorithms or systems − Faster computing hardware − Integrated cognitive architectures − Large-scale datasets − Logic-based systems − Robotics − Swarm intelligence − Whole brain emulation − Other method(s) currently known to at least one investigator − Other method(s) currently completely unknown − No method will ever contribute to this aim
 Future Concerning the above questions, how would you describe your own expertise?" (0 = none, 9 = expert) − Mean 5.85 2. "Concerning technical work in artificial intelligence, how would you describe your own expertise?" (0 = none, 9 = expert) − Mean 6.26 3. "What is your main home academic discipline?" (Select from list with 8 options: Biology/Physiology/Neurosciences -Computer Science -Engineering [non CS] -Mathematics/Physics -Philosophy -Psychology/Cognitive Science -Other academic discipline -None.) [Absolut numbers.] a. Biology/Physiology/Neurosciences 3
 Other method(s) currently known to at least one investigator Other method(s) currently completely unknown No method will ever contribute to this aim //www.fhi.ox.ac.uk/) (http://www.futuretech.ox.ac.uk) This brief questionnaire is directed towards researchers in artificial intelligence or the theory of artificial intelligence. It aims to gauge how people working in the field view progress towards its original goals of intelligent machines, and what impacts they would associate with reaching these goals.
 Vincent C. Müller (http://www.sophia.de) & Nick Bostrom (http://www.nickbostrom.com/) University of Oxford September 2012 A. The Future of AI Define a "high-level machine intelligence" (HLMI) as one that can carry out most human professions at least as well as a typical human. 1. In your opinion, what are the research approaches that might contribute the most to the development of such HLMI?: for the purpose of this question that human scientific activity continues without major negative disruption. By what year would you see a 10%/50%/90% probability for such HLMI to exist? 3. Assume for the purpose of this question that such HLMI will at some point exist. How likely do you then think it is that within (2 years / 30 years) thereafter, there will be machine intelligence that greatly surpasses the performance of any human in most professions? 4. Assume for the purpose of this question that such HLMI will at some point exist. How positive or negative would be the overall impact on humanity, in the long run? Please indicate a probability for each option. (The sum should be equal to 100%.) Total: 0% Questionnaire: Future Progress in Artificial Intelligence | Phil... http://www.pt-ai.org/questionnaire-future-progress-artificial-in...
 Baum et al. asked  for the ability to pass a Turing test, a third grade school year exam [i.e. for 9 year olds] and do Nobel Prize level research. They assume that all and only the intelligent behavior of humans is captured in the Turing test. The results they Within 10 years Between 11 and 25 years Between 26 and 50 years More than 50 years Never Totals got for the 50% probability point were: 2040 (Turing test), 2030 (third grade), and 11% 14% 22% 40% 14% 100% 7/19 2045 (Nobel). 4. Sandberg and Bostrom's first question was quite similar to our 2 nd (see below): "Assuming no global catastrophe halts progress, by what year would you assign a 3. Future Progress in Artificial Intelligence: A Poll Among Experts 10%/50%/90% chance of the development of human-level machine intelligence?"
 Clicks of the 'never' box. These answers did not enter in to the averages above. Future Progress in Artificial Intelligence: A Poll Among Experts Future Progress in Artificial Intelligence: A Poll Among Experts 10/19 11/19 10% 2020 2033 29 Algorithms revealed by computational Cognitive science 50% 2050 2097 200 Never no. % Integrated cognitive architectures 90% 2093 2292 675 10% 2 1.2 Artificial neural networks TOP100 Median Mean St. Dev. 50% 7 4.1 50% 10% 90% Embodied systems 2072 110 Large-scale datasets Faster computing hardware 2050 2024 2034 33 28 16.5 Other method(s) currently completely 90%: 2070 2168 342 ALL Whole brain emulation Median Mean St. Dev. Other method(s) currently known to at Evolutionary algorithms or systems 10%: 2022 2036 59 50%: 2040 Logic-based systems 2081 153 90%: Algorithmic complexity theory 2075 2183 396 No method will ever contribute to this Robotics Swarm intelligence Results sorted by percentage steps: 10% Bayesian nets Median Mean St. Dev. PT-AI 2023 2043 81 00% 10% 20% 30% 40% 50% 60% AGI 2022 2033 60 EETN 2020 2033 29 TOP100 2024 2034 33 ALL 2022 2036 59 50% Median Mean St. Dev. PT-AI 2048 2092 166 one-year increments] or check a box marked 'never'. AGI 2040 2073 144 EETN Results sorted by groups of respondents: 2050 2097 200 TOP100 2050 2072 110 PT-AI ALL Median Mean St. Dev. 2040 2081 153 10% 90% 2023 Median Mean St. Dev. 2043 81 50% PT-AI 2048 2080 2092 2247 166 515 90% AGI 2080 2065 2247 2130 515 202 AGI EETN Median Mean St. Dev. 2093 2292 675 10% TOP100 2070 2022 2033 2168 60 342 50% ALL 2040 2075 2073 2183 144 396 90% 2065 2130 202 EETN Median Mean St. Dev.
			 There is a collection of predictions on http://www.neweuropeancentury.org/SIAI-FHI_AI_predictions.xls
			 .11.12 10:29 OEZ

Abstract: There is, in some quarters, concern about high-level machine intelligence and superintelligent AI coming up in a few decades, bringing with it significant risks for humanity. In other quarters, these issues are ignored or considered science fiction. We wanted to clarify what the distribution of opinions actually is, what probability the best experts currently assign to high-level machine intelligence coming up within a particular time-frame, which risks they see with that development, and how fast they see these developing. We thus designed a brief questionnaire and distributed it to four groups of experts in 2012/2013. The median estimate of respondents was for a one in two chance that highlevel machine intelligence will be developed around 2040-2050, rising to a nine in ten chance by 2075. Experts expect that systems will move on to superintelligence in less than 30 years thereafter. They estimate the chance is about one in three that this development turns out to be 'bad' or 'extremely bad' for humanity.


Abstract: Most known regret bounds for reinforcement learning are either episodic or assume an environment without traps. We derive a regret bound without making either assumption, by allowing the algorithm to occasionally delegate an action to an external advisor. We thus arrive at a setting of active one-shot model-based reinforcement learning that we call DRL (delegative reinforcement learning.) The algorithm we construct in order to demonstrate the regret bound is a variant of Posterior Sampling Reinforcement Learning supplemented by a subroutine that decides which actions should be delegated. The algorithm is not anytime, since the parameters must be adjusted according to the target time discount. Currently, our analysis is limited to Markov decision processes with finite numbers of hypotheses, states and actions.


Xavier Abad
Robbin Scott Weyant
Michael J Imperiale
Arturo Casadevall

A New Approach to evaluating the risk-Benefit equation for Dual-Use and Gain-of-Function research of concern

The use of technology to gain an advantage in human conflict is ancient. Iron replaced bronze in weapons and history is replete with examples of how scientific and technological advances were adapted for war and the acquisition of power. In fact, all technologies are potentially dual use. This trend accelerated with the expansion of knowledge following the scientific and industrial revolutions and affected primarily the disciplines of physics and chemistry, domains of knowledge containing information that was then immediately useful in warfare. The application of biological knowledge to conflict came later even though biological warfare also has a long history. For example, the black death that devastated medieval European society may have begun by an act of biological warfare during the siege of Caffa in 1346  (Wheelis, 2002) . British settlers to North America brought smallpox-infested blankets as "gifts" to Native Americans  (Duffy, 1951) . However, it was not until the molecular biology revolution in the late twentieth century that the potential of biology, and in particular microbiology, in biological warfare and terrorism came into focus. These fears came true with a single act of terrorism involving the mailing of anthrax spores in the U.S. in 2001, which caused several deaths and considerable disruption to government facilities such as congressional offices and the postal system  (Bush and Perez, 2012) . This act of bioterrorism catalyzed a series of events that led to greater awareness of the potential of using biological knowledge in nefarious ways, new regulations in biological research in the U.S. such as the Select Agents and Toxins list, and heighted concerns about some types of microbiology research. One of the immediate effects of the post-2001 environment was increased scrutiny of published findings, and several papers were particularly noteworthy for eliciting discussion and concern.  In 2001, researchers  in Australia showed that insertion of the gene for IL-4 into the ectromelia virus genome defeated vaccine immunity  (Jackson et al., 2001) , a report that drew immediate concerns because of its possible implication for similar effects with variola major virus. Similarly, the finding that complement inhibition potentiated variola virus virulence suggested a mechanism for enhancing its pathogenicity  (Rosengard et al., 2002) . An analysis of the U.S. milk distribution system revealed vulnerable nodes where the introduction of botulinum toxin could lead to mass casualties  (Wein and Liu, 2005) , leading to an outcry about journals publishing research that could be exploited by terrorists. Advances in molecular biology led to the chemical synthesis of poliovirus  (Cello et al., 2002)  and the reconstruction of the 1918 pandemic influenza virus  (Tumpey et al., 2005) , which greatly heightened concern about the re-introduction of extinct, controlled, and new viruses to naïve human populations. In this environment, the U.S. Government created the National Science Advisory Board for Biosecurity (NSABB) in 2005 to provide guidance on issues relating to biological risk and security. Both authors of this article were inaugural members of the NSABB and served on that board for 9 years. When the NSABB first began their deliberations, one of the major early topics was that of dual-use research in the biological sciences. This issue had previously vexed the physics community in the twentieth century as advances in physics and associated technologies had led to radar, nuclear weapons, ballistic missiles, and other military hardware. However, there was little or no precedent for similar concerns in the biological research community, especially since biological weapons were outlawed by the Biological Weapons Convention in 1972. Moreover, the term dual use, which had meant civilian and military use in the physics arena, was redefined in its application to biology by an NRC report in 2004 to mean technologies and information that had both beneficial and maleficent use  (NR Council, 2004) . Nevertheless, among the early accomplishments of the NSABB was formulating a definition for dual use research of concern (DURC), which limited the scope of worrisome work to a small subset of biological research. The NSABB also generated a set of recommendations for communicating DURC in publications and other venues. The work of the NSABB proceeded quietly and in relative obscurity until 2011 when the U.S. government learned about two submitted manuscripts reporting that highly pathogenic avian influenza virus (HPAIV) could be made mammalian transmissible by laboratory passage in ferrets, and sought an opinion from the board as to whether publication should be permitted. the Great Gain-of-Function (GOF) controversy Before considering the GOF controversy involving influenza virus, it is worthwhile reviewing what is meant by "GOF" and the limitations of the lexicon in the controversy. At the most basic level, a GOF experiment, as the name implies, is one that gives a biological entity any new property. GOF experiments can be highly beneficial and can generate pest resistant crops, microbes expressing proteins for medical therapy such as recombinant human insulin, and new cancer therapies by enhancing lymphocyte function. The controversial aspects of GOF experiments in microbiology arise when microbes are modified to have new properties that can enhance their virulence and transmissibility because such experiments raise worries about new diseases, outbreaks, pandemics, biosafety, and biosecurity. In the experience of the authors, the NSABB probably did not anticipate what a problematic dual-use paper would look like until they had the opportunity to consider the two papers that reported the gain of mammalian transmissibility for HPAIV  (Herfst et al., 2012; Imai et al., 2012) . What made the two papers highly controversial was that they reported that only a few amino acid changes were sufficient to confer mammalian transmissibility to the H5N1 HPAIV, and there was concern that this information could be used to generate a virus capable of pandemic potential with high mortality by anyone with basic molecular biology and virology skills. The NSABB initially recommended redacting the sequence information, but this was considered not feasible given the legal framework in the U.S., and the two papers  (Herfst et al., 2012; Imai et al., 2012)  were eventually published in 2012 after some modifications to the text. The arguments, counterarguments, and events surrounding that episode are described in various publications  (Berns et al., 2012a,b; Fouchier et al., 2012a) , often by the participants themselves, and will not be repeated here. The major outcome of the great GOF controversy of 2012 is that it defined and crystallized some of the issues of dual-use research in biology by providing clear examples of experiments that were of great scientific value while also raising biosecurity and biosafety concerns. Although both papers were eventually published with full information, the debate and controversy did not solve the central questions of what work should be performed moving forward and how it should be reported. In fact, the controversy erupted again in the summer of 2014 following the publication of additional papers describing other GOF experiments (e.g.,  Watanabe et al., 2014)  as well as a series of biosafety lapses at U.S. government laboratories, which led the government to impose a pause on these types of experiments with certain viruses until the NSABB could formulate new recommendations and guidelines. The pause was lifted in mid-December, 2017, alongside policy guidance about how GOF funding decisions should be made going forward. 1
 Publication Quandaries One of the most vexing questions of the problem of dual-use research is publication of scientific findings. In the U.S., the research community is guided by National Security Decision Directive (NSDD) 189, formulated in 1985 by the Reagan admini stration, which states that all fundamental research results could be published freely. In other words, there was either open publication or classification, and nothing in between. Viewed from the context of NSDD-189 the papers involved in the 2012 GOF controversy were either to be published in their entirety or classified, with the latter alternative being impractical and unfeasible since the work had been carried out in an unclassified environment. With this binary policy, redacting papers because they might contain information useful for nefarious purposes is not an option unless export control regulations are invoked [for a full discussion of the legalities involved, see NAS EM DURC report (National Academies of Sciences Engineering and Medicine, 2017)]. Consequently, when faced with GOF papers containing information that could conceivably be used to enhance the pathogenicity or transmissibility of a virus, editors and journals have almost always opted for full publication, usually requiring more details from the authors about biosafety and biosecurity methods, and often publishing an accompanying editorial emphasizing the scientifically useful aspects of the research [for examples, see  Dermody et al. (2013 Dermody et al. ( , 2014a ]. Such decisions have sometimes elicited strong criticism from members of the virology community  (Dermody et al., 2014b; Wain-Hobson, 2014a) . A more recent situation illustrates the difficult issues faced by authors, editors, and journals when publishing papers that contain DURC. In 2013, the Journal of Infectious Disease faced the quandary of receiving a paper reporting a new botulinum toxin that was resistant to neutralization by the then-available antisera. In response, the journal took the remarkable decision of publishing the paper without the toxin sequence and vowed to keep the data confidential until an antidote was available  (Casadevall et al., 2013; Dover et al., 2013; Relman, 2013; Hooper and Hirsch, 2014) . A major factor in withholding the sequence data was the realization that this toxin could be easily produced using standard techniques if the toxin sequence was available and that the availability of such a toxin might pose a major public health risk. This decision was highly controversial since without the sequence information other investigators could not verify the findings reported in the publication or begin to develop medical countermeasures. In fact, subsequent work revealed that the new toxin was indeed neutralized by available sera, thus diminishing the initial concerns  (Enserink, 2015) . On one hand, withholding the sequence information from public scrutiny was the responsible action given the potential threat, while on the other hand, such an action delayed verification of the findings and the generation of countermeasures had the threat been real. Clearly, manuscripts containing DURC pose vexing problems for journals that often must make such decisions alone. At American Society for Microbiology journals, there are protocols in place for reviewing such manuscripts and the journals have available many individuals with microbiological and safety expertise who can provide input  (Casadevall et al., 2015) . Other situations are likely to pose different challenges, but it is clear that journals are often poorly equipped to handle the difficult problems involved in publishing DURC content. In this regard, we have argued for the need of a national board that can help journals make publication decisions  (Casadevall et al., 2014) , although to date no such body exists. Another major challenge to the control of DURC information is the emergence of preprint servers in biology that allow the posting of research findings before peer review and the proliferation of predatory open access journals that will publish essentially any paper for a fee. Consequently, there now exist alternative systems for publication even if standard journals decline to publish a particular article over DURC concerns. Bypassing traditional publication methods could also allow authors to avoid government scrutiny.
 the situation today It is not an exaggeration that the situation today regarding DURC and GOF experiments remains highly unsatisfactory. Experimental work involving GOF experiments on pathogens with pandemic potential is highly regulated with proponents of such work arguing that restrictions on experimental design are unwarranted  (Fouchier et al., 2012b)  while opponents of such experiments argue that such work cannot be justified (Wain-Hobson, 2014a,b). Proponents of GOF experiments have noted that data generated in such experiments is useful in epidemiological surveillance (Schultz-Cherry et al., 2014), whereas opponents have argued that such work cannot be morally justified because of its inherent risk  (Lipsitch and Galvani, 2014) . In the U.S., government-mandated pauses on certain types of experiments remain in effect for federally funded research but such prohibitions do not extend to other countries, or to work funded by other sources. Although the NSABB continues to analyze and debate the issues involved with DURC and GOF experiments in the U.S., there is no comparable body on the international scene, where much of this type of research is done. There is no consensus in the scientific community on whether the value of some experiments justifies the risks involved. The central problem in finding a consensus is that the value of the research cannot be measured in real time, while assessments of risk involve assumptions that can lead to widely divergent estimates. For example, some risk-benefit calculations have suggested that a high consequence accident is likely to occur in the near future  (Lipsitch and Bloom, 2012; Lipsitch and Inglesby, 2014) , while others have estimated such probabilities to be near zero  (Fouchier, 2015) . However, there is some evidence that the incessant debate, which shows no sign of resolution, is taking its toll on the virology community as shown by surveys suggesting that scientists in training may avoid these areas of research  (Pfeiffer, 2015) . This is of particular concern given that recent years have seen new viral threats in the forms of the West Africa Ebola outbreak, the emergence of Zika virus in the Americas, and Middle East Respiratory syndrome coronavirus in the Arabian peninsula, highlighting the importance of virology in human preparedness against pandemic threats  Casadevall, 2015a, 2016) . One of the most important developments in the past few decades has been a change in the zeitgeist of the field that concerns itself with DURC research and GOF experiments. In the early years of the twenty-first century, as the U.S. reeled from the 9/11 terrorist attacks, which were shortly followed by the anthrax spore attacks, the focus was primarily on biosecurity. At that time, the concern was that terrorists and state actors would use the tools of the molecular biology revolution to create new and more devastating biological weapons. However, as the years passed and no new attacks developed, combined with the attribution of the anthrax spores in mail attacks to a lone insider in a U.S. government laboratory  (Bhattacharjee and Enserink, 2008) , this security threat appeared to recede. At the same time, a series of unfortunate biosafety lapses in government laboratories heightened concerns about biosafety. Hence, today people (at least those outside the security community) appear to be more worried about an unintended release of a pathogen with pandemic potential than a deliberate biological attack. This change in emphasis from biosecurity to biosafety has developed slowly and could easily change if there is another deliberate biological attack or if becomes apparent that adversaries are developing biological weapons. A major problem contributing to the unsatisfactory nature of the current situation is that the NSABB definition of DURC does not work well in day-to-day practice. The NSABB defined DURC as "life sciences research that, based on current understanding, can be reasonably anticipated to provide knowledge, information, products, or technologies that could be directly misapplied to pose a significant threat with broad potential consequences to public health and safety, agricultural crops and other plants, animals, the environment, materiel, or national security. " Although the crafting of this definition was a major achievement at the time in being able to limit the scope of DURC to a relatively small portion of biomedical research, thus leaving most biological research to proceed unfettered, subsequent experience showed that the clause "reasonably anticipated" is so subjective in nature that it requires a risk assessment beyond the capabilities of most scientists, reviewers, and editors  (Casadevall et al., 2014) . In essence, the definition was helpful in keeping whole swaths of biological research outside the DURC category, such as cancer and immunology research, but it difficult to apply in determining what is included in the DURC definition. Also contributing to an unsatisfactory situation are concerns whether such regulations as the Select Agents and Toxins list help or hinder societal security. On one hand, placing great restrictions on the accessibility to a number of agents and toxins does increase security by making them more difficult to obtain, with the caveat that these are naturally occurring agents that could be obtained from nature by a determined actor. On the other hand, there is the concern that focusing on lists and a relatively short list of agents means that the overwhelming majority of microbial threats are not on the DURC radar screen. For example, Zika virus emerged as a pathogen of pandemic potential with little or no anticipation from experts. Hence, making lists and focusing attention on those agents in the list has the paradoxical potential to reduce biosecurity since regulations are tightened for listed agents, possibly hindering research, while other dangerous agents are neglected  (Casadevall and Relman, 2010) . A similar argument applies to the current DURC oversight policy of the U.S. government, which is focused on a specific list of agents. 2 This process, which is akin to searching under the lamppost, carries great danger because it ignores what could be significant threats. In this regard, it is noteworthy that fungal pathogens are seldom considered as biological threats despite the fact that members of this kingdom have significant weapon potential  (Casadevall and Pirofski, 2006; Casadevall, 2017) . Against this backdrop of dissatisfaction is the fact that science continues to progress very rapidly, introducing new technologies such CRISPR/Cas9, gene drives, and more efficient synthetic biology, each of which brings with it new possibilities for research that improves the human condition as well as new tools for nefarious purposes. Furthermore, as the technologies improve they are increasingly accessible to more individuals and countries for whom this type of research was previously beyond reach. Hence, the problem of DURC is likely to become significantly more urgent in the near future.
 the Way Forward In a world where new infectious agents that can rapidly spread among vulnerable populations are described on a regular basis, humanity needs a healthy research establishment focused on microbial threats. It is estimated that there are a minimum of 320,000 mammalian viruses  (Anthony et al., 2013) : some fraction of these are probably zoonotic events waiting to happen. As Ian Malcolm, the fictional character created by Michael Crichton in the novel Jurassic Park, stated, "Life finds a way. " Information and knowledge are the best defenses against these threats. In addition to terror from nature, a new crisis will almost certainly occur in the future arising from a deliberate attack, a new provocative paper, or another biosafety lapse. GOF experiments and DURC research are essential for preparedness and the question is not whether this research should be pursued but rather how to do it safely and mitigate risk. To date, each of the controversies has been reactive, with proponents and opponents of such experiments responding to a specific finding or study. After more than a decade of discussion on what constitutes DURC, benefits and risks of GOF experiments, regulations, pauses, and moratoriums, it is increasingly apparent that current approaches are inadequate for the challenges at hand. Given these limitations, we have proposed a new framework for DURC research that focuses on answering specific questions  (Imperiale and Casadevall, 2015b) . Hence, instead of prohibiting certain types of experimentation, we suggest that the way forward is to focus on specific scientific questions and the problems that need answers. For example, if there is a need to determine whether a particular feral virus pathogen has the capacity for mammalian infection and transmission, then GOF experiments performed in safe and controlled conditions can be justified. On the other hand, endowing HIV with new transmission properties is not a medically important GOF experiment (National Academies of Sciences Engineering and Medicine, 2017). All human activities that involve probing the unknown, ranging from space exploration to tissue culture procedures, carry some degree of risk, and it is nature of humanity to accept risk to attain progress. Opponents and proponents of this type of experimentation need to maintain open channels for continued discussion because the dialectic of ideas is likely to result in better decisions. Institutional bodies such as the NSABB need to be supported for they constitute important venues for such discussion and recommendations that mitigate risk. In fact, it is important to create similar institutions that can work at the international level since U.S.-based research is a small portion of all microbiological work done worldwide. Most importantly, we should remain optimistic that the research community can do the research necessary to obtain information critical to protect our species while minimizing the risks of such work. 			 https://grants.nih.gov/grants/guide/notice-files/NOT-OD-17-071.html.
			 https://www.phe.gov/s3/dualuse/documents/us-policy-durc-032812.pdf.

Abstract: In the twenty-first century, biology faces a problem that has previously vexed other disciplines such as physics, namely the prospect that its knowledge domain could be used to generate biological agents with altered properties that enhanced their weapon potential. Biological weapons bring the additional dimension that these could be self-replicating, easy to manufacture and synthesized with commonly available expertise. This resulted in increasing concern about the type of research done and communicated, despite the fact that such research often has direct societal benefits, bringing the dual-use dilemma to biology. The conundrum of dual use research of concern was crystallized by the so-called "gain-of-function" type of experiments in which avian influenza viruses were endowed with new properties in the laboratory such as increased virulence and the capacity for mammalian transmission. After more than a decade of intensive discussion and controversy involving biological experiments with dual-use potential, there is no consensus on the issue except for the need to carry out such experiments in the safest conditions possible. In this essay, we review the topic with the hindsight of several years and suggest that instead of prescribing prohibitions and experimental limitations the focus should be on the importance of scientific questions at hand. We posit that the importance of a scientific question for medical and scientific progress provides a benchmark to determine the acceptable level of risk in biological experimentation.