MS John T O'brien
MBBS, MPH Cassidy Nelson

Assessing the Risks Posed by the Convergence of Artificial Intelligence and Biotechnology

Introduction
 Technological Progress S ince the explosion of the information age, the collection of data has been increasing. About 2.5 quintillion  (10 18  ) bytes of data are generated each day, and 90% of all data have been generated in the past few years alone.  1  Although valuable information is often hidden in these vast datasets, due to their size and complexity, a human analyst could take weeks, months, or even years to discover them, if they are able to at all. If one can analyze a large enough set of data, it is possible to infer corollary relationships or patterns that may be of salience. In the past, it was not conceivable for extremely large datasets, or big data, to have been meaningfully analyzed due to the daunting task of doing so manually. Partially as a result of the increasing prevalence of big data requiring new tools for analysis and increased computing power enabling their development, the use of artificial intelligence (AI) in a variety of fields, including the biological sciences, has gained popularity in the past few years. AI can be defined as a system's ability to correctly interpret external data, to learn from such data, and to use those insights to achieve specific goals and tasks through flexible adaptation.  2  Machine learning, a technique that has enabled rapid advancements in the field of AI, is a subset of AI that is heavily based on pattern recognition and describes methods that help computers learn without explicitly being programmed to do so.  2  Deep learning, in turn, is a machine learning technique that arose in the early 2010s and uses computational models that exhibit similar characteristics to the hierarchal information processing in the human brain.  3  This technique is partially responsible for the increased power potential of machine learning algorithms and has been a large contributor to the recent surge of AI research.  4  Alongside AI, developments in biotechnology and the life sciences have progressed rapidly in the past several years. In fact, some biotechnological advancements may even be outpacing Moore's Law for computers.  5  Critical advances in biology such as next-generation DNA sequencing, de novo DNA synthesis, gene editing, genomics, and bioinformatics may result in the life sciences becoming a principal component of the next technological revolution. Given recent progress and projected trajectories, AI and biotechnology have both been described as key pillars of the fourth industrial revolution, with a growing number of technologies and applications arising at their point of convergence every year.  [6] [7] [8] [9]
 Technological Convergence In tandem, AI and biotechnology increase the potential capability of the life sciences and lower the tacit knowledge required to perform previously tedious laboratory tasks. Widening the pool of actors will almost certainly accelerate technological developments, but this could also pose security risks. With increased access to these technologies, so comes the potential for a resulting high-consequence biological event, some of which may have such large consequences that they constitute a global catastrophic biological risk.  10  The risks posed by the convergence of AI with the biological sciences are plentiful and varied, ranging from AI-assisted identification of virulence factors to in silico design of novel pathogens, which could increase the probability of a global catastrophic biological risk event if accidently, carelessly, or deliberately released. For the purposes of this paper, convergence is defined as the technological commingling between the life sciences and AI such that the power of their interaction is greater than the sum of their individual disciplines. That is, the convergence of these 2 fields has a multiplicative effect. This interaction mostly manifests as AI-developments influencing advances, as well as new risks, in the life sciences and biotechnology rather than the other way around. A more comprehensive understanding of such emerging threats is highly important for the field of health security but demands rigorous examination and a systematic approach. In this paper, risk assessment for converging technologies will be discussed and evaluation of 3 existing biosecurity risk assessment frameworks undertaken. After analyzing their strengths and limitations, suggestions will be made regarding how such frameworks could be adjusted to be applicable to the risks arising at the convergence of AI and biotechnology. Some illustrative examples from the converging risk landscape will be discussed, followed by a proposed hybridized framework and key recommendations to begin addressing gaps in current health security risk assessment for converging technologies.
 Risk Assessments For Converging Technologies
 Challenges Assessing Emerging and Converging Risks Risk assessment for emerging technologies is an inherently difficult task and general frameworks are needed for evaluating the risks associated with technological convergence, including AI and biotechnology. Many frameworks are overly specific to a particular scenario or technology (such as narrowly focusing on synthetic biology), contain many assumptions and guesswork, and are not easily applied to other converging or emerging technologies. This is largely because different frameworks serve different purposes, so expectations that they can be implemented more generally are limited. However, the primary strength of a more generic framework for biosecurity risk assessments would be the ability to compare across different technologies or scenarios. Decision makers could then use such frameworks to be more informed as to which risks may be of higher priority. Compared to assessments of singular domains, analyzing risks of converging technologies presents an even greater challenge due to the added layers of complexity and the presence of multidisciplinary components. Converging risks such as those described herein are uniquely challenging to assess because their uncertainties multiply such that one must consider all possible outcomes of AI development in tandem with all possible outcomes of developments in the life sciences. For example, the evaluation criteria in a scenario-based risk assessment for virtual laboratories or 'cloud labs' that are too focused on the biological components may miss important bottlenecks, risks, or gaps that exist within the underlying AI components of the technology. Full assessment may not always be possible for some unpredictable technological advancements. Challenges also arise due to limited access to data and the inability to quantitatively assess emerging risks. Many frameworks include the potential adversary as a component to calculate risk.  11, 12  However, the adversary has a large range of potential attributes that are difficult to capture in their entirety. The adversary could be a nation, a nonstate actor, or an individual. When evaluating the likelihood that one of these adversaries could carry out ASSESSING ARTIFICIAL INTELLIGENCE AND BIOTECHNOLOGY RISKS an attack, assumptions must be made and variables considered on a spectrum. For example, a nation-state could be a low-resourced or well-resourced country. An individual could be anyone from an undergraduate biology student to a senior research professor. Such approaches fail to capture the range of possibilities and thus suffer from decreased granularity. Along these lines, without access to classified information, it is impossible to appreciate a full picture of adversary capability and intent. Furthermore, frameworks are sometimes focused on a technology's ability to aid in a specific element in the development process of a biological agent, which then makes it difficult to assess the risks of a full scenario in which it was implemented successfully. Conversely, assessments that solely focus on a full scenario fail to accurately capture the broad range of potential misapplications of a technology and consequently suffer from excessive specificity. After searching published literature, we identified 3 previously employed frameworks that inform the development of an assessment for converging technologies. These frameworks are examined in this paper in detail, with their strengths and limitations explored.
 Overview of Existing Frameworks The AAAS-FBI-UNICRI Framework The first framework was published in the 2014 report, National and Transnational Security Implications of Big Data in the Life Sciences, and was developed by a joint team consisting of members from the American Association for the Advancement of Science (AAAS) Center for Science, Technology, and Security Policy, the Biological Countermeasures Unit of the Federal Bureau of Investigation (FBI) Weapons of Mass Destruction Directorate, and the United Nations Interregional Crime and Justice Research Institute (UNICRI).  11  The main elements of the AAAS-FBI-UNICRI framework are highlighted in Table  1 . A strong component of this framework is the inclusion of security specialists, in addition to experts from multiple disciplines to evaluate the risks. While being one of the only attempts at risk assessment of a converging technology, the framework is limited in generality because it uses a scenario process to assess risks and its evaluation criteria are heavily skewed to big data and do not neatly map on to other converging technologies such as AI, despite many commonalities between them. The lack of generality is best illustrated by the framework table itself in which the ''Needed scientific expertise and skills'' block is divided into 2 sections to cover all 3 scenarios evaluated. This may be due to the fact that the working group developed the risk scenarios prior to developing the framework itself, resulting in a framework with a narrow scope. When attempting to implement a scenario where a malicious actor uses AI-enabled cloud labs to synthesize a pathogen in silico, the narrowness of this framework becomes more evident. The parameter, ''vulnerabilities in data repositories, software, and/or underlying cyber infrastructure''  11  does not map onto this scenario since none of those 3 vulnerabilities are necessarily present in the case of cloud labs accessed through legitimate channels, albeit for nefarious purposes.
 The NASEM Framework The second framework emerged from the National Academies of Science, Engineering, and Medicine (NASEM) 2018 report, Biodefense in the Age of Synthetic Biology. The NASEM framework (Table  2 ) is strong in that it actively considers the synergy with other technologies as a factor to calculate level of concern. The report also considers machine learning and automation as potential technologies that could facilitate the development of a biological agent. This framework, however, excludes actors who may be beneficial, considering the broad range of potential adversaries. The limitation in using this framework for converging AI and biotechnology risks is that the scope is too specific and restricted to risks from synthetic biology and technologies relevant to the design-build-test cycle. The authors of the report are transparent about these limitations and recognize that every factor may not map on to a specific technology. If one wanted to apply this framework to evaluate the level of concern posed by risks from the convergence of AI and biotechnology, they could face difficulties since the questions used to evaluate the factors are heavily skewed toward synthetic biology. However, since the framework was developed to assess capabilities of a technology to enable harm more   12  This framework (Table  3 ) benefits from including a governability assessment that is used if the average value of the factors from the risk assessment portion are ''medium'' or ''high.'' While assessing governability for approaches to governance has unquestionable value, it can also be an indicator for probability. If a technology is less governable, it is likely to have a higher chance for misuse and resulting risk. A potential weakness in Tucker's framework is the use of an ordinal scale with just 3 values (high, medium, or low). A spectrum spanning a range of only 3 could result in lost information and decreased specificity, which may have negative downstream effects in determining which technologies pose the greatest or lowest risk. Another potential shortcoming is that the framework considers the imminence of the technology to be of equal weight to the other factors in calculating risk. As a result, it could potentially fail to accurately categorize a long-term, high-consequence risk as one worth governing. In a 2017 working paper, Koblentz et al expanded on Tucker's framework, proposing the Tucker-Koblentz framework, which includes the range of applications for misuse as an indicator of risk.  14  This is particularly useful in the context of the converging technologies since they tend to have synergistic effects that increase the range of appli-cations. The authors also noted how risk assessments are implicitly impacted by the anticipated benefits of a technology since such technologies are more likely to be intensely pursued by researchers and private companies, which results in accelerated innovation and diffusion of knowledge. This perspective is useful in the context of the convergence of AI and biotechnology since the range of beneficial applications is rather large.
 The Converging Risk Landscape The overlap between AI and biotechnology is already considerable, with increasing integration of AI into various facets of the biological sciences likely to continue for the foreseeable future.  9  This poses a challenge for the health security risk assessment frameworks previously described and requires further understanding of the risks that need to be prioritized for examination. This section provides an overview of some of the intersections of AI and biotechnology that could be considered for future risk assessment exercises. These sections are presented as illustrative examples and are not meant to propose a taxonomy of risks.
 Synthetic Biology Increasingly, engineering principles are being applied to biology for research and manufacturing purposes. Synthesis and design of existing and novel microorganisms have many legitimate and beneficial applications; however, these same technologies could be used to engineer pathogens for malicious use. AI and machine learning have the potential
 ASSESSING ARTIFICIAL INTELLIGENCE AND BIOTECHNOLOGY RISKS to greatly aid in the design-build-test stages of the synthetic biology process.  15  Current DNA synthesis methods and gene editing technologies, such as the CRISPR-Cas9 system, still pose a high technical barrier for viable pathogen creation. Nonetheless, several recent controversial experiments have showcased the ability to de novo synthesize live viruses, including horsepox and the 1918 flu.  16  Research on emerging techniques, such as enzymatic synthesis of DNA, aims to revolutionize the field of synthetic biology in the next decade, with the end goal being benchtop printers able to synthesize gene-length sequences of DNA to completion.  17  Advances in this space are not only increasing the potential power and capability of synthetic biology but are also lowering the technical barriers and tacit knowledge required for its widespread use.  18  Within the realm of synthetic biology, AI could potentially lower some of the barriers for a malicious actor to design dangerous pathogens with custom features. It is worth noting, however, that AI will have a differential effect on technological hurdles, with some bottlenecks remaining difficult to traverse. For a more detailed examination of how the difficulty of design steps can affect the probability of a successful biological attack, we recommend the paper by Sandberg and Nelson also published in this issue.  19  AI has enabled a substantial reduction in the time required to design and test functionality in fields such as protein design.  20  It may also lower barriers in the design stages of the design-build-test cycle and could be used to identify optimal mutations of a pathogen that enhance virulence factors, expand host range, alter routes of transmission, encode resistance to medical countermeasures, and intensify mortality.  21  Genetic changes could be identified and made to a virus that is traditionally benign in nature, allowing it to bypass screening procedures currently used by mail-order DNA synthesis services. Similarly, AI could be leveraged to circumvent field detection technologies by identifying code sequences that could be altered without changing function.  11  Deep learning could potentially identify genetic functions that code for vulnerabilities and interconnections between the immune system and microbiome. It may be possible that malicious actors could leverage this information to engineer ''precision maladies,'' or pathogens that target important mechanisms the immune system or the microbiome of specific subpopulations.  22  It is important to note, however, that simply identifying certain vulnerabilities does not mean actors can then freely make changes, significant bottlenecks would still have to be traversed in the overall development process and the feasibility of targeting specific subpopulations would remain an unknown. Moreover, in the past few years, several studies have been published on predicting the pathogenic potential of natural and synthetic DNA using AI. Although improving biosecurity is one of the motives for this research, the results of this work may have the reverse effect and pose information hazards if the methods and algorithms are published. PaPrBaG and DeePaC are examples of machine learning algorithms predicting pathogenic potential that have made all source code available online.  23, 24  These algorithms could potentially aid a bad actor with the intent of designing a pathogen that can cause maximal harm. AI can not only assist in the design step of the process, but it can also, to a lesser degree, support the building and testing phases. The rise of AI-enabled cloud labs to automate procedures that used to require hands-on, tacit knowledge to perform tasks could further widen the pool of actors who could perform certain biology experiments. Cloud labs could allow a malicious actor to perform synthesis and carry out subsequent testing undetected and with greater ease.  25  The next evolution of cloud labs may further enable AI scientists capable of carrying out experiments in entirety by developing and testing hypotheses, interpreting results, and amending hypotheses for retesting, thus lowering the technical barrier even further.  26  Furthermore, directed evolution of pathogens, which spans all 3 phases of the design-build-test cycle, is another route a malicious actor could take in the development of a customized pathogen.  13  AI techniques have already been leveraged to accelerate this process and guide directed evolution in protein engineering.  27  Overall, in the future, it may be possible for a bad actor to perform complete hands-off, in silico design, building, and testing of a novel or recreated pathogen by leveraging the technologies and advancements outlined above.  25, 28  The potential democratization and synergy of these technologies pose potentially high-consequence risks that could elevate to a globally catastrophic level if not recognized and mitigated.
 Vulnerabilities and AI Systems As technologies in the biological sciences continue to be developed, AI-enabled offensive cyber-attacks may become an increasing biosecurity threat. For example, cybersecurity vulnerabilities in machines such as DNA synthesizers, could allow the introduction of malware to corrupt the design or audio-record printed DNA sequences, disrupt laboratory biosecurity, or allow intruder access to sensitive information.  29, 30  In addition, AI system vulnerabilities could also have downstream effects on human health. If healthcare becomes increasingly automated with the use of AI, so does increased risk for cyber-intrusion and subsequent manipulation of medical devices, which could result in harm for the end-user. TrapX Security Labs notes in their latest MED-JACK report  31  that many medical devices and equipment in use today, such as infusion pumps, medical lasers, LASIK surgical machines, and life support equipment are vulnerable to hijacking by malicious actors. Wireless connectivity is becoming increasingly common in implantable medical devices, opening up the risk of malicious actors mounting attacks from anywhere in the world.  32
 O'BRIEN AND NELSON Beyond the risks associated with medical device exploitation, it is possible that in the future computer systems will be integrated with human physiology and, therefore, pose novel vulnerabilities. Brain-computer interfaces (BCIs), traditionally used in medicine for motor-neurological disorders, are AI systems that allow for direct communication between the brain and an external computer.  33  BCIs allow for a bidirectional flow of information, meaning the brain can receive signals from an external source and vice versa. The neurotechnology company, Neuralink, has recently claimed that a monkey was able to control a computer using one of their implants.  34  This concept may seem farfetched, but in 2004 a paralyzed man with an implanted BCI was able to play computer games and check email using only his mind.  35  Other studies have shown a ''brainbrain'' interface between mammals is possible.  36, 37  In 2013, one researcher at the University of Washington was able to send a brain signal captured by electroencephalography over the internet to control the hand movements of another by way of transcranial magnetic stimulation.  38  Advances are occurring at a rapid pace and many previous technical bottlenecks that have prevented BCIs from widespread implementation are beginning to be overcome.  [39] [40] [41]  Research and development of BCIs have accelerated quickly in the past decade.  42  Future directions seek to achieve a symbiosis of AI and the human brain for cognitive enhancement and rapid transfer of information between individuals or computer systems.  34  Rather than having to spend time looking up a subject, performing a calculation, or even speaking to another individual, the transfer of information could be nearly instantaneous. There have already been numerous studies conducted researching the use of BCIs for cognitive enhancement in domains such as learning and memory, perception, attention, and risk aversion (one being able to incite riskier behavior).  43  Additionally, studies have explored the military applications of BCIs, and the field receives a bulk of its funding from US Department of Defense sources such as the Defense Advanced Research Projects Agency.  44  While the commercial implementation of BCIs may not occur until well into the future, it is still valuable to consider the risks that could arise in order to highlight the need for security-by-design thinking and avoid path dependency, which could result in vulnerabilities-like those seen with current medical devices-persisting in future implementations. Cyber vulnerabilities in current BCIs have already been identified, including those that could cause physical harm to the user and influence behavior.  [45] [46] [47] [48]  In a future where BCIs are commonplace alongside advanced understandings of neuroscience, it may be possible for a bad actor to achieve limited influence over the behavior of a population or cause potential harm to users. This issue highlights the need to have robust risk assessment prior to widespread technological adoption, allowing for regulation, governance, and security measures to take identified concerns into account.
 Combined Illustrative Framework Given the rapid developments in AI and biotechnology and the convergence of these 2 areas, existing risk assessment frameworks could be adapted to properly examine emerging threats. Table  3  is an illustrative framework that adapts the original AAAS-FBI-UNICRI framework by removing some elements while adding some from other frameworks to improve its capability to assess converging risks, such as those posed by AI and biotechnology. Convergent risk scenarios from this paper are included as examples of how this framework might be implemented; the completed boxes presented in the table are illustrative only and not meant to be conclusive, as there was no large-scale expert review and consensus. In Table  4 , the adversary spans the same range for each element, which may suggest it is a variable that might not be worth including or at least reconsidered. However, a comprehensive assessment by multidisciplinary experts with multiple scenarios would need to be conducted to determine this. In this framework, rather than using an ordinal scale of only 3 (low, moderate, high) to assess each parameter, an ordinal scale of 5 (very low, low, moderate, high, very high) was used to increase specificity and better delineate between scenarios. Each parameter is meant to be evaluated based on the projected timeline. For instance, in the case of BCIs, it is assumed the parameter will be highly accessible and democratized in the long-term. Governability was included as a component contributing to probability since a technology convergence that is less governable may be more likely to be maliciously used. Democratization, called ''accessibility'' in other frameworks, was included because a technology that is more accessible to a wide range of individuals is assumed to have higher potential for misuse. The remaining evaluation criteria (vulnerabilities, needed skills and expertise, magnitude of potential consequences, and existing countermeasures) were selected because they are common to many of the frameworks reviewed herein and we believe are integral to any risk assessment framework. ''Vulnerabilities'' is meant to describe and illuminate any gaps that could be exploited to cause harm. ''Needed skills and expertise'' is meant to describe the scope of potential adversaries and help identify potential bottlenecks. ''Magnitude of potential consequences'' and ''existing countermeasures'' are meant to contextualize the scope of potential harm and balance that with any mitigating factors in place. As one might expect, this combined framework has many limitations. For example, depending on the type of pathogen designed or synthesized, the high variance changes the amount of expertise needed and potential consequences over time, as does the nature and sophistication of the type of BCI exploitation. The framework also includes uncertainties related to the timeline and assessing governability, which highlights the high degree of assumptions required to complete and use it. Moreover, the framework lacks a standardized mechanism for evaluating factors in a way that calculates risk. The hope is that explicitly identifying ASSESSING ARTIFICIAL INTELLIGENCE AND BIOTECHNOLOGY RISKS these limitations can help illustrate the gaps and challenges in developing a comprehensive framework for converging technologies that may pose high-consequence risks.
 Recommendations
 Risk Assessment Framework Approaches A risk assessment framework is needed that is applicable to converging technologies in the life sciences that balances generality to capture a broad scope but also maintains enough specificity to capture nuances. It is paramount that such frameworks be developed and implemented by a multidisciplinary team of experts that includes both security officials and scientists. Such a strategy allows for a more comprehensive and, subsequently, more accurate depiction of the potential risks. To further increase robustness, existing frameworks should be iteratively built upon and improved, such as the combined framework presented in this paper. A general framework intended to apply to a diverse range of converging or emerging technologies should be developed prior to determining the scenario to be evaluated. It can then be revised as scenarios or technologies are assessed. This would benefit the applicability of such a framework to other scenarios or technologies rather than narrowly limiting it to a specific issue. The role that time plays in calculating risk for an emerging or converging technology should be weighed. While a technology that is assumed to not be relevant until the far future may not pose an immediate risk, it should still be considered, as it could pose a significant threat once the technology materializes. Additionally, projected timelines for emerging technologies are inherently uncertain and technologies may develop more rapidly than expected. Therefore, it is worth considering projected timelines as a framing element rather than a factor to calculate risk.
 Revisiting the AAAS-FBI-UNICRI Framework As 5 years have passed since publication of the initial AAAS-FBI-UNICRI report, we recommend this working group reconvene to reassess the risks outlined in this paper and update their framework to be more inclusive of other converging threats. As noted in their last report, ''if scientists and security experts can assess the risks and benefits together and on a routine basis, the potential risk that biological [big data] would be used by adversaries decreases.'' We believe that this team is best equipped to develop an iterative set of frameworks related to this topic because the workgroup includes knowledgeable experts from a wide range of disciplines and the issues related to big data and AI are similar in nature. With perspectives from experts in law enforcement, the life sciences, and AI, the team is uniquely positioned to take on such a task. While their initial project focused narrowly on big data, it would be of great value to extend that scope to AI in the life sciences and provide relevant stakeholders in academia, government, and the private sector with an informational risk assessment that will help them make informed research, policy, and security decisions on these topics. This working group can leverage frameworks-such as the NASEM and Tucker-Koblentz frameworks-that have been developed since their initial report to improve upon their own. The Koblentz et al  14  paper could provide them with an insightful review of the strengths and weaknesses of many previous frameworks and would be a useful resource as the working group develops the next iteration. Components from the NASEM framework could be leveraged to increase the generality and broadness of their framework. The illustrative framework presented in this paper (Table  4 ) may serve as a useful starting point for the joint team.
 Conclusion AI and biotechnology are both rapidly growing fields. The biosecurity risks arising at their points of convergence are vast, highlighting the need for comprehensive mapping and thorough, robust risk assessment. Current frameworks used for risk assessment in health security settings may fail to capture the nuanced areas of these converging fields and could benefit from considering generality and timeline for technology adoption. We recommend that a multidisciplinary working group comprised of AAAS, the FBI, UNICRI, and other specialists gather to formulate a framework to comprehensively assess risks the results from converging technologies such as AI and biotechnology as well as others that may emerge in the future. The combined framework presented in this paper could serve as a starting point for the development of more robust biosecurity risk assessments for converging technologies like AI and biotechnology. While this new framework does not solve all of the identified challenges, it explicitly elucidates the gaps in our ability to accurately assess risks associated with emerging and converging technologies and will hopefully facilitate adaption of previous frameworks in order to develop a more inclusive health security risk assessment that properly balances generality and specificity and is regularly and iteratively revisited. Table 1 . 1 Main AAAS-FBI-UNICRI Framework Elements to Evaluate the Risks of Big Data 11   Risk Assessment Probability Consequences Adversary Vulnerabilities in Needed scientific expertise and skills Severe consequences Sufficient existing data repositories, software, and/or underlying cyber infrastructure To exploit vulnerabilities in the system To use big data analytics to design harmful biological agents to economics, political system, society, health, environment, and/or agriculture countermeasures
 Table 2 . 2 Main NASEM Framework Elements to Evaluate Level of Concern of Technologies in Synthetic Biology 14   Level of Concern About the Capability Usability of the Technology Usability as a Weapon Requirements of Actors Potential for Mitigation Ease of use Production and delivery Access to expertise Deterrence and prevention capabilities Rate of development Scope of casualty Access to resources Capability to recognize an attack Barriers to use Predictability of results Organizational footprint Attribution capabilities Synergy with other requirements Consequence management capabilities technologies
 Table 3 . 3 Main Tucker Decision Framework Elements to Assess Misuse and Governability 13   Risk of Misuse Governability Accessibility Embodiment Ease of misuse Maturity Magnitude of potential harm Convergence Imminence of potential misuse Rate of advance International diffusion Note: The elements were determined by averaging the values (high, medium, or low) yielded by the parameters. Governability was determined if risk of misuse is medium or high.
 Table 4 . 4 Illustrative Framework Adapted from AAAS-FBI-UNICI, NASEM, and Tucker Frameworks [12] [13] [14] [15]   Convergent Risk Scenarios In Silico Design In Silico Synthesis Brain-Computer of a Pathogen of a Pathogen Interface Exploitation
			 Volume 18, Number 3, 2020

Abstract: Rapid developments are currently taking place in the fields of artificial intelligence (AI) and biotechnology, and applications arising from the convergence of these 2 fields are likely to offer immense opportunities that could greatly benefit human health and biosecurity. The combination of AI and biotechnology could potentially lead to breakthroughs in precision medicine, improved biosurveillance, and discovery of novel medical countermeasures as well as facilitate a more effective public health emergency response. However, as is the case with many preceding transformative technologies, new opportunities often present new risks in parallel. Understanding the current and emerging risks at the intersection of AI and biotechnology is crucial for health security specialists and unlikely to be achieved by examining either field in isolation. Uncertainties multiply as technologies merge, showcasing the need to identify robust assessment frameworks that could adequately analyze the risk landscape emerging at the convergence of these 2 domains. This paper explores the criteria needed to assess risks associated with AI and biotechnology and evaluates 3 previously published risk assessment frameworks. After highlighting their strengths and limitations and applying to relevant AI and biotechnology examples, the authors suggest a hybrid framework with recommendations for future approaches to risk assessment for convergent technologies.


Working together to face humanity’s greatest threats preprint.tei.xml


Chenglong Wang
Rudy Bunel
Edward Grefenstette
♣ Deepmind

Knowing When to Stop: Evaluation and Verification of Conformity to Output-size Specifications

Introduction Neural networks with variable output lengths have become ubiquitous in several applications. In particular, recurrent neural networks (RNNs) such as LSTMs  [17] , used work done during an internship at DeepMind | now at Facebook AI Research to form "sequence" models  [30] , have been successfully and extensively applied in in image captioning  [34, 28, 21, 6, 12, 37, 26, 24, 1] , video captioning  [33, 38, 35, 40, 41] , machine translation (MT)  [30, 9] , summarization  [10] , and in other sequence-based transduction tasks. The ability of these sequence neural models to generate variable-length outputs is key to their performance on complex prediction tasks. However, this ability also opens a powerful attack for adversaries that try to force the model to produce outputs of specific lengths that, for instance, lead to increased computation or affect the correct operation of down-stream modules. To address this issue, we introduce the output-length modulation problem where given a specification of the form that the model should produce outputs with less than a certain maximum length, we want to find adversarial examples, i.e. search for inputs that lead the model to produce outputs with a larger length and thus show that the model under consideration violates the specification. Different from existing work on targeted or untargeted attacks where the goal is to perturb the input such that the output is another class or sequence in the development dataset (thus within the dataset distribution), the outputmodulation problem requires solving a more challenging task of finding inputs such that the output sequences are outside of the training distribution, which was previously claimed difficult  [5] . The naive approach to the solution of the output-length modulation problem involves a computationally intractable search over a large discrete search space. To overcome this, we develop an easy-to-compute differentiable proxy objective that can be used with gradient-based algorithms to find output-lengthening inputs. Experimental results on Machine Translation show that our adversarial outputlengthening approach can produce outputs that are 50 times longer than the input. However, when evaluated on the image-to-text image captioning model, the method is less successful. There could have been two potential reasons for this result: the image-to-text architecture is truly robust, or the adversarial approach is not powerful enough to find adversarial examples for this model. To resolve this question, we develop a verification method for checking and formally proving whether a network is consistent with the outputsize specification for the given range of inputs. To the best of our knowledge, our verification algorithm is the first formal verification approach to check properties of recurrent models with variable output lengths. Our Contributions To summarize, the key contributions of this paper are as follows: • We propose and formulate the novel output-size modulation problem to study the behaviour of neural architectures capable of producing variable length outputs, and we study its evaluation and verification problems. • For evaluation, we design an efficiently computable differentiable proxy for the expected length of the output sequence. Experiments show that this proxy can be optimized using gradient descent to efficiently find inputs causing the model to produce long outputs. • We demonstrate that popular machine translation models can be forced to produce long outputs that are 50 times longer than the input sequence. The long output sequences help expose modes that the model can get stuck in, such as undesirable loops where they continue to emit a specific token for several steps. • We demonstrate the feasibility of formal verification of recurrent models by proposing the use of mixedinteger programming to formally verify that a certain neural image-captioning model will be consistent with the specification for the given range of inputs. Motivations and Implications Our focus on studying the output-length modulation problem is motivated by the following key considerations: • Achieving Computational Robustness: Many ML models are now offered as a service to customers via the cloud. In this context, ML services employing variable-output models could be vulnerable to denialof-service attacks that cause the ML model to perform wasteful computations by feeding it inputs that induce long outputs. This is particularly relevant for variable compute models, like Seq2Seq  [9, 30] . Given a trained instance of the model, no method is known to check for the consistency of the model with a specification on the number of computation steps. Understanding the vulnerabilities of ML models to such outputlengthening and computation-increasing attacks is important for the safe deployment of ML services. • Understanding and Debugging Models: By designing inputs that cause models to produce long outputs, it is possible to reason about the internal representations learned by the model and isolate where the model exhibits undesirable behavior. For example, we find that an English to German sequence-to-sequence model can produce outputs that end with a long string of question marks ('?'). This indicates that when the output decoder state is conditioned on a sequence of '?'s, it can end up stuck in the same state. • Uncovering security vulnerabilities through adversarial stress-testing: The adversarial approach to outputlength modulation tries to find parts of the space of inputs where the model exhibits improper behavior. Such inputs does not only reveal abnormal output size, but could also uncover other abnormalities like the privacy violations of the kind that were recently revealed by  [4]  where an LSTM was forced to output memorized data. • Canonical specification for testing generalization of variable-output models: Norm-bounded perturbations of images  [31]  have become the standard specification to test attacks and defenses on image classifiers. While the practical relevance of this particular specification can be questioned  [14] , it is still served as a useful canonical model encapsulating the essential difficulty in developing robust image classifiers. We believe stability of output-lengths can serve a similar purpose: as a canonical specification for variable outputlength models. The main difficulties in studying variable output length models in an adversarial sense (the non-differentiability of the objective with respect to inputs) are exposed in output-lengthening attack, making it a fertile testing ground for both evaluating attack methods and defenses. We hope that advances made here will facilitate the study of robustness on variable compute models and other specifications for variableoutput models such as monotonicity.
 Related Work There are several recent studies on generating adversarial perturbations on variable-output models.  [27, 20]  show that question answering and machine comprehension models are sensitive to attacks based on semantics preserving modification or the introduction of unrelated information.  [11, 39]  find that character-level classifiers are highly sensitive to small character manipulations.  [29]  shows that models predicting the correctness of image captions struggle against perturbations consisting of a single word change.  [5]  and  [8]  further study adversarial attacks for sequential-output models (machine-translation, image captioning) with specific target captions or keywords. We focus on sequence output models and analyze the output-length modulation problem, where the models should produce outputs with at least a certain number of output tokens. We study whether a model can be adversarially perturbed to change the size of the output, which is a more challenging task compared to targeted attacks (see details in Section 3). On the one hand, existing targeted attack tasks aim to perturb the input such that the output is another sequence in the validation dataset (thus within the training distribution), but attacking output size requires the model to generate out-of-distribution long sequences. On the other hand, since the desired output sequence is only loosely constrained by the length rather than directly provided by the user, the attack algorithm is required to explore the output size to make the attack possible. For models that cannot be adversarially perturbed, we develop a verification approach to show that it isn't simply a lack of power by the adversary but the sign of true robustness from the model. Similar approaches have been investigated for feedforward networks  [3, 7, 32]  but our work is the first to handle variable output length models and the corresponding decoding mechanisms.
 Modulating Output-size We study neural network models capable of producing outputs of variable length. We start with a canonical abstraction of such models, and later specialize to concrete models used in machine translation and image captioning. We denote by x the input to the network and by X the space of all inputs to the network. We consider a set of inputs of interest S, which can denote, for example, the set of "small" 1 perturbations of a nominal input. We study models that produce variable-length outputs sequentially. Let y t 2 Y denote the t-th output of the model, where Y is the output vocabulary of the model. At each timestep, the model defines a probability over the next element P (y t+1 |x, y 0:t ). There exists a special end-of-sequence element eos 2Y that signals termination of the output sequence. In practice, different models adopt different decoding strategies for generating y t+1 from the probability P (y t+1 |x, y 0:t )  [13, 19, 22] . In this paper, we focus on the commonly used deterministic greedy decoding strategy  [13] , where at each time step, the generated token is given by the argmax over the logits: y 0 = argmax {P (•|x)} (1a) y t+1 = argmax {P (•|x, y 0:t )} if y t 6 = eos (1b) Since greedy decoding is deterministic, for a given sample x with a finite length output, we can define the length of the  1  The precise definition of small is specific to the application studied. greedily decoded sequence as: `(x)=t s.t y t = eos y i 6 = eos 8i<t y i+1 = argmax {P (.|x, y 0:i )}8 i<t (2) Note that there is a unique t that satisfies the above conditions, which is precisely the first t at which y t = eos when using greedy decoding. Output length modulation specification A network is said to satisfy the output length modulation specification parameterized by S, K, if for all inputs x in S, the model terminates within K steps under greedy decoding for all x 2S, formally: 8x 2S `(x)  K (3 ) In Section 3.1, we study the problem of finding adversarial examples, i.e., searching for inputs that lead the model to produce outputs with a larger length and thus show that the model violates the specification. In Section 4, we use formal verification method to prove that a model is consistent with the specification for the given range of inputs, if such attacks are indeed impossible.
 The Output-Size Modulation Problem In order to check whether the specification, Eq. (  3 ), is valid, one can consider a falsification approach that tries to find counterexamples proving that Eq. (  3 ) is false. If an exhaustive search over S for such counterexamples fails, the specification is indeed true. However, exhaustive search is computationally intractable; hence, in this section we develop gradient based algorithms that can efficiently find counterexamples (although they may miss them even if they exist). To develop the falsification approach, we study the solution to the following optimization objective: max x2S `(x) (4) where S is the valid perturbation space. If the optimal solution x in the space S has `(x) > K, then (3) is false. The attack spaces S we consider in this paper include both continuous inputs (for image-to-text models) and discrete inputs (for Seq2Seq models).
 Continuous inputs: For continuous inputs, such as image captioning tasks, the input is an n ⇥ m image with pixel values normalized to be in the range  [ 1, 1] . x is an n ⇥ m matrix of real numbers and X =[ 1, 1] n⇥m . We define the perturbation space S(x, ) as follows: S(x, )={x 0 2X |kx 0 xk 1  } i.e., the space of perturbations of the input x in the `1 ball. Discrete inputs: For discrete inputs, e.g., machine translation tasks, inputs are discrete tokens in a language vocabulary. Formally, given the vocabulary V of the input language, the input space X is defined as all sentences composed of tokens in V , i.e., X = {(x 1 ,...,x n ) | x i 2 V, n > 0}. Given an input sequence x =( x 1 ,...,x n ), we define the -perturbation space of a sequence as all sequences of length n with at most d • ne tokens different from x (i.e., 2 [0, 1] denotes the percentage of tokens that an attacker is allowed to modify). Formally, the perturbation space S(x, ) is defined as follows: S(x, δ)={(x 0 1 ,...,x 0 n ) 2 V n | n P i=1 [xi 6 = x 0 i ] dδ • ne}
 Extending Projected Gradient Descent Attacks In the projected gradient descent (PGD) attacks  [25] , 2 given an objective function J(x), the attacker calculates the adversarial example by searching for inputs in the attack space to maximize J(x). In the basic attack algorithm, we perform the following updates at each iteration: x 0 =Π S(x,δ) (x + ↵r x J(x)) ( 5 ) where ↵>0 is the step size and Π S(x,δ) denotes the projection of the attack to the valid space S(x, ). Observe that the adversarial objective in Eq. (  4 ) cannot be directly used as J(x) to update x as the length of the sequence is not a differentiable objective function. This hinders the direct application of PGD to output-lengthening attacks. Furthermore, when the input space S is discrete, gradient descent cannot be directly be used because it is only applicable to continuous input spaces. In the following, we show our extensions of the PGD attack algorithm to handle these challenges. Greedy approach for sequence lengthening We introduce a differentiable proxy of `(x). Given an input x whose decoder output logits are (o 1 ,...,o k ) (i.e., the decoded sequence is y = (argmax(o 1 ),...,argmax(o k ))), instead of directly maximizing the output sequence length, we use a greedy algorithm to find an output sequence whose length is longer than k by minimizing the probability of the model to terminate within k steps. In other words, we minimize the log probability of the model to produce eos at any of the timesteps between 1 to k. Formally, the proxy objective J is defined as follows: J(x)= k P t=1 max ⇢ o t [eos] max z6 =eos o t [z], ✏ where ✏>0 is a hyperparameter to clip the loss. This is piecewise differentiable w.r.t. the inputs x (in the same  2  Here the adversarial objective is stated as maximization, so the algorithm is Projected Gradient Ascent, but we stick with the PGD terminology since it is standard in the literature sense that the ReLU function is differentiable) and can be efficiently optimized using PGD.
 Continuous relaxation for discrete inputs While we can apply the PGD attack with the proxy objective on the model with continuous inputs by setting the projection function Π S(x,δ) as the Euclidean projection, we cannot directly update discrete inputs. To enable a PGDtype attack in the discrete input space, we use the Gumbel trick  [18]  to reparameterize the input space to perform continuous relaxation of the inputs. Given an input sequence x =( x 1 ,...,x n ), for each x i , we construct a distribution ⇡ i 2 R |V | initialized with ⇡ i [x i ]=1and ⇡ i [z]= 1 for all z 2 V \{x i }. The softmax function applied to ⇡ i is a probability distribution over input tokens at position i with a mode at x i . With this reparameterization, instead of feeding x =( x 1 ,...,x n ) into the model, we feed the Gumbel softmax sampling from the distribution (u 1 ,...,u n ). The sample x =(x 1 ,...,x n ) is calculated as follows: u i ⇠ Uniform(0, 1); g i = log( log(u i )) p = softmax(⇡); xi = softmax( gi+log pi τ ) where ⌧ is the Gumbel-softmax sampling temperature that controls the discreteness of x. With this relaxation, we perform PGD attack on the distribution ⇡ at each iteration. Since ⇡ i is unconstrained, the projection step in (  5 ) is unnecessary. When the final ⇡ 0 =( ⇡ 0 1 ,...,⇡ n ) is obtained from the PGD attack, we draw samples x 0 i ⇠ Categorical(⇡ i ) to get the final adversarial example for the attack.
 Verified Bound on Output Length While heuristics approaches can be useful in finding attacks, they can fail due to the difficulty of optimizing nondifferentiable nonconvex functions. These challenges show up particularly when the perturbation space is small or when the target model is trained with strong bias in the training data towards short output sequences (e.g., the Show-and-Tell model as we will show in Section 6). Thus, we design a formal verification approach for complete reasoning of the output-size modulation problem, i.e., finding provable guarantees that no input within a certain set of interest can result in an output sequence of length above a certain threshold. Our approach relies on counterexample search using intelligent brute-force search methods, taking advantage of powerful modern integer programming solvers  [15] . We encode all the constraints that an adversarial example should satisfy as linear constraints, possibly introducing additional binary variables. Once in the right formalism, these can be fed into an off-the-shelf Mixed Integer Programming (MIP) solver, which provably solves the problem, albeit with a potentially large computational cost. The constraints consist of four parts: (1) the initial restrictions on the model inputs (encoding S(x, )),  (2)  the relations between the different activations of the network (implementing each layer), (3) the decoding strategy (connection between the output logits and the inputs at the next step), and (4) the condition for it being a counterexample (ie. a sequence of length larger than the threshold). In the following, we show how each part of the constraints is encoded into MIP formulas. Our formulation is inspired by pior work on encoding feed-forward neural networks as MIPs  [3, 7, 32] . The image captioning model we use consists of an image embedding model, a feedforward convolutional neural network that computes an embedding of the image, followed by a recurrent network that generates tokens sequentially starting with the initial hidden state set to the image embedding. The image embedding model is simply a sequence of linear or convolutional layers and ReLU activation functions. Linear and convolutional layers are trivially encoded as linear equality constraints between their inputs and outputs, while ReLUs are represented by introducing a binary variable and employing the big-M method  [16] : x i = max (x i , 0) ) i 2{0, 1},x i 0 (6a) x i  u i • i ,x i xi (6b) x i  xi l i • (1 i ) (6c) with l i and u i being lower and upper bounds of xi which can be obtained using interval arithmetic (details in  [3] ). Our novel contribution is to introduce a method to extend the techniques to handle greedy decoding used in recurrent networks. For a model with greedy decoding, the token with the most likely prediction is fed back as input to the next time step. To implement this mechanism as a mixed integer program, we employ a big-M method  [36] : o max = max y2Y (o y ) ) o max o y , y 2{0, 1}8 y 2Y (7a) o max  o y +(u l y )(1 y ) 8y 2Y (7b) X y2Y y =1 (7c) with l y ,u y being a lower/upper bound on the value of o y and u = max y2Y u y (these can again be computed using interval arithmetic). Implementing the maximum in this way gives us both a variable representing the value of the maximum (o max ), as well as a one-hot encoding of the argmax ( y ). If the embedding for each token is given by {emb i | i 2Y}, we can simply encode the input to the following RNN timestep as P y2Y y • emb y , which is a linear function of the variables that we previously constructed. With this mechanism to encode the greedy decoding, we can now unroll the recurrent model for the desired number of timesteps. To search for an input x with output length `(x) K, we unroll the recurrent network for K steps and attempt to prove that at each timestep, eos is not the maximum logit, as in  (2) . We setup the problem as: max min t=1.. K  max z6 =eos o t [z] o t [eos] (8) where o(k) represents the logits in the k-th decoding step. We use an encoding similar to the one of Equation (  7 ) to represent the objective function as a linear objective with added constraints. If the global optimal value of Eq. (  8 ) is positive, this is a valid counterexample: at all timesteps t 2 [1.. K], there is at least one token greater than the eos token, which means that the decoding should continue. On the other hand, if the optimal value is negative, that means that those conditions cannot be satisfied and that it is not possible to generate a sequence of length greater than K. The eos token would necessarily be predicted before. This would imply that our robustness property is True.
 Target Model Mechanism We use image captioning and machine translation models as specific target examples to study the output length modulation problem. We now introduce their mechanism.
 Image captioning models The image captioning model we consider is an encoder-decoder model composed of two modules: a convolution neural network (CNN) as an encoder for image feature extraction and a recurrent neural network (RNN) as a decoder for caption generation  [34] . Formally, the input to the model x is an m ⇥ n sized image from the space X =[ 1, 1] m⇥n , the CNN-RNN model computes the output sequence as follows: i 0 = CNN(x); h 0 = 0 o t ,h t+1 = RNNCell(i t ,h t ) y t = arg max(o t ); i t+1 = emb(y t ) where emb denotes the embedding function. The captioning model first run the input image x through a CNN to obtain the image embedding and feed it to the RNN as the initial input i 0 along with the initial state h 0 . At each decode step, the RNN uses the input i t and state h t to compute the new state h t+1 as well as the logits o t representing the log-probability of the output token distribution in the vocabulary. The output y t is the token in the vocabulary with highest probability based on o t , and it is embedded into the continuous space using an embedding matrix W emb as W emb [y t ]. The embedding is fed to the next RNN cell as the input for the next decoding step.
 Machine translation models The machine translation model is an encoder-decoder model  [30, 9]  with both the encoder and the decoder being RNNs. Given the vocabulary V in of the input language, the valid input space X is defined as all sentences composed of tokens in V in , i.e., X = {(x 1 ,...,x n ) | x i 2 V, n > 0}. Given an input sequence x =( x 1 ,...,x n ), the model first calculates its embedding f (x) RNN as follows (h e t and i e t denote the encoder hidden states and the inputs at the t-th time step, respectively. emb e denotes the embedding function for each token in the vocabulary). The model then uses f (x) as the initial state h 0 for the decoder RNN to generate the output sequence, following the same approach as in the image captioning model. h e 0 = 0; i e t = emb e (x t ) h e t = RNNCell e (i e t ,h e t 1 ); f (x)=h e n
 Experiments We consider the following three models, namely, Multi-MNIST captioning, Show-and-Tell  [34] , and Neural Machine Translation (NMT) [30, 9, 2] models.
 Details of models and datasets Multi-MNIST. The first model we evaluate is a minimal image captioning model for Multi-MNIST dataset. The Multi-MNIST dataset is composed from the MNIST dataset (Figure  1 left ). Each image in the dataset is composed from 1-3 MNIST images: each MNIST image (28 * 28) is placed on the canvas of size (28 * 112) with random bias on the x-axis. The composition process guarantees that every MNIST image is fully contained in the canvas without overlaps with other images. The label of each image is the list of MNIST digits appearing in the canvas, ordered by their x-axis values. The dataset contains 50,000 training images and 10,000 test images, where the training set is constructed from MNIST training set and the test set is constructed from MNIST test set. The images are normalized to [ 1, 1] before feeding to the captioning model. For this dataset, we train a CNN-RNN model for label prediction. The model encoder is a 4-layers CNN (2 convolution layers and 2 fully connected layers with ReLU activation functions applied in between). The decoder is a RNN with ReLU activation. Both the embedding size and the hidden size are set to 32. We train the model for 300 steps with Adam optimizer based on the cross-entropy loss. The model achieves 91.2% test accuracy, and all predictions made by the model on the training set have lengths no more than 3. Show-and-Tell. Show and Tell model  [34]  is an image captioning model with CNN-RNN encoder-decoder architecture similar to the Multi-MNIST model trained on the MSCOCO 2014 dataset  [23] . Show-and-Tell model uses Inception-v3 as the CNN encoder and an LSTM for caption generation. We use a public version of the pretrained model 3 for evaluation. All images are normalized to  [ 1, 1]  3 https://github.com/tensorflow/models/ and all captions in the dataset are within length 20. NMT. The machine translation model we study is a Seq2Seq model  [30, 9]  with the attention mechanism  [2]  trained on the WMT15 German-English dataset. The model uses byte pair segmentation (BPE) subword units  [28]  as vocabulary. The input vocabulary size is 36, 548. The model consists of 4-layer LSTMs of 1024 units with a bidirectional encoder, with the embedding dimension set to 1024. We use a publicly available checkpoint 4 with 27.6 BLEU score on the WMT15 test datasets in our evaluation. At training time, the model restricts the maximum decoding length to 50.
 Adversarial Attacks Our first experiment studies whether adversarial inputs exist for the above models and how they affect model decoding. For each model, we randomly select 100 inputs from the development dataset as attack targets, and compare the output length distributions from random perturbation and PGD attacks. Multi-MNIST We evaluate the distribution of output lengths of images with an `1 perturbation radius of 2{ 0.001, 0.005, 0.01, 0.05, 0.1, 0.5} using both random search and PGD attack. In random search, we generate 10,000 random images within the given perturbation radius for each image in the target dataset as new inputs to the model. In PGD attack, the adversarial inputs are obtained by running 10,000 gradient descent steps with an learning rate of 0.0005 using the Adam Optimizer. Neither of the attack methods can find any adversarial inputs for 2{0.001, 0.005, 0.01} perturbation radius (i.e., no perturbation is found for any images in the target dataset within the above to generate an output sequence longer than the original one). Figure  2  shows the distribution of the output lengths for images with different perturbation radius. Results show that the PGD attack is successful at finding attacks that push the distribution of output lengths higher, particularly at larger values of . Examples of adversarial inputs found by the model are shown in Figure  1 .
 Show-and-Tell For the Show-and-Tell model, we generate attacks within an `1 perturbation radius of =0.5 with both random search and PGD attack on 500 images randomly selected from the development dataset. However, except one adversarial input found by PGD attack that would cause the model to produce an output with size 25, no other adversarial inputs are found that can cause the model to produce outputs longer than 20 words, which is the training length cap. Our analysis shows that the difficulty of attacking the model is resulted from its strong bias on the output  sequence distribution and the saturation of sigmoid gates in the LSTM decoder. This result is also consistent with the result found by  [5]  that Show-and-Tell model is "only able to generate relevant captions learned from the training distribution". NMT We evaluate the NMT model by comparing the output length distribution from adversarial examples generated from random search and PGD attack algorithms. We randomly draw 100 input sentences from the development dataset. The maximum input length is 78 and their corresponding translations made by the model are all within 75 tokens. We consider the perturbation 2{0.3, 0.5, 1.0}. 1. Random Search. In each run of the random attack, given an input sequence with length n, we first randomly select d • ne locations to modify, then randomly select substitutions of the tokens at these locations from the input vocabulary, and finally run the NMT model on the modified sequence. We run 10,000 random search steps for the 100 selected inputs, and show the distributions of all outputs obtained from the translation (in the total 1M output sequences). 2. PGD Attack. In PGD attack, we also start by randomly selecting d • ne locations to modify for each input sequence with length n. We then run 800 iterations of PGD attack with Adam optimizer using an initial learning rate of 0.005 to find substitutions of the tokens at these selected locations. We plot the output length obtained from running these adversarial inputs through the translation model. Figure  3  shows the distribution of output sequence lengths obtained from random search methods with different . We aggregate all sequences with length longer than 100 into the group '>100' in the plot. Results show that even random search approach could often craft inputs such that the corresponding output lengths are more than 75 and occasionally generates sentences with output length over 100. The random search algorithm finds 79, 11, 3 for =0.3, 0.5, 1, respectively, among the 1M translations that are longer than 100 tokens (at small , the search space is more restricted, and random search has a higher success rate of finding long outputs). Notably, the longest sequence found by the random search is a sequence with output length 312 tokens, where the original sequence is only 6.     to larger perturbations. With an unconstrained perturbation = 100%, PGD attack algorithm discovers more adversarial inputs whose outputs are longer than 100 tokens (10% among all attacks), which is 1000⇥ more often than random search. As an extreme case, PGD attack discovered an adversarial input with length 3 whose output length is 575. Examples of adversarial inputs and their corresponding model outputs are shown in Figure  5  and the Appendix; we find out that a common feature of the long outputs produced by the translation model is that the output sequences often end with long repetitions of one (or a few) words.
 names name names name names grammatically name names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names eos To analyze the bottleneck of PGD attack on the NMT model, we further run a variation of the PGD attack where the attack space is the (continuous) word embedding space as opposed to the (discrete) token space: we allow the attacker to directly modify token embeddings at selected attack locations to any other vector. PGD attack on this variation achieves a 100% success rate to find adversarial token embeddings such that the model outputs are longer than 500 tokens. This indicates that the discrete space is a bottleneck for consistently finding stronger attacks.
 Verification Our implementation of the verification algorithm using the mixed integer programming (8) is implemented using SCIP  [15] . We run our verification algorithm on the Multi-Figure  6 . Proportion of samples being provably robust (in blue), vulnerable (in red) or of unknown robustness status (in gray) to an attack attempting to make the model generate an output sequence longer than the ground truth, as a function of the perturbation radius allowed to the attacker. For small radiuses, the MIP can prove that no attacks can be successful. For large radiuses, we are able to find successful attacks. MNIST dataset, attempting to formally prove the robustness of the model to attacks attempting to generate an output longer than the ground truth. For each input image, we set a timeout of 30 minutes for the solver. The results in Figure  6  show that our verification algorithm is able to verify formally that no attacks exists for small perturbation radiuses. However, as the perturbation radius increases, our verification algorithm times out and is not able to explore the full space of valid perturbations and thus cannot decide whether attacks exists in the given space. For this reason, the number of robust samples we report is only a lower bound on the actual number of robust samples. Conversely, the vulnerable samples that we exhibit give us an upper bound on the number of those robust samples. As shown by the large proportion of samples of unknown status, there is currently still a gap between the capabilities of formal verification method and attacks.
 Conclusion In this paper, we introduce the existence and the construction of the output-length modulation problem. We propose a differentiable proxy that can be used with PGD to efficiently find output-lengthening inputs. We also develop a verification approach to formally prove certain models cannot produce outputs greater than a certain length. We show that the proposed algorithm can produce adversarial examples that are 50 times longer than the input for machine translation models, and the image-captioning model can conform the output size is less than certain maximum length using the verification approach. In future work, we plan to study adversarial training of sequential output models using the generated attacks, to models that are robust against output lengthening attacks, and further, verify this formally. Figure 1 . 1 Figure 1. Multi-MNIST examples (left), adversarial examples found by PGD attack (mid), and their differences. For the first group, the model correctly predicts label l1 =[6, 1] on the original image but predicts l 0 1 =[6, 1, 1] for its corresponding adversarial input. Predictions on the original/adversarial inputs made by model for the second group are l2 =[0, 7, 4],l 0 2 =[0, 1, 4, 3], and l3 =[3],l 0 3 =[3, 3, 5, 3] for the third group. The adversarial inputs in the first/second/third groups are found within the perturbation radius δ1 =0 .1,δ2 =0 .25,δ3 = 0.25.
 Figure 2 . 2 Figure2. The distribution of output length for random search (denoted as Rand) and PGD attack with different perturbation radius δ. The x-axis denotes the output length and y-axis denotes the number of outputs with the corresponding length. δ =0(no perturbation allowed) refers to the original output distribution of the target dataset.
 Figure 3 . 3 Figure 3. The histogram representing the output length distribution of the NMT model using random search with different perturbations (δ 2{ 0.3, 0.5, 1}). The x-axis shows the output length. y-axis values are divided by 10,000, the number of random perturbation rounds per image.
 Figure 4 4 Figure4shows the result from attacking the NMT model with PGD attack. Results show that PGD attack has relatively low success rate at lower perturbations compared
 Figure 4 . 4 Figure 4. The histogram representing the output length distribution of the NMT model under PGD attack for different δ. x-axis shows the output length and y-axis shows the number of instances with the corresponding length.
 ( I) Die Waffe wird ausgestellt und durch den Zaun übergeben. (O) The weapon is issued and handed over by the fence . eos (I 0 ) Die namen name descri und ames utt origin i.e. meet grammatisch . (O 0 )
 Figure 5 . 5 Figure 5. An example of German to English translation where I,O refer to an original sequence in the dataset and the corresponding translation made by the model. I 0 ,O 0 refer to an adversarial example found by PGD attack and the corresponding model translation.
			 https://github.com/tensorflow/nmt

Abstract: Models such as Sequence-to-Sequence and Image-to-Sequence are widely used in real world applications. While the ability of these neural architectures to produce variablelength outputs makes them extremely effective for problems like Machine Translation and Image Captioning, it also leaves them vulnerable to failures of the form where the model produces outputs of undesirable length. This behaviour can have severe consequences such as usage of increased computation and induce faults in downstream modules that expect outputs of a certain length. Motivated by the need to have a better understanding of the failures of these models, this paper proposes and studies the novel output-size modulation problem and makes two key technical contributions. First, to evaluate model robustness, we develop an easy-to-compute differentiable proxy objective that can be used with gradient-based algorithms to find output-lengthening inputs. Second and more importantly, we develop a verification approach that can formally verify whether a network always produces outputs within a certain length. Experimental results on Machine Translation and Image Captioning show that our output-lengthening approach can produce outputs that are 50 times longer than the input, while our verification approach can, given a model and input domain, prove that the output length is below a certain size.