Jess Whittlestone
Kai Arulkumaran
Matthew Crosby

The Societal Implications of Deep Reinforcement Learning

Introduction Artificial intelligence (AI) is already having an impact on many areas of society  (Whittaker et al., 2018; Whittlestone et al., 2019) , and further advances in AI research are likely to precipitate much greater impacts. There is a growing community of researchers and practitioners doing excellent work to identify and address ethical and societal issues raised by current applications of AI  (Schiff et al., 2020) . Given the pace of AI progress, it is crucial that this community also think ahead about what challenges might be raised by future advances, and how forms of governance, norms, and standards being developed around AI today can best prepare us for the future evolution of AI systems. Of course, predicting the future of any technology with certainty is difficult. One way to strike a balance between thinking ahead while still being grounded in reality is to ask the question: what might be the impacts on society if current successful trends in AI research continue and result in new and widespread societal applications? This paper explores the potential societal implications of one avenue of AI research currently showing promise: Deep Reinforcement Learning (DRL). The paper is primarily aimed at researchers and policy practitioners working on the ethical, societal and governance implications of AI, but may also be of interest to AI researchers concerned with the impacts of ©2021 AI Access Foundation. All rights reserved. their own work. Our discussion aims to provide important context and a clear starting point for the AI ethics and governance community to begin considering the societal implications of DRL in more depth. DRL has received increasing attention in recent years, leading to some high-profile successes, particularly in board and video games  (Mnih et al., 2015; Silver et al., 2016 Silver et al., , 2017 Berner et al., 2019; Vinyals et al., 2019) . DRL combines Reinforcement Learning (RL), an approach based on learning through interaction with an environment, with the potential to result in highly autonomous systems  (Sutton & Barto, 2018) , with Deep Learning (DL), an increasingly popular method that enables AI systems to scale to more complex problems and environments  (Goodfellow et al., 2016) . Several researchers have suggested that DRL is likely to be a key component of more general-purpose and autonomous AI systems in the future  (Arel, 2012; Chen & Liu, 2018; Popova et al., 2018) . Though there are still considerable barriers to large-scale real-world deployment of DRL systems, we are beginning to see potential for application in areas including robotics  (Ibarz et al., 2021) , online personalisation and targeting  (Zhao et al., 2019) , finance  (Fischer, 2018) , autonomous driving  (Tai et al., 2016) , healthcare  (Esteva et al., 2019) , and data centre cooling  (Gasparik et al., 2018) . These applications and others will likely become more widespread as the technology improves. This could have substantial implications for people's lives: for example, by speeding up automation in various sectors, changing the nature and potential harms of online influence, or introducing new safety risks in physical infrastructure. We therefore suggest that now is an ideal time to begin thinking about how more widespread application of DRL-based technologies might impact society, what ethical questions might arise or be made more pressing as a result, how progress on different areas of technical research will affect these ethical challenges, and what forms of governance might be required to mitigate any risks. One challenge for exploring the potential impacts of future AI progress is that doing so will often require considerable background knowledge about current AI techniques and research directions. Such understanding is often beyond the core expertise of AI ethics and governance scholars, and there is a lack of accessible resources. We therefore begin this paper by providing a broadly accessible introduction to DRL methods with links to further, more in-depth resources for the interested reader.  1  Following this background, we outline in Section 3 the ways in which plausible future applications of DRL may introduce or exacerbate pressing issues for society, ethics and governance, including by: raising questions for current approaches to human oversight; introducing new challenges for safety and reliability; and changing incentives for data collection. We then discuss how progress on different technical challenges might shape the landscape of these societal concerns: how might breakthroughs in different areas magnify or mitigate societal impacts and risks, or change the direction of the field such that currently underresearched risks become more important? Understanding and monitoring progress on the technical research areas crucial to widespread application of DRL will be key to successful and timely management of future societal impacts, and will require interdisciplinary collaboration between experts in AI, social science, and policy. Based on this analysis, we highlight Artificial Intelligence includes any approach that attempts to make machines perform intelligently. Machine Learning involves getting machines to learn to solve problems from, usually large amounts of, data.
 Deep Reinforcement Learning Reinforcement Learning DL models data with many layered (deep) neural networks. An RL agent learns from a reward signal using trial and error. some important avenues for future work and conclude with recommendations for researchers in the AI ethics and governance community.
 Deep Learning
 Combines
 Deep Reinforcement Learning: a Brief Overview DRL is an approach to AI falling within the subfield of machine learning (ML), which combines two popular approaches in current ML research: reinforcement learning (RL) and deep learning (DL). 2 In general, the purpose of ML is to find solutions to problems that we cannot directly write programs for. This involves using (often large amounts of) data to optimise the parameters of a mathematical model, which will then encode a solution  (Bishop, 2006) . This is a powerful approach to solving many different types of problems. For example, while we could easily write a computer program to emulate a medical questionnaire (which is essentially a flowchart), we cannot similarly distill the knowledge of a doctor to diagnose a patient on the basis of a CT (computerised tomography) scan. However, given a large amount of data from CT scans and corresponding expert diagnoses, we can use ML to train a computer program to imitate doctors on this task  (Kononenko, 2001; Fatima et al., 2017) . However, many of the problems that we have to solve in the real world require going beyond making predictions based on labelled inputs: we may also want to interact with our environment through sets of actions, where the choice of action depends on what goals we have. Part of how humans learn is through this type of trial and error: when we take an 2. For an excellent and accessible overview of ML also aimed at those interested in its implications, we encourage reading "Machine Learning for Policy Makers"  (Buchanan & Miller, 2017) . For more in-depth, technical overviews of DRL and its components, DL and RL, we refer readers to the numerous excellent resources already published  (Goodfellow et al., 2016; Arulkumaran et al., 2017; Sutton & Barto, 2018; François-Lavet et al., 2018; Kaelbling et al., 1996) .
 Agent learns from previous actions, states, and rewards
 Environment
 Its state determines the observations and reward
 Actions chosen by agent's policy
 Observations and Rewards Figure  2 : An agent interacts with its environment through its actions, and learns in response to receiving new observations and rewards. action or make a choice, we get feedback, which gives us some indication of whether we should take similar actions in the future. For example, a young child quickly learns that when they say or do certain things, adults give them positive attention, leading them to repeat those same behaviours for reward more often. Reinforcement learning (RL), the core of DRL, is based on this approach to problem-solving and learning: an RL agent 3 learns through interaction with an environment, and, through trial-and-error, can alter its own behaviour in response to the feedback it receives. More precisely, the goal of RL is to learn a policy that recommends the best action to take at any given moment in order to maximise total reward over time. The reward received at any time point is given by a reward function, which specifies how good or bad the agent's actions are (but not necessarily exactly what they should be) given the current state of the world. RL takes place in a feedback loop, incorporating perception, action, and learning, as illustrated in Figure  2 . An RL agent first perceives the current state of its environment, performs an action in response, and then observes the next state as well as a reward.  4  This loop continues until a predetermined terminal condition is reached, or can run on indefinitely. A good policy should give the agent an appropriate action to take in every possible state of the environment, including ones the agent has never seen before. For example, when AlphaGo  (Silver et al., 2016)  beat former world Go champion Lee Sedol, it encountered board positions that had never occurred during its training period, and still played moves at least as good as any human expert. The first generation of RL approaches learned polices in the form of large tables, which indicated what action to take in any given state. However, many interesting modern applications of RL have far too many possible states or actions for this tabular approach to be practical or effective. This has led to decades of work on using more complex functions to approximate policies, culminating in recent years in the use of deep learning for learning more complex policies. Deep learning (DL) involves using many-layered models-most commonly artificial neural networks-to represent complex, non-linear functions. 5 A neural network accepts an input (usually a large matrix of numerical values), and, through the repeated application of mathematical operations in each layer, will produce an output (another matrix of numerical values). The trainable parameters of the network, which can number in the billions  (Brown et al., 2020) , can be updated by a learning algorithm so that, over time, the network ends up producing the desired outputs. Key to the success of DL  (LeCun et al., 2015)  is its ability to learn high-level features or representations from raw sensory data (e.g., images, text, or audio), which can be used to find patterns in high-dimensional data and generalise to previously unseen inputs. DRL involves training a neural network using RL, where the network learns which actions to take given its sensory inputs and feedback in the form of rewards.  6  The RL algorithm specifies how to update the parameters of the neural network so that it outputs actions leading to higher rewards. DL has enabled RL to scale to previously intractable problems by improving the performance of RL algorithms in two key ways: through representation learning, allowing them to learn directly in complicated environments, and function approximation, enabling policies with complex inputs and/or outputs  (Arulkumaran et al., 2017) . Because neural networks can approximate any function, they can be used to encode the policy of an agent. This allows RL to extend to problems with state or action spaces that are too large for the tabular approach-which includes most interesting modern applications. Deep networks can also learn features directly from raw sensory data, which means that they can, at least theoretically, be applied in many environments, including the real worldwith appropriate sensors. Without deep networks, the inputs to a RL algorithm generally have to be pre-processed and provided by a human expert. By contrast, DeepMind's DQN algorithm-the work that kickstarted the field of DRL-was able to achieve human-level performance on some Atari 2600 games from the game screen and the score, but no additional domain knowledge about any of the games  (Mnih et al., 2015) . An important extension of RL is multi-agent RL (MARL), in which multiple agents learn to compete or co-operate in an environment  (Busoniu et al., 2008) . While some agents may be deployed in isolation, there are many scenarios where groups of agents will need to interact with each other. Some parts of MARL research also look at mixed human-AI interaction  (Bard et al., 2020) , where understanding the intent of other agents is especially important. As a subfield of RL, MARL inherits all of its technical issues, but also introduces extra emergent complexity. While a full treatment of the intersection of DRL and MARL is 5. DL can involve other types of deep models  (Damianou & Lawrence, 2013) , but is effectively synonymous with the use of (artificial) neural networks. While artificial neural networks were historically inspired by biological neurons found in the brain, they are far more simplistic, and it is more informative to think of artificial and biological neural networks as independent, loosely similar, systems. 6. Technically speaking the network may output a 'value' for each state and then a further rule will determine the action performed. out-of-scope for this work, it is worth noting that agents can have different, even conflicting reward functions, and hence a single 'optimal' solution does not exist in the general case, even in theory. Another implicit assumption in most ML research is that ML models are trained on exactly the same distribution of data that they will be tested on. In the context of DRL, an example would be training a robotic arm to pick and place a certain type of object in a given place in a factory line, 'freezing' the learned model, and then deploying it in the exact same scenario. However, such a model would most likely be unable to adapt to changes in the type of object or placement along the line. To do so, one would also have to incorporate continual learning  (Parisi et al., 2019) , where the deployed model could continue to learn and adapt to changes in its environment. While trained DRL agents could be deployed without continual learning capabilities, they would be limited to strictly regulated, static environments. As this condition is extremely limiting, we assume that most impactful future applications of DRL will need to be capable of some form of continual learning. In general, current DRL methods work best on problems that have: (a) a well-defined environment, where it is possible to provide observations of the environment as input, specify what sorts of actions can be taken, and model time as passing in discrete steps; (b) a clear reward function, which is appropriate to the task, provides enough information to learn from, and cannot be easily 'gamed'  (Amodei et al., 2016; Krakovna, 2018; Lehman et al., 2020) ; (c) plenty of data to train the deep neural network-in the context of RL, this means it must be able to interact with and gain feedback from the environment easily and quickly; and (d) lots of computing power available, as exemplified by the high profile successes in DRL  (Silver et al., 2016 (Silver et al., , 2017 Berner et al., 2019; Vinyals et al., 2019) . The latter requirement is exacerbated by the fact that ML algorithms also have 'hyperparameters' 7 that need to be tuned separately, requiring the training of many models to find the settings that work best. In reality, it is rare that all of these ideal conditions are met, which is currently preventing the widespread deployment of DRL systems. However, successful applications of DRL are already apparent, even in these early stages of technological development. These initial applications are already raising issues for society, ethics, and governance which will only become more important as progress is made. We give an overview of these issues in the next section, in particular noting how they relate to current and emerging applications of DRL.
 Challenges DRL Raises for Society, Ethics, and Governance Areas where DRL has begun to see practical application include: to send more personallyrelevant notifications on Facebook  (Gauci et al., 2018) ; to control Google's data centre cooling system, improving energy efficiency  (Gasparik et al., 2018) ; to automate aspects of electronic trading, such as in J.P. Morgan's LOXM  (Bacoyannis et al., 2018) ; and in robotics and manufacturing  (Ackerman, 2020) . In the future, we may see more substantial use of DRL in a variety of areas from clinical decision support to resource management, and online education to autonomous driving  (François-Lavet et al., 2018) . These potential applications of DRL could have many benefits. More flexible robotic systems could improve manufacturing and delivery processes beyond their current limitations, more adaptive recommender systems could better take account of individuals' personal 7. Hyperparameters are set before learning begins; 'standard' parameters are adjusted during learning. preferences and longer-term consequences, and improving the efficiency of city infrastructure could help preserve vital and scarce resources. The promise of these benefits ensures that we will see more resources invested in realising these applications in the future. At the same time, the possibility of more autonomous systems that operate flexibly across a range of real-world domains introduces and magnifies a number of problems for AI ethics and governance, which we discuss in this section.
 Human Oversight The Ethics Guidelines for Trustworthy AI report published in 2019 by the European Commission's High Level Expert Group on AI (High-level expert group on Artificial Intelligence, 2019) states that in order to support human autonomy and decision-making, AI systems must allow for human oversight. Most applications of AI in society today support existing, identifiable decisions taken by individuals and groups, such as doctors, stockbrokers, and lawyers  (Parson et al., 2019) . This makes human oversight a relatively straightforward matter of ensuring that there is always a human 'in-the-loop', i.e., the ability for a human to oversee the overall activity of the system and intervene in every decision made. Applications of DRL, which aim to increase the autonomy of machines, may pose a challenge for this approach to human oversight. DRL could potentially be used in ways that go beyond supplementing or even replacing identifiable decisions, towards taking over control of much larger-scale processes, systems and networks, such as those involved in resource management in cities  (Mohammadi et al., 2017) . What adequate 'human oversight' should look like in these contexts, where a system may be making hundreds of small decisions in a very short period of time, is unclear. The data flow that a system is acting upon may be incomprehensible to humans, or simply too large and fast-moving for meaningful oversight to be maintained. For example, a DRL system designed to monitor and adjust energy usage in a building may be constantly making so many small decisions that it is impractical for a human to review and alter decisions even after the fact, and the system may be sufficiently complex and opaque that it is not possible to identify mistakes and intervene in real-time. It may be possible to impose constraints while the system is being designed-ensuring that the system turns off and a human is alerted if levels go above or beyond a certain threshold, for example. However, there remains a question of whether this constitutes sufficient oversight-or whether in this or similar situations we should avoid deploying DRL systems until ways to maintain more substantive oversight are found. Another challenge for oversight is introduced by the fact that deployed DRL systems may continue to learn and adapt their behaviour once embedded in their environment, possibly at a pace that is challenging for humans to keep track of and therefore meaningfully oversee. The question of how to maintain human oversight over systems that are continually learning is not one that has been addressed in existing ethics and governance proposals, and will become more pressing as continual learning systems become more advanced and practical. Similarly, if MARL becomes more prevalent, it would make understanding and intervening at the level of a single agent more difficult, posing additional challenges for oversight. In response to the direction of AI research, the European Commission's more recent whitepaper on AI states that "the appropriate type and degree of human oversight may vary from one case to another"  (European Commission, 2020) . Even though it may not be possible for a human to explicitly approve every decision, it may be possible to retain meaningful oversight by ensuring a human is able to review decisions after they have been made, by monitoring an AI system in operation and intervening in real-time if necessary, or by imposing operational constraints in the design phase. Further work on flexible approaches to human oversight should take into account possible evolutions in the autonomy and behaviour of DRL systems discussed here. As systems become more autonomous and promise more high-impact applications, there will inevitably be economic pressures to trust systems to make more decisions and take over more processes, rather than waiting for, or relying on, slow human input. The benefits of doing so need to be traded off against the potential costs to human autonomy that come from reduced oversight, and we must begin thinking about how to navigate this tradeoff now before pressures to implement more autonomous systems grow.
 Safety and Reliability DRL agents learn by exploring their environment to discover the actions that lead to the highest reward over time. This trial-and-error approach to learning means that in order to learn how to avoid making a mistake, a DRL agent would have to first make that mistake  (Berger-Tal et al., 2014) . In many real-world contexts this is unacceptable: we cannot have self-driving cars running over pedestrians, or an energy control system accidentally switching off electricity in a hospital, before learning not to do these things. This can cause problems even when DRL is not being deployed in a physical environment: Microsoft's chatbot Tay 8 reproduced thousands of offensive tweets before the problem was noticed and it was taken down. Deploying DRL systems in the real world therefore requires new ways of ensuring the safety and reliability of systems, and in particular requires approaches to safe exploration  (Pecka & Svoboda, 2014; Garcıa & Fernández, 2015) : how can agents explore enough during training to learn robust behaviour, while avoiding the kinds of exploration that could cause harm? Ensuring the safety and reliability of DRL systems becomes even more pressing if they are deployed with increasing autonomy in high-stakes domains and systems. For example, it has been suggested that DRL may be able to make effective use of the data generated by 'smart cities' to improve resource management, such as by optimising power usage, increasing the efficiency of traffic signal control systems, and increasing crop productivity by monitoring and correcting parameters in agriculture systems  (Mohammadi et al., 2017; Mocanu et al., 2018; Genders & Razavi, 2016) . Systems such as energy, transportation, and agriculture are high-stakes in the sense that failures could result in loss of life, and incredibly complex, meaning that their dynamics cannot be fully understood by any single person or group. It is reasonable to suppose that DRL systems, with their adaptability and ability to process huge amounts of data, may ultimately be better equipped to manage these important processes than humans, and improving efficiency could help save critical resources across the world. However, the importance of these domains also means that the risks involved in deploying 8. Although the type of algorithm behind Tay is unknown, this is still a prime example of an agent that interacts with an environment and learns from feedback. DRL systems are huge, massively raising the stakes of mistakes and unintended consequences compared to smaller-scale and more constrained applications. Continual monitoring of DRL systems will be essential to ensuring their safety in highstakes domains, especially if those systems are trained in the real world and continue to learn from a constant stream of data after deployment. Even for systems that are 'frozen' before deployment (i.e., no longer continuing to learn), as initial deployments of DRL likely will be, it is widely accepted that we currently lack adequate methods for verifying safety and reliability  (Tarraf et al., 2019; Allen, 2020; Fiander & Blackwood, 2016) . For systems that learn over their lifetime, continual monitoring to ensure they do not learn behaviours outside of their intended use is even more challenging (Defense Innovation Board, 2019). Substantial progress on the testing, evaluation, verification and validation of AI systems in general, and DRL systems in particular, will be needed before continually-learning DRL systems can be safely deployed in high-stakes domains. One research area likely to contribute to this progress is explainable AI  (Gunning, 2017; Arrieta et al., 2020) . Explainable AI includes the development of models that are inherently easier for humans to understand, but also methods for post-hoc interpretation of standard models such as neural networks. While interpretability methods from the broader field of DL, such as saliency maps  (Simonyan et al., 2013)  or activation maximisation  (Erhan et al., 2009) , have been applied to study DRL agents  (Greydanus et al., 2018; Dai et al., 2019) , the intersection of interpretability and DRL is relatively underexplored. As research in this area matures, policymakers will need to make decisions around acceptable tradeoffs between the fidelity and interpretability of explanations given by AI systems  (Ribeiro et al., 2016) .
 Harms from Reward Function Design Beyond more immediate harms, it will also be important to consider the potential longerterm side effects of DRL systems optimising for a specific reward function, which may cause harms which are less obvious but potentially even greater in scale. The flexibility of reward functions-which is what makes DRL systems in theory extremely powerful-introduces much greater potential for unintended consequences, and we might not notice that a system is optimising for slightly the wrong objective before it is too late  (Clark & Amodei, 2016; Krakovna, 2018; Lehman et al., 2020) . The challenge of reward function design is a longstudied topic in RL and there are many examples where reward functions cause surprising behaviour  (Ng et al., 1999; Randløv & Alstrøm, 1998) . There are two related problems for reward function design that can result in unintended harms. First, many real-world tasks do not have easily-defined objectives, and it can be difficult to predict in advance what behaviour a system will exhibit as a result of optimising for a given reward. Second, even if reward functions do correctly capture what the designers or immediate users of a system intended, they may have broader consequences that are harmful to others, perhaps unintentionally. There are many documented cases of this first problem where DRL algorithms end up learning the 'easiest' way to achieve a stated objective  (Clark & Amodei, 2016; Krakovna, 2018) . For example, an RL agent trained to play a boat race game was rewarded for getting as many points as possible, under the assumption that this would be equivalent to winning the race  (Clark & Amodei, 2016) . However, due to the game layout, the agent was able to find a way to knock over targets by driving in circles, increasing its score but never actually finishing the game. This example is perhaps more amusing than concerning, but such misspecifications could be much more problematic in real-world settings where unintended actions could lead to economic or physical harm.  Everitt et al. (2019)  give the example of a stock prediction system that predicts the bankruptcy of a company, causing stakeholders to leave. Although it is undesired, creating 'self-fulfilling prophecies' is an effective way for the agent to maximise its predictive accuracy. To illustrate the second problem, consider the case of social media content-selection algorithms, designed to maximise the likelihood that a user clicks on a given item.  Russell (2019)  suggests that rather than helping users to find content they are more interested in, ubiquitous use of these algorithms has instead ended up changing users' preferences so that they are more predictable. Worse, this process gradually shifts user preferences towards more extreme content, and therefore contributes to greater conflict online. Russell even goes so far as to suggest the effect of these algorithms has contributed substantially to the resurgence of facism and the crumbling of international bodies like the European Union and NATO. In large part, the problem here arises as a result of companies optimising for a specific objective at the expense of the interests of individuals and wider society. From the perspective of a social media company, increased engagement can come from people finding more content they genuinely enjoy, or from people being shown more provocative and divisive content. As more companies develop and deploy DRL systems with wide-ranging impacts on users, we must consider both how to ensure that these systems behave as intended over the long-term, and whose interests they are serving. Potential conflicts between commercial and individual needs are particularly concerning here: a reward function by itself may not be sufficient to simultaneously maximise a company's long-term financial return while ensuring there is no negative impact on user agency or welfare, yet it is the former that will likely drive most commercial deployments of AI. Even with the best of intentions, reward functions are likely to be biased towards those designing or profiting from them. As AI systems come to be deployed more globally, what was once a localised problem becomes a problem of global value alignment: how do we ensure that the goals pursued, both directly and indirectly, by increasingly wide-ranging DRL systems, consider the values of all those impacted by them? Current AI ethics guidelines are predominantly issued by the US and EU, suggesting we have a way to go in ensuring a diversity of perspectives are accounted for  (Jobin et al., 2019) . To ensure the longer-term impacts of DRL systems are broadly beneficial, therefore, we need a broader sense of what ethical and responsible reward function design should look like, and how it could be enforced, especially where DRL is being deployed in domains with real implications for people's lives by companies whose goals may not be aligned with society as a whole.
 Incentives for Data Collection The type and amount of data required to train DRL systems creates incentives for data collection which raise particular ethical concerns. DRL methods tend to require very large amounts of data as well as ongoing data collection if the DRL system is continuing to learn and adapt its behaviour once deployed. This means that the collection of up-to-date and expansive data from across society could become a key factor for developing and deploying DRL systems. We might therefore be particularly concerned about progress in DRL creating greater incentives for more widespread surveillance. We suggest that particular attention should be paid to advances in continual learning combined with DRL, which would require more real-time data collection, and to advances in methods to learn from large-scale data sources for real-world deployment, such as we are beginning to see in work using DRL to navigate cities using Google street view  (Mirowski et al., 2018) . Some areas where DRL might be applied raise greater concerns about data collection and surveillance than others. In order to develop DRL systems for managing infrastructure, for example, it may be particularly important to generate and collect greater amounts of data through the installation of sensors and actuators across cities. This raises a challenging tradeoff between the potentially huge benefits of better resource management against the real threat to people's privacy and autonomy of increasingly 'smart' cities. Another potentially concerning application is the use of DRL to learn more sophisticated and granular models of different users of a digital product or service, and to adapt these models over time based on user behaviour  (Zheng et al., 2018) . While this could improve people's online experiences, it also might increase the incentive for companies to try to collect or infer increasingly sensitive information about online users, and raises the important ethical question about what data it is acceptable to use to build such models. There is ongoing technical work in areas such as federated learning and differential privacy attempting to address some of these issues  (Ji et al., 2014; Li et al., 2020) , and the European Union's General Data Protection Regulation (GDPR) represents an important legislative step towards protecting people's privacy. However, both technical and legislative approaches to data privacy will need to account for how incentives for data collection, and the resulting tradeoffs, may change as DRL systems become more widely deployed. Increased data collection may also have a disproportionate negative impact on groups who are already vulnerable or discriminated against. Even if surveillance systems are intended for specific, beneficial purposes such as better resource management, the collection of increased data from smart cities consolidates power over individuals' lives in the hands of technology companies and governments, who may well end up using that data for other purposes such as policing and tracking immigrants. Such uses of surveillance have a welldocumented history of being used, deliberately or otherwise, to entrench systemic discrimination against minority groups  (Crawford et al., 2019) . Similarly, increased modelling of users based on online behaviour could easily encode and entrench biased assumptions and stereotypes: Facebook's targeted advertising has been widely criticised for relying on gender and racial stereotypes  (Hao, 2019) . To the extent that applications of DRL incentivise greater surveillance and enable increasingly sophisticated online targeting, they are likely to exacerbate these problems.
 Security and Potential for Misuse Without advances in the robustness and security of DRL systems, they may be open to attacks from adversaries attempting to manipulate their performance and behaviour. For example, even a demonstrably safe DRL-based robot could be forced into dangerous collision scenarios by an adversary perturbing its sensory input or corrupting its reward function.  Behzadan and Munir (2018)  argue that there are unique challenges posed by ensuring the security of DRL systems. Compared to other ML approaches, it is much less straightforward to distinguish adversarial attacks from benign actions in DRL, since the exploration mechanism means that the training data distribution is continually changing, and delayed rewards may make it difficult to straightforwardly assess the performance of a system. Especially if DRL systems begin to be integrated into critical systems such as healthcare, energy, and transportation, it is essential that key security issues in DRL are well-understood and sufficiently addressed. There is also a risk that DRL-based systems may be deliberately misused in ways that cause harm. For example, due to their ability to more effectively tailor online content, DRLbased methods could easily be misused to improve the efficacy of online manipulation and disinformation by learning which messages are maximally persuasive to different individuals. In a recent review of online targeting systems, the UK Government's Centre for Data Ethics and Innovation recommend that there should be adequate transparency around online targeting (Centre for Data Ethics and Innovation, 2020). Given its potential use to tailor content more narrowly, it may be particularly important for policymakers and regulators to find ways to mandate transparency around where DRL in particular is used in recommender systems. Improved DRL-based models of human behaviour in particular could be misused to strengthen attempts at social manipulation. Advances in data efficiency and simulation quality would make it easier for those without access to large amounts of computing power and data to make use of DRL  (Tucker et al., 2020) . This could make it easier for small groups to misuse DRL capabilities for malicious purposes. For example,  Brundage, Avin et al. (2018)  describe how cyber attackers might use DRL to "craft attacks that current technical systems and IT professionals are ill-prepared for", emphasising how the feedback loop in RL enables comparison between different approaches to find the one that is most effective at evading security tools. However, DRL could also be used to improve defensive cybersecurity capacities. Whether the advantage lies on the side of the attackers or the defenders is unclear, and would benefit from further analysis, perhaps building on existing work on how technological progress may affect the offense-defense balance  (Garfinkel & Dafoe, 2019) .
 Automation and the Future of Work A clear implication of more adaptable and reliable DRL-based systems, especially in robotics and manufacturing, is in changing the susceptibility of different jobs to automation. Several analyses have suggested that many low-wage or manual jobs are unlikely to be automated in the near future because they require levels of dexterity, mobility and flexibility for which humans still vastly outperform machines  (Ford, 2015; Frey et al., 2016; Autor & Dorn, 2013) . While this advantage still clearly remains at present, current avenues of progress in robotics led by DRL methods suggest this gap could quickly diminish, a factor that is not considered explicitly by any of the analyses of automation the authors are aware of.  Frey and Osborne (2016) , for example, discuss areas where AI systems are far from human performance, including in perception and manipulation, and how this limits possibilities for automation at present, but do not consider the implications of potential progress in these areas.  Kai-Fu Lee (2018)  argues that manual jobs with the highest risk of replacement are those which are performed in structured environments, with little need for dexterity or social interaction. Prime examples include assembly line inspectors, fruit harvesters and dish washers. However, Lee does not specifically consider how advances in DRL could impact the ability of these jobs to be automated. Rather than leading to full automation of jobs, advances in DRL may instead ease or improve work in certain domains, by automating tedious or dangerous aspects of manual work. This could have a positive impact on existing employees if it makes their work more enjoyable, but this has to be traded off against potential job losses from improved efficiency. A more detailed analysis of how advances in DRL specifically might speed up automation or change the nature of work in different sectors, for example by improving robotic manipulation, would be valuable.
 Avenues of Progress in DRL and their Implications We have argued that widespread deployment of DRL systems in society could introduce new challenges for the ethics and governance of AI, such as by requiring new forms of oversight and new processes for responsible system design, as well as exacerbating the importance of existing challenges, such as ensuring the safety, reliability and security of systems deployed in safety-critical domains. However, current DRL methods are not yet seeing widespread deployment, due to several ongoing technical challenges in DRL research. This gives us time to anticipate and prepare for the societal impacts of DRL. An important part of doing so will involve tracking progress on different technical challenges currently faced by DRL methods, which will enable different kinds of applications. In this section, we therefore introduce some of the key technical challenges for the DRL research community, and discuss the possible implications of progress on these challenges for the ethical, societal, and governance challenges discussed in the previous section.
 Learning in the Real World DRL systems learn through interacting with an environment. In a simulation or a game the environment can be strictly controlled, but in real-world environments there is more complexity and noise, interaction is often slow and expensive, feedback can be delayed, and performing the wrong action can be dangerous  (Dulac-Arnold et al., 2019) . For example, in recommender systems the delay between recommending an item to a user and seeing them interact with it may be days or even weeks  (Mann et al., 2018) , and for robotic agents, a simple fall or crash could damage expensive equipment or even hurt someone. Because of the challenges and risks involved in training DRL in real-world environments, often it is desirable to instead use simulated environments for training. However, good simulations require enormous amounts of data and often still do not result in policies that transfer well to the real world  (Pan et al., 2017) . There are two key interrelated approaches to improving DRL systems so that they can be deployed in real-world environments. One approach, which has recently shown impressive results, is finding ways to transfer learned policies from simulation to the real world; these have recently been popularised as 'sim2real' methods. A popular method within these is 'domain randomisation'-randomising various aspects of the simulation (e.g., lighting, background clutter, friction coefficients, etc.), so that DRL agents learn policies that are robust to variations in these in the real world  (Tobin et al., 2017; James et al., 2017; Peng et al., 2018; OpenAI et al., 2019) . Other methods include system identification  (Chebotar et al., 2019)  and domain adaptation  (Bousmalis et al., 2018; Tzeng et al., 2020) . Should rapid progress be made in sim2real methods, this would mean that DRL agents could be trained primarily in private simulations, only to be released at the end of successful training. This means that there may be fewer public warning signs before a new system is introduced to the real world, potentially it more difficult to anticipate potential impacts of new applications in advance, as well as making it easier for companies to develop and deploy systems without oversight and accountability. The resulting systems may be relatively capable, but over-reliance on simulation could also result in the deployment of agents that perform well in simulation but fail catastrophically in the real world. As sim2real methods advance it will be important to consider how risk assessment and oversight of highrisk applications can be ensured. A second approach to improving the ability of DRL systems to learn in real-world environments is to improve data efficiency, i.e. the ability of DRL systems to learn from much smaller amounts of data. One approach to improving data efficiency in RL is to use modelbased RL (MBRL). In MBRL, the agent learns a model of the environment, which can allow the agent to explicitly or implicitly plan what to do without necessitating further interaction with the environment  (Sutton, 1990; Hamrick, 2019) . In this case, the agent effectively creates its own simulation of the world to learn in. Improvements in model-based RL could also aid with building more realistic simulators from available real-world data. However, substantial progress will be required before it is possible for agents to learn models of most complex real-world environments. The ability to train DRL systems entirely in simulation or from much smaller amounts of data would reduce the need for 'safe exploration' approaches discussed previously. Mistakes made in simulation would have little or no real cost, and the ability to learn from fewer interactions would significantly reduce (if not entirely mitigate) the risk of serious error. If we see advances in simulated environments then it may also be that the need for extensive real-world data collection is reduced for the purposes of initial training. However, even in such a case it would still be desirable to develop systems capable of continuing to learn once deployed, meaning constant streams of real-world data are still likely to be highly valued. Improvements in data efficiency would therefore also be valuable for reducing incentives for large-scale data collection.
 Safe and Reliable Reward Functions The behaviour of a DRL agent is highly dependent on its reward function, but as discussed in Section 3, it can be very difficult to specify what we actually want for many real-world tasks. This can make DRL systems unreliable in ways that compromise their safety. Several different technical approaches are attempting to solve the problem of ensuring the safety and reliability of reward functions. Some approaches focus on ways to train systems without interaction with the real world so that any unintended consequences can be spotted in advance: by training systems 'offline' on batches of historical data  (Wiering & Van Otterlo, 2012; Levine et al., 2020)  9 , or in simulated environments as discussed above. Other approaches focus on allowing DRL agents to explore safely by explicitly restricting them from visiting certain states or taking certain actions known or expected to be unsafe, or finding practically viable ways for humans to oversee exploration  (Hans et al., 2008; Garcıa & Fernández, 2015; Saunders et al., 2018) . These approaches to ensuring the safety of DRL systems will need to be combined with work on the broader problem of value alignment: designing AI systems whose behaviour is in line with human values. One popular approach to solving this problem is inverse reinforcement learning (IRL)  (Ng et al., 2000; Abbeel & Ng, 2004) , where the agent's goal is to infer a reward function from demonstrations, rather than being given one explicitly. This increases the likelihood of the agent learning to behave as intended (especially since it is often easier for us to demonstrate intended behaviour through examples than to provide rules for it). In some approaches, such as Bayesian IRL  (Ramachandran & Amir, 2007) , the agent also learns a probability distribution over reward functions, meaning learned policies take account of uncertainty, and are likely to be more robust across a wider range of possibilities. Many real-world tasks also have multi-dimensional objectives, where different subgoals need to be balanced against each other. For example, recommender systems might aim to produce recommendations that achieve some balance of relevance and novelty to the user, among other goals  (Aggarwal, 2016) . Even for an expert in a domain, it may be difficult to explicitly state what all those different objectives should be or how they should be balanced. An explicit way of dealing with this is multiobjective RL (MORL), where the aim is to either learn a single policy that can adapt to different preferences, or to learn several policies that optimally balance a set of preferences  (Liu et al., 2014) . Such an approach allows greater flexibility after training, and could be an option for improving interpretability and oversight. Even when rewards are well-defined, in most settings there is a delay between taking an action and receiving a reward. An agent may have performed thousands of actions immediately prior to receiving a reward and it may be impossible to infer which of those were the key steps that should be repeated in the future. This is known as the credit assignment problem, and is particularly challenging in real-world environments where rewards are sparse  (Sutton & Barto, 2018) . Current approaches to solving the credit assignment problem try to re-allocate  (Arjona-Medina et al., 2019)  or augment  (Hung et al., 2019)  the credit given to past events, but this is still a largely unsolved problem. If substantial progress is made in any of these methods for safe exploration, value alignment, and/or credit assignment, it will suddenly become possible to deploy DRL systems more safely and effectively in a wider range of real-world environments. These are therefore avenues of progress worth monitoring since they could lead to a very quick proliferation of applications. It should be emphasised that substantial progress in these areas will not be an easy task and is not expected in the near future. Nevertheless, progress is possible, and it is important to ensure that those working on the risks and governance of AI systems are ready for it. In order to conduct risk assessment for the use of DRL in a robotics system or autonomous vehicle and decide what kind of oversight is needed, for example, one will need to understand what guarantees can be made that the system will not perform certain types 9. In healthcare, offline evaluation is also of particular importance  (Gottesman et al., 2018) . of action, and how strict those guarantees are. In order to think through the scope of possible long-term impacts of a new DRL-based information filtering system, one will need to understand how the system balances multiple objectives and accounts for delayed rewards, and how effective those methods will be. As progress is made in all of these areas, therefore, greater communication will be needed between researchers and developers, and those deploying, assessing, and governing AI systems, around what assurances can be made about the behaviour of those systems and the limits of those assurances. Conversely, if we begin to see increased real-world deployment of DRL systems without seeing considerable progress in these different areas of reward function design, we might be concerned that such systems are being deployed without adequate guarantees that they will behave safely or have the consequences they are expected or intended to.
 Generalisation and Robustness Ensuring that DRL methods are sufficiently generalisable and robust-that is, they behave as intended across a wide range of scenarios-is a final key challenge for safe real-world deployment. In part, this is a problem inherited from the field of ML more broadly. Most ML algorithms struggle to reuse past knowledge and generalise to new situations  (Geirhos et al., 2018) , and especially those using deep learning can be frustratingly unstable  (Hutson, 2018) . DRL has its own approaches to improving generalisation  (Justesen et al., 2018; Cobbe et al., 2019; Witty et al., 2018; Zhang et al., 2018b Zhang et al., , 2018a , which have strong overlap with broader approaches for generalisation in ML. These include methods in transfer learning: developing algorithms that can effectively transfer knowledge from one domain to another  (Parisotto et al., 2015; Rusu et al., 2015 Rusu et al., , 2016 , and meta-learning, where more generalisable learning methods themselves are learned from data  (Schmidhuber, 1987; Wang et al., 2016; Duan et al., 2016; Finn et al., 2017) . A related problem for DRL is that results have proven particularly difficult to reproduce.  Henderson et al. (2018)  found that results can be significantly impacted by simple changes to the neural network architecture, or by simply multiplying the reward function by a constant value, among other factors. Improving the reproducibility of results may be more a matter of changing research and publication norms, rather than being a research challenge for new methods to overcome. While the ML community in general is particularly in favour of open source material and sharing code, issues of reproducibility still persist and not all research makes its way into the public domain. Reproducibility problems are exacerbated for real-world applications. Not only do network parameters need to be copied, but also the exact environment in which the agent learns may need to be reproduced faithfully. Until methods become much more robust, seemingly innocent changes in the real-world training environment may have large consequences for the final behaviour of an agent. Further, the expense of replicating real-world settings could be much larger than that of copying and running a simulated environment, further limiting public access to agents for testing and safety analysis. Without making considerable progress in the areas of generalisation and robustness, it will be difficult to ensure the reliability of DRL systems in real-world domains, which are often highly dynamic and variable. As with reward function design, if we start to see increased use of DRL systems in complex real-world domains without good evidence of substantial progress in areas such as transfer learning and meta-learning, this should raise concerns about whether such systems might be more brittle than they seem. On the other hand, if we see big breakthroughs in these areas, we may soon see DRL agents capable of performing a wide range of common everyday tasks. This could increase research incentives to solve other related tasks, such as robust real-world navigation and common sense reasoning, which, along with more generalisable DRL methods, could be used to build much more general-purpose systems such as personal assistants and cleaning robots.
 Discussion Progress in different areas of technical research will impact the challenges DRL raises for safety, ethics and governance. In general, any progress in research that enables DRL methods to be applied more widely in society will exacerbate these challenges, making them more urgent. For example, substantial progress in the data efficiency of DRL methods and/or the quality of simulated environments for training would likely enable applications in more complex real-world environments, where data is currently either sparse or expensive to acquire. Progress in simulated environments might make DRL for autonomous driving much more feasible, making work on safety and reliability for self-driving cars more pressing  (Lin, 2016; Stilgoe, 2018) . As a first step, we therefore suggest it would be valuable for those concerned with the wider societal impacts of AI to pay attention to progress in DRL, particularly to anticipate and prepare for the impact of DRL in key domains. The relationship between technical progress and ethics and governance issues is a little more nuanced than progress always making issues more pressing. In some cases, ethical and societal concerns are being directly addressed by avenues of technical research in DRL, and so progress in these areas could reduce these concerns. While improvements in data efficiency might make certain aspects of DRL safety and reliability more concerning, those same improvements would likely reduce incentives for widespread surveillance, since DRL systems would become less data-hungry. Advances in value alignment-designing reward functions in line with complex, multi-dimensional human values-would reduce concerns about misspecified reward functions having harmful unintended consequences, but might increase the importance of developing standards for reward function design that ensure those methods are used in practice. Improvements in the generalisability of DRL methods could lead to increased confidence that systems will perform reliably across a range of domains, but the resultant increase in applications of DRL could make other issues, such as ensuring that systems can learn safely, more urgent. It may therefore be particularly important to pay attention to signs of uneven progress across key technical areas in DRL, which might serve as a warning that we could see increased application of DRL without key challenges related to safety, alignment, and security being resolved. For example, if DRL systems begin to be integrated into critical functions in society such as energy or water management, without sufficient assurances about their security, then we might reasonably be concerned about those systems being vulnerable to attack. We need to be able to notice these concerns and work with experts in relevant domains about how to address them before mistakes or accidents happen. Beyond paying attention to progress in DRL methods and considering their implications, it will also be important for the AI ethics and governance community to explore specific domains where DRL may be applied in the near future. Throughout our analysis in this paper, we referred to a number of domains where DRL could be applied in the near future, including recommender systems, city infrastructure and manufacturing. Future research could explore the potential impacts of DRL in a variety of important domains not discussed in this paper, such as finance, healthcare, or autonomous vehicles, and attempt to more systematically prioritise which domains of application are likely to raise the most important issues for AI ethics and governance. While various avenues of technical research are addressing many of the concerns discussed in this paper, one of our key messages is that this will be insufficient by itself. In order to ensure developments in DRL are broadly beneficial, there are many issues and questions that the AI ethics and governance community must also consider. We suggest the two following broad areas will be particularly important as DRL methods move closer to widespread realworld deployment: 1. What kinds of oversight, risk assessment, and technical assurances should we require of DRL systems before deploying them in high-stakes domains, to ensure that they are safe, secure, reliable, and respect human autonomy? How can and should these requirements be implemented practically? How fit-for-purpose are current safety engineering approaches, and what can we learn from the safety engineering community about DRL safety and reliability? 2. How can we ensure that, where DRL systems are deployed with real consequences for individuals' lives, reward functions are designed in ways that take into account the values of all those impacted, and in ways that consider any potential long-term unintended consequences of optimising for a specific objective? What does responsible reward function design look like, and how can it be implemented in practice? Ideally questions like these would be explored by multidisciplinary teams, combining DRL experts with ethics and governance experts.
 Summary and Conclusion DRL is receiving increasing attention in the AI research community, and continued progress could lead to more widespread deployment with profound impacts across many sectors of society. This raises several broad concerns for AI safety, ethics and governance: • Advances in DRL promise more autonomous systems that can operate effectively with less human oversight, introducing a challenging tradeoff between the potential benefits of such systems and the potential threat to human autonomy and safety; • DRL systems learn through trial-and-error interaction with an environment, which introduces new challenges for ensuring their safety: we not only need to ensure that systems are safe when deployed, but that they can learn safely; • More broadly, advances in DRL could enable AI systems to be applied in an increasing number of safety-critical domains, increasing the potential harms of mistakes and therefore increasing the importance of being able to make robust assurances about how systems will behave, and about their security to outside attack; • The behaviour of a DRL system depends heavily on its reward function, and optimising for slightly the wrong objective can lead to harmful unintended consequences that are difficult to predict in advance-especially if those designing reward functions do not account for the values of all those affected by a system; • DRL systems currently require large amounts of data to learn effectively, meaning that incentives to deploy real-world DRL applications will likely increase incentives for surveillance and ongoing data collection; and • Progress in DRL is likely to speed up the susceptibility of jobs to automation in certain areas, such as robotics and manufacturing, which needs to be accounted for in analyses of how AI will impact the future of work. In order to address these concerns and future challenges, we recommend that the AI ethics and governance community: • Find ways to track progress in DRL and its applications. It is important that those concerned with the wider societal, ethical, and governance implications of AI find ways to track progress in DRL, paying particular attention to progress towards widespread practical deployment. This would help the community better anticipate and prepare for future societal impacts of DRL, and if progress in key areas is uneven, could identify 'warning signs' that DRL risks being deployed in unsafe or harmful ways. This could potentially be an extension of excellent existing initiatives which aim to measure progress in AI more broadly  (Perrault et al., 2019; Haynes & Gbedemah, 2019) . • Consider the implications of DRL progress for existing AI governance initiatives, including standards and regulation. We have suggested that applications of DRL may introduce new challenges for human oversight and making safety assurances about AI systems, and/or increase the importance of these issues by allowing AI to be applied in more high-stakes domains. A valuable next step would be to consider the practical implications of this for specific governance proposals, which tend to focus on very broad notions of 'AI'. For example, how can guidelines around autonomy in existing AI ethics principles be adapted to apply to DRL systems, which are potentially much more autonomous than any AI systems deployed today? How do notions of safety and security as emphasised in AI ethics proposals need to be adapted to recognise the particular challenges raised by DRL? • Establish notions of responsible DRL development. It could be extremely worthwhile for a multidisciplinary team, combining DRL experts with ethics and governance experts, to explore in more depth many of the questions raised here about how to develop and deploy DRL systems responsibly, especially in high-stakes contexts. What standards of reliability and generalisability should DRL systems be required to meet before deployment, especially where systems are being trained largely in simulation? What does responsible reward function design look like, especially in domains with consequences for individuals' lives, and how can norms of responsibility be embedded in DRL research practices? DRL is likely to play an important role in the future of AI, promising increasingly autonomous and flexible systems with the potential for application in increasingly highstakes domains. As a result, DRL introduces and exacerbates a number of concerns which will shape the debate around the safe and responsible use of AI and may be overlooked in more general treatments of AI impacts. Since the most pressing challenges for society will depend on the exact nature of progress of AI research, it will be important to build and maintain strong collaboration between groups working on AI progress and AI governance in order to prepare for the challenges of future AI systems. Figure 1 : 1 Figure 1: The relationship between deep reinforcement learning and other areas of AI research.
			 . RL systems are often referred to as 'agents', because they act autonomously in their environment. This does not mean that an RL agent is responsible or aware of their actions, just that the human designer is one step removed from the action selection process. 4. In standard RL formulations, the agent gets a reward at every timestep, but that reward may sometimes (or even often) be 0.

Abstract: Deep Reinforcement Learning (DRL) is an avenue of research in Artificial Intelligence (AI) that has received increasing attention within the research community in recent years, and is beginning to show potential for real-world application. DRL is one of the most promising routes towards developing more autonomous AI systems that interact with and take actions in complex real-world environments, and can more flexibly solve a range of problems for which we may not be able to precisely specify a correct 'answer'. This could have substantial implications for people's lives: for example by speeding up automation in various sectors, changing the nature and potential harms of online influence, or introducing new safety risks in physical infrastructure. In this paper, we review recent progress in DRL, discuss how this may introduce novel and pressing issues for society, ethics, and governance, and highlight important avenues for future research to better understand DRL's societal implications. 1. Readers already familiar with DRL could skip straight to Section 3.


Dj Dvijotham
Robert Stanforth
Sven Gowal
Timothy Mann
Pushmeet Kohli
Deepmind London
UK 4ag

A Dual Approach to Scalable Verification of Deep Networks

INTRODUCTION Neural networks and deep learning have revolutionized machine learning achieving state of the art performance on a wide range of complex prediction tasks  [Krizhevsky et al., 2012 . However, in recent years, researchers have observed that even state of the art networks can be easily fooled into changing their predictions by making small but carefully chosen modifications to the input data (known as adversarial perturbations)  [Szegedy et al., 2013 , Kurakin et al., 2016 , Carlini and Wagner, 2017a , Goodfellow et al., 2014 , Carlini and Wagner, 2017b . While modifications to neural network training algorithms have been proposed to mitigate this phenomenonMadry et al.  [2018] , a comprehensive solution that is fully robust to adversarial attacks remains elusive  Wagner, 2017b, Uesato et al., 2018] . Neural networks are typically tested using the standard machine learning paradigm: If the performance (accuracy) of the network is sufficiently high on a holdout (test) set that the network did not have access to while training, the network is deemed acceptable. This is justified by statistical arguments based on an i.i.d. assumption on the data generating mechanism, that is each input output pair is generated independently from the same (unknown) data distribution. However, this evaluation protocol is not sufficient in domains with critical safety constraints  [Marston and Baca, 2015] . In these cases, we may require a stronger test: for example, we may require that the network is robust against adversarial perturbations within certain bounds. Adversarial evaluation. In the context of adversarial examples, a natural idea is to test neural networks by checking if it is possible to generate an adversarial attack to change the label predicted by the neural network  [Kurakin et al., 2016]  and train them to be robust to these examples  Madry et al. [2018] . Generating adversarial examples is a challenging computational task itself, and the attack generated by a specific attack algorithm may be far from optimal. This may lead one to falsely conclude that a given model is robust to attacks even though a stronger adversary may have broken the robustness. Recent work  [Athalye et al., 2018 , Uesato et al., 2018  has shown that evaluating models against weak adversaries can lead to incorrect conclusions regarding the robustness of the model. Thus, there is a need to go beyond evaluation us-ing specific adversarial attacks and find approaches that provide provable guarantees against attacks by any adversary. Towards verifiable models. Verification of neural networks has seen significant research interest in recent years. In the formal verification community, Satisfiability Modulo Theory (SMT) solvers have been adapted for verification of neural networks  [Ehlers, 2017 , Huang et al., 2017 , Katz et al., 2017 . While SMT solvers have been successfully applied to several domains, applying them to large neural networks remains a challenge due to the scale of the resulting SMT problem instances. Furthermore, these approaches have been largely limited to networks with piecewise linear activation functions since most SMT solvers are unable to deal efficiently with nonlinear arithmetic. More recently, researchers have proposed a set of approaches that make use of branch and bound algorithms either directly or via mixed-integer programming solvers  [Bunel et al., 2017 , Cheng et al., 2017 , Tjeng and Tedrake, 2017 . While these approaches achieve strong results on smaller networks, scaling them to large networks remains an open challenge. These approaches also rely heavily on the piecewise linear structure of networks where the only nonlinearities are maxpooling and ReLUs. Towards scalable verification of general models. In this paper, we develop a novel approach to neural network verification based on optimization and duality. The approach consists of formulating the verification problem as an optimization problem that tries to find the largest violation of the property being verified. If the largest violation is smaller than zero, we can conclude that the property being verified is true. By using ideas from duality in optimization, we can obtain bounds on the optimal value of this problem in a computationally tractable manner. Note that this approach is sound but incomplete, in that there may be cases where the property of interest is true, but the bound computed by our algorithm is not tight enough to prove the property. This strategy has been used in prior work as well  [Kolter and Wong, 2018, Raghunathan et al., 2018] . However, our results improve upon prior work in the following ways: 1. Our verification approach applies to arbitrary feedforward neural networks with any architecture and any activation function and our framework recovers previous results  [Ehlers, 2017]  when applied to the special case of piecewise linear activation functions. 2. We can handle verification of systems with discrete inputs and combinatorial constraints on the input space, including cardinality constraints. 3. The computation involved only requires solving an unconstrained convex optimization problem (of size linear in the number of neurons in the network), which can be done using a subgradient method efficiently. Further, our approach is anytime, in the sense that the computation can be stopped at any time and a valid bound on the verification objective can be obtained. 4. For the special case of single hidden layer networks, we develop specialized verification algorithms with provable tightness guarantees. 5. We attain state of the art verified bounds on adversarial error rates on image classifiers trained on MNIST and CIFAR-10 under adversarial perturbations in the infinity norm.
 Related Work Certifiable training and verification of neural networks A separate but related thread of work is on certifiable training, ie, training neural networks so that they are guaranteed to satisfy a desired property (for example, robustness to adversarial examples within a certain radius)  [Kolter and Wong, 2018, Raghunathan et al., 2018] . These approaches use ideas from convex optimization and duality to construct bounds on an optimization formulation of verification. However, these approaches are limited to either a class of activation functions (piecewise linear models) or architectures (single hidden layer, as in  Raghunathan et al. [2018] ). Further, in  Kolter and Wong [2018] , the dual problem starts with a constrained convex formulation but is then converted into an unconstrained but nonconvex optimization problem to allow for easy optimization via a backprop-style algorithm. In contrast, our formulation allows for an unconstrained dual convex optimization problem so that for any choice of dual variables, we obtain a valid bound on the adversarial objective and this dual problem can be solved efficiently using subgradient methods. We also note that the ultimate goals of  [Kolter and Wong, 2018, Raghunathan et al., 2018]  are different from our paper: they modify the training procedure of the neural network so that the network is trained to be easily verifiable. In contrast, our work focuses on extending verification algorithms to apply to a broader class of architectures, activation functions and -in this sense, we view our work as complementary to  [Kolter and Wong, 2018, Raghunathan et al., 2018] . In fact, since the objective of our dual optimization is differentiable with respect to the network weights, we can extend our approach to training verifiable networks easily by simultaneously optimizing the network weights and dual variables to minimize the dual objective. We leave the study of such an extension for future work. Theoretical analysis of robustness Another related line of work has to do with theoretical analysis of adversarial examples. It has been shown that feedforward ReLU networks cannot learn to distinguish between points on two concentric spheres without necessarily being vulnerable to adversarial examples within a small radius  [Gilmer et al., 2018] . Under a different set of assumptions, the existence of adversarial examples with high probability is also established in  Fawzi et al. [2018] . In , the authors study robustness of nearest neighbor classifiers to adversarial examples. As opposed to these theoretical analyses, our approach searches computationally for proofs of existence or non-existence of adversarial examples. The approach does not say anything a-priori about the existence of adversarial examples, but can be used to investigate their existence for a given network and compare strategies to guard against adversarial attacks.
 VERIFICATION AS OPTIMIZATION
 NOTATION Our techniques apply to general feedforward architectures and recurrent networks, but we focus on layered architectures for the development in this paper. The input layer is numbered 0, the hidden layers are numbered 1, . . . , L − 1 and the output layer is numbered L. The size of layer l is denoted n l We denote by x in the input to the neural network, by z l the pre-activations of neurons at layer l before application of the activation function and by x l the vector of neural activations after application of the activation function (to z l−1 ). For convenience, we define x 0 = x in . We use x l (x in ), z l (x in ) to denote the activations at the l-th layer as a function of the input x in . Upper and lower bounds on the pre/post activations are denoted by x l , x l , z l , z l respectively. The activation function at layer l is denote h l and is assumed to be applied component-wise, ie, [h l (z l )] k = h l k (z l k ) Note that max-pooling is an exception to this rule -we discuss how max-pooling is handled separately in the Appendix section 6.2.1. The weights of the network at layer l are denoted W l and the bias is denoted b l , z l = W l x l + b l .
 VERIFICATION PROBLEM As mentioned earlier, verification refers to the process of checking that the output of the neural network sat-isfies a certain desirable property for all choices of the input within a certain set. Formally, this can be stated as follows: ∀x in ∈ S in (x nom ) x L (x in ) ∈ S out (1) where x in denotes the input to the network, x nom denotes a nominal input, S in (x nom ) defines the constrained subset of inputs induced by the nominal input, and S out denotes the constraints on the output that we would like to verify are true for all inputs in S in (x nom ). In the case of adversarial perturbations in image classification, x nom would refer to the nominal (unperturbed image), S in (x nom ) would refer to all the images that can be obtained by adding bounded perturbations to x nom , and x in would refer to a perturbed image. In this paper, we will assume that: S out is always a described by a finite set of linear constraints on the values of final layer ie. S out = ∩ m i=1 {x L : c i T x L + d i ≤ 0}, and S in (x nom ) is any bounded set such that any linear optimization problem of the form max x in ∈Sin(x nom ) c T x can be solved efficiently. This includes convex sets and also sets describing combinatorial structures like spanning trees, cuts in a graph and cardinality constraints.
 See the following examples for a concrete illustration of the formulation of the problem: Robustness to targeted adversarial attacks. Consider an adversarial attack that seeks to perturb an input x nom to an input x in subject to a constraint on the perturbation x in − x nom ≤ to change the label from the true label i to a target label j. We can map this to (1) as follows: S in (x nom ) = {x in : x in − x nom ≤ }, (2a) S out = {z : c T z ≤ 0} (2b) where c is a vector with c j = 1, c i = −1 and all other components 0. Thus, S out denotes the set of outputs for which the true label i has a higher logit value than the target label j (implying that the targeted adversarial attack did not succeed). Monotonic predictors. Consider a network with a single real valued output and we are interested in ensuring that the output is monotonically increasing wrt each dimension of the input x in . We can state this as a verifica-tion problem: S in (x nom ) = {x in : x in ≥ x nom } (3a) S out = {x L : x L (x nom ) − x L ≤ 0} (3b) Thus, S out denotes the set of outputs which are large than the network output at x nom . If this is true for each value of x nom , then the network is monotone. Cardinality constraints. In several cases, it makes sense to constrain a perturbation not just in norm but also in terms of the number of dimensions of the input that can be perturbed. We can state this as: S in (x nom ) = {x in : x in − x nom 0 ≤ k, x in − x nom ∞ ≤ } (4a) S out = {z : c T z ≤ 0} (4b) where x 0 denotes the number of non-zero entries in x. Thus, S out denotes the set of outputs which are larger than the network output at x nom . If this is true for each value of x nom , then the network is monotone.
 OPTIMIZATION PROBLEM FOR VERIFICATION Once we have a verification problem formulated in the form (1), we can easily turn the verification procedure into an optimization problem. This is similar to the optimization based search for adversarial examples  [Szegedy et al., 2013]  when the property being verified is adversarial robustness. For brevity, we only consider the case where S out is defined by a single linear constraint c T z + d ≤ 0. If there are multiple constraints, each one can be verified separately. max z 0 ,...,z L−1 x 0 ,...,x L c T x L + d (5a) s.t x l+1 = h l z l , l = 0, 1, . . . , L − 1 (5b) z l = W l x l + b l , l = 0, 1, . . . , L − 1 (5c) x 0 = x in , x in ∈ S in (x nom ) (5d) If the optimal value of this problem is smaller than 0 (for each c, d in the set of linear constraints defining S out ), we have verified the property (1). This is a nonconvex optimization problem and finding the global optimum in general is NP-hard (see Appendix section 6.4.1 for a proof). However, if we can compute upper bounds on the value of the optimization problem and the upper bound is smaller than 0, we have successfully verified the property. In the following section, we describe our main approach for computing bounds on the optimal value of (5).
 BOUNDING THE VALUE OF THE OPTIMIZATION PROBLEM We assume that bounds on the activations z l , x l , l = 0, . . . , L−1 are available. Section 6.1 discusses details of how such bounds may be obtained given the constraints on the input layer S in (x nom ). We can bound the optimal value of (5) using a Lagrangian relaxation of the constraints: max z 0 ,...,z L−1 x 0 ,x 1 ,...,x L−1 c T h L−1 z L−1 + d + L−1 l=0 µ l T z l − W l x l − b l + L−2 l=0 λ l T x l+1 − h l z l (6a) s.t. z l ≤ z l ≤ z l , l = 0, 1, . . . , L − 1 (6b) x l ≤ x l ≤ x l , l = 0, 1, . . . , L − 1 (6c) x 0 ∈ S in (x nom ) (6d) Note that any feasible solution for the original problem (  5 ) is feasible for the above problem, and for any such solution, the terms involving λ, µ become 0 (since the terms multiplying λ, µ are 0 for every feasible solution). Thus, for any choice of λ, µ, the above optimization problem provides a valid upper bound on the optimal value of (5) (this property is known as weak duality  [Vandenberghe and Boyd, 2004] ). We now look at solving the above optimization problem. Since the objective and constraints are separable in the layers, the variables in each layer can be optimized independently. For l = 1, . . . , L − 1, we have f l λ l−1 , µ l = max x l ∈[x l ,x l ] λ l−1 − W l T µ l T x l − b l T µ l which can be solved trivially by setting each component of x l to its upper or lower bound depending on whether the corresponding entry in λ l−1 − W l T µ l is non-negative. Thus, f l λ l−1 , µ l = λ l−1 − W l T µ l + T x l + λ l−1 − W l T µ l − T x l − b l T µ l where [x] + = max(x, 0), [x] − = min(x, 0) denote the positive and negative parts of x. Similarly, collecting the terms involving z l , we have, for l = 0, . . . , L − 1 fl (λ l , µ l ) = max z l ∈[z l ,z l ] µ l T z l − λ l T h l z l where λ L−1 = −c. Since h l is a component-wise nonlinearity, each dimension of z l can be optimized independently. For the k-th dimension, we obtain fl,k λ l k , µ l k = max z l k ∈[z l k ,z l k ] µ l k z l k − λ l k h l k z l k This is a one-dimensional optimization problem and can be solved easily-for common activation functions (ReLU, tanh, sigmoid, maxpool), it can be solved analytically, as discussed in appendix section 6.2. Finally, we need to solve f 0 (µ 0 ) = max x 0 ∈Sin(x nom ) − W 0 T µ 0 T x 0 − b 0 T µ 0 which can also be solved easily given the assumption on S in . We work out some concrete cases in 6.3. Once these problems are solved, we can construct the dual optimization problem: min λ,µ n L−1 k=0 fL−1,k −c k , µ l k + L−2 l=0 n l k=0 fl,k λ l k , µ l k + L−1 l=1 f l (λ l−1 , µ l ) + f 0 µ 0 + d (7) This seeks to choose the values of λ, µ so as to minimize the upper bound on the verification objective, thereby obtaining the tightest bound on the verification objective. This optimization can be solved using a subgradient method on λ, µ. Theorem 1. For any values of λ, µ, the objective of (  7 ) is an upper bound on the optimal value of (5). Hence, the optimal value of (7) is also an upper bound. Further, (7) is a convex optimization problem in (λ, µ). Proof. The upper bound property follows from weak duality  [Vandenberghe and Boyd, 2004] . The fact that (  7 ) is a convex optimization problem can be seen as each term f l , fl,k is expressed as a maximum overa set of linear functions of λ, µ  [Vandenberghe and Boyd, 2004] . Theorem 2. If each h is a ReLU function, then (7) is equivalent to the dual of the LP described in  Ehlers [2017] . Proof. See section 6.4. The LP formulation from  Ehlers [2017]  is also used in  Kolter and Wong [2018] . The dual of the LP is derived in  Kolter and Wong [2018]  -however this dual is different from (7) and ends up with a constrained optimization formulation for the dual (the details of this can be found in appendix section 6.4.2). To allow for an unconstrained formulation, this dual LP is transformed to a backpropagation-like computation. While this allows for folding the verification into training, it also introduces nonconvexity in the verification optimization -our formulation of the dual differs from  Kolter and Wong [2018]  in that we directly solve an unconstrained dual formulation, allowing us to circumvent the need to solve a nonconvex optimization for verification.
 TOWARDS THEORETICAL GUARANTEES FOR VERIFICATION The bounds computed by solving (7) could be loose in general, since (  5 ) is an NP-hard optimization problem (section 6.4.1). SMT solvers and MIP solvers are guaranteed to find the exact optimum for piecewise linear neural networks, however, they may take exponential time to do so. Thus, an open question remains: Are there cases where it is possible to perform exact verification efficiently? If not, can we approximate the verification objective to within a certain factor (known a-priori)? We develop results answering these questions in the following sections. Prior work: For any linear classifier, the scores of each label are a linear function of the inputs w T i x + b i . Thus, the difference between the predictions of two classes j (target class for an adversary) and class i (true label) is (w i − w j ) T x. Maximizing this subject to x − x 0 2 ≤ can be solved analytically to obtain the value (w i − w j ) T x 0 + w i − w j 2 . This observation formed the basis for algorithms in  [Raghunathan et al., 2018]  and  [Hein and Andriushchenko, 2017] . However, once we move to nonlinear classifiers, the situation is not so simple and computing the worst case adversarial example, even in the 2-norm case, becomes a challenging task. In  [Hein and Andriushchenko, 2017] , the special case of kernel methods and single hidden layer classifiers are considered, but the approaches developed are only upper bounds on the verification objective (just like those computed by our dual relaxation approach). Similarly, in  Raghunathan et al. [2018] , a semidefinite programming approach is developed to compute bounds on the verification objective for the special case of adversarial perturbations on the infinity norm. However, none of these approaches come with a-priori guarantees on the quality of the bound, that is, before actually running the verification algorithm, one cannot predict how tight the bound on the verification objective would be. In this section, we develop novel theoretical results that quantify when the verification problem (  5 ) can be solved either exactly or with a-priori approximation guarantees. Our results require strong assumptions and do not immediately apply to most practical situations. However, we believe that they shed some understanding on the conditions under which exact verification can be performed tractably and lead to specialized verification algorithms that merit further study. We assume the following for all results in this section: 1) We study networks with a single hidden layer, i.e. L = 2, with activation function h 0 = h and a linear mapping from the penultimate to the output layer x 2 = h 1 z 1 = z 1 = W 1 x 1 + b 1 . 2) The network has a differentiable activation function h with Lipschitz-continuous derivatives denoted h (tanh, sigmoid, ELU, polynomials satisfy this requirement). 3) S in (x nom ) = {x in : x in − x nom 2 ≤ }. Since the output layer is a linear function of the penultimate layer x 1 , we have c T x 2 = W 1 T c T x 1 + c T b 1 = W 1 T c T h 0 z 0 + c T b 1 For brevity, we simply denote W 1 T c as c, drop the constant term c T b 1 and let W = W 0 , b = b 0 , z nom = W x nom + b and W i denote the i-th row of W . Then, (5) reduces to: max x in : x in −x nom 2 ≤ i c i h i W i x in + b i (8) Theorem 3. Suppose that h has a Lipschitz continuous first derivative: h i (t) − h i ( t) ≤ γ i |t − t| Let ν = diag (c) W T h (z nom ) 2 L = σ max diag (c) W T σ max (diag (γ) W ) Then ∀ ∈ (0, ν 2L ), the iteration: x k+1 ← x nom + W T diag (c) h (W x k + b) W T diag (c) h (W x k + b) 2 starting at x 0 = x nom converges to the global optimum x of (8) at the rate x k − x ≤ L ν− L k Proof. Section 6.4 Thus, when is small enough, a simple algorithm exists to find the global optimum of the verification objective. However, even when is larger, one can obtain a good approximation of the verification objective. In order to do this, consider the following quadratic approxiimation of the objective from (8): max i c i (h i (z nom i ) + h i (z nom i ) (W i z)) + i c i 2 h i (z nom i ) (W i z) 2 (9a) s.t. z 2 ≤ (9b) This optimization problem corresponds to a trust region problem that can be solved to global optimality using semidefinite programming  [Yakubovic, 1971] : max z,Z i c i (h i (z nom i ) + h i (z nom i ) (W i z)) + i c i 2 h i (z nom i ) tr W T i W i Z (10a) s.t. tr (Z) ≤ , 1 z T z Z 0 (10b) where X 0 denotes that X is constrained to be a positive semidefinite matrix. While this can be solved using general semidefinite programming solvers, several special purpose algorithms exist for this trust region problem that can exploit its particular structure for efficient solution,  [Hazan and Koren, 2016]  Theorem 4. Suppose that h is thrice-differentiable with a globally bounded third derivative. Let ζ i = W i 2 , η i = sup t |h i (t)|, κ = 1 6 i η i c i ζ 3 i For each > 0, the difference between the optimal values of (10), (  8 ) is at most κ 3 . Proof. See Section 6.4
 EXPERIMENTS In this section, we present numerical studies validation our approach on three sets of verification tasks: Image classification on MNIST and CIFAR: We use our approach to obtain guaranteed lower bounds on the accuracy of image classifers trained on MNIST and CIFAR-10 under adversarial attack with varying sizes of the perturbation radius. We compare the bounds obtained by our method with prior work (in cases where prior work is applicable) and also with the best attacks found by various approaches. Classifier stability on GitHub data: We train networks on sequences of commits on GitHub over a collection of 10K repositories -the prediction task consists of predicting whether a given repository will reach more than 40 commits within 250 days given data observed until a certain day. Input features consist a value between 0 and 1 indicating the number of days left until the 250th day, as well as another value indicating the progress of commits towards the total of 40. As the features evolve, the prediction of the classifier changes (for example, predictions should become more accurate as we move closer to the 250th day). In this situation, it is desirable that the classifier provides consistent predictions and that the number of times its prediction switches is as small as possible. It is also desirable that this switching frequency cannot be easily be changed by perturbing input features. We use our verification approach combined with dynamic programming to compute a bound on the maximum number of switches in the classifier prediction over time. Digit sum task: We consider a more complex verification task here: Given a pair of MNIST digits, the goal is to bound how much the sum of predictions of a classifier can differ from the true sum of those digits under adversarial perturbation subject to a total budget on the perturbation across the two digits.
 IMAGE CLASSIFICATION: MNIST AND CIFAR We study adversarial attacks subject to an l ∞ bound on the input perturbation. An adversarial example (AE) is a pertubation of an input of the neural network such that the output of the neural network differs from the correct label for that input. An AE is said to be within radius if the ∞ norm of the difference between the AE and the original input is smaller than . We are interested in the adversarial error rate, that is, # Test examples that have an AE within radius Size of test set Computing this quantity precisely requires solving the NP-hard problem (5) for each test example, but we can obtain upper bounds on it using our (and other) verification methods and lower bounds using a fixed attack algorithm (in this paper we use a bound constrained LBFGS algorithm similar to  [Carlini and Wagner, 2017b] ). Since theorem 2 shows that for the special case of piecewise linear neural networks, our approach reduces to the basic LP relaxation from  Ehlers [2017]  (which also is the basis for the algorithms in  Bunel et al. [2017]  and  Kolter and Wong [2018] ), we focus on networks with smooth nonlinearities like tanh and sigmoid. We compare our approach with The SDP formulation from  Raghunathan et al. [2018]  (note that this approach only works for sin-gle hidden layer networks, so we just show it as producing vacuous bounds for other networks). Each approach gets a budget of 300 s per verifiation problem (choice of test example and target label). Since the SDP solver from  [Raghunathan et al., 2018]  only needs to be run once per label pair (and not per test example), its running time is amortized appropriately. Results on smooth activation functions: Figures  1a,1b  show that our approach is able to compute nearly tight bounds (bounds that match the upper bound) for small perturbation radii (up to 2 pixel units) and our bounds significantly outperform those from the SDP approach  [Raghunathan et al., 2018]  ( which is only able to compute nontrivial bounds for the smallest model with 20 hidden units). Results on models trained adversarially: We use the adversarial training approach of  [Uesato et al., 2018]  and train models on MNIST and CIFAR that are robust to perturbations from the LBFGS-style attack on the training set. We then apply our verification algorithm to these robust models and obtain bounds on the adversarial error rate on the test set. These models are all multilayer models, so the SDP approach from  [Raghunathan et al., 2018]  does not apply and we do not plot it here. We simply plot the attack versus the bound from our approach. The results for MNIST are plotted in figure  2b  and for CIFAR in figure  2a . On networks trained using a different procedure, the approaches from Raghunathan et al. [2018] and  Kolter and Wong [2018]  are able to achieve stronger results for larger values of (they work with = .1 in real units, which corresponds to = 26 in pixel units we use here). However, we note that our verification procedure is agnostic to the training procedure, and can be used to obtain fairly tight bounds for any network and any training procedure. In comparison, the results in  Kolter and Wong [2018]  and  Raghunathan et al. [2018]  rely on the training procedure optimizing the verification bound. Since we do not rely on a particular adversarial training procedure, we were also able to obtain the first non-trivial verification bounds on CIFAR-10 (to the best of our knowledge) shown in figure  2a . While the model quality is rather poor, the results indicate that our approach could scale to more complicated models.
 GITHUB CLASSIFIER STABILITY We allow the adversary to modify input features by up to 3% (at each timestep) and our goal is to bound the maximum number of prediction switches induced by each attack over time. We can model this within our verification framework as follows: Given a sequence of input features, we compute the maximum number of switches    [Raghunathan et al., 2018] . Each color represents a different network. The dashed lines at the bottom are lower bounds on the error rate computed using the best attack found using the LBFGS algorithm. achievable by first computing the target classes that are reachable through an adversarial attack at each timestep (using (  7 )), and then running a dynamic program to compute the choices of target classes over time (from within the reachable target classes) to maximize the number of switches over time. Figure  3a  shows how initially predictions are easily attackable (as little information is available to make predictions), and also shows how the gap between our approach and the best attack found using the LBFGS algorithm evolves over time.
 COMPLEX VERIFICATION TASK: DIGIT SUM In order to test our approach on a more complex specification, we study the following task: Given a pair of MNIST digits, we ask the question: Can an attacker perturb each image, subject to a constraint on the total perturbation across both digits, such that the sum of the digits predicted by the classifier differs from the true sum of those digits by as large an amount as possible? Answering this question requires solving the following optimization problem: where s is the true sum of the two digits. Thus, the adversary has to decide on both the perturbation to each digit, as well as the size of the perturbation. We can encode this within our framework (we skip the details here). The upper bound on the maximum error in the predicted sum from the verification and the lower bound on the maximum error computed from an attack for this problem (on an adversarially trained two hidden layer sigmoid network) is plotted in figure  3b . The results show that even on this rather complex verification task, our approach is able to compute tight bounds.
 CONCLUSIONS We have presented a novel framework for verification of neural networks. Our approach extends the applicability of verification algorithms to arbitrary feedforward networks with any architecture and activation function and to more general classes of input constraints than those considered previously (like cardinality constraints). The verification procedure is both efficient (given that it solves an unconstrained convex optimization problem) and practically scalable (given its anytime nature only required gradient like steps). We proved the first known (to the best of our knowledge) theorems showing that under special assumptions, nonlinear neural networks can be verified tractably. Numerical experiments demonstrate the practical performance of our approach on several classes of verification tasks. Figure 1 : 1 Figure 1: Figures show three curves per model -Dashed line: Lower bound from LBFGS attack. Solid line: Our verified upper bound. Dash-dot line: SDP verified upper bound from [Raghunathan et al., 2018] . Each color represents a different network. The dashed lines at the bottom are lower bounds on the error rate computed using the best attack found using the LBFGS algorithm.
 . x in a − x nom a ≤ a , x in b − x nom b ≤ b a + b ≤

Abstract: This paper addresses the problem of formally verifying desirable properties of neural networks, i.e., obtaining provable guarantees that neural networks satisfy specifications relating their inputs and outputs (robustness to bounded norm adversarial perturbations, for example). Most previous work on this topic was limited in its applicability by the size of the network, network architecture and the complexity of properties to be verified. In contrast, our framework applies to a general class of activation functions and specifications on neural network inputs and outputs. We formulate verification as an optimization problem (seeking to find the largest violation of the specification) and solve a Lagrangian relaxation of the optimization problem to obtain an upper bound on the worst case violation of the specification being verified. Our approach is anytime i.e. it can be stopped at any time and a valid bound on the maximum violation can be obtained. We develop specialized verification algorithms with provable tightness guarantees under special assumptions and demonstrate the practical significance of our general verification approach on a variety of verification tasks.