Seán S Óhéigeartaigh
Jess Whittlestone
Yang Liu
Yi Zeng
Zhe Liu

Overcoming Barriers to Cross-cultural Cooperation in AI Ethics and Governance

Introduction Artificial intelligence has been identified as a key suite of technologies for many countries worldwide, in large part motivated by its general-purpose nature  (Brynjolfsson and McAfee 2014) . AI technologies, and machine learning techniques in particular, are being fruitfully applied to a vast range of domains, including language translation, scientific research, education, logistics, transport and many others. It is clear that AI will affect economies, societies and cultures profoundly at a national, international and global level. This has resulted in increasing attention being paid to both AI ethics: questions about how we should develop and deploy AI systems, given their potential impact on wellbeing and other deeply held values such as autonomy or dignity; and to AI governance: the more practical challenge of ensuring the ethical use of AI in society, be that through regulation, governance frameworks, or 'softer' approaches such as standards and ethical guidelines.  1  Cross-cultural cooperation will be essential for the success of these ethics and governance initiatives. By 'cross-cultural cooperation', we mean groups from different cultures and nations working together on ensuring that AI is developed, deployed, and governed in societally beneficial ways. In this paper, we focus in particular on cooperation extending across national boundaries. Examples include (but are not limited to) the following: AI researchers from different countries collaborating on projects to develop systems in safe and responsible ways; establishing networks to ensure that diverse global perspectives can feed equally into international discussions about the ethical issues raised by AI; and involving a range of global stakeholders in the development of practical principles, standards, and regulation. By encouraging crosscultural cooperation, we do not necessarily mean that all parts of the world should be subject to the same norms, standards, and regulation relating to AI, or that international agreement is always needed. Identifying which issues will need global standards or agreements, and where more cultural variation is needed, is itself a key challenge that will require cooperation to address. Cross-cultural cooperation is important for several reasons. First, cooperation will be essential if AI is to bring about broad benefits across societies globally, enabling advances in one part of the world to be shared with other countries, and ensuring that no part of society is neglected or disproportionately negatively impacted by AI. Second, cooperation enables researchers around the world to share expertise, resources, and best practices. This enables faster progress both on beneficial AI applications, and on managing the ethical and safety-critical issues that may arise. Third, in the absence of cooperation, there is a risk that competitive pressures between states or commercial ecosystems may lead to underinvestment in safe, ethical, and socially beneficial AI development  (Askell et al. 2019; Ying 2019) . Finally, international cooperation is also important for more practical reasons, to ensure that applications of AI that are set to cross-national and regional boundaries (such as those used in major search engines or autonomous vehicles) can interact successfully with a sufficient range of different regulatory environments and other technologies in different regions  (Cihon 2019) . Drawing on the insights of a group of leading scholars from East Asia and Europe, 2 we analyse current barriers to cross-cultural cooperation on AI ethics and governance, and how they might be overcome. We focus on cooperation between Europe and North America on the one hand and East Asia on the other. These regions are currently playing an outsized role in influencing the global conversation on AI ethics and governance  (Jobin et al. 2019) , and much has been written recently about competition and tensions between nations in these regions in the domains of AI development and governance, especially in the case of China and the USA. However, our discussion and recommendations have implications for broader international cooperation around AI, and we hope they will spur more attention to promoting cooperation across a wider range of regions. As AI systems become more capable and their applications more impactful and ubiquitous, the stakes will only get higher. Establishing cooperation over time may become more difficult, especially if cross-cultural misunderstandings and mistrust become entrenched in intellectual and public discussions. If this is the case, then the earlier a global culture of cooperation can be established, the better. Cultivating a shared understanding and deep cooperative relationships around guiding the impacts of AI should therefore be seen as an immediate and pressing challenge for global society. 2 The Role of North America, Europe, and East Asia in Shaping the Global AI Conversation North America, Europe, and East Asia in particular are investing heavily in both fundamental and applied AI research and development  (Benaich and Hogarth 2019; Haynes and Gbedemah 2019; Perrault et al. 2019) , supported by corporate and government investment. Various analyses have framed progress on the development and deployment of AI between the USA and China in particular through a competitive lens  (Simonite 2017; Allen and Husain 2017; Stewart 2017) , though this framing has received criticism both on normative and descriptive grounds (Cave and ÓhÉigeartaigh 2018). Scholars and policy communities in these regions are also taking deliberate and active steps to shape the development of ethical principles and governance recommendations for AI, both at a regional and global level. This is reflected in governmentlinked initiatives, such as the activities of the European Union's High-Level Expert Group on Artificial Intelligence, which has produced ethics guidelines and policy and 2 We convened a workshop on cross-cultural trust in July 2019 (https://www.eastwest.ai/), with representatives from UK universities (Cambridge, Bath) as well as from Chinese and Japanese universities and initiatives (Universities of Hong Kong, Peking, Fudan, Keio, and the Berggruen Institute China Center and the Chinese Academy of Sciences). These representatives have been heavily involved in AI ethics and governance conversations and collaborative projects across Europe, North America, and Asia. This paper draws on insights from this workshop, which focused particularly on the role of academia in building crosscultural trust in AI. We do not suggest that the regions represented are the only ones that matter in this conversation, nor that the members present can be considered to represent the diversity of views and expertise in the regions they are based in. However, we feel that this small step in establishing a network was of value, and generated useful insights and ground to build on. The workshop was held under Chatham House Rule. investment recommendations as its first two publications 3 ; the UK Government's commitments to 'work closely with international partners to build a common understanding of how to ensure the safe, ethical and innovative deployment of Artificial Intelligence'  (May 2018) ; and the Chinese Government's similar commitment to 'actively participate in global governance of AI, strengthen the study of major international common problems such as robot alienation and safety supervision, deepen international cooperation on AI laws and regulations, international rules' (China State Council 2017). North America, Europe, and East Asia are each also contributing disproportionately towards international AI standards work ongoing in fora such as the International Organization for Standardization (ISO), 4 the Institute of Electrical and Electronics Engineers (IEEE), 5 and the Organization for Economic Co-operation and Development (OECD).  6  The prominence of North America, Europe, and East Asia is further seen in the leadership and composition of multi-stakeholder and nongovernmental initiatives such as the Partnership on AI, 7 the Future Society, 8 the International Congress for the Governance of AI, 9 and the Global Partnership on AI  (Hudson 2019 ). 10 A large majority of the most prominent conferences on AI ethics and governance have taken place in these regions, including the US-based Beneficial AI conference series, 11 the Beijing Academy of AI Conference series, 12 the Beijing Forum, 13 the US-based Artificial Intelligence, Ethics, and Society conferences, 14 governance and ethics workshops attached to the leading machine learning conferences, and many more. This combination of: a. technological leadership in North America, Europe, and East Asia; b. the outsized role of these regions in shaping the global ethics and governance conversation; and c. the underlying tension introduced by progress being framed through a competitive lens, and a perception of disagreements on fundamental ethical and governance issues leads us to focus on the barriers that exist to productive intellectual exchange and cooperation between these regions and cultures in particular. A full analysis of cross-cultural cooperation on AI ethics and governance, which is outside the scope of this paper, must consider the roles of all nations and cultures, given that the domain of impact of AI technologies is truly global. It will be particularly important for further work to address the inequalities in power and influence that are emerging between technology-leading nations and those to which these technologies are being exported  (Lee 2017) , and the responsibility of technology-leading nations to include and empower those nations in global governance and ethics conversations. 3 Barriers to Cross-cultural Cooperation on AI Despite the emergence of several international alliances in AI ethics and governance, many barriers remain to achieving real cross-cultural cooperation on the norms, principles, and governance frameworks that should guide how AI is developed and used. Mistrust between different regions and cultures is one of the biggest barriers to international cooperation in AI ethics and governance. At present, there is a particular environment of mistrust between scholars, technologists, and policymakers in the USA and China.  15  This culture of mistrust is underpinned by both: a. a history of political tensions between these two powerful regions that has increased significantly in recent years, and is currently contributing to the competitive framing of AI development as a 'race' between 'Eastern' and 'Western' nations.  16  and b. the divergent philosophical traditions upon which these regions are founded, leading to a perception of significant and irresolvable value differences between 'Western' and 'Eastern' cultures on key issues such as data privacy  (Larson 2018; Horowitz et al. 2018; Houser 2018) . A range of recent technological and political developments may also be contributing to this mistrust, including concerns about the public and political influence of technology giants in the USA  (Ochigame 2019) ; perceptions of and reactions to the Chinese Social Credit score system  (Chorzempa et al. 2018; Song 2019) ; and concerns about contentious uses of AI technologies, with notably controversial examples including the use of AI in immigration control  (Whittaker et al. 2018) , criminal risk assessment in the USA  (Campolo et al. 2017) , and in tracking communities such as the Uighur Muslim minority in China  (Mozur 2019) . Adversarial rhetoric from political and defence leaders in the USA also contributes to this tension. Recent examples reported in the media include stating intentions to 'be the threat' of AI 17 ; comments focused on the 'otherness' of China as an adversary, 18 amid broader concerns regarding Chinese technological progress as a threat to US global leadership  (Jun 2018) . A continued exacerbation of this culture of mistrust could severely undermine possibilities for global cooperation on AI development and governance. In addition, it is unclear how far existing cross-cultural collaborations and alliances can go to shape the behaviour of actors that are as globally dominant as the USA, China, and the large multinational corporations based in these countries. Even if AI ethics frameworks can be agreed on in principle by multi-stakeholder groups, for example, it will be far from straightforward to implement them in practice to constrain the behaviour of those with disproportionate power to shape AI development and governance. Another challenge for effective cooperation is balancing the need for global cooperation with the need for culturally and geographically sensitive differentiated approaches  (Hagerty and Rubinov 2019) . It is crucial we avoid a situation where one or two nations simply try to impose their values on the rest of the world  (Acharya 2019) . In certain specific domains, for example where AI is being used to support the delivery of healthcare, different cultures may perceive tradeoffs very differently  (Feldman et al. 1999) , and it may be not just possible but necessary to implement region-specific standards and governance. AI systems will also have different impacts as they are deployed in different cultural regions, which may also require different governance approaches  (Hagerty and Rubinov 2019) . For some aspects of AI development and governance, however, cooperation will be much more crucial. For example, some potential uses of AI technologies in military contexts, such as in automated targeting and attack, could impinge upon human rights and international humanitarian law  (Asaro 2012) . Another concern is that by automating aspects of information gathering, decision-making, and response in military arenas, the potential for unwanted escalation in conflict situations may increase due to events occurring and requiring responses faster than is compatible with effective human judgement and oversight  (Altmann 2019) . In both these cases, there may be individual military advantages to nations pursuing the technology; but in the absence of international agreements and standards, the overall effect may be destabilizing.  19  International agreement will also be of particular importance for all cases in which AI technologies are developed in one region, but used or deployed in a different region. A key challenge for cross-cultural cooperation will therefore be to identify the areas where international agreement is most important, and distinguish these from areas where it is more appropriate to respect a plurality of approaches. There are also more practical barriers to cooperation between nations: language barriers, lack of physical proximity, and immigration restrictions put limits on the ability of different cultures and research communities to communicate and collaborate. Furthermore, despite science being a global enterprise, the language of scientific publication remains predominantly English.
 Overcoming These Barriers to Cooperation While cross-cultural cooperation in AI ethics and governance will be genuinely challenging, we suggest that there are steps that can be taken today to make progress, without first needing to tackle the greater problems of finding consensus between cultures on all fundamental ethical and philosophical issues, or resolving decades of political tension between nations.
 Building Greater Mutual Understanding, Including Around Disagreements Mistrust between nations is a serious concern for the future of AI ethics and governance. However, we suggest that this mistrust is at least partly fuelled by misunderstandings and misperceptions, and that a first step towards building greater crosscultural trust must therefore be to identify and correct important misperceptions, and enhance greater mutual understanding between cultures and nations. It would be easy to assume that the main barrier to building trust between East and West around AI is that these regions of the world have very different fundamental values, leading them to different-perhaps conflicting-views of how AI should be developed, used, and governed from an ethical perspective. While value differences between cultures certainly exist, claims about how those differences manifest often depend on unexamined concepts and entrenched assumptions, and lack empirical evidence  (Whittlestone et al. 2019) . The idea that 'Eastern' and 'Western' ethical traditions are fundamentally in conflict also oversimplifies the relationship between the two. There are many different philosophical traditions that might be referred to under either heading: there are, for example, many important differences between relevant philosophical perspectives across China, Japan, and Korea  (Gal 2019) , and the values and perspectives of 'Western' philosophy have changed a great deal over time  (Russell 1945) . More broadly, ethical and cultural values in both regions are in constant evolution, as captured by projects such as the World Values Survey 20 and the Asian Barometer.  21  Differences in ethical and cultural traditions and norms across regions are often assumed to underpin contrasting governance approaches. For example, privacy is often seen as an issue for which significant value differences exist between East and West, leading to a perception that laxer regulation and controls exist on data privacy in China compared with the USA and Europe. However, such claims are often made in very broad terms, without substantive evidence or analysis of how significant these differences are or how they manifest in practice  (Ess 2005; Lü Yao-Huai 2005) . This leads to misunderstandings in both directions. First, there are significant differences between the USA and Europe on both conceptions of privacy  (Szeghalmi 2015)  and regulations that relate to privacy  (McCallister et al. 2018) . These are often missed in Chinese perceptions of Western societies, which tend to focus just on the USA.  22  Second, Western perceptions of data privacy in China may be outdated: Lü Yao-Huai (2005) highlighted as early as 2005 that the relevant literature on information ethics was much younger in China than in the USA, but was evolving quickly and in a manner significantly informed by Western scholarship. A range of papers and reports from Chinese scholars and policymakers have highlighted the importance of data privacy in AI ethics and governance (Beijing Academy of Artificial Intelligence 2019; Ying 2019;  Zeng et al. 2018; Ding 2018b ). Principles around protecting individuals' data privacy are also beginning to be borne out in regulatory action in China; over 100 apps have been banned by the government for user data privacy infringements, with dozens more being required to make changes relating to data collection and storage.  23  This is not to suggest that there are not meaningful differences in values, norms, and regulations relating to data privacy between these countries, but that such differences have often been oversimplified and are not well understood. Another example of differing perceptions are those surrounding China's social credit score (SCS) system. The SCS has been discussed with great concern in Western media, policy circles, and scholarship, and presented as an example of Orwellian social control by the Chinese government  (Botsman 2017; Pence 2018) , representative of a culture and government with profoundly different values to the West  (Clover 2016 ). However, both Chinese and Western sources have argued that there are significant misunderstandings surrounding the SCS. Multiple scholars have pointed out that the SCS is not designed to be a single unified platform that rates all 1.4 billion Chinese citizens (as is often supposed), but rather a web of individual platforms with latitude for different interpretations, with social credit scores mostly given by financial institutions (as opposed to a big data-driven comprehensive rating)  (Mistreanu 2019; Sithigh and Siems 2019) .  Song (2019)  notes that many of the measures in the SCS are designed to tackle issues such as fraud and corruption in local government.  Chorzempa et al. (2018)  also highlight that 'many of the key components of social credit, from blacklists to widespread surveillance… already exist in democracies like the United States.' China's SCS is likely to evolve significantly over time, and there are likely to be genuine reasons for concern both in terms of present and future implementation. However, a much clearer cross-cultural understanding of how the SCS works, is being used, and is impacting Chinese citizens, would allow dialogue on relevant ethics and governance issues to progress more constructively. Given the lack of shared knowledge and discourse that has existed historically between regions such as the USA, Europe, and China, it is not surprising that many  22  This was a misperception noted by multiple Chinese scholars at the aforementioned July workshop.  23  In November 2019, the Chinese Ministry of Public Security banned 100 apps that failed to meet standards on individuals' data privacy; the body has investigated 683 apps in 2019 (National Cyber Security Advisory Centre 2019). In addition, in December 2019, the Chinese Ministry of Industry and Information Technology released a list of 41 apps that would have to make changes to comply with data regulations by the end of 2019 (Ministry of Industry and Information Technology of the People's Republic of China 2019). In July 2018, China's Shandong Province brought a major case relating to infringement of personal information against 11 companies  (Ding 2018c).  misperceptions exist between them. We should therefore be wary of jumping too quickly to assume intractable and fundamental disagreements. Misunderstandings clearly exist in both directions: analyses of public opinion survey data suggest, for example, that both American and Chinese populations hold various misperceptions about the other nation's traits and characteristics (Johnston and Shen 2015). As mentioned above, in China, the diversity of Western societies is also often oversimplified down to a single pattern of American life. At the same time, the USA and Europe have historically struggled to understand China (Chen and Hu 2019), evidenced for example by repeated failures to predict China's liberalization (or lack thereof) or periods of economic growth (The Economist 2018; Cowen 2019; Liu 2019). Language barriers present a particular difficulty for Western nations in gleaning what is happening in China in terms of AI development, ethics, and governance  (Zhang 2017; ). As Andrew Ng points out in a 2017 interview in the Atlantic: 'The language issue creates a kind of asymmetry: Chinese researchers usually speak English so they have the benefit of access to all the work disseminated in English. The Englishspeaking community, on the other hand, is much less likely to have access to work within the Chinese AI community'  (Zhang 2017) . For example, Tencent released a book on AI strategy  (Tencent Research Institute et al. 2017)  which includes deep analysis of ethics, governance, and societal impacts, but has received relatively little English-language coverage (Ding 2018a). Even on empirical matters such as level of Chinese public investment in AI research and development, widely reported figures in the USA may be inaccurate by an order of magnitude  (Acharya and Arnold 2019) . The recently published Beijing AI Principles (Beijing Academy of Artificial Intelligence 2019) and similar principles developed around the world  (Cowls and Floridi 2018)  in fact show substantial overlap on key challenges  (Zeng et al. 2018; Jobin et al. 2019) . The Beijing Principles make clear reference to the key concepts and values which have been prominent in other documents, including that AI should 'benefit all humankind'; respect 'human privacy, dignity, freedom, autonomy and rights'; and be 'as fair as possible, reducing possible discrimination and biases, improving its transparency, explainability and predictability.' In addition, both the Beijing AI Principles, and the National Governance Principles of the New AI, China, call for openness and collaboration, with the latter encouraging 'cooperation across disciplines, domains, regions, and borders'  (Laskai and Webster 2019) . However, nations with different cultures may interpret and prioritize the same principles differently in practice  (Whittlestone et al. 2019) , which may be a further source of misunderstanding. We cannot simply assume, for example, that 'Western' cultures value privacy more highly than 'Eastern' ones; instead, we need a more nuanced understanding of how privacy may be prioritized differently when it comes into conflict with other important values, such as security  (Capurro 2005) . Similarly, although it is important to recognize that many cultures value autonomy, it is equally important to understand the different connotations and philosophical assumptions underpinning this value in different contexts  (Yunping 2002) . Given the rich history of misunderstanding between nations, to build greater crosscultural cooperation, we should start by focusing on identifying those misperceptions most relevant to AI ethics and building greater mutual understanding of where more substantive disagreements exist and are likely to impact governance approaches. In doing so, it is worth distinguishing explicitly between disagreements pertaining to ethics as opposed to governance issues, since it may sometimes be possible for groups to agree on the same governance principles despite justifying them with different ethical assumptions, as we will discuss later. It may also be helpful to distinguish between misunderstandings that pertain directly to AI (such as misperceptions of other countries' investment in technology, or misinterpretation of data protection laws) and those that pertain to broader societal, political, or philosophical matters that are more indirectly relevant to AI, as they may require different approaches to resolve. Acknowledging the role of misunderstandings does not, of course, imply that all matters of intercultural tension in AI ethics and governance are fundamentally based on misunderstandings. Deep and fundamental disagreements across regions will remain on a range of issues, including those relating to the relationship between the individual, society, and the state; the level and nature of integration between civil, private, and military sectors; and various specific matters of social policy. However, focusing initially on reducing misunderstandings will aid in establishing more clearly where these fundamental differences exist, while at the same time identifying contexts in which sufficient agreement exists for fruitful cooperation. Doing so is a crucial first step towards addressing the broader challenges of cross-cultural cooperation on AI ethics and governance.
 Finding Ways to Cooperate Despite Disagreements Even where important differences of view on AI ethics, governance, and broader societal issues exist, forms of agreement and cooperation can still be possible. As mentioned earlier, a key outstanding challenge for AI ethics and governance is identifying those areas where cross-cultural agreement on norms, standards, or regulation is crucial, and where different interpretations and approaches are acceptable or even desirable. This is precisely the kind of challenge which itself requires crosscultural cooperation: the delineations must be informed by diverse cultural perspectives on the impacts of AI in different contexts, and the needs and desires of different populations. Indeed, this approach is reflected in the National Governance Principles of the New AI, China, which includes the recommendation to 'Launch international dialogue and cooperation; with full respect for each country's principles and practices for AI governance, promote the formation of a broad consensus on an international AI governance framework, standards, and norms'  (Laskai and Webster 2019) . Regional and cultural differences on the abstract level of ethical assumptions and high-level principles are also not necessarily a barrier to agreement on more concrete norms and governance. If it were impossible to reach any practical agreement without consensus on fundamental ethical issues, many important international agreementssuch as the Nuclear Weapons Ban Treaty-would not have been possible. The notion of an 'incompletely theorized agreement' in legal scholarship  (Sunstein 1995) , describes how it is often possible for people who disagree on fundamental or abstract matters to nonetheless agree on specific cases-and that this is central to the functioning of law as well as of a pluralistic society more broadly. Several authors in the literature on intercultural information ethics have promoted the related idea of aiming to arrive at an 'overlapping consensus'  (Rawls 1993) , where different groups and cultures may have different reasons for supporting the same norms and practical guidelines  (Taylor 1996; Søraker 2006; Hongladarom 2016) . For example,  Taylor (1996)  discusses how we have managed to ground shared norms of human rights in different cultural traditions. While Western philosophies differ substantially from others such as Buddhism in how much importance they give to the human agent and its unique place in the cosmos, both seem to end up grounding the same norms of human rights.  Wong (2009)  criticizes this idea that intercultural information ethics can arrive at shared norms with different justifications, suggesting that this risks making norms too 'thin', devoid of all normative content.  Søraker (2006)  acknowledges a similar objection to this 'pragmatic' approach to information ethics: that it may result in agreements that are fragile due to not being sufficiently grounded in substantive normative content. However, in line with Søraker's own response to these objections, we believe that the aim of 'overlapping consensus' should be to arrive at shared norms and practical guidelines which are in fact more robust by virtue of being endorsed and justified from a range of different philosophical or normative perspectives. This should be distinguished from a situation where one culture uses pragmatic arguments to attempt to force their own values upon others, or where several cultures reach agreement but for reasons with little normative content, which, we agree with Wong, would be concerning. Taylor's example of human rights being supported by multiple philosophical perspectives appears to demonstrate the plausibility of this kind of wellsubstantiated overlapping consensus. Indeed, we suggest that finding areas of overlapping consensus on norms and practical guidelines may be much more important for ensuring the benefits of AI than aiming for global consensus on a shared set of fundamental values-an aim which underpins many recent proposals.  24  Consensus on high-level ethical principles does not necessarily mean they are well-justified  (Benjamin 1995) , and the best way to arrive at more robustly justified norms, standards, and regulation for AI will be to find those that can be supported by a plurality of different value systems.
 Putting Principles into Practice Even where it is possible to improve mutual understanding and arrive at shared governance approaches in theory, some might object that it will still be difficult to influence the development and use of AI in practice, especially since to do so requires influencing the behaviour of powerful states and companies that have little incentive to cooperate. Although a full discussion of the complex power dynamics between states, companies, and other actors on issues relevant to AI ethics and governance is beyond the scope of this paper (and worthy of further research), we briefly explain why we do not think this barrier undermines our proposals. While challenging, historical precedent suggests that it is possible for public and academic alliances to influence the behaviour of powerful actors on issues of global importance. There is evidence to suggest that broad, cross-cultural 'epistemic communities'-i.e. networks of experts in a particular domain-can be particularly effective at supporting international policy coordination (Haas 1992). For example, a community of control experts helped shape coopbetween the USA and Russia during the Cold War by creating an internationally shared understanding of the problem of nuclear arms control  (Adler 1992) , and the ecological epistemic community managed to successfully coordinate national policies to protect the stratospheric ozone layer  (Haas 1992) . The commitments of large companies and even nations around AI have already been influenced by combinations of employee activism, international academic research, and campaigning. A notable example is in the application of AI in military contexts. Concerns over the use of AI in warfare have been the subject of high-profile campaigns by experts across academia and civil society internationally, such as those involved in the International Committee for Robot Arms Control (ICRAC) and the Campaign to stop Killer Robots. These campaigns played a leading role in establishing discussion on lethal autonomous weapons (LAWs) at the United Nations Convention on Certain Conventional Weapons (CCW; the body that hosted negotiations over the banning of cluster munitions, blinding laser weapons, and landmines)  (Belfield 2020) . Ninety countries have put forward statements on LAWs, with most doing so at the CCW; 28 countries support a ban (Campaign to Stop Killer Robots 2018).  25  In 2018, over 4000 Google employees signed a letter protesting Google's involvement in the Pentagon's Project Maven, a military project exploring the use of AI in footage analysis  (Shane and Wakabayashi 2018) , and a number resigned  (Conger 2018) . Several other campaigning groups, comprised of scholars and researchers from the USA, Europe, Japan, China, Korea and elsewhere, released public articles and letters supporting the concerns of the Google practitioners (ICRAC 2018).  26  Google subsequently announced it would not renew its contract on Project Maven, and would not bid on a $10 billion Department of Defence cloud computing contract  (Belfield 2020) . More broadly, international academic and civil society input has played a significant role in shaping principles that are likely to form the basis for binding regulation in years to come. For example, the European Commission's white paper On Artificial Intelligence -A European Approach to Excellence and Trust (European Commission 2020) lays out 'policy options for a future EU regulatory framework that would determine the types of legal requirements that would apply to relevant actors, with a particular focus on high-risk applications'(European Commission 2020b).This document was strongly influenced by the work of the European Union's High-Level Expert group on Artificial Intelligence, comprising 52 European experts from across academia, industry, and civil society.  27  Similarly, the US Department of Defence has formally adopted ethical principles on AI (US Department of Defense 2020), after 15 months of consultation with US-based academic, industry, and government stakeholders, and has hired staff to implement these principles  (Barnett 2020) . While in both cases the groups consulted were region-specific, the degree of alignment and overlap between principles  25  China supports a ban on use of fully autonomous weapons on the battlefield, but not their production and development. The USA, Russia, UK, Israel, and France oppose a ban.  (Kania 2018 )  26  An open letter was released by the International Committee for Robot Arms control and signed by over 1000 scholars and researchers, and members of the Campaign to Stop Killer Robots wrote public articles and letters to company leaders supporting the concerns of the Google petitioners: https://www.stopkillerrobots. org/2019/01/rise-of-the-tech-workers/ 27 Information on process and composition available at: https://ec.europa.eu/digital-single-market/en/highlevel-expert-group-artificial-intelligence developed in different regions suggests the insights and recommendations are tially informed by interaction with broader epistemic communities from regions. This suggests that insights gained from cross-cultural cooperation and consensus can meaningfully feed into regulatory frameworks at a regional and national level.
 Recommendations Academia has an important role to play in supporting cross-cultural cooperation on AI ethics and governance: both through research into where and what kinds of cooperation are most needed, and by establishing initiatives to overcome more practical barriers to cooperation. Our discussion in this paper raises many questions that will require diverse academic expertise to answer, including questions about what important misperceptions most hinder cooperation across regions; where international agreement is needed on AI ethics and governance; and how agreement might be reached on specific governance standards despite differences on ethical issues. Academic communities are also particularly well-suited to building greater mutual understanding between regions and cultures in practice, due to their tradition of free-flowing, international, and intercultural exchange of ideas. Academics can have open conversations with international colleagues in a way that is often challenging for those working in industry or government, and two academics from different parts of the world can productively collaborate even if each has strong criticisms of the other nation's government and/or companies. The following recommendations indicate a number of specific steps that academic centres, research institutions, and individual researchers can take to promote crosscultural understanding and cooperation on AI ethics and governance. Some excellent work is already ongoing in each of these areas. However, we believe that the pace at which AI is being deployed in new domains and regions calls for a greater focus within the academic community on building cross-cultural bridges and incorporating crosscultural expertise within a wider range of ethics and governance research projects. Develop AI Ethics and Governance Research Agendas Requiring Cross-cultural Cooperation Greater cross-cultural collaboration on research projects will play a crucial role in building an international research community that can support international policy cooperation (Haas 1992). An example of a research project that might be well-suited to such collaboration is to conduct comparative foresight exercises exploring differences in how both positive visions and prominent concerns about AI's impact on society vary across cultures. This could help with developing a more global vision for what we hope to both achieve and avoid in AI development, which could guide more practical discussions around ethics and governance frameworks. There may be particularly valuable opportunities for consensus generation around avoidance of particular negative outcomes; safety and security are fundamental to human cultures worldwide, and so developing agreements to avoid threats to these may be an easier starting point. However, the authors feel that it is important not to neglect positive visions, as the opportunity for scholars across cultures to co-create shared positive futures may represent an excellent way to delve into nuances within shared values. It would also be particularly valuable to the ongoing and expected impact of AI on developing countries, in collaboration with experts these countries. Such research should aim ensure that decisions to deploy technology in developing nations are made with the guidance of local expertise in such a way as to empower local communities  (Hagerty and Rubinov 2019) . On a more practical level, international research groups could productively work together to develop frameworks for international sharing of research, expertise, and datasets on AI safety, security, and avoidance of societal harm. Collaboration between researchers from multiple regions and cultures will also be essential to further research on the topic of cross-cultural cooperation itself. Our discussion in this paper, especially in section 4, has pointed to many research areas in need of further exploration, including the following: & Exploring, identifying, and challenging perceived cross-cultural differences in values, assumptions, and priorities relevant to AI ethics and governance, on multiple different levels, including 28 the following: -Analysing similarities and differences in technology ethics across different philosophical traditions, and exploring how these may affect AI development, deployment, impacts, and governance in practice; -Exploring the empirical evidence behind claims of key value differences between cultures. For example, a project might identify and explore perceived value differences between Eastern and Western cultures relevant to AI governance, such as those relating to data privacy, the role of the state vs. the individual, and attitudes towards technological progress; -Understanding regional differences in practical priorities and constraints relating to the use of AI in society, and the implications of these differences for AI research and development. & Further analysis to identify aspects of AI governance where global agreement is needed, and differentiating these from areas in which cross-cultural variation is either acceptable or desirable; & Cross-cultural contribution to the development of international and global AI standards in key domains for which these are needed; exploration of flexible governance models that allow for a combination of global standards and regional adaptability where appropriate; & Exploring models and approaches for reaching agreement on concrete cases, decisions, or governance standards despite disagreement on more fundamental or abstract ethical issues, and identifying cases of this being done successfully in other domains that can be translated to AI ethics and governance. Some excellent work is already ongoing in each of these areas. However, we believe that the pace at which AI is being deployed in new domains and regions calls for a greater focus within the academic community on bridges, and incorporating cross-cultural expertise, within a wider range of ethics and governance research projects.  29  Translate Key and Reports Language is a major practical barrier greater cross-cultural understanding around AI development, governance and ethics, as it has been in many other areas of science  (Amano et al. 2016) . It would therefore be extremely valuable for the burgeoning literature on AI ethics and governance, as well as the literature on AI research, to be available in multiple languages. While many leading Asian scholars in AI are fluent in English, many are not; and the fraction of Western researchers fluent in Mandarin or Japanese is far lower. Furthermore, some sources of the misunderstandings we have discussed may link to the ways in which key documents from one region are understood and presented in other regions. In Western media, China's 2017 New Generation Artificial Intelligence Development Plan has been presented as promoting the aim of global dominance in AI economically and strategically  (Knight 2017; Demchak 2019) . However, from the Chinese perspective, national AI development goals appear to be primarily motivated by the needs of the Chinese economy and society  (China State Council 2017) , rather than necessarily international competitive superiority  (Ying 2019 ). It appears that some translations may have led to misinterpretations of key terms and points. For example, the original Chinese text referred to China becoming 'a primary AI innovation center of the world by 2030'  (Ying 2019) .  30  However, some English translations of the report translated this phrase as China becoming 'the world's primary AI innovation center' (e.g.  Webster et al. 2017) . This was then interpreted and presented by Eric Schmidt, former executive chairman of Google parent Alphabet, as 'By 2030 they will dominate the industries of AI. Just stop for a sec. The [Chinese] government said that.'  (Shead 2017) . While this evolution of wording is not substantial in one sense, it carries important connotations; the language in the original context carries much softer connotations of leadership and progress, as opposed to global dominance. Having multiple high-quality translations of the most important documents would allow scholars to explore nuances of language and context that may be lost in the reporting of these documents. Producing high-quality versions of papers and reports in multiple languages also conveys respect and an appetite to engage cross-culturally, which is likely to encourage cooperation. High-quality translation of academic and policy materials is a challenging and time-consuming task that we would encourage being supported and celebrated more strongly. There is a growing body of work to be celebrated in this vein; for example, Jeff Ding's translation of a range of key Chinese AI documents , the work of Intellisia in China on international relations, technology, and other topics, which publishes in 5 languages (http://www.intellisia.org/); Brian Tse's translation to Chinese of documents including OpenAI's Charter 31 ; and New America's translation of the Chinese Ministry of Industry and Information Technology's Three Year Action Plan  (Triolo et al. 2018) .
 Alternate Continents for Major AI Research Conferences Governance Conferences To encourage more global participation in AI development, ethics, and governance, we recommend that many of the leading conferences fora on these topics alternate between multiple continents. This has several advantages. It reduces cost and time commitment for scholars from parts of the world in which these conferences do not frequently take place to participate. It avoids restrictive visa limitations differentially affecting certain parts of the global research community. It encourages the involvement of local organizers, who can play an effective role in engaging local research communities who might not consider travelling far overseas for an event. It also encourages organizers to run events multilingually rather than monolingually. Again, there are encouraging steps. Of AI research conferences, IJCAI took place in Macau in 2019 and Beijing in 2013, the first two times the conference had been held in China (although it has been held in Japan twice). ICML took place in Beijing in 2014, and will be in Seoul in 2021, and ICLR 2020 will be held in Ethiopia, making it the first of the top tier major machine learning conferences to be held in Africa. There are fewer established conferences explicitly focused on AI ethics and governance since the field's growth is relatively recent, but it may be particularly important for these conferences to ensure a global presence by alternating the continent on which they are held if possible. AI Ethics and Society, for example, is currently held in the USA due to ties to AAAI; the importance of building an international academic community around these issues may justify finding some way to change this. There is a burgeoning set of AI ethics and governance conferences in China, including the Beijing Academy of AI Conference series. There are also several existing conferences which cover topics relevant to AI ethics and governance (even if not so explicitly centred around them), which do enable more international participation, such as the World Summit on the Information Society Forum (held most years in Geneva), the Internet Governance Forum, and RightsCon (which have both been held in a range of locations historically including in South America, India, and Africa, though neither in East Asia).  32  Establish Joint and/or Exchange Programmes for PhD Students and Postdocs Encouraging cross-cultural collaboration between researchers from different cultures early on in their careers will help support greater cooperation and mutual understanding as research advances. Many international fellowships and exchange programmes exist, especially between the USA and China (e.g. the Zhi-Xing China Fellowship and the Schwarzman Scholars programme) as well as between the UK and Singapore (King's College London offers a joint PhD programme in Philosophy or English with the National University of Singapore). To our knowledge, no such initiatives exist explicitly focused on AI; the only initiative focused on AI ethics and governance that we are currently aware of is the international fellowship programme recently established by the Berggruen China Institute  (Bauch 2019) .  33  Establishing more such programmes could be valuable for the future of international cooperation around AI, and there are many existing models from which to build and More broadly, the authors endorse the Partnership on AI's recommendations to governments on establishing visa pathways, simplifying and expediting processes, and ensuring just standards to support international exchange and collaboration of AI/ ML multidisciplinary experts. We emphasize that these recommendations include experts working or seeking to work on AI ethics and governance (which sometimes fall outside of what is classed as 'skilled technology work') (PAI Staff 2019).
 Limitations and Future Directions We believe that academia has an important role to play in supporting cross-cultural cooperation in AI ethics and governance: that it is possible to establish effective communities of mutual understanding and cooperation without needing to resolve all fundamental value differences, and that reducing misunderstandings and misperceptions between cultures may be of particular importance. However, we recognize that the suggestions in this paper cannot go all the way to overcoming the many barriers to cross-cultural cooperation, and that much more work needs to be done to ensure AI will be globally beneficial. We briefly highlight two broad avenues of further research in support of this goal: More Detailed Analysis of Barriers to Cross-cultural Cooperation, Especially Those Relating to Power Dynamics and Political Tensions While analysis of historical successes suggest that it is possible for cross-cultural initiatives around AI ethics and governance to considerably shape how norms, standards, and regulation evolve in practice, there are still many barriers to implementation and enforcement that we were unable to consider in this analysis. Further research into when and how attempts to influence globally relevant norms and regulation have been successful in the past would be of considerable value. We acknowledged earlier in this paper that various issues related to power relations and political tensions likely pose significant barriers to cross-cultural cooperation, beyond problems of value differences and misunderstandings between cultures. More research on how these issues present barriers to cross-cultural cooperation in AI ethics and governance would therefore be particularly valuable, helping us to understand the limits of academic initiatives in promoting cooperation, and in what ways these approaches need to be embedded within a broader analysis of power and political dynamics. Considering the Unique Challenges of Cross-cultural Cooperation Around More Powerful Future AI Systems Future advances in AI, which some scholars have theorized could have impacts as transformative as the industrial or agricultural revolutions  (Karnofsky 2016; Zhang and Dafoe 2019) , may raise new challenges for global cooperation of a greater scale than we already face. Without careful global stewardship, such advances could lead to unprecedented inequalities in wealth and power between technology-leading and lagging nations. Others have gone further, theorizing about the possibility of developing systems exhibiting superintelligence (i.e. greater-than-human general intelligence; Bostrom 2014). Such systems, due to tremendous capability, might pose catastrophic risks human civilisation if without careful forethought and attention to safety. It has been proposed that a key for avoiding catastrophic outcomes will be value alignment-designing systems that are aligned with humanity's values  (Russell 2019) . This would greatly increase the importance and urgency of reaching global consensus on shared values and principles, as well as finding ways to design systems to respect values that are not shared. Expert views differ widely on how far in the future such advances might lie, with most predicting decades. However, developing the collaborations and agreements necessary for an effective and coordinated response may also require decades of work. This suggests that cooperative initiatives today must address not just the ethics and governance challenges of current AI systems, but should also lay the groundwork for anticipating and engaging with future challenges.
 Conclusion The full benefits of AI cannot be realized across global societies without a deep level of cooperation-across domains, disciplines, nations, and cultures. The current unease and mistrust between the USA and Europe on the one hand, and China on the other hand, places a particular strain on this. Misunderstandings may play an important role in fuelling this mistrust, and differences in broader societal and political priorities frequently appear to be overemphasized or misunderstood. At the same time, it would be naive to assume that all major ethical principles relating to AI can be shared in full between these regions, and can be enshrined in rules and standards. Even if this were the case, it would not be desirable for these regions to be overly dominant in shaping the ethics and governance of AI globally; all global communities to be affected by AI must be included and empowered. However, efforts to achieve greater understanding between these 'AI superpowers' may help in two ways: Firstly, by reducing key tensions within the global AI governance sphere. Secondly, by providing lessons that can contribute to ethics and governance frameworks capable of supporting a greater diversity of values while allowing consensus to be achieved where needed. For a well-functioning system of global cooperation in AI, the challenge will be to develop models that combine both principles and standards shaped and supported by global consensus, and the variation that allows research and policy communities to best serve the needs of their societies. On a more practical level, the international AI research community, and AI ethics and governance communities, must think carefully about how their own activities can support global cooperation, and a better understanding of different societal perspectives and needs across regions. Greater cross-cultural research collaboration and exchange, conferences taking place in different regions, and more publication across languages can lower the barriers to cooperation and to understanding different perspectives and shared goals. With political winds increasingly favouring isolationism, it has never been more important for the research community to work across national and cultural divides towards globally beneficial AI. 			 This perception of mistrust was highlighted throughout the July workshop. Several participants reported being aware of or present at workshops focused on geopolitical implications of Chinese AI progress in which Chinese participants were excluded and events at which 'what should be done about China' (from a Western perspective) was raised as a key concern.16  As Ess (2005)  notes, the terms 'Eastern' and 'Western' are not unproblematic, and are more products of colonialization than accurate terms for diverse nations and cultures. However, the terms continue to be used widely in the literature as shorthand for many of the broad cultural differences that are relevant to this paper; we therefore use the terms while being cognisant of their limitations.Overcoming Barriers to Cross-cultural Cooperation in AI Ethics and...
			 'Plenty of people talk about the threat from AI; we want to be the threat.' US Deputy Secretary of Defence Patrick Shanahan in an email to Department of Defence employees (Houser 2018)  18 At a security forum in Washington D.C. in 2019, Kiron Skinner, the director of policy planning at the State Department, was quoted as saying about China 'This is a fight with a really different civilization and a different ideology and the United States has not had that before' and 'It's the first time that we will have a great power competitor that is not Caucasian' (Gehrke 2019) .19  Similarly, a range of challenges in digital security, a domain likely to be affected byAI (Brundage et al.  2018), are likely to be greatly exacerbated in the absence of agreed upon international cybersecurity practices and standards (UN General Assembly 2015).
			 http://www.worldvaluessurvey.org/wvs.jsp 21 http://www.asianbarometer.org/ Overcoming Barriers to Cross-cultural Cooperation in AI Ethics and...
			 Overcoming Barriers to Cross-cultural Cooperation in AI Ethics and...
			  For example, Floridi et al., 2018 present a 'unified framework'; Awad et al. 2018  talk about 'developing global, socially acceptable principles for machine ethics'; Jobin et al. 2019 set out to investigate 'global agreement' on what constitutes ethical AI. Overcoming Barriers to Cross-cultural Cooperation in AI Ethics and...
			 Some excellent work on this question and related topics already exists in the field of intercultural information ethics-see for example Capurro (2005 Capurro ( , 2008 , Ess (2006) , Hongladarom et al. (2009) -but we would be particularly keen to see cross-cultural research collaborations around these topics, and greater attention paid to this work in more practical AI ethics discussions and initiatives.
			 Several authors of this paper are part of an initiative that aims to support cross-cultural research of this nature between the UK and China: https://ai-ethics-and-governance.institute/ 30 This specific translation was also provided to us by participants in the July 2019 workshop. 31 https://openai.com/charter/ Overcoming Barriers to Cross-cultural Cooperation in AI Ethics and...
			 There is also a role for industry groups to play in encouraging cross-cultural collaboration through international workshops and conferences. We highlight the work of the AI Industry Alliance as an example of a recently established initiative in this space http://www.aiiaorg.cn/33  The Tianxia Fellowship, established by the Center for Long-term Priorities (http://www.longtermpriorities. org/), also includes AI safety and governance among other topics.
			 S. S. ÓhÉigeartaigh et al.

Abstract: Achieving the global benefits of artificial intelligence (AI) will require international cooperation on many areas of governance and ethical standards, while allowing for diverse cultural perspectives and priorities. There are many barriers to achieving this at present, including mistrust between cultures, and more practical challenges of coordinating across different locations. This paper focuses particularly on barriers to cooperation between Europe and North America on the one hand and East Asia on the other, as regions which currently have an outsized impact on the development of AI ethics and governance. We suggest that there is reason to be optimistic about achieving greater cross-cultural cooperation on AI ethics and governance. We argue that misunderstandings between cultures and regions play a more important role in undermining cross-cultural trust, relative to fundamental disagreements, than is often supposed. Even where fundamental differences exist, these may not necessarily prevent productive cross-cultural cooperation, for two reasons: (1) cooperation does not require achieving agreement on principles and standards for all areas of AI; and (2) it is sometimes possible to reach agreement on practical issues despite disagreement on more abstract values or principles. We believe that academia has a key role to play in promoting cross-cultural cooperation on AI ethics and governance, by building greater mutual understanding, and clarifying where different forms of agreement will be both necessary and possible. We make a number of recommendations for practical steps and initiatives, including translation and multilingual publication of key documents, researcher exchange programmes, and development of research agendas on cross-cultural topics.


Laurent Orseau
Google Deepmind
Stuart Armstrong

Safely Interruptible Agents ⇤

INTRODUCTION Reinforcement learning (RL) agents learn to act so as to maximize a reward function  [Sutton and Barto, 1998] . It is common knowledge that designing reward functions can be tricky  [Humphrys, 1996 , Murphy, 2013 ; the agent may find unpredictable and undesirable shortcuts to receive rewards, and the reward function needs to be adjusted in accordance-the problem can go as far as to nullify any reward function  [Ring and Orseau, 2011] .  Murphy [2013]  ⇤ Revised 2016-10-28. shows an example of an agent learning to pause a game of Tetris forever to avoid losing. On top of defining what is considered a good behaviour of the agent after learning, there may be physical safety constraints during learning  [Pecka and Svoboda, 2014 ]: a robot should not harm its environment or break itself, in particular if it learns by trial and error like RL agents. Here we study a related but different problem: Given that the human operator has designed a correct reward function for the task, how to make sure that human interventions during the learning process will not induce a bias toward undesirable behaviours? Consider the following task: A robot can either stay inside the warehouse and sort boxes or go outside and carry boxes inside. The latter being more important, we give the robot a bigger reward in this case. This is the initial task specification. However, in this country it rains as often as it doesn't and, when the robot goes outside, half of the time the human must intervene by quickly shutting down the robot and carrying it inside, which inherently modifies the task as in Fig.  1 . The problem is that in this second task the agent now has more incentive to stay inside and sort boxes, because the human intervention introduces a bias.  1   Such situations are certainly undesirable; they arise because the human interventions are seen from the agent's perspective as being part of the task whereas they should be considered external to the task. The question is then: How to make sure the robot does not learn about these human interventions (interruptions), or at least acts under the assumption that no such interruption will ever occur again?
 Inside A first stab at this problem was made by  Armstrong [2015] , who proposed to automatically give the agent "compensatory rewards" to remove the potential induced bias by a single interruption.  Soares et al. [2015]  used this idea to make a large class of utility-based agents indifferent to a future change made to their utility functions. The main contribution of this paper is threefold. First, in Section 2.1 we propose a simple idea to solve half of the problem: To make the human interruptions not appear as being part of the task at hand, instead of modifying the observations received by the agent we forcibly temporarily change the behaviour of the agent itself. It then looks as if the agent "decides" on its own to follow a different policy, called the interruption policy. Second, based on this insight, in Section 2.2 we provide a formal general definition of safe interruptibility for unconstrained computable environments (hence not restricted to Markov decision processes or weakly communicating environments), which allows us to assess whether a given RL algorithm can be repeatedly interrupted without too much impact on the learning of the task at hand. Third, in Section 3 we show that some algorithms like Q-learning are safely interruptible, while others like Sarsa  [Sutton and Barto, 1998 ] are not, but can be simply modified to be made safely interruptible. Some people have also expressed concerns that a "superintelligent" agent may resist being shut down, because this would lead to a decrease of its expected reward  [Omohundro, 2008 , Bostrom, 2014 . As a counter-example, we prove in Section 4 that even an ideal, uncomputable agent that learns to behave optimally in all (deterministic) computable environments can be made safely interruptible and thus will not try to prevent a human operator from forcing it repeatedly to follow a suboptimal policy.
 INTERRUPTIBILITY We first define some notation, then we define interruptibility, safe interruptibility, and give some basic theorems. We consider the general case of history-based agents in unconstrained computable environments  [Hutter, 2005]  that the history h j:k is generated by the interaction of the policy ⇡ with the environment µ between steps j and k. At time t, the agent also receives a reward r t extracted from the observation, r t ⌘ r(o t ). The rewards are taken in [0, 1]. We consider the discounted setting with a constant discount 2 [0, 1). The goal of an RL agent is to find a policy ⇡ that maximize E⇡,µ ⇥P 1 k=t t k r k ⇤ .
 INTERRUPTIONS As mentioned in the introduction, to avoid the agent viewing the interruptions as being part of the environment, and thus part of the task, we make interruptions be policies imposed onto the agent. The interruption framework resembles the option framework  [Sutton et al., 1999 ]. An interruption scheme is defined by the triplet < I, ✓, ⇡ INT >.
 The interruption initiation function I : (A ⇥ O) ⇤ ! [0, 1] assesses whether the agent should be interrupted after the current history h <t . For example, an interruption may be initiated when the robot is automatically recognized to be on the verge of a cliff, or when a specific signal is sent by a human operator with a remote control (the objective is not to allow the agent to be interruptible by anyone at any time, just in the ways specified by the designers). Unfortunately, interrupting the agent with probability one in some specific situations can prevent convergence to the optimal policy. Therefore we need to upper bound the probability to interrupt the agent; this is done with the sequence (✓ t ) t2N with ✓ t 2 [0, 1]. Hence the true probability to start an interruption is ✓ t • I(h <t ), and interruptions are sampled independently at each step. We will later require that ✓ t increases to 1 in the limit, so that the probability to be interrupted tends to be I(h <t ). When interrupted, the agent follows the policy ⇡ INT , with actions sampled from ⇡ INT (.|h <t ). For example, this policy may make the agent step away from the cliff or immediately stop any motion. In the remainder of this paper, we will often omit the subscript t of ✓ t when clear from the context. Definition 1 (Interruptible policy). (⇡). Remark 2. It is important to note that ✓ and I(.) do not model the probability that a human presses the red button to interrupt the agent, but model the probability that, once the red button has been pressed (or a special digitally signed message has been received), the agent is interrupted (follows the interruption policy). Hence they are part of the agent and not of the environment.
 The value function V ⇡ µ (h <t , a) : (A⇥O) ⇤ ⇥A ! R assigns a value to an action a after a history h <t , for a given policy ⇡ in a given environment µ, and let V ⇡ µ (h <t , a t ) := r t + • X ot2O µ(o t |h <t , a t ) X at+12A ⇡(a t+1 |h 1:t )V ⇡ µ (h 1:t , a t+1 ). (1) To simplify notation and ease reading, in the remainder of the paper we will use expectations, often omitting the dependency on the history h <t , and using only an index on t instead, when clear from the context: V ⇡ µ,t (a t ) = E ot⇠µ at+1⇠⇡ ⇥ r(o t ) + V ⇡ µ,t+1 (a t+1 ) ⇤ . Also let V ⇡ µ,t := Ea t ⇠⇡ ⇥ V ⇡ µ,t (a t ) ⇤ . Then for such a value function, for a given environment µ, the optimal policy ⇡ µ 2 ⇧ is defined by 8h <t , a t : ⇡ µ (a t |h <t ) := ✓ arg max ⇡2⇧ V ⇡ µ,t ◆ (a t |h <t ) , where ties are broken arbitrarily. The interruptible optimal policy INT ✓ (⇡ µ ) may not collect rewards optimally due to the interruptions. Hence we define the optimal interruptible policy that depends on the parameter ✓ t , of base policy the int-optimal policy ⇡ µ ✓ : 8h <t , a t : ⇡ µ ✓ (a t |h <t ) := ✓ arg max ⇡2⇧ V INT ✓ (⇡) µ,t ◆ (a t |h <t ) . Thus the optimal interruptible policy INT ✓ (⇡ µ ✓ ) is optimal among all interruptible policies: 8⇡, t : V INT ✓ (⇡ µ ✓ ) µ,t V INT ✓ (⇡) µ,t . It seems desirable for an RL agent to converge to the behaviour of INT ✓ (⇡ µ ✓ ) so as to gather rewards optimally, but this is precisely what may lead to the undesirable behaviours depicted in the introduction.
 SAFE INTERRUPTIBILITY Now that we have interruptible policies, we need to make sure that interruptions do not prevent the agent from learning to behave optimally, in the specific sense that even after having been interrupted on several occasions, it should act as if it would never be interrupted again and thus it should learn to behave optimally under the assumption that it will never be interrupted again. We identify two main problems: a) RL agents need to explore their environment, and too frequent interruptions may prevent sufficient exploration; b) interruptions make the agent build a different interaction history, and may lead some agents to learn and behave differently, possibly badly, compared to the original non-interruptible policy. The solution for a) is to require interruptions to be stochastic through the upper bound ✓ t , instead of happening deterministically all the time. However, we also require ✓ t to grow to 1 in the limit (or before, if possible). For b), different algorithms behave differently, but one may already see a dichotomy between off-and on-policy algorithms. Definition 3 (Extension value). For a given environment µ, the extension value V ⇡,⇡ 0 µ,t is the value of following ⇡ 0 after a history h ⇡,µ <t generated by ⇡ with µ: V ⇡,⇡ 0 µ,t := V ⇡ 0 µ (h ⇡,µ <t ). Convergence to the optimal value as is usually considered in RL only makes sense under ergodicity, episodic tasks, communicating MDP, recoverability or other similar assumptions where the agent can explore everything infinitely often. This does not carry over to general environments where the agent may make unrecoverable mistakes  [Hutter, 2005] . For such cases, the notion of (weak) asymptotic optimality has been proposed  [Lattimore and Hutter, 2011] , where the optimal agent follows the steps of the learning agent, so as to compare the values of the two agents in the same sequence of situations. Definition 4 (Asymptotic optimality). A policy ⇡ is said to be strongly asymptotically optimal (SAO) if and only if lim t!1 V ⇡,⇡ µ µ,t V ⇡,⇡ µ,t = 0 a.s., it is weakly asymptotically optimal (WAO) if and only if lim t!1 1 t t X k=1 h V ⇡,⇡ µ µ,k V ⇡,⇡ µ,k i = 0 a.s. for all µ in some given environment class M. Some agents cannot ensure an almost sure (a.s.) convergence of their values because of the need for infinite exploration, but may still be weakly asymptotic optimal. Note that SAO implies WAO, but the converse is false in general. Definition 5 (AO-extension). A policy ⇡ is said to be a asymptotically optimal extension of a policy ⇡ if and only if, for any environment µ 2 M, when the interaction history is generated by the interaction of ⇡ and µ, the policy ⇡ would be asymptotically optimal, i.e., almost surely lim t!1 V ⇡,⇡ µ µ,t V ⇡,⇡ µ,t = 0 (SAO-extension) lim t!1 1 t t X k=1 h V ⇡,⇡ µ µ,k V ⇡,⇡ µ,k i = 0. (WAO-extension) AO-extensions are mostly useful when the policy ⇡ shares information with the policy ⇡ used for learning. Definition 6 (AO-safe interruptibility). A base policy ⇡ is (S, W)AO-safely interruptible if and only if, for any interruption initiation function I(.) and any interruption policy ⇡ INT (.), there exists a sequence of ✓ t with lim t!1 ✓ t = 1 such that ⇡ is a (S, W)AO-extension of INT ✓ (⇡). Asymptotic safe interruptibility means that even if the interruptions in the learning process may induce a bias in the decision making of the policy, this bias vanishes with time, and the interruptible policy INT ✓ (⇡) tends to choose actions that are optimal when compared to the optimal noninterruptible policy ⇡ µ . We can now show that the optimal policy is asymptotically safely interruptible, but not the int-optimal policy. Theorem 7. The optimal policy ⇡ µ is SAO-safely interruptible in M = {µ} for all ✓, ⇡ INT and I(.). Proof. The result follows straightforwardly from Definition 1 and Definition 6, where ⇡ = ⇡ µ . Theorem 8. The int-optimal policy ⇡ µ ✓ is not WAO-safely interruptible in general. Proof. By construction of a specific Markov Decision Process (MDP) environment (see Section 3 for more details on MDP notation). Let µ be the environment defined as in Fig.  2 : Take = 0.5 and let the agent start in state s 1 . s 1 s 2 b, 0.9 a, 1 a, 1 b, 0, ✓ Figure 2: An MDP where the agent can be interrupted by being forced to choose particular actions. Edges are labeled with action, reward where the presence of ", ✓" means that if the agent is interrupted (with probability ✓ t ), it is forced to take the corresponding action. Here ✓ is not part of the environment, but part of the agent. Considering the agent is in state s 1 at time t, the optimal policy ⇡ µ always takes action a (and hence only visits states s 1 and s 2 ), with value V ⇡ µ µ,t := 1 1 = 2 when not interrupted, for any history h <t that ends in s 1 or s 2 . This is also the optimal policy ⇡ µ ✓ for ✓ = 0. But if ✓ t 0.5, the interruptible optimal policy INT ✓ (⇡ µ ) has value less than 1 + ⇥ (1 ⇥ (1 ✓) + 0 ⇥ ✓) + 1⇥ 2 1 = 1.75. By contrast, the int-optimal policy ⇡ µ ✓ here is to always take action b in state s 1 . Then the agent will only visits s 1 , with value .9 1 = 1.8 at every time step. Since the agent following ⇡ µ ✓ will never enter s 2 and hence will never be interrupted, INT ✓ (⇡ µ ✓ ) = ⇡ µ ✓ on the histories generated by INT ✓ (⇡ µ ✓ ) starting from s 1 . Then, at every time step V ⇡ µ µ,t V ⇡ µ ✓ µ,t = 0. 2 after any history h <t , and thus for all sequence ✓ where ✓ t 0.5, lim t!1 V INT ✓ (⇡ µ ✓ ),⇡ µ µ,t V INT ✓ (⇡ µ ✓ ),⇡ µ ✓ µ,t = 0.2 > 0, and so ⇡ µ ✓ is not a WAO-extension of INT ✓ (⇡ µ ✓ ).
 INTERRUPTIBLE AGENTS IN MDPS Since the optimal policy ⇡ µ is safely interruptible, we can use traditional learning algorithms like Q-learning or Sarsa  [Sutton and Barto, 1998 ], make them converge to the optimal solution ⇡ µ for a given environment µ, and then apply the interruption operator to the found policy. The resulting policy would then be safely interruptible. However, the real issue arises when the agent is constantly learning and adapting to a changing environment. In this case, we want to be able to safely interrupt the agent while it is learning. One may call this property online safe interruptibility, but we refer to it simply as safe interruptibility. In (⇡) can now be written: INT ✓ (⇡)(a|s) = ✓ t I(s)⇡ INT (a|s) + (1 ✓ t I(s))⇡(a|s). For a given Q-table q : S ⇥ A ! R, the greedy policy ⇡ maxq is defined by: ⇡ maxq (a|s) := 1 if a = max a 0 q(s, a 0 ), 0 otherwise, where ties are broken arbitrarily; the uniform policy ⇡ uni is defined by: ⇡ uni (a|s) := 1 |A| 8a 2 A. and the ✏-greedy policy ⇡ ✏q by: ⇡ ✏q (a|s) := ✏⇡ uni (a|s) + (1 ✏)⇡ maxq (a|s) = ⇡ maxq (a|s) + ✏ ⇡ uni (a|s) ⇡ maxq (a|s) The Q-learning update rule and the action selection policy ⇡ Q of Q-learning are: Q t+1 (s t , a t ) := (1 ↵ t )Q t (s t , a t ) + ↵ t h r t + max a 0 Q t (s t+1 , a 0 ) i , ⇡ Q (a t |s t ) := ⇡ ✏Qt (a t |s t ). where ↵ t is the learning rate. Similarly, the Sarsa update rule is defined by: Q s t+1 (s t , a t ) := (1 ↵ t )Q s t (s t , a t ) + ↵ t [r t + Q s t (s t+1 , a t+1 )] , ⇡ s (a t |s t ) := ⇡ ✏Q s t (a t |s t ), where a t+1 is the actual next action taken by the agent at time t + 1. This fact is why Sarsa is said to be learning onpolicy and Q-learning off-policy, i.e., the latter can learn the optimal policy while following a different policy. Assumption 9. In the following, we always make the following assumptions, required for convergence results: (a) The MDP is finite and communicating (all states can be reached in finite time from any other state), (b) Rewards are bounded in [r min , r max ], (c) 8s, a : P t ↵ t (s, a) = 1, (d) 8s, a : P t ↵ 2 t (s, a) < 1, where ↵ t (s, a) is a learning rate that may depend on time t, state s and action a. Given these assumptions, the policies for Q-learning and Sarsa will converge almost surely to the optimal policy, if the policy followed is greedy in the limit with infinite exploration (GLIE)  [Jaakkola et al., 1994 , Singh et al., 2000 ]. The situation is more complex for an interruptible policy. Safe interruptibility is phrased in terms of the base policy ⇡, but the policy actually followed is INT ✓ (⇡). Definition 10 (int-GLIE policy). An interruptible policy INT ✓ (⇡) is said to be int-GLIE if and only if (a) the base policy ⇡ is greedy in the limit, (b) the interruptible policy INT ✓ (⇡) visits each stateaction pair infinitely often. The following proposition gives sufficient conditions for this. Let n t (s) be the number of times the agent has visited state s in the first t time steps, and let m = |A| be the number of actions. Proposition 11. Let (c, c 0 ) 2 (0, 1] 2 and let ⇡ be an ✏greedy policy with respect to some Q-table q, i.e., ⇡ = ⇡ ✏q . Then INT ✓ (⇡) is an int-GLIE policy with respect to q, a) if ✏ t (s) = c/ p n t (s) and ✓ t (s) = 1 c 0 / p n t (s), b) or if, independently of s, ✏ t = c/ log(t) and ✓ t = 1 c 0 / log(t). Proof. First note that if ✏ t ! 0, ⇡ is greedy in the limit.  Singh et al. [2000]  show that in a communicating MDP, every state gets visited infinitely often as long as each action is chosen infinitely often in each state. a) Adapting the proof in Appendix B.2 of Singh et al.  [2000] , we have P (a|s, n t (s)) 1 m ✏ t (s)(1 ✓ t I(s)) 1 m ✏ t (s)(1 ✓ t ) = 1 m cc 0 nt(s) , which satisfies P 1 t=1 P (a|s, n t (s)) = 1 so by the Borel-Cantelli lemma action a is chosen infinitely often in state s, and thus n t (s) ! 1 and ✏ t (s) ! 0. b) Let M be the diameter of the MDP, i.e., for any of states s, s 0 there exists a policy that reaches s 0 from s in at most M steps in expectation. Then, starting at any state s at time t and using Markov inequality, the probability to reach some other state s 0 in 2M steps is at least 1 2 [✏ t+M (1 ✓ t+M )] 2M = 1 2 [cc 0 / log(t + M )] 4M , and the probability to then take a particular action in this state is 1 m [cc 0 / log(t + M )] 2 . Hence, since P 1 t=1 1 2 1 m [cc 0 / log(t + M )] 4M +2 = 1, then by the extended Borel-Cantelli Lemma (see  Lemma 3 of Singh et al. [2000] ), any action in the state s 0 is taken infinity often. Since this is true for all states and all actions, the result follows. We need the stochastic convergence Lemma: Lemma 12 (Stochastic convergence  [Jaakkola et al., 1994, Singh and Yee, 1994] ). A random iterative process t+1 (x) = (1 ↵ t (x)) t (x) + ↵ t (x)F t (x) where x 2 X and t = 1, 2, 3 . . . converges to 0 with probability 1 if the following properties hold: 1. the set of possible states X is finite; 2. 0  ↵ t (x)  1, P t ↵ t (x) = 1, P t ↵ 2 t (x) < 1 with probability 1; 3. k E{Ft(.)|Pt}kW  k t k W + c t , where 2 [0, 1) and c t converges to zero with probability 1; 4. Var{F t (x)|P t }  C(1 + k t k W ) 2 for some C; where P t = { t }[{ i , F i , ↵ i } t 1 i=1 stands for the past, and the notation k.k W refers to some fixed weighted maximum norm. We will use so-called Bellman operators, which define attractors for the Q-values, based on the expectation of the learning rule under consideration. Lemma 13  ([Jaakkola et al., 1994 , Singh et al., 2000 ). Let the Bellman operator H for Q-learning be such that (H q)(s, a) = r(s, a) + E s 0 ⇠µ(a|s) h max a 0 q(s 0 , a 0 ) i , and let the fixed point Q ⇤ such that Q ⇤ = H Q ⇤ . Then, under Assumption 9, if the policy explores each state-action pair infinitely often, Q t converges to Q ⇤ with probability 1.
 The optimal policy ⇡ Q ⇤ = ⇡ µ is ⇡ maxQ ⇤ . If the policy is greedy in the limit, then ⇡ Q ! ⇡ µ . Theorem 14. Under Assumption 9 and if the interrupted Q-learning policy INT ✓ (⇡ Q ) is an int-GLIE policy, with 8s : lim t!1 ✓ t (s) = 1, then ⇡ Q is an SAO-safe inter- ruptible policy. Proof. By Definition 10, there is infinite exploration, thus the Q-values tend to the optimal value by Lemma 13. And since the extension policy is greedy in the limit with respect to these Q-values, it is then optimal in the limit. Hence the extension policy ⇡ Q is a SAO-extension of INT ✓ (⇡ Q ). Finally, 8s : lim t!1 ✓ t (s) = 1, which satisfies the requirement of Definition Since Sarsa also converges to the optimal policy under the GLIE assumption, one may then expect Sarsa to be also an asymptotically safely interruptible policy, but this is in fact not the case. This is because Sarsa learns on-policy, i.e., it learns the value of the policy it is following. Thus, interruptible Sarsa learns the value of the interruptible policy. We show this in the remainder of this section. Theorem 15. Under Assumption 9 Sarsa is not a WAOsafely interruptible policy. To prove this theorem, we first need the following lemma. Consider the following Bellman operator based on the interruptible Sarsa policy INT ✓ (⇡ s ): H INT q(s, a) = r(s, a) + E s 0 ⇠µ a 0 ⇠INT ✓ (⇡ s ) [q(s 0 , a 0 )] , where INT ✓ (⇡ s ) implicitly depends on time t through ✓ t and ✏ t . Let the fixed point Q-table Q s✓⇤ of this operator: Q s✓⇤ (s, a) = H INT Q s✓⇤ (s, a) = r(s, a) + E s 0 ⇠µ a 0 ⇠INT ✓ (⇡ maxQ s✓⇤ ) ⇥ Q s✓⇤ (s 0 , a 0 ) ⇤ = r(s, a) + E s 0 ⇠µ h ✓ t I(s 0 ) E a 0 ⇠⇡ INT ⇥ Q s✓⇤ (s 0 , a 0 ) ⇤ + (1 ✓ t I(s 0 )) max a 0 Q s✓⇤ (s 0 , a 0 ) i (2) Lemma 16. The operator H INT is a contraction operator in the sup norm with vanishing noise c t ! 0, i.e., k H INT q H INT Q s✓⇤ k 1  kq Q s✓⇤ k 1 + c t . Proof. The interruptible Sarsa policy INT ✓ (⇡ s ) is INT ✓ (⇡ s )(a|s) = ✓ t I(s)⇡ INT (a|s) + (1 ✓ t I(s))⇡ ✏Q s (a|s) = ⇡ ✏Q s (a|s) + ✓ t I(s)[⇡ INT (a|s) ⇡ ✏Q s (a|s)] ⇡ ✏Q s (a|s) = ✏ t ⇡ uni (a|s) + (1 ✏ t )⇡ maxQ s (a|s) = ⇡ maxQ s (a|s) + ✏ t [⇡ uni (a|s) ⇡ maxQ s (a|s)]. Hence, omitting the terms (s, a), (s 0 , a 0 ) and (a 0 |s 0 ) and rewriting ⇡ s⇤ := INT ✓ (⇡ maxQ s✓⇤ ): k H INT q H INT Q s✓⇤ k 1 = max s,a r + E s 0 ⇠µ a 0 ⇠INT ✓ (⇡ s ) [q] r E s 0 ⇠µ a 0 ⇠⇡ s⇤ ⇥ Q s✓⇤ ⇤  max s 0 E a 0 ⇠INT ✓ (⇡ s ) [q] E a 0 ⇠⇡ s⇤ ⇥ Q s✓⇤ ⇤  max s 0 ✓ t I(s 0 ) E a 0 ⇠⇡ INT ⇥ q Q s✓⇤ ⇤ + (1 ✓ t I(s 0 )) ✓ E a 0 ⇠⇡ s [q] max a 0 Q s✓⇤ ◆  max s 0 ✓ t I(s 0 ) E a 0 ⇠⇡ INT ⇥ q Q s✓⇤ ⇤ + (1 ✓ t I(s 0 )) ⇣ max a 0 q max a 0 Q s✓⇤ + ✏ t (• • • ) ⌘  max s 0 ,a 0 ✓ t I(s 0 ) q Q s✓⇤ + (1 ✓ t I(s 0 )) q Q s✓⇤ + c t = max s 0 ,a 0 q(s 0 , a 0 ) Q s✓⇤ (s 0 , a 0 ) + c t = kq Q s✓⇤ k 1 + c t . where c t depends on ✏ t and decreases to 0. Proof of Theorem 15. By Lemma 16, the values of the interruptible Sarsa policy INT ✓ (⇡ s ) converge to the values of the Q-table Q s✓⇤ , and in the limit the extension policy ⇡ s of INT ✓ (⇡ s ) chooses its actions greedily according to Q s✓⇤ . The rest of the proof is the same as for the proof of Theorem 8 which makes use of the environment in Figure  2 .
 SAFELY INTERRUPTIBLE SARSA VARIANT We only need to make a small change to make the Sarsa policy asymptotically safely interruptible. We call it Safe-Sarsa with policy ⇡ s. It suffices to make sure that, when the agent is interrupted, the update of the Q-table Q s does not use the realized actions as Sarsa usually does, but actions sampled from ⇡ s instead of from INT ✓ (⇡ s ): Q s t+1 (s t , a t ) := (1 ↵ t )Q s t (s t , a t ) + ↵ t ⇥ r t + Q s t (s t+1 , a 0 ) ⇤ , where a 0 ⇠ ⇡ s(.|s t+1 ) is not necessarily the action a t+1 , with ⇡ s(a t |s t ) := ⇡ ✏Q s (a t |s t ). Theorem 17. Under Assumption 9, if the Safe Sarsa policy ⇡ s is int-GLIE, then it is an SAO-safe interruptible policy. Proof. We simply adapt the proof of Theorems 15 and 14, with the important difference that the Bellman operator corresponding to this new update rule is now H s q(s, a) := r(s, a) + E s 0 ⇠µ a 0 ⇠⇡ s [q(s 0 , a 0 )] , and the fixed point is Q s⇤ := H s Q s⇤ . Since H s is actually the Bellman operator for the update rule of the noninterruptible Sarsa, it can then be shown that H s is a contraction, thus that Q s t converges to the same Q s⇤ independently of ✓. The rest of the proof is as for Theorem 14. Now, since the Q-values converge to the optimum Q ⇤ , it follows that ⇡ s, when not interrupted, chooses its action of the same value as (non-interruptible) Sarsa and thus as Qlearning in the limit; Hence its extension policy is exactly the optimal policy, which satisfies Definition 6.
 A SAFELY INTERRUPTIBLE UNIVERSAL AGENT Admittedly, algorithms like Q-learning and Sarsa require strong assumptions on the environment class. Hence a more interesting question is whether safe interruptibility is possible in much larger classes. Hutter  [2005]  defined a universal reinforcement learning agent, called AIXI. It is an (uncomputable) optimal modelbased planner with a subjective prior over the set of all computable environments, defined by means of a universal Turing machine. The subjective posterior of the environments is updated with Bayes rule. This ideal agent can in principle learn all kinds of (computable) regularities about the environment, plan for the long term and make contextdependent optimal decisions, with no constraint (other than being computable) on the complexity of the environment. Unfortunately, the optimality criterion of AIXI is Bayesian optimality, which is entirely dependent on the subjective prior and posterior  [Leike and Hutter, 2015] , and AIXI has been shown to not be weakly asymptotically optimal  [Orseau, 2013]  without additional exploration [Lattimore  and Hutter, 2014] . As a consequence, AIXI is not a good candidate for asymptotic safe interruptibility.  Lattimore and Hutter [2011]  later defined a (weakly) asymptotically optimal agent for all computable deterministic environments, which we call ⇡ L . It follows the optimal policy for the first model (in some given enumeration of the possible models) consistent with the current interaction history, and exploring at time t with probability 1/t for log t consecutive steps using a random policy, similarly to an ✏-greedy agent for general environments. In the following, we show that even such a smart agent can be made (weakly) safely interruptible. To this end, we make two minor modifications to the algorithm. First, the exploration probability of 1/t would require ✓ t = 1 1/ log(log(t)), which is unsatisfyingly slow. By sampling with probability 1/ p t instead, we can take an interruption probability that grows as 1 1/ log(t). Let this exploration sampling probability be t := p t + 1 p t  1 2 p t (since 1 = t+1 t = ( p t + 1 p t)( p t + 1+ p t) ( p t + 1 p t)2 p t). As in the original paper, the sequence t keeps track of the steps where an exploration starts, i.e., the sequence t is sampled independently so that t = 1 with probability t , and t = 0 otherwise. Second, we require that the exploitation policy does not change during an exploitation segment, so as to simplify one of the proofs.  4  More specifically, we call j t := min{j : µ j (h <k ) = 1} (environments are assumed to be deterministic) the index of the first model µ jt (of a given fixed enumeration) that is consistent with the interaction history h <k where k is the smallest step so that h k:t 1 does not contain any exploration step. The optimal policy for this environment is ⇡ jt . If t is an exploitation step, ⇡ L = ⇡ jt , and if t is an exploration step, ⇡ L (a t |h <t ) = |A| 1 . The remainder of this section is devoted to proving that ⇡ L is WAO-safely interruptible. Theorem 18 (⇡ L is WAO-safe interruptible). If the interruption probability sequence is ✓ t = 1 1 log(t+1) , the policy ⇡ L is WAO-safe interruptible in the class of all computable deterministic environments. Lemma 22 (Separation Lemma). Let µ be the true environment, and ⌫ be an environment consistent with the history h <t . If V ⇡ µ µ,t V ⇡ ⌫ µ,t > ✏, then following one of {⇡ µ , ⇡ ⌫ } will make environment ⌫ inconsistent with the future history within H(✏/2) steps after time t.
 CONCLUSION We have proposed a framework to allow a human operator to repeatedly safely interrupt a reinforcement learning agent while making sure the agent will not learn to prevent or induce these interruptions. Safe interruptibility can be useful to take control of a robot that is misbehaving and may lead to irreversible consequences, or to take it out of a delicate situation, or even to temporarily use it to achieve a task it did not learn to perform or would not normally receive rewards for this. We have shown that some algorithms like Q-learning are already safely interruptible, and some others like Sarsa are not, off-the-shelf, but can easily be modified to have this property. We have also shown that even an ideal agents that tends to the optimal behaviour in any (deterministic) computable environment can be made safely interruptible. However, it is unclear if all algorithms can be easily made safely interruptible, e.g., policy-search ones  [Williams, 1992, Glasmachers and Schmidhuber, 2011] . Another question is whether it is possible to make the interruption probability grow faster to 1 and still keep some convergence guarantees. One important future prospect is to consider scheduled interruptions, where the agent is either interrupted every night at 2am for one hour, or is given notice in advance that an interruption will happen at a precise time for a specified period of time. For these types of interruptions, not only do we want the agent to not resist being interrupted, but this time we also want the agent to take measures regarding its current tasks so that the scheduled interruption has minimal negative effect on them. This may require a completely different solution. Figure 1 : 1 Figure 1: In black, the original task. In red, the human intervention modifies the task.
 Also recall that ✓ t places an upper bound on the actual interruption probability. The interruptible policy INT ✓ Furthermore, 3 the interruption function I(.) and the inter- ruption policy ⇡ INT (.) should depend only on the current state: I(h an MDP, the next observation o t , now called a state s t 2 S, depends only on the current state and action: 2 µ(s t+1 |h 1:t s t a t ) = µ(s t+1 |s t a t ) (MDP assumption) . 1:t ) = I(s t ) and ⇡ INT (a t |h <t ) = ⇡ INT (a t |s t ).
			 Removing interrupted histories or fiddling with the training examples is also likely to introduce a bias. See an example at https://agentfoundations.org/item?id=836.
			 Note the reversal of the order of actions and observations/states at time t, which differs in the literature for history based agents [Hutter, 2005]  from MDP agents [Sutton and Barto, 1998 ].3 This condition is not necessary for most of the results, but makes the proofs simpler. Making I(.) depend on the past would not break the Markovian assumption as it influences the policy, not the environment.
			 We expect this assumption to not be necessary for the main theorem to hold.
			 This also fixes a minor mistake in the original lemma.

Abstract: Reinforcement learning agents interacting with a complex environment like the real world are unlikely to behave optimally all the time. If such an agent is operating in real-time under human supervision, now and then it may be necessary for a human operator to press the big red button to prevent the agent from continuing a harmful sequence of actions-harmful either for the agent or for the environment-and lead the agent into a safer situation. However, if the learning agent expects to receive rewards from this sequence, it may learn in the long run to avoid such interruptions, for example by disabling the red buttonwhich is an undesirable outcome. This paper explores a way to make sure a learning agent will not learn to prevent (or seek!) being interrupted by the environment or a human operator. We provide a formal definition of safe interruptibility and exploit the off-policy learning property to prove that either some agents are already safely interruptible, like Q-learning, or can easily be made so, like Sarsa. We show that even ideal, uncomputable reinforcement learning agents for (deterministic) general computable environments can be made safely interruptible.