Who owns artificial intelligence? A preliminary analysis of corporate intellectual property strategies and why they matter

Table of Contents
 Introduction Artificial intelligence (AI) is increasingly a focal point of competition between leading firms, and between states. This paper focuses on a key, often under-examined component of the competitive strategies being employed by both corporate AI developers and national governments: the protection of their intellectual property. Intellectual property (IP) is a broad and flexible concept, referring to creations of the mind that are eligible for protection through law.  2  Today, companies are using a mix of patents, trade secrets, and open-source licensing agreements to protect their AI-related IP. Simultaneously, government patent offices, judiciaries, and national security apparatuses are deciding which aspects of AI should be patentable, and whether certain inventions should be restricted for military purposes. The IP policy choices that governments and corporations make can have profound implications for the development trajectory of a technology. For example, in the 1990s, the biotechnology industry was transformed after court decisions in the US enabled a broader range of biological compounds and processes to be protected by patent law. This development spurred additional private investment, but critically also allowed companies to claim ownership over what previously would have been considered basic academic research. This, in turn, encouraged higher levels of secrecy to protect valuable intellectual property.  3  More recently, one of the most prominent advances in biotechnology, the CRISPR gene editing mechanism (originally derived from a naturally occurring process in bacteria), has been subject to a protracted legal battle over overlapping patent claims in the US.  4  Changes in IP law and strategy may have a similarly large impact on the trajectory of AI development. What these impacts could be, however, have received little study. This paper aims to provide a preliminary analysis of the goals and strategies of corporations and governments focused on AI development, and what the implications of these strategies may be. First, we explain how corporate AI developers currently protect their AI-related intellectual property and why they choose the methods that they utilize. Second, we describe how governments use intellectual property law to pursue national goals related to AI. Finally, we describe three plausible scenarios for how IP strategies in AI may evolve. 2 World International Property Organization 3 World International Property Organization  4  Cohen, 2019
 Understanding Corporate AI Developer IP Strategies Corporate AI developers face two key decisions around how to protect their AI-related intellectual property: whether or not to patent AI techniques and systems, and whether to open-source models or keep them private as trade secrets. A prevalent strategy among top AI developers today involves accumulating patents while simultaneously sharing research with the open-source community.  5  For example, Microsoft holds the most number of machine learning patents in the US (see Figure  1 ), 6 but is also an active participant in the open-source community, 7 sharing source code for machine learning methods and under certain circumstances providing free licenses for their patents. Microsoft's strategy is not an anomaly. Amazon, Google, IBM, Facebook, Baidu, Tencent, and several other companies are prolific patent holders in AI (see Figures  1 and 2 ) while also open-sourcing substantial portions of their systems and sharing their work at academic conferences such as ICML and NeurIPS. 8  Notably, and perhaps unintuitively, some of the largest software patent holders in the world (Google, Amazon, and Facebook) signed an amicus brief to the Supreme Court in 2014 advocating that the court make it more challenging to patent abstract software patents, which includes a considerable percentage of more theoretical AI and ML patents: "Abstract software patents have become a plague on computer-related industries. The software industry developed and flourished without abstract patents before changes in the Federal Circuit's jurisprudence led to a flood of them. Far from promoting innovation, abstract software patents have impaired it by granting exclusive rights over high level ideas and thereby blocking others from undertaking the truly innovative task of developing specific applications." 9 These observations beg the question: why do so many of the top AI developers grow AI-related patent portfolios while simultaneously sharing their research at academic conferences, open-sourcing machine learning models, and advocating for the legal dissolution of many AI-related software patents? In this section we argue that these corporate IP strategies help to manage a variety of objectives that companies wish to pursue. AI developers apply for patents because of competitive pressures to do so. These same developers also often open-source AI models in order to build their reputation, attract talent, and incentivize customers to use paid products. AI developers can also use selective open-source licensing agreements as a hybrid strategy, enabling companies to participate in the open-source community while maintaining the legal threat of their patents. We discuss these incentives for both patenting and opensourcing in turn, along with limitations and drawbacks of each approach.
 Pressure to Patent Patents have several uses beyond simply enabling the patent holder to sue for patent infringement. In the case of big technology firms, there is a strong incentive to engage in "defensive patenting;" that is, patenting without the intention to offensively litigate for infringement, but rather to present a credible threat of counter-lawsuit to another company. As the "mutually assured destruction" analogy makes clear, patent litigation is extremely costly for all involved due to substantial legal fees and the stigma for investors of working with a company whose products are in legal purgatory. At the height of smartphone-related litigation in 2011, Apple and Google each spent more money on patent litigation (primarily in suits and countersuits against one another) than they did on research and development, a sum in the billions of dollars. 11 In that same year, Google spent $12 billion acquiring Motorola, which market analysts evaluated as being primarily for Motorola's substantial smartphone patent portfolio. 12 Perhaps if Google had acquired Motorola's patents before litigation began with Apple, the threat of a more substantial retaliation could have prevented the lawsuits. This defensive rationale is also the stated reason for Google's new AI patent filings in machine learning and neural networks. When asked about Google's new filings, spokespeople for Google and DeepMind stated that they "hold patents defensively, not with the intent to start fights with others." 13 This dynamic can also help explain why Google has advocated for narrowing the influence of software patents while simultaneously growing its own patent portfolio; while Google may prefer a world without expensive software patent litigation, its defensive patenting strategy is shaped by threats in the existing patent litigation regime. Beyond defensive patenting, corporate AI developers could also be incentivized to hold patents in order to gain leverage in other settings. For example, Google's patent sharing arrangement with the Chinese tech giant Tencent is paving the way for Google's entry into the Chinese market.  14  Google also allows other companies to enter into its "Open Patent Pledge" and make use of patents in a pool as long as they commit not to engage in patent litigation against  Google. 15  However, building and maintaining a large patent portfolio has its drawbacks. First, in the US, patent holders must pay substantial upkeep fees to keep their patents active; up to thousands of dollars a year depending on the size of the patent holding entity and age of the patent (see Table  1 ).  16  For large companies like Googlewhich has more than 50,000 active patents-these costs can be in the tens of millions of dollars. 17 Second, for companies that place a high premium on secrecy, the disclosure requirements and public nature of patent filings may be onerous. Finally, some in the AI research community are philosophically opposed to the idea of patenting AI concepts and techniques, particularly broad theoretical methods that are seen as mathematical truths rather than human inventions.  18  Companies that do choose to patent regardless may face pushback from those opposed, which may have flow-on effects on their ability to, for example, attract and retain research talent.
 Patent Fee Schedule (per patent) 19 Large Entity Fee Small Entity Fee Micro Entity Fee Patent Maintenance fee due 3.5 years after patent is issued $1,600 $800 $400 Patent Maintenance fee due at 7.5 years $3,600 $1,800 $900 Patent Maintenance fee due at 11.5 years $7,400 $3,700 $1,800
 Pressure to Open-Source Incentives for corporate open-sourcing are also complex, typically extending beyond an altruistic or philosophical belief in open science. For example, open-sourcing can be used to build a firm's reputation, generate goodwill among the research community, and encourage customers to use paid products. Apple's recent trend towards open-source and sharing more AI research demonstrates some of these incentives at work. Apple has a notorious culture of secrecy, with numerous internal mechanisms in place to prevent leaks and sequester information.  20  Apple has benefited from this culture of secrecy in consumer hardware design, a world where preventing leaks and copycat designs are critically important. However, this culture was also a barrier for Apple to recruit top ML researchers, who typically strongly value being able to publish and share their work at conferences. Notably, several of Apple's rival firms enabled researchers to do so.  21  In 2017, Apple changed its approach, launching a machine learning journal and enabling its researchers to publicly share findings at top ML conferences, including NeurIPS and ICML. Open-sourcing can also be used as a tool to generate more paying customers. For example, companies with substantial cloud computing businesses often offer free machine learning tools to encourage customers to design an application using the open-source tool. Then, these customers go on to pay for these computeintensive machine learning processes to be implemented on that same firm's cloud service. In some cases, this occurs through explicit lock-in. For example, Amazon's image recognition software "Rekognition" is only available on Amazon Web Services.  23  In other cases, companies aim to retain customers through brand loyalty; Google hopes that customers will use its ML open-source platform Governments around the world are grappling with how to best take advantage of the recent wave of advances in AI, with several nations releasing national plans on how their country intends to incentivize and capitalize on AI innovation.  30  These plans include ensuring that effective IP law regimes exist for AI and ML. This tends to break down into two objectives: making AI patentable, and regulating access to national security related IP. (For additional context on how national patent systems interact at the international level, see Box 1.)
 Box 1: The International Patent System and the World Intellectual Property Organization Patent systems are primarily domestic in nature rather than international. Each country has its own patent office and companies interested in seeking a patent for an invention must apply separately in all jurisdictions in which they wish to be awarded a patent.  31  A patent awarded in one country cannot be used to litigate infringement in another, though that patent does count as a form of "prior art" which can be used to prevent the award of a patent for that invention in another country. The World Intellectual Property Organization helps harmonize this process by assisting inventors to file their inventions in several jurisdictions at once. However, the decision to award a patent will ultimately fall to individual countries. While patent treaties such as "The Agreement on Trade-Related Aspects of Intellectual Property Rights" (TRIPS 32 ) have taken steps to harmonize intellectual property law across member nations, differences persist at every level of the process: what inventions are patentable, the level of scrutiny applied before a patent is granted, how patents are enforced and reviewed, the length for which patents are valid, and upkeep costs required to maintain the patent.
 Making AI Patentable A prominent element of several AI national plans is to ensure that AI-related inventions can receive patents in a timely fashion. The goal of this policy is to encourage research and development (R&D) investment in AI by rewarding that investment with a potentially lucrative patent. This strategy functions to both encourage domestic companies to invest in AI-related research, and entice corporations choosing between different IP systems to set up shop in their country rather than elsewhere. For example, US Patent and Trademark Office Chief Andrei Iancu recently expressed in a Senate hearing that the US needs to make sure that its IP rules adequately protect and incentivize innovation in AI.  33   state council plan, which declared the nation's intention to be the world leader in AI by 2030, one section advised that policy makers in China must "[s]trengthen the protection of intellectual property in the field of AI."  34  The European Patent Office also recently released specific guidance on how to successfully patent inventions in AI and machine learning,  35  and Singapore is allowing AI patents to be "fast tracked" for review through its patent system. 36 Proposals for increasing Britain's competitiveness in AI have also highlighted its patent system's challenges in protecting AI-related inventions as a liability.  37  In some ways, the question of how to create patent protections for AI is not a new one. AI patents mostly fall into the existing category of software patents, and countries have struggled for years to find regulatory structures that incentivize innovation without allowing individual companies to control overly broad, abstract, or obvious ideas.  38  In fact, despite a push to allow more patenting and offer more stringent protections for inventions, two recent major changes in IP law within the United States-the 2011 America Invents Act 39 and the 2014 Supreme Court decision Alice Corp. v. CLS Bank International 40 -made it more difficult to claim and enforce broad software patents. It will be difficult to change patenting rules in AI without also implicating these existing decisions on software patents. Furthermore, it is unclear whether expanding the range of patentable AI-relevant inventions would effectively incentivize innovation. For one, AI and ML commercial activity has experienced massive growth and international investment even while the patentability of innovations remains uncertain, suggesting that the ability to patent AI is not necessary for innovation. In addition, more patents increases the likelihood of litigation, which could act to disincentivize innovation and slow down industry growth. This is particularly a concern for some software patents due to their broad and abstract nature. China has outpaced the US in new patent applications related to the machine learning subfield of deep learning.  42  While some observers have interpreted this information as evidence of China's fast approaching superiority over the US in new AI innovation, three key pieces of information about China and the US's patent systems should make us view these statistics in a different light. First, patents in the US and China have very different standards, requirements and protections. The majority of technology patents in China are filed as "utility model" patents, a category of patent in China not extant in the US.  43  Utility patents require a smaller inventive step, are subject to less rigorous examination upon filing, and last only 10 years (in comparison to 21 years for American patents). Consequently, this has led to filers taking advantage of lax inspections and review. In a 2018 report, the Chinese state-owned Xinhua News Agency accused China's IP system of being characterized by "weak IP, fake demands and some companies fervent on phony innovation," according to a translation by Bloomberg News.  44  Chinese "invention patents" have requirements more similar to those in the US and last for 20 years rather than 10. However, they only comprise 23% of patent holdings in China. Second, in China, there are strong government financial incentives for researchers to file patents, regardless of the underlying patents' merit. 45 This is particularly true in AI, where the Chinese central government has committed substantial public funds to encourage inventions. Finally, it is important to look at the "discard rate" for Chinese patents-i.e. how quickly a patent is discarded by its holder-and how they compare to patents in the United States. A substantial percentage of Chinese patent holders allow their patents to expire before the patent's lifespan is complete-61% for utility patents and 37% for invention patents-in comparison to a discard rate of 15% in the US.  46  When viewed in context with the previous points, it becomes clear that there are many Chinese researchers who file for AI patents in order to claim government incentives without genuinely believing their invention is notable.  47  Regulating National-Security-Related IP Access Countries' patent regimes for AI are not only shaped by economic motivations, but also by national security interests. National governments typically pursue two primary interests on this front: to ensure that their own national security apparatuses have access to state-of-the-art technology, and to withhold that access from perceived rivals. On the goal of ensuring access, a US court decision in 2015 held that the federal government can use patented inventions without the permission of the patent holder and cannot be forced to cease usage of a patented invention; the only remedy is for the patent holder to request damages assessed at market rate (which amounts to compulsory licensing).  48  This means if a patent holder does not wish for their patent to be used by the government (e.g. a patent that has potential surveillance applications) their only recourse after suing for infringement is to force the government to pay for a reasonably costed license. This process is quite distinct from what happens when a private entity infringes on a patent, where the private entity can be enjoined to cease usage of the patent or be assigned additional damages. Additionally, over the last few years the Chinese government has passed broad laws on national security and cybersecurity with implications for access to intellectual property.  49  One of these laws mandates that network operators (broadly defined) provide "technical support and assistance" to national security relevant government offices.  50  The exact legal authority of the Chinese government to force cooperation with Chinese companies is difficult to know. The Center for a New American Security's Ashley Feng reports that "U.S. government officials, including at the FBI, interpreted this vague language to mean that all Chinese companies, including Huawei, are subject to the direct orders of the Chinese government." 51 However, The Diplomat's Jack Wagner reports that the main purpose of the law is to mandate additional data localization and storage on Chinese mainland servers and to set standards around cybersecurity.  52  Additional concerns around Chinese corporations acting as extensions of the state are conceivable, but more speculative in nature. On the goal of withholding access from perceived rivals, the US and the UK have long had government statutes that empower their patent offices to prevent public disclosure and bar the award of patents that have national security implications, regardless of their other merits.  53  In the US in 2018, 5,792 patent applications were covered within these so-called "secrecy orders," higher than at any point since the Cold War.
 Three Scenarios for the Future of AI Intellectual Property Strategies Given the observed corporate AI developer IP strategies and growing government interest in IP law as a lever for influencing AI development, how could the dynamics of AI intellectual property protection evolve? What would these dynamics then mean for the future of the AI industry, and in particular, on the competitive strategies employed by firms and states? Here we present three plausible scenarios which focus on how the IP strategies of corporate AI developers could evolve in the near future: (1) Open Research Continues: The status quo persists: open research alongside defensive patents remains the norm within the ML industry. (2) Patent Lawsuits and Secrecy: Offensive patent litigation breaks out within the ML community and prompts additional secrecy among developers. (3) Expansion of Patent Pools: In response to the threat of litigation, AI developers enter into additional patent sharing agreements. In the following section, we describe each scenario and present evidence to support its plausibleness. These scenarios are intended to be illustrative rather than predictive; indeed, there are several alternative and hybrid scenarios that could warrant further investigation as well.
 Path #1: Open Research Continues
 Scenario: Each of the major AI developers weighs the costs of engaging in patent litigation against their competitors, and decides that the threat of a countersuit and the symmetrical legal costs make litigation a poor choice. Each developer continues to file for patents in order to maintain a credible response, but analogous to a nuclear standoff, this "mutually assured destruction" framework holds. This equilibrium is bolstered by competition for researchers who want to work at companies that prioritize openness and cooperation with their peers. Some patent trolls-firms that profit from licensing and litigating on patents without producing any products of their own-may gain control of patents and engage in litigation without fear of reprisal, but these lawsuits remain relatively insignificant.
 Evidence in Favor: • Despite the recent flurry of activity on the subject, the machine learning community currently shows little sign of changing its open and non-litigious culture. Academics and individuals around the world can use cutting edge machine learning techniques from open-source platforms free of charge. There has been some litigation over trade secret theft in autonomous vehicles (Waymo vs. Uber,  55  Baidu vs. JingChi 56 ), but no large-scale patent litigation over broad concepts in machine learning.  55  Korosec, 2018  56  Borak, 2017 • Current patent rules in the US make AI-related software patents less threatening for the purposes of litigation than they were before the Supreme Court's 2014 decision in Alice Corp. v. CLS Bank International, which made software patents more likely to be classified as "abstract ideas" and thus unpatentable. Given this decision, it is likely that many existing AI patents, particularly those covering broad mathematical concepts, will be rejected during litigation or at the Patent Trial and Appeal Board. • While some AI-related patents in the US are being granted, the majority are not. Data from patent filings shows that in recent years, over 90% of AI-related patent applications in the US were initially rejected, many for being merely "abstract ideas" that are not eligible for patentability. 57 By comparison, the overall rejection rate for patents in the US is 48%.  58  Fewer AI-related patents mean fewer opportunities for companies to litigate over infringement, thus bolstering the incentives for open research. • If large tech companies choose to litigate with one another over AI-related IP, they do not just have to contend with a defendant's AI-related patents, but also with all of the other patents that would likely be used in a countersuit.  59  Google, Microsoft, Amazon, and IBM have expansive business operations across several verticals; this makes patent aggression with other large companies less attractive. Path #2: Patent Lawsuits and Secrecy
 Scenario: Major patent litigations break out between AI developers. While the previous open equilibrium may be preferable for the collective interest of major private AI developers, it may only take one large company deciding it is in its interest to pursue active litigation for this state of affairs to deteriorate. For instance, if IBM, with its trove of AI-related patents and its struggling core enterprise business, 60 chose to litigate against its rivals, it could provoke additional litigation. So-called "patent trolls" (entities that accumulate patents while not producing products of their own) could also threaten to disrupt the mutually assured destruction equilibrium and engage in lawsuits without fear of reprisal. This litigation could bleed over and affect academic research. While the EU has a research exemption that protects academic use from being deemed infringement, the US has no such exemption, and university researchers could thus potentially find themselves on the wrong end of litigation. 61 As litigation escalates, there is a substantial incentive for companies to ensure that potentially patentable inventions are kept secret from rivals. In a 2017 paper, Nick Bostrom discusses how the pursuit of patents in AI could cause companies to share research less often in order to prevent other entities from using  57  Decker, 2019 AI developers agree to cross-license their patents to one another in a "pool" to reduce the risk of litigation. We previously discussed patent sharing agreements in the context of companies like Google and Tencent using these deals to gain market entry into new countries. However, patent sharing agreements need not only be between two actors. There are several historical examples of large technology companies pooling their patents to protect against litigation and create advantageous licensing dynamics. The DVD6C Licensing Group was comprised of eight of the most high profile patent holders in DVD technology (including Samsung and Toshiba, among others). 66 Third-party manufacturers interested in using their technology could approach the group as a one-stop clearinghouse in order to obtain licenses instead of approaching each member individually. Similar arrangements would enable companies within the pool to share trade secrets and research and development, though too much coordination would come under the ire of antitrust enforcement. In this scenario, patent pools could be used not only to curtail litigation risks, but also to limit or promote certain applications of AI. As previously mentioned, companies are currently able to individually place stipulations on patent licensing. If a company decided that it wanted to refuse to license its patents to manufacturers of, for example, autonomous weapons, on moral grounds, it could certainly do so. Extending this to a patent pool, corporate AI developers could group together to share intellectual property and establish shared standards for how they wish to have their intellectual property used, perhaps based on certain ethical principles. These standards could then be implemented via, for example, selective licensing  62  Bostrom, 2017  63  Simonite, 2018  64  Ibid. 65 Ibid.
 DVS6C Licensing Group agreements which restrict use of the pool's IP to actors who commit to abiding by those standards. Membership of the patent pool could also be made conditional on abiding by the pool's standards. Indeed, patent pools used for licensing agreements by standard setting bodies such as the International Standards Organization are precedent for similar structures' success. It is worth noting, however, that such multilateral "refusals to deal" would need to be implemented with caution and appropriate due diligence in order to avoid potential infringements of antitrust law. 67 Evidence in Favor: • Existing software patent pools show the demand for and utility of this type of coordination. Google and several other tech companies participate in the Open Invention Network and Android Networked Cross-License Agreement in order to protect Linux and Android developers from infringement litigation.  68  As an alternative, the MPEG LA group is an example of a software patent pool that operates as a profitable licensing association for its members.  69  Facebook and Google's aforementioned existing patent non-aggression agreements could also be a path towards greater cooperation. • Increasing returns to scale in AI (meaning that more data improves AI systems and platforms, which attracts more users, which in turn generates more data) could increase the odds of industry centralization. A smaller number of relevant actors improves the feasibility of this kind of coordination. 67 Department of Justice 68 Open Invention Network, 2017 69 MPEG LA
 Conclusion AI is poised to be a critically impactful technology, and its development will be deeply affected by existing social and legal institutions. This paper has preliminarily explored an under-examined aspect of this infrastructure: the legal rules, norms, and strategies governing AI-related intellectual property. Leading corporate AI developers today have employed a dynamic and at times unintuitive IP strategy that allows them to respond to the shifting competitive landscape surrounding them. Governments are also seeking to shape IP systems that incentivize innovation around AI while also protecting their national security interests. How each actor balances its varying objectives in relation to AI and how they choose to wield IP strategies to achieve these objectives remains to be seen. This preliminary analysis scratches the surface of what may be an important element of the strategic landscape shaping competition and cooperation among AI firms and prominent national governments. Further investigation in this direction could be fruitful for better understanding the goals and strategies of actors seeking to protect AI-related intellectual property, and how these strategies have flow-on implications on the competitive dynamics that arise between AI developers. This, in turn, could shed light on questions related to the prospects for cooperation between these actors to achieve prosocial outcomes with respect to AI ethics and safety, and more broadly, the appropriate governance of AI going forward. Figure 1 : 1 Figure 1: Top Machine Learning Patent Holders, 2000-2015
 Figure 2 : 2 Figure 2: Top Neural Network Patent Holders, 2000-2015
 Table 1 : 1 USPTO patent upkeep costs, per patent
 2216 US Patent and Trademark Office 17 Regalado, 201318  For an example of this view, see Mark Riedl's quote in Simonite, 2018   19 US Patent and Trademark Office 20 Stone and Vance, 2009 21 Clark, 2015 22 Lewsing, 2017
 Tensorflow on Google Cloud, though they could also use it on Amazon Web Services or Microsoft's Azure. This proves to be a strong incentive for open-sourcing given that cloud services appear to be incredibly lucrative. In 2018, Amazon, Microsoft, and Google earned $25 billion, $23 billion, and $4 billion in revenue from their cloud businesses, respectively. 24 IBM's $34 billion acquisition of open-source cloud computing provider RedHat (which reportedly was also in acquisition talks with Google, Microsoft, and Amazon before selling to IBM 25 ) and Microsoft's $11 billion JEDI cloud computing contract with the Pentagon 26 further show how cloud business is a priority for large AI developers. Microsoft offered Azure cloud users access to a substantial portion of their patents as an incentive for users to join the platform.28  Facebook attempted to add a licensing stipulation to its popular open-source platform React, which would have caused users to retroactively lose their licenses if they ever engaged in patent litigation against Facebook.29     23 Amazon Web Services 24 Griswold, 2019; Microsoft, 2018; Novet, 2018 25 Peterson, 2018 The major downside of open-sourcing is the opportunity cost: open-sourcing in its purest form means forgoing licensing fees from users. It also means sharing what would otherwise be competitive secrets with rival companies. In the next section we discuss some ways that companies manage to avoid these drawbacks.A Hybrid Strategy: The Best of Both Worlds?Despite the apparent conflict between building a large patent portfolio and participating in the open-source community, selective licensing rights can enable a hybrid strategy where companies participate in the opensource community while maintaining their patents. Companies can and do create selective licensing terms for the use of their patents in open-source projects. These agreements can include defining permitted usage in a way that allows some users to utilise the code while disincentivizing competitors to include that code in a product.27    AI developers have also used selective open-source licensing agreements to achieve other ends. New York Times, 2019 27 E.g. The GNU Operating System GPL3 Open-Source license disincentivizes commercial use.28  Microsoft29  The plan was abandoned after developer backlash. See Wolff, 2017  3. Governments' Pursuit of AI Innovation and NationalSecurity Through IP Law
 Challenges Comparing Patent Filings: The US and China 41 34 Wang, 2018 35 European Patent Office, 2019 36 Spruson and Ferguson, 2019 37 Clark et al. 2019 38 Lee, 2013 39  Leahy-Smith America Invents Act, 2011 40 Alice Corp. v. CLS Bank International, 201441 Bessen, 2013   Box 2:
 58  Carley et al. 201559  For example, if Google were to engage in litigation against another company over machine learning patents, they would have to contend with its opposition's patents in E-commerce, search, drones, self-driving cars, smartphone hardware and software and telecommunications, because each of these are areas in which Google operates in and is thus capable of infringement.60  Imbert, 201861 Miller, 2002   intermediary research to obtain a patent first.62  Public research could also be used as evidence in litigation to prove infringement (e.g. releasing a model using a patented method), further dissuading companies in a litigious environment from engaging in open-source communities. As discussed previously, governments in the US, China, EU, and elsewhere are pushing to broaden the scope of patentable material in AI and encourage filings. If these efforts translate into more ostensibly defensible AI patents being filed successfully, this could increase incentives for litigation between patent-holders. • While Google reported that it is holding new AI patents on a defensive basis, other developers have been non-committal about their future intention to litigate with their patents.63  When asked about the issue, a Facebook spokesperson said that "its filings shouldn't be read to indicate current or future plans.' 64 IBM's patent counsel released a statement that said its large AI patent portfolio "reflects its commitment to fundamental research." 65 Evidence in Favor: • Path #3: Expansion of Patent Pools Scenario:
			  American Bar Association, 2014
			 Quinn, 2013 11 Duhigg and Lohr, 2012 12 Hardy, 2011 13 Simonite, 2018 14 Cadell, 2018 15 Google

Abstract:
This working paper is a preliminary analysis of the legal rules, norms, and strategies governing artificial intelligence (AI)-related intellectual property (IP). We analyze the existing AI-related IP practices of select companies and governments, and provide some tentative predictions for how these strategies and dynamics may continue to evolve in the future. In summary: • AI developers use a mix of patents, trade secrets, and open-source licensing agreements to protect their AI-related IP. • Many AI companies are pursuing what may seem like a counterintuitive IP strategy: aggressively patenting AI technologies while sharing them freely. They experience competitive pressure to patent in order to present the threat of a countersuit if another company sues them for IP infringement. However, they also experience pressure to open-source their work in order to attract top talent and entice consumers to use their platforms. • Governments broadly have two goals related to IP policy for AI that are at times in conflict with the goals of researchers and/or companies: to ensure that AI-related inventions can be patented, and to ensure that national-security-relevant AI inventions are restricted for government use and/or kept secret. • Significant uncertainty exists regarding how AI patentability, open-sourcing, and infringement litigation will evolve in the future. • There is an opportunity for patent pools to be used to facilitate pro-social behavior and ethical norms among AI developers. Existing patent pools and practices by international standards organisations represent possible models to replicate.
 Introduction 2.


Existential risks from a Thomist Christian perspective

whether Extinction or Survival will materialise. In fact it may depend on what we do today. We could now promote academic research on x-risks  (Bostrom and Ćirković 2008) , global political measures for peace, sustainability or AI safety  (Cave and ÓhÉigeartaigh 2019) , the development of asteroid defence systems  (Bucknam and Gold 2008) , shelters  (Hanson 2008) , risk-proof food technologies  (Denkenberger and Pearce 2015) , and so on. And while none of these measures will bring x-risks down to zero, they'll arguably at least reduce them. So all of this raises a very real practical question. How important is it, morally speaking, that we now take measures to make Survival more likely? The answer depends on the correct moral theory. It's most straightforward on standard utilitarianism. Suppose we increase the probability of Survival over Extinction by just 1 millionth of a percentage point. In terms of overall expected welfare (setting nonhuman sentience aside), this is equivalent to saving about 1 billion lives, with certainty. So according to utilitarianism, even such tiny increases in probability are astronomically important. Indeed, Nick Bostrom suggested that x-risk reduction 'has such high utility that standard utilitarians ought to focus all their efforts on it  ' (2003, 308ff. ; see also  Parfit 1984 , 452f., Beckstead 2013 . But this implication isn't restricted to utilitarianism. Something very similar will emerge on any view that assigns weight to expected impartial welfare increases. Consider Effective Altruism (or 'EA'). Effective Altruism is the project of using evidence and reasoning to determine how we can do the most good, and taking action on this basis (see  MacAskill 2015) . This doesn't presuppose any specific moral theory about what the 'good' is, or about what other reasons we have beyond doing the most good. But Effective Altruists typically give considerable weight to impartial expected welfare considerations. And as long as we do, the utilitarian rationale will loom large. Thus according to a 2018 survey, EA-leaders consider measures targeted at the far future (e.g. x-risk reduction) 33 times more effective than measures targeted at poverty reduction  (Wiblin and Lempel 2018) . The EA-organisation 80'000 hours suggests that 'if you want to help people in general, your key concern shouldn't be to help the present generation, but to ensure that the future goes well in the long-term'  (Todd 2017) . And many Effective Altruists already donate their money towards x-risk reduction rather than, say, short term animal welfare improvements. In this paper, I'll ask how the importance of x-risk reduction should be assessed on a Christian moral theory. My main claim will be that Christian morality largely agrees with EA that x-risk reduction is extremely important-albeit for different reasons. So Christians should emphatically support the abovementioned measures to increase the probability of Survival. Let me clarify. First, there's no such thing as the Christian moral doctrine. One of the philosophically most elaborate and historically most influential developments of Christian thought is the work of Aquinas. So I'll take this as my starting point, and argue first and foremost that core Thomist assumptions support x-risk reduction. Thomas's specific interpretation of these assumptions are often unappealing today. But I'll also claim that they can be interpreted more plausibly, that their core idea is still authoritative for many Christians, and that on any plausible interpretation they ground an argument for x-risk reduction. So while I'll start with Thomas, my conclusions should appeal to quite many contemporary Christians. Indeed, I'll ultimately indicate that these assumptions needn't even be interpreted in a specifically Christian manner, but emerge in some form or other from a number of worldviews (cf. section 4). Second, there are different x-risk scenarios, and they raise different moral issues. I think the case is clearest for ways in which humanity might literally go extinct, before being superseded by non-human intelligence, and as a direct consequence of our own actions. I'll refer to these cases as as 'non-transitionary anthropogenic extinction', and it's on these cases that I'll focus. It would be interesting to explore other scenarios: cases in which we're replaced by another form of intelligence, non-extinction 'x-risks' (which Bostrom's definition includes) like an extended stage of suffering, or scenarios of natural extinction like volcano eruptions. My arguments will have upshots for such cases too. But I won't explore them here. Third, there are different ways in which, or different agents for whom, x-risk reduction might be 'important'. In what follows, I'll mostly be considering a collective perspective. That is, I'll assume that we as humanity collectively do certain things. And I'll focus on whether we ought to do anything to mitigate x-risks-rather than on whether you individually ought to. The existence of this collective perspective might be controversial. But I think it's plausible (see e.g.  Dietz 2019 ). Christian moral theory, or at least Thomas, also seems to assume it (cf. section 2.1). And many important issues emerge only or more clearly from it. So I'll assume it in what follows. In short, my question is about how important it is, on roughly Thomist premises, for us to reduce risks of non-transitionary anthropogenic extinction. I'll first present three considerations to the effect that, if we did drive ourselves extinct, this would be morally very problematic-a hubristic failure in perfection with cosmologically bad effects (section 2). I'll then discuss some countervailing considerations, suggesting that even though such extinction would be bad, we needn't take measures against it-because God won't let it happen, or because we wouldn't intend it, or because at any rate it isn't imminent (section 3). I'll argue that none of these latter considerations cut any ice. So I'll conclude that on roughly Thomist premises it's extremely important for us to reduce x-risks.
 Three Thomist Considerations There are many Thomist considerations that would bear on x-risks. For instance, in driving ourselves extinct, we'd presumably kill the last representatives of humanity. On a Thomist perspective, those killings will be morally problematic simply as killings (see  ST, q64) . Yet this has nothing to do with the fact that those killings lead to extinction. So let's see whether there would be anything distinctly problematic about extinction, if we brought it about. There is.
 The natural law A first consideration follows from Thomas's teleology, or from the Aristotelian strand in his thinking. Recall that for Thomas the order of the cosmos is teleological. This teleology is grounded in the fact that the cosmos is subject to God. And it takes the form of a law for us: 'the universe is governed by Divine Reason. Wherefore the very Idea of the government of things in God [...] has the nature of a law.' (ST, i-II, q91 a1, co) For Thomas, following this 'eternal law'-or Divine 'will' (ST, i-II, q93 a4, ad1) or 'plan' (ST, i-II, q93 a3, co) for all things-is the ethical imperative for us. So what does it command? We can't know God's intent 'in itself' (ST, i-II, q93 a2, co), at least not in this earthly life (see ST, i-II, q93 a2, co; ST, I, q12 a11). But we can know it 'in its effect', through its manifestations in creation, or through the 'natural law' that we recognise as structuring the physical universe. In particular, we can detect God's plan for us through the natural inclinations He's implanted in us: 'all things partake somewhat of the eternal law[:] [...] from its being imprinted on them, they derive their respective inclinations to their proper acts and ends' (ST, i-II, q91 a2, co). In other words, our inclinations allow an indirect cognition of the essence of God's will, quite like sunrays allow an indirect cognition of the substance of the sun  (ST, q93 a2, co) . So what are our natural inclinations? Thomas speaks of three kinds: 'in man there is first of all an inclination [...] which he has in common with all substances: [...] the preservation of its own being  [...] . Secondly, there is in man an inclination [...] which he has in common with other animals: [...] sexual intercourse, education of offspring and so forth. Thirdly, there is in man an inclination to good, according to the nature of his reason [...]: [...] to know the truth about God' (ST, i-II, q94 a2, co). What's crucial for present purposes is the second type of inclination. Procreation or the 'preservation of the species' is firmly part of our 'natural good'  (SCG, III, , or of what the natural law commands us to do. Now Thomas doesn't believe that everyone must follow all of these inclinations, or that everyone must have offspring. It's permissible for some to remain celibate (see e.g. ST ii-II, q152). But we as a collective have a duty-a 'duty  [...]  to be fulfilled by the multitude' (ST ii-II, q152 a2, ad1)-to procreate. So this grounds a straightforward consideration for x-risk reduction. By going extinct, we'd fail to attain our end, or to accord with the natural law. Indeed, our failure would be profound. It wouldn't just be some of us flouting this law-the bad apples in an overall virtuous whole. We'd fail collectively, as an entire species, to attain our end. And (at least as far as this life's concerned) we wouldn't just fail in one aspect of perfection, while still able to excel in others. Our survival is a precondition for any aspect of our flourishing. So our extinction would mean we fail comprehensively, in all respects of our end. And of course we wouldn't just temporarily fall short of our calling. Once we've gone extinct, there's no second bite at the apple. It would mean we've foundered irreversibly. So the moral failure in anthropogenic extinction would seem complete. In short: Natural law: We ought to attain our natural end. Our extinction would prevent us from doing so-collectively, comprehensively, and irreversibly. Thus non-transitionary anthropogenic extinction would amount to a total moral failure of us as a species. That's a first consideration for why our extinction would be problematic. Now I suppose that this kind of rationale isn't parochially Thomist, but should appeal to Christians quite broadly. Thomas himself interprets the natural law pertaining to procreation very radically-e.g. as permitting a nonprocreative life only for the sake of the 'contemplation of truth' (ST ii-II, q152 a2, co; also ST ii-II, q153 a2, co), 2 and as prohibiting any intercourse known to be non-reproductive  (SCG, III, 122; q153 a2, co) . This would mean that our procreation-related obligations go much beyond preventing extinction. And it would mean that contemporary liberal moral thought is radically wrong about the good human life, and about intercourse among people of the same sex, or people who for whatever reasons cannot or don't want to reproduce. Many contemporary Christians will want to resist these implications. But we needn't understand the general idea of a 'natural law' in this manner. Plausibly, other aspects of the human end (athletic, social, emotional) are just as integral as 'the contemplation of truth'. So there are good lives without children beyond those of philosophers and priests (see e.g.  Nussbaum 1987 Nussbaum or 2011 . Also, perhaps there are other functions of human sexuality (e.g. bonding), such that non-reproductive intercourse isn't a misuse of sexual organs. After all, there's plenty of non-reproductive sexuality among non-human animals (see e.g.  Bagemihl 1999 ). The details of the natural law are a matter of large debate. 3 But I presume that the general idea of a 'natural law' is still quite prominent for Christians today. Indeed, assuming the universe manifests God's intentions, it's a very natural assumption. And I suppose that on any plausible interpretation, it will ground a consideration against extinction: whatever it implies about childfree individuals, contraception or homosexuality, it seems hard to square the natural law with an heirless self-eradication of our species. If the law commands anything, it seems, it commands us to 'be fruitful and multiply' (Gen 1:28). So a consideration along these lines should be authoritative to Christians quite broadly. Self-inflicted extinction would constitute a total failure in our executing our designated role. 4
 Humility However, there's more to anthropogenic extinction than a failure to reproduce. Let's look at the precise way in which, through anthropogenic extinction, we'd fail to attain our end. According to Thomas, there are different ways to fall short of perfection. One is through utter passivity, or sloth, or a 'sorrow' that 'so oppresses man as to draw him away entirely from good deeds' (ST, ii-II, q35 a2, co). So we might just not bother to do anything much at all, and therefore fail in perfection. Another way of failing lies in falling too low, or being overcome with 'the lower appetite, namely the concupiscible' (ST, ii-II, q153 a5, co). So we might behave like lower animals, and fail to live up to our standards. But drawing on Augustine, Thomas says that 'the most grievous of sins' (ST, ii-II, q162 a6, co) consists in aiming too high-in failing to respect our limitations, or acting as if we were God. To do so is to commit the sin of pride, hubris, or superbia. Following the church father (De Civ. Dei XIV, 13), Thomas characterises a prideful man as someone who 'aims higher than he is', or does not 'tend to that which is proportionate to him' (ST, ii-II, q162 a1, co). And what's proportionate to us is of course not so by coincidence, but due to Divine assignment. So pride is opposed specifically to humility: 'humility properly regards the subjection of man to God [...]. Hence pride properly regards lack of this subjection, in so far as a man raises himself above that which is appointed to him according to the Divine rule' (ST, ii-II, q162 a5, co). And in this sense, as far as the 'aversion from the immutable good' (ST, ii-II, q162 a6, co) is concerned, pride is the most grievous sin: it's not just a failure through ignorance or weakness or an innocent desire for another good, but an active 'withstanding' or 'resisting' or manifesting 'contempt of' God (ST, ii-II, q162 a6, co). What does this imply in practice? Thomas specifies what he means. Pride isn't any old undue desire. It's, specifically, an 'appetite for excellence in excess of right reason'-or an inordinate imitation of the powers or competences of God (ST, ii-II, q162 a1, ad2; emphasis added). This may take various forms. We may be pridefully curious about things we can't know, such as facts about good and evil (ST, ii-II, q163 a1, ad3). Or we may unduly discard our need for Divine grace, deeming ourselves capable of happiness on our own (ST, ii-II, q163 a2, co). But a more specific power that isn't appointed to us is decisions about life and death: 'it belongs to God alone to pronounce sentence of death and life' (ST, ii-II, q64 a5, co). Thus to kill someone, or (I take it) actively prevent them from coming into existence, is generally to show an appetite for a power that doesn't pertain to us. And if all of this is true, then non-transitionary anthropogenic extinction in particular would manifest an enormous form of pride. Note that on most such scenarios, we'd fall prey to technologies we were unable to control-nuclear weapons, artificial intelligence, biotechnology, or whatever. So our extinction would mean that we'd overestimated our mastery over these fabrications, and the invulnerability we could leverage in the face of them. It would mean we were prideful in the general sense of desiring an undue excellence. And the upshot of this would be, specifically, a life-death decision on an astronomical scale: preventing lives perhaps by the million billions. It would mean we were prideful in this more specific sense too. Or in short: Humility: Non-transitionary anthropogenic extinction would mean we overstrained our power as a species. And the upshot of it would be a life-death decision on an astronomical scale. Thus it would amount to an enormous form of superbia. That's a second consideration for why our extinction would be problematic. And here too, I suppose this consideration should appeal to Christians quite broadly. Again, Thomas's specific interpretation of God's authority over life/death-decisions should be controversial. He suggests that, while permitting us to execute death penalty (ST, ii-II, q64 a2, co), this authority absolutely prohibits suicide (ST, ii-II, q64 a5, co), and prohibits abortion from the moment of ensoulment (see e.g. ST, I, q118f.)-which some people have claimed, according to Thomas's metaphysical principles and the known facts about embryology, takes place at the moment of fertilisation  (Haldane and Lee 2003, 273) .  5  This would mean self-extinction can under no circumstances be permissible. And it would mean that contemporary liberal moral thought is radically wrong about death penalty, abortion, or suicide and euthanasia. Many contemporary Christians will want to resist these implications. But again we needn't interpret the general idea of superbia in this manner. After all, it's implausible that criminals are 'dangerous and infectious to the community' and must be cut away like an infected part of a body (ST, ii-II, q64 a2, co). Perhaps suicide can in some instances of extreme pain, or loss of autonomy or dignity, be a manifestation of self-love. And if so, perhaps we can then view ourselves as authorised by God-or His commandment to love ourselves (e.g. Mat 22:39)-to end our lives. After all, Thomas himself (defending Abraham's intention to kill Isaac) suggests that 'he who at God's command kills an innocent man does not sin' (ST, ii-II, q64 a6, ad1). Perhaps some forms of abortion (e.g. after rape) can also be seen as an expression of self-love or -respect. Or perhaps we must ascribe to Aquinas a different view of ensoulment  (Pasnau 2002) , or simply reject some of his metaphysics in light of more recent discoveries. Again, the details of superbia are contested.  6  But I assume that the pertinent general idea is still prominent among Christians today. Indeed, that playing God is a sin is a natural corollary of theism. The in-principle-ban on life/death-decisions has a very firm grounding in the Bible (Thomas cites Deut 32:39, for instance). And I suppose that on any plausible interpretation, these ideas ground a consideration against extinction: whatever they imply about death penalty, abortion, and suicide, the dictates of humility seem hardly compatible with our developing a technology that accidentally seals the fate of our whole species. So again, such a consideration should have import for Christians quite broadly. Self-inflicted extinction would constitute a shattering form of superbia.
 The value of humanity But we need to see another aspect of Thomas's view of humanity too, which is perhaps most distinctly Biblical, and which is implicit in the third kind of inclination he ascribes us. For sure, we're not God. But we are nonetheless made to know the truth about Him. In fact, of all corporeal things (i.e., disregarding angels) we're the only ones whose nature enables such knowledge. And this, for Thomas, marks our 'dignity'. It's this dignity which made it fitting for Jesus to adopt human nature, rather than becoming an animal or any other thing: 'in the irrational creature the fitness of dignity is wanting.' (ST, III, q4 a1, co) Indeed, it's these intellectual capacities that ground our likeness to Goda likeness greater than that of any non-reasoned creature (see ST, I, q93 a2, co), and sufficient to say we're His 'image': 'intellectual creatures alone  [...]  are made to God's image.' (ST, I, q93 a2, co) And it's this which ultimately manifests that God loves us more than any other thing. So Aquinas quotes Augustine approvingly: "'God loves all things that He has made, and amongst them rational creatures more, and of these especially those who are members of His only-begotten Son Himself."' (ST, I, q20 a3, sc) In other words, and as Thomas says very explicitly (ST, I, q20 a4, co), of all 6 There's a lot of recent literature concerning Thomas on humility. For longer treatments, see e.g.  Tadie (2006)  or  Fullam (2001) . As far as I see, Thomas himself doesn't discuss whether birth control or contraception infringes on God's dominion over life/death-decisions. But the Catholic church has since leveraged this argument. Yet perhaps a ban on contraception needn't follow from humility either. Perhaps Thomists may appeal to the doctrine of double effect, and compare well-intentioned prevention of fertilisation with well-intentioned killing in selfdefense (cf. section 3.2). Or perhaps they might again claim to be authorised in this decision by the commandment to love ourselves, and love our partners, and our (potential) children for whom we couldn't be sufficiently responsible parents. corporeal things we are the most valuable, or those with most 'goodness'. And this difference is categorical. So our extinction would have cosmological ramifications. For standard utilitarians, there's no categorical difference between a world populated by flourishing societies of people and a world populated by one lonely lizard basking in the sun and feeling a tinge of pleasure. The difference is a matter of degree. For Aquinas that's different. In a world devoid of higher intelligence, there's nothing that's made to God's image-nothing with our special dignity or perfection. This would radically undercut the perfection of the universe, as God created it, which depends on the varieties of goodness (see e.g. ST, I, q47). So our extinction would change the face of creation. Or in short: The value of humanity: Non-transitionary anthropogenic extinction would constitute the destruction of the most valuable aspect of creation-and thus a loss of categorical cosmological significance. That's a third consideration for why our extinction would be problematic. Again, I suppose that this consideration should appeal to Christians very broadly. Thomas interprets the order of creation very radically. He suggests that everything else exists for us (ST, ii-II, q64 a1, co)-such that we may use animals simply as means for our ends, say, and the self-interested killing of another person's ox wrongs at most its owner (ST, ii-II, q64 a1, co). This would mean that our extinction would literally destroy the purpose of the physical universe. And it would mean that much contemporary thought is radically wrong about animals, or other aspects of the natural world. Again, many Christians will want to resist these implications, or so I hope. But the distinct value of humans needn't be interpreted this radically. Perhaps our 'dominion' over animals (Gen 1:26) doesn't mean we can use them simply as means. Perhaps it means we must use our de facto power in the manner of a loving and respectful and liberal guard (see e.g. Linzey 2016, ch. 2). And similarly with every other aspect of creation-ecosystems, plants, and the climate. Again, the exact nature of our status is a large question. But the general idea of our special value still seems very prominent. It does seem hard to avoid if we take seriously that man is 'God's image', or has a special (perhaps responsibility-implying) dominion over the earth-and very plausible given God's human embodiment in Christ. And on any interpretation, our special status will ground a consideration against extinction: whatever it implies about our responsibilities towards animals, the face of creation will be changed categorically if we drive ourselves extinct. So this consideration should be authoritative to Christians very broadly. Self-inflicted extinction would constitute a destruction with cosmological import. Plausibly, there are other Christian considerations that bear on x-risks, at least once we move beyond Thomas. Most notably, perhaps Christian caritas simply gives us reasons to increase expected impartial welfare (see e.g. William Paley's utilitarian Christianity, in his Principles of  Moral and Political Philosophy; Paley 2014, esp. book II) . If it does, the EA rationale will be very pertinent to Christians as it stands. But whether that's an apt interpretation of the Christian virtue-or whether, as I personally find more likely, the latter is more about making people happy than about making happy people-will be controversial. And at any rate, I think these are the three most distinctly Christian concerns. So let me leave it at that for now.
 Discussion If all of this is right, non-transitionary anthropogenic extinction would be a morally highly problematic result. Such extinction wouldn't just amount to a regrettable form of imprudence-but to a failure to fulfill our God-given role. It wouldn't amount to any old failure of this kind-but to a prideful miss of our end. And it wouldn't be just an inconsequential overstepping of our dominionbut an unauthorised decision with categorical cosmological ramifications. 7 But as mentioned, humanity can now (in the form of the present generation) take measures to reduce the likelihood that it will eventually effect this result. So this suggests that we have very strong reasons to take some such measures. It suggests that Christians too have strong reasons to donate their money towards x-risk reduction rather than, say, disaster relief; to conduct x-risk research rather than enquiries about the cause of the grief of a neighbour; or to advocate political measures for long-term safety rather than for short-term caritative purposes. But let's not get ahead of ourselves. Some considerations in Thomas's philosophy seem to suggest that even if extinction would, as far as we know, be bad, it's not important that we now do much about it in practice. So let me turn to some of these theological and moral countervailing concerns. This will not simply corroborate the results so far. It will also help clarify the precise form and weight of what the above rationale implies.
 Theological considerations: Divine providence One apparent reason against x-risk activism is Divine providence. Again, for Thomas, everything is subject to God. And Thomas's God is all-knowing, allpowerful, and all-good: 'In God there exists the most perfect knowledge' (ST, I, q14 a1, co), 'He can do all things that are possible absolutely [or don't imply a contradiction in terms]' (ST, I, q25 a3, co); and He 'loves all existing things' (ST, I, q20 a2, co). One might conclude from this that God won't allow us to go extinct, or that if He does it's somehow just for the better. And from this one might infer that we needn't do anything to prevent our extinction in practice. X-risk reduction is obviated by the rule of a loving God. Now this is difficult and well-trodden terrain. But God's existence doesn't in general seem to warrant any heedlessness. Consider risks of road accidents, say. And suppose any accident accords with His will. It's an interesting question what precisely this implies. Perhaps it warrants a certain ultimate serenity or comfort: a reassurance at the thought that whatever happens on our roads has its mysterious rightness in the grander scheme of things. Perhaps it warrants a fundamental form of trust: a consolation in the belief that our diligence will be duly rewarded (see ST, ii-II, q22 a1, co). But God's unfathomable values cannot act as a guide for us, or as an excuse to depart from the norms and expectations we're given. So these forms of Christian hope must be distinguished from optimism that accidents won't occur, or from feeling exonerated from the need to take caution. Christians ought to see to it that they stop at red lights, respect speed limits, and check their brakes-and just as carefully as everyone else. Indeed, Thomas is the first to emphasise that we're obligated to such circumspection (see e.g. ST, ii-II, q64 a8, co). In short, Divine providence doesn't affect our need to reduce road risks in practice. But then it's unclear why it should do so with risks of human extinction.  8  One might suggest that there's something special about human extinction, setting it apart from everyday hazards, and making it particularly unlikely. Perhaps it's precisely because we're His 'image', or uniquely most valuable, that God won't allow us to perish. Perhaps Jesus's redemption of mankind would have been futile if some sorrowful two thousand years later He let us burn up or wither away. 9 Or perhaps, more specifically, there's evidence in scripture that we'll survive: the primal great flood precisely didn't erase us, and after that deluge there seems to have been a promise of survival-that 'never again will all life be destroyed by the waters of a flood.' (Gen 9:11) This isn't the place to dive deep into Christian theology. But it's hard to see why such reasonings should give us much confidence. The Lord moves in mysterious ways. If all known misery is compatible with His providence, it must surely at least be possible that He'd allow us go extinct. In particular, there are many more passages in scripture painting a grim prospect of extinction than promising a boundless glorious future. And many such visions of the 'end of time' indicate anthropogenic extermination-that 'the nations in the four corners of the earth' are gathered for 'battle' or 'war' (Rev 20:8), right before 'the earth and the heavens' are gone  (Rev 20:11) . If anything, according to the Bible, anthropogenic extinction seems a very live option. So Divine providence might have implications for the metaphysics and ultimate unfathomable significance of our doomsday. But it doesn't seem to warrant our heedlessness about x-risks in practice. If the above considerations are sound, we, just as any generation in the future, must see to it that it isn't us who bring a Johannine finale about.
 Moral considerations: deontology Let's look at moral considerations that might mitigate the importance of xrisk reduction. One such consideration concerns intentions. For Thomas, the permissibility of our actions can depend on what we intend. More precisely, he held, or indeed introduced, the doctrine of double effect (or 'DDE'). According to this doctrine, there's a difference between the effects we intend an action to have, and the effects we merely foresee but don't specifically intend. While it's always impermissible to intend evil effects, it can be permissible to cause them, if what you intend is good. For instance, you may kill in self-defence, if you intend to save your life and merely foresee the death of your aggressor (see ST, ii-II, q64 a7, co). This bears on x-risks. Perhaps non-transitionary anthropogenic extinction would be a very bad result. But one might suggest that as long as we don't intend to effect it, we wouldn't have acted impermissibly: an unintentional 8 In his contribution to this collection, Dominic Roser suggests that Christianity warrants a certain renunciation of control. I'm not sure whether he intends this to contradict my claims here-and to imply, say, that Christians may reasonably check their brakes half as often as atheists. If so, his view seems to me, among other things, unstable. It seems that God's existence would either require standard full-blown caution (as I think Thomas suggests), or would warrant total relinquishment of it. It seems hard to see why He would warrant a certain but only a certain trust in the course of events. At any rate, I'd think most Christians don't understand their faith in this manner. And even if Roser is right, any such limited trust in things would arguably still not undermine my practical conclusions. 9 I thank Felix Koch for mentioning this thought to me. mass life-prevention isn't prideful, and an unintentional thwarting of perfection or erasing of God's image isn't morally wrong. So if our intentions are fine, we needn't worry about x-risk reduction. 10 But this argument is a non-starter. It might well be worse to intend our extinction rather than to merely foresee it. For Thomas, it would mean that our action would lack 'goodness from its end', and not just from its circumstances or species  (ST, q18, esp. a7, co) . But this doesn't mean that unintentional extinction wouldn't be wrong, or even specifically prideful. There's no blanket permission for the well-intentioned. You mustn't speed through the cross-walk and put people at risk, even with the laudable aim of being punctual at the faculty meeting. Thomas says explicitly that you are 'in a sense guilty of voluntary homicide'-and thus (I take it) of superbia-if you kill someone unintentionally but without having taken 'due care' (ST, ii-II, q64 a8, co). He doesn't specify when precisely the DDE allows you to accept a foreseen evil. But he suggests it depends on (i) the expected goodness of the intended good, (ii) the expected badness of the foreseen evil-or on the 'proportion' between the two (ST, ii-II, q64 a7, co)-and (iii) on whether the harmful action is 'necessary' (ST, ii-II, q64 a7, co) for the good, or whether there are ways of securing the good without these bad effects. These criteria decidedly command x-risk reduction. Sure, many technologies that involve such risks also promise important goods. But if my arguments are right, the moral costs are potentially tremendous. And there are less risky courses of action which detract little from the expected good. The above-mentioned measures would reduce x-risks, but wouldn't really jeopardise the benefits these technologies promise. So it seems absolutely obligatory to take them. Going on as we do does seem like scorching through the cross-walk for timeliness at the meeting. That isn't killing for the sake of the killing. But it's a serious violation of the negative constraint of the DDE. And thus it's surely grave enough.  11  Let's consider another countervailing consideration. Perhaps extinction would be terrible. But for all we know, it isn't actually imminent. Indeed, compared to more immediate moral callings like global poverty, gender justice, or disaster relief, it seems very remote. According to utilitarianism or EA, such distance doesn't matter. These views are absolutely impartial. But perhaps Christian morality is different. One might suggest Christian morality is more concerned with visible, immediate, near-at-hand moral problems-the wounded person alongside the road (Luke 10:30), or perhaps the literal 'neighbor' (Mark 12:31)-and doesn't warrant too much concern about such far-distant matters. In fact, Thomas himself is explicit that we should be partial, at least in the virtue of caritas: 'Among our neighbors, we should love them more who are more closely connected to us' (DQV, q2 a9, co; cf. ST ii-II, q31 a3, co). So one might conclude that although our extinction would be bad, we needn't worry about x-risk reduction-or at least not now, or not with resources we could direct towards these more immediate concerns. But this would be a mistake. Christian morality may be partialist about caritas. So there may be a tension between Christian benevolence and the impartialist EA-directive to 'do the most good'. But on the considerations I've sketched, reducing x-risks isn't a form of 'doing good', or of charity, or of fulfilling any positive obligation. It's to ensure-or make more likely-that we don't flout our end, overstep our dominion, or wreck the crown of creation. Formally, it's to ensure we satisfy the constraint of the DDE. In other words, it's to respect a perfect negative duty. And Thomas doesn't seem to endorse partiality or discounting in such negative duties, and neither does Christian morality more broadly. On Christian morality, killing, say, is wrong. And doing what will kill someone in a month is presumably ceteris paribus just as wrong as doing what will kill someone in a year, or in a thousand years for that matter. So the sheer temporal distance of human extinction in itself doesn't seem to obviate x-risk reduction. On the contrary, note that the negative character of this obligation has implications for how it may be weighed against others. For utilitarians, the obligation to reduce x-risk is formally on a par with (other) obligations of benevolence, such as obligations to support the poor. So it ought to be weighed against them. For Christians, if I'm right, this is different. As a negative obligation, the obligation to reduce x-risks must not be weighed against positive imperfect obligations, or can't be discharged by doing enough by way of benevolence elsewhere. It's quite simply a perfect duty. There's a final point worth noting. For utilitarians, notoriously, we may take any means for the sake of the good end. If killing an innocent scientist reduces x-risks, we presumably ought to kill them. For Thomist Christians this will be different. Thomas explicitly accepts deontological constraints: 'some [actions] are evil, whatever their result may be.' (ST ii-II, q88 a2, ad2) So Thomist Christians must not do anything to reduce such risks. They generally mustn't kill or lie or steal for that purpose. In practice, however, this won't make much of a difference. The most salient means of x-risk reduction include academic research, or global measures for peace, sustainability or AI safety. None of these measures seem to violate any constraints. So even if Christian morality is deontological in nature, there's plenty of good ways to start making Survival more likely.
 Conclusion If all of this is right, it's not just that non-transitionary anthropogenic extinction would, as far as we know, be a disastrous result-a cosomologically salient prideful miss of our end. At least in practice, and as a matter of a negative constraint against lack of 'due care' that in principle seems neither discountable nor weighable against positive obligations, Thomist Christians have very strong reasons to take deontologically permissible means to prevent such extinction: to conduct research on risks and risk-reduction, promote political arrangements with an eye to the very far future, donate heir money towards x-risk mitigation, and so on. Indeed, given the gravity of the possible effect, and the perfect or negative form of the duty, they seem to have reason to do this even if it will considerably constrain their resources for classical forms of caritas. Or that's as far as our argument takes us. There are many considerations relevant to non-transitionary anthropogenic extinction that I haven't yet addressed. For instance, it would be interesting to explore the implications of Christian love for our question; to consider the relevance of other Christian virtues-such as temperance, diligence, or patience; or to integrate the assumption that in some form or other we'll always live on, or that in this sense we can't really die out. 12 That's beyond the scope of this paper. It would also be interesting to consider the implications of my arguments for issues beyond my core question. Take risks of natural extinction. If we ought to ensure the preservation of the species, we must generally guard ourselves also against asteroid hits. It would presumably be a form of superbia to presume that no such event could erase us. And to let it happen would mean to let creation's most valuable part be destroyed. Allowing natural extinction might not be as grave as actively extinguishing ourselves, but still seems a large-scale moral failure. Another, and more intricate question, concerns the issue of non-human higher intelligence. Suppose we turned into a nun-human AI and left mankind behind. Would we then miss our end (in the non-preservation of the species), or would we precisely fulfill it (in the perfection of our ingenuity)? Would it be a form of superbia to thus intervene in the nature of species, or would that belong to our natural proportionate capacities? And would the most valuable aspect of the universe be lost, or would the cosmos become more valuable if inhabited by a more perfect form of reason? 13 Again, these are questions for another occasion. There's also the question whether similar arguments emerge on other religious or non-religious worldviews. They arguably do. The normative core of the first two arguments is simply a kind of perfectionism; and the core of the third argument is the idea of a special value of humanity. In Christianity these assumptions are grounded in the existence of the Christian God. But they needn't be. In some form or other, you can adopt them without believing in that God, or indeed in any form of theism. We might see them as emerging, at bottom, from very general aspects of the Christian view of ourselves: from the ambivalent sense that there's something greater than us, but that we nonetheless are quite great indeed. Perfectionism expresses the idea that what structures the universe exceeds mankind and is somehow commanding for us-so that we ought to fulfill our role in the scheme of things, and accept the limitations of it. The idea of our special value expresses that we have a certain greatness nonetheless-that our role isn't any old role, and that it would matter if we were gone. It's at bottom this dichotomous nature, fallen and fragile yet valuable, which implies we should be exceedingly careful with ourselves. Christians, and everyone with a similar view of mankind, and of our obligations and limitations and value, should agree with Effective Altruists that x-risk reduction is extremely important. 			 Thomas doesn't say this explicitly in these passages. He says that the goods of the body are subordinate or 'directed' to the goods of the soul, and that virginity (or a lack of bodily good) is justified if done for the sake of Divine contemplation (or good of the soul; ST ii-II, q152 a2, co). I'm reading the conditional here as a biconditional.
			 For a classic systematic exploration of natural law, see Finnis (2011) . For an in-depth treatment of Aquinas's theory, see e.g.-among very many others-Lisska (1998).4 Historically, this broad rationale hasn't appealed to all Christians, of course. Paul himself seems to suggest it would be best if everyone was celibate (see 1 Cor 7:7). And the Cathars even thought reproduction was a moral evil. I thank Felix Timmermann and Christoph Halbig for pointing this out to me.
			 In the Scriptum super Sententiis, following Aristotle, Thomas himself suggests that the soul is infused after 40 days for males and after 90 days for females (SSS III, d3 q5 a2, co).
			 Perhaps it's worth noting that in principle, these considerations are logically independent. One might accept some but not all of them. For instance, one might doubt that our natural inclinations reveal a Divine law, but still accept the cosmological disvalue of human extinction.
			 I thank Eric Sampson and Jonathan Erhardt for mentioning this thought to me. 11 Strictly speaking, perhaps extinction wouldn't even be a 'foreseen' effect of our actions. By 'foreseen' [praecogitatus] Thomas seems to mean 'known with certainty to result' (see e.g. ST, i-II, q20 a5, co). But we aren't certain that promoting AI, say, will result in extinction. We're just not certain that it won't. In discussing the DDE, Thomas doesn't deal with uncertainty. But he does elsewhere. In ST, i-II, q20 a5, co, he says: 'if the consequences [of an action] are not foreseen, we must make a distinction. Because if they follow from the nature of the action and in the majority of cases [...] the consequences increase the goodness or malice of that action[...]. On the other hand, if the consequences follow by accident and seldom, then they do not increase the goodness or malice of the action.' This can't be right. Throwing stones off a cliff might kill passersby below in only 5% of cases. But then it's nonetheless wrong. Thomas lacked the concept of expected value. Today, we'd surely interpret the doctrine in terms of that concept, or some related one.
			 I thank Dominic Roser for mentioning this thought to me. 13 I thank Mara-Daria Cojocaru and Carin Ism for mentioning these questions to me. For a generally positive Christian stance on enhancement, see e.g. Keenan (2014) .

Abstract:
Let's say with Nick Bostrom that an 'existential risk' (or 'x-risk') is a risk that 'threatens the premature extinction of Earth-originating intelligent life or the permanent and drastic destruction of its potential for desirable future development ' (2013, 15). There are a number of such risks: nuclear wars, developments in biotechnology or artificial intelligence, climate change, pandemics, supervolcanos, asteroids, and so on (see e.g.  Bostrom and Ćirković 2008) . So the future might bring Extinction: We die out this century. In fact, Extinction may be more likely than most of us think. In an informal poll, risk experts reckoned that we'll die out this century with a 19% probability  (Sandberg and Bostrom 2008) . The Stern Review on the Economics of Climate Change, commissioned by the UK government, assumed a 9.5% likelihood of our dying out in the next 100 years  (UK Treasury 2006) . And a recent report by the Global Challenges Foundation suggests that climate change, nuclear war and artificial intelligence alone might ultimately result in extinction with a probability of between 5% and 10% (Pamlin and Armstrong 2015). 1 But the future needn't be so grim. It may also bring
 Survival: We survive for another billion years, and on average there are always 10 billion people, who live very good 100-year-long lives. So there are 100 million billion future people with very good lives. This may sound optimistic. But it's also possible. The earth seems to remain sustainably inhabitable by at least 10 billion people (United Nations 2001, 30), and for another 1.75 billion years  (Rushby et al. 2013 ). The quality of our lives seems to have increased continuously (see e.g. the data collected in Pinker 2018), and it seems possible for this trend to continue. Notably, it partly depends on us 1 More precisely, the report distinguishes between 'infinite impact', 'where civilisation collapses to a state of great suffering and does not recover, or a situation where all human life ends', and an 'infinite impact threshold', 'an impact that can trigger a chain of events that could result first in a civilisation collapse, and then later result in an infinite impact ' (2015,  40). The above numbers are their estimates for infinite impact thresholds.