Abstract:
an informal survey was circulated among participants, asking them to make their best guess at the chance that there will be disasters of different types before 2100. This report summarizes the main results. The median extinction risk estimates were:
 Risk At least 1 million dead
 At least 1 billion dead
 Human extinction Number killed by molecular nanotech weapons.

GLOBAL CATASTROPHIC RISKS SURVEY

25% 10% 5% Total killed by superintelligent AI.
 10% 5% 5% Total killed in all wars (including civil wars).
 98% 30% 4% Number killed in the single biggest engineered pandemic.
 30% 10% 2% Total killed in all nuclear wars. These results should be taken with a grain of salt. Non-responses have been omitted, although some might represent a statement of zero probability rather than no opinion.
 30% There are likely to be many cognitive biases that affect the result, such as unpacking bias and the availability heuristic--well as old-fashioned optimism and pessimism. In appendix A the results are plotted with individual response distributions visible.
 Other Risks The list of risks was not intended to be inclusive of all the biggest risks. Respondents were invited to contribute their own global catastrophic risks, showing risks they considered significant. Several suggested totalitarian world government, climate-induced disasters, ecological/resource crunches and "other risks"--specified or unknowable threats. Other suggestions were asteroid/comet impacts, bad crisis management, hightech asymmetric war attacking brittle IT-based societies, back-contamination from space probes, electromagnetic pulses, genocide/democides, risks from physics research and degradation of quality assurance.
 Suggestions Respondents were also asked to suggest what they would recommend to policymakers. Several argued for nuclear disarmament, or at least lowering the number of weapons under the threshold for existential catastrophe, as well as reducing stocks of highly enriched uranium and making nuclear arsenals harder to accidentally launch. One option discussed was formation of global biotech-related governance, legislation and enforcement, or even a global body like the IPCC or UNFCCC to study and act on catastrophic risk. At the very least there was much interest in developing defenses against misuses of biotechnology, and a recognition for the need of unbiased early detection systems for a variety of risks, be they near Earth objects or actors with WMD capabilities. Views on emerging technologies such as nanotech, AI, and cognition enhancement were mixed: some proposed avoiding funding them; others deliberate crash programs to ensure they would be in the right hands, the risks understood, and the technologies able to be used against other catastrophic risks. Other suggestions included raising awareness of the problem, more research on cyber security issues, the need to build societal resiliency in depth, prepare for categories of disasters rather than individual types, building refuges and change energy consumption patterns.
 Appendix A Below are the individual results, shown as grey dots (jittered for distinguishability) and with the median as a bar.


Abstract: Robot Hacking Games ( , RHG) are government-backed competitions that China uses to advance automatic software vulnerability discovery, patching, and exploitation technologies. 1 These tools offer both offensive and defensive capabilities that promise to increase the scale and pace of vulnerability discovery. If successful, countries could use these tools to find software vulnerabilities quicker than their adversaries. A fully developed capability would allow defenders to patch vulnerabilities as quickly as they are found; attackers could build new exploits equally fast. The Defense Advanced Research Project Agency's Cyber Grand Challenge in 2016 spurred China's interest in this area. The DARPA effort resulted in the creation of state-ofthe-art tools in each of these areas, which have since been siloed into separate programs. China, by contrast, has hosted at least seven competitions since 2017. China's competition structure embodies its military-civil fusion strategy, attracting a collection of academic, military, and privatesector teams. Just two years after the People's Liberation Army's National University of Defense Technology won the first competition in 2017, the military started managing competitions of its own. 2 By 2021, a laboratory run by the PLA Equipment Development Department hosted its first RHG competition. 3 These management and oversight roles situate the PLA in an ideal position to evaluate and attract the best tools and talent. Other state hacking teams, like those of the Ministry of State Security (MSS), will benefit from the technology's development, too. Leading Chinese cybersecurity experts and government strategy documents tie automated software vulnerability discovery, patching, and exploitation tools to Chinese President Xi Jinping's goal for China to become a "cyber powerhouse" ( ). 4 These policy documents create a de facto political mandate for China's cybersecurity community to develop the desired tools. Although they will not make China a "cyber powerhouse" on their own, their development illustrates one important capability that China has chosen in pursuit of its goal.

Robot Hacking Games China's Competitions to Automate the Software Vulnerability Lifecycle CSET Issue Brief

Introduction A collection of seven server racks, each with their team's color and logo splashed on the covers of the bulky boxes, stood on stage in a Las Vegas conference room in August 2016.  5  Professional commentators narrated as each team's Cyber Reasoning System (CRS) hacked away on their own code, and their competitors' servers, trying to find vulnerabilities. Over the course of the competition, these machines earned points by patching their own vulnerabilities while maintaining system performance and submitting successful attacks against opposing teams' servers.  6  But the event lacked the chatter of fingers furiously striking keys that normally accompanies hacking competitions. A few feet away from the flowing bits and bytes, a collection of PhDs, researchers, and private-sector innovators who created the CRSs watched the scoreboard update after every five-minute round. Like coaches at a swim meet, all they could do was sit back and watch. Source: DARPA DARPA hoped to show that software vulnerability discovery, patching, and exploitation could be automated. Together, these three phases constitute the "vulnerability lifecycle." Once a software vulnerability is found, what happens next depends on who found it. Attackers exploit those vulnerabilities, allowing them to access protected systems. Defenders patch those same vulnerabilities to prevent compromise. Both offense and defense want to automate software vulnerability discovery, a welldeveloped field of research consisting of corporate developers and cybersecurity experts using tools to find software flaws. Automated patching and exploitation are relatively less-developed and not as widely used. DARPA's CGC and China's RHGs lump these three distinct phases together because they rely on similar technical processes and techniques. This paper refers to these capabilities as "tools to automate the vulnerabilities lifecycle," or AVL tools. Currently, software vulnerability discovery, exploitation, and patching can be labor-intensive. 8 Software developers, often with years of experience, must pore over code looking for ways it can break. Even with existing tools and techniques, such as symbolic execution, it is impossible to consider all possible avenues of failure. Mistakes leave behind vulnerabilities that attackers may exploit. Open source fuzzing tools, such as American Fuzzy Lop, help researchers locate cracks in their code by generating inputs to cause software crashes. But the time dedicated to this process during software development is constrained by economics. Corporate requirements and shareholder value dictate the amount of time spent securing products, often resulting in insufficient attention. High labor costs and slow product development dent profits. AVL tools would pay huge dividends to companies and governments able to deploy the technology. The Cyber Grand Challenge provided a glimpse of the future by automatically identifying vulnerabilities, building and applying patches, and exploiting vulnerable programs. Although the event targeted relatively simple software compared to more widely-used programs, it demonstrated that AVL tools are viable. The day after CGC's machine-only event concluded, another important event unfolded. DEF CON, a conference for hackers that was co-located with DARPA's event, invited CGC's winning team to enter their system in a capture-the-flag game against DEF CON's human finalists.  9  ForAllSecure, the CGC's winning team from Carnegie Mellon University, agreed to submit their CRS, Mayhem. In the end, Mayhem lost to all of the competition's 14 human teams.  10  But it was not a resounding defeat. In the first 10 hours of the CTF, Mayhem led some of the human teams. In the hours that followed, the teams overtook the machine. By the end of the CTF, humans remained undefeated in hacking competitions. The events in Las Vegas set off a firestorm of articles touting the impact automation would have on cybersecurity.  11  The articles were optimistic and half right. ForAllSecure eventually received a contract to deploy their CRS on DOD systems.  12  Other competitors sold their systems to cybersecurity firms.  13  But AVL tools are still only deployed piecemeal-as specialized vulnerability discovery tools, not as fully developed vulnerability lifecycle products. On this front, the article's predictions were wrong. The technology is not trustworthy enough to automatically patch software, and most exploit generation requires a hands-on approach. Still, the competition and its results were so consequential that the Smithsonian National Museum of American History displayed Mayhem in an exhibition on innovation in defense technology.  14  CGC changed some fundamental assumptions about software and security that underpin the cyber domain. All the hard work spent engineering and fine-tuning the CRSs to do a human's job was sure to go somewhere impressive. But as far as is publicly known, the technology has not. DARPA never planned a second CGC. The agency pushed its research on automated exploitation and automated patching into siloed DARPA programs, reducing the public incentive to assemble such systems while simultaneously removing the technology's focal point for the cybersecurity community. The grand challenge model used for CGC isn't intended to support annual competitions, but rather tries to spur innovation and evaluate the best technology for a particular field at a single point in time. For Chinese Communist Party (CCP) policymakers, CGC did just that.
 China's Robot Hacking Games ( ) China's cybersecurity policymakers began monitoring the development of DARPA's CGC when it was first announced in 2014. Chinese policy publications and industry magazines hyped up the importance of the competition for cybersecurity.  15  That same year, Xi signaled that the party wanted China to become a "cyber powerhouse" ( ), an intentionally vague term meant to inspire. 16 Following Xi's announcement, CCP policymakers began releasing strategy documents to define what the political objective meant in technical terms.
 Xu Guibao ( ) was one of those policymakers. Xu served as a senior manager at a government think tank under China's Ministry of Industry and Information Technology. 17 Throughout his time in government service, he authored numerous policy documents related to China's 13th Five-Year Plan and received 10 patents for his technical innovations. When policymakers needed someone to serve as lead author for China's 2015 "Internet + Artificial Intelligence Three-Year Action Plan," to support China's AI-related technology development, he was a perfect fit. In the span of a single year, Xu witnessed three events that shaped his perspective on the technologies China needed to achieve its "cyber powerhouse" ambitions. The first event drew global attention. In early 2016, DeepMind's AlphaGo beat Lee Sedol, a world-champion Go player, in four of five games.  18  The event concentrated minds around the world on the potential impact of machine learning. A few months later in late 2016, DARPA's CGC concluded with a human versus machine competition, where ForAllSecure's Mayhem led two of the fourteen human teams from DEF CON before ultimately losing.  19  For Xu, Mayhem's short-lived lead over humans reminded him of Lee Sedol's loss. The influential academic referred to Mayhem's performance against those teams as the "AlphaGo incident in the field of cybersecurity"-a bit hyperbolic, but indicative of his thinking at the time.  20  As winter turned to spring, Xu watched as the WannaCry ransomware tore its way across networks and pillaged computers around the world, including networks in China. WannaCry shook Chinese policymakers. For a regime that prizes stability and the government's ability to solve society's problems, the WannaCry incident concentrated minds on the powerful and uncontrollable effects malware could have. The trend in cybersecurity and AI was clear to Xu-automation promised scale and capabilities that could best humans. Xu published an influential article in 2017 titled "U.S. Intelligent Cyberattack and Cyber Defense: Inspiration for China's Cyber Powerhouse Strategy."  21  The title alone was a clear indication Xu thought AVL tools should be one part of China's journey to becoming a "cyber powerhouse." Xu argued that China must "accelerate the development of a networked system of autonomous repair and offensive and defensive robots" to achieve its cyber powerhouse strategy.  The International Robot Hacking Game ( , RHG) attempted to recreate DARPA's CGC.  26  Teams even decorated their server racks with the same combination of lights, colored trim, and logos. Despite differences in scoring and structure, China's first RHG tested the same types of technologies as the CGC: automated vulnerability discovery, patching, and exploitation.  27  In the same way that DARPA oversaw the CGC, the Ministry of State Security's 13th Bureau, the Central Cyberspace Administration of China, and the Ministry of Education supervised the International RHG competition. 28 Despite including "international" in its name, China's first RHG attracted only three of the competition's 22 teams from abroad-one was a CGC finalist.  29  In a humorous twist, a source familiar with the competition claimed one Chinese team just copied open-source code published by a CGC participant and hoped for the best. 30 They didn't win. In the end, China's National University of Defense Technology, a PLA military academy, beat the CGC finalist and other entrants to win the competition.  31  The "international" component of RHG competitions has since been dropped. Chinese policymakers saw what they needed to see. In the months following its first RHG, China doubled down on Xu's recommendations in its "Internet +" Artificial Intelligence Three-Year Action Plan covering the 2018 to 2020 time period.  32  The plan stated that "in order to solve the security technology problems such as vulnerability discovery, security testing, threat warning, attack detection, and emergency response, enterprises should promote the advanced application of advanced AI technology in the field of cybersecurity." 33 By 2018, AVL tools had solidified their place in China's technology development strategies. The subsequent promulgation and standardization of RHG competitions was swift. Including preliminary rounds, China has hosted at least a dozen competitions for AVL technology since DARPA's CGC in 2016.  34  As an indication of its now prominent role, a 2019 article published by Civil-Military Integration in Cyberspace promoted the RHG model as a new standard for cybersecurity competitions in China, joining classic cyber games like capture the flag and jeopardy.  35  Center for Security and Emerging Technology | 8
 Implications of China's RHGs China's Pursuit Will Endure Xi wants China to become a "cyber powerhouse." Strategic policy documents signal that AVL tools are key to achieving Xi's ambitions.  37  As a result, the Party expects organizations able to research the technology to do so.  38  Efforts to develop AVL tools will persist until new strategic documents redefine what it means to be a "cyber powerhouse" or the technology meets the needs of the government. The widespread adoption of the RHG competition model provides strong incentives for Chinese academics, firms, and PLA laboratories to develop the technology. Although the prize money for winners of RHG competitions is paltry compared to private-sector competitions ($50K vs. $250K), party committees at universities and companies are able to encourage their organization's participation. In the United States, such small awards would fall short of the costs for just one researcher to work on AVL tools. In China, the CCP's political mandate to pursue the technology ensures that the competitions and technology remain a focal point for the cybersecurity community, regardless of the rewards offered. Organizations that are able to support the technology's development but choose not to would be out of step with the party-a politically untenable position. 39 China's crackdown on tech firms will concentrate minds on the need to be on the same team as party policymakers. The strong political signal by the CCP mobilizes resources across China to focus on the technologies' development.
 Increasing PLA Involvement The Ministry of State Security 13th Bureau and Ministry of Education served as government "steering organizations" ( ) responsible for managing the first three RHG competitions. 40 Some regional offices of the MSS 13th Bureau run cyber operations in partnership with regional State Security Bureaus. 41 But the 13th Bureau is also responsible for general cybersecurity issues within government agencies. The motivation behind the bureau's involvement in the first three RHGs is unclear-the Ministry of Education's involvement may suggest benign intentions. Although the MSS 13th Bureau has not hosted an RHG competition since late 2018, research on AVL tools may have been moved in-house. The technology's offensive and defensive uses, combined with the bureau's dual-purpose missions, obfuscate the nature of its interest. Few questions remain about the interest of the PLA, however. The Third Annual Qiangwang Cup ( ), which is self-described as having "a natural tendency towards military-civil fusion ( )," marked the shift towards PLA involvement. Qiangwang Cup was the first competition overseen by the Central Cyberspace Administration of China and PLA Information Engineering University.  42  The shift from MOE and MSS 13th Bureau oversight suggests increased military interest in the technology. PLA Information Engineering University is part of the PLA Strategic Support Force's Network Systems Department, which is responsible for military hacking operations.  43  The university's oversight of the RHG may reflect an interest in recruiting students with knowledge of AVL tools, since the Qiangwang Cup is a competition for college students.
 RHGs Are Evolving As long as AVL tools are central to the competition, hosts can change game structures and experiment with operational concepts. The Zongheng Cup introduced human-machine team competitions, where an automated AVL system supports two people in a 3-vs.-3 capture-the-flag style competition.  46  This human-in-the-loop concept is behind one of DARPA's follow-on programs to the CGC-Computers and Humans Exploring Software Security (CHESS).  47  Overseen by a lab affiliated with the PLA Equipment Development Department, the Zongheng Cup demonstrates converging operational concepts between the United States and China. RHGs are no longer changing their structures to match those of the Ministry of Education, but instead those of the PLA.
 Experience and Collaboration China's system of competitions attracts new participants, facilitates hands-on experience, and fosters relationships between institutions and competing teams. "Promot[ing] the training and selection of talents in the field of AI-based cybersecurity" was a key objective for China's first RHG and remains a goal of each subsequent competition.  48  Although automated software vulnerability discovery, patching, and exploitation promise to be more efficient than human professionals alone, these systems still require specialized knowledge to deploy. Operators with experience using the technology can more easily diagnose and solve errors as they arise during deployment. Competitions also encourage relationships between participants. These relationships can be formal, such as teams representing multiple institutions, or informal-social gatherings after the competition. Having a cohort of researchers familiar with the technology is crucial to its successful deployment. Close professional connections could provide networks for troubleshooting technical issues or helping the PLA deploy the technology.
 Conclusion China's state hacking teams, which involve the PLA and Ministry of State Security, stand ready to adopt AVL tools. A report from MIT Technology Review detailed how China's government monitored cybersecurity competitions for new tools and techniques, then rapidly acquired and deployed them against domestic surveillance targets in Xinjiang.  49  RHGs are likely no different. But a full lifecycle AVL tool has not been compiled yet. Instead, individual parts of the tools-like fuzzers, symbolic execution, or automatic exploit generation-may progress in a piecemeal fashion. Automated vulnerability tools are already widely deployed in software development, so improvements in the technology are building on past success. Still, the CEO of Qihoo360, the cybersecurity firm responsible for China's Cybersecurity Military-Civil Fusion Innovation Center-among other state ties-called automated vulnerability discovery tools an "Assassin's Mace" for China.  50  The arcane term references the military strategy of creating an asymmetric advantage over a more powerful enemy-in DOD jargon, it is the Chinese Offset Strategy. For China's military, attacking an adversary's command and control system to disrupt "system-of-systems" communication would fit the bill. 51 AVL tools could help the PLA foment such an attack. U.S. policymakers should consider whether current support for developing AVL technologies is enough. China's largest tech firms and universities are now competing at events hosted by the PLA's labs. Those competitions, in turn, spur innovation, connect researchers, and create a platform for iteratively testing and improving the technology. The United States, by contrast, supports three DARPA programs: Assured Micropatching, CHESS, and Harnessing Autonomy for Countering Cyberadversary Systems.  52  Combined with any classified programs or allocations, these three programs represent the USG's best efforts to develop AVL tools. To get the most out of the technology and maintain any lead over China in this technology, the United States may need to invest more in developing AVL tools. Public competitions with cash prizes large enough to turn winners into businesses could be a good first step. DARPA's CGC in 2016 helped launch a few new companies. But increasing investment and public interest in the technology by the cybersecurity community could yield even greater dividends. With some luck and more public investment, new businesses and a more secure U.S. cyber domain could be in the offing. Endnotes 1 , " phrack -." , September 26, 2017, https://perma.cc/W5VH-J7F5. The translation to "Robot Hacking Game" from Mandarin is both a direct translation, and the translation used in China's own translations. Figure  2  shows each server rack embossed with "RHG" in large white letters at the top to drive home the competition's branding. Although the name evokes thoughts of animated machinery moving about, the more appropriate English-language idea might be a "bot"-used to denote automated bits of software from virtual assistants to automated web scrapers. 2 " 'Halfbit' ," , November 8, 2017, https://perma.cc/ESL5-8YNL; --, " ," April 23, 2019, https://perma.cc/9E4N-CGY4; 419, "RHG , ," Sohu, March 30, 2021, https://perma.cc/7E53-2FBZ.
 3 , " IQ RHG ," , March 30, 2021, https://perma.cc/93CH-QXNN.  4  Translator's note: For a more in-depth discussion in English of the Chinese term , which can be rendered as "cyber powerhouse" or "cyber superpower," see Rogier Creemers et al., "Lexicon: Wǎngluo Qiángguó," New America, May 31, 2018, https://www.newamerica.org/cybersecurityinitiative/digichina/blog/lexicon-wangluo-qiangguo/. 5 Dustin Fraze, "Cyber Grand Challenge," Defense Advanced Research Projects Agency, accessed August 27, 2021, https://perma.cc/65W8-XEEK. 6 Defense Advance Research Projects Agency, "Cyber Grand Challenge Rules, Version 3," Massachusetts Institute of Technology. November 18, 2014, 12-13, https://archive.ll.mit.edu/cybergrandchallenge/docs/CGC_Rules_18_Nov_14_Ver sion_3.pdf. The "attacks" were, in fact, proofs of concepts that exploited other teams' vulnerabilities. An automated referee system evaluated whether the attacks would work as intended, and if so, awarded points. The victims were docked points, but no malware was installed on the targeted system. This structure prevented teams from permanently impairing their opponents and focused the game on vulnerability discovery, exploitation, and patching.  Figure 1 . 1 Figure 1. DARPA's CGC in Las Vegas 7
 22 (These so-called robots are the cyber reasoning systems tested at DARPA's CGC and Xu's inspiration). The CCP channeled Xu's recommendations in its New Generation Artificial Intelligence Development Plan, released around the same time in 2017, generically stating China must "strengthen AI cybersecurity technology research and development." 23 Chinese policymakers would later expressly echo Xu's recommendation. But by the time China's first Robot Hacking Game competitors filed into a Wuhan conference room in late 2017, it was already becoming clear that China viewed AVL tools as important to becoming a "cyber powerhouse." 24
 Figure 2 : 2 Figure 2: China's First RHG in 2017 25
 Figure 3 : 3 Figure 3: A Timeline of RHG Finals. 36
 In 2021, military oversight of RHGs expanded further. The Key State Laboratory for Information System Security Technology ( ), a lab administered by the PLA's Equipment Development Department, managed the 2021 Zongheng Cup ( ). 44 According to the U.S.-China Security and Economic Review Commission, the Equipment Development Department "plays a central role in military modernization by overseeing weapons development across the entirety of the PLA." 45 The lab's oversight of the competition indicates an uptick in the PLA's responsibility for developing, and possibly deploying, the technology.
 ( June 2014): 22-23.