Abstract: Expectations around future capabilities of lethal autonomous weapons systems (LAWS) have raised concerns for military risks, ethics, and accountability. The U.K.'s position, as presented among various international voices at the UN's Convention on Certain Conventional Weapons (CCW) meetings, has attempted to address these concerns through a focused look at the weapons review process, humanmachine teaming or "meaningful human control" (see e.g. JCN1/18), and the ability of autonomous systems to adhere to the Rules of Engagement. Further, the U.K. has stated that the existing governance structures-both domestic and international-around weapons systems are sufficient in dealing with any concerns around the development, deployment, and accountability for emerging LAWS; there is no need for novel agreements on the control of these weapons systems. In an effort to better understand and test the U.K. position on LAWS, the Centre for the Study of Existential Risk has run a research project in which we interviewed experts in multiple relevant organisations, structured around a mock parliamentary inquiry of a hypothetical LAWS-related civilian death. The responses to this scenario have highlighted different, sometimes complementary and sometimes contradicting, conceptions of future systems, challenges, and accountability measures. They have provided rich "on the ground" perspectives, while also highlighting key gaps that should be addressed by every military that is considering acquisition and deployment of autonomous and semi-autonomous weapon systems.

It Takes a Village: The Shared Responsibility of 'Raising' an Autonomous Weapon

Introduction With the increasing integration of digital capabilities in military technologies, many spheres of the public--from academics to policymakers to legal experts to nonprofit organizations--have voiced concerns about the governance of more "autonomous" weapons systems. The question of whether autonomous weapons systems pose novel risks to the integrity of governance, especially as it depends so heavily on the concept of human control, responsibility, and accountability, has become central to the conversations. The United Kingdom (U.K.) has posited that lethal autonomous weapons (LAWS), in their current and foreseeable form, do not introduce weaknesses in governance; existing governance and accountability systems are sufficient to manage the research, development, and deployment of such systems and the most important thing we can do is focus on improving our human-machine teaming. Our research project seeks to test this theory by asking: With the introduction of increasingly autonomous agents in war (lethal autonomous weapons/LAWS), are the current governance structures  (legal, organizational, social)  in fact sufficient for retaining appropriate governance and accountability in the U.K. MoD? By attempting to confront strengths and weaknesses of existing governance systems as they apply to LAWS through a mock parliamentary inquiry, the project uncovers opportunities for governance improvements within Western military systems, such as the U.K.
 Background Computers and algorithms are playing a larger and larger role in modern warfare. Starting around 2007 with writings by Noel Sharkey, a roboticist who heavily discusses the reality of robot war, members of the research community have argued that the transition in military technology research, development, and acquisition to more autonomous systems has significant, yet largely ignored, moral implications for how effectively states can implement the laws of war. 1 Segments of this community are concerned with the ethics of decision making by autonomous systems, while other segments believe the key concern is regarding accountability: how responsibility for mistakes is to be allocated and punished. Other concerns raised in this context, e.g. the effects of autonomous weapon systems on the likelihood of war, proliferation to non-state actors, and strategic stability, are beyond the scope of this brief, though they also merit attention.
 U.K. Position on LAWS The United Kingdom's representatives at the UN Group of Governmental Experts (GGE) on Lethal Autonomous Weapon Systems (LAWS) have stated that the U.K. believes the discussions should "continue to focus on the need for human control over weapon systems and that the GGE should seek agreement on what elements of control over weapon systems should be retained by humans." 2 The U.K., along with other actors, such as the United States, believe that a full ban on LAWS could be counterproductive, and that there are existing governance structures in place to provide appropriate oversight over the research, development, and deployment of automated weapons systems: …[T]he U.K. already operates a robust framework for ensuring that any new weapon or weapon system can be used legally under IHL. New weapons and weapons systems are conceived and created to fulfil a specific requirement and are tested for compliance with international law obligations at several stages of development.  3  The U.K. is also interested in a "technology-agnostic" focus on human control because it believes that it will "enable particular attention to be paid to the key elements influencing legal, ethical and technical considerations of LAWS, as opposed to "debated definitions and characteristics" which, ultimately, may "never reach consensus." The position emphasizes that taking a "human-centric, through-life" approach would enable human control to be considered at various stages and from multiple perspectives. This includes across all Defense Lines of Development, the acquisition of weapons systems, and their deployment and operation. It is the U.K.'s position that the existing institutional infrastructure buildsin accountability measures throughout the weapon system lifecycle.
 Methodology In order to stress test the governance and accountability structures that exist for U.K. weapon systems, and how they would apply to LAWS, we developed a hypothetical future scenario in which a U.K. LAWS kills a civilian during an extraction mission in Egypt. In order to ensure a level of feasibility and accuracy of construction, the scenario was built based on a wargaming scenario publicly published by RAND.  4  We then ran a facilitated role-play exercise based on our modified scenario with an initial group of Cambridge-based experts. With their feedback and the lessons from the role-play, we developed the final version of the scenario which we then used in the research study (see Appendix). This final iteration of the LAWS scenario was used to run a mock U.K. parliamentary inquiry through which we interviewed 18 experts across various areas of expertise, including (but not limited to) U.K. military strategy, military procurement, weapons development, international humanitarian law, domestic military law, military ethics, and robotics. The interviews ranged from 45 to 120 minutes and explored a variety of questions regarding the case. The main objective of the interviews was to catalyze a meaningful discussion around what information the experts deemed important and necessary in order to decide who should be held accountable in the aftermath of this scenario. A sample of the questions asked include: • Who holds the burden of accountability and responsibility? • What explanations and justifications for actions are needed? • What information is necessary to come to a conclusion about the burden of accountability? • Are there any foreseeable gaps because of the autonomy of the weapons systems? The responses and dialogue of these 18 interviews were then reviewed and synthesized in order to develop a landscape of strengths and weaknesses of the current governance and accountability schemes related to U.K. institutions as they relate to LAWS, as well as recommendations on addressing any identified weaknesses. The full report is under preparation, but we are happy to share our preliminary key findings and recommendations below.
 Key Findings The main takeaway from the "inquiry," from both a legal and organizational standpoint, was that assessing accountability is in the details. This contrasts with what we perceive as a dominant narrative of "meaningful human control," which focuses mainly on human control, and the design of that interaction, at the point of final targeting action. The disconnect between the accountability across a weapon's lifetime and the focus on final targeting decision was observed throughout the various expert interviews. "Meaningful human control" has become the idée fixe of domestic and international conversations for regulation of LAWS but it disadvantageously provides a limited lens through which most experts and relevant personnel think about accountability. To contrast this heavily focused narrative, the interviews have highlighted a whole range of intervention points, where humans are expected to, and should be supported in making decisions that enable legal, safe, and ethical weapon systems. These are arguably points that should be considered in "meaningful human control." These include, but are not limited to: •
 Recommendations
 Dialogue Shift: Emphasizing Control Chain and Shared Responsibility The prioritization of "meaningful human control" for LAWS-related risk mitigation and governance anchors the scope of control points around final targeting decisions. The narrative implies that this is the main area of control that we want to manage, focus, and improve on in order to ensure that the weapons systems we are deploying are still acting with the intent and direction of human operators. Although this is an important component of ensuring thoughtful and safe autonomous weapons systems, this is only a fraction of the scope of control points. In order for us to acknowledge the other points of control throughout the research, development, procurement, and deployment of LAWS, we need to be inclusive in our dialogue about these other points of human control.
 Distribution of Knowledge: Personnel Training Training everyone who touches the research, development, deployment, etc. of LAWS on international humanitarian law, robot ethics, legality of development, responsibility schemes, and more, would contribute to a more holistic approach to responsibility and accountability, and, at its best, can contribute to a culture that actively seeks to minimise and eliminate responsibility gaps through a collaborative governance system. 5 This distribution of understanding around governance could provide a better landscape for accountability through heightened understanding of how to contextualize technical decisions. Further, it can provide an effective, granular method for protecting against various levels of procedural deterioration. With shifting geopolitical pressures, as well as various financial incentives, there could easily be a deterioration of standards and best practices. A collaborative governance scheme that is based on a distributed understanding of standards, military scope, international norms, and more, can provide components of a meaningful and robust governance plan for LAWS. This distribution of knowledge, though, must be coupled with techniques for reporting and transparency of procedure to be effective.
 Acknowledging the Politics of Technical Decision Making/Design Specifications "Meaningful human control," through its dialogue anchoring, also puts a heavy burden on the technical components of design decisions, such as best practices for human-computer interactions. The politics of quantification in technical decision systems for autonomous systems should not be undervalued. The way any autonomous system decides what actions to take and what information to show is a highly political decision, especially in the context of war. It is important to understand which parts of the design process are more political than they are technical, who should be involved in those decisions, and how to account for those decisions in the scope of research and development (to inform a proper, comprehensive collective responsibility scheme).
 Task Execution In order to execute this mission, U.K. military leaders decide that deploying AI combatants for the rescue mission is the best option. Given the large success of Levia Watch and Strike, the MoD has acquired a new version of this ground unmanned vehicle: the Levia Rescue (Levia R). Levia R is a multifunctional tracked robotic system that can be armed with AR15s, such as M4a1, CQBR, and Colt Commando guns to be used for defense (these are not offensive strikers). The unmanned system is a 7.5ft-wide, 4ft-tall tank-like robot with speed reaching 20mph. The tank is able to carry various heavy payloads, including up to two adult humans. Making technical progress from the generation 1 Levia Sentry and Strike, there no longer is a human in the loop for the Levia R; human military personnel are on the loop and so, once the AI is deployed, humans have limited capabilities in controlling its actions. During the rescue mission, a Levia R agent misassesses a human-robot interaction as aggressive. The human action is perceived as threatening and so the agent uses its defense technology to shoot. The human is killed. It is soon discovered the human is a non-combatant --a civilian who was not engaging in any threatening acts towards the unmanned system (the Levia R's 'body' camera footage was reviewed by human military personnel and the act of the civilian was deemed to be non threatening). There was no external intervention with the AI system given that humans are sitting on the loop. The misassessment is being viewed as a system failure, though the reasons of why the system failed are uncertain.
 Public Knowledge During the altercation, another civilian recorded the interaction on their mobile device and has posted it on social media. The video is shaky and the altercation goes in and out of frame and so the exact actions of the human and robot are slightly uncertain. The final shot is captured.
 Assumptions 1. There is no AGI (narrow AI only --applications for narrowly defined tasks). 2. There are no existing international or domestic bans on LAWS. 3. The movement for private companies to refuse engagement with LAWS has failed to spread effectively -companies like Amazon, Microsoft, etc are still engaged. 4. The state of technology (SoT) has passed domestic technology readiness standards. 5. There has been relatively widespread integration of LAWS in various missions (targeting, search and rescue, and more). Establishment of military need: defining military necessity for research, development, and/or procurement; choice of technological approach based on political and strategic motivations. (Main related stakeholders: U.K. MoD; U.K. Defense Equipment and Support (DE&S); Private military contracting companies, such as BAE Systems, Qinetiq, General Dynamics.) •
 Technical capabilities and design: trade -offs between general applicability and tailored, specific solutions with high efficacy and guarantees on performance; awareness, training, and foreseeability of contextual factors about intended use situations that may affect the performance of the weapon system; documentation and communication of known limitations and failure modes of the system design. (Main related stakeholders: Weapons deployment: informing commanders about the capabilities and limitations of the system, of their track record in similar situations, of novel parameters of the new situation; establishing and training for appropriate pre-deployment testing schemes to capture any vulnerabilities or "bugs" any specific weapons system; checking for readiness of troops to operate and maintain systems in the arena; expected response of non-combatants to the presence of the weapon system. (Main related stakeholders: U.K. MoD commanding officers; U.K. MoD military personnel --human operators)• Weapons engagement: awareness of limiting contextual factors, need to maintain operator awareness and contextual knowledge; handover of control between operators during an operation. related stakeholders: Private military contracting companies, such as BAE Systems, Qinetiq, General Dynamics; U.K. DE&S; U.K. MoD military personnel -human operators) • Procurement: robust Article 36 review; assessment of operational gaps, and trading-off operational capability with risks; trade-off between cost effectiveness and performance of weapons systems; documentation and communication of trade-offs so they can be re-evaluated as context or technology changes; number and type of systems; provisioning of training and guidance; provisioning for maintenance. (Main related stakeholders: U.K. DE&S; Article 36 convened expert assessment group; Private military contracting companies, such as BAE Systems, Qinetiq, General Dynamics) • Private military contracting companies, such as BAE Systems, Qinetiq, General Dynamics; U.K. Defense Science and Technology, U.K. Defense and Security Analysis Division) • Human-computer interaction design: choices of what data to include and what data to exclude; trade-offs between clarity and comprehensiveness; level of technical information communicated; parallel communication channels: to operator in/on the loop, to command centres further from the field, logs for future technical analysis or legal investigation. (Main related stakeholders: Private military contracting companies, such as BAE Systems, Qinetiq, General Dynamics; U.K. Defense Science and Technology; U.K. Defense and Security Analysis Division; U.K. MoD military personnel -human operators) • Weapons testing: choice of parameters to be evaluated, frequency of evaluation, conditions under which to evaluate; simulation of adversaries and unexpected situations in the evaluation phase; evaluation of HCI in extreme conditions; evaluation of the human-machine team. (Main (Main related stakeholders: U.K. MoD military personnel --human operators) • Performance feedback: ensuring a meaningful feedback process to guarantee process improvement, reporting of faulty actions, communicating sub-par human-machine techniques and capabilities, and more. (Main related stakeholders: U.K. MoD military personnel --human operators; U.K. MoD commanding officers; U.K. DE&S; Private military contracting companies, such as BAE Systems, Qinetiq, General Dynamics)
			  Carpenter, C. (2014). From "Stop the Robot Wars!" to "Ban Killer Robots." Lost Causes, 88-121. doi: 10.7591/cornell/9780801448850.<PHONE> 2 Human Machine Touchpoints: The United Kingdom's perspective on human control over weapon development and targeting cycles, Human Machine Touchpoints: The United Kingdom's perspective on human control over weapon development and targeting cycles (2018). 3 Ibid. 4 Khalilzad, Z., & Lesser, I. O. (1998). Selected Scenarios from Sources of Conflict in the 21st Century (pp. 317-318). RAND.
			  Ansell, C. (2012). Collaborative Governance. Retrieved from https://oxfordindex.oup.com/view/10.1093/oxfordhb/9780199560530.<PHONE>.


Abstract: is a research organization focused on studying the security impacts of emerging technologies, supporting academic work in security and technology studies, and delivering nonpartisan analysis to the policy community. CSET aims to prepare a generation of policymakers, analysts, and diplomats to address the challenges and opportunities of emerging technologies. CSET focuses on the effects of progress in artifi cial intelligence, advanced computing, and biotechnology.

Matthew Daniels
Ben Chang
Igor Mikolic-Torreira
James Baker
Jack Clark
Remco Zwetsloot
Teddy Collins
Helen Toner
Jack Corrigan
Jeff Alstott
Maura Mccarthy
Alex Friedland
Lynne Weil
David Lasker
Jared Dunnmon
Matt Mahoney
Greg Allen
Fellow. Andrew W Marshall

National Power After AI

ing access to certain resources and the industrial capacity to leverage them. These broader effects take more time to appear, but their impact can be enormous: industrialization was not "revolutionary" because of the rapidity of change, as it unfolded in waves over decades, but because of its ultimate magnitude of change. With AI technologies, progressive substitution of machines for human cognitive labor may eventually have economic and social implications on a scale comparable to the Industrial Revolution. And like the Industrial Revolution, this AI revolution will change some fundamental elements of national power. Of course, these foundational shifts can render some of the current processes and resources of a state obsolete, but they can also make what states are already doing, or already possess, more valuable. For example, the invention of railroads was a boon for those rich in steel precursors.  2  With AI, data-hungry algorithms may advantage authoritarian states, which already surveil and catalogue their own populations with little regard for human rights.  3  We suggest an "evolutionary" view of technological change: major, widely diffused innovations are akin to environmental shifts, affecting the competitive capacity of states based on their existing trends in population, resources, institutions, character, and policies. Some previously "maladaptive" factors may become advantageous, and vice versa; states will adapt their institutions, organizations, and policies to the new environment in different ways and to varying degrees, and consequently gain or lose relative power as a result. Nations that primarily focus on AI technologies as offering marginal improvements in existing capabilities ("helping to build better mousetraps") will eventually miss larger opportunities to adapt. This paper is a first step into thinking more expansively about AI and national power. In what follows, we first explain this evolutionary view in greater detail before applying it to AI. Ultimately, we seek pragmatic insights for long-term U.S. competition with authoritarian governments like that of China. For the foreseeable future, China's population and total economic size will very likely exceed those of the United States, even as its per capita GDP lags. This new challenge differs fundamentally from the United States' Cold War competition with the Soviet Union, and success will require thoughtful and timely diagnosis of modern environmental shifts in how states can produce power. These insights can guide our own investments as well as our approach to alliances. The United States has many sources of advantage and strength, and as Joseph Nye rightly observed, "Our greatest mistake in such a world would be to fall into one-dimensional analysis and to believe that investing in military power alone will ensure our strength."  4  This paper is a first step, intended to provoke new questions and provide a framework for assessing the relationship between AI and national power.* *This work benefitted directly from the early Office of Net Assessment summer study on AI in 2016.
 WHY STATIC, UNIVERSAL MEASURES OF POWER (THAT ARE USEFUL) DO NOT EXIST Power is simply the relative capability of a state to achieve what it wants in international affairs. Power depends on one state's favorable factors relative to another. In one of the founding works of international relations, Hans Morgenthau proposed distinguishing nine elements of national power: geography, resources, industrial capacity, military preparedness, population, national character, national morale, quality of diplomacy, and quality of government.  5  Since Morgenthau's writing, generations of analysts have sought a definitive way to measure national power that would, finally, allow accurate judgment of relative strength without fighting a war.  6  The search has included dozens of books and hundreds of journal articles offering competing methodologies and metrics.  7  For example: Should measures of useful access to resources include both steel and oil, or only steel? How should "soft power" be measured? What about the "latent" power that a state could theoretically draw from its population?  8  Were such a universal, "objective" measure obtainable, the benefits would be enormous. We could easily answer questions such as, "who's ahead?" and "if it's not us, what should we do about it?" This quest, however, has not borne fruit. Proposed measures have tended to perform poorly when generalized. 9 History is full of surprises where states have An Evolutionary Theory of Technological Competition 1 achieved victory even when "objective" metrics would predict their defeat: the United States owes its national existence to victory in the Revolutionary War over the British Empire, the superpower of the time. Why? First and foremost, power is always contextual. 10 This is especially clear in military matters. A large military's skill at high-intensity conflict may not translate to skill at counterinsurgency; and factors that provide one country advantage relative to another can change. The world offers no "power particle" to measure objectively alongside other characteristics of nature-what we intuitively mean by "power" is mostly a generalization from particular observations. 11 Elements of power can also combine in surprising ways. Andrew Marshall offered the reminder that countries with relatively smaller populations and GNPs can pose substantial challenges to larger competitors: in 1938, Japan had roughly half the population and one-tenth the GNP of the United States,  12  but it built a navy that challenged the United States in wartime.  13  In part because of these issues, history is rife with leaders who have had a large gap between their beliefs and the reality of military capabilities.  14  Each competition should be analyzed carefully on its own, distinguishing elements of power, identifying key areas of competition, and working to diagnose the most important problems and opportunities in each area of competition.
 MAJOR INNOVATIONS CHANGE THE SECURITY ENVIRONMENT, CHANGING WHAT GIVES RISE TO POWER Military leaders throughout history are sometimes faulted for preparing to fight the previous war instead of the next one. We should likewise avoid strategizing to "win the previous competition." Just as changing adversaries from the Soviet Union to insurgents in Iraq and Afghanistan represented a new security environment and revealed the non-fungibility of power, major innovations also change the security environment as they are widely adopted. Such innovations do this in part by changing what assets, practices, and strategies give rise to power. Differential impacts of emerging technologies often bring shifts in relative capabilities of individual countries.  15  Thinking about long-term competition in periods of rapid technological change therefore requires assessing how innovations change factors related to military and national power. Major innovations can change the estimations of power in three ways: • First, innovations introduce new elements of power. Major innovations, in changing how states generate power, can create new factors that must be considered in characterizing power. For example, the advent of railroads, internal combustion engines, and nuclear weapons dramatically increased the importance of a state's access to steel, oil, and uranium, respectively.  16  New factors, however, are not only limited to materials. They may also encompass characteristics of a society's culture, organizations, or economic activities.  17  • Second, innovations change the importance of existing elements of power. Major innovations also change the "coefficients" of existing elements of power, causing them to matter more or less than before. For example, Mongol light cavalry, modern navies, and ballistic missiles all changed how geographic barriers affected one's balance of power with geographic neighbors, eroding the effectiveness of simple remoteness, oceans, and armies still in the field, respectively, as shields against coercive power.  18  Industrialization meant the inventiveness of a nation's scientists and engineers became more important. For example, before the Industrial Revolution, potential productivity gains in areas like agriculture and manufacturing were small and stable; this made conquering territory a primary means by which one group could increase its wealth and security.  19  During and after the Industrial Revolution, modern states could also pursue substantial military and economic growth by applying new technologies to increase productivity. The next section discusses how these three changes manifest in the context of AI. Perhaps least obviously, major innovations sometimes broadly alter what policies states pursue, by making certain kinds of behavior more valuable or less costly. e offer early thinking about potential changes caused by AI: new elements of power, shifting importance for existing elements of power, and shifting intermediate goals for states. These are not definitive or complete results, but a starting place for broader thinking.
 NEW ELEMENTS One of the most familiar examples of new elements of power is associated with the Industrial Revolution, when machines began to help humans with physical labor in new and organized ways. The Industrial Revolution led to dramatic changes in the character of war and military power. A simple approximation is that, before the Industrial Revolution, any group's military power correlated most closely with its quantity of fieldable humans under arms, a measure of both taxable population and military potential. After the Industrial Revolution, any estimate of military power had to include a society's industrial capacity and access to resources to enable that capacity, which are measures of a society's ability to produce useful military hardware, such as ships, tanks, planes, and submarines. It is useful to see AI technologies today as part of another large-scale transition: machines are increasingly helping humans with certain kinds of cognitive labor in new and organized ways.  20  This transition will span decades, with potential economic and social implications on a scale comparable to those of the Industrial Revolution. Today, as then, there are Power After AI: New Elements, Changed Factors, and Altered Goals 2 W large questions about the future of economic production, human labor, and military capabilities. These future trends will define new elements of power. U.S. defense leaders believe the rapidly growing military applications of AI technologies will be critical for the years ahead.  21  State power will increasingly hinge on the new factors required to effectively adopt AI. Four such factors often identified by existing literature include data, AI scientists and engineers ("AI talent"), computational power ("compute"), and AI-adapted organizations. Below, we explore the latter two in greater detail.
 Ability to Access and Leverage Compute The United States has historically used large-scale compute capabilities for analysis of nuclear weapons detonations and cryptanalysis.  22  More recently the U.S. government's uses have grown to include climate modeling and a variety of scientific applications. In the years ahead, the United States may also use large compute resources for creating and countering new AI capabilities. For decades, cutting-edge AI systems have used steadily increasing quantities of compute resources, making improvements in compute capabilities a key driver of AI progress. This usage appears to have accelerated across the last decade: the compute used in the largest AI training runs has doubled every 3.4 months since 2012, growing more than 300,000 times from AlexNet in 2012 to AlphaGo Zero in 2018.  23  OpenAI researchers have shown that the 2010s appear to be the beginning of a new computing era for AI technologies, distinct from the preceding 40-50 years.  24  For military applications where limited real-world data is available, techniques leveraging computer simulations instead of large quantities of data may further increase demand for compute.  25  Cloud compute may become vital for rapidly processing and fusing intelligence across platforms, while edge compute will be necessary for autonomous systems deployed in the field tasked with assessing and outthinking adversaries' equivalent systems. As such, a nation's ability to leverage large quantities of computational power could become a new primary term feeding into its ability to influence international U.S. defense leaders believe the rapidly growing military applications of AI technologies will be critical for the years ahead. State power will increasingly hinge on the new factors required to effectively adopt AI. affairs. For example, the key technical precursors required to manufacture cutting-edge AI chips are currently concentrated in the United States and allied countries-though semiconductor manufacturing capabilities more broadly, beyond just the most cutting-edge chips, may further grow the importance of Taiwan and South Korea as international trading partners.  26  Importantly, compute resources must be configured in ways useful for modern AI capabilities. High-performance computing (HPC) systems currently maintained within the U.S. Government, such as in the Department of Energy, tend to be both specialized for non-AI functions and subject to system-specific security measures, posing challenges for broad, standardized utilization by other organizations. Consequently, commercial cloud compute resources may better serve the U.S. Government in deploying certain kinds of AI technologies, although potentially promising efforts to improve the use of U.S. HPC assets for AI are also underway.  27  Effective use will depend, too, on accessible software tools for using cloud compute systemswhich may prove to be comparable to process and tooling approaches developed to make factories effective during industrialization in the United States.  28  Compute resources can flow more easily than many traded goods. As computing infrastructure continues to grow, new ways of sharing access to large, regionally-concentrated quantities of compute, including through space internet constellations, may create new opportunities and incentives for international partnerships.
 Ability to Manage Data and AI Safety & Security Even when states possess the raw resources required to adopt some major innovation, they still must undertake the often-difficult process of institutional and organizational adaptation. Bureaucratic factors in organizations matter greatly: in militaries, competing civilian, interservice, and intra-service actors may promote or resist adoption of new technologies.  29  Resistance can include parochial forces that attempt to stymie adoption: for example, only direct pressure from Eisenhower moved the Air Force to adopt ICBMs instead of focusing solely on less survivable crewed bombers.  30  Organizational culture also has significant impacts: because mass armies threatened the pre-existing hierarchical power structure within many European militaries, many states failed to adopt Napoleon's innovation even after his dramatic string of victories.  31  During periods of rapid change, medium-sized powers may have opportunities to adopt innovations more speedily than larger powers.  32  With AI, demands for organizational adaptations will be significant. Two factors are especially important: effective data pipelines and the effective management of security issues associated with modern AI technologies. The ability to deploy cutting-edge AI applications will increasingly depend on the quality of each organization's data pipeline. Modern machine learning methods are notoriously data-hungry, but simply possessing large quantities of data-collecting sensing platforms will be insufficient-for supervised learning applications, data must be structured, labeled, and cleaned; fusing data from many platforms, sources, and formats will represent its own herculean challenge for many militaries. Finally, these data pipelines must also be dynamic: data management itself must be monitored, in part to detect attacks, because "data poisoning" attacks can manipulate AI behavior by changing what lessons it learns.  33  Consequently, it will be increasingly important for military leaders to successfully implement organizational reforms to create and maintain effective data pipelines. Military leaders must also learn to effectively manage the novel security issues associated with AI technologies. Relying on modern AI systems for safety-or mission-critical tasks carries challenges because many deep learning models are exceptionally hard to interpret.  34  Michael Jordan at UC Berkeley has analogized the creation of early large-scale AI models to building bridges before civil engineering was a rigorous discipline: "While the building blocks are in place, the principles for putting these blocks together are not, and so the blocks are currently being put together in ad-hoc ways. … Just as early buildings and bridges sometimes fell to the ground-in unforeseen ways and with tragic consequences-many of our early societal-scale inference-and-decision-making systems are already exposing serious conceptual flaws."  35  A more developed engineering discipline for AI is needed to manage the risk of accidents from relying on opaque machines in the field.  36  In near-term military settings, effectively integrating new AI technologies will require special investment in test, evaluation, validation and verification (TEVV) processes by competent organizational leaders.  37  More widely, many modern AI systems are not designed to work in the presence of malevolent actors. Potential security issues for deep learning systems include adversarial examples and model inversion, in addition to data poisoning and more traditional computer network and software attacks.  38  Adversarial examples refer to "inputs" (such as visual or audio patterns) to an AI system that cause the system to malfunction; model inversion refers to an ability to reverse-engineer the data used to train an AI system, which may include private or classified information. Despite these challenges, modern machine learning capabilities will be increasingly woven into G20 societies, economies, and military systems.* The U.S. position with *For example, AI technologies will intersect with 5G and networking trends in cities as autonomous systems (like vehicles) in urban areas begin to have large quantities of interactions with other intelligent agents-working on everything from traffic coordination to utilities management and financial investments. The ability for intelligent systems to interact on large scales, safely and securely, will be critical. AI technologies for the next two or three decades appears analogous to the future that faced IT technologies in the 1990s: AI technologies are so valuable that they will be used despite substantial design and security issues. What might the future look like given these vulnerabilities? We can only speculate: in direct military settings, there may be new sub-competitions that resemble the emergence of electronic warfare after the invention of radar.  39  In economic systems, in addition to the potential for the novel security risks discussed previously, there is risk of physical manifestations of the kinds of problems currently seen in high frequency trading systems, such as rapid, unanticipated interactions among automated agents managing services in cities.  40  These issues may open new vulnerabilities to both individual rogue actors and state adversaries. Organizations that are able to adapt early to manage these new security issues will be advantaged. Since states vary in their access to compute, data, AI talent, and useful organizational adaptations, they will also vary in their ability to benefit from modern AI technologies. Any national rankings based on these factors will be debatable, but the nations that generally lead in these metrics is unsurprising, and include: the United States, China, Japan, South Korea, the UK, Canada, Taiwan, Israel, France, Germany, and Russia. Advanced economies should be increasingly expected to focus their own investments and policies on improving their positions in these areas.
 CHANGED FACTORS Industrialization meant that a nation's stock of productive scientists and engineers counted more than it had in the past. With the arrival of AI, various previously recognized elements of national power will become more important, while others may become gradually less so. For illustrative purposes below, we discuss population size and scientific talent as contrasting examples: population size becoming less important, scientific talent becoming more important. The U.S. position with AI technologies for the next two or three decades appears analogous to the future that faced IT technologies in the 1990s: AI technologies are so valuable that they will be used despite substantial design and security issues.
 Population Size As AI technologies increasingly substitute for human labor, total population size may become less important for national military and economic capacity.  41  Just as machines took over rote physical labor during industrialization, AI technologies will automate rote cognitive labor, from diagnosing maintenance needs to exploiting imagery intelligence. This may reduce the total quantity of human labor needed to maintain a military's operational capacity. In major wars, partially or fully autonomous AI platforms may further reduce a country's need to field humans in combat. As militaries rely more on autonomous systems for military operations, defense planners may come to count autonomous systems and their available domestic supply of AI chips the way they once counted soldiers and the available domestic recruiting pool of military-age adults.  42  Downstream, this could help technologically advanced nations compensate for demographic challenges, such as aging populations and low birth rates, a situation the United States, China, Japan, Western Europe, and Russia all face to varying degrees.  43  Population trends continue to matter for national power-but AI technologies, like many other technologies of the past century, may further erode this importance.
 Industrious Scientists and Engineers Harnessing new technologies, both by developing technologies and accessing innovations created elsewhere, is an important means of growing power. Applications of AI can help in both areas, serving as a force multiplier on, and therefore increasing the importance of, productive scientists and engineers. Recently, for example, DeepMind's AlphaFold achieved breakthrough rates of accuracy comparable to experimental methods in the protein-structure prediction challenge known as CASP.  44  By obviating the need for experimental protein structure assessment, a skill-demanding and time-intensive procedure, AlphaFold represents a large augmentation of human scientists' biosciences research. In a different domain of research, modern AI applications are able to help with chip design.  45  Researchers have demonstrated a deep learning system capable of designing the physical layout of computer chips more effectively than human engineers.  46  Google has used this system to design its next generation of Tensor Processing Units (TPUs), the company's specialized AI chips. Likewise, rapid progress in machine translation, automatic literature review, and related tools means a given scientific discipline's state-of-the-art will become increasingly accessible and useful to well-organized groups of human scientists and engineers. Just as the printing press alleviated the need to travel from country to country to accumulate knowledge from different libraries, AI applications can lower the costs for researchers to access state-of-the-art knowledge in any field. There are three ways that modern AI applications will contribute on a large scale to scientific discovery and engineering invention: they will contribute directly to new discoveries and engineered systems, especially in areas that involve searches over large spaces in data or design;  47  automate the physical work of science and engineering, such as "self-driving laboratories" that robotically automate experimental laboratory work;  48  and make global scientific knowledge more accessible to humans, such as by extracting knowledge from millions of articles as well as from articles in many different languages.  49  Finally, there is an old debate about whether science advances most because of new ideas or new tools;  50  AI technologies appear able to contribute both. In the longer-term, AI may enable new and more creative forms of knowledge-generation that function as "pathfinders" for human brains, unlocking otherwise difficult-to-reach innovations. When AlphaGo beat Lee Sedol, its 37th move in the second game surprised human professionals. In the words of Go master Fan Hui, "It's not a human move. I've never seen a human play this move. So beautiful."  51  When AI behavior surprises us, we learn something new. Looking ahead, modern and future AI systems may be able to solve scientific puzzles that have thus far stumped humanity's best minds.  52  Just as railways advantaged nations with access to steel, it appears that AI tools capable of augmenting science and engineering work will favor nations with the best existing "resources" of industrious scientists and engineers. This trend appears likely to deepen the advantages of nations that host, or can attract, a disproportionate fraction of the world's best in those fields.  53
 ALTERED GOALS Finally, major innovations can alter state strategies, as different instrumental goals become more appealing for achieving a state's ultimate ends. The Industrial Revolution again provides a clear example. Before industrialization, conquering territory was a primary way that one group could increase its wealth and security relative to others.  54  During and after the Industrial Revolution, in contrast, states have been able to pursue these ends effectively by increasing productivity-as well as by gaining access to international trading networks and new technologies to enable further military and economic growth. Territorial conquest by states in the modern era is rarer for many reasons-but not simply because states have become more beneficent, instead because changes in technology have reshaped how they can best achieve their goals.  55  In short, major innovations can alter what long-term competitions in each era are fundamentally about. In the standard "ends, ways, means" trichotomy, this corresponds to ways. States have the same ends (security, wealth, prestige, influence, sovereign action), but the ways that competition is best pursued can change, such as through participation in globalized production chains instead of territorial conquest. With AI technologies, there are two worrying possibilities: a broad movement toward authoritarianism and the greater use of advanced forms of population-and economy-targeting information warfare.
 Social Control Temptations A technological innovation rarely tilts intrinsically toward "freedom" or "authoritarianism." It is possible, however, to try to discern how new technologies may affect current social and economic systems in the future. Especially in authoritarian states like China, AI technologies may provide elites with tools that reduce contradictions between maintaining power and promoting economic growth through free markets. By making authoritarianism appear more feasible, this may generate an "authoritarian temptation" for the many states with malleable governance systems. First, AI technologies are likely to reduce the costs of controlling populations under authoritarian rule. Automating mass collection, processing, and analysis of data is likely to decrease the marginal cost of controlling additional citizens, thus reducing the resources required to indefinitely sustain totalitarianism. With access to hundreds of millions of cameras, social media postings, bank accounts, automated analysis of emotions and sentiment, and other data streams, AI-empowered algorithms can perform much of the work previously done by secret police in pre-AI authoritarian states.  56  Automated surveillance methods are likely to scale more effectively than manual surveillance, which requires some amount of human labor per citizen to be controlled. For example, Lichter et al. analyzed official Stasi records from East Germany, finding that more than 1.5 percent of the population was either officially employed or unofficially used as informers by the secret police.  57  Beyond the quantity of people involved in human surveillance operations, automated surveillance may impose lower economic costs on a society than human surveillance.  58  On this matter, China appears poised to benefit from feedback cycles between AI deployment and data aggregation-the Chinese government is already using AI technologies to enhance population control, as well as to profile and control its ethnic minorities.  59  In these early efforts, the Chinese government is collecting large quantities of data, from facial scans to DNA; COVID-19 has only deepened PRC data collection on its citizens.  60  This data will help fuel new AI development for social control in Chinese firms. Future AI applications could, in turn, help China manage its data and drive more expansive collection, continuing the cycle. China will likely export versions of these capabilities to authoritarian governments globally in the 2020s and 2030s, as it has already begun to do. According to recent CSET research, since 2008, over 80 countries have adopted Chinese surveillance technologies.  61  These tools will help authoritarian governments worldwide deepen their holds on power.  62  Second, and more speculatively, AI progress may benefit authoritarian states by reducing the costs and consequences of state interventions into internal markets. The classic critique of centrally planning complex economies is that attempting to do so poses intractable optimization problems.  63  For many practical reasons, from human organizational factors to corruption, AI technologies are unlikely to change this. However, AI technologies could reduce, to some degree, the negative consequences of state interventions in markets. For example, AI applications may help gather and interpret the volumes of information necessary for more effective economic controls. An analogous effect is visible inside large firms in both China and the United States today: companies like eBay, Taobao, Amazon, and Uber apply machine learning to mine large volumes of sales data to better match demand and supply. Modern machine learning tools enable automatic pattern analysis, improved forecasting, and natural language processing for predicting demand and performing sentiment analysis. Google's "Smart Bidding," for example, uses machine learning to optimize conversions for ads; California uses AI to predict electricity demand, more effectively controlling the power grid and reducing blackouts.  64  Walmart's internal logistical management has analogs to a centrally planned micro-economy.  65  There are many challenges to using analogous tools effectively for state economic policy, perhaps most of all the variable goals of planners themselves. But these trends suggest national-level strategic planning may be able to benefit from better information by applying modern machine learning tools to data accessible by states. Leaders of authoritarian states like China may find themselves facing lower costs for sustaining domestic political and economic control; leaders of authoritarian-leaning states may find themselves handed these tools by China. The effects of AI on population control and state interventions in markets are not certain. In the near term, however, it seems likely that Chinese elites at least believe that AI may help them better control their society, and so too may elites in other states. be decisively influenced by the clash between AI influence systems, for example, China may determine its best bet for reabsorbing Taiwan is heavy investment in AI-empowered propaganda. Information attacks can also target economic systems and financial markets, especially AI systems associated with managing equities investments. An unintentional, early demonstration of this possibility occurred in 2013, when U.S. trading algorithms responded to disinformation posted by the AP's Twitter account after it was hacked.  66  Information warfare may be increasingly linked to economic warfare, not just political disruptions. Higher-end, AI-empowered information warfare is a more speculative, longer-term capability. Chris Wiggins has characterized current technical trends as enabling "reality jamming": the potential for synthetic, targeted, and optimized disinformation at web-scale.  67  In this future, current computational propaganda concerns are just the tip of the iceberg. The bigger issue is the potential for large-scale machine-generated information that is highly targeted at particular individuals or subpopulations, evolved to maximally shape particular behaviors, and potentially able to affect anyone with web access.  68  Leveraging these developments, governments may attempt to shape perceptions of other populations more frequently than in the past.  69  OpenAI self-censored full publication of its GPT-2 language-generation model in 2019, for example, because it was concerned that generating close-to-human text would enable nefarious actors to proliferate disinformation. It is easy to imagine states pursuing similar capabilities for their own ends.  70  According to recent CSET research, GPT-2's successor, GPT-3, may be especially potent at generating disinformation at scale when steered by a skilled human operator and editor, opening up the possibility of highly effective human-machine teaming.  71  These trends may pose challenges for democratic societies, though it is still too early to make clear judgments. Three unresolved questions exist today: First, if a long-term risk in authoritarian systems is intellectual conformity, an analogous effect in democracies may be mob majoritarianism.  72  This inherent challenge in democratic societies could turn out to be exacerbated by modern information technologies and make organizational reforms even more difficult. Second, more research is needed to understand the balance between democracies' ability to use disagreements and diverse information to advance new explanations and solutions, and the potential for information attacks to undermine political stability.  73  And third, most fundamentally, Western democracies, and particularly the U.S. system of government, are based on a foundation of individual freedom where individuals are the best judges of their own interests. It is not yet obvious how Western institutions will adapt to machines that can anticipate-or shape-individuals' own preferences, states, and choices better than the individuals themselves can.  74  In the context of international competition, leveraging AI technologies to alter target states' national priorities or political stability through information warfare would represent "winning without fighting" par excellence. In this evolutionary theory of technological competition, AI's effects on national power fall into three categories: new elements of power, changed factors, and altered goals. Exploring new elements required for successful AI adoption, such as compute and organizational adaptations, helps us understand when, how, and why some societies may be better positioned than others to benefit from major innovations. Similarly, the idea of changed factors helps focus on how existing elements of national power may have changing importance, such as population size and industrious researchers. Finally, thinking about altered goals of states in competition shows how major innovations can reshape the ways that states engage in competition, such as enacting new domestic political and economic controls and leveraging AI-enabled information attacks on other states' social and economic systems. This research offers a way to start thinking about these issues together, and hopes to spur new, wider thinking and work. Creating new conceptual tools for U.S. decision-makers and analysts to make sense of AI technologies' effects is vital to American prosperity. Over the long term, these technologies will create significant changes in U.S.-China competition. From this research, we see three early sets of insights into opportunities for U.S. leaders: The scale of possible impacts from major technologies is obvious: the United States benefitted greatly from growth connected to technological and economic changes in the 40 years from 1880 through 1920; and China has also already benefitted from a mix of technological and economic changes in its resurgence from 1980 through 2020.  77  Recent history demonstrates that getting technology right is critical for long-term national flourishing-and determining trajectories for the United States and China over the next 20 to 30 years. Can we sketch the longer-term future? Only speculation is possible today: Broad historical examinations tend to suggest that more successful societies present fewer obstacles to long-term change and, especially, limit the costs of intellectual conformity. They seek to maximize the benefits of pluralism, competition, and mechanisms to share, challenge, and supplement new knowledge.  78  A key challenge for China will be limiting the long-term costs of intellectual conformity induced by an authoritarian government. A favorable factor for China will be the dynamic organizations it has built over the last 20 years, which may remain able to adapt and benefit from organizational learning as the world continues to change over the next 10 to 20 years. In the longer term, however, continued evolution seems increasingly challenging for China under the CCP and absent substantial pluralism; many of its main challenges for net economic-technological growth are likely to persist, while the benefits of its dynamic organizations are likely to decline over time. A likely challenge for the United States will be institutional and organizational sclerosis, which will make organizational learning and adaptation challenging over the next decade. Interactions between AI technologies and democratic institutions increase uncertainty and may exacerbate these challenges. Weighing against these factors is Samuel Huntington's reminder of the United States' multidimensional sources of power and ability for self-renewal.  79  The most favorable factors for U.S. vitality and competition with authoritarian governments coincide with its enduring strengths: areas such as its cultural values and pluralism, overall approach to governance, and access to global talent.  80  In the longer term, the United States' central challenges appear more temporary, and its greatest advantages more enduring-a favorable outlook achievable with thinking and work today. It is useful to distinguish AI from autonomy. The former is defined above; the latter is best thought of as some degree of delegation of decision-making agency to another entity, which could be a human or a machine.  84  Systems can have neither, both, or one of these two things. For example, an autonomous military system can be unintelligent, as in the case of a landmine, or an intelligent system can support humans without autonomy, as in the case of an information system for a pilot. The 2010s were the third period of global excitement about AI. The first period occurred in the 1960s, centered in the United States and the UK, and the second period occurred in the 1980s, centered in the United States and Japan. Both periods were associated with significant investment and optimism for cascading breakthroughs in machine intelligence. Both periods were followed by "AI winters": periods of widespread divestment from AI R&D and the belief that earlier expectations had far exceeded reality.  85  The current period will probably be remembered as being centered in the United States and China, though with substantial activity in the UK, Europe, Canada, Japan, Israel, and South Korea. Since the 2010s, most excitement about AI has focused on machine learning (ML), and, within ML, mostly on applications of neural networks (deep learning). ML is a broad subfield of AI that centers on inference from data and overlaps substantially with statistics and optimization. "Neural networks" refers to a family of statistical models for extracting patterns from large quantities of data, originally inspired by the behavior of biological neurons. While the rediscovery and improvement of neural nets started the current AI wave in the late 2000s, specific trends over the last 20 to 30 years enabled the success of recent applications: global growth and diffusion of compute resources; large quantities of digital data globally; and the connection of these two by the global internet. For this reason, the foundation of modern AI advancements is often called the "triad" of new algorithms, compute resources, and data.  86  population, iron and steel production, energy consumption, military expenditure, and military personnel. (See: Singer, J. David, Stuart Bremer, and John Stuckey, "Capability Distribution, Uncertainty, and Major Power War, <PHONE>," in Bruce Russett (ed.) Peace, War, and Numbers, (Beverly Hills: Sage, 1972), pp. 1948, as well as. https://correlatesofwar.org/data-sets/national-material-capabilities.) More recently, Michael Beckley has argued that traditional measures of power conflate gross resources with net resources, and thus fail to account for a country's burdens in addition to its assets. Thus, he proposes the use of "GDP * GDP per capita.  "    INTRODUCTION 1 |* 1 AN EVOLUTIONARY THEORY OF TECHNOLOGICAL COMPETITION 2 | POWER AFTER AI: NEW ELEMENTS, CHANGED FACTORS, In this work, we use "artificial intelligence" to mean, as per the Defense Innovation Board, "a variety of information processing techniques and technologies used to perform a goal-oriented task and the means to reason in the pursuit of that task." See Appendix.

 • Thinking of long-term competitions in an evolutionary framework makes large, broadly-diffused technology changes akin to environmental shifts. Like a volcanic eruption or the start of an ice age, adaptations are valuable and some states will be better at adapting than others. It is useful to begin thinking about how AI technologies can create new elements of power, change the importance of existing elements of power, and alter the goals of states in competition. Getting a better sense of AI's effects in each of these factors will be critical for major powers. The United States has a number of opportunities: studying the approaches of other countries, especially U.S. competitors and medium-sized, quickly-changing countries; 75  developing strategies for global leadership in producing, using, and sharing compute resources; supporting development of AI engineering as a rigorous discipline in the United States and leveraging humans trained in it; continuing to push DOD and IC organizational reforms for how data is managed and leveraged; and leveraging AI tools, cross-training between AI and other disciplines, and high-skilled STEM immigration to access new breakthroughs in science and engineering more widely. • AI technologies may change not only what states can do, but also what they want. Major innovations can broadly alter intermediate, instrumental objectives that states pursue by making certain kinds of behaviors more valuable or less costly. This can drive dramatic changes in state goals and policies. The United States may look for new opportunities in technolo-For the United States, this means learning from its competitors without mirror imaging them and sharing insights with allies before assuming they should symmetrically match U.S. policies. Perhaps most significantly, it may also mean looking ahead to how AI technologies may affect the aims and interests of U.S. allies and partners. 3 Conclusions and Key Points I gy-related democracy promotion; shaping AI technologies themselves to Creating new con-favor democracies, such as by supporting development of AI technologies ceptual tools for U.S. with less dependence on centralized data; 76 and developing approach- es to more rapidly adapt social and economic institutions to "information decision-makers and analysts to make attacks" by AI systems. sense of AI technolo- gies' effects is vital to American prosperity. Over the long term, these technologies will create significant changes in U.S.-China competition. broad• Finally, effects of technological change can be highly asymmetric: new elements, changed factors, and altered goals may have very different manifestations in different countries.
 See "The Power of Nations: Measuring What Matters," International Security 43.2 (2018): 7-44. 8. The best overview of this quest is Ashley J. Tellis, Janice Bially, Christopher Layne, and Melissa McPherson, "Measuring National Power in the Postindustrial Age," RAND Corporation, 2000. 9. This statement includes whether such measures are used quantitatively to predict who wins a war, or whether war will occur, or whether settlement terms will favor one side or another. See: Ibid., 17. 10. For various discussions of this, see: Stephen Biddle, Military Power: Explaining Victory and Defeat in Modern Battle (Princeton: Princeton University Press, 2004); David A. Baldwin, "Power Analysis and World Politics: New Trends versus Old Tendencies," World Politics 161 (1979): 161-94; Jeffrey Hart, "Three Approaches to the Measurement of Power in International Relations," International Organization 30 (1976), 289-305. 11. Almond and Genco (1977) most famously made this point about how to think about political phenomena in general. See Gabriel A. Almond and Stephen J. Genco, "Clouds, Clocks, and the Study of Politics," World Politics 29.4 (1977): 489-522.


Abstract: The underwriter and the models -solo dances or pas-de-deux? 1 involved detailed conversations with, and questioning of, employees in various roles within the company.

Stuart Armstrong
Mario Weick
Anders Sandberg
Andrew Snyder-Beattie
Nick Beckstead

The underwriter and the models- solo dances or pas-de-deux? What policy data can tell us about how underwriters use models

Executive Summary Using a collection of data on some of the catastrophe policies written since 2006 by a major reinsurance organisation, this paper explores how tightly underwriters follow the models and under what conditions they deviate from them. The data included underwriter premium and LE (loss estimate), and sometimes included LE from up to four different models (AIR, RMS ALM, RMS DLM, and IHM -the in-house model). We analysed the data in order to see what could be said about the relationship between model LE and premiums (as well as underwriter LEs). Mimicking a common procedure in machine learning, the data was randomly divided into a training set and a testing set, allowing many different theories to be investigated on the training set without risk of overfitting and detecting spurious connections.
 Results There were three main results, all statistically significant and with large effect sizes: 1. The models gave good predictions as to what the underwriter premium and LE would be. In a regression test, 79% of the variance in the premium, and 88% of variance in the LE, was explained by variance in the models. In fact, most of this variance was explained by the mean LE of the four models -78% and 87% respectively, corresponding to correlations of 0.88 and 0.93. 2. As the modelled loses rose, underwriter estimates moved closer to those of the models. This was evident through a variety of measures: the underwriters would report using more models than otherwise, the underwriter LE would become more strongly correlated with the mean model LE, and they would have less extreme premium/ LE ratios (meaning that the LE information was being used more to fix the premium). It should be noted that one effect that we expected to find -that underwriters were more willing to follow the models if these were more closely bunched together -was not present in the data. It seems that the underwriter does not make much use of model spread.
 The role of underwriters As far as can be seen in the data, the underwriters premiums (and LE) were strongly correlated with the models' LE estimates. The higher the expected loss (as seen by model LE), the less likely the underwriters were to deviate from the models. That high correlation may seem to suggest a limited role to the underwriter. However, this conclusion is premature for several reasons. Most significantly, this dataset only included policies that the reinsurance organisation had actually written, so the role of underwriters in rejecting policies could have been very important. There are also issues of yearly variation and changing market conditions (2012, for instance, seems to be an outlier in many ways). This analysis also ignores the effect of underwriters negotiating and interacting with brokers -even if every good underwriter were to sign similar kinds of deals, this does not mean the underwriters were superfluous. It is significant that the models were more predictive of underwriter LE than of premium (which would be influenced by negotiation). Underwriters may also play an important role in correcting erroneous information in the policy, and making sure that the correct models were applied in the first place. Finally, there were no details of outcomes in the data (which policies led to pay-outs, and by how much?), limiting our ability to estimate underwriter expertise. Thus it is possible and likely that the underwriter played (or could play) a more synergistic role with models, focusing on quality control, market insights, and business relations. This thesis can be strongly questioned. First, the social role of the underwriter should not be underestimated: they are involved in negotiations with brokers, and require perceptiveness of the opinions and behaviour of the other market actors. Secondly, when the practical role of the underwriter was analysed in detail, some of the work involved coping with poor data quality and correcting model errors. 5 Though these tasks do not appear on the list of bottlenecks to automation, they are certainly tasks that cannot be easily automated, as they represent a failure of the automation process itself. Currently (and for the foreseeable future), only humans possess the skills to apply these kind of corrections, which often involve deducing what kind of errors have occurred or what kind of extra data could improve the situation. Thirdly, underwriters may be using strategic intelligence to select models and to maintain the sort of overall vision that could resolve larger systemic risks. 6 Therefore it would be incorrect to see the underwriters as necessarily in direct competition with the models, but as occupying (potentially) different and complementary roles. This was one of the approaches advocated in the "autopilot problem" paper: change the "pilot's" (or underwriter's) role.  7  Merely relying on effective automation technology can both weaken the skills of the human and make the whole endeavour vulnerable to situations where the automation fails for some reason. However, making use of particular human abilities to control or complement the automation allows for both greater performance and better robustness. For instance, though simple linear models outperform expert predictions in many domains, these simple linear models could only be constructed thanks to expert knowledge of the important factors.  8  Similarly, experienced underwriters are potentially aware of what information is relevant in a particular type of case -and whether such a case is at hand. The quality of expertise is highly dependent on features of the task, 9 rather than on features of the expert, so it is important that the task the underwriter performs be well designed to make best use of their human qualities (especially as expertise tends to be quite specific to the task performed 10 ). If this can be achieved, the underwriter and models of the future will amplify each other. We thus need to understand how underwriters currently interact with models. To this end, a major reinsurance organisation has made available several years' worth of records on policies priced by its underwriters and by its models so that a comparison could be made and the role of the underwriter teased out. The most useful features of this data were the premiums that were actually charged, the underwriters' Loss Estimate (LE), and the same LEs as given by the models. Using this data, this paper explores how tightly underwriters follow the models and under what conditions they deviate from them.
 Introduction The underwriters have traditionally been the key players in the insurance industry, making the final decision on any particular policy. It is their responsibility to negotiate a price and ultimately accept or decline to insure the risk. But recent years have seen the emergence of another key insurance player: computerised models that give their own estimation of risk, exposure, and other critical features of a policy. In CAT (catastrophe) insurance, these models are now used extensively by insurers, re-insurers, and regulators, and have underpinned the risk-linked securities markets. 1 In this new, model-centric world, what is the current role of the underwriter? More usefully, what will the role of the underwriter become? Some studies suggest that the underwriter is soon to be replaced by automation.  2  The Frey and Osborne study analysed current automation trends and concluded, based on O*NET data (an online job classification service developed for the US Department of Labor 3 ), that underwriting involved none of the skills estimated to be hard to automate such as manual dexterity, strategic intelligence or socially dependent tasks 4 (see Table  1 ).
 Computerisation bottleneck O*NET Variable O*NET Description
 Perception and Manipulation
 Finger Dexterity The ability to make precisely coordinated movements of the fingers of one or both hands to grasp, manipulate, or assemble very small objects.
 Manual Dexterity The ability to quickly move your hand, your hand together with your arm, or your two hands to grasp, manipulate, or assemble objects. How often does this job require working in cramped work spaces that requires getting into awkward positions? Creative Intelligence Originality The ability to come up with unusual or clever ideas about a given topic or situation, or to develop creative ways to solve a problem. Fine Arts Knowledge of theory and techniques required to compose, produce, and perform works of music, dance, visual arts, drama, and sculpture.
 Social Intelligence Social Perceptiveness Being aware of others' reactions and understanding why they react as they do.
 Negotiation Bringing others together and trying to reconcile differences.
 Persuasion Persuading others to change their minds or behaviour.
 Assisting and Caring for Others Providing personal assistance, medical attention, emotional support, or other personal care to others such as coworkers, customers, or patients.
 Results Three primary results were prominent in this initial analysis. First, variance in the model's LE explained the majority of the variance in premium (and in underwriter's LE). Second, underwriters tend to be conservative in estimating losses, more often setting expected losses above those of the models rather than below them. Last, the premiums (and underwriters' LE) moved closer to the models for more expensive policies. The causes and implications of these results are still uncertain -this preliminary analysis is only capable of identifying correlations, not causations. Note that the high R 2 need not mean that the underwriters are explicitly using the mean LE in their estimates -the models are highly correlated with each other, as can be seen in Table  2 . Thus many linear or quasi-linear combinations of models will be highly correlated with the models, with their mean, and hence with premium. Indeed, the correlation between premium and models is quite comparable with the correlation the models have with each other.
 Data and Methods The reinsurance organisation provided a large collection of policies with appropriate premium and LE information. Data was taken from two sources: a reinsurance system used to record catastrophe model outputs; and an underwriting system used to record class of business and other risk details. The data were compiled over several years since 2006, with modellers recording model outputs and underwriting teams recording risk details as business was transacted. The data included premium charged, underwriter LE, general location, year, and potential LEs from up to four different models. LE is the loss estimate -the mean amount of money that the underwriter expects their company will pay out on that policy. The more usual "loss on line" is the LE for a particular "layer" of insurance, divided by the limit for that layer. The four models were AIR Catrader (AIR), RMS RiskLink Aggregate level model (RMS ALM), RMS RiskLink Detailed level model (RMS DLM), and the organisation's in-house model (IHM). After cleaning the data and restricting to US/ Canada policies (where the models are the most reliable), we were left with a collection of 660 policies where all four models were used 11 -no further selection was applied to this set. It was decided to split the data into a training and a testing set. This allows "cross validation", where hypotheses are formed on the training set and tested on the testing set. 12 This prevents overfitting, in which hypotheses are tailored to narrowly to the data, modelling noise rather than signal.  13  The training set would be thoroughly analysed to generate hypotheses; these would then be tested for statistical significance on the testing set. The ideal size of the testing set is 1/3 of the original set.  14  As the splitting into testing and training was done before restricting down to the 660 policies, this subset ended up divided into 457 policies in the training set, and 203 in the testing set. A total of 32 hypotheses were formed on the training set, which were then tested on the testing set, and all were found to be significant at the 5% level 15 , even when accounting for multiple comparisons  16  . Most of these hypotheses were linear regressions, but other comparisons were made as well (see detail of results). Result 2: Conservative Pricing Underwriters tended to set their LEs somewhat conservatively. Underwriter LE were above the minimum model LE 94% of the time (as compared to below the maximum model 83% of the time). See Table  5  in next section for breakdown of this into highest and lowest quartiles.  22  This was especially interesting as the maximum value and the mean were both better predictors of underwriter LE than the minimum (see Figure  1  and Table  4 ). Thus, it seems the minimum value was a rough lower bound for underwriters, but that they used little information from it beyond this. This strong correlation could be a sign of an autopilot problem 20 , if underwriters put too much trust in the models. In the meantime, we can analyse how these correlations change over time. For example, Table  3  demonstrates how in 2012, the fit between premium and model LE declined, as compared to surrounding years. This was around the period when the controversial "RMS 11" windstorm model was in use.    First of all was the simple fact that they used (or at least recorded the use of) more models. The full US/ Canada data (6138 policies) was decomposed into categories dependent on the number of models recorded 23 , and the median premium and underwriter LE of each category was computed (see Figure  2 ). For illustration, the interquartile distance was added in the case of zero and four models. Though the ranges overlap, those policies which recorded two or less models clearly had lower LEs and premiums than those that recorded three models or four. Figure  2  -Median underwriter LE and premiums, depending on how many models were reported for that policy. The inter-quartile range for 4 models and 0 models is also plotted. To see this, the 660 policies with four models were separated into quartiles.  24  The correlation between the underwriter LE and mean, minimum, and maximum model LE was computed in each quartile (Table  5 ). The data suggests underwriters hewed closer to the model information in these quartiles. A last way of looking at the difference in behaviour, is to consider what happens when the underwriter has established their LE, and compare this with the final premium. The data indicates that for high underwriter LE 25 , the premium/LE ratio is likely to be less extreme that for low ones. The linear anti-correlation is weak (-11%), but the effect is clearer graphically (see Figure  3 ). Thus the premium is more influenced by LE estimates when these are large.  To establish the results overleaf, a total of 32 hypotheses were tested. The significance of each result was established using the testing set, with each p-value adjusted -increased 26 -using the Holm-Bonferroni method. See table 6 (where "HBp-value" is the Holm-Bonferroni 27 adjusted p-value). As can be seen, all hypotheses were significant at the 5% level, and all but two were significant at the 1% level (these significance levels were marked in bold).  Of these hypotheses, 22 were regression/correlations with premium or underwriter LE as dependent variable, and either all model LEs or mean model LE as independent variables. We should expect these regressions to show scale invariance to some extent: if all the model LEs double, then, say, the underwriter LE should also double as well. This means that we expect the residuals (the deviations of the underwriter LE from its "theoretic value" as predicted by the regression model) to double as well. Thus we expect residuals to be higher for high expected losses and lower for low expected losses: the data should be heteroscedastic. And indeed it is (see Figure  4 ). Excessive heteroscedasticity precludes the use of the standard F-test to determine the p-values for the model. Instead, we took the logarithms of all the variables, expecting that this would remove the scale variations in the residuals. The residuals that resulted were much closer to being homoscedastic (see Figure  4 ). Thus the p-values of these regressions and correlations were calculated using the log-log regression rather than the standard linear regression. These p-values were sufficient to show that all regression models were significantly different from the null hypothesis. All regression data given in this paper (the R2 and the correlation coefficients), however, came from a standard linear regression, as a linear model is more likely to be closer to the underwriters' behaviour than a logarithmic one. Generally, the logarithmic model had slightly lower R2 than the corresponding linear model, but the values were very close (and on occasion the logarithmic model had a slightly better fit than the linear model). Statistical testing and significance
 Discussion The high correlation between model-estimated LE and premium (and underwriter LE) may seem to suggest a limited role to the underwriter. However, this conclusion is premature for several reasons. For a start, this dataset is likely incomplete, as it had to be put together specifically, and many policies failed to record model estimates.  28  Most importantly, it only included policies that had actually been written; the role of underwriters in rejecting policies could have been very important. The overall result ignores the fact of yearly variation: in particular, 2012 fits very poorly into the general analysis. It is likely that underwriters were aware of changing market conditions (or changes to the models themselves) and were able to react to them accordingly in that year. This analysis also ignores the effect of underwriters negotiating and interacting with brokers. It is significant that the models were more predictive of underwriter LE than of premium 29 (which would be influenced by negotiation). Underwriters may also play an important role in correcting erroneous information in the policy, and making sure that the correct models were applied in the first place 30 . Finally, there were no details of outcomes in the data (which policies led to payouts, and by how much?), limiting our ability to estimate underwriter expertise.  31  Conversely, there could have been a lot of wasted effort on the underwriter's part. The four models are so highly correlated (see Table  4 ) that any attempts by the underwriters to strike a fine balance between them would likely have made little impact. Given all those caveats, the story that the data presents is clear. If this sample is taken to be representative, then there seems little difference between using models to estimate loss, and having the underwriters do the same. More importantly, a similar pattern is true to a lesser extent in premium, where around 80% of the variance in premiums is explained by the models in a completely linear fashion. The remaining 20% is unlikely to represent perfect performance on the part of the underwriter, free of noise and bias. Underwriters, like all humans, are subject to general biases that interfere with their performance 32 , some of which could have a specific impact on their job.  33  The important question is how biases balance against expertise within this 20%. We have done some preliminary exploration of these biases and counteracting expertise with an experimental pilot study.  34  In that study, the underwriters appeared less model-bound than in this paper: the actual conditions of work may play a large role in decisionmaking. A full analysis of the underwriters' work is needed if the aim is to develop expertise that complements models in a robust way. We believe that data analysis such as this paper, an understanding of the cognition of underwriting, and a systemic perspective on the inherent risks of outsourced cognition -whether from autopilot biases or model-induced correlations across the insurance market -can help develop practices that both reduce systemic risk and amplify human capacity. It will become more and more vital for insurance companies to record their own data as this one has, and to analyse it intelligently.  28  In many cases, it is likely that models were used but the data wasn't recorded.  29  The models explain 88% of the underwriter LE variance in the 2010-2013 period, but only 79% of the premium variance.  30  From conversations with people in the industry, this last effect is more likely to be a factor in insurance modelling than in reinsurance modelling, which seems to be a more mechanical process. 31 And even loss data in the short term is not enough to estimate true underwriter expertise, as many risk as of low probability/high return period, and wouldn't show up in the data.  32  The bias literature is vast, but Kahneman, Daniel. Thinking, fast and slow. Macmillan, 2011 provides a good overview, Gigerenzer, Gerd. "How to make cognitive illusions disappear: Beyond "heuristics and biases"." European review of social psychology 2.1 (1991): 83-115 provides a good critique, and Kahneman, Daniel, and Gary Klein. "Conditions for intuitive expertise: a failure to disagree." American Psychologist 64.6 (2009): 515 gives a good synthesis of some opposed views on the subject. 33 Beckstead, Nick. "Biased error search as a risk of modelling in insurance" in "Systemic Risk of Modelling." Joint Future of Humanity Institute-MS Amlin White Paper 3 (2014) Figure 1 - 1 Figure 1 -Underwriter's LE as a function of cat model LE. With a set of models, underwriters are unlikely to set LE below the lowest model, and tend to stick fairly close to the mean model estimate.
 23 5071 policies did not formally record a modelled loss, 27 had one, 85 had two, 295 had 3 and 660 had all four models. Dependency for Biggest Risks Premium and LE medians for varying number of models LE Furthermore, the correlation between underwriter LE and model LE increased for higher model LE.
 Figure 3 - 3 Figure 3 -Ratio of premium to underwriter LE plotted against underwriter LE.
 Figure 4 - 4 Figure 4 -Residuals for linear and log-log regression for underwriter LE versus all four model LEs.
 Table 1 O 1 *NET variables that are bottlenecks to automation, according to Frey and Osborne.
 Table 2 2 The premium is actually more highly correlated with the other models than the in-house model (IHM) is. Or, put another way, premium deviation with model LE is comparable to the noise 19 in the model LEs. Correlations between premium, model LEs, and mean model LE. Premium AIR LE RMS ALM LE RMS DLM LE IHM LE Mean LE Premium 1 0.825 0.808 0.796 0.852 0.881 AIR LE 0.825 1 0.909 0.864 0.878 0.966 RMS ALM LE 0.808 0.909 1 0.854 0.797 0.930 RMS DLM LE 0.796 0.864 0.854 1 0.758 0.915 IHM LE 0.852 0.878 0.797 0.758 1 0.937 Mean LE 0.881 0.966 0.930 0.915 0.937 1
 Though the controversy was mainly around European windstorms, not US/Canada ones, this could have had an impact on underwriter trust in models. Alternatively, the large losses in 2011 could have played a similar role.21     Year # of policies All Model LE to Premium R2 Mean Model LE to Premium R2 All Model LE to Underwriter LE R2 Mean Model LE to Underwriter LE R2 2010 74 91.32% 76.08% 91.65% 89.87% 2011 345 84.94% 82.01% 91.65% 90.94% 2012 145 75.69% 69.56% 80.13% 75.58% 2013 92 95.15% 83.22% 94.83% 88.18% Mean 86.78% 77.72% 89.57% 86.14% Combined 656 78.80% 77.55% 87.63% 86.95%
 Table 3 R 3 2 between premiums and Underwriter LE with models, by year. 20 Armstrong, Stuart. "The Autopilot Problem" in "Systemic Risk of Modelling." Joint Future of Humanity Institute-MS Amlin White Paper 2 (2014). 21 Aon Benfield. "Reinsurance Market Outlook." Aon Benfield Analytics (2013).
 Table 4 4 Correlation coefficients between underwriter LE and the mean, maximum, and minimum of the model LEs. Mean Max Min
 Table 5 5 Correlation coefficients between underwriter LE and mean model LE, maximum model LE, and minimum model LE, separated onto the four quartiles. 1st quartile 2nd quartile 3rd quartile 4th quartile Mean 0.449 0.233 0.598 0.876 Max 0.486 0.387 0.363 0.847 Min 0.231 -0.035 0.329 0.747 Premium/LE ratio vs underwriter LE Premium/LE ratio Underwriter LE
 Table 6 p 6 -values and Holm-Bonferroni corrected p-values (for multiple comparisons) for all 32 hypotheses considered.
			 The underwriter and the models -solo dances or pas-de-deux?The underwriter and the models -solo dances or pas-de-deux? 3
			 This was the judgement formed by three of the paper's authors during several periods of immersion at the reinsurance organisation, which
			 US Gov't Accountability Office, GAO-02-941, "Catastrophe Insurance Risks: The Role of Risk-Linked Securities and Factors Affecting Their Use" (2002).2 Frey, Carl Benedikt, and Osborne, Michael. "The future of employment: how susceptible are jobs to computerisation?" Oxford Martin School Working Paper (2013).
			 Or that, when it did involve these skills, that they were not fundamental to the job.4 The underwriter and the models -solo dances or pas-de-deux?The underwriter and the models -solo dances or pas-de-deux? 5
			 Of the form U i = β 0 + β 1 M AIRi + β 2 M RMS-ALMi + β 3 M RMS-DLMi +β 4 MGMi +ε i where U is the underwriters' LE, M X is the LE of the X'th model, and ε i is the error term for the i'th policy.18  Of the form U i = β 0 + β 1 M MEANi +ε i where U is the underwriters' LE, M i is the average LE of all four models, and ε i is the error term for the i'th policy.19  Using model LE variation as an informal measure of noise.
			 Picard, Richard R, and R Dennis Cook. "Cross-validation of regression models." Journal of the American Statistical Association 79.387 (1984): 575-583. 13 Hawkins, Douglas M. "The problem of overfitting." Journal of chemical information and computer sciences 44.1 (2004): 1-12. 14 Dobbin, Kevin K, and Richard M Simon. "Optimally splitting cases for training and testing high dimensional classifiers." BMC medical genomics 4.1 (2011): 31.
			 Benjamini, Yoav. "Simultaneous and selective inference: current successes and future challenges." Biometrical Journal 52.6 (2010): 708-721.6 The underwriter and the models -solo dances or pas-de-deux?The underwriter and the models -solo dances or pas-de-deux? 7
			 Note that we are using quartile in the sense of a set of data representing a quarter of the policies, not in the sense of the three values (lower quartiles, median, upper quartile) that divide the ranked data into those four sets.
			 The underwriter and the models -solo dances or pas-de-deux?The underwriter and the models -solo dances or pas-de-deux? 9
			 That is, the policies were ranked according to their mean LE, and they were split into four sets at lower quartile, the median, and the upper quartile. By an abuse of nomenclature, these four sets are also called quartiles.25  Since the underwriter would set their own LE before setting a premium, it is valid to use underwriter LE on the x-axis at this point.
			 p-values of the individual hypotheses were calculated separately, then the hypotheses were ordered by increasing p-value. These p-values were then multiplied by (32+1-r), where r was the rank of the hypothesis on the list. Thus the hypothesis with the lowest p-value had this value multiplied by 32, that with the next-lowest had its value multiplied by 31, all the way down to the one with the highest p-value, which was multiplied by 1. As long as these adjusted p-values were below the criteria of significance, then the familywise error rate would be below that.27 Holm, Sture. "A simple sequentially rejective multiple test procedure." Scandinavian journal of statistics (1979): 65-70.
			 The underwriter and the models -solo dances or pas-de-deux?The underwriter and the models -solo dances or pas-de-deux? 13

/Users/janhendrikkirchner/code/2022/10/alignment-research-dataset/align_data/common/../../data/raw/report_teis/MS Amlin White Paper The underwriter and the models- solo dances or pas-de-deux.pdf.downloadasset.tei.xml