Artificial Intelligence in Strategic Context: an Introduction

 Download as PDF

Introduction: AI Advances, Impacts, and Governance Concerns
Artificial intelligence (AI), particularly various methods of machine learning (ML), has achieved landmark advances over the past few years in applications as diverse as playing complex games, language processing, speech recognition and synthesis, image identification, and facial recognition. These breakthroughs have brought a surge of popular, journalistic, and policy attention to the field, including both excitement about anticipated advances and the benefits they promise, and concern about societal impacts and risks – potentially arising through whatever combination of accident, malicious or reckless use, or just social and political disruption from the scale and rapidity of change.
Potential impacts of AI range from the immediate and particular to the vast and transformative. While technical and scholarly commentary on AI impacts mainly concerns near-term advances and concerns, popular accounts are dominated by vivid scenarios of existential threats to human survival or autonomy, often inspired by fictional accounts of AI that has progressed to general super-intelligence, independent volition, or some other landmark similar to or far surpassing human capabilities. Expert opinions about the likelihood and timing of such extreme further advances vary widely.1 Yet it is also increasingly clear that advances like these are not necessary for transformative impacts – for good or ill, or more likely for good and ill – including the prospect of severe societal disruption and threats.
The potential societal impacts of AI, and their associated governance challenges, are in significant ways novel. Yet they also lie in the context of prior concerns with assessing and managing technology-related risks, which has been an active area of research and policy debate since controversies over technologies related to energy, environment, weapons, computation, and molecular biology in the 1960s and 1970s.2 This work has generated many insights into societal impacts and control of technology, of which two in particular stand out. First, the societal impacts of technology are not intrinsic to the technology, but emerge from the social processes by which technologies are developed and applied. It is thus not possible to assess or manage societal impacts by examining a technology divorced from its economic, political, and social context.3 Second, these linked pathways of technological and social development are complex and uncertain, so societal impacts cannot be confidently projected in advance, no matter how obvious they may appear in retrospect. There is thus a structural tension that hinders efforts to manage the impacts of technology for social benefit. The process of developing, applying, and reacting to any new technology both gradually clarifies its effects, and also builds constituencies with interests in its unhindered continuance and expansion. Efforts to manage impacts thus move from an early state in which they are limited in knowledge because the nature of impacts is not clear, to a later state in which impacts are clearer but politically difficult to manage.4 This paradox is not absolute or categorical, but does describe a real tension and a real challenge to effective assessment and management of technological risks – which is clearly applicable to efforts to manage AI risks today.5
While every technological area is in some respects unique, AI is likely to be more challenging in its potential societal effects and governance needs than even other contentious, high-stakes, rapidly-developing technologies. There are at least three reasons for this, rooted in characteristics of AI and applications that are likely to be enduring. First, AI is weakly defined. It includes diverse methods and techniques, derived from multiple prior areas of inquiry, which have fuzzy boundaries with each other and with multiple other areas of technological advance and challenge. The weak definition and boundaries of AI make it difficult to precisely localize and specify related objects of concern and their boundaries, and thus difficult to define regulatory authority or other forms of governance response. Second, AI has a foundational character. AI advances promise to transform, and interact strongly with, other areas of technological advance such as computational biology, neuroscience, and others. AI's foundational role suggests comparison to historical examples of transformative technologies on the scale of those that drove the industrial revolution – electricity, the steam engine and fossil fuels. Considered together, the diffuse boundaries and foundational character of AI give it a vast breadth of potential application areas, including other areas of scientific and technology research – raising the possibility of an explosion of AI-augmented research rapidly transforming multiple fields of inquiry. Third, many currently prominent AI algorithms and applications, particularly those involving deep learning and reinforcement learning, are opaque in their internal operations, such that it is difficult even for experts to understand how they work.6 Even distinguished practitioners of these methods have expressed concern that recent advances are unsupported by general principles, often dependent on ad hoc adjustments in specific applications, and generative of outputs that are difficult to explain.7 Two prominent commentators characterized the state of the field as alchemical.8
One consequence of these challenges is uncertainty over the causes and implications of recent large advances. Do they represent foundational advances that put general understanding and reproduction of intelligence within reach?9 Or do they just reflect the result of continuing advances in several parallel areas – data, algorithms, computational capacity – that are important in practical power, without necessarily representing major scientific breakthroughs or auguring imminent advances in dissimilar problems or reproducing general intelligence?10 Expert surveys estimating how soon such major landmarks will be achieved show a wide range of views.11
A second consequence is deep, high-stakes uncertainty about AI's societal impacts, risks, and governance responses – arguably even more than in other parallel areas of rapid, high-stakes, contentious technologies. This deep uncertainty is driven both by uncertainty about the rate and character of future technological advances in AI, and by uncertainty about how people, enterprises, and societies will interact with these rapidly advancing capabilities: how new capabilities will be used and applied; how people and organizations will react and adjust; how capabilities will further change in response to these adjustments; and how, and how well, societal institutions will manage the consequences to promote the beneficial, limit or mitigate the harmful, and decide in time – competently, prudently, legitimately – which consequences are beneficial and which harmful.
AI Impacts and Governance: Major areas of present inquiry
In the face of this deep uncertainty, current study and speculation on AI impacts and governance has a few salient characteristics. In some respects these are similar to characteristics often found in rapidly growing fields of inquiry and commentary, yet they also reflect the distinct, perhaps unique, characteristics of AI. In broad terms, the approach of many researchers and commentators to AI reflects their prior interests, concerns, capabilities, and disciplinary traditions. Given the diffuse and labile character of AI and its applications, this is both familiar and sensible. Indeed, it is a phenomenon so familiar as to be nicely captured by old aphorisms and folk tales. AI is the elephant, and we who are trying to reason through its societal impacts and potential responses are the blind men, each feeling around one piece of it.12 Or alternatively, we all come to the problem of AI with our various forms of hammers, so it looks like a nail.
Attempting to give aggregate characterizations of such a heterogeneous and rapidly growing field is risky, yet necessary. Much present commentary falls into two clusters, mainly distinguished by the immediacy of the concerns they address. The first and larger cluster examines extant or imminently anticipated AI applications that interact with existing legal, political, or social concerns. Prominent examples include how liability regimes must be modified to account for AI-supported decisions (e.g., in health care, employment, education, and finance), or AI-embedded physical objects that interact with humans (autonomous vehicles, robots, internet-of-things devices);13 racial, gender, or other biases embedded in algorithms, whether in high-stakes decision settings14 or in more routine but still important matters like search prompts or image labeling;15 defending privacy under large-scale data integration and analysis;16 and transparency, explainability, or other procedural values in AI-enabled decisions.17
The second cluster of current work concerns the existential risks of extreme AI advances, whether characterized as progressing to general superintelligence, a singularity beyond which AI controls its own further advances, or in other similar terms. Although this perspective is less frequent in scholarly work, it draws support from the argument that risks of catastrophic or existential consequence merit examination even if they appear temporally distant or unlikely.18 These prospects loom large in press and popular treatments of AI19 – to such a degree that researchers arguing for the importance of more immediate risks initially faced an uphill battle gaining similar levels of attention to these concerns.20
Between these two clusters lies a wide gap receiving less attention: potential impacts, risks, and governance challenges that are intermediate in time-scale and magnitude, lying between the near-term and the existential. Some scholarship does target this intermediate zone, typically by examining impact mechanisms that are of both immediate and larger-scale, long-term significance. For example, many studies aim to identify technical characteristics of AI systems likely to make them either riskier or more benign.21 In addition, certain domains and mechanisms of AI social impact – for example, livelihood displacement,22 social reputation or risk scoring, and autonomous lethal weapons – are of both immediate concern and larger future concern as applications expand. The broad diversity of methods and foci of inquiry in AI impacts is a sensible response to present deep uncertainty about AI capabilities and impacts, and there is great value in this diversity of work. This is a context where interdisciplinarity is at a premium, and the most significant advances are likely to come not just from deep specialization, but also from unexpected connections across diverse fields of inquiry.
The AI PULSE project at UCLA School of Law: A focus on mid-range impacts
In this busy, diverse, and rapidly growing space, the AI PULSE Project at UCLA Law is a newcomer. Like many others, we aim to advance understanding on the societal impacts of current and anticipated advances in AI, the good, bad, and dangerous; the causes and mechanisms of those impacts; and potential law and other governance responses to inform decision-making, both within traditional legal and regulatory settings, and in new institutional mechanisms and settings.
In looking for areas on which to focus our attention and permit us to make useful contributions, we have used two loose and provisional criteria. First, we have sought issues and questions that can effectively draw on our prior expertise in law and policy, and on prior experiences with other technology law and policy areas such as energy, environment, internet governance, and cybersecurity. We aim to attend carefully to points where scientific or technology matters interact strongly with societal or governance issues – not aiming to focus centrally on technology, which is not our comparative advantage, but rather to recognize and connect with needed expertise via collaborators. Second, we have looked for areas of potential importance that are receiving relatively less attention, and where there is less risk of simply re-treading familiar ground.
This orientation has led us to the intermediate scale of AI impacts, time horizons, and implications, as outlined above. We do not focus principally on immediate concerns already attracting substantial study and attention, nor on existential endpoints. This is not because we judge these uninteresting or unimportant – they are emphatically not – but because so much excellent work is already being done here. This intermediate range, roughly defined in some combination of time-scale and of intensity of impacts and potential disruptions, is unavoidably a bit diffuse in its boundaries. We characterize it by rough conditions that separate it from immediate concerns, and from singularity, existential, or ultimate concerns.
We propose one principal criterion to distinguish this middle range from the applications, consequences, and concerns that characterize the large volume of important work being done on current and near-term AI challenges. AI applications now deployed and in advanced development sit within the context of existing configurations of decision-makers with associated capabilities, interests, and goals. They are being embedded in commercial products and services marketed by existing firms to identified consumers and businesses. They are supporting, and may in some applications replace, human expertise and agency in existing decisions now taken by individual humans, in a wide variety of professional and employment settings – e.g., drivers, machine tool operators, pharmacists, stockbrokers, librarians, doctors, and lawyers. And they are similarly supporting, advising, and perhaps replacing current decisions now made by groups or organizations – i.e., actors larger than one person – but still recognized, abstracted, and sometimes held accountable as an individual, more or less human-like actor, such as corporations, courts, boards, offices, or departments.
But the fact that current AI applications are presently slotting into these existing decisions by existing actors is a social and historical contingency that reflects immediate opportunities to deploy, and sell, AI-infused products and services. There is no reason to expect that AI's capabilities, or its future applications, will necessarily follow the same patterns. The same characteristics that pose challenges to the prediction and governance of AI's societal impacts – its diffuse, labile character, fuzzy boundaries, broad connections to other technologies and fields of inquiry, and foundational nature – also suggest that it is capable of doing things of greater scale, scope, novelty, or complexity than any presently identified decision by a presently identified actor.
It is this greater scale of application, along with associated changes in scope, complexity, and integration, that we propose as the principal criterion distinguishing near-term impacts and governance challenges from medium-term ones. In this hypothesized mid-range, AI is, at a minimum, doing things that to some degree resemble functions being discharged by present actors, but which due to greatly expanded scale or scope are qualitatively changed in their impacts, for example by divesting current decision-makers of power or fundamentally transforming their aims and inter-relationships. Alternatively, AI might be doing things that are not presently done by any identified single actor, but by larger-scale social processes or networks of multiple actors and institutions – e.g., markets, normative systems, diffuse non-localized institutions, and the international system. Deployed in such settings, AI would take outcomes that are now viewed as emergent properties, equilibria, or other phenomena beyond the reach of any individual decision or centralized control, and make them subject to explication, intentionality, or control. Or as a third alternative, AI might be deployed to carry out actions that are not presently done at all, for various reasons – including that they are beyond the ability of any individual actor to imagine, perceive, or carry out, yet at the same time are not the objects of any linked systems of multi-actor decision-making. In any such settings, we expect the societal impacts and disruptions/transformations of AI, and the associated challenges of governance (indeed, the meaning of governance) to be profoundly transformed, in scale, meaning, and possibly also speed.
Yet this is not the singularity.23 We also distinguish this middle range from existential or singularity-related risks and by the limitation that AI is not self-directed or independently volitional, but rather is still to a substantial degree developed and deployed under human control. Of course, the practical extent of human control in specific applications may be ambiguous, and the details matter a lot. Moreover, as noted above, even with AI not fully autonomous but practically, or formally, under human control, there may still be transformative impacts, including vast public as well as private effects and the potential for large-scale disruptions and harms – in addition to the large benefits that are intended and anticipated.
This intermediate range is thick with potential areas and mechanisms of high-stakes AI impacts. In addition to those noted above, involving mechanisms of influence already operating but subject to transformation from increased scale and speed (e.g., livelihood displacement), there are multiple other possibilities. These are substantially more heterogeneous than those already evident in present practice. Even brief reflection suggests a wide range of potential AI applications, impacts, bases for concern, and associated governance challenges. We outline a few below. Many others, similarly plausible, could readily be generated.

AI as a manager and coordinator of economic functions at larger scale than the scope typical of current enterprises producing and selling goods and services;
AI as a component of, and to varying degrees supplanting, various forms and functions of state decision-making – legislative, executive, administrative, judicial, or electoral.24 Small-scale instances of this, and more ambitious proposals, already abound. Implications are profound, both for material outcomes, such as substance of decisions, character and quality of service and program delivery, efficiency and cost; and for legal processes, associated rights, and political principles.
AI as a disruptor of competitive relations, in commercial competition and other domains of competitive interactions, with the prospect of major shifts in the distribution of power among individual, commercial, public, and other institutions, perhaps also driving large changes in the meaning and determinants of such power.
AI as an enabler of increased effectiveness and new forms of influence over people in commercial, political, and other societal contexts. Early signs, in 2016 and since, of the potency of algorithmic targeting and tuning of messages for both domestic and international political campaigns suggest much broader and more potent tools of influence to come. These of course can include AI-empowered influence and manipulation for conventional commercial and political ends, perhaps allowing greater concentration of power than has been possible by other means: one major recent report identified "robust totalitarianism" as one of the major social and political risks of AI.25 But multiple other forms of influence are possible, including manipulation of behavior, preference, or values, with aims and effects ranging from the benign to the malign. Consider, for example, the possibilities of AI-enabled manipulation for purposes of healthier living, other forms of self-improvement, civic virtue, or conformity, docility, or subordination to the will of another; AI-empowered psychotherapy, new political or social movements, new religions (the first church centered on AI is already established),26 or cults.
AI as a scrambler of human individual and collective self-conception, including possibly undermining basic assumptions on which liberal capitalist democratic states and societies are founded – either truly altering the assumed conditions, or altering large-scale confidence or belief in them through increasingly powerful tools of deception such as "deep fakes."27 These shared assumptions, which include both positive and normative beliefs, exercise a powerful influence on the acceptance and viability of democratic government and market economies. The potential implications of their disruption, for practice of democratic government, and for social cohesion and identity, are profound.

These and similar potential intermediate-range impacts can be read to varying degrees as benign or malign. Many are most likely some of each. But they are clearly large and novel enough to require careful study and assessment, without a strong prior presumption that they predominantly fall one way or the other. Although it may be tempting to view any large change as harmful, at least initially, we do not presume that the societal impacts of AI will be entirely, or even predominantly, harmful. In fact, there are strong grounds to anticipate large benefits. Innovations in commercial settings generally pursue improved production processes or products and services that people want. We thus expect that the initial, direct effects of AI deployment will mainly be benefits, since they will be realized through voluntary commercial transactions.28 The axiom of revealed preference is not the last word on societal values, in view of imperfect knowledge and anticipation of individual welfare, identified pathologies of choice, emergent collective outcomes, constrained choices, and manipulation – but neither is it to be arbitrarily rejected. Moreover, even outside purely commercial transaction, a few areas of AI application hold clear prospects for large societal benefits, including medical diagnosis and treatment, scientific and technical research, and environmental monitoring, management, and restoration. Even when technical advances bring some harm – canonically, the harm that incumbents suffer when their business is disrupted by innovations – these local disruptions are often consistent with larger-scale economic and societal benefits.
Yet at the same time, there are well-founded grounds for concern. The greater the new technical capabilities and resultant transformations, the less the resultant disruptions are likely to be confined to the commercial sphere and the more they are likely to implicate larger-scale network and public effects, and political, social, and ethical values separate from those of the market. Moreover, many applications of AI will be in non-market contexts – whether in classic areas of state decision-making or something brand new. In these other contexts, the comforting invisible-hand assumptions about the consonance of individual self-interested choice and aggregate societal benefit do not apply. Individuals and firms developing and applying AI may be sensitive to these broader impacts, but lack the scale of view, knowledge, or authority to address them effectively. Doing so will require some larger scale apparatus, perhaps including deployment of state regulatory authority, perhaps involving multi-party collaboration among state and non-state actors, including enterprises, technical associations, educational institutions, research funders, scientific and professional associations, civil society organizations, and governmental decision-makers at multiple levels.
Even focusing on this middle range rather than more distant and extreme possibilities, it is difficult to do research about future conditions, risks, or requirements. Future events, conditions, and outcomes are not observable, except via proxies, trends, or enduring conditions and relationships in the present or past. Saying anything useful about future effects unavoidably requires speculative reasoning and acknowledgement of uncertainty, and also disciplining that speculation by reference to current knowledge. Present conditions and trends, scientific knowledge about enduring mechanisms of causation and influence in the world, and the properties and limits of current technologies, all provide useful inputs to reasoning about future conditions and possibilities, but with limits. Structured methods for exploration and contingency planning such as model projections, scenario exercises, and robust decision-making approaches can help stimulate the needed balance of imagination and discipline, but do not surmount deep uncertainties. These challenges are all particularly acute in reasoning through potential impacts, risks, and governance responses for AI, in view of the uniquely challenging characteristics of AI noted above.
One promising way to get insights into AI impacts and responses over this intermediate scale is – instead of focusing on present applications or the technical properties of algorithms and associated data and tools – to focus on decisions: the decisions to develop and refine capabilities, train and test them, and deploy them for specific purposes in specific settings. Focusing on decisions in turn implies focusing on the actors who make those decisions, whether individuals, teams, or private or public organizations, and the factors that influence their decisions: the interests they seek to advance; the capabilities and resources that characterize their decision choice sets and associated constraints; and the strategic environment in which they make these decisions, including interactions with other decision-makers with resultant emergent properties.
A survey of present AI applications and actors suggests a wide range of interests may be motivating development, adoption, and application decisions. The principal actors developing new methods and capabilities are private firms and academic researchers. But the firms are not typical commercial firms. Several of the leading firms are so big and rich, and so secure in dominant positions in current markets, that they are able to invest in speculative research some distance from commercial application.29 Even accounting for firms' ostensible profit motives, their need to recruit and retrain scarce, often idiosyncratic, top-rank talent may also shift their mix of interests to some degree, to include scientific interest, technological virtuosity, plus whatever other ambitions or commitments motivate this talent pool.30 Motivations become even more diverse on the international scale. Several market-leading AI firms are based in China, under varying degrees of government influence – suggesting a blurring of lines between commercial competition and state objectives, both domestic (securing and sustaining political control) and international (geopolitical rivalry, through multiple channels not limited to military). Moreover, not all the major developers are for-profit firms. One major AI enterprise is organized as a not-for-profit, with a stated commitment to development of AI capabilities for social benefit – although the practical implications of this different commitment are not yet fully evident.31
In considering medium-term possibilities for AI applications, risks, and other impacts, it is thus necessary to consider a range of interests and incentives: in addition to commercial competition, potentially important interests might include political competition through electoral or other channels; competition for fame, notoriety, or influence; pursuit of technological advance for its intrinsic pleasures and for professional status; and multiple forms of rivalrous or malicious aims, among individuals, firms, other organizations, and states.32 This broad list of interests implicates harm mechanisms associated with intentional design choices as well as unforeseen accidents, or design or application failures.
The papers collected here represent the results of an early attempt to examine AI impacts, risks, and governance from this actor and decision-centered perspective. They are based on a May 2018 workshop, "AI in Strategic Context," at which preliminary versions of papers were presented and discussed. In extending their workshop presentations into these papers, we have asked authors to move a little outside their normal disciplinary perspectives, with the aim of making their papers accessible both to academic readers in other fields and disciplines, and to sophisticated policy-oriented and other non-academic audiences. In the spirit of the larger-scale project, we also encouraged authors to be a little more speculative than they would normally be able to do when writing for their scholarly peers.
This collection includes seven of the resultant papers, spanning a broad range of applications, impacts, disciplinary perspectives, and views of governance. Sterbenz and Trager use an analytic approach rooted in game theory to characterize the effects of a particular class of AI, autonomous weapons, on conflicts and crisis-escalation situations. Ram identifies a new potential mechanism of harmful impact based on increasing pursuit of the technical capability, "one-shot learning." She argues that increased use of one-shot learning might exacerbate present concerns about both bias and opacity, and assigns a central role to trade secrecy in these harms. Grotto distills a set of concrete lessons for governance of AI by analogizing to a prior political conflict over another high-stakes and contentious technology, GMOs, drawing particularly on the variation provided by disparate policy outcomes in the United States and the European Union. Marchant argues both that effective governance of AI is needed and that conventional governmental regulation is unlikely in the near term, and instead proposes a soft-law approach to governance, including a specific institutional recommendation to address some of the most obvious limitations of soft-law governance. Moving from regulatory practice to political theory, Panagia explores the alternative ways algorithms can be regarded as political artifacts. He argues that algorithms are devices that order the world, in relations between people and objects and among people, and that this is a distinct, and broader, role than that conventionally proposed by critical thinkers of the left, who tend to view algorithms predominantly as instruments of political domination among people. Osoba offers a provocative sketch of a new conceptual framework to consider global interactions over AI and its governance, arguing that there exist distinct linked technical and cultural systems, or "technocultures," around AI, and that current diversity of technocultures is likely to be a persistent source of regulatory friction among regional governance regimes. And finally, Lempert provides a provocative vision of the capability of AI to drive large-scale divergence in human historical trajectories. Drawing on historical analogies explored in the scholarship of Elizabeth Anderson, he presents two scenarios in which large-scale AI deployment serves either to restrict or to strengthen human agency, and speculates on the technical structures of algorithms that might tend to nudge a world deploying them toward those two divergent futures.
Closing Observations:
These papers represent the first output from what we aim to make a continuing inquiry. In our view, they nicely illustrate what needs to happen to build a coherent and progressive body of work in a new and diffuse, but high-stakes area: targeting interesting and important questions and effectively deploying discipline-based concepts and frameworks, yet also striving for clear communication across disciplinary lines and accessibility both outside the field and outside the academy – while still maintaining precision. At the same time, they also illustrate the challenges facing this area of inquiry: its vast breadth, the diversity of relevant disciplinary languages and perspectives – and the difficulty of directing focus forward beyond immediate concerns, of being willing to engage in speculation yet keeping the speculation disciplined and connected to current knowledge and expertise. These are the aims and the challenges of the project, which we plan to further explore in additional workshops, collaborations, and publications.
By Edward Parson, Dan and Rae Emmett Professor of Environmental Law, Faculty Co-Director of the Emmett Institute on Climate Change and the Environment, and Co-Director of the AI Law and Policy Program, UCLA School of Law, UCLA Dept
Richard Re, Assistant Professor Co-Director of PULSE and the AI Law and Policy Program
Alicia Solow-Niederman, PULSE Fellow in AI Law and Policy, UCLA School of Law
and Elana Zeide, PULSE Fellow in AI Law and Policy, UCLA School of Law.
 The post Artificial Intelligence in Strategic Context: an Introduction first appeared on AI Pulse.


Bezos World or Levelers: Can We Choose Our Scenario?

 Download as PDF

Robert Lempert1
Artificial intelligence (AI) augurs changes in society at least as large as those of the industrial revolution.  But much of the policy debate seems narrow – extrapolating current trends and asking how we might manage their rough edges.  This essay instead explores how AI might be used to enable fundamentally different future worlds and how one such future might be enabled by AI algorithms with different goals and functions than those most common today. 2
AI machine learning algorithms now meet or surpass capabilities once regarded as uniquely human and will grow more capable over time.  The algorithms draw their power from their ability to learn from vast amounts of data and their own experiences. For instance, autonomous vehicles, while still imperfect, roam our roads each learning not only from its own encounters, but also from the data gathered by other such vehicles.  Amazon and other online services, learning from the choices of millions of customers, recommend books we might read and music we might enjoy. In 2016, the machine learning AI program AlphaGo beat Lee Sedol, the world's best human Go player, in a five match tournament, exhibiting strategies that displayed an astounding level of non-human creativity (Krieg, Proudfoot, and Rosen 2017). The AlphaGo version that beat Sodol learned its craft from records of previous human Go games. AlphaGo's even more powerful successor learned from millions of games it played against itself.
Such capabilities portend vast social, economic, and political transformations. Alluringly, we can imagine a future world of vast material wealth and convenience.  Freed from human error and shortcomings, services such as transportation and medicine become much safer and more efficient. Localized, on demand, customized 3D manufacturing satisfies human wants and reduces environmental footprints. Freed from the drudgery of work that the machines can better handle, people embrace more meaningful tasks.
Yet AI also has dystopian portents.  Most concretely, the technology threatens to destroy vast numbers of jobs.  By some reckonings, 40 percent of the world's jobs could be replaced in the next 15 years (Roose 2019).  AI may threaten privacy, as exemplified by the digital assistant devices that increasingly serve, and monitor, us in our homes.  The power and wealth flowing to the individuals and firms that successfully commercialize AI may also exacerbate the income inequality straining our society.
The policy debate has begun to engage with these challenges (Alden and Taylor-Kale 2018).  Some discussions focus on skill training to allow people to take new jobs.  Others suggest a universal basic income (UBI) to reduce the adverse consequences of unemployment.  Social insurance programs predicated on full-time employment might be expanded to include those in part-time jobs.
But such policy responses only nibble around the edges of an unfolding societal transformation.  Extrapolating current trends, we might imagine a world in 2050 with AI offering unparalleled convenience and widespread material comfort while making most people economically irrelevant and concentrating power and wealth in the small number of firms and individuals that create and own the machines.  No amount of job training or UBI would make such a world remotely similar to our own.
Envisioning scenarios offers one means to grapple with societal changes as fundamental as those augured by AI.  Scenarios represent "focused descriptions of fundamentally different futures, often presented in a coherent script-like or narrative fashion" (Schoemaker 1993), often crafted to inform decision making.  In this spirit, we might envision a scenario called Bezos World in which current trends unfold into a world with all the wealth, power, and robots concentrated into a few hands.  We can then posit other scenarios to help explore the extent to which Bezos World is foreordained, or whether we might imagine, understand, and influence the creating of alternative worlds that some people might find more to their liking.
The Levelers Scenario
Among its profound implications, the industrial revolution of the 19th and early 20th centuries reshaped the way citizens of liberal societies experienced agency and freedom. For many centuries, proponents of unfettered markets had also been natural advocates for popular sovereignty and economic and social equality (Anderson 2017).  Markets helped to shatter aristocratic hierarchies and government chartered monopolies.  In a feudal world, artisans labored for lords in a relationship of deference.  A market economy made the two more equal, since an artisan could choose his customers with a freedom similar to that of a lord choosing his vendors.  As late as the mid-19th century, proponents of freedom such as Abraham Lincoln could envision a world of independent proprietors, whose few employees would be young apprentices on their way to running their own small firms.
The industrial revolution sundered this connection by creating economies of scale that required vast enterprises to exploit.  These economies of scale generated immense material wealth, vastly expanding people's freedom as consumers.  The choices of food to eat, clothes to wear, places to travel, and (with electric lighting) when to be awake and active expanded greatly.  But the enterprises that swelled abundance and choice also generated new hierarchies of power and wealth.  In the new managerial capitalism, assembly line workers and office clerks labored for their bosses in a relationship of deference.  Industrial age production also required large agglomerations of capital, further expanding inequality among citizens.  In response, the U.S. and other industrialized countries transformed their governments and new non-governmental organizations emerged, such as labor unions, to more equitably share wealth and political power.  The resulting social contract generated history's greatest expansion of freedom and material wealth.
But this social contract is now unraveling.  As one important driver, new information technology threatens many jobs, and firms increasingly use technology to replace managerial capitalism with a new economy much less reliant on full-time workers.  For instance, organizations such as Uber foster a gig-economy in which firms contract with independent workers for short-term engagements, thereby capturing more of the profits and control of the workplace for themselves.  Uber has roughly 10,000 full time employees and in the US 750,000 drivers operating as independent proprietors.  While these drivers often enjoy the benefits of setting their own schedules, they exercise little control over their working conditions, have few benefits of employment, and exist at the bottom of an economic hierarchy which is investing heavily in automation to eliminate their jobs.
AI seems poised to exacerbate such trends.  Today's firms actively pursue the Bezos World scenario, deploying AI towards a future in which machines do all the work and technology is used to maximize power and wealth in the hands of the small number of people who own the machines (O'Reilly 2017).
We might, however, envision a purposely very different scenario.  Rather than using technology to automate away workers, this scenario envisions a world in which society uses technology to unwind the firm.  We might call this scenario Levelers, based loosely on a political movement of that name during the English Civil War, which was among the first to advocate markets as a means towards equality and popular sovereignty (Anderson 2017).  In this Levelers scenario, a combination of populist uprising and appropriate technology would establish a 21st century version of what Lincoln envisioned for his time – a society of prosperous, independent proprietors without large concentrations of wealth – a world with many Uber drivers, but no Ubers.  In this Levelers scenario, technology enables new kinds of work and is designed to spread power and wealth more equally across society.
Economists point to transaction costs to explain the necessity of firms (Coase 1960).  Many economic arrangements, for instance running a railroad (Chandler 1977) or the design and production of complex goods such as automobiles and airplanes, require webs of connections among people and capital stock too complex to be organized and managed entirely by market forces.  Prior to AI, these productive assets needed to be organized within managerial hierarchies associated with large stocks of capital.  Today, firms also create vast wealth from the symbiotic relationship among their centrally managed, proprietary data (often gathered from customers), the services they sell that enables them to collect the data, and the AI learning from that data in order to make the services more efficient and effective.
AI technology can reduce many transaction costs, which enables a reimagining of the firm.  Today, however, these capabilities are being used by firms to reimagine themselves for their own ends.  For instance, ride-sharing services such as Uber and Lyft use ubiquitous web connectivity, databases, and route optimization algorithms to shift most of their workforce to independent proprietors and then to machines.
In contrast, the Levelers Scenario envisions a future in which widely deployed AI provides great wealth and convenience, but with the power relationships of the Bezos World scenario reversed.  In the Levelers Scenario AI helps labor hire capital, instead of the other way around.  For instance, the firm Gigster currently uses AI to help its corporate clients efficiently identify ideal teams of temporary workers, thereby reducing the need to nurture well-known teams of full-time staff. 3 In the Levelers Scenario, small teams of people might join together to make a car.  Using a Gigster-like capability, they could find others with the skills they need and rent 3D manufacturing facilities to produce their design.  In the service sector, AI could help drivers and passengers use competing databases to find each other without the need for an Uber to own the network.  Achieving or maintaining the Levelers Scenario might require that a strong government use its authority to break up and prevent any large concentrations of economic power, such as those arising from data or network monopolies.  But supported by AI created for the task, the Levelers Scenario envisions by 2050 a gig economy of radical social and economic equality, a world with material abundance, choice, and convenience but without managerial hierarchies and without large concentrations of wealth and power.
How AI algorithms might enhance human agency
This essay offers the Levelers as a normative scenario, which describes one way in which the transformative power of AI might lead to a future that enhances rather than reduces human agency.  The scenario raises numerous questions, not least of which include the mix of government policies and social conditions necessary to bring such a future into being, the policies and conditions required to make such a future stable against the pressures that might undermine it, as well as the many potential unintended consequences that a world without many large commercial institutions might entail.  But this essay explores one particular issue – how AI algorithms with different goals and functions than those most common today might help enable the Levelers Scenario.
In thinking about normative future scenarios, a first step requires being explicit about what values constitute the good.  We focus here on the capabilities framework, which Amartya Sen (2009) developed and then used to support a framework for measuring and helping to bring about a more just world.  The capabilities framework offers an alternative to value systems based entirely on welfare economics.  Outcomes matter, but so does the process by which the outcomes are achieved.  For instance, choosing to spend a quiet day at home is different than spending a day under house arrest though the physical outcomes may appear similar.  A just society, as emphasized by the capabilities framework, enables individuals to make reasoned and consequential choices about their own lives, to act on those choices, and to evaluate the results in terms of their own values and goals, respecting the diversity of such goals and values.  In addition, a just society enables individuals to participate meaningfully in shaping their society, sharing in an informed and consequential way in choices made about its economic, social, and political attributes.
The Levelers scenario envisions widespread use of AI algorithms that enhance, rather than reduce, such human agency.  But many of today's most touted algorithms fall in the latter category.  They seek some best outcome and are less concerned with enriching the decision-making processes of the humans with whom they interact.  In operating on the world, such algorithms start with a clear set of objectives to achieve and a set of actions they can take.  They then assess a current situation, predict the consequences of various actions, and choose those actions that best achieve the objectives.  For instance, AlphaGo seeks to win a Go game, can make moves consistent with the game's rules, and each turn makes the move that most increases its chances of victory.  Autonomous vehicles seek to travel safely and efficiently to a desired destination; create an understanding of the objects around them; and each moment decide whether their objectives are best met by turning, accelerating, or braking.  The Amazon website aims to display those products to viewers that result in the highest probability of a sale to a satisfied customer.  While humans may set the objectives (e.g. the destination of the AV or a high sales volume), the algorithm aims to reduce human involvement in the choices that lead to those outcomes.  The algorithm's success is judged by the extent to which it enhances human welfare, based on the outcomes for at least some humans.
What would it look like for an algorithm to enhance human agency?  The algorithm would certainly help achieve welfare-improving outcomes.  But the algorithm would also help individuals gather the resources – including skills, capital, and material resources – relevant to their goals, team with appropriate human collaborators, help all the individuals involved make good choices on how best to coordinate their activities and deploy their resources in pursuit of multiple goals, explore how these goals might be expanded or modified in light of what is possible, examine all these steps from different vantages and points of view, and explain their choices to themselves and to others.  For instance, a Gigster-like algorithm that helps people to form teams could be designed, not to promote just efficiency and profit by adding skills to an existing organization, but to help people assemble new teams and networks around a common purpose, help them to assemble and manage the resources needed to achieve their objectives, and help the participants' understanding of their common and individual purposes grow and deepen as they worked together.
Algorithms do exist that help to enhance humans' ability to make reasoned and consequential choices about their lives.  This is most apparent in the field of decision support, in which algorithms, much less capable than today's AI, are used to help groups of stakeholders deliberate about contentious policy challenges such as improving the resilience of a community to climate change and then seek consensus on actions to address the challenge (Marchau et al. 2019).  Such algorithms are inherently multi-scenario and multi-objective, the former to reflect multiple ways to view and interpret the world, the latter to reflect alternative ways of judging outcomes due to different interests and different ethical frameworks (Lempert, Groves, and Fischbach 2013).  The algorithms, often adapted from those in the classification and robust optimization literatures, are configured to support what is called agree-on-decision analysis, because the analytics aims to help people with different objectives and expectations about the future nonetheless reach consensus on near-term actions. 4 In contrast, algorithms are more commonly developed and used to support predict-then-act analysis, which assumes that all the parties to a decision will accept a single, often computer-generated understanding of the future and then seek prescriptive recommendations from the computer on the best actions to take.
Such agree-on-decision algorithms are often embedded in a process called "deliberation with analysis" in which stakeholders deliberate on their objectives, options, and problem framings; algorithms generate decision-relevant information; and the parties revisit their objectives, options, and problem framings influenced by the algorithms information products (NRC 2009).  The process envisions that participants' understanding and views will evolve over time in response to interactions with each other and with scientific information.  In brief, the algorithms aim to facilitate a democratic process of social choice, in which diverse parties agree on actions based on a Habermasean discourse in which parties recognize the inescapable plurality of competing views; facilitate clear explication of reasoning and logic; and accept the legitimacy of multiple views.  The process, and the algorithms that support them, are judged not only by the welfare outcomes they help achieve, but also by the extent to which they empower the people involved in the process to make what they regard as meaningful choices about their lives and society.
Human agency-enhancing AI algorithms are clearly not on their own sufficient to bring about a Levelers scenario.  But it nonetheless remains useful to ask to what extent such algorithms could be designed to facilitate such a future?  It remains hard to know, in part because the requisite technology landscape remains under-explored.  Many people and institutions currently developing AI have incentives to reduce human agency, since excluding humans best serves the purposes of those most involved in designing the algorithms.  Firms' goals generally do not focus on the agency of their customers or workers.  Rather firms seek control over the latter and want their customers to make choices good for the corporation, not engage in self-reflection and enlightenment. Researchers seek objective truth as scientists and technological virtuosity as engineers, so seek algorithms that operate independently from human subjectivity and influence.
The experience with multi-scenario, multi-objective decision support may be, however, instructive.  Explicitly changing their goals from predict-then-act to agree-on-decision analysis enabled researchers and institutions focused on the latter to recraft existing classification and robust optimization algorithms, originally developed to operate independently of humans, and develop interlocking processes and algorithmic tools that enhance human agency.  While at best a necessary condition, and certainly not a sufficient one, decision support processes using such agency-enhancing algorithms do seem to help generate improved social outcomes (Knopman and Lempert 2016).
Moving towards a Levelers scenario?
If those seeking alternatives to the Bezos World scenario wanted to take near-term actions that might steer society towards a Levelers scenario, what might they do?
As one step, they might establish research activities and institutions that have the incentives and resources to develop human agency enhancing AI algorithms.  People might then launch pilot programs in a few sectors of the economy in which such algorithms might have the most success in helping people replace the firm rather than the firm replacing workers.  The government could institute policies that create space in the economy for such experiments to thrive, perhaps akin to the renewable portfolio standards in the energy sector.
The industrial revolution and its enabling technologies created vast material abundance but exacerbated tensions among the several dimensions of human freedom.  Today's AI augurs social transformations at least as profound. Much AI research, development, and deployment currently seeks to replace humans in pursuit of technological virtuosity and economic efficiency.  The Levelers, and speculation regarding the algorithms that might support it, are offered as one of many potential scenarios intended to help people systematically explore whether and how AI might be configured to facilitate a future in which machines collaborate with humans to enhance the latter's capabilities, agency, and freedom.
REFERENCES
Alden, Edward, and Laura Taylor-Kale. 2018. "The Work Ahead: Machines, Skills, and U.S. Leadership in the Twenty-First Century." In, New York: Council on Foreign Relations.
Anderson, Elizabeth. 2017. Private Government: How Employers Rule Our Lives (and Why We Don't Talk about It) (Princeton University Press).
Chandler, Alfred Dupont. 1977. The Visible Hand: The Managerial Revolution in American Business (Harvard University Press).
Coase, Ronald H. 1960. "The Problem of Social Cost." In, Classic Papers in Natural Resource Economics (Palgrave Macmillan: London).
Knopman, Debra, and Robert Lempert. 2016. Urban Responses to Climate Change: Framework for Decisionmaking and Supporting IndicatorsRAND. RR-1144-MCF
Krieg, Gary, Kevin Proudfoot, and Josh Rosen. 2017. "AlphaGo." In, RO*CO FILMS.
Lempert, Robert, David G. Groves, and Jordan Fischbach. 2013. "Is it Ethical to Use a Single Probability Density Function?" In, RAND.
Lempert, Robert J., Steven W. Popper, and Steven C. Bankes. 2003. Shaping the Next One Hundred Years: New Methods for Quantitative, Long-term Policy Analysis (RAND Corporation).
Marchau, Vincent A. W. J., Warren E. Walker, Pieter J. T. M. Bloemen, and Steven W. Popper. 2019. Decision Making Under Deep Uncertainty: From Theory to Practice (Springer) (in press).
National Research Council. 2009. "Informing Decisions in a Changing Climate." In, Panel on Strategies and Methods for Climate-Related Decision Support, Committee on the Human Dimensions of Climate Change, Division of Behavioral and Social Sciences and Education (The National Academy Press).
O'Reilly, Tim. 2017. WTF?: What's the Future and Why It's Up to Us. (Random House).
Pendleton-Julian, Ann, and Robert Lempert. 2019. "World Building Workshop on Technology and Work Worth Doing (Not Jobs) Post AlphaGo." In, RAND CF-398 (forthcoming).
Roose, Kevin. 2019. 'The Hidden Automation Agenda of the Davos Elite', New York Times, Jan 29, 2019.
Schoemaker, Paul J.H. 1993. "Multiple Scenario Development:  Its conceptual and Behavioral Foundation". In, Strategic Management Journal, 14: 193-213.
Sen, Amartya. 2009. The Idea of Justice (Belknap Press).
Shapiro, Ian. 2003. Moral Foundation of Politics (Yale University Press).
Wu, Tim. 2018. The Curse of Bigness: Antitrust in the New Gilded Age (Columbia Global Reports).The post Bezos World or Levelers: Can We Choose Our Scenario? first appeared on AI Pulse.


Autonomous Weapons and Coercive Threats

 Download as PDF

Governments across the globe have been quick to adapt developments in artificial intelligence to military technologies. Prominent among the many changes recently introduced, autonomous weapon systems pose important new questions for our understanding of conflict generally, and coercive diplomacy in particular. These weapons dramatically decrease the cost of employing military force, in human terms on the battlefield, in financial and material terms, and in political terms for leaders who choose to pursue conflict. In this article, we analyze the implications of these new weapons for coercive diplomacy, exploring how they will influence the course of international crises. We argue that drones have different implications for relationships between relatively equal states than they do for unbalanced relationships where one state vastly overpowers the other. In asymmetric relationships, these weapons exaggerate existing power disparities. In these cases, the strong state is able to use autonomous weapons to credibly signal, avoiding traditional and more costly signals such as tripwires. At the same time, the introduction of autonomous weapons puts some important forms of signaling out reach. In symmetric conflicts where states maintain the ability to inflict heavy damages on each other, autonomous weapons will have a relatively small effect on crisis dynamics. Credible signaling will still require traditional forms of high-cost signals, including those that by design put military and civilian populations at risk.
Increasingly, governments are using artificial intelligence technologies to revolutionize their military capabilities. In many ways, these technologies present the potential to transform the conduct of war and, in so doing, to alter the nature of state-to-state interactions. Prominent among the many changes recently introduced, autonomous weapon systems (AWS) pose important new questions for our understanding of conflict generally, and coercive diplomacy in particular. Automation, the most novel trait of these systems, allows states to deploy military force remotely at startlingly low cost. Automated weapons can enter enemy territory without endangering the lives of soldiers, maintain constant surveillance on important targets without risk of fatigue, and deliver deadly and highly precise strikes in an instant. Already, militaries employ remote-operated technologies to capitalize on similar advantages. As the introduction of automation streamlines and centralizes the planning and conduct of conflict, militaries will come to rely ever more on systems which build upon and extend these features. For example, U.S. military planning emphasizes the need to develop a "global surveillance and strike network" (GSS). This is expected to rely heavily on autonomous weapons. According to a key planning document: "while many elements of the U.S. would have important roles to play in a future GSS network, it would rely disproportionately upon air and maritime forces in general and unmanned platforms in particular."1
Most importantly, AWS appear to have dramatically decreased the costs of fighting a war. First, the ratio of capital to labor inputs for the conduct of war has drastically shifted. Although they require an investment cost upfront in their design and development, once built, these weapons impose minimal risk to the lives of their operators and require remarkably less labor for their effective performance. For the first time in history, the soldier who pulls the trigger need not be present on the battlefield. Consequently, AWS save on the enormous costs that have been spent in human sacrifice throughout the tragic history of conflict. This is important not only for the direct reason that it reduces danger to military personnel, but also, more broadly, because it offers a more "palatable" way to conduct war. As domestic publics grow increasingly distasteful of violence and casualties, leaders, particularly those held directly accountable to their publics, have found it increasingly difficult to pursue conflict.2 AWS offer a way to do so with minimal risk to the lives of soldiers, thereby greatly reducing the political and reputation costs suffered by states that choose to go to war. For example, while the US lost a full 10% of its aerial personnel in Vietnam, there was not a single pilot causality among a total of 568 drone strikes conducted by the US from 2002 to 2015.3 By avoiding large-scale causalities, leaders who engage in conflict can also avoid the domestic and international opprobrium that might otherwise impose heavy costs. Additionally, these systems are significantly cheaper to employ. While upfront investment costs are certainly not small, when viewed in comparison with the costs invested in other advanced manned-aircraft, missile technology, or nuclear capabilities, AWS are a relatively cheap technology.4 Thus, while revolutionary in a number of regards, automated weapons are particularly notable for the ways in which they decrease the costs of conflict financially, politically, and in terms of human sacrifice. Because these costs are extremely important to the existing understanding of coercive diplomacy, we explore how these weapons affect the use and perception of threats in international crises.
In this article, we analyze the implications of AWS for signaling between states. A well-established set of results from the literature on interstate bargaining asserts that credible communication between adversaries sometimes requires signals to be costly. All states, including those unwilling to actually follow through on a threat, stand to benefit from successfully coercing an opponent. As a result, states on the receiving end find it difficult to determine whether or not a given coercive threat demonstrates the sender's genuine intent and willingness to engage in conflict. When these signals are costly to make, however, those who are not truly resolved to follow through will be unwilling to undertake them, thereby allowing the receiver to distinguish between genuine and non-genuine threats. Key to this argument is that the costs of signaling must be sufficiently high such that bluffing states will prefer not to make an empty threat, even when it would be believed. Some rationalist theories have therefore argued that only high-cost signals which risk causalities or impose a hefty financial burden are sufficient. Given that the conduct of war with autonomous weapons involves little risk to human life and is drastically cheaper than ever before, are such high-cost signals still necessary? Is it possible to use these weapons to credibly signal intent and successfully coerce opponents when they are so remarkably cheap both financially and politically to deploy?
We focus on how a challenger's acquisition of AWS against a target without these capabilities will affect signaling. We argue that, where AWS lower costs of conflict, states are able to credibly signal intent with certain types of low-cost signals. When powerful states develop these technologies and face much weaker opponents, they will find that communicating credible intent requires less need to pay costs in the form of mobilizations, tripwires and the staking of bargaining reputations. In limited-war contexts where the anticipated costs of war are low, the mobilization of drones and other commensurately low-cost autonomous weapons systems can still send a credible signal of resolve. The new technologies influence not just the level of cost associated with credibility, however, but also the availability of certain other types of costly signals. This might suggest that states will turn toward costless diplomatic signaling in response, but we argue that power asymmetries will hinder the effectiveness of diplomatic signals as well. Rather, in crises over specific issues, states are likely to depend upon staking and defending reputations for resolve more than they did in the past; in some cases, this will imply a greater likelihood of conflict.
We argue further that while the costs of employing force with autonomous weapons may be dramatically low, the costs of war do not necessarily lower significantly. Where an adversary retains the ability to impose substantial damage on a state's society, the costs of war remain high. As a result, conflicts between AWS-endowed challengers and non-AWS-armed targets do not necessarily involve a large shift in relative war costs. We argue that threats backed by the mobilization of autonomous weapons systems are unlikely to be able to demonstrate resolve when the costs of war are moderately high. In these cases, the inevitable large societal sacrifice that would result from costly conflict entails the continued need for signals of resolve to be associated with high costs. We therefore expect that, when facing a relatively strong opponent, states with AWS capabilities will rely on certain traditional forms of costly signaling. These include voluntarily placing tripwire forces in harm's way and risking heavy casualties or investing in financially burdensome preparations.5
The Bargaining Model of War
To understand how autonomous weapons might change the conduct of international conflict, one must first start with an understanding of how crises have been conducted and understood in their absence. A fundamental dilemma in any given international dispute arises from two essential facts. First, no opponent can ever truly know another state's willingness to go to war. Second, unresolved states always have incentives to bluff, and issue empty threats, when doing so will successfully coerce an opponent. If a challenger knows its opponent will concede to a threat, it has enormous incentive to make this threat as convincingly as possible, even if it is not actually willing to follow through. This behavior, in turn, makes all target states dubious that a coercive threat against them is genuine. The crisis bargaining literature has long sought to address how states can escape this problem and make credible threats that successfully communicate a genuine intention to follow-through.
A core thesis of this literature contends that threats are only credible when they are costly to make. Typically, these game theoretic models involve two players, a target who receives some form of threat, commonly referred to as a signal, and a challenger who makes this threat. States make decisions rationally, based on cost-benefit calculations, and both seek to maximize their share of some good in dispute. Challengers who value the good highly or have a high probability of winning compared to the costs of fighting are resolved to fight should the conflict escalate that far. Threats made by resolved challengers are therefore genuine. On the other hand, when costs of war outweigh the benefits of fighting and winning the good, challengers are unresolved. While these states wish to possess the good, they are unwilling to suffer the costs of conflict required to fight for it. Threats made by unresolved challengers are therefore disingenuous bluffs that they will not see through. Generally, a target on the receiving end of a coercive signal has no way of knowing whether or not the threat is genuine because the challenger's value of the good or costs of war are private information, only known to the challenger. When threats are costly to make, however, unresolved challengers will be dissuaded from bluffing because they are unwilling to bear the costs required to send the coercive signal. By weeding out the unresolved bluffing states in this way, costly signaling allows a resolved challenger to credibly communicate its intent and allows the target to distinguish between genuine and non-genuine threats.
Traditional models of costly signaling have focused on two main ways to incur costs and communicate resolve. The first focuses on sunk-cost signals where the challenging state invests in efforts that are, in themselves, costly to undertake (Fearon 1997). Typically, these costs are borne through mobilization and preparation for war. By showing a willingness to undertake actions that, by their nature are costly to perform and difficult to reverse, states can credibly communicate their willingness to fight.6 This logic helps to explain why states often forfeit the benefits of a surprise attack and instead make overt preparations for war that are insufficient to give the state any realistic strategic advantage. For example, the U.S. has sent military assets into a tense region even though the addition of these assets does not greatly affect the balance of power. Further, investing in arms building and certain weapons technology may be more significant in its capacity to demonstrate a willingness to bear costs in preparation for conflict than in the ways these weapons influence military effectiveness and capability on the battlefield.
The second form of costly signaling operates through a so-called tying-hands mechanism. These signals do not impose any cost when they are made initially, but in the event that the issuer backs down they impose a heavy cost. In this way, they tie the hands of those who make them. Fearon argued that one prominent form of tying-hands signal comes through public statements and domestic audiences. According to this argument, domestic observers punish a leader for issuing threats, in so doing, "engaging the national honor," and subsequently backing down.7 Particularly in democratic states where leaders are held directly accountable to their domestic public through regular elections, leaders are thought to be highly concerned with public opinion and keen to avoid incurring any political and reputation costs for failing to follow through on a threat. As a result, leaders who are not truly resolved to fight will be unwilling to expose themselves to the risk of having to go back on their word and of incurring these so-called audience costs. In this way, verbal threats which in themselves do not carry a cost can still credibly demonstrate resolve when publicly issued.
Many real-world signals have both sunk cost and tying-hands aspects to them. Mobilization for war incurs costs that are paid whether or not the war is fought, making it a sunk cost signal. But if a significant part of the cost of conflict involves moving troops and military hardware into the theater of conflict, mobilization is also a tying-hands signal because it affects the relative value of choosing peace or war in the future.8 Tripwire forces are another example of a signal that contain elements of each type. In these cases, troops are deployed near the border with the target but are much smaller in size and capability than the target's forces. As such, these forces give little military advantage to the challenging state and would be easily wiped out by the target were war to break out. Any conflict in the region would essentially guarantee a substantial loss of the challenger's troops, and this, in turn, would quickly galvanize the challenging state and its domestic public into a full-scale war effort. The main purpose of deploying these troops is therefore not to gain any strategic advantage but rather to demonstrate willingness to enter conflict. Since this is done by influencing the relative political cost of entering the conflict or staying out in the future, this is a tying-hands signal. But since the risk of loss of life incurred and the costs of maintaining and mobilizing the troops must be paid regardless of whether conflict occurs, it is also a sunk cost signal.
Bargaining with Autonomous Weapons Systems
In contexts where revolutionary, low-cost military technology dramatically reduces war costs, how is credible signaling conducted? Given that autonomous weapons systems are exceptionally cheap to deploy, are leaders still able to forward deploy these weapons as a form of sunk-cost signal? Can leaders use AWS to send hand-tying signals when the use of these weapons draws far less public and political attention? We argue that the answer to these questions depends on the type of conflict involved. Autonomous weapons have very different consequences in asymmetric conflicts where relative costs of war are highly skewed than they do in more balanced, symmetric disputes.
It is important to note that the concept of symmetry of war costs between the target and the challenger is distinct from the resolve of the players. In asymmetric relationships, the challenger, regardless of its resolve to fight, faces very low relative costs of war. In symmetric relationships, however, the costs of war are fairly balanced between the target and the challenger, again regardless of the challenger's resolve. This is most easily understood through the simple equation relating the costs of war to the benefits of winning. Let the probability that the challenger wins the war be p, its value for the good in dispute be vc, and its costs of war be cc. We can then state, according to expected utility theory, that if pvc −cc > 0 the challenger is resolved to fight. Similarly, the cost-benefit calculus for the target is to be resolved to fight if (1−p)vt −ct > 0. In a symmetric relationship cc ≈ ct, and in an asymmetric relationship cc is much less than ct. In each relationship type, we can further categorize challengers as resolved or not by evaluating whether pvc −cc is greater than zero. In other words, two challengers might have exactly the same war costs cc such that the symmetry of the relationship to the target is the same, but different values of pvc such that one is willing to fight, and the other is not. In the following sections, we discuss how the introduction of autonomous weapons affects the dynamics of credible signaling within symmetric and asymmetric relationships separately. The challenge of credibly communicating resolve remains in each case since there are resolved and unresolved challengers in both, but the effects of AWS diverge.
Effective Signals in Asymmetric Relationships
When the costs of conflict for the challenger decrease, the costliness of the signal can be smaller as well. This can be seen in models analyzed in Fearon (1997). In both sunk-cost and tying-hands signaling models, the equilibrium cost that highly resolved states employ to credibly signal their resolve decreases as the costs of war decrease. With low war costs, states are more willing to contest issues, making their threats inherently more credible. The intuition of this result is straightforward. When it is not very costly for an opponent to follow through on a threat, the receiver is more likely to take it seriously. For example, imagine that a challenging state mobilizes autonomous weapons and readies them at the border with the target. In the asymmetric case, the opponent can keep war limited to only these low cost autonomous forces and would therefore suffer only minimal cost to follow on the threat and engage in conflict. The target, knowing how easy and cheap it would be for the challenger to follow through, is therefore very likely to take the threat seriously. In other words, because war itself is not highly costly, states do not need to demonstrate a willingness to bear large sacrifices in order to demonstrate their resolve to fight. Thus, it is not self-evidently the case that AWS make signaling resolve harder, as some have argued.9 Specific capabilities of AWS enhance these effects. Two of the most important are the ability to sustain operations at low cost and target objectives precisely, reducing costs that may arise from collateral damage (Zegart 2018).
Autonomous weapons do not merely reduce the costs of conflict, however. They also reduce the costs of deploying force abroad, an action that has traditionally been used as a sunk-cost signal of resolve to fight. Because AWS are costly to build but relatively cheap to mobilize and deploy, the cost and time required to deploy force abroad has been greatly reduced. Further, AWS capabilities are also replacing traditional force projection capabilities in military planning. For example, air defense and missile capabilities make more traditional surface ships and aircraft increasingly vulnerable.10 According to the so-called "Third Offset Strategy" unveiled by US Secretary of Defense Chuck Hagel in 2014, AWS technologies are the appropriate response to these "anti-access / area denial" or "A2AD" capabilities being developed by other powers.11
Because militaries have long used mobilization and preparation for war to send high-cost signals of resolve to fight, the replacement of traditional military hardware with AWS carries significant implications for signaling. This is particularly true for sunk-cost signals that are applied to particular foreign policy issues in shorter-term crises. To be clear, many sorts of sunk cost signals will remain, such as the production and forward installment of arms from AWS to missile silos. These actions, however, are typically tied to signals of general resolve to defend broad interests and regions, or sustain power projection capabilities and great power status. Such signals are less relevant to signaling about particular issues in a specific crisis between two opposing states. Deployment and mobilization decisions have often played important roles in signaling intent and resolve in such cases. As the costs of utilizing these traditional, labor-intensive signals grow comparatively larger and larger next to the costs of employing AWS, however, it is unlikely that states will continue to use them. As cheaper autonomous weapons technologies advance, traditional, high-cost signals like tripwires are likely to be seen as far too costly for the scope of the challenge. Game theorists will point out that these high-cost, traditional signals will always exist since, theoretically, even the burning of money may serve as a costly signal. In reality, however, no political leader has ever been willing to do this directly. Political leaders will always look for signals that achieve the desired end at minimal cost. What might these cheaper alternatives be?
One might expect that states would turn to closed-door diplomacy associated with costless signaling mechanisms. If AWS increase power asymmetries, however, many forms of diplomatic signaling will be unavailable as well. Diplomacy typically convinces by taking on some form of risk. For example, making a demand may risk that an adversary forms an opposing coalition, strikes first, or builds more arms than it otherwise would have. Similarly, insisting on a highly favorable outcome in negotiations may jeopardize the possibility of achieving an intermediate compromise, or may increase the likelihood that negotiations will break down leading to the outbreak of conflict. However, when the power differential between two states is extremely large, as it is in asymmetrical relationships, the adverse consequences risked through the conduct of forceful diplomacy disproportionately rest on the weak state. The powerful state, on the other hand, has little to lose because even if diplomacy fails entirely and war breaks out, its overwhelming power likely ensures that it will easily win the conflict. This can make it very difficult for powerful states to take on the risk necessary to facilitate credible diplomacy. Where AWS exacerbate already extreme power asymmetries, this problem is likely to intensify. The fact that the target of coercion is less willing to resist because of the sheer overwhelming power of the coercing state, or that the target's countering actions are less consequential, can imply that the most powerful states have less ability to use costless diplomacy to avoid conflict.12 The result is a "Goliath's Curse" – states with extreme power lose the ability to signal in certain ways.13
In cases where AWS create asymmetries of power and cost, a clear signaling possibility remains, namely, tying-hands signaling based on reputation.14 The essential task for the target in any signaling interaction is to form accurate beliefs and expectations about the challenger's true intentions based upon the information it gathers by observing the challenger's behavior. The game-theoretic crisis bargaining models often allow the challenger only one action to affect the target's beliefs. In the real world of international politics, however, these expectations and beliefs are not limited to this one-shot interaction in the context of a specific dispute. On the contrary, the target often has a long history of direct interaction with the challenging state and has observed its behavior with others over time and across diverse contexts. In this way, reputation and status produce expectations about a challenger's behavior generally which then shape expectations about its resolve to fight in particular circumstances. As such, developing and maintaining a certain reputation remains an important way to establish the credibility of one's stated intent in any given crisis. A reputation for following through on commitments, for resolve, for military strength and capability, for high cost tolerance, or for perseverance could convince a target that concession is optimal.
Compared to other forms of signaling, reputation signals of resolve have the property that they often increase the likelihood of conflict (Fearon 1997, Sartori 2005). Importantly, reputation is built upon a consistent pattern of behavior. Acting in ways that are inconsistent with or harmful to one's current reputation will alter it. This may impose costs later on as observers form new, different expectations. As a result, the incentive to maintain a reputation often provides an additional incentive for war, and one that can apply to both sides at once. Because all states desire a reputation for resolve, target states which might otherwise concede to the challenger's demand may be pressured to stand firm in order to maintain their reputation. Similarly, challengers who initially miscalculate such that the target refuses to accept a demand when the challenger had counted on a concession may wish to negotiate a different mutually-agreeable settlement. If backpedaling and renegotiating damages the challenger's reputation by making it appear pliable or irresolute in the face of defiance, however, the challenger may also prefer to go to war to preserve its reputation. That said, reputation signaling also reduces the states' need to fight by allowing actors to better communicate what they are willing to fight for. Nevertheless, the use of reputation signals instead of others usually increases the likelihood of conflict. Thus, as AWS exaggerate asymmetries of power and other traditional forms of signaling appear enormously costly in comparison, states may turn more and more to reputation as the basis of credibility and this, in turn, may tragically increase the incentives for war for both the target and the challenger.
Finally, autonomous weapons systems may alter the frequency and form of asymmetric conflict. A reduction in war costs corresponds directly to an increase in resolve through a basic cost-benefit analysis. Simply put, a state is more likely to find that the benefits of pursuing both conflict and coercive diplomacy outweigh the costs when these costs are lowered. This is of particular interest for borderline issues where the sudden reduction in war costs changes the state's preferences for pursuing the issue. Prior to the acquisition of autonomous weapons, a state would have seen the cost of pursuing these issues as greater than the benefits, and therefore preferred to let the status quo continue. After the introduction of autonomous weapons, however, a state might suddenly find the benefits outpacing the now drastically reduced costs and would thus choose to pursue the issue. An implication may be that states that gain autonomous weapons will begin to pursue a new set of issues for which they have relatively low resolve, but where the costs of war are even lower as a result of AWS capabilities. This may mean that AWS-endowed states issue more threats over relatively low-level issues compared to their conventionally armed counterparts. Future work should focus on exploring the observable indications of this change in behavior empirically. U.S. involvement in affairs around the globe, outside of a Cold War context, is often thought of as a new imperative resulting from the War on Terror, but it may also represent the decreased costs of involvement through drones in Pakistan, Afghanistan, Yemen and elsewhere. Indeed, existing literature suggests that as war costs lower, the probability of conflict increases.15 AWS capabilities will likely increase this trend.
In sum, states possessing AWS in asymmetric conflicts can credibly signal at low-cost, but this is far from the only implication of this technological revolution. States only need to demonstrate a willingness to engage in behavior that is proportionately costly to their anticipated costs of war. Because AWS dramatically lowers a challenger's war costs in an asymmetric conflict, low-cost signals are credible in these cases. At the same time, however, low costs of mobilizing for particular conflicts, combined with power asymmetries, imply that some traditional signals will lose their efficacy. Traditional signals,  particularly those involving the risk of fatalities, will be seen as imposing far greater cost than appropriate or necessary. As AWS further exacerbates power inequalities, strong states may find it more and more difficult to conduct some forms of closed-door diplomacy. As a result, states may become increasingly more dependent on the hand-tying signals of staking and preserving their bargaining reputations, even though this will sometimes lead to unwanted conflicts. Finally, powerful states with AWS capabilities are likely to face new incentives to pursue low-level disputes. Given that this technology makes it easier and cheaper for strong states to deploy force and make credible low-cost threats, we may observe an increase in the number of new issues pursued by strong states toward their weak counterparts.
Effective Signals in Symmetric Relationships
While possession of autonomous weapons drastically lowers a state's costs to deploy force abroad, it may not necessarily decrease the costs of war at home. It is important to remember that a state's costs of war are largely driven by the harm imposed by the opponent. An adversary may still employ tactics or capabilities that result in very costly destruction and loss of human life, and these costs are not likely affected by the introduction of autonomous weapons. In particular where intensive attacks occur on one's own soil, devastating destruction is likely not reduced by the development of AWS. We define cases where opposing sides possess a similar potential to impose substantial costs on each other to be symmetric relationships. No matter how advanced autonomous weaponry becomes, the basic fact remains that no defense system on the horizon can ever be impenetrable; for the foreseeable future, the proverbial AWS or missile "can always get through." Thus, even if only one side possess AWS, the basic fact that both retain the ability to impose heavy war costs on each other means that traditional, high-cost signals will remain important.
When symmetric conflicts threaten large-scale warfare and heavy war costs, a proportionately larger cost is necessary to demonstrate a willingness to bear these costs and the resolve to go to war. Take the extreme example of two nuclear powers: in this type of symmetric conflict, the costs of war loom extremely large. Mobilizing low cost autonomous weapons against a foe with the potential to unleash nuclear holocaust would be vastly insufficient to demonstrate a willingness to bear such costly devastation. Rather, in cases of large, symmetric war costs, tripwire forces will remain relevant and useful to challengers. As autonomous weapons drive the likelihood that conflict will require human sacrifice lower and lower, willingness to accept the risk of battlefield causalities will become an increasingly potent signal of resolve. Moreover, as AWS replace traditional labor-intensive force projection methods and make force projection less costly, the sunk-cost signaling aspect of military mobilization and preparation will increasingly fail to meet the high cost threshold necessary to credibly signal resolve. As a result, signals will become increasingly important when they demonstrate a willingness to suffer heavy human sacrifice, tying hands by ensuring that states would retaliate and fully engage in conflict.
Further, just as in the Cold War, a key set of questions surrounds when AWS-capable adversaries can keep conflicts limited to particular weapons technologies and levels of violence, or whether the risks of inadvertent wars will remain. These risks may be compounded by the speed of AWS technologies and the potential for first mover advantages in combat. Indeed, a crucial advantage of autonomous weapons is that they can observe and act more quickly than humans. This speed, combined with the potential for AWS technologies to initiate disabling strikes may be destabilizing. That is, AWS have the potential to make first strikes against powerful adversaries more attractive military options. As adversaries develop ways of interrupting communications systems, these weapons may become increasingly autonomous. In fact, credible deterrents may require systems that have the autonomy to act even after governments have been destroyed. These factors could increase the risks of crisis escalation and war. AWS systems interacting in unforeseen ways may even produce dynamics like the one that resulted in the 2010 "flash crash" in the U.S. stock market when high-frequency trading algorithms generated an extreme change in asset prices.16 On the other hand, autonomous systems, in reducing human error and being free from aspects of human crisis psychology, may also reduce the likelihood of conflict. These are critical areas for future research.
Conclusion
In this article, we have argued that autonomous weapons have important implications for threat-making and coercive diplomacy in certain types of conflicts. In asymmetric relationships where a strong state threatens a much weaker target, AWS capabilities can drastically lower the costs for the strong state to follow through on a coercive threat. As a result, high-cost signals traditionally used in interstate bargaining are no longer necessary to communicate resolve. This is simply because a credible signal must only demonstrate a willingness to bear costs to a commensurate degree with war costs. That said, autonomous weapons may reduce the effectiveness of signals which previously communicated resolve through costly mobilizations for war. Strong states may also find it difficult to turn to diplomatic solutions in asymmetric conflicts as AWS capabilities exaggerate power disparities, making it harder for the strong state to leverage its willingness to undertake risk to credibly communicate. This may precipitate a switch to establishing credibility through staking reputation, which will increase the incidence of conflict. Finally, by lowering the costs of war, autonomous weapons may spur states to pursue a new set of issues that were previously just below the cost-benefit threshold.
We see less near term change in the signaling dynamics of symmetric conflicts. While these weapons substantially lower the costs to deploy force abroad, they do not guarantee equivalently low war costs. In conflicts between states of similar power and capability, conflict entails heavy costs not only through the deployment of force, but also, more prominently, through the costs of destruction on society. Where an opposing state can impose heavy damage within one's border, war costs remain high in spite of AWS capabilities. As such, credible signaling in these disputes will still rely on a set of traditional, high-cost signals. In particular, as military technologies grow ever cheaper to deploy, willingness to risk fatalities and sacrifice human life is likely to become an increasingly salient, high-cost signal.
References
Banks, Jeffrey S. 1990. "Equilibrium Behavior in Crisis Bargaining Games." American Journal of Political Science 34(3):599-614.
Braumoeller, Bear. 2013. "Is War Disappearing?" Unpublished Manuscript.
Cheung, Tai Ming and Thomas G. Mahnken. 2017. The Gathering Pacific Storm. Amherst, NY: Cambria Press.
Dafoe, Allan, Jonathan Renshon and Paul Huth. 2014. "Reputation and Status as Motives for War." Annual Review of Political Science 17:371-93.
Fearon, James D. 1994. "Domestic Political Audiences and the Escalation of International Disputes." American Political Science Review 88(3):577-92.
Fearon, James D. 1997. "Signaling Foreign Policy Interests: Tying Hands Versus Sinking Costs." Journal of Conflict Resolution 41(1):68-90.
Fey, Mark and Kristopher W. Ramsay. 2011. "Uncertainty and Incentives in Crisis Bargaining: Game-Free Analysis of International Conflict." American Journal of Political Science 55(1):149-69.
Martinage, Robert. 2014. Toward a New Offset Strategy: Exploiting US Long-Term Advantages to Restore US Global Power Projection Capability. Center for Strategic and Budgetary Assessments.
Office of Chief Financial Officer. 2017. "Department of Energy FY 2018 Congressional Budget Request." National Nuclear Security Administration.
Office of the Undersecretary of Defense. 2017. "Comptroller/CFO, Program Acquisition Cost by Weapons System." United States Department of Defense FY 2018 Budget Request.
Pinker, Steven. 2011. The Better Angels of Our Nature: Why Violence Has Declined. Vol. 75 Viking New York.
Sartori, Anne E. 2005. Deterrence by Diplomacy. Princeton, NJ: Princeton University Press.
Scharre, Paul and Michael Horowitz. 2015. "Keeping Killer Robots on a Tight Leash." Defense One.
Schelling, Thomas C. 1980. The Strategy of Conflict. Harvard University Press.
Sechser, Todd S. 2010. "Goliath's Curse: Coercive Threats and Asymmetric Power." International Organization 64(04):627-60.
Serle, Jack and Jessica Purkiss. 2017. "Drone Wars: the Full Data." The Bureau of Investigative Journalism. URL: https://www.thebureauinvestigates.com/stories/2017-01-01/drone-wars-the-full-data
Slantchev, Branislav. 2010. Military Threats: the Costs of Coercion and the Price of Peace. New York, NY: Cambridge University Press.
Trager, Robert F. 2011. "Multi-Dimensional Diplomacy." International Organization 65:469-506.
Trager, Robert F. 2017. Diplomacy: Communication and the Origins of International Order. Cambridge, England: Cambridge University Press.
Zegart, Amy. 2018. "Cheap Fights, Credible Threats: The Future of Armed Drones and Coercion." Journal of Strategic Studies 0(0):1-41.
By Ciara Sterbenz, PhD Candidate, UCLA Dept. of Political Science and Robert Trager, Associate Professor UCLA Dept. of Political ScienceThe post Autonomous Weapons and Coercive Threats first appeared on AI Pulse.

Autonomous Weapons and Coercive Threats