Genetically Modified Organisms: A Precautionary Tale for AI Governance

 Download as PDF

The fruits of a long anticipated technology finally hit the market, with promise to extend human life, revolutionize production, improve consumer welfare, reduce poverty, and inspire countless yet-imagined innovations. A marvel of science and engineering, it reflects the cumulative efforts of a generation of researchers backed by research funding from the U.S. government and private sector investments in (predominantly American) technology companies. Though most scientists and policy elites consider the fruits of this technology to be safe, and the technology itself as a game-changer, there is still widespread acknowledgment that certain applications raise deeply challenging ethical issues, with some commentators even warning that careless or malicious applications could cause planet-wide catastrophes. Indeed, the technology has long been a fixture of science fiction, as an antagonist in allegories about hubris and science run amok—a narrative not lost on policy makers in the United States, Europe and elsewhere as they navigate the challenges and opportunities of this potentially world-changing new technology.
I'm referring to genetically modified organisms (GMOs), circa 1996, the year they entered the commercial market, and the biotechnologies used to produce them. By this time, the governance regimes in Europe and the United States for GMOs had diverged sharply, with Europe hardening as anti-GMO and the United States as permissive. The story behind why GMO policy in both places evolved the way it did, presented below, has important lessons for thinking about AI governance. Among other lessons, a consensus among technologists and other elites that a new technology is safe, and that its benefits outweigh its risks, does not guarantee its broader societal acceptance.
1. Comparative U.S.-European GMO Regulatory Policy
The World Health Organization defines GMOs as "organisms (i.e. plants, animals or microorganisms) in which the genetic material (DNA) has been altered in a way that does not occur naturally by mating and/or natural recombination."1 Instead, GMOs are the product of genetic engineering methods. Commercial cultivation of genetically modified crops began in 1996.2 By the end of that year, there were 1.7 million hectares of such crops worldwide, most of it in the United States.3 As of 2016, worldwide plantings reached 185 million hectares, but with 90% of the hectares confined to just five countries (the United States, Brazil, Argentina, Canada and India) and most of the remaining plantings (98%) in just five additional countries (Paraguay, Pakistan, China, South Africa and Uruguay).4 The vast majority of scientists and other mainstream experts in biotechnology assess that GMOs present no inherent risks to health or the environment compared to non-GMO alternatives. This view is generally reflected in U.S. government policy towards GMOs since the 1980s, and has helped make the United States the world leader in GMO crop production, with around 40% of the world's total plantings.5
European regulators, however, have taken a very different approach, one involving the application of a "precautionary principle." The concept of precaution is mentioned in Article 191 of the 1992 Treaty on European Union, and was first defined eight years later in a February 2000 communication from the European Commission.6 At its most basic, the precautionary principle holds that scientific uncertainty about risk due to insufficient or inconclusive data should not bar regulatory action when the activity or conduct in question implicates significant and irreversible threats to human health or the ecosystem. The precautionary principle forms the substantive basis for European regulator hostility towards the cultivation and sale of GMOs in Europe, where less than .1% of worldwide GMO plantings are located, most EU Member States have outright bans on GMO cultivation, EU law requires labeling of products containing .9% or more GMOs, and GMO imports are mainly used in animal feed.7
In developing the account of GMO governance that follows, I draw heavily on Pollack and Shaffer's definitive comparative study of American and European policies towards GMOs, which emphasizes "the ability of interest groups to capitalize on preexisting cultural and institutional differences, with an important role played by contingent events."8 As we shall see, American tolerance and European hostility towards GMOs were not inevitable: an observer in the early 1980s might well have predicted that American policy towards GMOs would trend hostile and that European policy would trend permissive.
A. Origins of U.S. Regulatory Policy
As Pollack and Shaffer document, U.S. regulatory policy towards GMOs reached an inflection point in the mid-1980s, as the technology appeared headed towards eventual commercialization. The decision for U.S. policy makers was whether to adopt a regulatory approach advocated by the Environmental Protection Agency (EPA) that emphasized the "newness" of GMOs, owing to the genetic engineering techniques used to create them, compared to products created through conventional, non-genetically engineered processes. EPA's proposed process-based approach to GMO regulation would have brought GMOs squarely within EPA's authority under the Toxic Substances Control Act to regulate "new" chemicals. EPA sought to distinguish "new" GMOs from "natural" organisms, and identified the process by which GMOs are created as the logical differentiator.
The EPA had allies for this process-based approach in Congress. For example, the then-chairman of the House Commerce Committee's Subcommittee on Investigations and Oversight, Al Gore, Jr., urged in 1983 for a precautionary approach to biotechnology. He asserted that "[w]hile there are certainly benefits to be reaped from this technology, I am concerned that we have a proper understanding of all potential environmental ramifications before a genetically novel organism is released, rather than having to learn about them after the damage has occurred."9
As late as 1984, the EPA was asserting such precautionary, process-based regulatory jurisdiction over GMOs, even as its partner agencies with regulatory jurisdiction over food and agriculture production, the Food and Drug Administration and the U.S. Department of Agriculture, began to emphasize product-based approaches to regulating biotechnology.10 As Pollack and Shaffer observed, "During the first half of the 1980s, therefore, it appeared as if the U.S. might take a highly precautionary, process-based approach to GMO regulation."11
The pro-business Reagan Administration weighed in decisively in 1986, when the White House Office of Science and Technology Policy (OSTP) issued a "Coordinated Framework for the Regulation of Biotechnology" that effectively resolved the process/product debate in favor of a product-based approach. The Coordinated Framework was the culmination of an interagency process led by OSTP. It essentially dismissed the proposition that products produced by biotechnological processes pose any inherent human health or environmental risks.12
The Coordinated Framework established the industry-friendly USDA as the lead U.S. regulator for introducing new GMOs into the environment, and confined the EPA's role to regulating GMOs with certain pesticidal traits. In doing so, the Coordinated Framework cabined the EPA's scope of influence on GMO regulation, and shifted the center of gravity in the Congress from committees with wide-ranging oversight responsibilities over technology and the environment to the Congressional agriculture committees. As the 1980s came to a close, congressional interest in agricultural biotechnology shifted from the early 1980s focus on risks, and towards the benefits.13
Meanwhile, the George H.W. Bush Administration preserved the White House's central role in setting policy on biotechnology.14 Finally, the Coordinated Framework's rejection of process-based distinctions set the table for the FDA's Statement of Policy in 1992, that it would approve new foods based solely on whether the product itself presented health risks. Using parallel logic, the FDA also decided that year that GMOs did not require any market approvals or labeling requirements.15 Thus, by the mid-1990s, just as commercialization neared, the three regulatory agencies with some claim of jurisdiction over GMOs were firmly in the camp of product-based regulation, while the political mood in Congress was mainly preoccupied with the upside of GMOs.
The net effect of these developments was to make it "more difficult for GM skeptics to use the existing regulatory and political framework to impede approval of GM crops and foods in the U.S."16 As Pollack and Shaffer conclude, "the U.S. system for biotechnology regulation has been determined almost exclusively by regulators operating under existing statutory authority, while the legislature (Congress) has played a relatively passive oversight role."17 In addition, regulation is primarily at the Federal level, limiting the ability of states to intervene. Advocates seeking change to U.S. policy towards GMOs would have to overcome a regulatory consensus codified in a policy established by a White House-led interagency process among the three key agencies with relevant responsibilities and/or persuade a Congress with countervailing voices to legislate. As a result, U.S. policy is relatively resistant to change in an anti-GMO direction.
It is important to recognize how contingent this policy outcome was on particular sets of institutional factors in place in the mid- and late-1980s. The Reagan Administration, under the leadership of EPA Administrator Anne Burford Gorsuch, had made a determined effort to weaken the EPA as part of the administration's overall deregulatory agenda upon entering office in 1980.
For example, the EPA's budget in 1981 reflected a 35% cut compared to 1980, and over the course of Reagan's first three years in office the agency shed nearly over 20% of its workforce.18 The agency's budget and workforce levels gradually recovered over the course of the decade, but during the crucial period of <PHONE>, when the die was being cast on the Administration's policy towards regulating GMOs, EPA was hardly in a position—fiscally or politically—to argue for an expansive approach to regulating GMOs that would have required substantial investments in EPA's investigatory, adjudicatory, and enforcement functions. Thus, the U.S. agency most inclined to closely regulate GMOs—and to advocate this approach within Executive Branch deliberations and externally to Congressional and other audiences—was marginalized and relatively weak during this formative period.
This was also the same period when Gorsuch's successor, William Ruckelshaus, transitioned the agency's posture towards assessing environmental dangers to the scientific risk assessment principles identified in the landmark 1983 National Academies study "Risk Assessment in the Federal Government,"19 which articulated an approach to risk management philosophically consistent with the risk-based, product-oriented approach of the Coordinated Framework. And the American biotechnology industry was already well-organized in the 1980s, with a willing partner in the deregulatory Reagan Administration for taking a relatively light-touch approach to regulating biotechnology.
Finally, the commercialization of GMO products in 1986, the year of the Coordinated Framework, remained several years in the future, which meant that the overall political salience of the issue was relatively low. Indeed, it is worth noting that during this formative period, and onward into the 1990s as GMOs began arriving in the marketplace, the United States enjoyed a period of relative calm in terms of health and safety scandals. The relatively marginal political salience of food and agriculture issues generally for most Americans throughout the decade, and the overall lack of major health and safety scandals relating to food, agriculture and the environment during the crucial period when GMOs were finally hitting the market, left few opportunities for policy entrepreneurs to seize in order to mobilize efforts to change U.S. policy. And as indicated earlier, even if opportunities had presented themselves, policy entrepreneurs would have limited avenues for their advocacy, due to the collective embrace by the three key federal regulatory agencies—FDA, USDA and EPA—of product-based regulation over process-based precaution and the presence of countervailing, pro-biotechnology actors and interest groups in the Congress.
These countervailing interest groups included the biotechnology industry and, increasingly, farmers who had planted GM crops and experienced such benefits as higher yields and reduced need for pesticides. As Pollack and Shaffer conclude, the decisions made in the 1980s around regulation of GMOs may have been contingent, but once made, those decisions initiated a certain path dependency that favors the status quo product-based approach to GMO regulation.20
B. Origins of EU Regulatory Policy
As noted earlier, Europe has taken a very different, and far more interventionist, approach to regulating GMOs. Initially, however—and in a somewhat mirror image to early 1980s skepticism about GMOs in the United States—much of European interest in biotechnology and GMOs in the early 1980s was motivated by a desire to support the competitiveness of the EU's biotechnology industry in the face of burgeoning competition from the American companies. And much as the Reagan Administration had leaned on the OSTP to lead the interagency process that yielded the Coordinated Framework in 1986, when the European Commission began exploring frameworks for supporting and regulating biotechnology in the early 1980s, it leaned primarily on the research-focused Science, Research and Development DG (DG Science) to lead the Commission's efforts.21
The center of institutional gravity with the Commission began to shift away from DG Science, however, "as biotechnology moved out of the laboratory to planting in crop trials and the marketing of GM seeds and foods." The anticipated commercialization of the technology created demand for legislation on agricultural biotechnology as the 1980s were coming to a close.22
The institutional actors within the Commission best suited to draft legislation, due to their mission and competencies, were DG Environment and DG Agriculture. DG Agriculture, however, was preoccupied by challenges associated with its lead role in Europe's Common Agriculture Policy (CAP). The CAP was one of the three core pillars of what was then still called the European Economic Community, and its framework of agriculture subsidies and other interventions had contributed to significant agricultural surpluses that required further policy interventions, consuming DG Agriculture's institutional bandwidth.23
That left DG Environment to take the lead on biotechnology regulatory policy, which it naturally framed as an environmental challenge squarely within its jurisdiction. In 1986, the Commission issued its first major policy document on biotechnology, a communication on biotechnology regulation, largely authored by DG Environment. The communication urged a European-level regulatory response to biotechnology. The communication was followed in 1988 by a proposal for a Directive on "deliberate release" of biotechnology products into the environment. The Directive emphasized a lack of scientific data about the risks of biotechnology and called for a regulatory framework requiring case-by-case assessments of new GMOs before they are released into the environment. In other words, the Commission proposed a process-based approach to regulating GMOs: bioengineered products are regulated within this schema because of the distinctive process used to create them.24
As Shaffer and Pollack note, "the biotech industry was not as well organized in Europe" during this critical period of policy development in Europe, "and was unable to mobilize political resources to prevent the process-based GM regulation that was framed in environmental terms."25 In addition, the European Parliament criticized the proposal as too lenient, firmly establishing it as a reliable, strongly pro-regulatory pole in European debates about biotechnology.
The European Council rejected most of the Parliament's proposed amendments in its final Directive 90/220, but changed the Commission's proposed decision rules to give Member States additional avenues to contest the Commission's decisions in two key areas: approving new GMOs for release into the environment and reviewing decisions by Member States to implement "safeguards" that "provisionally restrict or prohibit the use and/or sale" of a GMO from its territory.26 In 1997, the Commission issued a follow-on measure, Regulation 258/97, establishing labeling requirements and an approval process for "novel foods," including GMOs that had "not hitherto been used for human consumption to a significant degree within the Community." Unlike the approval process for new GMOs in Directive 90/220, which vested the Commission with the authority to approve or deny applications for new GMOs, Regulation 258/97 vested this authority in Member States.
Directive 90/220 and Regulation 258/97 established the initial institutional framework for how decisions about GMOs would be made in Europe. As Pollack and Shaffer summarize:
In comparison with the U.S. system, the regulatory structure established by Directive 90/220 and Regulation 258/97 was more complex, more decentralized, and more politicized than the U.S. system. It was more decentralized because of the key role of member states to start, oppose, and reject (through the imposition of safeguards) the approval of a GM seed or food. It was more politicized because of the involvement of politicians in the approval process. And it was more complex in that it created more institutional "veto points," where the approval of new GM varieties or the release and marketing of EU-approved varieties could be blocked.27
Thus, when contingent events occurred that boosted the political salience of biotechnology issues, interest groups had numerous options—the Commission, the Parliament, and the various political, regulatory and policy-making institutions within each Member State—for where to target their efforts to influence policy.
The debate about biotechnology in the 1990s occurred during a period of intensive economic and political integration in Europe. The growth and success of the single market throughout this period meant that Europeans were exposed to goods and services from other EU Member States—and any safety risks that may have accompanied them. This put pressure on Member States to act decisively when products from other Member States were exposed as having safety risks, and created a competitive dynamic between Member States and Brussels to demonstrate which was tougher on protecting public health and safety.
The late 1990s in particular were marked by a series of significant food and health safety episodes in Europe, including mad cow disease, asbestos problems at a major French university, and dioxin in Belgian chicken feed. These episodes undermined the public's confidence in the ability of their governments, of industry, and of the scientific community to understand and manage safety risks.28
Mad cow disease, or bovine spongiform encephalopathy (BSE), stands out in particular. Mad cow disease was first detected in cows in the UK in the early 1980s. The UK's Ministry of Agriculture assured the public and the European Commission that the disease did not pose a threat to humans. A significant outbreak of the disease among cattle occurred in <PHONE>, and the EU banned consumption of cattle sick with the disease. Over the course of the 1990s, as public concern within the UK grew about the health effects of eating beef from mad cow-diagnosed cattle, the government, its scientists, and the beef industry continued to reassure the public that the disease posed no meaningful threat to human health. It also persuaded the European Commission that it should not restrict the sale of British beef.
Thus, when the UK government announced in 1996 that ten people had been diagnosed with the human variant of mad cow disease, and that they probably contracted the disease by coming into contact with infected cattle, the announcement flew in the face of more than fifteen years of reassurances from British and European Commission regulators, and from industry, that the disease posed no threat to human health. The horrifying nature of the disease—inevitable death, preceded by graphic neurological decline—only amplified its impact. 1996 was also the year that a Scottish scientist touched off a lively debate about the ethics of biotechnology when he announced the cloning of a sheep named "Dolly," and the year that the WTO authorized Canada and the United States to implement retaliatory tariffs against EU farm products in response to the EU's ban on hormone-treated beef, triggering an anti-trade/anti-globalization backlash from farmers and activists such as Jose Bové, who also tended to be anti-GMO as well.29
1996 was a notable year in at least one additional respect: it was the first year of commercialization for GMOs. In April of that year, the Commission approved the sale of products containing a certain kind of bioengineered soy, despite objections from European Member States. When the product was imported into Europe later that year, it triggered protests by Greenpeace and other activist groups. As Pollack and Shaffer put it:
The close succession of these events illustrates how the popular understanding of GM products in Europe became associated with consumer anxieties related to food safety crises, distrust of regulators and scientific assessments, disquiet over corporate control of agricultural production, ethical unease over genetic modification techniques, environmental concerns, and anger over the use of international trade rules by the U.S. to attempt to force "unnatural" foods on Europeans.30
Public opinion in Europe, already tepid towards GMOs, soured drastically in the aftermath of these events. Reviewing Eurobarometer poll results, Gaskell and co-authors report that "[a]ll the EU countries, with the exception of Spain and Austria, showed moderate to large declines in support for GM crops over the period <PHONE>," and similar declines during this period for support of GM foods.31
The episode repeated in 1997, when the Commission approved another GMO over Member State objections. This time, a succession of Member States moved to block the product from their territory by invoking their right under Directive 90/220 to implement safeguards.32 Meanwhile, activists launched successful campaigns to pressure retailers and major European food processors to renounce the sale and use of GMOs,33 and in 1999 a coalition of Member States comprised of Denmark, France, Greece, Italy and Luxembourg succeeded in an effort to impose a de facto moratorium across the EU on approving new GMOs. The moratorium lasted for six years.34
2. Lessons for AI Governance
From this history, I draw five lessons that are relevant for AI governance. First, a consensus among technologists and other elites that a new technology is safe, and that its benefits outweigh its risks, is no guarantee of broader societal acceptance. Societal attitudes about the benefits and drawbacks of technology can change over time, with institutional, cultural, and contingent event factors enabling or constraining, as the case may be, how the institutions of governance adapt to, and even themselves shape, these attitudes. As Pollack and Shaffer demonstrate that with respect to GMOs, Europe's adoption of a process-based, precautionary approach to regulating GMOs was not inevitable. Had DG Agriculture assumed a greater role in shaping the Commission's initial communication on GMOs, European biotechnology interests been better organized, or European farmers seized on the benefits of GMOs in terms of higher yields and lower pesticide use and become a countervailing interest the same way their American brethren did, the institutional conditions under which regulatory and policy decisions were made in Europe might have taken a different, more permissive path.
Similarly, if Europe hadn't experienced a perfect storm of public health and safety crises in the 1990s, it is conceivable that these institutional conditions might have eventually yielded a more permissive approach to GMO governance. Consider the case of France—a country popularly associated with traditional foodways and cultural preservation, and an outsized driver of agricultural policy in Europe. Through the first half of the 1990s, France actually had by far the most GMO field trials in Europe, and ranked third in the world for such field trials during this period, behind only the United States and Canada.35 It even attracted forum shopping by GMO producers as the friendliest country in Europe for seeking regulatory approval for new GMOs, and was the only Member State in 1996 to vote in favor of approving a variety of genetically-modified corn.36 In the face of the health and environmental scandals of the late 1990s and the concurrent backlash to globalization, however, France abruptly reversed course and became a reliably staunch backer of aggressive regulatory action against the introduction of GMOs in Europe.37 And as noted earlier, in the early 1980s, an observer at the time might have predicted that the United States would be the one to adopt a skeptical approach to GMO governance, not Europe.
This could happen to AI, and not just in Europe, where politicians and regulators have already signaled a tepid view towards AI, citing concerns ranging from privacy to its effects on labor markets. The United States, as of this writing, is in the midst of what could turn out to be a significant shift in political and policy elite attitudes towards Silicon Valley. For the past couple decades, information technology has been celebrated by American elites as both a democratizing force for ordinary people to assume greater control of their economic and political fortunes, and as an essential enabler of "disruptive" innovation fueling economic growth and improved consumer welfare. Internet platforms in particular enjoyed strong presumptions of competency and good faith, especially on the American political left. These presumptions on the left, and the traditional anti-regulatory sentiment of the political right, formed the basis for what had been a tentative bipartisan consensus that technology regulation was, with limited exceptions, either premature or unnecessary, with arguments about the negative effects of regulation on innovation and investment typically prevailing over health, safety, and other equities.
A series of developments since Russia's weaponization of social media to spread fake news during the 2016 election, however, has given rise to a so-called "techlash" in Washington, with progressives and conservatives alike adopting a far more hostile, skeptical, and confrontational posture towards "Silicon Valley." Though the two sides have different critiques, it is safe to say that whatever presumptions of good faith and competence Silicon Valley enjoyed among policy and political elites in Washington before 2016 are badly damaged, and that the political antibodies against regulation are weakening. Of course, there is no guarantee that this will coalesce into a governing coalition with an affirmative policy agenda, but it marks a major shift in attitudes in the United States towards Silicon Valley.
Second, governance decisions made today about technology policy domains relevant to AI may have durable, long-lasting impacts on how policy evolves in the future. For both Europe and the United States, key decisions about GMO governance were made more than a decade before the technology was commercialized. And today, some 30 years after these initial decisions were made, these decisions continue to define the framing assumptions behind the two regulatory regimes. Europe's initial position on GMOs, for example, was contingent—had France's preferences towards GMOs up until the 1990s prevailed, the continent's regulatory framework might be more permissive. Once the EU enacted Directive 90/220, however, it created an institutional framework that proved highly prone to a race to the top (or bottom, depending on your perspective) as the varying actors involved in decision-making about GMOs sought to demonstrate their commitments to health and safety in response to contingent events.
For AI governance, policy and regulatory decisions about privacy, security, and safety seem especially important in establishing framing assumptions about how to weigh the costs and benefits of AI applications. Constituencies threatened by deployments of automated vehicles, for example, might meet arguments about the safety and efficiency benefits of automated vehicles with concerns about how personal data is collected and used by the vehicles. Already, the contours of the global privacy landscape are being formed, in these relatively early days of commercial deployments of AI. China's Cybersecurity Law went into effect in 2017, and considerable additional work is going into complementary initiatives, such as the Personal Information Security Specification and the Security Impact Assessment Guide of Personal Information. Europe's General Data Privacy Regulation (GDPR) went into effect last year, along with the Network Information Security Directive, with the latter likely to emerge as a focus of refinement and elaboration in the years ahead. In the United States, 2019 figures to be a seminal year for technology governance, with a tough new privacy law set to go into effect in California in 2020, creating a de facto deadline for the U.S. Congress to preempt it with Federal comprehensive consumer privacy legislation that could shape data privacy practices in the United States for generations to come.
Third, the intuition behind precaution—the notion that uncertainty about cause and effect attributable to data limitations should not bar regulatory intervention as a precautionary measure, especially when the negative effects may be substantial and irreversible—is a powerful rhetorical tool for justifying regulatory interventions in any domain with complex questions about risk. As Wiener and Rogers note in their comparative study of precaution in the United States and Europe, the precautionary principle is not a formal component of U.S. law, like it is in Europe, but there are regulatory actions in the United States that have been colored by shades of precaution, such as the USDA's early (and, as it turns out, prescient) import restrictions on British beef in 1989 due to concerns about mad cow disease and the FDA's ban on blood donations from would-be donors who had lived in the UK for a period of time.38 Similarly, while the precautionary principle is formally enshrined in European law, its application varies in Europe—not all regulatory domains are marked with the same degree of precaution as GMOs in Europe. As Wiener and Rogers conclude, "[s]ometimes Europe does take a more precautionary stance than the U.S., but sometimes the U.S. is the more precautionary regulator…Ultimately, neither Europe nor America can claim to be the more precautionary actor across the board."39
Certain deployments of AI may be especially vulnerable to application of a precautionary principle, in the United States as well as Europe, due to the challenges associated with explainability. Deep learning techniques, for example, rely on neural networks or similar architectures and large data sets to train an algorithm to perform a variety of complex tasks, such as driving a car. These algorithms are so complex that it may be impossible to isolate a cause or reason for a particular action. For those seeking to delay or interfere with deployments of AI, invoking precaution may prove to be a powerful strategy, particularly when the deployments in question implicate important societal values, such as privacy, security and safety.
Fourth, the institutional characteristics of how decisions are made about governance, such as the presence and configuration of veto points, establish the parameters around how and even whether interest groups can meaningfully influence policy making, especially in the face of contingent events. In the case of GMOs and Europe, contingent events had major effects in hardening European regulator sentiment against GMOs in significant part due to the institutional characteristics of the decision-making processes in Directive 90/220 and Regulation 258/97—namely, a set of processes that created multiple veto points. To this day, the European Union has approved just one GMO for cultivation in Europe, which four countries in Europe cultivate.40 Strong majorities agreed in the last Eurobarometer poll on this subject that GMOs are "unnatural" and disagreed that GMOs are safe and that development should be encouraged.41
With respect to AI governance, the fact that GDPR devolves enforcement authority to member states and their respective data protection authorities and judiciaries creates many opportunities for policy entrepreneurs to advance their preferences through enforcement actions and litigation. Similarly, the California Consumer Privacy Act of 2018 gives the State of California, through its elected Attorney General, enforcement authority over that law's requirements, and also establishes a private right of action for data breaches.
Finally, the nature and sequencing of the benefits and costs of AI deployment may also impact the resilience and adaptability of AI governance frameworks, especially in the face of contingent events. For example, if the benefits of AI are felt deep and wide by key stakeholders, when costs do emerge, there are more likely to be countervailing constituencies to offset advocacy by those feeling the costs. This was the case with respect to GMOs in the United States, where farmers adopted the technology relatively early, creating what Pollack and Shaffer quipped "facts in the ground."42 On the other hand, if the benefits of AI are distant or diffuse and thus diluted, or accrue to narrow constituencies, and costs emerge, the countervailing constituencies may be disorganized and/or weak. For example, it is conceivable that many of the initial society-wide benefits of automation will be diffuse—for example, a statistically lower risk of car accidents in a given population. The costs, however, may be concentrated in certain groups within that population, such as people whose professions as drivers are at risk due to automation. A constellation of costs and benefits along these lines could favor the emergence of organized political opposition to automation.
3. Conclusion
The transatlantic divergence over GMO governance ought to stand as a precautionary tale for technologists and policy makers that the benefits of a new technology seldom speak for themselves. Policy entrepreneurs, using contingent events and incumbent institutions, have a say too.
By Andy Grotto, William J. Perry International Security Fellow at the Center for International Security and Cooperation,  Research Fellow at the Hoover Institution, Stanford University
 The post Genetically Modified Organisms: A Precautionary Tale for AI Governance first appeared on AI Pulse.


The Algorithm Dispositif (Notes towards an Investigation)

 Download as PDF

Davide Panagia
Professor of Political Science
University of California, Los Angeles
<EMAIL>
How can we speak of algorithms as political?
The intuitive answer disposes us to presume that algorithms are not political. They are mathematical functions that operate to accomplish specific tasks. In this regard, algorithms operate independently of a specific belief system or of any one system's ideological ambitions. They may be used for political ends, in the manner in which census data may be used for voter redistricting, but in and of themselves algorithms don't do anything political.
In recent years, with the development of a field of research generally referred to as "critical algorithm studies," the sense of the politically neutral standing of algorithms has been placed under suspicion. Scholars from diverse fields – including cultural, film, media, and literary studies as well as race and ethnicity studies, sociology, philosophy, and the law – have begun to explore the extent to which, as socially effective structures, algorithms aren't merely abstract recipes for task completion, but they also create and exacerbate extant conditions of inequality, exploitation and social domination. An algorithm contains within it a "cultural logic" (as David Golumbia has named it) that carries with it, in its coded programming, a social imaginary of how things ought to be classified, organized, and operationalized.1 Taina Bucher, in her recent book If… Then: Algorithmic Power and Politics, also raises the issue of how algorithmic structures, once they are embedded in everyday life and practice, don't simply help us complete mundane tasks more efficiently, but also produce (and – crucially – reproduce) everyday conditions of perceptibility and intelligibility.2 In short, this growing area of research shows how algorithms are constituent participants in everyday life management. More than abstract practical instruments, they are life coefficients that, as the political geographer Louise Amoore has argued, are tasked with managing uncertainty through probability calculations and risk assessment. The end result is that not only present life, but future events too, may be managed and administered. As Amoore states, "the emphasis of risk assessment ceases to be one of the balance of probability of future threat and occupies instead the horizon of actionable decisions, making possible action on the basis of uncertainty."3 The shift that Amoore notes is an ontological one: uncertainty used to be a reason not to act, both morally and politically. We would wait to act until we had all the facts. But now, thanks to the deployment of probabilism in everyday life, uncertainty is a legitimate justification for preemptive action. That is, we act when we are uncertain precisely so as to mitigate possible outcomes.
As a contributor to this area of inquiry and research, I wish to raise some issues regarding the difficulties and challenges of thinking about the politicality of algorithms. Specifically, I wish to consider how an algorithm is a medium (first) and a political medium (second). My very rudimentary and initial notes towards such an investigation stem from a general frustration that begins with the following question: are all media political in the same way? Here's what I mean by asking that question. One has the sense, when thinking critically about the status of algorithms in everyday life, that if they are to be considered a political medium, then they operate no differently than a microphone, or a television, or a film. That is, their status as a political medium is located in their ability to transmit information. And as instruments of transmission, they are "influence machines."4 Thus, the effectivity and extent of their influence (otherwise imagined as their power of coercion) is what makes them political.
"Influence machine" is a term coined by the Viennese neurologist and psychologist Victor Tausk (<PHONE>) who, prior to his work in the field of psychoanalysis, was a distinguished jurist and journalist. Tausk defines the influence machine as a "delusional instrument" that "serves to persecute the patient and is operated by its enemies."5 Typically, patients describe such devices as possessing the following characteristics: 1. It makes individuals see pictures; 2. It manipulates the mind by inducing and removing thoughts; 3. It has physical effects upon the body that are beyond a person's control; 4. It creates strange and indescribable sensations – that is, new sensations that have yet to be named in language; 5. It produces physical and pathological maladies. According to Tausk, patients recount how these machines are immensely complicated, with many parts, and that they operate by means of obscure constructions. They are, to use modern parlance, "black boxes" – devices that operate effectively but are also fantastical. Finally, as Tausk accounts for it, the influence machine is perceived by the patient as a "hostile object"6 or a "diabolical apparatus."7
When we consider algorithms critically and reflect on their status as political media, we tend to treat them as influence machines in the Tauskian sense. That is, the critical paradigms we deploy to analyze the status of algorithms carry within their critical imaginary an account of algorithms as influence machines, hostile objects that manipulate mind and soul, not to mention the body. Hence the indisputable persuasiveness of the "black box" metaphor. In part, this treatment of algorithms arises from a characteristic of the dominant critical apparatus throughout the humanities and social sciences, as well as critical legal studies, that considers the task of criticism to be one of negating various forms of structural domination through the exposition or the unearthing of the mystical operations of power that sustain and proceduralize practices of subjection. In this regard, the image of the Tauskian influence machine is both normatively and conceptually provocative and helpful to our critical investigations. This, because that image corresponds to our sense that domination operates through channels of coercive influence, as Thomas Hobbes reminds us in his Leviathan when he describes human psychology as inclined to limit the freedom of others for the purposes of self-aggrandizement.
In recent years, scholars have developed an alternative, and compelling, account of criticism that isn't reducible (but is also not adverse) to the view of criticism outlined above. This novel approach to criticism is more experientially focused – that is, it looks to activities, practices, and actions – rather than ideational specters. As the literary scholar, Toril Moi, accounts for it, "actions aren't objects, and they don't have surfaces or depths."8 This view of criticism is less concerned with unmasking underlying structures of domination, and thus imagining that there exists a hidden world of power beneath the surface of experience. Rather, it considers experience as its starting point. In this respect, it is a radically empiricist mode of criticism that does not depend exclusively on the cognitive expertise of the critic to see things that others, uninitiated in the epistemic ambitions of a specific school of criticism, cannot.9 If a task of criticism is to develop an understanding of what something does, and how, then treating the doings of technical objects as if they only perpetuate the operations of domination seems to go against the idea of an activity as an embodied practice. This doesn't mean that activities and practices, including the political effectivity of technical objects, are transparent or self-evident. It is, rather, to treat practices and activities as things done in the world and not merely as delusional, automated habits like those characterized by Tausk's influence machine.
What does this alternative approach to the practice of criticism mean for the political study of algorithms? It means that alongside our understandings of algorithms as complicit in ideological domination operating along the same lines, and within the same register, as other media like television or film, we also consider algorithms in terms of their technical milieu and, crucially for my purposes, we examine the forms of participation they enable, disable, constrain, and proliferate.10 By "participation" I mean something like the ways in which algorithms take part in everyday life. In short, the political study of algorithms that I am proposing looks to the ways in which new forms of relationality are introduced in a specific lived context, and how extant or already existing modes of association are reproduced or rearticulated within that same context.
How might this be understood as a specifically political form of criticism? Politics (as I propose to analyze it – though, of course, not just me) isn't merely the exercise of domination (as it has been classically defined), but is fundamentally a pluralist activity for the creation of value through the forging and fomenting of relations between peoples, things, places, and times. Politics exists when things exist in relation to one another, and this fact of relationality is itself based on the sense that our individual and collective worlds are constituted by a plurality of beings, both human beings and non-human objects. This fact of pluralism – of there being not just something rather than nothing (as Plato had famously noted), but multiple somethings (or what the philosopher William James calls the pluriverse) – creates the possibility of relationality and hence, of things and people coming together and wrenching apart. In short, relationality creates worlds. As James affirms, "knowledge of sensible reality thus comes to life inside the tissue of experience. It is made; and made by relations that unroll themselves in time."11
In this respect algorithms are political because a fundamental function of the algorithm is to generate world-making relations, and what seems to me to be of central political import are the experiences of relationality that algorithms generate. Consider, in this regard, something as basic as a sorting algorithm like the purchasing recommendation algorithm on Amazon.com. Anyone who has shopped on Amazon has experienced both the frustration and the excitement of these recommendations. And clearly, there is an element of the influence machine built into these sorting mechanisms: through the correlational realist magic of artificial intelligence, we receive a suggestion about how to extend (or reproduce, or replicate, or alter) our experiential pluriverse. The algorithm sorts our previous views, purchases, and (crucially) our attentions (not just mine, by the way, but those of all who have attended to the same object) in such a way as to generate an expectation of future taste as invested in this other (perhaps previously unimagined … by me) object of enjoyment. That the magic of correlation functions within a capitalist climate of profit maximization is surely a contextual truth about the sorting algorithm, but that insight tells me little or nothing about the politicality of the algorithm. It simply confirms what I already know: that most everything created and operationalized in my world is done so for the purpose of augmenting the revenues of a particular organization – in this case, Amazon.com.
But there is something else interesting going on here with this sorting algorithm: by presenting its recommendations as it does, it articulates relations not just between me and another commodity of desire, but also between an expectation of taste (based on something I may have enjoyed in the past) and a future value. Now, regardless of whether this recommendation is accurate or not, worthwhile or not, or ultimately profitable or not for the company, the simple fact that a relation has been posited is a politically relevant fact about algorithms. And this is a politically relevant fact independent of (though not innocent of) the particular ideological context of its operation.
To treat an algorithm more broadly as a relational medium allows us to say this about them: algorithms exist in the human condition of separateness. They are technical media that have been invented in order to mediate separateness – of time, of space, of awareness, of attention. In short, algorithms intermediate the separateness of the in-between which is the condition sine qua non of human pluralism. And this radical empiricist insight helps get at a possible answer to the question, how do algorithms participate in politics? They participate by partaking in scenes of intermediation that exist in the in-between of peoples, places, things, and events. When we think of a sorting algorithm as an intermediator of separateness we begin to appreciate that the algorithm is political because what it is actively doing is participating in the arrangement of worlds. Our worlds. The worlds we experience in the here and now. The political matter for me, then, is not one of how algorithms constrain my freedoms. But, rather, how do algorithms participate in the formation of worlds, including the worlds within which I participate on a daily and hourly basis? Where "participation in the formation of worlds" stands as a short-form for a coming-to-understanding of the algorithm's powers of arrangement, association, and dissociation.
In this respect, I consider algorithms not simply as tools of domination but as "sentimental" media. Sentimental here is not synonymous with emotions and feelings (although emotions and feelings emerge out of a sentimental operation). By sentimental I refer to the ordering, structuring, and arranging of sensibilities: emotions and feelings (to be sure), but also perceptibilities and intelligibilities. In their capacities as sentimental media, algorithms first and foremost coordinate attention and awareness and make it so that we exist differently in relation to one another. An acknowledgment of the algorithm's claim on our all too human condition of separateness brings us face to face with their standing as political media. They are political because they arrange worlds. And out of these arrangements, intermedial power dynamics that may include (but aren't exclusive to) domination emerge.
It is for this reason, then, that rather than speaking about algorithms in general – or about any one specific algorithm – I prefer to think about the "algorithm dispositif." What is the algorithm dispositif? In part I have answered this question above. But a few words on this Frenchism might help clarify things further. "Dispositif" is a Latinate word that arrives to English from France and is typically untranslatable – though it has often been mistranslated as "apparatus." Elaborating the distinction between "dispositif" and "apparatus" must be deferred for another discussion, but the distinction more or less rests on the difference between an influence machine and an intermedial object. The term dispositif has its root in the Latin dispositio that refers to practices of arrangement and, to use a cognate English word, dispositions. More specifically still, the dispositif comes to us from the tradition of rhetoric – its classical sources are Aristotle's Rhetoric, Cicero's De Oratore, and Quintillian's Institutio Oratoria. The dispositio in rhetoric refers to the arrangements of the parts of speech in an oration, and how the order of ideas, of words, and of formulations, may be organized in such a way as to maximize persuasion. The dispositio is that part of an experienced oration that disposes the audience to attend to the speaker's words – not to listen, understand, or interpret them – but to attend to them, to lend them attention, to orient one's attention to them. Listening, understanding, interpreting may follow from this – indeed, usually do follow from this if the dispositio is successful. But the principal aim of the dispositio is not the transmission of an intention; this, because the dispositio is not a demonstrative proof.12 It is, rather, oriented towards the disposing (in the sense of attuning) of one's perceptibilities and forms of intelligibility. Consider in this regard the first line of Mark Antony's famous funeral oration from William Shakespeare's Julius Caesar, "[f]riends, Romans, countrymen, lend me your ears." (Act III, Scene 2) To lend one's ears – the disposing of the ears towards speech – is the exhortation of the dispositio. What matters here is not language as expressing intention, but how what is said is posed (and poised) so as to call attention and bestow notice: dispositio is a modality of collective participation, an active placing upon of parts, one in relation to the other, resting between and among each other.
It's in this sense of dispositio that the algorithm dispositif is a sentimental medium. As the sentimental philosophers of the eighteenth century showed, David Hume chief among them, sentiments are the forces that connect us to one another, through technical media like language (i.e., promising) and contracts, and to political life as a whole. The sentimental, in other words, is a category of experience that is world making. As a sentimental medium, the algorithm dispositif arranges and disposes us to the world. In doing so, it organizes worlds by orienting relations of time and space, subject and object. This is what I mean when I say that algorithms exist in the human condition of separateness. Their dispositional powers operate in such a way as to coordinate and negotiate the in-between of separateness – just like a sentiment like sympathy is what organizes my separateness from other humans so that I may build something like social trust.13
To be clear, I'm not saying the algorithms are emotional devices, though there is much evidence to suggest that algorithms are emotion-triggering devices. What I am proposing is that the social and political study of algorithms proceed in a manner akin to how we understand the dispositional powers of the sentiments – powers that dispose us to move and extend ourselves within the in-between space of separateness that conditions human existence. The algorithm dispositif is political, in other words, because it operates in the intervening spaces of separateness and does so by a power of mediation that is dispositional. And this, I wish to say further, is substantially different from claiming that algorithms are structures of domination and that their political function is one of subjection. Humans dominate one another. Of this there is no doubt. But the work of arrangement in and of political societies is not reducible to domination.
The preceding offers notes towards the possibility of asking the following question: What are the conditions in and through which we can think the politicality of the algorithm dispositif? The ambition for broaching this question rests on what I take to be a unique impasse for the history of critical thinking that the algorithm dispositif affords. Much of our critical tradition rests on two important – indeed, essential – gestures. The first, inherited from Plato, is that to think critically about the politicality of technical experience requires the capacity to turn away (through reflection, cognition, rationalization) from the coercive operations of power implied or presumed in technical objects. The second is akin to the first: our sense or acceptance of the workings of a technical object rests on a reflective experience we may have of it. We experience a film by viewing it, a musical score by listening to it, a food morsel by eating it, a novel by reading it. The impasse that the algorithm affords our critical tradition challenges both these premises: the fact of algorithmic ubiquity in everyday life makes turning away an unavailable critical response; moreover, we don't experience algorithms. We experience inputs and outputs, data and data's mediation.14 But we don't experience the technical medium of the algorithm, not in the way we appreciate our experiences of other, more established, media. At the interstice of these impasses in our critical traditions we may begin to reflect anew on the tissues of experience that the algorithm dispositif affords.
By Davide Panagia, Professor, UCLA Dept. of Political ScienceThe post The Algorithm Dispositif (Notes towards an Investigation) first appeared on AI Pulse.