AI & Agency

 Download as PDF

Introduction
In July of 2019, at the Summer Institute on AI and Society in Edmonton, Canada (co-sponsored by CIFAR and the AI Pulse Project of UCLA Law), scholars from across disciplines came together in an intensive workshop. For the second half of the workshop, the cohort split into smaller working groups to delve into specific topics related to AI and Society.
I proposed deeper exploration on the topic of "agency," which is defined differently across domains and cultures, and relates to many of the topics of discussion in AI ethics, including responsibility and accountability. It is also the subject of an ongoing art and research project I'm producing. As a group, we looked at definitions of agency across fields, found paradoxes and incongruities, shared our own questions, and produced a visual map of the conceptual space. We decided that our disparate perspectives were better articulated through a collection of short written pieces, presented as a set, rather than a singular essay on the topic. The outputs of this work are shared here.
This set of essays, many of which are framed as provocations, suggests that there remain many open questions, and inconsistent assumptions on the topic. Many of the writings include more questions than answers, encouraging readers to revisit their own beliefs about agency. As we further develop AI systems, and refer to humans and non-humans as "agents"– we will benefit from a better understanding of what we mean when we call something an "agent" or claim that an action involves "agency." This work is under development and many of us will continue to explore this in our ongoing AI work.
– Sarah Newman, Project Lead, August 2019
1. Characterizing Agency
Jon Bowen
PhD student in Philosophy, Western University
Some of the beings we encounter in our environment are inanimate. These things may be pushed and pulled, they may collapse or disintegrate. In each of these cases, the entities are fundamentally passive–if they move or change, one suspects that these movements and changes will be exhaustively explained by appealing to mechanical forces within or without.
But there is another kind of entity in our environment. These beings seem to be fundamentally goal-directed. To appearances, they are spontaneous initiators of their own actions. These are animate beings, or agents. The movements of these entities seem to be best explained not by appeal to mechanical causes of their activity, but to the goals that they are striving towards, the beliefs they have about the world, and their desires.
Giving a precise definition of what animacy or agency consists of is no easy task for the philosopher, but nonetheless we appear to have no difficulty at all recognizing animate motion and distinguishing it from the motion of inanimate objects. Even human infants, it seems, can detect animate motion and differentiate it from inanimate motion in point-light displays, even when occlusions are present.
But why should this be the case? Why would it be so difficult to give a theory of intentional action, and yet so easy to detect it? I will set out one suggestion. We do not, as has been proposed, infer intentions, beliefs, and desires as a part of a theory for explaining or predicting behavior. Instead, intentional action is behavior with certain distinctive, overt characteristics, which our perceptual systems have evolved to directly perceive. Goal-directed behaviors, I will suggest, are a very real kind of behavior out there in the world with distinctive characteristics. Furthermore, it is important that animals perceive and understand this particular kind of behavior, and sure enough, they are able to do so with astonishing acuity.
What are we saying when we explain the activities of another person (or of a non-human animal) by appealing to their intentions? Here I will draw on an analysis from Dennis Walsh: "A teleological explanation is one that explains the nature or activities of an entity, or the occurrence of an event, by citing the goal it subserves. A system has goal, E, just in case it exhibits goal-directed behavior toward E. Goal-directed behavior is a gross property of a system as a whole." (p. 177)
What this amounts to is not an account of the intrinsic causal etiology of the agent's behavior. Instead, we are locating that behavior in a chain of events that show a certain distinctive pattern. If an agent is trying to do X, then its behavior will flexibly reconfigure itself in the service of that goal. When a dropped object encounters the ground, it will stop. When an agent's initial attempts to pursue some goal are thwarted, that agent will spontaneously and flexibly reconfigure its behavior so as to continue to pursue its goal. A human need not stop at the ground–they can retrieve a shovel, and perhaps a jackhammer or a drill if called for (if they really want to!) This is to say, when an agent is engaging in goal-directed activity, its behavior is robust against perturbations and obstacles in a way characteristically not present in inanimate objects.
If there are such systems in nature–systems that will reliably produce effects by marshaling their intrinsic causal capacities in the service of goals–then clearly the perceiving animal would be at an advantage if they could detect them when present! The challenge, from the perspective of an animal's perceptual system, then, is to detect or pick up the information which specifies what the goals of other agents in their acting are. While this might sound like quite a feat, again, this is something we all seem to be very good at.
If agency amounts to the capacity for intentional action, and the preceding account of goal-directed behavior is sound, what basis might there be to deny that such a thing exists as a real phenomenon in nature, and a real attribute of natural beings?
2. The Value of the Concept of Agency in an Increasingly Rational World
Osonde Osoba
Information scientist, RAND Corporation
Professor, Pardee RAND Graduate School
Let us concede that different traditions of thought have different definitions and perspectives on what it means to be an agent or to have agency. There are some common threads that may be useful to highlight. I will focus on one. Most conceptions of Agency are rooted in action, in doing, in affecting a substrate environment.
A working definition for the purposes of this discussion could go thus:
An agent is an entity that is capable of causing or effecting change in its world in pursuit of private (personal) goals.
This definition has a couple of features worth highlighting:
The primacy of causality:
We focus on the idea of causal influence as a defining characteristic. An entity whose whole existence consists of internal ruminations (e.g. Ibn Tufayl's floating man) does not meet our criteria. However much sophisticated intelligence it applies to its sense perceptions, it has no influence over its environment. It can achieve no goals in its world no matter how intensely it wills them.
Contextual worlds:
Context determines the relevant world over which the agent aims to exert influence. Entities can be part of numerous worlds or environments. An entity's agency in each of these worlds is determined by how much causal influence it can exert in each one. We can imagine a measure of power based on what fraction of an agent's environment it can influence.
Private goals:
Private goals may be related to Aristotle's idea of a "final cause," the reason for which a thing exists. The capacity for pure action without goals requires no planning, interiority, or intentionality. We will argue that tracking that sort of capacity is not useful.
The concept of agency has proven useful for rooting responsibility and/or liability in entities capable of modifying their actions in response to external influence. Such a capacity for redress or accountability can arguably only be supported by entities capable of goal-oriented behavior.1 Responsibility can be moral or legal (more coercive/backed by institutional power). Agency likely serves other important functions. But the responsibility-rooting function of agency is crucial for influencing or controlling behavior in social structures.
This view of agency is explicitly not about independence or autonomy. Agency, in this conception, is closer to a useful fiction that enables the clean assignment of responsibility and dessert. And the default assumption is that agents exist within networks of influence. A degree of external manipulation of agents is the norm, not a novel pattern.
Historically, the use of agency for allocating moral responsibility has been a useful but imperfect device: the assignment of moral responsibility has not always tracked causal responsibility. The long tradition of arguments for the justice of gods (theodicies) is a case in point. If evil befalls a person, it must be because that person has misused his agency ("sinned") and therefore deserves or is morally responsible for his lot.2 Some superstitions may also be construed to serve a similar function. These failures in causal attribution happen because the world is complex, causal attribution is notoriously difficult, & causal influences can be very subtle when they exist. By contrast, gods are simpler, more convenient causal explanations.
Our modern conception of moral responsibility is becoming more rational, more scientific. Part of the goal of rational thought is to focus on the true causes of observed phenomena. Weber goes so far as to argue that scientific inquiry is just a rational incarnation of theodicy.3 We have moved from agency based on imperfect beliefs towards a more causal conception of responsibility.
But what happens when our rational understanding of reality expands to the point where we are able to track causal influences as finely as possible?4 E.g. recent literature has begun to undermine agency-based explanations of individual behavior in favor of longer chains of causal influence that reach past the mask of more person-focused conceptions of agency. How do we ground responsibility and liability when large swathes of action can be explained away via causal factors outside the individual (e.g. the larger explaining value of social influence or manipulation, genetics, environmental factors, etc.)?
Does the concept of agency survive this trend?
3. Human agency in the age of AI
Abeba Birhane
PhD Candidate, School of Computer Science, University College Dublin
Provocation:
The question of agency necessarily provokes the question of what it means to be a person and, in particular, what it means to be a person in the age of ubiquitous artificial intelligence (AI) systems. We are embodied beings that inherently exist in a web of relations within political, historical, cultural, and social norms. Increasingly, seemingly invisible AI systems permeate most spheres of life, mediating and structuring our communications, interactions, relations, and ways of being. Since we do not exist in a social, political, historical, and AI-mediated vacuum, it is imperative to ground agency as inherently inseparable from the person as construed in such contingent constituent factors. Depending on the context and the space we occupy in the social world, all these dynamic and contingent factors serve as enabling constraints for our capacity to act. Our capacity to act within these contextual factors varies in degree depending on the space we occupy at a certain time, in a certain socioeconomic context; the more privileged we are, the fewer the potential constraints, and the greater our degrees of agency.
Essay:
The individual is never a fully autonomous entity: rather, they come into being and maintain that sense of existence through dynamic, intersubjective, and reciprocal relations with others.5 Our biology, current social and cultural norms, historical, and contextual contingencies, as well as our physical and technological environment, constitute who we are and our degrees of agency within a given time and context. Increasingly, AI systems are becoming an integral part of our environment – be it the search engines that we interact with, our social media activities, the facial recognition systems that we come in contact with, or the algorithmic systems that sift through our job applications – further adding enabling, or limiting, constraints. (Enabling constraints here might include having a common Western male name, or other demographic traits, that the job application algorithm chooses to include, rather than exclude. These are still constraints, but in certain instances they increase opportunity, rather than decrease them.)
We are embodied beings that necessarily exist in a web of relations with others, within certain social and cultural norms as well as through emerging technologies. This means our sense of being, as well as our capacity to act, are inextricably intertwined and continually changing as we move between various spheres taking on various roles. The various factors that constitute (and sustain) who we are influence the varying degrees of agency we are afforded. As we go on about our daily lives, we move between various social and cultural conventions, physical environmental enablers (or disablers) of certain behaviors and actions (as opposed to others), and technological tools that shape, reinforce, and nudge behavior and actions in certain directions (and not others). As a PhD student, my role, responsibility, and capacity to act in my academic environment, for example, is different than that of my role, responsibility, and capacity for action when I am at a social gathering within the immigrant community. Furthermore, my interaction with others through Twitter is different from both these other contexts, and is partially determined by the ways the medium affords possible actions and interactions. Our sense of agency, then, is fluid, dynamic, and continually negotiated within these various physical, mental, psychological, technological, and cultural spaces. Discussion of agency, consequently, cannot emerge in a social, technological, and contextual vacuum. Nor is it something we can view as stable or pin on individual persons due to the complex, contingent, and changing factors that constitute and sustain personhood.
Conversely, agency cannot be an abstract term that we attempt to define and analyze in a general, one-size-fits-all manner but one that needs to be grounded in people. People, due to their embeddedness in context, culture, history, and socio-economic status, are afforded varying degrees of enabling constraints. Agency, therefore, is not an all-or-nothing phenomenon but something that varies in degrees depending on individual factors, circumstances and situations. Individuals at the top of the socio-economic hierarchy, for example, face relatively fewer disabling constraints, consequently resulting in a higher degree of agency, and the reverse holds for those at the lower end of society. For example, depending on their socio-economic and educational background, one may be labelled "eccentric" vs. "insane", a "lone wolf" vs. a "radicalized extremist", a "freedom fighter" vs. a "terrorist".
Agency, AI, and ethical considerations
Living in a world of ubiquitous networked communication, a world where AI technologies are interwoven into the social, political, and economic sphere means living in a world where who we are, and subsequently our degree of agency, is partially influenced by automated AI systems.
The concept of AI often provokes the idea of (future and imaginary) sentient artificial beings, or autonomous vehicles such as self-driving cars or robots. These preconceptions often assume (implicitly or otherwise) that AI systems are entities that exist independently of humans in a machine vs. human dichotomy. This view, which dominates academic and public discourse surrounding AI is a deeply misconceived, narrow, and one-dimensional conception of AI. What AI refers to in the present context is rather grounded in current systems and tools that operate in most spheres of life. These are seemingly invisible tools and systems that mediate communication, interaction with others and other technological infrastructures that alter the social fabric. These AI systems make life effortless, as they disappear into the background to the extent that we forget their very existence. They have become so inextricably integrated with our daily lives that life without them seems unimaginable. As Weiser6 has argued, these are the most profound and powerful technologies. "The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it."
These systems sort, classify, analyze, and predict our behaviors and actions. Our computers, credit card transactions, phones, and the cameras and sensors that proliferate public and private spaces are recording and codifying our "habits", "behaviors", and "experiences". Such ubiquitous interlinked technological milieu continually maps out the where, when, what, and how of our behaviors and actions, which provide superficial patterns that infer who we are.7 Whether we are engaging in political debate on Facebook, connecting to "free" wi-fi, using Google Maps to get from point A to B, searching for sensitive health information on Google, ordering grocery shopping, posting selfies on Instagram, or out in the park for a jog; our actions and behaviors produce a mass flow of data that produce pattern-based actionable indices about "who we are". These superficial extrapolations, in turn, feed models that predict how we might behave in various scenarios, whether we might be a "suitable" candidate for a job, or are likely to commit crimes, or are risks that should be denied loans or mortgages. Questions of morality (often misconceived as technical questions in need of a technical fix) are increasingly handed over to engineers and commercial industries developing and deploying AI systems as they are bestowed with sorting, pattern detecting, and predicting behaviors and actions. These predictive systems give options and opportunities to act or they limit what we see and the possible actions we can take. And as O'Neil8 reminds us, each individual person does not pass through these processes to the same degree nor do they suffer the consequences equally. "The privileged are processed by people, the masses by machines."
These systems not only predict behavior based on observed similar patterns, they also alter the social fabric and reconfigure the nature of reality in the process. Through "personalized" ads and recommender systems, for example, the level and amount of options put in front of us varies depending on the AI's decision of "who we are," which reflects the place we occupy in the social hierarchy. The constraints that provide us with little or great room to act in the world are closely related to our socio-economic status and, increasingly, to who our data says we are. Unsurprisingly, the more privileged we are, the more we are afforded the capacity to overrule algorithmic identification and personalization (or not be subjected to them at all), maximizing our degrees of agency.
Since agency is inextricably linked to subjecthood, which is necessarily political, moral, social, and increasingly digital, the impact of power structures is inescapable. These power relations and the capacity to minimize the potential constraints AI imposes on agency, is starkly clear when we look at the lifestyle choices that powerful agents in Silicon Valley, who make and deploy technology, are afforded. For example, while screen-aided education is pushed towards mainstream schools, the rich on the other hand are reluctant to adopt such practices.9 Agency, the capacity to act in a given technological environment and context varies in degree from person to person. Silicon Valley tech developers, those with power and awareness of technology as constraining powers, are reluctant to let it infiltrate their children's surroundings. Some go so far as banning their nannies from the use of screens.10
Agency is not an all-or-nothing phenomenon that we either do or do not have. Rather, agency is inextricably linked to our social, political, and historical contexts, which are increasingly influenced by technological forces. These forces grant people varying degrees of agency. In an increasingly AI-powered society our capacity to act is limited or expanded based on our privilege; agency is increasingly becoming a commodity that only the privileged can afford.
4. Agency to Change the World
Mike Zajko
Assistant Professor, Department of History and Sociology, University of British Columbia
Abstract
Social theory has identified agency with social change and dynamism, bringing tension and possibility to a world where social structures are reproduced. The concept of agency can rescue us from the notion that we are simply the product of our conditioning (zombies of embodied habits), and stands in opposition to ontologies that foreground practices at the expense of subjects. While a humanist conception links agency to purposive action, an expansive (post-humanist) definition elides the question of intentionality, and links agency with action, irrespective of purpose. According to this view, rather than being an exclusive human property, agency is all around us, and society has always consisted of relations between human and non-human actors. We should keep in mind that agency is not absolute or independent, but contextual and relational. If we conceive of agency in this way, we can see the stakes of some of the current debates about AI: to what extent will these systems act as agents of change in our world, and how will AI affect (enhance, extend, supplant, or constrain) human agency?
Agency to Change the World
How did Western intellectuals go from believing agency to be the exclusive property of the human subject, to considering whether algorithmic agents, or AI systems, also have agency? One understanding is that as AI increasingly approximates human intelligence, it attains attributes formerly reserved for humanity. But an argument can be made that AI today, even in its narrowest forms, already exercises agency, and that humans were never particularly special to begin with.
It is commonly said that people exercise agency to achieve their desires, goals, and interests. In sociology, agency has long been seen as the source of change in society. Agency is why society does not remain in a steady state, despite all the ways that social structures are reproduced. Without agency, we would all be pawns shaped and manipulated by larger forces that often precede our existence: children molded into reproductions of their parents; compliant, orderly workers reproduced by the educational system to passively accept ideologies that justify why the existing order is natural, desirable, or worthy of being preserved. Agency refers to our ability to change this social structure, to disagree with our parents, use education to advance knowledge, achieve social mobility, critique ideology, and challenge government.
There is a longstanding debate in social theory about the relationship between agency and social structure,11 which has largely gone stale and unresolved. But agency continues to provide the tension that prevents a totalizing view of structure. Not everyone agrees that agency is required to understand humanity or the relationship between individuals and society, but social theories that do without agency, or that provide an impoverished view of agency, paint a deterministic picture. Individuals are conceived not as subjects, but through their habits and practices, or as the effects of the social structures that produce them. Without agency, we are zombies, automata, or cultural dupes.12
Amidst some of the debates about agency and structure in the 1980s and 1990s, a new conception (often associated with Bruno Latour and Actor-Network Theory)13 began to take hold. The provocative argument was that agency was not confined to humans, but that society was composed of both human and non-human "actants."14 Agency was defined roughly with action, and the ability to affect the world. If a human worker was replaced by an object (even an inanimate one)15 that could play the same role, then that object similarly exercised agency. Because in many of our interactions with the world, whether in laboratory experiments or farming,16 humans cannot fully predict or control the outcome, nature also has agency – co-creating the world with us. Questions of intentionality and purposiveness are elided through this focus on action.
Whether in its humanist or post-humanist form, Western theory's interest in agency has also been subjected to significant critique. The idea of an autonomous human subject is arguably a historical invention – a distinctly Western, masculine, individualistic vision of man. Feminist theorists advanced these arguments decades ago, pointing to the often unacknowledged work (disproportionately performed by women) of nurturing and caring for 'autonomous' subjects. Complicating but not necessarily rejecting the ideal of autonomy, these authors advanced a concept of agency that situates it firmly within social relationships.17 Relational and non-human conceptions of agency are common in Indigenous and non-Western ontologies,18 rooted in the understanding that the world is agentially alive, and that humanity is inexorably linked to and dependent on these forces. In an article section titled Columbus Discovers Non-Human Agency,19 three authors influenced by Indigenous feminist literature point to the Eurocentric and settler colonial bias of a recent turn in social theory. In this 'new materialism', authors influenced by Latour and feminist STS have made expansive claims about agency that may be innovative for social theory, but which are quite traditional for unacknowledged indigenous ontologies.
At this point it is worth reflecting on these divergent conceptions of agency. Along one dimension outlined above, they run the range from treating agency as a distinctly human property, linked to subjectivity, consciousness and intentionality, to a broader view of agency as whatever has effects on the world (and ourselves). At its broadest, we are not agents at all: distinctions of subjects and objects are dissolved, and the entire universe becomes a quantum soup of intra-active becoming.20 But somewhere between this posthuman extreme and the reassurance of conventional humanism, we can return to a view of agency that encompasses both humans and AI, as agents that change the world, and are entwined in relations with one another.
Today, developers are building robots that learn about and interact with their environment – an environment that includes other robots as well as humans. Machine learning enables AI systems to pursue goals in ways that humans could not anticipate, even if their goals were initially formulated by humans. We now regularly interact with various kinds of AI, or are subject to decisions made by these systems. Finally, the distinctiveness or exceptionality of the human subject has been repeatedly problematized by advances in AI and in our understanding of other organisms. In this context, conceiving of agency as the ability to change the world remains valuable for considering issues common to humanity and AI.
Conceptualized as a means of social change, we can see that agency is not a human birthright, and is not equally distributed across humanity. Structured inequality provides opportunities to some, which are denied to others. Where a person is born, and how they are nurtured or socialized, has great consequences for the choices and capacities available to them – including the impact a person can have on reshaping pre-existing structures. Agency depends on our relationship to these structures, as well as to each other. Hence, agency varies across positions in society and is subject to change. We can engineer technologies and social systems to enhance human agency, to provide capabilities for transformation of individual or collective conditions; or we can design to preserve and reinforce existing power structures. Similarly, it is valuable to conceptualize the agency of AI through its ability to affect the world, change itself, and change human lives, irrespective of consciousness or intentionality. If we conceive of agency in this way, we can see the stakes of some of the current debates about AI: to what extent will these systems be agents of change in our world, and how will AI affect human agency? What decisions will AI make on behalf of humans, and how will these sociotechnical systems reconfigure the possibilities available to us?
5. Can (and Should) AI Be Considered an Agent
Gabriel Lima
Computer science undergraduate student, KAIST, South Korea
Provocation
In this short essay, I share my thoughts on the relationship between artificial intelligence (AI) and various definitions of agency. Can AI be considered an agent? More specifically, does AI fulfill requirements set forth in various definitions of agency? Depending on the perspective and definition taken by the reader, the agency of AI could be controversial, unimaginable, or an unquestionable truth. A question that is often neglected, however, is whether AI should be given any agency. Even though we often derive normative statements of value (e.g., should, ought to) from descriptive statements of fact (e.g., can, is), their distinction is extremely important and has been discussed by many philosophers who argue this relation is not necessarily valid and advisable. Finally, I conclude my essay by raising the open question whether AI should indeed be an agent in our society independent of the fulfilment of agency requirements set by various definitions. Instead of focusing on the abilities of an AI, what if we first ask whether it would be beneficial to treat an AI as an agent in society?
Introduction
Agency has never been clearly defined across, or even within, disciplines. Even though it is often related to autonomy, responsibility, or causality, no clear definition agrees on every detail around the complicated issue of who (or what) is an agent.
In this short essay, I share my thoughts on the relationship between artificial intelligence (AI) and agency. Can AI be considered an agent? More specifically, does AI fulfill requirements set in various definitions of agency? Depending on the perspective taken by the reader, the agency of AI could be controversial, unimaginable, or an unquestionable truth. A question that is often neglected, however, is whether AI should be given agency. Even though we often derive normative statements of value (e.g., should, ought to) from descriptive statements of fact (e.g., can, is), their distinction is important and many philosophers have argued that this connection is not valid or advisable. Finally, I conclude my essay by raising the open question whether AI should indeed be an agent in our society independent of the fulfilment of agency requirements set by various definitions.
As introduced above, agency is not clearly defined and thus, tackling whether AI could qualify as an agent following every single proposed idea of agency is infeasible. In the following short subsections, I will deal with some common sociological, legal, philosophical, and technological definitions of agency and share my thoughts on whether AI could be considered an agent under each definition.
An Agent Is a Goal-Oriented Entity
Does an AI have a goal? From a computer science perspective, this is often how we create and train AIs. For instance, in reinforcement learning we teach AIs by rewarding them depending on whether or not they have achieved a set goal. The goals of an AI are not intrinsic, but extrinsic; the programmer sets its goals following his or her needs. This does not, however, disqualify AI as a candidate for agency. According to the idea that agency is based on a goal-oriented behavior, AI could be seen as an agent.
An Agent Can Act and Modify Its Behavior Depending on the Environment
This definition is often used in computer science when dealing with reinforcement learning, a method used to train AIs. In this setting, we define AI as an agent in an environment with a set of policies and actions. Given that AI is defined as an agent from its conception, it is easy to imagine an AI as an agent after its deployment.
An Agent Has an Effect on the World and Drives Social Change
Following this more sociological perspective, an agent must make a difference in society to qualify for agency. In the current "AI Summer," AI is affecting society in ways many did not expect – or did expect, but unfortunately neglected. AI has been disruptive in diverse sections of society. Job markets having to adapt to the insertion of these electronic entities, and recommendation algorithms controlling what kind of information a certain part of society has access to, are among many examples of novel consequences AI is imposing on society. It is not hard to see an AI as an agent considering its impact on society.
An Agent Can Engage With or Resist Colonial Power
Even though sci-fi scenarios give us the idea that AI can resist the power of its creators, this possibility is far from us. AI cannot resist and turn against its own creator, due to both lack of ability and the high level of control creators still have over their creations. AIs are distant from engaging with (or inverting) the power pyramid, where they are at the very bottom. More importantly, how can they even set that as a goal, if an AI is not currently able to have intrinsic goals? By this conception, AI cannot be an agent since it does not engage in any action dealing with its creators and its hierarchical position.
An Agent Is an Entity That Acts on Behalf of a Principal
We often build AIs as entities to complete a certain task for humans. These systems act on behalf of a principal, which can be their programmers, manufacturers, or users. The principal sets the AI's goals and the system works towards achieving them. By this conception of agency, an AI is clearly an agent. Some authors even argue that AI could be a "perfect agent," since it does not have intentions or goals that could deviate from its principal's goals.
As issued raised by many legal scholars about AI agency is the usual requirement of a contract to establish a principal-agent relationship. Since AI has not (yet) been granted any kind of legal personhood, it cannot be a party to a legal contract. Consequently, while an AI could be seen as an agent under a principal in economic terms, it cannot qualify as one legally.
An Agent Can Bear Responsibility for Its Actions
Can an AI be responsible for its actions? How would this responsibility even be assigned to an entity that cannot be held accountable for its actions? If an AI causes damage, how can it be punished? These issues are raised by many law scholars when dealing with the liability assignment of an act with legal consequences by an AI. At present, liability usually goes towards the manufacturer or user of an AI, so the AI system itself cannot be seen as an agent.
But Should AI Be Considered an Agent?
As I have argued above, depending on how you define agency, the idea of AI being an agent can be seen as either reasonable or completely absurd. Given that it is a possibility, should we consider AI as an agent? Even though we often derive whether an entity should receive any consideration from its ontology and capabilities, should we apply the same reasoning when dealing with AI? Would that be beneficial to our society, our legal systems, or even humanity as a whole? Should we even ask that question?
With the fast development of AI, we keep dwelling on what each system can and cannot do; we thereby neglect the question of whether this consideration is the right one to focus on. What if, instead of focusing on what an AI can do, we center discussion on whether these entities can be seen as agents no matter how complex, intelligent, or autonomous they might be? Although the abilities and inabilities of current AI systems are important to the discussion of the position of AI in society, this might better be left as a follow-up question to the most immediate inquiry: given the lack of agreement on the definition of agency and regardless of the abilities of these newly developed entities, is it socially beneficial or possible to consider AIs as agents?
6. How does AI affect human Autonomy?
Carina Prunkl
Senior Research Scholar, Future of Humanity Institute, University of Oxford
Autonomy (autos = self; nomos = law) in the context of human beings refers to the capacity of self-governance or self-determination. This also implies that an individual's actions are neither the product of external manipulation, nor imposition of external forces. Autonomy in this sense plays an important role in Western culture and is often considered desirable for the individual. When we speak about 'autonomous systems' in the context of artificial intelligence, we similarly refer to some sort of 'self-governance', but in contrast to human case, this 'autonomy' has little to do with acting true to one's own beliefs, desires or motivations. Instead, it refers to the capacity of the system to learn and perform certain tasks without human guidance or supervision. A well-known example of such 'autonomous systems' are self-driving cars that navigate themselves through traffic to bring their passengers from A to B. But of course this type of 'autonomy' is not limited to the mechanical realm and we may easily conceive of virtual 'autonomous systems', such as virtual assistants that organize our lives by making appointments, doing (online) grocery shopping, taking notes, etc. By outsourcing seemingly trivial tasks such as driving and grocery shopping – not to mention some highly non-trivial tasks, such as those now performed by soldiers but that might at some point become automated – we are handing over more and more responsibilities to 'autonomous systems.' How will such a development affect our own autonomy? It is difficult to imagine that at least those of us who are somewhat indifferent to the joy of driving, will feel or be less autonomous by having a car that takes us to where we want to go faster and safer. This is at least in part because it is we, after all, who decide where to go and when. But what about when such 'autonomous systems' not only navigate us through traffic, but also through life? When they learn from our behavioral patterns, our preferences, our relationships, to make predictions about, say, what groceries we would like to eat next week? Here the situation is much less clear. Do we gain autonomy by not having to be bothered with boring grocery planning and shopping, and instead having time for the things we would really like to do? Or do we instead forfeit autonomy by not being the ones who make the choices about our nutrition, returning almost to the childlike state of not having to take responsibility for certain aspects of our lives? These are questions we urgently need to ask ourselves.
7. The Myth of Agency
Sarah Newman
Senior Researcher, Principal at metaLAB at Harvard
Fellow, Berkman Klein Center for Internet & Society, Harvard University
"Ultimately, nothing or almost nothing about what a person does seems to be under his control."
– Thomas Nagel, Moral Luck
We look, critically, at how our technologies work, and yet we make assumptions about how we work. What motivates our choices? Are we in control of our actions – and if so, all of them, or only some of them? As our interactions with and dependence on new technologies, including AI, become both increasingly common and invisible, what, if any, agency are we giving up? If we better understand our agency, how does this connect to our responsibility for the technological world we are creating, and the natural world we are destroying? What responsibilities should we have for our own behaviors, and where does accountability reside in automated systems?
We use the term "agency" to refer to humans, to current and future AI systems, as part of a framework for responsibility and accountability. But what do we mean by agency? Agency is defined differently across disciplines–from computer science to philosophy to sociology to law. Recent developments in neuroscience and AI both call into question the accepted notion of volitional agency as the willed proximate cause of a thought or an action. How might exploring frameworks of agency affect our approaches to ethical standards in the development of AI? A potential blind spot in our analysis of the development of AI lies in the assumptions we make about our own agency, freedom of will, and moral capabilities.
Are we are actually more accurate when describing the behavior of machines–mechanistic, physical, governed by the laws of nature and programming–than we are when we describe ourselves? Things get fuzzy as the mysteries of consciousness and subjectivity arise. What is true – and what, if not true, is useful to believe?
We believe that we, as humans, have at least some agency. We acknowledge that our degrees of agency differ across individuals and circumstances, increasing or decreasing based on certain constraints, and governed by physical laws–at least those outside of our brains. Most people don't believe that they could defy physical laws: the laws of gravity, survival without food, etc. We accept these physical constraints, those that appear to affect all beings and appear to be external to us, or at least external to our physical bodies. Yes, this agency is highly variable: a healthy adult has more agency, people tend to agree, than a baby, or someone who is very old or unwell.
We tend to agree that we do not have the agency to fly, or to travel in time, or countless other fantastical things (barring of course certain mental illnesses, or other illnesses which impinge on mental capacities, which have their own unique relationship to agency and thus also to responsibility). And yet most people now, and throughout history–across cultures, ages, and every other demographic factor–have a distinct sense of being in control of (at least some of) our behaviors and actions. Even though it is difficult to explain, there is a distinct and overwhelming sense that I am choosing to write these words, that I will choose what to have for dinner, that I could choose to clap my hands, or nod my head, or close my eyes. This sense, as inexplicable, biologically and physically, as it may be (as a being comprised of physical matter that came into existence in a way that I certainly did not will), from where did it arise? Is the sense of agency I possess merely a myth? Perhaps a useful, or even inescapable myth? If so, is considering such questions useful or productive?
For me, reflecting on such questions is enriching: it enriches my daily life and my experiences. Paying attention to this deep and abiding mystery, somewhat ironically, feels empowering – as if I am curiously contemplating whether the backdrop is a facade, whether this sense of agency is indeed an illusion. I acknowledge the possible privilege of this perspective. Perhaps, if I do indeed have some sort of inexplicable agency, contemplating it is enjoyable because I have (if I have it at all) a relatively high degree of it. But perhaps not.
Such topics have fascinated philosophers, theologists, and most humans for so long as we have records of such contemplation. Debates on free will or the existence of agency–nevertheless have barely made their way into discussions of the new sophisticated technologies we are creating–particularly AI, in terms of how it already is acting in the world, as well as how it could impact the future. We talk about autonomy and responsibility, but can we use this moment to also reflect back on our assumptions about ourselves?The post AI & Agency first appeared on AI Pulse.

AI & Agency


Could AI drive transformative social progress? What would this require?

 Download as PDF

AI's Transformative Social Impacts and their Determinants
The potential societal impacts of artificial intelligence (AI) and related technologies are so vast, they are often likened to those of past transformative technological changes such as the industrial or agricultural revolutions. They are also deeply uncertain, presenting a wide range of possibilities for good or ill – as indeed the diverse technologies lumped under the term AI are themselves diffuse, labile, and uncertain. Speculation about AI's broad social impacts ranges from full-on utopia to dystopia, both in fictional and non-fiction accounts. Narrowing the field of view from aggregate impacts to particular impacts and their mechanisms, there is substantial (but far from total) agreement on some – e.g., profound disruption of labor markets, with the prospect of unemployment that is novel in scale and breadth – but great uncertainty on others, even as to sign. Will AI concentrate or distribute economic and political power – and if concentrate, then in whom? Will it make human lives and societies more diverse or more uniform? Expand or contract individual liberty? Enrich or degrade human capabilities? On all these points, the range of present speculation is vast.
What outcomes actually come about will depend partly on characteristics of the technologies, partly on the social, economic, and political context – what specific technical capabilities, with what attributes, are developed and deployed, and how people adjust behavior around the capabilities. It is a basic doctrine of technology studies to reject technological determinism: technological and socio-political factors interact, and to the extent either predominates in shaping outcomes it tends to be the social and political factors. The interplay between these underpins the well-known "Collingridge paradox," which states a structural challenge to managing technology's societal impacts1: early in development, control efforts are hindered by limited knowledge, because impacts are indeterminate until a technology is stabilized, deployed, and used; while later in development, control efforts are hindered by limited power, because the same development processes that determine and clarify impacts also build political interests in the technology's unhindered expansion.
In correctly rejecting naïve or extreme forms of technological determinism, however, these characterizations are often deployed too starkly and universally. Collingridge's paradox of knowledge and control is better understood as a persistent tension than as a categorical statement of impossibility. Moreover, without disparaging the power of social context, technological processes and artifacts are not infinitely malleable: particular technologies have characteristics, which in some cases tend to favor particular uses, applications, or consequences. Kranzberg's (slightly whimsical) first law of technology aptly captures the tension: "Technology is neither good nor bad; nor is it neutral."2 It is subject to influence, and that puts responsibility onto humans to wisely guide its development and application.
AI may be a class of technologies for which serious consideration of the role of technical characteristics in shaping impacts is especially needed, in view of its labile nature and its potential for profound societal disruption. Two examples from widely separated parts of present debates about AI impacts illustrate the point. First, concerns about impacts of extreme AI advances to general, beyond-human intelligence – and related efforts to develop "Friendly" or "Safe" AI, or align its objectives with human values (assuming these are known and agreed) – are entirely concerned with attributes of the technology. These efforts seek to ensure good consequences, or at least avoid the worst ones, by embedding reliable determinants of benevolent aims, prudence, or other virtues into the technical artifacts themselves. To the extent this program succeeds – a huge assumption, to be sure – it would move concerns about these extreme forms of AI impact out of the social and political domains entirely.
It is widely noted, of course, that focusing predominantly on such hypothetical future super-AI risks misleading, by distracting from addressing nearer-term uses and impacts that are also potentially transformative for good or ill – including both the "now" and the mid-term.3 Technical characteristics, even abstracted from social context, also matter for these near and medium time horizons – i.e., well before development of AGI or super-AI – when AI will clearly have transformative possibilities but still, at least formally, be under human control. The importance of technological characteristics is evident even in current AI controversies, in both what technical capacities allow and what they require. As an example of impacts driven by what technical capacities allow, AI-enabled advances in data integration and surveillance, especially facial recognition, already present significant threats to privacy and autonomy. These capabilities are being deployed because current actors find advantage in them, of course – a matter of social and political context. But it is the technical performance characteristics that create these new capabilities and make them visible. As an example of impacts driven by technical requirements, present machine learning algorithms require training on large labeled datasets. This requirement has driven two powerful effects and points of concern. It has steered many near-term commercial applications toward decision domains such as criminal justice and health, in which huge individual-level datasets with clearly labeled outcomes are available, with little advance consideration of the high personal, legal, and societal stakes – and high costs of error – that are intrinsic to these domains. And it has replicated, by some accounts even magnified, pre-existing biases present in these training data, and projected them forward into future decisions.
Our workgroup reflected on the question of AI impacts in broad historical context: in effect, we took seriously the analogy to past technology-fueled revolutionary transformations of human society such as the industrial revolution. But we did this with a perspective opposite to much current debate, considering the prospect for societal impacts that are transformative in scale but beneficial in valence. Speculations about huge societal benefits from AI are common, but tend to be superficial and conclusory, often based on speculative gains in single areas such as medical care or scientific research. By contrast, speculations on AI-driven dystopias are frequent and attention-getting, often with their causal mechanisms characterized in some detail. 4
A Historical Analogy
In our inquiry, we drew insight and inspiration from a line of commentary on past societal transformations that gets insufficient attention in current debates on technology impacts – despite being a prominent theme in the work of a few distinguished scholars such as Albert Hirschman and Elizabeth Anderson.5 These scholars point out that at the time modern liberal states, market capitalism, and associated technological changes were emerging, these trends were widely heralded as drivers of political and economic progress, relative to aristocratic social hierarchies, promising not just greater liberty – one part of the argument that remains prominent in modern political discourse – but also increased equality (in some accounts also fraternity or comity – to complete the revolutionary triad). These promised and briefly realized happy trends reversed, as technologies of the industrial revolution and their economies of scale drove vast accumulations of capital and separated the previously tight connection between markets and equality. Progressive reactions from governments (e.g., anti-trust) and new organizations (e.g., labor unions, charitable foundations) mitigated these trends to better balance autonomy, prosperity, and equality – a balance that current technological and economic trends are disrupting.
Our group aimed to re-open this question in the current context of rapid advances of AI. Can these transformative capabilities deliver on the old promise of technology as both liberator and equalizer? Can they do so in a way that is compatible with foundational moral and constitutional principles, and democratic institutions: e.g., freedoms of speech, association, religion, and the press; and private property rights with markets allocating resources through voluntary transactions, except insofar as these implicate external harms or public values (and as Mill reminds us, without drawing these public bases for concern so broadly as to undermine the basic liberty presumptions).6 And if this is all possible, what would it require: what are the key conditions that would mediate the ability of AI to help advance such a happy social vision?7
In considering this question, we did not elevate technological characteristics to the exclusion of social and political context; but we did consider technical and political forms of the question separately. First, what technological characteristics of AI systems and applications are likely to promote good societal outcomes? And second, what economic, social, and political conditions – including, concretely, what feasible business models – are likely to promote AI technology developing in these benevolent directions, and be sustainable over time?
Promising Directions: Technological Characteristics
In considering the technological part of the question, we focused on two broad technical attributes that we speculate may help direct AI's transformative societal impact toward the good: one related to the form and structure of decision-making, and one related to the distribution, scope, and number of separate AI agents.
Decision-making structure: Single-valued optimization, versus robustness and pluralism?
Virtually all automated decision systems – modern machine-learning systems and conventional algorithms alike – operate by optimizing a single-valued scoring or objective function. This is most obvious in the case of known preferences and conditions of full certainty, but similar approaches are used under uncertainty: maximizing an expected payoff or expected utility function, based on specified probability distributions, sampling from specified uncertain parameter inputs, or data assimilation from concurrent observations. These approaches all optimize a single-valued function relative to a single characterization, deterministic or stochastic, of conditions in the world.
There is an alternative, less unitary approach to decision-making, which initially grew out of concepts of satisficing, bounded rationality, and multi-criteria decision-making.8 This alternative approach, one prominent form of which is called "robust and adaptive decision-making" (RDM), seeks decisions that perform acceptably well over a wide range of possible realizations of uncertainties, rather than performing maximally well under any single specification, whether deterministic or probabilistic. RDM has extensive experience in diverse decision application areas. It has not been used in AI or machine learning, but we conjecture that it may have powerful implications, broadly consistent with the progressive social values we aim to advance.
The seed for this hopeful speculation lies in the fact that RDM is not just robust over alternative realizations of uncertainty about the world: it is also robust to uncertainty in the decision's goals or the range of values it implicates. RDM thus holds the potential to be more pluralistic, more compatible with both uncertainty and diversity of values – and thus, perhaps, with more inclusive and more equitable AI-driven decision-making. We realize that this is hopeful speculation about potential technical capabilities and the societal implications of their application, not a demonstrated characteristic of AI systems. But while the capabilities and associated questions remain largely unexamined, they clearly merit high-priority investigation.
The Number and Orientation of AI agents: What actors, and what aims, do they serve?
There is a wide range of speculation on the number, deployment scale, and objectives of future AI systems, ranging from each person commanding multiple AI agents for different purposes, through one integrated AI that does everything for everyone. Present AI developments consistently show a much narrower pattern, which is not necessarily well aligned with broadly distributed societal benefits. Most current efforts and most important recent advances have come from large, well-funded organizations: for-profit corporations, free-standing laboratories and institutes, and universities, some surrounded by clusters of small startup firms, with widely varying levels of government financial support and control among countries.
The most prominent current deployments of algorithmic decision-making are offered by private, for-profit firms, many of them in settings where the deploying party has a dominant position in the relevant interactions: Amazon toward purchasers and 3rd-party vendors; Facebook toward social-media users; Google toward its service users and data providers; Uber toward drivers. In these settings, users can observe only a small slice of the system's performance in its interactions with them, but have virtually no information about its broader operations, including what it is optimizing. In all these interactions, commercial or not, the available evidence – plus common sense – suggest that the systems are optimizing for the interests of the dominant actor, taking lesser account of the interests or welfare of the user only as needed to advance the primary aim – and, moreover, are doing so in ways that take advantage of the dominant actor's market power.
But this structure of relationships is not a necessary consequence of algorithmic decision-making or decision-support systems. One can imagine a wide range of other possibilities for how AI systems are deployed, some of them more compatible with a reduced concentration of power. An obvious and widely discussed possibility would be general-purpose AI assistants serving individual people, either unitary systems or integrations of multiple special-purpose systems. Such agents could act as information source, advocate, and negotiator for their clients in multiple interactions. They could provide suggestions and recommendations, manage the mechanics of transactions, and bargain on your behalf in consumption and other commercial interactions, both present ones and new ones it would enable – e.g., renting out your car, tools, or other costly assets when you do not need them. They could play similar roles in financial and investment decisions, and in labor-market participation. In situations of conflict or interaction with authorities, they could aid you in negotiations and advise you on your legal rights. They could support and advise your political participation, whether through existing channels such as voting and candidate support or through new, AI-enabled processes that combine elements of representative and direct democracy, such as issue-specific proxy delegation or other forms of "liquid democracy." And they could act as a personal coach, helping you make decisions and manage your time in line with your goals and values. The biggest challenge in creating such systems – as we discuss below – would be defining their objectives to reliably align with their user's welfare and values.
Alternatively, rather than serving individuals, AI systems could operate enterprises or collections of assets, to perform specified functions or advance specified interests aligned with social good. For example, AI systems – perhaps self-owned or self-controlled – might operate businesses or parts thereof such as individual factories; apartment buildings or larger-scale collections of housing or other buildings; public transit systems or other infrastructure components; or specific functions of government decision-making, in cases where the delegation of authority and the specification of relevant values to advance are unproblematic.
AI agents could also be deployed at higher levels of aggregation, to inform or guide the joint actions of groups of people in pursuit of their shared interests in some specific domain – whether commercial, political, recreational, expressive, religious, or something else. Relative to personal AI assistants, these agents would operate at a scale that is broader in the people whose interests are served, but narrower in the range of functions being pursued or interests being advanced.
Finally, one could imagine AI agents deployed at the level of the entire polity in some jurisdiction, centralizing decision-making on state functions in pursuit of some legitimate and widely agreed conception of the aggregate social good. There is of course some tension in using AI this way to increase human liberty and agency. Can we really claim to advance liberty and agency by centralizing state control? But these are largely the same tensions as attend state authority guided by humans. The state is a strong centralizer of power. But in liberal democratic states, this centralization serves the interests of order and security, including displacing other, less legitimate forms of concentrated power that are likely to arise in the absence of the state. And moreover, liberal states exercise this power lightly, so as to enhance liberty and welfare, only coercing citizens as needed to pursue legitimate public purposes.
Considered overall, this collection of potential AI deployments might tend to have an hourglass structure. As the scale of deployment moves from individuals to groups, the functional scope of the AI narrows to specific aims of particular groups; then at the highest level of jurisdictional aggregation, the scope of AI decision-making returns (or can return) to the comprehensively broad, imperfectly known set of interests that are the legitimate purview of state authority.
Examples of large-scale social reorganizations that would potentially be feasible with such a collection of AI agents would include the following:
– Breaking up monopolistic social-media platforms into multiple distinct platforms, each managing members' interactions by internally agreed rules and mediating the interactions between insiders and outsiders. AI would be used to facilitate such a breakup by overcoming the incumbent advantage due to network externalities. This is already happening in a small way, with the growth of new social media platforms such as Diaspora and Mastodon with commitments to stronger privacy protections than present platforms.
– New ride-sharing platforms, in which AI is deployed not as an instrument of the network's monopoly (or the Uber/Lyft duopoly) over drivers, but instead deployed to serve drivers and driver collectives, interacting with multiple counter-parties (current ride-share companies, potential new entrants, and others), and with riders, who in turn might be interacting with the system through their personal AI assistants. AI in such settings could optimize contractual terms to maximize shared value, while also equitably distributing surplus value between drivers, riders, and providers of other factors of production – including paying returns to capital and managerial services, but on competitive rather than monopolistic terms.
– A similar but broader labor-force model, not for people providing one service to one business (e.g., drivers and Uber), but with AI intermediaries enabling groups of individuals to come together to offer products and services to the market without the need for corporate intermediaries. Such groups could utilize distributed supply chains, to which they submit opportunities and find others who wish to join and bid to offer their services. As in the narrower, ride-sharing case, AI agents could optimize contractual terms, including provisions for duration and modification, based on the preferences of the participating individuals.
– AI-mediated political interaction – among citizens, activists, politicians, and political parties – to provide more civil and substantive deliberations, and more effective, informed, and flexible translation of citizen preferences into collective decisions. Such systems might aim to provide equal opportunity for political participation to all citizens, to motivate and reward virtue and moderation rather than vice and extremism, and to be dynamic – able to update, including updating collective understanding of what counts as virtue or vice. In contrast to the preceding examples, which would replace commercial transactions, this one would require more manipulation of incentives for individual behavior in pursuit of collective interests of civility, moderation, and reasoned debate. It could be designed to reward – with more influence and scope to reach a broad audience – those who best exhibit those virtues, rather than rewarding volume, belligerence, extreme views, personal attacks, skilled manipulation, or outrage.
– Within this context of AI-mediated political interactions, AI could discharge certain administrative functions of the state, mitigating the long-standing tension between expert and democratic control noted by Weber. AI's exercise of these functions would be guided by objective functions tuned by democratic deliberation. Through AI-facilitated direct deliberations or some equivalent quasi-legislative process, citizens would define large-scale aims and principles, set parameters for AI objective functions, then observe the results and iteratively adjust those parameters to steer toward a preferred balance of multiple societal aims. Such an administrative AI would in effect act like regulatory agencies under present administrative law, but with more explicit and more consistent parsing of authority between high-level democratic goal-setting and technically skilled implementation.
Realizing any of these alternative models of AI deployment would pose major challenges, which include significant technological elements even though they are not exclusively technological in character. A central challenge, perhaps the fundamental one, is appropriately defining the AI's objective function. Even with the shift toward a more robust and pluralistic approach as discussed above, this would imply three additional subtle and related requirements.
First, in any application the AI must act as a faithful agent of its intended beneficiary, whether this is an individual or a group of any size. The AI pursues its beneficiaries' values and interests – not the interests of its maker, not even when it must resolve ambiguities or indeterminacies in its understanding of its beneficiaries' values. This would represent a major departure from presently deployed AI systems, including those that are approaching the role of general-purpose personal assistants. These are developed by firms with interests in the user's behavior, and thus in manipulating that behavior or harvesting the user's information – even if that manipulation may be subtle and the systems seem to optimize for the user's preferences. These systems are also developed in the context of various commercial and state interests in creating over-rides or back doors, in order to allow surveillance and control contrary to the user's interests.
Even assuming this first condition is met, so the decision scales are not tilted to favor the maker's interests, systems interacting with individuals face a second challenge of understanding the determinants of the user's true values, interests, or welfare, as distinct from their immediate impulses or desires. This is hard to define, imperfectly inferable from observed behavior, prone to error, and in need of continual adjustment and updating. Like a wise parent or a skilled life coach, such systems would nudge the user's choices in directions judged likely to be compatible with their long-term flourishing – with the key difference from parenting (although not from coaching) that the ultimate authority in the relationship lies with the user. This would require a delicate balance, by which the system pushes against immediate preferences and desires when these appear to be at odds with the client's values or long-term interests. But to do this, the AI assistant must build a model of the client's values and long-term interests, based on data available to it. The system will thus sometimes make mistakes, and so will need to recognize uncertainty, make some of its recommendations tentative, and sometimes consult and ask for help – while also still using its present, uncertain knowledge to configure the choice space in ways likely to tend to beneficial outcomes. There will thus be a core design tension, between allowing human over-ride of AI recommendations and putting some degree of burden or barrier in front of instant, effortless, or wholesale over-ride.
A related but even sharper tension will be present in the case of people with destructive, malicious, or criminal preferences. Even liberal states do not honor or aim to fulfill the preferences of every citizen, independent of collectively exercised moral judgments. One can readily imagine a sexual predator or other criminal wanting their AI agent to help identify victims, assess the threat of detection or apprehension, manipulate victims to not resist or not report, or pursue other clearly nefarious aims. One problem here is defining the boundaries of permissible preferences – a challenge similar but not identical to that in non-AI contexts of defining the boundaries of criminal or civil wrongdoing – except that, as in so many domains, making scoring or objective functions explicit can be troublesome in cases where maintaining ambiguity provides needed social cohesion or moral comfort. Even assuming appropriate definition of the boundaries of permissible user preferences, a related design problem will be protecting AI systems against hacking or manipulation to enable such uses – either by intentionally disabling the AI's "conscience" functions, or by misrepresenting intentions in planning or multi-step execution of bad acts. We want individual AI agents that can distinguish their user's seeking an out-of-the-way place for a quiet picnic, or to carry out a murder.
Additional requirements and challenges would apply to AI agents managing enterprises or assets: e.g., self-directed AI corporations, housing developments, or transit systems. First, should the substantive decision scope of such agents be narrowly circumscribed and fixed? This raises issues analogous to those in current law regarding charities or other non-profit organizations seeking to change their original missions. Narrow and fixed goals would risk restricting behavior so the AI cannot respond appropriately to changed knowledge and conditions; but changeable goals risks letting the AI transit system decide to go into the adult film business instead – whether because it judges the change would make more money, generate more happiness, or better promote peace and order. Second, can behavior be constrained to be legal and ethical in a way that is sufficiently clearly defined and does not put such enterprises at competitive disadvantage relative to others playing by looser rules? Third, can objectives be tuned to not accumulate rents in excess of the costs of all factors of production? If so, these enterprises might be able to out-compete others that are pursuing and taking rents, and so form the kernel of a gradual erosion of concentrated economic power – unless the others are pursuing an Amazon strategy, taking losses for a long time to secure a dominant market position thereafter. Alternatively, if rents do accrue – as they sometimes will – what should be done with these? Presumably they should not be retained within the individual enterprise, but instead distributed in line with the system's large-scale aims. But does this mean to the Treasury? Or perhaps to a pool dedicated to financing the capital needs of the broad "social-progress-through-AI" enterprise, as discussed below? Finally, if these bodies sometimes go bankrupt – as seems likely, given the constraints imposed on them – how can one ensure that they quietly accept this fate, and what should happen to their assets when they do? As UCLA law professor Dan Bussell argues in a forthcoming paper, AI enterprises may need a new kind of bankruptcy court.
When AI systems are deployed to serve multiple people, to inform people's interactions with each other or advance group interests and values, additional challenges and design tensions will arise. These problems are similar, whether the structural approach to decisions involves collective decision-making or bargaining among individuals' AI agents, or some separate AI agents operating at a higher, collective scope of authority. The challenges all follow from a basic fact: in any decision situation involving multiple people, there are multiple measures of welfare. These are sometimes aligned, but they can also exhibit disagreements, rivalrous claims on the same resources, collective-action problems, or other tensions. Most often, there is some mixture of aligned and opposing interests. In such situations, even formal game-theoretic outcomes can be ambiguous due to the existence of multiple Nash equilibria. There can also be inferior collective outcomes from individual choices that are locally advantageous, or inequitable outcomes in distributive negotiations that favor the most aggressive bargaining tactics.
Even assuming AI agents reflect individual values well, guiding or informing such multi-person interactions presents several additional design requirements. The systems would need to identify and avoid collectively inferior outcomes – even if they are equilibria – by providing coordinated nudges to steer parties toward collectively superior outcomes. They would need to apply the same gentle resistance against self-destructive impulses as at the individual level, now with the added requirement to steer groups against choices driven by collective-level pathologies such as envy, malice, hostile stereotypes, or escalation dynamics and other entrapment mechanisms. And they would need to address the problem of aggressive bargaining behavior, recognizing that this often succeeds at securing favorable one-time outcomes in divide-the-pie negotiations, at the cost of inferior collective outcomes and damaged relationships. The systems would have to both refrain from such behavior on behalf of individuals, and not reward it in determining collective outcomes. These requirements and the associated bargaining pathologies are best understood in commercial interactions, but they have close analogies in other domains. A salient current example is maintaining civil discourse, in politics and online, in the presence of powerful attention-getting advantages in being colorful, extreme, and uncivil – a domain in which a few experiments have shown that AI agents can make the problem worse, if they are trained on the actual content of current discourse.
Achieving these aims would require that an AI system managing collective decision outcomes would need both the knowledge to identify collectively superior and inferior outcomes, and the ability to apply defensible principles for fair division of surpluses and resolution of conflicting preferences. If collective decisions are handled by collectively accountable AI agents, these would need to reliably observe the preferences and values of all affected people, plus relevant information about the world that shapes the set of feasible outcomes – a tall order. On the other hand, if collective decisions are handled by interactions among individual AI agents – each presumably with better information about its own user's preferences and values — then the individual agents' bargaining behavior must be subject to constraints guided by collective welfare: e.g., seeking to maximize joint gains; not pursuing these by shifting negative externalities onto others not present in the interaction; fair dealing with each other, in both process and substance; and refraining from destructive bargaining tactics even when these promise a one-time advantage.
Some form of regulation at the collective level appears to be needed, but defining (and automating) precise rules will pose severe challenges. In different decision domains, the needed functions might be characterized as mediator-arbitrators, content moderators, or judges. Should these be AIs, humans, or machine-human partnerships? How can these processes be made robust against sophisticated attempts to capture them for partisan advantage? If the aim of these is to advance widely (but perhaps not universally) held collective values, how broadly should they be binding in domains such as political discourse that implicate free speech and other liberty values? And to the extent these processes supplant human decision-making – which traditionally advances collective aims by some combination of formal regulation and propagation and maintenance of social norms – might widespread assumption of these duties by AI risk atrophy of the associated skills, sense of duty, and other virtues in humans?
Promising Directions: Social, Political, and Strategic Issues
Summarizing the above, the technical AI characteristics we speculate likely to be associated with good societal impacts include the following:
– AI does not irreversibly alienate individual human agency in any domain;
– AI objective functions are tentative and pluralistic, along the lines of RDM, rather than single-minded and dogmatic; they admit multiple possibilities in outcomes and values, recognize limits to their knowledge of these, and know when and how to ask for additional information or guidance;
– AI performance is monitored and adjusted over time with significant input from people, acting alone for their personal AI's or in democratic, deliberative groups for AIs with collective or society-wide responsibilities;
– AI agents must be are trustworthy in all respects. Individual AI agents pursue the interests of their client rather than any developer or vendor; and they pursue the true, long-term interests and values of their client, via recommendations, nudges, and exhortations – acting like a wise parent or friend. AI agents acting, mediating, or arbitrating on behalf of collections of people follow principles of fair dealing and equitable distribution of surpluses among participating parties, and incorporate interests of other actors or values outside the participating parties only insofar as these represent real externalities.
Having speculatively identified these desirable technical characteristics of AI systems, we then asked how such technical systems might be developed, deployed, scaled, and sustained over time. These are questions of political and economic strategy. The proposed innovations – in addition to being uncertain and weakly characterized – would represent attacks on existing concentrations of wealth and power and the rents that sustain these. There are thus likely to face, at a minimum, challenges in securing the resources they need to be created, established, grow, and sustain; and more likely, will face determined and strategically sophisticated opposition.
Getting a Start:
In this situation, the first challenge will be getting such systems developed and deployed. What this requires will depend on the details of the relevant systems and the inputs needed to produce them – the production function for AI capabilities – all aspects of which are deeply uncertain.
On this, an initial issue to consider is whether systems with the desired characteristics can be reliably developed by modifying other systems that were developed by and for current commercial actors – assuming these can be legally acquired. If they can to some degree, then the key questions are, first, trust and reliability – how can we verifiably assure that the systems so ported do not sneakily import the interests of their developers – and second, what additional resources and inputs are needed to modify systems and deploy them for their new purposes?
At best, the desired systems would need training procedures and data for their newly targeted uses, related to the individual or collective values to be served. This might be cheap and easy; it might be expensive and difficult; or it might be impossible, at least initially, because data relevant to the newly targeted uses and goals might not exist. Oddly, there is likely to be more and better data available to serve vendors' commercial interests – which depend on observable matters such as attention, time spent, and purchasing and other behavior – than is available to serve individual and collective values. Data presents other challenges as well, including the possibility that no truly general-application AI can be developed given jurisdictional divisions and restrictions on data access and use; and the present dependence of AI progress on a huge volume of labeled data, which in turn depends on a huge, low-wage workforce doing this essential step.
The less fortunate case would be that new systems with the desired characteristics must be developed from scratch. In this case, the same data concerns identified above would still apply. But there would also be a greater need for other inputs, for initial system development and deployment and for continuing maintenance, adaptation, and upgrades. These needs are probably similar for key advances in multiple areas of AI development, independent of the specific form of objective or the scope of application. In addition to suitable training data, these include highly skilled technical personnel; hardware-based computing power; and capital – lots of capital judging from present industry structure, although this could change.
The premise of the new AI developments we seek is that, unlike the present system, successful development of useful capabilities, even achieving crucial technical advances, will not create fabulous wealth for developers or their employees, collaborators, or investors. So how can the needed developments be effectively motivated? The recent case of OpenAI reconstituting itself as a for-profit corporation because it could not raise enough capital as a not-for-profit AI developer provides a germane cautionary example.
Our discussions identified several promising elements of potential development models. The first concerns identifying early targets, current products or present or potential uses to displace. Promising targets might include products that are now gathering the largest rents, or that are targets of the strongest current objections and political threats, or for some other reason are ripe for raiding. Other promising factors would include consumers' willingness to incur a little inconvenience from switching costs; perhaps also a preference for local providers and small-scale relationships. The aim would be to target early penetration there, with alternative products that distribute the rents or other values to their users, not the vendors.
The second element is assembling and mobilizing the needed factors of production. On this, the initiative could start with crowd-sourcing, philanthropy, or other sources of capital motivated by social goal rather than profit – although these sources are usually much smaller than investment-motivated capital. An open-source development model may hold advantages, including facilitating engagement of top technical talent and mobilizing utopian and anarchic strains within the technical community. Such an initiative would provide an opportunity to probe the depth and sincerity of the revolts by high-tech workers against narrow conceptions of their employers' self-interest, inviting them to put their money and skills where their mouths are.
All aspects of this strategy – including, crucially, attracting capital and pro bono talent – would benefit from well-branded, highly attractive initial projects: e.g., the faithful individual AI helper, or the AI facilitator of civil political discussion and collective action (both of which may represent compellingly attractive aspirations, but would clearly need better names).
Not all philanthropy pursues aims that are clearly benign and universally agreed, of course. Sometimes it makes sense to worry about limited or partisan social objectives in philanthropy: For example, don't solicit support for your climate-change campaign from the Koch Foundation. But this concern might be less serious for AI than in more established policy areas with well-known lines of political alliance and opposition. Libertarian philanthropists – yes, perhaps even the Koch Foundation – may well support the aim of empowering individual liberty and agency with individual-level AI agents. As for group-level AI agents advising different decisions to advance different aims, these will be multiple overlapping agents operating in a pluralistic setting, so the risks of capture by any limited or partisan view of the public interest may be less severe.
Persisting and Scaling:
Once socially beneficial AI capabilities are deployed, they still need mechanisms and resources to persist, scale, and sustain their position. Moreover, they must do this in a way that maintains their alignment with citizen and public values and remains attractive to users – even once the initial novelty of the initiative has passed, with possible decline in the enthusiasm of pioneer supporters and developers. The initial sources of capacity may not be enough to persist under these conditions, or to overcome the sustained advantage of strategically sophisticated and ruthlessly self-interested incumbents, who might respond by deploying cheap attractive systems as loss-leaders to secure longer-term advantage. The enterprise will need to maintain needed access to technical expertise and capital, whether from associated revenues or from investors.
Some present business models, such as relying on advertising, clearly appear not to be viable for this project, but several others appear plausible. One possibility would be subscription or purchase, although the implications of alternative ownership models and their compatibility with the large-scale aims – do I purchase my AI assistant and related supporting systems or rent them, and from whom – would require careful thought. If the services provided by AI systems include facilitating transactions or cooperative activity with exchange of money, the system could take a fee to cover development costs, provided the fee is perceived as reasonable and its basis fully disclosed. Another possibility would be a co-operative enterprise model. These organizations reach large scale in some jurisdictions with strong historical traditions of self-organized cooperative activity and supportive policy environments.
There might be bootstrapping possibilities, based upon the use of AI. Early AI's might be developed to help identify targets and strategies for subsequent expansion. They might provide information, services, and access to resources that have traditionally been provided by venture capitalists or other early-stage private investors. They might help identify points in current supply chains or production models that are rigid or constrained, or where market power is hindering rapid development and deployment.
Another novel approach might be to turn the widely denounced short-termism of capital markets to advantage, by deploying AI agents that pre-commit to change their behavior over time. An AI raider could initially pursue maximum short-term competitive advantage, but with a binding commitment to change course in the future. If its short-term competitive advantage is based on strong IP, for example, the commitment might be to unlimited free licensing after the initial period expires. A policy change to support this might be a new form of IP, based on modifying either patent or copyright, that combines highly advantageous short-term protections with an iron-clad, non-contestable commitment to expiry and full release to the public domain thereafter.
As the endeavor succeeds and grows, it will encounter changes in its strategic and competitive conditions. Some of these will work to its benefit: for example, open networked organizations pursuing broadly public aims are likely to have an easier time pooling and sharing data than rivalrous commercial organizations. Other changes will represent new challenges that increase costs or other barriers. As the new systems grow to mediate decisions that channel large sums of money, they will attract hackers and others interested in subverting them, and will have to develop robust security protections. Stringent open-source review can provide part of the needed protection, but some risk will remain. It will also be necessary to be vigilant about the interests of continuing sources of finance: any source motivated by financial return will present ongoing risks of subtle distortions of aims, and the associated prospect of simply replacing old centers of concentrated power by new ones just as determined to sustain their position.
Finally, if the endeavor succeeds so well that some combination of individual AI assistants, autonomous AI enterprises, and AI-mediated collective interactions – all with the desired characteristics – becomes the dominant model for societal deployment of AI, it will be necessary to grapple with the question of innovation. Current law and policy assume that the main incentive to innovate comes from the pecuniary motive of earning rents, from the innovations themselves and from IP protection around them. With AI agents eschewing most or all of the rents that provide enormous financial rewards to present market actors, where will the motivation and resources to support innovation come from? Several alternatives might be possible. Innovation might still come from people, businesses, or other organizations, including AI-facilitated innovation, stimulated by some combination of the pecuniary rewards that remain under the new model (which will be smaller than under the present system, but probably not negligible), plus intrinsic motivation to innovate and create – which the present system largely overlooks. AI agents might be able to fully take over the huge volume of prosaic, small-scale innovations now done for profit in enterprises seeking IP assets – many of small or questionable merit. AI agents could take over the pedestrian activities of searching through current technologies, patents, and scientific publications that power much such innovative activity, but do so with better information and processing capability and with objectives better aligned with the broad public interest – and with results placed in the public domain for free further exploitation. For larger-scale scientific, technological, artistic, and social innovation, intrinsic motivations have long been the dominant driver and it is reasonable to expect they will still be present in the new world. Indeed, they might be effectively aided by AI support tools.
Challenges, next steps:
The technological-political program of societal transformation we sketch here is bold, under-specified, and incomplete. It can be viewed as an attempt to update Alinsky's Rules for Radicals, Scott's Weapons of the Weak, and the Ethical hacking movement, for a new technological environment of greatly increased power for autonomous and semi-autonomous systems. It is bold in that we are proposing a new technological model of AI and its deployment that opposes the interests of present dominant incumbents – both private-sector actors whose revenues and business models would be threatened, and government institutions that would hold different and less extensive and exclusive as some decision authority shifts to networks of citizens and autonomous decision-making systems. It is essential not to be naïve about how large the barriers to entry are, or about the determination and resources of incumbents seeking to strangle the new model in its crib. The new model also opposes certain structural characteristics in the economy that tend to favor scale, and thus centralization. These include technical factors such as economies of scale and network externalities that are strongly shaped by characteristics of production technologies; and factors more institutional and political in origin, such as fixed costs from regulatory obligations advancing various public values such as environment health and safety, consumer protection, etc.
The new model is under-specified, both in its technical and its political/strategic dimensions. Technically, we sketch a couple of salient system characteristics that appear likely to push in the desired direction, but the devil is in the details. A wide range of systems and design approaches is now being pursued and developed in parallel, with capabilities – depending on multiple factors in the systems and their contexts – that might favor or oppose liberty, privacy, agency, and equality. Even current developments have had a mix of centralizing and decentralizing effects, empowering many distributed activities even as they create great new centers of wealth and power, including new forms of power not yet exploited or even well understood.
As a strategy to move toward this vision, we have identified a few possible pathways to pursue it through private action. But it is also worth asking whether appropriate government policies would be necessary or helpful, and if so, what form. Possible points of leverage might include data ownership policies such as clear conferral of data property rights on individuals, or limits on concentrated holding of data; limits on or new forms of IP; or more expansive definition and robust enforcement of anti-trust policies. To the extent the desired transformation does require public policy, one might also consider which jurisdictions would be most promising to seek an early strategic foothold. Perhaps the social democracies of Europe, which are already leaders in data and privacy policies? Or perhaps major developing countries with strong technological capacity – who would have the advantage of large domestic markets for early scaling, but might also be ambivalent toward the leveling ambition, depending whether leveling is construed as between countries (in which case they would presumably be keen advocates) or within countries (in which case, maybe not). In this global context, one must also consider risks posed by opportunistic geopolitical adversaries, including the possibility of surreptitious early support for the development of the new systems coupled with efforts to bias or undermine its aims – although this threat might become less salient over time if one consequence of the spread of the new systems is a decrease in international rivalries.
Finally, the proposed new model is incomplete. It is unlikely to address all impacts and social disruptions caused by rapid advances in AI. In particular, we can't necessarily expect it to avoid large-scale displacement of livelihoods by AI. It might, however, make mass unemployment less individually and socially destructive, perhaps even make it desirable. If leveling of power implies different bases for distribution of economic output, no longer coupled to employment, then loss of employment might cease to be catastrophic. This might seem inconceivable, but it could be analogous to the treatment of health insurance across nations: in the United States it is tightly coupled to employment and thus highly unequally distributed, while in all other advanced democracies it is uncoupled from employment and more equally distributed. It is even possible that mass unemployment – not under present social organization, but in a levelers world – could be profoundly liberating, enabling people to work, individually or cooperatively, on endeavors they value that are not necessarily related to the production of material goods and services. As AI facilitates efficient production, it could also facilitate effective pursuit of these other aims.
In closing, AI is likely to have huge, transformative societal impacts, for good or ill, but present patterns of development and deployment suggest that small "AI for good" movements are likely to be overwhelmed by massive developments that serve concentrated commercial, political, and strategic interests. With such labile technology and such potentially vast impacts, the possibilities for positive transformative change are real, but highly uncertain in their detailed requirements and pathways – and are not being pursued with resources commensurate with their importance, or with the resources directed to systems serving private or rivalrous advantage. With such huge stakes, it is clearly worth pursuing even ill-defined and speculative investigations of how to effectively shift the balance toward the good. We have identified a few possibilities that seem promising, but we fully realize that these are the output of just two days of discussions and are speculative, incomplete, and under-specified. Yet despite all the challenges, further pursuit of these questions, drawing on more breadth of relevant expertise, is a high and urgent priority.The post Could AI drive transformative social progress? What would this require? first appeared on AI Pulse.