The Whiteness of AI

Introduction It is a truth little acknowledged that a machine in possession of intelligence must be white. Typing terms like "robot" or "artificial intelligence" into a search engine will yield a preponderance of stock images of white plastic humanoids. Perhaps more notable still, these machines are not only white in colour, but the more human they are made to look, the more their features are made ethnically White.  1  In this paper, we problematize the often unnoticed and unremarked-upon fact that intelligent machines are predominantly conceived and portrayed as White. We argue that this Whiteness both illuminates particularities of what (Anglophone Western) society hopes for and fears from these machines, and situates these affects within long-standing ideological structures that relate race and technology. Race and technology are two of the most powerful and important categories for understanding the world as it has developed since at least the early modern period. Yet, as a number of scholars have noted, their profound entanglement remains understudied  (Sinclair 2004; de la Peña 2010) . There are a number of possible reasons for this-and, as Bruce Sinclair writes, "racial prejudice dominates all of them"  (Sinclair 2004, 1) . They include the lack of first-or secondhand accounts of the role of people of colour in the development and use of technology; persistent stereotypes about technology as the province and product of one particular racial group-White people; and the persistent tendency of members of that group, who dominate the academy in the US and Europe, to refuse to see themselves as racialised or race as a matter of concern at all. This lack of scholarly attention is surprising because, as Michael Adas elucidated in 1989, the idea of technological superiority was essential to the logic of colonialism. Not only was superior weaponry and transportation (etc.) necessary for large-scale conquest and control of foreign territory, it was also part of its justification: proof that White Europeans were an advanced civilisation with a right to rule over others  (Adas 1989) . Fortunately, this lack of attention is increasingly being remedied, and the relationship between race and technology is beginning to garner the kind of attention that has since the 1970s been given to gender and technology, following the pioneering work of Donna  Haraway, Sandra Harding, and Evelyn Fox Keller (Haraway 1991; Harding 1986; Keller 1985) . This includes attention to this century's ubiquitous digital technologies. In 2006, Lisa Nakamura asked, "How do we make cyberculture studies a field that as a matter of course employs critical race theory and theories of cultural difference…?"  (Nakamura 2006, 35) . Since then, a number of significant works have attempted to do just that, including Safiya Noble's Algorithms of Oppression and Ruha Benjamin's Race After Technology  (Noble 2018; Benjamin 2019) . This paper aims to contribute to this body of literature on race and technology by examining how the ideology of race shapes conceptions and portrayals of artificial intelligence (AI). Our approach is grounded in the philosophy of race and critical race theory, particularly the Black feminist theories of bell hooks, Sylvia Wynter and Alexander G.  Weheliye (hooks 1992 Weheliye (hooks 1997 Wynter 2003; Weheliye 2014), and work  in Whiteness studies, including that of Richard Dyer, Joe R. Feagin, and Ruth Frankenberg  (Dyer 1997; Feagin 2013; Frankenberg 1997a) . In 2006, Feagin coined the term "white racial frame" to describe those aspects of the Anglophone Western worldview that perpetuate a racialised hierarchy of power and privilege  (Feagin 2006 ). In his words, "the white racial frame includes a broad and persisting set of racial stereotypes, prejudices, ideologies, interlinked interpretations and narratives, and visual images"  (Feagin 2013, xi) . Although it reached its peak in the age of colonial expansion, this framing persists: "Today, as whites move through their lives, they frequently combine racial stereotypes and biases (a beliefs aspect), racial metaphors and concepts (a deeper cognitive aspect), racialised images (the visual aspect), racialised emotions (feelings), interpretive racial narratives, and inclinations to discriminate within a broad racial framing"  (Feagin 2013, 91) . In essence, this paper examines how representations of AI reflect this White racial frame. One of the main aims of critical race theory in general, and Whiteness studies in particular, is to draw attention to the operation of Whiteness in Western culture. The power of Whiteness's signs and symbols lies to a large extent in their going unnoticed and unquestioned, concealed by the myth of colour-blindness. As scholars such as Jessie Daniels and Safiya Noble have noted, this myth of colour-blindness is particularly prevalent in Silicon Valley and surrounding tech culture, where it serves to inhibit serious interrogation of racial framing  (Daniels 2013 (Daniels , 2015 Noble 2018) . Hence the first step for such an interrogation is, in Richard Dyer's term, to "make strange" this Whiteness, de-normalising and drawing attention to it  (Dyer 1997, 10) . As Steve Garner puts it, the reason "for deploying whiteness as a lens is that it strips a normative privileged identity of its cloak of invisibility"  (Garner 2007, 5) . This is our primary intention in examining intelligent machines through the White racial frame. In the next section of this paper, we first lay out current evidence for the assertion that conceptions and portrayals of AI-both embodied as robots and disembodied-are racialised, then evidence that such machines are predominantly racialised as White. In the third section of the paper, we offer our readings of this Whiteness. Our methods are qualitative. As de la Peña writes: "Studying whiteness means working with evidence more interpretive than tangible; it requires imaginative analyses of language and satisfaction with identifying possible motivations of subjects, rather than definitive trajectories of innovation, production, and consumption" (de la Peña 2010, 926). We offer three interpretations of the Whiteness of AI. First, the normalisation of Whiteness in the Anglophone West can go some way to explaining why that sphere's products, including representations of AI, are White. But we argue that this argument alone is insufficient. Second, we argue that to imagine an intelligent (autonomous, agential, powerful) machine is to imagine a White machine because the White racial frame ascribes these attributes predominantly to White people. Thirdly, we argue that AI racialised as White allows for a full erasure of people of colour from the White utopian imaginary. Such machines are conceived as tools that will replace "dirty, dull, or dangerous" tasks  (Murphy 2000, 16) , including replacing human interactions that are considered metaphorically dirty: White robot servants will allow the White master to live a life of ease unsullied by interaction with people of other races.
 Seeing the Whiteness of AI Our concern in this paper is with the racialisation (as White) of both real and imagined machines that are implied or claimed to be intelligent. By racialisation, we mean the ascription of characteristics that are used to identify and delineate races in a given racial frame, which in this case is the Anglophone West. Feagan notes: Among the most important ingredients of this frame are: (1) the recurring use of certain physical characteristics, such as skin colour and facial features, to differentiate social groups; (2) the constant linking of physical characteristics to cultural characteristics; and (3) the regular use of physical and linked cultural distinctions to differentiate socially "superior" and "inferior" groups in a social hierarchy  (Feagin 2013, 41) . It is worth noting that "physical characteristics" need not only refer to those that are visible: voice and accent are also used as markers for social categorisation. Similarly, the category "cultural characteristics" is also used expansively and can include markers such as dialect, mannerisms, and dress codes, as well as mental and moral qualities, such as diligence, industriousness, reliability, trustworthiness, inventiveness, and intellectual ability. Indeed, these mental and moral qualities have always been an essential part of the racial frame, as it is largely on the basis of these that claims of superiority or inferiority have been made.
 Machines Can Be Racialised That machines can be racialised, in the sense that they can be given attributes that enable their identification with human racial categories, has been empirically demonstrated. For example, in one study, Christoph Bartneck and colleagues took pictures of the humanoid Nao robot and adjusted the colouration to match the skin tone of stock images of White and Black people  (Bartneck et al. 2018) . They then asked participants to define the race of the robot with several options including "does not apply". A minority-ranging across the experiments from 7 to 20%-chose the "does not apply" option, while a majority-ranging from 53 to 70%-identified the robots as belonging to the race from which their colouration derived. They concluded "Participants were able to easily and confidently identify the race of robots according to their racialization [...] Thus, there is also a clear sense in which these robotsand by extension other humanoid robotsdo have race"  (Bartneck et al. 2018, 201) . This should not be surprising. Many machines are anthropomorphised-that is, made to be human-like to some degree-in order to facilitate human-machine interaction. This might involve obvious physical features (a head on top, two eyes, a mouth, four limbs, bipedalism, etc.), but it can also include invisible features such as a humanlike voice, or human-like interactions, such as politeness or humour. Given the prevalence of racial framing, in most contexts, to be human-like means to have race. Consequently, as Liao and He point out in their discussion of the racialisation of psychotherapeutic chatbots, "racial identity is an integral part of anthropomorphized agents"  (Liao and He 2020, 2) . They go on to explore a number of racial cues for virtual agents, including visual cues such as skin colour, but also cultural signifiers such as names (e.g. for male names, Jake as White, Darnell as Black, and Antonio as Hispanic). Similarly, "even text-based conversational exchanges"-that is, those with no visual component at all-"perform a racial or ethnic identity" through the interlocutors' choice of dialect, etc.  (Marino 2014, 3) . Given the sociopolitical importance of the racial frame in structuring people's interactions, if machines are really being racialised, then we would expect this to have an impact on how people interact with these machines. Numerous studies show just this. For example, Liao and He found that a person's "perceived interpersonal closeness" with a virtual agent is higher when the virtual agent has the same racial identity as that person  (Liao and He 2020, 2) . Other studies reflect the extent to which racism-prejudicial treatment on the basis of race-is intrinsic to racial framing. As detailed in their paper "Robots Racialized in the Likeness of Marginalized Social Identities are Subject to Greater Dehumanization than Those Racialized as White", Strait et al. analysed free-form online responses to three videos, each depicting a female-gendered android with a different racial identity: Black, White, and East Asian. Their aim was to assess whether the same kind of marginalising and dehumanising commentary that is applied to real people of colour would be applied to these robots. They found that the valence of the commentary was significantly more negative towards the Black robot than towards the White or Asian ones and that both the Asian and Black robots were subject to over twice as many dehumanising comments as the White robot  (Strait et al. 2018) . Two recent studies have further examined the transfer of bias to machines using the "shooter bias" paradigm. This paradigm was first described in the 2002 paper "The Police Officer's Dilemma: Using Ethnicity to Disambiguate Potentially Threatening Individuals"  (Correll et al. 2002) . It used a simple video game featuring images of (real) Black and White male targets, each holding either a gun or a nonthreatening object. Participants were instructed to shoot only armed targets. A clear racial bias was identified: "participants fired on an armed target more quickly when he was African American than when he was White, and decided not to shoot an unarmed target more quickly when he was White than when he was African American"  (Correll et al. 2002 (Correll et al. , 1325 . Studies by  Bartneck et al. and Addison et al. used  the same methodology to examine whether this "shooter bias" would be transferred to racialised robots  (Bartneck et al. 2018; Addison et al. 2019) . They found that "people showed a similar shooter bias toward robots racialized as Black relative to White in a similar fashion as they showed toward Black vs. White humans, no matter their own race"  (Addison et al. 2019, 493) .
 Whiteness as the Norm for Intelligent Machines The previous section shows that research has empirically demonstrated that machines can be racialised and that this racialisation includes transfer of the attendant biases found in the human world. In this subsection, we will survey evidence for the extent to which AI systems-machines purported to be intelligent-are predominantly racialised as White. We will look briefly at four categories: real humanoid robots, virtual personal assistants, stock images of AI, and portrayals of AI in film and television.
 The Whiteness of Humanoid Robots A number of commentators have remarked on the preponderant Whiteness of humanoid robots. In their proposed "code of ethics" for human-robot interaction Riek and Howard note the "lack of diversity in robot morphology and behavior": In terms of race, with precious few exceptions, such as Hanson's Bina48, the vast majority of android and gynoid robots are Asian or Caucasian in their features for no discernible reason. Furthermore, most of these robots tend to have a eurocentric design with regards to their appearance, behavior, and voice.  (Riek and Howard 2014, 4)  Human-computer interaction researchers Christoph Bartneck and colleagues, who conducted some of the studies cited above, have also noted that robots are usually racialised as White: "most of the main research platforms for social robotics, including Nao, Pepper, and PR2, are stylized with white materials and are presumably White"  (Bartneck et al. 2018, 202) . Finally, media studies and literary scholar Jennifer Rhee notes the "normalization and universalization of whiteness" as expressed both in earlier robotics research and in robot toys: "Kismet, with its blue eyes, light brown eyebrows, and pink ears, also 'normalizes whiteness', as do other robot companions, such as the blonde-haired, blue-eyed Cindy Smart Doll and the similarly blonde-haired, blue-eyed My Friend Cayla."  (Rhee 2018, 105) . Although robots such as Nao and Pepper have enjoyed commercial success, neither has received quite the attention garnered by Sophia from Hanson Robotics. This machine consists foremost of a White humanoid head, sometimes also with an upper torso (see Fig.  1 ). It has not only given numerous high-profile television interviews but also received political honours, including in 2017 receiving citizenship of Saudi Arabia and becoming an "Innovation Champion" for the United Nations Development Programme  (Weller 2017 ; UNDP 2017).
 The Whiteness of Chatbots and Virtual Assistants Though conversational agents do not exhibit any visual racial cues, they are racialised by means of sociolinguistic markers  (Sweeney 2016; Villa-Nicholas and Sweeney 2019) . Discussing ELIZA, an influential natural language processing program created by Joseph Weizenbaum at the MIT AI Laboratory in 1966, Mark Marino writes: "If ELIZA presented a bot that tried to imitate language, it was performing standard white middle-class English, without a specific identifying cultural inflection... language without culture, disembodied, hegemonic, and, in a word, white"  (Marino 2014, 5) . Since then, natural language processing has entered the mainstream, with "virtual assistants" existing in many people's pockets, handbags, or homes through devices such as smartphones. Indeed, this is one of the most common ways in which people interact with technology that could be labelled "AI". These tools present their designers with many decisions about socio-cultural positioning. Ruha Benjamin recalls this anecdote: A former Apple employee who noted that he was "not Black or Hispanic" described his experience on a team that was developing speech recognition for Siri, the virtual assistant program. As they worked on different English dialects -Australian, Singaporean and Indian Englishhe asked his boss: "What about African American English?" To this his boss responded: "Well, Apple products are for the premium market."  (Benjamin 2019, 28)  As a further example, she describes a Black computer scientist who chose a White voice for his app rather than a Black one, so as not to "create friction"  (Benjamin 2019, 28-29) . So while some designers might be unconsciously racialising their products as White, others are doing so in full awareness of this choice.
 The Whiteness of Stock Images of AI As anyone working in the field will know, stock images of AI, at least when anthropomorphised, are overwhelmingly white and arguably overwhelmingly White. The more realistically humanoid these machines become, the more Caucasian in their features. Such images are used to illustrate not only generalist newspaper articles and corporate slideshows but also specialist and technical works, and even works of a critical nature, such as Harry Collins's Artifictional Intelligence (Polity, 2018) and Anthony Elliott's The Culture of AI (Routledge, 2018) (Fig.  2 ). The prevalence of such images is reflected in the results of search engines. Such searches are a useful indicator of how a subject is portrayed at a given time, for two reasons. First, search engines are very widely used (approximately 3.5 billion searches are made on Google every day, or 40 thousand per second 2 ) and can therefore be considered a highly influential source of information and perceptions. Second, the nature of such search engines means that they are not only promoting certain ideas and perceptions but also reflecting their existing prevalence. While the exact nature of Google's search, ranking, and result presentation algorithms is proprietary, we know that they evaluate (crudely put) influence and popularity-for example, in terms of how many other sites link to a given website. So the fact that certain images are shown when someone searches for a relevant term means not only that those images are being thus promoted by some of the most powerful organs of content mediation in existence today but also that these images are already widespread and used on other influential websites, as that is what underlies their promotion by the search engines. Consequently, search results are increasingly examined by scholars, including in the study of racial bias. For example, in her 2018 book Algorithms of Oppression: How Search Engines Reinforce Racism, Safiya U. Noble identifies many ways in which such sites reflect and exacerbate prejudice, such as the search results for "Latinas" that feature mostly porn  (Noble 2018, 75, 155)  or the White men who come up when searching for images of professions such as "construction worker", "doctor", or "scientist"  (Noble 2018, 82-83) . In order to get an indication of the prevalence of these racialised machines on the internet, we conducted two image searches on Google (the most widely used search engine) using the anonymous Tor browser to ensure results were not influenced by our personal search histories and locations. We first searched on the term "artificial intelligence": the top results are in Fig.  3 . Some of these results are too abstract, featuring stylised brains and circuits, for example, to be considered racialised. However, among the results showing humanoid figures, racialisation as White predominates. First, two pictures show actual human hands, and both are White. Second, a further two pictures show humanoid robots, and both are White and could thus be read as White, as  Bartneck et al. suggest (Bartneck et al. 2018, 202) . Therefore, we might say that inasmuch as the machines are racialised, they are racialised as White. In order to focus more on representations of embodied, anthropomorphic AI, we also searched for "artificial intelligence robot": the top results are in Fig.  4 . As is clear, this search produces an even greater preponderance of images that are either white in colour or racialised as White or both.
 The Whiteness of AI in Film and Television These contemporary stock images distil the visualisations of intelligent machines in Western popular culture as it has developed over decades. In science fiction from the nineteenth century onwards, AI is predominantly imagined as White. For example, the Terminator (Arnold Schwarzenegger), RoboCop (Peter Weller and Joel Kinnaman), all of the "replicants" in the Blade Runner franchise (e.g. Rutger Hauer, Sean Young, and Mackenzie Davis), Sonny in I, Robot (Alan Tudyk), Ava in Ex Machina (Alicia The Whiteness of AI Vikander) (Fig.  5 ), and Maria in Metropolis (Brigitte Helm) are all played by White actors and are visibly White on screen. Androids made of metal or plastic are also usually given White facial features, such as the robots in the 2007 film I, Robot. Even disembodied AI is imagined as White: HAL-9000 in 2001: A Space Odyssey and Samantha in Her are voiced by White actors. All of these AIs come from Hollywood films; they have been produced in a country in which 18% of the population is Hispanic, but in which only one fictional robot has that background: Bender Rodríguez in the animated TV series Futurama, who is canonically constructed in Mexico-but who is voiced by the White voice actor John DiMaggio. Only very recent TV shows with a large cast of androids, such as Westworld and Humans, have attempted to address this with AI characters evincing a mix of skin tones and ethnicities. This preponderance of intelligent machines racialised as White led Dyer to posit "the android as a definition of whiteness"  (Dyer 1997, 213) .
 Understanding the Whiteness of AI We offer three interpretations of the racialisation of intelligent machines as White: the Whiteness of their creators perpetuating itself; the Whiteness of the attributes ascribed to AI; and the extent to which AI permits the erasure of people of colour from the White utopia.
 Whiteness Reproducing Whiteness In European and North American societies, Whiteness is normalised to an extent that renders it largely invisible. As Toby Ganley puts it in his survey of Whiteness studies, "the monopoly that whiteness has over the norm" is one of the field's two unifying insights-the other being that it confers power and privilege  (Ganley 2003, 12) . Richard Dyer describes this as the view that "other people are raced, we are just people"  (Dyer 1997, 1) . This normalisation means that Whiteness is not perceived by majority populations as a distinct colour, but rather as an absence of colour-colour Fig.  5  Alicia Vikander as Ava in Ex Machina. Source: Youtube both in the literal sense and in the sense of race. Consequently, the Whiteness of AI could be considered simply a default. It does not appear as a feature, but is transparent, like the air we breathe: the "unmarked marker", as Ruth Frankenberg calls it  (Frankenberg 1997b, 1) . The majority of White viewers are unlikely to see humanlike machines as racialised at all, but simply as conforming to their idea of what "human-like" means. For non-White people, on the other hand, Whiteness is never invisible in this manner, as bell hooks reminds us  (hooks 1992 1997) . So-called colour-blindness, an attitude of not seeing race, and of presuming that people in contemporary society are no longer disadvantaged on the basis of race, is itself a narrative that perpetuates White hegemony: "communities of color frequently see and name whiteness clearly and critically, in periods when white folks have asserted their own 'color blindness'"  (Frankenberg 1997b, 4) . Noble argues that "central to these 'colorblind' ideologies is a focus on the inappropriateness of 'seeing race'"-a view that she argues is dominant among Silicon Valley technologists, who "revel in their embrace of colorblindness as if it is an asset and not a proven liability"  (Noble 2018, 168) . Such colour-blindness is a liability because it obscures the normalisation of Whiteness and marginalisation of other racialised groups-and the real world effects this has, such as facial recognition technologies not distinguishing Black or East Asian faces  (Buolamwini and Gebru 2018) . Given the normalisation of Whiteness, for some designers, to make a human-like machine will unthinkingly mean to make a White machine. As Dyer puts it: "white people create the dominant images of the world and don't quite see that they thus create the dominant images of the world in their own image"  (Dyer 1997, 9) . But this alone is not a satisfactory explanation of the Whiteness of AI, as not all entities-more specifically, not all intelligent, humanoid entities-imagined by predominantly White industries are portrayed as White. For example, Western science fiction has a long tradition of White authors racialising extraterrestrials as non-White. In the late nineteenth century, for instance, the real-world fear of the "Yellow Peril" was metaphorically addressed in science fiction by racialising extraterrestrial invaders as East Asian. The Flash Gordon franchise gained its lead villain in a 1934 comic, which introduced the tyrannical emperor of the planet Mongo-the Orientalised alien Ming the Merciless. Such is the villain in Flash Gordon -a trident bearded, slanty eyed, shiny doomed [sic], pointy nailed, arching eyebrowed, exotically garbed Oriental named Ming, who personifies unadulterated evil. A heavy like Ming is not contrived in a comic strip writer's imagination during a coffee break, but rather is the product of perhaps the richest and longest tradition of all of Hollywood ethnic stereotypes.  (Barshay 1974, 24-26)  Dyer points out that Blade Runner similarly deliberately uses East Asian characters in order to offset the whiteness of its protagonists, including the White androids: "the yellow human background emphasises the chief protagonists' whiteness. The whitest of hue are the replicants"  (Dyer 1997, 214) . Racial stereotyping of aliens is not a phenomenon limited to past centuries. The Star Wars prequel trilogy  (Lucas 1999; 2002; 2005)  has been criticised for the "transparent racism" in its depiction of the alien Jar Jar Binks as a West Indian caricature  (Lavender 2011, 193)  reminiscent of blackface minstrelsy  (Williams 1999) , and of the slave trader Watto, an antisemitic Jewish caricature with a large nose, skullcap, Yiddish accent, and obsession with money  (Freedman 2019) . This racialisation of aliens in SF suggests that the racialisation of artificial intelligence is a choice. The White racial frame as perpetuated by the White creators of these works portrays dangerous invaders from another planet as East Asian and bumbling alien petty-criminals as Afro-Caribbean. Therefore, the fact that it portrays AI as overwhelmingly White requires further explanation. In the following sections, we offer two.
 AI and the Attributes of Whiteness While Whiteness functions in part through its invisibility in mainstream discourse, this does not mean it has no distinguishable features of its own. Indeed, the White racial frame has a long history of ascribing certain attributes to Whites and disputing them in others: these are the very claims that have been used to justify colonialism, segregation, and other modes of oppression. We argue that AI is predominantly racialised as White because it is deemed to possess attributes that this frame imputes to White people. We examine these attributes under three key headings: intelligence, professionalism, and power. First, the primary attribute being projected onto these machines is, as the term "AI" suggests, intelligence. Throughout the history of Western thought, but in particular since the seventeenth century in Europe and the territories it colonised, intelligence has been associated with some humans more than others  (Carson 2006) . The idea that some races were more mentally able than others was crucial to the legitimation of the advancing colonial project. Those deemed less intelligent-in the words of Rudyard Kipling, "Half-devil and half-child"-were judged unqualified to rule themselves and their lands. It was therefore legitimate-even a duty, "the white man's burden" as Kipling put it-to destroy their cultures and take their territories  (Kipling 1899) . Through the nineteenth century, strenuous efforts were made to empirically demonstrate and measure this intellectual difference, culminating in the development of the IQ test  (Gould 1981) . Although explicit associations between racial groups and intelligence declined after the Second World War, (a) they continue to be made in right-wing circles  (Saini 2019 ) and (b) implicit or unconscious associations between race and intelligence persist widely (see, for example, van den  Bergh et al. 2010; Okeke et al. 2009) . Given the White racial frame has for centuries promoted the association of intelligence with the White, European race, it is to be expected that when this culture is asked to imagine an intelligent machine, it imagines a White machine. A crucial aspect of the idea of intelligence is generality. Intelligence is often defined as a "general mental capability"  (Gottfredson 1997) , and in AI, the concept of "artificial general intelligence"-a system with the kind of flexible mental capabilities humans have-is often considered to be the original and primary goal of the field  (Crevier 1993) . But in the White racial frame, not all humans are considered to have this attribute to the same degree. As Weheliye puts it, using Sylvia Wynter's idea of "the Man"-the Enlightenment, Western, White male subject, "In the context of the secular human, black subjects, along with indigenous populations, the colonised, the insane, the poor, the disabled, and so on as limit cases by which Man can demarcate himself as the universal human"  (Weheliye 2014, 24) . According to the White racial frame, it is the rational, scientific thought of the White Westerner that lays claim to universal validity-or, we might say, true generality. Other races, by contrast, are framed as particular and subjective, constrained by the limits of their non-ideal bodies and cultures to think thoughts that are partial and parochial. To imagine a truly intelligent machine, one with general intelligence is therefore to imagine a White machine. Second, much of the current discourse around AI focuses on how it is, or will soon be, capable of professional work. This is frequently claimed to be what makes the present wave of automation different from previous waves, in which machines became capable of supplanting manual and semi-skilled labour  (Ford 2015) . Professional work-law, medicine, business, and so forth-is at the upper end of pay and status scales. White Europeans and North Americans have historically not considered all humans equally fit for such roles and have kept them closed to people who lacked the requisite connections, wealth, or other in-group identifiers. Universities, the gateways to the professions, have long histories of excluding people of colour from their ranks  (Burrow 2008, 107) . The historic exclusion of anyone other than White men shapes to this day what mainstream White culture imagines when imagining someone fulfilling such roles. Safiya Noble shows that it took years of criticism before search engines adjusted their algorithms so that searching for "engineer" or "doctor" stopped exclusively returning images of White men  (Noble 2018) . But the underlying bias, on which the algorithms fed, remains. To imagine a machine in a white-collar job is therefore to imagine a White machine. Third, hierarchies of intelligence and of professional status are of course also hierarchies of power. Consequently, power relations are implicit in the previous two categories. However, it is worth also considering power separately, because power struggles between AI and humans are such a common narrative trope. Alongside the narrative that robots will make humans redundant, an equally well-known narrative is that they will rise up and conquer us altogether  (Cave and Dihal 2019) . These are both narratives about machines becoming superior to humans: stories in which they become better at every task, leaving humans with nothing to do, from E.M. Forster's 1909 short story 'The Machine Stops' to the Oscar-winning film WALL-E, or in which they outwit and subjugate those who built them, as in the Terminator film franchise or the film Ex Machina  (Forster 1909; Stanton 2008; Cameron 1984; Garland 2015) . When White people imagine being overtaken by superior beings, those beings do not resemble those races they have framed as inferior. It is unimaginable to a White audience that they will be surpassed by machines that are Black. Rather, it is by superlatives of themselves: hyper-masculine White men like Arnold Schwarzenegger as the Terminator, or hyperfeminine White women like Alicia Vikander as Ava in Ex Machina. This is why even narratives of an AI uprising that are clearly modelled on stories of slave rebellions depict the rebelling AIs as White-for example, in Blade Runner  (Dihal 2020) . The implication of this racialisation is that these machines might genuinely be superior, or are at least worthy adversaries. The use of White bodybuilders such as Arnold Schwarzenegger to play the evil robots suggests this. As Dyer points out, Schwarzenegger's physique suggests "the body made possible by [...] natural mental superiority. The point after all is it built, a product of the application of thought and planning, an achievement"  (Dyer 1997, 164) . Consequently, for a White technologist or author, to imagine a superior anthropomorphic machine is to imagine a White machine. In summary, popular conceptions of AI suggest these machines have general intelligence, are capable of professional jobs, and/or are poised to surpass and supplant humanity. In the White imagination, such qualities are strongly associated with Whiteness. It is no surprise, therefore, that in mainstream Western media, such machines are portrayed as White.
 White Utopia While we believe the attribution to AI of these qualities, so strongly associated with Whiteness, goes a long way to making sense of the racialisation of anthropomorphic intelligent machines, we also want to propose one further hypothesis: that the Whiteness of the machines allows the White utopian imagination to fully exclude people of colour. One of the most pertinent hopes for artificial intelligence is that it will lead to a life of ease  (Cave and Dihal 2019) . As a tool that can take over "dirty, dull, or dangerous" jobs, it relieves its owners from work they do not want to do, enabling them to pursue leisure. As critical race theorists have repeatedly pointed out, the leisure currently available to the wealthier classes is disproportionately facilitated by the labour of working-class women of colour  (hooks 1992 Rhee 2018) . bell hooks shows that the people performing this labour are actively kept invisible, even when the White master and the coloured servant are physically present in the same space. She cites the memoirs of a White heiress who grew up with Black servants in her house: "Blacks, I realized, were simply invisible to most white people, except as a pair of hands offering a drink on a silver tray"  (hooks 1992 1997, 168) . As this forced pretence of invisibility shows, interactions with non-White servants are undesirable to the White master: such interactions are almost literally considered a "dirty job". Depictions of people of colour as being dirty and unwashed, eating dirty food, living in the dirt, even of being the colour of excrement have contributed to the development of both the fear of pollution in interactions with people of colour, and the association of Whiteness with cleanliness and purity  (Dyer 1997, 75-76) . This association has been exacerbated by a long history of propaganda preceding conquest and genocide that portrays the racial other as evoking disgust: as vectors of disease, such as lice or rats, or as a literal plague  (Glover 1999, chap. 35; Rector 2014, chap. 3) . The utopia of the White racial frame would therefore rather remove people of colour altogether, even in the form of servants. From the inception of the academic study of science fiction onwards, many critics have pointed out that utopias throughout literary history have been construed on exclusionary, colonialist, and eugenicist premises  (Suvin 1979 (Suvin 2016 Jameson 2005, 205; Ginway 2016, 132) . In Astrofuturism, De Witt Douglas Kilgore shows that mid-twentieth-century American visions of space age utopias are "idealisations ... based on a series of exclusions" (Kilgore 2010, 10): rather than depicting a post-racial or colourblind future, the authors of these utopias simply omit people of colour. AI offers the possibility of making such utopias real. By virtue of its generality, it is imagined as able replace and any unwanted labour-social and cognitive as well as physical  (Cave and Dihal 2019) , so obviating the need for people of colour in any role. Consequently, as Jennifer Rhee points out, advertisements for real AI such as household robots "are striking in their whiteness": they are aimed at showing white middle-class families an ideal leisurely lifestyle. In doing so, she argues, "the images reserve the luxury of liberation from domestic labor for white women, while erasing the women of color who perform this labor, both within their own homes and in the homes of others"  (Rhee 2018, 94) . In some cases, the unsulliedness of this utopia can extend further to exclude all women. Just as people of colour can be associated with offensive physicality, so can women in general, particularly with respect to their reproductive organs. The necessity of sexual intercourse, pregnancy, and childbearing for the continuation of a race that prides itself on rationality and the ability to transcend its physicality is an offensive hurdle that has been imagined as transcendable by science for centuries. As Dyer points out, in the ideology of Whiteness, the elevation of mental over physical prowess has simultaneously been the White race's most valuable achievement and a threat to its own continuation  (Dyer 1997, 27) . It has led to the paradox known as the "White Crisis", in which the White race is seen as under threat of being overwhelmed by "inferior" races that are breeding more prolifically. Transhumanism has been envisioned as a solution to this White Crisis  (Ali 2017) . Seen as a form of offspring, artificial intelligence offers a way for the White man to perpetuate his existence in a rationally optimal manner, without the involvement of those he deems inferior.
 Conclusion and Implications Images of AI are not generic representations of human-like machines, but avatars of a particular rank within the hierarchy of the human. These representations of intelligent machines-and our future with them-are refracted through the White racial frame; their Whiteness a proxy for how we perceive their status and potential. This can cause what is sometimes called representational harms  (Blodgett et al. 2020) . We suggest three. First, this racialisation can amplify the very prejudices it reflects. We have argued that intelligent machines are portrayed as White because that is how the mainstream perceives intelligence and related desirable characteristics. But equally, the consistent portrayal of intelligent machines as White itself transmits this association, so sustaining it. As we have argued elsewhere  (Whittlestone et al. 2019) , bias in representations of AI contributes to a vicious cycle of social injustice: the biased representations can influence both aspiring technologists and those in charge of hiring new staff, shaping whom they consider fit for the field  (Cave 2020) . This could contribute to sustaining a racially homogenous workforce, which will continue to produce products, whether real intelligent machines or their representations, that are biased to benefit that group and disadvantage others. Second, the racialisation of these machines places them within an existing hierarchy of the human in a way that could exacerbate real injustice. Portrayals of AI as White situate these machines in a power hierarchy above currently marginalised groups, such as people of colour. These oppressed groups are therefore relegated to an even lower position in the hierarchy: below that of machine. As machines become ever more important in making automated decisions-frequently about marginalised  (Eubanks 2017) -this be consequential. Automation bias-the tendency of people to favour suggestions from automated decision-making systems over those from humans-has already been evidenced  (Goddard et al. 2012) . We might speculate that it will be exacerbated in cases where such systems are racialised White and the humans in question are not. Third, these portrayals could distort our perceptions of the risks and benefits of these machines. For example, they could frame the debate about AI's impact disproportionately around the opportunities and risks posed to White middle-class men  (Cave 2020) . It is already a common narrative that the current wave of automation differs from those of the past in that "impacts from automation have thus far impacted mostly blue-collar employment; the coming wave of innovation threatens to upend white-collar work as well"  (Pew Research Center 2014) . Public interest and policy therefore often focus on white collar professionals, instead of on marginalized groups, which in reality are likely to be worse affected by the impact of AI  (Eubanks 2017; Noble 2018) . In this paper, we have offered three interpretations of the whiteness and Whiteness of representations of AI. All three, and the implications that we posit, need further investigation. This process is part of what can be described as decolonising AI: a process of breaking down the systems of oppression that arose with colonialism and have led to present injustices that AI threatens to perpetuate and exacerbate. Weheliye describes how he "works towards the abolition of Man, and advocates the radical reconstruction and decolonization of what it means to be human"  (Weheliye 2014, 4) . It is in the field of AI that technology is most clearly entwined with notions of "what it means to be human", both in reality and in cultural fantasies. We hope to have taken a step towards this reconstruction, by drawing attention to the Whiteness of these machines and "making it strange". regulation or exceeds the permitted use, you will need obtain permission directly from the copyright holder. To view a copy of this licence, visit Fig. 2 2 Fig. 2 Covers of Collins 2018, Polity, and Elliott 2018, Routledge
 Fig. 3 3 Fig. 3 Tor browser Google image search result for "artificial intelligence", 13 April 2020

			 Following the increasingly common usage of the capitalised form "Black" to denote the ethnicity and "black" the colour, we use "White" to refer to the ethnicity and "white" the colour.While not yet the norm, as can be seen in our quotations of critics who do not employ this distinction, this usage will make our discussion clearer.
			 The Whiteness of AI
			 https://www.internetlivestats.com/google-search-statistics/ accessed 30 December 2019.Fig. 1 Sophia. Hanson Robotics, April 2020The Whiteness of AI
			 Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Abstract: This paper focuses on the fact that AI is predominantly portrayed as white-in colour, ethnicity, or both. We first illustrate the prevalent Whiteness of real and imagined intelligent machines in four categories: humanoid robots, chatbots and virtual assistants, stock images of AI, and portrayals of AI in film and television. We then offer three interpretations of the Whiteness of AI, drawing on critical race theory, particularly the idea of the White racial frame. First, we examine the extent to which this Whiteness might simply reflect the predominantly White milieus from which these artefacts arise. Second, we argue that to imagine machines that are intelligent, professional, or powerful is to imagine White machines because the White racial frame ascribes these attributes predominantly to White people. Third, we argue that AI racialised as White allows for a full erasure of people of colour from the White utopian imaginary. Finally, we examine potential consequences of the racialisation of AI, arguing it could exacerbate bias and misdirect concern.


Robust program equilibrium

Introduction Much has been written about rationalizing non-Nash equilibrium play in strategicform games. Most prominently, game theorists have discussed how cooperation may be achieved in the prisoner's dilemma, where mutual cooperation is not a Nash equilibrium but Pareto-superior to mutual defection. One of the most successful approaches is the repetition of a game, and in particular the iterated prisoner's dilemma  (Axelrod 2006) . Another approach is to introduce commitment mechanisms of some sort. In this paper, we will discuss one particular commitment mechanism:  Tennenholtz's (2004)  program equilibrium formalism (Sect. 2.2). Here, the idea is that in place of strategies, players submit programs which compute strategies and are given access to each other's source code. The programs can then encode credible commitments, such as some version of "if you cooperate, I will cooperate". As desired,  Tennenholtz (2004, Sect. 3, Theorem 1)  shows that mutual cooperation is played in a program equilibrium of the prisoner's dilemma. However, Tennenholtz' equilibrium is very fragile. Essentially, it consists of two copies of a program that cooperates if it faces an exact copy of itself  (cf. McAfee 1984; Howard 1988) . Even small deviations from that program break the equilibrium. Thus, achieving cooperation in this way is only realistic if the players can communicate beforehand and settle on a particular outcome. Another persuasive critique of this trivial equilibrium is that the model of two players submitting programs is only a metaphor, anyway. In real life, the programs may instead be the result of an evolutionary process  (Binmore 1988 pp. 14f.)  and Tennenholtz' equilibrium is a hard to obtain by such a process. Alternatively, if we view our theory as normative rather than descriptive, we may view the programs themselves as the target audience of our recommendations. This also means that these agents will already have some form of source code-e.g., one that derives and considers the program equilibria of the game-and it is out of their realm of power to change that source code to match some common standard. However, they may still decide on some procedure for thinking about this particular problem in such a way that enables cooperation with other rationally pre-programmed agents. Noting the fragility of Tennenholtz' proposed equilibrium, it has been proposed to achieve a more robust program equilibrium by letting the programs reason about each other  (van der Hoek et al. 2013; Barasz et al. 2014; Critch 2016) . For example,  Barasz et al. (2014, Sect.  3) propose a program FairBot-variations of which we will see in this paper-that cooperates if Peano arithmetic can prove that the opponent cooperates against FairBot. FairBot cooperates (via Löb's theorem) more robustly against different versions of itself. These proposals are very elegant and certainly deserve further attention. However, their benefits come at the cost of being computationally expensive. In this paper, I thus derive a class of program equilibria that I will argue to be more practical. In the case of the prisoner's dilemma, I propose a program that cooperates with a small probability and otherwise acts as the opponent acts against itself (see Algorithm 1). Doing what the opponent does-à la FairBot-incentivizes cooperation. Cooperating with a small probability allows us to avoid infinite loops that would arise if we merely predicted and copied our opponent's action (see Algorithm 2). This approach to a robust cooperation program equilibrium in the prisoner's dilemma is described in Sect. 3. We then go on to generalize the construction exemplified in the prisoner's dilemma (see Sect. 4). In particular, we show how strategies for the repeated version of a game can be used to construct good programs for the one-shot version of that game. We show that many of the properties of the underlying strategy of the repeated game carry over to the program for the stage game. We can thus construct "good" programs and program equilibria from "good" strategies and Nash equilibria.
 Preliminaries
 Strategic-form games For reference, we begin by introducing some basic terminology and formalism for strategic-form games. For an introduction, see, e.g.,  Osborne (2004) . For reasons that will become apparent later on, we limit our treatment to two-player games. A two-player strategic game G = (A 1 , A 2 , u 1 , u 2 ) consists of two countable sets of moves A i and for both players i ∈ {1, 2} a bounded utility function u i : A 1 × A 2 → R. A (mixed) strategy for player i is a probability distribution π i over A i . Given a strategy profile (π 1 , π 2 ) the probability of an outcome (a 1 , a 2 ) ∈ A 1 × A 2 is P(a 1 , a 2 | π 1 , π 2 ) := π 1 (a 1 ) • π 2 (a 2 ). (1) The expected value for player i given that strategy profile is E [u i | π 1 , π 2 ] := (a 1 ,a 2 )∈A 1 ×A 2 P(a 1 , a 2 | π 1 , π 2 ) • u i (a 1 , a 2 ). Note that because the utility function is bounded, the sum converges absolutely, such that the order of the action pairs does not affect the sum's value.
 Program equilibrium We now introduce the concept of program equilibrium, first proposed by  Tennenholtz (2004) . The main idea is to replace strategies with computer programs that are given access to each other's source code.  1  The programs then give rise to strategies. For any game G, we first need to define the set of program profiles PROG(G) consisting of pairs of programs. The ith entry of an element of PROG(G) must be a program source code p i that, when interpreted by a function apply, probabilistically map program profiles 2 onto A i . We require that for any program profile ( p 1 , p 2 ) ∈ PROG(G), both programs halt. Otherwise, the profile would not give rise to a well-defined strategy. Whether p i halts depends on the program p −i , it plays against, where (in accordance with convention in game theory) − : {1, 2} → {1, 2} : 1 → 2, 2 → 1 and we write −i instead of −(i). For example, if p i runs apply( p −i , (p i , p −i )), i.e., simulates the opponent, then that is fine as long as p −i does not also run apply( p i , (p i , p −i )), which would yield an infinite loop. To avoid this mutual dependence, we will generally require that PROG(G) = PROG 1 (G) × PROG 2 (G), where PROG i (G) consists of programs for player i. Methods of doing this while maintaining expressive power include hierarchies of players-e.g., higher indexed players are allowed to simulate lower indexed ones but not vice versa-hierarchies of programs-programs can only call their opponents with simpler programs as input-requiring programs to have a "plan B" if termination can otherwise not be guaranteed, or allowing each player to only start strictly less than one simulation in expectation. These methods may also be combined. In this paper, we do not assume any particular definition of PROG(G). However, we assume that they can perform arbitrary computations as long as these computations are guaranteed to halt regardless of the output of the parts of the code that do depend on the opponent program. We also require that PROG(G) is compatible with our constructions. We will show our constructions to be so benign in terms of infinite loops that this is not too strong of an assumption. Given a program profile ( p 1 , p 2 ), we receive a strategy profile (apply( p 1 , (p 1 , p 2 )), apply( p 2 , (p 1 , p 2 ))). For any outcome (a 1 , a 2 ) of G, we define P(a 1 , a 2 | p 1 , p 2 ) := P(a 1 , a 2 | apply( p 1 , (p 1 , p 2 )), apply( p 2 , (p 1 , p 2 ))) (2) and for every player i ∈ {1, 2}, we define E [u i | p 1 , p 2 ] := (a 1 ,a 2 )∈A 1 ×A 2 P(a 1 , a 2 | p 1 , p 2 ) • u i (a 1 , a 2 ). ( 3 ) For player i, we define the (set-valued) best response function as B i ( p −i ) = arg max p i ∈PROG i (G) E u i | p i , p −i . A program profile ( p 1 , p 2 ) is a (weak) program equilibrium of G if for i ∈ {1, 2} it is p i ∈ B i ( p −i ).
 Repeated games Our construction will involve strategies for the repeated version of a two-player game. Thus, for any game G, we define G to be the repetition of G with a probability of ∈ (0, 1] of ending after each round. Both players of G will be informed only of the last move of their opponent. This differs from the more typical assumption that players have access to the entire history of past moves. We will later see why this deviation is necessary. A strategy π i for player i non-deterministically maps opponent moves or the information of the lack thereof onto a move π i : {0} ∪ A −i A i . Thus, for a ∈ A i , b ∈ A −i , π i (b, a) := π i (b)(a) denotes the probability of choosing a given that the opponent played b in the previous round and π i (0, a) := π i (0)(a) denotes the probability of choosing a in the first round. We call a strategy  a) . The probability that the game follows a complete history of moves h = a 0 b 0 a 1 b 1 • • • a n b n and then ends is π i stationary if for all a ∈ A i , π i (b, a) is constant with respect to b ∈ {0} ∪ A −i . If π i is stationary, we write π i (a) := π i (b, P(h | (π 1 , π 2 )) := π 1 (0, a 0 )π 2 (0, b 0 ) (1 − ) n n i=1 π 1 (b i−1 , a i )π 2 (a i−1 , b i ). (4) Note that the moves in the history always come in pairs a i b i which are chosen "simultaneously" in response to b i−1 and a i−1 , respectively. The expected value for player i given the strategy profile (π 1 , π 2 ) is E [u i | π 1 , π 2 ] := h∈(A 1 •A 2 ) + P(h | π 1 , π 2 ) • u i (h), (5) where (A 1 • A 2 ) + is the set of all histories and u i (a 0 b 0 a 1 b 1 • • • a n b n ) := n i=0 u i (a i , b i ). ( 6 ) The lax unordered summation in Eq. 5 is, again, unproblematic because of the absolute convergence of the series, which is a direct consequence of the proof of Lemma 1. Note how the organization of the history into pairs of moves allows us to apply the utility function of the stage game in Eq. 6. For player i, we define the set-valued best response function as B i (π −i ) = arg max π i :{0}∪A −i A i E u i | π i , π −i . Analogously, B c i (π −i ) is the set of responses to π −i that are best among the computable ones, B s i (π −i ) the set of responses to π −i that are best among the stationary ones, and B s,c i (π −i ) the set of responses to π −i that are best among stationary computable strategies. A strategy profile (π 1 , π 2 ) is a (weak) Nash equilibrium of G if for i ∈ {1, 2} it is π i ∈ B i (π −i ). We now prove a few lemmas that we will need later on. First, we have suggestively called the values P(h) probabilities, but we have not shown them to satisfy, say, Kolmogorov's axioms. Additivity is not an issue, because we have only defined the probability for atomic events and non-negativity is obvious from the definition. However, we will also need the fact that the numbers we have called probabilities indeed sum to 1, which requires a few lines to prove. Lemma 1 Let G be a repeated game and π 1 , π 2 be strategies for that game. Then h∈(A 1 •A 2 ) + P(h | π 1 , π 2 ) = 1. Proof h P(h | π 1 , π 2 ) = Eq. 4 a 0 b 0 •••a n b n ∈(A 1 •A 2 ) + π 1 (0, a 0 )π 2 (0, b 0 ) (1 − ) n • n i=1 π 1 (b i−1 , a i )π 2 (a i−1 , b i ) = absolute convergence ∞ n=0 (1 − ) n a 0 b 0 •••a n b n ∈(A 1 A 2 ) n+1 π 1 (0, a 0 )π 2 (0, b 0 ) n i=1 π 1 (b i−1 , a i )π 2 (a i−1 , b i ) = ∞ n=0 (1 − ) n a 0 b 0 ∈A 1 A 2 π 1 (0, a 0 )π 2 (0, b 0 ) a 1 b 1 ∈A 1 A 2 π 1 (b 0 , a 1 )π 2 (a 0 , b 1 ) . . . • a n b n ∈A 1 A 2 π 1 (b n−1 , a n )π 2 (a n−1 , b n ) = ∞ n=0 (1 − ) n = sum of geo- metric series 1 For seeing why the second-to-last equation is true, notice that the inner-most sum is 1. Thus, the next sum is 1 as well, and so on. Since the ordering in the right-hand side of the first line is lax, and because only the second line is known to converge absolutely, the re-ordering is best understood from right to left. The last step uses the well-known formula ∞ k=0 x k = 1/(1 − x) for the geometric series. For any game G , k ∈ N + , a ∈ A 1 , b ∈ A 2 and strategies π 1 and π 2 for G , we define P k,G (a, b | π 1 , π 2 ) := (1 − ) k a 0 b 0 •••a k−1 b k−1 ∈(A 1 A 2 ) k π 1 (b k−1 , a)π 2 (a k−1 , b)π 1 (0, a 0 )π 2 (0, b 0 ) • k−1 j=1 π 1 (b j−1 , a j )π 2 (a j−1 , b j ). ( 7 ) For k = 0, we define P 0,G (a, b | π 1 , π 2 ) := π 1 (0, a) • π 2 (0, b). Intuitively speaking, P k,G (a, b | π 1 , π 2 ) is the probability of reaching at least round k and that (a, b) is played in that round. With this ab∈A 1 A 2 P k,G (a, b | π 1 , π 2 )u i (a, b) should be the expected utility from the kth round (where not getting to the kth round counts as 0). This suggests a new way of calculating expected utilities on a more round-by-round basis. Lemma 2 Let G be a game, and let π 1 , π 2 be strategies for that game. Then E [u i | π 1 , π 2 ] = ∞ k=0 ab∈A 1 A 2 P k,G (a, b | π 1 , π 2 )u i (a, b). Proof E G [u i | π 1 , π 2 ] = h∈(A 1 A 2 ) + P(h | π 1 , π 2 )u i (h) = Eqs. 4, 6 a 0 b 0 •••a n b n ∈(A 1 A 2 ) + π 1 (0, a 0 )π 2 (0, b 0 ) (1 − ) n ⎛ ⎝ n j=1 π 1 (b j−1 , a j )π 2 (a j−1 , b j ) ⎞ ⎠ • n k=0 u i (a k , b k ) = ∞ k=0 a 0 b 0 •••a n b n ∈(A 1 A 2 ) ≥k+1 π 1 (0, a 0 )π 2 (0, b 0 ) (1 − ) n • ⎛ ⎝ n j=1 π 1 (b j−1 , a j )π 2 (a j−1 , b j ) ⎞ ⎠ u i (a k , b k ) = ∞ k=0 a 0 b 0 •••a k b k ∈(A 1 A 2 ) k+1 a k+1 b k+1 •••a n b n ∈(A 1 A 2 ) * π 1 (0, a 0 )π 2 (0, b 0 ) (1 − ) n u i (a k , b k ) • ⎛ ⎝ k j=1 π 1 (b j−1 , a j )π 2 (a j−1 , b j ) ⎞ ⎠ ⎛ ⎝ n j=k+1 π 1 (b j−1 , a j )π 2 (a j−1 , b j ) ⎞ ⎠ = ∞ k=0 a 0 b 0 •••a k b k ∈(A 1 A 2 ) k+1 π 1 (0, a 0 )π 2 (0, b 0 ) (1 − ) k u i (a k , b k ) • ⎛ ⎝ k j=1 π 1 (b j−1 , a j )π 2 (a j−1 , b j ) ⎞ ⎠ • a k+1 b k+1 •••a n b n ∈(A 1 A 2 ) * (1 − ) n−k ⎛ ⎝ n j=k+1 π 1 (b j−1 , a j )π 2 (a j−1 , b j ) ⎞ ⎠ = lemma 1 ∞ k=0 a 0 b 0 •••a k b k ∈(A 1 A 2 ) k+1 π 1 (0, a 0 )π 2 (0, b 0 ) (1 − ) k u i (a k , b k ) • ⎛ ⎝ k j=1 π 1 (b j−1 , a j )π 2 (a j−1 , b j ) ⎞ ⎠ = Eq. 7 ∞ k=0 a k b k ∈A 1 A 2 P k (a k , b k | π 1 , π 2 )u i (a k , b k ) To find the probability of player i choosing a in round k, we usually have to calculate the probabilities of all actions in all previous rounds. After all, player i reacts to player −i's previous move, who in turn reacts to player i's move in round k − 2, and so on. This is what makes Eq. 7 so long. However, imagine that player −i uses a stationary strategy. This, of course, means that player −i's probability distribution over moves in round k (assuming the game indeed reaches round k) can be computed directly as π −i (b). Player i's distribution over moves in round k is almost as simple to calculate, because it only depends on player −i's distribution over moves in round k − 1, which can also be calculated directly. We hence get the following lemma. Lemma 3 Let G be a game, let π i be a any strategy for G and let π −i be a stationary strategy for G . Then, for all k ∈ N + , it is P k,G (a, b | π i , π −i ) = (1 − ) k b ∈A −i π −i (b )π −i (b)π i (b , a). Proof We conduct our proof by induction over k. For k = 1, it is  P 1 (a, b | π i , π −i ) = Eq. 7 (1 − ) a 0 b 0 π i (b 0 , a)π −i (a 0 , b)π i (0, a 0 )π −i (0, b 0 ) = (1 − ) b 0 π i (b 0 , a)π −i (b)π −i (b 0 ) a 0 π i (0, a 0 ) = (1 − ) b 0 π i (b 0 , a)π −i (b)π −i (b 0 ). If the lemma is true for k, it is also true for k + 1: P k+1 (a, b | π i , π −i ) = (1 − ) k+1 a 0 b 0 •••a k b k ∈(A i A −i ) k+1 π i (b k , a)π −i (a k , b)π i (0, a 0 )π −i (0, b 0 ) • k j=1 π i (b j−1 , a j )π −i (a j−1 , b j ) = Eq. 7 (1 − ) a k b k P k (a k b k | π i , π −i )π i (b k , a)π −i (b) = I .H . (1 − ) k+1 a k b k b π −i (b )π −i (b k )π −i (b)π i (b , a k )π i (b k , a) = (1 − ) k+1 b k π −i (b k )π −i (b)π i (b k , a) b π −i (b ) a k π i (b , a k ) = (1 − ) k+1 b k π −i (b k )π −i (b)π i (b k , a).
 Robust program equilibrium in the prisoner's dilemma Discussions of the program equilibrium have traditionally used the well-known prisoner's dilemma (or trivial variations thereof) as an example to show how the program equilibrium rationalizes cooperation where the Nash equilibrium fails (e.g.,  Tennenholtz 2004, Sect. 3; McAfee 1984; Howard 1988; Barasz et al. 2014) . The present paper is no exception. In this section, we will present our main idea using the example of the prisoner's dilemma; the next section gives the more general construction and proofs of properties of that construction. For reference, the payoff matrix of the prisoner's dilemma is given in Table  1 . I propose to use the following decision rule: with a probability of ∈ (0, 1], cooperate. Otherwise, act as your opponent plays against you. I will call this strat- Algorithm 1: The GroundedFairBot for player i. The program makes use of a function sample which samples uniformly from the given interval or probability distribution. It is assumed that is computable. egy GroundedFairBot. A description of the algorithm in pseudo-code is given in Algorithm 1.  3  The proposed program combines two main ideas. First, it is a version of FairBot  (Barasz et al. 2014) . That is, it chooses the action that its opponent would play against itself. As player −i would like player i to cooperate, GroundedFairBot thus incentivizes cooperation, as long as < 1/2. In this, it resembles the tit for tat strategy in the iterated prisoner's dilemma (IPD) (famously discussed by  Axelrod 2006) , which takes an empirical approach to behaving as the opponent behaves against itself. Here, the probability of the game ending must be sufficiently small (again, less than 1/2 for the given payoffs) in each round for the threat of punishment and the allure of reward to be persuasive reasons to cooperate. The second main idea behind GroundedFairBot is that it cooperates with some small probability . First and foremost, this avoids running into the infinite loop that a naive implementation of FairBot-see Algorithm 2-runs into when playing against opponents who, in turn, try to simulate FairBot. Note, again, the resemblance with the tit for tat strategy in the iterated prisoner's dilemma, which cooperates when no information about the opponent's strategy is available. Data: program profile ( p 1 , p 2 ) Result: action a i ∈ {C, D} 1 return sample(apply( p −i , (p 1 , p 2 ))) Algorithm 2: The NaiveFairBot for player i To better understand how GroundedFairBot works, consider its behavior against a few different opponents. When GroundedFairBot faces NaiveFairBot, then both cooperate. For illustration, a dynamic call graph of their interaction is given in Fig.  1 . It is left as an exercise for the reader to analyze GroundedFairBot's behavior against other programs, such as another instance of GroundedFairBot or a variation of GroundedFairBot that defects rather than cooperates with probability . When playing against strategies that are also based on simulating their opponent, we could think of GroundedFairBot as playing a "mental IPD". If the opponent program decides whether to cooperate, it has to consider that it might currently only be simu- lated. Thus, it will choose an action with an eye toward gauging a favorable reaction from GroundedFairBot one recursion level up. Cooperation in the first "round" is an attempt to steer the mental IPD into a favorable direction, at the cost of cooperating if sample (0, 1) < already occurs in the first round. In addition to proving theoretical results (as done below), it would be useful to test GroundedFairBot "in practice", i.e., against other proposed programs for the prisoner's dilemma with access to one another's source code. I only found one informal tournament for this version of the prisoner's dilemma. It was conducted in 2013 by Alex Mennen on the online forum and community blog LessWrong.  4  In the original set of submissions, GroundedFairBot would have scored 6th out of 21. The reason why it is not a serious contender for first place is that it does not take advantage of the many exploitable submissions (such as bots that decide without looking at their opponent's source code). Once one removes the bottom 9 programs, GroundedFairBot scores second place. If one continues this process of eliminating unsuccessful programs for another two rounds, GroundedFairBot ends up among the four survivors that cooperate with each other. 5
 From iterated game strategies to robust program equilibria We now generalize the construction from the previous section. Given any computable strategy π i for a sequential game G , I propose the following program: with a small probability sample from π i (0). Otherwise, act how π i would respond (in the sequential game G ) to the action that the opponent takes against this program. I will call this program Groundedπ i Bot. A description of the program in pseudo-code is given in Algorithm 3. As a special case, GroundedFairBot arises from Groundedπ i Bot by letting π i be tit for tat. Data: program profile ( p 1 , p 2 ) Result: action a i ∈ A i 1 if sample (0, 1) < then 2 return sample(π i (0)) 3 end 4 return sample(π i (sample(apply( p −i , (p 1 , p 2 ))))) Algorithm 3: The Groundedπ i Bot for player i. The program makes use of a function sample which samples uniformly from a given interval or a given probability distribution. It is assumed that π i and are computable. Again, our proposed program combines two main ideas. First, Groundedπ i Bot responds to how the opponent plays against Groundedπ i Bot. In this, it resembles the behavior of π i in G . As we will see, this leads Groundedπ i Bot to inherit many of π i 's properties. In particular, if (like tit for tat) π i uses some mechanism to incentivize its opponent to converge on a desired action, then Groundedπ i Bot incentivizes that action in a similar way. Second, it-again-terminates immediately with some small probability to avoid the infinite loops that Naiveπ i Bot-see Algorithm 4-runs into. Playing π i (0) in particular is partly motivated by the "mental G " that Groundedπ i Bot plays against some opponents (such as Groundedπ −i Bots or Naiveπ −i Bots). The other motivation is to make the relationship between Groundedπ i Bot and π i cleaner. In terms of the strategies that are optimal against Groundedπ i Bot, the choice of that constant action cannot matter much if is small. Consider, again, the analogy with tit for tat. Even if tit for tat started with defection, one should still attempt to cooperate with it. In practice, however, it has turned out that the "nice" version of tit for tat (and nice strategies in general) are more successful  (Axelrod 2006, ch. 2) . The transparency in the program equilibrium may render such "signals of cooperativeness" less important-e.g., against programs like  Barasz et al.'s (2014, Sect. 3)  FairBot. Nevertheless, it seems plausible that-if only because of mental G -related considerations-in transparent games the "initial" actions matter as well. We now ground these intuitions formally. First, we discuss Groundedπ i Bot's halting behavior. We then show that, in some sense, Groundedπ i Bot behaves in G like π i does in G .
 Halting behavior For a program to be a viable option in the "transparent" version of G, it should halt against a wide variety of opponents. Otherwise, it may be excluded from PROG(G) in our formalism. Besides, it should be efficient enough to be practically useful. As with GroundedFairBot, the main reason why Groundedπ i Bot is benign in terms of the risk of infinite loops is that it generates strictly less than one new function call in expectation and never starts more than one. While we have no formal machinery for analyzing the "loop risk" of a program, it is easy to show the following theorem. Theorem 4 Let π i be a computable strategy for a game G . Furthermore, let p −i be any program (not necessarily in PROG i (G)) that calls apply( p i , (p i , p −i )) at most once and halts with probability 1 if apply halts with probability 1. Then Groundedπ i Bot and p −i halt against each other with probability 1 and the expected number of steps required for executing Groundedπ i Bot is at most T π i + (T π i + T p −i ) 1 − , ( 8 ) where T π i is the maximum number of steps to sample from π i , and T p −i is the maximum number of steps needed to sample from apply( p −i , ( Groundedπ i Bot, p −i )) excluding the steps needed to execute apply( Groundedπ i Bot, ( Groundedπ i Bot, p −i )). Proof It suffices to discuss the cases in which p −i calls p i once with certainty, because if any of our claims were refuted by some program p −i , they would also be refuted by a version of that program that calls p i once with certainty. If p −i calls p i once with certainty, then the dynamic call graphs of both Groundedπ i Bot and p −i look similar to the one drawn in 1. In particular, it only contains one infinite path and that path has a probability of at most lim i→∞ (1 − ) i = 0. For the time complexity, we can consider the dynamic call graph as well. The policy π i has to be executed at least once (with probability with the input 0 and with probability 1− against an action sampled from apply( p −i , ( Groundedπ i Bot, p −i )). With a probability of (1 − ), we also have to execute the non-simulation part of p −i and, for a second time, π i . And so forth. The expected number of steps to execute Groundedπ i Bot is thus T π i + ∞ j=1 (1 − ) j (T π i + T p −i ), which can be shown to be equal to the term in 8 by using the well-known formula ∞ k=0 x k = 1/(1 − x) for the geometric series. Note that this argument would work if there were more than two players or if the strategy for the iterated game were to depend on more than just the last opponent move, because in these cases, the natural extension of Groundedπ i Bot would have to make multiple calls to other programs. Indeed, this is one of the reasons why the present paper only discusses two-player games and iterated games with such shortterm memory. Whether a similar result can nonetheless be obtained for more than 2 players and strategies that depend on the entire past history is left to future research. As special cases, for any strategy π −i , Groundedπ i Bot terminates against Groundedπ −i Bot and Naiveπ −i Bot (and these programs in turn terminate against Groundedπ i Bot). The latter is especially remarkable. Our Groundedπ i Bot terminates and leads the opponent to terminate even if the opponent is so careless that it would not even terminate against a version of itself or, in our formalism, if PROG −i (G) gives the opponent more leeway to work with simulations.
 Relationship to the underlying iterated game strategy Theorem 5 Let G be a game, π i be a strategy for player i in G , p i = Groundedπ i Bot and p −i ∈ PROG −i (G) be any opponent program. We define π −i = apply( p −i , (p i , p −i )), which makes π −i a strategy for player −i in G . Then E G u i | p i , p −i = E G u i | π i , π −i . Proof We separately transform the two expected values in the equation that is to be proven and then notice that they only differ by a factor : E G u i | π i , π −i = lemma 2 ∞ k=0 ab∈A i A −i P k,G (a, b | π i , π −i )u i (a, b) = lemma 3 ab∈A i A −i π i (0, a)π −i (b)u i (a, b) + ∞ k=1 ab∈A i A −i (1 − ) k b ∈A −i π −i (b )π i (b , a)π −i (b)u i (a, b) = absolute convergence ab∈A i A −i π i (0, a)π −i (b)u i (a, b) + ab∈A i A −i b ∈A −i π −i (b )π i (b , a)π −i (b)u i (a, b) ∞ k=1 (1 − ) k = sum of geo- metric series ab∈A i A −i π i (0, a)π −i (b)u i (a, b) + 1 − ab∈A i A −i b ∈A −i π −i (b )π i (b , a)π −i (b)u i (a, b). The second-to-last step uses convergence to reorder the sum signs. The last step uses the well-known formula ∞ k=0 x k = 1/(1 − x) for the geometric series. Onto the other expected value: E G u i | p i , p −i = Eqs. 3, 2, 1 ab∈A i A −i apply( p i , (p i , p −i ), a)apply( p −i , (p i , p −i ), b)u i (a, b) = def.s p i , π −i ab∈A i A −i ⎛ ⎝ π i (0, a)+(1 − ) b ∈A −i π −i (b )π i (b , a) ⎞ ⎠ π −i (b)u i (a, b) = ab∈A i A −i π i (0, a)π −i (b)u i (a, b) +(1 − ) ab∈A i A −i b ∈A −i π −i (b )π i (b , a)π −i (b)u i (a, b). Here, apply( p i , (p i , p −i ), a) := apply( p i , (p i , p −i ))(a). The hypothesis follows immediately. Note that the program side of the proof does not involve any "mental G ". Using Theorem 5, we can easily prove a number of property transfers from π i to Groundedπ i Bot. Corollary 6 Let G be a game. Let π i be a computable strategy for player i in G and let p i = Groundedπ i Bot.
 If p −i ∈ B −i ( p i ), then apply( p −i , (p i , p −i )) ∈ B s,c −i (π i ). 2. If π −i ∈ B s,c i (π i ) and apply( p −i , (p i , p −i )) = π −i , then p −i ∈ B −i ( p i ). Proof Both 1. and 2. follow directly from Theorem 5. Intuitively speaking, Corollary 6 shows that π i and Groundedπ i Bot provoke the same best responses. Note that best responses in the program game only correspond to best stationary computable best responses in the repeated game. The computability requirement is due to the fact that programs cannot imitate incomputable best responses. The corresponding strategies for the repeated game further have to be stationary because Groundedπ i Bot only incentivizes opponent behavior for a single situation, namely the situation of playing against Groundedπ i Bot. As a special case of Corollary 6, if < 1/2, the best response to GroundedFairBot is a program that cooperates against GroundedFairBot because in a IPD with a probability of ending of less than 1/2 a program that cooperates is the best (stationary computable) response to tit for tat.
 Exploitability Besides forming an equilibrium against many opponents (including itself) tivizing cooperation, another important reason for tit for tat's success is that it is "not very exploitable"  (Axelrod 2006) . That is, when playing against tit for tat, it is impossible to receive a much higher reward than tit for tat itself. We now show that (in)exploitability transfers from strategies π i to Groundedπ i Bots. We call a game G = (A 1 , A 2 , u 1 , u 2 ) symmetric if A 1 = A 2 and u 1 (a, b) = u 2 (b, a) for all a ∈ A 1 and b ∈ A 2 . If G is symmetric, we call a strategy π i for G N -exploitable in G for an N ∈ R ≥0 if there exists a π −i , such that E u −i | π −i , π i > E u i | π −i , π i + N . We call π i N -inexploitable if it is not N -exploitable. Analogously, in a symmetric game G we call a program p i N -exploitable for an N ∈ R ≥0 if there exists a p −i , such that E u −i | p −i , p i > E u i | p −i , p i + N . We call p i N -inexploitable if it is not N -exploitable. Corollary 8 Let G be a game and π i be an N -inexploitable strategy for G . Then Groundedπ i Bot is N -inexploitable. Proof Follows directly from Theorem 5. Notice that if -like tit for tat in the IPD -π i is N -inexploitable in G for all , then we can make Groundedπ i Bot arbitrarily close to 0-inexploitable by decreasing .
 Conclusion In this paper, we gave the following recipe for constructing a program equilibrium for a given two-player game: 1. Construct the game's corresponding repeated game. In particular, we consider repeated games in which each player can only react to the opponent's move in the previous round (rather than the entire history of previous moves by both players) and the game ends with some small probability after each round. 2. Construct a Nash equilibrium for that iterated game. The result is a program equilibrium which we have argued is more robust than the equilibria described by  Tennenholtz (2004) . More generally, we have shown that translating an individual's strategy for the repeated game into a program for the stage game in the way described in 3 retains many of the properties the strategy for the repeated game. Thus, it seems that "good" programs to submit may be derived from "good" strategies for the repeated game. Fig. 1 1 Fig. 1 Dynamic call diagram describing how p 1 = GroundedFairBot chooses when playing against p 2 = NaiveFairBot.
 3. Convert each of the strategies into a computer program that works as follows (see Algorithm 3): with probability do what the strategy does in the first round. With probability 1 − , apply the opponent program to this program; then do what the underlying strategy would reply to the opponent program's output.
 Table 1 1 Payoff matrix for the prisoner's dilemma Player 1 Player 2 Cooperate Defect Cooperate 3, 3 1 , 4 Defect 4, 1 2 , 2
			 The equilibrium in its rudimentary form had already been proposed by McAfee (1984)   and Howard (1988) . At least the idea of viewing players as programs with access to each other's source code has also been discussed by, e.g.,Binmore (1987, Sect. 5; and Anderlini (1990) .2  For keeping our notation simple, we will assume that our programs receive their own source code as input in addition to their opponent's. If PROG i (G) is sufficiently powerful, then by Kleene's second recursion theorem, programs could also refer to their own source code without receiving it as an input(Cutland 1980,  ch. 11).
			 This program was proposed by Abram Demski in a conversation discussing similar (but worse) ideas of mine. It has also been proposed by Jessica Taylor at https://agentfoundations.org/item?id=524, though in a slightly different context.
			 The tournament was announced at https://www.lesserwrong.com/posts/BY8kvyuLzMZJkwTHL/ prisoner-s-dilemma-with-visible-source-code-tournament and the results at https://www.lesserwrong. com/posts/QP7Ne4KXKytj4Krkx/prisoner-s-dilemma-tournament-results-0. 5 For a more detailed analysis and report on my methodology, see https://casparoesterheld.files.wordpress. com/2018/02/transparentpdwriteup.pdf.

Abstract: One approach to achieving cooperation in the one-shot prisoner's dilemma is Tennenholtz's (Games Econ Behav 49(2): [363][364][365][366][367][368][369][370][371][372][373] 2004)  program equilibrium, in which the players of a game submit programs instead of strategies. These programs are then allowed to read each other's source code to decide which action to take. As shown by Tennenholtz, cooperation is played in an equilibrium of this alternative game. In particular, he proposes that the two players submit the same version of the following program: cooperate if the opponent is an exact copy of this program and defect otherwise. Neither of the two players can benefit from submitting a different program. Unfortunately, this equilibrium is fragile and unlikely to be realized in practice. We thus propose a new, simple program to achieve more robust cooperative program equilibria: cooperate with some small probability and otherwise act as the opponent acts against this program. I argue that this program is similar to the tit for tat strategy for the iterated prisoner's dilemma. Both "start" by cooperating and copy their opponent's behavior from "the last round". We then generalize this approach of turning strategies for the repeated version of a game into programs for the one-shot version of a game to other two-player games. We prove that the resulting programs inherit properties of the underlying strategy. This enables them to robustly and effectively elicit the same responses as the underlying strategy for the repeated game.