# The AI revolution and international politics (Allan Dafoe) - EA Forum

Artificial intelligence (AI) is rapidly improving. Superhuman AI in strategically relevant domains is likely to arrive in the next several decades; some experts think two. This will transform international politics, could be profoundly destabilizing, and could pose existential risks. Urgent research is needed on AI grand strategy.

This requires a careful examination of humanity’s highest interests in the era of [transformative AI](https://www.openphilanthropy.org/blog/some-background-our-views-regarding-advanced-artificial-intelligence#Sec1), of the international dynamics likely to arise from AI, and of the most promising strategies for securing a good future. Much work will be required to design and enact effective global AI policy.

Below is a transcript of Allan Dafoe's talk on this topic from EA Global: Boston 2017. We've lightly edited the transcript for clarity.

## The Talk

**Nathan Labenz:** It is my honor to introduce Professor Allan Dafoe. He is an Assistant Professor of Political Science at Yale University and a Research Associate at the Future of Humanity Institute at Oxford, where he is involved in building the AI Politics and Policy Group. His research seeks to understand the causes of world peace and stability. Specifically, he has examined the causes of the liberal peace and the role of reputation and honor as motives for war. Along the way, he has developed methodological tools and approaches to enable more transparent, credible, causal inference. More recently, he has focused on artificial intelligence grand strategy, which he believes poses existential challenges and also opportunities, and which requires us to clearly perceive the emerging strategic landscape in order to help humanity navigate safely through it. Please welcome Professor Allan Dafoe to discuss his work on AI and international politics.

**Allan Dafoe:** Thank you, Nathan.

I will start by talking about war, and then we'll get to AI, because I think there are some lessons for effective altruism.

War is generally understood to be a bad thing. It kills people. It maims them. It destroys communities and ecosystems, and is often harmful to economies. We’ll add an asterisk to that because in the long run, war has had dynamic benefits, but in the current world, war is likely a negative that we would want to avoid.

So if we were going to start a research group to study the causes of war for the purposes of reducing it, we might ask ourselves, “What kinds of war should we study?” There are many kinds of wars that have different causes. Some are worse than others.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/3704d0af382f2c6f6a81f5e343d8b235de7194c13eadd941.png)

One classification that we might employ is how many people different kinds of wars kill. There are some wars that kill only 1,000 to 10,000, and some that kill 100,000 to a million. And so the x-axis \[on this slide\] shows the battle deaths in wars, and the y-axis is the fraction of wars of those kinds.

An instinct we might have is to say, “Let's study the wars that are most common — those in the first bin.” These are the wars that are happening today in Syria: civil wars. Availability bias would suggest that those are the wars we should worry about. Some of my colleagues have argued that great-power war — the big world wars — are a thing of the past. They say that the liberal peace, democracy, capitalism, and nuclear weapons have all rendered great-power war obsolete, and that we're not going to have them again. The probability is too low.

But as effective altruists, we know that you can't just round a small number down to zero. You don't want to do that. You want to try to think carefully about the expected value of different kinds of actions. And so it's important that even though the probability of a war killing a million, 10 million, 100 million, or a billion is very small, it's not zero.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/88d31db53c751e30c56002f433b71e481555d3f991354ca0.png)

If we look at the past 70 years of history, World War II stands out as the source of most of the battle deaths that have been experienced. That would suggest that fatalities are a good enough proxy for whatever metric of importance you have, and that we first want to make sure we understand World War II and the kinds of wars that are like it (i.e. that are likely to kill many people).

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/46adadd40e60253045d8682e04506a9575f94933db1214e5.png)

We can zoom out more, and see that World War I comes out of the froth of civil wars. And really, those two wars loom above everything else as what we want to explain, at least if we're prioritizing importance.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/2cbb5a0b533a6a66bae3fa4b71cf03a0487096bd8552ac85.png)

We can see that in this graph, which again has these bins depicting the size of violent quarrels. Relative to homicides, we see that the world wars in the “10 million” bin on the right contain, again, most of the deaths in violent quarrels. That also suggests that it's really important that we understand these big wars.

But of course, the next war need not limit itself to 99 million deaths. There could be a war that kills hundreds of millions or even 6.5 billion. The problem, empirically speaking, is that we don't have those wars and that data set, so we don't know how to estimate, non-parametrically, the expected value in those. We can try to extrapolate from what's very close to a power-law distribution. And no matter how we do it, unless we're extremely conservative, we get a distribution like this, which shows that most of the harm from war comes from the wars that kill a billion people or more:

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/3ad408f0ce60ad9764934ed99fe9672932aaedf48ec33101.png)

Of course, we haven't had those wars. Nevertheless, this follows from other reasoning.

We can go still further. The loss from a war that killed 6.5 billion is not about just those 6.5 billion people who die. It's also about future people. And this is where the idea of existential risk and existential value comes in. We have to ask ourselves, “What is the value of the future? How much worth do we give to future human lives?”

There are many ways to answer that question. And you want to discount for model uncertainty and various things. But one thing that drives concern with existential risk, which I'm concerned with and the Future of Humanity Institute is concerned with, is that there's so much potential in the future. There are so many potential lives that could be lived. Anything that cuts those lives off has tremendous disvalue.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/5df0afb5e3d84f15b7548845bf803a545534af84b4272b07.png)

One estimate that’s quite conservative is this: If there are only a billion people living on Earth, and they continue living in a sustainable way for a billion years (which is doable), as long as we don't muck it up, that yields 10,000 trillion lives. That is a lot. And so anything that reduces the probability of extinction and of losing those 10,000 trillion lives, by even a little bit, has a lot of expected value.

Now, that's a very small number, one ten trillionth (0.0000000000001), and it's hard to know if you're making less of an effect than that. It looks close to zero. The numbers aren't meant to be taken too seriously. They're more thinking heuristics. They illustrate that if you give value to the future, you really want to worry about anything that poses a risk of extinction. And one thing that I and others, such as the Open Philanthropy Project, have identified as a risk to the future is artificial intelligence.

Before telling you about the risk, I'm going to first tell you about what's up with AI these days.

For a long time, artificial intelligence consisted of what we would now call good old-fashioned AI. There was a programmer who wrote if-then statements. They tried to encode some idea of what was a good behavior that was meant to be automated. For example, with chess algorithms, you would have a chessmaster say, “Here are the heuristics I use. Here's the value function.” You put those heuristics in a machine, and the machine runs it more reliably and more quickly than a human. And that's an effective algorithm. But it turns out that good old-fashioned AI just couldn't hack a number of problems — even simple ones that we do in an instant, like recognizing faces, images, and other things.

More recently, what has sort of taken over is what's called “machine learning.” This means what it sounds like it means: machines learning, for themselves, solutions to problems. Another term for this is “deep learning,” which is especially flexible machine learning. You can think of it as a flexible optimization procedure. It's an algorithm that's trying to find a solution and has a lot of parameters. “Neural networks” is another term you've probably heard.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/9bc2e02dcbd7ccd91aecd035ada6d068e67b0489d534d628.png)

This slide shows the breakthrough recently in image classification arising from neural networks and the year-on-year improvements to the point where machines are now better than humans at image classification.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/c7874af0d4a75ba434c4f1e4d8e89d9d26c205afa815b9e1.png)

Another domain is generalized game-playing or arcade game-playing. Here are our target games. Probably not many of us have played these. DeepMind is a leading AI group within Google that learned to play Atari games at a superhuman level with no instructions about the nature of the universe (e.g., “What is time and space?”, “What is a bad guy?”, “Here's Pac-Man — what is a pellet and what is a ghost?”). All the machine is told or given is pixel input. It just sees the screen from a blank-state type of beginning. And then it plays the game again and again and again, getting its score, and it tries to optimize for the score. Gradually, over the span of about a day, it learns to make sense of this pixel input. It derives concepts of the bad guy, the ghosts, and the pellets. It devises strategies and becomes superhuman at a range of games.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/476507ddedcab38aee856c03dc8c9426c2ec63bd7e506a01.png)

Another domain where we see this is Go. The solution to Go is very similar. You take a blank-slate neural network that's sufficiently flexible and expose it to a lot of human games, and it learns to predict the human moves. Then you have the machine play itself again and again, on the order of about 10 million games. And it becomes superhuman.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/964474271c838297f75c82280f0e5e5f5b30189aca07ddd2.png)

Here's Lee Sedol and AlphaGo, which is another product of DeepMind. And Lee Sedol was the first really excellent Go player who publicly played AlphaGo. And here he's saying in February 2016, “I think I won the game by a near-landslide this time.” Well, as I've alluded to, that didn't work out.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/d4c8c30a4d907469baff3f5ec0c68dade29901fef0bec641.png)

Here he is after the first game: “I was very surprised because I didn't think I would lose.”

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/06f9a5d06e11384568202d4866d87f907c207c8217ed6f93.png)

And unfortunately, he lost again: “I'm quite speechless. I'm in shock. I can admit that. The third game is not going to be easy for me.” He lost the third game. He did win the fourth game and he talks about how it was sort of the greatest moment of his life. And it's probably the last and only game played against a level of AlphaGo of that level or better that a human will ever win.

I'm bringing up Lee Sedol and his losses for a reason. I think it serves as an allegory for humanity: We don't want to be caught off-guard when it's our AlphaGo moment. At some point, machine intelligence will be better than we are at strategically relevant tasks. And it would be prudent for us to see that coming — to have at least a few years’ notice, if not more, to think through how we can adapt our international systems, our politics, our notion of the meaning of life, and other areas.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/5c1463d2ae0a9796f9c804527a085b944314b6c0d463f1f2.png)

What's driving this progress? Algorithms, talent, and data, but another big thing driving it is hardware. Computing inputs keep getting better at an exponential rate. This is sort of a generalized Moore's law across a range of inputs. And it's this persistent progress that makes Kurzweil's graph from 2001 seem not totally absurd.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/8340d08346567587be5442e31eafa1bf3da9396555268898.png)

Here we have, on the first y-axis, calculations per second, per thousand dollars, and I added four recent dots from the past 17 years. And you see that basically we're on track. We have exponential improvements. What we don't know is when we get to transformative AI.

Now, Kurzweil has this evocative second y-axis, where you have organisms we recognize: mice, humans, and all humans. It's not obvious what the mapping should be between calculations per second and transformative AI. Intelligence is not a single dimension, so we could get superhuman AI in some domains long before other domains. But what I think is right about this graph is that at some point between 2020 or 2025 and 2080, big things are going to happen with machine intelligence.

In our work, we want to be a bit more systematic about these timelines, and there are various ways to do it. One way is to survey AI researchers. This is the result of a recent survey.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/c3cbdda22da9887bdf8d12ef450f76ebdc91408359111991.png)

Some takeaways are:

1.  There's huge disagreement about when human-level machine intelligence, defined here as machines better than all humans at every task, will come, as you can see by the gray S-curves.
2.  The group that we surveyed gives enough probability mass that by 100 years, there still won't be human-level machine intelligence.
3.  But in the next 10 or 20 years, this group gives a 10-20% chance that we will have reached it already. And if that probability seems right (and upon consideration, I think it does, and not just from using this as evidence), there’s more than sufficient warrant for us to invest a lot of resources into thinking very hard about what human-level AI would mean for humanity.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/b8af218dbc98985f543afd818a90274a072ee5cf7b07c0e0.png)

Here are some tasks. I like to think of them as milestones and canaries. What are some things that machines will eventually achieve that are either noteworthy or strategically relevant (i.e. milestones)? And the canaries are those things that when they happen, signal to us that we’d better be paying attention because things are going to change quickly. I expect most of these tasks on the right-hand column will soon be moving over to the left-hand column, if they're not there already. So this is something else we're working on.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/95de8fd3d4eed18e5e5d86d36460f888ff5bf369bb13628e.png)

More generally, there's a whole host of challenges — near, medium, and long term — that we will be working on as a society, but also at the Future of Humanity Institute. There's a range of near-term issues that I'm not going to talk about. Each of those could occupy a workshop or a conference. I will say that when thinking about long-term issues, we also confront the near-term issues for one reason: because the long-term issues often look like the near-term issues, magnified a hundred-fold, and because a lot of our long-term insights, strategies, and policy interventions’ most appropriate place of action is in the near term.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/9bb96db8553ccebce756846c9bc7d53a3dea39b7b027d822.png)

But what are the long-term opportunities and risks? The opportunities are tremendous. We often just think it's greater wealth, and that maybe Google stock will go up. But it's a lot more: longevity, health, preventative medicine, material bounty that could be used to end poverty ( though it need not, because it will also likely come in a more unequal world), reduced environmental impact. DeepMind was able to reduce energy usage at Google data centers by 40%. AI could basically help with anything of value that is either the product of intelligence or depends on intelligence for its protection. And so if we have superhuman intelligence, then in principle, we can use that to achieve all of our goals.

I'll also emphasize the last point: resilience to other existential risks. We're likely to face those in the next 100 years. And if we solve AI — if we build it well — then that could reduce those other risks by a large margin.

But of course, there are also risks with bringing into the ecosystem a creature that is better than we are at the thing that matters most for our survival and flourishing. I'm not going to go through this topology.

I will illuminate the risk by quoting Max Tegmark and others:

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/003a94eb08240aea330c5281af1f484627f25c2097eb4424.png)

I will also appeal to another authority from my world:

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/abaf1bc735f5bd59ac1ae90686093b9ac58ecc08c3b94b26.png)

I'm a scholar of international relations. Henry Kissinger, as you know, is also very worried about AI. And this is sort of the definition of a grand strategist.

In addition to these quotes, the AI researchers we surveyed agree. We asked them what the long-term prospects are for human-level machine intelligence. And while most of the probability mass was on “extremely good” or on “balanced good,” the median is a 5% probability of AI being extremely bad or leading to human extinction.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/2cd2c30fca6d274c2de7972fa289e67a0f78a2769beaa7a8.png)

It's not often you get an industry that says that their activities give rise to a 5% chance of human extinction. I think it should be our goal to take whatever the real and most frequent number is, and push it as close to zero, getting as much of that probability mass up to the top.

There are two broad ways we can do that. One is to work on what's called AI safety. The computer scientists in the room and mathematicians can help build AI systems that are unlikely to misbehave without our intentions. And the other way is AI strategy, which I'm going to talk about.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/bdceccf65b1f386dc1e3fbd36650b6d5c92100a2b5e0f5b3.png)

So here's Stuart Russell explaining that we're not worried about machines suddenly waking up and deciding they want to build their own machine utopia and get rid of humans. It's not some kind of emergent consciousness. The worry is that they will be hyper optimizers, which is what we're building them to do — and that we will have not specified their value function correctly. Therefore, they could optimize for the wrong thing. This is broadly called “the control problem” or “the value alignment problem.” Here are some groups working on this or funding it.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/eba0d7dbb8307d5c2222e169316a99be1837f4eafbfb9b57.png)

I'm going to keep moving. Here are some people who you can't really make out, but I will tell you that they are leading researchers and industrialists from the most prominent AI groups in the world. And they came together at the Asilomar Conference to really think seriously about AI safety, which I think is a tremendous achievement for the community — that we've brought everyone together like this. I'm showing this as a reflection of how exciting a time it is for you to be involved in this.

___

One conjecture that's been posed is that AI value alignment is actually not that hard of a problem. As long as we have enough time, once we have the final system we want to deploy, we can test it, right? It's like drug tests. You have the thing you are thinking about deploying in the population. You just make sure it undergoes sufficient scrutiny. However, this conjecture goes, it is almost impossible to test AI value alignment if we don't have enough time. And if that's right, which seems plausible, then it directs attention to the world in which this system is being deployed. Is it one where the developers have the incentives and the time to do these safety tests? This is one of the big things people working on AI strategy think about — this issue of how to prevent a race in AI.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/4bae8114220ae884f5fd4e047931e5cec40546b525140be3.png)

Here's the CEO of DeepMind, basically reinforcing this point: “We want to avoid a harmful race to the finish, where corner-cutting starts happening and safety gets cut. This is a big issue on a global scale, and it's extra hard when you're talking about national governments.” It's not just a race between companies in the US. It's a race between countries.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/527b6c489177f0ab61b9b3026f96fb446aa122f86555d657.png)

So what do we think about in AI strategy? A lot. One topic we think a lot about is what AI races and AI arms races would look like. Another whole class of issues is what AI could look like militarily. What are some implications of AI for the military in terms of balance of power, crisis stability, and uncertainty over capabilities?

Another thing we think about is what it means economically if we live in a world where AI is the engine of growth and value in societies, which increasingly seems to be the case. Of the top 10 firms by market capitalization, either five or six are AI companies: Google, Amazon, Apple, Microsoft. In such a world, what do countries do like Saudi Arabia or France, which don't have their own Google or Amazon, but want to be part of that value chain? We may be entering an era of AI nationalism, where countries want to build their own national champion. China is certainly in the business of doing this.

The last high-level category I’ll mention is the massive challenge of AI governance. This is an issue on a small, near-term scale. For example, how do we govern algorithms that are being used for judicial sentencing or self-driving cars? It’s also an issue on a long-run scale, when we consider what kind of electoral system or voting rules we want to use for the organization that's going to be deciding how to test and deploy superintelligence. These are very hard questions.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/69205e89fb37c8e4749f2e54ee44a600396973c900e71eb4.png)

I want to make clear that these are questions being asked today by the leading AI groups. Here are Sam Altman and Demis Hassabis asking questions like “What is the voting rule that we should use for control of our superintelligence once we build it?”

The site for governance today is the [Partnership on AI](https://www.partnershiponai.org/). This is a private-sector organization that has recently brought in NGOs, including the Future of Humanity Institute. And it's plausible that this will do a good job guiding AI in the near term, and could grow in the longer term. At some point, governments are likely to get more involved. So that's a site for study and for intervention.

Another thing we can do is try to articulate principles of good governance of AI. Here are some principles that came out of the Asilomar Conference.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/de51d0a3c02f9140c300db799aceae13f6a419c83787f971.png)

Again, a hat tip to Max Tegmark for putting that together, and especially these principles. We might want to work to identify important principles that we can get different communities to agree on, and then formalize and institutionalize them to make sure they stick.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/28fb0ad537936cd3c1fe895a57c8d0c4d3e5f0765af1b0bf.png)

In summary, what's to be done? A lot of work. What kind of skills do we need? Virtually every skill set — people who can help us grow the community, conduct operations and administration, or just be enthusiastic and efficient. We need people to do policy engagement, outreach, and media strategy. For example, what should the media strategy of a group like the Future of Humanity Institute be when there's another self-driving car incident, or when truckers are being massively displaced from their sites of employment? These are important near-term issues, but also they're sites for having conversations about longer-term issues.

We're doing work on strategy, theoretical work, and mathematical modeling of tech races. We’re trying to understand AI development and what predicts innovation, measuring actual capabilities in different sites in the world's countries to understand the value chain and the supply chain of AI. We're surveying publics and elites around the world. We're trying to design safety standards and working with AI safety researchers, and tackling a range of other issues.

If this seems important and interesting to you, I strongly encourage you to get involved. The [AI Policy Career Guide](https://80000hours.org/articles/ai-policy-guide/) has some texts that can point you in the right direction. There’s also a reading list. And in general, just reach out to me. There are also people working on this at a range of sites, and we'd be very happy to help you be productively engaged. Thanks.

## Q&A

**Nathan:** Let’s have a seat. Thank you for the talk and for being here. We'll give a second for questions to come in from the audience \[via the conference app\].

One thing that I'm struck by is it seems like we're making a lot of progress. I've been involved in this community, sort of from the edges, for a number of years. And there was a time (seven to 10 years ago) when to even talk about something like this was extremely “fringe.” Only weirdos seemed to be willing to go there. Now we have respectable people like you and a growing body of academics who are getting involved. So it seems like there has been a lot of social progress.

What about technical or practical progress on these issues? It seems that we're bringing people together, but what are those people producing so far? And should we feel any safer than we did 10 years ago?

**Allan:** I can speak to the strategy side of it. One comment that was made at the Asilomar Conference that resonated as true to many people is that AI strategy today is where AI safety was two years ago. The first beneficial meeting for AI safety was in Puerto Rico. Just a handful of individuals in the world were thinking seriously and full-time about it. That has changed today. Now the leading AI groups have safety teams. There are people doing PhDs with an eye towards AI safety. And that's very exciting. And there has been a lot of technical progress that's coming out of that.

AI strategy is just where AI safety was two years ago, but I think it's rapidly scaling up. A lot of thinking has been done, but it's not public yet. I think in the coming year you will start to see a lot of really insightful strategic analysis of AI existential risk.

I will also say I've given some talks like this (e.g., political science workshops) to other respectable audiences and none of the PhD students think this is crazy. Some of them think, “Oh, I don't have to quit smoking because of this.” That was one comment. But they all think this is real. And the challenge for them is that the discipline doesn't entirely support work on the future.

One question I got when I presented at Yale was “How will you be empirical about this?” Because we're social scientists, we like data, and we like to be empirical. And I remarked that it is about the future, and it's hard to get data on that, but we try. So I think it's a challenge for currently existing disciplines to adapt themselves to this problem, but increasingly we're finding good people who recognize the importance of the problem enough to take the time to work on it.

**Nathan:** So, the first audience question, which I think is a good one, is this: Could you provide more detail on what an AI governance board might look like? Are you thinking it will be a blue-ribbon panel of experts or more of a free-for-all, open-democracy type of structure, where anyone can contribute?

**Allan:** I think there are a lot of possibilities and I don't have a prescription at this point, so I'm not going to answer your question. But these are the issues we need to think through. There are trade-offs that I can talk about.

There's the issue of legitimacy. The UN General Assembly, for example, is often seen as a legitimate organization because every country gets a vote, but it's not necessarily the most effective international body. Also, you have to weigh your institutions in terms of power holders. So if you have the most ideal governance proposal, it might be rejected by the people who actually have the power to enact it. So you need to work with those who have power, make sure that they sign onto a regime, and try to build in sites of intervention — the key properties of whatever this development and governance regime is — so that good comes out of it.

I'll mention a few suggestions. Whatever development regime it is, I think it should have:

1.  A constitution — some explicit text that says what it's about and what it’s trying to achieve.
2.  Enough transparency so that the relevant stakeholders — be that the citizens of the country, if not citizens of the world — can see that the regime is, in fact, building AI according to the constitution. And I should say that the constitution should be a sort of common-good principle type thing.
3.  Accountability, so that if the regime isn’t working out, there's a peaceful mechanism for changing the leadership.

These are basic principles of institutional design, but it's important to get those built in.

**Nathan:** So speaking of people in power, I haven’t seen this clip, but President Obama apparently was asked about risks related to artificial intelligence. And the answer that he gave seemed to sort of equate AI risk with cybersecurity risks that we know and love today. Do you think that there is a sufficient understanding at the highest levels of power to even begin to make sense of this problem? Or do we have a fundamental lack of understanding that may be quite hard to overcome?

**Allan:** Yeah. That's a great clip. I wish I could just throw it up really quickly. We don't have the AI yet to just do that. I encourage you to watch it. It's hard to find it. [_Wired_ put it on their page](https://www.youtube.com/watch?v=72bHop6AIcc). He's asked about superintelligence and if he's worried about it and he pauses. He hesitates and gives sort of a considered “hmm” or sigh. And then he says, “Well, I've talked to my advisors and it doesn't seem to be a pressing concern.” But the pause and hesitation is enough to suggest that he really did think about it seriously. I mean, Obama is a science fiction fan. So I think he probably would have been in a good place to appreciate other risks as they arise.

But I think a lot of people in government are likely to dismiss it. Many reports from the military or government have put superintelligence worries at least sufficiently distant enough that we don't really need to think about or address it now. I will say though, that cybersecurity is likely to be a site where AI is transformative, at least in my assessment. So, that's one domain to watch in particular.

**Nathan:** Here’s another question from the audience: If there is a 5% chance of extinction due to AI, one would not be unreasonable to jump to the conclusion that maybe we should just not do this at all. It's just too hot to touch. What do you think of that idea? And second, is there any prospect of making that decision globally and somehow sticking to it?

**Allan:** Yeah. I'll flick back to the slide on opportunities. I actually had a conversation the other day with family members and friends, and one person at the table asked that question: If it's so risky, why don't we not do it? And then another friend of the family asked, “What are the impacts of AI for medicine and health, and for curing diseases?” And I think in many ways those are two sides of the policy decision. There are tremendous opportunities from AI, and not just material ones, but opportunities like curing Alzheimer's. Pretty much any problem you can imagine that's the product of intelligence could \[be solved with\] machine intelligence. So there's a real trade-off to be made.

The other issue is that stopping AI progress is politically infeasible. So I don't think it's a viable strategy, even if you thought that the trade-off weighed in favor of doing so. And I could talk a lot more about that, but that is my position.

**Nathan:** Probably the last question that we can take due to time constraints is this: Thinking about the ethical direction that we want to take as we go forward into the future — the value alignment problem — you had posed that notion that if we have two years, we can probably figure it out.

**Allan:** Yeah.

**Nathan:** But if we don't, maybe more than likely we can't. That strikes someone in the audience and I would say me too, as maybe a little too optimistic, because we've been working for thousands of years on what it means to have a good life and what good is.

**Allan:** Right.

**Nathan:** Do you think that we are closer to that then than maybe I think we are? Or how do you think about the kind of fundamental question of “What is good?” in the first place?

**Allan:** Right. To be clear, this conjecture is about whether a single person who knows what they want can build a superintelligent machine to advance those interests. It says nothing about whether we all know what we want and could agree on what we want to build. So can we agree on the political governance question: Even if we all have fundamental preferences, how do we aggregate those in a good way? And then there's the deeper question of what should we want? And yeah, those are hard questions that we need people working on, as I mentioned. In terms of a moral philosophy in politics, what do we want? We need your help figuring that out.

**Nathan:** Well, thank you for wrestling with these issues and for doing your best to protect our future. Professor Allan Dafoe, thank you very much.

The AI revolution and international politics _ Allan Dafoe _ EAG 2017 Boston-by Centre for Effective Altruism-video_id Zef-mIKjHAk-date 20170618.md


# Baobao Zhang How social science research can inform AI governance - EA Forum

_Political scientist Baobao Zhang explains how social science research can help people in government, tech companies, and advocacy organizations make decisions regarding artificial intelligent (AI) governance. After explaining her work on public attitudes toward AI and automation, she explores other important topics of research. She also reflects on how researchers could make broad impacts outside of academia._

_We’ve lightly edited Baobao’s talk for clarity. You can also watch it on_ [_YouTube_](https://www.youtube.com/watch?v=eTkvtHymI9s) _and read it on_ [_effectivealtruism.org_](https://effectivealtruism.org/articles/baobao-zhang-how-social-science-research-can-inform-ai-governance)_._

## The Talk

**Melinda Wang (Moderator):** Hello, and welcome to this session on how social science research can inform AI governance, with Baobao Zhang. My name is Melinda Wang. I'll be your emcee. Thanks for tuning in. We'll first start with a 10-minute pre-recorded talk by Baobao, which will be followed by a live Q&A session.

Now I'd like to introduce you to the speaker for this session, Baobao Zhang. Baobao is a fellow at the [Berkman Klein Center for Internet and Society](https://cyber.harvard.edu/) at Harvard University and a research affiliate with the [Centre for the Governance of AI](https://www.fhi.ox.ac.uk/govai/) at the University of Oxford. Her current research focuses on the governance of artificial intelligence. In particular, she studies public and elite opinions toward AI, and how the American welfare state could adapt to the increasing automation of labor. Without further ado, here's Baobao.

**Baobao:** Hello, welcome to my virtual presentation. I hope you are safe and well during this difficult time. My name is Baobao Zhang. I'm a political scientist focusing on technology policy. I'm a research affiliate with the Centre for the Governance of AI at the Future of Humanity Institute in Oxford. I'm also a fellow with the Berkman Klein Center for Internet and Society at Harvard University.

Today I will talk about how social science research can inform AI governance.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/fb440c6dca78eaffcdd72fe55393333b640267fc6ec1cb69.png)

Advances in AI research, particularly in machine learning (ML), have grown rapidly in recent years. Machines can outperform the best human players in strategy games like [Go](https://en.wikipedia.org/wiki/Go_(game)) and poker. You can even generate synthetic videos and news articles that easily fool humans.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/0f845f3709e38afe2a292ab5b2e946e6e8d9215c84b91c7c.png)

Looking ahead, ML researchers believe that there's a 50% chance of AI outperforming humans in all tasks by 2061. This \[estimate\] is based on a survey that my team and I conducted in 2016.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/060864652249468dd881dd8d05b2f98a822b6a5355d2c684.png)

The EA community has recognized the potential risks, even existential risks, that unaligned AI systems pose to humans. Tech companies, governments, and civil society have started to take notice as well. Many organizations have published AI ethics principles to guide the development and deployment of the technology.

A [report by the Berkman Klein Center](https://cyber.harvard.edu/publication/2020/principled-ai) counted 36 prominent sets of AI principles.

Now we're entering a phase where tech companies and governments are starting to translate these principles into policy and practice.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/5822e0b2f9709ee135eaa0b886440c1311d886f434e3c5d0.png)

At the Centre for the Governance of AI (GovAI), we think that social science research — whether it’s in political science, international relations, law, economics, or psychology — can inform decision-making around AI governance.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/9a3f2cf2c41107d7b6629b938f562100c79b819cb3b5a2a6.png)

For more information about our research agenda, please see “[AI Governance: A Research Agenda](https://www.fhi.ox.ac.uk/wp-content/uploads/GovAI-Agenda.pdf)” by Allan Dafoe. It's also a good starting place if you're curious about the topic and are new to it.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/0934fcedb98c6f697a5d01d1afa7359faaefb082da4dd962.png)

Here's a roadmap for my talk. I’ll cover:

-   My research on public opinion toward AI
-   EA social science research highlights on AI governance
-   Research questions that I've been thinking about a lot lately
-   How one can be impactful as a social scientist in this space

### Why study the public’s opinion of AI?

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/e0106a04f566e148745c8c8dc5880e9ea1152ac5306427a8.png)

From a normative perspective, we need to consider the voices of those who will be impacted by AI. In addition, public opinion has shaped policy in many other domains, including climate change and immigration; therefore, studying public opinion could help us anticipate how electoral politics may impact AI governance.

The research I'm about to present comes from [this report](https://isps.yale.edu/sites/default/files/files/Zhang_us_public_opinion_report_jan_2019.pdf).

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/d63fd689a349045a7260a648497f5b9eb3ec024d894829ea.png)

It's based on a nationally representative survey of 2,000 Americans that Allan Dafoe and I conducted in the summer of 2018.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/d71f6b1b571a5c127b8e7808e157f6bc613de4701e111ebc.png)

Here are the main takeaways from the survey:

1\. An overwhelming majority of Americans think that AI should be carefully managed.
2\. They considered all 13 governance challenges that we presented to them to be important.
3\. However, they have only low-to-moderate levels of trust in the actors who are developing and managing AI.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/f64b1b9c399746f28f4b8bb98bc7ba062a7200d364e3306a.png)

Now, on to some results. Here's a graph of Americans' view of AI governance challenges. Each respondent was randomly assigned to consider five challenges randomly selected from 13. The x-axis shows the respondents’ perceived likelihood that the governance challenge would impact large numbers of people around the world. The y-axis shows the perceived importance of the issue.

Those perceived to be high in both dimensions include protecting data privacy, preventing AI-enhanced cyber attacks, preventing mass surveillance, and preventing digital manipulation — all of which are highly salient topics in the news.

I’ll point out that the respondents consider all of these AI governance challenges to be important for tech companies and governments to manage. But we do see some variations between respondents when we break them down by subgroups.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/eedca4761990c8164513480745457644cb21962ee78fec14.png)

Here we've broken it down by age, gender, race, level of education, partisanship, etc. We're looking at the issue’s perceived importance in these graphs. Purple means greater perceived issue importance. Green means lesser perceived issue importance.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/9660a291d9489697ffe72699b383e0e3830f8a2f014bcc21.png)

I'll highlight some differences that really popped out. In this slide, you see that older Americans, in contrast to younger Americans, perceive the governance challenges presented to them to be more important.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/4517087c727b1965c6872371b66055cdef7399f465882c53.png)

Interestingly, those who have CS \[computer science\] or engineering degrees, in contrast to those who don't, perceive all of the governance challenges to be less important. We also observed this techno-optimism among those with CS or engineering degrees in other survey questions.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/c256c6159734d52f2836096b8c18f17390cbaa4abab0ec69.png)

Despite Americans perceiving that these AI governance challenges are important, they have low to moderate levels of trust in the actors who are in a position to shape the development and deployment of AI systems.

There are a few interesting observations to point out in these slides:

-   While trust in institutions has declined across the board, the American public still seems to have relatively high levels of trust in the military. This is in contrast to the ML community, whose members would rather not work with the US military. I think we get this seemingly strange result because the public relies on heuristics when answering this question.
-   The American public seems to have great distrust of Facebook. Part of it could be the fallout from the [Cambridge Analytica scandal](https://en.wikipedia.org/wiki/Facebook%E2%80%93Cambridge_Analytica_data_scandal). But when we ran a previous survey before the scandal broke, we observed similarly low levels of trust in Facebook.

I'm sharing just some of the results from our report. I encourage you to read it. We're currently working on a new iteration of the survey. And we're hoping to launch it concurrently in the US, EU, and China, \[which will allow us\] to make some interesting cross-country comparisons.

I would like to highlight two works by my colleagues in the EA community who are also working in AI governance.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/8f91c170f28bfdb93332b1219f8a3606b422840d6f966aef.png)

First, there’s “[Toward Trustworthy AI Development](https://arxiv.org/abs/2004.07213).” This paper came out recently. It's a massive collaboration among researchers in different sectors and fields, \[with the goal of determining\] how to verify claims made by AI developers who say that their algorithms are safe, robust, and fair. This is not merely a technical question. Suggestions in the report include creating new institutional mechanisms like bounties for detecting bias and safety issues in AI systems, and creating AI incident reports.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/52527536eb1165dcbab706872defa17ddbb73c1ec298e298.png)

Second, there’s “[The Windfall Clause](https://www.fhi.ox.ac.uk/windfallclause/)” by my colleagues at GovAI. Here, the team considers this idea of a “windfall clause” as a way to redistribute the benefits from transformative AI. This is an ex-ante agreement, where tech companies — in the event that they make large profits from their AI systems — would donate a massive portion of those profits. The report combines a lot of research from economic history and legal analysis to come up with an inventive policy proposal.

### New research questions

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/28134c847ff756822ffa4845ea7fb0d3239afcd0a64b010c.png)

There are a lot of new and interesting research questions that keep me up at night. I’ll share a few of them with you, and let's definitely have a discussion about them during the Q&A.

-   **How do we build incentives for developing safe, robust, and fair AI systems — and avoid a race to the bottom?** I think a lot of us are rather concerned about the rhetoric of an AI arms race. But it's also true that even the EU is pushing for competitiveness in AI research and development. I think the “[Toward Trustworthy AI Development](https://arxiv.org/abs/2004.07213)” paper gives some good recommendations on the R&D \[research and development\] front. But what will the market and policy incentives be for businesses and the public sector to choose safer AI products? That's still a question that I and many of my colleagues are interested in.
-   **How can we transition to an economic system where AI can perform many of the tasks currently done by humans?** I've been studying perceptions of automation. Unfortunately, a lot of workers underestimate the likelihood that their jobs will be automated. They actually have an optimism bias. Even correcting workers’ \[false\] beliefs about the future of work in my studies has failed to make them more supportive of redistribution. And it doesn't seem to decrease their hostility toward globalization. So certainly there's a lot more work to be done on the political economy around the future of work.
-   **How do other geopolitical risks make AI governance more difficult?** I think about these a lot. In Toby Ord's book, [_The Precipice_](https://www.amazon.com/dp/B07V9GHKYP/), he talks about the risk factors that could increase the probability of existential risk. And one of these risks is great-power war. We're not at \[that point\], but there has certainly been a rise in aggressive nationalism from some of these great powers. Instead of coming together to combat the COVID pandemic, many countries are pointing fingers at each other. And I think these trends don't bode well for international governance. Therefore, thinking about how these trends might shape international cooperation around AI governance is definitely a topic that my colleagues and I are working on.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/ec30e6c2c3b8cc47b7f2c179a9067bc7f525eb4117acf089.png)

I’ll conclude this presentation by talking about how one can be impactful as a social scientist. I have the great luxury of working in academia, where I have plenty of time to think and carry out long-term research projects. At the same time, I have to constantly remind myself to engage with the world outside of academia — the tech industry and the policy world — by writing op eds, doing consulting, and communicating with the media.

Fortunately, social scientists with expertise in AI and AI policy are also in demand in other settings. Increasingly, tech companies have sought to hire people to conduct research on how individual humans interact with AI systems, or what the impact of AI systems may be on society. Geoffrey Irving and Amanda Askell have published a paper called “[AI Safety Needs Social Scientists](https://openai.com/blog/ai-safety-needs-social-scientists/).” I encourage you to read it if you're interested in this topic.

To give a more concrete example, some of my colleagues have worked with [OpenAI](https://openai.com/) to test whether their [GPT-2 language model](https://openai.com/blog/better-language-models/) can generate news articles that fool human readers.

Governments are also looking for social scientists with expertise in AI. Policymakers in both the civilian government and in the military have an AI literacy gap. They don't really have a clear understanding of the limits and the potentials of the technology. But advising policymakers does not necessarily mean that you have to work in government. Many of my colleagues have joined think tanks in Washington, DC, where they apply their research skills to generate policy reports, briefs, and expert testimony. I recommend checking out the [Center for the Security and Emerging Technology](https://cset.georgetown.edu/) or CSET, based at Georgetown University. They were founded about a year ago, but they have already put out a vast collection of research on AI and US international relations.

Thank you for listening to my presentation. I look forward to your questions during the Q&A session.

**Melinda:** Thank you for that talk, Baobao. The audience has already submitted a number of questions. We're going to get started with the first one: \[Generally speaking\], what concrete advice would you give to a fresh college graduate with a degree in a social science discipline?

**Baobao:** That's a very good question. Thank you for coming to my talk.

One of the unexpected general pieces of advice that I would give is to have strong writing skills. At the end of the day, you need to translate all of the research that you do for different audiences: readers of academic journals, policymakers, tech companies \[that want you to produce\] policy reports.

Besides that, I think \[you may need to learn certain skills depending on\] the particular area in which you specialize. For me, learning data science and statistics is really important for the type of research that I do. For other folks it might be game theory, or for folks doing qualitative research, it might be how to do elite interviews and ethnographies.

But overall, I think having strong writing skills is quite critical.

**Melinda:** Great. \[Audience member\] Ryan asks, “What are the current talent gaps in AI safety right now?”

**Baobao:** That's a good question. I must confess: I'm not an AI safety expert, although I did talk about a piece that folks at OpenAI wrote called “[AI Safety Needs Social Scientists.](https://openai.com/blog/ai-safety-needs-social-scientists/)” And I definitely agree with the sentiment, given that the people who are working on AI safety want to run experiments. You can think of them as psychology experiments. And a lot of computer scientists are not necessarily trained on how to do that. So if you have skills in running surveys or psych experiments, that’s a skill set that I hope tech companies will acknowledge and recognize as important.

**Melinda:** That's really interesting. Would you consider psychology to be within the realm of social sciences, \[in terms of how\] people generally perceive it? Or do you mean fields related to STEM \[science, technology, engineering, math\]?

**Baobao:** I think psychology is quite interesting. People who are more on the neuroscience side might be \[considered to be in\] STEM. I work with some experimental psychologists, particularly social psychologists, and they read a lot of the literature in economics, political science, and communications studies. I do think that there's a bit of overlap.

**Melinda:** How important do you consider interdisciplinary studies to be, whether that's constrained within the realm of social science, or social science within STEM, etc.?

**Baobao:** I think it's important to work with both other social scientists and with computer scientists.

This is more \[along the lines of\] career advice and not related to AI governance, but one of the realizations I had in doing recent work on several COVID projects is that it’s important to have more than just an “armchair” \[level of understanding\] of public health. I try to get my team to talk with those who are either in vaccine development, epidemiology, or public health. And I'd like to see more of that type of collaboration in the AI governance space. We do that quite well at that, I think, at GovAI, where we can talk to in-house computer scientists at the [Future of Humanity Institute](https://www.fhi.ox.ac.uk/).

**Melinda:** Yes. That's an interesting point and relates to one of the questions that just came in: How can we more effectively promote international cooperation? Do you have any concrete strategies that you can recommend?

**Baobao:** Yes. That's a really good question. I work a bit with folks in the European Union, and bring my AI governance expertise to the table. The team that I'm working with just submitted a consultation to the EU Commission. That type of work is definitely necessary.

I also think that collaboration with folks who work on AI policy in China is fruitful. I worry about the decoupling between the US and China. There's a bit of tension. But if you're in Europe and you want to collaborate with Chinese researchers, I encourage it. I think this is an area that more folks should look into.

**Melinda:** Yes, that's a really good point.

In relation to the last [EAGxVirtual talk on biosecurity](https://www.youtube.com/watch?v=BbOHQLrVSX4), can you think of any information hazards within the realm of AI governance?

**Baobao:** That's a good question. We do think a lot about all of our publications. At GovAI, we talk about being careful in our writing so that we don't necessarily escalate tensions between countries. I think that's definitely something that we think about.

At the same time, there is the [open science movement](https://en.wikipedia.org/wiki/Open_science) in the social sciences. It’s tricky to \[find a good\] balance. But we certainly want to make sure our work is accurate and speaks to our overall mission at GovAI of promoting beneficial AI and not doing harm.

**Melinda:** Yes. A more specific question someone has asked is “Do you think this concept of info hazards — if it's a big problem in AI governance — would prohibit one from spreading ideas within the \[discipline\]?”

**Baobao:** That's a good question. I think the EA community is quite careful about not spreading info hazards. We're quite deliberate in our communication. But I do worry about a lot of the rhetoric that other folks who are in this AI governance space use. There are people who want to drum up a potential AI arms race — people who say, “Competition is the only thing that matters.” And I think that's the type of dangerous rhetoric that we want to avoid. We don't want a race to the bottom where, whether it's the US, China, or the EU, researchers only care about \[winning, and fail to consider\] the potential risks of deploying AI systems that are not safe.

**Melinda:** Yes. Great. So we're going to shift gears a little bit into more of the nitty-gritty of your talk. One question that an audience member asked is “How do you expect public attitudes toward AI to differ by nationality?”

**Baobao:** That's a good question. In the talk, I mentioned that at GovAI, we're hoping to do a big survey in the future that we run concurrently in different countries.

Judging from what I've seen of the literature, [Eurobarometer](https://ec.europa.eu/commfrontoffice/publicopinion/index.cfm) has done a lot of good surveys in the EU. As you would expect, folks living in Europe — where there are tougher privacy laws — tend to be more concerned about privacy. But it’s not necessarily so that Chinese respondents are totally okay with a lack of privacy.

That’s why we're hoping to do a survey in which we ask \[people in different countries\] the same questions, around the same time. I think it’s really important to make these cross-national comparisons. With many questions, you get different responses because of how the questions are framed or worded, so \[ensuring a\] rigorous approach will yield a better answer to this question.

**Melinda:** Yes. I'd like you to unpack that a bit more. In what concrete ways can we be more culturally sensitive in stratifying these risks when it comes to international collaboration?

**Baobao:** I think speaking the \[appropriate\] language is really important. I can provide a concrete example. As my team worked on the EU consultation that I mentioned, we talked to folks who work at the European Commission in order to understand their particular concerns and “speak their language.”

They're concerned about AI competition and the potential for an arms race, but they don't want to use that language. They also care a lot about human rights and privacy. And so when we make recommendations, since we’ve read the \[relevant\] reports and spoken to the people involved in the decision-making, those are the two things that we try to balance.

And in terms of the survey research that we're hoping to do, we're consulting with folks on the ground so that our translation work is localized and \[reflects\] cultural nuances.

**Melinda:** Yes, that's a really good point about using the right language. It also \[reminds\] me of how people in the social sciences often think differently, and may use different \[terminology\] than people from the STEM realm, for example. The clash of those two cultures can sometimes result in conflicts. Do you have any advice on how to mitigate those kinds of conflicts?

**Baobao:** That's a good question. Recently, GovAI published a guide to \[writing the\] impact statements required for the [NeurIPS](https://nips.cc/) conference. One of the suggestions was that computer scientists trying to \[identify\] the societal impact of their research talk with social scientists who can help with this translational work. Again, I think it’s quite important to take an interdisciplinary approach when you conduct research.

**Melinda:** Yes. Do you think there's a general literacy gap between these different domains? And if so, how should these gaps be filled?

**Baobao:** Yes, good question. My colleagues, Mike Horowitz and \[Lauren\] Kahn at the University of Pennsylvania, have written a [piece about the AI literacy gap in government](https://warontherocks.com/2020/01/the-ai-literacy-gap-hobbling-american-officialdom/). They acknowledge that it's a real problem. Offering crash courses to train policymakers or social scientists who are interested in advising policymakers is one way to \[address this issue\]. But I do think that if you want to work in this space, doing a deep dive — not just taking a crash course — can be really valuable. Then, you can be the person offering the instruction. You can be the one writing the guides.

So yes, I do think there's a need to increase the average level of AI literacy, but it’s also important for the social science master's degree and PhD programs to train people.

**Melinda:** Yes, that's interesting. It brings up a question from \[audience member\] Chase: “Has there been more research done on the source of technical optimism from computer scientists and engineers? \[Is this related to\] overconfidence in their own education or in fellow developers?”

**Baobao:** That's a good question. We \[work with\] machine learning researchers at GovAI, and \[they’ve done\] research that we hope to share later this year. I can't directly address that research, but there may be a U-shape curve. If the x-axis represents your level of expertise in AI, and the y-axis represents your level of concern about AI safety or risks from AI, those who don't have a lot of expertise are kind of concerned. But those who have CS or engineering degrees are perhaps not very worried.

But then, if you talk to the machine learning researchers themselves, many are concerned. I think they're recognizing that what they work on can have huge societal impacts. You have folks who work in AI safety who are very concerned about this. But in general, I think that the machine learning field is waking up to these potential risks, given the proliferation of AI ethics principles coming from a lot of different organizations.

**Melinda:** Yes, it’s interesting how the public is, for once, aligned with the experts in ML. I suppose in this instance, the public is pretty enlightened.

I'm not sure if this question was asked already, but are organizations working on AI governance more funding-constrained or talent-constrained?

**Baobao:** That's a good question. May I \[mention\] that at GovAI, we're looking to hire folks in the upcoming months? We're looking for someone to help us on survey projects. And we're also looking for a project manager and other researchers. So, in some sense, we have the funding — and that's great. And now we just need folks who can do the research. So that's my plug.

I can't speak for all organizations, but I do think that there is a gap in terms of training people to do this type of work. I'm going to be a faculty member at a public policy school, and they're just beginning to offer AI governance as a course. Certainly, a lot of self-study is helpful. But hopefully, we'll be able to get these courses into the classrooms at law schools, public policy schools, and different PhD or master's degree programs.

**Melinda:** Yes. I think perhaps the motivation behind that question was more about the current general trend around a lot of organizations in EA being extremely competitive to get into — especially EA-specific job posts. I'm wondering whether you can provide a realistic figure for how talent-constrained AI governance is, as opposed to being funding-constrained.

**Baobao:** I can't speak to the funding side. But in terms of the research and the human-resource side, I think we're beginning to finally \[arrive at\] some concrete research questions. At the same time, we're building out new questions. And it's hard to predict what type of skill set you will need.

As I mentioned in my talk, GovAI has realized that social science skills are really important — whether you can do survey research, legal analysis, elite interviews on the sociology side, or translational work. So, although I don't have a good answer to that question, I think getting some sort of expertise in one of the social sciences and having expertise in AI policy or AI safety are the types of skills that we're looking for.

**Melinda:** Okay. Given that we have six minutes left, I'm going to shift gears a little bit and try to get through as many questions as possible. Here’s one: “What insights from nuclear weapons governance can inform AI governance?”

**Baobao:** I think that's a really good question. And I think you've caught me here. I have some insights into the dual-use nature of nuclear weapons. People talk about AI as a “general-purpose technology.” It could be both beneficial and harmful. And speaking from my own expertise in public opinion research, one of the interesting findings in this space is that people tend to reject nuclear energy because of its association with nuclear weapons. There's a wasted opportunity, because nuclear energy is quite cheap and not as harmful as, say, burning fossil fuels. But because of this negative association, we’ve sort of rejected nuclear energy.

\[Turning to\] trust in AI systems, we don't want a situation where people's association of AI systems is so negative that they reject applications that could be beneficial to society. Does that make sense?

**Melinda:** Yes.

**Baobao:** I recommend \[following the work of\] some of my colleagues in international relations, at organizations like [CSET](https://cset.georgetown.edu/) who have written about this. I'm sorry — it's not my area of expertise.

**Melinda:** Okay. The next question is “Is there a way we can compare public trust in different institutions, specifically regarding AI, compared to a general baseline trust in that institution — for example, in the case of the public generally having greater trust in the US military?”

**Baobao:** That's a good question. I think that, because AI is so new, the public relies on their heuristics rather than what they know. They're just going to rely on what they think of as a trustworthy institution. And one thing that I've noticed, and that I wrote about in my [Brookings report](https://www.brookings.edu/research/public-opinion-lessons-for-ai-regulation/), is how some areas of political polarization around AI governance map to what you would expect to find in other domains. It’s concerning, in the context of the US at least, that if we can't agree on the right policy solution, we might just \[rely on\] partisan rhetoric about AI governance.

For example, acceptance of facial recognition software maps to race and partisanship. African Americans really distrust facial recognition. Democrats tend to distrust facial recognition. Republicans, on the other hand, tend to have a greater level of acceptance. So you see this attitude towards policing being mapped to facial recognition.

You also see it in terms of regulating algorithmically curated social media content. There seems to be a bipartisan backlash against tech, but when you dig more deeply into it, as we've seen recently, Republicans tend to think about content moderation as censorship and \[taking away\] a right, whereas Democrats tend to see it as combating misinformation. So unfortunately, I do think that partisanship will creep into the AI governance space. And that's something that I'm actively studying.

**Melinda:** Yes. We only have one minute left. I'm going to go through the last questions really quickly. With regards to the specific data that you presented, you mentioned that the public considers all AI governance challenges important. Did you consider including a made-up governance challenge to check response bias?

**Baobao:** Oh, that's a really good suggestion. We haven't, but we certainly can do that in the next round.

**Melinda:** Okay, wonderful. And that concludes the Q&A part of this session. \[...\] Thanks for watching.

How social science research can inform AI governance _ Baobao Zhang _ EAGxVirtual 2020-by Centre for Effective Altruism-video_id eTkvtHymI9s-date 20200615.md