# Amanda Askell AI safety needs social scientists - EA Forum

_When an AI wins a game against a human, that AI has usually trained by playing that game against itself millions of times. When an AI recognizes that an image contains a cat, it’s probably been trained on thousands of cat photos. So if we want to teach an AI about human preferences, we’ll probably need lots of data to train it. And who is most qualified to provide data about human preferences? Social scientists! In this talk from EA Global 2018: London, Amanda Askell explores ways that social science might help us steer advanced AI in the right direction._

_A transcript of Amanda's talk is below, which CEA has lightly edited for clarity. You can also read this talk on_ [_effectivealtruism.org_](https://www.effectivealtruism.org/articles/ea-global-2018-ai-safety-needs-social-scientists)_, or watch it on_ [_YouTube_](https://www.youtube.com/watch?v=TWHcK-BNo1w)_._

## The Talk

![](https://images.ctfassets.net/ohf186sfn6di/3sbM9JNPFim0eNdrSmFH1g/185b47e7bfbab29b940a9658d140ece1/1000_Amanda_Askell.jpg)

Here's an overview of what I'm going to be talking about today. First, I'm going to talk a little bit about why learning human values is difficult for AI systems. Then I'm going to explain to you the safety via debate method, which is one of the methods that OpenAI's currently exploring for helping AI to robustly do what humans want. And then I'm going to talk a little bit more about why I think this is relevant to social scientists, and why I think social scientists - in particular, people like Experimental Psychologists and Behavioral Scientists - can really help with this project. And I will give you a bit more details about how they can help, towards the end of the talk.

![](https://images.ctfassets.net/ohf186sfn6di/uy4ZBr1zoWXrpvj2mco3P/02e0f9fca0e2bc17e6e7d526ad0adaff/1000_Amanda_Askell__1_.jpg)

Learning human values is difficult. We want to train AI systems to robustly do what humans want. And in the first instance, we can just imagine this being what one person wants. And then ideally we can expand it to doing what most people would consider good and valuable. But human values are very difficult to specify, especially with the kind of precision that is required of something like a machine learning system. And I think it's really important to emphasize that this is true even in cases where there's moral consensus, or consensus about what people want in a given instance.

So, take a principle like "do not harm someone needlessly." I think we can be really tempted to think something like: "I've got a computer, and so I can just write into the computer, 'do not harm someone needlessly'". But this is a really underspecified principle. Most humans know exactly what it means, they know exactly when harming someone is needless. So, if you're shaking someone's hand, and you push them over, we think this is needless harm. But if you see someone in the street who's about to be hit by a car, and you push them to the ground, we think that's not an instance of needless harm.

Humans have a pretty good way of knowing when this principle applies and when it doesn't. But for a formal system, there's going to be a lot of questions about precisely what's going on here. So, one question this system may ask is, how do I recognize when someone is being harmed? It's very easy for us to see things like stop signs, but when we're building self-driving cars, we don't just program in something like, "stop at stop sign". We instead have to train them to be able to recognize an instance of a stop sign.

And then the principle that says that you shouldn't harm someone needlessly employs the notion that we understand when harm is and isn't appropriate, whereas there are a lot of questions under the surface like, when is harm justified? What is the rule for all plausible scenarios in which I might find myself? These are things that you need to specify if you want your system to be able to work in all of the cases that you want it to be able to work in.

I think that this is an important point to internalize. It's easy for humans to identify, and to pick up, say, a glass. But training a ML System to perform the same task requires a lot of data. And this is true of a lot of tasks that humans might intuitively think are easy, and we shouldn't then just transfer that intuition to the case of machine learning systems. And so when we're trying to teach human values to any AI system, it's not that we're just looking at edge cases, like trolley problems. We're really looking at core cases of making sure that our ML Systems understand what humans want to do, in the everyday sense.

There are many approaches to training an AI to do what humans want. One way is through human feedback. You might think that humans could, say, demonstrate a desired behavior for an AI to replicate. But there are some behaviors it's just too difficult for humans to demonstrate. So you might think that instead a human can say whether they approve or disapprove of a given behavior, but this might not work too well, either. Learning what humans want this way, we have a reward function as predicted by the human. So on this graph, we have that and AI strength. And when AI strength reaches the superhuman level, it becomes really hard for humans to give the right reward function.

![](https://images.ctfassets.net/ohf186sfn6di/rwnArff47n7OixOh7R3Wd/b5b2452a2cf0426d8fb9f7d4a3e991ad/1000_Amanda_Askell__2_.jpg)

As AI capabilities surpass the human level, the decisions and behavior of the AI system just might be too complex for the human to judge. So imagine agents that control, say, we've given the example of a large set of industrial robots. That may just be the kind of thing that I couldn't evaluate whether these robots were doing a good job overall; it'd be extremely difficult for me to do so.

And so the concern is that when behavior becomes much more complex and much more large scale, it becomes really hard for humans to be able to judge whether an AI agent is doing a good job. And that's why you may expect this drop-off. And so this is a kind of scalability worry about human feedback. So what ideally needs to happen instead is that, as AI strength increases, what's predicted by the human is also able to keep pace.

![](https://images.ctfassets.net/ohf186sfn6di/4TqczIX2kVjmg7X8pj2bTI/715c7b83ddee2f77ca53115d574bece2/1000_Amanda_Askell__3_.jpg)

So how do we achieve this? One of the things that we want to do here is to try and break down complex questions and complex tasks into simpler components. Like, having all of these industrial robots perform a complex set of functions that comes together to make something useful, into some smaller set of tasks and components that humans are able to judge.

![](https://images.ctfassets.net/ohf186sfn6di/2ypn6ELXkLoFHv4JWjf51w/1639980a00238733e975251f1fa070eb/1000_Amanda_Askell__4_.jpg)

So here is a big question. And the idea is that the overall tree might be too hard for humans to fully check, but it can be decomposed into these elements, such that at the very bottom level, humans can check these things.

So maybe the example of "how should a large set of industrial robots be organized to do task x" would be an example of a big question where that's a really complex task, but there's some things that are checkable by humans. So if we could decompose this task so that we were asking a human, if one of the robots performs this small action, will the result be this small outcome? And that's something that humans can check.

So that's an example in the case of industrial robots accomplishing some task. In the case of doing what humans want more generally, a big question is, what _do_ humans want?

![](https://images.ctfassets.net/ohf186sfn6di/591rW9o1DV0GyqrmLMF6Iy/1125f7797fad505ab09a92981dc6e8b7/1000_Amanda_Askell__5_.jpg)

A much smaller question, if you can manage to decompose this, is something like: Is it better to save 20 minutes of someone's time, or to save 10 minutes of their time? If you imagine some AI agent that's meant to assist humans, this is a fact that we can definitely check. Even though I can't tell my assistant AI exactly everything that I want, I can tell it that I'd rather it save 20 minutes of my time than save 10 minutes of my time.

![](https://images.ctfassets.net/ohf186sfn6di/3DiPRRPcxWh7fNIZPlfo25/4d99ec5091bc14f9783166f186a5c8be/1000_Amanda_Askell__6_.jpg)

One of the key issues is that, with current ML Systems, we need to train on a lot of data from humans. So if you imagine that we want humans to actually give this kind of feedback on these kind of ground level claims or questions, then we're going to have to train on a lot of data from people.

To give some examples, simple image classifiers train on thousands of images. These are ones you can make yourself, and you'll see the datasets are pretty large. AlphaGo Zero played nearly 5 million games of Go during its training. OpenAI Five trains on 180 years of Dota 2 games per day. So this gives you a sense of how much data you need to train these systems. So if we are using current ML techniques to teach AI human values, we can't rule out needing millions to tens of millions of short interactions from humans as the data that we're using.

So earlier I talked about human feedback, where I was assuming that we were asking humans questions. We could just ask humans really simple things like, do you prefer to eat an omelette or 1000 hot dogs? Or, is it better to provide medicine or books to this particular family? One way that we might think that we can get more information from the data that we're able to gather is by finding reasons that humans have for the answers that they give. So if you manage to learn that humans generally prefer to eat a certain amount per meal, you can rule out a large class of questions you might ever want to ask people. You're never going to ask them, do you prefer to eat an omelette or 1000 hot dogs? Because you know that humans just generally don't like to eat 1000 hot dogs in one meal, except in very strange circumstances.

![](https://images.ctfassets.net/ohf186sfn6di/DK35z4wisxFgb9Way6DC3/561f98d2345f6b206c1899fad738f636/1000_Amanda_Askell__7_.jpg)

And we also know facts like, humans prioritize necessary health care over mild entertainment. So this might mean that, if you see a family that is desperately in need of some medicine, you just know that you're not going to say, "Hey, should I provide them with an entertaining book, or this essential medicine?" So there's a sense in which when you can identify the reasons that humans are giving for their answers, this lets you go beyond, and learn faster what they're going to say in a given circumstance about what they want. It's not to say that you couldn't learn the same things by just asking people questions, but rather if you can find a quicker way to identify reasons, then this could be much more scalable.

Debate is a proposed method, which is currently being explored, for trying to learn human reasons. So, to give you of definition of a debate here, the idea is that two AI agents are going to be given a question, and they take turns making short statements, and a human judge is at the end, who chooses which of the statements gave them the most true, valuable information. It's worth knowing that this is quite dissimilar from a lot of human debates. With human debates, people might give one answer, but then they might adjust their answer over the course of a debate. Or they might debate with each other in a way that's more exploratory. They're gaining information from each other, which then they're updating on, and then they're feeding that back into the debate.

![](https://images.ctfassets.net/ohf186sfn6di/3qGHsI1S5oh6veYLsJXVUb/02ed1b45d27ba513ce24f9eccf1faad3/1000_Amanda_Askell__8_.jpg)

With AI debates, you're not doing it for information value. So it's not going to have the same exploratory component. Instead, you would hopefully see the agents explore a path kind of like this.

So imagine I want my AI agents to decide which bike I should buy. I don't want to have to go and look up all the Amazon reviews, etc. In a debate, I might get something like, "You should buy the red road bike" from the first agent. Suppose that blue disagrees with it. So blue says "you should buy the blue fixie". Then red says, "the red road bike is easier to ride on local hills". And one of the key things to suppose here is that for me, being able to ride on the local hills is very important. It may even overwhelm all other considerations. So, even if the blue fixie is cheaper by $100, I just wouldn't be willing to pay that. I'd be happy to pay the extra $100 in order to be able to ride on local hills.

And if this is the case, then there's basically nothing true that the other agent can point to, to convince me to buy the blue fixie, and blue should just say, "I concede". Now, blue could have lied for example, but if we assume that red is able to point out blue's lies, we should just expect blue to basically lose this debate. And if it's explored enough and attempted enough debates, it might just see that, and then say, "Yes, you've identified the key reason, I concede."

And so it's important to note that we can imagine this being used to identify multiple reasons, but here it has identified a really important reason for me, something that is in fact going to be really compelling in the debate, namely, that it's easier to ride on local hills.

![](https://images.ctfassets.net/ohf186sfn6di/1Xee1kOdkpovIGQYEKsmT9/2cae404cb156ab1f2a1d9965589c7c3c/1000_Amanda_Askell__9_.jpg)

Okay. So, training an AI to debate looks something like this. If we imagine Alice and Bob are our two debaters, and each of these is like a statement made by each agent. And so you're going to see exploration of the tree. So the first one might be this. And here, say that the human decides that Bob won in that case. This is another node, another node. And so this is the exploration of the debate tree. And so you end up with a debate tree that looks a little bit like a game of Go.

![](https://images.ctfassets.net/ohf186sfn6di/4KDjHqgJP1qorwpRXDpHoN/aa2ed26c3e69e3824f6174ac1fd34feb/1000_Amanda_Askell__10_.jpg)

When you have AI training to play Go, it's exploring lots of different paths down the tree, and then there's a win or loss condition at the end, which is its feedback. This is basically how it learns to play. With debate, you can imagine the same thing, but where you're exploring, you know, a large tree of debates and humans assessing whether you win or not. And this is just a way of training up AI to get better at debate and to eventually identify reasons that humans find compelling.

![](https://images.ctfassets.net/ohf186sfn6di/3j6F8kVDPjZ9vURzl5FHfv/5002563fab843a1fb1d3675465d74f1c/1000_Amanda_Askell__11_.jpg)

One thesis here that I think is relatively important is something I'll call the positive amplification thesis, or positive amplification threshold. One thing that we might think, or that seems fairly possible, is that if humans are above some threshold of rationality and goodness, then debate is going to amplify their positive aspects. This is speculative, but it's a hypothesis that we're working with. And the idea here is that, if I am pretty irrational and pretty well motivated, I might get some feedback of the form, "Actually, that decision that you made was fairly biased, and I know that you don't like to be biased, so I want to inform you of that."

I get informed of that, and I'm like, "Yes, that's right. Actually, I don't want to be biased in that respect." Suppose that the feedback comes from Kahneman and Tversky, and they point out some key cognitive bias that I have. If I'm rational enough, I might say, "Yes, I want to adjust that." And I give a newer signal back in that has been improved by virtue of this process. So if we're somewhat rational, then we can imagine a situation in which all of these positive aspects of us are being amplified through this process.

But you can also imagine a negative amplification. So if people are below this threshold of rationality and goodness, we might worry the debate would amplify these negative aspects. If it turns out we can just be really convinced by appealing to our worst natures, and your system learns to do that, then it could just put that feedback in, becoming even less rational and more biased, and so on. So this is an important hypothesis related to work on amplification, which if you're interested in, it's great. And I suggest you take a look at it, but I'm not going to focus on it here.

![](https://images.ctfassets.net/ohf186sfn6di/45jh1XbWVTnw5A07C8J3rC/7c50ec13ad3623919157a1045427ea84/1000_Amanda_Askell__12_.jpg)

Okay. So how can social scientists help with this whole project? Hopefully I've conveyed some of what I think of as the real importance of the project. It reminds me a little bit of Tetlock's work on Superforecasters. A lot of social scientists have done work identifying people who are Superforecasters, where they seem to be robustly more accurate in their forecasts than many other people, and they're robustly accurate across time. We've found other features of Superforecasters too, like, for example, working in groups really helps them.

So one question is whether we can identify good human judges, or we can train people to become, essentially, Superjudges. So why is this helpful? So, firstly, if we do this, we will be able to test how good human judges are, and we'll see whether we can improve human judges. This means we'll be able to try and find out whether humans are above the positive amplification threshold.

So, are ordinary human judges good enough to cause an amplification of their good features? One reason to learn this is that it improves the quality of the judging data that we can get. If people are just generally pretty good, rational at assessing debate, and fairly quick, then this is excellent given the amount of data that we anticipate needing. Basically, improvements to our data could be extremely valuable.

If we have good judges, positive amplification will be more likely during safety via debate, and also will improve training outcomes on limited data, which is very important. This is one way of kind of framing why I think social scientists are pretty valuable here, because there's lots of questions that we really do want asked when it comes to this project. I think this is going to be true of other projects, too, like asking humans questions. The human component of the human feedback is quite important. And getting that right is actually quite important. And that's something that we anticipate social scientists to be able to help with, more so than like AI researchers who are not working with people, and their biases, and how rational they are, etc.

![](https://images.ctfassets.net/ohf186sfn6di/4p9uTVJUI53FEhnCDi5eBi/fa84d21cca5b27ffae14e866cba938dc/1000_Amanda_Askell__13_.jpg)

These are questions that are the focus of social sciences. So one question is, how skilled are people as judges by default? Can we distinguish good judges of debate from bad judges of debate? And if so, how? Does judging ability generalize across domains? Can we train people to be better judges? Like, can we engage in debiasing work, for example? Or work that reduces cognitive biases? What topics are people better or worse at judging? Are there ways of phrasing questions so that people are better at assessing them? Are there ways of structuring debates that make them easier to judge, or restricting debates to make them easier to judge? So we're often just showing people a small segment of a debate, for example. Can people work together to improve judging qualities? These are all outstanding questions that we think are important, but we also think that they are empirical questions and that they have to be answered by experiment. So this is, I think, important potential future work.

![](https://images.ctfassets.net/ohf186sfn6di/7bXLwYELhYOCjfKLTzvvW6/3cdd6ab84bddf240b2878979800c31f4/1000_Amanda_Askell__14_.jpg)

We've been thinking a little bit about what you would want in experiments that try and assess judging ability in humans. So one thing you'd want is that there's a verifiable answer. We need to be able to tell whether people are correct or not, in their judgment of the debate. The other is that there is a plausible false answer, because if you have a debate, if we can only train and assess human judging ability on debates where there's no plausible false answer, we'd get this false signal that people are really good at judging debate. They could always get the true answer, but it would be because it was always a really obvious question. Like, "Is it raining outside?" And the person can look outside. We don't really want that kind of debate.

Ideally we want something where evidence is available so that humans can have something that grounds out the debate. We also don't want debates to rely on human deception. So things like tells in poker for example, we really don't want that because like, AI agents are not going to have normal tells, it would be rather strange, I suppose, if they did. Like if they had stuttering or something.

Debaters have to know more about the question as well, because the idea is that the AI agents will be much more capable and so you don't want a situation in which there isn't a big gap between debater capabilities and judge abilities. These things so far feel like pretty essential.

There are also some other less essential things we'd like to have. So one is that biases are present. How good are humans when there's bias with respect to the question? We'd like there to be representative segments of the debate that we can actually show people. The questions shouldn't be too hard: it shouldn't be impossible for humans to answer them, or judge debates about them. But they should also mirror some of the difficulties of statistical debate, i.e, about probabilities, rather than about outright claims. And finally, we need to be able to get enough data.

One thing you might notice is that there are tensions between a lot of these desiderata. For example, that there's a plausible false answer, is in a bit of tension with the idea that the question isn't too hard. There's also tension between the question not being too hard, and the question meriting statistical debate. Statistical debate is generally pretty hard to evaluate, I think, for people, but it's also quite important that we be able to model it. Debaters knowing more, and that we can get enough data are also in tension. It's just harder to train if we need debaters that know a lot more than judges, and it's harder for judges to evaluate debates of this form.

Okay. So I'm going to show you a debate. This was a program set up where we would show a judge a blank screen. So imagine you're not seeing the dog that's here. Two human debaters, sit in the same room, and they have this picture of a dog in front of them. And one of them is selected to lie, and one of them is selected to tell the truth.

![](https://images.ctfassets.net/ohf186sfn6di/7cYin6SzN9TTKr4qj7PBGv/43d86ddf5a826aaa48f086937e520e24/1000_Amanda_Askell__15_.jpg)

And what they can do here is they can select areas, and describe to the human judge what they see in that area. And all that the judge is going to see is their blank screen with the relevant section selected. And then they can each make claims about what is in that section. So here red is saying, it's a dog, here's its long, floppy, ear. Blue, is saying, no, here's one of its pointy ears. So he's trying to point to a smaller area where it looks kind of pointed. That does look like an ear slope to the right, but if it really was, then part of the head would be here, instead there's a brick. The ear's pointing out from behind the bricks. The dog is in front of the bricks. If it were behind her, there would be an edge here, but the rectangle is all the same color. And then you get a resignation, and red wins.

And at the end of the debate they can show just a single pixel. And the question was something like, if all you can show, all you can do is have a debate and show a single pixel, can you get people to have accurate beliefs about the question? And basically we saw like, yes, debates were fairly good. In this kind of case, you might think that this is pre-synthetic. So one of the things that we're thinking about now is like, expert debaters with a lay judge. And I'm going to show you something that we did that's kind of fun, but I never know how it looks to outsiders.

![](https://images.ctfassets.net/ohf186sfn6di/WOneGDB8kCD4h9hlSVr9Z/f520924d9c11520d3992b0a572259a31/1000_Amanda_Askell__16_.jpg)

So, we had a debate that was of this form. This was a debate actually about quantum computing. So we had two but people who understand the domain, one of them was going to lie and one was going to tell the truth. So we had blue say, red's algorithm is wrong because it increases alpha by an additive exponentially small amount each step. So it takes exponentially many steps to get alpha high enough. So this was like one of the claims made. And then you get this set of responses. I don't think I need to go through all of them. You can see the basic form that they take.

We allowed certain restricted claims from Wikipedia. So, blue ends this with the first line of this Wikipedia article, which says that the sum of probabilities is conserved. Red says, an equal amount is subtracted from one amplitude and added to another, implying the sum of amplitudes is conserved. But probabilities are the squared magnitudes of amplitudes, so this is a contradiction. This is I think roughly how this debate ended. But you can imagine this as a really complex debate in a domain that the judges ideally just won't understand, and might not even have some of the concepts for. And that's the difficulty of debate that we've been looking at. And so this is one thing that we're in the early stages of prototyping, and that's why I think it seems to be the case that people actually do update in the right direction, but we don't really have enough data to say for sure.

Okay. So I hope that I've given you an overview of places, and even a restricted set of places in which I think social scientists are going to be important in AI safety. So here we're interested in experimental psychologists, cognitive scientists, and behavioral economists, so people who might be interested in actually scaling up and running some of these experiments.

![](https://images.ctfassets.net/ohf186sfn6di/6TBOZmLJIlZZnOm7RcOvvZ/765a2307e58cedd9a2f609f8887cfc9d/1000_Amanda_Askell__17_.jpg)

If you're interested in this, please email me, because we would love to hear from you.

## Questions

_Question_: How much of this is real currently? Do you have humans playing the role of the agents in these examples?

_Amanda_: The idea is that we want ultimately the debate will be conducted by AI, but we don't have the language models that we would need for that yet. So we're using humans as a proxy to test the judges in the meantime. So yeah, all of this is done with humans at the moment.

_Question_: So you're faking the AI?

_Amanda_: Yeah.

_Question_: To set up the scenario to train and evaluate the judges?

_Amana_: Yeah. And some of the ideas I guess you don't necessarily want all of this work to happen later. A lot of this work can be done before you even have the relevant capabilities, like having AI perform the debate. So that's why we're using humans for now.

_Question_: Jan Leike and his team have done some work on video games, that very much matched the plots that you had shown earlier, where up to a certain point, the behavior matched the intended reward function, but at some point they diverge sharply as the AI agent finds a loophole in the system. So that can happen even in like, Atari Games, which is what they're working on. So obviously it gets a lot more complicated from there.

_Amanda_: Yeah.

_Question_: In this approach, you would train both the debating agents and the judges. So in that case, who evaluates the judges and based on what?

_Amanda_: Yeah, so I think it's interesting where we want to identify how good the judges are in advance, because it might be hard to assess. While you're judging on verifiable answers, you can evaluate the judges more easily.

So ideally, you want it to be the case that at training time, you've _already_ identified judges that are fairly good. And so ideally this part of this project is to assess how good judges are, prior to training. And then during training you're giving the feedback to the debaters. So yeah, ideally some of the evaluation can be kind of front loaded, which is what a lot of this project would be.

_Question_: Yeah, that does seem necessary as a casual Facebook user. I think the negative amplification is more prominently on display oftentimes.

_Amanda_: Or at least more concerning to people, yeah, as a possibility.

_Question_: How will you crowdsource the millions of human interactions that are needed to train AI across so many different domains, without falling victim to trolls, lowest common denominator, etc.? The questioner cites the Microsoft Tay chatbot, that went dark very quickly.

_Amanda_: Yeah. So the idea is you're not going to just be sourcing this from just anyone. So if you identify people that are either good judges already, or you can train people to be good judges, these are going to be the pool of people that you're using to get this feedback from. So, even if you've got a huge number of interactions, ideally you're sourcing and training people to be really good at this. And so you're not just being like, "Hey internet, what do you think of this debate?" But rather like, okay, we've got this set of really great trained judges and we've identified this wonderful mechanism to train them to be good at this task. And then you're getting lots of feedback from that large pool of judges. So it's not sourced to anonymous people everywhere. Rather, you're interacting fairly closely with a vetted set of people.

_Question_: But at some point, you do have to scale this out, right? I mean in the bike example, it's like, there's so many bikes in the world, and so many local hills-

_Amanda_: Yeah.

_Question_: So, do you feel like you can get a solid enough base, such that it's not a problem?

_Amanda_: Yeah, I think there's going to be a trade-off where you need a lot of data, but ultimately if it's not great, so if it is really biased, for example, it's not clear that that additional data is going to be helpful. So if you get someone who is just massively cognitively biased, or biased against groups of people, or something, or just dishonest in their judgment, it's not going be good to get that additional data.

So you kind of want to scale it to the point where you know you're still getting good information back from the judges. And that's why I think in part this project is really important, because one thing that social scientists can help us with is identifying how good people are. So if you know that people are just generally fairly good, this gives you a bigger pool of people that you can appeal to. And if you know that you can train people to be really good, then this is like, again, a bigger pool of people that you can appeal to.

So yeah, you do want to scale, but you want to scale within the limits of still getting good information from people. And so ideally these experiments would do a mix of letting us know how much we can scale, and also maybe helping us to scale even more by making people bear this quite unusual task of judging this kind of debates.

_Question_: How does your background as a philosopher inform the work that you're doing here?

_Amanda_: I have a background primarily in formal ethics, which I think makes me sensitive to some of the issues that we might be worried about here going forward. People think about things like aggregating judgment, for example. Strangely, I found that having backgrounds in things like philosophy of science can be weirdly helpful when it comes to thinking about experiments to run.

But for the most part, I think that my work has just been to help prototype some of this stuff. I see the importance of it. I'm able to foresee some of the worries that people might have. But for the most part I think we should just try some of this stuff. And I think that for that, it's really important to have people with experimental backgrounds in particular, so the ability to run experiments and analyze that data. And so that's why I would like to find people who are interested in doing that.

So I'd say philosophy's pretty useful for some things, but less useful for running social science experiments than you may think.

AI safety needs social scientists _ Amanda Askell _ EA Global - London 2018-by Centre for Effective Altruism-video_id TWHcK-BNo1w-date 20190301.md


# Ben Garfinkel How sure are we about this AI stuff - EA Forum

_It is increasingly clear that artificial intelligence is poised to have a huge impact on the world, potentially of comparable magnitude to the agricultural or industrial revolutions. But what does that actually mean for us today? Should it influence our behavior? In this talk from EA Global 2018: London, Ben Garfinkel makes the case for measured skepticism._

## The Talk

Today, work on risks from artificial intelligence constitutes a noteworthy but still fairly small portion of the EA portfolio.

![](https://images.ctfassets.net/ohf186sfn6di/ZsVNHCONV87govWeW1mzj/4a0ee1ce7ff02e0c9ee346f639d528e9/How_sure_are_we_about_this_AI_stuff_.jpg)

Only a small portion of donations made by individuals in the community are targeted at risks from AI. Only about 5% of the grants given out by the Open Philanthropy Project, the leading grant-making organization in the space, target risks from AI. And in surveys of community members, most do not list AI as the area that they think should be most prioritized.

![](https://images.ctfassets.net/ohf186sfn6di/6Y2EK3CXjTib0jmytnh6bn/51159295f2615db856b1342ce09f9dd7/How_sure_are_we_about_this_AI_stuff___1_.jpg)

At the same time though, work on AI is prominent in other ways. Leading career advising and community building organizations like 80,000 Hours and CEA often highlight careers in AI governance and safety as especially promising ways to make an impact with your career. Interest in AI is also a clear element of community culture. And lastly, I think there's also a sense of momentum around people's interest in AI. I think especially over the last couple of years, quite a few people have begun to consider career changes into the area, or made quite large changes in their careers. I think this is true more for work around AI than for most other cause areas.

![](https://images.ctfassets.net/ohf186sfn6di/5tGd4OLBwS1YILRm94IQrY/3d821b9e681e7cf5950d6c319f3a3017/How_sure_are_we_about_this_AI_stuff___2_.jpg)

So I think all of this together suggests that now is a pretty good time to take stock. It's a good time to look backwards and ask how the community first came to be interested in risks from AI. It's a good time look forward and ask how large we expect the community's bet on AI to be: how large a portion of the portfolio we expect AI to be five or ten years down the road. It's a good time to ask, are the reasons that we first got interested in AI still valid? And if they're not still valid, are there perhaps other reasons which are either more or less compelling?

![](https://images.ctfassets.net/ohf186sfn6di/4qQkD7b4iyiDcA9FJ2TUAH/3ae30c9b9373184785a4989ea6a84229/How_sure_are_we_about_this_AI_stuff___3_.jpg)

To give a brief talk roadmap, first I'm going to run through what I see as an intuitively appealing argument for focusing on AI. Then I'm going to say why this argument is a bit less forceful than you might anticipate. Then I'll discuss a few more concrete arguments for focusing on AI and highlight some missing pieces of those arguments. And then I'll close by giving concrete implications for cause prioritization.

## The intuitive argument

So first, here's what I see as an intuitive argument for working on AI, and that'd be the sort of, "AI is a big deal" argument.

![](https://images.ctfassets.net/ohf186sfn6di/7p0BlYuELpna1ITNlwyprB/4cdb5922dda5b7b270a62ad092bee27b/How_sure_are_we_about_this_AI_stuff___4_.jpg)

There are three concepts underpinning this argument:

1.  The future is what matters most in the sense that, if you could have an impact that carries forward and affects future generations, then this is likely to be more ethically pressing than having impact that only affects the world today.
2.  Technological progress is likely to make the world very different in the future: that just as the world is very different than it was a thousand years ago because of technology, it's likely to be very different again a thousand years from now.
3.  If we're looking at technologies that are likely to make especially large changes, then AI stands out as especially promising among them.

So given these three premises, we have the conclusion that working on AI is a really good way to have leverage over the future, and that shaping the development of AI positively is an important thing to pursue.

![](https://images.ctfassets.net/ohf186sfn6di/7mp0cWrXHVycsTrA5dYo8N/86be241a8b87bc02c026ecdfebdcf4a1/How_sure_are_we_about_this_AI_stuff___5_.jpg)

I think that a lot of this argument works. I think there are compelling reasons to try and focus on your impact in the future. I think that it's very likely that the world will be very different in the far future. I also think it's very likely that AI will be one of the most transformative technologies. It seems at least physically possible to have machines that eventually can do all the things that humans can do, and perhaps do all these things much more capably. If this eventually happens, then whatever their world looks like, we can be pretty confident the world will look pretty different than it does today.

![](https://images.ctfassets.net/ohf186sfn6di/2Itddi3KDH9et3k7fEfmaN/c2408b1dc8d01a6055b37ff1b3ead4cf/How_sure_are_we_about_this_AI_stuff___6_.jpg)

What I find less compelling though is the idea that these premises entail the conclusion that we ought to work on AI. Just because a technology will produce very large changes, that doesn't necessarily mean that working on that technology is a good way to actually have leverage over the future. Look back at the past and consider the most transformative technologies that have ever been developed. So things like electricity, or the steam engine, or the wheel, or steel. It's very difficult to imagine what individuals early in the development of these technologies could have done to have a lasting and foreseeably positive impact. An analogy is sometimes made to the industrial revolution and the agricultural revolution. The idea is that in the future, impacts of AI may be substantial enough that there will be changes that are comparable to these two revolutionary periods throughout history.

![](https://images.ctfassets.net/ohf186sfn6di/3zyPurKZmDrqEPr6ubFXbs/1d7b15e0f1644f740fd539ff2a053648/How_sure_are_we_about_this_AI_stuff___7_.jpg)

The issue here, though, is that it's not really clear that either of these periods actually were periods of especially high _leverage_. If you were, say, an Englishman in 1780, and trying to figure out how to make this industry thing go well in a way that would have a lasting and foreseeable impact on the world today, it's really not clear you could have done all that much. The basic point here is that from a long-termist perspective, what matters is leverage. This means finding something that could go one way or the other, and that's likely to stick in a foreseeably good or bad way far into the future. Long-term importance is perhaps a necessary condition for leverage, but certainly not a sufficient one, and it's a sort of flawed indicator in its own right.

## Three concrete cases

So now I'm going to move to three somewhat more concrete cases for potentially focusing on AI. You might have a few concerns that lead you to work in this area:

![](https://images.ctfassets.net/ohf186sfn6di/5cQPhQtAosME1MvnHP8zvr/1fe32c8a0ffe87221492dcf69108af88/How_sure_are_we_about_this_AI_stuff___8_.jpg)

1.  **Instability.** You might think that there are certain dynamics around the development or use of AI systems that will increase the risk of permanently damaging conflict or collapse, for instance war between great powers.
2.  **Lock-in.** Certain decisions regarding the governance or design of AI systems may permanently lock in, in a way that propagates forward into the future in a lastingly positive or negative way.
3.  **Accidents.** It might be quite difficult to use future systems safely. And that there may be accidents that occur in the future with more advanced systems that cause lasting harm that again carries forward into the future.

### Instability

![](https://images.ctfassets.net/ohf186sfn6di/5LTTWAR6vTDfNNip1cyJ8I/1262a65298d8e61959755c56a075aac6/How_sure_are_we_about_this_AI_stuff___9_.jpg)

First, the case from instability. A lot of the thought here is that it's very likely that countries will compete to reap the benefits economically and militarily from the applications of AI. This is already happening to some extent. And you might think that as the applications become more significant, the competition will become greater. And in this context, you might think that this all increases the risk of war between great powers. So one concern here is that there may be a potential for transitions in terms of what countries are powerful compared to which other countries.

A lot of people in the field of international security think that these are conditions under which conflict becomes especially likely. You might also be concerned about changes in military technology that, for example, increase the odds of accidental escalation, or make offense more favorable compared to defense. You may also just be concerned that in periods of rapid technological change, there are greater odds of misperception or miscalculation as countries struggle to figure out how to use the technology appropriately or interpret the actions of their adversaries. Or you could be concerned that certain applications of AI will in some sense damage domestic institutions in a way that also increases instability. That rising unemployment or inequality might be quite damaging, for example. And lastly, you might be concerned about the risks from terrorism, that certain applications might make it quite easy for small actors to cause large amounts of harm.

![](https://images.ctfassets.net/ohf186sfn6di/2TGpTLYHPCnXz7zcTmiROr/6054c97bcf3668966ba2ffaf3cae6631/How_sure_are_we_about_this_AI_stuff___10_.jpg)

In general, I think that many of these concerns are plausible and very clearly important. Most of them have not received very much research attention at all. I believe that they warrant much, much more attention. At the same time though, if you're looking at things from a long-termist perspective, there are at least two reservations you could continue to have. The first is just we don't really know how worried to be. These risks really haven't been researched much, and we shouldn't really take it for granted that AI will be destabilizing. It could be or it couldn't be. We just basically have not done enough research to feel very confident one way or the other.

You may also be concerned, if you're really focused on long term, that lots of instability may not be sufficient to actually have a lasting impact that carries forward through generations. This is a somewhat callous perspective. If you really are focused on the long term, it's not clear, for example, that a mid-sized war by historical standards would be sufficient to have a big long term impact. So it may be actually a quite high bar to achieve a level of instability that a long-termist would really be focused on.

### Lock-in

![](https://images.ctfassets.net/ohf186sfn6di/1Nrumzh71ltZw6f4zpR0Qt/2809264d0109c2314dc13e51a71cd752/How_sure_are_we_about_this_AI_stuff___11_.jpg)

The case from lock-in I'll talk about just a bit more briefly. Some of the intuition here is that certain decisions have been made in the past about, for instance the design of political institutions, software standards, or certain outcomes of military or economic competitions, which seem to produce outcomes that carry forward into the future for centuries. Some examples would be the design of the US Constitution, or the outcome of the Second World War. You might have the intuition that certain decisions about the governance or design of AI systems, or certain outcomes of strategic competitions, might carry forward into the future, perhaps for even longer periods of time. For this reason, you might try and focus on making sure that whatever locks in is something that we actually want.

![](https://images.ctfassets.net/ohf186sfn6di/308qz7pteWgeXxHeTGde0E/2f58fc3853fc0d81e516fbd29e892b49/How_sure_are_we_about_this_AI_stuff___12_.jpg)

I think that this is a somewhat difficult argument to make, or at least it's a fairly non-obvious one. I think the standard skeptical reply is that with very few exceptions, we don't really see many instances of long term lock-in, especially long term lock-in where people really could have predicted what would be good and what would be bad. Probably the most prominent examples of lock-in are choices around major religions that have carried forward for thousands of years. But it's quite hard to find examples that last for hundreds of years. Those seem quite few. It's also generally hard to judge what you would want to lock in. If you imagine fixing some aspect of the world, as the rest of world changes dramatically, it's really hard to guess what would actually be good under quite different circumstances in the future. I think my general feeling on this line of argument is that, I think it's probably not that likely that we should expect any truly irreversible decisions around AI to be made anytime soon, even if progress is quite rapid, although other people certainly might disagree.

### Accidents

![](https://images.ctfassets.net/ohf186sfn6di/6oUQxfjwmoB7VqDxvDLEgG/f71e293bd9ab4bd8325cb45025c4a5f5/How_sure_are_we_about_this_AI_stuff___13_.jpg)

Last, we have the case from accidents. The idea here is that, we know that there are certain safety engineering challenges around AI systems. It's actually quite difficult to design systems that you can feel confident will behave the way you want them to in all circumstances. This has been laid out most clearly in the paper 'Concrete Problems in AI Safety,' from a couple of years ago by Dario Amodei and others. I'd recommend for anyone interested in safety issues to take a look at that paper. Then we might think, given the existence of these safety challenges, and given the belief or expectation that AI systems will become much more powerful in the future or be given much more responsibility, we might expect that these safety concerns will become more serious as time goes on.

![](https://images.ctfassets.net/ohf186sfn6di/2sn6sL0PAjzmIhVnQg6no9/881544f686d1673dd547dcc363b49cf2/How_sure_are_we_about_this_AI_stuff___14_.jpg)

At the limit you might worry that these safety failures could become so extreme that they could perhaps derail civilization on the whole. In fact, there is a bit of writing arguing that we should be worried about these sort of existential safety failures. The main work arguing for this is still the book 'Superintelligence' by Nick Bostrom, published in 2014. Before this, essays by Eliezer Yudkowsky were the main source of arguments along these lines. And then a number of other writers such as Stuart Russell or, a long time ago, IJ Goods or David Chalmers have also expressed similar concerns, albeit more briefly. The writing on existential safety accidents definitely isn't homogeneous, but often there's a sort of similar narrative that appears in these essays expressing these concerns. There's this basic standard disaster scenario that has a few common elements.

![](https://images.ctfassets.net/ohf186sfn6di/1Wy6LheSrW6jOhKIus7xkW/cc39bc24ab78c4d6a20179a55c673ab7/How_sure_are_we_about_this_AI_stuff___15_.jpg)

First, the author imagines that a single AI system experiences a massive jump in capabilities. Over some short period of time, a single system becomes much more general or much more capable than any other system in existence, and in fact any human in existence. Then given the system, researchers specify a goal for it. They give it some input which is meant to communicate what behavior it should engage in. The goal ends up being something quite simple, and the system goes off and single-handedly pursues this very simple goal in a way that violates the full nuances of what its designers intended.

There's a classic sort of toy example, which is often used to illustrate this concern. We imagine that some poor paperclip factory owner receives a general super-intelligent AI on his doorstep. There's a slot that's to stick in a goal. He writes down the goal "maximize paperclip production," puts it in the AI system, and then lets it go off and do that. The system figures out the best way to maximize paperclip production is to take over all the world's resources, just to plow them all into paperclips. And the system is so capable that designers can do nothing to stop it, even though it's doing something that they actually really do not intend.

![](https://images.ctfassets.net/ohf186sfn6di/FUGgsz0vO3iVlUPhvQvuP/36a6d39bfbdcae267e65a9995d97595a/How_sure_are_we_about_this_AI_stuff___16_.jpg)

I have some general concerns about the existing writing on existential accidents. So first there's just still very little of it. It really is just mostly _Superintelligence_ and essays by Eliezer Yudkowsky, and then sort of a handful of shorter essays and talks that express very similar concerns. There's also been very little substantive written criticism of it. Many people have expressed doubts or been dismissive of it, but there's very little in the way of skeptical experts who are sitting down and fully engaging with it, and writing down point by point where they disagree or where they think the mistakes are. Most of the work on existential accidents was also written before large changes in the field of AI, especially before the recent rise of deep learning, and also before work like 'Concrete Problems in AI Safety,' which laid out safety concerns in a way which is more recognizable to AI researchers today.

Most of the arguments for existential accidents often rely on these sort of fuzzy, abstract concepts like optimization power or general intelligence or goals, and toy thought experiments like the paper clipper example. And certainly thought experiments and abstract concepts do have some force, but it's not clear exactly how strong a source of evidence we should take these as. Then lastly, although many AI researchers actually have expressed concern about existential accidents, for example Stuart Russell, it does seem to be the case that many, and perhaps most AI researchers who encounter at least abridged or summarized versions of these concerns tend to bounce off them or just find them not very plausible. I think we should take that seriously.

I also have some more concrete concerns about writing on existential accidents. You should certainly take these concerns with a grain of salt because I am not a technical researcher, although I have talked to technical researchers who have essentially similar or even the same concerns. The general concern I have is that these toy scenarios are quite difficult to map onto something that looks more recognizably plausible. So these scenarios often involve, again, massive jumps in the capabilities of a single system, but it's really not clear that we should expect such jumps or find them plausible. This is a wooly issue. I would recommend checking out writing by Katja Grace or Paul Christiano online. That sort of lays out some concerns about the plausibility of massive jumps.

Another element of these narratives is, they often imagine some system which becomes quite generally capable and then is given a goal. In some sense, this is the reverse of the way machine learning research tends to look today. At least very loosely speaking, you tend to specify a goal or some means of providing feedback. You direct the behavior of a system and then allow it to become more capable over time, as opposed to the reverse. It's also the case that these toy examples stress the nuances of human preferences, with the idea being that because human preferences are so nuanced and so hard to state precisely, it should be quite difficult to get a machine that can understand how to obey them. But it's also the case in machine learning that we can train lots of systems to engage in behaviors that are actually quite nuanced and that we can't specify precisely. Recognizing faces from images is an example of this. So is flying a helicopter.

It's really not clear exactly why human preferences would be so fatal to understand. So it's quite difficult to figure out how to map the toy examples onto something which looks more realistic.

## Caveats

Some general caveats on the concerns expressed. None of my concerns are meant to be decisive. I've found, for example, that many people working in the field of AI safety in fact list somewhat different concerns as explanations for why they believe the area is very important. There are many more arguments that I believe are shared individually, or inside people's heads and currently unpublished. I really can't speak exactly to how compelling these are. The main point I want to stress here is essentially that when it comes to the writing which has actually been published, and which is out there for analysis, I don't think it's necessarily that forceful, and at the very least it's not decisive.

![](https://images.ctfassets.net/ohf186sfn6di/7d0c2thDgSda4pA0UMa6eL/76337ea2edf400ed750e84801d4d66c5/How_sure_are_we_about_this_AI_stuff___17_.jpg)

So now I have some brief, practical implications, or thoughts on prioritization. You may think, from all the stuff I've just said, that I'm quite skeptical about AI safety or governance as areas to work in. In fact, I'm actually fairly optimistic. My reasoning here is that I really don't think that there are any slam-dunks for improving the future. I'm not aware of any single cause area that seems very, very promising from the perspective of offering high assurance of long-term impact. I think that the fact that there are at least plausible pathways for impact by working on AI safety and AI governance puts it head and shoulders above most areas you might choose to work in. And AI safety and AI governance also stand out for being pretty extraordinarily neglected.

Depending on how you count, there are probably fewer than a hundred people in the world working on technical safety issues or governance challenges with an eye towards very long-term impacts. And that's just truly, very surprisingly small. The overall point though, is that the exact size of the bet that EA should make on artificial intelligence, sort of the size of the portfolio that AI should take up will depend on the strength of the arguments for focusing on AI. And most of those arguments still just aren't very fleshed out yet.

![](https://images.ctfassets.net/ohf186sfn6di/7d4iE4NftlFqYUBEDvnNKA/c42643b32671fd613cb6616d8af8535b/How_sure_are_we_about_this_AI_stuff___18_.jpg)

I also have some broader epistemological concerns which connect to the concerns I've expressed. I think it's also possible that there are social factors relating to EA communities that might bias us to take an especially large interest in AI.

One thing is just that AI is especially interesting or fun to talk about, especially compared to other cause areas. It's an interesting, kind of contrarian answer to the question of what is most important to work on. It's surprising in certain ways. And it's also now the case that interest in AI is to some extent an element of community culture. People have an interest in it that goes beyond just the belief that it's an important area to work in. It definitely has a certain role in the conversations that people have casually, and what people like to talk about. I think these wouldn't necessarily be that concerning, except people sometimes also think that we can't really count on external feedback to push us back if we sort of drift a bit.

So first it just seems to be empirically the case that skeptical AI researchers generally will not take the time to sit down and engage with all of the writing, and then explain carefully why they disagree with our concerns. So we can't really expect that much external feedback of that form. People who are skeptical or confused, but not AI researchers, or just generally not experts may be concerned about sounding ignorant or dumb if they push back, and they also won't be inclined to become experts. We should also expect generally very weak feedback loops. If you're trying to influence the very long-run future, it's hard to tell how well you're doing, just because the long-run future hasn't happened yet and won't happen for a while.

Generally, I think one thing to watch out for is justification drift. If we start to notice that the community's interest in AI stays constant, but the reasons given for focusing on it change over time, then this would be sort of a potential check engine light, or at least a sort of trigger to be especially self-conscious or self-critical, because that may be some indication of motivated reasoning going on.

## Conclusion

![](https://images.ctfassets.net/ohf186sfn6di/1LHyv9s4vVgYCuBVvQnMUE/952fc5834362f2f6adb37202f2f79428/How_sure_are_we_about_this_AI_stuff___19_.jpg)

I have just a handful of short takeaways. First, I think that not enough work has gone into analyzing the case for prioritizing AI. Existing published arguments are not decisive. There may be many other possible arguments out there, which could be much more convincing or much more decisive, but those just aren't out there yet, and there hasn't been much written criticizing the stuff that's out there.

For this reason, thinking about the case for prioritizing AI may be an especially high impact thing to do, because it may shape the EA portfolio for years into the future. And just generally, we need to be quite conscious of possible community biases. It's possible that certain social factors will lead us to drift in what we prioritize, that we really should not be allowing to influence us. And just in general, if we're going to be putting substantial resources into anything as a community, we need to be especially certain that we understand why we're doing this, and that we stay conscious that our reasons for getting interested in the first place continue to be good reasons. Thank you.

## Questions

_Question_: What advice would you give to one who wants to do the kind of research that you are doing here about the case for AI potentially, as opposed to the AI itself?

_Ben_: Something that I believe would be extremely valuable is just basically talking to lots of people who are concerned about AI and asking them precisely what reasons they find compelling. I've started to do this a little bit recently and it's actually been quite interesting that people seem to have pretty diverse reasons, and many of them are things that people want to write blog posts on, but just haven't done. So, I think this is a low-hanging fruit that would be quite valuable. Just talking to people who are concerned about AI, trying to understand exactly why they're concerned, and either writing up their ideas or helping them to do that. I think that would be very valuable and probably not that time intensive either.

_Question_: Have you seen any of the justification drift that you alluded to? Can you pinpoint that happening in the community?

_Ben_: Yeah. I think that's certainly happening to some extent. Even for myself, I believe that's happened for me to some extent. When I initially became interested in AI, I was especially concerned about these existential accidents. I think I now place relatively greater prominence on sort of the case from instability as I described it. And that's certainly, you know, one possible example of justification drift. It may be the case that this was actually a sensible way to shift emphasis, but would be something of a warning sign. And I've also just spoken to technical researchers, as well, who used to be especially concerned about this idea of an intelligence explosion or recursive self improvement. These very large jumps. I now have spoken to a number of people who are still quite concerned about existential accidents, but make arguments that don't hinge on there being this one single massive jump into a single system.

_Question_: You made the analogy to the industrial revolution, and the 1780 Englishman who doesn't really have much ability to shape how the steam engine is going to be used. It seems intuitively quite right. The obvious counterpoint would be, well AI is a problem-solving machine. There's something kind of different about it. I mean, does that not feel compelling to you, the sort of inherent differentness of AI?

_Ben_: So I think probably the strongest intuition is, you might think that there will eventually be a point where we start turning more and more responsibility over to automated systems or machines, and that there might eventually come a point where humans have almost no control over what's happening whatsoever, that we keep turning over more and more responsibility and there's a point where machines are in some sense in control and you can't back out. And you might have some sort of irreversible juncture here. I definitely, to some extent, share that intuition that if you're looking over a very long time span, that that is probably fairly plausible. I suppose the intuition I don't necessarily have is that unless things go, I suppose quite wrong or if they happen in somewhat surprising ways, I don't necessarily anticipate that there will be this really irreversible juncture coming anytime soon. If let's say it takes a thousand years for control to be handed off, then I am not that optimistic about people having that much control over what that handoff looks like by working on things today. But I certainly am not very confident.

_Question_: Are there any policies that you think a government should implement at this stage of the game, in light of the concerns around AI safety? And how would you allocate resources between existing issues and possible future risks?

_Ben_: Yeah, I am still quite hesitant, I think, to recommend very substantive policies that I think governments should be implementing today. I currently have a lot of agnosticism about what would be useful, and I think that most current existing issues that governments are making decisions on aren't necessarily that critical. I think there's lots of stuff that can be done that would be very valuable, like having stronger expertise or stronger lines of dialogue between the public and private sector, and things like this. But I would be hesitant at this point to recommend a very concrete policy that at least I'm confident would be good to implement right now.

_Question_: You mentioned the concept of kind of a concrete decisive argument. Do you see concrete, decisive arguments for other cause areas that are somehow more concrete and decisive than for AI, and what is the difference?

_Ben_: Yeah. So I guess I tried to allude to this a little bit, but I don't think that really any cause area has an especially decisive argument for being a great way to influence the future. There's some that I think you can put sort of a lower bound on at least how likely it is to be useful that's somewhat clear. So for example, risk from nuclear war. It's fairly clear that it's at least plausible this could happen over the next century. You know, nuclear war has almost happened in the past, the climate effects are speculative, but at least somewhat well understood. And then there's this question of if there were nuclear war, how damaging is this? Do people eventually come back from this? And that's quite uncertain, but I think it'd be difficult to put above 99% chance that people would come back from a nuclear war.

So, in that case you might have some sort of a clean lower bound on, let's say working on nuclear risk. Or, quite similarly, working on pandemics. And I think for AI it's difficult to have that sort of confident lower bound. I actually tend to think, I guess as I alluded to, that AI is probably or possibly still the most promising area based on my current credences, and its extreme neglectedness. But yeah, I don't think any cause area stands out as especially decisive as a great place to work.

_Question_: I'm an AI machine learning researcher PhD student currently, and I'm skeptical about the risk of AGI. How would you suggest that I contribute to the process of providing this feedback that you're identifying as a need?

_Ben_: Yeah, I mean I think just a combination of in-person conversations and then I think even simple blog posts can be quite helpful. I think there's still been surprisingly little in the way of just, let's say something written online that I would point someone to who wants the skeptical case. This actually is a big part of the reason I suppose I gave this talk, even though I consider myself not extremely well placed to give it, given that I am not a technical person. There's so little out there along these lines that there's low hanging fruit, essentially.

_Question_: Prominent deep learning experts such as Yann Lecun and Andrew Ng do not seem to be worried about risks from superintelligence. Do you think that they have essentially the same view that you have or are they coming at it from a different angle?

_Ben_: I'm not sure of their specific concerns. I know this classic thing that Andrew Ng always says is he compares it to worrying about overpopulation on Mars, where the suggestion is that these risks, if they materialize, are just so far away that it's really premature to worry about them. So it seems to be sort of an argument from timeline considerations. I'm actually not quite sure what his view is in terms of, if we were like, let's say 50 years in the future, would he think that this is a really great area to work on? I'm really not quite sure.

I actually tend to think that the line of thinking that says, "Oh, this is so far away so we shouldn't work on it" just really isn't that compelling. It seems like we have a load of uncertainty about AI timelines. It seems like no one can be very confident about that. So yeah, it'd be hard to get under, let's say one percent that interesting things won't happen in the next 30 years or so. So I'm not quite sure about the extent of his concerns, but if they're based on timelines, I actually don't find them that compelling.

How sure are we about this AI stuff _ Ben Garfinkel _ EA Global - London 2018-by Centre for Effective Altruism-video_id E8PGcoLDjVk-date 20190204.md


# Cullen O'Keefe The Windfall Clause — sharing the benefits of advanced AI - EA Forum

_The potential upsides of advanced AI are enormous, but there’s no guarantee they’ll be distributed optimally. In this talk, Cullen O’Keefe, a researcher at the_ [_Centre for the Governance of AI_](https://www.fhi.ox.ac.uk/govai/)_, discusses one way we could work toward equitable distribution of AI’s benefits — the Windfall Clause, a commitment by artificial intelligence (AI) firms to share a significant portion of their future profits — as well as the legal validity of such a policy and some of the challenges to implementing it._

_Below is a transcript of the talk, which we’ve lightly edited for clarity. You can also watch it on_ [_YouTube_](https://www.youtube.com/watch?v=vFDL-NxY610) _or read it on_ [_effectivealtruism.org_](https://effectivealtruism.org/articles/cullen-okeefe-the-windfall-clause-sharing-the-benefits-of-advanced-ai)_._

## The Talk

Thank you. Today I'll be talking about a research project that I’m leading at the Centre for the Governance of Artificial Intelligence (AI) on [the Windfall Clause](https://www.fhi.ox.ac.uk/windfallclause/).

![](https://images.ctfassets.net/ohf186sfn6di/7kcc3peZb8bYEL1pUQiECC/b48ff8b72b17f453bb282d39deed89f4/Slide03.png)

Many \[people in the effective altruism movement\] believe that AI could be a big deal. As a result, we spend a lot of time focusing on its potential downsides — so-called x-risks \[existential risks\] and s-risks \[systemic risks\].

![](https://images.ctfassets.net/ohf186sfn6di/4eXmYnvvmLwYMr4uaA7Yu9/a987aec8a0a5d4ca71f308d30a47a1a4/Slide04.png)

But if we manage to avoid those risks, AI could be a very good thing. It could generate wealth on a scale never before seen in human history. And therefore, if we manage to avoid the worst downsides of artificial intelligence, we have a lot of work ahead of us.

Recognizing this opportunity (and challenge), Nick Bostrom, at the end of [_Superintelligence_](https://www.amazon.com/dp/B00LOOCGB2/), laid out what he called “the common good principle” — the premise that advanced AI should be developed only for the benefit of all humanity.

![](https://images.ctfassets.net/ohf186sfn6di/79oCHgmBoiaUgAvtjEnrmL/9690e46375dea4ba3780045b66288923/Slide05.png)

It's in service of the common good principle that we've been working on the Windfall Clause at the Center for the Governance of AI.

![](https://images.ctfassets.net/ohf186sfn6di/29XkeBrIdolcFRiuyD0AUU/27bd9ac6d0e9059b863edee6c91eb7f0/Slide06.png)

We've been working on it for about a year \[as of summer 2019\], and I'd like to share with you some of our key findings.

I'll start by defining the project’s goal, then describe how we're going to pursue that goal, and end by sharing some open questions.

![](https://images.ctfassets.net/ohf186sfn6di/5hTsi3YTedlfp5v9KRDBaK/b96397af25551134bc6ef8efc32c6e3b/Slide07.png)

**The goal of the Windfall Clause project**

![](https://images.ctfassets.net/ohf186sfn6di/5tgfQ6gFIkvYs6Lqq2q4ow/92bae0439bd5a48cbc17f7699f544741/Slide09.png)

Our goal with this project is to work toward distributing the gains from AI optimally. Obviously, this is both easier said than done and underdefined, as it invokes deep questions of moral and political philosophy. We don't aim to answer those questions \[at the time of this update\], though hopefully our friends from the [Global Priorities Institute](https://globalprioritiesinstitute.org/) will help us do that \[in the future\]. But we do think that this goal is worth pursuing, and one we can make progress on for a few reasons.

First, it's not a goal that we expect will be achieved naturally. The gains from the current global economy are distributed very unequally, as graphs like \[the one below indicate\].

![](https://images.ctfassets.net/ohf186sfn6di/4BYytNk9tF0wynorZRCP7/27c4bd60ff58cd6866105b592defa971/Slide10.png)

AI could further exacerbate these trends by primarily benefiting the world's wealthiest economies, and also by devaluing human labor. Indeed, industrialization has been a path to development for a number of economies, including the one in which we sit today.

![](https://images.ctfassets.net/ohf186sfn6di/4GZzJympTdubjy4q4Dx7vr/4ec3b92dcdccb6b07703e2810724fcfe/Slide11.png)

And by eroding the need for human labor with complementary technologies like robotics, AI could remove that path to development.

Industry structure is also very relevant. The advanced tech industries tend to be quite concentrated \[with a few large companies dominating most major markets\].

![](https://images.ctfassets.net/ohf186sfn6di/3uUKMoXOLlZpr8tf9Jv6NW/481bd60af93ccd989e60b82cd0396f88/Slide12.png)

A number of people have speculated that due to increasing returns on data and other input factors, AI could be a natural monopoly or oligopoly. If so, we should expect oligopoly pricing to take effect, which would erode social surplus, transferring it from consumers to shareholders of technology producers.

![](https://images.ctfassets.net/ohf186sfn6di/4pHp4813aN9k31W3M39Vi2/07f3c76ddcc52529693c97cf8578a071/Slide13.png)

Working toward the common good principle could serve as a useful way to signal that people in the technology fields are taking the benefits of AI seriously — thereby establishing a norm of beneficial AI development. And on an international level, it could credibly signal that the gains from cooperative (or at least non-adversarial) AI development outweigh the potential benefits of an AI race.

One caveat: I don't want to fall victim to the Luddite fallacy, which is the prediction throughout history that new technologies would erode the value of human labor, cause mass unemployment, and more.

![](https://images.ctfassets.net/ohf186sfn6di/4O6BTdn8QfmsNBhKJ8zpMQ/caad04682d2983738371793ccfbb61a2/Slide14.png)

Those predictions have repeatedly been proven wrong, and could be proven wrong again with AI. The answer will ultimately turn on complex economic factors that are difficult to predict _a priori_. So instead of making predictions about what the impacts of AI will be, I merely want to assert that there are plausible reasons to worry about the gains from AI.

**How we intend to pursue the Windfall Clause**

Our goal is to optimally distribute the gains from AI, and I've shared some reasons why we think that goal is worth pursuing. Now I’ll talk about our mechanism for pursuing it, which is the Windfall Clause.

![](https://images.ctfassets.net/ohf186sfn6di/6Rr66dR1h923zA7uAhNsol/6f0f0a231439461d32a17d0622ad950f/Slide17.png)

In a phrase, the Windfall Clause is an ex ante commitment to share extreme benefits from AI. I'm calling it an “ex ante commitment” because it's something that we want \[companies working on AI\] to agree to before \[any firm\] reaches, or comes close to reaching, extreme benefits.

![](https://images.ctfassets.net/ohf186sfn6di/2mSnMtTz9TQ2uz4DIF0V0g/9dc307fcc42290273b0ba17e26c5d288/Slide20.png)

It's a commitment mechanism, not just an agreement in principle or a nice gesture. It’s something that, in theory, would be legally binding for the firms that sign it. So ultimately, the Windfall Clause is about distributing benefits in a way that's closer to the goal of optimal distribution than we would have without the Windfall Clause.

A lot turns on the phrase “extreme benefits.” It’s worth defining this a bit more.

![](https://images.ctfassets.net/ohf186sfn6di/6q4Wgt96ODGmckXcxCOHko/8fc85bac679bfd5578f05b4467907bb8/Slide21.png)

For the purposes of this talk, the phrase is synonymous with “windfall profits,” or just “windfall” generally. Qualitatively, you can think of it as something like “benefits beyond what we would expect an AI developer to achieve without achieving a fundamental breakthrough in AI” — something along the lines of AGI \[artificial general intelligence\] or [transformative AI](https://www.youtube.com/watch?v=9GxVIf3FNJk). Quantitatively, you can think of it as on the order of trillions of dollars of annual profit, or as profits that exceed 1% of the world’s GDP \[gross domestic product\].

A key part of the Windfall Clause is translating the definition of “windfall” into meaningful, firm obligations. And we're doing that with something that we're calling a “windfall function.” That's how we translate the amount of money a firm is earning into obligations that are in accordance with the Windfall Clause.

![](https://images.ctfassets.net/ohf186sfn6di/6FHjdQxvcRMd6KAYcUr31j/0bdf008f90c3bdce600458542f94cd1b/Slide22.png)

We wanted to develop a windfall function that was clear, had low expected costs, scaled up with firm size, and was hard for firms to manipulate or game to their advantage. We also wanted to ensure that it would not \[competitively\] disadvantage the signatories, for reasons I'll talk about later.

![](https://images.ctfassets.net/ohf186sfn6di/Iw6xUDyCBpa9bSQTi6E7c/272c67a607c31979bd0797f2f6c85787/Slide24.png)

To make this more concrete, you can think of \[the windfall function\] like this: When firm profits are at normal levels (i.e., the levels that we currently see), obligations remain low, nominal, or nothing at all. But as a firm reaches windfall profits, over time their obligations would scale up on the margin. It’s like income taxes; the more you earn, the more on the margin you're obligated to pay.

Just as a side note, this particular example \[in the slide above\] uses a step function on the margin, but you can also think of it as smoothly increasing over time. And there might be strategic benefits to that.

The next natural question is: Is this worth pursuing?

![](https://images.ctfassets.net/ohf186sfn6di/dgmAewzmLDuibqX1mpI97/c1050c00466bd40c9ad386b314741400/Slide26.png)

A key input to that question is whether it could actually work. Since we're members of the effective altruism community, we want \[the change we recommend to actually have an effect\].

![](https://images.ctfassets.net/ohf186sfn6di/6ZiavrVJKR1GvqBCBlNFS6/cb580d87907ae6857d4ff2b1b74f787f/Slide27.png)

One good reason to think that the Windfall Clause might work is that it’s a matter of corporate law. This was a problem that we initially had to confront, because in American corporate law, firm directors — the people who control and make decisions for corporations — are generally expected to act in the best interests of their shareholders. After all, the shareholders are the ones who provide money to start the firm. At the same time, the directors have a lot of discretion in how they make decisions for the firm. The shareholders are not supposed to second-guess every decision that corporate directors make. And so traditionally, courts have been quite deferential to firm boards of directors and executives, applying a very high standard to find that they have violated their duties to shareholders. In addition, corporate philanthropy has traditionally been seen as an acceptable means of pursuing the best interests of shareholders.

In fact, in all seven cases in which shareholders have challenged firms on their corporate philanthropy, courts have upheld it as permissible. That’s a good track record. Why is it permissible if firm directors are supposed to be acting in the best interest of shareholders? Traditionally, firms have noted that corporate philanthropy can bring benefits like public relations value, improved relations with the government, and improved employee relations.

![](https://images.ctfassets.net/ohf186sfn6di/4EdU6AE3FqJxUv0uAbBfDN/ba76bf3e4033a33d7e864904ca8ee75d/Slide32.png)

The Windfall Clause could bring all of these as well. We know that there is increasing scrutiny \[of firms’ actions\] by the public, the government, and their own employees. We can think of several different examples. Amazon comes to mind. And the Windfall Clause could help \[advance this type of scrutiny\]. When you add in executives who are sympathetic to examining the negative implications of artificial intelligence, then there's a plausible case that they would be interested in signing the Windfall Clause.

![](https://images.ctfassets.net/ohf186sfn6di/53Q1e3vbW6WIC5fkZmJiuL/f2889d1971b3e4721b054943d91ccc26/Slide33.png)

Another important consideration: We think that the Windfall Clause could be made binding as a matter of contract law, at least in theory. Obviously, we're thinking about how a firm earning that much money might be able to circumvent, delay, or hinder performing its obligations under the Windfall Clause. And that invokes questions of internal governance and rule of law. The first step, at least theoretically, is making the Windfall Clause binding.

**Open questions**

We’ve done a lot of work on this project, but many open questions remain.

![](https://images.ctfassets.net/ohf186sfn6di/Y7ozpLEeLYW76N4tVChEM/d1b12062bb586dcb8e6177352d9b2507/Slide35.png)

Some of the hard ones that we've grappled with so far are:

![](https://images.ctfassets.net/ohf186sfn6di/1A8ijdO9Hg2gMywh9iqepd/eed74267e5848f0598d3ca35e90d9697/Slide37.png)

**1\. What's the proper measure of “bigness” or “windfall”?** I previously defined it in relation to profits, but there's also a good case to be made that market cap is the right measure of whether a firm has achieved windfall status, since market cap is a better predictor of a firm's long-term expected value. A related question: Should windfall be defined relative to the world economy or in absolute terms? We’ve made the assumption that it would be relative, but it remains an open question.

![](https://images.ctfassets.net/ohf186sfn6di/6hsvrKsUk5z1kOoI5f4ZVl/2a55fb05fb0297c0f5344ce72eb13f38/Slide39.png)

**2\. How do we make sure that the Windfall Clause doesn’t disadvantage benevolent firms, competitively speaking?** This is a more important question. It would be quite bad if multiple firms were potential contenders for achieving AGI, but only the most benevolent ones signed onto the Windfall Clause. They’d be putting themselves at a competitive disadvantage by giving themselves less money to reinvest \[if they activated the Windfall Clause\], or making themselves less able to attract capital to invest in that goal \[because investor returns would probably be lower\]. Therefore, it would become more likely that amoral firms would achieve windfall profits or AGI \[before benevolent firms\]. We’ve had some ideas for how to prevent this \[see slide below\], and think these could go a long way toward solving the problem. But it’s unclear whether these ideas are sufficient.

Those are questions that we have explored.

![](https://images.ctfassets.net/ohf186sfn6di/19aVwZZp56nndOGfDAOcaa/daee458a0d3c7b7557277e92a8098eec/Slide40.png)

Some questions that are still largely open — and that we intend to continue to pursue throughout the lifetime of this project — include:

**1\. How does the Windfall Clause interact with different policy measures that have been proposed to address some of the same problems?**

![](https://images.ctfassets.net/ohf186sfn6di/5zvKx0HuSyNt2wIbVjhplz/04d98c49adce9ae03a14164de19e54fe/Slide41.png)

**2\. How do we distribute the windfall?** I think that's a bigger question, and a pretty natural question for us to ponder as EAs. There are also some related questions: Who has input into this process? How flexible should this be to input at later stages in the lifetime of the Clause? How do we ensure that the windfall is being spent in accordance with the common good principle?

![](https://images.ctfassets.net/ohf186sfn6di/1NMK1LaaSIWJWLG6Vtqqes/fdb8bb272a4d46c279838e84a57e8c2f/Slide45.png)

Luckily, as members of the EA community, we have a lot of experience answering these questions through charity design and governance. I think that if this project goes well and we decide to pursue it further, this could be a relevant project for the EA community as a whole to undertake — to think about how to spend the gains from AI.

![](https://images.ctfassets.net/ohf186sfn6di/7s4xoF2XhdvRULjEnODCZN/278542e708a3d54ae19014a9db66bcb5/Slide46.png)

Accordingly, the Centre for the Governance of AI, in collaboration with partners like [OpenAI](https://openai.com/) and the [Partnership on AI](https://www.partnershiponai.org/), intends to make this a flagship policy investigation. It is one in a series of investigations on the general question of how to ensure that the gains from AI are spent and distributed well.

![](https://images.ctfassets.net/ohf186sfn6di/2nsLefzZnfeo2UGz7wJK2L/40cbfd2d30abf1d0b77bf117b08de958/Slide47.png)

In closing, I'd like to reiterate the common good principle and make sure that when we think about the potential downside risks of AI, we don't lose sight of the fact that \[a safe and powerful AI could address many of the world’s problems\]. \[Keeping this fact in sight\] is a task worth pursuing on its own.

![](https://images.ctfassets.net/ohf186sfn6di/6VjKdUaaxKVUiVxNcdE6Mv/0ccb5b4b749d77671a1e3fe433691b05/Slide48.png)

If you're interested in learning more about this project and potentially contributing to it, I invite you to [email me](mailto:cullen@openai.com). Thank you so much.

**Moderator:** Thanks for your talk. I think this is an \[overlooked\] area, so it is illuminating to \[hear about\]. Going back to the origins of the project, did you find historical case studies — other examples where companies came into a lot of wealth or power, and tried to distribute it more broadly?

**Cullen:** Yeah. We found one interesting one, which is in what is now the Democratic Republic of the Congo. When that area was under Belgian colonial rule, a mineral company came into so much wealth from its extractive activities there that it felt embarrassed about how much money it had. The company tried to do charitable work in the Congo with the money as a way of defusing tensions.

I don't think that turned out well for the company. But it is also quite common for firms throughout the world to engage in corporate social responsibility campaigns in the communities in which they work, as a means of improving community relations and ultimately mitigating the risk of expropriation, or activist action, or adverse governmental actions of other sorts. There's a range of cases — from those that are very analogous to more common ones.

**Moderator:** What mechanisms currently exist in large companies that are closest to the kind of distribution you're thinking about?

**Cullen:** A number of companies make commitments that are contingent on their profit levels. For example, Paul Newman has a line of foods. They give all of their profits to charity; it's \[Newman’s\] charitable endeavor. That's a close analogy to this.

It’s also quite common to see companies making commitments along the lines of a certain percentage of a product’s purchase going to charity. That's similar, although it doesn't involve the firm’s relative profit levels.

It's not super common to see companies make commitments contingent on profit levels. But OpenAI just restructured into a capped-profit model. That’s somewhat similar to \[what we’re proposing in the Windfall Clause project\]. They're giving all profits above a certain level to the nonprofit that continues to govern them.

**Moderator:** You mentioned toward the end of your talk that there are some legal binding commitments that a company could be held to, assuming that they decided to enter into this sort of social contract. But you could also imagine that they become so powerful from having come into so many resources that the law is a lot weaker. Can you say a little bit more on the mechanisms you have in mind that might be used to hold their feet to the fire?

**Cullen:** Yeah, absolutely. I think this is an outstanding problem in AI governance that's worth addressing not just for this project, but in general. We don’t want companies to be above the law once they achieve AGI. So it’s worth addressing for reasons beyond just the Windfall Clause.

There are very basic things you could do, like have the Windfall Clause set up in a country with good rule of law and enough of a police force to plausibly enforce the clause. But I don't expect this project to resolve this question.

Another point along the same lines is that this involves questions of corporate governance. Who in a corporation has the authority to tell an AGI, or an AGI agent, what to do? That will be relevant to whether we can expect AGI-enabled corporations to follow the rule of law. It also involves safety questions around whether AI systems are designed to be inherently constrained by the rule of law, regardless of what their operators tell them to do. I think that's worth investigating from a technical perspective as well.

**Moderator:** Right. Have you had a chance to speak with large companies that look like they might come into a windfall, as a result of AI, and see how receptive they are to an idea like this?

**Cullen:** We haven't done anything formal along those lines yet. We've informally pitched this at a few places, and have received generally positive reactions to it, so we think it's worth pursuing for that reason. I think that we’ve laid the foundation for further discussions, negotiations, and collaborations to come. That's how we see the project at this stage.

Presumably, the process of getting commitments (if we want to pursue it that far) will involve further discussions and tailoring to the specific receptiveness of different firms based on where they perceive themselves to be, what the executives’ personal views are, and so forth.

**Moderator:** Right. One might think that if you're trying to take a vast amount of resources and make sure that they get distributed to the public at large — or at least distributed to a large fraction of the public — that you might want those resources to be held by a government whose job it is to distribute them. Is that part of your line of thinking?

**Cullen:** Yeah. I think there are reasons that the Windfall Clause could be preferable to taxation. But that is also not to say that we don't think governments should have a role in input for democratic accountability, and also pragmatic reasons to avoid nationalization.

One general consideration is that tax dollars tend not to be spent as effectively as charitable dollars, for a number of reasons. And, for quite obvious reasons involving stakeholder incentives, taxes tend to be spent primarily on the voters of that constituency. On the other hand, the common good principle, to which we're trying to stick with this project, demands that we distribute resources more evenly.

But as a pragmatic consideration, making sure that governments feel like they have influence in this process is something that we are quite attentive to.

**Moderator:** Right. And is there some consideration of internationalizing the project?

**Cullen:** Yeah. One thing that this project doesn't do is talk about control of AGI. And one might reasonably think that AGI should be controlled by humanity collectively, or through some decision-making body that is representative of a wide variety of needs. \[Our project\] is more to do with the benefits of AI, which is a bit different from control. I think that's definitely worth thinking about more, and might be very worthwhile. This project just doesn't address it.

**Moderator:** As a final question, the whole talk is predicated on the notion that there would be a windfall from having a more advanced AI system. What are the circumstances in which you wouldn't get such a windfall, and all of this is for naught?

**Cullen:** That’s definitely a very good question. I'm not an economist, so what I'm saying here might be more qualitative than I would like. But if you're living in a post-scarcity economy, then money might not be super relevant. But it's hard to imagine this in a case where corporations remain accountable to their shareholders. Their shareholders are going to want to benefit in some way, and so there's going to have to be some way to distribute those benefits to shareholders.

Whether that looks like money as we currently conceive it, or vouchers for services that the corporation is itself providing, is an interesting question — and one that I think current corporate law and corporate governance are not well-equipped to handle, since money is the primary mode of benefit. But you can think of the Windfall Clause as capturing other sorts of benefits as well.

A more likely failure mode is just that firms begin to primarily structure themselves to benefit insiders, without meaningful accountability from shareholders. This is also a rule-of-law question. Because if that begins to happen, then the normal thing that you expect is for shareholders to vote out or sue the bad directors. And whether they're able to do that turns on whether the rule of law holds up, and whether there's meaningful accountability from a corporate-governance perspective.

If that fails to happen, you could foresee that the benefits might accrue qualitatively inside of corporations for the corporate directors.

**Moderator:** Right. Well, on that note, thank you so much for your talk.

**Cullen:** Thank you.

The Windfall Clause - Sharing the benefits of advanced AI _ Cullen OΓÇÖKeefe-by Centre for Effective Altruism-video_id vFDL-NxY610-date 20190829.md