# Rohin Shah on Effective altruism, AI safety, and learning human preferences from the world’s state by Jeremie Harris on the Towards Data Science Podcast

If you walked into a room filled with objects that were scattered around somewhat randomly, how important or expensive would you assume those objects were?

What if you walked into the same room, and instead found those objects carefully arranged in a very specific configuration that was unlikely to happen by chance?

These two scenarios hint at something important: human beings have shaped our environments in ways that reflect what we value. You might just learn more about what I value by taking a 10 minute stroll through my apartment than by spending 30 minutes talking to me as I try to put my life philosophy into words.

And that’s a pretty important idea, because as it turns out, one of the most important challenges in advanced AI today is finding ways to communicate our values to machines. If our environments implicitly encode part of our value system, then we might be able to teach machines to observe it, and learn about our preferences without our having to express them explicitly.

The idea of leveraging deriving human values from the state of an human-inhabited environment was first developed in a paper co-authored by Berkeley PhD and incoming DeepMind researcher Rohin Shah. Rohin has spent the last several years working on AI safety, and publishes the widely read AI alignment newsletter — and he was kind enough to join us for this episode of the Towards Data Science podcast, where we discussed his approach to AI safety, and his thoughts on risk mitigation strategies for advanced AI systems.

Here were some of my favourite take-homes from our conversation:

-   Like many, Rohin was driven towards AI alignment and AI safety work in part through his exposure to members of the Effective Altruism community. Effective Altruism is a philosophical movement focused on determining how people can make the maximum positive impact on the world, either through charitable giving or their professional career. It’s focused on asking questions like: what is the move I could make, that would maximize the expected value of my contribution to the world? For Rohin, the possible harm that could be done by powerful AI systems in the future — and the possible benefits that these systems could have if they’re developed safely — made AI alignment appealing.
-   Rohin discussed two important potential failure modes of advanced AI systems, which already appear in different forms in current systems.
-   First, he highlighted the risk of bad generalization: AIs learning the wrong lessons from their training data, leading them to generalize in ways that humans might not anticipate or want. As an example, he cited an initial concern with OpenAI’s GPT-3 model, which surfaced when a developer prompted the model with a nonsense question (like, “how many glubuxes are in a woolrop?”). Rather than answering this query “honestly”, by saying something like, “I don’t know — I’m not familiar with those words,” GPT-3 tried to answer with a best guess, like “there are 3 glubuxes in a woolrop.” You could argue that this actually wasn’t a bad generalization: GPT-3 basically answered like a student taking a test, who wants to hide the fact that they don’t know the answer to a question by taking a guess based purely on the context of the question. But if our hope was to make an honest language model — one that admits its ignorance when appropriate — GPT-3 in its non-fine-tuned form seemed to fail the test. As AI systems get more powerful, this kind of behavior could get more harmful, so Rohin thinks it’s worth paying attention to.
-   Second, Rohin discussed the challenge of communicating human preferences to AIs. This is a hard problem: most humans don’t actually know what they want out of life, and are even less capable of communicating those desires and values to other humans — let alone to machines, which are currently less context-aware and flexible in their reasoning. That’s where Rohin’s work on teaching machines to infer human preferences from their environment comes in: he thinks the strategy shows promise as an additional source of data on human preferences that machines might be able to use to decipher human values without needing us to be able to articulate them explicitly. We talked about many interesting strengths and weaknesses of this strategy.

## Chapters:

-   0:00 Intro
-   1:44 Effective altruism
-   6:50 Rohin’s introduction to AI safety work
-   11:18 Why AI risk is so serious
-   18:33 Child rearing analogy
-   22:15 Statistical learning theory
-   25:09 What is preference learning?
-   32:23 Application to higher levels of abstraction
-   34:45 Revealed preferences and the state of the world
-   36:26 Broken vase metaphor
-   44:10 Rule of time horizons
-   49:03 Wrap-up

## Please find below the transcript for Season 2 Episode 4:

Jeremie (00:00):
Hey, everyone. Welcome to another episode of the Towards Data Science podcast. My name is Jeremie and, apart from hosting the podcast, I’m also on the team at the SharpestMinds data science mentorship program. I’m really excited about today’s episode because I’ve been thinking about getting today’s guest on the podcast for a very long time. I’m so glad we finally made it happen.

Jeremie (00:16):
He’s actually in between a transition right now from Berkley where he’s wrapping up his PhD. He’s working at the Center for Human Compatible AI, and he’s transitioning over into DeepMind where he’ll be doing some alignment work. His name is Rohin Shah. And, apart from being a very prolific researcher in AI and AI alignment in particular, he is also the publisher of the AI alignment newsletter which is a really, really great resource if you’re looking to get oriented in the space to learn about some of the open problems and open questions in AI alignment. I really recommend checking it out.

Jeremie (00:47):
We’re going to talk about a whole bunch of different things, including the philosophy of AI, the philosophy of machine learning and AI alignment, ways in which it can be accomplished, some of the challenges that exist, and we’re going to explore one of the most interesting proposals, I think, that Rohin’s come up with, which is an idea about extracting human preferences from the state of the environment. So, basically, the idea here is that humans, through their activity, have encoded their preferences implicitly in their environments, we do a whole bunch of different things, different actions, that reveal our preferences. And it would be great if we could have AIs look at the world and figure out what our preferences are implicitly based on the state of that world. And that might be a great way to bootstrap AI alignment efforts.

Jeremie (01:30):
We’ll be talking about that proposal in depth, along with a whole bunch of other things. I’m really looking forward to the episode, so I hope you enjoy. Without further ado, here we go.

Jeremie (01:39):
Hello, and thank you so much for joining us for the podcast.

Rohin (01:41):
Yeah, thanks for having me here. I’m excited.

Jeremie (01:44):
Well, I’m very excited to have you. There’s so many interesting things that you’ve been working on in the alignment space in general. But, before we tackle some of the more technical questions, there’s an observation that I think anybody who’s spent any time working on alignment or talking to alignment researchers is going to end up making at some point, which is that the vast majority of people in the space seem to come from the effective altruism community. And I’d love to get your take on, number one, what the effective altruism community is, what effective altruism is, and number two, why you think there’s this deep connection between EA, effective altruism, and AI alignment and AI safety research.

Rohin (02:20):
Yeah, sure. The overarching idea effective altruism, the very easy to defend one that doesn’t get into specifics, is with whatever money, time, resources, whatever you’re willing to spend altruistically, you should try to do the most good you can with it, rather than… And you should think about that. It’s pretty hard to argue against this. I don’t think I’ve really seen people disagree with that part.

Rohin (02:56):
Now, in practice, effective altruism the movement has whole bunch of additional premises that are meant to be in support of the skill, but are more controversial. I think the really fundamental big idea of effective altruism is that of cause prioritization. Many people will say, “Okay, I want to have, say, clean water in Africa. I will work towards that.” And they’ll think about different ways in which you can get clean water in Africa, maybe you could try sanitizing the water that people already get, or you could try building some new treatment plants in order to provide fresh, flowing water that’s drinkable to everyone. And they’ll think about how best can they achieve their goal of delivering clean water.

Rohin (03:47):
It’s much, much, much less common for people to think, “Okay, well, should I work on getting clean water to people in Africa, or combating racism in the US? Which of these should I put my effort into? Or my money into?” The main premise of effective altruism is you can, in fact, do this. There are actually significant differences between causes and, by thinking about it, have much more impact by selecting the right cause to work on in the first place.

Rohin (04:20):
It’s very focused on this take weird… Take ideas seriously, actually evaluate them, figure out whether or not they are true and not whether or not they sound crazy. Whether or not they sound crazy does have some relation to whether or not they are true, but they are not necessarily the same. I think that ties into why it’s also such a hotbed for AI safety research. The EA case for AI safety, for work on AI safety, is that AI has a good chance of being extremely influential in the next century, let’s say.

Rohin (05:00):
There is some argument that is debatable, but it doesn’t seem like you can rule it out. It seems like at least moderately likely that if we don’t take care of how we do this, the AI system might “take over” in the sense of all of the important decisions about the world are made by the AI system and not by humans. And one possible consequence of this is humans go extinct. I’ll go into this argument later, I’m sure, but-

Jeremie (05:36):
So \[crosstalk 00:05:37\]-

Rohin (05:37):
\[crosstalk 00:05:37\] believe this argument somewhat, and so then it becomes extremely important and impactful to work on. It sounds crazy, but one of the EA’s, to me, strengths is that it separates what sounds crazy from what is true.

Jeremie (05:52):
It seems like, really, the focus is there’s this extra missing step that a lot of people don’t apply to their thinking when deciding what causes to contribute to, what to work on, what to spend their lives on, and that is the step of going, “What areas are going to give outsized returns on my time?”

Rohin (06:09):
Yep, exactly.

Jeremie (06:12):
I can really think back to most of the conversations I ever had with people about causes, about charity, and it’s usually focused on stuff like what are the administrative fees associated with this charity? Oh, I want to donate to a place where all my dollars go to the cause, rather than asking the more fundamental question, is this cause actually going to give the best ROI from the standpoint of benefiting everyone or benefiting humanity? It’s interesting that that kind of thinking, a more first principles approach, leads a lot of people to the area of AI alignment and AI safety. As you said, it makes sense, you’ve got this super high risk high reward profile.

Jeremie (06:50):
What was it that drew you, for example, to AI alignment, AI safety work in particular, rather than any of the other, I could imagine, bio terrorism, I could imagine all kinds of horrible things that could happen to us, but why AI alignment in particular?

Rohin (07:05):
Yeah, so, my story is kind of weird. It may be a classic AI story in that convinced by weird very weird arguments. I got into effective altruism in 2014. I heard the arguments for AI risk within a year of that, probably. I was deeply unconvinced by them. I just did not buy them.

Rohin (07:37):
And, so, until 2017, I basically didn’t engage very much with AI safety. I was also unconvinced of, basically, there’s this field of ethics called population ethics which tries to deal with the question of how do you compare how good different worlds are when they have different populations of people in them? We don’t need to go into the details, but it’s a very confusing area. Lots of impossibility results that say you might want these six very intuitive properties, but, no, you can’t actually have all of them at the same time, stuff like this. So you’re \[crosstalk 00:08:21\]-

Jeremie (08:20):
Would the idea here be like is a world with 100 decently happy people better than a world with 1,000 decent minus epsilon happy people? Is that the kind of calculation?

Rohin (08:31):
Yes. That’s an example of the question that I would deal with, yeah.

Rohin (08:35):
So, anyway, I was thinking about this question a lot in the summer of 2017. And, eventually, I was like, “Okay, I think I should actually put a fair amount of weight,” not certainty, certainly, but a fair amount of weight on the view that more happy people is, in fact, just means a better world, even if they’re in the future. Once you put a decent probability on that, it starts looking overwhelmingly important to ensure that the future continues to exist and have happy people because it’s just so big relevant to the present.

Rohin (09:21):
And so, then, I wanted to do something that was more future oriented and I had a ton of skills in computer science and math and, basically, everything you would want to work in AI alignment. I still was not very convinced of AI risk but I was like, “Okay, a bunch of smart people have thought about this, maybe I should work on it for a while, see whether or not it makes sense.” That’s what caused me to actually switch, and a year later I actually started believing the arguments.

Jeremie (09:57):
That’s so interesting.

Rohin (09:58):
I also saw different arguments.

Jeremie (10:00):
You were led by… Is it the quality of the people who were drawn to the problem more so than necessarily the initial arguments themselves? Do you remember an ah-ha moment as you were working on this stuff where you’re like, “Well, wait a minute. This is actually for real.” I can now see why Nick Bostrom, and maybe Eliezer Yudkowsky, and whoever else is talking about it back then was on to something?

Rohin (10:21):
I never really had an ah-ha moment. I remember, at one point, I was like, “I guess I now believe these arguments,” but it wasn’t like I… I guess I now believe that AI risk is substantial and real. I can’t point to a specific point in time where yes, now I believe it. I just, one day, was reflecting on it and noticed, “Oh, yeah. I used to not believe this. And now I do.”

Jeremie (10:52):
That’s so interesting. It seems like there’s a bifurcation between people who they read Superintelligence, or they read less wrong, and they get really excited about the problem and really scared of it right off the bat because, for whatever reason, they’re wired in such ways to have that happen. And then people, yeah, who are like you. It’s like a slow burn and you ease into it. I guess this is part of the problem, almost, of articulating the problem if it takes that long to get people to think of this as a really important thing.

Jeremie (11:18):
Do you have a strategy that you use when you try to explain to people why is AI risk so serious? Why is the probability nontrivial that you think might’ve worked on you back then to accelerate the process?

Rohin (11:32):
Yeah, I should note that I still… I’m not super happy with the arguments in like Superintelligence, for example. I would say that it’s slightly different arguments that are motivating it for me with still a fair amount of emphasis on things that were in Superintelligence.

Jeremie (11:50):
I think a lot of people won’t have heard of Superintelligence, by the way.

Rohin (11:54):
Oh, yes.

Jeremie (11:55):
If you want to address any of the arguments that you raise, please feel free to give that background, too.

Rohin (12:00):
Yeah, maybe I’ll just talk about the arguments I personally like since I can explain them better. But, just for context, Superintelligence is a book by a professor at Oxford named Nick Bostrom. It was published in 2014 and it was the first \[inaudible 00:12:19\] treatment of why AI risk is a thing that might occur and why we should think it might be plausible, what solutions might seem like they should be investigated and stuff like that.

Rohin (12:33):
And then, for me, personally, the argument I would give… So, A, we’re going to \[inaudible 00:12:45\] as a premise that we build AI systems that are intelligent, like as intelligent as a human, let’s say. We can talk about that later, but that’s a whole other discussion. I’ll just say that I think it is not… I think it is \[inaudible 00:13:03\] reasonably likely to happen in the next century. But, for now, take it as an assumption.

Rohin (13:10):
One thing about intelligence is it means that you can adapt to new situations, you’re presented with a new situation, you learn about it and you do something and that something is coherent. It makes sense. One example I give of this, where we see this even with current neural nets, is a specific example from GPT-3. I believe viewers will be familiar with GPT-3… Listeners, not viewers. But if not, GPT-3 is a \[inaudible 00:13:47\] language generation \[inaudible 00:13:48\] that OpenAI developed and released recently.

Rohin (13:51):
I think I like one particular example which ones from the the post giving GPT-3 a Turing test. Where the context to GPT-3 was a bunch of questions and answers. And GPT-3 would pose a question, how may bonks are in a quoit. These are nonsense words, you did not mishear me. GPT-3, in some sense, this is outside of its training distribution. It has never seen this sentence in its training corpus, presumably. It may not have even seen the words bonk and quoit ever. It’s actual distribution shift and you’re relying on some sort of generalization out of distribution.

Rohin (14:43):
Nonetheless, I think we can all predict that GPT-3 is not going to output some random string of characters. It’s going to probably say something sensible. In fact, the thing that it says is that, “You know, there are three bonks in a quoit.” Why three? I have no idea. But, you know, it’s sensible in some sense. It produced an answer that sounds like English.

Jeremie (15:10):
And we’ve all been there to some degree, if we write exams or whatever, we’re asked how many bonks are in a quoit, we haven’t done our studying, and, hey, there are three bonks in a quoit. There we go.

Rohin (15:18):
Exactly, right? In some sense, GPT-3 did generalize, it generalized the way a student taking a test would. In the original post, this was taken as evidence of GPT-3 not actually being reasonable because it doesn’t know how to say, “This question is nonsense.”

Rohin (15:39):
But then a followup post was like, “Actually, you totally can get GPT-3 to do that!” If you tell GPT-3 that… If in the context, you say whenever it sees a nonsense question the AI responds, “Yo, be real.” Then, when it’s asked how many bonks are in a quoit? It says, “Yo, be real.” So you know it’s got the ability to tell that this is nonsense, it just turned out that it generalized in a way where it was more like a test taker and less like somebody in conversation. Did we know that ahead of time? No, we did not. We had to actually run GPT-3 in order to figure this out.

Rohin (16:23):
I think AI risk is basically like this, but supercharged where your AI system, if it is human level intelligent, it’s definitely going to be deployed in new areas, in new situations, that we haven’t seen before. We just don’t really have a compelling reason to believe that it will continue to do the thing that we were training it to do as opposed to something else. In GPT-3, what were we training it to do? Well, on the training data set, at least, we were training it to do whatever a human would write in that context.

Rohin (17:05):
When you see there are three bonks in a… Sorry, how many bonks are in a quoit? What would a human do in that circumstance? I don’t know. It’s not really well defined, and GPT-3 did something sensible. I don’t think you could reasonably say it wasn’t doing what we trained it to do, it just did something that was coherent. And, similarly, if you’ve got AI systems that are human level intelligent or more, taking super impactful actions upon the world and they are put in these new situations where there’s not really a fact of the matter about how they will generalize then they might take actions that have a high impact on the world that aren’t what we want.

Rohin (17:49):
And then, as maybe intuition \[inaudible 00:17:52\] for why this could be really, really bad, like human extinction level bad. One particular distribution shift is you go from the training setting where humans have more power than the AI and can turn off the AI to the setting where the AI is sufficiently intelligent and sufficiently widely deployed, but no human can… Or humanity as a whole cannot turn it off. In that situation, that’s a new situation. AI has never been in a situation where it had this sort of power before. Will it use it in some way that… Will it generalize in some way that was different than what we expected during training? We don’t really have a reason to say no, it won’t do that.

Jeremie (18:33):
Is there an analogy you think here with child rearing? I’m just thinking of here intergenerational human propagation where our ancestors in the 1600s, at least in the west, I’m sure would be absolutely disgusted by our vile ways today the way that we deal with sex, the way that we communicate to our elders, the way that we manage our institutions and so on, all our hierarchies are just completely different. And, in many ways, we’re \[inaudible 00:19:00\] to the moral frameworks that were applied in the middle ages or the early renaissance.

Jeremie (19:07):
I guess there is a difference here in the sense that at least we are still running on the same fundamental hardware, or something very similar. Maybe that ensures a minimum level of alignment, but does this analogy break apart in some way?

Rohin (19:18):
I think that’s a pretty good intuition. There are some ways that the analogy breaks, like for example, well… The analogy doesn’t break so much as I would say put a little bit less weight on it for these reasons. One is, in child rearing, you have some influence over children, but you don’t get to do a full training process where you give them gradients for every single time step of action that they ever do. You might hope that, given that you can have way, way, way more selection pressure on AI systems, you would be able to avoid this problem.

Rohin (20:00):
But, yes, I think that that is the same fundamental dynamic that I’m pointing at. You have some amount of influence over these agents, but those agents that encounter new situations and they do something in those situations and you didn’t think about those situations ahead of time and you didn’t train them to do the right thing.

Jeremie (20:23):
I definitely buy the idea here that this AI risk, this is really significant risk. The stakes are very high. When it comes to the solutions or the strategies that you think are most promising, you yourself are specialized, obviously, in one category, everyone has to be, in one subspace within the alignment problem domain. What is it that the area that you’ve decided to focus on and why do you think that is most deserving of the attention at this point?

Rohin (20:50):
The story that I’ve told you so far is one of generalization. The main issue is we don’t know how to generalize and, plausibly, you could get AI systems that are single mindedly pursuing power and that’s similar to the Superintelligence story and those could cause human extinction. The fundamental mechanism is bad generalization, or generalization that’s like your capabilities generalize. You do something coherent and high impact, but the thing you’re trying to do doesn’t generalize, relative to what humans wanted.

Rohin (21:31):
A lot of the things I’m most excited about are somehow generalization related. One thing that I’m interested in is can we get a better understanding of empirically how do neural nets tend to generalize? Can we say anything about this? There’s a lot of theory that tries to explain why neural nets have as good generalization power as they do. It can’t be explained by statistical learning theory because neural nets can memorize random noise, but nonetheless seem to generalize reasonably well on when the labels are not random noise.

Jeremie (22:15):
And do you mind explaining statistical learning theory as a reference? I’m actually not so sure that I can to the connection.

Rohin (22:23):
Statistical learning theory is like a branch of machine learning theory that tries to do several things. But, among other things, try to prove that if we train a machine learning model on such and such training data with such and such training properties, then we know that it will generalize in such and such way and it proves theorems about this.

Rohin (22:50):
Importantly, most approaches right now focus on making assumptions about your model, your hypothesis class. These assumptions usually preclude the ability to overfit to an arbitrary sized data set because if you could, then you can’t really say anything about generalization. But the fact of the matter is neural nets really can actually overfit to any data set. They can memorize labels that are literal random noise. And, so, these assumptions just don’t apply to neural nets.

Rohin (23:28):
The thing I’m excited about is can we talk about assumptions on the data set, rather than just the model? And using, if we think about assumptions on the data set and assumptions on the model, then can we say something about how neural nets tend to generalize? This is like a super vague not fleshed out hope that I have not really started working on, nor to my knowledge has anyone else.

Rohin (23:55):
There’s just so many empirical things about neural nets that are so deeply confusing to me, like deep double descent. I don’t get it. It’s an empirical phenomenon. If you don’t know, you can look it up. It’s probably not that worth me going into, just so confusing. I don’t know why it happens. It makes no sense to me. And I will want to know why, and I think that if we understood things like this, we might be able to start making statements about how neural nets tend to generalize that maybe that translates into things we can say about safety.

Jeremie (24:27):
That’s interesting, because the generalization story seems to be one ingredient of the problem, of course, and then the other ingredient, which, I mean, there’s some overlap, but it does seem like they have distinct components. Is this challenge of telling machines what human preferences even are, our ability to tell each other what we want out of life is already so limited, and it’s something that, I mean, at least I personally find somewhat jarring as a prospect, having to actually not only express our preference, but quantify them and etch them into some kind of loss function that we then feed to a model. You’ve done a lot of interesting work on this.

Jeremie (25:09):
And, actually, there’s one of your papers that I wanted to talk about. We discussed this before we started recording and I was so glad to hear that it was also the one that you thought was the most interesting. We have compatible views, at least on that. It was this idea of… Well, the paper’s title is Preferences Implicit in the State of the World. I guess, first, I wanted to ask a question to set the scene a little bit. What is preference learning? What is that concept?

Rohin (25:34):
This is actually the next thing I was going to say I was excited by.

Jeremie (25:38):
Oh, great.

Rohin (25:39):
Which is I’ve talked about generalization, but before you get to generalization, you want to train on the right thing in the first place. That seems like a good starting point for AI system. If you don’t have that, you’re probably toast. Lots of ink has been spilled on how it’s actually very typical to specify what you want by writing a program or an equation that captures it in a number which, as you know, how deep reinforcement learning, or any deep learning system, works. But it’s most commonly associating with deep reinforcement learning.

Rohin (26:21):
The idea of preferenced learning is rather than having to specify what you want by writing down an equation, you specify it by some easier method. For example, you could look at two trajectories in a reinforcement setting, you can look at two behaviors that the agent took, and you can say, “Ah, yes, the left one. That one was better.” That’s giving the agent some feedback about what it should be doing. It’s not trying to write down an equation that captures the ideal behavior in every possible scenario. It’s just saying, out of these two, which one is better? You would imagine that that’s easier for humans to do and more likely to be correct.

Rohin (27:08):
This preferenced learning field, I think of it as the field of how do we design mechanisms for humans to give feedback to an AI system, such that we can actually give feedback that incentivizes the behavior we actually want and we don’t make as many mistakes in specification as we do with reward functions.

Jeremie (27:34):
So, what I find really exciting about that aspect, too, is there’s this well known difference in humans between expressed desires and revealed desires, or expressed intent and revealed intent. I’ll say I want to work out for three hours today, I want to do a bunch of coding, and I want to have a bunch of vegan meals for the next month. And then if you check in on me next month, I will have not have done all those things, I wouldn’t have done nearly all those things. And the question is well, which me is me? Am I the aspirational self that said hey, I would love to be that person. Or am I the jackass who actually sat on his couch and watched Netflix the entire time?

Jeremie (28:16):
This seems to really scratch that itch in the sense that if probes are revealed preferences, for better or for worse, I guess that could also be a failure mode. Is that something that you see as valuable in this approach?

Rohin (28:29):
Yeah, I think you want to use both sources of information and not do either one. Actually, let me take a step back and distinguish between two different things you could be trying to do. There’s one thing where you’re trying to learn what humans value, which is the sort of thing that you’re talking about, and there’s another framing where you’re just like, “I want my AI system to do such and such task and I want to train it to do that, but I can’t write down a reward function for that task.”

Rohin (29:01):
I’m actually more interested in the latter, honestly, but the former is also something I’ve spent a lot of time on and I’m excited by. Right now, we’re talking about the former.

Jeremie (29:13):
Can I ask a naïve question? I think I understand what the difference is, but I just want to put it to you to tackle it explicitly. What is the difference between those two things?

Rohin (29:24):
One thing is maybe I want my AI system to vacuum my floors, or something. The task of vacuuming my floors is not well specified just by that sentence. Anyone who has a Roomba will tell you stories of the Roomba being super dumb. Some of those are just the Roomba being not intelligent enough, but some are also the task is not super well specified.

Rohin (29:57):
Should the agent vacuum underneath a Christmas tree where there’s a bunch of needles that might ruin their vacuum? Who knows. If there’s some random shiny button on the floor, should it be vacuumed up or left alone? Because maybe that button’s important. What sorts of things, should the cat ever be vacuumed? The cat has a lot of hair that gets everywhere. If you vacuum the cat, that seems like it would make your house cleaner.

Rohin (30:31):
There’s lots of ambiguity here. I wouldn’t really say that these are human values, like teaching your Roomba how to vacuum does not seem to be the same thing as teaching the Roomba about human values. For one thing, you can’t really talk that much about revealed preferences here because I don’t vacuum my house very often. If an AI system were going to queue vacuum, I might have it vacuum more often.

Jeremie (31:06):
Would you say this is a narrow application of human preferences? It almost seems like the distinction between narrow AI and AGI somehow maps onto this.

Rohin (31:16):
Yeah, and I think I agree with that. I would say, but in this sense, everything is narrow AI. You just get narrow AI that becomes more and more general, and at some point we decide to stop calling it narrow AI and start to call it AGI because of how broad it has become.

Rohin (31:34):
I like the idea of you start with something that can be applied to systems today and you just scale it up. It becomes more and more capable, more and more general, but it’s always the same technique. Eventually, the systems that we create with it, we would label them as AGI or human level intelligence or super intelligent. It’s the same technique, it’s the same general principle. That’s why I’m more excited by this framing of the problem, rather than the human value spamming.

Rohin (32:07):
As you get to more general systems, it merges in with the human values. Once you get AI systems that are designing government policies or something, whatever feedback you’re giving them, it better teach them about human values.

Jeremie (32:23):
Yeah, and hopefully, I guess, we start to do that at higher and higher levels of abstraction as you say as we climb that ladder. We fill in the convolutional filters in a sense as we go up.

Rohin (32:34):
Yes, exactly. You had asked a question about revealed preferences versus spoken preferences, or express preferences. I think, yes, this is an important distinction. I definitely want any method that we propose to not be dependent on one or the other but to instead be using both, and there will be conflicts, I’m mostly hoping that we can just have AI systems that set aside the conflicts and do things that are robustly good according to either set. Probably, you’ll have to have some amount of conflict resolution mechanism, but humans already have to do this in some sense. It seems plausible that we could do it.

Rohin (33:28):
I think it is a good, very nice aspect of this is that you don’t have to commit yourself to finding the behavior up front in every possible situation. We just don’t know this. Our values are not well defined enough, honestly, for that to be right. Our values are constantly in the process of being updated as we encounter new situations. Right now, we talk about democracy, one vote per person. If, someday, in the transhumanist future, if someday it becomes possible to just make copies of people, I think we would pretty quickly no longer want to have one vote per person. Because otherwise you can just pay to have anyone elected if you are sufficiently rich.

Jeremie (34:19):
Yeah. Or, I guess, just in the limited better information about brain states, we could say well, sure, this policy makes the majority of people happier, but the people it makes more unhappy, I mean, look at that horrible dopamine cycle. Those people are really taking a big hit and you wake those responses.

Rohin (34:37):
Yep, yeah, you could definitely optimize better for social welfare potentially, and maybe then you don’t want to just have one vote per person.

Jeremie (34:45):
Right, now, I guess this brings us back to preferences implicit in the state of the world, there are things, presumably, about the structure of the world that reveal our, I guess this is revealed preferences, mostly, right?

Rohin (34:57):
Yes.

Jeremie (34:57):
What we’ve actually done.

Rohin (34:58):
Yep, this is definitely a revealed preferences method. I think an important aspect of this is people will… I think one of the reasons I’m especially excited about this, which I want to say as a prelude, is that it’s not trying to do the hard things. When people think about value learning, they think about should a self driving car if it has a choice between running into two passengers, or killing the driver, what should it do? Those are hard, ethical questions. I’m less interested in them. I want to start with can we get an AI system that reliably knows that it should not kill humans. If there are two options where, yeah…

Rohin (35:50):
Anyway, the basic stuff that we all agree on or nearly all agree on, and so I think looking at the state of the world is a good way to see this and the basic intuition here is that we’ve been acting in the world for so long. We have preferences, we’ve been rearranging the world to fit the way that we want the world to be. As a result, you can invert that process to figure out what things we probably want.

Rohin (36:26):
There’s this nice toy example that illustrates this. Suppose there’s a room, and in the middle of the room there is this breakable vase. And vases, once they’re broken, they can never be repaired. We assume that the AI knows this. We’re going to assume that the AI knows all empirical facts. It knows how the world works, it knows what actions the human can take, it knows what actions the human can take, it knows what actions it itself can take, it knows what states of the world are possible, but it doesn’t know anything about the \[inaudible 00:36:58\] function which is the equivalent of human values.

Rohin (37:02):
It knows empirical facts. It knows that this vase, once broken, cannot be fixed. We’re going to leave aside glue and things like that. It then looks at the fact that it sees, it is deployed in this room, and it sees that its human, who I’ll call Alice, is in the room, the vase is unbroken. Now you can pose hypothetical questions like all right, well, what would I have expected to see if Alice wanted to break the vase? Well, I would have seen a broken vase. What would I have expected if Alice didn’t care about the vase? Well, probably, at some point, the most efficient way would have been to just walk through the room while knocking over the vase. So, probably in that case, also I would have seen the broken vase.

Rohin (37:58):
What would I expect to see if Alice did not want the vase to be, or actively wanted the vase to not be broken? In that case, I actually see an unbroken vase, probably. Since I actually see an unbroken vase, that tells me that of those three situations, only the last one seems consistent with my observations. So, probably, Alice did not want to break the vase. You can infer this fact about that Alice doesn’t want to break the vase just by looking at the state of the world and seeing that the vase is not broken.

Jeremie (38:33):
It seems like there’s a very deep connection here to the second law of thermodynamics and the universe has there’s so many more ways to end up in a situation where you have a broken vase, but the fact that there isn’t a broken vase is a huge piece of information.

Rohin (38:52):
Yeah, that’s exactly right.

Jeremie (38:57):
Basically, to the extent \[crosstalk 00:38:59\]-

Rohin (38:58):
I don’t think I have anything to add.

Jeremie (38:59):
Well, it just strikes me, the physicist instinct in me, but to the extent that the world looks any different from what we would expect with pure thermodynamic randomness, the assumption here is those differences come from human preferences. Would that be a fair way to characterize the…

Rohin (39:17):
Yep, that’s right.

Jeremie (39:19):
And does that imply certain failure modes then? Because I guess we encode information in our environment, I guess this is \[inaudible 00:39:25\] revealed preferences thing, but implicitly, I’ve hard coded my brain state into my apartment, every arrangement of things, any misogyny, any racism, any foot fetishes, the whole laundry list of weird quirks that may or may not be part of my personality are implicitly encoded in the room. Is this part of the risk of applying a technique like this?

Rohin (39:54):
Yeah, so, in theory, if you \[inaudible 00:39:59\] this method, it would… Is this going to get everything that is a revealed preference? And there are, well, I don’t know that it gets everything. But to a first approximation, it gets your revealed preferences. I’m sure there is some that it does not get. Sometimes, you just don’t like your revealed preferences and you think they should be different.

Rohin (40:27):
You have a revealed preference, many people have a revealed preference, to procrastinate that they probably do not in fact endorse and they wouldn’t want their AI system giving them more and more addictive material so that they can procrastinate better which is plausibly something that could happen. I would have to think significantly harder about how exactly concretely that could happen, but I could believe that that would be an effect.

Rohin (41:03):
Similarly, the technique as I’ve explained it so far it seems that there is only one human in the world and things get a lot more complicated if there are multiple humans and I have just ignored that case so far.

Jeremie (41:20):
It’s what you need to get the thing off the ground, right?

Rohin (41:22):
Yeah.

Jeremie (41:26):
In this context, I imagine at least, there’s another risk mode which is if, by pure chance, let’s say, in the example with the vase, let’s say that the human actually doesn’t care about the vase but just happens, in her demonstration, to have avoided the vase. Is there the risk that, I guess this is always a risk in machine learning, it sounds like just a case of added distribution sampling, like you would learn-

Rohin (41:57):
Yep.

Jeremie (41:57):
Okay.

Rohin (41:58):
Yeah, that’s right. If the vase is kept in an inconspicuous out of the way location where it’s not actually that likely that Alice would have broken the vase on the course of moving around the room, we actually have this on the paper, we show that in that environment you actually don’t learn anything significant about the vase. You’re just like, “Eh, she probably didn’t want it broken.” You infer that she did not deeply desire for the vase to be broken, but you don’t infer anything stronger than that. You’re uncertain between whether it’s bad to break vases rather versus yeah, it doesn’t really matter.

Jeremie (42:44):
Interesting.

Rohin (42:45):
It’s more like if there were efficiency gains to be had by breaking the vase, then you infer and you observe that actually the vase wasn’t broken, then you infer that it’s bad to break vases. It’s still possible that humans, we’re not perfect optimal people, we might not pick up an efficiency gain and so we might go around a vase even though it would be faster to go to the vase even if we didn’t care about the vase. And, yeah, this method would make an incorrect inference. In general, in preference learning, there’s a big tension between you’re assuming that humans do the things that the humans do to reflect what they want. And not always true.

Jeremie (43:37):
Right, and sometimes just for reasons of pure stupidity as well, I guess. We may want a thing, just not know how to make it happen.

Rohin (43:45):
Yup, exactly. That is a big challenge in preference learning and people have tried to tackle it, including me, actually. But I wouldn’t say that there has been super substantial progress on separating stupidity from the things you actually wanted.

Jeremie (44:10):
I think we’d end up solving a lot of other problems instead \[crosstalk 00:44:12\] if we do that. Actually, there’s one more aspect I wanted to ask you about, with respect to the paper. The rule of time horizons, or the rule that time horizons play in the paper is, I think, just really interesting because there’s certain assumptions that the robot makes or the AI makes about what is the time horizon that the human has in mind for this action that if there’s assumptions about that time horizon shift, you start to see different behavior. I’d love to hear you expand on that and describe that setting a little bit.

Rohin (44:43):
I think the main takeaway from that I think I would have about time horizons is if you assume a short time horizon, then cases where the state hasn’t been fully optimized are much more excusable because the human just hasn’t had enough time to fully optimize the state towards what would be optimal for them and so you can-

Jeremie (45:11):
And so maybe I should fill in that gap, I realize it was a little ambiguous, but by time horizon, I guess we’re talking about the amount of time the human would have to, say, go from one point in the room to a desired end point, right? \[crosstalk 00:45:24\]

Rohin (45:23):
Yeah, it’s like the amount of time that the robot assumes the human has been acting in the environment before the robot showed up. In the room case, it’s robot is deployed and sees an intact vase and it’s like, “Ah, yes, the human has been walking around this room for an hour,” or something like that.

Jeremie (45:40):
Right, if you’ve been walking around the room for a full hour and the vase is still there, you can then assume that the vase is probably pretty important.

Rohin (45:48):
Yeah, something along those lines, exactly. The actual setting in the paper is slightly different, but that’s the right intuition. Yeah, and so this isn’t illustrated best with the vase example, but imagine you’re trying to build a house of cards. This is another example where the state of the world is really very informative about your preferences. House of cards are super, super not entropic. You can infer a lot.

Jeremie (46:21):
Yeah, the more specific the arrangement, I guess the more… Which is interesting because that’s exactly what’s so challenging about preserving humanity in general, there’s almost a philosophically conservative streak to this approach in the sense that we’re assuming that we’ve gotten to somewhere that’s worth preserving because we’ve encoded so much of ourselves, so much of what’s good about us already in the environment and it almost seems like what I love about this time horizon stuff is the political philosophy behind it, it almost gives you a dial that you can tune from the progressive to the conservative end of the spectrum just by assuming different time horizons. If you assume that we just got here and it’s sort of blank slate, then, hey, we can try anything. We’re not really certain about what humans want in this environment, conversely…

Rohin (47:12):
Yeah.

Jeremie (47:12):
Right? Yeah.

Rohin (47:13):
That’s true, I’ve never actually thought about it that way, but you’re right. That is basically what it is. Another way of thinking about it is the way I actually got to this point, to the point of writing this paper, was asking myself why is it that we privilege the do nothing action? We’ll say that the safe action is to do nothing. Why? It’s just an action. This is an answer, we’ve already optimized the environment, random actions are probably, so… The current state is high on our preference ranking. Random actions take us out of that state into some random other state, so probably an expectation to go lower in our ranking whereas the do nothing action preserves it and so it’s good. The longer the time horizon, the stronger you want to do nothing as a default.

Jeremie (48:07):
Yeah, yeah, I remember reading that in the paper, actually, as almost a derivation of that intuition which is it’s so beautiful when you can see it laid out like that.

Rohin (48:15):
I know, so good.

Jeremie (48:18):
Yeah, yeah. In a way, it makes me think of so many arguments among and between people of different political stripes could be so much easier if we applied toy models like this where you can say, well, hey, there is value to the conservative. There is value to the progressive. We end up in a dystopia either way, and here’s the parameter we can tune to see how dystopic things get one way or the other, depending on how much we value things.

Jeremie (48:43):
Yeah, anyway I love the work and I thought it was… Anyway, again, one of these ah-ha moments, for anybody who’s interested in the intersection between philosophy, moral philosophy, and then an AI, anyway, it’s just such a cool piece of work.

Rohin (48:58):
Yeah, thanks. I like it for basically the same reasons.

Jeremie (49:03):
Sweet. Well, I’m glad we have compatible \[inaudible 00:49:06\], then. Awesome. Well, I think we’ve covered a lot of the bases here, but was there anything else you wanted to talk about? One thing I do want to make sure we get to is a reference to the alignment newsletter that you put out. I think everybody should check that out, especially if you’re looking to get into the space. Rohin puts out this amazing newsletter and anyway, we’ll link to it on the blog post that will come with the podcast.

Jeremie (49:30):
Did you have any social media links or things like that that you want to share?

Rohin (49:35):
I think the alignment newsletter is the best way to get my current thinking on things. If you’re new to the space, I’d probably recommend other things. Specific ranking of mine that I like as more introductory… It’s not exactly introductory, but more timeless material, on the alignment forum there is a sequence of blog posts called the value learning sequence that I wrote. I like that as a good introduction, there are two other recommended sequences on that forum that I also recommend, that I also think are pretty great.

Rohin (50:21):
In terms of social media, I have a Twitter. It’s @RohinMShah, but mostly it just sends out the alignment newsletter links. People can also feel free to e-mail me. My e-mail’s on my website, can’t guarantee that I will send you a response because I do actually get a lot of e-mail but I think I have a fairly high response rate.

Jeremie (50:50):
Yeah, well, I can vouch for that at my end. Thanks for making the time, really appreciate it, and I’m really looking forward actually to putting this out and also good luck with DeepMind because you’re heading over there in a couple of days, really, right?

Rohin (51:04):
Yeah, I’m heading there on Monday. It’s two more business days.

Jeremie (51:09):
All right, yeah, enjoy the long weekend such as it is.

Rohin (51:13):
Cool. Thanks.

Jeremie (51:13):
Awesome, thanks so much, Rohin.

Rohin Shah - Effective altruism, AI safety, and learning human preferences from the world_s state-by Towards Data Science-video_id uHiL6GNXHvw-date 20201028.md


# Jade Leung Why companies should be leading on AI governance - EA Forum

_Are companies better-suited than governments to solve collective action problems around artificial intelligence? Do they have the right incentives to do so in a prosocial way? In this talk, Jade Leung argues that the answer to both questions is "yes"._

_A transcript of Jade's talk is below, which CEA has lightly edited for clarity. You can also watch this talk on_ [_YouTube_](https://www.youtube.com/watch?v=AVDIQvJVhso)_, or read its transcript on_ [_effectivealtruism.org_](https://www.effectivealtruism.org/articles/ea-global-2018-why-companies-should-be-leading-on-ai-governance)_._

## The Talk

In the year 2018, you can walk into a conference room and see something like this: There's a group of people milling around. They all look kind of frantic, a little bit lost, a little bit stressed. Over here, you've got some people talking about GDPR and data. Over there, you've got people with their heads in their hands being like, "What do we do about China?" Over there, you've got people throwing shade at Zuckerberg. And that's when you know you're in an AI governance conference.

The thing with governance is that it's the kind of word that you throw around, and it feels kind of warm and fuzzy because it's the thing that will help us navigate through all of these kind of big and confusing questions. There's just a little bit of a problem with the word "governance," in that a lot of us don't really know what we mean when we say we want to be governing artificial intelligence.

So what do we actually mean when we say "governance?" I asked Google. Google gave me a lot of aesthetically pleasing, symmetrical, meaningless management consulting infographics, which wasn't very helpful.

![](https://images.ctfassets.net/ohf186sfn6di/0QF4aqkkDrIDQKTo4vMJo/fa5db8b93b528ab2f29a5bb9934e6e07/jade_slide_1.PNG)

And then I asked it, "What is AI governance?" and then all the humans became bright blue glowing humans, which didn't help. And then the cats started to appear. This is actually what comes up when you search for AI governance. And that's when I just gave up, and I was like, "I'm done. I need a career change. This is not good."

![](https://images.ctfassets.net/ohf186sfn6di/2Ivakk5wkUquB12FarVQEe/5a5ecbe6542781095b2fa6c3de9581e2/jade_slide_2.PNG)

So it seems like no one really knows what we mean when we say, "AI governance." So I'm going to spend a really quick minute laying some groundwork for what we mean, and then I'll move on into the main substantive argument, which was, "Who should actually be leading on this thing called governance?"

![](https://images.ctfassets.net/ohf186sfn6di/6TgqeZBUAbBNm8LCWXTAY6/d19b7c98fa82625e8d33442c72373e23/jade_slide_3.PNG)

So governance, global governance, is a set of norms, processes, and institutions that channel the behavior of a set of actors towards solving a collective action problem at a global or transnational scale. And you normally want your governance regime to steer you towards a set of outcomes. When we talk about AI governance, our outcome is something like the robust, safe, and beneficial development and deployment of advanced artificial intelligence systems.

![](https://images.ctfassets.net/ohf186sfn6di/4S3XPxYuB52keZpivYY1jF/ce68e1ee2a08d3159ce21a23f672e0fe/jade_slide_4.PNG)

Now that outcome needs a lot of work, because we don't actually really know what that means either. We don't really know how safe is safe. We don't really know what benefits we're talking about, and how they should be distributed. And us answering those questions and adding granularity to that outcome is going to take us a while.

So you can also put in something like a placeholder governance outcome, which is like the intermediate outcome that you want. So the process of us getting to the point of answering these questions can include things like being able to avoid preemptive locking, so that we don't have a rigid governance regime that can't adapt to new information. It could also include things like ensuring that there are enough stakeholder voices around the table so that you are getting all of your opinions in. So those are examples of intermediate governance outcomes that your regime can lead you towards.

![](https://images.ctfassets.net/ohf186sfn6di/6yurMnugW4fN34drvdAeXF/01b8c15c094c42b00b49259b744cf72a/jade_slide_5.PNG)

And then in governance you also have a set of functions. So these are the things that you want your regime to do so that you get to the set of outcomes that you want. So common sets of functions that you talk about would be things like setting rules. What do we do? How do we operate in this governance regime? Setting context, creating common information and knowledge, doing common benchmarks and measurements. You also have implementation, which is both issuing and soliciting commitments from actors to do certain things. And it's also about allocating resources so that people can actually do the things. And then finally, you've got enforcement and compliance, which is something like making sure that people are actually doing the thing that they said that they would do.

So these are examples of functions. And the governance regime is something like these norms, processes, and institutions that get you towards that outcome by doing some of these functions.

So the critical question today is something like, how do we think about who should be taking the lead on doing this thing called AI governance?

I have three propositions for you.

![](https://images.ctfassets.net/ohf186sfn6di/3aBi5cF4bmBdAMJGBP3Aek/257d867d0d6911ddb2f6e30fc7c6557e/jade_slide_6.PNG)

One: states are ill-equipped to lead in the formative stages of developing an AI governance regime. Two: private AI labs are better, if not best-placed, to lead in AI governance. And three: private AI labs can and already, to some extent, are incentivized to do this AI governance thing in a prosocial way.

I'll spend some time making a case for each one of these propositions.

### States are Ill-Equipped

When we normally think about governance, you consider states as the main actors sitting around the table. You think about something like the UN: everyone sitting under a flag, and there are state heads who are doing this governance thing.

![](https://images.ctfassets.net/ohf186sfn6di/14y9bJyzhoLTk3tTnNjE0j/961bf1667043952d4ddda5072c32c28e/jade_slide_7.PNG)

You normally think that because of three different reasons. One is the conceptual argument that states are the only legitimate political authorities that we have in this world, so they're the only ones who should be doing this governance thing. Two is you've got this kind of functional argument: states are the only ones who can pass legislation, design regulation, and if you're going to think about governance as regulation and legislation, then states have to be the ones doing that function. And three, you've got something like the incentives argument, which is that states are set up, surely, to deliver on these public goods that no one else is going to care about as a result. So states are the only ones that have the explicit mandate and the explicit incentive structure to deliver on these collective action problems. Otherwise, none of this mess would get cleaned up.

Now all of those things are true. But there are trends and certain characteristics about a technology governance problem, which means that states are particularly increasingly undermined in their ability to do governance effectively, despite all of those things being true.

![](https://images.ctfassets.net/ohf186sfn6di/2mCyYhctogvxzVWjItUS6t/e433c20b26e360acdd5514fd57104211/jade_slide_8.PNG)

Now the first is that states are no longer the sole source of governance capacity. And this is a general statement that isn't specifically about technology governance. You've got elements like globalization, for example, creating the situation where these collective action problems are at a scale which states have no mandate or control over. And so states are increasingly unable to do this governance thing effectively within the scope of the jurisdiction that they have.

You also have non-state actors emerging on the scene, most notably civil society and multi-national corporations are at this scale that supersedes states. And they also are increasingly demonstrating that they have some authority, and some control, and some capacity to exercise governance functions. Now their authority doesn't come from being voted in. The authority of a company, for example, plausibly comes from something like their market power and the influence on public opinion. And you can argue about how legitimate that authority is, but it is exercised and it does actually influence action. So states are no longer the sole source of where this governance stuff can come from.

Specifically for technology problems, you have this problem that technology moves really fast, and states don't move very fast. States use regulatory and legislative frameworks that hold technology static as a concept. And technology is anything but static: it progresses rapidly and often discontinuously, and that means that your regulatory and legislative frameworks get out of date very quickly. And so if states are going to use that as the main mechanism for governance, then they are using irrelevant mechanisms very often.

Now the third is that you have emerging technologies specifically being a challenge. Emerging technologies have huge bars of uncertainty around the way that they're going to go. And to be able to effectively govern things that are uncertain, you need to understand the nature of that uncertainty. In the case of AI, for example, you need deep in-house expertise to understand the nature of these technology trajectories. And I don't know how to say this kindly, but governments are not the most technology-literate institutions that are around, which means that they don't have the ability to grapple with that uncertainty in a nuanced way, which means you see one of two things: you either see preemptive clampdown out of fear, or you see too little too late.

So states are no longer the sole source of governance capacity. And for technology problems that move fast and are uncertain, states are particularly ill-equipped.

## Private Labs are Better Placed

Which leads me to proposition two, which is that instead of states, private AI labs are far better-placed, if not the best-placed, actors to do this governance thing, or at least form the initial stages of a governance regime.

![](https://images.ctfassets.net/ohf186sfn6di/47lFfvgscAaYFssV338Lkm/83ec08c55b572a5b944ce88d317301a0/jade_slide_9.PNG)

Now this proposition is premised on an understanding that private AI labs are the ones at the forefront of developing this technology. Major AI breakthroughs have come from private companies, privately funded nonprofits, or even academic AI labs that have very strong industrial links.

Why does that make them well-equipped to do this governance thing? Very simply, it means that they don't face the same problems that states do. They don't face this pacing problem. They have in-house expertise and access to information in real time, which means that they have the ability to garner unique insights very quickly about the way that this technology is going to go.

![](https://images.ctfassets.net/ohf186sfn6di/MRIgUjKZM1xoDLtukUlVj/a6f8834edec27bb3557bfdc392b16787/jade_slide_10.PNG)

So of all the actors, they are most likely to be able to slightly preemptively, at least, see the trajectories that are most plausible and be able to design governance mechanisms that are nuanced and adaptive to those trajectories. No other actor in this space has the ability to do that except those at the forefront of leading this technology development.

![](https://images.ctfassets.net/ohf186sfn6di/dliu1XeCIrLxZgDbWP80e/ad791977f72ac0087d93d6458bf8d02b/jade_slide_11.PNG)

Now secondly, they also don't face the scale mismatch problem. This is where you've got a massive global collective action problem, and you have states which are very nationally scaled. What we see is multinational corporations which from the get-go are forced to be designed globally because they have global supply chains, global talent pools, global markets. The technology they are developing is proliferated globally. And so, necessarily, they both have to operate at the scale of global markets, and they also have experience, and they attribute resources to navigating at multiple scales in order to make their operations work. So you see a lot of companies scale at local, national, regional, transnational levels, and they navigate those scales somewhat effortlessly, and certainly effortlessly compared to a lot of other actors in this space. And so, for that reason, they don't face the same scale mismatch problem that a lot of states have.

So you've got private companies that both have the expertise and also the scale to be able to do this governance thing.

Now you're probably sitting there thinking, "This chick has drunk some private sector Kool-Aid if she thinks that private sector, just because they have the capacity, means that they're going to do this governance thing. Both in terms of wanting to do it, but also being able to do it well, in a way that we would actually want to see it pan out."

### Private Labs are Incentivized to Lead

Which leads me to proposition three, which is that private labs are already and can be more incentivized to lead on AI governance in a way that is prosocial. And when I say "prosocial" I mean good: the way that we want it to go, generally, as an altruistic community.

Now I'm not going to stand up here and make a case for why companies are actually a lot kinder than you think they are. I don't think that. I think companies are what companies are: they're structured to be incentivized by the bottom line, and they're structured to care about profit.

![](https://images.ctfassets.net/ohf186sfn6di/75u4VpesxfLDLY3HSvdDwA/3cbf76dbeb3c4822e9fdfa347da62178/jade_slide_12.PNG)

All that you need to believe in order for my third proposition to fly is that companies optimize for their bottom line. And what I'm going to claim is that that can be synonymous with them driving towards prosocial outcomes.

Why do I think that? Firstly, it's quite evidently in a firm's self-interest to lead on shaping the governance regime that is going to govern the way that their products and their services are going to be developed and deployed, because it costs a lot if they don't.

![](https://images.ctfassets.net/ohf186sfn6di/1IGpA6dC3YtDm6Uw3S2DX7/523fd902a9d58e6e9bd42487e3954a56/jade_slide_13.PNG)

How does that cost them? Poor regulation, and when I say "poor", I mean poor in terms of costly for firms to engage with, is something where you see a lot of costs incurred to firms when that happens across a lot of technology domains. And the history of technology policy showcases a lot of examples where firms haven't been successful in preemptively engaging with regulation and preemptively engaging with the governance, and so they end up facing a lot of costs. In the U.S, and I'll point to the U.S. because the U.S. is not worst example of it, but they have a lot of poor regulation in place particularly when it comes to things like biotechnology. In biotechnology, you've got blanket bans on certain types of products, and you also have things like export controls, which have caused a lot of loss of profit for these firms. You also have a lot of examples of litigation across a number of different technology domains where firms have had to battle with regulation that has been put in place.

Now it wasn't in the firms' interests to incur those costs. And so the most cost-effective way, in hindsight, would be for these firms to engage with the governance as they were shaping regulation, shaping governance, and doing what that would be.

Now just because it's costly doesn't mean that it's going to go in a good way. What are the reasons why them preemptively engaging is likely to lead to prosocial regulation? Two reasons why. One: the rationale for a firm would be something like, "We should be doing the thing that governance will want us to do, so that they don't then go in and put in regulation that is not good for us." And if you assume that governance has that incentive structure to deliver on public goods, then firms, at the very least, will converge on the idea that they should be mitigating their externalities and delivering on prosocial outcomes in the same way that the state regulation probably would.

The more salient one in the case of AI is that public opinion actually plays a fairly large role in dictating what firms think are prosocial. You've seen a lot of examples of this in recent months where you've had Google, Amazon, and Microsoft face backlash from the public and from employees where they've developed and deployed AI technologies that grate against public values. And you've seen reactions from these firms respond to those actions as well. It's concrete because it actually affects their bottom line: they lose consumers, users, employees. And that, again, ties back to their incentive structure. And so if we can shore up the power of something like the public opinion that translates into incentive structures, then there are reasons to believe that firms will engage preemptively in shaping things that will go more in line with what public opinion would be on these issues.

So the second reason is that firms already do a lot of governance stuff. We just don't really see it, or we don't really think about it as governance. And so I'm not making a wacky case here in that business as usual currently is already that firms do some governance activity.

Now I'll give you a couple of examples, because I think when we think about governance, we maybe hone in on the idea that that's regulation. And there are a lot of other forms of governance that are private sector-led, which perform governance functions, but aren't really called "governance" by the traditional term.

So here are some examples. When you think about the function of governance of implementing some of these commitments, you can have two different ways of thinking about private sector leading on governance. One is establishing practices along the technology supply chain that govern for outcomes like safety.

Again, in biotechnology, you've got an example of this where DNA synthesis companies voluntarily self-initiated schemes for screening customer orders so that they were screening for whether customers were ordering for malicious use purposes. The state eventually caught up. And a couple of years after most DNA synthesis companies had been doing this in the U.S., it became U.S. state policy. But that was a private sector-led initiative.

Product standards are another really good example where private firms have consistently led at the start for figuring out what a good product looks like when it's on the market.

Cryptographic products, the first wave of them, is a really good example of this. You had firms like IBM and a firm called RSI Security Inc., in particular, do a lot of early-stage R&D to ensure that strong encryption protocols made it onto the market and took up a fair amount of global market share. And for the large part, that ended up becoming American standards for cryptographic products, which ended up scaling across the global markets.

So those are two examples of many examples of ways in which private firms can lead on the implementation of governance mechanisms.

![](https://images.ctfassets.net/ohf186sfn6di/Wx52C5oyqcoMO9371kRPE/12cbb7572f38f2f1b080827ef1eaebef/jade_slide_14.PNG)

The second really salient function that they play is in compliance. So making sure that companies are doing what they do. There are a lot of examples in this space of climate change, in particular where firms have either sponsored or have directly started initiatives that are about disclosing the things that they're doing to ensure that they are in line with commitments that are made on the international scale. Whether that's things like divestment, or disclosing climate risk, or carbon footprints, or various rating and standards agencies, there is a long list of ways in which the private sector is delivering on this compliance governance function voluntarily, without necessarily needing regulation or legislation.

So firms already do this governance thing. And all that we have to think of is how can they lead on that and shape it in a more preemptive way.

And the third reason to think that firms could do this voluntarily is that, at the end of the day, particularly for transformative artificial intelligence scenarios, firms rely on the world existing. They rely on markets functioning. They rely on stable sociopolitical systems. And if those don't end up being what we get because we didn't put in robust governance mechanisms, then firms have all the more reason to want us to not get to those futures. And so, for an aspirationally long-term thinking firm, this would be the kind of incentive that would lead them to want to lead preemptively on some of these things.

So these are all reasons to be hopeful, or to think at least, that firms can do and can be incentivized to lead on AI governance.

![](https://images.ctfassets.net/ohf186sfn6di/3aBi5cF4bmBdAMJGBP3Aek/257d867d0d6911ddb2f6e30fc7c6557e/jade_slide_6.PNG)

So here are the three propositions again. You've got states who are ill-equipped to lead on AI governance. You've got private AI labs who have the capacity to lead. And finally, you've got reasons to believe that private AI labs can lead in a way that is prosocial.

Now am I saying that private actors are all that is necessary and sufficient? It wouldn't be an academic talk if I didn't give you a caveat, and the caveat is that I'm not saying that. It's only that they need to lead. There are very many reasons why the private sector is not sufficient, and where their incentive structures can diverge from what prosocial outcomes are.

More than that, there are some governance functions which you actually need non-private sector actors to play. They can't pass legislation, and then you often need like a third party civil society organization to do things like monitoring compliance very well. And the list goes on of a number of things that private sector can't do on their own.

So they are insufficient, but they don't need to be sufficient. The clarion call here is for private sector to recognize that they are in a position to lead on demonstrating what governing artificial intelligence can look like if it tracks technological progress in a nuanced, adaptive, flexible way, if it happens at a global scale and scales across jurisdictions easily, and finally avoids costly conflict between states and firms, which tends to precede a lot of costly governance mechanisms that are ineffective being put in place.

So firms and private AI labs can demonstrate how you can lead on artificial intelligence governance in a way that achieves these kinds of outcomes. The argument is that others will follow. And what we can look forward to is shaping the formative stages of an AI governance regime that is private sector-led, but publicly engaged and publicly accountable.

![](https://images.ctfassets.net/ohf186sfn6di/2CMgEPWgeJ0HFSPewNQYKr/ed9612653ca8344e0bb054b32c29cd3d/jade_slide_15.PNG)

Thank you.

## Questions

_Question_: Last time you spoke at EA Global, which was just a few months ago, it was just after the Google engineers' open letter came out saying, "We don't want to sell AI to the government". Something along those lines. Since then, Google has said they won't do it. Microsoft has said they will do it. It's a little weird that rank and file engineers are sort of setting so much of this policy, and also that two of the Big Five tech companies have gone so differently so quickly. So how do you think about that?

_Jade_: Yeah. It's so unclear to me how optimistic to be about these very few data points that we have. I think also last time when we discussed it, I was pretty skeptical about how effective research communities can be and technical researchers within companies can be in terms of affecting company strategy.

I think it's not surprising that different companies are making different decisions with respect to how to engage with the government. You've historically seen this a lot where you've got some technology companies that are slightly more sensitive to the way that the public thinks about them, and so they make certain decisions. You've got other companies that go entirely under the radar, and they engage with things like defense and security contracts all the time, and it's part of their business model, and they operate in the same sector.

So I think the idea that you can have the private sector operate in one fashion, with respect to how they engage with some of these more difficult questions around safety and ethics, isn't the way it pans out. And I think the case here is that you have some companies that can plausibly care a lot about this stuff, and some companies that really just don't. And they can get away with it, is the point.

And so I think, assuming that there are going to be some leading companies and some that just kind of ride the wave if it becomes necessary is probably the way to think about it, or how I would interpret some of these events.

_Question_: So that relates directly, I think, to a question about the role of small companies. Facebook, obviously, is under a microscope, and has a pretty bright spotlight on it all the time, and they've made plenty of missteps. But they generally have a lot of the incentives that you're talking about. In contrast, Cambridge Analytica just folded when their activity came to light. How do you think about small companies in this framework?

_Jade_: Yeah. That's a really, really good point.

I think small companies are in a difficult but plausibly really influential position. As you said, I think they don't have the same lobbying power, basically. And if you characterize a firm as having power as a result of their size, and their influence on the public, and their influence on the government, then small companies, by definition, just have far less of that power.

There's this dynamic where you can point to a subset of really promising, for example, startups or up-and-coming small companies that can form some kind of critical mass that will influence larger actors who, for example, in a functional, transactional sense, would be the ones that would be acquiring them. E.g., like DeepMind had a pretty significant influence on the way that safety was perceived within Google as a result of being a very lucrative acquisition opportunity, in a very cynical framing.

And so I think there are ways in which you can get really important smaller companies using their bargaining chips with respect to larger firms to exercise their influence. I would be far more skeptical of small companies being influential on government and policy makers. I think historically it's always been large industry alliances or large big companies that get summoned to congressional hearings and get the kind of voice that they want. But I think certainly, like within the remit of private sector, I think small companies, or at least medium-size companies, can be pretty important, particularly in verticals where you don't have such dominant actors.

_Question_: There have been a lot of pretty well-publicized cases of various biases that are creeping into algorithmic systems that sort of can create essentially racist or otherwise discriminatory algorithms based on data sets that nobody really fully understood as they were feeding it into a system. That problem seems to be far from solved, far from corrected. Given that, how much confidence should we have that these companies are going to get these even more challenging macro questions right?

_Jade_: Yeah. Whoever you are in the audience, I'm not sure if you meant that these questions are not naturally incentivized to be solved within firms. Hence, why can we hope that they're going to get solved at the macro level? I'm going to assume that's what the question was.

Yeah, that's a very good observation that within... unless you have the right configuration of pressure points on a company, there are some problems which maybe haven't had the right configuration and so aren't currently being solved. So put aside the fact that maybe that's a technically challenging problem to solve, and that you may not have the data sets available, etc. And if you assume that they have the capacity to solve that problem internally but they're not solving it, why is that the case? And then why does that mean that they would solve bigger problems?

The model of private sector-led governance requires, and as I alluded to, pressure points that are public-facing that the company faces. And with the right exertion of those pressure points, and with enough of those pressure points translating into effects on their bottom line, then that would hopefully incentivize things like this problem and things like larger problems to be solved.

In this particular case, in terms of why algorithmic bias in particular hasn't faced enough pressure points, I'm not certain what the answer is to that. Although, I think you do see a fair amount more like things like civil society action and whatnot popping up around that, and a lot more explicit critique about that.

I think one comment I'll say is that it's pretty hard to define and measure when it's gone wrong. So there's a lot of debate in the academic community, for example, and the ProPublica debate comes to mind too, where you've got debates literally about what it means for this thing to have gone fairly or not. And so that points to the importance of a thing like governance where you've got to have common context, and common knowledge, and common information about your definitions simply, and your benchmarks and your metrics for what it means for a thing to be prosocial in order for then you to converge on making sure that these pressure points are exercised well.

And so I think a lot of work ahead of us is going to be something like getting more granularity around what prosocial behavior looks like, for firms to take action on that. And then if you know basically what you're aiming for, then you can start to actually converge more on the kind of pressure points that you want to exercise.

_Question_: I think that connects very directly to another question from somebody who said, basically, they agree with everything that you said, but still have a very deep concern that AI labs are not democratic institutions, they're not representative institutions. And so will their sense of what is right and wrong match the broader public's or society's?

_Jade_: I don't know, people. I don't know. It's a hard one.

There are different ways of answering this question. One is that it's consistently a trade off game in terms of figuring out how governance is going to pan out or get started in the right way. And so one version of how you can interpret my argument is something like, look, companies aren't democratic and you can't vote for the decisions that they make. But there are many other reasons why they are better. And so if you were to trade off the set of characteristics that you would want in an ideal leading governance institution, then you could plausibly choose to trade off, as I have made the case for trading off, that they are just going to move faster and design better mechanisms. And so you could plausibly be able to trade off some of the democratic elements of what you would want in an institution. That's one way of answering that question.

In terms of ways of making... yeah, in terms of ways of aligning some of these companies or AI labs: so aside from the external pressure point argument... which if I were critiquing myself on that argument, there are many ways in which pressure points don't work sometimes and it kind of relies on them caring enough about it and those pressure points actually concretizing into kind of bottom line effects that actually makes that whole argument work.

But particularly in the case of AI, there are a handful of AI labs that I think are very, very important. And then there are many, many more companies that I think are not critically important. And so the fact that you can identify a small group of AI labs makes it an easier task to both kind of identify at almost like an individual founder level where some of these common views about what good decisions are can be lobbied to.

And I think it's also the case that there are a number of AI labs... we're not entirely sure how founders think or how certain decision makers think. But there are a couple who have been very public and have gone on record about, and have been pretty consistent actually, about articulating the way that they think about some of these issues. And I think there is some hope that at least some of the most important labs are thinking in quite aligned ways.

Doesn't quite answer the question about how do you design some way of recourse if they don't go the way that you want. And that's a problem that I haven't figured out how to solve. And if you've got a solution, please come tell me.

Yeah, I think as a starting point, there's a small set of actors that you need to be able to pin down and get them to articulate what the kind of mindset is around that. And also that there are an identifiable set of people that really need to buy in, particularly to get transformative AI scenarios right.

Why companies should be leading on AI governance _ Jade Leung _ EA Global - London 2018-by Centre for Effective Altruism-video_id AVDIQvJVhso-date 20190207.md