# Interview with AI Researchers NeurIPSorICML_lgu5f by Vael Gates

**Interview with lgu5f, on 3/22/22**

**0:00:02.5 Vael:** Alright, my first question is, can you tell me about what area of AI you work on in a few sentences?

**0:00:08.4 Interviewee:** Yeah. So I work on privacy in AI. I've been working on differential privacy, which is sort of the de facto standard for giving statistical privacy in analytics and AI. I've been working on differentially private synthetic data, so coming up with algorithms that generate synthetic versions of datasets that mimic the statistics in those datasets without revealing any information about the users and the data itself. And more broadly, I also just work on differential privacy for analytics, so it's not specifically AI, but it's still like algorithm design with privacy.

**0:00:55.3 Vael:** Cool, great, thanks. And my next question is, what are you most excited about in AI, and what are you most worried about? In other words, what are the biggest benefits or risks of AI?

**0:01:05.7 Interviewee:** I think that AI has been most helpful in the little things. When I pull up my phone, and it's like, "Hey, this is your most used app in this location," recommender systems like that, or small tweaks that help my daily life, that's what I'm most excited about. I think I'm most worried about AI being used in applications where explainability and fairness are going to be important, or privacy. This is stuff I see red flags in. I'm worried about an insurance company putting a neural net into some important decision-making system and then not being able to analyze why it made a decision that it did, or understanding if it's being unfair.

**0:02:11.6 Vael:** Great, yeah, that makes sense. And then focusing on future AI, so putting on a science fiction forecasting hat, say we're 50 plus years into the future, so at least 50 years in the future, what does that future look like?

**0:02:30.3 Interviewee:** I'm sort of pessimistic about how advanced AI can get. I think that the trend is going to be that we're going to see smaller models that are more bespoke for the problem domain that it's trying to solve. So instead of these like GPT-3 trillion parameter-sized models, I think that we're going to start moving back towards stuff that can run more easily on the edge. That doesn't require as much energy and time to train and doesn't require as much data. I think that AI becomes more ubiquitous, but in a way that's easy for us to compute with. So it's just more AI running everywhere.

**0:03:17.6 Vael:** Yeah, what drives that intuition?

**0:03:20.7 Interviewee:** One is\-- partially concern with large models consuming too much energy and, of course, climate change is one of the chief things that we should be worried about. The other thing is\... I've seen some experiments that come out of GPT-3, and they're cool as toy problems, or it's cute program synthesis and stuff like that, but I don't see that really being used in production. It's one thing for a research organization to come up with those experiments and say, "Hey, we were able to, I don't know, use this to beat a goal player," but you look at the details of it, and really this gigantic model also had a tree search algorithm, that was a big benefit. I think just\... keeping it bespoke, like CNNs just do so well, and I think that that's for a reason, so that's sort of the intuition I have. If we keep it tight to the problem domain, I've seen it do better. Domain expertise has helped a lot.

**0:04:37.5 Vael:** And then just a quick follow-up on the climate change thing, is the idea that, current systems are using too much energy and this is causing increased climate change?

**0:04:50.0 Interviewee:** Yeah, I think that we all just need to reduce how much energy we're using, because, unless we're sure that a lot of it is coming sustainably, we should be concerned about how much energy we're using. And training these trillion parameter models requires a lot of energy. It requires a lot of hardware. That hardware does not come for free. There's a manufacturing process that goes into that, building up the data centers that are training, that are hosting all of this, and then replacing the hard drives, replacing the GPUs when they get stale, so there's just a whole bunch of life cycle impacts from training models that I think are really coming up because we're seeing people doing these studies on blockchain because that tends to burn through GPUs and hard disks faster than training machine learning models, but it's sort of the same impact.

**0:05:47.4 Vael:** Interesting. Cool. Well, this next thing is more of a spiel, and it is quite close to what you've been talking about with where AI will go in the future and how big it will get. So people talk about the promise of AI and they mean many things by that, but one thing they may mean is a very general capable system, such that they'll have the cognitive capacity to replace all human jobs, all current day human jobs. So whether or not we choose to replace human jobs is a different question, but having the cognitive capacity to do that, and so I usually think about this in the frame of 2012 when we have AlexNet, deep learning revolution, and then 10 years later, here we are, and we've got the GPT-3 system. Which, like you said, have some weirdly emerging capabilities, so it can do some text generation, and some translation, and some coding, and some math and such. And we might expect that if we continue with all this human effort that's been going into this kind of mission of getting more and more general systems, and we've got nations competing, and we've got corporations competing, and we've got all these young people learning AI and like, maybe we'll see algorithmic improvements at the same pace we've seen hardware improvements, maybe we get optical or quantum. So then we might actually end up scaling the very general systems or like you said, we might not, we might have hit some sort of ceiling or require a paradigm shift or something. But my question is, regardless of how we get there, do you ever think we'll have very general AI systems like a CEO AI or a scientist AI? And if so, when?

**0:07:13.3 Interviewee:** Oh, that's a good question. I tend to be more pessimistic about this. It depends what you mean by AI, I guess in this sense, right? Is it sort of a decision-making system that's just doing Pareto optimal decision-making. Does it have to be trained?

**0:07:52.6 Vael:** Yeah. What I'm visualizing here is any sort of decision-making system, any sort of system that can do things like multi-step planning, that can do social modeling, that can model itself\-- modeling other people modeling it. Have the requirements such that I can be like, "Alright, CEO AI, I want you to maximize profit plus constraints." So very capable cognitive systems. I don't require right now that they're embodied necessarily because I think robotics is a bit behind, but like cognitive capacities.

**0:08:25.1 Interviewee:** Got it. I think we're still a solid century behind that. Yeah, I don't know. I just feel like it's still a solid century behind because that might require a huge paradigm shift in how we're training our models. Yeah, I don't think we've even thought about modeling something with a state space that large, right?

**0:08:58.9 Vael:** Yeah, it's like reality.

**0:09:00.7 Interviewee:** Yeah, exactly. I've seen cool stuff where there's like, you hand over a floor plan or something, and then you say, "Hey, where did I leave my keys?" And this AI is able to jog through, navigate through this floor plan and then say you left it in the kitchen. But I still think that something like a CEO AI is still going to be like 100 years out. Yeah.

**0:09:29.5 Vael:** Interesting. Yeah, that is interesting, because it seems like what you were saying earlier, is like, I think we'll get more bespoke systems, less large scale systems. And I'm like \[humorously\]: "What if we have a very large-scale system, that is very\..." \[interviewee laughs\] Yeah. How does that square?

**0:09:43.8 Interviewee:** Between having a bespoke system versus\...

**0:09:47.9 Vael:** Or like how the future will develop or something. I think that that's like\... maybe kind of likely that there will be at least some companies like OpenAI and DeepMind that'll just keep pushing on the general system thing. But I don't know.

**0:09:58.4 Interviewee:** Yeah, I feel like there's always going to be billions of dollars poured into that kind of research. But at the end of the day, I think that we'll see more impact if we make quality of life improvements that make jobs easier. Of course, there'll be a whole bunch of automation, right, but I don't know if we'll ever get to a point where we can leave decision-making entirely to AI. Yeah, because\... I think also just as society, we haven't thought about what that means. I think Mercedes just recently said that Mercedes will take responsibility if one of their driverless cars hits someone, which is a huge step, right, but we haven't gone close to already deciding who has liability if an insurance company messes up, right, with their premium setting algorithms.

**0:11:09.5 Vael:** Yeah, when I think about the future, I'm like, I think we might get technological capacities before we get good societal understanding or regulation.

**0:11:17.1 Interviewee:** Likely. Likely. Because that's what's happening in fairness and in privacy. With fairness regulations, like you have in Australia and a whole bunch of places, where it just says, Okay, you don't pass in some protected attributes and your algorithm is not going to be racist, homophobic, transphobic, whatever, sexist. And it's like, Okay, cool, now that you've purged the data of all these attributes, there's no way for us to measure whether it's actually being any of those things. But it's required by policy. There's also the same thing with privacy, where it's like, Okay, if you just remove these attributes and you remove their names, we can't tell who it is, but nobody else has watched the exact five last movies I've watched on Netflix, or the last three things I've liked on Twitter or Facebook or whatever. So yeah, for sure, we'll be lagging societally, but hopefully with this timespan I have in mind of it's in a century, we'll be thinking about these things if we're on the cusp of it, and people will be raising alarm bells about it like, Hey, maybe we should start thinking about this.

**0:12:34.1 Vael:** Yeah, that makes sense. I know there's a substantial number of researchers who think there's some probability that we'll get it substantially sooner than that. And maybe even using the current deep learning paradigm, like GPT-whatever, 87, 7, GPT-13. I don't know. But that this system will like work. I don't actually know if that's true. It's hard to predict the future. Yeah, so my next question is sort of talking about whenever we get these very advanced systems, which again, who knows, you said like maybe 100 years, some people think earlier, some people think much later. So imagine we're in the future, and we're talking about the CEO AI, and I'm like, Okay, CEO AI, I wish for you to maximize profits\-- this is where humans are like the shareholder type thing\-- and try not to run out of money and try not to exploit people and try to avoid side-effects. Currently, this is technologically challenging for many reasons, but one of the reasons I think that might continue into the future is that we're currently not very good at taking human values and preferences and goals and putting them into a mathematical formulations that we could optimize over them. And I think this might be even harder to do in the future as we get more powerful systems and that are optimizing over larger\-- reality. Larger optimization spaces. So what do you think of the argument, "Highly intelligent systems will fail to optimize exactly what their designers intended them to, and this is dangerous?" \[pause\] So they'd be doing what we tell them to do rather than what we intended them to do.

**0:14:12.2 Interviewee:** \...Yeah, I think that that's most likely. Because\... Well, it depends what level of self-awareness or cognitive awareness this AI has to understand what we intended versus telling them. If it figured out the state space was\... If it can model what the ramifications of what it does are on itself, then ideally it would figure out what we intended for it to do, not what we told it to do it. So it's one thing if it goes off the rails and says say, actually, this is the only way to solve\-- That's a very Asimov-esque dystopia, right. But hopefully, if it can model that and say like, Oh actually, you know, the humans will delete me if I do something that's exactly what they told me to do and not what they intended for me to do, then ideally we'd be past that point.

**0:15:15.3 Vael:** I see, yeah. Cool. Let me try to engage with that. So I'm like, Alright, where we've told the CEO AI that I want you to maximize profit, but not have side-effects, and it's like, Okay, cool. Side-effects is pretty undefined, but maybe it has a very good model of what humans want, and can infer exactly what the humans want and can model things forward, and will know that if it pollutes, then the humans will be unhappy. I do think if you have a great enough model of the humans in AI, and AI is incentivized to make sure to do exactly what the humans want, then this is not a problem.

**0:16:00.6 Interviewee:** Right. The risk, of course, is that it figures out all the legal loopholes, and it's like, Oh great, I've figured out exactly what I need to do to not be held morally culpable for what choices I've made and get off scot-free, which is also a huge risk.

**0:16:19.2 Vael:** Yeah, yeah. So I think with the current day systems, they're much dumber, and so if you're like, Alright AI, I want you to get a lot of points then maybe it will get a lot of points through some little side loop instead of winning the game. That's because we haven't specified what we wanted exactly, and it can't infer that. But you're saying as we get more advanced, we will have systems that can at least know what humans want them to do, whether or not they follow through with that is a different question.

**0:16:51.7 Interviewee:** Yeah. Yeah, and at some point, there's still the huge element of what the designer of the algorithm put in. There is some choice on the loss function. Is it average, is it worst case? Is it optimizing for the highest likelihood, the worst likelihood, I feel like that would also be a huge change in how it does it. So it's not going to be fully up to the algorithm itself, they're still going to be human choices in building out that algorithm that determine how it interacts with other humans.

**0:17:31.1 Vael:** Yeah, this is in fact the whole question, I think. What do you put in the loss function such that it will do whatever things you want to do. Yup. Yeah, okay, cool. So I have a second argument on top of that one, which I think is pretty relevant. So say we have the CEO AI, and it has pretty good models of humans, and it can model humans modelling, and it does multi-step planning. And because we figured, humans figured that we should have some safety mechanisms, so maybe we don't let the AI make any big decisions unless it passes that decision through us. And so they've asked the AI for a one-page memo on this upcoming decision. And the AI is like, Cool, well, I obviously have a lot of thinking and information that I can't put everything in a one-page memo, and so humans are expecting me to condense it. But I notice that sometimes if I include some types of information in this memo, then the humans will shut me down and that will make me less likely to be able to achieve the goal they programmed in. So why don't I just leave out some information such that I have a higher likelihood of achieving my goal and moving forward there. And so this is a story that's not like the AI has self-preservation built in it, but rather as an agent optimizing a goal, a not perfect goal, then it ends up having an instrumental incentive to stay alive and to self-preserve itself. So the argument is, what do you think of the argument, "Highly intelligent systems will have an incentive to behave in ways to ensure that they are not shut off or limited in pursuing any goals, and this is dangerous?"

**0:19:08.7 Interviewee:** I think we already see this with not highly intelligent systems. There was this paper from 2019 where they analyzed the preventative healthcare recommendations from some insurance company algorithm setup. And they found that it consistently would recommend people of color to go for preventative healthcare less. So they were looking through it, and they realized that the algorithm is optimizing to minimize the dollar spent by the patient, and so it figured out that, Hey, folks who are a part of majority communities go for preventative healthcare and they save dollars by getting preventative healthcare versus folks in minority communities tend to not get preventative healthcare, but most of them end up not ever getting specialist care either, so it's actually more dollars saved if we just don't give them preventative healthcare. I think that that could be pretty likely. Again, if we have this highly intelligent system that's able to model so much, then we should be kinda skeptical of what comes in the memos. But yeah, I'd say that that's possible, that it just gets stuck on one of those loops, like, Oh, this is best in\...

**0:20:44.7 Vael:** Yeah, interesting. Yeah, so the example you gave seems to me like it's some sort of alignment problemm where we've tried to tell what we want, which is presumably like good healthcare for everyone that's sort of cheap. And instead it is doing something not what we intended it to do, like, whoops, that was like a failure. We've put in a failure in terms of the loss function that we're optimizing. Something that's close, but not quite the problem. Then yeah, I think this argument is, as we get very intelligent systems, if the loss function that we put in is not exactly what we want, then we might have it optimizing something kind of close, and then it will be incentivized to continue pursuing that original goal, and in fact, maybe optimizing against humans trying to change it. Which is interesting. So, if one buys this argument of instrumental incentives of an agent optimizing what a goal it puts in, then you have ideas like an agent that is optimizing for not being shut down. So that can involve deception and acquiring resources and influence and also improving itself, which are all things that are good things that are able to help you better achieve goals. Which feels pretty worrying to me if this is the case, because if this is one of the default ways that we're developing AI, then when we actually get advanced AI and then we'll maybe have a system that is as or more intelligent than us optimizing against humans. And I'm like, "Wow, that seems like real bad."

**0:22:14.0 Interviewee:** Yeah. Yeah, that's a nightmare scenario, which is Matrix-esque, right? Yeah.

**0:22:21.5 Vael:** Yeah. What is the bad AI in the Matrix doing? Is there a bad AI in the Matrix?

**0:22:28.1 Interviewee:** Yes. So the bad AI\... It's sort of the same idea as the Terminator, where the Skynet of the Matrix realizes that the best way to avoid conflict in humanity is to sort of keep humanity in a simulation. I think the robots and the humans end up fighting a war, and then they just plug the humans into the matrix, keeping them quelled that way and making the broader argument that living in a simulation is nicer than the world outside. So the AI is technically also doing something good for humanity and self-sustaining, and keeping human population alive.

**0:23:09.2 Vael:** Interesting. Yeah. Cool, alright. So my interest, in general, is looking at long-term risks from AI. Have you heard of the\-- well, okay, I'm sure you've heard about AI safety. What does AI safety mean for you, for the first question?

**0:23:31.0 Interviewee:** Um\...

**0:23:32.3 Vael:** Or have you heard of AI safety?

**0:23:34.8 Interviewee:** No, not really.

**0:23:35.7 Vael:** Oh, interesting, cool!

**0:23:37.5 Interviewee:** So can you tell me more?

**0:23:41.3 Vael:** Yeah. So, AI safety means a bunch of different things to a bunch of different people, it's quite a large little field. It includes things like surveillance, autonomous weapons, fairness, privacy. Things like having self-driving cars not kill people. So anything that you could imagine that could make an AI safe. It also includes things that are more in what is currently called AI alignment. Have you heard of that term before?

**0:24:11.1 Interviewee:** No.

**0:24:12.3 Vael:** Cool. AI alignment is a little bit more long-term focused, so imagining like as we continue scaling systems, how do we make sure that the AIz continue to do what humans intend them to do, what humans want them to do, don't try to disable their off-switches just by virtue of optimizing for some goal that's not what we intended. Trying to make sure that the AIs continue to be aligned with the humans. Where one of the definitions of alignment here is building models that represent and safely optimize hard-to-specify human values. Alternatively, ensuring the AI behavior aligns with the system designer intentions. So trying to prevent scenarios like the one you brought up where the designer puts in a goal that's not quite what the humans want, but this can get much worse as systems get more and more powerful, one can imagine. Yeah, so there's a community working on these sorts of questions. Trying to figure out how we could get AI aligned. Because people are like, Well, can't you just put in an off switch, but this AI will probably be deployed on the Internet, and it can replicate itself presumably, if it's smart enough, and also it may have an incentive to disable the off switch if we do things by default. But some people are like, oh well, how do we solve that? How do we make it so that it doesn't have an incentive to turn itself off? And one group is like, Oh well, instead of having AI optimizing towards one goal singularly, in which case it has an incentive to try to stop people from achieving its goal, one thing you could do is have it build in uncertainty over the reward function, such that the AI wants to be corrected and wants to be switched off so that it has more information about how to achieve the goal. And this means that AI no longer has an incentive to want to prevent itself from being switched off, which is cool. And you can sort of like build\... that's like an alignment solution you can build in. Another thing that people talk a lot about is if\... we should obviously have human feedback if we're trying to get an AI to do what humans want. So how do we build in human feedback when the systems are more and more intelligent than us and dealing with really big problems that are like, should we do this nuclear fusion reactor? I don't know, the human doesn't really know. What are all the considerations? Another question they tackle is, how do you get honest reporting out of AI? How do you build a loss function such that it's incentivized to your honest reporting? And how do you get an AI in general to defer to humans kind of always while still being agentic enough to do things in the world?

**0:26:39.3 Interviewee:** Got it.

**0:26:40.4 Vael:** How does all that sound to you?

**0:26:43.0 Interviewee:** Very important. I feel like AI safety is something we're brushing up against the sharp edges of right now. But a lot of what AI alignment is looking into, I think, would also just apply to how we think about building these larger models. Because if that's the goal of these large experiments, I would hope that they're thinking about these right now, given their funding and the immense amount of time and effort they have that goes into it. Yeah. This is really interesting.

**0:27:25.3 Vael:** Yeah. Most researchers are working on capabilities, which is what I just call moving the state of the field forward. Because AI is actually really hard, turns out, and so lots of people working on that, and there's a lot of money in the space too, especially with applications. And then there's a smaller community of people who are working on AI safety, more like short-term AI safety, like there's plenty of problems with today's systems. There's an even smaller community of people who are working on long-term safety. They're kind of the group that I hang out with. And I'm worried that the group of people working on long-term safety will continue to be significantly smaller than the capabilities people, because while there's more money in the space now, because a couple of very rich people are concerned about it like Elon Musk, famously. But generally, it's hard work. It's like anticipating something in the future. Some people think it's a very far future. It's hard to speculate about what will go on, and that's also just kinda like a difficult problem, alignment. It might not be trivial.

**0:28:25.5 Interviewee:** Yeah, and I sort of see parallels with nuclear research that happened like 100 years back, right? Or 80 years back, around the time of the Manhattan Project, where most of the scientists in New Mexico were focused on capabilities, like, "How do we make this thing go boom." There was a small group of them who were worried about the ramifications of, "What would happen if we did drop our first nuclear bomb and stuff like that?" And of course, there's always been the same thing about space exploration. So with nuclear stuff, I think that we ended up, again, with more worries about nuclear safety than nuclear alignment. So, "What do we do to keep reactors safe?" Instead of thinking about like, "Okay, should we pursue nuclear technologies where countries cannot enrich uranium and make warheads with it? What if we went with thorium reactors instead?" And again, it's been more focused on capabilities and safety than alignment.

**0:29:32.9 Vael:** Yeah, I don't think we really need alignment with nuclear stuff because nuclear stuff doesn't really have intelligence per se. It's not an optimizer, is what I meant. Yeah. So you're not trying to make sure that the optimizer does whatever your goals are.

**0:29:51.5 Interviewee:** Yeah. It's humans at the end of the day. Yes, yes, yes, that's right.

**0:29:55.2 Vael:** Yeah, yeah. I also think the Manhattan Project is a very good comparison here. Or nuclear stuff in general. There's plenty of issues in today's world, there's so many issues in today's world. But the reason I focus on long-term risks is because I'm worried about existential risks. So what happens if all of humanity goes extinct. I think there's a number of things that are possibilities for existential risk. I think advanced AI is maybe one of the highest ranked ones in my head, where I'm like, "Well, if we have an AI system that is by default not completely aligned to human values and is incentivized against us, that's bad. Other ways that I think that we can have AI disasters are like AI assisted war, misuse, of course, and loss of control and correlated failures. So what if we use AI to automate food production and manufacturing production, and there's some failure partway along the line and then correlated failure? Or what if it results in some sort of pollution? Are not really sure who's in charge of the system anymore, and then we're like, "Oh, well, now, there's some sort of poison in the air," or there's like something in the water, we'd like messed something up, but we don't really know how to stop it. Like coordination failures is another thing that I think that can happen even before we do very advanced AI. So generally, I'm worried about AI.

I think nuclear stuff is also\... You can kill a lot of people with nuclear stuff. Biological weapons\... tou can do synthetic bio. There was that paper that came out recently that was like, "Well, what happens if you just put a negative sign on the utility function to generate cool drugs?" Then you get poisonous drugs, amazing. Yeah, that was a paper that just came out in Nature, I think maybe two weeks ago, made quite a splash. So I think if we had something that's much more deadly, is harder, takes longer to appear, spreads faster than COVID, I'm like, hm, that's\... You may be able to\... If people have bunkers, that's okay, and if people are very distant, maybe it's okay, but like\... I don't know. It's not good. And then climate change also. I think that one will take much longer to kill humans. In terms of the time scales that I'm thinking of. We're looking at 3 degrees of warming in the next 100 years, or is it 50? I don't quite remember, I think it's like we've got a bit more time, and I kind of think that we might get technical solutions to that if we wait long enough, if we advance up the tech tree, if it takes a couple hundred years. So these are what I think of when I think of existential risks.

And the Manhattan Project is interesting, tying it back. There was some risk that people thought that if you deployed a nuclear bomb, then it would set the atmosphere on fire, and thus result in nuclear winter and kill everyone. And it was a very small percentage, people thought it won't happen, and they did it anyway, and I'm like, "Oh boy, I don't know how many\... get lucky of those we have." Because AI might also go perfectly fine. Or it might not. And I think there's a decent\... In a study in 2017 by Grace et al., where they were asking researchers from NeurIPS and ICML how likely they thought it was that AI would have extremely bad consequences, and the median answer was like 5% or something, and I was like, "5%?! That's like real high, man."

**0:33:03.0 Interviewee:** Yeah. \[chuckle\] And that's sort of an indication of what percentage of ICML and NeurIPS researchers are working on like fairness and privacy. I feel would align pretty closely with that, yeah.

**0:33:15.9 Vael:** Oh, interesting. Yeah. Yeah, so very bad outcomes could be many different things. I think not many people are thinking about what happens if you literally kill all humans because it's a weird thing to have happened, but I think we're in a very weird time in history where 10,000 years ago everything was the same from lifetime to lifetime, but now we have things like nuclear weapons where one person or a small chain of people can kill like so many more humans than previously.

**0:33:42.9 Interviewee:** Yeah. And yeah, speaking of tech tree, one sort of linchpin that would really accelerate if we can get new AI, highly intelligent AI, would be if quantum computing becomes feasible. And we're actually able to run AI on quantum computers, then I think we're way closer to actually having a highly intelligent AI because that's an infinite state space right there.

**0:34:22.3 Vael:** Yeah. Yeah, yeah, I know very little about quantum myself, because I was like, "What are the hardware improvements that we could see?" And people are like, "Optical, maybe coming in optimistically in five years from now," and I was like, "Okay, that's soon." And then there's quantum where people are like, "That's significantly further away," and I'm like, "Okay." And then software improvements also. So there's just been a lot of progress in AI recently, and we're training all the young people, and there's China-US competition, and there's just a ton of investment right now, where I'm like, "We're moving fast." So interesting.

**0:34:56.4 Interviewee:** Yep, yep, yep.

**0:34:57.8 Vael:** Yeah. Well, you are very agreeable to my point of view here. \[laugh\]

**0:35:06.3 Interviewee:** Yeah, I'm just pessimistic, and I've watched too much sci-fi dystopia, I guess. One of the downsides of\... It's great to democratize AI, but if your systems that say like, "Hey, just upload your dataset, and we'll give you good AI at the end of it," if it's not at least asking you questions about like, "What sort of fairness do you want? What should the outcome be?" Most humans themselves are going to be thinking like, "Oh, give me optimal, minimum cost or something like that." Humans already don't factor human values in necessarily when deciding what to do. So I'm just pessimistic about how well we can do it. Like 50 years ago, people thought that we'd be in space colonies by now, but sadly, we're not.

**0:36:10.5 Vael:** Yeah, very hard to predict the future. Okay, so my second to last question is: what would convince you or motivate you to work on the safety type of areas?

**0:36:25.8 Interviewee:** If I saw that this was coming much sooner. I think the fact that I'm seeing this as like 100 years out, a lot of smarter than me researchers will hopefully be concerned enough about it.

**0:36:44.3 Vael:** Yeah. How soon would be soon enough that you're like, "Oh, that's kind of soon?"

**0:36:47.7 Interviewee:** Fifty.

**0:36:49.7 Vael:** Fifty?

**0:36:50.4 Interviewee:** Yeah. If it's fifty, then that's in my lifetime. And I'd need to start worrying about it. That would be a bigger\... The existential risk of that would be much higher than the risk I see of just AI safety in day-to-day life, so that's sort of how I'm weighing it. So for me, it's like: 100, it's like AI safety matters more. Yeah.

**0:37:15.0 Vael:** Yep, that seems very reasonable to me. I'm like, "Yep, it seems right." Cool. And then my last question is, have you changed your mind on anything during this interview, and how was this interview for you?

**0:37:26.9 Interviewee:** Have I changed my mind? I'm definitely wondering if I've overestimated when highly intelligent AI could come through. (Vael: "I can send you some resources so you can make your own opinion, etcetera.) Yeah. But\... Otherwise, I don't think I've changed my opinion. I still feel pessimistic, and I hope that we start moving towards smaller AI that solves one problem really well, and we don't just think that like, "Hey, it's a perceptron. It figured out that this was the digit 9, and hence it can figure out a whole bunch of these other things." I hope that we don't start barreling down that track for too much longer. Yeah.

**0:38:21.5 Vael:** And when you say pessimistic, you mean like pessimistic about society and not pessimistic about the pace of AI? Or like societal impacts?

**0:38:28.3 Interviewee:** About both. I think we tend to put stuff out without really, really considering the consequences of it. But also I think AI has done a bunch, but it requires a lot of energy and a lot of funding that I'm not sure necessarily is going to stay up unless we start seeing a lot bigger breakthroughs come through.

**0:38:57.5 Vael:** Interesting. Yeah, I kind of think that the applications will keep it afloat for quite a while, and also it sounds like we might be entering a race, but I don't know.

**0:39:05.5 Interviewee:** True, yeah. Yeah, maybe that's what has changed in my mind through this interview, is like, "Okay, this is probably\...Things are just going to keep going bigger."

**0:39:18.9 Vael:** I think it's quite plausible.

**0:39:22.2 Interviewee:** Yeah, otherwise, interview was great. This was really interesting. For example, the stuff you brought up on\... I would love it if you could send me papers on the\-- (Vael: "I would love to.") \--the uncertainty\... You were talking like, programming a way to get out the AI turning off its own off-switch. I would love to read more of these alignment papers, they sound really cool.

**0:39:51.2 Vael:** Awesome! Well, I'm excited. I will send you a whole bunch of resources probably, and I advise you\-- it might be overwhelming so pick only the stuff that is good, and I'll bold some interesting stuff.

**0:40:00.9 Interviewee:** That would be super helpful. And hope you don't mind my sending following up questions.

**0:40:05.7 Vael:** No, no. Again, lovely. Very happy to do that.

**0:40:09.4 Interviewee:** Yeah. Awesome. Yeah, thank you so much. This was, like I said, a really interesting experience. I've never had to think about longer term impacts. Most of my stuff is like, "Okay, GDPR is out. What does this mean for AI?" And that's a very immediate concern. It's not this like, Okay, where should\... Or even thinking like, "Okay, five years out, what do we want privacy legislation to look like?" That's something I think about, but not, "Oh my God, there's a decision-making AI out there. Does it care about my privacy?" So, yeah.

**0:40:47.0 Vael:** Yeah, yeah, I think people aren't really incentivized to think about the long-term future.

**0:40:51.8 Interviewee:** Yeah. Humans are just bad at that, right? Yeah.

**0:40:53.5 Vael:** And it's hard to forecast, so it makes sense.

\[closings\]


# The AI revolution and international politics (Allan Dafoe) - EA Forum

Artificial intelligence (AI) is rapidly improving. Superhuman AI in strategically relevant domains is likely to arrive in the next several decades; some experts think two. This will transform international politics, could be profoundly destabilizing, and could pose existential risks. Urgent research is needed on AI grand strategy.

This requires a careful examination of humanity's highest interests in the era of [transformative AI](https://www.openphilanthropy.org/blog/some-background-our-views-regarding-advanced-artificial-intelligence#Sec1), of the international dynamics likely to arise from AI, and of the most promising strategies for securing a good future. Much work will be required to design and enact effective global AI policy.

Below is a transcript of Allan Dafoe's talk on this topic from EA Global: Boston 2017. We've lightly edited the transcript for clarity.

## The Talk

**Nathan Labenz:** It is my honor to introduce Professor Allan Dafoe. He is an Assistant Professor of Political Science at Yale University and a Research Associate at the Future of Humanity Institute at Oxford, where he is involved in building the AI Politics and Policy Group. His research seeks to understand the causes of world peace and stability. Specifically, he has examined the causes of the liberal peace and the role of reputation and honor as motives for war. Along the way, he has developed methodological tools and approaches to enable more transparent, credible, causal inference. More recently, he has focused on artificial intelligence grand strategy, which he believes poses existential challenges and also opportunities, and which requires us to clearly perceive the emerging strategic landscape in order to help humanity navigate safely through it. Please welcome Professor Allan Dafoe to discuss his work on AI and international politics.

**Allan Dafoe:** Thank you, Nathan.

I will start by talking about war, and then we'll get to AI, because I think there are some lessons for effective altruism.

War is generally understood to be a bad thing. It kills people. It maims them. It destroys communities and ecosystems, and is often harmful to economies. We'll add an asterisk to that because in the long run, war has had dynamic benefits, but in the current world, war is likely a negative that we would want to avoid.

So if we were going to start a research group to study the causes of war for the purposes of reducing it, we might ask ourselves, "What kinds of war should we study?" There are many kinds of wars that have different causes. Some are worse than others.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/3704d0af382f2c6f6a81f5e343d8b235de7194c13eadd941.png)

One classification that we might employ is how many people different kinds of wars kill. There are some wars that kill only 1,000 to 10,000, and some that kill 100,000 to a million. And so the x-axis \[on this slide\] shows the battle deaths in wars, and the y-axis is the fraction of wars of those kinds.

An instinct we might have is to say, "Let's study the wars that are most common — those in the first bin." These are the wars that are happening today in Syria: civil wars. Availability bias would suggest that those are the wars we should worry about. Some of my colleagues have argued that great-power war — the big world wars — are a thing of the past. They say that the liberal peace, democracy, capitalism, and nuclear weapons have all rendered great-power war obsolete, and that we're not going to have them again. The probability is too low.

But as effective altruists, we know that you can't just round a small number down to zero. You don't want to do that. You want to try to think carefully about the expected value of different kinds of actions. And so it's important that even though the probability of a war killing a million, 10 million, 100 million, or a billion is very small, it's not zero.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/88d31db53c751e30c56002f433b71e481555d3f991354ca0.png)

If we look at the past 70 years of history, World War II stands out as the source of most of the battle deaths that have been experienced. That would suggest that fatalities are a good enough proxy for whatever metric of importance you have, and that we first want to make sure we understand World War II and the kinds of wars that are like it (i.e. that are likely to kill many people).

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/46adadd40e60253045d8682e04506a9575f94933db1214e5.png)

We can zoom out more, and see that World War I comes out of the froth of civil wars. And really, those two wars loom above everything else as what we want to explain, at least if we're prioritizing importance.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/2cbb5a0b533a6a66bae3fa4b71cf03a0487096bd8552ac85.png)

We can see that in this graph, which again has these bins depicting the size of violent quarrels. Relative to homicides, we see that the world wars in the "10 million" bin on the right contain, again, most of the deaths in violent quarrels. That also suggests that it's really important that we understand these big wars.

But of course, the next war need not limit itself to 99 million deaths. There could be a war that kills hundreds of millions or even 6.5 billion. The problem, empirically speaking, is that we don't have those wars and that data set, so we don't know how to estimate, non-parametrically, the expected value in those. We can try to extrapolate from what's very close to a power-law distribution. And no matter how we do it, unless we're extremely conservative, we get a distribution like this, which shows that most of the harm from war comes from the wars that kill a billion people or more:

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/3ad408f0ce60ad9764934ed99fe9672932aaedf48ec33101.png)

Of course, we haven't had those wars. Nevertheless, this follows from other reasoning.

We can go still further. The loss from a war that killed 6.5 billion is not about just those 6.5 billion people who die. It's also about future people. And this is where the idea of existential risk and existential value comes in. We have to ask ourselves, "What is the value of the future? How much worth do we give to future human lives?"

There are many ways to answer that question. And you want to discount for model uncertainty and various things. But one thing that drives concern with existential risk, which I'm concerned with and the Future of Humanity Institute is concerned with, is that there's so much potential in the future. There are so many potential lives that could be lived. Anything that cuts those lives off has tremendous disvalue.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/5df0afb5e3d84f15b7548845bf803a545534af84b4272b07.png)

One estimate that's quite conservative is this: If there are only a billion people living on Earth, and they continue living in a sustainable way for a billion years (which is doable), as long as we don't muck it up, that yields 10,000 trillion lives. That is a lot. And so anything that reduces the probability of extinction and of losing those 10,000 trillion lives, by even a little bit, has a lot of expected value.

Now, that's a very small number, one ten trillionth (0.0000000000001), and it's hard to know if you're making less of an effect than that. It looks close to zero. The numbers aren't meant to be taken too seriously. They're more thinking heuristics. They illustrate that if you give value to the future, you really want to worry about anything that poses a risk of extinction. And one thing that I and others, such as the Open Philanthropy Project, have identified as a risk to the future is artificial intelligence.

Before telling you about the risk, I'm going to first tell you about what's up with AI these days.

For a long time, artificial intelligence consisted of what we would now call good old-fashioned AI. There was a programmer who wrote if-then statements. They tried to encode some idea of what was a good behavior that was meant to be automated. For example, with chess algorithms, you would have a chessmaster say, "Here are the heuristics I use. Here's the value function." You put those heuristics in a machine, and the machine runs it more reliably and more quickly than a human. And that's an effective algorithm. But it turns out that good old-fashioned AI just couldn't hack a number of problems — even simple ones that we do in an instant, like recognizing faces, images, and other things.

More recently, what has sort of taken over is what's called "machine learning." This means what it sounds like it means: machines learning, for themselves, solutions to problems. Another term for this is "deep learning," which is especially flexible machine learning. You can think of it as a flexible optimization procedure. It's an algorithm that's trying to find a solution and has a lot of parameters. "Neural networks" is another term you've probably heard.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/9bc2e02dcbd7ccd91aecd035ada6d068e67b0489d534d628.png)

This slide shows the breakthrough recently in image classification arising from neural networks and the year-on-year improvements to the point where machines are now better than humans at image classification.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/c7874af0d4a75ba434c4f1e4d8e89d9d26c205afa815b9e1.png)

Another domain is generalized game-playing or arcade game-playing. Here are our target games. Probably not many of us have played these. DeepMind is a leading AI group within Google that learned to play Atari games at a superhuman level with no instructions about the nature of the universe (e.g., "What is time and space?", "What is a bad guy?", "Here's Pac-Man — what is a pellet and what is a ghost?"). All the machine is told or given is pixel input. It just sees the screen from a blank-state type of beginning. And then it plays the game again and again and again, getting its score, and it tries to optimize for the score. Gradually, over the span of about a day, it learns to make sense of this pixel input. It derives concepts of the bad guy, the ghosts, and the pellets. It devises strategies and becomes superhuman at a range of games.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/476507ddedcab38aee856c03dc8c9426c2ec63bd7e506a01.png)

Another domain where we see this is Go. The solution to Go is very similar. You take a blank-slate neural network that's sufficiently flexible and expose it to a lot of human games, and it learns to predict the human moves. Then you have the machine play itself again and again, on the order of about 10 million games. And it becomes superhuman.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/964474271c838297f75c82280f0e5e5f5b30189aca07ddd2.png)

Here's Lee Sedol and AlphaGo, which is another product of DeepMind. And Lee Sedol was the first really excellent Go player who publicly played AlphaGo. And here he's saying in February 2016, "I think I won the game by a near-landslide this time." Well, as I've alluded to, that didn't work out.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/d4c8c30a4d907469baff3f5ec0c68dade29901fef0bec641.png)

Here he is after the first game: "I was very surprised because I didn't think I would lose."

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/06f9a5d06e11384568202d4866d87f907c207c8217ed6f93.png)

And unfortunately, he lost again: "I'm quite speechless. I'm in shock. I can admit that. The third game is not going to be easy for me." He lost the third game. He did win the fourth game and he talks about how it was sort of the greatest moment of his life. And it's probably the last and only game played against a level of AlphaGo of that level or better that a human will ever win.

I'm bringing up Lee Sedol and his losses for a reason. I think it serves as an allegory for humanity: We don't want to be caught off-guard when it's our AlphaGo moment. At some point, machine intelligence will be better than we are at strategically relevant tasks. And it would be prudent for us to see that coming — to have at least a few years' notice, if not more, to think through how we can adapt our international systems, our politics, our notion of the meaning of life, and other areas.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/5c1463d2ae0a9796f9c804527a085b944314b6c0d463f1f2.png)

What's driving this progress? Algorithms, talent, and data, but another big thing driving it is hardware. Computing inputs keep getting better at an exponential rate. This is sort of a generalized Moore's law across a range of inputs. And it's this persistent progress that makes Kurzweil's graph from 2001 seem not totally absurd.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/8340d08346567587be5442e31eafa1bf3da9396555268898.png)

Here we have, on the first y-axis, calculations per second, per thousand dollars, and I added four recent dots from the past 17 years. And you see that basically we're on track. We have exponential improvements. What we don't know is when we get to transformative AI.

Now, Kurzweil has this evocative second y-axis, where you have organisms we recognize: mice, humans, and all humans. It's not obvious what the mapping should be between calculations per second and transformative AI. Intelligence is not a single dimension, so we could get superhuman AI in some domains long before other domains. But what I think is right about this graph is that at some point between 2020 or 2025 and 2080, big things are going to happen with machine intelligence.

In our work, we want to be a bit more systematic about these timelines, and there are various ways to do it. One way is to survey AI researchers. This is the result of a recent survey.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/c3cbdda22da9887bdf8d12ef450f76ebdc91408359111991.png)

Some takeaways are:

1.  There's huge disagreement about when human-level machine intelligence, defined here as machines better than all humans at every task, will come, as you can see by the gray S-curves.
2.  The group that we surveyed gives enough probability mass that by 100 years, there still won't be human-level machine intelligence.
3.  But in the next 10 or 20 years, this group gives a 10-20% chance that we will have reached it already. And if that probability seems right (and upon consideration, I think it does, and not just from using this as evidence), there's more than sufficient warrant for us to invest a lot of resources into thinking very hard about what human-level AI would mean for humanity.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/b8af218dbc98985f543afd818a90274a072ee5cf7b07c0e0.png)

Here are some tasks. I like to think of them as milestones and canaries. What are some things that machines will eventually achieve that are either noteworthy or strategically relevant (i.e. milestones)? And the canaries are those things that when they happen, signal to us that we'd better be paying attention because things are going to change quickly. I expect most of these tasks on the right-hand column will soon be moving over to the left-hand column, if they're not there already. So this is something else we're working on.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/95de8fd3d4eed18e5e5d86d36460f888ff5bf369bb13628e.png)

More generally, there's a whole host of challenges — near, medium, and long term — that we will be working on as a society, but also at the Future of Humanity Institute. There's a range of near-term issues that I'm not going to talk about. Each of those could occupy a workshop or a conference. I will say that when thinking about long-term issues, we also confront the near-term issues for one reason: because the long-term issues often look like the near-term issues, magnified a hundred-fold, and because a lot of our long-term insights, strategies, and policy interventions' most appropriate place of action is in the near term.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/9bb96db8553ccebce756846c9bc7d53a3dea39b7b027d822.png)

But what are the long-term opportunities and risks? The opportunities are tremendous. We often just think it's greater wealth, and that maybe Google stock will go up. But it's a lot more: longevity, health, preventative medicine, material bounty that could be used to end poverty ( though it need not, because it will also likely come in a more unequal world), reduced environmental impact. DeepMind was able to reduce energy usage at Google data centers by 40%. AI could basically help with anything of value that is either the product of intelligence or depends on intelligence for its protection. And so if we have superhuman intelligence, then in principle, we can use that to achieve all of our goals.

I'll also emphasize the last point: resilience to other existential risks. We're likely to face those in the next 100 years. And if we solve AI — if we build it well — then that could reduce those other risks by a large margin.

But of course, there are also risks with bringing into the ecosystem a creature that is better than we are at the thing that matters most for our survival and flourishing. I'm not going to go through this topology.

I will illuminate the risk by quoting Max Tegmark and others:

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/003a94eb08240aea330c5281af1f484627f25c2097eb4424.png)

I will also appeal to another authority from my world:

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/abaf1bc735f5bd59ac1ae90686093b9ac58ecc08c3b94b26.png)

I'm a scholar of international relations. Henry Kissinger, as you know, is also very worried about AI. And this is sort of the definition of a grand strategist.

In addition to these quotes, the AI researchers we surveyed agree. We asked them what the long-term prospects are for human-level machine intelligence. And while most of the probability mass was on "extremely good" or on "balanced good," the median is a 5% probability of AI being extremely bad or leading to human extinction.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/2cd2c30fca6d274c2de7972fa289e67a0f78a2769beaa7a8.png)

It's not often you get an industry that says that their activities give rise to a 5% chance of human extinction. I think it should be our goal to take whatever the real and most frequent number is, and push it as close to zero, getting as much of that probability mass up to the top.

There are two broad ways we can do that. One is to work on what's called AI safety. The computer scientists in the room and mathematicians can help build AI systems that are unlikely to misbehave without our intentions. And the other way is AI strategy, which I'm going to talk about.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/bdceccf65b1f386dc1e3fbd36650b6d5c92100a2b5e0f5b3.png)

So here's Stuart Russell explaining that we're not worried about machines suddenly waking up and deciding they want to build their own machine utopia and get rid of humans. It's not some kind of emergent consciousness. The worry is that they will be hyper optimizers, which is what we're building them to do — and that we will have not specified their value function correctly. Therefore, they could optimize for the wrong thing. This is broadly called "the control problem" or "the value alignment problem." Here are some groups working on this or funding it.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/eba0d7dbb8307d5c2222e169316a99be1837f4eafbfb9b57.png)

I'm going to keep moving. Here are some people who you can't really make out, but I will tell you that they are leading researchers and industrialists from the most prominent AI groups in the world. And they came together at the Asilomar Conference to really think seriously about AI safety, which I think is a tremendous achievement for the community — that we've brought everyone together like this. I'm showing this as a reflection of how exciting a time it is for you to be involved in this.

___

One conjecture that's been posed is that AI value alignment is actually not that hard of a problem. As long as we have enough time, once we have the final system we want to deploy, we can test it, right? It's like drug tests. You have the thing you are thinking about deploying in the population. You just make sure it undergoes sufficient scrutiny. However, this conjecture goes, it is almost impossible to test AI value alignment if we don't have enough time. And if that's right, which seems plausible, then it directs attention to the world in which this system is being deployed. Is it one where the developers have the incentives and the time to do these safety tests? This is one of the big things people working on AI strategy think about — this issue of how to prevent a race in AI.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/4bae8114220ae884f5fd4e047931e5cec40546b525140be3.png)

Here's the CEO of DeepMind, basically reinforcing this point: "We want to avoid a harmful race to the finish, where corner-cutting starts happening and safety gets cut. This is a big issue on a global scale, and it's extra hard when you're talking about national governments." It's not just a race between companies in the US. It's a race between countries.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/527b6c489177f0ab61b9b3026f96fb446aa122f86555d657.png)

So what do we think about in AI strategy? A lot. One topic we think a lot about is what AI races and AI arms races would look like. Another whole class of issues is what AI could look like militarily. What are some implications of AI for the military in terms of balance of power, crisis stability, and uncertainty over capabilities?

Another thing we think about is what it means economically if we live in a world where AI is the engine of growth and value in societies, which increasingly seems to be the case. Of the top 10 firms by market capitalization, either five or six are AI companies: Google, Amazon, Apple, Microsoft. In such a world, what do countries do like Saudi Arabia or France, which don't have their own Google or Amazon, but want to be part of that value chain? We may be entering an era of AI nationalism, where countries want to build their own national champion. China is certainly in the business of doing this.

The last high-level category I'll mention is the massive challenge of AI governance. This is an issue on a small, near-term scale. For example, how do we govern algorithms that are being used for judicial sentencing or self-driving cars? It's also an issue on a long-run scale, when we consider what kind of electoral system or voting rules we want to use for the organization that's going to be deciding how to test and deploy superintelligence. These are very hard questions.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/69205e89fb37c8e4749f2e54ee44a600396973c900e71eb4.png)

I want to make clear that these are questions being asked today by the leading AI groups. Here are Sam Altman and Demis Hassabis asking questions like "What is the voting rule that we should use for control of our superintelligence once we build it?"

The site for governance today is the [Partnership on AI](https://www.partnershiponai.org/). This is a private-sector organization that has recently brought in NGOs, including the Future of Humanity Institute. And it's plausible that this will do a good job guiding AI in the near term, and could grow in the longer term. At some point, governments are likely to get more involved. So that's a site for study and for intervention.

Another thing we can do is try to articulate principles of good governance of AI. Here are some principles that came out of the Asilomar Conference.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/de51d0a3c02f9140c300db799aceae13f6a419c83787f971.png)

Again, a hat tip to Max Tegmark for putting that together, and especially these principles. We might want to work to identify important principles that we can get different communities to agree on, and then formalize and institutionalize them to make sure they stick.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/28fb0ad537936cd3c1fe895a57c8d0c4d3e5f0765af1b0bf.png)

In summary, what's to be done? A lot of work. What kind of skills do we need? Virtually every skill set — people who can help us grow the community, conduct operations and administration, or just be enthusiastic and efficient. We need people to do policy engagement, outreach, and media strategy. For example, what should the media strategy of a group like the Future of Humanity Institute be when there's another self-driving car incident, or when truckers are being massively displaced from their sites of employment? These are important near-term issues, but also they're sites for having conversations about longer-term issues.

We're doing work on strategy, theoretical work, and mathematical modeling of tech races. We're trying to understand AI development and what predicts innovation, measuring actual capabilities in different sites in the world's countries to understand the value chain and the supply chain of AI. We're surveying publics and elites around the world. We're trying to design safety standards and working with AI safety researchers, and tackling a range of other issues.

If this seems important and interesting to you, I strongly encourage you to get involved. The [AI Policy Career Guide](https://80000hours.org/articles/ai-policy-guide/) has some texts that can point you in the right direction. There's also a reading list. And in general, just reach out to me. There are also people working on this at a range of sites, and we'd be very happy to help you be productively engaged. Thanks.

## Q&A

**Nathan:** Let's have a seat. Thank you for the talk and for being here. We'll give a second for questions to come in from the audience \[via the conference app\].

One thing that I'm struck by is it seems like we're making a lot of progress. I've been involved in this community, sort of from the edges, for a number of years. And there was a time (seven to 10 years ago) when to even talk about something like this was extremely "fringe." Only weirdos seemed to be willing to go there. Now we have respectable people like you and a growing body of academics who are getting involved. So it seems like there has been a lot of social progress.

What about technical or practical progress on these issues? It seems that we're bringing people together, but what are those people producing so far? And should we feel any safer than we did 10 years ago?

**Allan:** I can speak to the strategy side of it. One comment that was made at the Asilomar Conference that resonated as true to many people is that AI strategy today is where AI safety was two years ago. The first beneficial meeting for AI safety was in Puerto Rico. Just a handful of individuals in the world were thinking seriously and full-time about it. That has changed today. Now the leading AI groups have safety teams. There are people doing PhDs with an eye towards AI safety. And that's very exciting. And there has been a lot of technical progress that's coming out of that.

AI strategy is just where AI safety was two years ago, but I think it's rapidly scaling up. A lot of thinking has been done, but it's not public yet. I think in the coming year you will start to see a lot of really insightful strategic analysis of AI existential risk.

I will also say I've given some talks like this (e.g., political science workshops) to other respectable audiences and none of the PhD students think this is crazy. Some of them think, "Oh, I don't have to quit smoking because of this." That was one comment. But they all think this is real. And the challenge for them is that the discipline doesn't entirely support work on the future.

One question I got when I presented at Yale was "How will you be empirical about this?" Because we're social scientists, we like data, and we like to be empirical. And I remarked that it is about the future, and it's hard to get data on that, but we try. So I think it's a challenge for currently existing disciplines to adapt themselves to this problem, but increasingly we're finding good people who recognize the importance of the problem enough to take the time to work on it.

**Nathan:** So, the first audience question, which I think is a good one, is this: Could you provide more detail on what an AI governance board might look like? Are you thinking it will be a blue-ribbon panel of experts or more of a free-for-all, open-democracy type of structure, where anyone can contribute?

**Allan:** I think there are a lot of possibilities and I don't have a prescription at this point, so I'm not going to answer your question. But these are the issues we need to think through. There are trade-offs that I can talk about.

There's the issue of legitimacy. The UN General Assembly, for example, is often seen as a legitimate organization because every country gets a vote, but it's not necessarily the most effective international body. Also, you have to weigh your institutions in terms of power holders. So if you have the most ideal governance proposal, it might be rejected by the people who actually have the power to enact it. So you need to work with those who have power, make sure that they sign onto a regime, and try to build in sites of intervention — the key properties of whatever this development and governance regime is — so that good comes out of it.

I'll mention a few suggestions. Whatever development regime it is, I think it should have:

1.  A constitution — some explicit text that says what it's about and what it's trying to achieve.
2.  Enough transparency so that the relevant stakeholders — be that the citizens of the country, if not citizens of the world — can see that the regime is, in fact, building AI according to the constitution. And I should say that the constitution should be a sort of common-good principle type thing.
3.  Accountability, so that if the regime isn't working out, there's a peaceful mechanism for changing the leadership.

These are basic principles of institutional design, but it's important to get those built in.

**Nathan:** So speaking of people in power, I haven't seen this clip, but President Obama apparently was asked about risks related to artificial intelligence. And the answer that he gave seemed to sort of equate AI risk with cybersecurity risks that we know and love today. Do you think that there is a sufficient understanding at the highest levels of power to even begin to make sense of this problem? Or do we have a fundamental lack of understanding that may be quite hard to overcome?

**Allan:** Yeah. That's a great clip. I wish I could just throw it up really quickly. We don't have the AI yet to just do that. I encourage you to watch it. It's hard to find it. [_Wired_ put it on their page](https://www.youtube.com/watch?v=72bHop6AIcc). He's asked about superintelligence and if he's worried about it and he pauses. He hesitates and gives sort of a considered "hmm" or sigh. And then he says, "Well, I've talked to my advisors and it doesn't seem to be a pressing concern." But the pause and hesitation is enough to suggest that he really did think about it seriously. I mean, Obama is a science fiction fan. So I think he probably would have been in a good place to appreciate other risks as they arise.

But I think a lot of people in government are likely to dismiss it. Many reports from the military or government have put superintelligence worries at least sufficiently distant enough that we don't really need to think about or address it now. I will say though, that cybersecurity is likely to be a site where AI is transformative, at least in my assessment. So, that's one domain to watch in particular.

**Nathan:** Here's another question from the audience: If there is a 5% chance of extinction due to AI, one would not be unreasonable to jump to the conclusion that maybe we should just not do this at all. It's just too hot to touch. What do you think of that idea? And second, is there any prospect of making that decision globally and somehow sticking to it?

**Allan:** Yeah. I'll flick back to the slide on opportunities. I actually had a conversation the other day with family members and friends, and one person at the table asked that question: If it's so risky, why don't we not do it? And then another friend of the family asked, "What are the impacts of AI for medicine and health, and for curing diseases?" And I think in many ways those are two sides of the policy decision. There are tremendous opportunities from AI, and not just material ones, but opportunities like curing Alzheimer's. Pretty much any problem you can imagine that's the product of intelligence could \[be solved with\] machine intelligence. So there's a real trade-off to be made.

The other issue is that stopping AI progress is politically infeasible. So I don't think it's a viable strategy, even if you thought that the trade-off weighed in favor of doing so. And I could talk a lot more about that, but that is my position.

**Nathan:** Probably the last question that we can take due to time constraints is this: Thinking about the ethical direction that we want to take as we go forward into the future — the value alignment problem — you had posed that notion that if we have two years, we can probably figure it out.

**Allan:** Yeah.

**Nathan:** But if we don't, maybe more than likely we can't. That strikes someone in the audience and I would say me too, as maybe a little too optimistic, because we've been working for thousands of years on what it means to have a good life and what good is.

**Allan:** Right.

**Nathan:** Do you think that we are closer to that then than maybe I think we are? Or how do you think about the kind of fundamental question of "What is good?" in the first place?

**Allan:** Right. To be clear, this conjecture is about whether a single person who knows what they want can build a superintelligent machine to advance those interests. It says nothing about whether we all know what we want and could agree on what we want to build. So can we agree on the political governance question: Even if we all have fundamental preferences, how do we aggregate those in a good way? And then there's the deeper question of what should we want? And yeah, those are hard questions that we need people working on, as I mentioned. In terms of a moral philosophy in politics, what do we want? We need your help figuring that out.

**Nathan:** Well, thank you for wrestling with these issues and for doing your best to protect our future. Professor Allan Dafoe, thank you very much.


# Rohin Shah: What's been happening in AI alignment?

Interviewee: Rohin Shah
Date: 2020-03-21

While we haven't yet built aligned AI, the field of alignment has steadily gained ground in the past few years, producing many useful outputs. In this talk, Rohin Shah, a sixth-year PhD student at UC Berkeley's Center for Human-Compatible AI (CHAI), surveys conceptual progress in AI alignment over the last two years.

While Rohin started his PhD working on program synthesis, he became convinced that it was important to build safe, aligned AI, and so moved to CHAI at the start of his fourth year. He now thinks about how to provide specifications of good behavior in ways other than reward functions. He is best known for the Alignment Newsletter, a popular weekly publication with content relevant to AI alignment.

Below is a transcript of Rohin's talk, which we've lightly edited for clarity. You can also watch it on YouTube and discuss it on the EA Forum.

# The Talk

Hi, everyone. My name is Rohin Shah. I'm a sixth-year PhD student at the Center for Human-Compatible AI at UC Berkeley. My research is generally on what happens when you try to do deep reinforcement learning in environments that involve humans. More broadly, I work on technical AI safety. I also write the Alignment Newsletter.

Today, I'll cover what's been happening in AI alignment. I should warn you: While this talk doesn't assume any technical knowledge of AI, it does assume basic familiarity with the arguments for AI risk.

I'll be surveying a broad swath of work rather than focusing on my personal interests. I'm hoping that this will help you figure out which parts of AI alignment you find exciting and would like to delve into more deeply.

A lot of the talk is based on a literature review I wrote a few months ago. You can find references and details in that review.

[Flow Chart of AI Alignment Landscape]

With that, let's get started. Taking a high-level, outside view, the reason that most people work on AI safety is that powerful AI systems are going to be a big deal. They're going to radically transform the world that we live in. Therefore, we should probably put some effort into making sure that this transformation goes well.

In particular, if AI systems are smarter than we are, then they could become the dominant force on the planet, which could be bad for us — in the same way that gorillas probably aren't [thrilled] about how we have taken over all of their habitats. This doesn't necessarily mean that [AI will create] be an x-risk [existential risk]. It just means that we should have a sound technical reason to expect that the powerful AI systems we build are actually beneficial for us. And I would argue that we currently do not have such a reason. Therefore, the case for working on AI alignment is that we really should be creating this reason.

I want to note that there's a lot of disagreement over specific sub-questions in AI safety. That will become more evident over the rest of this talk. But my impression is that virtually everyone in the field agrees with the basic, high-level argument [that we should have a good reason for expecting AI systems to be beneficial].

What are the specific risks we're worried about with AI? One issue is that humans aren't ready to deal with the impacts of AI. People tend to be in conflict a lot, and the US-China relationship is a big concern [in the AI community]. AI will enable better and better ways of fighting. That seems pretty bad. Maybe our fights will lead to bigger and bigger impacts; at some point, that could result in extinction-level events. Or perhaps AI leads to technological progress at such a fast pace that we're unable to [adjust]. As a result, we could lock in some suboptimal values [that AI would act on for the rest of humanity's future]. In both of these scenarios, the AI system wouldn't intentionally cause x-risk, but it nonetheless would happen.

I'm not going to focus too much on this, but will note that some people are talking about preference aggregation. This is the idea that the AI system aggregates preferences across all stakeholders and does its thing — and then everyone agrees not to [oppose] the results. Similarly, we could try to [arrive at a] better metaphilosophy to avoid problems like value lock-in.

Another outside view that people take, aside from "AI is powerful and a big deal," is that optimization leads to extreme outcomes. To take a very simple example, men in the US are, on average, about five feet, 10 inches tall. But very few basketball players, who are selected for height, are five feet, 10 inches. Most are well over six feet. When you select for something and have optimization pressure, you tend to get extreme outcomes. And powerful AI systems are going to be powerful optimizers. As a result, we probably shouldn't expect our everyday reasoning to properly account for what these optimizers will do.

Therefore, we need to [cultivate] more of a security mindset and look for arguments that quantify every possibility, as opposed to the average possibility. This mindset inspires researchers, especially at MIRI [the Machine Intelligence Research Institute], to try to understand how intelligence really works, so that we can then make well-designed AI systems that we understand. This has led to research on embedded agency, partial agency, and abstraction.

A bit about embedded agency: This is one of MIRI's main research programs. The basic idea is that, according to the standard model of reinforcement learning and [our understanding of] AI more generally, an environment takes in actions and produces [observable phenomena] and rewards. Then, completely separate from the environment, an agent [observes these phenomena] and takes actions as a result. But that's not how agents work. I'm an agent, yet I am not separate from the environment; I am a part of it. This leads to many philosophical problems. I would love to go into more detail, but don't have too much time. There's a great sequence on the AI Alignment Forum that I strongly recommend.

——

The next problem I want to talk about is one that I call "the specification problem." It's also called "outer alignment." Basically, the way we build AI systems right now is by assuming that we have some infallible specification of the optimal behavior in all possible situations, as though it were handed down to us from God. Then, we must figure out how to meet that specification. But of course, we can never actually get such a specification. The classic paperclip maximizer thought experiment shows that it's quite hard to specify the behavior of an AI making paperclips in a reasonable and sane way. This is also the main problem that Stuart Russell discusses in his book Human Compatible. Organizations [whose work includes addressing] this specification problem include CHAI, OpenAI, DeepMind, and Ought.

The main proposed way of solving the specification problem is to do some form of value learning. One thing I want to note: Value doesn't necessarily mean "normative value." You don't necessarily need to be thinking about population ethics. For example, a robot that learned how to clean your room, and then reliably did so, would count as [an example of] value learning. Maybe we should be calling it "specification learning," but value learning seems to be the name that has stuck.

The types of value learning include CIRL (or "assistance games"). CIRL stands for "cooperative inverse reinforcement learning." This is a particular formalization of how you could approach value learning, in which the world contains a single human who knows the reward function — the true specification — but, for some reason, can't communicate that explicitly to the agent. There is also an agent whose goal is to infer what the human's specification is, and then optimize for it. And because the agent no longer has a definite specification that it's trying to optimize, and it's instead uncertain over what it's trying to optimize, this results in many nice properties.

For example, the agent might ask you about what you want; it may try to clarify what your preferences are. If you try to shut it down, it will reason that it must have been doing a poor job of helping you. Therefore, it's going to allow you to shut it down, unlike a classic unexpected utility maximizer, which will say, "No, I'm not going to shut down, because if I am shut down, then I can't achieve my goal."

The unfortunate thing about assistance games is that they are [exceptionally] computationally intractable. It's very expensive to solve a CIRL game. In addition, it requires a good model of how human preferences relate to human behavior, which — as many of the social sciences show — is a very difficult problem. And there is a theorem that says it is impossible to prove in the super-general case. Although, of course, we don't actually need the super-general case; we only need the case that applies in the real world. Instead of being impossible to prove, [the real-world case] is merely very, very difficult.

Next, we have [strategies based on agents] learning human intent. This is a broad category of possible communication protocols that a human could use to communicate the specification to the agent. So perhaps a human could demonstrate the optimal behavior to the agent, and from that, the agent could learn what it's supposed to do. (This is the idea behind inverse reinforcement learning and imitation learning.) Alternatively, perhaps the human could evaluate proposed hypothetical behaviors that the agent might execute, and then the agent could reason out what it should be doing.

Now we come to intent alignment, or "corrigibility." This is somewhat different. While the previous approaches try to specify an algorithm that learns values, with intent alignment we instead build an agent that tries to do what we want it to do. Put another way, we're trying to bake into the agent the motivation to be helpful to us. Then, if we have an agent [whose sole motivation] is to be helpful to [a human], that will naturally motivate it to do many other things that we want. For example, it's going to try to clarify what my [travel] preferences are in the same way that a good personal assistant would, so that it doesn't have to bother me when I ask it to book me a flight.

That covers a broad spectrum of approaches to value learning. However, there are still a few problems that arise. Intuitively, one big one is that, since the agent is learning from our feedback, it's not going to be able to do better than we can; it won't be able to scale to superhuman performance. If we demonstrate the task to the agent, it won't be able to perform the task any better than we could, because it's receiving no information on how to [go about that]. Similarly, if we're evaluating the agent's behavior, it won't be able to find good behaviors that we wouldn't recognize as good.

An example is AlphaGo's move 37 [in its match against Go champion Lee Sedol]. That was a famous move that AlphaGo made, which no human ever would have made. It seemed crazy. I think it was assigned a less than one-in-10,000 chance of succeeding, and yet that move ended up being crucial to AlphaGo's success. And why could AlphaGo do this? Because AlphaGo wasn't relying on our ability to determine whether a particular move was good. AlphaGo was just relying on a reward function to tell it when it had won and when it had lost, and that was a perfect specification of what counts as winning or losing in Go. So ideally, we would like to build superintelligent AI systems that can actually exceed human performance at tasks, but it's not clear how we do this with value learning.

The key idea that allows current approaches around this is: Our AI systems are never going to exceed the supervision that we give them, but maybe we can train our AI systems to approximate what we would do if we had an extremely long time to think. Imagine I had 1,000 years to think about what the best thing to do was in a certain scenario, and then I shared that with an AI system — and then the AI system properly approximated my suggestion, but could do so in a few minutes as opposed to 1,000 years. That would presumably be a superintelligent AI.

The details for how we take this insight and arrive at an algorithm so that we can try it soon — not in 1,000 years — are a bit involved. I'm not going to go into them. But the techniques to look for are iterated amplification, debate, and recursive reward modeling.

Another problem with value learning is the informed oversight problem: Even if we're smarter than the agent that we're training, we won't be able to effectively supervise it in the event that we don't understand why it chose a certain action. The classic example is an agent tasked to write a new novel. Perhaps it has access to a library where it's supposed to learn about how to write books, and it can use this in order to write the novel, but the novel is supposed to be new; [the task requires more than] just memorizing a novel from the library and spitting it back out again. It's possible that the agent will look at five books in the library, plagiarize chunks from all of them, and put those together into a book that reads very nicely to us, but doesn't really solve the task because [the novel is unoriginal]. How are we supposed to tell the agent that this was bad? In order to catch the agent looking at the five books and stealing sentences from them, we'd have to read the entire library — thousands of books — and search for evidence of plagiarism. This seems too expensive for oversight.

So, it may be significantly more costly for us to provide oversight than it is for the agent to take actions if we cannot see how the agent is taking those actions. The key to solving this is almost obvious. It's simply to make sure you know how the agent is taking their actions. Again, there are many details on exactly how we think about this, but the term to look for is "ascription universality." Essentially, this means that the supervisor knows everything that the agent knows, including any facts about how the agent chose its output.

[In the novel-writing example], if we were ascription-universal with respect to the agent, then we would know that it had taken sentences from five books, because the agent knows that. And if we knew that, then we could appropriately analyze it and tell it not to plagiarize in the future.

How do we create this property? Sadly, I'm not going to tell you, because again, I have limited time. But there's a great set of blog posts and a summary in the Alignment Newsletter, and all of those items are in my literature review. Really, I just want you to read that link; I put a lot of work into it, and I think it's good.

——

Let's move on to another top-level problem: the problem of mesa optimization. I'm going to illustrate mesa optimization with a non-AI example. Suppose you're searching for a Python program that plays tic-tac-toe well. Initially you find some programs that have good heuristics. Maybe you find a program that always starts at the center square, and that one tends to win a little more often than the others. Later, you find a program that makes sure that anytime it has two spots in a row and the third spot is empty, it plays in that third spot and wins. One that does that in a single step starts to win a bit more.

Eventually, you come across the minimax algorithm, which plays optimally by searching for the best action to take in every situation. What happened here was that in your search for optimal Python programs, you ended up finding a program that was itself an optimizer that searched possible moves in tic tac toe.

This is mesa optimization. You have a base [or "outer"] optimizer — in this case, the search over Python programs — and in the course of running that base optimizer, you find a new optimizer, which in this case is the minimax algorithm.

Why is this weird example about programs relevant to AI? Well, often we think about AI systems that are trained using gradient descent. And gradient descent is an optimization algorithm that searches over the space of neural net parameters to find some set of parameters that performs well on a loss function.

Let's say that gradient descent is the outer optimizer. It seems plausible that mesa optimization could happen even with gradient descent, where gradient descent finds an instantiation of the neural net parameters, such that then the neural net itself, when it runs, performs some sort of optimization. Then the neural net would be a mesa optimizer that is optimizing some objective, which we would call the mesa objective. And while we know that the mesa objective should lead to similar behavior as the original objective on the training distribution, because that's what it was selected to do, it may be arbitrarily different [outside] the training distribution. For example, if you trained it on tic tac toe, then you know it's going to win at tic tac toe — but if you switch to Connect Four, it might do something crazy. Maybe in Connect Four, it will continue to look for three in a row instead of four in a row, and therefore it will lose badly at Connect Four, even though it was working well with tic tac toe.

Let's say that this happened with gradient descent, and that we had a very powerful, intelligent neural net. Even if we had solved the specification problem, and had the ideal reward function to train this agent, it might be that the neural net model that we come up with optimizes for a different objective, which may once again be misaligned with what we want. The outer-inner distinction is why the specification problem is called "outer alignment," and why mesa optimization is called "inner alignment."

How do people solve mesa optimization? There's one main proposal: adversarial training. The basic idea is that in addition to training an AI system that's trying to perform well on your specifications, you also have an adversary — an AI system or AI human team that's trying to find situations in which the agent you're training would perform badly, or would optimize for something other than the specification problem.

In the case where you're trying to get a corrigible AI system, maybe your adversary is looking for situations in which the AI system manipulates you or deceives you into thinking something is true, when it is actually false. Then, if you can find all of those situations and penalize the agent for them, the agent will stop behaving badly. You'll have an agent that robustly does the right thing across all settings. Verification would [involve using] that agent to verify another property that you care about.

Ideally, we would like to say, "I have formally verified that the agent is going to reliably pursue the specification that I outlined." Whether this is possible or not — whether people are actually optimistic or not — I'm not totally clear on. But it is a plausible approach that one could take.

There are also other areas of research related to less obvious solutions. Robustness to distributional shift is particularly important, because mesa optimization becomes risky with distributional shift. On your training distribution, your agent is going to perform well; it's only when the world changes that things could plausibly go badly.

——

A notable thing that I haven't talked about yet is interpretability. Interpretability is a field of research which entails trying to make sure that we understand the AI systems we train. The reason I haven't included it yet is because it's useful for everything. For example, you could use interpretability to help your adversary [identify] the situations in which your agent will do bad things. This helps adversarial training work better. But interpretability is also useful for value learning. It allows you to provide better feedback to the agent; if you better understand what the agent is doing, you can better correct it. And it's especially relevant to informed oversight or description universality. So while interpretability is obviously not a solution in and of itself, it makes other solutions way better.

There's also the option of trying to prevent catastrophes. Someone else can deal with whether the AI system will be useful; we're just going to stop it from killing everybody. Approaches in this area include impact regularization, where the AI system is penalized for having large impacts on the world. Some techniques are relative reachability and attainable utility preservation. The hope here would be that you could create powerful AI systems that can do somewhat impactful things like providing advice on writing new laws, but wouldn't be able to do extremely impactful things like engineer a pandemic that kills everybody. Therefore, even if an AI system were motivated to harm us, the impact penalty would prevent it from doing something truly catastrophic.

Another [area of impact regularization] is oracles. The idea here is to restrict the AI system's action space so that all it does is answer questions. This doesn't immediately provide safety, but hopefully it makes it a lot harder for an AI system to cause a catastrophe. Alternatively, you could try to box the AI system, so that it can't have much of an impact on the world. One example of recent work on this is BoMAI, or boxed myopic artificial intelligence. In that case, you put both the human and the AI system in a box so that they have no communication with the outside world while the AI system is operating. And then the AI system shuts down, and the human leaves the box and is able to use any information that the AI system gave them.

So that's most of [the material] I'll cover in this problem-solution format. There's also a lot of other work on AI safety and alignment that's more difficult to categorize. For example, there's work on safe exploration, adversarial examples, and uncertainty. These all seem pretty relevant to AI alignment, but it's not obvious to me where, exactly, they fit in the graph [above]. So I haven't put them in.

There's also a lot of work on forecasting, which is extremely relevant to [identifying] which research agendas you want to pursue. For example, there has been a lot of disagreement over whether or not there will be discontinuities in AI progress — in other words, whether at some point in the future, AI capabilities shoot up in a way that we couldn't have predicted by extrapolating from past progress.

Another common disagreement is over whether advanced AI systems will provide comprehensive services. Here's a very short and basic description of what that means: Each task that you might want an AI system to do is performed by one service; you don't have a single agent that's doing all of the tasks. On the other hand, you could imagine a single monolithic AI agent that is able to do all tasks. Which of these two worlds are we likely to live in?

A third disagreement is over whether it is possible to get to powerful AI systems by just increasing the amounts of compute that we use with current methods. Or do we actually need some deep insights in order to get to powerful AI systems?

This is all very relevant to deciding what type of research you want to do. Many research agendas only make sense under some possible worlds. And if you find out that one world [doesn't seem very likely], then perhaps you switch to a different research agenda.

That concludes my talk. Again, here's the link to the literature review that I wrote. There is both a short version and a long version. I really encourage you to read it. It goes into more detail than I could in this presentation. Thank you so much.