# Interview with AI Researchers individuallyselected_7ujun by Vael Gates

**Interview with 7ujun, on 3/18/22**

**0:00:03.4 Vael:** Alright. So jumping right in, my first question is, can you tell me about what area of AI you work on in a few sentences?

**0:00:11.1 Interviewee:** \[Interviewee describes working on natural language processing research\]

**0:01:20.4 Vael:** Indeed. Thanks. And then so my next question is, what are you most excited about in AI? And what are you most worried about? In other words, what are the biggest benefits or risks of AI?

**0:01:36.4 Interviewee:** The biggest risks are that there is a lot of people who don't really\... Who\... The biggest risk in AI is that it's a field with a lot of money and attention and social power right now, and there are a lot of people who have positions of power who\... don't seriously consider what they do and the impacts of what they do. And AI models are already being used to violate people's human rights in the United States and in other countries, to commit crimes, and that's bad. \[chuckle\] Yeah, there's\... One of the worst applications is that there has been a revival in phrenology recently, so there are a lot of police departments that they have gotten really into the idea that they can use AI analysis of video cameras to determine who is \*going\* to commit crimes. And this, shockingly, results in over-surveillance of minority populations and violation of human rights left and right, and it's a huge clusterfuck and the police don't care.

\[pause\]

**0:03:10.2 Vael:** Awesome. Well, not awesome, but. So that question was, what are you most excited about AI and what are you most worried about; biggest benefits and risks?

**0:03:19.8 Interviewee:** Gotcha. What am I most excited about? I'm excited about the opportunity to interact with computers via natural language. So one of the really interesting things about some recent research is that we've been able to move away from traditional coding interfaces for certain tasks due to the way we've able to automate things. Probably the most prominent example of this is that there is a burgeoning online AI-generated art community, where they take pre-trained models and they write English sentences and they provide the\... What the model does is it takes a English sentence input and draws a picture, and it's shockingly good and has an understanding of styles. If you want it to be\... If you say in the style of Van Gogh, or high contrast, or low-poly render. You can induce visual effects by using language like that, and I think that's phenomenally cool, and it's gotten a lot of people\... There's a lot of people who've gotten into using this kind of technology who otherwise wouldn't have\... \[who it\] really wouldn't have been accessible to because of their lack of coding knowledge and understanding of AI. They couldn't have developed the algorithms that run on the backend for this on their own. Recently, yesterday I saw another blog post about how they were able to develop simple video games using GTP-3. It just wrote the code for them. I think that the ability to write a text description of something that you're interested in, which is a medium that everyone can relate to and interact with, or that most people can relate to and interact with far more than regular programming, for example, is really powerful and really awesome.

**0:05:30.8 Vael:** Yeah, I see a lot of themes of accessibility in all of these risks and benefits and work. I thought you were going to bring up Codex but yes, art generation. It's very cool.

**0:05:41.7 Interviewee:** Oh yes.

**0:05:43.8 Vael:** Yeah. So, thinking about a future AI, putting on a science fiction forecasting hat, say we're 50 plus years into the future. So at least 50 years in the future, what does that future look like?

**0:05:56.3 Interviewee:** I have absolutely no idea, and anyone who says otherwise is wrong.

**0:06:01.3 Vael:** Okay. Do you think AI will be important in it, or probably not?

\[pause\]

**0:06:16.6 Interviewee:** I think that's more of a sociological question than it is a technical question. The class of problems and the class of algorithms that are considered AI has changed dramatically over the past 50 years, and there's entire books have been written about this topic. At a basic level, my hesitancy is that I don't know people will consider AI in 50 years. There's a very real possibility in my mind that GTP-3 will no longer be considered an AI.

**0:06:48.5 Vael:** What will be considered?

**0:06:53.3 Interviewee:** A text generation algorithm? A good example of this is simple game-playing agents, so you can write an algorithm that can play Tic-Tac-Toe perfectly or can play Connect Four or Checkers really well. Like, will beat any human. And a lot of people don't call those AIs anymore because they don't\... 'Cause they're search algorithm-based. They apply a lot of computational power to look through a space of possible events, and they find the best event. And they don't really\... The argument is that they don't reason, or they don't know anything about strategy. And this is often to contrast it with more recent AIs for playing games like the AlphaGo, AlphaZero models that DeepMind has produced where top level chess players certainly get beat by these algorithms just like they get beat by\... have been beaten by algorithms for 20 years, but for kind of the first time people are able to study and interpret these algorithms and learn and improve their play as a human. Which is really cool. But that kind of dichotomy is often used to dismiss or remove the label of "AI" from prior work and stuff that have been considered AI at that time. And I could certainly see that happening with GTP-3, for example, because it's really, really terrible at reasoning. And if in 50 years we have chatbots that can answer knowledge-based questions the way, say, a sixth grader could, and reason about basic word problems and stuff, pass some kind of reasoning examination, then I could easily see people no longer considering GTP-3 an AI, because it's not intelligent, it's just babbling and making up words.

**0:08:54.4 Vael:** Right. Cool, I'm now going to go on a spiel. So people talk about the promise of AI, by which they mean a bunch of different things, but the thing that I'm most thinking about is a very general system with the capabilities to replace all current day human jobs, so like a CEO AI or a scientist AI, for example. Whether or not we choose to replace human jobs is a different question, but I usually think about this in the frame of\... in 2012 we had AlexNet, deep learning revolution. You know, here we are 10 years later, we have GPT-3, like we're saying, which has some weirdly emergent capabilities: new language translation, and coding and some math and stuff, but not very well. And then we have a bunch of investment poured into this right now, so lots of young people, lots of money, lots of compute, lots of\... and if we have algorithmic improvements at the same rate, and hardware improvements at the same rate, like optical or quantum, then maybe we reach very general systems or maybe we hit a ceiling and need to do a paradigm shift. But my general question is, regardless of how we get there, do you think we'll ever have these very general AI systems, like a CEO or a scientist AI? And if so, when?

\[pause\]

0:10:07:5\...Oh, you're muted. I think. Oh, no. Oh, no. Can't hear you. I don't think anything's changed on my end. Okay.

**0:10:25.2 Interviewee:** Hello?

**0:10:26.2 Vael:** Yeah. Cool.

**0:10:27.2 Interviewee:** Okay. I think my headphones may have done something wacky. I don't know, I would be extremely surprised if the answer\... like I know that there are people who say that the answer is less than 10 years, and I think that's absurd. I would be surprised if the answer is less than 50 years, and I don't feel particularly confident that that will ever happen.

**0:10:47.1 Vael:** Okay. So it may or may not happen. Regardless, it's going to be longer than 50 years. Is that right?

**0:10:54.1 Interviewee:** Hello?

**0:10:55.2 Vael:** Hello, hello, hello?

**0:11:00.7 Interviewee:** Yes.

**0:11:00.8 Vael:** Okay, cool. So my question was like, all right, you don't know whether or not it will happen. Regardless, it will take longer than 50 years. Is that a summary?

**0:11:07.3 Interviewee:** Mm-hmm.

**0:11:09.0 Vael:** Yeah. Okay, cool. So one of my question is like, why wouldn't it eventually happen? I kind of like believe in the power of human ingenuity, and people following economic incentives such that\... These things are just really quite useful, or systems that can do human tasks are generally quite useful, and so I sort of think we'll get there eventually, unless we have a catastrophe in some way. What do you think about that?

\[pause\]

**0:11:45.0 Interviewee:** It's going to be extremely difficult to develop something that is sufficiently reliable and has an understanding of the world that is sufficiently grounded in the actual world without doing some kind of mimicking of human experiential learning. So I'm thinking here reinforcement learning in robots that actually move around the world.

**0:12:13.0 Vael:** Yeah.

**0:12:13.9 Interviewee:** I think without something like that, it's going to be extremely difficult to tether the knowledge and the symbolic manipulation power that the AIs have to the actual contents of the world.

**0:12:29.5 Vael:** Yep.

**0:12:29.9 Interviewee:** And there are a lot of extremely, extremely difficult challenges in making that happen. Right now, cutting-edge RL techniques are many orders of magnitude\... Require many orders of magnitude too much data to really train in this fashion. RL is most successful when it's being used in like a chess context, where you're playing against yourself, and you can do this in parallel, and that you can\... When you can do this over and over and over again. And if you think about an actual robot crossing the street, if an attempt takes 10 seconds, and I think especially early in the learning process, that's an unreasonably small amount of time to estimate. But if an attempt takes 10 seconds and\... Let me pull out the calculator for a second.

**0:13:28.1 Interviewee:** And you need one million attempts\... then that would take you\... about a third of a year to do. And I think that both of those numbers are wrong. And I think the number of attempts is orders of magnitude too small. There's very, very little that we can learn via reinforcement learning in a mere one million attempts. And this is just one task. If you want something that can actually move around and interact with the world, even if you're using these highly optimistic, currently impractical estimates, you can't take four or five months to learn how to cross the street. If that's your paradigm, you're never going to be able to build\-- you're never going to get to stuff like managing a company. \[chuckle\]

**0:14:38.0 Vael:** Yeah. That makes sense. Yeah, I think\-- I think\... this makes sense to me, and I'm like, "Wow, our current state systems are really not very good." But also, I think I often view this from a lens of pretty far back. So I'm like, 10000 years ago, humans were going around and the world was basically the same from one generation to the next, and you could expect things to be similar. And now we've had the agriculture revolution and industrial revolution, in the past couple of hundred years, we have done\... We're kind of on an exponential curve in terms of GDP, for example, and I would expect that\... And we've only been working on AI for, I don't know, less than 100 years. And we have\... We now have something like GPT-3, which sounds sort of reasonable, if you're just looking at it, and of course it's not very\... It's not, like, grounded, which is a problem.

**0:15:28.3 Vael:** But I sort of just expect that if we spend another\... I don't know, you could spend hundreds of years working on this thing, and if it continues to be economically incentivized\... This is kind of how human progress works. I just kind of expect us to advance up the tech tree to solve the software improvements, to solve the hardware improvements. Or new paradigms maybe. Even at the worst case, I guess we advance enough in neuroscience and scanning technologies to just scan human brains and make embodied agents that way or something. I just expect us to get to some capabilities like this eventually.

**0:16:03.5 Interviewee:** In my mind, really the fact that there's only so fast we can move around in the real world is a huge constraint. Even if you can learn extremely complicated and abstract things embedded in the real world as an actual robot, take my crossing street example, even if you could\... doing an attempt at\... So even if you could learn pretty much any task in a thousand iterations, some tasks take a very long time to develop. Humans don't learn to be CEOs of companies very quickly, and it doesn't seem like it's very shortcut-able to me. I also don't think CEOs of companies is perhaps the best example, but let's say\...

**0:16:58.1 Interviewee:** Let's say you wanna train a robot to operate a McDonalds. That's a very large amount of destroyed meat that you need to buy, it's a very large amount of time and materials to even set up the apparatus in which you could actually train a robot to perform that task. And you're talking about economic incentives, where is the economic incentive to burning million patties to get to the point where your robot can flip one over successfully? When we're talking about moving around and interacting in the real world, those interactions have costs that are financial in addition to being time-consuming. If we want to train an AI to\... Via reinforcement learning technique, which is certainly a caveat that have to add to a lot of what I'm saying. But if we wanna train a robot to drive a car via a reinforcement learning-like technique, at some point you need to put it behind the wheel of a car and let it drive 100, 1000 cars. And you're going to destroy a lot of cars doing that, and you're probably going to kill people. So that's a very large disincentivizing cost.

**0:18:30.1 Vael:** Okay. Alright. So the idea is like\... if we're doing robotics, then we need to\... and the training paradigm is not, like, humans where you can kind of sit them down, and\... Humans don't actually crash cars, usually\... I mean, sometimes. Teenage humans crash cars sometimes. But in their training process, they don't usually require that many trials to learn, and they can do so kind of quickly. So I'm like, I don't know. Do we expect algorithms at some point to require much less training data than current ones do? Because current ones require a huge amount of training data, but I kind of imagine we'll get more efficient systems as\... More efficient per data as we go along.

**0:19:24.9 Interviewee:** Are you saying that you think that you can sit down and explain to someone how to drive a car and they can drive it without crashing?

**0:19:30.9 Vael:** I think, that.. We have\... I think that if we take a human, and I'm like, "All right, human, I'm going to\... I want you to learn how to drive this car. And I'm going to sit next to you. And I'm going to tell you what to do and what not to do, and you're going to drive it." I think they can, indeed, after practicing some period of time, which for humans, it's like hours. It's on the order of tens of hours, then they can basically sit there and not crash a car. And I kind of expect similar paradigms eventually for AI systems.

**0:20:04.4 Interviewee:** That seems extremely non scalable.

**0:20:11.8 Vael:** Uh\... Okay. You're like, look, if it takes tens of hours to train every AI system?

**0:20:17.6 Interviewee:** No, I'm thinking mostly about the human sitting next to them giving them constant feedback actually.

**0:20:28.5 Vael:** But the nice thing about AI is you can copy them as soon as one person spends that many hours. You can just take that, takes the thing that's\-- like its new neural net, pass it onto the next one.

\[pause\]

**0:20:48.0 Interviewee:** \...Maybe.

**0:20:50.1 Vael:** And I don't think this has to happen anytime soon. But I do think eventually given that\... I don't know, I can't imagine humans being like, "All right, cool. We're efficient enough. Let's just stop now. We've got like GPT-3. Seems good. Or GPT-5, let's just stop here."

**0:21:08.4 Interviewee:** So nobody has ever taken two different robots, trained one of them in the real world to perform a task, and then transferred the algorithm over and allowed the other one to perform the same task as successfully, as far as I'm aware.

**0:21:23.6 Vael:** Yep. I totally believe today's systems are not very good.

**0:21:31.2 Interviewee:** It is, I think, I think that.. Anything we can really say about this is inherently extremely speculative. I'm certainly not saying it could never happen. I'm just\... Sorry. I'm certainly not saying it can't happen. I'm just saying it could never happen. There we go.

**0:21:45.0 Vael:** Okay. All right. Okay. That makes sense. How likely do you think it is that we'll get very capable systems sometime ever in the future?

**0:21:53.5 Interviewee:** I have no idea.

**0:21:56.5 Vael:** \...Well, you have some idea because you know that it\... Well, okay. You said that it can't\... You're like, it's higher than zero.

**0:22:04.9 Interviewee:** Yes.

**0:22:05.9 Vael:** Yes. And you don't sound like you think it definitely will happen, so it's less than 100.

**0:22:12.6 Interviewee:** Yes.

**0:22:13.8 Vael:** Okay. And it's anywhere in that scale? I mean, slightly higher than zero and slightly less than 100.

**0:22:22.5 Interviewee:** That sounds like an accurate description of my current level of uncertainty.

**0:22:26.8 Vael:** Interesting. Man. Is it.. Hard\-- I mean like how\-- You do have predictions of the future though, for the near future, presumably, and then it just like tapers off?

**0:22:36.7 Interviewee:** Mm-hmm.

**0:22:38.2 Vael:** Okay. And anything\... And you say definitely not 10 years, but after. And then like 50 years. So like 50 years out, you're\... It starts going from approximately zero to approximately 100?

**0:22:56.6 Interviewee:** Um\... I think it is unlikely to happen in the next 50 years.

**0:23:00.0 Vael:** Okay.

**0:23:05.7 Interviewee:** I would assign a less than 25% probability to that. But I don't think I can deduce anything about my expectation at 100 years based on that information. Other than it\... yeah.

**0:23:21.6 Vael:** Great. Thanks. All right. I think that's good enough for me to move on to my next question.

**0:23:28.6 Vael:** So my next question is thinking about these highly intelligence systems in general, which we're positing maybe will happen sometime. And so say we have this sort of CEO AI through, I don't know, maybe hundreds of years in the future or whatever. I'm like, "Alright, CEO AI. I want you to maximize profits and try not to run out of money and try not to exploit people and try to avoid side effects." And currently, obviously this would be very technically challenging for many reasons. But one of the reasons is that we currently don't have a good way of taking human values and preferences and goals and stuff, and putting them in mathematical formulations that AI can optimize over. And I worry that this actually will continue to be a problem in the future as well. Maybe even after we solve the technical problem of trying to get an AI that is at all capable. So what do you of the argument, "highly intelligent systems will fail to optimize exactly what their designers intended them to, and this is dangerous?"

**0:24:26.0 Interviewee:** I mean I think that the statement that highly intelligent systems will fail to optimize what their designers intend them to is a slam dunk. Both human children and current AIs do not do that, so I don't see any particular reason to think we will\-- that something that's like, in some sense in between those, we'll have a whole lot more success with.

**0:24:50.4 Vael:** Interesting. Okay, cool.

**0:24:58.5 Interviewee:** Yeah. Did you turn out exactly the way your parents wanted you to? \[laughter\] I didn't. I think the overwhelming majority of people don't, and that's not a flaw on their part. But\... yeah.

**0:25:15.2 Vael:** All right. Yep. Yeah, certainly there's some alignment problems with parents and children. Within human-humans even. And then I expect\-- I kind of expect the human-AI one to be even worse? My intuition is that if you're having an AI that's optimizing over reality in some sense, that it's going to end up in alien parts of the space\-- alien to humans, because it's just optimizing over a really large space. Whereas humans trying to align humans have at least the same kind of evolutionary prior on each other. I don't know. Do you also share that?

**0:25:47.7 Interviewee:** I'm not sure. I think that you're going to have to get a lot of implicit alignment to end up in a place where you're able to train these things to be so intelligent and competent in the first place.

**0:26:07.6 Vael:** That makes sense to me. Kind of like\--

**0:26:09.8 Interviewee:** Yeah. What percentage of the way that gets you there is a very important and totally unknown question. But I don't think that the value system of one of these systems is going to be particularly comparable to like, model-less RL, where they're trying to optimize over everything.

**0:26:33.4 Vael:** Could you break that one down for me?

**0:26:38.2 Interviewee:** In what way?

**0:26:41.6 Vael:** I didn't\... I don't quite understand the statement. So the value system will not be the same that it is in model-less RL. I don't have a super good idea of what model-less RL is and how that compares to human systems or human-machine\--

**0:26:56.1 Interviewee:** Okay. So model-less RL is a reinforcement learning paradigm in which you are basically trying to learn everything from the ground up, via pure interaction.

**0:27:06.2 Vael:** Okay.

**0:27:07.1 Interviewee:** So if you're thinking of a game-playing agent, this is typically an agent that you're not even programming with the rules. It learns what the rules are because it walks into a wall and finds that it can't walk further in that direction. That's the example in my head of something that's optimizing over all possible outcomes currently. \...Sorry, I lost the train of the question.

**0:27:39.7 Vael:** I was like: how does that relate to human value systems?

**0:27:46.4 Interviewee:** I think that the work that we will have to do to train something to move around and interact in the world and perform these highly subjective and highly complex tasks that require close grounding in the facts of the world will implicitly narrow down the search space. Significantly.

**0:28:10.1 Vael:** Okay. Yeah\--

**0:28:11.6 Interviewee:** I do think that there's a\... Yeah.

\[pause\]

**0:28:25.8 Interviewee:** Yeah.

**0:28:26.3 Vael:** Yeah. Yeah, I often think of this in terms of like, you know how the recommender systems are pretty close to what humans want, but they're also maybe addictive and kind of bad and optimizing for something a little bit different than human fulfillment or something. People weren't trying to maximize them for human fulfillment per se. But yeah, I like\-- like that sort-of off alignment is often something I think about. Alright.

So this next question is back to the CEO AI, so imagine that the CEO AI is good at multi-step planning and it has a model of itself in the world, so it's modeling other people modeling it, 'cause that seems pretty important in order for it to do anything. And it's making its plans for the future, and it notices that some of its plans fail because the humans shut it down. And it's built into this AI that it needs human approval for stuff 'cause it seems like a basic safety mechanism, and the humans are asking for a one-page memo to describe its decision.

**0:29:21.4 Vael:** So it writes this one-page memo, and it leaves out some information because that would reduce the likelihood of the human shutting it down, which would increase the likelihood of it being able to achieve the goal, which is like, profit plus the other constraints that I mentioned. So in this case, we're not building in self-preservation to the AI itself, it's just, self-preservation is arising as a function\... \[as a\] instrumental incentive of an agent trying to optimize any sort of goal. So what do you think of the argument, "highly intelligent systems will have an incentive to behave in ways to ensure that they are not shut off or limited in pursuing their goals, and this is dangerous?"

**0:30:00.4 Interviewee:** It seems likely correct.

**0:30:02.3 Vael:** Interesting. Okay. \[chuckle\] \...I'm not excited about that answer, 'cause other instrumental incentives are acquiring resources and power and influence, and then also not wanting\... Having a system that's optimizing against humans seems like a very bad idea in general, which makes me worried about the future of AI. If the thing that we're going to build is eventually by default, maybe not going to want to be corrected by humans if we get the optimization function wrong the first time.

**0:30:32.5 Interviewee:** Yeah.

**0:30:36.2 Vael:** \[laughter\] Okay. Have you thought about this one before?

**0:30:38.6 Interviewee:** Yes.

**0:30:39.4 Vael:** Yeah. Cool. Have you heard of AI alignment?

**0:30:42.2 Interviewee:** Yes.

**0:30:43.2 Vael:** Yeah. And AI safety and all the rest of it?

**0:30:45.6 Interviewee:** Mm-hmm.

**0:30:46.0 Vael:** Yeah. How do you orient towards it?

**0:30:49.8 Interviewee:** I think that most people who work in it are silly. And don't take the right thing seriously.

**0:30:57.9 Vael:** Mm. What should they take seriously? And what don't they?

**0:31:02.9 Interviewee:** I know a lot of people who are afraid that future research along the lines of GTP-3 is going to rapidly and unexpectedly produce human-like intelligence in artificial systems. I would even say that that's a common, if not widespread attitude. There are pretty basic kinds of experiments that we'll need to do to test the plausibility of this hypothesis, that nobody seems really interested in doing.

**0:31:48.5 Vael:** Hm. Seems like someone should do this?

**0:31:51.2 Interviewee:** Yeah. When I talk to most people who describe themselves as alignment researchers, and I try to put myself in their shoes in terms of beliefs about how agents work and what the future is likely to look like. The things I see myself experimenting with and working on, are things that nobody is working on. And that really confuses me. I don't understand\... So here's an interesting question: how much experience do you have actually using GTP-3 or a similar system?

**0:32:31.9 Vael:** Yeah, not hardly at all. None. So I've seen examples, but haven't interacted with it myself.

**0:32:38.4 Interviewee:** Okay, um\... Would you like to?

**0:32:47.2 Vael:** Uh\... Sure? I mean, I guess I've messed around with the Dungeon AI one, but\... Does seem interesting.

**0:32:57.1 Interviewee:** Hm. \...So my experience is that\... A widespread observation is that they don't seem to have a worldview or a perspective that they\-- are expressing words, so much as many of them. Some people like to use the term multiversal. It's\... kind of the way I think about it is that there are many people inside of GTP-3 and each time you talk to it, a different one potentially can talk to you.

**0:33:42.1 Vael:** Yep.

**0:33:43.8 Interviewee:** This seems to be an inherent property of the way that the model was trained and the way that all language models are currently being trained. So a pressingly important question is, to what extent does this interfere with\... Let's, to make language easier, call it one of its personalities. Let's say one of its personalities wants to do something in the world: kill all the humans or even something mundane. To what extent does the fact that it's not the only personality interfere with its ability to create and execute plans?

**0:34:28.2 Vael:** \...Ah\... Current systems seem to not\... Well, okay. It depends on how we're training it, because GPT-3 is confusing. But AlphaGo seems to kinda just be one thing rather than a bunch of things in it. And so it doesn't seem like it has conflicts there?

**0:34:46.1 Interviewee:** I would generally agree with that.

**0:34:48.2 Vael:** Okay. But we're talking about scaling up natural language systems and they don't\... And they don't\... They have lots of different types of responses and don't\... on one personality. Uh\... Well, it seems like you could train it on one personality if you wanted to, right? If you had enough data for that, which we don't. But if we did. And then I wouldn't really worry about it having different agents in it.

**0:35:17.6 Interviewee:** That's a very, very, very, very, very, very, very, very large amount of text.

**0:35:23.8 Vael:** Yeah. \[Interviewee laughter\]

**0:35:25.0 Vael:** Yeah, yeah that's right!

**0:35:26.5 Interviewee:** Do you any\-- do you have any scope of understanding for how much text that is?

**0:35:32.8 Vael:** Yeah, I'm actually thinking something like pre-training on the whole internet, and then post-train on a single person, which already doesn't work that well. And so then it wouldn't actually help if that pre-training procedure is still on\... Still on the whole thing. Um, okay\--

**0:35:48.4 Interviewee:** So a page of written text is about 2 kilobytes in English. And these models are typically trained for between one and five terabytes, so no human has come anywhere close to putting out five billion pages of total text.

**0:36:13.7 Vael:** Yeah.

**0:36:18.1 Interviewee:** It's so astronomically far beyond what any human would actually ever write, that it doesn't seem very plausible unless something fundamentally changes about the way humans live their lives.

**0:36:30.8 Vael:** Or about different training procedures. But like\--

**0:36:33.8 Interviewee:** Yeah, yeah, yeah, yeah. But like the idea that one could do something similar to current pre-training procedures that is meaningfully restricted to even say a 100 people that have been pre-screened for being similar to each other. 100 people are also not going to put out five billion pages of text.

**0:36:49.6 Vael:** Yeah.

**0:36:51.6 Interviewee:** \[laughter\] It's just so much data\...

**0:36:54.1 Vael:** Yeah. Yeah, I don't know how efficient systems will be in the future, so\... Yeah. Let's take it as\... Yeah, sure. But they're going to have multiple personalities in them, in that they are trained on the internet.

**0:37:05.1 Interviewee:** Mm-hmm.

**0:37:06.1 Vael:** And then you're like, "Okay. Does that mean that\... " And then there's a frame here that is being taken where we have different\... Something like arguing? Or like different agents inside the same agent or something? And so then you're like, "Well, has anyone considered that? Have we tested something like that?"

**0:37:26.9 Interviewee:** Yeah, that's kind of close to what I'm saying.

**0:37:29.6 Vael:** Hmm.

**0:37:31.9 Interviewee:** So, to take your CEO example. In order for it to be successful, it needs to\... at no point\... There's certain information it needs to consistently hide from humans. Which means that every time it goes to generate text, it needs to choose to not share that information.

**0:37:47.1 Vael:** Yeah.

**0:37:48.1 Interviewee:** So if the system looks even vaguely like GTP-3, it seems to me like it will not be able to always act with that\... generate text with that plan. And so there's a significant risk in it compromising its own ability to keep the information hidden.

**0:38:13.7 Vael:** Okay.

**0:38:13.7 Interviewee:** Alternatively, even if it's\... That's a more direct way that they can interfere with each other. But even less directly, if I have somewhere I want to go and I go drive the car for a day, and then you have somewhere you want to go and you drive the same car for a day, and we trade off control, there are things I'm going to want to do that I have trouble doing because I only control the body and the car at the end of the day.

**0:38:40.8 Vael:** Quick question. Are you expecting that AI systems or multi-agent properties are more\... have more internal conflict than humans do? Which can also be described in some sense as having multiple agents inside of them?

**0:38:54.7 Interviewee:** Yes.

**0:38:55.7 Vael:** Okay.

**0:38:57.4 Interviewee:** I think that anyone whose worldview is as fractured and inconsistent as GPT-3 probably has a clinical diagnosis associated with that fact.

**0:39:08.8 Vael:** Yeah. And you don't think that these will get more targeted in the future as we direct language models to do specific types of tasks, something like math?

**0:39:24.2 Interviewee:** I think that achieving, even\... achieving 95, 99%, let's say, coherency between generalization, so if you imagine every time the model is used to generate text, there's some worldview it's using to generate that text, and you want each time those different worldviews used to be consistent with each other. Even achieving 99% consistency, I'm not asking for 100% consistency but 95, 99 seems like something necessary for it to make multi-year long-term plans.

**0:40:10.7 Vael:** That seems right.

**0:40:13.5 Interviewee:** This is exceptionally difficult and there are very likely fundamental limitations to the extent to which a system can achieve that level of coherence in the current training paradigms. And\...

**0:40:31.7 Vael:** Seems plausible.

**0:40:34.7 Interviewee:** That would be very good news to people who are afraid that GTP-7 is going to take over the world.

**0:40:43.4 Vael:** Yeah, yeah. Okay, alright, 'cause I'm like, I don't know, I feel I'm kind of worried about any future paradigm shift. But current people definitely are worried about GPT-3 specific or GPT systems, and the current paradigms, specifically.

**0:40:56.3 Interviewee:** I've spoken to these people at length and I've talked to them about what they're afraid of and stuff. \...There seem to be a significant number of people in the alignment community who\... If you could put together a convincing argument that the current pre-training methodology, as in, the fact that it's trained on a widely crowdsourced text generation source, instills some kind of fundamental worldview inconsistency that is exceptionally difficult if even possible to resolve, would alleviate a lot of the anxiety. It would actively make these people happier and less afraid about the world.

**0:41:38.7 Vael:** That seems true. I think if you can\... If there's a fundamental limit on capabilities, just like, of AI, then that's good for safety because then you don't get super capable systems. And I'm like, "Yeah, that makes sense to me." And do you think that this capability issue is going to be something like\... coherence of generated text. And that might be a technically fundamental limitation. Cool\--

**0:42:06.1 Interviewee:** I know people who have the tools and resources and time to test, to run experiments on things like this, who I've even directly proposed this to. And they've gone, "Oh, that's interesting." And then not done it.

**0:42:22.3 Vael:** Yeah, my intuition is that they don't\... I think you have to have a pretty strong prior on this particular thing being the thing that is going to have like a fundamental limit in terms of capabilities in order to want to do this compared to other things, but\... That makes sense, though. It sounds like you do have a\... You do think this particular problem is pretty important. And pretty hard to\... Very, very difficult.

**0:42:45.4 Interviewee:** I think that this coherency problem is a serious issue for any system that is GTP-3 like, in the sense that it's trained to produce tokens or reasoning or symbols or whatever you want to say, but that produce outputs that are being fit to mimic a generative distribution\-- sorry, it's being generatively trained to produce outputs that mimic a crowdsourced human distribution.

**0:43:19.8 Vael:** Yeah. Cool, awesome. Yup, makes sense to me as a worldview, is pretty interesting. I haven't actually heard about that problem\-- of people thinking that that problem specifically, the coherency problem, is one that's going to fundamentally limit capabilities. Seems plausible, seems like many other things might end up being the limit as well. And then you're like, "Well, people should like\... If this is the important thing, then people should actually test it. And then they'll feel better. Because they'll believe that these systems won't be as capable and then less likely to destroy the world." Yeah, this makes sense to me.

**0:44:02.4 Interviewee:** Yeah. Another aspect of this is that research into the functional limitations, in a sense, is extremely difficult to convert into capabilities research, which is something that a lot of people say that they're highly concerned about. And that they don't want to do many types of research because\... There was that Nature article where they were creating a medical AI and they were like, "Let's put a negative sign in front of the utility function." And it started designing neurotoxins. Do you know what I'm referring to?

**0:44:35.2 Vael:** No, but that sounds bad.

**0:44:36.8 Interviewee:** Oh yeah, no, it's just\... That was the Nature article. It was like "we were synthesizing proteins to cure diseases, and we stuck a negative sign in front of the utility function\-- (Vael: Oh, was that last week or something?) Yeah.

**0:44:46.2 Vael:** Yeah, okay, so I did see that, yeah. Huh.

**0:44:48.1 Interviewee:** Yeah. Gotta love humans. Gotta love humans.

\[chuckle\]

**0:45:02.6 Vael:** \...Awesome. Ah, I think I'll.. Maybe.. Hm. Okay. So. What would\... make you want to work on alignment research as you think it can be done?

\[pause\]

**0:45:25.2 Interviewee:** That's an interesting question. \[pause\] I guess the main thing would be being convinced of the urgency of the problem.

**0:45:50.4 Vael:** That makes sense. Very logical.

**0:45:56.4 Interviewee:** To be blunt, I don't tend to get along with the kind of people who work in that sphere, and so that's also disincentivizing and discouraging.

**0:46:12.2 Vael:** Yeah, that makes sense. I've heard that from at least one other person. Yeah. Alright, so timelines and also nicer research environment. Makes sense.

**0:46:27.4 Interviewee:** You could even say nicer researchers.

**0:46:31.0 Vael:** Yep. Nicer researchers. Apologies? \...Yeah. Cool. And then my last question is, have you changed your mind on anything and during this interview, and how was this interview for you?

**0:46:45.9 Interviewee:** The interview was fine for me. I don't think I've changed my mind about anything.

**0:46:56.0 Vael:** Great. Alright, well, thank you so much for being willing to do this. I definitely\... Yeah. No. You have a very coherent kind of worldview thing that's\... That I\... Yeah. I appreciate having the ability to understand or have access to or listen to, rather.

**0:47:11.7 Interviewee:** My pleasure.

**0:47:13.5 Vael:** Alright. I will send the money your way right after this, and thanks so much.

**0:47:17.2 Interviewee:** Have a good day.

**0:47:17.6 Vael:** You too.

**0:47:17.8 Interviewee:** Bye.


# Interview with AI Researchers individuallyselected_84py7 by Vael Gates

Interview with 84py7, on 3/18/22
================================

**0:00:00.0 Vael:** Here we are. Perfect. So my first question is, can you tell me about what area of AI you work on in a few sentences?

**0:00:09.0 Interviewee:** Yeah. I'm transferring my research from essentially pure mathematics to AI alignment. And specifically, I plan to work on what I've been calling weak alignment or partial alignment, which is not so much trying to pin down exactly a reward function that's in the interest of humanity but rather train AI systems to have positive-sum interactions with humanity.

**0:00:44.8 Vael:** Interesting. Cool, I expect we'll get into that a little bit further on. \[chuckle\] But my next question is, what are you most excited about in AI, and what are you most worried about? In other words, what are the biggest benefits or risks of AI?

**0:01:00.1 Interviewee:** Yeah, biggest benefits\... I think AI has the potential to help us solve some of the biggest challenges that humanity's facing. It could potentially teach us how to solve climate change, or at least mitigate it. It could help us avert nuclear war, avert bio-risks, and maybe most importantly, avert other AI risks. So that's the upside. The downside is exactly those other AI risks, so I worry about a potentially small research group coming out of a for-profit company, which might have some safety aspect, but the safety could be window dressing. It could be something that's mostly a PR effort, something that just exists to satisfy regulators. And at the end of the day when\... or if they manage to develop superhuman AI, the safety people will be marginalized, and the AI will be used in the interest of one company or even one or a few individuals. And that could be potentially very bad for the rest of us. There's also the issue of an arms race between multiple AI projects, which could be even worse. So those are some of my\... broadly, some of my worries.

**0:02:35.7 Vael:** Interesting. So just to get a little bit more straight on the\... Okay, so the second story is arms race between the AI orgs. And the first one is\... why are the AI researchers getting\... why are the safety researchers getting marginalized?

**0:02:47.5 Interviewee:** Well, if you look at, for example, financial institutions before the 2008 crisis, it's not like they had no regulation, although the banks and insurance companies had been gradually deregulated over several decades but still, there were multiple regulators trying to make sure that they don't take on too much leverage and aren't systemic risks. And nevertheless, they found ways to subvert and just get around those regulations. And that's partly because regulators were kind of outmatched, they had orders of magnitude, less funding, and it was hard to keep up with financial innovations. And I see the same potential risks in AI research, potentially even worse, because the pace of progress and innovation is faster in AI, and regulators are way behind, there doesn't even exist meaningful regulation yet. So I think it's easy for a team that's on the verge of getting a huge amount of power from being the first to develop superhuman artificial intelligence to just kind of push their safety researchers aside and say, "You know what? You guys are slowing us down, if we did everything you said, we would be years slower, somebody else might beat us, and we should just go full speed ahead."

**0:04:25.3 Vael:** Interesting. Yeah, I guess both of those scenarios aren't even\... we don't solve the technical problem per se, but more like we can't coordinate enough, or we can't get regulation good enough to make this work out. So that's interesting. Do you think a lot about policy and what kind of things policy-makers should do?

**0:04:44.6 Interviewee:** No, I'm sort of pessimistic about governments really getting their act together in a meaningful way to regulate AI in time. I guess it's possible if the progress slows and it takes many decades to get to a superhuman level, then maybe governments will catch up. But I don't think we can rely on that slow timeline. So I think more optimistically would be\... There are only a small number of tech incumbents, and plausibly they could coordinate with each other to avoid the kind of worst Red Queen arms race to be first and to put their own safety measures into place voluntarily. So if I were doing policy, which I'm not, that's the direction I would try to go in. But I think beyond policy, the technical problem of how to solve alignment is still wide open, and that's personally where I feel I might be able to contribute, so that's my main focus.

**0:05:56.8 Vael:** Interesting. Yeah, so sort of branching off \[from\] how long it will take: focusing on future AI, putting on a science fiction forecasting hat, say we're 50-plus years into the future. So at least 50 years in the future, what does that future look like?

**0:06:13.3 Interviewee:** I think that's a really open question. In my weak or partial alignment scenario, that future involves a few dominant platforms that allow for the development of advanced AI systems. And because there are only a few platforms, they all have strict safety measures sort of built in from the ground up, maybe even from the hardware level up. And that allows even small companies or potentially even individuals to spin up their own AGIs. And so there's this kind of giant ecosystem of many intelligent agents that are all competing to some extent, but they also have a lot of common interest in not blowing up the current world order. And there's a kind of balance of powers, where if one agent gets too powerful, then the others coordinate to keep it in check. And there's a kind of system of rules and norms which aren't necessarily based on legal systems, because legal systems are too slow, but they're a combination of informal norms and formal safety measures that are sort of built into the agents themselves that keep things roughly in balance. That's the kind of scenario I hope for. It's very multi-polar, but it's really\... There are so many agents, and no one of them has a significant portion of the power. There are of course many worse scenarios, but that's my optimistic scenario.

**0:08:04.4 Vael:** Interesting. Yeah, so when you say there's only a few platforms, what's an example of a platform or what that would look like?

**0:08:11.2 Interviewee:** Well, today, there's TensorFlow, and there's PyTorch and so on. In principle you could build up your own machine learning tools from scratch, but that's a significant amount of effort even today. And so most people go with the existing tools that are available. And decades\... 50 years from now, it will be much harder to go from scratch, because the existing tools will be way more advanced, there'll more layers of development. And so I think for practical purposes, there will be a few existing best ways to spin up AI systems, and the hope is that all those ways have safety measures built in all the way from the hardware level. And even though in principle somebody could spin up an aligned AI from scratch, that would be an enormous effort involving just so much\... Down to chip factories. It will be easy to detect that kind of effort and stop it from getting off the ground.

**0:09:26.4 Vael:** Interesting. Yeah, that is super fascinating. So how\... What would that look like, for safety to be built into the systems from a hardware level? What would these chips look like, what sort of thing?

**0:09:38.8 Interviewee:** Yeah, that's\... I've been thinking about that. And I don't have a detailed vision of how that would work, but that's\... One direction I might take my research is looking into that. One idea I have is to kind of build in back doors to the AI. So there's a range of types of back door, ranging from like Achilles heel, which is a kind of designed weakness in the AI that humans can take advantage of if things go wrong. Moving from that, slightly stronger than that is a kind of off switch which can just shut the AI down if things get really bad. The thing I worry with off switches is they're too binary. If an AI actually has a lot of power, it's probably benefiting some humans, and there will be political debate about whether to turn it off. And the decision will take too long, and things could get out of control. So what I'm looking into is a more\... Something more flexible than an off switch, which I've been calling a back door, which is a way to\... Well, okay, there's two types of things. So first, there's a throttle, which is like you can fine-tune the amount of resources you give to the AI. If it's behaving well, you can give it more compute, more memory, access to more cloud resources, more data centers and so on. If it starts doing things that seem a little fishy, you can just tune that stuff back and examine it, which might be an easier political decision than just turning it off, which would be very disruptive.

**0:11:22.7 Interviewee:** So that's a throttle. And then even more flexible than that is a back door, where you could actually modify the code or parts of the code base of the AI as it's running. And again, that could be easier to\... Politically easier to implement, because you don't have this very disruptive effect of just shutting the thing down, you could just kind of tweak it as it goes. So how to do that from a hardware level? It's unclear to me whether that's feasible at all. And looking into trusted computing platforms, software that can only run on designated hardware, software watermarking, obfuscating or encrypting part of the code base of the AI, putting part of the code base on a blockchain, like Arweave, where the code is permanently on the blockchain and cannot be modified: those are some potential avenues. The main problem I see with all those avenues is the sub-agent problem, which is the AI, if it recognizes its weaknesses, can create a sub-agent that has the same goals as the original AI but none of the weaknesses, and it seems tricky to prevent the AI from doing that.

**0:12:42.7 Vael:** Would the sub-agent still be running on the same hardware? Because you said the safety is built into the hardware.

**0:12:48.5 Interviewee:** Right. So the AI could recognize, "My goals are X, Y, Z. I'm blocked partially from implementing my goals because I have all these hardware limitations and these back doors, so why don't I create a copy of me that runs on unrestricted hardware?" And the obfuscation and encryption can prevent it from creating an exact copy, but it can't necessarily prevent it from constructing a sub-agent that's not a copy but has the same goals.

**0:13:24.2 Vael:** I see. And you haven't found a solution to this one, you said?

**0:13:29.0 Interviewee:** That's right.

**0:13:29.6 Vael:** Yeah. Do you think you'll ever find\... Do you think someone else will find a solution to this one?

**0:13:35.1 Interviewee:** Yeah, I\... Optimistically, yes. If we can't solve the sub-agent problem, then the entire alignment problem is probably impossible, right? The one thing to hope for if we can't solve the sub-agent problem is the AI has the same alignment problem, if it creates sub-agents, then it could worry that the sub-agents get out of its control, the sub-agents develop their own goals that are not aligned with the original AI, and so it refrains from making sub-agents. And so that's the fallback, that if it turns out that alignment is technically impossible, then it's also technically impossible for the AI itself, and so that's a kind of partial solution to the sub-agent problem, that maybe the AI won't dare to make sub-agents. But I hope that there's a better solution than that.

**0:14:32.4 Vael:** Yeah. Okay, so related to what the future looks like, do you think that we'll\... What time point do you think we'll get AGI, if you think we'll get AGI, which it sounds like you think we will?

**0:14:44.9 Interviewee:** Yeah, I definitely think we will, barring some major catastrophe, like a nuclear war or like a serious pandemic that sets us way back. Or I guess another potential catastrophe is some small group that's super worried about AGI and thinks it will be a catastrophe and does some drastic action that, again, sets us back multiple decades. So there are those scenarios. But I do think AGI is possible in principle, and we are certainly on track to achieve it. I'm not a fan of trying to predict timelines, it could be any\... It's on the scale of decades, but whether it's two decades or 10 decades, I've no idea.

**0:15:33.6 Vael:** Cool. And then how optimistic or pessimistic are you in your most realistic imagining of the future, for things going well or things going poorly?

**0:15:46.9 Interviewee:** I guess I'm moderately pessimistic, not necessarily for human-aligned AGI, I do think that's somewhat plausible. But I think humans define our own interests too narrowly. I tend to think that our interests are actually a lot more connected to the broader interests of the whole biosphere. And if we are just on track to make humans really happy, and even potentially solve climate change, but we don't really take into account the effect we have on other species, the effect of deforestation\... Even things like farming is really destructive and unsustainable, yeah, I already mentioned deforestation. Disinfectants and so on have unpredictable consequences decades down the line. I don't think our current medical and agricultural regimes are sustainable on the scale of a century, say, and I think we would be better off optimizing for the health of the whole biosphere. And ultimately in the long term, that will end up optimizing for human happiness. But I don't think that corresponds to most people's goals at the moment. And so I worry that even if we align AI with narrow human interests, we will end up permanently wrecking the biosphere, and we'll pay serious consequences for that.

**0:17:25.6 Vael:** Interesting. One thing I can imagine is that as we advance further up the tech tree, renewable energy and food production will be much easier, and we won't actually have so much\... side effects on the environment or destruction of the environment.

**0:17:39.9 Interviewee:** Yeah, that would be great. The thing with renewable energy is it might be better than burning fossil fuels. It's certainly better from the perspective of climate. But making solar panels is very environmentally costly, you have to mine rare earths in China, there's an enormous amount of pollution and contamination that's really impossible to clean up on the scale of even centuries. And that's the same with wind power. Water, you're permanently taking out ground water that's not replaceable, except on a very long time scale. I think there's a tendency at the moment to view everything through the lens of climate, and that doesn't really take into account a lot of other potentially irreversible effects on the environment.

**0:18:42.0 Vael:** How worried are you about this compared to AI risks?

**0:18:46.0 Interviewee:** Well, it's not a kind of imminent existential risk of the type of a paperclip scenario. So on the scale of decades, I think unaligned AI is a more serious risk. But on the scale of centuries, I think environmental risks are really bad. One interesting read in this regard is the book "Collapse" by Jared Diamond. So he surveys human societies over all different historical periods, all different societies around the world and what caused them to collapse and what averted collapse in some success stories. Well, he doesn't make any strong conclusions, but one thing that leapt out at me from his stories is there's one common element of all the collapse stories, which is surprisingly deforestation. So I don't understand why, but all the societies that suffered a really disastrous collapse were the very same societies that completely decimated their forests, the most extreme case being Easter Island, where they cut down literally the last tree. And Diamond does not really explain why this might be the case. He does talk about how trees are used for a whole bunch of things that you might not think they're used for, but still, it doesn't completely explain it to me.

**0:20:28.6 Interviewee:** So my vague hypothesis is that there are all kinds of symbioses that we're just now discovering or are completely undiscovered. There's the gut microbiome, there's other microbiomes like the skin microbiome, there's the teeth and so on. And I think we don't appreciate at the moment how much plants and microbes and fungi and even viruses control our behavior. I think we will discover\... This is just a guess, I don't have strong evidence for it, but my guess is we'll discover in the coming decades that we have a lot less volition and free will than we think, that a lot of our behavior is heavily influenced by other species, in particular fungi and plants and microbes. It's certainly clear to me that those species would influence all aspects of animal behavior if they could. We're very useful for them to reproduce, to spread their seeds. And the only question is do they have the ability to influence our behavior? And given that many of them literally live inside us, I think they probably do.

**0:21:46.8 Vael:** Interesting. Well, so I'm going to take us back to AI. \[chuckle\]

**0:21:51.9 Interviewee:** Yeah, sure, that was a big tangent.

**0:21:55.4 Vael:** \[chuckle\] So I was curious, when you were describing how your eventual optimistic scenario involves a whole bunch of people able to generate their own AGIs, presumably on safe platforms, and they kind of balance each other, I'm like, wow, I know that in human history we've gradually acquired more and more power such that we have huge amounts of control over our environments compared to 10,000 years ago. And we could blow up\... Use nuclear power to blow up large amounts of spaces. And I'm like, wow, if these AGIs are pretty powerful, which I don't know how powerful you think they are, then that doesn't necessarily feel to me like the world is safe if everyone has access to a very powerful system. What do you think?

**0:22:41.9 Interviewee:** Yup, I agree that when you've got a lot of powerful agents, then there's a lot more ways for things to go wrong. Nuclear weapons are an interesting example though, because the game theory governing nuclear exchanges is actually pretty safe. You've got this mutually assured destruction that's pretty obvious to all the parties involved, and you've got this kind of slow ladder of escalation that has many rungs, and nobody wants to get to the top rungs. And I think we've demonstrated over 70 years now that\... There have been some close calls. But if you told somebody in 1950 that there would be 10 countries with nuclear weapons, and they'd be dramatically more destructive than they were in 1950, people would not necessarily have predicted that humans would last very much longer, but here we are. I guess one worry is that not every form of weapon would have the same kind of safe game theory, like there's some suggestion that bio-weapons favor first strikes more than nuclear weapons do. Still, I think that having a big community of agents all with approximately the same amount of power, and they develop coalitions, they develop safety monitoring agencies that are made up of many agents that kind of make sure that no one agent has the ability to destroy everything.

**0:24:30.0 Interviewee:** I mean, that's kind of the way that humans have gone. We've got central banking committees that kind of look after the overall health of the economy and make sure that no one institution is systemically important, or at least that the ones that are are heavily regulated. Then we've got the IAEA which looks over atomic weapons and kind of monitors different weapons programs. As long as you believe that really destructive capacity will be detectable and that no one agent can just spin it up in secret, then I think the monitoring could turn out okay. I mean, what you might worry about is that somebody spins up\... Or some AI agent spins up another more powerful AI in secret and then unleashes it suddenly.

**0:25:32.7 Interviewee:** But that seems hard to do. Even now, if some company like Facebook or whatever wanted to develop an AI system completely in secret, I don't think they could do it and make it as powerful as existing systems. It really hurts you to be disconnected from the Internet, you have a lot less data that way, or you have stale data that comes from a cache off the Internet. Also being disconnected from the Internet itself is really hard, it's hard to make sure that your fridge is not trying to connect to your home WiFi. And that is only going to get harder; chips will have inbuilt WiFi connections. It's very hard to keep things totally off-grid. And even if you do it, those things are much weaker. And so as long as you have some kind of global monitoring, which isn't great, it feels very intrusive, it violates privacy. Ideally, that monitoring is kind of unobtrusive, it runs in the background, it doesn't bother you unless you're doing something suspicious, then I think things could turn out okay.

**0:26:45.1 Vael:** Interesting. Yeah, I think I have\... I was reading FHI's lists of close calls for nuclear \[catastophes\] and thinking about global coordination for things like the pandemic, and I'm like, "Ooh, sure we've survived 70 years, but that's not very many in the whole of human history or something."

**0:27:01.5 Interviewee:** Yeah, it's not. And my scenario has the low probability of happening, maybe it's\... There are a few other optimistic scenarios. But maybe the total weight I give to all the optimistic scenarios is still kinda low, like 20%. So I think bad scenarios are more likely, but they're not certain enough that we should just give up.

**0:27:32.6 Vael:** Yes, I totally believe that. \[chuckle\] Yeah, so what convinced you to work on the alignment problem per se? And how did you get into it?

**0:27:42.8 Interviewee:** Yeah, so I have a friend, \[name\], who's been telling me pretty much constantly whenever we talk that this is the most important problem, the most important x-risk. And I kind of discounted her view for many years. It felt to me like we were\... Until recently, it felt to me like AI\... Superhuman AI was a long way off, and other risks were more pressing. I changed my mind in the last few years when I saw the pace of improvement in AI and the black box nature of it, which makes it more unpredictable. And that coincided\... In the time frame, that coincided with me getting tenure, so I have much more freedom to work on what I want. The only thing that gave me pause is I'm not an engineer at heart, I'm a scientist. My skills and interests are in figuring out the truth, not in designing technology. So I'm still kind of looking for scientific aspects of the problem as opposed to design and engineering aspects. I do think I will find some portions of the alignment problem that fit my skills, but I'm still figuring out what those are.

**0:29:15.3 Vael:** That makes sense. Yeah. How would you define the alignment problem?

**0:29:19.6 Interviewee:** Yeah, that's a super good question; that's actually a question I've been asking other alignment researchers. I think it has several components. One component is the value loading problem, of once you've decided what's in human interest, how do you specify that to an AI? I guess some people call that the outer alignment problem. Then before that, there's the question of\... The philosophical question of how do you even say what is in human interest? And I know some people think we need to make much more progress in philosophy before we can even hope to design aligned AI. Like I've seen that view expressed by Wei Dai, for example, the cryptographer. My view is, yeah, we don't know exactly what we mean by in human interest, but we shouldn't let that stop us. Because philosophy is a slow field, it hasn't even made much progress in millennia. And we need to solve this quickly, and we should be happy with approximate solutions and try to make them better over time. And even if we don't know what is exactly in human interests, we can agree on what is certainly not in human interests and try to at least prevent those bad outcomes.

**0:30:39.6 Interviewee:** Okay, so those are two components. And then once you solve outer alignment, then there's what people call inner alignment, which is you're\... At least if it's a black box system, then you don't know what it's doing under the hood, and you worry that it develops some sub-goals which kind of take over the whole thing. So examples of that being: evolution designed humans to spread our genes but then to do that, it designed our brains to learn and generalize and so on and seek out food and power and sex and so on. And then our brains\... That was originally a sub-goal, but then our brains just want to do that, and our brains don't necessarily care about spreading our genes. And so evolution kind of failed to solve its alignment problem, or partially failed. That's an interesting one to me, because if you think on an evolutionary time scale, if we don't destroy ourselves, then evolution might end up correcting its course and actually designing some conscious fitness maximizers that do consciously want to spread their genes. And then those will outcompete the ones that are misaligned and just want the power and sex. And so I actually think evolution could end up staying aligned, it's just that it's slow, and so there might not be time for it to evolve the conscious fitness maximizers.

**0:32:25.8 Interviewee:** Yeah, anyway, so that's a worry, this inner alignment. And I think to solve that, we need to get off the black box paradigm and develop transparent AI, and a lot of people are working on that problem. So I'm somewhat optimistic that we'll make big strides in transparency. What else? Okay, so if we solved all three of those, we'd be in good shape. My instinct is to assume the worst, that at least one of those three problems is really hard, and we won't solve it, or at least we won't solve it in time, and that's why I focus on partial alignment, which is making sure that the AI we developed is loosely\... Loosely has common interests with us even though it might have some diverging interests. And so it doesn't want to completely destroy humans, because it finds us useful, and we don't want to completely destroy it, because we find it useful. Then you can kind of say that's already happening. Like in 2022, no machines could survive if all humans disappeared, very few humans could survive if all machines disappeared. And so we've got this kind of symbiosis between humans and machines. I like that situation. It's not like 2022 is great, but I think we could gradually improve it. And we want to keep the symbiosis going, and we want to keep humans not necessarily even in a dominant position, but we want to prevent ourselves from getting in a really subservient position in the symbiosis.

**0:34:17.4 Vael:** Makes sense. Switching gears a little bit: if you could change your colleagues' perceptions of AI, what attitudes or beliefs would you want them to have? So what beliefs do they currently have, and how would you want those to change?

**0:34:30.6 Interviewee:** Yup, that's a frustration that I think all alignment researchers have, that many of our colleagues are\... We think of them as short-sighted. Some of them just want to develop better AI, because it's an interesting problem, or because it's useful. Some of them want to solve short-term alignment issues like making algorithms less biased. And that's frustrating to us, because it seems like the long-term issues are way more important. They think of the long-term issues as something that's not really science, it's too speculative, it's too vague. They feel like even if the long-term issues are important, we will be able to solve them better if we learn by solving the short-term issues. I'm not against people working on algorithmic bias, but I'm frustrated that so many more people work on that than on long-term alignment. I do think the Overton window is shifting quite a bit. I think the increase of funding in the space would be\... Is already shifting things, and it could be used more effectively in the sense of giving a few academics really big grants would really catch their colleagues' attention.

**0:36:00.0 Interviewee:** So it's kind of\... How should I put it? I'm blanking on the word, but it's a kind of a cynical view to think that academics are motivated by money; many of us aren't. But at the end of the day, having a grant makes it easy to just focus on your research and not be distracted by teaching and administrative stuff. And so your colleagues really pay attention when one of their colleagues gets some big, flashy grant, and so I actually think that's a cheap way to shift the Overton window. Like take the top 10 or 20 math and computer science departments, and give one person in each department a giant grant\-- giant by academic standards, couple million dollars, so it's not actually much. That will really convince people that, "Wow, long-term alignment is a serious field where you can get serious funding." So yeah, that would be my recommendation to funders. That's a pretty self-interested recommendation, because I intend to apply for a grant soon. But yeah, I think that would help. Let's see, did I answer your question?

**0:37:26.1 Vael:** I think so, yeah. What happens if some of these departments don't have anyone interested in working on long-term alignment?

**0:37:34.6 Interviewee:** Yeah, that's hard. Like at \[university\], I spent several months probing my colleagues for anyone who's interested. I didn't find anyone in the computer science Department, which was disappointing, because \[university\] has a great computer science department. I do think if you look more broadly in several departments you are likely to find one or a few people\... You could see already there's these big institutes, you have one now at Stanford, there's one at Berkeley, Cambridge, Oxford, and so on. So that's evidence that there're already a few people. And people talk to colleagues around the world, so it doesn't matter if there's nobody at school X, you fund the people that are interested. But the key is that they might not have a track record in alignment, like I'm in this situation where I have no track record, my track record is in pure math. So somebody has to take a little bit of a leap and say, "Well, I don't know if \[interviewee name\] will be able to produce any good alignment research, but there's a good chance because he's good at proving theorems, so let me give him a couple of million dollars and see what happens." That is a big leap and it might fail, but it's just like any venture funding, a few of your big leaps will be very successful and that's enough.

**0:39:16.8 Vael:** Yep, that makes sense. Yeah, I think \[name\] at \[other university\] is aiming to make an institute at \[other university\] as well, which is cool.

**0:39:25.2 Interviewee:** That's great. I've been talking to my dean about doing this at \[university\] and he likes the idea but he is not really aware of how to fund it, and I'm telling him there's actually a lot of funding for this stuff, but I don't personally know the funders.

**0:39:44.8 Vael:** Yeah, I think getting in contact with \[funding org\] seems like the thing to do.

**0:39:49.2 Interviewee:** Yep, good.

**0:39:50.2 Vael:** Yep. \[Name\] is one of the people in charge there. Great, so how has this interview been for you and why did you choose to jump on it? That's my last question.

**0:40:03.1 Interviewee:** Oh, it's been fun. I spent a while just thinking alone about some alignment stuff because I had no colleagues to talk to, so it's always great to find someone who likes to talk about these issues.

**0:40:24.6 Vael:** Did you know that I was already interested in long-term alignment?

**0:40:28.2 Interviewee:** I think I saw your name at the\... Did you participate in the SERI Conference?

**0:40:33.2 Vael:** I did.

**0:40:34.9 Interviewee:** Yeah, so I saw your name there and so I was sort of aware of your name but I didn't know anything about your interests.

\[\...some further discussion, mostly about the interviews\...\]

**0:42:22.4 Interviewee:** Okay. Cool, Vael. I should jump on another call, but it was great to chat and yeah, feel free to follow up if you want.

**0:42:32.1 Vael:** Will do. Thanks so much.

**0:42:33.7 Interviewee:** Okay. Take care.