Word Document

Interview with zlzai, on 3/18/22

[Note: Th interviewee's Zoom connection was very bad since they were calling in from an Uber, so there's a lot of missing content (and "I lost you..." parts of the conversation that have been removed for clarity).]

0:00:02.5 Vael: Cool. All right. So my first question is: can you tell me about what area of AI you work on in a few sentences?

0:00:09.2 Interviewee: Definitely. The way that I describe it is, first of all, I work on deep learning, let's get that out of the way. And then beyond that, I'm interested in understanding [inaudible - long]. If I had to describe in a word, I would call it the science of deep learning. This is a topic that there are few other researchers interested in. We try to understand... We treat these as complex systems that have emergent properties due to that complexity, in the same way you might think of biology as an emergent property of physics. It's not something you might predict for first principles, but once it's there, there's a lot you can say about it that's pretty interesting. And then from there, I'm interested in trying to understand how do these things learn and how can you make them learn more efficiently. You can think of that... A metaphor I like to use is like pharmaceuticals. Once you understand a biological organism, you can design an intervention that takes advantage of the patterns that you've seen in order to get a certain behavior or a certain outcome.

0:01:00.8 Vael: Great. Cool. I missed maybe the second sentence there. I assume that was like large NLP systems or foundations or something?

0:01:08.1 Interviewee: Yes. So I work on deep learning, so I'm interested in the neural networks as they are in practice.

0:01:12.7 Vael: Got it. Cool. All right. And then my next question is, what are you most excited about in AI? And what are you most worried about? In other words, what are the biggest benefits or risks of AI?

0:01:23.4 Interviewee: So the thing that I'm most excited about is that this can do extraordinary things that we don't know how to program, but we know we can do. Let me try to elaborate on that a little bit. What I mean by that is there's a world of things that we don't know whether anything can accomplish, tasks that are sufficiently complicated or tasks where given the input data, you may not know whether you can predict the output data. But neural networks are closing the gap between things that we know are possible to do because we do them as humans all the time, but we don't know how to program, or there's no kind of discreet handwritten program that we can write easily that will describe this. And so I'm really excited about the fact that we can take data and actually do something that starts to resemble things that require human complexity in order to do that. So to me, that's the exciting part. Self-diving cars are one example, though we're not that good at that, but even just a mere handwritten digit recognition like the MNIST task, which is the most basic machine learning task in the world, now that I'm in the field, but when I first took a machine learning class and saw that task, my mind was blown that you could just, with this tiny little data set, learn how to do handwritten digit recognition. That's pretty profound, as far as I'm concerned, and I've never quite lost my fascination for that.

0:02:35.9 Interviewee: On the risk side, I'm concerned about a few things. I'm not an AGI believer or an AGI concerned person, so you can cross that one off. I think [inaudible] huge risks, I am always concerned about... [inaudible - very long]. [Vael saying what was missed] ...Let me start over. It was me trying to search for the right words so you missed absolutely nothing.

0:03:47.2 Interviewee: Now that I found the right words though, the two things that scare me most, number one, the fact that the systems fail in ways that are unintuitive to humans or that we wouldn't be able to reason about intuitively. So the example I give is, you think an automobile driver is likely to drive more poorly at night or when they're tired or when they're impaired or what have you. It's really hard to have an intuition about when the Tesla is going to think of the bus as a cloud, and then slam into the back of it. And it's those rare failures when we use a system a lot of times that lead to really bad outcomes and lead to mistrust. At the end of the day, AI needs to be trustworthy and we need to, as humans, trust it, otherwise, it will never be useful, and we'll never accept it or terrible things will happen, and so I think a lot about the notion of what does it mean for it to be trusted; where are the gaps between where we are right now and what it would take to be trustworthy.

0:04:41.3 Interviewee: The other big concern I have is that I think we don't take the problem of understanding how these systems work seriously. It used to be that when we design a big complicated piece of software, think like Windows or PowerPoint or a browser or something like that, we assumed it would take 1000 engineers to build it, and then maybe let's say 50 or 100 to test it. Deep learning, we have this really incredible situation where [inaudible - long] ... the way that I would put it, the system kind of [?verb] on its own based on the data. And we're lazy in computer science, we like to think that, "Oh, the system came to life on its own, so we should have this one hammer that we can bang against it and see whether it's good or not, or whether it's fair or not, or use the explainability bot to understand how to explain the system.

0:05:32.1 Interviewee: But in reality, I think the ratios are just going to be reversed. The software, if you're using... 100 researchers may be able to build a very exciting piece of software. You're going to need 1000 or 2000 engineers to tear it apart down to the studs and try to understand how each individual piece of one specific system works. So I think the entire literature around explainability and understanding of deep learning is completely wrong-headed because people are looking for general solutions to a problem that doesn't have general solutions. It's like writing a program that will debug any program for you. There's just no such thing. It's not how this game works. So that's scares me a lot, that we're thinking about this completely the wrong way. And we do need some degree of explainability, we need some degree of understanding of how these systems work, but we're not going to get there from the way that we're currently thinking about this, and it's just going to take a lot more effort and resources than we're currently remotely considering giving to it.

0:06:24.5 Vael: Interesting. And it sounded like... So how many engineers per researcher do we need, or per model do we need, do you think?

0:06:33.8 Interviewee: Well, this is going to be complicated. The 1000 and 100 are just kind of... Every piece of software needs a different number of engineers, and three engineers can actually build a pretty big piece of software, but the amount of... I'll go out there on a limb and say it's going to be a 10 to 1 ratio.

0:06:53.8 Interviewee: That's for every 10... Let's call it 10 testers. Whatever we call them, whether it's engineers or researchers or what have you, let's say that 10 people need to try to understand the system or 100 people need to understand the system. For every [inaudible] building the system. ...Feel free to tell me I'm wrong if [inaudible].

0:07:16.5 Vael: No, I'm just trying to actually hear you, so it's 10 to... Or 10 testers or 100 testers to one capabilities person making it happen. Wow. ...Are you still here? (Interviewee: "Yep, still here.")

0:07:34.0 Vael: Great, alright, so that's pretty extreme ratio. Do you think we'll get there? And what happens if we don't get there?

0:07:42.3 Interviewee: I think we'll have to get there. At the end of the day, we evaluate these systems by their capabilities. If you have an autopilot system and it crashes the plane a lot, at the end of the day, you're not going to use it and it's not going to be allowed to fly. In order to get the kinds of capabilities we're hoping for, and to get the degree of understanding of failures that we typically expect in any high assurance system, we're going to need to have... If your self-driving car crashes enough times, someone will require you to tear that thing down to the studs to understand that, and you're going to have to do this regardless of whether you decide to do this proactively, or eventually your system won't be allowed on the road until you do. So I think there's going to be a huge amount of manual labor in understanding how these systems work, and I think that either through failures that lead to bans of certain uses of technology until it's better understood or until it's more resilient, or someone who's being proactive and actually wants to ensure their self-driving car works. Either way, you're going to end on this situation. No better way to put it.

0:08:48.5 Vael: Okay, that's super fascinating. Okay, so you think that we are going to need... The way we're doing interpretability is not good right now, it needs to be specific to each system, and currently, we're probably doing much... Yeah, we're not even going in the right direction. I kind of expect that self-driving cars will be deployed without this huge amount of intervention on it, but you say that for most systems or something, we will just continue to have failures until it becomes obvious from society that we need people to do interpretability and explainability type of work in this correct way.

0:09:22.0 Interviewee: I think so, I think that's a good summary. The only thing I'll add to this, and I spend a big chunk of my life doing AI policy, so I have to think about this stuff a lot. Is that the amount of work we put into the assurance is proportional to the risk that the system has if it fails. So my favorite example of this is-- I've done a lot of work in the past on facial recognition policy. Obviously a hot topic, facial bias issues, etc. So let me give you two examples. Google Photos has this clustering algorithm where it will basically find all the faces of grandma and cluster them so that you click on one picture of grandma, and you can find them all if you want to.

0:09:58.4 Interviewee: Suppose there's racial bias in that algorithm, and suppose that instead the police department is using a facial recognition system to try to identify the perpetrator of a crime via a driver's license database. These are two applications that might both have the same technology underneath, they might both have the same biases. We're going to be much more worried about one than the other because one could lead people to lose their freedom, and the other may lead to people rightly feeling offended and people rightly feeling hurt, but not someone being thrown in jail potentially. So we have to handle the consequences of the systems in line with their risks. I'm a lot less concerned about interpretability of Google's facial recognition system than I am about a police's facial recognition system.

0:10:47.8 Vael: I see. And since they need to be treated differently anyway, according to your view... Yeah, just do it-- do the interpretability in accordance with their importance. That makes sense.

0:10:56.7 Interviewee: Exactly.

0:10:57.7 Vael: Yeah. Cool. Alright, so my next question is about future AI. So putting on a science fiction forecasting hat, say we're 50 plus years into the future. So at least 50 years in the future, what does that future look like?

0:11:11.4 Interviewee: I think it's honestly not going to look that different from the present, just a little bit more extreme. I don't think we'll be in an AGI world. I don't think we will have systems that rival human intelligence. I think we will be able to specify what we want out of systems in much higher level terms and actually get them to do this-- [I'm] thinking, like, a Roomba that is actually a useful intelligent Roomba. A Roomba that knows when to vacuum, knows when not to disturb you, that you can say, "Hey, can you please do an extra touch-up on that area," and it'll go do it. I think the current wave of machine learning is great at pattern recognition; I don't think pattern recognition gets us to AGI. I think the technology is going to have to look fundamentally different, and I don't know that we're any closer now than we were 50 years ago in that respect, beyond that we know there are a lot of dead ends.

0:12:05.0 Interviewee: That's not to say what we're doing right now isn't exceedingly useful. And [that] we're not going to push this to the nth degree, and [that] we won't have the New York Times of the future where you click on a headline and it will generate an article for you based on your interests, your knowledge level, your background on the topic and the amount of time you have to read it; I think that's an application that might be 10 or 15 years in the future, if not sooner than that. But I don't think we'll be in a place where we're worried about giving robots rights or things like that. I don't think that a huge amount of pattern recognition will get us to cognition, and I don't think the current track we're on will get us there. That doesn't bother me personally because I don't really care about that. I'm much more interested in what we can do for people now and in the near future, and not how we create intelligence. That's more of a scary question for me than an exciting question for me, but it's... [inaudible - short]

0:13:00.2 Interviewee: ...thinks that, heading to an AGI world, even if... And we may be in a world where we do have self-driving cars that actually work, even if it may not be here for 10 or 15 years, 'cause it's a hard problem. We may have actual smart digital assistants, but that's [inaudible - short] intelligent beings who are peers or [hard to parse] in that respect.

0:13:25.1 Vael: Interesting. I don't know if it will help for me to turn my video off, but I'm going to try it to hope it gets a little bit less choppy. Great. Okay. So you're talking about within the frame of AGI, like lots of people presumably talk about AGI around you. What are people's opinions? What do you think of their opinions? Etcetera.

0:13:46.5 Interviewee: Oh, I think there are a lot of nut cases. I mean, there are also a lot of optimists and a lot of people who have read a lot of science fiction and want to bring that to life, which is great. And a lot of people who have spent too much time in San Francisco and are surrounded by peer groups who... Basically, there's a lot of monoculture in San Francisco. And I say this having just come back to San Francisco, and as someone who refuses to move [there] despite [...]. I have friends who, if I recall correctly, go and worship at the Church of the AGI or something like that, from what I understand, under the belief that eventually there will be artificial intelligences that are smarter and more powerful than us, and therefore they should get ahead of the game and start worshipping them now, since they'll be our overlords later.

0:14:32.4 Interviewee: I'm a little more concerned with the here and now, and I think that the technological leaps between great pattern recognizes that we have today and that are way far away. Just because you're a nutcase doesn't mean you can't change the world. And there are plenty of examples of that, and it's really good to have that point of view constantly echoed in the community. But I don't worry about the end of the world because of AGI in the way that I think some of my friends do. I don't consider that to be the biggest existential risk that we have, far from it. And I don't think it's a risk that we should really be spending any time worrying about at the moment.

0:15:07.6 Vael: Got it. Is that because, like you said, there's other existential risks to prioritize, or you don't think this could be an existential risk?

0:15:15.1 Interviewee: I don't think this is... In the very, very long tail of ultra low probability risks to civilization, this is so far out in the tail that it's not worth spending any time on, independent of the fact that there are also much greater risks.

0:15:29.6 Interviewee: It's not just a matter of priority. It's also a matter of... it's not a good use of any resources.

0:15:34.8 Vael: Got it. That makes sense, yeah. Unfortunately, a lot of my questions are about AGI so you... [chuckle] Here we go.

0:15:42.1 Interviewee: No, I'm happy to give you a strong contrasting opinion to many that I'm sure you've heard. So come at it.

0:15:48.1 Vael: Lovely, lovely. Okay, cool. So how I'm defining AGI here is like any sort of very general system that could, for example, replace all human jobs, current-day human jobs, whether or not we choose to or don't choose to do that. And the frame I usually take is like, 2012, deep learning revolution, here we are. We've only been doing AI for like 70 years or something, and here we are 10 years later, and we have got systems like GPT-3, which have some weirdly general capabilities. But regardless of how you get there, because I can imagine that we hit some ceiling on the current deep learning revolution and we need to have paradigm shifts-- my impression is generally that if we keep on pouring in the amount of human talent and have software-- algorithmic improvements at the rate we've seen, hardware improvements, etcetera, and just like, the driving human desire to continue to follow economic incentives to earn money and replace things and make life more convenient, which I think is what a lot of what ML is aimed at right now, that eventually we will get some sort of AGI system. I don't really know when. Do you think we will at some point get some very general system?

0:16:55.8 Interviewee: I think if you take the time to infinity and humanity lasts in time going to infinity, yes, we will. I do think we have one example of a truly general intelligence system, namely human beings. And eventually we will probably get to the point where we could replicate that intelligence manually, if we have to literally photocopy somebody's brain and the transistors at some point, when we get technology advanced enough for that. Do I think that will happen this century? No. Do I think it will happen next century? Probably not. Do I think it might happen in the ones after that? Maybe.

0:17:28.8 Interviewee: So the answer is yes, in the limit, but no in any kind of limit that you or I would think about or be able to conceptualize.

0:17:35.7 Vael: Yeah, that makes sense. So you think there's probably going to be a bunch more paradigm shifts needed before we get there?

0:17:42.8 Interviewee: I think at least one-- it's hard to know how many paradigm shifts because it's hard to know what they are, but I do not think this paradigm is the right one. I think this paradigm is amazing, and I'm really excited about the kinds of machines we can build. But you can be excited about the kinds of machines we can build and recognize the limits of those machines. In the same way that-- how are we going to get to AGI in 10 years if we've been pouring... You were talking about economic incentives. How much investment do you think has gone into self driving cars over the past, let's say, 10 years?

0:18:14.2 Interviewee: Let's call it $100 billion plus or minus. That's probably the largest single investment in AI technology for an application anywhere. Ever, very likely. And where are we today on self driving cars? They're 90% of the way there, [inaudible - short] 10 times as difficult as the first 90%. And I don't think you'll be seeing fully autonomous self-driving cars in the general case on general roads in the next 10 or 15 years. So the thought that there would be an AGI at that point, let alone in 50 years is completely nonsensical to me, personally.

0:18:56.0 Vael: Yeah, that makes sense. Especially since self-driving cars are, like, robotics and robotics is behind as well. But even GPT and stuff doesn't really have good grounding with anything that's happening in the world and how--

0:19:09.2 Interviewee: GPT's capabilities are also wildly overstated. You can pull out a lot of good examples out of GPT if you really want to, and you can pull out a lot of crappy ones. But we're not going to just brute force large language models to get our way to general intelligence. That's BS you'll only hear from someone who works at OpenAI who wants their equity to be worth more, quite frankly. The only people who say this are the ones who have an economic incentive to say this, and the people who follow the hype. Otherwise, I don't really know of anyone who thinks GPT is the road to AGI, especially given that we can't scale up any bigger. I mean, this is something, this is my whole push right now, is that the only way that Nvidia is going to come out with new GPUs next week, and they are going to come out with new GPUs next week that will be twice as fast as the ones that came out two years ago, is if they doubled the amount of power. It's not like we're doubling the amount of hardware we have available.

0:20:02.4 Vael: Got it. Do you know how good optical or quantum computing will be? I know that those are in the pipeline.

0:20:08.9 Interviewee: They're in the pipeline. Quantum's been in the pipeline for a long time, and we're up to what, four qubits? Cool. Again, this is one of those cases where we've been awful close to nuclear fusion for a long time. We're going to have nuclear fusion in the limit. I'm a 100% certain of that. When we have that, call me.

0:20:32.7 Interviewee: It could be next year. I mean, we're very close to crossing that threshold, but we've been really close to crossing that threshold for several decades. And so, trying to call the year that it's going to happen within a five, 10 or 50-year time horizon, it's really, really tough when it comes to these technologies. And I'm in that same place about quantum. We are making progress on quantum and I'm really excited about it. I'm a little bit scared of it, but I'm excited about it, but that doesn't mean... Like, crossing that threshold is really difficult. The difference between 10 years and 50 years and a 100 years away may be very small improvements in technology, but it may take an exceedingly long time for us to accomplish. Optical, same thing. So I'm not sitting here expecting that breakthroughs are just going to happen left and right. These breakthroughs often take a very long time; a lot of incremental advances and all sorts of other technical advances in other fields to make them happen. Material science especially in the case of quantum. And we may get there in five years, or we may get there in 50 or 100 years or longer, and it's hard to say.

0:21:36.6 Vael: That makes sense. So what would convince you that you should start working on AI alignment? It sounds like there's probably going to be some breakthrough that would make you think that it's important, but we're not necessarily anywhere near that breakthrough right now. Do you have an idea of what that might be?

0:21:56.9 Interviewee: Give me your personal definition of AI alignment.

0:22:00.0 Vael: Yeah. Well, actually I want your definition first. [chuckle]

0:22:03.3 Interviewee: So this is not a field that I follow that closely. The entire concept of alignment has really come to the fore in the past three or four months while I've been trying to [job-related task]. So I haven't been paying as much attention, so I'd actually appreciate your definition. I can tell you the kinds of people who I see talking about alignment and the kinds of papers that I've seen, the paper titles that I've seen go across my desk. But I couldn't give you a good definition even if [inaudible - short].

0:22:33.9 Vael: Yeah. So one of the definitions I use, and I'll give you a problem setting I usually think about as well-- so, [the] definition is building models that represent and safely optimize [inaudible] specify human values. Alternatively, ensuring that AI behavior aligns with system designer intentions. And one of the examples I use for what an alignment problem would be is the idea that highly intelligent systems would fail to optimize exactly what their designers intended them to and instead do what we tell them to do. So the example of OpenAI, trying to... Have that boat win a race and then it getting caught on like some little-- but optimizing instead for a number of points and instead ending up in this little-- side, collecting-points area, instead of winning the race. So doing what the designers told it to do instead of what it intended them to.

0:23:26.2 Interviewee: So, I mean, this sounds like a... If you want my honest frank opinion.

0:23:34.8 Vael: Yeah.

0:23:35.3 Interviewee: A BS-ey rebranding of a simple fact of life in computing for... Since the dawn of computing, the computer does what you tell it to do, not what you want it to do. (Vael: "Yes.")

0:23:47.5 Interviewee: And so, I don't... I know a lot of people at OpenAI think they're very deep and profound for calling it AI alignment. [inaudible - very long].

0:25:02.7 Interviewee: So picking up where I was, there are probably a lot of folks at OpenAI, and I know exactly who they are, who think they're very deep and profound who are wondering about this question, but this is kind of the obvious fundamental thing that every first year programmer learns and is a question that everybody has been asking in every context whenever they develop a loss function for a neural network. Your loss function never reflects exactly the value that you want the system to carry exactly, even just the outcome. We optimize language models to be good at predicting the next word and yet they somehow also generate... We want them to have properties that go above and beyond that, we want them to be able to even just transfer the representations for downstream tasks.

0:25:44.0 Interviewee: This is just a question of to what extent is the thing you're optimizing a system for going to actually align with the task that you want. I just, I don't see what's interesting or new about this question from a research perspective. I think of course it's an important one, but it's not a... [inaudible - long] Saying, oh, we should worry about computer security. Well, yes, but there's no security robot that fixes all computer security. It's a complex context-specific problem. Again, people say things that are aligned with their economic incentives and that is certainly true for my friends at OpenAI but I don't see any profundity in this observation that's just the nature of computing.

0:26:38.1 Vael: Yeah. I think it has to be paired with the idea that if you have a very intelligent system that can plan ahead, that can model itself in the world, that it may have an incentive to preserve itself just as an agent pursuing any goal, because it doesn't want to decrease its chance of it succeeding at its goal. I think that has to be paired. Otherwise, it's not particularly special. But I do think creating an agent that in the far future, whenever AGI develops, that has an incentive to not be modified makes it much more dangerous than anything we've seen previously.

0:27:16.2 Interviewee: I agree with that factor, when we get to a point where this becomes a problem. But I think you can see that the amount I care about this question is proportional to the amount that I think that AGI is or will be a concern in your lifetime, or your grandchildren's lifetime. And so I think there are a lot more fundamental basic questions. Like, we don't even understand why a neural network actually is able to recognize handwritten digits in the first place. And until we get these basic things down, we're never going to get to build the kind of systems that have these properties anyway.

0:27:44.7 Interviewee: So, I would make the analogy of, let me see, I'm trying to think of the right metaphor... I don't know, for the folks who worry about how we're going to communicate with the alien civilizations we inevitably come into contact with-- we better build some space ships that can get us to space first before we start worrying. And then figure out whether there are alien civilizations out there before we start worrying about how we're going to communicate with them.

0:28:05.6 Vael: Yeah, this sounds like a very coherent worldview, I'm like, yep, makes sense, is logical.

0:28:11.7 Interviewee: Yeah, I have strong opinions, and researchers are always incentivized to have strong opinions; it's what moves science forward. We argue and we disagree with each other, and we're all right, and we're all wrong. But these are my strong opinions. And if you were to chat with any of my friends at OpenAI, you'd hear the opposite view and we could still go out for drinks and have a good time.

0:28:32.5 Vael: Great. So I think this line of questioning was originally developed by me asking what would you see... What would you want to see in the world before you're like, "Oh actually, people are right, this thing is coming earlier than I expected."

0:28:47.6 Interviewee: That's a good question. Honestly, I would want to see a... I'm thinking of examples of a task, so like a Turing test of some sort that would fit here. Obviously, the Turing Test is not a very effective test given that GPT-3 probably passes it with flying colors, despite the fact that GPT-3 is a very good language model. What I would look for is long-lived machine learning systems that can learn continually. We don't even have learning continually down. We have reinforcement learning systems that eventually become effective agents at solving one particular task; they can't move on to a second task, we don't have any general kind of learning process. Even the reinforcement learning agents that we have have to be really, really, specially hand-calibrated for a specific task, to the point where even the specific random seed you use is an important hyperparameter in determining whether the agent will succeed or fail at a given task.

0:29:50.0 Interviewee: If we're in that world, we're very far away from an agent that can learn many tasks or has a generalized learning process. This is I guess why people are excited about meta-learning in general. Again, I think we shouldn't worry about light speed travelling until we can get off the planet, but that's a whole different... You know where I stand on that. So... [inaudible - short] and improve themselves over time; reinforcement learning is kind of a shadow of what we would expect of a real system. A smart home assistant, where you can teach it new tasks by explaining it new tasks, and it will be able to figure out how to accomplish them. We don't have any kind of abstract or general learning capabilities right now. And we talk a very big game about things like AlphaFold or what have you, or any of the Alpha stuff coming out of DeepMind. But these are requiring lots and lots and lots of engineers to get a system to work on one very specific setting in ways that don't transfer to any other setting or any other task. There are principles we... [inaudible short] being really hard trying a bunch of stuff. And that's a... [inaudible long] real general learning process that will keep me up at night.

0:31:03.5 Vael: Got it. Okay. Interesting. So do you think you and your colleagues probably have the same information but are just reacting to it differently? Other people are also not seeing these very general systems or maybe-- yeah, presumably, because you have access to the same information, but you're just interpreting it differently based on the incentives that your company or whatever is in?

0:31:29.9 Interviewee: I wouldn't always blame the incentives and the all, again, throwing some shade at San Francisco in particular. When you spend all of your waking hours and all of your sleeping hours around exclusively people who are working on the same [inaudible], same field. And when you work as they do not only in the tech industry, but especially in the very tight knit AI community, where if we all go to the same birthday parties when I'm out there, and everybody goes to the same spas, and runs into each other on the street, despite the fact that San Francisco is a big city. When you live in that kind of monoculture, quite frankly, you'll lose touch with reality, to some extent. And I think, a lot of people start believing that... [inaudible very long]

0:32:31.9 Vael: Okay, cool. My question was, although I missed that last bit, but my question was, how do people stay in touch with reality, how do they...

0:32:43.7 Interviewee: I think you have to talk to people who are doing things other than building AI systems. You have to actually maybe interact with someone who works in healthcare or someone who works in finance, or someone who works in any other field besides AI or besides machine learning. And you have to remember that they are real human beings out there. I think this is just a general San Francisco problem. You see people, you see 20-somethings who are wealthy, producing systems that are convenient for them. We see this all the time, startups coming out of the Bay Area.

0:33:24.1 Interviewee: My experience in Boston and New York is it's a very different start up culture of healthtech, fintech, edtech, what have you, things that are quite frankly real and useful. So when I hang out with... The times I've been working at [large tech company] in California, for example, all of my friends are worried that the sky is falling for some reason. Maybe we're about to enter an AI winter or we're about to create AGI, it's usually one or the other. Depending on who you talk to and what mood they're in, and what's going on in their organization at that different point in time. But people are just so caught up in the inside baseball, they've lost the bigger context for everything we've accomplished, but also how far away we are from some of the things people talk about. And you gotta actually taste the real world from time to time, and remember that you don't just live in this little bubble of lots of neural networks and lots of people with PhDs working on neural networks.

0:34:19.7 Vael: And interacting with other people and seeing the wider world gives the context to believe more... mainline views? Because it sounds like people end up in either direction, you said they believe that everything is about AI, everything's going to go badly: everything is going to go badly and too fast or everything's going too badly and too slow. And there's some regularity effect that happens if you're hanging out with other people?

0:34:50.4 Interviewee: It's almost the opposite of... [inaudible - short] in that respect. I think it's partially because people will say things like, "Well, there hasn't been a big advance or there hasn't been a big breakthrough in 18 months." And my reaction is, it's been a... Yes, the transformer paper came out 18 months ago. Yeah, let's be patient, and the field does not always continually accelerate. No scientific field continually accelerates. We have periods of big progress and then periods of stagnation and paradigm shift. This is the structure of scientific revolution. If you want to... For the extent to which you take that view of the world seriously, that does, in my impression, simply how science moves, in fits and starts, not at a steady pace. And in computers we're used to exponentials. That doesn't mean that these systems are going to get exponentially better over time. It may mean we had 10 really good years of progress based on a bunch of different factors all coming together, between big data being accessible, lots of computing accessible, and some improvements in deep learning, that all came together really nicely to give us a big burst of progress.

0:36:00.1 Interviewee: Maybe things will slow down a little bit. Or maybe there will be a new architecture in five years that will be another big burst, the way that convolutional neural networks were around the AlexNet breakthrough and then the way that transformers were. But this happens in fits and starts. But around [large tech company], partially because of [large tech company]'s promotion process and partially because of the very internally competitive atmosphere that that fosters, people are always dour and feel like the sky is falling. And at OpenAI, the incentive is to continue the hype and build really big systems. It's partially in the culture, partially in the leadership, partially in the brand of the place and what it takes to make your stock go up in value. So I think people really are shaped by their environments in that respect, and you gotta get out and talk to other kinds of people and get other perspectives, and I don't think there's that much you can do when you're in a big organization like [large tech company], and there's not a lot you can do when you're in San Francisco and everybody you talk to. It's not "what you do". It's "which tech company do you work for" or more likely "which AI project do you work on".

0:37:04.4 Vael: I see, so this is probably, like you said, why you haven't moved and also presumably why you have friends who are not AI researchers.

0:37:12.4 Interviewee: Yeah, my office right now is at [university] [non-science field]. I just sit by a bunch of [non-science field] professors. I've worked there at times in the past. I live in [city] right now, and they were kind enough to offer me a place to sit and work for a little bit. So I'm hanging out with a bunch of [non-science field] faculty all day. It's a very different perspective on the world. They're worried about very mundane things compared to AGI coming and killing us all, or a quantum computing breakthrough that leads to the end of civilization when we train quantum neural networks or something like that. They're much more worried about bail reform, or very basic mundane day-to-day problems that are actually affecting people around them. It's hard to get caught up in the hype.

0:37:56.5 Vael: Yeah, and it seems like it's important to make sure that people are working on things that actually affect more than just their local environment. I think you mentioned something like that earlier, right?

0:38:05.9 Interviewee: Exactly. And there's also a little bit of... as computer scientists, we love to calculate things, we love to measure things. Effective Altruism is one great example of this. We love to say, "What is the way that I can put my dollar the furthest along?" And I think sometimes people lose touch with the fact that there are humans out there. There are people right now, you ask questions like, well, either I can help the humans today or I can put all my resources toward saving the entirety of civilization. And civilization and the future is obviously-- there are many more humans there than there are here today, so I'll worry about, on the 0.1% chance that we develop AGI and it comes to threaten us all, I'll worry about that problem, because in expectation, that will save more lives than what have you. And I think--

0:38:51.5 Vael: --Yeah, and this is the wrong way to think about things?

0:38:56.5 Interviewee: I don't think this is necessarily the wrong way to think about things, but I think it's a little bit... Sometimes, it's more important to focus on what's right in front of you and what's tangible and what's here and now. Or the other way I'll put it is, MIT is pathologically bad as an institution at dealing with topics that can't be measured for exactly this reason. It's really hard to talk to people at MIT about values, unless a value can be measured or optimized.

0:39:25.0 Interviewee: And this is just, for a lot of folks who get empirical education and who are training systems all day, for whom this is a problem, they are just as much... The people are just as much subject to this problem as the systems they build. So, if we want to talk about alignment, it's an issue not just of the systems, but of the people building the systems.

0:39:43.5 Vael: Alright, well, we are at time. Thank you so much, this is a very different opinion than the ones I've received so far, and yeah, very well presented. Awesome.

0:39:52.6 Interviewee: Thank you, that's what I'm here for. You know where to find me if you need to chat more. I'm very exited to see what the outcome of the project is.

0:39:58.4 Vael: Great. Alright. Well, thank you so much.


Word Document

Interview with 7ujun, on 3/18/22

0:00:03.4 Vael: Alright. So jumping right in, my first question is, can you tell me about what area of AI you work on in a few sentences?

0:00:11.1 Interviewee: [Interviewee describes working on natural language processing research]

0:01:20.4 Vael: Indeed. Thanks. And then so my next question is, what are you most excited about in AI? And what are you most worried about? In other words, what are the biggest benefits or risks of AI?

0:01:36.4 Interviewee: The biggest risks are that there is a lot of people who don't really... Who... The biggest risk in AI is that it's a field with a lot of money and attention and social power right now, and there are a lot of people who have positions of power who... don't seriously consider what they do and the impacts of what they do. And AI models are already being used to violate people's human rights in the United States and in other countries, to commit crimes, and that's bad. [chuckle] Yeah, there's... One of the worst applications is that there has been a revival in phrenology recently, so there are a lot of police departments that they have gotten really into the idea that they can use AI analysis of video cameras to determine who is *going* to commit crimes. And this, shockingly, results in over-surveillance of minority populations and violation of human rights left and right, and it's a huge clusterfuck and the police don't care.

[pause]

0:03:10.2 Vael: Awesome. Well, not awesome, but. So that question was, what are you most excited about AI and what are you most worried about; biggest benefits and risks?

0:03:19.8 Interviewee: Gotcha. What am I most excited about? I'm excited about the opportunity to interact with computers via natural language. So one of the really interesting things about some recent research is that we've been able to move away from traditional coding interfaces for certain tasks due to the way we've able to automate things. Probably the most prominent example of this is that there is a burgeoning online AI-generated art community, where they take pre-trained models and they write English sentences and they provide the... What the model does is it takes a English sentence input and draws a picture, and it's shockingly good and has an understanding of styles. If you want it to be... If you say in the style of Van Gogh, or high contrast, or low-poly render. You can induce visual effects by using language like that, and I think that's phenomenally cool, and it's gotten a lot of people... There's a lot of people who've gotten into using this kind of technology who otherwise wouldn't have... [who it] really wouldn't have been accessible to because of their lack of coding knowledge and understanding of AI. They couldn't have developed the algorithms that run on the backend for this on their own. Recently, yesterday I saw another blog post about how they were able to develop simple video games using GTP-3. It just wrote the code for them. I think that the ability to write a text description of something that you're interested in, which is a medium that everyone can relate to and interact with, or that most people can relate to and interact with far more than regular programming, for example, is really powerful and really awesome.

0:05:30.8 Vael: Yeah, I see a lot of themes of accessibility in all of these risks and benefits and work. I thought you were going to bring up Codex but yes, art generation. It's very cool.

0:05:41.7 Interviewee: Oh yes.

0:05:43.8 Vael: Yeah. So, thinking about a future AI, putting on a science fiction forecasting hat, say we're 50 plus years into the future. So at least 50 years in the future, what does that future look like?

0:05:56.3 Interviewee: I have absolutely no idea, and anyone who says otherwise is wrong.

0:06:01.3 Vael: Okay. Do you think AI will be important in it, or probably not?

[pause]

0:06:16.6 Interviewee: I think that's more of a sociological question than it is a technical question. The class of problems and the class of algorithms that are considered AI has changed dramatically over the past 50 years, and there's entire books have been written about this topic. At a basic level, my hesitancy is that I don't know people will consider AI in 50 years. There's a very real possibility in my mind that GTP-3 will no longer be considered an AI.

0:06:48.5 Vael: What will be considered?

0:06:53.3 Interviewee: A text generation algorithm? A good example of this is simple game-playing agents, so you can write an algorithm that can play Tic-Tac-Toe perfectly or can play Connect Four or Checkers really well. Like, will beat any human. And a lot of people don't call those AIs anymore because they don't... 'Cause they're search algorithm-based. They apply a lot of computational power to look through a space of possible events, and they find the best event. And they don't really... The argument is that they don't reason, or they don't know anything about strategy. And this is often to contrast it with more recent AIs for playing games like the AlphaGo, AlphaZero models that DeepMind has produced where top level chess players certainly get beat by these algorithms just like they get beat by... have been beaten by algorithms for 20 years, but for kind of the first time people are able to study and interpret these algorithms and learn and improve their play as a human. Which is really cool. But that kind of dichotomy is often used to dismiss or remove the label of "AI" from prior work and stuff that have been considered AI at that time. And I could certainly see that happening with GTP-3, for example, because it's really, really terrible at reasoning. And if in 50 years we have chatbots that can answer knowledge-based questions the way, say, a sixth grader could, and reason about basic word problems and stuff, pass some kind of reasoning examination, then I could easily see people no longer considering GTP-3 an AI, because it's not intelligent, it's just babbling and making up words.

0:08:54.4 Vael: Right. Cool, I'm now going to go on a spiel. So people talk about the promise of AI, by which they mean a bunch of different things, but the thing that I'm most thinking about is a very general system with the capabilities to replace all current day human jobs, so like a CEO AI or a scientist AI, for example. Whether or not we choose to replace human jobs is a different question, but I usually think about this in the frame of... in 2012 we had AlexNet, deep learning revolution. You know, here we are 10 years later, we have GPT-3, like we're saying, which has some weirdly emergent capabilities: new language translation, and coding and some math and stuff, but not very well. And then we have a bunch of investment poured into this right now, so lots of young people, lots of money, lots of compute, lots of... and if we have algorithmic improvements at the same rate, and hardware improvements at the same rate, like optical or quantum, then maybe we reach very general systems or maybe we hit a ceiling and need to do a paradigm shift. But my general question is, regardless of how we get there, do you think we'll ever have these very general AI systems, like a CEO or a scientist AI? And if so, when?

[pause]

0:10:07:5...Oh, you're muted. I think. Oh, no. Oh, no. Can't hear you. I don't think anything's changed on my end. Okay.

0:10:25.2 Interviewee: Hello?

0:10:26.2 Vael: Yeah. Cool.

0:10:27.2 Interviewee: Okay. I think my headphones may have done something wacky. I don't know, I would be extremely surprised if the answer... like I know that there are people who say that the answer is less than 10 years, and I think that's absurd. I would be surprised if the answer is less than 50 years, and I don't feel particularly confident that that will ever happen.

0:10:47.1 Vael: Okay. So it may or may not happen. Regardless, it's going to be longer than 50 years. Is that right?

0:10:54.1 Interviewee: Hello?

0:10:55.2 Vael: Hello, hello, hello?

0:11:00.7 Interviewee: Yes.

0:11:00.8 Vael: Okay, cool. So my question was like, all right, you don't know whether or not it will happen. Regardless, it will take longer than 50 years. Is that a summary?

0:11:07.3 Interviewee: Mm-hmm.

0:11:09.0 Vael: Yeah. Okay, cool. So one of my question is like, why wouldn't it eventually happen? I kind of like believe in the power of human ingenuity, and people following economic incentives such that... These things are just really quite useful, or systems that can do human tasks are generally quite useful, and so I sort of think we'll get there eventually, unless we have a catastrophe in some way. What do you think about that?

[pause]

0:11:45.0 Interviewee: It's going to be extremely difficult to develop something that is sufficiently reliable and has an understanding of the world that is sufficiently grounded in the actual world without doing some kind of mimicking of human experiential learning. So I'm thinking here reinforcement learning in robots that actually move around the world.

0:12:13.0 Vael: Yeah.

0:12:13.9 Interviewee: I think without something like that, it's going to be extremely difficult to tether the knowledge and the symbolic manipulation power that the AIs have to the actual contents of the world.

0:12:29.5 Vael: Yep.

0:12:29.9 Interviewee: And there are a lot of extremely, extremely difficult challenges in making that happen. Right now, cutting-edge RL techniques are many orders of magnitude... Require many orders of magnitude too much data to really train in this fashion. RL is most successful when it's being used in like a chess context, where you're playing against yourself, and you can do this in parallel, and that you can... When you can do this over and over and over again. And if you think about an actual robot crossing the street, if an attempt takes 10 seconds, and I think especially early in the learning process, that's an unreasonably small amount of time to estimate. But if an attempt takes 10 seconds and... Let me pull out the calculator for a second.

0:13:28.1 Interviewee: And you need one million attempts... then that would take you... about a third of a year to do. And I think that both of those numbers are wrong. And I think the number of attempts is orders of magnitude too small. There's very, very little that we can learn via reinforcement learning in a mere one million attempts. And this is just one task. If you want something that can actually move around and interact with the world, even if you're using these highly optimistic, currently impractical estimates, you can't take four or five months to learn how to cross the street. If that's your paradigm, you're never going to be able to build-- you're never going to get to stuff like managing a company. [chuckle]

0:14:38.0 Vael: Yeah. That makes sense. Yeah, I think-- I think... this makes sense to me, and I'm like, "Wow, our current state systems are really not very good." But also, I think I often view this from a lens of pretty far back. So I'm like, 10000 years ago, humans were going around and the world was basically the same from one generation to the next, and you could expect things to be similar. And now we've had the agriculture revolution and industrial revolution, in the past couple of hundred years, we have done... We're kind of on an exponential curve in terms of GDP, for example, and I would expect that... And we've only been working on AI for, I don't know, less than 100 years. And we have... We now have something like GPT-3, which sounds sort of reasonable, if you're just looking at it, and of course it's not very... It's not, like, grounded, which is a problem.

0:15:28.3 Vael: But I sort of just expect that if we spend another... I don't know, you could spend hundreds of years working on this thing, and if it continues to be economically incentivized... This is kind of how human progress works. I just kind of expect us to advance up the tech tree to solve the software improvements, to solve the hardware improvements. Or new paradigms maybe. Even at the worst case, I guess we advance enough in neuroscience and scanning technologies to just scan human brains and make embodied agents that way or something. I just expect us to get to some capabilities like this eventually.

0:16:03.5 Interviewee: In my mind, really the fact that there's only so fast we can move around in the real world is a huge constraint. Even if you can learn extremely complicated and abstract things embedded in the real world as an actual robot, take my crossing street example, even if you could... doing an attempt at... So even if you could learn pretty much any task in a thousand iterations, some tasks take a very long time to develop. Humans don't learn to be CEOs of companies very quickly, and it doesn't seem like it's very shortcut-able to me. I also don't think CEOs of companies is perhaps the best example, but let's say...

0:16:58.1 Interviewee: Let's say you wanna train a robot to operate a McDonalds. That's a very large amount of destroyed meat that you need to buy, it's a very large amount of time and materials to even set up the apparatus in which you could actually train a robot to perform that task. And you're talking about economic incentives, where is the economic incentive to burning million patties to get to the point where your robot can flip one over successfully? When we're talking about moving around and interacting in the real world, those interactions have costs that are financial in addition to being time-consuming. If we want to train an AI to... Via reinforcement learning technique, which is certainly a caveat that have to add to a lot of what I'm saying. But if we wanna train a robot to drive a car via a reinforcement learning-like technique, at some point you need to put it behind the wheel of a car and let it drive 100, 1000 cars. And you're going to destroy a lot of cars doing that, and you're probably going to kill people. So that's a very large disincentivizing cost.

0:18:30.1 Vael: Okay. Alright. So the idea is like... if we're doing robotics, then we need to... and the training paradigm is not, like, humans where you can kind of sit them down, and... Humans don't actually crash cars, usually... I mean, sometimes. Teenage humans crash cars sometimes. But in their training process, they don't usually require that many trials to learn, and they can do so kind of quickly. So I'm like, I don't know. Do we expect algorithms at some point to require much less training data than current ones do? Because current ones require a huge amount of training data, but I kind of imagine we'll get more efficient systems as... More efficient per data as we go along.

0:19:24.9 Interviewee: Are you saying that you think that you can sit down and explain to someone how to drive a car and they can drive it without crashing?

0:19:30.9 Vael: I think, that.. We have... I think that if we take a human, and I'm like, "All right, human, I'm going to... I want you to learn how to drive this car. And I'm going to sit next to you. And I'm going to tell you what to do and what not to do, and you're going to drive it." I think they can, indeed, after practicing some period of time, which for humans, it's like hours. It's on the order of tens of hours, then they can basically sit there and not crash a car. And I kind of expect similar paradigms eventually for AI systems.

0:20:04.4 Interviewee: That seems extremely non scalable.

0:20:11.8 Vael: Uh... Okay. You're like, look, if it takes tens of hours to train every AI system?

0:20:17.6 Interviewee: No, I'm thinking mostly about the human sitting next to them giving them constant feedback actually.

0:20:28.5 Vael: But the nice thing about AI is you can copy them as soon as one person spends that many hours. You can just take that, takes the thing that's-- like its new neural net, pass it onto the next one.

[pause]

0:20:48.0 Interviewee: ...Maybe.

0:20:50.1 Vael: And I don't think this has to happen anytime soon. But I do think eventually given that... I don't know, I can't imagine humans being like, "All right, cool. We're efficient enough. Let's just stop now. We've got like GPT-3. Seems good. Or GPT-5, let's just stop here."

0:21:08.4 Interviewee: So nobody has ever taken two different robots, trained one of them in the real world to perform a task, and then transferred the algorithm over and allowed the other one to perform the same task as successfully, as far as I'm aware.

0:21:23.6 Vael: Yep. I totally believe today's systems are not very good.

0:21:31.2 Interviewee: It is, I think, I think that.. Anything we can really say about this is inherently extremely speculative. I'm certainly not saying it could never happen. I'm just... Sorry. I'm certainly not saying it can't happen. I'm just saying it could never happen. There we go.

0:21:45.0 Vael: Okay. All right. Okay. That makes sense. How likely do you think it is that we'll get very capable systems sometime ever in the future?

0:21:53.5 Interviewee: I have no idea.

0:21:56.5 Vael: ...Well, you have some idea because you know that it... Well, okay. You said that it can't... You're like, it's higher than zero.

0:22:04.9 Interviewee: Yes.

0:22:05.9 Vael: Yes. And you don't sound like you think it definitely will happen, so it's less than 100.

0:22:12.6 Interviewee: Yes.

0:22:13.8 Vael: Okay. And it's anywhere in that scale? I mean, slightly higher than zero and slightly less than 100.

0:22:22.5 Interviewee: That sounds like an accurate description of my current level of uncertainty.

0:22:26.8 Vael: Interesting. Man. Is it.. Hard-- I mean like how-- You do have predictions of the future though, for the near future, presumably, and then it just like tapers off?

0:22:36.7 Interviewee: Mm-hmm.

0:22:38.2 Vael: Okay. And anything... And you say definitely not 10 years, but after. And then like 50 years. So like 50 years out, you're... It starts going from approximately zero to approximately 100?

0:22:56.6 Interviewee: Um... I think it is unlikely to happen in the next 50 years.

0:23:00.0 Vael: Okay.

0:23:05.7 Interviewee: I would assign a less than 25% probability to that. But I don't think I can deduce anything about my expectation at 100 years based on that information. Other than it... yeah.

0:23:21.6 Vael: Great. Thanks. All right. I think that's good enough for me to move on to my next question.

0:23:28.6 Vael: So my next question is thinking about these highly intelligence systems in general, which we're positing maybe will happen sometime. And so say we have this sort of CEO AI through, I don't know, maybe hundreds of years in the future or whatever. I'm like, "Alright, CEO AI. I want you to maximize profits and try not to run out of money and try not to exploit people and try to avoid side effects." And currently, obviously this would be very technically challenging for many reasons. But one of the reasons is that we currently don't have a good way of taking human values and preferences and goals and stuff, and putting them in mathematical formulations that AI can optimize over. And I worry that this actually will continue to be a problem in the future as well. Maybe even after we solve the technical problem of trying to get an AI that is at all capable. So what do you of the argument, "highly intelligent systems will fail to optimize exactly what their designers intended them to, and this is dangerous?"

0:24:26.0 Interviewee: I mean I think that the statement that highly intelligent systems will fail to optimize what their designers intend them to is a slam dunk. Both human children and current AIs do not do that, so I don't see any particular reason to think we will-- that something that's like, in some sense in between those, we'll have a whole lot more success with.

0:24:50.4 Vael: Interesting. Okay, cool.

0:24:58.5 Interviewee: Yeah. Did you turn out exactly the way your parents wanted you to? [laughter] I didn't. I think the overwhelming majority of people don't, and that's not a flaw on their part. But... yeah.

0:25:15.2 Vael: All right. Yep. Yeah, certainly there's some alignment problems with parents and children. Within human-humans even. And then I expect-- I kind of expect the human-AI one to be even worse? My intuition is that if you're having an AI that's optimizing over reality in some sense, that it's going to end up in alien parts of the space-- alien to humans, because it's just optimizing over a really large space. Whereas humans trying to align humans have at least the same kind of evolutionary prior on each other. I don't know. Do you also share that?

0:25:47.7 Interviewee: I'm not sure. I think that you're going to have to get a lot of implicit alignment to end up in a place where you're able to train these things to be so intelligent and competent in the first place.

0:26:07.6 Vael: That makes sense to me. Kind of like--

0:26:09.8 Interviewee: Yeah. What percentage of the way that gets you there is a very important and totally unknown question. But I don't think that the value system of one of these systems is going to be particularly comparable to like, model-less RL, where they're trying to optimize over everything.

0:26:33.4 Vael: Could you break that one down for me?

0:26:38.2 Interviewee: In what way?

0:26:41.6 Vael: I didn't... I don't quite understand the statement. So the value system will not be the same that it is in model-less RL. I don't have a super good idea of what model-less RL is and how that compares to human systems or human-machine--

0:26:56.1 Interviewee: Okay. So model-less RL is a reinforcement learning paradigm in which you are basically trying to learn everything from the ground up, via pure interaction.

0:27:06.2 Vael: Okay.

0:27:07.1 Interviewee: So if you're thinking of a game-playing agent, this is typically an agent that you're not even programming with the rules. It learns what the rules are because it walks into a wall and finds that it can't walk further in that direction. That's the example in my head of something that's optimizing over all possible outcomes currently. ...Sorry, I lost the train of the question.

0:27:39.7 Vael: I was like: how does that relate to human value systems?

0:27:46.4 Interviewee: I think that the work that we will have to do to train something to move around and interact in the world and perform these highly subjective and highly complex tasks that require close grounding in the facts of the world will implicitly narrow down the search space. Significantly.

0:28:10.1 Vael: Okay. Yeah--

0:28:11.6 Interviewee: I do think that there's a... Yeah.

[pause]

0:28:25.8 Interviewee: Yeah.

0:28:26.3 Vael: Yeah. Yeah, I often think of this in terms of like, you know how the recommender systems are pretty close to what humans want, but they're also maybe addictive and kind of bad and optimizing for something a little bit different than human fulfillment or something. People weren't trying to maximize them for human fulfillment per se. But yeah, I like-- like that sort-of off alignment is often something I think about. Alright.

So this next question is back to the CEO AI, so imagine that the CEO AI is good at multi-step planning and it has a model of itself in the world, so it's modeling other people modeling it, 'cause that seems pretty important in order for it to do anything. And it's making its plans for the future, and it notices that some of its plans fail because the humans shut it down. And it's built into this AI that it needs human approval for stuff 'cause it seems like a basic safety mechanism, and the humans are asking for a one-page memo to describe its decision.

0:29:21.4 Vael: So it writes this one-page memo, and it leaves out some information because that would reduce the likelihood of the human shutting it down, which would increase the likelihood of it being able to achieve the goal, which is like, profit plus the other constraints that I mentioned. So in this case, we're not building in self-preservation to the AI itself, it's just, self-preservation is arising as a function... [as a] instrumental incentive of an agent trying to optimize any sort of goal. So what do you think of the argument, "highly intelligent systems will have an incentive to behave in ways to ensure that they are not shut off or limited in pursuing their goals, and this is dangerous?"

0:30:00.4 Interviewee: It seems likely correct.

0:30:02.3 Vael: Interesting. Okay. [chuckle] ...I'm not excited about that answer, 'cause other instrumental incentives are acquiring resources and power and influence, and then also not wanting... Having a system that's optimizing against humans seems like a very bad idea in general, which makes me worried about the future of AI. If the thing that we're going to build is eventually by default, maybe not going to want to be corrected by humans if we get the optimization function wrong the first time.

0:30:32.5 Interviewee: Yeah.

0:30:36.2 Vael: [laughter] Okay. Have you thought about this one before?

0:30:38.6 Interviewee: Yes.

0:30:39.4 Vael: Yeah. Cool. Have you heard of AI alignment?

0:30:42.2 Interviewee: Yes.

0:30:43.2 Vael: Yeah. And AI safety and all the rest of it?

0:30:45.6 Interviewee: Mm-hmm.

0:30:46.0 Vael: Yeah. How do you orient towards it?

0:30:49.8 Interviewee: I think that most people who work in it are silly. And don't take the right thing seriously.

0:30:57.9 Vael: Mm. What should they take seriously? And what don't they?

0:31:02.9 Interviewee: I know a lot of people who are afraid that future research along the lines of GTP-3 is going to rapidly and unexpectedly produce human-like intelligence in artificial systems. I would even say that that's a common, if not widespread attitude. There are pretty basic kinds of experiments that we'll need to do to test the plausibility of this hypothesis, that nobody seems really interested in doing.

0:31:48.5 Vael: Hm. Seems like someone should do this?

0:31:51.2 Interviewee: Yeah. When I talk to most people who describe themselves as alignment researchers, and I try to put myself in their shoes in terms of beliefs about how agents work and what the future is likely to look like. The things I see myself experimenting with and working on, are things that nobody is working on. And that really confuses me. I don't understand... So here's an interesting question: how much experience do you have actually using GTP-3 or a similar system?

0:32:31.9 Vael: Yeah, not hardly at all. None. So I've seen examples, but haven't interacted with it myself.

0:32:38.4 Interviewee: Okay, um... Would you like to?

0:32:47.2 Vael: Uh... Sure? I mean, I guess I've messed around with the Dungeon AI one, but... Does seem interesting.

0:32:57.1 Interviewee: Hm. ...So my experience is that... A widespread observation is that they don't seem to have a worldview or a perspective that they-- are expressing words, so much as many of them. Some people like to use the term multiversal. It's... kind of the way I think about it is that there are many people inside of GTP-3 and each time you talk to it, a different one potentially can talk to you.

0:33:42.1 Vael: Yep.

0:33:43.8 Interviewee: This seems to be an inherent property of the way that the model was trained and the way that all language models are currently being trained. So a pressingly important question is, to what extent does this interfere with... Let's, to make language easier, call it one of its personalities. Let's say one of its personalities wants to do something in the world: kill all the humans or even something mundane. To what extent does the fact that it's not the only personality interfere with its ability to create and execute plans?

0:34:28.2 Vael: ...Ah... Current systems seem to not... Well, okay. It depends on how we're training it, because GPT-3 is confusing. But AlphaGo seems to kinda just be one thing rather than a bunch of things in it. And so it doesn't seem like it has conflicts there?

0:34:46.1 Interviewee: I would generally agree with that.

0:34:48.2 Vael: Okay. But we're talking about scaling up natural language systems and they don't... And they don't... They have lots of different types of responses and don't... on one personality. Uh... Well, it seems like you could train it on one personality if you wanted to, right? If you had enough data for that, which we don't. But if we did. And then I wouldn't really worry about it having different agents in it.

0:35:17.6 Interviewee: That's a very, very, very, very, very, very, very, very large amount of text.

0:35:23.8 Vael: Yeah. [Interviewee laughter]

0:35:25.0 Vael: Yeah, yeah that's right!

0:35:26.5 Interviewee: Do you any-- do you have any scope of understanding for how much text that is?

0:35:32.8 Vael: Yeah, I'm actually thinking something like pre-training on the whole internet, and then post-train on a single person, which already doesn't work that well. And so then it wouldn't actually help if that pre-training procedure is still on... Still on the whole thing. Um, okay--

0:35:48.4 Interviewee: So a page of written text is about 2 kilobytes in English. And these models are typically trained for between one and five terabytes, so no human has come anywhere close to putting out five billion pages of total text.

0:36:13.7 Vael: Yeah.

0:36:18.1 Interviewee: It's so astronomically far beyond what any human would actually ever write, that it doesn't seem very plausible unless something fundamentally changes about the way humans live their lives.

0:36:30.8 Vael: Or about different training procedures. But like--

0:36:33.8 Interviewee: Yeah, yeah, yeah, yeah. But like the idea that one could do something similar to current pre-training procedures that is meaningfully restricted to even say a 100 people that have been pre-screened for being similar to each other. 100 people are also not going to put out five billion pages of text.

0:36:49.6 Vael: Yeah.

0:36:51.6 Interviewee: [laughter] It's just so much data...

0:36:54.1 Vael: Yeah. Yeah, I don't know how efficient systems will be in the future, so... Yeah. Let's take it as... Yeah, sure. But they're going to have multiple personalities in them, in that they are trained on the internet.

0:37:05.1 Interviewee: Mm-hmm.

0:37:06.1 Vael: And then you're like, "Okay. Does that mean that... " And then there's a frame here that is being taken where we have different... Something like arguing? Or like different agents inside the same agent or something? And so then you're like, "Well, has anyone considered that? Have we tested something like that?"

0:37:26.9 Interviewee: Yeah, that's kind of close to what I'm saying.

0:37:29.6 Vael: Hmm.

0:37:31.9 Interviewee: So, to take your CEO example. In order for it to be successful, it needs to... at no point... There's certain information it needs to consistently hide from humans. Which means that every time it goes to generate text, it needs to choose to not share that information.

0:37:47.1 Vael: Yeah.

0:37:48.1 Interviewee: So if the system looks even vaguely like GTP-3, it seems to me like it will not be able to always act with that... generate text with that plan. And so there's a significant risk in it compromising its own ability to keep the information hidden.

0:38:13.7 Vael: Okay.

0:38:13.7 Interviewee: Alternatively, even if it's... That's a more direct way that they can interfere with each other. But even less directly, if I have somewhere I want to go and I go drive the car for a day, and then you have somewhere you want to go and you drive the same car for a day, and we trade off control, there are things I'm going to want to do that I have trouble doing because I only control the body and the car at the end of the day.

0:38:40.8 Vael: Quick question. Are you expecting that AI systems or multi-agent properties are more... have more internal conflict than humans do? Which can also be described in some sense as having multiple agents inside of them?

0:38:54.7 Interviewee: Yes.

0:38:55.7 Vael: Okay.

0:38:57.4 Interviewee: I think that anyone whose worldview is as fractured and inconsistent as GPT-3 probably has a clinical diagnosis associated with that fact.

0:39:08.8 Vael: Yeah. And you don't think that these will get more targeted in the future as we direct language models to do specific types of tasks, something like math?

0:39:24.2 Interviewee: I think that achieving, even... achieving 95, 99%, let's say, coherency between generalization, so if you imagine every time the model is used to generate text, there's some worldview it's using to generate that text, and you want each time those different worldviews used to be consistent with each other. Even achieving 99% consistency, I'm not asking for 100% consistency but 95, 99 seems like something necessary for it to make multi-year long-term plans.

0:40:10.7 Vael: That seems right.

0:40:13.5 Interviewee: This is exceptionally difficult and there are very likely fundamental limitations to the extent to which a system can achieve that level of coherence in the current training paradigms. And...

0:40:31.7 Vael: Seems plausible.

0:40:34.7 Interviewee: That would be very good news to people who are afraid that GTP-7 is going to take over the world.

0:40:43.4 Vael: Yeah, yeah. Okay, alright, 'cause I'm like, I don't know, I feel I'm kind of worried about any future paradigm shift. But current people definitely are worried about GPT-3 specific or GPT systems, and the current paradigms, specifically.

0:40:56.3 Interviewee: I've spoken to these people at length and I've talked to them about what they're afraid of and stuff. ...There seem to be a significant number of people in the alignment community who... If you could put together a convincing argument that the current pre-training methodology, as in, the fact that it's trained on a widely crowdsourced text generation source, instills some kind of fundamental worldview inconsistency that is exceptionally difficult if even possible to resolve, would alleviate a lot of the anxiety. It would actively make these people happier and less afraid about the world.

0:41:38.7 Vael: That seems true. I think if you can... If there's a fundamental limit on capabilities, just like, of AI, then that's good for safety because then you don't get super capable systems. And I'm like, "Yeah, that makes sense to me." And do you think that this capability issue is going to be something like... coherence of generated text. And that might be a technically fundamental limitation. Cool--

0:42:06.1 Interviewee: I know people who have the tools and resources and time to test, to run experiments on things like this, who I've even directly proposed this to. And they've gone, "Oh, that's interesting." And then not done it.

0:42:22.3 Vael: Yeah, my intuition is that they don't... I think you have to have a pretty strong prior on this particular thing being the thing that is going to have like a fundamental limit in terms of capabilities in order to want to do this compared to other things, but... That makes sense, though. It sounds like you do have a... You do think this particular problem is pretty important. And pretty hard to... Very, very difficult.

0:42:45.4 Interviewee: I think that this coherency problem is a serious issue for any system that is GTP-3 like, in the sense that it's trained to produce tokens or reasoning or symbols or whatever you want to say, but that produce outputs that are being fit to mimic a generative distribution-- sorry, it's being generatively trained to produce outputs that mimic a crowdsourced human distribution.

0:43:19.8 Vael: Yeah. Cool, awesome. Yup, makes sense to me as a worldview, is pretty interesting. I haven't actually heard about that problem-- of people thinking that that problem specifically, the coherency problem, is one that's going to fundamentally limit capabilities. Seems plausible, seems like many other things might end up being the limit as well. And then you're like, "Well, people should like... If this is the important thing, then people should actually test it. And then they'll feel better. Because they'll believe that these systems won't be as capable and then less likely to destroy the world." Yeah, this makes sense to me.

0:44:02.4 Interviewee: Yeah. Another aspect of this is that research into the functional limitations, in a sense, is extremely difficult to convert into capabilities research, which is something that a lot of people say that they're highly concerned about. And that they don't want to do many types of research because... There was that Nature article where they were creating a medical AI and they were like, "Let's put a negative sign in front of the utility function." And it started designing neurotoxins. Do you know what I'm referring to?

0:44:35.2 Vael: No, but that sounds bad.

0:44:36.8 Interviewee: Oh yeah, no, it's just... That was the Nature article. It was like "we were synthesizing proteins to cure diseases, and we stuck a negative sign in front of the utility function-- (Vael: Oh, was that last week or something?) Yeah.

0:44:46.2 Vael: Yeah, okay, so I did see that, yeah. Huh.

0:44:48.1 Interviewee: Yeah. Gotta love humans. Gotta love humans.

[chuckle]

0:45:02.6 Vael: ...Awesome. Ah, I think I'll.. Maybe.. Hm. Okay. So. What would... make you want to work on alignment research as you think it can be done?

[pause]

0:45:25.2 Interviewee: That's an interesting question. [pause] I guess the main thing would be being convinced of the urgency of the problem.

0:45:50.4 Vael: That makes sense. Very logical.

0:45:56.4 Interviewee: To be blunt, I don't tend to get along with the kind of people who work in that sphere, and so that's also disincentivizing and discouraging.

0:46:12.2 Vael: Yeah, that makes sense. I've heard that from at least one other person. Yeah. Alright, so timelines and also nicer research environment. Makes sense.

0:46:27.4 Interviewee: You could even say nicer researchers.

0:46:31.0 Vael: Yep. Nicer researchers. Apologies? ...Yeah. Cool. And then my last question is, have you changed your mind on anything and during this interview, and how was this interview for you?

0:46:45.9 Interviewee: The interview was fine for me. I don't think I've changed my mind about anything.

0:46:56.0 Vael: Great. Alright, well, thank you so much for being willing to do this. I definitely... Yeah. No. You have a very coherent kind of worldview thing that's... That I... Yeah. I appreciate having the ability to understand or have access to or listen to, rather.

0:47:11.7 Interviewee: My pleasure.

0:47:13.5 Vael: Alright. I will send the money your way right after this, and thanks so much.

0:47:17.2 Interviewee: Have a good day.

0:47:17.6 Vael: You too.

0:47:17.8 Interviewee: Bye.