# Interview with AI Researchers NeurIPSorICML_q243b by Vael Gates

**Interview with q243b, on 3/18/22**

**0:00:00.0 Vael:** Cool. Alright. So my first question is, can you tell me about what area of AI you work on in a few sentences?

**0:00:09.0 Interviewee:** Of course. I did my PhD in optimization and specifically in non-convex optimization. And after that I switched topics quite a lot. I worked in search at \[company\] \[\...\] And now actually I work in \[hard to parse\] research, so I kind of come back to optimization, but it's more like a kind of weird angles of optimizations such as like meta-learning or future learning, kind of more novel trends like that.

**0:00:42.1 Vael:** Cool. And next question is: what are you most excited about in AI and what are you most worried about? In other words, what are the biggest benefits or risks of AI?

**0:00:51.1 Interviewee:** Well, the benefit is that AI has slowly but surely kind of taken over a lot of problems in the world. And in fact pretty much any problem where you need to optimize something, you can use AI for that. I'm more of a traditional machine learning person, in the sense that\... Currently I include everything, including not necessarily neural networks, also like logistic regressions, decision trees, all those things. And I think those things have been grossly unutilized, I would say, because a lot of problems right now in machine learning that people think, "Okay, should this solve this neural networks?" but in reality, just in decision trees. But stay the same because of the current trends and because of the current hype of neural networks as well, they kind of came along with it as well. All the publicity and marketing that, you know, that AI should have, honestly. And I think more and more companies realize that you can solve the problem with just simple solutions. And I think that will be a really exciting part. So just like, to answering to your question yeah, I'm excited about the fact that machine learning has just become more and more and more ubiquitous. It becomes like almost prerequisites for any big company or even a smaller company. And the second part, what I'm worried the most. I don't know, that's a good question.

**0:02:07.7 Interviewee:** I mean, I guess like, I mean, I don't share those worries that AI would dominate us and we would all be exterminated by overpowerful AI. I don't think AGI is coming anytime soon. I think we're still doing statistics in a way, like some kind of belong to this camp will just think, we're still doing linear models and I don't believe the system is conscious or anything of those sorts. I think the most malicious use, like, I mean, especially now currently with the war, I see more and more people using AI for malicious sense. Like not necessarily they will be, you know, we're going to have next SkyNet coming, but in the bad hands, in the bad actors, you know, AI can serve not a good purpose in war. Like for example, you know, like now with drones, with, you know, the current war, for example, in Ukraine is more and more done with drones. And drones have automatic targeting, automatic navigation. And yeah, so that's kind of not necessarily a good thing and they can become more and more dramatic, and more automatized and they can lead to harm.

**0:03:06.4 Vael:** Yeah, it makes sense. Lot of strong misuse cases. So focusing on future AI, putting on like a science fiction, forecasting hat, say we're 50 plus years into the future, so at least 50 years in the future, what does that future look like?

**0:03:19.5 Interviewee:** Well, I hope\... I mean I kind of, you know, I really like that, my favorite quote in a\... And probably that one of the quotes that I really like is that the arc of justice is nonlinear, but it kind of bends towards justice. I really like it. And I really hope in 50 years we would actually figure out first of all, the way to harness our current problems and make not necessarily disappear, but at least make it controllable such as nuclear weapons and the global warming. And that's in 50 years, I think it's a reasonable time. Again, not to solve them, but just to figure out how to harness these issues. And AI should definitely help that. And do you want me to answer more specifically, like more, like give you ideas of how I think in 50 years the world would look like? (Vael: "Yeah, I'll take some more specifics.")

**0:04:01.3 Interviewee:** Alright. I mean, I think that one of my exciting areas that I think that right now is already kind of flourishing a little bit, and it's large language models. It's a current trend, so it's kind of an easy, like on the surface thing to talk about, and I think I\... As a \[large tech company\] employee, I can see how they've been developed over like two years, things changing dramatically. And I think these kind of things are pretty exciting. Like having a system that can talk to you, understand you, respond not necessarily with just one phrase, but like accomplish tasks that you wanted to accomplish. Like now it's currently in language scenarios, but I also, within 50 years definitely anticipate it could happen in like robotics, like a personal assistant next to you or something like that. Another area I'm really excited about is medicine.

**0:04:46.1 Interviewee:** I think once we figure out all the privacy issues that surround medicine right now, and we're able to create like, clean up database, so to speak, of patients diagnoses. And I hope that it'll be enough for a machine learning model to solve like cancer as we know it and things like that. I'm just hopeful. I mean, I hope it's going to happen in 50 years and it's going to, I don't know if I want to place my bet there, but I'm hoping that would happen. So I guess in robotics, as well, as I said, one of the things that we're kind of inching there, but not quite there, but I think in 50 years, we'll solve it. So I think these three things: personal assistant, solving medicine and robotics, these three things.

**0:05:28.2 Vael:** Wow. Yeah. I mean, solving robotics would be huge. Like what's an example of\... could you do anything that a human could do as a robot or like less capable than that?

**0:05:36.0 Interviewee:** I think so. I mean, it depends. It depends what you mean by human, right? I mean the\... Well, if you try to drive a car for the last 20 years. We've been trying to do that, but honestly, I think this problem is really, really hard because you have to interact with other agents as well. That's kind the main thing, right? You have to interact with other humans mostly. I mean, I think interaction between robots, it's one thing, interaction between robots and robots is much easier. So I think whatever task that doesn't involve humans is actually going to be pretty useful. Well, again, actually pretty easy because it hasn't been solved yet, but I think it's much easier than solving with humans.

**0:06:05.8 Interviewee:** And like for example organizing your kitchen, organizing your room, cleaning your room, cooking for you, I think all the things should be pretty straightforward. Again, the main issue here is that every kitchen is different, so although we can train a robot to a particular kitchen or like do some particular kitchen, like once it's presented with a novel kitchen with novel mess\... mess is very personal. So it'd be harder for the robots to do. But I think that's something that would be kinda within the reach, I think.

**0:06:37.7 Vael:** Interesting, and for like, solving cancer, for example, I imagine that's going to involve a fair amount of research per se, so do we have AIs doing research?

**0:06:47.2 Interviewee:** So research, I want to distinguish research here because there is research in machine learning and there is research in medicine. And they are two different things. The research in medicine, and I'm not a doctor at all, but from what I understand, it's very different, in the sense that you research particular forms of cancer, very empirical research. Like hey we have\... Cancer from what I understand, one of the main issues with cancer is that every cancer is more or less unique.

**0:07:11.8 Interviewee:** It's really hard to categorize it, it is really hard to do AB testing. The main research tool that medical professionals use is AB testing, right, you have this particular group of cancers, group of people, that suffer from this particular cancer. Okay, let's just come up with a drug that you can try to put these people on trial, and do that. But because every cancer is unique, it's pretty hard to do that. So, and how to do this research is data, and that's what we need for machine learning, we need to have sufficient data such that machine learning can leverage that and utilize it. So they're now asking questions in two perspectives, one is do we need more data? Yes, absolutely.

**0:07:46.3 Interviewee:** Moreover we not only need data, we need to have rules for which these machine learning agents, that of a company, university, would have access to this data differentially private right in the sense that this should be available to them. But is possibly private, of course privacy is a big issue. Which right now doesn't really happen, plus there are other bureaucratic reasons for this not to happen, like for example hospitals withholding the data because they don't want to share it and stuff like that.

**0:08:16.3 Interviewee:** So if we can solve this problem, the research and medical part would be not necessarily\... Not necessary for the machine learning. And on the machine learning side, there is also, as well, it's very big hurdles in the sense that current machine learning algorithms needs tons of data. Like for the self-driving cars, they're still talking about we need millions and millions of hours cars driving on the road, and they still don't have enough. So for cancer, that's kind of not be the case hopefully. Right? So hopefully we're going to come up with algorithms that work with fewer data. Like one of the algorithms is so called few-shot algorithms, so when you have algorithms that learn on somebody's language, but when you want to apply to a particular patient in mind, you just need to use specific markers to adjust your model to the specific patient. So there are some advancement in this way too but I think we are not there yet.

**0:09:07.1 Vael:** Interesting. Cool, alright, so that's more\... It's not like the AI is doing research itself, it's more that it is, like you're feeding in the data, to the similar types of algorithms that already exist. Cool, that makes sense. Alright so, I don't know if you're going to like this, but, people talk about the promise of AI, by which thy mean many things but one of the things is that\... The frame that I'm using right now is like having a very general capable system, such that they have the cognitive capacities to replace all current day human jobs. So whether or not we choose to replace human jobs is a different question. But I usually think of this as in the frame of like, we have 2012, we've the neural net\... we have AlexNet, the deep learning revolution, 10 years later with GPT-3 which has some weirdly emergent capabilities, so it can do text generation, language translations, some coding, some math.

**0:09:51.9 Vael:** And so one might expect that if we continued pouring all the human effort that has been going into this. And nations competing and companies competing and like a lot of talent going into this and like young people learning all this stuff. Then we have software improvements and hardware improvements, and if we get optical and quantum at the rate we've seen that we might actually reach some sort of like very general system. Alternatively we might hit some ceiling and then we'd need do a paradigm shift. But my general question is, regardless of how we get there, do you think we will ever get to a very general AI system like a CEO or a scientist AI? And if so, when?

**0:10:23.6 Interviewee:** So, my view on that is that it's really hard to extrapolate towards the future. Like my favorite example is I guess Elon Musk\... I heard it first from Elon Musk but it's a very known thing. Is that, "Hey we had like Pong 40 years ago, it was the best game that ever created, which was Pong, it was just like pixels moving and now we have a realistic thing and VR around the corner, so of course in 100 years we will have like a super realistic simulation of everything, right? And of course in a 1,000 years we'll have everything, \[hard to parse\] everything."

**0:10:53.0 Interviewee:** But again it doesn't work this way. Because the research project is not linear, the research progress is not linear. Like 100 years ago Einstein developed this theory of everything, right, of how reality works. And then yet, we hit a very big road block of how exactly it works with respect to the different scale, like micro, macro and micro and we're still not there, we propose different theories but it's really hard. And I think that science actually works this way pretty much all around the history it's been like that, right. You have really fast advancement and then slowing down. And in some way you have advancement in different things.

**0:11:26.8 Interviewee:** Plus the cool thing about research is that sometimes you hit a road block that you can't anticipate. Not only there are road blocks that you maybe don't even imagine there are, but you don't even know what they could be in the future. And that's the cool part about science. And honestly, again, I think if we are indeed developing AGI soon, I think it's actually a bad sign. Honestly, I think it's a bad sign because it means that it's\... It's like too easy, then I'll be really scared: okay what's next? Because if we developed some really super powerful algorithm that can essentially super cognitive\... Better and better cognition of humans, I think that will be scary because then I don't even know. First of all, my imagination doesn't go further than that, because exactly by definition it will be smarter than me, so I don't even know how to do that. But also I think it means that my understanding of science is wrong. Another example I like is that someone said, if you're looking for aliens in the universe right now and then this person says, if we actually do discover aliens right now, it's actually a very bad sign. It's a bad sign in the sense that\...

**0:12:29.7 Interviewee:** If they're there, it means that the great filter, that whatever concept of great filter, right, that we're kind of in front of it, not some behind us, it is in front of us. Just means there's some really big disaster coming up, because it actually, if aliens made it as well, this mean that they also pass all the filters behind us, it mean that some bigger filters in front of us. So I kind of belong to that camp. Like I'm\... I'm kinda hoping that the science will slow down. And we'll not be able to get there. Or there's going to be something\... It's not that I think that human mind is unique and we can't reproduce it. I just think that it's not as easy as we think it would be, or like in our lifetime at least.

**0:13:05.0 Vael:** I see. So maybe not in our lifetime. Do you think it'll happen in like childrens' lifetime?

**0:13:10.1 Interviewee:** Which children? Our children hopefully not. But I mean, at some point I think so. But again, I think it'll be very different form. Humans' intellect is very, very unique, I think, and because it's shaped by evolution, shaped by specific things, specific rules. So I also kind of believe in this, in the theory that in a way computers already\... They are better than us because they are faster, to start with, and then they can\... Another example I really like is that if you remember the AlphaGo playing Go with Lee Sedol, like one of the best two players of Go. And there was a\...

**0:13:43.0 Interviewee:** If you remember the Netflix show, there was like in one room they actually have all the journalists and they were sitting next to Lee Sedol playing with the person that represents DeepMind. And then all the DeepMind engineers and scientists, they were in a different room. And in that room, when they were watching the game playing, \[in\] that room the computer said by the move number 30, very early in the game, it says, okay, I won. And it took Lee Sedol another like half an hour or more, another like a hundred moves to confirm that statistic. And they were\... The DeepMind guys were celebrating and these guys were like all thinking about the game, how to\... But the game was already lost.

**0:14:16.6 Interviewee:** So computers are already bett\--\... I mean, of course it's a very constrained sandbox around the Go game. I think it's true for many things, computers are already better than us. We are more general in our sense of generality, I guess. So maybe they will go in different direction\... But the world is really multidimensional and the problems that we solve are very multi-dimensional. So I think it's too simplistic to say that, then you're universally better than us, or we are clearly subset and they are superset of our cognition. It's, I don't know\... I think it \[hard to parse\].

**0:14:44.9 Vael:** Great. Yeah. I'm going to just hark back to the original point, which was when do you think that we'll have systems that are able to like be a CEO or a scientist AI?

**0:14:55.0 Interviewee:** Okay. Yeah, sure. Again, sorry\-- sorry for not giving you a simple answer. Maybe that's what you're looking for, but let me know if this is\... (Vael: "Nah, it's fine,")

**0:15:06.5 Interviewee:** Yeah. I don't know. In a way like\... The work that like accountant does right now, it's very different than what accountant did 30 years ago. Did we replace it? No, we didn't. We augmented work. We changed the work of accountant so that the work is now simpler. So replacing completely accountant, in a way, yes, we also\... Because the current, the set of tasks that accountant did 30 years ago, it's automated already. Do we still need accountants? Yes. So same here. Maybe the job that CEO is doing right now in 30 or 40 years, everything that right now, as of today, CEO is doing in 40, 30 years, we will still\... The computer will do it. Would we still need the human there? Yes. If this answers your question.

**0:15:45.1 Vael:** Will we need the humans? I can imagine that we can have like, eventually AI might be good enough that we could have it do all of our science and then, or it's just so much smarter than us then we're just like, well, you're much faster at doing science. So I'll just let you do science.

**0:15:57.8 Interviewee:** So let me rephrase your question a bit. So what you're saying right now is it is a black box right now, that right now, that's CEO right now, that's a CEO job. That's what CEO is doing. There's some input, then some output. So what you're saying that now we can automate it. And now the input and output will also feed through something to computer let's say, but then what is, what would be the\... We'll have to refine the input and output then, because it still should serve humans. Right?

**0:16:21.9 Interviewee:** So previously you need to have drivers, like for the tram you're having to have a driver. Now instead of drivers, you have computers, but you still need to have a person to supervise the system in a way. Or, but then you're talking about even that being automated. But in same time, you cannot\... Like the system, like for example, self-driving car, it's become a tool for someone else. So you're removing the work of a driver, but you replace it with a system that now it's called something else. Like Uber. Previously, you had to call a taxi. Now you have an app to do it for you.

**0:16:49.7 Interviewee:** There's an algorithm that does it for you. So the system morph into something else. So same thing here. I think as CEOs, in a sense, they might be replaced, but the system also would change as well. So it won't be the same. It won't be like, okay, we'll have a Google. And there is a CEO of Google who is like robot. Now the Google will morph in a way that the task the CEO is doing would be given the computer. But the Google will still\... Like also, by the way, Google, even Google. Google works on its own. In fact, if you, right now, fire all the employees, it'll still work a few days. Everything that we do, we do for the future. Like it's pretty unique moment in history, right? 'Cause previously, like before the industrial revolution, you had to do things yourself. Then with the factories and factories well then, okay, you're helping factory to do its work. And now there is a third wave, whatever, fourth wave, industrial revolution. We don't even do anything. It's on\... In a way Google doesn't have a CEO, the Google CEO doesn't work for the today Google. Google CEO works for the Google in a year which is\... So Google work\... Google is already that. Google doesn't have a CEO. So that's what I mean.

**0:17:56.8 Vael:** Alright. Uh, I'm going to do the science example, because that feels easier to me, but like, like, okay. So we're like doing\... We're attempting to have our AI solve cancer and normally humans would do the science. And be like, okay, cool I'm going to like do this experiment, and that experiment, and that experiment. And then at some point, like we'll have an AI assistant and at some point we'll just be like, alright, AI solve cancer. And then it will just be able to do that on its own. And it's still like serving human interests or something, but it is like kind of automated in its own way. Okay. So do you think that will\... When do you think that will happen?

**0:18:34.0 Interviewee:** The question is how with\... The question is, can this task be, sorry I know you're asking about the timeline and I want to be, I know\... I don't want to ramble too much. But I think I want to be specific enough, what kind of problem we're talking about. If we're talking about the engineering problem, that we're talking about the timeframe of our lifetime. If you're talking about a problem that involves more creativity, like for example, come up with a new vaccine for the new coronavirus? Sorry, it's automatic. I think that work we could do in the 20\... 20-30 years. Right, because we have tools, we know engineering tools, what needs to be done, where you can do ABC, you're going to get D. Once you have D you need to pass it through some tests and you're going to get E and that pretty much automate. I think this we can do in 20-30 years. Solving cancer, I just don't know enough. How much creativity needs to be there. So more harder, probably, yeah. Yeah, yeah.

**0:19:21.0 Vael:** Yeah, no, no, that's great. And yeah, and you don't know when we'll\... For\... So probably more than our lifetime, or more than 30 years at least for creating those?

**0:19:28.9 Interviewee:** Mm-hmm. Mm-hmm.

**0:19:30.1 Vael:** Alright, great. Cool. Alright, I'm moving on to my next set of questions. So imagine we have a CEO AI. This is\... I'm still going back to the CEO AI even though\... (Interviewee: "Sure, of course.")

**0:19:40.8 Vael:** And I'm like, "Okay CEO AI, I want you to maximize profits and try not run out of of money or exploit people or\... try to avoid side effects." And this currently is very technically challenging for a bunch of reasons, and we couldn't do it. But I think one of the reasons why it's particularly hard is we don't actually know how to put human values into AI systems very well, into the mathematical formulations that we can do. And I think this will continue to be a problem. And maybe, and\... I think it seems like it would get worse in the future as like the AI is optimizing over, kind of more reality, or a larger space. And then we're trying to instill what we want into it. And so what do you think of the argument: highly intelligent systems will fail to optimize exactly what their designers intended them to, and this is dangerous.

**0:20:28.3 Interviewee:** Okay, so this is a quite big misconception of the public is that, and I think actually, it's rightfully so, because I am actually working on this, is that\... So what you said right now is like the famous paperclip example, right? Which is going to turn the whole world in particular, and that's kind of what you said more or less, right? So the problem here is that current system, a lot of them, it's true, they work on a very specific one dimensional object. There's a loss function that we're trying to minimize. And it's true, like GPT-3 and all these system, currently they are there, they have only one number, it want to minimize. And this is, if you think about it, is way too simplistic for this very reason, right? Exactly, because if you want to just maximize number of paperclips, you're just going to turn the whole word into the paperclip machine factory. And that's the problem. But the reality is much more complicated. And in fact, we are moving there, like my research was, we're moving away from that. We're trying to understand, we're trying to understand if first of all intelligence can be emerge on its own without it being minimized explicitly.

**0:21:20.2 Interviewee:** And second of all, these pitfalls, the pitfalls that you're just minimizing one number and of course, not going to work. So answering your question, yeah, yeah I think it will fail, because it will face the real world scenario unless it has specific checks and balances. For example, there's also online learning paradigm, right, where you basically learn from every example as they come in the time series. I think this system need to be revamped to work in a larger scale obviously, but this is the kind of system that potentially could work, where you don't just minimize, maximize one objective, but you have a set of objectives or you have a set of actors with their own goals and their intelligence is emerging from them. Or you learn online. You just fail and you learn and you forget what you learned, you have\... learn in a continuous fashion. So like all of these things that we as a humans do, could be applicable for AI as well. Like we are the humans, we don't have\... You don't spend your day like, go to sleep, okay, today, day was like 26. You don't do that. And even if you do that, you probably will have multiple numbers. This was 26, this was 37. Okay, it doesn't matter the day.

**0:22:19.3 Vael:** Yeah, that makes sense. So, one thing I'm kind of curious about is there's\... like, when you say, it won't work, just optimizing one number. And will it not work in the sense that, we're trying to advance capabilities, we're working on applications, Oh, turns out we can't actually do this because we can't put the values in. And so it's just going to fail, and then it will get handled as part of the economic process of progress. Or do you think we'll go along pretty well, we'll go along fine, things seem fine, things seems like mostly fine. And then like if things just diverge, kind of? Kind of like how we have recommender systems that are kind of addictive even though we didn't really mean to, but people weren't really trying to make it fulfill human values or something. And then we'd just have the sort of drift that wouldn't get corrected unless we had people explicitly working on safety? Yeah, so, yeah, what do you think?

**0:23:06.4 Interviewee:** I think people who work on safety, and you could see it yourself, people who work on safety, people who work on the fairness, people who work on all the things, checks and balances, so to speak, right? They are becoming more and more prominent. Do I know it's enough? No, it's not enough, obviously we need to do more for different reasons. For obviously DEI, but also for just privacy and safety and other things. And also for things we just talked about, right? Just because it's true that the fact that\... By the way, the fact that we have only the current system like GPT-3 or many other algorithms minimize only one value, it's not a feature, it's a bug. It's just convenience because we use the machine learning optimization algorithms that work in this way, and we just don't have other ways to do that. And I hope in the future other things would come up.

**0:23:46.0 Interviewee:** In answering your question, I don't think it will necessarily diverge, we'll just hit a roadblock and we're already hitting them. You've heard about AlexNet like 10 years ago, and now, sure, we have acute applications and like filters on the phone, but like, did AI actually enter your life, daily life? Well, not true, I mean, you have better phone, they're more capable, but actually AI, like in terms of that we all dream about, does it enter your life? Well, not really. We can live without it, right? So, we're already hitting all these roadblocks, even like medical application. Google 10 years ago, claimed they'd solved like the skin cancer when they can detect it, and it didn't\... It didn't really see the light of day except for some hospitals in India, unfortunately. So we're already hitting tons of roadblocks, and I don't think it's\... It's like for this reason precisely, because when you face reality, you just don't work as good as you expect for multiple reasons.

**0:24:30.4 Vael:** Interesting. Cool. So do you think that others\... This\... Have you heard of the alignment problem? Question one.

**0:24:38.1 Interviewee:** No.

**0:24:40.7 Vael:** No. Cool, all right, so you've definitely heard of AI safety, right?

**0:24:43.5 Interviewee:** Mm-hmm.

**0:24:43.9 Vael:** Yeah. Alright. So one of the definitions of alignment is building models that represent and safely optimize hard-to-specify human values, alternatively ensuring that AI behavior aligns with system designer intentions. So I'm\... So one of the questions, the question that I just kind of asked, was trying to dig at, do you think that the alignment problem per se, of trying to make sure that we are able to encode human values in a way that AIs can properly optimize, that that'll just kind of be solved as a matter of course, in that we're going to get stuck at various points, then we're going to have to address it. Or do you think that'll just like\... We will\... It won't get solved by default? Things will continue progressing in capabilities, and then we'll just have it be kind of unsafe, but like, "Uh, you know, it's good enough. It's fine."

**0:25:21.1 Interviewee:** I think a bit of both. There's so much promise and so much hype and so much money in pushing AI forward. So I think a lot of companies will try to do that. These various\... We live in democracy, fortunately or unfortunately or actually we live in more like value of dollar, unfortunately, our society. At least in some countries are valued by progress. And especially companies, they have to progress, they have to advance and this is one of the easiest ways to advance. But I think some companies may be bad actors, whatever, they will try to push it to the limit. But these questions are ultimately unsolved, in a way this current system are designed, for the reason we discussed. So I think it will be a bit of both. Some companies will back down, some companies will try to push it to the limit, so we'll see. Depends on applications as well. I mean, some applications are safe. If you\... I'm sorry, but\... Sorry to bring it up, but for example, there was a case of AI in Microsoft when they released the bot and the bot start cursing. Which is okay. It's a cute example, they should have done a bit of PR loss, it's fine, but it's different from the car crashing you into that tree. So depends, depends on the application.

**0:26:27.4 Vael:** Yeah, it seems definitely true. Alright, so next argument is focusing on our CEO AI again, which can do multi-step planning, and it has a model of itself in the world. So it's modeling other people modeling it, because it feels like that's pretty important for having any sort of advanced AI that's acting as a CEO. So the CEO is making plans for the future, and it's noticing as it's making plans for its goal of maximizing profits with constraints that some of its plans fail because it gets shut down. So we built this AI so that it has the default thing where you have to check with humans before executing anything, because that seems like a basic safety measure. And so the humans are looking for a one page memo on the AI's decision. So the AI is thinking about writing this memo, and then it notices at some point that if it just changes the memo to not include all the information, then maybe the AI will be\... I mean, then the humans will be more likely to approve it, which means it would be more likely to succeed in its original goal.

**0:27:22.8 Vael:** And so, the question here is\... So this is not building self preservation into the AI, it's just like AI as an agent that's optimizing any sort of goal and self preservation is coming up as an instrumental incentive. So, what do you think of the argument, "Highly intelligent systems will have an incentive to behave in ways to ensure that they are not shut off or limited in pursuing their goals and this is dangerous?"

**0:27:42.9 Interviewee:** Well, you already put intelligence and the human level into the sentence as well.

**0:27:48.3 Vael:** Yes.

**0:27:48.6 Interviewee:** Yeah, and it kind of\... I'm already against that, because I don't think the system would actually behave in a way, like a sneaky way to avoid. Well, first of all, the current AI, even currently AI systems are highly uninterpretable. It's very hard to interpret what exactly is going on, right? But it still work within the bounds, right? So the example I gave you was Lee Sedol, when the Go game, already knew it's won, but it couldn't explain why it won, and it take human to parse it. Or another example that I like, in the chess, AI, at some point, put a queen in the corner, something you just never do as a human. And it couldn't explain, obviously, in fact. But it never pick up the board and crash it on the\... It will work within the bounds.

**0:28:29.1 Interviewee:** So of course, if the bounds of this program allows the program to cheat in a way and withhold information, then yes, but\... Again, it kinda works pair in pair. On one side it's really hard to interpret, so this one-pager that AI has to provide, it also has to be curated. And it will not include all information because it's impossible to digest all the computer memory and knowledge into one page. So on the one side, this page will always be limited and with all this lossy compression of the state of computer. But on the other side, I don't think computers can on purpose cheat on this page. Actually they might, depending on the algorithm\[?\], again, but I think it's a valid concern. That's an easy question, an easy answer, but it's depending on the system, depends how you design it.

**0:29:09.5 Vael:** Yeah, so I think I'm trying to figure out what exactly is in this design system. So one thing is it has to be very capable, and it has to\... I want it to be operating over like reality in a way that I expect that CEOs would, so its task is interacting\... It's doing online learning, I expect, and it's interacting with actual people. So it's giving them text and it's taking in video and text, and interacting with them like a CEO would. And it does have to, I think, have a model of itself in the world in order for this to happen and to model how to interact with people. But if we have AI to that extent, which I kind of think that eventually we'll develop AI with these capabilities. I don't know how long it will take, but I assume that these are commercially valuable enough that people will eventually have this sort of system that\... This is an argument that like\... And I don't know if this is true, but any agent that's optimizing really hard for a single goal will at some point, like, figure out if it's smart enough to\... That it should like try to eliminate things that are going to reduce the likelihood of its plan succeeding, which in this case may be humans in the way.

**0:30:10.5 Interviewee:** I think you're right. Actually, while you were talking I also came up this example that I really like about, maybe you saw it as well, where there's an agent that optimized solving a particular race game and you control the car. At some point found a loophole, you remember \[hard-to-parse\] example, and find a loophole and just goes in circles. And the answer to that, you need to have explicit test goals. But in online learning settings, it's really hard. Plus, again, coming back to question number one, which is like, the system is so large at some point, you're just not able to cover all the cases. So yeah, yeah, I think so, I think it's possible, especially today in online learning fashion when you can't really have a complete system that you have all integration tests possible, and then you ship it. Once a system that automatically learns and updates, then it becomes\... That could be a problem. Yeah, I agree.

**0:30:52.8 Vael:** Yeah. So this boat driving race is actually one of the canonical examples of the alignment problem, as it were, which is like\-- you put in the right example into it. There's another part of the\... so that's one version of "outer alignment", which is where the system designer isn't able to input what it wants into the system well enough, which I think gets harder as the AI gets more powerful. And then there's an "inner alignment" kind of issue that people are hypothesizing might happen, which is where you have an optimizer that spins up another optimizer inside of it, where the outer optimizer is aligned with what the human wants, but it now has this inner optimizer. The canonical example used here is how evolution is in some sense the top-level optimizer and it's supposed to create agents that are good at doing inclusive reproductive fitness and instead we have humans. And we are not optimizing very hard for inclusive reproductive fitness and we've invented contraceptives and stuff, which is like not very helpful. And so people are worried that maybe that would happen in very advanced AIs too, as we get like more and more generalizable systems.

**0:31:51.0 Interviewee:** Yeah. I think that there are two things to say, first of all, is what you said about loop in the loop. Now we're talking about what exactly system, what kind of a system can an algorithm create, right? Because for example, if you look at the current machine learning, we do know that for example, convolutional layers, they are good at computing equivariances, like transitional equivariances. If you move the object, they're supposed to be indifferent, but \[hard to parse\]-connected layers don't do that. So this \[hard to parse\] behavior you described? You need to have a system, first of all, that's capable enough for this kind of behavior. That's a big if, but okay. Once we get there, if we get there, the second question is, okay, can you cover for that? Can you figure out that these cases are eliminated completely from existence?

**0:32:32.4 Interviewee:** And the example of CEO is maybe is a good example. For me, the interesting example that is\... that I can definitely see and envision, not even CEO, but like for example, application that controls your behavior in a way. Like, for example, curate your Tinder profile or curate your inbox sorting. And control you through that. Then yeah, for sure. Yeah. If you don't control for everything, it can be smarter than us and kind of figure out, back-engineer, how humans work, because we're not \[hard to parse\] than that. And curate the channel for us and even\... Get us to the local minimum we might not want, but we'll still maximize its profits, whatever the profit means for the computer.

**0:33:07.2 Vael:** Okay. Yeah. So I'm one of those people who is really concerned about long term risk from AI, because I think there's some chance that if we continue along the developmental pathway that we have so far, that if we won't solve the alignment problem, we won't figure out a good way in order to like encode human values in properly and include the human in the loop properly. Like one of the easiest solutions here is like trying to figure out how to put a human in that loop with a AI, even if the AI is vastly more intelligent than a human. And so people are like, Oh, well maybe if you just train an AI to like work with the human and translate the big AI so that like the human understands it, and this is interpretability, and then you have like a system who's training that system and maybe we can recursively loop up for something. But.

**0:33:52.2 Interviewee:** The problem here, why they see, like, for example, let's take a very specific example. There's AI systems that, for example, historically curated, curated by humans, obviously without bad intent, but it gives bad credit scores to black population say, like people of color. That's really bad behavior. And this behavior is kinda easy to check because you have statistics and you look at statistics. It's one, let's call it one hop away. So you take the data, you take statistics, is done. The second hop, the two hops away would be: it creates dynamics that you can check, not right away, but later that potentially show something like that. It would be harder for humans to check. You can also, if you think about it for a while, you can come up with like three hop, right. Something that creates something that creates something that it does. So it's much easier and much harder to check. You don't know until it happens, and that's the point. So you have this very complicated dynamic you can't\... There's not even flags that you can check, red flags that you can check in your model. That might be the issue.

**0:34:49.3 Vael:** Yeah. There's also\... Yeah. Interpretability seems really\... It seems really important, especially as very, very intelligent systems, and we don't know how to do that. So possible versions of things going badly, I think, are\... So if you have an AI system that's quite powerful and it's going to be instrumentally incentivized to not let itself be modified, then that means that you don't have that many chances to get it right in some sense, if you're dealing with a sufficiently intelligent system. Especially also because instrumental incentives are to acquire resources and influence and so\... and also improve itself. Which could be a problem maybe of recursive self-improvement. And then it, like, can get rid of humans very quickly if it wanted to, via like synthetic biology. Another kind of\-- this is not as advanced AI, but what if you put AI in charge of the manufacturing systems and food production systems, and there's some sort of correlated failure. And then we have misuse, and then the AI assisted-war as, like, various concerns about AI. What do you think about this?

**0:35:37.9 Interviewee:** Yeah. So one thing I want to say is that the AI system are kinda too lazy. In a sense that the reason why this loophole worked with the car is because this solution is easier in some mathematical way to find the proper solution.

**0:35:52.7 Interviewee:** So one thing humans can address this problem is just looking at\... Use a supervisor as a human, or with a test, which is pretty much like supervisors\-- check all the check and balances. We can create the system for which the, maybe mathematical even, finding proper solution is easier for the computer than finding its loopholes. That's one thing I want to say. And the second thing I want to say, which is now coming to nature because now we're talking about the rules of nature. But if it so happened that we design a system for which finding the loophole\... for example, we have laws of physics\-- we also have laws in behavior. Like if you have an algorithm that, you know, wants to organize human behavior or something, there's also laws of behavior. So it might be an interesting question that once we have this algorithm, we can get a bunch of sociologists, for example, people who are familiar with it, to study this algorithm and figure out that if, for example, loopholes are actually more probable than normal behavior quote on quote. So for example, being a bad person is better than being a good person\-- or easier. Not necessarily better, but easier than being a good person. Which we kind of see it in society sometimes.

**0:36:51.5 Interviewee:** So it's curious if an algorithm will actually discover that. And finding this loophole with a car is easy because it's there, you just need to move a little bit. So for a computer is easy to find it, like it's a local minimum that's really easy to fall in. But if you design the system for which these loopholes are hard, that might be easier. Or the question is, can we define a problem for which the proper solution is easier?

**0:37:11.9 Vael:** Yeah. And I think the problems are going to get much more\... harder and harder. Like if you have an AI that's doing a CEO thing, then I imagine like, just as humans figure out, there's many loopholes in society, many loopholes in how you achieve goals that are much cheaper in some sense. So I do think it's probably going to have to be designed in the system rather than being in the problem per se, as the problems get more and more difficult. Yeah.

**0:37:32.1 Vael:** So there's this community of people working on AI alignment per se and long term AI risk, and are kind of trying to figure out how you can build\... How you can solve the alignment problem, and how you can build optimization functions or however you need to design them, or interpretability or whatever the thing is, in order to have an AI that is more guaranteed, or less likely to end up doing this kind of instrumental incentive to deceive or not want to\... not self-preservation, maybe get rid of humans. So I think my question right now is, and there's money in this space now, there's a lot of interest in it and I'm sure you've heard of it\-- I mean, I'm not sure you've heard about it, but there's\... Yeah. There's certainly money and attention these days now that there wasn't previously. So what would cause you to work on these problems?

**0:38:12.4 Interviewee:** \...Well, ultimately a lot of people are motivated by actually just very few things and one of them is kids. Kids cause here you want to have a good future for yourself and the kids.

**0:38:24.8 Interviewee:** You want to live in a better and better human, better and better society, better and better everything. So that, and actually it hits home. The examples we discussed even during this call are, could be pretty grim. If we don't make this right. So putting resources there I think is really important. If people, before they come up with atomic bomb, they will figure out situations of which we are facing now, people might not even come up with atomic bomb, but do it in a safer way. Or like people knew about Chernobyl before it happened, obviously they would make it better. So having this hindsight, even though we don't know what's going to happen in the future, but putting the resources there, I think is definitely a smart move.

**0:38:57.9 Vael:** Got it. And what would cause you specifically to work on this?

**0:39:03.2 Interviewee:** Yes, I know, you asked this question. Yeah, well again, particular problems. I mean, it's an interesting problem. For me I'm actually\... Since I work in optimization, so I like to have well-formulated problems. And making sure this one goes, this problem is more\... now it's kind of vague. I mean, even now we discuss it. I agree with you that this problem is valid. It makes sense. It exists. But it's still vague, like how you study it. My PhD was also on the way to interpret, to come up with a way to interpret data.

**0:39:30.2 Interviewee:** In fact, like maybe it's\... I don't want to spend too much time on it, but basically the idea is to visualize the data. If you have a very high dimension of data, you want to visualize it. But it's very lossy. You just visualize something, you just do something and it doesn't represent everything. And it's in a way it was actually, it was well-formulated, because there is a mathematical formula to minimizing. And of course it comes with conditions like \[hard to parse\] and loss and stuff, but in a way it's there, the problem is defined. So the same here. Like in a way, there's a mathematical apparatus. \...Maybe actually I'm going to be the one developing it as well. So I'm not saying that, "come give me the one I'm going to work on!" I think that's a \[hard to parse - "would be direct"?\], so problem that I would be excited about.

**0:40:06.1 Vael:** Yeah. And I mean, I think like what this field really needs right now is someone to specify the problem more precisely. Just because it's like, Oh, this is a future system, it's like at least 50 years, well I don't know at least\-- it's far away. It's not happening immediately and we don't have very good like frameworks for it. And so it makes it hard to do research on. Cool. Alright. Well, I'll send you some resources afterwards if you feel like looking into it, but if not, regardless, thank you so much for doing this call with me.

**0:40:33.1 Interviewee:** Yeah, I appreciate it. Thank you for your time, it was really fun.


# Interview with AI Researchers NeurIPSorICML_7oalk by Vael Gates

**Interview with 7oalk, on 3/20/22**

**0:00:03.2 Vael:** Alright, my first question is, can you tell me about what area of AI you work on in a few sentences?

**0:00:08.5 Interviewee:** Yeah, I study biologically-inspired artificial intelligence of building models of biological intelligence, mostly with the visual system, but also with cognitive functions, and using those models to understand differences between humans and machines. And upon finding those differences, the hope is that we can build more human-like artificial intelligence and in the process develop models that can better explain the brain.

**0:00:46.1 Vael:** Interesting, yep. I was in that space for a while. The dual\-- AI\--

**0:00:49.1 Interviewee:** Oh, cool. I saw you worked with Tom Griffiths.

**0:00:54.0 Vael:** Yeah, that's right, yep.

**0:00:55.0 Interviewee:** Yeah, very much so. Cool.

**0:00:56.5 Vael:** Alright, so my next question is, what are you most excited about in AI, and what are you most worried about? In other words, what are the biggest benefits or risks of AI?

**0:01:06.9 Interviewee:** So I'm most excited about AI, not for most of the applications that are getting a lot of praise and press these days, but more so for convergences to biomedical science. I think that the main driver of progress in AI over the past decade has been representation learning from large-scale data. And this is still an untapped area for biomedical science, so we really don't know what elements of biology you can learn predictive mappings for. So, for instance, given an image of a cell's morphology, can you predict its genome? Can you predict its transcriptome? Etcetera. So I think that what I'm most excited about is the potential for AI to completely transform the game of drug discovery, of our ability to identify the causes and identify targets for treating disease.

**0:02:18.4 Vael:** Nice.

**0:02:18.7 Interviewee:** What I'm most afraid of is in the game of AI that is most popular right now. The NeurIPS CDPR game, which\... lots of people in the field have pointed out the issues with biases. And also\... it's late here, so I'm not thinking of a good term, but this Clever Hans, Mechanical Turk-ish nature of AI, where the ability to solve a problem that seems hard to us can give this sheen of intelligence to something that's explained a really trivial solution. And sometimes those trivial solutions, which are available to these deep learning networks which exploit correlations, those can be really painful for people. So in different applications of using AI models for, let's say screening job applicants, there's all these ethical issues about associating features not relevant to a job, but rather just regularities in the data with a predictive outcome. So that's a huge issue. I have no expertise in that. It's definitely something I'm worried about. Something that's related to my work that I'm super worried about\... I mentioned I do this biological to artificial convergence, this two-way conversation between biological and artificial intelligence. The best way to get money, funding in academia, for that kind of research is going through defense departments. So one of the ARPA programs\-- IARPA, DARPA, Office of Naval Research. So I know that the algorithms built for object tracking, for instance, could be extremely dangerous. And so I build them to solve biological problems, and then I scale them to large-scale data sets, large data sets to get into conferences. But, you know, easily, somebody could just take one of those algorithms and shove them into a drone. And you do pretty bad stuff pretty easily.

**0:04:41.2 Vael:** Yeah.

**0:04:43.2 Interviewee:** Yeah. I guess one more point related to the biomedical research. There's this fascinating paper in Nature and Machine Intelligence. One of the trends right now for using AI in biology is to predict protein folding, or to predict target activity for a molecule or a small molecule. So when you do biomedical research, you want to screen small molecules for their ability to give you a phenotype in the cell. And so you can just use a multilayer perceptron or graph neural network, transformer, whatever, to learn that mapping, and do a pretty good job it turns out; it's shocking to me. But what you can do is you can either predict\-- let's find the molecule that's going to have the best target site activity, that's the best therapeutic candidate. But you can also flip it, you can optimize it in the opposite direction. (Vael: "Yeah\--") You saw that.

**0:05:48.2 Vael:** Yeah. I saw that \[paper\].

**0:05:49.0 Interviewee:** It's so obvious, but easily, easily, easily you could use any of these biomedical applications for the nefarious things.

**0:06:00.5 Vael:** Yeah, yeah, that's right. That was a very recent paper right?

**0:06:02.5 Interviewee:** Yeah, it just came out like last week.

**0:06:05.1 Vael:** Got it. Yeah. Alright, we've got excited and risks. My next question is talking about future AI. So putting on a science fiction, forecasting hat. So we're 50 plus years into the future, so at least 50 years in the future, what does that future look like?

**0:06:22.6 Interviewee:** Okay. Optimistically, we're all still here. So the reason why I got into this field is because I think it's the biggest force multiplier there is. So all of the greatest problems that\-- the existential crises we face today, there's a potential for machine learning to be part of the solution to those problems. So focusing just for a moment on health, biomedical research, which is where I'm mostly interested right now, 50 years, I think that we're going to be to a point where we know we have drugs that are completely uninterpretable to humans. We have solved disease, we have cured disease, but we don't know why. We've also created a potential issue, byproducts from that, adversarial effects, for lack of a better phrase, by curing cancer through some black box method. That's going to yield other problems, kind of like putting your finger in the dam. So I think that in 50 years from now, we will have machine learning-based solutions to some of the greatest health problems: cancer, aging, neurodegenerative disease that face us, but we won't understand those solutions. And so that's potentially the next frontier, which is to develop interpretable drugs. So I imagine that science will change. The paradigm of science will shift where there are no longer bench scientists, but instead robotics is how science is done. So scientists ask high-level questions, and then biology becomes a search problem where you just, akin to a tricorder from Star Trek where it just solves, and you move onto the next question. You can imagine that framework being applied to other problems. Climate change, hopefully. I'm not optimistic, but hopefully, you could have solutions to climate change. But there will be a black box. And when you have a black box, there's always issues with unintended effects. So I guess 50 years from now, we'll have, rather than thinking about AGI and all that stuff, we'll have solutions to some of our grandest challenges today, but those solutions may bear costs that are unexpected.

**0:09:40.8 Vael:** Yeah, that makes sense. I have few follow-ups. So you mentioned existential risks facing humanity. What are those?

**0:09:51.5 Interviewee:** So there's climate change. There's nuclear war, which is the byproduct of power. There is poverty and hunger, famine. With the risk of sounding like some VC, I think that there are chances to disrupt solutions to each of those problems using machine learning. For instance, for energy, already folks at DeepMind are using reinforcement learning to drive the development of new forms of fusion to search for better forms of fusion. Who knows if this is scalable, but I think with\... it's not going to be better algorithms, it's going to be better integration of the toolset for studying fusion and these machine learning methods to make it more rapid to test and explore those questions that are related to that field, which I don't know. I think that will lead to positive solutions. That would potentially yield lower-emission or no-emission, high-energy solutions that can power homes without any emissions. Similarly, with cars\... self-driving, as much as I think that Elon Musk is barking up the wrong tree\-- going LIDAR-free, \[inaudible\] approach to self-driving, that will be a solved problem within 10 years. And that's going to help a lot the emissions issue. So energy and global warming are kind of tied together. And I think that there's lots of different applications of machine learning and AI that can help address those problems. Likewise for famine, for world hunger. Genetically-engineered organisms are a good approach to that problem. What I mentioned about for solutions to biomedical problems, disease, etc.\-- treating those as search problems and bringing to bear the power of artificial intelligence. I think you could probably adopt a similar approach for engineering organisms that are tolerant to weather, especially the changing climate and other diseases and bugs, etcetera. I think there's going to be some overlap between the methods of work for each of these areas. I guess I didn't mention COVID and pandemics. But again, that falls into the same framework of treating these existential problems as search problems, no longer doing brute force, then science. And instead resorting to black box solutions that are discovered by large scale neural network algorithms. Did I miss any? Oh, war. Yeah, I don't know. Okay, so here's an answer. Whoever has these technologies, who's ever able to scale these technologies most rapidly, that's going to be almost like an arms race. So whoever has these technologies will be able to generate so much more value than countries that don't have these technologies that economies will be driven by the ability to wield appropriate artificial intelligence tools for searching for solutions to these existential issues.

**0:14:36.1 Vael:** Got it, thanks. Alright, so my next question is\... so this is kind of a spiel. Some people talk about the promise of AI, by which they mean many things, but that the thing I'm referring to is something like a very general capable system. So the cognitive capacities that can be used to replace all current day human jobs. Whether or not we choose to replace human jobs is a different question, but having the kind of capabilities to do that. So like, imagine\... I mostly think about this in the frame of 2012, we have the deep learning revolution with AlexNet, and then 10 years later, here we are, and we have systems like GPT-3, which have kind of weirdly emergent capabilities, that can be used in language translation and some text generation and some coding and some math. And one might expect that if we can continue pouring all of the human effort that we've been pouring into this, with all the young people and the nations competing and corporations competing and algorithmic improvements at the rate we've seen, and like hardware improvements, maybe people get optical or quantum, that we might get\... scale to very general systems, or we might hit some sort of ceiling and need to do a paradigm shift. But my question is regardless of how we'd get there, do you think we'll ever get very general systems like a CEO AI or a scientist AI and if so, when?

**0:15:55.3 Interviewee:** I don't think it's worth it. I see the AI as\... look, I grew up reading Asimov books, so I love the idea of "I, Robot", not with Will Smith, but solving all these galaxy conquest issues and questions. But I don't think it's worth it because\... I see this as a conversation between human experts, the main experts, and artificial intelligence models that are just going to force multiply the ability of human experts to explore problems. And maybe in a 100 years, when we've explored all of these problems, then we'll be so bored that we say, "Can we engineer something that contains all of these different domain expert AIs in one, in a way that can\-- those AIs can respond to a question and affect behavior in the world and have their own free will." It's more of a philosophical pursuit that I don't even, I don't\... okay, so from an economic standpoint, I don't know who's going to pay for that. Even the military would be like, "All I want is a drone that's going to kill everybody."

**0:17:30.9 Vael:** Yeah, Yeah, cool. So I have some counterpoints to that. So I'm not actually imagining necessarily that we have AI that is conscious or has free will or something. I think I'm just imagining an AI that is, has a, very capable of doing tasks. If you have a scientist AI then\...We were talking about automating science, but in this case, it's not even relying on an expert anymore. Maybe it defers to an expert. It's like, do you want me\... it looks to the expert's goals, like, "Do you want me to solve cancer?" Or like a CEO, it has shareholders. And I think that there is actually economic incentive to do this. DeepMind and OpenAI have explicitly said\... I think that they're trying to make artificial general intelligence, or like things that can replace jobs, and more and more replace jobs. And I think we're going in that direction with Codex and stuff, which is currently an assistant tool, but I wouldn't be surprised if in the end, we didn't do much coding ourselves.

**0:18:24.9 Interviewee:** Okay, so yeah, I agree with what you said. Right, it's going to change the job\... So strike from the record what I just said. It's going to change the job requirements. That would be a good feature in my opinion, where we no longer have to code because instead we describe our program. And again, it's a search problem to find the right piece of code to execute the concept that we have. We describe the medical problem that we're interested in, and we search within whatever imaging modality or probe that we're using to study that system. We can search for that solution. Where I'm not so sure is, we have this huge service industry, and you mentioned shareholders and I count that as one of them. I think when it comes to service, there's this quality, there's this subjectiveness where for instance, in medicine, talking about radiology or pathology or even family medicine. I think there will be tools to give you diagnoses, but they're always going to come with a human second opinion. So, from that standpoint, it's going to be a bit of a science fair, kind of a sideshow to the expert. I don't think that human experts in service, medical or business, etcetera, are going to be pushed aside by AI, because ultimately, I think if there's a different answer given by the artificial pathologist versus the human pathologist, the human pathologist will double check and say, "You are wrong, AI" or "You are right, AI". But the patient will never receive the artificial intelligence answer. So I think in some fields, there's going to be a complete paradigm shift in how research is done, or how work is done, research for one. Driving, supply chain, transportation, that's another example. Which will be great. I think that would be great, it will push down costs, push down fatalities, morbidity, and probably be a net good. But yeah, for a shareholder, I cannot imagine a scenario in which a CEO who has to figure out the direction of the company, has to respond to shareholders would ever say, like, "This time around, I'm just going to give an AI a GPT-6 report for the markets that we should enter, the new verticals we should work on, etcetera." Maybe it would be just be a tool to get feedback.

**0:21:54.2 Vael:** Yeah. So it seems pretty true to me that we started\... our current level of AI, which is machine learning, that they started as tools and we use them as tools. I do\-- it sort of feels to me that as we progr\--\... we've been working on AI for under 100 years, and deep learning revolution is quite new. And it seems to me like there are economic incentives towards making things more convenient, towards more automation, towards freeing up human labor to do whatever it is that people could do. So I wouldn't be surprised if eventually we reached a point where we had systems that were capable enough to be CEO. We could be like, "Alright AI, I want you to spin me up a company," and maybe you would be able to do that in some very different future, I would imagine.

**0:22:38.2 Interviewee:** Ah, so see\-- okay, so this is a question. Creativity versus capturing the regularities or the regulations of a field. So I can imagine an AI VC, easy. Or in finance, yeah. Finance, I can imagine that being overtaken by AI. I'm thinking CEO, spinning up your own company, identifying weakness in a field, or maybe some potential tool that can monopolize a field. That requires creativity. And I know there's this debate about creativity versus just interpolation in high-dimensional space of what GPT-3 is doing, and yes, I've seen some cool-looking art, but I don't know. That seems like a\... I think if you talk to me about this, if we talked about this for another week, maybe you would convince me. But I'm not quite there yet that I can imagine deep learning being able to do that. Although, now that I say that, maybe I'm just thinking about it wrong. Where most businesses just amount to, "What has worked somewhere else? Can that frame of reference work in this field? Can you disrupt this field with what's worked there?" So maybe, maybe. Okay, I agree with what you're saying. I'm evolving here, on the spot. I can see it, I can see that. I guess the one field that completely, that I'm going to push back on is medicine, where I think that part of service need a human to tell you what is up. If you don't have a human, then I can't imagine a machine diagnosis would ever fly. Maybe in 200 years, that would fly. There would have to be a lot of development of the goal for that to happen.

**0:24:45.3 Vael:** Yeah, I think there's a number of things going on here. So I don't know that we'll be able to achieve\... I do think a CEO is very high cognitive capabilities, you need to be modeling other people, you need to be talking to other people, you need to be modeling other people modeling you, you need to have multi-step planning, you need to be doing a whole bunch of very advanced\...

**0:25:03.6 Interviewee:** Yeah, convincing people, interpersonal relationships. Yeah.

**0:25:06.2 Vael:** Yeah, there's a whole bunch of stuff there, and I don't that we're\... we're not near there with today's systems for sure. And I don't know that we can get there with the current deep learning paradigm, maybe we'll need something different. I think that's a possibility. I think what I'm doing is zooming way out where I'm like, "Okay, evolution evolved humans, we have intelligence." I think that humans are trying pretty hard to make this thing happen, make up whatever intelligence is happen. And like, if we just continue working on this thing, absent some sort of huge disaster that befalls humanity, I kind of expect that we'll get there eventually.

**0:25:39.6 Interviewee:** Yeah.

**0:25:40.2 Vael:** And it sounds like, you said that like, maybe eventually, in like 200 years or something?

**0:25:45.1 Interviewee:** Definitely, definitely. So, I agree with what you said, science is incremental. And I think we have not found any ceilings in the field of modeling intelligence. We've continued to move. Even though we're kind of arguably asymptoting with the current paradigm of deep learning, we have a proof of concept that it's possible to do better, which is our own brains. And so, maybe we just need to have neuromorphic hardware or whatever, who knows, who knows? So I think you're right, and if you can operationalize, just like you did very quickly, what it takes to be a CEO. If you can build models of each of those problems and if you can do what people do, which is benchmark those models and just start to make them better and better, yeah, I can imagine having some\... It sounds crazy to even visualize that, but yeah, having an automated CEO, I think so. I think the part that's probably the most tractable is finding the emerging markets or the, like I said, the technologies that have proven to be flexible, flexibly applied, or the tricks, let's say, that have worked in multiple fields, and then identifying the field where it has just not been applied yet. Kind of that first mover type advantage. I think that's where AI could\... I can definitely imagine an AI identifying those opportunities. Almost like drug discovery, right? It's just a hit. And you'd have a lot of bad hits, and like, "This is bullshit. Nobody cares about knee pads on baby pants," which is like the WeWork founder before WeWork, but you also have some good hits. So the interpersonal stuff, yeah, that's like 200 years off.

**0:27:50.6 Interviewee:** Although you have chat bots. So I'm working on my own startups, a lot of this is trying to convince people to spend their time, that this is worthwhile, that they should leave their current stable positions to do this, that they should work\... They should do the work for you. They should drop what they're doing and do what you need them to do right now, and that you're going to help them eventually. And a lot of that is empathy and connection, and there's no evidence that we have yet that we can build a model that can do that stuff. So yeah, that's a really interesting idea, though. And I do think if you go through and operationalize each of those problems, you could make progress. So, 200 years, yes, I agree with you.

**0:28:56.7 Vael:** Alright, so maybe we'll get something like this in 200 years.

**0:29:00.1 Interviewee:** Yeah.

**0:29:00.6 Vael:** This is mainly an argument about the belief in\... like, faith in humans to follow economic incentives towards making life more convenient for themselves. That's where it feels like it's coming from in my mind. Like 200 years\... Humans have existed for a long time, and things are moving very quickly. 10,000 years ago, nothing changed from year to year, from lifetime to lifetime. Now, things are moving very quickly, and I'm like, "Hm."

**0:29:23.1 Interviewee:** I'm happy to talk with you, like, add time to this interview. But I'm just curious, what do you think? Do you think it's going to be sooner than 200 years? 10 years? 20 years?

**0:29:37.1 Vael:** I think there's a possibility that it could happen\... sooner than 200 years. \[chuckle\] I'm trying to figure out how much of my personal opinion I want to put in here. There's a study, which was like, "Hello, AI researchers. What do you think timelines?" And there's several other kind of models of this, and a lot of the models are earlier than 70 years. So, this is\... possibilities. I think it could be within the next 30 years or something. But I don't actually know, there's like probability estimates over all of these things. I don't know how these things work.

**0:30:13.2 Interviewee:** Okay. Yeah. Some McKinsey consultant put like little probability estimates.

**0:30:18.9 Vael:** Yeah, a little. It's a little bit more than that. There's a lot of surveying of AI researchers and then some people have some more fancy models, I can send you to them afterwards, you can see\...

**0:30:27.3 Interviewee:** Please, please. Yeah.

**0:30:28.6 Vael:** Great.

**0:30:29.4 Interviewee:** And like I said, I'm happy to add time to this, so\...

**0:30:32.3 Vael:** Awesome. Alright, so my next question is talking about these very\... highly intelligent systems. So in your mind, maybe like 200 years in the future. And so I guess, yeah, this argument probably feels a little bit more close to home if your timelines are shorter, but like, regardless. So, imagine we're talking about the future. We have our CEO AI, and I'm like, "Alright, CEO AI, I wish for you to maximize profits and try not to run out of money or try not to exploit people and try to not have these side effects." And this currently, obviously, is very technically challenging for many reasons. But one of the reasons I think, is that we're currently not very good at taking human values and human preferences and human goals, and then putting them into mathematical formulations such that AIs can optimize over them. And I worry that AIs in the future will continue to do what we tell them to do and not as we intend them to do. So, we're not that we won't be able to take all of our preferences\... it will continue to be hard to take our preferences and put them into something that can optimize over. So what do you think of the argument, "Highly intelligent systems will fail to optimize exactly what their designers intended them to, and this is dangerous"?

**0:31:46.6 Interviewee:** \...Yeah. Well, so\... Okay. Exactly what they're intended to do, so presumably this means, you start to test generalization of your system and some of holdout set, and it does something funky on that set. Or there's some attack and it does something funky. The question is\... so presumably it's going to do superhuman\... well, let's just imagine it's superhuman everywhere else, except for this one set. Now it's a real issue if humans are better than this AI on that one problem that it's being tested on, this one holdout problem\...

**0:32:32.4 Vael:** I think in this argument, maybe we have something like human-level intelligence in our AI and we're just trying to get it to do any goal at all, and we're like, "Hello, I would like you to do a goal, which is solve cancer." But we are unable to properly specify what we want and all the side effects and all the things.

**0:32:47.0 Interviewee:** Yeah, yeah, yeah. Yeah. No, that's exactly where I'm thinking in. So imagine that you are studying a degenerative disease and you figure out, or you ask your AI, "Can you tell me why ALS happens in focal cells" So your AI gives you an answer, "Well, it's because of some weird transcription that happens on these genes." So if you have a small molecule that addresses that, then you're going to have normal-acting genes and normal phenotype in those cells. Okay, so byproduct could be, after 10 years, some of what was happening in those genes is natural for aging, and so you introduce some cancer, because what you did to fix neurodegenerative disease is so unnatural that it has that kind of horrible side effect. Now, is that a problem? (Vael: "Yeah, I think\--") Yes. Yeah, but\--

**0:33:52.7 Vael:** I think in this particular scenario, maybe the AI would know that it was a problem. That like 10 years out, you would not be unhappy. I mean, you would be unhappy.

**0:34:03.6 Interviewee:** You would be unhappy. Yeah, so okay, even if it does know it. Is that\... So yes, that's an issue, but it's also significantly better than where we are today. And let's say the expected lifespan for ALS is five years, and now you're saying 10 years, but you're going to have this horrible cancer. That's a bad trade-off, bad decision to force somebody to make, but is undoubtedly progress. So I think black boxes.. and that would not be a black box, I believe. Black box would be like when\... you don't know, you find out about this 10-year cancer 10 years after it passes clinical trials, you know? So this would be like, almost a very good version to AI that can tell you like, "I found a solution, but given the current technology, this is the best we can do." So I would think that is a fantastic future that we\...

**0:35:00.8 Vael:** I think that's right.

**0:35:02.0 Interviewee:** The black box version, I think is also a fantastic future because it would still represent meaningful progress that would hopefully continue to be built on by advances in biomedical technologies and AI. So for this specific domain, there's obvious issues with black box. Like, if you're going to use AI to make any decisions about people, whether they can pass a bar for admissions, this or that or the other thing, there's going to be problems there. But for medicine at least, I think black boxes should be embraced.

**0:35:45.2 Vael:** Yeah, I think the scenario you outlined is actually what\-- I would say that that system is doing exactly what the designers intended them to. Like the designers wanted a solution, and even if we currently don't have best solution, it gave it the best we had. I'm like, "That seems successful." I think an unsuccessful version would be something like, "Alright, I want you to solve this disease." But you forget to tell it that it shouldn't cause any other side effects that it certainly knows about and just doesn't tell you, and then it makes a whole bunch of things worse. And I think that would be more of an example of an unsuccessful system doing something that didn't optimize exactly what they're intended.

**0:36:19.0 Interviewee:** Yeah, yeah, so that's just a failure. Okay, so here. Let's say it only does that on a certain type of people, racial profiles. Okay, so then the upshot of automating AI is that, sorry, automating science, is that it should make it a lot cheaper to test a far wider range of people. So just like, now the current state of the art for image\-- object classification is to pre-train on this 300 million image data set, this JFT-300 or whatever. Likewise, you could develop one-billion-person data sets of IPS cells, which are like stem cells. Cool science fiction there that had never happened, but then you would have this huge sample where you would avoid these kinds of horrible side effects.

**0:37:28.7 Vael:** Cool. I still think\-- yeah. I'll just\-- (Interviewee: "No, tell me. Tell me, you still think what?") Ah, no, no, I, well. I'm trying to think of how I want to order this, and we're running out of time and I do actually need to go afterwards. (Interviewee: "Oh, okay.") So I think I want to get to my next question, which does relate to how I think this previous\... where I'm like, "Hmm" about this previous question. So assume we have this powerful AI CEO system, and it is capable of multi-step planning and is modeling other people modeling it, so it has like a model of itself in the world, which I think is probably pretty important for anything to be deployed as a CEO AI, it needs to have that capability. And it's planning for\... And its goal, people have told it, "Alright, I want you to optimize profit with a bunch of these constraints," and it's planning for the future. And built into this AI is the idea that we want\... That it needs to have human approval for making decisions, because that seems like a basic safety feature. And the humans are like, "Alright, I want a one-page memo telling us about this potential decision." And so the AI is thinking about this memo and it is planning, and it's like, "Hmm, I noticed that if I include some information about what's going to happen, then the humans will shut me down and that will make me less likely to succeed in the goal that's been programmed into me, and that they nominally want. So why don't I just lie a little bit, or not even a lie, just omit some information on this one-page memo, such that I'm more likely to succeed in the plan." And so this is not like a story about an AI being programmed with something like self-preservation, but is a story about an agent trying to optimize any goal and then\...

**0:38:55.2 Interviewee:** Like shortcuts, yeah.

**0:38:56.5 Vael:** And then having an instrumental incentive to preserve itself. And this is sort of an example that I think is paired with the previous question I was asking, where you've built an AI, but you haven't fully aligned it with human values, so it isn't doing exactly what the humans want, it's doing what the humans told it to do. And so, what do you think of the argument, "Highly intelligent systems will have an incentive to behave in ways to ensure that they are not shut off or limited in pursuing their goals, and this is dangerous."

**0:39:38.5 Interviewee:** Okay, so if we are going to be using supervision to train these systems, what you're describing is a shortcut. So, highly intelligent systems exploiting a shortcut like you're talking about, which would be like, "Let's achieve a high score by doing the wrong thing\..."

**0:40:00.4 Vael:** Yeah, yep.

**0:40:00.5 Interviewee:** Can be corrected by having the right data set. Not saying that it would be this simple, input-output, learn the mapping between the two, but we'll just imagine that it is. If you have any kind of low-level bias in your data set, or high level bias, if this is the case, neural networks will learn to find that bias. So it's a matter of having a broad enough data set where that just basically, statistically, what you're talking about would not exist.

**0:40:35.4 Vael:** Interesting. I feel like\--

**0:40:37.3 Interviewee:** I don't think that would be a problem if you can design the system appropriately.

**0:40:41.2 Vael:** Yes, I totally agree this would not be a problem if you can design the system appropriately.

**0:40:44.9 Interviewee:** The data set appropriately.

**0:40:46.6 Vael:** Ah, the data set appropriately. Interestingly, on one of the examples that I know of where we have, like you set the AI, it's like a reinforcement learning agent, and you're like, "Reinforcement learning agent, optimize the number of points." And then instead of trying to win the race, which is the thing you actually wanna do, it just like finds this weird local maximum.

**0:41:02.8 Interviewee:** Yeah, so for RL, that's like the ultimate example of shortcuts. Because there, what you have are extremely\... So, for video games, if you're talking about a race, you have extremely limited sensor data, sorry, low dimensional sensor data. And you have few ways of controlling, manipulating that sensor data. So, let's say there's two local minima that can be found via optimization, and one is, "go for red, because red wins," and the other is, "Drive your car to avoid these obstacles and pass the finish line," which just happens to be red. Of course, the shortest solution will be the one chosen by the model. So I agree, shortcuts are ubiquitous, and of course, they will become more and more advanced as we move into these other domains. But I think what you're describing is just a problem with shortcuts. And so it becomes a question of, can you induce enough biases in your model architecture to ignore those shortcuts? Can you design data sets such that those shortcuts wash out via noise? Or can you have human intervention in the training loop that says, "Keep working, keep working. I don't accept this answer?"

**0:42:30.3 Vael:** Great, yeah. Cool. Have you heard of AI alignment?

**0:42:37.1 Interviewee:** Yeah, yeah, I have. Well, I've seen OpenAI talking about that a ton these days.

**0:42:42.3 Vael:** Great. Can you explain the term for me?

**0:42:46.5 Interviewee:** So this is like, you have some belief about how a system should work, and the model is going to do what it does, and so alignment means you're going to bring the system into alignment with your belief about how it should work. It's essentially what you're talking about with the shortcut problem. So, that's it, yeah.

**0:43:09.4 Vael:** Great, thanks. Yeah, I think I mostly think the problem of aligning systems with humans is going to be\... One of my central thing is that I think it will be more difficult than it seems like you think, which you were like, "Oh well, we can do it with having better data sets or we can do it with denoising things," and I'm like, I don't actually know if\...

**0:43:31.1 Interviewee:** I don't know about denoising\-- I don't mean to make this sound trivial, because it certainly is not. But to give you an example, MNIST is so confounded with low-level cues. You don't need to know "six" to recognize six, you just need to know this contour, right? Is that\... only in six do you have the top of the six and then a continuous contour. So, that's what I mean. So if you can design a data set where those don't exist, then you're golden. But usually\-- In the real world, that doesn't happen that much because we have these long-tailed distributions of stuff. So then you have to induce biases in your architecture, and this comes back to human vision, sorry, human intelligence. It's alignment with human decision-making, because we have all these biases through development through our natural\-- our genomes, through neural development that make us able to interact with the world in such a way where we don't just go for the red thing, where we're not vulnerable to adversarial examples. So I don't mean to trivialize it, I only mean to say computationally, the problem is biases within the data set, and when you use gradient descent, if that is the learning algorithm that you used to train the model that you're talking about, then that's what you're going to be fighting against.

**0:45:06.0 Vael:** Cool. Yep, makes sense.

**0:45:07.6 Interviewee:** Yeah.

**0:45:08.3 Vael:** Alright. I think my last question is just, how has this interview been for you and have you changed your mind on anything during it?

**0:45:19.1 Interviewee:** Yeah, well, it's great. I never thought of an AI CEO. I think that's super fascinating. Yeah, it was super fun. Yeah, it was great.

**0:45:33.2 Vael:** Great. Wonderful. I think I'm going to call it now, just because we both have places to be. And I will send along the money, and then I'll also send along some resources since you said you were curious about my opinions on timelines.

**0:45:47.6 Interviewee:** I definitely am. Thanks so much for your time, I really appreciate it.

**0:45:51.1 Vael:** Yeah, thanks so much for yours.

**0:45:53.9 Interviewee:** Okay. All right, take care. Have a good night.

**0:45:55.7 Vael:** Bye.