# Interview with AI Researchers individuallyselected_zlzai by Vael Gates

**Interview with zlzai, on 3/18/22**

\[Note: Th interviewee\'s Zoom connection was very bad since they were calling in from an Uber, so there\'s a lot of missing content (and \"I lost you\...\" parts of the conversation that have been removed for clarity).\]

**0:00:02.5 Vael:** Cool. All right. So my first question is: can you tell me about what area of AI you work on in a few sentences?

**0:00:09.2 Interviewee:** Definitely. The way that I describe it is, first of all, I work on deep learning, let\'s get that out of the way. And then beyond that, I\'m interested in understanding \[inaudible - long\]. If I had to describe in a word, I would call it the science of deep learning. This is a topic that there are few other researchers interested in. We try to understand\... We treat these as complex systems that have emergent properties due to that complexity, in the same way you might think of biology as an emergent property of physics. It\'s not something you might predict for first principles, but once it\'s there, there\'s a lot you can say about it that\'s pretty interesting. And then from there, I\'m interested in trying to understand how do these things learn and how can you make them learn more efficiently. You can think of that\... A metaphor I like to use is like pharmaceuticals. Once you understand a biological organism, you can design an intervention that takes advantage of the patterns that you\'ve seen in order to get a certain behavior or a certain outcome.

**0:01:00.8 Vael:** Great. Cool. I missed maybe the second sentence there. I assume that was like large NLP systems or foundations or something?

**0:01:08.1 Interviewee:** Yes. So I work on deep learning, so I\'m interested in the neural networks as they are in practice.

**0:01:12.7 Vael:** Got it. Cool. All right. And then my next question is, what are you most excited about in AI? And what are you most worried about? In other words, what are the biggest benefits or risks of AI?

**0:01:23.4 Interviewee:** So the thing that I\'m most excited about is that this can do extraordinary things that we don\'t know how to program, but we know we can do. Let me try to elaborate on that a little bit. What I mean by that is there\'s a world of things that we don\'t know whether anything can accomplish, tasks that are sufficiently complicated or tasks where given the input data, you may not know whether you can predict the output data. But neural networks are closing the gap between things that we know are possible to do because we do them as humans all the time, but we don\'t know how to program, or there\'s no kind of discreet handwritten program that we can write easily that will describe this. And so I\'m really excited about the fact that we can take data and actually do something that starts to resemble things that require human complexity in order to do that. So to me, that\'s the exciting part. Self-diving cars are one example, though we\'re not that good at that, but even just a mere handwritten digit recognition like the MNIST task, which is the most basic machine learning task in the world, now that I\'m in the field, but when I first took a machine learning class and saw that task, my mind was blown that you could just, with this tiny little data set, learn how to do handwritten digit recognition. That\'s pretty profound, as far as I\'m concerned, and I\'ve never quite lost my fascination for that.

**0:02:35.9 Interviewee:** On the risk side, I\'m concerned about a few things. I\'m not an AGI believer or an AGI concerned person, so you can cross that one off. I think \[inaudible\] huge risks, I am always concerned about\... \[inaudible - very long\]. \[Vael saying what was missed\] \...Let me start over. It was me trying to search for the right words so you missed absolutely nothing.

**0:03:47.2 Interviewee:** Now that I found the right words though, the two things that scare me most, number one, the fact that the systems fail in ways that are unintuitive to humans or that we wouldn\'t be able to reason about intuitively. So the example I give is, you think an automobile driver is likely to drive more poorly at night or when they\'re tired or when they\'re impaired or what have you. It\'s really hard to have an intuition about when the Tesla is going to think of the bus as a cloud, and then slam into the back of it. And it\'s those rare failures when we use a system a lot of times that lead to really bad outcomes and lead to mistrust. At the end of the day, AI needs to be trustworthy and we need to, as humans, trust it, otherwise, it will never be useful, and we\'ll never accept it or terrible things will happen, and so I think a lot about the notion of what does it mean for it to be trusted; where are the gaps between where we are right now and what it would take to be trustworthy.

**0:04:41.3 Interviewee:** The other big concern I have is that I think we don\'t take the problem of understanding how these systems work seriously. It used to be that when we design a big complicated piece of software, think like Windows or PowerPoint or a browser or something like that, we assumed it would take 1000 engineers to build it, and then maybe let\'s say 50 or 100 to test it. Deep learning, we have this really incredible situation where \[inaudible - long\] \... the way that I would put it, the system kind of \[?verb\] on its own based on the data. And we\'re lazy in computer science, we like to think that, \"Oh, the system came to life on its own, so we should have this one hammer that we can bang against it and see whether it\'s good or not, or whether it\'s fair or not, or use the explainability bot to understand how to explain the system.

**0:05:32.1 Interviewee:** But in reality, I think the ratios are just going to be reversed. The software, if you\'re using\... 100 researchers may be able to build a very exciting piece of software. You\'re going to need 1000 or 2000 engineers to tear it apart down to the studs and try to understand how each individual piece of one specific system works. So I think the entire literature around explainability and understanding of deep learning is completely wrong-headed because people are looking for general solutions to a problem that doesn\'t have general solutions. It\'s like writing a program that will debug any program for you. There\'s just no such thing. It\'s not how this game works. So that\'s scares me a lot, that we\'re thinking about this completely the wrong way. And we do need some degree of explainability, we need some degree of understanding of how these systems work, but we\'re not going to get there from the way that we\'re currently thinking about this, and it\'s just going to take a lot more effort and resources than we\'re currently remotely considering giving to it.

**0:06:24.5 Vael:** Interesting. And it sounded like\... So how many engineers per researcher do we need, or per model do we need, do you think?

**0:06:33.8 Interviewee:** Well, this is going to be complicated. The 1000 and 100 are just kind of\... Every piece of software needs a different number of engineers, and three engineers can actually build a pretty big piece of software, but the amount of\... I\'ll go out there on a limb and say it\'s going to be a 10 to 1 ratio.

**0:06:53.8 Interviewee:** That\'s for every 10\... Let\'s call it 10 testers. Whatever we call them, whether it\'s engineers or researchers or what have you, let\'s say that 10 people need to try to understand the system or 100 people need to understand the system. For every \[inaudible\] building the system. \...Feel free to tell me I\'m wrong if \[inaudible\].

**0:07:16.5 Vael:** No, I\'m just trying to actually hear you, so it\'s 10 to\... Or 10 testers or 100 testers to one capabilities person making it happen. Wow. \...Are you still here? (Interviewee: \"Yep, still here.\")

**0:07:34.0 Vael:** Great, alright, so that\'s pretty extreme ratio. Do you think we\'ll get there? And what happens if we don\'t get there?

**0:07:42.3 Interviewee:** I think we\'ll have to get there. At the end of the day, we evaluate these systems by their capabilities. If you have an autopilot system and it crashes the plane a lot, at the end of the day, you\'re not going to use it and it\'s not going to be allowed to fly. In order to get the kinds of capabilities we\'re hoping for, and to get the degree of understanding of failures that we typically expect in any high assurance system, we\'re going to need to have\... If your self-driving car crashes enough times, someone will require you to tear that thing down to the studs to understand that, and you\'re going to have to do this regardless of whether you decide to do this proactively, or eventually your system won\'t be allowed on the road until you do. So I think there\'s going to be a huge amount of manual labor in understanding how these systems work, and I think that either through failures that lead to bans of certain uses of technology until it\'s better understood or until it\'s more resilient, or someone who\'s being proactive and actually wants to ensure their self-driving car works. Either way, you\'re going to end on this situation. No better way to put it.

**0:08:48.5 Vael:** Okay, that\'s super fascinating. Okay, so you think that we are going to need\... The way we\'re doing interpretability is not good right now, it needs to be specific to each system, and currently, we\'re probably doing much\... Yeah, we\'re not even going in the right direction. I kind of expect that self-driving cars will be deployed without this huge amount of intervention on it, but you say that for most systems or something, we will just continue to have failures until it becomes obvious from society that we need people to do interpretability and explainability type of work in this correct way.

**0:09:22.0 Interviewee:** I think so, I think that\'s a good summary. The only thing I\'ll add to this, and I spend a big chunk of my life doing AI policy, so I have to think about this stuff a lot. Is that the amount of work we put into the assurance is proportional to the risk that the system has if it fails. So my favorite example of this is\-- I\'ve done a lot of work in the past on facial recognition policy. Obviously a hot topic, facial bias issues, etc. So let me give you two examples. Google Photos has this clustering algorithm where it will basically find all the faces of grandma and cluster them so that you click on one picture of grandma, and you can find them all if you want to.

**0:09:58.4 Interviewee:** Suppose there\'s racial bias in that algorithm, and suppose that instead the police department is using a facial recognition system to try to identify the perpetrator of a crime via a driver\'s license database. These are two applications that might both have the same technology underneath, they might both have the same biases. We\'re going to be much more worried about one than the other because one could lead people to lose their freedom, and the other may lead to people rightly feeling offended and people rightly feeling hurt, but not someone being thrown in jail potentially. So we have to handle the consequences of the systems in line with their risks. I\'m a lot less concerned about interpretability of Google\'s facial recognition system than I am about a police\'s facial recognition system.

**0:10:47.8 Vael:** I see. And since they need to be treated differently anyway, according to your view\... Yeah, just do it\-- do the interpretability in accordance with their importance. That makes sense.

**0:10:56.7 Interviewee:** Exactly.

**0:10:57.7 Vael:** Yeah. Cool. Alright, so my next question is about future AI. So putting on a science fiction forecasting hat, say we\'re 50 plus years into the future. So at least 50 years in the future, what does that future look like?

**0:11:11.4 Interviewee:** I think it\'s honestly not going to look that different from the present, just a little bit more extreme. I don\'t think we\'ll be in an AGI world. I don\'t think we will have systems that rival human intelligence. I think we will be able to specify what we want out of systems in much higher level terms and actually get them to do this\-- \[I\'m\] thinking, like, a Roomba that is actually a useful intelligent Roomba. A Roomba that knows when to vacuum, knows when not to disturb you, that you can say, \"Hey, can you please do an extra touch-up on that area,\" and it\'ll go do it. I think the current wave of machine learning is great at pattern recognition; I don\'t think pattern recognition gets us to AGI. I think the technology is going to have to look fundamentally different, and I don\'t know that we\'re any closer now than we were 50 years ago in that respect, beyond that we know there are a lot of dead ends.

**0:12:05.0 Interviewee:** That\'s not to say what we\'re doing right now isn\'t exceedingly useful. And \[that\] we\'re not going to push this to the nth degree, and \[that\] we won\'t have the New York Times of the future where you click on a headline and it will generate an article for you based on your interests, your knowledge level, your background on the topic and the amount of time you have to read it; I think that\'s an application that might be 10 or 15 years in the future, if not sooner than that. But I don\'t think we\'ll be in a place where we\'re worried about giving robots rights or things like that. I don\'t think that a huge amount of pattern recognition will get us to cognition, and I don\'t think the current track we\'re on will get us there. That doesn\'t bother me personally because I don\'t really care about that. I\'m much more interested in what we can do for people now and in the near future, and not how we create intelligence. That\'s more of a scary question for me than an exciting question for me, but it\'s\... \[inaudible - short\]

**0:13:00.2 Interviewee:** \...thinks that, heading to an AGI world, even if\... And we may be in a world where we do have self-driving cars that actually work, even if it may not be here for 10 or 15 years, \'cause it\'s a hard problem. We may have actual smart digital assistants, but that\'s \[inaudible - short\] intelligent beings who are peers or \[hard to parse\] in that respect.

**0:13:25.1 Vael:** Interesting. I don\'t know if it will help for me to turn my video off, but I\'m going to try it to hope it gets a little bit less choppy. Great. Okay. So you\'re talking about within the frame of AGI, like lots of people presumably talk about AGI around you. What are people\'s opinions? What do you think of their opinions? Etcetera.

**0:13:46.5 Interviewee:** Oh, I think there are a lot of nut cases. I mean, there are also a lot of optimists and a lot of people who have read a lot of science fiction and want to bring that to life, which is great. And a lot of people who have spent too much time in San Francisco and are surrounded by peer groups who\... Basically, there\'s a lot of monoculture in San Francisco. And I say this having just come back to San Francisco, and as someone who refuses to move \[there\] despite \[\...\]. I have friends who, if I recall correctly, go and worship at the Church of the AGI or something like that, from what I understand, under the belief that eventually there will be artificial intelligences that are smarter and more powerful than us, and therefore they should get ahead of the game and start worshipping them now, since they\'ll be our overlords later.

**0:14:32.4 Interviewee:** I\'m a little more concerned with the here and now, and I think that the technological leaps between great pattern recognizes that we have today and that are way far away. Just because you\'re a nutcase doesn\'t mean you can\'t change the world. And there are plenty of examples of that, and it\'s really good to have that point of view constantly echoed in the community. But I don\'t worry about the end of the world because of AGI in the way that I think some of my friends do. I don\'t consider that to be the biggest existential risk that we have, far from it. And I don\'t think it\'s a risk that we should really be spending any time worrying about at the moment.

**0:15:07.6 Vael:** Got it. Is that because, like you said, there\'s other existential risks to prioritize, or you don\'t think this could be an existential risk?

**0:15:15.1 Interviewee:** I don\'t think this is\... In the very, very long tail of ultra low probability risks to civilization, this is so far out in the tail that it\'s not worth spending any time on, independent of the fact that there are also much greater risks.

**0:15:29.6 Interviewee:** It\'s not just a matter of priority. It\'s also a matter of\... it\'s not a good use of any resources.

**0:15:34.8 Vael:** Got it. That makes sense, yeah. Unfortunately, a lot of my questions are about AGI so you\... \[chuckle\] Here we go.

**0:15:42.1 Interviewee:** No, I\'m happy to give you a strong contrasting opinion to many that I\'m sure you\'ve heard. So come at it.

**0:15:48.1 Vael:** Lovely, lovely. Okay, cool. So how I\'m defining AGI here is like any sort of very general system that could, for example, replace all human jobs, current-day human jobs, whether or not we choose to or don\'t choose to do that. And the frame I usually take is like, 2012, deep learning revolution, here we are. We\'ve only been doing AI for like 70 years or something, and here we are 10 years later, and we have got systems like GPT-3, which have some weirdly general capabilities. But regardless of how you get there, because I can imagine that we hit some ceiling on the current deep learning revolution and we need to have paradigm shifts\-- my impression is generally that if we keep on pouring in the amount of human talent and have software\-- algorithmic improvements at the rate we\'ve seen, hardware improvements, etcetera, and just like, the driving human desire to continue to follow economic incentives to earn money and replace things and make life more convenient, which I think is what a lot of what ML is aimed at right now, that eventually we will get some sort of AGI system. I don\'t really know when. Do you think we will at some point get some very general system?

**0:16:55.8 Interviewee:** I think if you take the time to infinity and humanity lasts in time going to infinity, yes, we will. I do think we have one example of a truly general intelligence system, namely human beings. And eventually we will probably get to the point where we could replicate that intelligence manually, if we have to literally photocopy somebody\'s brain and the transistors at some point, when we get technology advanced enough for that. Do I think that will happen this century? No. Do I think it will happen next century? Probably not. Do I think it might happen in the ones after that? Maybe.

**0:17:28.8 Interviewee:** So the answer is yes, in the limit, but no in any kind of limit that you or I would think about or be able to conceptualize.

**0:17:35.7 Vael:** Yeah, that makes sense. So you think there\'s probably going to be a bunch more paradigm shifts needed before we get there?

**0:17:42.8 Interviewee:** I think at least one\-- it\'s hard to know how many paradigm shifts because it\'s hard to know what they are, but I do not think this paradigm is the right one. I think this paradigm is amazing, and I\'m really excited about the kinds of machines we can build. But you can be excited about the kinds of machines we can build and recognize the limits of those machines. In the same way that\-- how are we going to get to AGI in 10 years if we\'ve been pouring\... You were talking about economic incentives. How much investment do you think has gone into self driving cars over the past, let\'s say, 10 years?

**0:18:14.2 Interviewee:** Let\'s call it \$100 billion plus or minus. That\'s probably the largest single investment in AI technology for an application anywhere. Ever, very likely. And where are we today on self driving cars? They\'re 90% of the way there, \[inaudible - short\] 10 times as difficult as the first 90%. And I don\'t think you\'ll be seeing fully autonomous self-driving cars in the general case on general roads in the next 10 or 15 years. So the thought that there would be an AGI at that point, let alone in 50 years is completely nonsensical to me, personally.

**0:18:56.0 Vael:** Yeah, that makes sense. Especially since self-driving cars are, like, robotics and robotics is behind as well. But even GPT and stuff doesn\'t really have good grounding with anything that\'s happening in the world and how\--

**0:19:09.2 Interviewee:** GPT\'s capabilities are also wildly overstated. You can pull out a lot of good examples out of GPT if you really want to, and you can pull out a lot of crappy ones. But we\'re not going to just brute force large language models to get our way to general intelligence. That\'s BS you\'ll only hear from someone who works at OpenAI who wants their equity to be worth more, quite frankly. The only people who say this are the ones who have an economic incentive to say this, and the people who follow the hype. Otherwise, I don\'t really know of anyone who thinks GPT is the road to AGI, especially given that we can\'t scale up any bigger. I mean, this is something, this is my whole push right now, is that the only way that Nvidia is going to come out with new GPUs next week, and they are going to come out with new GPUs next week that will be twice as fast as the ones that came out two years ago, is if they doubled the amount of power. It\'s not like we\'re doubling the amount of hardware we have available.

**0:20:02.4 Vael:** Got it. Do you know how good optical or quantum computing will be? I know that those are in the pipeline.

**0:20:08.9 Interviewee:** They\'re in the pipeline. Quantum\'s been in the pipeline for a long time, and we\'re up to what, four qubits? Cool. Again, this is one of those cases where we\'ve been awful close to nuclear fusion for a long time. We\'re going to have nuclear fusion in the limit. I\'m a 100% certain of that. When we have that, call me.

**0:20:32.7 Interviewee:** It could be next year. I mean, we\'re very close to crossing that threshold, but we\'ve been really close to crossing that threshold for several decades. And so, trying to call the year that it\'s going to happen within a five, 10 or 50-year time horizon, it\'s really, really tough when it comes to these technologies. And I\'m in that same place about quantum. We are making progress on quantum and I\'m really excited about it. I\'m a little bit scared of it, but I\'m excited about it, but that doesn\'t mean\... Like, crossing that threshold is really difficult. The difference between 10 years and 50 years and a 100 years away may be very small improvements in technology, but it may take an exceedingly long time for us to accomplish. Optical, same thing. So I\'m not sitting here expecting that breakthroughs are just going to happen left and right. These breakthroughs often take a very long time; a lot of incremental advances and all sorts of other technical advances in other fields to make them happen. Material science especially in the case of quantum. And we may get there in five years, or we may get there in 50 or 100 years or longer, and it\'s hard to say.

**0:21:36.6 Vael:** That makes sense. So what would convince you that you should start working on AI alignment? It sounds like there\'s probably going to be some breakthrough that would make you think that it\'s important, but we\'re not necessarily anywhere near that breakthrough right now. Do you have an idea of what that might be?

**0:21:56.9 Interviewee:** Give me your personal definition of AI alignment.

**0:22:00.0 Vael:** Yeah. Well, actually I want your definition first. \[chuckle\]

**0:22:03.3 Interviewee:** So this is not a field that I follow that closely. The entire concept of alignment has really come to the fore in the past three or four months while I\'ve been trying to \[job-related task\]. So I haven\'t been paying as much attention, so I\'d actually appreciate your definition. I can tell you the kinds of people who I see talking about alignment and the kinds of papers that I\'ve seen, the paper titles that I\'ve seen go across my desk. But I couldn\'t give you a good definition even if \[inaudible - short\].

**0:22:33.9 Vael:** Yeah. So one of the definitions I use, and I\'ll give you a problem setting I usually think about as well\-- so, \[the\] definition is building models that represent and safely optimize \[inaudible\] specify human values. Alternatively, ensuring that AI behavior aligns with system designer intentions. And one of the examples I use for what an alignment problem would be is the idea that highly intelligent systems would fail to optimize exactly what their designers intended them to and instead do what we tell them to do. So the example of OpenAI, trying to\... Have that boat win a race and then it getting caught on like some little\-- but optimizing instead for a number of points and instead ending up in this little\-- side, collecting-points area, instead of winning the race. So doing what the designers told it to do instead of what it intended them to.

**0:23:26.2 Interviewee:** So, I mean, this sounds like a\... If you want my honest frank opinion.

**0:23:34.8 Vael:** Yeah.

**0:23:35.3 Interviewee:** A BS-ey rebranding of a simple fact of life in computing for\... Since the dawn of computing, the computer does what you tell it to do, not what you want it to do. (Vael: \"Yes.\")

**0:23:47.5 Interviewee:** And so, I don\'t\... I know a lot of people at OpenAI think they\'re very deep and profound for calling it AI alignment. \[inaudible - very long\].

**0:25:02.7 Interviewee:** So picking up where I was, there are probably a lot of folks at OpenAI, and I know exactly who they are, who think they\'re very deep and profound who are wondering about this question, but this is kind of the obvious fundamental thing that every first year programmer learns and is a question that everybody has been asking in every context whenever they develop a loss function for a neural network. Your loss function never reflects exactly the value that you want the system to carry exactly, even just the outcome. We optimize language models to be good at predicting the next word and yet they somehow also generate\... We want them to have properties that go above and beyond that, we want them to be able to even just transfer the representations for downstream tasks.

**0:25:44.0 Interviewee:** This is just a question of to what extent is the thing you\'re optimizing a system for going to actually align with the task that you want. I just, I don\'t see what\'s interesting or new about this question from a research perspective. I think of course it\'s an important one, but it\'s not a\... \[inaudible - long\] Saying, oh, we should worry about computer security. Well, yes, but there\'s no security robot that fixes all computer security. It\'s a complex context-specific problem. Again, people say things that are aligned with their economic incentives and that is certainly true for my friends at OpenAI but I don\'t see any profundity in this observation that\'s just the nature of computing.

**0:26:38.1 Vael:** Yeah. I think it has to be paired with the idea that if you have a very intelligent system that can plan ahead, that can model itself in the world, that it may have an incentive to preserve itself just as an agent pursuing any goal, because it doesn\'t want to decrease its chance of it succeeding at its goal. I think that has to be paired. Otherwise, it\'s not particularly special. But I do think creating an agent that in the far future, whenever AGI develops, that has an incentive to not be modified makes it much more dangerous than anything we\'ve seen previously.

**0:27:16.2 Interviewee:** I agree with that factor, when we get to a point where this becomes a problem. But I think you can see that the amount I care about this question is proportional to the amount that I think that AGI is or will be a concern in your lifetime, or your grandchildren\'s lifetime. And so I think there are a lot more fundamental basic questions. Like, we don\'t even understand why a neural network actually is able to recognize handwritten digits in the first place. And until we get these basic things down, we\'re never going to get to build the kind of systems that have these properties anyway.

**0:27:44.7 Interviewee:** So, I would make the analogy of, let me see, I\'m trying to think of the right metaphor\... I don\'t know, for the folks who worry about how we\'re going to communicate with the alien civilizations we inevitably come into contact with\-- we better build some space ships that can get us to space first before we start worrying. And then figure out whether there are alien civilizations out there before we start worrying about how we\'re going to communicate with them.

**0:28:05.6 Vael:** Yeah, this sounds like a very coherent worldview, I\'m like, yep, makes sense, is logical.

**0:28:11.7 Interviewee:** Yeah, I have strong opinions, and researchers are always incentivized to have strong opinions; it\'s what moves science forward. We argue and we disagree with each other, and we\'re all right, and we\'re all wrong. But these are my strong opinions. And if you were to chat with any of my friends at OpenAI, you\'d hear the opposite view and we could still go out for drinks and have a good time.

**0:28:32.5 Vael:** Great. So I think this line of questioning was originally developed by me asking what would you see\... What would you want to see in the world before you\'re like, \"Oh actually, people are right, this thing is coming earlier than I expected.\"

**0:28:47.6 Interviewee:** That\'s a good question. Honestly, I would want to see a\... I\'m thinking of examples of a task, so like a Turing test of some sort that would fit here. Obviously, the Turing Test is not a very effective test given that GPT-3 probably passes it with flying colors, despite the fact that GPT-3 is a very good language model. What I would look for is long-lived machine learning systems that can learn continually. We don\'t even have learning continually down. We have reinforcement learning systems that eventually become effective agents at solving one particular task; they can\'t move on to a second task, we don\'t have any general kind of learning process. Even the reinforcement learning agents that we have have to be really, really, specially hand-calibrated for a specific task, to the point where even the specific random seed you use is an important hyperparameter in determining whether the agent will succeed or fail at a given task.

**0:29:50.0 Interviewee:** If we\'re in that world, we\'re very far away from an agent that can learn many tasks or has a generalized learning process. This is I guess why people are excited about meta-learning in general. Again, I think we shouldn\'t worry about light speed travelling until we can get off the planet, but that\'s a whole different\... You know where I stand on that. So\... \[inaudible - short\] and improve themselves over time; reinforcement learning is kind of a shadow of what we would expect of a real system. A smart home assistant, where you can teach it new tasks by explaining it new tasks, and it will be able to figure out how to accomplish them. We don\'t have any kind of abstract or general learning capabilities right now. And we talk a very big game about things like AlphaFold or what have you, or any of the Alpha stuff coming out of DeepMind. But these are requiring lots and lots and lots of engineers to get a system to work on one very specific setting in ways that don\'t transfer to any other setting or any other task. There are principles we\... \[inaudible short\] being really hard trying a bunch of stuff. And that\'s a\... \[inaudible long\] real general learning process that will keep me up at night.

**0:31:03.5 Vael:** Got it. Okay. Interesting. So do you think you and your colleagues probably have the same information but are just reacting to it differently? Other people are also not seeing these very general systems or maybe\-- yeah, presumably, because you have access to the same information, but you\'re just interpreting it differently based on the incentives that your company or whatever is in?

**0:31:29.9 Interviewee:** I wouldn\'t always blame the incentives and the all, again, throwing some shade at San Francisco in particular. When you spend all of your waking hours and all of your sleeping hours around exclusively people who are working on the same \[inaudible\], same field. And when you work as they do not only in the tech industry, but especially in the very tight knit AI community, where if we all go to the same birthday parties when I\'m out there, and everybody goes to the same spas, and runs into each other on the street, despite the fact that San Francisco is a big city. When you live in that kind of monoculture, quite frankly, you\'ll lose touch with reality, to some extent. And I think, a lot of people start believing that\... \[inaudible very long\]

**0:32:31.9 Vael:** Okay, cool. My question was, although I missed that last bit, but my question was, how do people stay in touch with reality, how do they\...

**0:32:43.7 Interviewee:** I think you have to talk to people who are doing things other than building AI systems. You have to actually maybe interact with someone who works in healthcare or someone who works in finance, or someone who works in any other field besides AI or besides machine learning. And you have to remember that they are real human beings out there. I think this is just a general San Francisco problem. You see people, you see 20-somethings who are wealthy, producing systems that are convenient for them. We see this all the time, startups coming out of the Bay Area.

**0:33:24.1 Interviewee:** My experience in Boston and New York is it\'s a very different start up culture of healthtech, fintech, edtech, what have you, things that are quite frankly real and useful. So when I hang out with\... The times I\'ve been working at \[large tech company\] in California, for example, all of my friends are worried that the sky is falling for some reason. Maybe we\'re about to enter an AI winter or we\'re about to create AGI, it\'s usually one or the other. Depending on who you talk to and what mood they\'re in, and what\'s going on in their organization at that different point in time. But people are just so caught up in the inside baseball, they\'ve lost the bigger context for everything we\'ve accomplished, but also how far away we are from some of the things people talk about. And you gotta actually taste the real world from time to time, and remember that you don\'t just live in this little bubble of lots of neural networks and lots of people with PhDs working on neural networks.

**0:34:19.7 Vael:** And interacting with other people and seeing the wider world gives the context to believe more\... mainline views? Because it sounds like people end up in either direction, you said they believe that everything is about AI, everything\'s going to go badly: everything is going to go badly and too fast or everything\'s going too badly and too slow. And there\'s some regularity effect that happens if you\'re hanging out with other people?

**0:34:50.4 Interviewee:** It\'s almost the opposite of\... \[inaudible - short\] in that respect. I think it\'s partially because people will say things like, \"Well, there hasn\'t been a big advance or there hasn\'t been a big breakthrough in 18 months.\" And my reaction is, it\'s been a\... Yes, the transformer paper came out 18 months ago. Yeah, let\'s be patient, and the field does not always continually accelerate. No scientific field continually accelerates. We have periods of big progress and then periods of stagnation and paradigm shift. This is the structure of scientific revolution. If you want to\... For the extent to which you take that view of the world seriously, that does, in my impression, simply how science moves, in fits and starts, not at a steady pace. And in computers we\'re used to exponentials. That doesn\'t mean that these systems are going to get exponentially better over time. It may mean we had 10 really good years of progress based on a bunch of different factors all coming together, between big data being accessible, lots of computing accessible, and some improvements in deep learning, that all came together really nicely to give us a big burst of progress.

**0:36:00.1 Interviewee:** Maybe things will slow down a little bit. Or maybe there will be a new architecture in five years that will be another big burst, the way that convolutional neural networks were around the AlexNet breakthrough and then the way that transformers were. But this happens in fits and starts. But around \[large tech company\], partially because of \[large tech company\]\'s promotion process and partially because of the very internally competitive atmosphere that that fosters, people are always dour and feel like the sky is falling. And at OpenAI, the incentive is to continue the hype and build really big systems. It\'s partially in the culture, partially in the leadership, partially in the brand of the place and what it takes to make your stock go up in value. So I think people really are shaped by their environments in that respect, and you gotta get out and talk to other kinds of people and get other perspectives, and I don\'t think there\'s that much you can do when you\'re in a big organization like \[large tech company\], and there\'s not a lot you can do when you\'re in San Francisco and everybody you talk to. It\'s not \"what you do\". It\'s \"which tech company do you work for\" or more likely \"which AI project do you work on\".

**0:37:04.4 Vael:** I see, so this is probably, like you said, why you haven\'t moved and also presumably why you have friends who are not AI researchers.

**0:37:12.4 Interviewee:** Yeah, my office right now is at \[university\] \[non-science field\]. I just sit by a bunch of \[non-science field\] professors. I\'ve worked there at times in the past. I live in \[city\] right now, and they were kind enough to offer me a place to sit and work for a little bit. So I\'m hanging out with a bunch of \[non-science field\] faculty all day. It\'s a very different perspective on the world. They\'re worried about very mundane things compared to AGI coming and killing us all, or a quantum computing breakthrough that leads to the end of civilization when we train quantum neural networks or something like that. They\'re much more worried about bail reform, or very basic mundane day-to-day problems that are actually affecting people around them. It\'s hard to get caught up in the hype.

**0:37:56.5 Vael:** Yeah, and it seems like it\'s important to make sure that people are working on things that actually affect more than just their local environment. I think you mentioned something like that earlier, right?

**0:38:05.9 Interviewee:** Exactly. And there\'s also a little bit of\... as computer scientists, we love to calculate things, we love to measure things. Effective Altruism is one great example of this. We love to say, \"What is the way that I can put my dollar the furthest along?\" And I think sometimes people lose touch with the fact that there are humans out there. There are people right now, you ask questions like, well, either I can help the humans today or I can put all my resources toward saving the entirety of civilization. And civilization and the future is obviously\-- there are many more humans there than there are here today, so I\'ll worry about, on the 0.1% chance that we develop AGI and it comes to threaten us all, I\'ll worry about that problem, because in expectation, that will save more lives than what have you. And I think\--

**0:38:51.5 Vael:** \--Yeah, and this is the wrong way to think about things?

**0:38:56.5 Interviewee:** I don\'t think this is necessarily the wrong way to think about things, but I think it\'s a little bit\... Sometimes, it\'s more important to focus on what\'s right in front of you and what\'s tangible and what\'s here and now. Or the other way I\'ll put it is, MIT is pathologically bad as an institution at dealing with topics that can\'t be measured for exactly this reason. It\'s really hard to talk to people at MIT about values, unless a value can be measured or optimized.

**0:39:25.0 Interviewee:** And this is just, for a lot of folks who get empirical education and who are training systems all day, for whom this is a problem, they are just as much\... The people are just as much subject to this problem as the systems they build. So, if we want to talk about alignment, it\'s an issue not just of the systems, but of the people building the systems.

**0:39:43.5 Vael:** Alright, well, we are at time. Thank you so much, this is a very different opinion than the ones I\'ve received so far, and yeah, very well presented. Awesome.

**0:39:52.6 Interviewee:** Thank you, that\'s what I\'m here for. You know where to find me if you need to chat more. I\'m very exited to see what the outcome of the project is.

**0:39:58.4 Vael:** Great. Alright. Well, thank you so much.

individuallyselected_zlzai-by Vael Gates-date 20220318.md


# Jaime Sevilla on Projecting AI progress from compute trends by Jeremie Harris on the Towards Data Science Podcast

## Jaime Sevilla on timelines for transformative AI and general intelligence

There’s an idea in machine learning that most of the progress we see in AI doesn’t come from new algorithms of model architectures. instead, some argue, progress almost entirely comes from scaling up compute power, datasets and model sizes — and besides those three ingredients, nothing else really matters.

Through that lens the history of AI becomes the history f processing power and compute budgets. And if that turns out to be true, then we might be able to do a decent job of predicting AI progress by studying trends in compute power and their impact on AI development.

And that’s why I wanted to talk to Jaime Sevilla, an independent researcher and AI forecaster, and affiliate researcher at Cambridge University’s Centre for the Study of Existential Risk, where he works on technological forecasting and understanding trends in AI in particular. His work’s been cited in a lot of cool places, including Our World In Data, who used his team’s data to put together [an exposé on trends in compute](https://ourworldindata.org/grapher/ai-training-computation). Jaime joined me to talk about compute trends and AI forecasting on this episode of the TDS podcast.

Here were some of my favourite take-homes from the conversation:

-   Jaime’s work involves projecting trends in AI to estimate when we might reach a level of AI capability known as transformative AI (TAI). TAI is roughly the stage at which AI is so powerful and generally capable that it triggers a process analogous to the Industrial Revolution, whose effects on human civilization are profound and entirely unpredictable. Interestingly, it probably doesn’t matter how exactly TAI is defined, because the general consensus among researchers and forecasters is that by the time a stage like TAI is reached, technological progress will be happening so fast that even fairly major differences in capability will be separated by short periods of time. As a result, disagreements over how precisely to define TAI are likely to result in estimates that differ only by a few years, rather than decades or centuries, as was the case with the Industrial Revolution.
-   Jaime’s work identified three distinct eras in the history of AI. In the first, AI progress is driven by academic labs with relatively fixed budgets. Still, thanks to Moore’s Law, compute power was getting twice as cheap every 20 months during this time, so labs tended to double their compute usage every two years. But around 2010, things started to change: deep learning was showing real potential to solve a variety of problems in vision and natural language processing — but deep learning is highly compute-intensive. As a result, companies began throwing an accelerating amount of resources at AI development, and therefore, at compute. Notably, Jaime argues that this actually started to happen slightly before AlexNet (the canonical and widely-celebrated “kick-off” moment for deep learning). This deep learning era left us with impressive systems for tasks as diverse as vision and game-playing, but they still couldn’t quite crack language generation. As GPT-3 dramatically showed, the path to better language models would be a third scaling phase — a trend that Jaime argues began sometime in 2015/2016, half a decade before GPT-3 itself.
-   There’s quite a bit of debate about how compute investments will translate into new AI capabilities, and therefore on when we can expect to reach TAI. One method is to look at biological intelligence, and try to estimate how much computational power has gone into creating human brains. There are several ways to do that: for example, you could imagine estimating the number of computations that a human needs to carry out to reach adulthood from birth, or the compute expended by nature as it evolved human brains from inanimate matter billions of years ago. These are known as “biological anchors” for AI.
-   Jaime identifies two main drivers of compute trends. The first is essentially Moore’s Law: compute power is getting exponentially cheaper, reducing in cost by a factor of two every 20 months as humans come up with new ways to squeeze more processing power or of less matter. The second driver is corporate investment in compute: as companies recognize the value of AI, and the crucial role that processing power plays in its development, they’ve been ploughing more and more resources into compute at an accelerating rate.
-   This second driver is currently the most significant, sure to the violent expansion of corporate AI budgets in recent years, but it’s unclear if those investments can continue to grow at their current rate indefinitely. Jaime estimates that this budget growth phase may fizzle out around the end of the decade, as AI related compute becomes a prohibitively large fraction of corporate budgets. That is, unless AI-related revenues grow fast enough to keep it going, which is also a distinct possibility.

Jaime Sevilla - Projecting AI progress from compute┬átrends-by Towards Data Science-video_id 2NXagVA3yzg-date 20220413.md


# Interview with AI Researchers NeurIPSorICML_q243b by Vael Gates

**Interview with q243b, on 3/18/22**

**0:00:00.0 Vael:** Cool. Alright. So my first question is, can you tell me about what area of AI you work on in a few sentences?

**0:00:09.0 Interviewee:** Of course. I did my PhD in optimization and specifically in non-convex optimization. And after that I switched topics quite a lot. I worked in search at \[company\] \[\...\] And now actually I work in \[hard to parse\] research, so I kind of come back to optimization, but it\'s more like a kind of weird angles of optimizations such as like meta-learning or future learning, kind of more novel trends like that.

**0:00:42.1 Vael:** Cool. And next question is: what are you most excited about in AI and what are you most worried about? In other words, what are the biggest benefits or risks of AI?

**0:00:51.1 Interviewee:** Well, the benefit is that AI has slowly but surely kind of taken over a lot of problems in the world. And in fact pretty much any problem where you need to optimize something, you can use AI for that. I\'m more of a traditional machine learning person, in the sense that\... Currently I include everything, including not necessarily neural networks, also like logistic regressions, decision trees, all those things. And I think those things have been grossly unutilized, I would say, because a lot of problems right now in machine learning that people think, \"Okay, should this solve this neural networks?\" but in reality, just in decision trees. But stay the same because of the current trends and because of the current hype of neural networks as well, they kind of came along with it as well. All the publicity and marketing that, you know, that AI should have, honestly. And I think more and more companies realize that you can solve the problem with just simple solutions. And I think that will be a really exciting part. So just like, to answering to your question yeah, I\'m excited about the fact that machine learning has just become more and more and more ubiquitous. It becomes like almost prerequisites for any big company or even a smaller company. And the second part, what I\'m worried the most. I don\'t know, that\'s a good question.

**0:02:07.7 Interviewee:** I mean, I guess like, I mean, I don\'t share those worries that AI would dominate us and we would all be exterminated by overpowerful AI. I don\'t think AGI is coming anytime soon. I think we\'re still doing statistics in a way, like some kind of belong to this camp will just think, we\'re still doing linear models and I don\'t believe the system is conscious or anything of those sorts. I think the most malicious use, like, I mean, especially now currently with the war, I see more and more people using AI for malicious sense. Like not necessarily they will be, you know, we\'re going to have next SkyNet coming, but in the bad hands, in the bad actors, you know, AI can serve not a good purpose in war. Like for example, you know, like now with drones, with, you know, the current war, for example, in Ukraine is more and more done with drones. And drones have automatic targeting, automatic navigation. And yeah, so that\'s kind of not necessarily a good thing and they can become more and more dramatic, and more automatized and they can lead to harm.

**0:03:06.4 Vael:** Yeah, it makes sense. Lot of strong misuse cases. So focusing on future AI, putting on like a science fiction, forecasting hat, say we\'re 50 plus years into the future, so at least 50 years in the future, what does that future look like?

**0:03:19.5 Interviewee:** Well, I hope\... I mean I kind of, you know, I really like that, my favorite quote in a\... And probably that one of the quotes that I really like is that the arc of justice is nonlinear, but it kind of bends towards justice. I really like it. And I really hope in 50 years we would actually figure out first of all, the way to harness our current problems and make not necessarily disappear, but at least make it controllable such as nuclear weapons and the global warming. And that\'s in 50 years, I think it\'s a reasonable time. Again, not to solve them, but just to figure out how to harness these issues. And AI should definitely help that. And do you want me to answer more specifically, like more, like give you ideas of how I think in 50 years the world would look like? (Vael: \"Yeah, I\'ll take some more specifics.\")

**0:04:01.3 Interviewee:** Alright. I mean, I think that one of my exciting areas that I think that right now is already kind of flourishing a little bit, and it\'s large language models. It\'s a current trend, so it\'s kind of an easy, like on the surface thing to talk about, and I think I\... As a \[large tech company\] employee, I can see how they\'ve been developed over like two years, things changing dramatically. And I think these kind of things are pretty exciting. Like having a system that can talk to you, understand you, respond not necessarily with just one phrase, but like accomplish tasks that you wanted to accomplish. Like now it\'s currently in language scenarios, but I also, within 50 years definitely anticipate it could happen in like robotics, like a personal assistant next to you or something like that. Another area I\'m really excited about is medicine.

**0:04:46.1 Interviewee:** I think once we figure out all the privacy issues that surround medicine right now, and we\'re able to create like, clean up database, so to speak, of patients diagnoses. And I hope that it\'ll be enough for a machine learning model to solve like cancer as we know it and things like that. I\'m just hopeful. I mean, I hope it\'s going to happen in 50 years and it\'s going to, I don\'t know if I want to place my bet there, but I\'m hoping that would happen. So I guess in robotics, as well, as I said, one of the things that we\'re kind of inching there, but not quite there, but I think in 50 years, we\'ll solve it. So I think these three things: personal assistant, solving medicine and robotics, these three things.

**0:05:28.2 Vael:** Wow. Yeah. I mean, solving robotics would be huge. Like what\'s an example of\... could you do anything that a human could do as a robot or like less capable than that?

**0:05:36.0 Interviewee:** I think so. I mean, it depends. It depends what you mean by human, right? I mean the\... Well, if you try to drive a car for the last 20 years. We\'ve been trying to do that, but honestly, I think this problem is really, really hard because you have to interact with other agents as well. That\'s kind the main thing, right? You have to interact with other humans mostly. I mean, I think interaction between robots, it\'s one thing, interaction between robots and robots is much easier. So I think whatever task that doesn\'t involve humans is actually going to be pretty useful. Well, again, actually pretty easy because it hasn\'t been solved yet, but I think it\'s much easier than solving with humans.

**0:06:05.8 Interviewee:** And like for example organizing your kitchen, organizing your room, cleaning your room, cooking for you, I think all the things should be pretty straightforward. Again, the main issue here is that every kitchen is different, so although we can train a robot to a particular kitchen or like do some particular kitchen, like once it\'s presented with a novel kitchen with novel mess\... mess is very personal. So it\'d be harder for the robots to do. But I think that\'s something that would be kinda within the reach, I think.

**0:06:37.7 Vael:** Interesting, and for like, solving cancer, for example, I imagine that\'s going to involve a fair amount of research per se, so do we have AIs doing research?

**0:06:47.2 Interviewee:** So research, I want to distinguish research here because there is research in machine learning and there is research in medicine. And they are two different things. The research in medicine, and I\'m not a doctor at all, but from what I understand, it\'s very different, in the sense that you research particular forms of cancer, very empirical research. Like hey we have\... Cancer from what I understand, one of the main issues with cancer is that every cancer is more or less unique.

**0:07:11.8 Interviewee:** It\'s really hard to categorize it, it is really hard to do AB testing. The main research tool that medical professionals use is AB testing, right, you have this particular group of cancers, group of people, that suffer from this particular cancer. Okay, let\'s just come up with a drug that you can try to put these people on trial, and do that. But because every cancer is unique, it\'s pretty hard to do that. So, and how to do this research is data, and that\'s what we need for machine learning, we need to have sufficient data such that machine learning can leverage that and utilize it. So they\'re now asking questions in two perspectives, one is do we need more data? Yes, absolutely.

**0:07:46.3 Interviewee:** Moreover we not only need data, we need to have rules for which these machine learning agents, that of a company, university, would have access to this data differentially private right in the sense that this should be available to them. But is possibly private, of course privacy is a big issue. Which right now doesn\'t really happen, plus there are other bureaucratic reasons for this not to happen, like for example hospitals withholding the data because they don\'t want to share it and stuff like that.

**0:08:16.3 Interviewee:** So if we can solve this problem, the research and medical part would be not necessarily\... Not necessary for the machine learning. And on the machine learning side, there is also, as well, it\'s very big hurdles in the sense that current machine learning algorithms needs tons of data. Like for the self-driving cars, they\'re still talking about we need millions and millions of hours cars driving on the road, and they still don\'t have enough. So for cancer, that\'s kind of not be the case hopefully. Right? So hopefully we\'re going to come up with algorithms that work with fewer data. Like one of the algorithms is so called few-shot algorithms, so when you have algorithms that learn on somebody\'s language, but when you want to apply to a particular patient in mind, you just need to use specific markers to adjust your model to the specific patient. So there are some advancement in this way too but I think we are not there yet.

**0:09:07.1 Vael:** Interesting. Cool, alright, so that\'s more\... It\'s not like the AI is doing research itself, it\'s more that it is, like you\'re feeding in the data, to the similar types of algorithms that already exist. Cool, that makes sense. Alright so, I don\'t know if you\'re going to like this, but, people talk about the promise of AI, by which thy mean many things but one of the things is that\... The frame that I\'m using right now is like having a very general capable system, such that they have the cognitive capacities to replace all current day human jobs. So whether or not we choose to replace human jobs is a different question. But I usually think of this as in the frame of like, we have 2012, we\'ve the neural net\... we have AlexNet, the deep learning revolution, 10 years later with GPT-3 which has some weirdly emergent capabilities, so it can do text generation, language translations, some coding, some math.

**0:09:51.9 Vael:** And so one might expect that if we continued pouring all the human effort that has been going into this. And nations competing and companies competing and like a lot of talent going into this and like young people learning all this stuff. Then we have software improvements and hardware improvements, and if we get optical and quantum at the rate we\'ve seen that we might actually reach some sort of like very general system. Alternatively we might hit some ceiling and then we\'d need do a paradigm shift. But my general question is, regardless of how we get there, do you think we will ever get to a very general AI system like a CEO or a scientist AI? And if so, when?

**0:10:23.6 Interviewee:** So, my view on that is that it\'s really hard to extrapolate towards the future. Like my favorite example is I guess Elon Musk\... I heard it first from Elon Musk but it\'s a very known thing. Is that, \"Hey we had like Pong 40 years ago, it was the best game that ever created, which was Pong, it was just like pixels moving and now we have a realistic thing and VR around the corner, so of course in 100 years we will have like a super realistic simulation of everything, right? And of course in a 1,000 years we\'ll have everything, \[hard to parse\] everything.\"

**0:10:53.0 Interviewee:** But again it doesn\'t work this way. Because the research project is not linear, the research progress is not linear. Like 100 years ago Einstein developed this theory of everything, right, of how reality works. And then yet, we hit a very big road block of how exactly it works with respect to the different scale, like micro, macro and micro and we\'re still not there, we propose different theories but it\'s really hard. And I think that science actually works this way pretty much all around the history it\'s been like that, right. You have really fast advancement and then slowing down. And in some way you have advancement in different things.

**0:11:26.8 Interviewee:** Plus the cool thing about research is that sometimes you hit a road block that you can\'t anticipate. Not only there are road blocks that you maybe don\'t even imagine there are, but you don\'t even know what they could be in the future. And that\'s the cool part about science. And honestly, again, I think if we are indeed developing AGI soon, I think it\'s actually a bad sign. Honestly, I think it\'s a bad sign because it means that it\'s\... It\'s like too easy, then I\'ll be really scared: okay what\'s next? Because if we developed some really super powerful algorithm that can essentially super cognitive\... Better and better cognition of humans, I think that will be scary because then I don\'t even know. First of all, my imagination doesn\'t go further than that, because exactly by definition it will be smarter than me, so I don\'t even know how to do that. But also I think it means that my understanding of science is wrong. Another example I like is that someone said, if you\'re looking for aliens in the universe right now and then this person says, if we actually do discover aliens right now, it\'s actually a very bad sign. It\'s a bad sign in the sense that\...

**0:12:29.7 Interviewee:** If they\'re there, it means that the great filter, that whatever concept of great filter, right, that we\'re kind of in front of it, not some behind us, it is in front of us. Just means there\'s some really big disaster coming up, because it actually, if aliens made it as well, this mean that they also pass all the filters behind us, it mean that some bigger filters in front of us. So I kind of belong to that camp. Like I\'m\... I\'m kinda hoping that the science will slow down. And we\'ll not be able to get there. Or there\'s going to be something\... It\'s not that I think that human mind is unique and we can\'t reproduce it. I just think that it\'s not as easy as we think it would be, or like in our lifetime at least.

**0:13:05.0 Vael:** I see. So maybe not in our lifetime. Do you think it\'ll happen in like childrens\' lifetime?

**0:13:10.1 Interviewee:** Which children? Our children hopefully not. But I mean, at some point I think so. But again, I think it\'ll be very different form. Humans\' intellect is very, very unique, I think, and because it\'s shaped by evolution, shaped by specific things, specific rules. So I also kind of believe in this, in the theory that in a way computers already\... They are better than us because they are faster, to start with, and then they can\... Another example I really like is that if you remember the AlphaGo playing Go with Lee Sedol, like one of the best two players of Go. And there was a\...

**0:13:43.0 Interviewee:** If you remember the Netflix show, there was like in one room they actually have all the journalists and they were sitting next to Lee Sedol playing with the person that represents DeepMind. And then all the DeepMind engineers and scientists, they were in a different room. And in that room, when they were watching the game playing, \[in\] that room the computer said by the move number 30, very early in the game, it says, okay, I won. And it took Lee Sedol another like half an hour or more, another like a hundred moves to confirm that statistic. And they were\... The DeepMind guys were celebrating and these guys were like all thinking about the game, how to\... But the game was already lost.

**0:14:16.6 Interviewee:** So computers are already bett\--\... I mean, of course it\'s a very constrained sandbox around the Go game. I think it\'s true for many things, computers are already better than us. We are more general in our sense of generality, I guess. So maybe they will go in different direction\... But the world is really multidimensional and the problems that we solve are very multi-dimensional. So I think it\'s too simplistic to say that, then you\'re universally better than us, or we are clearly subset and they are superset of our cognition. It\'s, I don\'t know\... I think it \[hard to parse\].

**0:14:44.9 Vael:** Great. Yeah. I\'m going to just hark back to the original point, which was when do you think that we\'ll have systems that are able to like be a CEO or a scientist AI?

**0:14:55.0 Interviewee:** Okay. Yeah, sure. Again, sorry\-- sorry for not giving you a simple answer. Maybe that\'s what you\'re looking for, but let me know if this is\... (Vael: \"Nah, it\'s fine,\")

**0:15:06.5 Interviewee:** Yeah. I don\'t know. In a way like\... The work that like accountant does right now, it\'s very different than what accountant did 30 years ago. Did we replace it? No, we didn\'t. We augmented work. We changed the work of accountant so that the work is now simpler. So replacing completely accountant, in a way, yes, we also\... Because the current, the set of tasks that accountant did 30 years ago, it\'s automated already. Do we still need accountants? Yes. So same here. Maybe the job that CEO is doing right now in 30 or 40 years, everything that right now, as of today, CEO is doing in 40, 30 years, we will still\... The computer will do it. Would we still need the human there? Yes. If this answers your question.

**0:15:45.1 Vael:** Will we need the humans? I can imagine that we can have like, eventually AI might be good enough that we could have it do all of our science and then, or it\'s just so much smarter than us then we\'re just like, well, you\'re much faster at doing science. So I\'ll just let you do science.

**0:15:57.8 Interviewee:** So let me rephrase your question a bit. So what you\'re saying right now is it is a black box right now, that right now, that\'s CEO right now, that\'s a CEO job. That\'s what CEO is doing. There\'s some input, then some output. So what you\'re saying that now we can automate it. And now the input and output will also feed through something to computer let\'s say, but then what is, what would be the\... We\'ll have to refine the input and output then, because it still should serve humans. Right?

**0:16:21.9 Interviewee:** So previously you need to have drivers, like for the tram you\'re having to have a driver. Now instead of drivers, you have computers, but you still need to have a person to supervise the system in a way. Or, but then you\'re talking about even that being automated. But in same time, you cannot\... Like the system, like for example, self-driving car, it\'s become a tool for someone else. So you\'re removing the work of a driver, but you replace it with a system that now it\'s called something else. Like Uber. Previously, you had to call a taxi. Now you have an app to do it for you.

**0:16:49.7 Interviewee:** There\'s an algorithm that does it for you. So the system morph into something else. So same thing here. I think as CEOs, in a sense, they might be replaced, but the system also would change as well. So it won\'t be the same. It won\'t be like, okay, we\'ll have a Google. And there is a CEO of Google who is like robot. Now the Google will morph in a way that the task the CEO is doing would be given the computer. But the Google will still\... Like also, by the way, Google, even Google. Google works on its own. In fact, if you, right now, fire all the employees, it\'ll still work a few days. Everything that we do, we do for the future. Like it\'s pretty unique moment in history, right? \'Cause previously, like before the industrial revolution, you had to do things yourself. Then with the factories and factories well then, okay, you\'re helping factory to do its work. And now there is a third wave, whatever, fourth wave, industrial revolution. We don\'t even do anything. It\'s on\... In a way Google doesn\'t have a CEO, the Google CEO doesn\'t work for the today Google. Google CEO works for the Google in a year which is\... So Google work\... Google is already that. Google doesn\'t have a CEO. So that\'s what I mean.

**0:17:56.8 Vael:** Alright. Uh, I\'m going to do the science example, because that feels easier to me, but like, like, okay. So we\'re like doing\... We\'re attempting to have our AI solve cancer and normally humans would do the science. And be like, okay, cool I\'m going to like do this experiment, and that experiment, and that experiment. And then at some point, like we\'ll have an AI assistant and at some point we\'ll just be like, alright, AI solve cancer. And then it will just be able to do that on its own. And it\'s still like serving human interests or something, but it is like kind of automated in its own way. Okay. So do you think that will\... When do you think that will happen?

**0:18:34.0 Interviewee:** The question is how with\... The question is, can this task be, sorry I know you\'re asking about the timeline and I want to be, I know\... I don\'t want to ramble too much. But I think I want to be specific enough, what kind of problem we\'re talking about. If we\'re talking about the engineering problem, that we\'re talking about the timeframe of our lifetime. If you\'re talking about a problem that involves more creativity, like for example, come up with a new vaccine for the new coronavirus? Sorry, it\'s automatic. I think that work we could do in the 20\... 20-30 years. Right, because we have tools, we know engineering tools, what needs to be done, where you can do ABC, you\'re going to get D. Once you have D you need to pass it through some tests and you\'re going to get E and that pretty much automate. I think this we can do in 20-30 years. Solving cancer, I just don\'t know enough. How much creativity needs to be there. So more harder, probably, yeah. Yeah, yeah.

**0:19:21.0 Vael:** Yeah, no, no, that\'s great. And yeah, and you don\'t know when we\'ll\... For\... So probably more than our lifetime, or more than 30 years at least for creating those?

**0:19:28.9 Interviewee:** Mm-hmm. Mm-hmm.

**0:19:30.1 Vael:** Alright, great. Cool. Alright, I\'m moving on to my next set of questions. So imagine we have a CEO AI. This is\... I\'m still going back to the CEO AI even though\... (Interviewee: \"Sure, of course.\")

**0:19:40.8 Vael:** And I\'m like, \"Okay CEO AI, I want you to maximize profits and try not run out of of money or exploit people or\... try to avoid side effects.\" And this currently is very technically challenging for a bunch of reasons, and we couldn\'t do it. But I think one of the reasons why it\'s particularly hard is we don\'t actually know how to put human values into AI systems very well, into the mathematical formulations that we can do. And I think this will continue to be a problem. And maybe, and\... I think it seems like it would get worse in the future as like the AI is optimizing over, kind of more reality, or a larger space. And then we\'re trying to instill what we want into it. And so what do you think of the argument: highly intelligent systems will fail to optimize exactly what their designers intended them to, and this is dangerous.

**0:20:28.3 Interviewee:** Okay, so this is a quite big misconception of the public is that, and I think actually, it\'s rightfully so, because I am actually working on this, is that\... So what you said right now is like the famous paperclip example, right? Which is going to turn the whole world in particular, and that\'s kind of what you said more or less, right? So the problem here is that current system, a lot of them, it\'s true, they work on a very specific one dimensional object. There\'s a loss function that we\'re trying to minimize. And it\'s true, like GPT-3 and all these system, currently they are there, they have only one number, it want to minimize. And this is, if you think about it, is way too simplistic for this very reason, right? Exactly, because if you want to just maximize number of paperclips, you\'re just going to turn the whole word into the paperclip machine factory. And that\'s the problem. But the reality is much more complicated. And in fact, we are moving there, like my research was, we\'re moving away from that. We\'re trying to understand, we\'re trying to understand if first of all intelligence can be emerge on its own without it being minimized explicitly.

**0:21:20.2 Interviewee:** And second of all, these pitfalls, the pitfalls that you\'re just minimizing one number and of course, not going to work. So answering your question, yeah, yeah I think it will fail, because it will face the real world scenario unless it has specific checks and balances. For example, there\'s also online learning paradigm, right, where you basically learn from every example as they come in the time series. I think this system need to be revamped to work in a larger scale obviously, but this is the kind of system that potentially could work, where you don\'t just minimize, maximize one objective, but you have a set of objectives or you have a set of actors with their own goals and their intelligence is emerging from them. Or you learn online. You just fail and you learn and you forget what you learned, you have\... learn in a continuous fashion. So like all of these things that we as a humans do, could be applicable for AI as well. Like we are the humans, we don\'t have\... You don\'t spend your day like, go to sleep, okay, today, day was like 26. You don\'t do that. And even if you do that, you probably will have multiple numbers. This was 26, this was 37. Okay, it doesn\'t matter the day.

**0:22:19.3 Vael:** Yeah, that makes sense. So, one thing I\'m kind of curious about is there\'s\... like, when you say, it won\'t work, just optimizing one number. And will it not work in the sense that, we\'re trying to advance capabilities, we\'re working on applications, Oh, turns out we can\'t actually do this because we can\'t put the values in. And so it\'s just going to fail, and then it will get handled as part of the economic process of progress. Or do you think we\'ll go along pretty well, we\'ll go along fine, things seem fine, things seems like mostly fine. And then like if things just diverge, kind of? Kind of like how we have recommender systems that are kind of addictive even though we didn\'t really mean to, but people weren\'t really trying to make it fulfill human values or something. And then we\'d just have the sort of drift that wouldn\'t get corrected unless we had people explicitly working on safety? Yeah, so, yeah, what do you think?

**0:23:06.4 Interviewee:** I think people who work on safety, and you could see it yourself, people who work on safety, people who work on the fairness, people who work on all the things, checks and balances, so to speak, right? They are becoming more and more prominent. Do I know it\'s enough? No, it\'s not enough, obviously we need to do more for different reasons. For obviously DEI, but also for just privacy and safety and other things. And also for things we just talked about, right? Just because it\'s true that the fact that\... By the way, the fact that we have only the current system like GPT-3 or many other algorithms minimize only one value, it\'s not a feature, it\'s a bug. It\'s just convenience because we use the machine learning optimization algorithms that work in this way, and we just don\'t have other ways to do that. And I hope in the future other things would come up.

**0:23:46.0 Interviewee:** In answering your question, I don\'t think it will necessarily diverge, we\'ll just hit a roadblock and we\'re already hitting them. You\'ve heard about AlexNet like 10 years ago, and now, sure, we have acute applications and like filters on the phone, but like, did AI actually enter your life, daily life? Well, not true, I mean, you have better phone, they\'re more capable, but actually AI, like in terms of that we all dream about, does it enter your life? Well, not really. We can live without it, right? So, we\'re already hitting all these roadblocks, even like medical application. Google 10 years ago, claimed they\'d solved like the skin cancer when they can detect it, and it didn\'t\... It didn\'t really see the light of day except for some hospitals in India, unfortunately. So we\'re already hitting tons of roadblocks, and I don\'t think it\'s\... It\'s like for this reason precisely, because when you face reality, you just don\'t work as good as you expect for multiple reasons.

**0:24:30.4 Vael:** Interesting. Cool. So do you think that others\... This\... Have you heard of the alignment problem? Question one.

**0:24:38.1 Interviewee:** No.

**0:24:40.7 Vael:** No. Cool, all right, so you\'ve definitely heard of AI safety, right?

**0:24:43.5 Interviewee:** Mm-hmm.

**0:24:43.9 Vael:** Yeah. Alright. So one of the definitions of alignment is building models that represent and safely optimize hard-to-specify human values, alternatively ensuring that AI behavior aligns with system designer intentions. So I\'m\... So one of the questions, the question that I just kind of asked, was trying to dig at, do you think that the alignment problem per se, of trying to make sure that we are able to encode human values in a way that AIs can properly optimize, that that\'ll just kind of be solved as a matter of course, in that we\'re going to get stuck at various points, then we\'re going to have to address it. Or do you think that\'ll just like\... We will\... It won\'t get solved by default? Things will continue progressing in capabilities, and then we\'ll just have it be kind of unsafe, but like, \"Uh, you know, it\'s good enough. It\'s fine.\"

**0:25:21.1 Interviewee:** I think a bit of both. There\'s so much promise and so much hype and so much money in pushing AI forward. So I think a lot of companies will try to do that. These various\... We live in democracy, fortunately or unfortunately or actually we live in more like value of dollar, unfortunately, our society. At least in some countries are valued by progress. And especially companies, they have to progress, they have to advance and this is one of the easiest ways to advance. But I think some companies may be bad actors, whatever, they will try to push it to the limit. But these questions are ultimately unsolved, in a way this current system are designed, for the reason we discussed. So I think it will be a bit of both. Some companies will back down, some companies will try to push it to the limit, so we\'ll see. Depends on applications as well. I mean, some applications are safe. If you\... I\'m sorry, but\... Sorry to bring it up, but for example, there was a case of AI in Microsoft when they released the bot and the bot start cursing. Which is okay. It\'s a cute example, they should have done a bit of PR loss, it\'s fine, but it\'s different from the car crashing you into that tree. So depends, depends on the application.

**0:26:27.4 Vael:** Yeah, it seems definitely true. Alright, so next argument is focusing on our CEO AI again, which can do multi-step planning, and it has a model of itself in the world. So it\'s modeling other people modeling it, because it feels like that\'s pretty important for having any sort of advanced AI that\'s acting as a CEO. So the CEO is making plans for the future, and it\'s noticing as it\'s making plans for its goal of maximizing profits with constraints that some of its plans fail because it gets shut down. So we built this AI so that it has the default thing where you have to check with humans before executing anything, because that seems like a basic safety measure. And so the humans are looking for a one page memo on the AI\'s decision. So the AI is thinking about writing this memo, and then it notices at some point that if it just changes the memo to not include all the information, then maybe the AI will be\... I mean, then the humans will be more likely to approve it, which means it would be more likely to succeed in its original goal.

**0:27:22.8 Vael:** And so, the question here is\... So this is not building self preservation into the AI, it\'s just like AI as an agent that\'s optimizing any sort of goal and self preservation is coming up as an instrumental incentive. So, what do you think of the argument, \"Highly intelligent systems will have an incentive to behave in ways to ensure that they are not shut off or limited in pursuing their goals and this is dangerous?\"

**0:27:42.9 Interviewee:** Well, you already put intelligence and the human level into the sentence as well.

**0:27:48.3 Vael:** Yes.

**0:27:48.6 Interviewee:** Yeah, and it kind of\... I\'m already against that, because I don\'t think the system would actually behave in a way, like a sneaky way to avoid. Well, first of all, the current AI, even currently AI systems are highly uninterpretable. It\'s very hard to interpret what exactly is going on, right? But it still work within the bounds, right? So the example I gave you was Lee Sedol, when the Go game, already knew it\'s won, but it couldn\'t explain why it won, and it take human to parse it. Or another example that I like, in the chess, AI, at some point, put a queen in the corner, something you just never do as a human. And it couldn\'t explain, obviously, in fact. But it never pick up the board and crash it on the\... It will work within the bounds.

**0:28:29.1 Interviewee:** So of course, if the bounds of this program allows the program to cheat in a way and withhold information, then yes, but\... Again, it kinda works pair in pair. On one side it\'s really hard to interpret, so this one-pager that AI has to provide, it also has to be curated. And it will not include all information because it\'s impossible to digest all the computer memory and knowledge into one page. So on the one side, this page will always be limited and with all this lossy compression of the state of computer. But on the other side, I don\'t think computers can on purpose cheat on this page. Actually they might, depending on the algorithm\[?\], again, but I think it\'s a valid concern. That\'s an easy question, an easy answer, but it\'s depending on the system, depends how you design it.

**0:29:09.5 Vael:** Yeah, so I think I\'m trying to figure out what exactly is in this design system. So one thing is it has to be very capable, and it has to\... I want it to be operating over like reality in a way that I expect that CEOs would, so its task is interacting\... It\'s doing online learning, I expect, and it\'s interacting with actual people. So it\'s giving them text and it\'s taking in video and text, and interacting with them like a CEO would. And it does have to, I think, have a model of itself in the world in order for this to happen and to model how to interact with people. But if we have AI to that extent, which I kind of think that eventually we\'ll develop AI with these capabilities. I don\'t know how long it will take, but I assume that these are commercially valuable enough that people will eventually have this sort of system that\... This is an argument that like\... And I don\'t know if this is true, but any agent that\'s optimizing really hard for a single goal will at some point, like, figure out if it\'s smart enough to\... That it should like try to eliminate things that are going to reduce the likelihood of its plan succeeding, which in this case may be humans in the way.

**0:30:10.5 Interviewee:** I think you\'re right. Actually, while you were talking I also came up this example that I really like about, maybe you saw it as well, where there\'s an agent that optimized solving a particular race game and you control the car. At some point found a loophole, you remember \[hard-to-parse\] example, and find a loophole and just goes in circles. And the answer to that, you need to have explicit test goals. But in online learning settings, it\'s really hard. Plus, again, coming back to question number one, which is like, the system is so large at some point, you\'re just not able to cover all the cases. So yeah, yeah, I think so, I think it\'s possible, especially today in online learning fashion when you can\'t really have a complete system that you have all integration tests possible, and then you ship it. Once a system that automatically learns and updates, then it becomes\... That could be a problem. Yeah, I agree.

**0:30:52.8 Vael:** Yeah. So this boat driving race is actually one of the canonical examples of the alignment problem, as it were, which is like\-- you put in the right example into it. There\'s another part of the\... so that\'s one version of \"outer alignment\", which is where the system designer isn\'t able to input what it wants into the system well enough, which I think gets harder as the AI gets more powerful. And then there\'s an \"inner alignment\" kind of issue that people are hypothesizing might happen, which is where you have an optimizer that spins up another optimizer inside of it, where the outer optimizer is aligned with what the human wants, but it now has this inner optimizer. The canonical example used here is how evolution is in some sense the top-level optimizer and it\'s supposed to create agents that are good at doing inclusive reproductive fitness and instead we have humans. And we are not optimizing very hard for inclusive reproductive fitness and we\'ve invented contraceptives and stuff, which is like not very helpful. And so people are worried that maybe that would happen in very advanced AIs too, as we get like more and more generalizable systems.

**0:31:51.0 Interviewee:** Yeah. I think that there are two things to say, first of all, is what you said about loop in the loop. Now we\'re talking about what exactly system, what kind of a system can an algorithm create, right? Because for example, if you look at the current machine learning, we do know that for example, convolutional layers, they are good at computing equivariances, like transitional equivariances. If you move the object, they\'re supposed to be indifferent, but \[hard to parse\]-connected layers don\'t do that. So this \[hard to parse\] behavior you described? You need to have a system, first of all, that\'s capable enough for this kind of behavior. That\'s a big if, but okay. Once we get there, if we get there, the second question is, okay, can you cover for that? Can you figure out that these cases are eliminated completely from existence?

**0:32:32.4 Interviewee:** And the example of CEO is maybe is a good example. For me, the interesting example that is\... that I can definitely see and envision, not even CEO, but like for example, application that controls your behavior in a way. Like, for example, curate your Tinder profile or curate your inbox sorting. And control you through that. Then yeah, for sure. Yeah. If you don\'t control for everything, it can be smarter than us and kind of figure out, back-engineer, how humans work, because we\'re not \[hard to parse\] than that. And curate the channel for us and even\... Get us to the local minimum we might not want, but we\'ll still maximize its profits, whatever the profit means for the computer.

**0:33:07.2 Vael:** Okay. Yeah. So I\'m one of those people who is really concerned about long term risk from AI, because I think there\'s some chance that if we continue along the developmental pathway that we have so far, that if we won\'t solve the alignment problem, we won\'t figure out a good way in order to like encode human values in properly and include the human in the loop properly. Like one of the easiest solutions here is like trying to figure out how to put a human in that loop with a AI, even if the AI is vastly more intelligent than a human. And so people are like, Oh, well maybe if you just train an AI to like work with the human and translate the big AI so that like the human understands it, and this is interpretability, and then you have like a system who\'s training that system and maybe we can recursively loop up for something. But.

**0:33:52.2 Interviewee:** The problem here, why they see, like, for example, let\'s take a very specific example. There\'s AI systems that, for example, historically curated, curated by humans, obviously without bad intent, but it gives bad credit scores to black population say, like people of color. That\'s really bad behavior. And this behavior is kinda easy to check because you have statistics and you look at statistics. It\'s one, let\'s call it one hop away. So you take the data, you take statistics, is done. The second hop, the two hops away would be: it creates dynamics that you can check, not right away, but later that potentially show something like that. It would be harder for humans to check. You can also, if you think about it for a while, you can come up with like three hop, right. Something that creates something that creates something that it does. So it\'s much easier and much harder to check. You don\'t know until it happens, and that\'s the point. So you have this very complicated dynamic you can\'t\... There\'s not even flags that you can check, red flags that you can check in your model. That might be the issue.

**0:34:49.3 Vael:** Yeah. There\'s also\... Yeah. Interpretability seems really\... It seems really important, especially as very, very intelligent systems, and we don\'t know how to do that. So possible versions of things going badly, I think, are\... So if you have an AI system that\'s quite powerful and it\'s going to be instrumentally incentivized to not let itself be modified, then that means that you don\'t have that many chances to get it right in some sense, if you\'re dealing with a sufficiently intelligent system. Especially also because instrumental incentives are to acquire resources and influence and so\... and also improve itself. Which could be a problem maybe of recursive self-improvement. And then it, like, can get rid of humans very quickly if it wanted to, via like synthetic biology. Another kind of\-- this is not as advanced AI, but what if you put AI in charge of the manufacturing systems and food production systems, and there\'s some sort of correlated failure. And then we have misuse, and then the AI assisted-war as, like, various concerns about AI. What do you think about this?

**0:35:37.9 Interviewee:** Yeah. So one thing I want to say is that the AI system are kinda too lazy. In a sense that the reason why this loophole worked with the car is because this solution is easier in some mathematical way to find the proper solution.

**0:35:52.7 Interviewee:** So one thing humans can address this problem is just looking at\... Use a supervisor as a human, or with a test, which is pretty much like supervisors\-- check all the check and balances. We can create the system for which the, maybe mathematical even, finding proper solution is easier for the computer than finding its loopholes. That\'s one thing I want to say. And the second thing I want to say, which is now coming to nature because now we\'re talking about the rules of nature. But if it so happened that we design a system for which finding the loophole\... for example, we have laws of physics\-- we also have laws in behavior. Like if you have an algorithm that, you know, wants to organize human behavior or something, there\'s also laws of behavior. So it might be an interesting question that once we have this algorithm, we can get a bunch of sociologists, for example, people who are familiar with it, to study this algorithm and figure out that if, for example, loopholes are actually more probable than normal behavior quote on quote. So for example, being a bad person is better than being a good person\-- or easier. Not necessarily better, but easier than being a good person. Which we kind of see it in society sometimes.

**0:36:51.5 Interviewee:** So it\'s curious if an algorithm will actually discover that. And finding this loophole with a car is easy because it\'s there, you just need to move a little bit. So for a computer is easy to find it, like it\'s a local minimum that\'s really easy to fall in. But if you design the system for which these loopholes are hard, that might be easier. Or the question is, can we define a problem for which the proper solution is easier?

**0:37:11.9 Vael:** Yeah. And I think the problems are going to get much more\... harder and harder. Like if you have an AI that\'s doing a CEO thing, then I imagine like, just as humans figure out, there\'s many loopholes in society, many loopholes in how you achieve goals that are much cheaper in some sense. So I do think it\'s probably going to have to be designed in the system rather than being in the problem per se, as the problems get more and more difficult. Yeah.

**0:37:32.1 Vael:** So there\'s this community of people working on AI alignment per se and long term AI risk, and are kind of trying to figure out how you can build\... How you can solve the alignment problem, and how you can build optimization functions or however you need to design them, or interpretability or whatever the thing is, in order to have an AI that is more guaranteed, or less likely to end up doing this kind of instrumental incentive to deceive or not want to\... not self-preservation, maybe get rid of humans. So I think my question right now is, and there\'s money in this space now, there\'s a lot of interest in it and I\'m sure you\'ve heard of it\-- I mean, I\'m not sure you\'ve heard about it, but there\'s\... Yeah. There\'s certainly money and attention these days now that there wasn\'t previously. So what would cause you to work on these problems?

**0:38:12.4 Interviewee:** \...Well, ultimately a lot of people are motivated by actually just very few things and one of them is kids. Kids cause here you want to have a good future for yourself and the kids.

**0:38:24.8 Interviewee:** You want to live in a better and better human, better and better society, better and better everything. So that, and actually it hits home. The examples we discussed even during this call are, could be pretty grim. If we don\'t make this right. So putting resources there I think is really important. If people, before they come up with atomic bomb, they will figure out situations of which we are facing now, people might not even come up with atomic bomb, but do it in a safer way. Or like people knew about Chernobyl before it happened, obviously they would make it better. So having this hindsight, even though we don\'t know what\'s going to happen in the future, but putting the resources there, I think is definitely a smart move.

**0:38:57.9 Vael:** Got it. And what would cause you specifically to work on this?

**0:39:03.2 Interviewee:** Yes, I know, you asked this question. Yeah, well again, particular problems. I mean, it\'s an interesting problem. For me I\'m actually\... Since I work in optimization, so I like to have well-formulated problems. And making sure this one goes, this problem is more\... now it\'s kind of vague. I mean, even now we discuss it. I agree with you that this problem is valid. It makes sense. It exists. But it\'s still vague, like how you study it. My PhD was also on the way to interpret, to come up with a way to interpret data.

**0:39:30.2 Interviewee:** In fact, like maybe it\'s\... I don\'t want to spend too much time on it, but basically the idea is to visualize the data. If you have a very high dimension of data, you want to visualize it. But it\'s very lossy. You just visualize something, you just do something and it doesn\'t represent everything. And it\'s in a way it was actually, it was well-formulated, because there is a mathematical formula to minimizing. And of course it comes with conditions like \[hard to parse\] and loss and stuff, but in a way it\'s there, the problem is defined. So the same here. Like in a way, there\'s a mathematical apparatus. \...Maybe actually I\'m going to be the one developing it as well. So I\'m not saying that, \"come give me the one I\'m going to work on!\" I think that\'s a \[hard to parse - \"would be direct\"?\], so problem that I would be excited about.

**0:40:06.1 Vael:** Yeah. And I mean, I think like what this field really needs right now is someone to specify the problem more precisely. Just because it\'s like, Oh, this is a future system, it\'s like at least 50 years, well I don\'t know at least\-- it\'s far away. It\'s not happening immediately and we don\'t have very good like frameworks for it. And so it makes it hard to do research on. Cool. Alright. Well, I\'ll send you some resources afterwards if you feel like looking into it, but if not, regardless, thank you so much for doing this call with me.

**0:40:33.1 Interviewee:** Yeah, I appreciate it. Thank you for your time, it was really fun.

NeurIPSorICML_q243b-by Vael Gates-date 20220318.md