# Interview with AI Researchers NeurIPSorICML_cvgig by Vael Gates

**Interview with cvgig, on 3/24/22**

**0:00:02.5 Vael:** Awesome. Alright. So my first question is, can you tell me about what area of AI you work on in a few sentences?

**0:00:09.8 Interviewee:** Yeah. So I\'m what\'s technically called a computational neuroscientist, which is studying, using mathematics, AI and machine learning techniques to study the brain. Rather than creating intelligent machines, it\'s more about trying to understand the brain itself. And I study specifically synaptic plasticity, which is talking about how the brain itself learns.

**0:00:44.0 Vael:** So these questions are like, AI questions, but feel free to like\-- (Interviewee: \"No, go ahead.\") Okay, cool. Sounds good. Alright. What are you most excited about in AI and what are you most worried about? In other words, what are the biggest benefits or risks of AI?

**0:00:55.9 Interviewee:** Right. So in terms of benefits, I think that my answer might be a little bit divergent again, because I\'m a computational neuroscientist. But I think that AI and the tools surrounding AI give us a huge amount of power to understand both the human brain, cognition itself, and more general phenomena in the world. I mean, you see AI used in physics and in other areas. I think that it is just a very powerful tool in general for building understanding. In terms of risks, I think that it\'s, again, by virtue of being a very powerful tool, also something that can be used for just a huge number of nefarious things like governmental surveillance, to name one, military targeting technology and things like that, that could be used to kill or harm or disenfranchise large numbers of people in an automated way.

**0:02:04.2 Vael:** Awesome, makes sense. Yeah, and then focusing on future AI, putting on a science fiction forecasting hat, say we\'re 50-plus years into the future. So at least 50 years in the future, what does that future look like? This is not necessarily in terms of AI, but if AI is important, then include AI.

**0:02:22.6 Interviewee:** Yeah, so 50-plus years in the future. I always have trouble speculating with things like this. \[chuckle\] I think it\'ll be way harder than people tend to be willing to extrapolate. And also, I think that AI is not going to play as large of a role as someone might think. I think that\... I don\'t know, I mean in much the same way, I think it\'ll just be the same news with a different veneer. So we\'ll have more powerful technology, we\'ll have artificial intelligence for self-driving cars and things like that. I think that the technologies that we have available will be radically changed, but I don\'t think that AI is really going to fundamentally change the way that people\... Whether people are kind or cruel to one another, I guess. Yeah, is that a good answer? I don\'t know. \[chuckle\]

**0:03:21.8 Vael:** I\'m looking for your answer. So\...

**0:03:26.3 Vael:** Yes. 50 years in the future, you\'re like, it will be\... Society will basically kind of be the same as it is today. There will be some different applications than exists currently.

**0:03:36.4 Interviewee:** Yeah, unless it\'s\... It\'s perfectly possible society will utterly collapse, but I don\'t really think AI will be the reason for that. \[chuckle\] So, yeah, right.

**0:03:47.5 Vael:** What are you most worried about?

**0:03:50.9 Interviewee:** In terms of societal collapse? I\'d say climate change, pandemic or nuclear war are much more likely. But I don\'t know, I\'m not really betting on things having actually collapsed in 50 years. I hope they don\'t, yeah. \[chuckle\]

**0:04:07.0 Vael:** Alright, I\'m gonna go on a bit of a spiel. So people talk about the promise\...

**0:04:10.7 Interviewee:** Yeah, yeah.

**0:04:12.6 Vael:** \[chuckle\] Yeah, people talk about the promise of AI, by which they mean many things, but one of the things they may mean is whether\... The thing that I\'m referencing here is having a very generally capable system, such that you could have an AI that has the cognitive capacities that could replace all current day jobs, whether or not we choose to have those jobs replaced. And so I often think about this within the frame of like 2012, we had the deep learning revolution with AlexNet, and then 10 years later, here we are and we have systems like GPT-3, which have some weirdly emergent capabilities, like they can do some text generation and some language translation and some coding and some math.

**0:04:42.7 Vael:** And one might expect that if we continue pouring all of the human effort that has been going into this, like we continue training a whole lot of young people, we continue pouring money in, and we have nations competing, we have corporations competing, that\... And lots of talent, and if we see algorithmic improvements at the same rate we\'ve seen, and if we see hardware improvements, like we see optical or quantum computing, then we might very well scale to very general systems, or we may not. So we might hit some sort of ceiling and need a paradigm shift. But my question is, regardless of how we get there, do you think we\'ll ever get very general systems like a CEO AI or a scientist AI? And if so, when?

**0:05:20.6 Interviewee:** Yeah, so I guess this is somewhat similar to my previous answer. There is definitely an exponential growth in AI capabilities right now, but the beginning of any form of saturating function is an exponential. I think that it is very unlikely that we are going to get a general AI with the technologies and approaches that we currently have. I think that it would require many steps of huge technological improvements before we reach that stage. And so things that you mentioned like quantum computing, or things like that.

**0:06:00.1 Interviewee:** But I think that fundamentally, even though we have made very large advances in tools like AlexNet, we tend to have very little understanding of how those tools actually work. And I think that those tools break down in very obvious places, once you push them beyond the box that they\'re currently used in. So, very straightforward image recognition technologies or language technologies. We don\'t really have very much in terms of embodied agents working with temporal data, for instance. I think that\...

**0:06:42.2 Interviewee:** I essentially think that even though these tools are very, very successful in the limited domains that they operate in, that does not mean that they have scaled to a general AI. What was the second half of your question? It was like kind of, Given that we\... Do you have it what it\'ll look like, or\...

**0:06:57.4 Vael:** Nah, it was actually just like, will we ever get these kind of general AIs, and if so, when? So\...

**0:07:03.2 Interviewee:** Yeah, so I would essentially say that it\'s too far in the future for me to be able to give a good estimate. I think that it\'s 50 plus years, yeah.

**0:07:13.2 Vael:** 50 plus years. Are you thinking like a thousand years or you\'re thinking like a hundred years or?

**0:07:19.7 Interviewee:** I don\'t know. I mean, I hope that it\'s earlier than that. I like the idea of us being able to create such things, whether we would and how we would use them. I would not, \[chuckle\] I don\'t think I would want to see a CEO AI, \[chuckle\] but there are many forms of general artificial intelligence that could be very interesting and not all that different from an ordinary person. And so I would be perfectly happy to see something like that, but I just, you know, and I guess in some sense, my work is hopefully contributing to something along those lines, but I don\'t think that I could guess when it would be, yeah.

**0:08:00.2 Vael:** Yeah. Some people think that we might actually get there via by just scaling, like the scaling hypothesis, scale our current deep learning system, more compute, more money, like more efficient, more like use of data, more efficiency in general, yeah. And do you think this is like basically misguided or something?

**0:08:15.9 Interviewee:** Yeah, let me take a moment to think about how to articulate that properly. I think\... Yeah, you know, let me just take a moment. I think that when you hear people like, for instance, Elon Musk or something along these lines saying something like this, it reflects how a person who is attempting to get these things to come to pass and has a large amount of money would say something, right. It\'s like, what I\'m doing is I\'m pouring a large amount of money into this system and things keep on happening, so I\'m happy with that. But I think that from my position of seeing how much work and effort goes into every single incremental advance that we see, I think that it\'s just, there are so many individual steps that need to be made and any one of them could go wrong and provide a, essentially a fundamental sealing on the capabilities that we\'re able to reach with our current technologies. And so it just seems a little, a little hard to extrapolate that far in the future.

**0:09:25.5 Vael:** Yeah. What kind of things do you think we\'ll need in order to have something like, you know, a multi-step planner can do social modeling, can model all of the things modeling it like that kind of level of general.

**0:09:35.5 Interviewee:** Yeah. So I think that one of the main things that has made vision technologies work extremely well is massive parallelization in training their algorithms. And I think that, what this reflects is the difficulty involved in training a large number\... So essentially, when you train an algorithm like this, you have a large number of units in the brain like neurons or something like that, that all need to change their connections in order to become better at performing some task. And two things really tend to limit these types of algorithms, it\'s the size and quality of the data set that\'s being fed into the algorithm and just the amount of time that you are running the algorithm for. So it might take weeks to run a state-of-the-art algorithm and train it now. And you can get big advances by being able to train multiple units in parallel and things like that.

**0:10:33.5 Interviewee:** And so I think that the easiest way to get very large data sets and have everything run in parallel is with specialized hardware called, you know, people would call that wetware or neuromorphic computing or something along those lines. Which is currently very, very new and has not really, as far as I know, been used for anything particularly revolutionary up to this point. You can correct me if I\'m wrong on that. I would expect that you would have to have essentially embodied agents before you can get\... in a system that is learning and perceiving at the same time before you could get general intelligence.

**0:11:12.5 Vael:** Well, yeah, that\'s certainly very interesting to me. So, it\'s not\... So people are like, \"We definitely need hardware improvements.\" And I\'m like, \"Yup, current day systems are not very good at stuff. Sure, we need hardware improvements.\" And you\'re saying, are you saying we need to like branch sideways and do wetware\-- these are like biological kind of substrates, or are they different types of hardware?

**0:11:37.3 Interviewee:** I guess different types of hardware is maybe the shorter term goal on something like that. Like you would expect circuits in which individual units of your circuit look a little bit like neurons and are capable to adapt their connections with one another, running things in parallel like that can save a lot of energy and allows you to kind of train your system in real time. So it seems like that has some potential, but it\'s such a new field that, this is when I, when I think about what time horizon you would need for something like this to occur, it seems like you would need significant technological improvements that I just don\'t know when they\'ll come.

**0:12:20.4 Vael:** Yeah. So I haven\'t heard of this wetware concept. So like it\'s a physical substrate that like\... It like creates, it creates new physical connections like neurons do or it just like, does, you know\...

**0:12:33.5 Interviewee:** No, it doesn\'t create physical connections. You could just imagine this like\... So, you know, computer systems have programs that they run in kind of an abstract way.

**0:12:43.8 Vael:** Yep.

**0:12:44.8 Interviewee:** And the hardware itself is logic circuits that are performing some kind of function.

**0:12:48.9 Vael:** Yep.

**0:12:49.8 Interviewee:** And neuromorphic computing is individual circuits in your computer have been specially designed to individually look like the functions that are used in neural networks. So you have\... Basically, the circuit itself is a neural network, and because you don\'t have these extra layers of programming added in on top, you can run them continuously and have them work with much lower energy and stuff like that. It\'s just\... It\'s limiting because they can\'t implement arbitrary programs, they can only do neural network functions, and so it\'s kind of like a specialized AI chip. People are working on developing that now\... Yeah.

**0:13:32.7 Vael:** Okay, cool, so this is one of the new hardware-like things down the line. Cool, that makes sense. Alright, so you\'d like to see better hardware, probably you\'d say that you\'d probably need more data, or more efficient use of data. Presumably for this\-- because the kind of continuous learning that humans do, you need to be able to have it acquire and process continuous streams of both image and text data at least. Yeah, what else is needed?

**0:14:03.8 Interviewee:** Oh, I think that\... Yeah, more fundamentally than either of those things. It\'s just the fact that we don\'t understand what these algorithms are doing at all. And so we\'re\... You can train it, you can train an algorithm and say, \"Okay, you know, it does what I want it to do, it performs well,\" and most machine learning techniques are not very good at actually interrogating what a neural network is actually doing when it\'s processing images. And there are many instances recently, I think the easiest example is adversarial networks, if you\'ve heard of those?

**0:14:41.8 Vael:** Mm-hmm.

**0:14:42.2 Interviewee:** I don\'t know what audience I\'m supposed to be talking to in this interview.

**0:14:46.4 Vael:** Yeah, just talk to me I think.

**0:14:49.2 Interviewee:** Okay, okay.

**0:14:50.1 Vael:** I do know what adversarial\... Yeah.

**0:14:52.9 Interviewee:** Okay, so, adversarial networks are\... You perturb images in order to get your network to output very weird answers. And the ability of making a network do something like that, where you are able to change its responses in a way that\'s very different from the human visual system by artificial manipulations, makes me worried that these systems are not really doing what we think they\'re doing, and that not enough time has been invested in actually figuring out how to fix that, which is currently a very active area of research, and it\'s partly limited by the data sets that we\'ve been showing our neural networks. But I think in general, there\'s been too much of an emphasis on getting short-term benefits in these systems, and not enough effort on actually understanding what they\'re learning and how they work.

**0:15:43.5 Vael:** That makes sense. Do you think that the trend\... So if we\'re at the point where people are deploying things that you don\'t understand very well, do you think that this trend will continue and we\'ll continue advancing forward without having this understanding, or do you think it would catch up or\...

**0:16:00.4 Interviewee:** Yeah, well, I think it\'s reflective of the huge pragmatic influence that is going on in machine learning, which is essentially, corporations can make very large amounts of money by having incremental performance increases over their preferred competitors. And so, that\'s what\'s getting paid right now. And if you look at major conferences, the vast majority of papers are not probing the details of the networks that they\'re training, but are only showing how they compare it to competitors. They\'ll say, \"Okay, mine does better, therefore, I did a good job,\" which is really not\... It\'s a good way to get short-term benefits to perform, essentially, engineering functions, but once you hit a boundary in the capabilities of your system, you really need to have understanding in order to be able to be advanced further. And so I really think it\'s the funding structure, and the incentive structure for the scientists that\'s limiting advancement.

**0:17:02.2 Vael:** That makes sense. Yeah, and again, I hear a lot of thoughts that the field is this way and they have their focus on benchmarks is maybe not\... and incremental improvements in state-of-the-art is not necessarily very good for\... especially for understanding. When I think about organizations like DeepMind or OpenAI, who\'re kind of exclusively or\... explicitly aimed at trying to create very capable systems like AGI, they\... I feel like they\'ve gotten results that I wouldn\'t have expected them to get. It doesn\'t seem like you could should just be able to scale a model and then you get something that can do text generation that kind of passes the Turing Test in some ways, and do some language translation, a whole bunch of things at once. And then we\'re further integrating with these foundational models, like the text and video and things. And I think that those people will, even if they don\'t understand their systems, will continue advancing and having unexpected progress. What do you think of that?

**0:18:09.6 Interviewee:** Yeah, I think it\'s possible. I think that DeepMind and OpenAI have basically had some undoubtedly, extremely impressive results, with things like AlphaGo, for instance. What\'s it called, AlphaStar, the one that plays StarCraft. There are lots of really interesting reinforcement learning examples for how they train their systems. Yeah, I think it just remains to be seen, essentially. It would be nice\-- Well, maybe it wouldn\'t be nice, it would be interesting to see if you can just throw more at the system, throw more computing capabilities at problems, and see them end up being fixed, but I\...

**0:19:04.0 Interviewee:** I\'m just skeptical, I guess. It\'s not the type of work that I want to be doing, which is maybe biasing my response, and I don\'t think that we should be doing work that does not involve understanding for ethical reasons and advancing general intelligence. For reasons that I stated, that essentially, if you hit a wall you\'ll get very stuck. But yeah, you\'re totally right that there have had been some extremely, extremely impressive examples in terms of the capability capabilities of DeepMind. And, yeah, there\'s not too much to be said for me on that front.

**0:19:46.8 Vael:** Yeah. So you said it would be interesting, you don\'t know if it would will be nice. Because one of the reasons that it maybe wouldn\'t be nice is that you said that there\'s ethical considerations. And then you also said there\'s this other thing; if you don\'t understand things then when you get stuck, you really get stuck though.

**0:20:01.5 Interviewee:** Yeah.

**0:20:04.4 Vael:** Yeah, it seems right. I would kind of expect that if people really got stuck, they would start pouring effort into interpretability work for other types of things.

**0:20:12.7 Interviewee:** Right. You would certainly hope so. And I think that there has been some push in that direction, especially there\'s been a huge\... I keep on coming back to the adversarial networks example, because there have actually been a huge number of studies trying to look at how adversarial examples work and how you can prevent systems from being targeted by adversarial attacks and things along those lines. Which is not quite interpretability, it\'s still kind of motivated by building secure, high performance systems. But I think that you\'re right, essentially, once you hit a wall, things come back to interpretability. And this is, again, circling back to this idea of every saturating function looks like an exponential at the beginning, is that the deep learning is currently in a period of rapid expansion, and so we might be coming back to these ideas of interpretability in 10 years or so, and we might be stuck in 10 years ago or so, and the question of how long it\'ll take us to get general artificial intelligence will seem much more inaccessible. But who knows.

**0:21:26.8 Vael:** Interesting. Yeah, when I think about the whole of human history or something, like 10,000 years ago, things didn\'t change in lifetime to lifetime. And then here we are today where we have probably been working on AI for under 100 years, like about 70 years or something, and we made a remarkable amount of progress in that time in terms of the scope of human power over their environment, for example. So yeah, there certainly have been several booms and bust of cycles, so I wouldn\'t be surprised if there is a bust of cycle for deep learning. Though I do expect us to continue on the AI track just because it\'s so economically valuable, which especially with all the applications that are coming out.

**0:22:04.1 Interviewee:** Yeah, you don\'t have to be getting all the way to AI for there not to be plenty of work to be\... General artificial intelligence, for there to be plenty of work to be done. There are hundreds of untapped ways to use, I\'m sure, even basic AI that are currently the reason that people are getting paid so well in the field, and there\'s a lack of people to be working in the field, so there\'s\... I don\'t know, there are tons of opportunities, and it\'s gonna be a very long time before people get tired of AI. So yeah, that\'s not gonna happen anytime soon.

**0:22:36.6 Vael:** True. Alright, I\'m gonna switch gears a little bit, and ask a different question. So now, let\'s say we\'re in whatever period we are where we have this advanced AI systems. And so we have a CEO AI. And a CEO AI can do multi-step planning and as a model of itself modelling it and here we are, yeah, as soon as that happens. And so I\'m like, \"Okay, CEO AI, I wish for you to maximize profits for me and try not to run out of money and try not to exploit people and try to avoid side-effects.\" And obviously we can\'t do this currently. But I think one of the reasons that this would be challenging now, and in the future, is that we currently aren\'t very good at taking human values and preferences and goals and turning them into optimizations\-- or, turning them into mathematical formulations such that they can be optimized over. And I think this might be even harder in the future\-- there\'s a question, an open question, whether it\'s harder or not in the future. But I imagine as you have AI that\'s optimizing over larger and larger state spaces, which encompasses like reality and the continual learners and such, that they might alien ways of\... That there\'s just a very large shared space, and it would be hard to put human values into them in a way such that AI does what we intended to do instead of what we explicitly tell it to do.

**0:23:57.9 Vael:** So what do you think of the argument, \"Highly intelligent systems will fail to optimize exactly what their designers intended them to and this is dangerous?\"

**0:24:07.1 Interviewee:** Oh, I completely agree. I think that no matter how good of an optimization system you have, you have to have articulated it well and clearly the actual objective function itself. And to say that we as a collective society or as an individual corporation or something along those lines, could ever come to some kind of clear agreement about what that objective function should be for an AI system is very dubious in my opinion. I think that it\'s essentially\... Such an AI system would have to, in order to be able to do this form of optimization, would essentially have to either be a person, in order to give people what they want, or it would have to be in complete control of people, at which point it\'s not really a CEO anymore, it\'s just a tool that\'s being used by people that are in a system of controlling the system like that. I don\'t think that that would solve the problem. There are lots of instances of corporate structures and governmental structures that are disenfranchising and abusing people all around the world, and it becomes a question of values and what we think these systems should be doing rather than their effectiveness in actually doing what we think they should be doing. And so, yeah, I basically completely agree with the question in saying that we wouldn\'t really get that much out of having an AI CEO. Does that\...

**0:25:50.8 Vael:** Interesting. Yeah, I think in the vision of this where it\'s not just completely dystopian, what you maybe have is an AI that is very frequently checking in on human feedback. And that has been trained very well with humans such that it is\... So there\'s a question of how hard it is to get an AI to be aligned with one person. And then there\'s a question of how hard it is to get an AI to be aligned with a multitude of people, or a conglomerate of people, or how we do democracy or whatever that\'s, yeah, complicated. But even with one person, you still might have trouble, is my intuition here? And just trying to have it\-- still with the access to human feedback, still have human feedback in a way that it\'s fast enough that the AI is still doing approximately what you want.

**0:26:41.7 Interviewee:** Yeah, yeah, I agree. Yeah. I just think that the question of interpretability becomes a very big issue here as well where you really want to know what your system is doing, and you really need to know how it works. And with the way things are currently going we\'re nowhere near that. And so, if we have a large system that we don\'t understand how it works and is operating on limited human feedback and is relatively inscrutable, the list of problems that could result from that is very very long. Yeah. \[chuckle\]

**0:27:15.6 Vael:** Awesome. Yeah, and my next question is about presumably one of those problems. So, say we have our CEO AI, and it\'s capable of multi-step planning and can do people modelling it, and it is trying to\... I\'ve given it its goal, which is to optimize for profit with a bunch of constraints, and it is planning and it\'s noticing that some of its plans are failing because it gets shut down by people. So as a basic mechanism, we have basically\--

**0:27:44.4 Interviewee:** Because it\'s what by people?

**0:27:46.2 Vael:** Its plans are getting\... Or it is getting shut down by people. So this AI has been put\... There\'s a basic safety constraint in this AI, which is that any big plans it does has to be approved by humans, and the humans have asked for a one-page memo. So this AI is sitting there and it\'s like, \"Okay, cool, I need to write this memo. And obviously, I have a ton of information, and I need to condense it into a page that\'s human comprehensible.\" And the AI is like, \"Cool, so I noticed that if I include some information in this memo then the human decides to shut me off, and that would make my ultimate plan of trying to get profit less likely to happen, so why don\'t I leave out some information so that I decrease the likelihood of being shut down and increase the likelihood of achieving the goal that\'s been programmed into me?\" And so, this is a story about an AI that hasn\'t had self-preservation built into it, but it is arising as an instrumental incentive of it being an agent optimizing towards any goal. So what do you think of the argument, \"Highly intelligent systems will have an incentive to behave in ways to ensure that they are not shut off or limited in pursuing their goals, and this is dangerous?\"

**0:28:53.1 Interviewee:** Well, right. It\'s very dependent on the objective function that you select for the system. I think that a system\... It seems, at face value, pretty ridiculous to me that the CEO of a company, the CEO robot, would have its objective function being maximizing profit rather than maximizing individual happiness within the company or within the population on the whole. But even in a circumstance like that, you can imagine very, very, very many pathological circumstances arising. This is the three laws of robotics from Isaac Asimov, right? It\'s just very simplified objective functions produce pathological consequences when scaled to very large complex systems. And so, in much the same way you can train a neural network to recognize an image which produces the unintended consequence that tiny little perturbations of that image can cause it to radically change its output when you have improperly controlled what the system is doing at a large scale, the number of tiny unintended consequences that you could have essentially explodes many-fold. And yeah, I certainly wouldn\'t do this. That\'s certainly not something that I would do, yeah.

**0:30:20.6 Vael:** Yeah. Have you heard of AI Safety?

**0:30:24.3 Interviewee:** AI\... Yeah, yeah.

**0:30:26.0 Vael:** Cool. What does that term mean for you?

**0:30:27.2 Interviewee:** You\'re talking\... What does it mean for me? Well, I guess it\'s closely related to AI ethics. AI safety would mainly be a set of algorithms, or a set of protocols intended to ensure that a AI system is actually doing what it\'s supposed to do and that it behaves safely in a variety of circumstances. Is that correct?

**0:30:52.2 Vael:** Well, I don\'t\-- there\'s not one definition in fact, it seems like it\'s a sprawling field. And then, have you heard of the term AI alignment?

**0:31:00.7 Interviewee:** No, I don\'t know what that is.

**0:31:01.5 Vael:** Cool. This is more long-term focused AI safety. And one of their definitions they use is building models that represent and safely optimize hard-to-specify human values. Alternatively, ensuring that AI behavior aligns with the system designer intentions. Although there are a lot of different definitions of alignment as well. So there\'s a whole bunch of people who are thinking about long-term risks from AI, so as AI gets more and more powerful. I think the example we just talked about, like the ones where adversarial examples can really change the output of a system very easily, is a little bit different than the argument made here, which is something like: if you have an agent that\'s optimizing for a goal and it\'s good enough at planning then it\'s going to be instrumentally incentivized to acquire resources and power and not be shut down and kind of optimize against you, which is a problem when you have an AI that is similarly as smart as humans. And I think in that circumstance, one of the arguments is that this constitutes an existential risk, like having a system that\'s smarter than you constituting against you would be quite bad. What do you think of that?

**0:32:04.1 Interviewee:** Yeah, I was only using the adversarial example to give an example of how easily and frequently this does happen at even the level that we\'re currently working at. I think it would be much, much, much worse at the level of the general artificial intelligence that would have essentially long-term dynamic interactions with people, rather than a system that\'s just taking an image and outputting a response. When the consequences of such a system can have long term effects on the health and well-being of people, this kind of thing becomes very different and much more important.

**0:32:43.4 Vael:** Yeah. And like with the problem I was outlining earlier, which is like, how do we get to do exactly what they intended to do? The idea that you have of like trying\... Like why would you create a system that wasn\'t optimizing for all of human values? I was like, wow, ahead of the game there. That is in some sense the goal. So there is a community who\'s working on AI alignment kind of research, there\'s money in this community. It\'s fairly new\-- although much more popular, or like, AI safety haw grown a lot more over the years. What would cause you to work on trying to prevent long-term risks from AI systems?

**0:33:18.5 Interviewee:** What would cause me to do work on it?

**0:33:20.6 Vael:** Yeah.

**0:33:29.6 Interviewee:** To be honest, I think that it would have to be\... I guess I would really have to be convinced that the state of the field in the next few years is tending towards some type of existential risk. I feel like\... You don\'t have to convince me too much, but I personally don\'t think that the field of study that I\'m currently occupying is one that\'s really contributing to this problem. And so I would become much more concerned if I felt like the work that I was doing was actively contributing to this problem, or if there was huge evidence of the near advent of these types of generally intelligent systems to be terribly worried about.

**0:34:28.6 Vael:** Yeah. That makes sense. Yeah, I don\'t actually expect computational neuroscience to be largely contributing to this in any way. I feel like the companies that are gonna be doing this are the ones who are aiming for AGI. I do expect them to kind of continue going that way, regardless of what is happening. And I expect the danger to happen not immediately, not in the next couple of years. Certainly people have like different ranges, but like 2060 is like an estimate on some paper I believe that I can send along. It probably won\'t be a while, won\'t be for a while.

**0:35:00.5 Interviewee:** Sure. I don\'t know, I think that people who understand these algorithms in the way that they work do have in some sense a duty to stand up to these types of problems if they present themselves. And there are many instances of softer forms of AI being used for horrible things currently, which I certainly could be doing more in my daily life to prevent. But for now, I don\'t know. I guess I just have, I have my own interests and priorities. And so it\'s kind of a\... It\'s something to get to eventually.

**0:35:42.9 Vael:** Yeah, yeah. For sure. I think these technical AI safety is important. And am I working in technical AI safety? Nope. So like we all do the things that we want to do.

**0:35:54.8 Interviewee:** Yeah.

**0:35:54.9 Vael:** Great, cool. So that was my last question, my downer of an interview here \[chuckle\], but how do you think\...

**0:36:02.3 Interviewee:** No, no.

**0:36:04.1 Vael:** But yeah. Okay. So my actual last question is, have you changed your mind in anything during this interview and how was this interview for you?

**0:36:08.9 Interviewee:** No, it was a good interview. I don\'t think I\'ve particularly changed my mind about anything. I think that it was good to work through some of these questions and yeah, I had a good time.

**0:36:24.2 Vael:** Amazing. Yeah, why\--

**0:36:25.3 Interviewee:** I typically don\'t expect it to change my mind too much in interviews, so \[chuckle\].

**0:36:28.8 Vael:** Absolutely. Yeah, yeah, yeah. Okay. Why do\... People tell me they have a good time and I\'m like, are you lying? Did you really have\... Why is this a good time?

**0:36:37.2 Interviewee:** No, it\'s nice to talk about your work. It\'s nice to talk about long-term impacts that you don\'t talk about in your daily basis. I don\'t know. I don\'t need to be paid to do something like this for instance.

**0:36:51.7 Vael:** All right. Well, thank you so much. Yeah. If you think of any questions for me, I\'m here for a bit. I\'m also happy to send any resources if you\'re curious about, like, my takes on things, but yeah, generally just very appreciate this.

**0:37:04.4 Interviewee:** Yeah, sure. I\'m a little curious about what this interview is for. Is it for just you, or is it, like a\... You mentioned something about some type of AI alignment group or is there some kind of\... I\'m just curious about what it\'s for.

**0:37:20.9 Vael:** Yeah. So I am interested\... I\'m part of the AI alignment community, per se, although I\'m not doing direct work. The people there often work on technical solutions to try to\... to the alignment problem, which is just trying to come up with good ways of making sure that AIs in the future will be responsive, do what humans want. And examples of that include trying to build in feedback, human feedback, in a way that is scalable with current systems and works with uninterpretable systems, and interpretability\-- certain types of interpretability work. There\'s teams like DeepMind Safety, OpenAI Safety, different, like, separate alignment community. So I\'m like in that space. And I\'ve been doing interviews with AI researchers to see what they think about the safety arguments. And whether\... instrumental incentives. And just like, when do you think we\'ll get AGI, if you think we will. Get a lot of different opinions, a lot of different ways.

\[\...\]

**0:38:47.5 Interviewee:** Cool. Anyway, that makes a lot of sense and, yeah, I hope that things go well. Thanks for having me. Yeah.

**0:38:55.5 Vael:** Yeah. Thanks so much, really appreciate it. Alright, bye.

**0:38:59.1 Interviewee:** Bye, see you.

NeurIPSorICML_cvgig-by Vael Gates-date 20220324.md


# Interview with AI Researchers individuallyselected_84py7 by Vael Gates

Interview with 84py7, on 3/18/22
================================

**0:00:00.0 Vael:** Here we are. Perfect. So my first question is, can you tell me about what area of AI you work on in a few sentences?

**0:00:09.0 Interviewee:** Yeah. I\'m transferring my research from essentially pure mathematics to AI alignment. And specifically, I plan to work on what I\'ve been calling weak alignment or partial alignment, which is not so much trying to pin down exactly a reward function that\'s in the interest of humanity but rather train AI systems to have positive-sum interactions with humanity.

**0:00:44.8 Vael:** Interesting. Cool, I expect we\'ll get into that a little bit further on. \[chuckle\] But my next question is, what are you most excited about in AI, and what are you most worried about? In other words, what are the biggest benefits or risks of AI?

**0:01:00.1 Interviewee:** Yeah, biggest benefits\... I think AI has the potential to help us solve some of the biggest challenges that humanity\'s facing. It could potentially teach us how to solve climate change, or at least mitigate it. It could help us avert nuclear war, avert bio-risks, and maybe most importantly, avert other AI risks. So that\'s the upside. The downside is exactly those other AI risks, so I worry about a potentially small research group coming out of a for-profit company, which might have some safety aspect, but the safety could be window dressing. It could be something that\'s mostly a PR effort, something that just exists to satisfy regulators. And at the end of the day when\... or if they manage to develop superhuman AI, the safety people will be marginalized, and the AI will be used in the interest of one company or even one or a few individuals. And that could be potentially very bad for the rest of us. There\'s also the issue of an arms race between multiple AI projects, which could be even worse. So those are some of my\... broadly, some of my worries.

**0:02:35.7 Vael:** Interesting. So just to get a little bit more straight on the\... Okay, so the second story is arms race between the AI orgs. And the first one is\... why are the AI researchers getting\... why are the safety researchers getting marginalized?

**0:02:47.5 Interviewee:** Well, if you look at, for example, financial institutions before the 2008 crisis, it\'s not like they had no regulation, although the banks and insurance companies had been gradually deregulated over several decades but still, there were multiple regulators trying to make sure that they don\'t take on too much leverage and aren\'t systemic risks. And nevertheless, they found ways to subvert and just get around those regulations. And that\'s partly because regulators were kind of outmatched, they had orders of magnitude, less funding, and it was hard to keep up with financial innovations. And I see the same potential risks in AI research, potentially even worse, because the pace of progress and innovation is faster in AI, and regulators are way behind, there doesn\'t even exist meaningful regulation yet. So I think it\'s easy for a team that\'s on the verge of getting a huge amount of power from being the first to develop superhuman artificial intelligence to just kind of push their safety researchers aside and say, \"You know what? You guys are slowing us down, if we did everything you said, we would be years slower, somebody else might beat us, and we should just go full speed ahead.\"

**0:04:25.3 Vael:** Interesting. Yeah, I guess both of those scenarios aren\'t even\... we don\'t solve the technical problem per se, but more like we can\'t coordinate enough, or we can\'t get regulation good enough to make this work out. So that\'s interesting. Do you think a lot about policy and what kind of things policy-makers should do?

**0:04:44.6 Interviewee:** No, I\'m sort of pessimistic about governments really getting their act together in a meaningful way to regulate AI in time. I guess it\'s possible if the progress slows and it takes many decades to get to a superhuman level, then maybe governments will catch up. But I don\'t think we can rely on that slow timeline. So I think more optimistically would be\... There are only a small number of tech incumbents, and plausibly they could coordinate with each other to avoid the kind of worst Red Queen arms race to be first and to put their own safety measures into place voluntarily. So if I were doing policy, which I\'m not, that\'s the direction I would try to go in. But I think beyond policy, the technical problem of how to solve alignment is still wide open, and that\'s personally where I feel I might be able to contribute, so that\'s my main focus.

**0:05:56.8 Vael:** Interesting. Yeah, so sort of branching off \[from\] how long it will take: focusing on future AI, putting on a science fiction forecasting hat, say we\'re 50-plus years into the future. So at least 50 years in the future, what does that future look like?

**0:06:13.3 Interviewee:** I think that\'s a really open question. In my weak or partial alignment scenario, that future involves a few dominant platforms that allow for the development of advanced AI systems. And because there are only a few platforms, they all have strict safety measures sort of built in from the ground up, maybe even from the hardware level up. And that allows even small companies or potentially even individuals to spin up their own AGIs. And so there\'s this kind of giant ecosystem of many intelligent agents that are all competing to some extent, but they also have a lot of common interest in not blowing up the current world order. And there\'s a kind of balance of powers, where if one agent gets too powerful, then the others coordinate to keep it in check. And there\'s a kind of system of rules and norms which aren\'t necessarily based on legal systems, because legal systems are too slow, but they\'re a combination of informal norms and formal safety measures that are sort of built into the agents themselves that keep things roughly in balance. That\'s the kind of scenario I hope for. It\'s very multi-polar, but it\'s really\... There are so many agents, and no one of them has a significant portion of the power. There are of course many worse scenarios, but that\'s my optimistic scenario.

**0:08:04.4 Vael:** Interesting. Yeah, so when you say there\'s only a few platforms, what\'s an example of a platform or what that would look like?

**0:08:11.2 Interviewee:** Well, today, there\'s TensorFlow, and there\'s PyTorch and so on. In principle you could build up your own machine learning tools from scratch, but that\'s a significant amount of effort even today. And so most people go with the existing tools that are available. And decades\... 50 years from now, it will be much harder to go from scratch, because the existing tools will be way more advanced, there\'ll more layers of development. And so I think for practical purposes, there will be a few existing best ways to spin up AI systems, and the hope is that all those ways have safety measures built in all the way from the hardware level. And even though in principle somebody could spin up an aligned AI from scratch, that would be an enormous effort involving just so much\... Down to chip factories. It will be easy to detect that kind of effort and stop it from getting off the ground.

**0:09:26.4 Vael:** Interesting. Yeah, that is super fascinating. So how\... What would that look like, for safety to be built into the systems from a hardware level? What would these chips look like, what sort of thing?

**0:09:38.8 Interviewee:** Yeah, that\'s\... I\'ve been thinking about that. And I don\'t have a detailed vision of how that would work, but that\'s\... One direction I might take my research is looking into that. One idea I have is to kind of build in back doors to the AI. So there\'s a range of types of back door, ranging from like Achilles heel, which is a kind of designed weakness in the AI that humans can take advantage of if things go wrong. Moving from that, slightly stronger than that is a kind of off switch which can just shut the AI down if things get really bad. The thing I worry with off switches is they\'re too binary. If an AI actually has a lot of power, it\'s probably benefiting some humans, and there will be political debate about whether to turn it off. And the decision will take too long, and things could get out of control. So what I\'m looking into is a more\... Something more flexible than an off switch, which I\'ve been calling a back door, which is a way to\... Well, okay, there\'s two types of things. So first, there\'s a throttle, which is like you can fine-tune the amount of resources you give to the AI. If it\'s behaving well, you can give it more compute, more memory, access to more cloud resources, more data centers and so on. If it starts doing things that seem a little fishy, you can just tune that stuff back and examine it, which might be an easier political decision than just turning it off, which would be very disruptive.

**0:11:22.7 Interviewee:** So that\'s a throttle. And then even more flexible than that is a back door, where you could actually modify the code or parts of the code base of the AI as it\'s running. And again, that could be easier to\... Politically easier to implement, because you don\'t have this very disruptive effect of just shutting the thing down, you could just kind of tweak it as it goes. So how to do that from a hardware level? It\'s unclear to me whether that\'s feasible at all. And looking into trusted computing platforms, software that can only run on designated hardware, software watermarking, obfuscating or encrypting part of the code base of the AI, putting part of the code base on a blockchain, like Arweave, where the code is permanently on the blockchain and cannot be modified: those are some potential avenues. The main problem I see with all those avenues is the sub-agent problem, which is the AI, if it recognizes its weaknesses, can create a sub-agent that has the same goals as the original AI but none of the weaknesses, and it seems tricky to prevent the AI from doing that.

**0:12:42.7 Vael:** Would the sub-agent still be running on the same hardware? Because you said the safety is built into the hardware.

**0:12:48.5 Interviewee:** Right. So the AI could recognize, \"My goals are X, Y, Z. I\'m blocked partially from implementing my goals because I have all these hardware limitations and these back doors, so why don\'t I create a copy of me that runs on unrestricted hardware?\" And the obfuscation and encryption can prevent it from creating an exact copy, but it can\'t necessarily prevent it from constructing a sub-agent that\'s not a copy but has the same goals.

**0:13:24.2 Vael:** I see. And you haven\'t found a solution to this one, you said?

**0:13:29.0 Interviewee:** That\'s right.

**0:13:29.6 Vael:** Yeah. Do you think you\'ll ever find\... Do you think someone else will find a solution to this one?

**0:13:35.1 Interviewee:** Yeah, I\... Optimistically, yes. If we can\'t solve the sub-agent problem, then the entire alignment problem is probably impossible, right? The one thing to hope for if we can\'t solve the sub-agent problem is the AI has the same alignment problem, if it creates sub-agents, then it could worry that the sub-agents get out of its control, the sub-agents develop their own goals that are not aligned with the original AI, and so it refrains from making sub-agents. And so that\'s the fallback, that if it turns out that alignment is technically impossible, then it\'s also technically impossible for the AI itself, and so that\'s a kind of partial solution to the sub-agent problem, that maybe the AI won\'t dare to make sub-agents. But I hope that there\'s a better solution than that.

**0:14:32.4 Vael:** Yeah. Okay, so related to what the future looks like, do you think that we\'ll\... What time point do you think we\'ll get AGI, if you think we\'ll get AGI, which it sounds like you think we will?

**0:14:44.9 Interviewee:** Yeah, I definitely think we will, barring some major catastrophe, like a nuclear war or like a serious pandemic that sets us way back. Or I guess another potential catastrophe is some small group that\'s super worried about AGI and thinks it will be a catastrophe and does some drastic action that, again, sets us back multiple decades. So there are those scenarios. But I do think AGI is possible in principle, and we are certainly on track to achieve it. I\'m not a fan of trying to predict timelines, it could be any\... It\'s on the scale of decades, but whether it\'s two decades or 10 decades, I\'ve no idea.

**0:15:33.6 Vael:** Cool. And then how optimistic or pessimistic are you in your most realistic imagining of the future, for things going well or things going poorly?

**0:15:46.9 Interviewee:** I guess I\'m moderately pessimistic, not necessarily for human-aligned AGI, I do think that\'s somewhat plausible. But I think humans define our own interests too narrowly. I tend to think that our interests are actually a lot more connected to the broader interests of the whole biosphere. And if we are just on track to make humans really happy, and even potentially solve climate change, but we don\'t really take into account the effect we have on other species, the effect of deforestation\... Even things like farming is really destructive and unsustainable, yeah, I already mentioned deforestation. Disinfectants and so on have unpredictable consequences decades down the line. I don\'t think our current medical and agricultural regimes are sustainable on the scale of a century, say, and I think we would be better off optimizing for the health of the whole biosphere. And ultimately in the long term, that will end up optimizing for human happiness. But I don\'t think that corresponds to most people\'s goals at the moment. And so I worry that even if we align AI with narrow human interests, we will end up permanently wrecking the biosphere, and we\'ll pay serious consequences for that.

**0:17:25.6 Vael:** Interesting. One thing I can imagine is that as we advance further up the tech tree, renewable energy and food production will be much easier, and we won\'t actually have so much\... side effects on the environment or destruction of the environment.

**0:17:39.9 Interviewee:** Yeah, that would be great. The thing with renewable energy is it might be better than burning fossil fuels. It\'s certainly better from the perspective of climate. But making solar panels is very environmentally costly, you have to mine rare earths in China, there\'s an enormous amount of pollution and contamination that\'s really impossible to clean up on the scale of even centuries. And that\'s the same with wind power. Water, you\'re permanently taking out ground water that\'s not replaceable, except on a very long time scale. I think there\'s a tendency at the moment to view everything through the lens of climate, and that doesn\'t really take into account a lot of other potentially irreversible effects on the environment.

**0:18:42.0 Vael:** How worried are you about this compared to AI risks?

**0:18:46.0 Interviewee:** Well, it\'s not a kind of imminent existential risk of the type of a paperclip scenario. So on the scale of decades, I think unaligned AI is a more serious risk. But on the scale of centuries, I think environmental risks are really bad. One interesting read in this regard is the book \"Collapse\" by Jared Diamond. So he surveys human societies over all different historical periods, all different societies around the world and what caused them to collapse and what averted collapse in some success stories. Well, he doesn\'t make any strong conclusions, but one thing that leapt out at me from his stories is there\'s one common element of all the collapse stories, which is surprisingly deforestation. So I don\'t understand why, but all the societies that suffered a really disastrous collapse were the very same societies that completely decimated their forests, the most extreme case being Easter Island, where they cut down literally the last tree. And Diamond does not really explain why this might be the case. He does talk about how trees are used for a whole bunch of things that you might not think they\'re used for, but still, it doesn\'t completely explain it to me.

**0:20:28.6 Interviewee:** So my vague hypothesis is that there are all kinds of symbioses that we\'re just now discovering or are completely undiscovered. There\'s the gut microbiome, there\'s other microbiomes like the skin microbiome, there\'s the teeth and so on. And I think we don\'t appreciate at the moment how much plants and microbes and fungi and even viruses control our behavior. I think we will discover\... This is just a guess, I don\'t have strong evidence for it, but my guess is we\'ll discover in the coming decades that we have a lot less volition and free will than we think, that a lot of our behavior is heavily influenced by other species, in particular fungi and plants and microbes. It\'s certainly clear to me that those species would influence all aspects of animal behavior if they could. We\'re very useful for them to reproduce, to spread their seeds. And the only question is do they have the ability to influence our behavior? And given that many of them literally live inside us, I think they probably do.

**0:21:46.8 Vael:** Interesting. Well, so I\'m going to take us back to AI. \[chuckle\]

**0:21:51.9 Interviewee:** Yeah, sure, that was a big tangent.

**0:21:55.4 Vael:** \[chuckle\] So I was curious, when you were describing how your eventual optimistic scenario involves a whole bunch of people able to generate their own AGIs, presumably on safe platforms, and they kind of balance each other, I\'m like, wow, I know that in human history we\'ve gradually acquired more and more power such that we have huge amounts of control over our environments compared to 10,000 years ago. And we could blow up\... Use nuclear power to blow up large amounts of spaces. And I\'m like, wow, if these AGIs are pretty powerful, which I don\'t know how powerful you think they are, then that doesn\'t necessarily feel to me like the world is safe if everyone has access to a very powerful system. What do you think?

**0:22:41.9 Interviewee:** Yup, I agree that when you\'ve got a lot of powerful agents, then there\'s a lot more ways for things to go wrong. Nuclear weapons are an interesting example though, because the game theory governing nuclear exchanges is actually pretty safe. You\'ve got this mutually assured destruction that\'s pretty obvious to all the parties involved, and you\'ve got this kind of slow ladder of escalation that has many rungs, and nobody wants to get to the top rungs. And I think we\'ve demonstrated over 70 years now that\... There have been some close calls. But if you told somebody in 1950 that there would be 10 countries with nuclear weapons, and they\'d be dramatically more destructive than they were in 1950, people would not necessarily have predicted that humans would last very much longer, but here we are. I guess one worry is that not every form of weapon would have the same kind of safe game theory, like there\'s some suggestion that bio-weapons favor first strikes more than nuclear weapons do. Still, I think that having a big community of agents all with approximately the same amount of power, and they develop coalitions, they develop safety monitoring agencies that are made up of many agents that kind of make sure that no one agent has the ability to destroy everything.

**0:24:30.0 Interviewee:** I mean, that\'s kind of the way that humans have gone. We\'ve got central banking committees that kind of look after the overall health of the economy and make sure that no one institution is systemically important, or at least that the ones that are are heavily regulated. Then we\'ve got the IAEA which looks over atomic weapons and kind of monitors different weapons programs. As long as you believe that really destructive capacity will be detectable and that no one agent can just spin it up in secret, then I think the monitoring could turn out okay. I mean, what you might worry about is that somebody spins up\... Or some AI agent spins up another more powerful AI in secret and then unleashes it suddenly.

**0:25:32.7 Interviewee:** But that seems hard to do. Even now, if some company like Facebook or whatever wanted to develop an AI system completely in secret, I don\'t think they could do it and make it as powerful as existing systems. It really hurts you to be disconnected from the Internet, you have a lot less data that way, or you have stale data that comes from a cache off the Internet. Also being disconnected from the Internet itself is really hard, it\'s hard to make sure that your fridge is not trying to connect to your home WiFi. And that is only going to get harder; chips will have inbuilt WiFi connections. It\'s very hard to keep things totally off-grid. And even if you do it, those things are much weaker. And so as long as you have some kind of global monitoring, which isn\'t great, it feels very intrusive, it violates privacy. Ideally, that monitoring is kind of unobtrusive, it runs in the background, it doesn\'t bother you unless you\'re doing something suspicious, then I think things could turn out okay.

**0:26:45.1 Vael:** Interesting. Yeah, I think I have\... I was reading FHI\'s lists of close calls for nuclear \[catastophes\] and thinking about global coordination for things like the pandemic, and I\'m like, \"Ooh, sure we\'ve survived 70 years, but that\'s not very many in the whole of human history or something.\"

**0:27:01.5 Interviewee:** Yeah, it\'s not. And my scenario has the low probability of happening, maybe it\'s\... There are a few other optimistic scenarios. But maybe the total weight I give to all the optimistic scenarios is still kinda low, like 20%. So I think bad scenarios are more likely, but they\'re not certain enough that we should just give up.

**0:27:32.6 Vael:** Yes, I totally believe that. \[chuckle\] Yeah, so what convinced you to work on the alignment problem per se? And how did you get into it?

**0:27:42.8 Interviewee:** Yeah, so I have a friend, \[name\], who\'s been telling me pretty much constantly whenever we talk that this is the most important problem, the most important x-risk. And I kind of discounted her view for many years. It felt to me like we were\... Until recently, it felt to me like AI\... Superhuman AI was a long way off, and other risks were more pressing. I changed my mind in the last few years when I saw the pace of improvement in AI and the black box nature of it, which makes it more unpredictable. And that coincided\... In the time frame, that coincided with me getting tenure, so I have much more freedom to work on what I want. The only thing that gave me pause is I\'m not an engineer at heart, I\'m a scientist. My skills and interests are in figuring out the truth, not in designing technology. So I\'m still kind of looking for scientific aspects of the problem as opposed to design and engineering aspects. I do think I will find some portions of the alignment problem that fit my skills, but I\'m still figuring out what those are.

**0:29:15.3 Vael:** That makes sense. Yeah. How would you define the alignment problem?

**0:29:19.6 Interviewee:** Yeah, that\'s a super good question; that\'s actually a question I\'ve been asking other alignment researchers. I think it has several components. One component is the value loading problem, of once you\'ve decided what\'s in human interest, how do you specify that to an AI? I guess some people call that the outer alignment problem. Then before that, there\'s the question of\... The philosophical question of how do you even say what is in human interest? And I know some people think we need to make much more progress in philosophy before we can even hope to design aligned AI. Like I\'ve seen that view expressed by Wei Dai, for example, the cryptographer. My view is, yeah, we don\'t know exactly what we mean by in human interest, but we shouldn\'t let that stop us. Because philosophy is a slow field, it hasn\'t even made much progress in millennia. And we need to solve this quickly, and we should be happy with approximate solutions and try to make them better over time. And even if we don\'t know what is exactly in human interests, we can agree on what is certainly not in human interests and try to at least prevent those bad outcomes.

**0:30:39.6 Interviewee:** Okay, so those are two components. And then once you solve outer alignment, then there\'s what people call inner alignment, which is you\'re\... At least if it\'s a black box system, then you don\'t know what it\'s doing under the hood, and you worry that it develops some sub-goals which kind of take over the whole thing. So examples of that being: evolution designed humans to spread our genes but then to do that, it designed our brains to learn and generalize and so on and seek out food and power and sex and so on. And then our brains\... That was originally a sub-goal, but then our brains just want to do that, and our brains don\'t necessarily care about spreading our genes. And so evolution kind of failed to solve its alignment problem, or partially failed. That\'s an interesting one to me, because if you think on an evolutionary time scale, if we don\'t destroy ourselves, then evolution might end up correcting its course and actually designing some conscious fitness maximizers that do consciously want to spread their genes. And then those will outcompete the ones that are misaligned and just want the power and sex. And so I actually think evolution could end up staying aligned, it\'s just that it\'s slow, and so there might not be time for it to evolve the conscious fitness maximizers.

**0:32:25.8 Interviewee:** Yeah, anyway, so that\'s a worry, this inner alignment. And I think to solve that, we need to get off the black box paradigm and develop transparent AI, and a lot of people are working on that problem. So I\'m somewhat optimistic that we\'ll make big strides in transparency. What else? Okay, so if we solved all three of those, we\'d be in good shape. My instinct is to assume the worst, that at least one of those three problems is really hard, and we won\'t solve it, or at least we won\'t solve it in time, and that\'s why I focus on partial alignment, which is making sure that the AI we developed is loosely\... Loosely has common interests with us even though it might have some diverging interests. And so it doesn\'t want to completely destroy humans, because it finds us useful, and we don\'t want to completely destroy it, because we find it useful. Then you can kind of say that\'s already happening. Like in 2022, no machines could survive if all humans disappeared, very few humans could survive if all machines disappeared. And so we\'ve got this kind of symbiosis between humans and machines. I like that situation. It\'s not like 2022 is great, but I think we could gradually improve it. And we want to keep the symbiosis going, and we want to keep humans not necessarily even in a dominant position, but we want to prevent ourselves from getting in a really subservient position in the symbiosis.

**0:34:17.4 Vael:** Makes sense. Switching gears a little bit: if you could change your colleagues\' perceptions of AI, what attitudes or beliefs would you want them to have? So what beliefs do they currently have, and how would you want those to change?

**0:34:30.6 Interviewee:** Yup, that\'s a frustration that I think all alignment researchers have, that many of our colleagues are\... We think of them as short-sighted. Some of them just want to develop better AI, because it\'s an interesting problem, or because it\'s useful. Some of them want to solve short-term alignment issues like making algorithms less biased. And that\'s frustrating to us, because it seems like the long-term issues are way more important. They think of the long-term issues as something that\'s not really science, it\'s too speculative, it\'s too vague. They feel like even if the long-term issues are important, we will be able to solve them better if we learn by solving the short-term issues. I\'m not against people working on algorithmic bias, but I\'m frustrated that so many more people work on that than on long-term alignment. I do think the Overton window is shifting quite a bit. I think the increase of funding in the space would be\... Is already shifting things, and it could be used more effectively in the sense of giving a few academics really big grants would really catch their colleagues\' attention.

**0:36:00.0 Interviewee:** So it\'s kind of\... How should I put it? I\'m blanking on the word, but it\'s a kind of a cynical view to think that academics are motivated by money; many of us aren\'t. But at the end of the day, having a grant makes it easy to just focus on your research and not be distracted by teaching and administrative stuff. And so your colleagues really pay attention when one of their colleagues gets some big, flashy grant, and so I actually think that\'s a cheap way to shift the Overton window. Like take the top 10 or 20 math and computer science departments, and give one person in each department a giant grant\-- giant by academic standards, couple million dollars, so it\'s not actually much. That will really convince people that, \"Wow, long-term alignment is a serious field where you can get serious funding.\" So yeah, that would be my recommendation to funders. That\'s a pretty self-interested recommendation, because I intend to apply for a grant soon. But yeah, I think that would help. Let\'s see, did I answer your question?

**0:37:26.1 Vael:** I think so, yeah. What happens if some of these departments don\'t have anyone interested in working on long-term alignment?

**0:37:34.6 Interviewee:** Yeah, that\'s hard. Like at \[university\], I spent several months probing my colleagues for anyone who\'s interested. I didn\'t find anyone in the computer science Department, which was disappointing, because \[university\] has a great computer science department. I do think if you look more broadly in several departments you are likely to find one or a few people\... You could see already there\'s these big institutes, you have one now at Stanford, there\'s one at Berkeley, Cambridge, Oxford, and so on. So that\'s evidence that there\'re already a few people. And people talk to colleagues around the world, so it doesn\'t matter if there\'s nobody at school X, you fund the people that are interested. But the key is that they might not have a track record in alignment, like I\'m in this situation where I have no track record, my track record is in pure math. So somebody has to take a little bit of a leap and say, \"Well, I don\'t know if \[interviewee name\] will be able to produce any good alignment research, but there\'s a good chance because he\'s good at proving theorems, so let me give him a couple of million dollars and see what happens.\" That is a big leap and it might fail, but it\'s just like any venture funding, a few of your big leaps will be very successful and that\'s enough.

**0:39:16.8 Vael:** Yep, that makes sense. Yeah, I think \[name\] at \[other university\] is aiming to make an institute at \[other university\] as well, which is cool.

**0:39:25.2 Interviewee:** That\'s great. I\'ve been talking to my dean about doing this at \[university\] and he likes the idea but he is not really aware of how to fund it, and I\'m telling him there\'s actually a lot of funding for this stuff, but I don\'t personally know the funders.

**0:39:44.8 Vael:** Yeah, I think getting in contact with \[funding org\] seems like the thing to do.

**0:39:49.2 Interviewee:** Yep, good.

**0:39:50.2 Vael:** Yep. \[Name\] is one of the people in charge there. Great, so how has this interview been for you and why did you choose to jump on it? That\'s my last question.

**0:40:03.1 Interviewee:** Oh, it\'s been fun. I spent a while just thinking alone about some alignment stuff because I had no colleagues to talk to, so it\'s always great to find someone who likes to talk about these issues.

**0:40:24.6 Vael:** Did you know that I was already interested in long-term alignment?

**0:40:28.2 Interviewee:** I think I saw your name at the\... Did you participate in the SERI Conference?

**0:40:33.2 Vael:** I did.

**0:40:34.9 Interviewee:** Yeah, so I saw your name there and so I was sort of aware of your name but I didn\'t know anything about your interests.

\[\...some further discussion, mostly about the interviews\...\]

**0:42:22.4 Interviewee:** Okay. Cool, Vael. I should jump on another call, but it was great to chat and yeah, feel free to follow up if you want.

**0:42:32.1 Vael:** Will do. Thanks so much.

**0:42:33.7 Interviewee:** Okay. Take care.

individuallyselected_84py7-by Vael Gates-date 20220318.md


# Danijar Hafner on Gaming Our Way to AGI. Danijar Hafner on procedural game generation for reinforcement learning agents by Jeremie Harris on the Towards Data Science Podcast

Until recently, AI systems have been narrow — they’ve only been able to perform the specific tasks that they were explicitly trained for. And while narrow systems are clearly useful, the holy grain of AI is to build more flexible, general systems.

But that can’t be done without good performance metrics that we can optimize for — or that we can at least use to measure generalization ability. Somehow, we need to figure out what number needs to go up in order to bring us closer to generally-capable agents. That’s the question we’ll be exploring on this episode of the podcast, with Danijar Hafner. Danijar is a PhD student in artificial intelligence at the [University of Toronto](http://learning.cs.toronto.edu/) with [Jimmy Ba](https://scholar.google.com/citations?user=ymzxRhAAAAAJ&hl=en&oi=ao) and [Geoffrey Hinton](https://scholar.google.com/citations?user=JicYPdAAAAAJ&hl=en&oi=ao) and researcher at [Google Brain](https://research.google/teams/brain/) and the [Vector Institute](https://vectorinstitute.ai/).

Danijar has been studying the problem of performance measurement and benchmarking for RL agents with generalization abilities. As part of that work, he recently released Crafter, a tool that can procedurally generate complex environments that are a lot like Minecraft, featuring resources that need to be collected, tools that can be developed, and enemies who need to be avoided or defeated. In order to succeed in a Crafter environment, agents need to robustly plan, explore and test different strategies, which allow them to unlock certain in-game achievements.

Crafter is part of a growing set of strategies that researchers are exploring to figure out how we can benchmark and measure the performance of general-purpose AIs, and it also tells us something interesting about the state of AI: increasingly, our ability to define tasks that require the right kind of generalization abilities is becoming just as important as innovating on AI model architectures. Danijar joined me to talk about Crafter, reinforcement learning, and the big challenges facing AI researchers as they work towards general intelligence on this episode of the TDS podcast.

Here were some of my favourite take-homes from the conversation:

-   The Crafter environment includes a wide range of achievable goals, each of which involves performing a specific in-game task for the first time. Some of these tasks are simple, and can be achieved fairly consistently by current state-of-the-art RL agents (for example, finding sources of food and harvesting them). But some are more challenging, because they involve dependencies on other tasks: a “collect iron” task can only be achieved once a “make stone pickaxe” task has already been completed, for example. The result is a fairly deep tech tree that can only be completed by agents that have learned to plan.
-   Because different tasks require different abilities, Crafter environments can give developers a way to profile their RL agents. By measuring an agent’s average performance across the full distribution of achievable tasks, they obtain a fingerprint of its capabilities that indicates the extent to which it’s picked up skills like planning and exploration, which are closely linked to generalization.
-   One of the challenging aspects of developing benchmarks for machine learning is to ensure that they’re tuned to the right level of difficulty. Good benchmarks are challenging to master (so that they can motivate and direct progress in the field), yet tractable (so that developers have enough signal to iterate and improve their models).
-   One interesting debate about the future of AI has to do with the extent to which further progress will come from dramatic improvements to algorithms, or from the increased availability of compute resources. Increasingly, we’ve seen cutting-edge results in reinforcement learning come from model-based systems that make heavier use of compute than their model-free counterparts, and progress in RL has closely tracked increases in compute budgets (e.g. MuZero and EfficientZero). To some, that’s an indication that current AI techniques might be sufficient to reach human-level performance, with relatively minor tweaks, if they’re only scaled up with more compute. While Danijar sees some merit to this argument, he does think that there remain fundamental advances in algorithm design that we’ll have to make before reaching human-level AI that go beyond leveraging raw compute horsepower.

Danijar Hafner - Gaming our way to┬áAGI-by Towards Data Science-video_id Bgz9eMcE5Do-date 20220112.md