# Interview with AI Researchers NeurIPSorICML_a0nfw by Vael Gates

**Interview with a0nfw, on 3/18/22**

**0:00:02.3 Vael:** Alright. So my first question is, can you tell me about what area of AI you work on in a few sentences?

**0:00:08.7 Interviewee:** Yes. More recently I\'ve been working on AI for mathematical reasoning and applications in navigation, mathematical navigation.

**0:00:18.1 Vael:** Great. And then, what are you most excited about in AI and what are you most worried about? In other words, what are the biggest benefits or risks of AI?

**0:00:25.8 Interviewee:** The biggest benefit of AI as a tool that now we have and that sort of works, is that it just enabled a lot of new applications when we are able to define goals for it. The broad story that I\'m pursuing recently is like, suppose you have an AI expert for a certain domain, then how can you leverage that expert to teach people to be better at the same domain? And the reason why we\'re working more on mathematical education, it\'s exactly because the mathematical domains are easier to define as a formal problem. Training solvers for interesting mathematical domains is not possible with AI tools, that \[hard to parse\] not be so much 10 years ago. In a lot of other fields, it\'s also been the case that we as humans knew what we wanted and wrote down a bunch of heuristics to accomplish certain goals. But we were always sure that those weren\'t the best, and now AI is allowing us to replace a lot of those by just better algorithms.

The main worry is exactly when we don\'t know exactly what the goal is, or when we don\'t know how to specify it. Or in a lot of cases, in research, people have been stuck with these proxy tasks that are supposed to represent some kind of behavior that we want, but a lot of people are just unsure of what exactly what we want out of these systems. It\'s a worry for me that a lot of people are spending a lot of time and resources on those problems without knowing exactly what to expect. I come from sort of a systems background, so I\'m more used to people having proxy tasks that are very directly related to the actual task. For example, in compilers, usually one goal is, make programs faster, and then you can create benchmarks. \"Okay, here is the set of programs, let\'s run your compiler and it optimize\[?\] and see how long it takes.\" And you can argue about whether those programs are representative of what actual programs that people will run it on, if the benchmark that I have is representative or not, but the goal was very clear, it\'s to make things faster. Now in language for example, in NLP, we have a lot of these tasks about language understanding and benchmarks that try to capture some form of understanding. But it\'s not entirely clear what the goal is, if we solve this benchmark\-- and this is being revised all the time. People propose a benchmark that\'s hard for current-day models, then a few months later, someone comes up with a solution. Then people say, \"Oh, but that\'s actually not exactly what we wanted \'cause look, the model doesn\'t do this other thing.\" And then we enter this cycle of refining exactly what\'s the problem and developing models, but without a clear goal in a lot of cases.

**0:03:47.7 Vael:** Got it. Okay, so that question was, what are you most excited about AI and what are you most worried about in AI? Could you summarize the culmination of all that?

**0:03:57.1 Interviewee:** Yeah. It\'s centered around goals. So AI lets us pose richer and new kinds of goals more formally and optimize for those. When those are clear, those are cases where I\'m very excited about. When they\'re not clear, then I\'m worried about it.

**0:04:16.3 Vael:** Got it. Cool, that makes sense. Awesome. Alright, so focusing on future AI, putting on a science fiction forecasting hat, say we\'re 50 plus years into the future. So at least 50 years in the future, what does that future look like?

**0:04:32.8 Interviewee:** It can look like\... a lot of\... it can go in a lot of ways, I think. I think, in one possible future\-- do you want me to just list the possible futures?

**0:04:48.1 Vael:** I think I\'m most interested in your realistic future, but also like optimistic, pessimistic; it\'s a free-form question.

**0:04:57.8 Interviewee:** Okay. Yeah, maybe optimistically what would happen is that AI lets us solve problems that are important for society and that we just don\'t have the right tools to at this point. I think a lot of exciting applications in\... In language, we can imagine some interesting applications where a lot of computer interfaces become much easier, just for doing complex tasks, to specify a natural language. Like programming right now gives a lot of power to people, but it also takes a lot of time to learn, so it would be awesome to enable a lot more people to automate tasks using natural language. So in one future, a lot of those applications will be enabled and things will be great. In another future, which is actually\-- it\'s probably going to be a combination of these things, but a very likely future, which is already kind of happening, but maybe at a smaller scale than is possible, is that people will start replacing existing tools and systems with AI for not very clear reasons and get not very clear outcomes. And then we don\'t really know exactly how they\'re misbehaving in a lot of cases. And the thing is that the incentives for deploying the systems are at odds with broader societal goals in a lot of cases. So, for example, private companies like Google and Facebook, they have all the incentives possible to deploy these systems to optimize their metrics that ultimately correlate with revenue. I don\'t know.

Facebook has a lot of metrics which ultimately relate to both how much time people spend on the platform, and that translates to how much money they make. And they basically have thousands of engineers trying different combinations of features and things to suggest users to do, and like very small tweaks a lot of times to try to optimize for those through metrics. And a lot of that process became much faster with AI because now you can take much more fine-grain decisions for individual users. And sure, we can look at the metrics and see that, \"Yeah, they keep improving, that\'s a new tool that now exists and enables them to optimize for this goal.\" At the same time, it\'s not exactly to clear to me what is happening, that AI is choosing things for individual users. It might be like closing people off in their bubbles, it might be\... all the things we know about, but also potentially a lot of other things, other behaviors. Exactly because they\'re not the same behavior for everyone, that also makes it harder to study and understand. So I like the future, is that AI will replace a lot of these systems, but that will involve people losing control of exactly what it is doing. And that will probably come before us having a good understanding of what\'s going on, just because there is already an incentive to deploy those systems, like a financial incentive.

**0:08:40.8 Vael:** Yeah, interesting. You\'re making a lot of the arguments that I often make later in this interview. Alright. This next thing is, so people talk about the promise of AI, by which they mean many things, but one thing that I think they mean, and that I\'m talking about here, is that the idea of having a very general, capable system, such that the systems would have cognitive capacities that we could use to replace all current-day human jobs. Which we could or could not, but the cognitive capacities do that. And I usually think about this in the frame of like, here we have\... in 2012, we have AlexNet and the deep learning revolution, and then 10 years later, here we are, and you\'ve got systems like GPT-3, which have some weirdly emergent capabilities like text generation and language translation and math and coding and stuff. And so one might expect that if we continued pouring all the human effort that\'s been going into this, with nations competing and corporations competing and lots of young people and lots of talent and algorithmic improvements at the same rate we\'ve seen and hardware improvements, maybe we\'ll get optical or quantum computing, then we might scale to very general systems. Or we might not, and we might hit some sort of ceiling and need to paradigm shift. But my question is, regardless of how we get there, do you think we\'ll ever have very general systems like a CEO or a scientist AI, and if so, when?

**0:11:00.9 Interviewee:** Yeah. The thing that has been happening with these large language models, that they objectively become more and more capable, but we at the same time reassess what we consider to be general intelligence. That will probably keep happening for quite a while, I would say at least like 10-15 years, just because\...I don\'t think we know exactly what\... Human intelligence is still very poorly understood in a lot of ways. It\'s not clear that higher intelligence is possible without some of the shortcomings that human intelligence has. So I think that cycle of getting more powerful systems but also realizing some of their shortcomings and saying, \"Oh, maybe\...\" Basically shifting the goal and what is it that we consider general intelligence will probably keep happening.

**0:12:07.1 Vael:** Yeah, my question is, do you think that we\'ll get systems like a CEO AI or a scientist AI or be able to replace most cognitive jobs at some point, and when that will be?

**0:12:18.6 Interviewee:** Yeah. Let\'s see. We\'ll probably have systems that can act like a CEO or a scientist. Whether they\'ll have the same goals that a CEO or a scientist have, like a human CEO or a human scientist have in the real world, that is a different question, which is not that clear to me.

**0:12:47.0 Vael:** Interesting. Yeah. So there\'s the question of whether AIs develop like consciousness, for example. So I\'m assuming not consciousness, and I\'m like, \"Okay, say we have a system that can do multi-step planning, it can do models of other people modeling it, it\'s interacting with humans by text and video, or whatever,\" and I\'m like, \"Alright, I need you to solve cancer for me, AI,\" or, \"I want you to run this company,\" and we put in its goal, optimization function so that it just does that at some point. Yeah.

**0:13:24.4 Interviewee:** Yeah. You\'re right. So you asked about the cognitive capability. I think that will probably be there in like 30-40 years, would be my estimate. For at least a subset of what it means to run a company. Now, my question is, if we have a system that does that, is it also possible for the system to simply obey everything we tell it to do? Because, of course, a person is capable of doing that, but you wouldn\'t expect to tell the person, \"Okay, go do that.\" They might complain. They might say \"No, I\'m in a bad mood today,\" or something like that.

**0:14:03.3 Vael:** Yeah, and I don\'t even think you could tell any given human\... Like if someone is like, \"Alright, Vael, go be the CEO of a company,\" I\'d be like, \"What? I don\'t know how to do that.\"

**0:14:09.9 Interviewee:** Exactly, exactly. Yeah.

**0:14:12.0 Vael:** Yeah, yeah. Interesting. Okay, so that leads into my next set of questions. So imagine we have a CEO AI and I\'m like, \"Alright, CEO AI, I wish for you to maximize profits, and try not to run out of money and try not to exploit people and try to avoid side effects.\" And so currently, this would be very technically challenging for a number of reasons. One reason is that we aren\'t very good at taking human values and preferences and stuff and putting them into mathematical formulations such that AI can optimize over. And so I worry that AI in the future will continue to do this, and then we\'ll have AI that continue to do what we tell them to do instead of what we intend them to do, but as our expressions get more and more ambiguous, and higher scale and operating on more of the world. And so what do you think of the argument, \"Highly intelligent systems will fail to optimize exactly what their designers intended them to, and this is dangerous?\"

**0:15:22.1 Interviewee:** Yeah, if I understand what you\'re saying, that is very clear to me. That\... specifying\... If you just hope to give a short description of what you want, you\'ll most likely fail.

**0:15:35.9 Vael:** Even a long definition would be fine. I just want there to be any system of putting what we want into the AI.

**0:15:46.0 Interviewee:** Yeah, so I think there might be a safe way to do that, but it might require a lot of interaction and will be hard. I have thought a little bit about that, because I was thinking about the question of how do we\... So if we want to use these systems for interfacing with people, then we have to let them handle ambiguity in a way that\'s similar to how people do. And in cognitive science, there is a lot of knowledge about the mechanisms that people use to resolve ambiguity, like there is context, prior knowledge, common ground, and all that. Some of that could be replicated now to fully align the way that AI resolves ambiguity with how people do it. We\'ll also probably take a lot of work in understanding how people do it, which is not there yet.

**0:16:50.3 Vael:** Interesting. Interesting. Do you think that we\'ll have to know enough about human psychology? Like, what fields need to advance in order for us to have systems that do what we intend them to do?

**0:17:06.1 Interviewee:** Yeah, if you\'re talking about aligning a system with human values, part of that problem is understanding human values and like how people agree on values. And there\'s just so much that we assume that we don\'t have to tell people, it\'s assumed that they were created in the same way and exposed to the same contexts. All that goes away with AI, or at least most of it. And I\'m not an expert in these fields, so I might not know the exact names. I guess pragmatic reasoning is one thing that we have a lot of high-level insights on how it works, but to operationalize it into an NLP system, for example, we\'re still very far away from that.

\[Person\] worked on this model for pragmatic reasoning, which can explain very neatly some very simple cases. Like, you have a few words and a few people, and some words describe these people. You have a three-by-three matrix that predicts exactly what people would guess the word is referring to with very high accuracy. But just extending that to a sentence and the three people to be an image is a very complicated problem. So I think that is something that would need to advance a lot. Other fields of psychology that try to understand ambiguity like of cognitive psychology. Yeah. I don\'t know their names, but I think\... Oh well, yeah. I guess another one in linguistics that is relevant for this is human convention formation. Like when we are talking to a person over time, we develop these partner-specific conventions. That\'s very natural, enables very efficient communication. And if you\'re writing a description of what you want in a long form to an AI\-- Oh, if you\'re doing so for a person, you assume that the person is picking up on what we want and forming the same conventions over time. So at first you might say, \"I want to optimize the revenues for XYZ company.\" And then later you might say \"the company,\" dropping the XYZ, but you assume that\-- the only company that I mentioned was XYZ, so the only interpretation possible for a company is attribute the XYZ company. Well, that\'s because humans form conventions that way, but if you have kind of an adversary AI, then suddenly that\'s not an assumption you can build off of anymore. So the ways in which humans communicate with conventions, with ambiguity and all the computational tools that we need to do that are open problems, if that makes sense.

**0:20:05.3 Vael:** Yeah. That does seem right. Thanks. Alright, so this next question is still about the CEO AI. So imagine we have the CEO AI that is capable of multi-step planning and has a model of itself in the world. So it\'s modeling other people modeling it because that seems really important in order to have a CEO. So it\'s making these plans for the future and it\'s trying to optimize for profit with the constraints I\'ve mentioned. And it\'s noticing that some of its plans fail because the humans shut it down. And so we built into this AI and its loss function that it has to get human approval for things because it seems like a basic safety measure. (Interviewee: \"Sorry, it has to get what?\") It has to get approval for any action from humans, like a stakeholder-type thing. And so the humans have asked for a one-page memo from the AI just to discuss what it\'s doing. And so the AI is thinking about what to put in this memo, and it\'s like, \"Maybe I should omit some relevant information that the humans would want because that would reduce the chance that the AIs \[note: I meant to say \"humans\"\] would shut me down and increase the likelihood of my plans succeeding, of optimizing my goal.\" And so in this case, we\'re not building self-preservation into the AI, what we\'re doing is having an AI that\'s an agent that is optimizing over a goal. And so instrumentally, it has the goal of self-preservation. So what do you think of the argument, \"Highly intelligent systems will have an incentive to behave in ways to ensure that they are not shut off or limited in pursuing their goals, and this is dangerous?\"

**0:21:30.7 Interviewee:** Yeah. I think if we get systems that have that capability to do the things that we said in the beginning, like planning, in the most extreme form, then I agree with this statement.

**0:21:44.6 Vael:** Interesting. Do you think we\'ll ever\-- hm, so\... We were talking about whether you thought that we would get to that point. What was the answer to that?

**0:21:54.9 Interviewee:** The answer is that I think so, although I\'m not sure to what extent. I\'m not sure if the most powerful possible version will actually happen in 30-40 years. But I think there\'s, there\'s like a small chance, but\-- it\'s not impossible.

**0:22:11.1 Vael:** Interesting. So yeah, within our lifetimes. Yeah, so I\'m worried about the possibility that we don\'t get the alignment problem right per se. And we end up with systems that are now optimizing against humans, which seems bad, especially if they are smarter than humans. Yeah. How likely do you think this sort of scenario would happen?

**0:22:34.7 Interviewee:** The scenario of having super-capable AI that is---

**0:22:38.3 Vael:** That has an instrumental incentive to do self-preservation or other instrumental incentives like power-seeking or acquiring resources or improving itself.

**0:22:48.1 Interviewee:** Yeah, I think it is small but not zero.

**0:22:57.9 Vael:** Yeah. How bad do you think that would be if such a thing happened?

**0:23:03.4 Interviewee:** Yeah, I think pretty bad.

**0:23:05.9 Vael:** Pretty bad. Yeah. Like what would happen?

**0:23:10.4 Interviewee:** Yeah, it\'s not exactly clear to me how that would look like. But it is a little scary to think about humans losing control in some ways. The worry is exactly that we can\'t\... like if we\'re not under control, we can\'t know exactly what comes.

**0:23:30.6 Vael:** Yeah. That seems true. Have you heard of the\-- What does AI safety mean to you? That\'s my first question.

**0:23:39.3 Interviewee:** AI safety. I understand it as a field that\... When I think of AI safety, the first defining problem that comes for me is AI alignment. This question, \"How do you specify your goals in a way that aligns with human values?\" and all that.

**0:24:00.6 Vael:** That makes sense. Yeah. When did you start learning about AI alignment or when did you start caring about it?

**0:24:05.9 Interviewee:** Ah, it was mostly when I came to the \[position\] at \[university\], because I really wasn\'t that much of an AI person before. I kind of incidentally got into AI because it gave me tools to think about problems that were hard before. And I happened to get a bit more involved with the AI safety community. \[Friend\'s name\], you might know how him. (Vael: \"Ah! Nice.\") We\'re very close friends. We lived together for almost three years, and had a lot of conversations.

\[short removed segment, discussing situation and friend\]

**0:25:09.2 Vael:** Yeah. Some visions that I have, why it could be bad, is if we have an AI that\'s like very powerful, then if it\'s incentivized to get rid of humans, then maybe it\'s intelligent enough to do synthetic biology against humans, or nanotechnology or whatever, or maybe just increase the amount of pollution. I don\'t know. I just feel like there\'s ways in which you could make the environment uninhabitable for humans via like putting things in the water or in the air. It\'d be pretty hard to do worldwide, but seems maybe possible if you\'re trying. Yeah. Also we\'ve got like misuse and AI-assisted war, and like maybe if we put AI in charge of like food production or manufacturing production, then we can have a lot of correlated supply chain failures. That\'s another way, but I don\'t know if that would kill everyone of course, but that could be quite bad.

**0:26:12.5 Interviewee:** Yeah, that is something that I worry more about in the short-term, which is AI in the hands of\... Even currently, AI can already be pretty bad with like spreading misinformation. That certainly happened here in \[country\], but in \[other country\] in the last elections, where this really bad president got elected, he used a lot of misinformation campaigns funded by a lot of people. And it\'s super hard to track because it\'s on WhatsApp and it\'s like the double-edged sword, the sword of privacy. It aligns with things that we care about, but which also allows these campaigns to run at massive scale and not be tracked. Even currently, AI personalizing wordings and messages to different people at large scale could be really pretty \[hard to parse, \"good\"?\].

**0:27:20.1 Vael:** Got it. Do you work on AI alignment stuff?

**0:27:26.0 Interviewee:** So currently, I\'m working more on the education side of research. I still keep in the back of my head some of the problems about the ambiguity thing. We did have one paper last year that started building some of those ideas but I\'m not actively working on that. But it\'s still like a realm of problems that I think is important for a number of reasons.

**0:27:56.2 Vael:** Yeah. That makes sense. What would cause you to work on it?

**0:28:04.4 Interviewee:** Sorry, what what would make me work on it?

**0:28:08.3 Vael:** Yeah, like in what circumstance would you be like: Whelp, it is now five year later, it is now one year later, and I happen to be somehow working on some element of the AI alignment problem. Or AI safety or something. How did that happen?

**0:28:27.8 Interviewee:** Yeah, it\'s interesting. From my trajectory, I have always tried to work on the thing that I feel is not being done and that I\'m in a certain position to do. And that has over time shifted towards education for a number of reasons. And I don\'t know exactly where that comes from, but I still feel that I\'m more able to contribute there than with other AI problems. Although I guess that kind of thing is hard to assess, but it\'s still my feeling on when deciding what things to work on.

**0:29:19.9 Vael:** Hm, so it sounds like there\'s maybe something like\... Okay, so I don\'t quite know if it\'s a matter of whether the education bit is neglected by the world, or whether it\'s like, you feel like it would be a better fit per se, or both?

**0:29:35.6 Interviewee:** Yeah, it\'s the combination of both. Like if it\'s neglected. It is important, important in the short-term. And I think there are like things that I could do in the next five years. If the plan that I have for the \[position\] I have works out, I think it would be great. Like we would have tools that people would be able to use, and they need technical advances that are in the intersection of things that I care about or know for very weird reasons. But I feel like I\'m in a position to make progress there and very immediately. So yeah, I just feel very urged to see how that goes.

**0:30:21.3 Vael:** Yeah. Makes sense. Yeah, do you ever think you\'ll go get into the AI alignment thing or you\'ll just be like, \"Woo, seems good for someone to work on it?\" I get that\-- that\'s an impression that I get.

**0:30:32.6 Interviewee:** Uh huh, yeah. I think I\'m getting a taste of it through \[friend\]. We actually wrote this project together two months ago. Okay, I actually met with \[friend\] to discuss this a few times but I have no idea where it\'s exactly came from, I think from someone at LessWrong that posted like a challenge. So it\'s a program called ELK, which I forgot even what ELK means, but I met with \[friend\] and we ended up coming up with some ideas, and he submitted it. \[\...\]

**0:31:16.9 Interviewee:** So I guess I\'m getting a little bit of flavor of the kind of work that that entails. I still don\'t feel exactly comfortable in doing it myself just because it requires this very hypothetical and future reasoning that I\'m not used to, not that I couldn\'t. But I understand that thinking about that kind of problem today requires you to think of systems that we can\'t use today and we can\'t run and test today. And I\'m very used to ideas that I can write down the code and see how they work and then make progress from there. So I don\'t know, my opinion and comfort might change.

**0:32:18.5 Vael:** Yeah, I also encountered this where I was like, \"I don\'t wanna think about far-future programs or far-future things, where there\'s no feedback loops and it\'s very hard to tell what\'s happening and there\'s only a limited number of people, and then\-- now there\'s money, but there wasn\'t necessarily very much money before\-- and there\'s not really journals, and everything\'s pre-pragmatic.\" And then, I think I was won over by the argument that, I don\'t know, I think existential risk is probably the thing that I\'ll just work on for most of my life because it seems like really important, and like, probable. But then I tried technical research for a while, and I thought it was not my cup of tea. And then I got into AI community building, which is what I\'m kind of currently doing, and trying to get more people to become AI technical researchers. But not me! It\'s been a good fit in some sense.

**0:33:10.1 Interviewee:** I see, that makes sense.

**0:33:11.1 Vael:** Yeah, and I think it\'s pretty interesting as well. Well, the current thing I\'m doing is talking to AI researchers and being like, \"What do you think of things? What about this argument?\" \...Cool. Well, we\'ve got a little bit of time left, so, see if I can ask some of my more unusual questions, which are\... How much do you think about policy and what do you think about policy related to AI?

**0:33:34.6 Interviewee:** Yeah, I think I\'ve\... I think a lot about policy in general, about just related to AI. I\'m currently at this phase in my life where I think a lot of the immediate problems that we have, unfortunately in a sense, we\'ll have to resolve by good politics, and I\'m not exactly sure yet how to do that. Because one other thing that I think about, when I think about AI safety, is that one thing is to have the technical ability to do certain things. The other is to get the relevant players to implement them. So, if there is an AI safety mechanism that can like limit the reach of Google\'s super AI that will do all things, but it also means that it will decrease revenue to some extent\... I\'m not sure the decision that they will make which would be to implement the safe version. Think about VC\'s that are sitting in these boards and pushing decisions for the company. I\'m not sure if all of them actually care about this and realize that it\'s a problem. So I think part of the solution is also on the political side, of like us as a society sitting down and deciding, \"What do we gotta do?\" More than just the technical proposal. I\'ve decided that this year, I will try to have some political involvement in \[country\], which is a place where I can. I\'m currently in the planning phase, but I\'ve committed to doing something this year, because we have elections this year.

**0:35:39.2 Vael:** Interesting, yeah. I think AI governance is very, very important. I think the community\-- the AI safety community also, like\... or, I don\'t know, the Effective Altruism community \[hard to parse\] a lot in their listings. Alright, question: if you could change your colleagues\' perceptions of AI, what attitudes or beliefs would you want them to have? So what beliefs do they currently have and how would you want those to change?

**0:36:05.9 Interviewee:** I\'m not sure if this is the kind of attitude that you\'re talking about, but one attitude that I really don\'t like and is very prevalent in papers is that people kind of self-identify with the method that they\'re proposing so much that they call it \"our method\". In the sense that the goal should not necessarily be to show off better numbers, and show a curve where the color that\'s associated with you is like higher than the other. But it should really be to understand what\'s going on, right? If we have a system that has some sort of behavior that\'s desirable, that\'s awesome, and we should also look at how it maybe doesn\'t. Like people include a lot of cases in their papers where, \"Oh look, the model is doing this.\" But then when we actually try out the model, it\'s very easy to find cases where it doesn\'t or it fails in weird ways. And the reluctance to put that comes from this attachment from like, \"Oh, this is like me, but in form of a method that is written in this paper.\" And I wish that people didn\'t have that attitude, that they were more detached and scientific about it.

**0:37:24.9 Vael:** Yeah, makes sense. Cool. And then my last question is, why did you choose this interview, and how has this interview been for you?

**0:37:32.6 Interviewee:** Yeah, it\'s been really fun. I had thought of some of the questions before, and not of others. Yeah, the exercise that I maybe haven\'t done that much is to think super long-term, like 50-100 years in the future. I thought about it some but I don\'t think I have the right tools yet to think about them, so I get in this hard-to-process state. Yeah, so I came mostly out of curiosity, I didn\'t know exactly what to expect. If you don\'t mind sharing, but if you do, that\'s okay, how did you pick the people to interview?

**0:38:28.3 Vael:** Yeah, so this is a pretty boring answer, but I originally selected people who\'d submitted papers to NeurIPS or ICML in 2021, and like yeah, then some proportion of people replied back to me.

**0:38:44.5 Interviewee:** I see.

**0:38:47.2 Vael:** Yeah, great, cool. That makes a lot of sense. Thanks you so much for your time. I will send you the money and some additional resources if you\'re curious about it. I\'m sure \[friend\] has already shown you some, but if you\'re curious. And thanks so much for doing this.

**0:39:00.7 Interviewee:** Yeah, no, definitely. Thank you for the invite, this was fun.

**0:39:05.1 Vael:** All right. Bye.

**0:39:05.8 Interviewee:** Bye.

NeurIPSorICML_a0nfw-by Vael Gates-date 20220318.md


README-by Vael Gates-date 20220509

# Interview with AI Researchers by Vael Gates

Table of Contents
=================

**[Table of Contents](#table-of-contents) 1**

**[Interview Information](#interview-information) 2**

> [Individually-selected](#individually-selected) 2
>
> [NeurIPS-or-ICML](#neurips-or-icml) 2

**[Intended Script (all interviews)](#intended-script-all-interviews) 3**

**[Post-Interview Resources Sent To Interviewees](#post-interview-resources-sent-to-interviewees) 5**

> [Master list of resources](#master-list-of-resources) 5

**[Informal Interview Notes](#informal-interview-notes) 10**

> [Thoughts from listening to myself doing these interviews](#thoughts-from-listening-to-myself-doing-these-interviews) 10
>
> [Content analysis](#content-analysis) 11

Interview Information
=====================

These interviews are associated with the LessWrong Post: [[Transcripts of interviews with AI researchers]{.ul}](https://www.lesswrong.com/posts/LfHWhcfK92qh2nwku/transcripts-of-interviews-with-ai-researchers).

(Please do not try to identify any interviewees from any remaining peripheral information.)

###

### Individually-selected

"Five of the interviews were with researchers who were informally categorized as 'particularly useful to talk to about their opinions about safety' (generally more senior researchers at specific organizations)."

-   7ujun

-   zlzai

-   92iem

-   84py7

-   w5cb5

###

### NeurIPS-or-ICML

"Six of the interviews were with researchers who had papers accepted at NeurIPS or ICML in 2021."

-   a0nsf (this is the interview in which I most straightforwardly get through my questions)

-   q243b

-   7oalk

-   lgu5f

-   cvgig

-   bj9ne (language barriers, young)

Intended Script (all interviews)
================================

There was a fixed set of questions that I was attempting to walk people through, across all of the interviews. It's a sequence, so I generally didn't move onto the next core question until I had buy-in for the previous core question. The core questions were: "do you think we'll get AGI" (if yes, I moved on; if not I interacted with the beliefs there, sometimes for the entire interview), "\[alignment problem\]", and "\[instrumental incentives\]". I was reacting to the researchers' mental models in all cases. I was trying to get to all of the core questions during the allotted time, but early disagreements often reappeared if the interviewee and I didn't manage to reach initial agreement. I prioritized the core questions, and brought other questions up if they seemed relevant.

The questions (core questions are highlighted):

-   "What are you most excited about in AI, and what are you most worried about? (What are the biggest benefits or risks of AI?)"

-   "In at least 50 years, what does the world look like?"

-   "When do you think we'll get AGI / capable / generalizable AI / have the cognitive capacities to have a CEO AI if we do?"

    -   Example dialogue: "All right, now I\'m going to give a spiel. So, people talk about the promise of AI, which can mean many things, but one of them is getting very general capable systems, perhaps with the cognitive capabilities to replace all current human jobs so you could have a CEO AI or a scientist AI, etcetera. And I usually think about this in the frame of the 2012: we have the deep learning revolution, we\'ve got AlexNet, GPUs. 10 years later, here we are, and we\'ve got systems like GPT-3 which have kind of weirdly emergent capabilities. They can do some text generation and some language translation and some code and some math. And one could imagine that if we continue pouring in all the human investment that we\'re pouring into this like money, competition between nations, human talent, so much talent and training all the young people up, and if we continue to have algorithmic improvements at the rate we\'ve seen and continue to have hardware improvements, so maybe we get optical computing or quantum computing, then one could imagine that eventually this scales to more of quite general systems, or maybe we hit a limit and we have to do a paradigm shift in order to get to the highly capable AI stage. Regardless of how we get there, my question is, do you think this will ever happen, and if so when?"

-   "What do you think of the argument 'highly intelligent systems will fail to optimize exactly what their designers intended them to, and this is dangerous'?"

    -   Example dialogue: "Alright, so these next questions are about these highly intelligent systems. So imagine we have a CEO AI, and I\'m like, \"Alright, CEO AI, I wish for you to maximize profit, and try not to exploit people, and don\'t run out of money, and try to avoid side effects.\" And this might be problematic, because currently we\'re finding it technically challenging to translate human values preferences and intentions into mathematical formulations that can be optimized by systems, and this might continue to be a problem in the future. So what do you think of the argument \"Highly intelligent systems will fail to optimize exactly what their designers intended them to and this is dangerous\"?

-   "What do you think about the argument: 'highly intelligent systems will have an incentive to behave in ways to ensure that they are not shut off or limited in pursuing their goals, and this is dangerous'?"

    -   Example dialogue: "Alright, next question is, so we have a CEO AI and it\'s like optimizing for whatever I told it to, and it notices that at some point some of its plans are failing and it\'s like, \"Well, hmm, I noticed my plans are failing because I\'m getting shut down. How about I make sure I don\'t get shut down? So if my loss function is something that needs human approval and then the humans want a one-page memo, then I can just give them a memo that doesn\'t have all the information, and that way I\'m going to be better able to achieve my goal.\" So not positing that the AI has a survival function in it, but as an instrumental incentive to being an agent that is optimizing for goals that are maybe not perfectly aligned, it would develop these instrumental incentives. So what do you think of the argument, \"Highly intelligent systems will have an incentive to behave in ways to ensure that they are not shut off or limited in pursuing their goals and this is dangerous\"?"

-   "Have you heard of the term "AI safety"? And if you have or have not, what does that term mean for you?"

-   "Have you heard of AI alignment?"

-   "What would motivate you to work on alignment questions?"

-   "If you could change your colleagues' perception of AI, what attitudes/beliefs of theirs would you like to change?"

-   "What are your opinions about policy oriented around AI?"

I also had content prepared if we got to the end of the interview, based on [[Clarke et al. (2022)]{.ul}](https://www.alignmentforum.org/posts/WiXePTj7KeEycbiwK/survey-on-ai-existential-risk-scenarios), [[RAAPs]{.ul}](https://www.lesswrong.com/posts/LpM3EAakwYdS6aRKf/what-multipolar-failure-looks-like-and-robust-agent-agnostic), some of Critch's content on pollution, and my general understanding of the space. My notes: "Scenarios here are about loss of control + correlated failures... can also think about misuse, or AI-assisted war. Also a scenario where the AI does recursive-self-improvement, and ends up actually able to kill humans via e.g. synthetic biology or nanotechnology or whatever, pollution."

Post-Interview Resources Sent To Interviewees
=============================================

I sent most interviewees resources after the interviews.

-   I usually floated the idea of sending them resources during the interview, and depending on their response, would send different amounts of resources.

-   I did **not** send resources if the interviewee seemed like they would be annoyed by them.

-   I only sent a couple of resources if they seemed not very open to the idea.

-   For people who were very interested, I often sent them different content that was more specific to them getting involved. These were the people who I sometimes sent the EA / Rationalist material at the end-- I very rarely included EA/Rationalist-specific content in emails, only if they seemed like they'd be very receptive.

Here's my master list of notes, which I selected from for each person based on their interests. I sometimes sent along copies of Human Compatible, the Alignment Problem, or the Precipice.

### Master list of resources

Hello X,

Very nice to speak to you! As promised, some resources on AI alignment. I tried to include a bunch of stuff so you could look at whatever you found interesting. Happy to chat more about anything, and thanks again!

**Introduction to the ideas:**

-   **The** **[[Most Important Century]{.ul}](https://www.cold-takes.com/most-important-century/) and specifically \"[[Forecasting Transformative AI]{.ul}](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/)\" by Holden Karnofsky, blog series and podcast. Most recommended for description of AI timelines**

-   [[Introduction]{.ul}](https://www.vox.com/future-perfect/2018/12/21/18126576/ai-artificial-intelligence-machine-learning-safety-alignment) piece by Kelsey Piper (Vox)

-   A short [[interview]{.ul}](https://www.vox.com/future-perfect/2019/10/26/20932289/ai-stuart-russell-human-compatible) from Prof. Stuart Russell (UC Berkeley) about his book, [[Human-Compatible]{.ul}](https://smile.amazon.com/Human-Compatible-Artificial-Intelligence-Problem-ebook/dp/B07N5J5FTS/ref=sr_1_1?dchild=1&keywords=human+compatible&qid=1635910751&s=digital-text&sr=1-1) (the other main book in the space is [[The Alignment Problem]{.ul}](https://smile.amazon.com/Alignment-Problem-Machine-Learning-Values-ebook/dp/B085T55LGK/ref=sr_1_1?dchild=1&keywords=alignment+problem&qid=1635910676&s=digital-text&sr=1-1), by Brian Christian, which I actually like more!)

**Technical work on AI alignment:**

-   Some [[empirical work]{.ul}](https://deepmindsafetyresearch.medium.com/scalable-agent-alignment-via-reward-modeling-bf4ab06dfd84) by DeepMind\'s Safety team about the alignment problem

-   [[Empirical work]{.ul}](https://arxiv.org/pdf/2112.00861.pdf) by an organization called Anthropic (mostly OpenAI\'s old Safety team) on alignment solutions

-   [[Podcast (and transcript)]{.ul}](https://futureoflife.org/2021/11/01/rohin-shah-on-the-state-of-agi-safety-research-in-2021/) by Rohin Shah, describing the state of AI value alignment (probably want the first half or so)

-   [[Talk (and transcript)]{.ul}](https://forum.effectivealtruism.org/posts/63stBTw3WAW6k45dY/paul-christiano-current-work-in-ai-alignment) by Paul Christiano describing the AI alignment landscape in 2020

-   [[Alignment Newsletter]{.ul}](https://rohinshah.com/alignment-newsletter/) for alignment-related work

-   A much more hands-on approach to [[ML safety]{.ul}](https://arxiv.org/abs/2109.13916), focused on current systems

-   Interpretability work aimed at long-term alignment: [[Elhage (2021)]{.ul}](https://transformer-circuits.pub/2021/framework/index.html), by Anthropic and [[Olah (2020)]{.ul}](https://distill.pub/2020/circuits/zoom-in/)

-   Ah, and one last report, which outlines one small research organization\'s ([[Alignment Research Center]{.ul}](https://alignmentresearchcenter.org/)) [[research direction]{.ul}](https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit) and offers prize money for solving it: [[https://www.lesswrong.com/posts/QEYWkRoCn4fZxXQAY/prizes-for-elk-proposals]{.ul}](https://www.lesswrong.com/posts/QEYWkRoCn4fZxXQAY/prizes-for-elk-proposals)

**Introduction to large-scale, long-term risks from humanity\-- including \"existential risks\" that would lead to the extinction of humanity:**

-   The [[first third of this book summary]{.ul}](https://ndpr.nd.edu/reviews/the-precipice-existential-risk-and-the-future-of-humanity/), or the book [[The Precipice]{.ul}](https://www.amazon.com/Precipice-Existential-Risk-Future-Humanity/dp/031648492X/), by Toby Ord (not about AI particularly, more about long-term risks)

> Chapter 3 is on natural risks, including risks of asteroid and comet impacts, supervolcanic eruptions, and stellar explosions. Ord argues that we can appeal to the fact that we have already survived for 2,000 centuries as evidence that the total existential risk posed by these threats from nature is relatively low (less than one in 2,000 per century).
>
> Chapter 4 is on anthropogenic risks, including risks from nuclear war, climate change, and environmental damage. Ord estimates these risks as significantly higher, each posing about a one in 1,000 chance of existential catastrophe within the next 100 years. However, the odds are much higher that climate change will result in non-existential catastrophes, which could in turn make us more vulnerable to other existential risks.
>
> Chapter 5 is on future risks, including engineered pandemics and artificial intelligence. Worryingly, Ord puts the risk of engineered pandemics causing an existential catastrophe within the next 100 years at roughly one in thirty. With any luck the COVID-19 pandemic will serve as a \"warning shot,\" making us better able to deal with future pandemics, whether engineered or not. Ord\'s discussion of artificial intelligence is more worrying still. The risk here stems from the possibility of developing an AI system that both exceeds every aspect of human intelligence and has goals that do not coincide with our flourishing. Drawing upon views held by many AI researchers, Ord estimates that the existential risk posed by AI over the next 100 years is an alarming one in ten.
>
> Chapter 6 turns to questions of quantifying particular existential risks (some of the probabilities cited above do not appear until this chapter) and of combining these into a single estimate of the total existential risk we face over the next 100 years. Ord\'s estimate of the latter is one in six.

-   [[How to Reduce Existential Risk]{.ul}](https://80000hours.org/articles/how-to-reduce-existential-risk/) by 80,000 Hours or \"[[Our current list of pressing world problems]{.ul}](https://80000hours.org/problem-profiles/)\" blog post

**Governance:**

-   [[AI Governance: Opportunity and Theory of Impact]{.ul}](https://www.allandafoe.com/opportunity), by Allan Dafoe and [[GovAI](https://www.governance.ai/)]{.ul} generally

-   [[AI Governance: A Research Agenda]{.ul}](https://www.fhi.ox.ac.uk/wp-content/uploads/GovAI-Agenda.pdf), by Allan Dafoe and [[GovAI]{.ul}](https://www.governance.ai/)

-   [[The longtermist AI governance landscape: a basic overview]{.ul}](https://forum.effectivealtruism.org/posts/ydpo7LcJWhrr2GJrx/the-longtermist-ai-governance-landscape-a-basic-overview) if you\'re interested in getting involved, also [[more personal posts of how to get involved]{.ul}](https://forum.effectivealtruism.org/tag/governance-of-artificial-intelligence) including [[Locke_USA - EA Forum]{.ul}](https://forum.effectivealtruism.org/users/locke_usa)

```{=html}
<!-- -->
```
-   [[The case for building expertise to work on US AI policy, and how to do it](https://80000hours.org/articles/us-ai-policy/)]{.ul} by 80,000 Hours

**How AI could be an existential risk:**

-   [[AI alignment researchers disagree a weirdly high amount about how AI could constitute an existential risk]{.ul}](https://www.alignmentforum.org/posts/WiXePTj7KeEycbiwK/survey-on-ai-existential-risk-scenarios), so I hardly think the question is settled. Some plausible ones people are considering (from the paper)

```{=html}
<!-- -->
```
-   \"Superintelligence\"

    -   A single AI system with goals that are hostile to humanity quickly becomes sufficiently capable for complete world domination, and causes the future to contain very little of what we value, as described in "[[Superintelligence]{.ul}](https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies)\". (Note from Vael: Where the AI has an instrumental incentive to destroy humans and uses its planning capabilities to do so, for example via synthetic biology or nanotechnology.)

-   Part 2 of "[[What failure looks like]{.ul}](https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-failure-looks-like)"

    -   This involves multiple AIs accidentally being trained to seek influence, and then failing catastrophically once they are sufficiently capable, causing humans to become extinct or otherwise permanently lose all influence over the future. (Note from Vael: I think we might have to pair this with something like \"and in loss of control, the environment then becomes [[uninhabitable to humans]{.ul}](https://www.lesswrong.com/posts/LpM3EAakwYdS6aRKf/what-multipolar-failure-looks-like-and-robust-agent-agnostic) through pollution or consumption of important resources for humans to survive)

-   Part 1 of "[[What failure looks like]{.ul}](https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-failure-looks-like)"

    -   This involves AIs pursuing easy-to-measure goals, rather than the goals humans actually care about, causing us to permanently lose some influence over the future. (Note from Vael: I think we might have to pair this with something like \"and in loss of control, the environment then becomes [[uninhabitable to humans]{.ul}](https://www.lesswrong.com/posts/LpM3EAakwYdS6aRKf/what-multipolar-failure-looks-like-and-robust-agent-agnostic) through pollution or consumption of important resources for humans to survive)

-   War

    -   Some kind of war between humans, exacerbated by developments in AI, causes an existential catastrophe. AI is a significant risk factor in the catastrophe, such that no catastrophe would be occurred without the developments in AI. The proximate cause of the catastrophe is the deliberate actions of humans, such as the use of AI-enabled, nuclear or other weapons. See Dafoe ([[2018]{.ul}](https://www.fhi.ox.ac.uk/wp-content/uploads/GovAIAgenda.pdf)) for more detail. (Note from Vael: Though there\'s a recent argument that it may be [[unlikely for nuclear weapons to cause an extinction event]{.ul}](https://www.lesswrong.com/posts/sT6NxFxso6Z9xjS7o/nuclear-war-is-unlikely-to-cause-human-extinction), and instead it would just be catastrophically bad. One could still do it with synthetic biology though, probably, to get all of the remote people.)

-   Misuse

    -   Intentional misuse of AI by one or more actors causes an existential catastrophe (excluding cases where the catastrophe was caused by misuse in a war that would not have occurred without developments in AI). See Karnofsky ([[2016]{.ul}](https://www.openphilanthropy.org/blog/potential-risks-advanced-artificial-intelligence-philanthropic-opportunity)) for more detail.

-   Other

**Off-switch game and corrigibility**

-   [[Off-switch game]{.ul}](https://arxiv.org/abs/1611.08219) and [[corrigibility]{.ul}](https://intelligence.org/files/Corrigibility.pdf) paper, about incentives for AI to be shut down. This article from DeepMind about \"[[specification gaming]{.ul}](https://deepmindsafetyresearch.medium.com/specification-gaming-the-flip-side-of-ai-ingenuity-c85bdb0deeb4)\" isn\'t about off-switches, but also makes me feel like there\'s currently maybe a tradeoff in task specification, where more building more generalizability into a system will result in novel solutions but less control. Their [[follow-up paper]{.ul}](https://deepmindsafetyresearch.medium.com/scalable-agent-alignment-via-reward-modeling-bf4ab06dfd84) where they outline a possible research to this problem makes me feel like encoding human preferences is going to be quite hard, and all of the other discussion in AI alignment, though we don\'t know how hard the alignment problem will be.

**There\'s also a growing community working on AI alignment**

-   The strongest academic center is probably UC Berkeley\'s [[Center for Human-Compatible AI]{.ul}](https://humancompatible.ai/about/). Mostly there are researchers distributed at different institutions e.g. [[Dylan Hadfield-Menell]{.ul}](https://scholar.google.com/citations?hl=en&user=4mVPFQ8AAAAJ&view_op=list_works&sortby=pubdate) at MIT, [[Jaime Fisac]{.ul}](https://scholar.google.com/citations?hl=en&user=HvjirogAAAAJ&view_op=list_works&sortby=pubdate) at Princeton, [[David Krueger]{.ul}](https://twitter.com/davidskrueger) in Oxford, Sam Bowman at NYU, Alex Turner at Oregon, etc. Also, a good portion of the work is done by industry / nonprofits: [[Anthropic]{.ul}](https://www.anthropic.com/), [[Redwood Research]{.ul}](https://www.redwoodresearch.org/), OpenAI\'s safety team, DeepMind\'s Safety team, [[ARC]{.ul}](https://alignmentresearchcenter.org/), independent researchers in various places.

-   **There is money in the space! If you want to do AI alignment research, you can be funded by either Open Philanthropy ([[students]{.ul}](https://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/the-open-phil-ai-fellowship),** **[[faculty]{.ul}](https://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/request-for-proposals-for-projects-in-ai-alignment-that-work-with-deep-learning-systems)\-- one can also just email them directly instead of going through their grant programs) or** **[[LTFF]{.ul}](https://funds.effectivealtruism.org/funds/far-future) or** **[[FTX-]{.ul}](https://ftxfuturefund.org/area-of-interest/)- this is somewhat competitive and you do have to show good work, but it\'s less competitive than a lot of sources of funding in academia.**

-   If you wanted to rapidly learn more about the theoretical technical AI alignment space, walking through this [[curriculum]{.ul}](https://www.eacambridge.org/technical-alignment-curriculum) is one of the best resources. A lot of the interesting theoretical stuff is happening online, at [[LessWrong]{.ul}](https://www.lesswrong.com/posts/P3Yt66Wh5g7SbkKuT/how-to-get-into-independent-research-on-alignment-agency) / [[Alignment Forum]{.ul}](https://www.alignmentforum.org/posts/Yp2vYb4zHXEeoTkJc/welcome-and-faq) (Introductory [[Content]{.ul}](https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ)), since this field is still pretty pre-paradigmatic and people are still working through a lot of the ideas.

**There\'s also two related communities who care about these issues, who you might find interesting:**

-   [[Effective Altruism](https://www.effectivealtruism.org/articles/introduction-to-effective-altruism)]{.ul} community, whose strong internet presence is on the [[EA Forum]{.ul}](https://forum.effectivealtruism.org/posts/fd3iQRkmCKCCL289u/new-start-here-useful-links). [[Longtermism]{.ul}](https://80000hours.org/articles/future-generations/) is a concept they care a lot about, and you can schedule a one-on-one [[coaching]{.ul}](https://80000hours.org/speak-with-us/) call here.

-   [[Rationalist]{.ul}](https://www.lesswrong.com/tag/rationalist-movement) community\-- the best blog from this community is from Scott Alexander ([[first blog]{.ul}](https://slatestarcodex.com/about/), [[second blog]{.ul}](https://astralcodexten.substack.com/?sort=top)), and they\'re present on [[LessWrong]{.ul}](https://www.lesswrong.com/bestoflesswrong). Amusingly, they also write fantastic fanfiction ([[e.g. Harry Potter and the Methods of Rationality]{.ul}](https://www.lesswrong.com/hpmor)) and I think some of their [[nonfiction]{.ul}](https://mindingourway.com/guilt/) is fantastic.

Happy to chat more about anything, and good to speak to you!

Best,

Vael

Informal Interview Notes
========================

### Thoughts from listening to myself doing these interviews

-   There's obviously strong differences in what content gets covered, based on the interviewees' opinions and where they're at. I didn't realize that another important factor in what content gets covered is the interviewee's general attitude towards people / me. Are they generally agreeable? Do they take time to think over my statement, find examples that match what I'm saying? Do they try to speak over me? Are they curious about my opinions? How much time do they have? Separate from rapport (and my rapport differs with different interviewees), there's a strong sense of spaciousness in some interviews, while many feel like they're more rapid-fire exchanges of ideas. I often end up talking more in the agreeable / spacious interviews.

-   Participants differ in how much they want to talk in response to one question. I tended to not interrupt my interviewees, though I think that's a good thing to do. ("I'm sorry, this is very interesting, but we really need to get to the next question.") That meant that for participants who tended to deliver long answers, I had fewer chances to ask questions, which meant I often engaged less with their previous responses and tried to move them on to new topics more abruptly.

-   I make a lot of agreeable sounds, and try to rephrase what people say. People differ with how many agreeable sounds they make during my speech as well, and how much they're looking at the camera and looking for cues.

-   I tended to adjust my talking speed to the interviewee somewhat, but usually ended up talking substantially more quickly. This made my speech harder to parse because of all the "like"s that get inserted while I'm thinking and talking at the same time. (I don't think I realized this at the time; it's more obvious when listening back through the interviews. I've removed a fair amount of the "like"s in the transcripts because it's harder to read than hear.) Generally, I found it useful to try to insert technical vocabulary and understanding as early as possible, so researchers would explain more complicated concepts and be calibrated on my level of understanding. I did somewhat reduce speaking speed and vocabulary when speaking with interviewees whose grasp of English was obviously weaker, though in those cases I think it's maybe not worth having the interview, since I found it quite hard to communicate across a concept gap under time and communication constraints. (These concepts are complicated, and hard enough to cover in 40m-60m even without being substantially limited by language.)

-   When I'm listening to these interviews, I'm often like: Vael, how did you completely fail to remember something that the interviewee said one paragraph up, what's up with your working memory? And I think that's mostly because there's a lot to track during interviews, so my working memory gets occupied. I often found my attention on several things:

    -   Trying to take on the framework of their answer, and fit it into the framework of how I think about these issues. Some people had substantially different frames, so this took a lot of mental energy.

    -   Trying to figure out what counterpoint to respond with when I disagreed with something, so -- fitting their answer into a mental category, flitting through my usual replies to that category, and then holding my usual replies in mind for when they were done, if there were multiple replies lined up.

    -   Trying to figure out whether I should reply to their answer, or move on. One factor here was whether they tended to take up a lot of talking space, so I needed to be very careful with what I used my conversational turn for. Another factor was how much agreement I had with my previous question, so that I could move on to the second. A third factor was tracking time-- I spent a lot of time tracking time in the interview, and holding in mind where we were in the argument tree, and where I thought we could get to.

    -   If they'd said something that was actually surprising to me, and seemed true, rather than something I'd heard before and needed to reformulate my answer to, this often substantially derailed a lot of the above processing. I then needed to do original thinking while on a call, trying to evaluate whether something said in a different frame was true in my frames. In those cases I usually just got the interviewee to elaborate on their point, while throwing out unsophisticated, gut-level "but what if..." replies and seeing how they responded, which shifted the conversation towards more equality. I think thinking about these points afterwards (and many more things were new to me in the beginning of the interviews, compared to the end) was what made my later interviews better than my earlier interviews.

    -   Trying to build rapport / be responsive / engage with their points well / make eye contact with the camera / watch my body language / remember what was previously said and integrate it. This was mostly more of a background process.

-   Conversations are quite different if you're both fighting for talking time than if you're not. Be ready for both, I think? I felt the need to think and talk substantially faster the more interruptions there were in a conversation. I expected my interviewees to find the faster-paced conversations aversive, but many seemed not to and seemed to enjoy it. In conversations where the interviewee and I substantially disagreed, I actually often found faster-pace conversations more enjoyable than slower-paced conversations. This was because it felt more like an energetic dialogue in the faster conversations, and I often had kind of slow, sinking feeling that "we both know we disagree with each other but we're being restrained on purpose" feel in the slower conversations.

-   My skill as an interviewer at this point seems quite related to how well I know the arguments, which like... I could definitely be better on that front. I do think this process is helpful for my own thinking, especially when I get stuck and ask people about points post-interview. But I do read these interviews and think: okay, but wouldn't this have been better if I had had a different or fuller understanding? How good is my thinking? It feels hard to tell.

### Content analysis

I have a lot to say about typical content in these types of interviews, but I think the above set of interviews is somewhat indicative of the spread. Hoping to have more information on these eventually once I finish sorting through more of my data.


# Interview with AI Researchers NeurIPSorICML_bj9ne by Vael Gates

**Interview with bj9ne, on 3/24/22**

\[Note: This transcript has been less edited than other due to language barriers. The interviewee is also younger than typical interviewees.\]

**0:00:00.0 Vael:** Alright. My first question is, can you tell me about what area of AI you work on in a few sentences?

\[Discusses research in detail\]

**0:03:31.9 Vael:** Got it. Great. Yeah, so thinking about the future, so thinking about what will happen in the future, maybe you think AI is important or maybe you don\'t think AI is important, but people talk about the ability to generate an AI that is a very capable general system, so one can imagine that in 2012, we had AlexNet and the deep learning revolution, and here we are 10 years later, and we\'ve got systems like GPT-3, which have a lot of different capabilities that you wouldn\'t expect it to, like it can do some text generation and language translation and coding and math, and one might expect that if we continue pouring in all of the investment with nations competing and companies competing and algorithmic improvements or software improvements and hardware improvements, that eventually we might reach a very powerful system that could, for example, replace all current human jobs, and we could have CEO AIs, and we could have scientist AIs, so do you think this will ever happen, and if so, when?

**0:04:43.6 Interviewee:** Honestly speaking, I don\'t think this is realistic in the future, like 10 years or 20 years. So basically, I agree that AI will replace more jobs, but in my mind, this replacement, I don\'t say that\... For example, in a factory, there were 100 employees, and in the future, we may replace that 100 employees with AI, and five employees, but this five may be advanced engineers, rather than regular people. Because I have gone to some factories that manufacture something we use in our daily life, and my impression is that there were very, very few people, but the factory is very large, and mostly those people are just sitting in a room with air-conditioning. But those robotics, which is guided by AI and processing maybe those request from customers very efficiently, but they tell me that, for example, for some security issues, they need to watch those machines, so that that won\'t happen something unexpected. So also as an example, cars, and automated cars, auto drive is very, very \[inaudible\] in these years, but this is a issue that there are a lot of security issues.

**0:06:39.3 Interviewee:** So basically, just, I think the manufacturer and auto drive has a similarity, that is in many scenarios, we do want to guarantee a high level of security, but maybe for scenarios like recommendations, so making mistakes, that doesn\'t matter, and also I think one issue is very important, that is interpretability. Because for many, many models we designed or we have designed, and maybe less, there are many popular or famous papers in top conferences, but these models, many of these models lack interpretability.

**0:07:30.0 Interviewee:** So this is not a\... Sometimes it is unacceptable to deploy these models into some scenarios that requires us to \[inaudible\] something, especially in the scenarios like in hospitals, but we can\'t accept this because sometimes, an error is\... For example, when we do salaries, and some errors is not acceptable. So I think this is a very good question, so for my mind, I think this should be discussed by classifications, so for scenarios like recommendations, where errors are acceptable, then AI will replace more and more work of human, and this work was done by maybe some data scientists, because with our models become more and more intelligent and our systems become more and more efficient, also our data become more and more efficient, so the work of data scientists will be reduced by a great margin. And I expect this growth to\... I think this is sustainable.

**0:08:49.9 Vael:** Yeah\--

**0:08:50.2 Interviewee:** For example, in probably one or two decades, this will keep growing, but for some scenarios like auto drive, then on my mind, maybe this is controversial, but I don\'t think that auto drive is really something so realistic that we can expect like L4 or L5 auto drive to be realistic, to be used by us in a decade.

**0:09:20.0 Vael:** Yes\--

**0:09:20.9 Interviewee:** So in these scenarios, I think AI is a very important system for us. It can reduce our work, reduce our burden, but human is needed.

**0:09:30.0 Vael:** Yeah. Great. Cool, so my question is, do you think we\'ll ever get very, very capable AI, regardless of whether we deploy it or not? Do you ever think that we\'ll have systems that could be a CEO AI or a scientist AI, and if so, when? Like it could be like 50 years from now, it could be like 1,000 years from now, when do you think that would happen?

**0:10:00.3 Interviewee:** Oh, this is a tough question. Let\'s see. At least, based on the method we have taken, I think the way we develop AI now will not lead us to that future, but maybe the human will find some different ways to develop AI. But through my mind, I guess is to first we talk about maybe 50 years or a century, I think that\'s not very possible, but in the future, this may be a question about some knowledge about our brains. So maybe human at the moment, we are not\... I mean the investigation or research into our brain is not very clear.

**0:11:01.0 Interviewee:** So it\'s quite hard to imagine if the machine can evolve into some stages, that is the machine can be as complex, as powerful like human brains. So maybe in a century or even in two centuries, I tend not to believe that this will happen. but in the very long run, it\'s very hard to tell because I think that the way we think about something and the way the machine difference or train themselves are totally different. They work in different ways. For example, machines may require very large amount of data to find some internal principles. But with human, we are very good at generalizations, so I think for this point, if then maybe we will achieve that after 1,000 years, but I\'m not very optimistic about this. I tend not to accept this, but I can\'t deny that entirely.

**0:12:13.3 Vael:** Yeah. There are some people who argue that the current way of scaling may eventually produce AI, like very advanced AI, like OpenAI and DeepMind. No? You don\'t think so? Cool.

**0:12:29.1 Interviewee:** I don\'t think so, because based on our current trend, the development of the models comes from the development of computing power. But computing power, essentially that comes from the increasing density of changes account. So according to a research by Princeton, we know that for computer, computing power per transistor hasn\'t changed a lot during the last few decades, and also, you know the end of dinosaurs and maybe in the future, we\'ll see the end of Moore\'s Law. And I know, I have solid GPT, GPT 2022 \[inaudible\] Nvidia has powered the AI and speed up by a million x in the last decade. But I don\'t think that this development is sustainable because you see, over here I spent GPT-3 with maybe \$4 million, if I don\'t get it wrong. This scale is not good because not only we need a larger cluster but also we need more money and more time to train that model. So I remember that OpenAI has forged some arcs\[?\] in this way, but we are not able to train that again, and I have some\... There are some articles about the scaling of deep-learning models based on the parameter count of the models and they found that, I think maybe before 2022\... No, before 2020, the growth is 75X per 18 months, but after that the growth has slowed down, has greatly slowed down because, lots of issues, like the scaling of GPU or the scaling of clusters to deploy those models in very large cluster that is not narrow anything.

**0:14:53.0 Interviewee:** So I think so, this is partly\... This is the main reason why I shift my interest from data to system, because I believe that AI has a bright future, but to make that future brighter, we need to make our system run faster. For example, can we achieve the same result using less parameters or using less power so that we can\... We can\'t have these hardware resources but can we make better use of it? So I think the future of AI largely depends on system people. That is, can we improve the system for AI? I mean, when I talk about system, I actually, from the perspective \[inaudible\] AI, I talk about three things, the hardware, software system, and the platforms. In my mind, those models are just like the top layer, that is the application. I think the lower three levels are the key to the future success of AI, but indeed I think AI is\... And also \[inaudible\] so, I think the future of AI also means we need more application scenarios, like dark\[?\] intervention or something like that, and also robotics, and I think this is very promising and it can bring us a lot of fortune or something like that.

**0:16:25.8 Vael:** Yeah, great. I\'m concerned about when we eventually\... I think that we may get AI, AGI a little faster than that. And I\'m concerned that the optimization functions that we\'ll be optimizing are not going to reflect human values well because humans aren\'t able to put their values and goals and preferences perfectly into AI, and I worry that problem will get solved less fast than the problem of how we get more and more capable AI. What do you think of that?

**0:17:00.6 Interviewee:** When you talk about values, it\'s something. Yeah, so although it\'s not a technical issue, but this is really what the technicians should care about. So this is a very important issue in open drives\[?\], when the cars make you a hero\[?\], a person first\[?\]. So maybe we should find some method or some way to plug our rules, plug our values to guide the models, guide AI, but I think it\'s also an issue of interpretability because now we don\'t have\... Sometimes we have no idea why some models can work so well. For example, when\... Because I have worked on GNN, and this year GCN and its variants, it\'s very popular, but many of that, when the researcher comes up with that model, they don\'t know why this is good.

**0:18:12.5 Interviewee:** So maybe similarly it\'s quite hard to guide AI to follow the rules, so I think this is also an important issue and important obstacle for the applications of AI. For example, we cannot put some non-CV skills to recognize human faces because sometimes it may violate the law. Yeah, I think this is an important issues, but will this stop AI? I think for my mind, this may be an obstacle for AI in some scenarios, but in many scenarios, this is not a\... This will generate some issues for us to think about, but in the end I think we will deal with this. For example, some people may use federated learning to deal with privacy and there are some techniques to deal with these issues. So yeah, I think we should put more emphasis on the values and the rules or even the assets about AI so that this community will grow faster. Yes, this is an important aspect. Although I don\'t put much emphasis on this.

**0:19:44.1 Vael:** Why not?

**0:19:49.8 Interviewee:** That\'s because, I guess, because my research and my internship mainly focus on recommendation and there is not much issues about this except for privacy, and because when we got those data, we don\'t know the meaning of data. When I get a little data intention, I just send out important numbers, or maybe sometimes this is a one \[inaudible\] issue, and I don\'t know much about that, so let\'s say that these privacy issues has been\... Maybe this has been dealt with by those data scientists, not by people like us, so this is because of my research interest, but I think for those people who do\...

**0:20:34.1 Interviewee:** Yes, I have taken some courses about AI and then teachers say that they have developed some robotics to help the elderly. But let\'s say that sometimes you cannot use a camera because using a camera will generate some privacy issues, so maybe sometimes we can just use something to catch its audio rather than video or something like this, but because\... I guess that\'s most of students\... Most of my schoolmates don\'t put\... Most of my classmates haven\'t paid much attention on these issues, but this is a very\... I think this issue will become more and more important in the future if we want to generate AI to more and more scenarios, so thank you for raising these points. I will think more about it in the future.

**0:21:37.0 Vael:** Yeah. I\'m happy to send you resources. One extra thing, one other additional thing. So I think probably, I think by default the systems we train will not do what we intend them to do, and instead will do what we tell them to do, so we\'ll have trouble putting all of our preferences and goals into mathematical formulations that we can optimize over. I think this will get even harder as time goes on. I\'m not sure if this is true, but I think it might get harder as the AI is optimizing over more and more complex things, state spaces.

**0:22:13.6 Interviewee:** So you mean that because in the future we will have more and more requirements, and that\'s so\...

**0:22:22.6 Vael:** No, no, the AI will just be operating under larger state spaces, so I will be like, \"Now I want you to be a CEO AI,\" or, \"Now I want you to be a manager AI.\"

**0:22:34.3 Interviewee:** Oh, did you say that we need to encode those requirements into optimization function so that AI will operate like what we want them to do? Did I get it wrong? Oh, that\'s a quite good question. I have discussed that with my roommates. Yeah, so yes, it is the optimization, the loss function that guides the model to do something that we want, and sometimes it\'s hard to find an appropriate function, especially for newbies. Sometimes we chose a wrong loss function, and the way the model is totally unusable.

**0:23:16.8 Vael:** Yeah, and I have one more worry about that scenario, so\...

**0:23:20.5 Interviewee:** Yeah, yeah, yeah. I think this is also a very important issues, and I\'m not very optimistic about this because it\'s really hard. And lots of things, because like a CEO AI, we not only need to care about the revenue of this company, but also learn maybe the reputation, and we may also want them to abide by the laws, and maybe when there is new business and we want to inject new rules into that loss function core business. Great.

**0:24:05.6 Vael:** Yeah, and I have one more twist. Alright, so imagine that we had a CEO AI, and it takes human feedback because we\'ve decided that that\'s a good thing for it to do, and it needs to write a memo to the humans so that they make sure that its decision is okay, and the AI is trying to optimize a goal. It\'s trying to get profit and trying not to hurt people and stuff, and it notices that when it writes this\... That when it writes, sometimes humans will shut it down. And it doesn\'t want that to happen because if humans shut it down, then it can\'t achieve its goal, so it may lie on this memo or omit or not include information on this memo, such that it is more likely to be able to pursue its goal. So then we may have an AI that is trying to keep itself alive, not because that was programmed into it, but because it is an agent optimizing a goal, and anything that interfered with its goal is like not achieving the goal. What do you think of this?

**0:25:16.3 Interviewee:** Oh, very good question, but it\'s very hard. It basically mean that we need to establish enough rules so that when\... Sometimes it\'s very hard to come up with some common cases that AI may\... Yes, it is optimizing towards its goal, but there may be something that we want it to do, so maybe we need to have a mechanism so that we can switch from the AI mode, from manual mode, that we can take control of AI or take control of\... For example, the company in the last second was guided by an AI, and for the next second, we want to guide in, we want to lead the company manually, so I think if we ask \[inaudible\] enough and we may establish maybe a thorough mechanism so that we can guarantee that, it is possible to take control back, and the AI will not lie to us. And yes, theoretically, this is possible and this should be the case. But correctly speaking, I think this can be done, but it\'s quite hard to estimate the cost. The engineering cost for us to make such a AI that is complete enough, that is secure enough to help us achieve the goal. So maybe in this point, if we want to do something like recommendations, we can be very radical, we can develop or deploy radical models. But maybe in the scenarios like a CEO AI, I think we should be conservative because some internal principles of AIs are not known entirely by the humans, so I think sometimes we need to be conservative to prevent some bad things from happening.

**0:27:45.1 Vael:** Yeah. I think that the problem\... I think it\'s not just an engineering problem. I think it\'s also a research problem, where people don\'t know how to construct optimization functions such that AI will be responsive to humans and won\'t be incentivizing against humans. And there\'s a population of people who are working on this, who are worried that if we have very intelligent AIs that are optimizing against us, then humans may be wiped out, and so you really don\'t want\... You really wanna make sure the loss function is such that it isn\'t optimizing against us. Have you heard of the AI safety community or AI alignment community?

**0:28:25.9 Interviewee:** No. I don\'t know much about that, but let me say the scenarios you have mentioned, that AI may optimize against us and even wipe out the human, I have seen some films, some movies about this, and yes, this is possible if we are too careless, I think this is possible.

**0:28:48.8 Vael:** Yeah.

**0:28:49.0 Interviewee:** But at least it\'s right at\... I haven\'t paid much attention on these issues, but I think this is an important question.

**0:28:58.9 Vael:** Yeah, I personally think that advanced AI may happen in the next 50 years, and this is just from looking at some surveys of experts. I\'m very unsure about it. But if that does happen, then I think that currently we have groups like\... China and the US are going to be competing, I expect, and we\'ll have lots of different corporations competing, and maybe DeepMind and OpenAI are not competing, but maybe they\'re just going really hard for the goal. And I worry that we\'re not going to spend enough effort on safety and we\'re going to be spending much more effort on trying to make the AI do the cool things, and if the safety problems are hard, then we may end up with a very unsafe, powerful AI.

**0:29:55.2 Interviewee:** Or this then may come up with the competition on nuclear weapons.

**0:30:01.1 Vael:** Yeah.

**0:30:01.4 Interviewee:** How likely \[inaudible\] just like nuclear weapons. Yes, the power of AI in the future may be something like the power of nuclear weapons, that is quite hard to control if the real war or something\... Maybe not so severe like a war, but it is possible, so maybe I think we need some international association about the use of AI. But both in China and the US, the government has established more and more rules about, for example, privacy, security, and what you can do and what you can\'t do. So yes, if we don\'t place enough emphasis on this, this may be a question, but well, most\--

**0:30:51.3 Vael:** I think they\'re placing emphasis on issues like privacy and fairness, but they\'re not placing emphasis on trying to develop loss functions that do what humans want. I think that is a more different type of research that is not being invested in.

**0:31:08.8 Interviewee:** Yes, you\'re right. So maybe the community should do more things about this because you can\'t count on those people in the government to realize that this is really a case, they don\'t know much about this, so yes, yes, the community should, like the conference or a workshop, we should talk more about this, yes, before that is too late. Yes, I agree. Before that is too late, or everything will be a disaster.

**0:31:38.5 Vael:** Amazing. Yeah, so there is actually a community of people working on this. I don\'t actually know\... I know fewer of the people who are working on it in \[country\], although I do think there are people working on it. I\'m curious, what would cause you to work on these sort of issues if you felt like it?

**0:32:00.0 Interviewee:** You say issues that we have just mentioned?

**0:32:01.9 Vael:** Yeah, long-term. Well, like trying to make the AI aligned with a human, trying to make sure that the things that the AI does is what the humans wanted to do, long-term issues from AI, anything like that.

**0:32:16.4 Interviewee:** So I try to, although from the bottom of my heart, I think I believe that this is an important issue, frankly speaking as a student, a student researcher, or as an engineer, I don\'t have much resources about this. So I think this is the most important issue why most of my schoolmates just like improving the models and then don\'t care about if the AI may optimize against us because\... I know this sound not good, but most of student just care about, \"So if I can graduate with a PhD degree or so.\" Yeah, so maybe\... I think for me, maybe\... because I guess I will be an engineer in the future, so maybe if I have enough resources or I have enough influence in the community, I\'m willing to spend my time on this, but if I just a low-level coder and I don\'t have much power to ask my superintendent that we should place more emphasis on this, they just take\...

**0:33:35.2 Interviewee:** For example, if I intend and they just say, \"Oh, this model, the accuracy is good enough, but the speed is not, so optimize the model so that it can run fast enough as maybe the customers\' requirement.\" So yeah, this is basically the entire ecosystem, both in the academia and the industry that force the researchers and the employees in the company that they will not put much emphasis on this, and also, most of the time they just focus on short-term issues, short-term profits, or in the universities, student just care about, \"Oh, can I have some publications so that\... \" I really don\'t know any of my class schoolmates who have publication on these issues.

**0:34:32.4 Interviewee:** So I don\'t know whether there are lots of the researchers who cares about this and will spend maybe several months on these issues where they will have some publications about this. I know the top conference has asked about some ethical issues, but yeah, we really don\'t pay enough attention on this. This is a very good point. I think we need some more incentives on the future of AI. For example, like environmental issues, some factories, they will not care about environmental issues if the government doesn\'t force them to do so. For example, now we have the trade on the carbon dioxide budget. That is, the government tell the factories that you shouldn\'t emit more carbon dioxide than maybe more than this threshold or you will be fined. Maybe we need some, yes, we need some incentives to force us to think about these issues or otherwise I think this is not optimistic because not many people will be guided to do this because maybe those on the other levels they don\'t care about this.

**0:36:05.7 Vael:** Yeah, yeah. That seems right. Yep. It does seem like it\'s not currently as popular in the top journals as it could be, seems like a pretty small community right now. I will look around to see if I can find any researchers in \[country\] who are working on this sort of thing, because I know a bunch of them in \[country\] and I know some of them in \[country\], but not as many in \[country\], and I\'ll send you some resources if you\'re interested. There is a group of people who pay attention to this a lot, and they\'re called the Effective Altruism community, and right now, one of the things they care about is trying to make sure that existential risks don\'t happen so that humans are okay. Some other things they\'re worried about are pandemics, nuclear stuff, climate change, stuff like this, and also many other things. Interesting. Alright, cool. I think my last question is, have you changed your mind on anything during this interview, and how was this interview for you?

**0:37:16.9 Interviewee:** Oh. Yeah, I think maybe the greatest change to my mind is, you say that if we want a CEO AI, we need to, maybe we need to encode those requirements into the optimization function, and maybe someday an advanced AI will optimize against us. Yes, basically, in the past, I think this may be an ethical issue, and now I\'ve realized that it\'s both an ethical and a social, as well as a technical issues and we have\...

**0:37:53.4 Vael:** Yeah.

**0:37:55.3 Interviewee:** Yes. I know we haven\'t paid enough emphasis on this, but yeah, now I think that it\'s time for us to do more things, and this is a very wonderful, wonderful idea for me to think about. Thank you very much.

**0:38:14.3 Vael:** Yeah. Well, I mean, it\'s super cool that you\'re interested in this, so I\'m very enthused by you being like, \"This does seem like a problem, this does seem like a technical\... \" Yeah, I\'m very excited about that. Cool. Alright. Well, I will send you some resources then and I\'ll see if I can find anyone who is doing anything like this and send anything I find your way. But thank you so much, and feel free to reach out if you have any questions or if there\'s anything I can help you with.

**0:38:41.8 Interviewee:** Oh, okay. Also, thank you. So no, I don\'t have much questions. I will read more about this in the future. I think this is very important and also very interesting. Thank you. Thank you.

**0:38:54.8 Vael:** Yeah, I\'ll send you some resources. Alright, email you soon. Bye.

**0:39:01.3 Interviewee:** Okay.

NeurIPSorICML_bj9ne-by Vael Gates-date 20220324.md