# Interview with AI Researchers NeurIPSorICML_lgu5f by Vael Gates

**Interview with lgu5f, on 3/22/22**

**0:00:02.5 Vael:** Alright, my first question is, can you tell me about what area of AI you work on in a few sentences?

**0:00:08.4 Interviewee:** Yeah. So I work on privacy in AI. I\'ve been working on differential privacy, which is sort of the de facto standard for giving statistical privacy in analytics and AI. I\'ve been working on differentially private synthetic data, so coming up with algorithms that generate synthetic versions of datasets that mimic the statistics in those datasets without revealing any information about the users and the data itself. And more broadly, I also just work on differential privacy for analytics, so it\'s not specifically AI, but it\'s still like algorithm design with privacy.

**0:00:55.3 Vael:** Cool, great, thanks. And my next question is, what are you most excited about in AI, and what are you most worried about? In other words, what are the biggest benefits or risks of AI?

**0:01:05.7 Interviewee:** I think that AI has been most helpful in the little things. When I pull up my phone, and it\'s like, \"Hey, this is your most used app in this location,\" recommender systems like that, or small tweaks that help my daily life, that\'s what I\'m most excited about. I think I\'m most worried about AI being used in applications where explainability and fairness are going to be important, or privacy. This is stuff I see red flags in. I\'m worried about an insurance company putting a neural net into some important decision-making system and then not being able to analyze why it made a decision that it did, or understanding if it\'s being unfair.

**0:02:11.6 Vael:** Great, yeah, that makes sense. And then focusing on future AI, so putting on a science fiction forecasting hat, say we\'re 50 plus years into the future, so at least 50 years in the future, what does that future look like?

**0:02:30.3 Interviewee:** I\'m sort of pessimistic about how advanced AI can get. I think that the trend is going to be that we\'re going to see smaller models that are more bespoke for the problem domain that it\'s trying to solve. So instead of these like GPT-3 trillion parameter-sized models, I think that we\'re going to start moving back towards stuff that can run more easily on the edge. That doesn\'t require as much energy and time to train and doesn\'t require as much data. I think that AI becomes more ubiquitous, but in a way that\'s easy for us to compute with. So it\'s just more AI running everywhere.

**0:03:17.6 Vael:** Yeah, what drives that intuition?

**0:03:20.7 Interviewee:** One is\-- partially concern with large models consuming too much energy and, of course, climate change is one of the chief things that we should be worried about. The other thing is\... I\'ve seen some experiments that come out of GPT-3, and they\'re cool as toy problems, or it\'s cute program synthesis and stuff like that, but I don\'t see that really being used in production. It\'s one thing for a research organization to come up with those experiments and say, \"Hey, we were able to, I don\'t know, use this to beat a goal player,\" but you look at the details of it, and really this gigantic model also had a tree search algorithm, that was a big benefit. I think just\... keeping it bespoke, like CNNs just do so well, and I think that that\'s for a reason, so that\'s sort of the intuition I have. If we keep it tight to the problem domain, I\'ve seen it do better. Domain expertise has helped a lot.

**0:04:37.5 Vael:** And then just a quick follow-up on the climate change thing, is the idea that, current systems are using too much energy and this is causing increased climate change?

**0:04:50.0 Interviewee:** Yeah, I think that we all just need to reduce how much energy we\'re using, because, unless we\'re sure that a lot of it is coming sustainably, we should be concerned about how much energy we\'re using. And training these trillion parameter models requires a lot of energy. It requires a lot of hardware. That hardware does not come for free. There\'s a manufacturing process that goes into that, building up the data centers that are training, that are hosting all of this, and then replacing the hard drives, replacing the GPUs when they get stale, so there\'s just a whole bunch of life cycle impacts from training models that I think are really coming up because we\'re seeing people doing these studies on blockchain because that tends to burn through GPUs and hard disks faster than training machine learning models, but it\'s sort of the same impact.

**0:05:47.4 Vael:** Interesting. Cool. Well, this next thing is more of a spiel, and it is quite close to what you\'ve been talking about with where AI will go in the future and how big it will get. So people talk about the promise of AI and they mean many things by that, but one thing they may mean is a very general capable system, such that they\'ll have the cognitive capacity to replace all human jobs, all current day human jobs. So whether or not we choose to replace human jobs is a different question, but having the cognitive capacity to do that, and so I usually think about this in the frame of 2012 when we have AlexNet, deep learning revolution, and then 10 years later, here we are, and we\'ve got the GPT-3 system. Which, like you said, have some weirdly emerging capabilities, so it can do some text generation, and some translation, and some coding, and some math and such. And we might expect that if we continue with all this human effort that\'s been going into this kind of mission of getting more and more general systems, and we\'ve got nations competing, and we\'ve got corporations competing, and we\'ve got all these young people learning AI and like, maybe we\'ll see algorithmic improvements at the same pace we\'ve seen hardware improvements, maybe we get optical or quantum. So then we might actually end up scaling the very general systems or like you said, we might not, we might have hit some sort of ceiling or require a paradigm shift or something. But my question is, regardless of how we get there, do you ever think we\'ll have very general AI systems like a CEO AI or a scientist AI? And if so, when?

**0:07:13.3 Interviewee:** Oh, that\'s a good question. I tend to be more pessimistic about this. It depends what you mean by AI, I guess in this sense, right? Is it sort of a decision-making system that\'s just doing Pareto optimal decision-making. Does it have to be trained?

**0:07:52.6 Vael:** Yeah. What I\'m visualizing here is any sort of decision-making system, any sort of system that can do things like multi-step planning, that can do social modeling, that can model itself\-- modeling other people modeling it. Have the requirements such that I can be like, \"Alright, CEO AI, I want you to maximize profit plus constraints.\" So very capable cognitive systems. I don\'t require right now that they\'re embodied necessarily because I think robotics is a bit behind, but like cognitive capacities.

**0:08:25.1 Interviewee:** Got it. I think we\'re still a solid century behind that. Yeah, I don\'t know. I just feel like it\'s still a solid century behind because that might require a huge paradigm shift in how we\'re training our models. Yeah, I don\'t think we\'ve even thought about modeling something with a state space that large, right?

**0:08:58.9 Vael:** Yeah, it\'s like reality.

**0:09:00.7 Interviewee:** Yeah, exactly. I\'ve seen cool stuff where there\'s like, you hand over a floor plan or something, and then you say, "Hey, where did I leave my keys?" And this AI is able to jog through, navigate through this floor plan and then say you left it in the kitchen. But I still think that something like a CEO AI is still going to be like 100 years out. Yeah.

**0:09:29.5 Vael:** Interesting. Yeah, that is interesting, because it seems like what you were saying earlier, is like, I think we\'ll get more bespoke systems, less large scale systems. And I\'m like \[humorously\]: \"What if we have a very large-scale system, that is very\...\" \[interviewee laughs\] Yeah. How does that square?

**0:09:43.8 Interviewee:** Between having a bespoke system versus\...

**0:09:47.9 Vael:** Or like how the future will develop or something. I think that that\'s like\... maybe kind of likely that there will be at least some companies like OpenAI and DeepMind that\'ll just keep pushing on the general system thing. But I don\'t know.

**0:09:58.4 Interviewee:** Yeah, I feel like there\'s always going to be billions of dollars poured into that kind of research. But at the end of the day, I think that we\'ll see more impact if we make quality of life improvements that make jobs easier. Of course, there\'ll be a whole bunch of automation, right, but I don\'t know if we\'ll ever get to a point where we can leave decision-making entirely to AI. Yeah, because\... I think also just as society, we haven\'t thought about what that means. I think Mercedes just recently said that Mercedes will take responsibility if one of their driverless cars hits someone, which is a huge step, right, but we haven\'t gone close to already deciding who has liability if an insurance company messes up, right, with their premium setting algorithms.

**0:11:09.5 Vael:** Yeah, when I think about the future, I\'m like, I think we might get technological capacities before we get good societal understanding or regulation.

**0:11:17.1 Interviewee:** Likely. Likely. Because that\'s what\'s happening in fairness and in privacy. With fairness regulations, like you have in Australia and a whole bunch of places, where it just says, Okay, you don\'t pass in some protected attributes and your algorithm is not going to be racist, homophobic, transphobic, whatever, sexist. And it\'s like, Okay, cool, now that you\'ve purged the data of all these attributes, there\'s no way for us to measure whether it\'s actually being any of those things. But it\'s required by policy. There\'s also the same thing with privacy, where it\'s like, Okay, if you just remove these attributes and you remove their names, we can\'t tell who it is, but nobody else has watched the exact five last movies I\'ve watched on Netflix, or the last three things I\'ve liked on Twitter or Facebook or whatever. So yeah, for sure, we\'ll be lagging societally, but hopefully with this timespan I have in mind of it\'s in a century, we\'ll be thinking about these things if we\'re on the cusp of it, and people will be raising alarm bells about it like, Hey, maybe we should start thinking about this.

**0:12:34.1 Vael:** Yeah, that makes sense. I know there\'s a substantial number of researchers who think there\'s some probability that we\'ll get it substantially sooner than that. And maybe even using the current deep learning paradigm, like GPT-whatever, 87, 7, GPT-13. I don\'t know. But that this system will like work. I don\'t actually know if that\'s true. It\'s hard to predict the future. Yeah, so my next question is sort of talking about whenever we get these very advanced systems, which again, who knows, you said like maybe 100 years, some people think earlier, some people think much later. So imagine we\'re in the future, and we\'re talking about the CEO AI, and I\'m like, Okay, CEO AI, I wish for you to maximize profits\-- this is where humans are like the shareholder type thing\-- and try not to run out of money and try not to exploit people and try to avoid side-effects. Currently, this is technologically challenging for many reasons, but one of the reasons I think that might continue into the future is that we\'re currently not very good at taking human values and preferences and goals and putting them into a mathematical formulations that we could optimize over them. And I think this might be even harder to do in the future as we get more powerful systems and that are optimizing over larger\-- reality. Larger optimization spaces. So what do you think of the argument, \"Highly intelligent systems will fail to optimize exactly what their designers intended them to, and this is dangerous?\" \[pause\] So they\'d be doing what we tell them to do rather than what we intended them to do.

**0:14:12.2 Interviewee:** \...Yeah, I think that that\'s most likely. Because\... Well, it depends what level of self-awareness or cognitive awareness this AI has to understand what we intended versus telling them. If it figured out the state space was\... If it can model what the ramifications of what it does are on itself, then ideally it would figure out what we intended for it to do, not what we told it to do it. So it\'s one thing if it goes off the rails and says say, actually, this is the only way to solve\-- That\'s a very Asimov-esque dystopia, right. But hopefully, if it can model that and say like, Oh actually, you know, the humans will delete me if I do something that\'s exactly what they told me to do and not what they intended for me to do, then ideally we\'d be past that point.

**0:15:15.3 Vael:** I see, yeah. Cool. Let me try to engage with that. So I\'m like, Alright, where we\'ve told the CEO AI that I want you to maximize profit, but not have side-effects, and it\'s like, Okay, cool. Side-effects is pretty undefined, but maybe it has a very good model of what humans want, and can infer exactly what the humans want and can model things forward, and will know that if it pollutes, then the humans will be unhappy. I do think if you have a great enough model of the humans in AI, and AI is incentivized to make sure to do exactly what the humans want, then this is not a problem.

**0:16:00.6 Interviewee:** Right. The risk, of course, is that it figures out all the legal loopholes, and it\'s like, Oh great, I\'ve figured out exactly what I need to do to not be held morally culpable for what choices I\'ve made and get off scot-free, which is also a huge risk.

**0:16:19.2 Vael:** Yeah, yeah. So I think with the current day systems, they\'re much dumber, and so if you\'re like, Alright AI, I want you to get a lot of points then maybe it will get a lot of points through some little side loop instead of winning the game. That\'s because we haven\'t specified what we wanted exactly, and it can\'t infer that. But you\'re saying as we get more advanced, we will have systems that can at least know what humans want them to do, whether or not they follow through with that is a different question.

**0:16:51.7 Interviewee:** Yeah. Yeah, and at some point, there\'s still the huge element of what the designer of the algorithm put in. There is some choice on the loss function. Is it average, is it worst case? Is it optimizing for the highest likelihood, the worst likelihood, I feel like that would also be a huge change in how it does it. So it\'s not going to be fully up to the algorithm itself, they\'re still going to be human choices in building out that algorithm that determine how it interacts with other humans.

**0:17:31.1 Vael:** Yeah, this is in fact the whole question, I think. What do you put in the loss function such that it will do whatever things you want to do. Yup. Yeah, okay, cool. So I have a second argument on top of that one, which I think is pretty relevant. So say we have the CEO AI, and it has pretty good models of humans, and it can model humans modelling, and it does multi-step planning. And because we figured, humans figured that we should have some safety mechanisms, so maybe we don\'t let the AI make any big decisions unless it passes that decision through us. And so they\'ve asked the AI for a one-page memo on this upcoming decision. And the AI is like, Cool, well, I obviously have a lot of thinking and information that I can\'t put everything in a one-page memo, and so humans are expecting me to condense it. But I notice that sometimes if I include some types of information in this memo, then the humans will shut me down and that will make me less likely to be able to achieve the goal they programmed in. So why don\'t I just leave out some information such that I have a higher likelihood of achieving my goal and moving forward there. And so this is a story that\'s not like the AI has self-preservation built in it, but rather as an agent optimizing a goal, a not perfect goal, then it ends up having an instrumental incentive to stay alive and to self-preserve itself. So the argument is, what do you think of the argument, \"Highly intelligent systems will have an incentive to behave in ways to ensure that they are not shut off or limited in pursuing any goals, and this is dangerous?\"

**0:19:08.7 Interviewee:** I think we already see this with not highly intelligent systems. There was this paper from 2019 where they analyzed the preventative healthcare recommendations from some insurance company algorithm setup. And they found that it consistently would recommend people of color to go for preventative healthcare less. So they were looking through it, and they realized that the algorithm is optimizing to minimize the dollar spent by the patient, and so it figured out that, Hey, folks who are a part of majority communities go for preventative healthcare and they save dollars by getting preventative healthcare versus folks in minority communities tend to not get preventative healthcare, but most of them end up not ever getting specialist care either, so it\'s actually more dollars saved if we just don\'t give them preventative healthcare. I think that that could be pretty likely. Again, if we have this highly intelligent system that\'s able to model so much, then we should be kinda skeptical of what comes in the memos. But yeah, I\'d say that that\'s possible, that it just gets stuck on one of those loops, like, Oh, this is best in\...

**0:20:44.7 Vael:** Yeah, interesting. Yeah, so the example you gave seems to me like it\'s some sort of alignment problemm where we\'ve tried to tell what we want, which is presumably like good healthcare for everyone that\'s sort of cheap. And instead it is doing something not what we intended it to do, like, whoops, that was like a failure. We\'ve put in a failure in terms of the loss function that we\'re optimizing. Something that\'s close, but not quite the problem. Then yeah, I think this argument is, as we get very intelligent systems, if the loss function that we put in is not exactly what we want, then we might have it optimizing something kind of close, and then it will be incentivized to continue pursuing that original goal, and in fact, maybe optimizing against humans trying to change it. Which is interesting. So, if one buys this argument of instrumental incentives of an agent optimizing what a goal it puts in, then you have ideas like an agent that is optimizing for not being shut down. So that can involve deception and acquiring resources and influence and also improving itself, which are all things that are good things that are able to help you better achieve goals. Which feels pretty worrying to me if this is the case, because if this is one of the default ways that we\'re developing AI, then when we actually get advanced AI and then we\'ll maybe have a system that is as or more intelligent than us optimizing against humans. And I\'m like, \"Wow, that seems like real bad.\"

**0:22:14.0 Interviewee:** Yeah. Yeah, that\'s a nightmare scenario, which is Matrix-esque, right? Yeah.

**0:22:21.5 Vael:** Yeah. What is the bad AI in the Matrix doing? Is there a bad AI in the Matrix?

**0:22:28.1 Interviewee:** Yes. So the bad AI\... It\'s sort of the same idea as the Terminator, where the Skynet of the Matrix realizes that the best way to avoid conflict in humanity is to sort of keep humanity in a simulation. I think the robots and the humans end up fighting a war, and then they just plug the humans into the matrix, keeping them quelled that way and making the broader argument that living in a simulation is nicer than the world outside. So the AI is technically also doing something good for humanity and self-sustaining, and keeping human population alive.

**0:23:09.2 Vael:** Interesting. Yeah. Cool, alright. So my interest, in general, is looking at long-term risks from AI. Have you heard of the\-- well, okay, I\'m sure you\'ve heard about AI safety. What does AI safety mean for you, for the first question?

**0:23:31.0 Interviewee:** Um\...

**0:23:32.3 Vael:** Or have you heard of AI safety?

**0:23:34.8 Interviewee:** No, not really.

**0:23:35.7 Vael:** Oh, interesting, cool!

**0:23:37.5 Interviewee:** So can you tell me more?

**0:23:41.3 Vael:** Yeah. So, AI safety means a bunch of different things to a bunch of different people, it\'s quite a large little field. It includes things like surveillance, autonomous weapons, fairness, privacy. Things like having self-driving cars not kill people. So anything that you could imagine that could make an AI safe. It also includes things that are more in what is currently called AI alignment. Have you heard of that term before?

**0:24:11.1 Interviewee:** No.

**0:24:12.3 Vael:** Cool. AI alignment is a little bit more long-term focused, so imagining like as we continue scaling systems, how do we make sure that the AIz continue to do what humans intend them to do, what humans want them to do, don\'t try to disable their off-switches just by virtue of optimizing for some goal that\'s not what we intended. Trying to make sure that the AIs continue to be aligned with the humans. Where one of the definitions of alignment here is building models that represent and safely optimize hard-to-specify human values. Alternatively, ensuring the AI behavior aligns with the system designer intentions. So trying to prevent scenarios like the one you brought up where the designer puts in a goal that\'s not quite what the humans want, but this can get much worse as systems get more and more powerful, one can imagine. Yeah, so there\'s a community working on these sorts of questions. Trying to figure out how we could get AI aligned. Because people are like, Well, can\'t you just put in an off switch, but this AI will probably be deployed on the Internet, and it can replicate itself presumably, if it\'s smart enough, and also it may have an incentive to disable the off switch if we do things by default. But some people are like, oh well, how do we solve that? How do we make it so that it doesn\'t have an incentive to turn itself off? And one group is like, Oh well, instead of having AI optimizing towards one goal singularly, in which case it has an incentive to try to stop people from achieving its goal, one thing you could do is have it build in uncertainty over the reward function, such that the AI wants to be corrected and wants to be switched off so that it has more information about how to achieve the goal. And this means that AI no longer has an incentive to want to prevent itself from being switched off, which is cool. And you can sort of like build\... that\'s like an alignment solution you can build in. Another thing that people talk a lot about is if\... we should obviously have human feedback if we\'re trying to get an AI to do what humans want. So how do we build in human feedback when the systems are more and more intelligent than us and dealing with really big problems that are like, should we do this nuclear fusion reactor? I don\'t know, the human doesn\'t really know. What are all the considerations? Another question they tackle is, how do you get honest reporting out of AI? How do you build a loss function such that it\'s incentivized to your honest reporting? And how do you get an AI in general to defer to humans kind of always while still being agentic enough to do things in the world?

**0:26:39.3 Interviewee:** Got it.

**0:26:40.4 Vael:** How does all that sound to you?

**0:26:43.0 Interviewee:** Very important. I feel like AI safety is something we\'re brushing up against the sharp edges of right now. But a lot of what AI alignment is looking into, I think, would also just apply to how we think about building these larger models. Because if that\'s the goal of these large experiments, I would hope that they\'re thinking about these right now, given their funding and the immense amount of time and effort they have that goes into it. Yeah. This is really interesting.

**0:27:25.3 Vael:** Yeah. Most researchers are working on capabilities, which is what I just call moving the state of the field forward. Because AI is actually really hard, turns out, and so lots of people working on that, and there\'s a lot of money in the space too, especially with applications. And then there\'s a smaller community of people who are working on AI safety, more like short-term AI safety, like there\'s plenty of problems with today\'s systems. There\'s an even smaller community of people who are working on long-term safety. They\'re kind of the group that I hang out with. And I\'m worried that the group of people working on long-term safety will continue to be significantly smaller than the capabilities people, because while there\'s more money in the space now, because a couple of very rich people are concerned about it like Elon Musk, famously. But generally, it\'s hard work. It\'s like anticipating something in the future. Some people think it\'s a very far future. It\'s hard to speculate about what will go on, and that\'s also just kinda like a difficult problem, alignment. It might not be trivial.

**0:28:25.5 Interviewee:** Yeah, and I sort of see parallels with nuclear research that happened like 100 years back, right? Or 80 years back, around the time of the Manhattan Project, where most of the scientists in New Mexico were focused on capabilities, like, \"How do we make this thing go boom.\" There was a small group of them who were worried about the ramifications of, \"What would happen if we did drop our first nuclear bomb and stuff like that?\" And of course, there\'s always been the same thing about space exploration. So with nuclear stuff, I think that we ended up, again, with more worries about nuclear safety than nuclear alignment. So, \"What do we do to keep reactors safe?\" Instead of thinking about like, \"Okay, should we pursue nuclear technologies where countries cannot enrich uranium and make warheads with it? What if we went with thorium reactors instead?\" And again, it\'s been more focused on capabilities and safety than alignment.

**0:29:32.9 Vael:** Yeah, I don\'t think we really need alignment with nuclear stuff because nuclear stuff doesn\'t really have intelligence per se. It\'s not an optimizer, is what I meant. Yeah. So you\'re not trying to make sure that the optimizer does whatever your goals are.

**0:29:51.5 Interviewee:** Yeah. It\'s humans at the end of the day. Yes, yes, yes, that\'s right.

**0:29:55.2 Vael:** Yeah, yeah. I also think the Manhattan Project is a very good comparison here. Or nuclear stuff in general. There\'s plenty of issues in today\'s world, there\'s so many issues in today\'s world. But the reason I focus on long-term risks is because I\'m worried about existential risks. So what happens if all of humanity goes extinct. I think there\'s a number of things that are possibilities for existential risk. I think advanced AI is maybe one of the highest ranked ones in my head, where I\'m like, \"Well, if we have an AI system that is by default not completely aligned to human values and is incentivized against us, that\'s bad. Other ways that I think that we can have AI disasters are like AI assisted war, misuse, of course, and loss of control and correlated failures. So what if we use AI to automate food production and manufacturing production, and there\'s some failure partway along the line and then correlated failure? Or what if it results in some sort of pollution? Are not really sure who\'s in charge of the system anymore, and then we\'re like, \"Oh, well, now, there\'s some sort of poison in the air,\" or there\'s like something in the water, we\'d like messed something up, but we don\'t really know how to stop it. Like coordination failures is another thing that I think that can happen even before we do very advanced AI. So generally, I\'m worried about AI.

I think nuclear stuff is also\... You can kill a lot of people with nuclear stuff. Biological weapons\... tou can do synthetic bio. There was that paper that came out recently that was like, \"Well, what happens if you just put a negative sign on the utility function to generate cool drugs?\" Then you get poisonous drugs, amazing. Yeah, that was a paper that just came out in Nature, I think maybe two weeks ago, made quite a splash. So I think if we had something that\'s much more deadly, is harder, takes longer to appear, spreads faster than COVID, I\'m like, hm, that\'s\... You may be able to\... If people have bunkers, that\'s okay, and if people are very distant, maybe it\'s okay, but like\... I don\'t know. It\'s not good. And then climate change also. I think that one will take much longer to kill humans. In terms of the time scales that I\'m thinking of. We\'re looking at 3 degrees of warming in the next 100 years, or is it 50? I don\'t quite remember, I think it\'s like we\'ve got a bit more time, and I kind of think that we might get technical solutions to that if we wait long enough, if we advance up the tech tree, if it takes a couple hundred years. So these are what I think of when I think of existential risks.

And the Manhattan Project is interesting, tying it back. There was some risk that people thought that if you deployed a nuclear bomb, then it would set the atmosphere on fire, and thus result in nuclear winter and kill everyone. And it was a very small percentage, people thought it won\'t happen, and they did it anyway, and I\'m like, \"Oh boy, I don\'t know how many\... get lucky of those we have." Because AI might also go perfectly fine. Or it might not. And I think there\'s a decent\... In a study in 2017 by Grace et al., where they were asking researchers from NeurIPS and ICML how likely they thought it was that AI would have extremely bad consequences, and the median answer was like 5% or something, and I was like, \"5%?! That\'s like real high, man.\"

**0:33:03.0 Interviewee:** Yeah. \[chuckle\] And that\'s sort of an indication of what percentage of ICML and NeurIPS researchers are working on like fairness and privacy. I feel would align pretty closely with that, yeah.

**0:33:15.9 Vael:** Oh, interesting. Yeah. Yeah, so very bad outcomes could be many different things. I think not many people are thinking about what happens if you literally kill all humans because it\'s a weird thing to have happened, but I think we\'re in a very weird time in history where 10,000 years ago everything was the same from lifetime to lifetime, but now we have things like nuclear weapons where one person or a small chain of people can kill like so many more humans than previously.

**0:33:42.9 Interviewee:** Yeah. And yeah, speaking of tech tree, one sort of linchpin that would really accelerate if we can get new AI, highly intelligent AI, would be if quantum computing becomes feasible. And we\'re actually able to run AI on quantum computers, then I think we\'re way closer to actually having a highly intelligent AI because that\'s an infinite state space right there.

**0:34:22.3 Vael:** Yeah. Yeah, yeah, I know very little about quantum myself, because I was like, \"What are the hardware improvements that we could see?\" And people are like, \"Optical, maybe coming in optimistically in five years from now,\" and I was like, \"Okay, that\'s soon.\" And then there\'s quantum where people are like, \"That\'s significantly further away,\" and I\'m like, \"Okay.\" And then software improvements also. So there\'s just been a lot of progress in AI recently, and we\'re training all the young people, and there\'s China-US competition, and there\'s just a ton of investment right now, where I\'m like, \"We\'re moving fast.\" So interesting.

**0:34:56.4 Interviewee:** Yep, yep, yep.

**0:34:57.8 Vael:** Yeah. Well, you are very agreeable to my point of view here. \[laugh\]

**0:35:06.3 Interviewee:** Yeah, I\'m just pessimistic, and I\'ve watched too much sci-fi dystopia, I guess. One of the downsides of\... It\'s great to democratize AI, but if your systems that say like, \"Hey, just upload your dataset, and we\'ll give you good AI at the end of it,\" if it\'s not at least asking you questions about like, \"What sort of fairness do you want? What should the outcome be?\" Most humans themselves are going to be thinking like, \"Oh, give me optimal, minimum cost or something like that.\" Humans already don\'t factor human values in necessarily when deciding what to do. So I\'m just pessimistic about how well we can do it. Like 50 years ago, people thought that we\'d be in space colonies by now, but sadly, we\'re not.

**0:36:10.5 Vael:** Yeah, very hard to predict the future. Okay, so my second to last question is: what would convince you or motivate you to work on the safety type of areas?

**0:36:25.8 Interviewee:** If I saw that this was coming much sooner. I think the fact that I\'m seeing this as like 100 years out, a lot of smarter than me researchers will hopefully be concerned enough about it.

**0:36:44.3 Vael:** Yeah. How soon would be soon enough that you\'re like, \"Oh, that\'s kind of soon?\"

**0:36:47.7 Interviewee:** Fifty.

**0:36:49.7 Vael:** Fifty?

**0:36:50.4 Interviewee:** Yeah. If it\'s fifty, then that\'s in my lifetime. And I\'d need to start worrying about it. That would be a bigger\... The existential risk of that would be much higher than the risk I see of just AI safety in day-to-day life, so that\'s sort of how I\'m weighing it. So for me, it\'s like: 100, it\'s like AI safety matters more. Yeah.

**0:37:15.0 Vael:** Yep, that seems very reasonable to me. I\'m like, \"Yep, it seems right.\" Cool. And then my last question is, have you changed your mind on anything during this interview, and how was this interview for you?

**0:37:26.9 Interviewee:** Have I changed my mind? I\'m definitely wondering if I\'ve overestimated when highly intelligent AI could come through. (Vael: \"I can send you some resources so you can make your own opinion, etcetera.) Yeah. But\... Otherwise, I don\'t think I\'ve changed my opinion. I still feel pessimistic, and I hope that we start moving towards smaller AI that solves one problem really well, and we don\'t just think that like, \"Hey, it\'s a perceptron. It figured out that this was the digit 9, and hence it can figure out a whole bunch of these other things.\" I hope that we don\'t start barreling down that track for too much longer. Yeah.

**0:38:21.5 Vael:** And when you say pessimistic, you mean like pessimistic about society and not pessimistic about the pace of AI? Or like societal impacts?

**0:38:28.3 Interviewee:** About both. I think we tend to put stuff out without really, really considering the consequences of it. But also I think AI has done a bunch, but it requires a lot of energy and a lot of funding that I\'m not sure necessarily is going to stay up unless we start seeing a lot bigger breakthroughs come through.

**0:38:57.5 Vael:** Interesting. Yeah, I kind of think that the applications will keep it afloat for quite a while, and also it sounds like we might be entering a race, but I don\'t know.

**0:39:05.5 Interviewee:** True, yeah. Yeah, maybe that\'s what has changed in my mind through this interview, is like, \"Okay, this is probably\...Things are just going to keep going bigger.\"

**0:39:18.9 Vael:** I think it\'s quite plausible.

**0:39:22.2 Interviewee:** Yeah, otherwise, interview was great. This was really interesting. For example, the stuff you brought up on\... I would love it if you could send me papers on the\-- (Vael: \"I would love to.\") \--the uncertainty\... You were talking like, programming a way to get out the AI turning off its own off-switch. I would love to read more of these alignment papers, they sound really cool.

**0:39:51.2 Vael:** Awesome! Well, I\'m excited. I will send you a whole bunch of resources probably, and I advise you\-- it might be overwhelming so pick only the stuff that is good, and I\'ll bold some interesting stuff.

**0:40:00.9 Interviewee:** That would be super helpful. And hope you don\'t mind my sending following up questions.

**0:40:05.7 Vael:** No, no. Again, lovely. Very happy to do that.

**0:40:09.4 Interviewee:** Yeah. Awesome. Yeah, thank you so much. This was, like I said, a really interesting experience. I\'ve never had to think about longer term impacts. Most of my stuff is like, \"Okay, GDPR is out. What does this mean for AI?\" And that\'s a very immediate concern. It\'s not this like, Okay, where should\... Or even thinking like, "Okay, five years out, what do we want privacy legislation to look like?\" That\'s something I think about, but not, \"Oh my God, there\'s a decision-making AI out there. Does it care about my privacy?\" So, yeah.

**0:40:47.0 Vael:** Yeah, yeah, I think people aren\'t really incentivized to think about the long-term future.

**0:40:51.8 Interviewee:** Yeah. Humans are just bad at that, right? Yeah.

**0:40:53.5 Vael:** And it\'s hard to forecast, so it makes sense.

\[closings\]

NeurIPSorICML_lgu5f-by Vael Gates-date 20220322.md