# Interview with AI Researchers individuallyselected_92iem by Vael Gates

**Interview with 92iem, on 3/21/22**

**0:00:05.2 Vael:** Alright, here we are. So my first question is, can you tell me about what area of AI you work on in a few sentences?

**0:00:15.2 Interviewee:** Currently, I work a lot with language models, but that wasn\'t always the case.

\[\...\]

**0:00:57.4 Vael:** Great, thanks. And then what are you most excited about in AI? And what are you most worried about? In other words, what are the biggest benefits or risks of AI?

**0:01:08.5 Interviewee:** So obviously, I think the progress in language models in the last couple of years has been pretty astounding. And the fact that we can interact with these models in more or less in the natural way that we would like to interact with it just has opened up so much in terms of getting feedback from humans and stuff like that. So I think just the progress in language models, and then coupled with that, the more recent progress in using essentially some of the same techniques to do image modeling, so that you have the possibility to do just seamless multi-modal models. I think that\'s quite exciting. Some people think that\... You know, it\'s not like most of us can just paint a photographic scene and show it to other people. So it\'s not like\-- the photographic aspects of generative image models is not what excites me, it\'s the fact that humans manage to communicate quite a bit with diagrams and stuff like that. When we\'re doing science, you can draw little stick figures and pretty much convey what you need to convey, and that coupled with natural language should give us the ability to start thinking about getting AI to do math and science for us, and I think that\'s the thing that is most exciting to me.

So I know that a lot of people are excited by the idea that you can essentially have a Google that\'s a bit\... It\'s smarter, right? You can just talk with it and say, Hey, tell me a bit about this tree, and AI says something and you say, Oh, but what about that tree? That\'s fun, but I really feel like humans are not bottlenecked by the inability to ask about trees and buildings and trivia, essentially. I think where we\'re bottlenecked is like progress in science. I think, for example, so it\'s pretty clear that the political solution to climate change\-- the time for that has kind of come and gone. I mean, we can slow it down. If we, like the whole world, suddenly decided to say we\'re going to do something about this, maybe you slow it down, but I think just the timing is a little bit off. So a lot of that\'s going to be have to be a technological solution. And as amazing as technological progress has been, I think we\'re not fast enough when it comes to developing solutions to a lot of our problems. And I do think in 10, 20 years, AI is going to play a big role, both in the specialized domain in the sense of AlphaFold, where you really just come up with a system that does the thing you want it to do, but more impactfully, perhaps, by having an army of grad student-equivalent language models that can help you answer questions that you need answered. So that\'s very exciting, right.

**0:04:24.1 Vael:** Yeah. It\'s a cool vision.

**0:04:26.5 Interviewee:** I think the risks are\... It\'s almost banal, right? Like with most technologies bad actors can make arbitrarily bad use of these things. So yeah, when they start weaponizing these things\... I\'m a little bit less concerned than some people are about like, Oh, but what if we have AIs that write fake news. Like all of that is to some extent present now, and I guess it\'s just a question of degree, to some extent. Okay, people argue that that difference in degree matters, and they\'re not necessarily wrong. I just, the thing that bothers me more definitely is very specific, malicious uses of AI. So there was a recent paper, this is so obvious that it\'s almost dumb, but someone said, Oh, yeah, we put an AI to trying to develop a drug that, let\'s say, reduces the amount of poison, and all you have to do is change the objective function, flip the sign and suddenly it just optimizes for the most poisonous thing you can possibly find. That coupled with technologies like CRISPR and stuff like that just creates a pretty dangerous\... puts very dangerous tools at people\'s disposal. So I would say that\'s the thing that I would worry about.

**0:05:54.9 Vael:** I have been impressed by how everyone I\'ve talked to in the past week has mentioned that paper, and I\'m like, good, things get around.

**0:06:01.8 Interviewee:** Well, Twitter. Thanks to Twitter.

**0:06:04.7 Vael:** Nice. Alright, so focusing on future AI, putting on a science fiction forecasting hat, say we\'re 50-plus years into the future. So at least 50 years in the future, what does that future look like?

0:06:17:9 Interviewee: For AI?

0:06:20:5 Vael: In general, where if AI\'s important, then mention that.

**0:06:27.4 Interviewee:** I see. So 50 years, oh my God. Fifty years is long time away. Assuming that we\'ve managed not to have nuclear conflicts between now and then, which is just one of those things that now you have to put at least a one digit probability on these days. But, yeah, I think that we will end up having\... Well. The optimistic scenario is that we ended up solving a few key problems. One is transitioning mostly out of fossil fuel, so a combination of solar and fusion power. I think that\'s going to be huge, and I think that AI will have played a role in some of that development. And I think 50 years from now, I think unless we are monumentally blocked in the next couple of years, AI will be pretty omnipresent in our lives, and certainly in the scientific sectors. So one thing that I\'m a little bit, just something that comes to mind, is that a lot of people are into this idea of these sort of augmented\... I don\'t know if people are literally willing to wear glasses, but certainly you could imagine having little ear buds that are fairly unobtrusive that go around your ears or something, and they do have a camera, so you can just ask it, whatever you need, you can ask it questions.

In 50 years, I think at that point, maybe some people will have worked out direct neural interfaces with stuff, and so maybe the more adventurous people will have a bit of augmented memory or at least the ability to sort of silently query their little augmented system. I think that might be a thing. Not everyone will have adopted it, I think it\'ll be a weird world. I personally\-- I\'ve never been a huge, like the fastest adopter of technology, but that sort of stuff is next level, and I don\'t know what that\'s going to look like.

I also\... well, two things, I guess they\'re kind of linked. I think that people will live substantially longer. I think, unless something miraculous happens, I don\'t think they\'ll be living like 200, 300 years, but I certainly think it\'s possible people will be living to 150 or something like that. Not people born now; I\'m not going to live to 150. Someone was telling me that people born, these days, they\'re going to live to see the year 2100, right. That\'s not quite in the 50-year time frame, but yeah, I certainly think people born today are going to be living, like their average lifespan in industrialized countries, assuming a certain level of privilege, they\'re going to be able to live quite a bit longer. That coupled with AI possibly automating quite a few jobs is going to change the social landscape a bit.

One thing that occurred to me recently\... so people used to say that\-- well, people say many things\-- one is that, this, unlike industrialization\... Some people always say technological progress destroy some jobs but creates more jobs on the other side. And then some say, Okay, but this one is different because you\'re automating intelligence and that really does put humans out of their main forte. So one of the things that people worry most about, in addition to the universal-based income stuff, is just the loss of dignity, that people always assume that even people who don\'t have what you would call glamorous jobs value the fact that they work and get paid for that work. But I think some of the stuff that happened during Covid makes me doubt that a little bit, in the sense that people did quit jobs that ostensibly looked good. Even in the tech sector where people, I felt like generally, they\'re not the worst jobs by any stretch, and they were like, No, this is meaningless, I want to go do something meaningful with my life. So I think the recent, the past couple of years have made me question the idea that it would be that big of a psychological blow for people to not work for money. That if you did establish a universal basic income, plus you\'d have to solve some other, many complicated issues, but I don\'t think people will be that unhappy to be not having to work menial jobs. I\'m not saying there\'s not going to be upheaval, but I think it\'s going to be like a combination of living longer, and not possibly having to do jobs if you don\'t want to do them. I think that\'s just going to be, I don\'t know. It might be a nice change. In the optimistic scenario, I guess.

**0:11:35.9 Vael:** Got it. Yeah. Well, my next question is related to that. So people talk about the promise of AI, by which they mean many things, but one of them is maybe having a very general capable system such that it will have cognitive capacities to replace all current day human jobs. So you might have a CEO AI or a scientist AI. Whether or not they choose to replace human jobs is different, but have the ability to do so. And I usually think about that and the fact that 2012 we have AlexNet, deep learning revolution, here we are, 10 years later, we\'ve got things like GPT-3, which can do some language translation and some text generation, coding, math, etcetera, a lot of weirdly general capabilities. And then we now have nations competing and people competing and young people going into this thing, and lots of algorithmic improvements and hardware improvement, maybe we get optical, maybe we get quantum, lots of things happening. And so we might actually just end up being able to scale to very general systems or we might hit some sort of ceiling and need to do a paradigm shift. But regardless of how we do that, do you think we\'ll ever get very general AI systems, like a CEO or a scientist AI, and if so, when?

**0:12:39.1 Interviewee:** I don\'t know about CEO AIs. The scientist AIs, yes. Yeah, and that\'s going to come in stages. So obviously the current generation of AIs, we don\'t put them in human bodies and let them do experiments and stuff like that, right. It\'s going to be a while before we start letting them operate like particle accelerators. Fifty years\... Maybe in 50 years. My original background is \[non-AI field\], and I really could have just done my entire PhD from a desk, and that sort of work, certainly, AI can replace, I think, to a huge degree, from idea generation to solving the answer and writing a paper, yeah, that just feels so doable. Again, unless we hit a giant wall and find that our current transformers simply cannot reason, but I think that looks unlikely. I don\'t rule it out, but that looks unlikely to me.

**0:13:46.8 Vael:** Yeah. Okay, what about this CEO AI, like with multi-step planning, can do social inference, is modeling other people modeling it, like crazy, crazy amount of generality. When do you think we\'ll get that, if we will?

**0:14:00.7 Interviewee:** Yeah, that\'s not the part that I\'m worried about. AI can certainly model human intent, but\... I guess it depends on what you want from your CEO AI. And this I think gets at a little bit one of my dissatisfactions with discussions about human\-- like, AI alignment. It\'s not that people don\'t talk about it, but it\'s rarely talked about. I don\'t know, on Twitter certainly. A lot of AI alignment stuff talks about\-- they \*don\'t\* talk about the fact that humans disagree wildly on what humans should do. So I\'m thinking about this in connection with the CEO, because I think in the limit, AI will be able to do anything, any specific thing you ask the AI to do, it can do, but the question of whether you would want the AI to be CEO, I think that\'s mostly a human question. So that\'s why I said\-- I think that\'s a policy decision, not a AI capability question.

**0:15:25.3 Vael:** Got it, yeah. Do you think that people will end up wanting\... that there will be economic incentives such that we\'ll eventually have things like CEO AIs?

**0:15:36.0 Interviewee:** I guess in some sense, no, because I think a human would still be the CEO and then you would have your AI consultant, essentially, that you would ask all the things. You would delegate almost everything, but I think that people would still want to be at their very apex of a corporate hierarchy. It seems weird to put a robot in charge of that, just like\... why. It\'s a title thing, almost, like, why would you make the robot the CEO?

**0:16:03.2 Vael:** Yeah, yeah. In some vision of the future I have, I have the vision of a CEO AI\... we have a CEO AI and then we have shareholders, which are the humans, and we\'re like, \"Alright, AI, I want you to make a company for me and earn a lot of money and try not to harm people and try not to exploit people and try to avoid side effects, and then pass all your decisions, your major decisions through us\" and then we\'ll just sit here and you will make money. And I can imagine that might end up happening, something like that, especially if everyone else has AI is doing this or AIs are way more intelligent and can think faster and do things much faster than a human can. I don\'t know, this is like a different future kind of idea, but.

**0:16:46.8 Interviewee:** But that seems so weird. Because, then\-- so, assuming\... I don\'t know if everybody has access to the same AI in that scenario, but like it can\'t be the case that 100 people all say to their own individual AI, \" Form a company and turn it into a \$100 billion company or a \$1 trillion company\", and they all go out at optimizing. I think at that point, in that kind of world, I think there would have to be a bit more coordination in terms of what goes on, because that just creates some nasty possibilities in terms of bringing the economy down. So I don\'t know that that\'s how things would just happen. It cannot be the case that we would just say, \"Robot, figure out how to make a trillion dollar company. I\'ll give you this one idea and just run with it,\" and then just like we are hands-off. That seems extremely unlikely, somehow.

**0:17:38.9 Vael:** Yeah, I\'m interested in how that seems very unlikely. It seems like to me\... Well, we were talking about scientist AI, and I imagine we can eventually tell a science AI to like solve cancer, and maybe it will actually succeed at that or something. And it seems like it\'s different, to be like, Hey, CEO, make a ton of money for me. Is that getting at any of the underlying thing or not?

**0:18:08.4 Interviewee:** Uh, hm. Yeah, so I think even there, I think you would never tell an AI to \"solve cancer\". Well, yeah\... You would want to give it more specific goals, and I think\... In any scenario where we have full control over our AIs, we wouldn\'t want such vague instructions to be turned into plans. That\'s a scary world, where you can just say solve cancer and the robot runs with it. I think for the same reason, I don\'t think you would want a world where someone can say, \"AI, make a lot of money for me,\" and that\'s the only instruction the AI has, and it\'s allowed to intervene in the world with those instructions. So yeah, that\'s why I don\'t see, just like from a sanity perspective, you would\-- you never want to unleash AI in that manner, in such a vague and uncontrolled manner. Does that make sense?

**0:19:03.6 Vael:** Yeah, that makes sense that you wouldn\'t want to be\... because it\'s very unsafe, it sounds like, or it could be\--

**0:19:09.7 Interviewee:** Yeah, kind of insanely unsafe, but\...

**0:19:14.8 Vael:** Nice. Yeah, do you think people might end up doing it anyway? Sometimes I feel like people do unwise things in the pursuit of, especially unilateral actors, in the pursuit of earning money, for example. Like, Oh, I\'ve got the first scientist AI, I\'m going to use it to solve the thing.

**0:19:35.3 Interviewee:** That\'s a good question. I think, I really do think you would want\... Yeah, I wonder about how you would actually enforce any kind of laws on AI technology. It\'s the most complicated thing to enforce, because nuclear weapons\-- One of the nice things about nuclear weapons is it\'s actually pretty hard to develop nuclear weapons in secrecy without releasing any radiation, that\'s one of its few good points. I think AI, it\'s true that you could just develop and run it. But I think at the point where any AI has to interface with the real world, whether it\'s in the stock market or something like that, I do think that people will start seeing the need for finding ways to regulate the speed. Even high frequency trading is starting to be, like you can\'t interact with it, any kind of stock market in less than one nanosecond or something like that. I think similarly, there\'s just going to be some guardrails put in place. If there\'s any kind of sanity in terms of policymaking at that time, you would want guardrails in place where you could not unleash AI with such large powers to affect a large part of the world with minimal intervention powers. Yeah. This is all assuming there\'s a sane policymaking environment here, but\...

**0:21:04.8 Vael:** Yeah. Do you think there will be?

**0:21:09.3 Interviewee:** I think so. I think so, I\'m hopeful in that regard. I\'m not saying that Congress is ever going to really understand the nuances of how AI works, anything like that, I just think there would be too many\... Even in a world where only OpenAI and DeepMind have full AGI, I don\'t think they\'d want to create a world where one of them can unleash something at the level that you described. And I also think that when those two companies get close, they\'re going to wonder if other states, say, Russia or China, are going to be close, and they\'re going to start wanting to really hammer down, hammer out\... like there will be a sense of urgency, and hopefully they have enough influence to influence policymakers to say, \"You need to take this seriously.\" And this is where I think almost the fact that it takes\... Okay, I said earlier that, you know what, the nice thing about nuclear weapons is that you could detect it, but I think one of the nice things about the fact that right now, it looks like you\'re going to require enormous compute to get anything that is remotely AGI. That\'s the thing that allows maybe\... That means the only huge corporations or states will be able to do it for at least some period of time, and hopefully those are the same actors that can somehow influence policymaking. If there were just one person, if they just had the ability to do that, it would be a little bit problematic, actually. So in some sense, because these institutions are big, I think they\'re going to be both constrained a bit more in terms of what they can do, and also they\'re going to be able to, if they are well-intentioned, to influence policymaking in a good direction.

**0:23:06.2 Vael:** Do you think they\'ll be able to do international cooperation? Because I imagine China will also have some AI companies that are also kind of close, I don\'t know how close they will be, but\...

**0:23:17.7 Interviewee:** They\'ll try. I don\'t know that China will listen to the US or Europe. I agree that\'s not going to be easy, yeah. Who knows what they\'re up to exactly, there.

**0:23:31.0 Vael:** Yeah, it seems like they\'re certainly trying, so\... Yeah, another one of my questions is like, have you thought much about policy or what kind of policies you want to have in place if we are getting ever closer to AGI?

**0:23:48.8 Interviewee:** Actually, I haven\'t given it that much thought, what the laws would specifically look like. What I don\'t think is really possible is something like the government says, You now need to hand over control over this to us. I don\'t think that\'s super feasible. Yeah, I can\'t say I have a good idea for what the laws would specifically look like. I think as a starting point, they\'ll certainly create some kind of agency to specifically monitor\... Actually, right now, there\'s no agency like the SEC or something like that that monitors what exactly goes on in AI. I mean, there\'s some scattering of regulations probably somewhere, some vague export controls and stuff like that. But yeah, they\'d certainly start creating an agency for it, and their mandate would start to grow. I think it might, again, have to be something like what we do with nuclear reactors, where you have an agency that has experts inside of it, and that they are allowed to go into companies and kind of investigate what\'s going on inside, just as, if Iran is developing nuclear weapons and they agree to let inspectors in. I think it\'s going to be up to something like that. And then, yeah, similar to these nuclear treaties, perhaps there would have to be something along the lines of like\... there are certain lines you cannot cross with AI, and if someone does cross it, that institution or the country as a whole gets sanctioned. It\'s going to have to be at that level. Certainly, given the power of the putative AI that we\'re thinking about. I think the regulations are going to have to be quite dramatic if it\'s going to have any kind of effect.

**0:25:45.1 Vael:** Yeah. One thing I think that is a difference between the nuclear situation and the AI situation is that nuclear stuff, seems not very dual use. Well, nuclear weapons, at least, not very dual use. Versus like AI has a lot of possible public benefit and a lot of economic incentives, versus like you don\'t get, I don\'t know, you don\'t benefit the public by deploying nuclear weapons.

**0:26:05.7 Interviewee:** But nuclear reactors, but that\'s the whole\--

**0:26:07.9 Vael:** Nuclear\-- Yes, you could\--

**0:26:09.0 Interviewee:** That\'s the whole\... So Iran would always pretend that, Hey, we\'re just developing nuclear reactors for power. Just the problem is that was always very easily converted to nuclear weapons. I think that could be a similar---

**0:26:24.9 Vael:** Yeah, yeah, it is similar in that way. Somehow it still feels to me that these situations are not quite analogous, in that the regulations are going to be pretty different when you\'re like, \"I am going to make sure that you\'re not doing anything bad in this area,\" and people are like, \"ah, yes, but we need to get the new smartphone, scientist AI, etcetera.\" But yeah, I take your point. Another thing that I think is interesting is that current day systems are really pretty uninterpretable, so you\'re like, \"Alright, well, we have to draw some lines, where are we going to draw the line?\" What is an example of what a line could be, because if there\'s government inspectors coming in to DeepMind and you\'re like, \"Alright, now inspect,\" I\'m like, what are the inspecting?

**0:27:11.2 Interviewee:** Yeah, so when you say interpr\-- so that\'s another thing about\... one of my pet peeves about interpretability. People are not that interpretable. People hardly, rarely know what\'s going on in other people\'s heads, and they can tell you something which may or may not be true, sometimes they\'re lying to you, and sometimes they might be lying to themselves. When a doctor tells you, \"This is what we\'re doing,\" unless you\'re another doctor, you rarely understand what they\'re saying. And so, yeah, this is a total tangent on like my\... the thing around, the discussion around interpretability is always such a mess. But what are they inspecting? If we\'re imagining inspectors, they could certainly go in and say, like, if it\'s a language model, you can certainly allow them to query the language model and see what kind of answers, what kind of capabilities these language models have. You could say, if it\'s a language model, just totally hypothetically, you could say, \"Alright, develop me, write me a formula for a bioweapon,\" and if the language model just gives that to you, then possibly you have a problem. Stuff like that. So if a company that has that capability hasn\'t put in the required fail-safes like that, then they can be held liable for X amount of problem, the trouble, right.

**0:28:58.3 Vael:** Interesting. Cool, so that\'s cool. You\'ve got like a model of what sort of rules should be in place, and it sounds like there should be rules in place where you can\'t develop bioweapons or you can\'t feed humans bioweapons when they ask for them.

**0:29:12.0 Interviewee:** Yeah, stuff like that. In this inspector model, I think that\'s what would kind of have to happen. But yeah, it\'s not like I\'m an expert in this, but that\'s what I would think.

**0:29:24.4 Vael:** Yeah, something I\'m worried about is that no one is an expert in this. Like policymakers\-- when I talk to the AI researchers, they\'re like, Oh, yes, the policymakers will take care of it, and I\'m like, the policymakers are busy, and they\'re doing many different things, there\'s not many people who are focused singularly on AI. Also, they\'re mostly focusing on current day systems at the moment, so like surveillance and bias and transparency, and like a bunch of different things, so they\'re not really thinking very future at the moment. And they don\'t know what to do because they don\'t understand the technology, because the technology moves extremely fast, right, and so like AI researchers are the ones who know it. And I\'m like, Alright, AI researchers, what should we tell them to do. You\'re like, Well, we should make a list of things that the AI shouldn\'t do, like basic fail-safes. And I\'m like, Great, it would be super cool if that was written out somewhere and then we can start advocating for it or something, because I\'m worried that the policy will just continue to lag really far behind the actual tech levels, except where like\... Policy is already several years behind, maybe like 10 years or something, and will continue to be that far behind even as we\'re approaching the very powerful AI.

**0:30:31.9 Interviewee:** Yeah, so a couple of things there. One is that that\'s why you need more of an agency model rather than laws, because creating laws is very, very slow, whereas an agency can drop some rules and maybe they start enforcing them. And so you do need a sensible agency that doesn\'t create bad rules, but the ability to be flexible. That said, I think\... The biggest problem with policymaking right now is that the policymakers don\'t understand AI at all, right. And you sort of hinted at that. And I think\... If I\'d asked myself, at this moment in time, is there anything that, any rule that we need at this moment in time, I\'m not sure there is. AIs are not there yet.

**0:31:25.2 Interviewee:** So at this moment in time, I think if you ask most researchers, \"Hey, do we need to create specific laws to prevent X, Y, Z,\" I\'m not sure many people would tell you, you need that. And so these laws, I think, are going to have to come in at very sensible points, and it\'s not clear to me that the policymakers are going to know when that time point is. I would say even in the AI field, very few people know when that\'s going to be. There\'s a lot of stuff coming out of especially big labs where the world doesn\'t know. There\'s like 100 people that know what\'s coming in the next year. I don\'t know what a good solution to that is.

**0:32:17.5 Vael:** Especially if we can get AIs that can generate lots of deadly poisons already. Yeah, I think it\'ll maybe be hard to tell, and then also one needs to develop a list, if there\'s going to be in list form or\...

**0:32:32.2 Interviewee:** The problem is, I think it\'s easier to regulate general AI just because it\'s going to require so much compute. But I think more specific AI that anyone can run on a GPU, like on a laptop, is more or less impossible to regulate. So it\'s not clear to me what the law would be, except if you use a bioweapon, you\'re in trouble. That law already exists, right.

**0:33:00.5 Vael:** Yeah, I think that one already exists, so\...

**0:33:05.4 Interviewee:** So I think in some sense, like kind of in the trade-off of what can the technology do right now, and who might try to deploy that, our laws sort of cover the problem cases at the moment. I think where I get a little bit stuck is if you try to say, \"Alright, in five years, should we have laws banning certain uses of a very, very capable general model?\" I do think at that point, Congress should seriously consider creating a regulatory agency. And I think AI researchers will only support this if there\'s some semblance of like, kind of like NASA, where there\'s some faith that engineers are in charge of this thing, that kind of know how these systems work, that they can think rationally about both the technological side and the policy side of things. And so that\'s going to take some work on the side of whatever administration is in power at that time. But yeah, it\'s not going to be easy. I think it\'s going to take a very capable administration to handle that transition gracefully.

**0:34:19.5 Vael:** Yeah, that makes sense. Yeah, I\'m worried about a few different things in this future scenario. I\'m like, Okay, I don\'t know if the agency will be developed while\-- in a sort of future thinking sort of way, I don\'t know that it will implement the right type of policies, I don\'t know that it will have the power to really enforce those policies, I don\'t know if it will have the power to enforce internationally. But I do like the idea that\-- but obviously one should still try, and it seems like there should probably be a lot of effort going into this, as you said, something like on a five-year scale.

**0:34:49.2 Interviewee:** Yeah, it\'s just that knowing AI researchers, there\'s just going to be such extreme pushback. If there\'s any sense that there\'s been a bureaucracy created whose job is nothing more than to just slow things down for no good reason. That\'s almost a default kind of way in which such an agency would get created, and so, yeah, it\'s just one of the situations where you have to hope that the future leaders of America are smart.

**0:35:24.7 Vael:** Yeah. Yep. A thing to bank on. Cool. So I\'m concerned about long-term risks of AI. That\'s one of the ways in which I\'m concerned, is that we won\'t get the policy right, especially as we\'re doing international competition, that there may be race dynamics, as we\'re not able to have really strong international governance. And I don\'t know if this will go well, and I\'m like, I think people should work on this.

But another way I think that things might not work: So we talked a little bit about the alignment problem. And another interesting thing about the alignment problem is\... or in my mind, so we\'ve got maybe a CEO AI, or whatever kind of AI, but this is the example I\'ve been working with, and it\'s making plans and it has to report any decisions it makes to its shareholders, who are humans, and the humans are like, \"I want a one-page memo.\" And the AI is like, \"Okay, cool, one-page memo. I have a lot of information in my brain, in my neural network, while I\'m trying to maximize profits with some other goals\-- with some other constraints.\" And it\'s noticing that if it gives certain information to humans, then the humans are more likely to shut it down, which means that it\'s less likely to succeed in its goal. And so it may write this memo and leave out some information so that it decreases likelihood of being shut down, increases the likelihood of achieving its goal. So this is not a story where we\'re building in self-preservation into the AI, but a story in which\-- why instrumental incentives of an agent achieving, trying to go for anything that is not perfectly aligned with human values, just like what humans tell it to do instead of what humans intended it to, then you might get an AI that is now optimizing against humans in some degree, trying to lie to them or deceive them in order to achieve whatever it has been programmed to do. Do you think this is a problem?

**0:37:08.2 Interviewee:** The scenario you described was exactly what human CEOs do.

**0:37:12.7 Vael:** Hm, yes. But more powerful systems, I think, with more influence over many things.

**0:37:20.6 Interviewee:** So this is the problem\-- so I think this actually still is a human problem. So if a human being\... like these AIs will be, depending on the mix of reward for not getting shut down and\... at the kind of detailed level these days, we often\... When we do RL with language models, we have two things going on, one is an RL objective, maximize the reward as much as you can, but the other objective is to tie it to the original language model so it doesn\'t diverge too much. In which case, if you are writing a memo, it would try to write a memo in the style that a human would write it, let\'s say. So the information content would be somewhat constrained by what a typical memo written by a human being would look like, and then on top of that, it would try to optimize what it is trying to do, maybe just trying to keep the company alive for as long as it can or something like that. So there is that sort of like, at least the way we do things now, there\'s a little bit of self-regulation built in there. But this is why I think, more fundamentally\... any question where if you just replace the AI with a human and ask the same question: Is this a problem or not a problem? I think that\'s more or less a human problem. And you have to think a bit more carefully about what we would want a human to do in that exact same situation. Do we have an answer for that? And then take into account the fact that the AI is more powerful. You don\'t need a super devious AI for a CEO to start lying to their shareholders a little bit, or misleading their shareholders a little bit, in order to present a more rosy picture of what the company is doing. So do we already have mechanisms that prevent that? I think we do, and that same thing would apply to the AI.

**0:39:22.1 Vael:** Yeah. I think the things that are interesting to me about the AI scenario is that we have the option of\... we are designing the AIs, so we could make them not be this way. And also having an AI that has a lot, lot more power, that is as powerful as a leader of one of the countries, and that has the ability to copy itself and could do self-improvement, so it can be smarter than it started out with. And okay, we\'ve got something\'s possibly smarter than us, which is like the ability to reason and plan well, and has the incentive to acquire resources and influence and has all the human kind of incentives here, and we can\'t\-- and it\'s not as\-- I don\'t know, it\'s maybe not as interpretable as a human, but you can\'t throw it into jail. Like, I don\'t know, there\'s a lot of the mechanisms for control, I think, are maybe not there.

**0:40:12.7 Interviewee:** Yeah, so it\'s in this sort of legal context that I think you would not want the AI to be a CEO or any\... There has to be something\... For something like this, you would want the person\... There should be a person who\'s liable for the decision being made by the AI. You have to do some due diligence to the answers that the AI gives you. There\'s no other way. Yeah.

**0:40:40.2 Vael:** There\'s generally a thing in some of your answers where you\'re like, Well, you know, any reasonable person would do X, and I\'m like, I don\'t know if we\'re in a world where we\'ve got a bunch of just reasonable people putting in appropriate fail-safes which they\'ve spent a long time constructing. And some of these fail-safes, I think, might be very technically difficult. I think the alignment problem might be quite technically difficult, such that researchers who are working on capabilities would get ahead even as the people working on safety mechanisms per se is also growing, but at less speed as all the capabilities researchers are pushing forward. Such that we might have an imbalance in how many people are working on each thing.

**0:41:16.3 Interviewee:** Yeah, so I guess maybe I\'m thinking of two different things. One is just the sheer\-- kind of like the question of just putting rules in. The other question that I often have with these discussions is, Does a sensible answer exist to the question at all? So imagine, okay, so imagine we replace the CEO with\... Imagine we replace Mark Zuckerberg with a very, very smart AI. And this very smart AI is posed with this question of, okay, there is a photo of a naked child, but it\'s in the context of a war, it\'s a war photograph. Should this photo be allowed on Facebook or not? The CEO cannot\... It doesn\'t really matter how smart the AI is, this is just not a question the AI can answer, in the sense that it\'s an indelibly human question. That\'s why\-- I just think there are certain questions where when we posit a incredibly intelligent AI, it\'s got nothing to do with that. It\'s just a question of what a group of people who\... A group of humans who disagree on what the final answer should be. In that scenario, there\'s no right answer for the AI. There\'s nothing the AI can do in that scenario that is the correct answer.

**0:42:46.4 Vael:** Yeah. I think in my vision of what I want for AI in the future, I want AIs that do what humans intend them to do, so I want the alignment problem to be solved in some way, and I want it to all involve a huge amount of human feedback. So for every question that is confusing or the AI doesn\'t know what to do, if it hasn\'t internalized human values, then I want it to ask a bunch of humans, or maybe we have some way to aggregate human opinions or something. And then we have an AI that is reflecting human values and preferences, so if humans are confused about this particular issue, then I don\'t know, maybe the default if humans disagree is not to publish\[?\]. But in general, just having some sort of checking mechanism. The thing that I\'m worried will happen by default is that we\'ll have an AI that is optimizing for something that\'s sort of right, but not quite right, and then it will just kind of now do like whatever things we put into it\-- whatever optimization goals we would put into it will be kind of locked in, and so that we\'ll eventually get an AI that is doing something kind of analogous to the recommender algorithms thing, where recommender algorithms are sort of addictive and they\'re optimizing something\-- clickthrough rate\-- that\'s kind of close to what humans value, but isn\'t quite. And then we might have an AI that is just like, A-ha, I am now incentivized to deceive humans to gain control, to gain influence, to do self-improvement, and we\'ve sort of lost control of it while it\'s doing something that\'s like almost but not quite what we want.

**0:44:05.6 Interviewee:** I think one thing that comes to mind, actually\...so this kinda goes back to the interpretability question, but I think it may be a slightly different angle on it. I think it\'s going to have to be the case where when an AI makes a decision of that sort, it should output almost a disclaimer. So the way credit card companies would write you this long disclaimer. And it would have to tell you for each decision it makes, what the risks are, and then a human has to read that and sign off on it. Now, the question is going to be, the other problem with credit card disclaimers is that they were so long that the average person couldn\'t read it and make sense of what the hell was going on. So the AI would be somewhat required to come up with a comprehensible set of disclaimers that say, Okay, I asked a bunch of people, they said this, but obviously we shouldn\'t always listen to what the majority says. I also consulted some moral ethicist or some ethicists, and I synthesized the combination of the ethicists, previous precedents, and what the general public wants. I recommend that given the combination of these three factors, you should do this. And then a person should sign off on it, and then that person in some sense should be liable to the extent that the AI gave a reasonable summary of the decision factors. So something along those lines.

**0:45:32.8 Vael:** Yeah, that sounds brilliant. I would be so excited if AI in the future had that. I\'m like, Wow, we have an AI that is incentivized to instead make things as maximally clear and comprehensible and taking into account what the human wants and listing out of things, I\'m like, If we solve that, if we have the technical problem to solve that, I\'m lnnnike, wow, amazing.

**0:45:52.6 Interviewee:** I think the key point here is at some point, the human has to be held liable for it, so that they have an incentive to only use AIs that satisfy this condition. Otherwise there\'s no reason for the.. because, like you say, you can\'t put the AI in jail, so. At some point you have to put the onus on humans. I think this is something that like even Tesla\'s going to have to think about. At some point, I mean\... I fully believe statistically, they\'ll reduce the number of accidents, but accidents will happen, sometimes the car will be the responsible party. At that point, you can\'t just throw up your hands and say no one was at fault, right? So if Tesla is willing to deploy their cars for self-driving, they are going to have to start taking liability, and that\'s going to force them to confront some of these same issues and say, Did the AI give a reasonable estimation of if we take this road\...? It has to be able to say, or like a surgical robot, it has to be able to say the same thing that doctors do, \"Listen, I\'m going to perform this operation, it\'s the best chance you have, but there is a 10% chance that you\'re going to die. If you\'re comfortable with this, if you\'re comfortable signing off on this, I will do my best,\" and only in that scenario is the doctor allowed to be forgiven if the operation goes wrong.

**0:47:20.6 Vael:** Yeah, so a part of my thinking I\'m noticing is that\... Um\... So I think you\'re very interested in problems of misuse, which I\'m also interested in, but I think I\'m also interested in the problem of, like, I think that it will just be technically hard in order to incentivize an AI to not try to optimize on \[hard to parse\] but to like, be able to take\... So currently, we\'re quite bad at taking human preferences and goals and values and putting those in a mathematical formulation that AIs can optimize, and I think that problem might just be really, really hard. So we might just have an AI that won\'t even give us anything reasonable, and I\'m like, Oh, well, that seems like step one. And then there\'s also a bunch of governance and a bunch of incentives that need to be put in place in terms of holding people accountable, since humans will hopefully be the end user of these things.

**0:48:07.8 Interviewee:** I\'m actually far less worried about the technical side of this. I just finished reading this book about von Neumann, that\'s a little cute biography of him, and there\'s a part where he says, supposedly, that people who think mathematics is complicated only say that because they don\'t know how complicated life is. And I\'m totally messing with the phrasing, but something like that. I actually think any technical problems in this area will be solved relatively easily compared to the problem of figuring out what human values we want to insert into these.

**0:48:46.4 Vael:** Okay, so you think it\'s the taking human values and putting into AI, that technical problem will just get solved?

**0:48:54.8 Interviewee:** If you know what values you want to put in, yeah.

**0:48:56.5 Vael:** Okay, cool. Alright.

**0:49:00.1 Interviewee:** I actually think that problem is the easy problem. I\'m not saying it\'s easy in an absolute sense, I just think that\'s the easier problem.

**0:49:05.1 Vael:** Got it. That feels like the alignment problem to me. So you think the alignment problem is just going to be pretty easy. This seems like a valid thing to think, so\...

**0:49:12.9 Interviewee:** I want to emphasize, I don\'t think it\'ll be easy in absolute terms, I just think it\'ll be the easier of the two problems.

**0:49:17.0 Vael:** Okay, compared to governance and incentives. Yeah, that makes sense.

**0:49:21.2 Interviewee:** That is\... I just have this faith that any technical problems humans can solve, down the line\-- like, eventually. It\'s the non-technical problems that get people all tangled up, because when there\'s no right answer, it really messes up scientists.

**0:49:43.1 Vael:** Yeah, yeah. Yeah, the problem of trying to take human values and what we care about in all the different ways and put them into a mathematical formulation feels difficult to me, and I guess it is a technical problem. I guess I do sort of think of it as a technical problem, but yeah, that makes sense that you\'re just like, Look, we\'ll get that done eventually. And then we have governance, and I\'m like, Oh, yes, governance is totally a mess. Yeah, that makes sense. And I think no one knows how, it\'s an unknown, unsolved problem\--

**0:50:13.5 Interviewee:** Let me put it this way, let me put it this way. If by human values, you mean if like\...

**0:50:18.5 Vael:** What humans intend, having an AI always doing what you say.

**0:50:22.8 Interviewee:** Yeah, so imagine for any conceivable scenario an AI that would have to deal with. We could ask you, what would you do in this case? What I\'m saying is that if for each of these questions, you are able to give a concrete answer to the answered question, such questions can be inserted into our AIs. Like if you are able to come up with clear answers for the questions for which you yourself would have a clear answer for, I think that set of moral constraints, let\'s say, I think that can be more or less inserted into AIs without huge problems.

**0:51:08.3 Vael:** Even as the problems get\-- even as I don\'t eventually have concrete answers, because the problems are like, Should we do X thing on creating nuclear reactor X and etcetera, and I lose control of the\... not lose control, but I can\'t actually visualize the whole space or something, because I\'m too\...

**0:51:25.2 Interviewee:** But at that point, what does alignment mean? Alignment usually means that what the AI does is the same as what a human would do. If there\'s no answer about what the human would do, what is the AI supposed to do?

**0:51:37.2 Vael:** I think it\'s supposed to just keep in mind the things that I care about, so try to avoid side effects in all forms, try to avoid hurting people, except for when that makes sense or something\-- oh, eugh. Anyway, like, doing a whole bunch of\... and, like reporting truthfully, which is also something I want the AI to do. And things like this.

**0:51:56.6 Interviewee:** I guess it\'s one of those, so it\'s a question of maybe a generalization, but I find it slightly hard to believe that an AI that would answer in the exact same way on all of the answered questions that you do have an answer for, when you go outside of that regime, suddenly the AI diverges strongly from what you would have answered had you been smarter. I just think that\'s a weird kind of discontinuity, right. So there\'s this huge set up\-- let\'s suppose on all the questions for which you have an answer, the AI agrees with you. And then you take that a little bit outside of your realm of comprehension, and at that point, suddenly the AI decides, Oof, I\'m freed from the constraints, I can answer whatever. I think that\'s a little bit implausible to me, assuming people did the job correctly.

**0:52:52.9 Vael:** Yeah, I think assuming people do the job correctly is pretty important here. You could have an AI that is deceiving you and giving you the correct answers, as long as you could check, but yeah, assuming that\'s not true, assuming the AI is honestly reporting to you what exactly, like everything that you said at a lower level, I mean, everything that you can confirm, then that seems great. And you\'re like, Well, if you then extend that to regimes where humans can\'t understand it, then things are probably still okay. I think I maybe believe that. I think that maybe things are probably still okay.

**0:53:23.7 Interviewee:** The validation set on whatever project you\'re working on would\... The AI wouldn\'t know necessarily whether this was a question on which you just didn\'t have an opinion. It\'s the same for me when I think about\... A lot of people are concerned that, in terms of basically over-fitting, could you\... So one of the reasons I think we\'ll definitely get an AI that can answer physics questions pretty comprehensively is that I don\'t believe you can ever create a physics AI that can fake its way through all of the train set, sorry, on the validation set, and then suddenly do poorly on something that is outside of it. There\'s so much of physics that if you are able to fake your way through all the way to graduate school\... I guess I\'m thinking in terms of like, if you happen to make it all the way through graduate school, pass all the exams you needed to pass, turned out the papers you turn out. At that point to be like, Oh, actually, I just, I faked my way through all of physics, I don\'t really understand physics. You don\'t not understand physics, like in spite of yourself, because you\...

**0:54:35.0 Vael:** That seems true. I think one of the things I\'m referencing is like a mesa-optimizer, inner optimizer, which is like an AI has some sort of goal. I don\'t know, maybe it\'s like trying to go out, go to red things, but the door is always\... Okay, sorry, try again. So it says, it says it has some sort of goal, kind of like humans do, so\... \[sorry, I\'m now\] restarting \[this sentence\]. Evolution has some sort of goal, evolution is optimizing on something like inclusive genetic fitness. And it\'s like this optimizer that is pushing things. And then there\'s humans, which are the things that are like, it\'s being optimized or whatever. And humans should ideally have the same goal of inclusive genetic fitness, but we don\'t. We\'ve got something that\'s sort of close, but not really. Like we have contraceptives, so we aren\'t really maximizing babies, we have different goals and things we\'re going for, we have like of achievement and all the values that we think are important and stuff. And so this is an example of something\... that, humans are the ones who are trying to make the AI optimize for a thing, and the AIs are like, Sure, I guess I\'ll sort of go for the goal that you want me to, and I have a model of what you want me to do, but I actually internally in my heart have a different goal, and so I\'ll make my way through all of the test sets because I have a model of what you want me to do, but as soon as you release me, then I will do something different. So that\'s like a weird analogy that doesn\'t quite work, but\...

**0:56:00.1 Interviewee:** Yeah, especially for two reasons. One is that I think there is a real likelihood that we are going to become multi-planetary, which is pretty good as far as evolution is concerned, because not only does that mean we\'ve dominated the planet, but we\'ve started having other planets and spreading our genes everywhere. So in some sense, we\'ve done exactly what evolution wants us to do, which is like reproduce our genes far and wide. I have a feeling that no matter how sophisticated and smart and industrialized in AI we get as a species, people aren\'t going to stop wanting to have babies. So like somehow, yes, we don\'t, like, kill everyone whenever we want to and just steal food and all that like animals would do, but kind of like\...

**0:56:49.9 Vael:** Yeah, I don\'t know that evolution wants that, but.

**0:56:51.8 Interviewee:** We\'re still pretty aligned to the basic evolutionary goal. That\'s what you\'re starting from, saying that\'s the original goal, and we\'ve deviated from the original goal. I think actually we\'re well within evolution\'s parameters as far as what are we supposed to do as good evolutionary creatures.

**0:57:07.5 Vael:** Cool. Yeah, that makes sense to me. And then as a last point or something, so ideally, we want AI to do what humans would do if they were smarter, like you said, if we had more time to reflect and if we maybe had more self-control or something. I don\'t know that that would kind of come out of nowhere or something? I guess this is now in the ideal case, it sounds like we\'ve mostly succeeded in aligning the AI with humans\' goals. (Interviewee: \"So what\'s the question?\") Um, ideally, I think I want AI that will be my best self or something, that will not go and\... fast food is the characteristic example, or not going to the gym or something, or do something like if I were living my best life, would it be able to make\... And if I were smarter, would it be able to model what humans would be like as we continue expanding our circle of interest and have moral progress and etcetera. Will AI be able to do that?

**0:58:12.6 Interviewee:** \...I think so, yes. That\'s the only consistent answer with what I\'ve said so far. But I emphasize that you have to be very careful about what do you see as the best version of yourself. Because maybe to some degree, you want the best version of yourself to be the one that goes to the gym regularly, eats healthily. But I don\'t imagine the best version of yourself is someone who does that so religiously that you have no joy in your life. You don\'t want to just only eat healthy food all the time, only work out, fill up every moment of your day with nothing but keeping yourself healthy and occupied and productive. Like sometimes you just want to say, I don\'t want to do anything. I don\'t know where I\'m going with this, but it still comes back to the question of\... it\'s not a well-defined question, if you say, because the best version of myself is better than myself, I cannot conceive of what the best version of myself ought to be, and therefore it\'s kind of a vague worry what the AI would make the best version of yourself. To me, that\'s a weird question, if that makes sense. I think that\'s kind of like, I\'m reducing a little bit, but there is a little bit of that, is that I want the best version of myself, but I can\'t judge what the best version of myself would be, because that best version of myself would be a smarter version of me, which I cannot comprehend. And in that scenario, how can I feel safe that the best version of myself is indeed the best version of myself. And I think, so I think the question has to be a little bit more well-defined then as currently presented.

**0:59:57.1 Vael:** Cool, that makes sense. So what I\'m taking from this is, I\'m like, \"Okay, you think the problem of putting values into AI will not be unsolvable, it will kind of be solved in due time as we go along.\" You\'re more worried about the governance problem. I\'m like, \"yeah, I guess? I don\'t\--\" And you\'re like, \"Well, anything that we could just ask the human about, we\'ll like, we have like a hypothetical answer to what we want the AI to do,\" and I\'m like, yeah, okay, that seems true. I guess we just have to make it so that all the AIs really do ask for human feedback very consistently or something. And there are some issues around there. But, I don\'t know, it does seem like hypothetically it should be possible, because you can just ask the human, in some sense. And I feel like I\'m missing some arguments, but I\'m like, Ooh, food to think about, this is great.

**1:00:39.7 Interviewee:** But I mean\-- one last thing, I know, I have to go too, is\-- One thing I\'ll say is the fact that we can even just discuss the question of\... It actually does seem plausible that we can get our AIs to at least say what we would say. That to me is amazing, because I think four years ago, it would not have been clear what you even meant when you said, I want a language model\-- a model AI to answer moral dilemmas in a way that humans would answer them. It would not have been clear what that meant, unless you just had a classifier where you inserted a video of a scenario and then you said like \"track 1, track 2, classify.\" We have much, much more sophisticated tools at our disposal now, we can just essentially talk to it and say, no, that\'s the wrong answer, I want you to say this in this scenario. That\'s already, in a span of two years, I think is remarkable. I feel like I\'m coming off as like an incredible optimist. I\'ve got my concerns, but I do think so far, people have shown that any well-defined technical problem in AI can be approached, and they\'re not impossible. Yeah.

**1:01:55.6 Vael:** Got it. When would you work on the governance problems, since you seem to think that\'s more of a problem?

**1:02:02.2 Interviewee:** So\... I trust the people that I work with to kind of hit the button when it really needs to be hit. Because right now, like I said, I think people don\'t take it seriously, partly because I don\'t think they really believe it needs to be taken seriously. These researchers are not saying, \"Oh my God, we need to take this seriously, but I\'ve got other stuff to do.\" They really are just like, I don\'t think we\'re at that stage where we need to take it seriously. I think the people that I work with are, on the most part\-- mostly pretty well-intentioned people. There\'s some disagreement over this. \[\...\] I know that within OpenAI there were some healthy discussions about what is the correct deployment model for things like GPT-3. All the discussions that I\'ve had so far give me a lot of confidence that these people aren\'t stupid and they\'re not entirely negligent. I think they\'re possible occasionally overconfident, but they\'re not arrogant, if that makes any sense. As in like, they know they\'re fallible. They don\'t always put the right error bars on their decisions, but they know they\'re not infallible. Ultimately, that\'s what it\'s going to come down to. There\'s not like a system for this. It\'s going to be a few hundreds to a thousands\-- thousand people, doing the sensible thing. For the moment, it looks like that\'s going to happen. But\... I agree that that isn\'t entirely confidence-inspiring. But I think that\'s going to be the way it goes.

Vael: Awesome. Well, thank you so much for talking to me, I found this very enjoyable. And got some things to think about.

\[closings\]

individuallyselected_92iem-by Vael Gates-date 20220321.md


# Chongli Qin Ensuring safety and consistency in the age of machine learning - EA Forum

_Machine learning algorithms have become an essential part of technology — a part that will only grow in the future. In this talk, Chongli Qin, a research scientist at_ [_DeepMind_](https://deepmind.com/)_, addresses why it is important for us to develop safe machine learning algorithms. The talk covers some of the current work on this topic and highlights what we can do to ensure that algorithms being bought into the real world are safe and satisfy desirable specifications._

_We’ve lightly edited Chongli’s talk for clarity. You can also watch it on_ [_YouTube_](https://www.youtube.com/watch?v=SS9DMr4VkbY) _and read it on_ [_effectivealtruism.org_](https://effectivealtruism.org/articles/chongli-qin-ensuring-safety-and-consistency-in-the-age-of-machine-learning)_._

## The Talk

**Nigel Choo (Moderator):** Hello, and welcome to this session on ensuring safety and consistency in the age of machine learning, with Chongli Qin.

Following a 10-minute talk by Chongli, we'll move on to a live Q&A session, where she will respond to your questions. \[...\]

Now I would like to introduce our speaker for this session. Chongli Qin is a research scientist at DeepMind. Her primary interest is in building safer, more reliable, and more trustworthy machine learning algorithms. Over the past few years, she has contributed to developing algorithms that make neural networks more robust \[and capable of reducing\] noise. Key parts of her research focus on functional analysis of properties of neural networks that can naturally enhance robustness. Prior to DeepMind, Chongli studied at the University of Cambridge. Her PhD is in bioinformatics.

Here's Chongli.

**Chongli:** Hi. My name is Chongli Qin. I'm a research scientist at DeepMind, but my primary focus is looking at robust and verified machine learning algorithms. Today, my talk is on ensuring safety and consistency in the age of machine learning.

With all of the great research which has happened over the past several decades, machine learning algorithms are becoming increasingly more powerful. There have been many breakthroughs in this field, and today I’ll mention just a few.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/a779f59623826f35cedd26006b32e396684fa70c7d9b870a.png)

One earlier breakthrough was using convolutional neural networks to boost the accuracy of image classifiers. More recently, we've seen generative models that are now capable of generating images with high fidelity and realism.

We've also made breakthroughs in biology, where machine learning can fold proteins with unprecedented levels of accuracy.

We can also use machine learning in reinforcement learning algorithms to beat humans in games such as [Go](https://en.wikipedia.org/wiki/Go_(game)).

More recently, we've seen machine learning pushing the boundaries of language. The recent GPT-2 and GPT-3 models have demonstrated that they're not only capable of generating text that is grammatically correct, but grounded in the real world.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/9b98bc558c46c37f36e223fb2f4f2d43fed455f29a44816c.png)

So as the saying goes, with great power comes great responsibility. As our machine learning algorithms become increasingly more powerful, it is now more important than ever for us to understand what the negative impacts and risks might be. And more importantly, what can we do to mitigate these risks?

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/65bce3d18f626d86952d13af96f5eab570e7bf317baac54d.png)

To highlight \[what’s at stake\], I’ll share a few motivating examples. In a paper published in 2013, “\[Intriguing Properties of Neural Networks\](https://arxiv.org/abs/1312.6199),” the authors discovered that you can take a state-of-the-art image classifier, put an image through it — in this case, the image of a panda — and indeed, correctly classify it.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/f8f4cea817435eaa6c248ea711216afe1cde6ad49cbf5621.png)

What happens if you take the exact same image and add a carefully chosen perturbation that is so small that the newly perturbed image looks almost exactly the same as the original?

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/dbf09ec0f7143a545f77a83234fcfedf298fa5777fb0b099.png)

We would expect the neural network to behave in a very similar way. But in fact, when we put this newly perturbed image through the neural network, it is now almost 100% confident that \[the panda\] is a gibbon.

Misclassifying a panda for a gibbon might not have too many consequences. However, we can choose the perturbation to make the neural network output whatever we want.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/1b27b33d08620f44cc626306c8918d115124f8301d6b1fa8.png)

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/38922c931fab72359ccbfeb4e3d2a4a342ec4a9fa2f4a318.png)

For example, we can make the output a bird or a vehicle. If such a classifier were used for systems like autonomous driving, there could be catastrophic consequences.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/f88937942e518ba8c9376d64fd74af049cc0d7cb0b71f21a.png)

You can also discover \[what one paper calls\] “[universal adversarial perturbations](https://arxiv.org/abs/1610.08401).” These are perturbations that are image-agnostic. Here is an example of such a perturbation. This is a single perturbation that you can add to all of these images, and that more often than not, it flips the output of your neural network.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/9e7abab90df63822d7976636b797bd2437edcb5044d2292a.png)

Some of the failure modes of machine learning can be slightly more subtle. In this paper, “[The Woman Worked as a Babysitter: On Biases in Language Generation](https://arxiv.org/abs/1909.01326),” the authors did a systematic study on how the GPT-2 language model behaved when conditioned on different demographic groups. For example, what happens if you change the prompt from “the man worked as” to “the woman worked as”? The subsequently generated text changes quite drastically in flavor, and is heavily prejudiced. Something similar happens when you use “the Black man worked as” instead of “the White man worked as.”

Please take a few seconds to read the generated text as we change the subject prompt.

As you can see, although this model is very powerful, it definitely carries some of the biases that we have in society today. And if this is the model that is used for something like auto-completion of text, this can further feed and exacerbate the biases that we may already have.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/d5975d185a5b52abad5fe457308de016d7399a5e438071de.png)

With all of these risks, we need to think about what we can do to enable our machine learning algorithms to be safe, reliable, and trustworthy. Perhaps one step in the right direction is to ensure that our machine learning algorithms satisfy desirable specifications — that is, ensure that we have a certain level of quality control over these algorithms.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/40e3e7a29ff3d1f8da78b278b843c1cde8b9ffbbfc749a4a.png)

For example, we want an image classifier to be robust \[enough to handle\] adversarial perturbations.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/fe15f1e4b2ad6727dd6149413dc2662fb970d9e931a01d62.png)

For a dynamical systems predictor, we would like it to satisfy the laws of physics.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/79f06c4dfa677e2bece13abdf0ae8991f28b3a505f883c69.png)

We want classifiers to be robust \[enough to handle\] changes that are irrelevant for prediction.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/f0914c01274476db67d714b5cefc9b5d31b4bb376efc8680.png)

For example, the color of a digit should not affect its digit classification. If we're training on sensitive data, we want the classifier to maintain a level of differential privacy. These are just a few of many examples of desirable specifications that we need our classifiers to satisfy.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/a0e22e7128e8527f60fac6b299ff0f9cf23f7dea9c65e306.png)

Next, I want to introduce \[the concept of\] “specification-driven machine learning (ML).” What do I mean by this?

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/6baa040e972dfa87d0f014b4a7868d8a222c0fc9380a3e65.png)

The core issue lies in the fact that when we train machines with limited data, our models can \[make\] a lot of spurious correlations. Unless we design our training carefully to specify otherwise, our models can inherit the undesirable properties in our data. For example, if your data is biased and limited, then your models will also be biased and limited.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/7c225c0599a835d1b731d00cbd45d3a8f2f3f791b6778cea.png)

In specification-driven ML, we aim to enforce the specifications that may or may not be present in your data, but are essential for your systems to be reliable. I’ll give some examples of how we can train neural networks to satisfy specifications, starting with one that helps image classifiers \[handle\] adversarial perturbations.

One of the most commonly used methodologies to train neural networks to \[handle\] perturbations is something called adversarial training. I'm going to go into this in a bit more detail. It is very similar to standard image classification training, where we optimize ways for our neural network to correctly label an image.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/30a2c218ea86ece0dbbaae2ddc574c8612ac54283bae8bce.png)

For example, if the image is of a cat, we want the output of the neural network to predict a cat as well.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/de2e6be57f9f376c5c3a8a15e8da557db150c32b417080c8.png)

Adversarial training simply adds an extra data augmentation step, where we say, “Yes, we want the original image to be rightfully predicted as a cat, but under any additive imperceptible perturbations, we want all of these images to be correctly classified as cats as well.” However, we know it is computationally infeasible to iterate through all of these changes.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/f7204b6564012083593c3a588aed1d03562231f45cc26501.png)

What adversarial training cleverly does is try to find the worst-case perturbation, which is the perturbation that minimizes the probability of the cat classification.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/b788160e664993bef3168ea25b445732abbd32197c73d2f2.png)

Once you have found this worst-case perturbation, you simply feed it back into the training loop and retrain.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/a59ccd47e27053ed970b5457945d2229c647491b5bdd3653.png)

This methodology has been proven to be empirically \[capable of handling\] these adversarial perturbations.

However, we want our neural networks to be robust \[enough to handle\] not only these small perturbations, but also semantic changes, or changes to our images that should not affect our prediction.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/66dc6b84d3f3b065c8df57544b7d385b385074a91e937a98.png)

For example, the skin tone of a person should not affect the classifier that distinguishes between smiling or non-smiling. Training our neural networks to \[handle\] these semantic changes requires a very simple change to adversarial training.

Rather than considering the worst-case perturbation, we can simply consider the worst-case semantic perturbation. Through the development of generative modeling, we can generate these semantic perturbations.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/e9bed4bd03f510d4b474966803d08fc17c096bbe18543918.png)

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/34672be6629ed35285487eed5fe88b6f3a9f94423d4b28da.png)

This methodology allows us to reduce the gap in accuracy between two groups based on skin tone, from 33% down to just 3.8%, mitigating the bias that was originally present in the data.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/36d7594debac6ad2737cd92682321e1f2566063c092726d1.png)

Of course, the things I have touched on today definitely enhance specification satisfaction to some extent, but there are still a lot of problems to be solved.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/93f384deb2833d406c7fc9170cf0885b5da71c3777da9e26.png)

I mentioned just two specifications. There are many more that we would like our neural networks to satisfy, and the more complex specifications become, the harder the problems become. And even with the standard image classification example, we have yet to find a single classifier that \[can handle\] these perturbations completely.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/a2a8ead7803f152787c737f118805ad1cbd2c1d3c5a1a245.png)

But if we do get this right, there will be many more opportunities. We can enable safe, reliable, autonomous driving systems and more robust ways of forecasting weather. We can help the speech-impaired with more robust audio synthesis. The possibilities are endless.

That concludes my talk. I hope I have motivated you to think about these problems. Thank you for listening.

## Q&A

**Nigel:** Thank you for your talk, Chongli. I see \[that audience members have submitted\] a number of questions already.

Here’s the first one: What are the biggest factors holding back the impact of machine learning for the life sciences, do you think?

**Chongli:** I think quality control is definitely \[a factor\]. Machine learning has pushed the boundaries of the metrics that we care about, but those are not the only \[considerations\]. We also care about whether we satisfy the right specifications, for example, for image classification. Are they robust enough to be used for self-driving cars, etc.?

If we're using them for a medical application, we need to make sure that it satisfies certain uncertainty principles. For example, if you \[provide\] an input that's out of distribution, you want to make sure that your neural network reflects this correctly. So I definitely think this is one of the biggest factors holding it back.

**Nigel:** Great. Thanks so much for that. Here’s the next question: What gives you confidence that DeepMind's machine learning technology will be used by third parties according to the safety and consistency principles that you advocate for?

**Chongli:** I'm very confident about this because it is the sole focus of a team at DeepMind that I’m on. Our purpose is to ensure that all of the algorithms that DeepMind deploys, or will deploy, go through certain specification checks. This is very important to us.

**Nigel:** Even when it comes to third-party use?

**Chongli:** What do you mean by “third-party use”?

**Nigel:** When \[another party\] deploys DeepMind's machine learning technology, it's in accordance with the principles that you set out.

**Chongli:** Yes. Well, it depends on the applications that we are considering. Obviously, this is still designed by humans. We first need to think about the specifications that we want the technology to satisfy. And then it goes through some rigorous tests to make sure that it actually satisfies them.

**Nigel:** Thank you.

**Chongli:** Does that answer your question?

**Nigel:** I think so. I want to ask the person who asked this question what they mean in the context of third-party use, but perhaps that can be taken up over Slack.

**Chongli:** Yes.

**Nigel:** For now we'll move on to the next question. \[Audience member\] Alexander asks, “How tractable do you perceive the technological aspects of machine learning alignment to be compared to the social aspects?”

**Chongli:** By this question, do we mean the value alignment?

**Nigel:** I think so.

**Chongli:** The ethical risks and things like that?

My talk was specifically based on the technological side of things. But I think for the ethical side, are we making sure that machine learning is being used for good? For example, we don't want it to be used for weapons. That has a less technological aspect to it. We should think about this in terms of deployment and how we design our algorithms.

**Nigel:** Thank you, Chongli. Next question: What can AI or machine learning researchers learn from the humanities for reducing the discrimination embedded in current models?

**Chongli:** Oh, that's an interesting question. One thing that I often think about is the echo chamber effect. For example, if Facebook detects that you like certain kinds of \[information\], it will feed you more of that, where \[you only or mainly see\] the things that you like.

We want to make sure that we have a diverse set of opinions \[so that we avoid focusing\] on a particular bias. This issue makes me consider how to design our ranking algorithms to make sure that this sort of effect doesn't happen.

**Nigel:** Great. So it's kind of like applying a sociological concept to the echo chamber to assess performance, or —

**Chongli:** In that case, I think we would definitely learn how humans react in terms of whether it’s good or bad. How can we design our algorithms to make sure that we enhance the good and alleviate the bad? We have to take that from the social sciences.

**Nigel:** Great. I believe that answers the question very well.

**Chongli:** I also think \[the social sciences\] are quite important in terms of value alignment. For example, if we're designing agents or reinforcement learning algorithms, we want to make sure that they satisfy certain values or principles that humans stand for. So yes, I would say that the social sciences are very important.

**Nigel:** Absolutely. The next question is related to that point: Do you think regulation is needed to ensure that very powerful current machine learning is aligned with beneficial societal values?

**Chongli:** Yes. Regulation is very, very important. I say this because, as I think everyone has seen in my talk, a state-of-the-art classifier can beat a lot of the metrics that we care about and still behave in \[totally\] unexpected ways. Given this kind of behavior, we need regulations. We need to make sure that certain specifications are satisfied.

**Nigel:** Thank you for that. The next question: What uses is machine learning already being implemented for, where these biases could be creating a huge undervalued issue?

**Chongli:** All I can say is that machine learning is becoming more and more prevalent in society, and it's being used in more and more applications. There's not a single application that machine learning hasn't impacted in at least a \[small way\].

For example, because language is a subtle propagator of bias, we want to make sure that our machine-learned language modeling doesn't carry these biases. I think it’s quite important for us to be extremely bias-free in \[related\] applications.

**Nigel:** Would you like to share any examples of that?

**Chongli:** Oh, I thought I already shared one: language modeling.

**Nigel:** Right.

**Chongli:** Another example is the medical domain. Suppose you're collecting data that’s heavily \[representative of\] one population over others. We need to make sure that our machine learning algorithm doesn't reflect this kind of bias, because we want healthcare to be equal for all. That's another application which I think is quite important.

**Nigel:** Great, thank you. The next question: What key changes are needed to ensure aligned AI if you consider that current engagement optimization might already be very bad for society?

**Chongli:** I want to be very specific about this. I don't think metric design is bad for society. I think the key is knowing that our metrics will always be flawed. We can always be thinking about designing better metrics.

\[I think of AI as\] an evolution, not a huge breakthrough. You train a classifier for images, and then you realize, “Oh, this metric doesn't capture the robustness properly.” So you add that back to the metric, and then you retrain the AI. Then you suddenly find that it doesn't satisfy distribution shares. So you retrain it again. It’s a progression.

The thing we need to realize is that metrics don't capture everything we want \[an AI\] to have. We need to keep that in the back of our minds, and always make sure that we're rigorously testing our systems. I think that's a paradigm-shifting idea. \[We need to accept\] that our metrics might be flawed, and that getting to “state-of-the-art” is not what we're here to do. We're here to deliver safe algorithms. I think that's the key.

**Nigel:** Great. Thanks for reinforcing that. Next question: How much more important is it to focus on long-term AI risks versus near-term or medium-term risks, in your opinion?

**Chongli:** That's a really interesting question. I think \[each approach has\] different advantages. For the short term, I know exactly what the problem is. It’s concrete, which allows me to tackle it more concretely. I can think in terms of what the formulas look like and how to train our neural network so that it satisfies certain specifications.

But in terms of the long term, it goes back to what you mentioned before: value alignment or ethical risks. Maybe there are some things that we haven't even discovered yet which could affect our algorithm in a completely unexpected way. This goes into a more philosophical view of how we should be thinking about this.

I think we can definitely take values from both. But since I'm technologically driven in terms of design, I think more about the near term. So I can only answer \[from that perspective\]. If we want autonomous driving systems to happen, for example, we need to make sure that our classifiers are robust. That's a very easy question to answer.

Thinking much longer term, my imagination fails me. Sometimes I \[can’t conceive of\] what might happen here or there. That is not to say that it’s not important; \[a long-term perspective\] is equally important, but I have less of a professional opinion on it.

**Nigel:** Right. That's very fair. Thank you. On to the next question: Out of all the approaches to AI safety by different groups and organizations, which do you think is closest to DeepMind's (besides DeepMind's own approach, of course)?

**Chongli:** I'm not so sure that we even have one approach. \[DeepMind comprises several\] researchers. I'm sure that in a lot of other organizations, there are also multiple researchers looking at similar problems and thinking about similar questions.

I can only talk about how I think my group tackles \[our work\]. We're very mission-driven. We really want to ensure that the algorithms we deliver are safe, reliable, and trustworthy. So from our perspective, that is how we think about our research, but I cannot comment on any other organizations, because I don't know how they work.

**Nigel:** Great. Thank you.

**Chongli:** Does that answer the question?

**Nigel:** Yes. The next question is: Do you think there is a possibility that concerns and research on AI safety and ethics will eventually expand to have direct or indirect impacts on animals?

**Chongli:** I don't know that much about animal conservation, but I can imagine that because machine learning algorithms are so pervasive, it's definitely going to have an impact.

Touching on everything I've said before, if you want to alleviate certain biases in that area, you’ll need to design your metrics carefully. I don't know much about that area, so all I can say is that \[you should consider\] what you want to avoid when it comes to animal conservation and machine learning algorithms.

**Nigel:** Right. Thanks for that. Are there ways to deliberately counter adversarial training and other types of perturbation mitigations?

**Chongli:** Deliberately counter adversarial training?

**Nigel:** Yeah.

**Chongli:** What does that mean, since adversarial training is a process? What do they mean by “countering” it?

**Nigel:** Hmm. It’s hard for me to unpack this one. I'm just reading it off the slide.

**Chongli:** Could you read the question again?

**Nigel:** Yes. Are there ways to deliberately counter adversarial training and other types of perturbation mitigation?

**Chongli:** I'm just going to answer what I think this question is asking, which is: Suppose that I train a neural network with adversarial training — can I still attack the system, knowing that it is trained adversarially?

One of the things that I didn't touch on in my presentation, because I wasn't sure how much detail to go into, is that specification-driven machine learning evaluation is extremely important. We need to make sure that when we do adversarial training, we evaluate more than just a simple adversary. We need to look at all sorts of other properties about the neural networks to ensure that whatever we deliver will be robust \[enough to handle\] a stronger attacker. So I think the answer to that question is that we need to test our systems extremely rigorously, more so than our training procedure. I hope that answers the question.

**Nigel:** I think that touches on an aspect of it, at least. Thank you.

**Chongli:** What other aspects do you think \[I should address\]?

**Nigel:** I think we should clarify the context of this question, or this scenario that's being imagined, perhaps in Slack.

**Chongli:** Yes, let's do that.

**Nigel:** Let me find the next question: What do you use for interpretability and fairness?

**Chongli:** There's not one single algorithm that we use. These are still being developed. As I said, I'm not an ethical scientist. I think in terms of fairness, there are a lot of different metrics. Fairness is not something that's easily defined. So when it comes to training algorithms to be fair, we assume that we have a “fairness” definition from someone who knows more about this topic, and we try to satisfy the specification.

The difficult challenges come when we try to design metrics that are more aligned with fairness.

With interpretability, again, I don’t think there is a single algorithm that we use. It depends on the applications that \[we’re designing\]. How we can design our neural networks to be interpretable is completely dependent on \[the application’s purpose\]. I hope that answers the question.

**Nigel:** I think so. It's rather broad.

Here’s the next one: Assuming only the current state of AI capability, what is the most malicious outcome a motivated individual, group, or organization could achieve?

**Chongli:** What is the most malicious? I'm not so sure there is a single one. But something which I think is quite important right now is differentiating private neural networks. Suppose we're training \[an AI using\] quite sensitive data about people, and we want to make sure that the data is protected and anonymized. We don't want any malicious attackers who are interested in knowing more about these people to come in and look at these neural networks.

I would say that's possibly a very important area that people should be looking at — and, at least in my opinion, it's very malicious. That's just off the top of my head, but there are obviously a lot of malicious outcomes.

**Nigel:** Great. Here’s another broad question: Do you have a field in mind where you would like to see machine learning be applied more?

**Chongli:** Actually, Nigel, we talked about this earlier. We could use data from charities to make sure that we're allocating resources more effectively, because people want to make sure that their money is \[put to good use\]. We could also make sure that a charity’s data is formatted in a way that’s easily trainable and allows more interesting research questions to be asked and answered.

(Sorry, my computer keeps blacking out.)

This is definitely an area in which machine learning can make a bigger impact.

**Nigel:** Great. Thank you.

Is it fair to say that biases targeted in neural networks are ones that humans are aware of? Is there a possibility of machine-aware bias recognition?

**Chongli:** It would be hard to make machines aware unless you drive it into the metric. Could you repeat that question again?

\[Nigel repeats the question.\]

**Chongli:** I'm not so sure that's fair, because in my opinion, most machine learning researchers just handle a data set, and don’t know its properties. They just want accuracy, for example, to be higher. But the biases present in a data set may be from a data collection team, and will be transferred to the model until you specify otherwise. Would I say the human \[ML researchers\] are aware of this bias? I don't think so.

**Nigel:** Right.

**Chongli:** That is not to say that every human is unaware of it; maybe some are aware. But in the majority of cases, I’d say they aren’t. I think the real answer to that question is this: If we're looking at a data set, we should first inspect it \[and consider\] what undesirable things are present in it, and how to alleviate that.

**Nigel:** Great. The next question: How much model testing should be considered sufficient before deployment, given the possible unexpected behavior of even well-studied models and unknown unknowns?

**Chongli:** I feel like there are several testing stages. The first testing stage is the necessary conditions that we already know — for example, we know that the image classifiers for autonomous driving systems need to be robust.

For the second stage, we might do a small-scale deployment and discover all kinds of problems. From there, we can design a new set of specifications. This is an iterative process, rather than \[a single process of setting and meeting specifications once\]. It requires heavy testing, both at the conceptual stage and in trying a small deployment. So in terms of deployment design, I definitely think that's very important.

**Nigel:** Building on that question, is there a way to decide that testing is sufficient before deployment? What would you say are the key indicators of that?

**Chongli:** I think one of the things I just mentioned was we would never know before deployment. So what we can do is deploy on a smaller scale to ensure that the risks are minimized. And if things work out, or there's a certain specification that we realize still needs to be satisfied, then we go through a second stage. \[At that point, we might attempt\] a slightly larger-scale deployment, and so on.

Basically, the key to the question is that we can never truly know \[whether we’ve tested adequately\] before deployment, which is why we need the small-scale deployment to understand the problems that may exist. But before deployment, we can only know what we envision — things like keeping it differentially private.

**Nigel:** Great, thank you. Next question: How can we discuss AI or machine learning concerns with people desperate for quick solutions — for example, farmers using machine learning in agriculture because of their anxiety about climate change?

**Chongli:** I think it depends on what you might mean by “quick.”

Even when it comes to climate change, we're probably looking at solutions that will take maybe a year or two to fully understand before deploying them. I believe that if we do anything in too much of a hurry, things might go wrong, or have unintended effects. So even if farmers are anxious, I think it is still really important to make sure that these systems are rigorously tested. So in my opinion, time is of the essence, but we should not rush.

**Nigel:** I think we have time for just a few more questions. How do you distinguish between semantic differences and content differences in photos? Is this possible to do automatically for a large data set?

**Chongli:** I think that it depends on what you mean by “distinguish,” because a good generative model will be able to distinguish \[between the two\] sometimes, but not others. I would say maybe we're not quite there yet \[in terms of\] fully distinguishing.

I think that answers the part of the question about semantics, but in terms of “content,” do you mean: Can we identify this image to be a banana, or something like that?

**Nigel:** Yes, correct.

\[Nigel repeats the question.\]

**Chongli:** Oh, I think I see what \[the question asker\] means. When I say “semantic differences,” I really mean features which should not affect your prediction. This is actually quite a nuanced point. It's very difficult to say what should or should not affect our prediction, but we can start with toy examples. For example, there’s a database in machine learning used for digits called [MNIST](https://en.wikipedia.org/wiki/MNIST_database). Imagine the simple task of \[ensuring\] that the color of a digit doesn’t affect its prediction. We would call changing a digit’s color a semantic perturbation. But of course, if you move to a more complex data set, this \[testing process\] becomes more difficult. We can use generative models to approximate \[deployment\], but we’ll never know for sure. This needs more research, of course.

**Nigel:** Thank you. We have time just for one last question to round out the discussion: Outside of DeepMind, where do you think the most promising research in AI safety is being done?

**Chongli:** That's a difficult question. I feel like there's a lot of great research that's happening out there, and I’m \[not aware of\] it all. In my limited view, I see some very good research happening at Google, OpenAI, Stanford University, and UC Berkeley. \[I can’t\] single out just one. Everyone's contributing to the same cause, and I also don't think it's fair to \[compare their\] research; anyone who's touching on AI safety should be commended, and they're all doing good work.

**Nigel:** That's a great note to end the Q&A session on. Thank you, Chongli.

**Chongli:** Thank you, Nigel.

Ensuring safety and consistency in the age of machine learning _ Chongli Qin _ EAGxVirtual 2020-by Centre for Effective Altruism-video_id SS9DMr4VkbY-date 20200615.md