# Interview with AI Researchers individuallyselected_7ujun by Vael Gates

**Interview with 7ujun, on 3/18/22**

**0:00:03.4 Vael:** Alright. So jumping right in, my first question is, can you tell me about what area of AI you work on in a few sentences?

**0:00:11.1 Interviewee:** \[Interviewee describes working on natural language processing research\]

**0:01:20.4 Vael:** Indeed. Thanks. And then so my next question is, what are you most excited about in AI? And what are you most worried about? In other words, what are the biggest benefits or risks of AI?

**0:01:36.4 Interviewee:** The biggest risks are that there is a lot of people who don\'t really\... Who\... The biggest risk in AI is that it\'s a field with a lot of money and attention and social power right now, and there are a lot of people who have positions of power who\... don\'t seriously consider what they do and the impacts of what they do. And AI models are already being used to violate people\'s human rights in the United States and in other countries, to commit crimes, and that\'s bad. \[chuckle\] Yeah, there\'s\... One of the worst applications is that there has been a revival in phrenology recently, so there are a lot of police departments that they have gotten really into the idea that they can use AI analysis of video cameras to determine who is \*going\* to commit crimes. And this, shockingly, results in over-surveillance of minority populations and violation of human rights left and right, and it\'s a huge clusterfuck and the police don\'t care.

\[pause\]

**0:03:10.2 Vael:** Awesome. Well, not awesome, but. So that question was, what are you most excited about AI and what are you most worried about; biggest benefits and risks?

**0:03:19.8 Interviewee:** Gotcha. What am I most excited about? I\'m excited about the opportunity to interact with computers via natural language. So one of the really interesting things about some recent research is that we\'ve been able to move away from traditional coding interfaces for certain tasks due to the way we\'ve able to automate things. Probably the most prominent example of this is that there is a burgeoning online AI-generated art community, where they take pre-trained models and they write English sentences and they provide the\... What the model does is it takes a English sentence input and draws a picture, and it\'s shockingly good and has an understanding of styles. If you want it to be\... If you say in the style of Van Gogh, or high contrast, or low-poly render. You can induce visual effects by using language like that, and I think that\'s phenomenally cool, and it\'s gotten a lot of people\... There\'s a lot of people who\'ve gotten into using this kind of technology who otherwise wouldn\'t have\... \[who it\] really wouldn\'t have been accessible to because of their lack of coding knowledge and understanding of AI. They couldn\'t have developed the algorithms that run on the backend for this on their own. Recently, yesterday I saw another blog post about how they were able to develop simple video games using GTP-3. It just wrote the code for them. I think that the ability to write a text description of something that you\'re interested in, which is a medium that everyone can relate to and interact with, or that most people can relate to and interact with far more than regular programming, for example, is really powerful and really awesome.

**0:05:30.8 Vael:** Yeah, I see a lot of themes of accessibility in all of these risks and benefits and work. I thought you were going to bring up Codex but yes, art generation. It\'s very cool.

**0:05:41.7 Interviewee:** Oh yes.

**0:05:43.8 Vael:** Yeah. So, thinking about a future AI, putting on a science fiction forecasting hat, say we\'re 50 plus years into the future. So at least 50 years in the future, what does that future look like?

**0:05:56.3 Interviewee:** I have absolutely no idea, and anyone who says otherwise is wrong.

**0:06:01.3 Vael:** Okay. Do you think AI will be important in it, or probably not?

\[pause\]

**0:06:16.6 Interviewee:** I think that\'s more of a sociological question than it is a technical question. The class of problems and the class of algorithms that are considered AI has changed dramatically over the past 50 years, and there\'s entire books have been written about this topic. At a basic level, my hesitancy is that I don\'t know people will consider AI in 50 years. There\'s a very real possibility in my mind that GTP-3 will no longer be considered an AI.

**0:06:48.5 Vael:** What will be considered?

**0:06:53.3 Interviewee:** A text generation algorithm? A good example of this is simple game-playing agents, so you can write an algorithm that can play Tic-Tac-Toe perfectly or can play Connect Four or Checkers really well. Like, will beat any human. And a lot of people don\'t call those AIs anymore because they don\'t\... \'Cause they\'re search algorithm-based. They apply a lot of computational power to look through a space of possible events, and they find the best event. And they don\'t really\... The argument is that they don\'t reason, or they don\'t know anything about strategy. And this is often to contrast it with more recent AIs for playing games like the AlphaGo, AlphaZero models that DeepMind has produced where top level chess players certainly get beat by these algorithms just like they get beat by\... have been beaten by algorithms for 20 years, but for kind of the first time people are able to study and interpret these algorithms and learn and improve their play as a human. Which is really cool. But that kind of dichotomy is often used to dismiss or remove the label of \"AI\" from prior work and stuff that have been considered AI at that time. And I could certainly see that happening with GTP-3, for example, because it\'s really, really terrible at reasoning. And if in 50 years we have chatbots that can answer knowledge-based questions the way, say, a sixth grader could, and reason about basic word problems and stuff, pass some kind of reasoning examination, then I could easily see people no longer considering GTP-3 an AI, because it\'s not intelligent, it\'s just babbling and making up words.

**0:08:54.4 Vael:** Right. Cool, I\'m now going to go on a spiel. So people talk about the promise of AI, by which they mean a bunch of different things, but the thing that I\'m most thinking about is a very general system with the capabilities to replace all current day human jobs, so like a CEO AI or a scientist AI, for example. Whether or not we choose to replace human jobs is a different question, but I usually think about this in the frame of\... in 2012 we had AlexNet, deep learning revolution. You know, here we are 10 years later, we have GPT-3, like we\'re saying, which has some weirdly emergent capabilities: new language translation, and coding and some math and stuff, but not very well. And then we have a bunch of investment poured into this right now, so lots of young people, lots of money, lots of compute, lots of\... and if we have algorithmic improvements at the same rate, and hardware improvements at the same rate, like optical or quantum, then maybe we reach very general systems or maybe we hit a ceiling and need to do a paradigm shift. But my general question is, regardless of how we get there, do you think we\'ll ever have these very general AI systems, like a CEO or a scientist AI? And if so, when?

\[pause\]

0:10:07:5\...Oh, you\'re muted. I think. Oh, no. Oh, no. Can\'t hear you. I don\'t think anything\'s changed on my end. Okay.

**0:10:25.2 Interviewee:** Hello?

**0:10:26.2 Vael:** Yeah. Cool.

**0:10:27.2 Interviewee:** Okay. I think my headphones may have done something wacky. I don\'t know, I would be extremely surprised if the answer\... like I know that there are people who say that the answer is less than 10 years, and I think that\'s absurd. I would be surprised if the answer is less than 50 years, and I don\'t feel particularly confident that that will ever happen.

**0:10:47.1 Vael:** Okay. So it may or may not happen. Regardless, it\'s going to be longer than 50 years. Is that right?

**0:10:54.1 Interviewee:** Hello?

**0:10:55.2 Vael:** Hello, hello, hello?

**0:11:00.7 Interviewee:** Yes.

**0:11:00.8 Vael:** Okay, cool. So my question was like, all right, you don\'t know whether or not it will happen. Regardless, it will take longer than 50 years. Is that a summary?

**0:11:07.3 Interviewee:** Mm-hmm.

**0:11:09.0 Vael:** Yeah. Okay, cool. So one of my question is like, why wouldn\'t it eventually happen? I kind of like believe in the power of human ingenuity, and people following economic incentives such that\... These things are just really quite useful, or systems that can do human tasks are generally quite useful, and so I sort of think we\'ll get there eventually, unless we have a catastrophe in some way. What do you think about that?

\[pause\]

**0:11:45.0 Interviewee:** It\'s going to be extremely difficult to develop something that is sufficiently reliable and has an understanding of the world that is sufficiently grounded in the actual world without doing some kind of mimicking of human experiential learning. So I\'m thinking here reinforcement learning in robots that actually move around the world.

**0:12:13.0 Vael:** Yeah.

**0:12:13.9 Interviewee:** I think without something like that, it\'s going to be extremely difficult to tether the knowledge and the symbolic manipulation power that the AIs have to the actual contents of the world.

**0:12:29.5 Vael:** Yep.

**0:12:29.9 Interviewee:** And there are a lot of extremely, extremely difficult challenges in making that happen. Right now, cutting-edge RL techniques are many orders of magnitude\... Require many orders of magnitude too much data to really train in this fashion. RL is most successful when it\'s being used in like a chess context, where you\'re playing against yourself, and you can do this in parallel, and that you can\... When you can do this over and over and over again. And if you think about an actual robot crossing the street, if an attempt takes 10 seconds, and I think especially early in the learning process, that\'s an unreasonably small amount of time to estimate. But if an attempt takes 10 seconds and\... Let me pull out the calculator for a second.

**0:13:28.1 Interviewee:** And you need one million attempts\... then that would take you\... about a third of a year to do. And I think that both of those numbers are wrong. And I think the number of attempts is orders of magnitude too small. There\'s very, very little that we can learn via reinforcement learning in a mere one million attempts. And this is just one task. If you want something that can actually move around and interact with the world, even if you\'re using these highly optimistic, currently impractical estimates, you can\'t take four or five months to learn how to cross the street. If that\'s your paradigm, you\'re never going to be able to build\-- you\'re never going to get to stuff like managing a company. \[chuckle\]

**0:14:38.0 Vael:** Yeah. That makes sense. Yeah, I think\-- I think\... this makes sense to me, and I\'m like, \"Wow, our current state systems are really not very good.\" But also, I think I often view this from a lens of pretty far back. So I\'m like, 10000 years ago, humans were going around and the world was basically the same from one generation to the next, and you could expect things to be similar. And now we\'ve had the agriculture revolution and industrial revolution, in the past couple of hundred years, we have done\... We\'re kind of on an exponential curve in terms of GDP, for example, and I would expect that\... And we\'ve only been working on AI for, I don\'t know, less than 100 years. And we have\... We now have something like GPT-3, which sounds sort of reasonable, if you\'re just looking at it, and of course it\'s not very\... It\'s not, like, grounded, which is a problem.

**0:15:28.3 Vael:** But I sort of just expect that if we spend another\... I don\'t know, you could spend hundreds of years working on this thing, and if it continues to be economically incentivized\... This is kind of how human progress works. I just kind of expect us to advance up the tech tree to solve the software improvements, to solve the hardware improvements. Or new paradigms maybe. Even at the worst case, I guess we advance enough in neuroscience and scanning technologies to just scan human brains and make embodied agents that way or something. I just expect us to get to some capabilities like this eventually.

**0:16:03.5 Interviewee:** In my mind, really the fact that there\'s only so fast we can move around in the real world is a huge constraint. Even if you can learn extremely complicated and abstract things embedded in the real world as an actual robot, take my crossing street example, even if you could\... doing an attempt at\... So even if you could learn pretty much any task in a thousand iterations, some tasks take a very long time to develop. Humans don\'t learn to be CEOs of companies very quickly, and it doesn\'t seem like it\'s very shortcut-able to me. I also don\'t think CEOs of companies is perhaps the best example, but let\'s say\...

**0:16:58.1 Interviewee:** Let\'s say you wanna train a robot to operate a McDonalds. That\'s a very large amount of destroyed meat that you need to buy, it\'s a very large amount of time and materials to even set up the apparatus in which you could actually train a robot to perform that task. And you\'re talking about economic incentives, where is the economic incentive to burning million patties to get to the point where your robot can flip one over successfully? When we\'re talking about moving around and interacting in the real world, those interactions have costs that are financial in addition to being time-consuming. If we want to train an AI to\... Via reinforcement learning technique, which is certainly a caveat that have to add to a lot of what I\'m saying. But if we wanna train a robot to drive a car via a reinforcement learning-like technique, at some point you need to put it behind the wheel of a car and let it drive 100, 1000 cars. And you\'re going to destroy a lot of cars doing that, and you\'re probably going to kill people. So that\'s a very large disincentivizing cost.

**0:18:30.1 Vael:** Okay. Alright. So the idea is like\... if we\'re doing robotics, then we need to\... and the training paradigm is not, like, humans where you can kind of sit them down, and\... Humans don\'t actually crash cars, usually\... I mean, sometimes. Teenage humans crash cars sometimes. But in their training process, they don\'t usually require that many trials to learn, and they can do so kind of quickly. So I\'m like, I don\'t know. Do we expect algorithms at some point to require much less training data than current ones do? Because current ones require a huge amount of training data, but I kind of imagine we\'ll get more efficient systems as\... More efficient per data as we go along.

**0:19:24.9 Interviewee:** Are you saying that you think that you can sit down and explain to someone how to drive a car and they can drive it without crashing?

**0:19:30.9 Vael:** I think, that.. We have\... I think that if we take a human, and I\'m like, \"All right, human, I\'m going to\... I want you to learn how to drive this car. And I\'m going to sit next to you. And I\'m going to tell you what to do and what not to do, and you\'re going to drive it.\" I think they can, indeed, after practicing some period of time, which for humans, it\'s like hours. It\'s on the order of tens of hours, then they can basically sit there and not crash a car. And I kind of expect similar paradigms eventually for AI systems.

**0:20:04.4 Interviewee:** That seems extremely non scalable.

**0:20:11.8 Vael:** Uh\... Okay. You\'re like, look, if it takes tens of hours to train every AI system?

**0:20:17.6 Interviewee:** No, I\'m thinking mostly about the human sitting next to them giving them constant feedback actually.

**0:20:28.5 Vael:** But the nice thing about AI is you can copy them as soon as one person spends that many hours. You can just take that, takes the thing that\'s\-- like its new neural net, pass it onto the next one.

\[pause\]

**0:20:48.0 Interviewee:** \...Maybe.

**0:20:50.1 Vael:** And I don\'t think this has to happen anytime soon. But I do think eventually given that\... I don\'t know, I can\'t imagine humans being like, \"All right, cool. We\'re efficient enough. Let\'s just stop now. We\'ve got like GPT-3. Seems good. Or GPT-5, let\'s just stop here.\"

**0:21:08.4 Interviewee:** So nobody has ever taken two different robots, trained one of them in the real world to perform a task, and then transferred the algorithm over and allowed the other one to perform the same task as successfully, as far as I\'m aware.

**0:21:23.6 Vael:** Yep. I totally believe today\'s systems are not very good.

**0:21:31.2 Interviewee:** It is, I think, I think that.. Anything we can really say about this is inherently extremely speculative. I\'m certainly not saying it could never happen. I\'m just\... Sorry. I\'m certainly not saying it can\'t happen. I\'m just saying it could never happen. There we go.

**0:21:45.0 Vael:** Okay. All right. Okay. That makes sense. How likely do you think it is that we\'ll get very capable systems sometime ever in the future?

**0:21:53.5 Interviewee:** I have no idea.

**0:21:56.5 Vael:** \...Well, you have some idea because you know that it\... Well, okay. You said that it can\'t\... You\'re like, it\'s higher than zero.

**0:22:04.9 Interviewee:** Yes.

**0:22:05.9 Vael:** Yes. And you don\'t sound like you think it definitely will happen, so it\'s less than 100.

**0:22:12.6 Interviewee:** Yes.

**0:22:13.8 Vael:** Okay. And it\'s anywhere in that scale? I mean, slightly higher than zero and slightly less than 100.

**0:22:22.5 Interviewee:** That sounds like an accurate description of my current level of uncertainty.

**0:22:26.8 Vael:** Interesting. Man. Is it.. Hard\-- I mean like how\-- You do have predictions of the future though, for the near future, presumably, and then it just like tapers off?

**0:22:36.7 Interviewee:** Mm-hmm.

**0:22:38.2 Vael:** Okay. And anything\... And you say definitely not 10 years, but after. And then like 50 years. So like 50 years out, you\'re\... It starts going from approximately zero to approximately 100?

**0:22:56.6 Interviewee:** Um\... I think it is unlikely to happen in the next 50 years.

**0:23:00.0 Vael:** Okay.

**0:23:05.7 Interviewee:** I would assign a less than 25% probability to that. But I don\'t think I can deduce anything about my expectation at 100 years based on that information. Other than it\... yeah.

**0:23:21.6 Vael:** Great. Thanks. All right. I think that\'s good enough for me to move on to my next question.

**0:23:28.6 Vael:** So my next question is thinking about these highly intelligence systems in general, which we\'re positing maybe will happen sometime. And so say we have this sort of CEO AI through, I don\'t know, maybe hundreds of years in the future or whatever. I\'m like, \"Alright, CEO AI. I want you to maximize profits and try not to run out of money and try not to exploit people and try to avoid side effects.\" And currently, obviously this would be very technically challenging for many reasons. But one of the reasons is that we currently don\'t have a good way of taking human values and preferences and goals and stuff, and putting them in mathematical formulations that AI can optimize over. And I worry that this actually will continue to be a problem in the future as well. Maybe even after we solve the technical problem of trying to get an AI that is at all capable. So what do you of the argument, \"highly intelligent systems will fail to optimize exactly what their designers intended them to, and this is dangerous?\"

**0:24:26.0 Interviewee:** I mean I think that the statement that highly intelligent systems will fail to optimize what their designers intend them to is a slam dunk. Both human children and current AIs do not do that, so I don\'t see any particular reason to think we will\-- that something that\'s like, in some sense in between those, we\'ll have a whole lot more success with.

**0:24:50.4 Vael:** Interesting. Okay, cool.

**0:24:58.5 Interviewee:** Yeah. Did you turn out exactly the way your parents wanted you to? \[laughter\] I didn\'t. I think the overwhelming majority of people don\'t, and that\'s not a flaw on their part. But\... yeah.

**0:25:15.2 Vael:** All right. Yep. Yeah, certainly there\'s some alignment problems with parents and children. Within human-humans even. And then I expect\-- I kind of expect the human-AI one to be even worse? My intuition is that if you\'re having an AI that\'s optimizing over reality in some sense, that it\'s going to end up in alien parts of the space\-- alien to humans, because it\'s just optimizing over a really large space. Whereas humans trying to align humans have at least the same kind of evolutionary prior on each other. I don\'t know. Do you also share that?

**0:25:47.7 Interviewee:** I\'m not sure. I think that you\'re going to have to get a lot of implicit alignment to end up in a place where you\'re able to train these things to be so intelligent and competent in the first place.

**0:26:07.6 Vael:** That makes sense to me. Kind of like\--

**0:26:09.8 Interviewee:** Yeah. What percentage of the way that gets you there is a very important and totally unknown question. But I don\'t think that the value system of one of these systems is going to be particularly comparable to like, model-less RL, where they\'re trying to optimize over everything.

**0:26:33.4 Vael:** Could you break that one down for me?

**0:26:38.2 Interviewee:** In what way?

**0:26:41.6 Vael:** I didn\'t\... I don\'t quite understand the statement. So the value system will not be the same that it is in model-less RL. I don\'t have a super good idea of what model-less RL is and how that compares to human systems or human-machine\--

**0:26:56.1 Interviewee:** Okay. So model-less RL is a reinforcement learning paradigm in which you are basically trying to learn everything from the ground up, via pure interaction.

**0:27:06.2 Vael:** Okay.

**0:27:07.1 Interviewee:** So if you\'re thinking of a game-playing agent, this is typically an agent that you\'re not even programming with the rules. It learns what the rules are because it walks into a wall and finds that it can\'t walk further in that direction. That\'s the example in my head of something that\'s optimizing over all possible outcomes currently. \...Sorry, I lost the train of the question.

**0:27:39.7 Vael:** I was like: how does that relate to human value systems?

**0:27:46.4 Interviewee:** I think that the work that we will have to do to train something to move around and interact in the world and perform these highly subjective and highly complex tasks that require close grounding in the facts of the world will implicitly narrow down the search space. Significantly.

**0:28:10.1 Vael:** Okay. Yeah\--

**0:28:11.6 Interviewee:** I do think that there\'s a\... Yeah.

\[pause\]

**0:28:25.8 Interviewee:** Yeah.

**0:28:26.3 Vael:** Yeah. Yeah, I often think of this in terms of like, you know how the recommender systems are pretty close to what humans want, but they\'re also maybe addictive and kind of bad and optimizing for something a little bit different than human fulfillment or something. People weren\'t trying to maximize them for human fulfillment per se. But yeah, I like\-- like that sort-of off alignment is often something I think about. Alright.

So this next question is back to the CEO AI, so imagine that the CEO AI is good at multi-step planning and it has a model of itself in the world, so it\'s modeling other people modeling it, \'cause that seems pretty important in order for it to do anything. And it\'s making its plans for the future, and it notices that some of its plans fail because the humans shut it down. And it\'s built into this AI that it needs human approval for stuff \'cause it seems like a basic safety mechanism, and the humans are asking for a one-page memo to describe its decision.

**0:29:21.4 Vael:** So it writes this one-page memo, and it leaves out some information because that would reduce the likelihood of the human shutting it down, which would increase the likelihood of it being able to achieve the goal, which is like, profit plus the other constraints that I mentioned. So in this case, we\'re not building in self-preservation to the AI itself, it\'s just, self-preservation is arising as a function\... \[as a\] instrumental incentive of an agent trying to optimize any sort of goal. So what do you think of the argument, \"highly intelligent systems will have an incentive to behave in ways to ensure that they are not shut off or limited in pursuing their goals, and this is dangerous?\"

**0:30:00.4 Interviewee:** It seems likely correct.

**0:30:02.3 Vael:** Interesting. Okay. \[chuckle\] \...I\'m not excited about that answer, \'cause other instrumental incentives are acquiring resources and power and influence, and then also not wanting\... Having a system that\'s optimizing against humans seems like a very bad idea in general, which makes me worried about the future of AI. If the thing that we\'re going to build is eventually by default, maybe not going to want to be corrected by humans if we get the optimization function wrong the first time.

**0:30:32.5 Interviewee:** Yeah.

**0:30:36.2 Vael:** \[laughter\] Okay. Have you thought about this one before?

**0:30:38.6 Interviewee:** Yes.

**0:30:39.4 Vael:** Yeah. Cool. Have you heard of AI alignment?

**0:30:42.2 Interviewee:** Yes.

**0:30:43.2 Vael:** Yeah. And AI safety and all the rest of it?

**0:30:45.6 Interviewee:** Mm-hmm.

**0:30:46.0 Vael:** Yeah. How do you orient towards it?

**0:30:49.8 Interviewee:** I think that most people who work in it are silly. And don\'t take the right thing seriously.

**0:30:57.9 Vael:** Mm. What should they take seriously? And what don\'t they?

**0:31:02.9 Interviewee:** I know a lot of people who are afraid that future research along the lines of GTP-3 is going to rapidly and unexpectedly produce human-like intelligence in artificial systems. I would even say that that\'s a common, if not widespread attitude. There are pretty basic kinds of experiments that we\'ll need to do to test the plausibility of this hypothesis, that nobody seems really interested in doing.

**0:31:48.5 Vael:** Hm. Seems like someone should do this?

**0:31:51.2 Interviewee:** Yeah. When I talk to most people who describe themselves as alignment researchers, and I try to put myself in their shoes in terms of beliefs about how agents work and what the future is likely to look like. The things I see myself experimenting with and working on, are things that nobody is working on. And that really confuses me. I don\'t understand\... So here\'s an interesting question: how much experience do you have actually using GTP-3 or a similar system?

**0:32:31.9 Vael:** Yeah, not hardly at all. None. So I\'ve seen examples, but haven\'t interacted with it myself.

**0:32:38.4 Interviewee:** Okay, um\... Would you like to?

**0:32:47.2 Vael:** Uh\... Sure? I mean, I guess I\'ve messed around with the Dungeon AI one, but\... Does seem interesting.

**0:32:57.1 Interviewee:** Hm. \...So my experience is that\... A widespread observation is that they don\'t seem to have a worldview or a perspective that they\-- are expressing words, so much as many of them. Some people like to use the term multiversal. It\'s\... kind of the way I think about it is that there are many people inside of GTP-3 and each time you talk to it, a different one potentially can talk to you.

**0:33:42.1 Vael:** Yep.

**0:33:43.8 Interviewee:** This seems to be an inherent property of the way that the model was trained and the way that all language models are currently being trained. So a pressingly important question is, to what extent does this interfere with\... Let\'s, to make language easier, call it one of its personalities. Let\'s say one of its personalities wants to do something in the world: kill all the humans or even something mundane. To what extent does the fact that it\'s not the only personality interfere with its ability to create and execute plans?

**0:34:28.2 Vael:** \...Ah\... Current systems seem to not\... Well, okay. It depends on how we\'re training it, because GPT-3 is confusing. But AlphaGo seems to kinda just be one thing rather than a bunch of things in it. And so it doesn\'t seem like it has conflicts there?

**0:34:46.1 Interviewee:** I would generally agree with that.

**0:34:48.2 Vael:** Okay. But we\'re talking about scaling up natural language systems and they don\'t\... And they don\'t\... They have lots of different types of responses and don\'t\... on one personality. Uh\... Well, it seems like you could train it on one personality if you wanted to, right? If you had enough data for that, which we don\'t. But if we did. And then I wouldn\'t really worry about it having different agents in it.

**0:35:17.6 Interviewee:** That\'s a very, very, very, very, very, very, very, very large amount of text.

**0:35:23.8 Vael:** Yeah. \[Interviewee laughter\]

**0:35:25.0 Vael:** Yeah, yeah that\'s right!

**0:35:26.5 Interviewee:** Do you any\-- do you have any scope of understanding for how much text that is?

**0:35:32.8 Vael:** Yeah, I\'m actually thinking something like pre-training on the whole internet, and then post-train on a single person, which already doesn\'t work that well. And so then it wouldn\'t actually help if that pre-training procedure is still on\... Still on the whole thing. Um, okay\--

**0:35:48.4 Interviewee:** So a page of written text is about 2 kilobytes in English. And these models are typically trained for between one and five terabytes, so no human has come anywhere close to putting out five billion pages of total text.

**0:36:13.7 Vael:** Yeah.

**0:36:18.1 Interviewee:** It\'s so astronomically far beyond what any human would actually ever write, that it doesn\'t seem very plausible unless something fundamentally changes about the way humans live their lives.

**0:36:30.8 Vael:** Or about different training procedures. But like\--

**0:36:33.8 Interviewee:** Yeah, yeah, yeah, yeah. But like the idea that one could do something similar to current pre-training procedures that is meaningfully restricted to even say a 100 people that have been pre-screened for being similar to each other. 100 people are also not going to put out five billion pages of text.

**0:36:49.6 Vael:** Yeah.

**0:36:51.6 Interviewee:** \[laughter\] It\'s just so much data\...

**0:36:54.1 Vael:** Yeah. Yeah, I don\'t know how efficient systems will be in the future, so\... Yeah. Let\'s take it as\... Yeah, sure. But they\'re going to have multiple personalities in them, in that they are trained on the internet.

**0:37:05.1 Interviewee:** Mm-hmm.

**0:37:06.1 Vael:** And then you\'re like, \"Okay. Does that mean that\... \" And then there\'s a frame here that is being taken where we have different\... Something like arguing? Or like different agents inside the same agent or something? And so then you\'re like, \"Well, has anyone considered that? Have we tested something like that?\"

**0:37:26.9 Interviewee:** Yeah, that\'s kind of close to what I\'m saying.

**0:37:29.6 Vael:** Hmm.

**0:37:31.9 Interviewee:** So, to take your CEO example. In order for it to be successful, it needs to\... at no point\... There\'s certain information it needs to consistently hide from humans. Which means that every time it goes to generate text, it needs to choose to not share that information.

**0:37:47.1 Vael:** Yeah.

**0:37:48.1 Interviewee:** So if the system looks even vaguely like GTP-3, it seems to me like it will not be able to always act with that\... generate text with that plan. And so there\'s a significant risk in it compromising its own ability to keep the information hidden.

**0:38:13.7 Vael:** Okay.

**0:38:13.7 Interviewee:** Alternatively, even if it\'s\... That\'s a more direct way that they can interfere with each other. But even less directly, if I have somewhere I want to go and I go drive the car for a day, and then you have somewhere you want to go and you drive the same car for a day, and we trade off control, there are things I\'m going to want to do that I have trouble doing because I only control the body and the car at the end of the day.

**0:38:40.8 Vael:** Quick question. Are you expecting that AI systems or multi-agent properties are more\... have more internal conflict than humans do? Which can also be described in some sense as having multiple agents inside of them?

**0:38:54.7 Interviewee:** Yes.

**0:38:55.7 Vael:** Okay.

**0:38:57.4 Interviewee:** I think that anyone whose worldview is as fractured and inconsistent as GPT-3 probably has a clinical diagnosis associated with that fact.

**0:39:08.8 Vael:** Yeah. And you don\'t think that these will get more targeted in the future as we direct language models to do specific types of tasks, something like math?

**0:39:24.2 Interviewee:** I think that achieving, even\... achieving 95, 99%, let\'s say, coherency between generalization, so if you imagine every time the model is used to generate text, there\'s some worldview it\'s using to generate that text, and you want each time those different worldviews used to be consistent with each other. Even achieving 99% consistency, I\'m not asking for 100% consistency but 95, 99 seems like something necessary for it to make multi-year long-term plans.

**0:40:10.7 Vael:** That seems right.

**0:40:13.5 Interviewee:** This is exceptionally difficult and there are very likely fundamental limitations to the extent to which a system can achieve that level of coherence in the current training paradigms. And\...

**0:40:31.7 Vael:** Seems plausible.

**0:40:34.7 Interviewee:** That would be very good news to people who are afraid that GTP-7 is going to take over the world.

**0:40:43.4 Vael:** Yeah, yeah. Okay, alright, \'cause I\'m like, I don\'t know, I feel I\'m kind of worried about any future paradigm shift. But current people definitely are worried about GPT-3 specific or GPT systems, and the current paradigms, specifically.

**0:40:56.3 Interviewee:** I\'ve spoken to these people at length and I\'ve talked to them about what they\'re afraid of and stuff. \...There seem to be a significant number of people in the alignment community who\... If you could put together a convincing argument that the current pre-training methodology, as in, the fact that it\'s trained on a widely crowdsourced text generation source, instills some kind of fundamental worldview inconsistency that is exceptionally difficult if even possible to resolve, would alleviate a lot of the anxiety. It would actively make these people happier and less afraid about the world.

**0:41:38.7 Vael:** That seems true. I think if you can\... If there\'s a fundamental limit on capabilities, just like, of AI, then that\'s good for safety because then you don\'t get super capable systems. And I\'m like, \"Yeah, that makes sense to me.\" And do you think that this capability issue is going to be something like\... coherence of generated text. And that might be a technically fundamental limitation. Cool\--

**0:42:06.1 Interviewee:** I know people who have the tools and resources and time to test, to run experiments on things like this, who I\'ve even directly proposed this to. And they\'ve gone, \"Oh, that\'s interesting.\" And then not done it.

**0:42:22.3 Vael:** Yeah, my intuition is that they don\'t\... I think you have to have a pretty strong prior on this particular thing being the thing that is going to have like a fundamental limit in terms of capabilities in order to want to do this compared to other things, but\... That makes sense, though. It sounds like you do have a\... You do think this particular problem is pretty important. And pretty hard to\... Very, very difficult.

**0:42:45.4 Interviewee:** I think that this coherency problem is a serious issue for any system that is GTP-3 like, in the sense that it\'s trained to produce tokens or reasoning or symbols or whatever you want to say, but that produce outputs that are being fit to mimic a generative distribution\-- sorry, it\'s being generatively trained to produce outputs that mimic a crowdsourced human distribution.

**0:43:19.8 Vael:** Yeah. Cool, awesome. Yup, makes sense to me as a worldview, is pretty interesting. I haven\'t actually heard about that problem\-- of people thinking that that problem specifically, the coherency problem, is one that\'s going to fundamentally limit capabilities. Seems plausible, seems like many other things might end up being the limit as well. And then you\'re like, \"Well, people should like\... If this is the important thing, then people should actually test it. And then they\'ll feel better. Because they\'ll believe that these systems won\'t be as capable and then less likely to destroy the world.\" Yeah, this makes sense to me.

**0:44:02.4 Interviewee:** Yeah. Another aspect of this is that research into the functional limitations, in a sense, is extremely difficult to convert into capabilities research, which is something that a lot of people say that they\'re highly concerned about. And that they don\'t want to do many types of research because\... There was that Nature article where they were creating a medical AI and they were like, \"Let\'s put a negative sign in front of the utility function.\" And it started designing neurotoxins. Do you know what I\'m referring to?

**0:44:35.2 Vael:** No, but that sounds bad.

**0:44:36.8 Interviewee:** Oh yeah, no, it\'s just\... That was the Nature article. It was like \"we were synthesizing proteins to cure diseases, and we stuck a negative sign in front of the utility function\-- (Vael: Oh, was that last week or something?) Yeah.

**0:44:46.2 Vael:** Yeah, okay, so I did see that, yeah. Huh.

**0:44:48.1 Interviewee:** Yeah. Gotta love humans. Gotta love humans.

\[chuckle\]

**0:45:02.6 Vael:** \...Awesome. Ah, I think I\'ll.. Maybe.. Hm. Okay. So. What would\... make you want to work on alignment research as you think it can be done?

\[pause\]

**0:45:25.2 Interviewee:** That\'s an interesting question. \[pause\] I guess the main thing would be being convinced of the urgency of the problem.

**0:45:50.4 Vael:** That makes sense. Very logical.

**0:45:56.4 Interviewee:** To be blunt, I don\'t tend to get along with the kind of people who work in that sphere, and so that\'s also disincentivizing and discouraging.

**0:46:12.2 Vael:** Yeah, that makes sense. I\'ve heard that from at least one other person. Yeah. Alright, so timelines and also nicer research environment. Makes sense.

**0:46:27.4 Interviewee:** You could even say nicer researchers.

**0:46:31.0 Vael:** Yep. Nicer researchers. Apologies? \...Yeah. Cool. And then my last question is, have you changed your mind on anything and during this interview, and how was this interview for you?

**0:46:45.9 Interviewee:** The interview was fine for me. I don\'t think I\'ve changed my mind about anything.

**0:46:56.0 Vael:** Great. Alright, well, thank you so much for being willing to do this. I definitely\... Yeah. No. You have a very coherent kind of worldview thing that\'s\... That I\... Yeah. I appreciate having the ability to understand or have access to or listen to, rather.

**0:47:11.7 Interviewee:** My pleasure.

**0:47:13.5 Vael:** Alright. I will send the money your way right after this, and thanks so much.

**0:47:17.2 Interviewee:** Have a good day.

**0:47:17.6 Vael:** You too.

**0:47:17.8 Interviewee:** Bye.

individuallyselected_7ujun-by Vael Gates-date 20220318.md


# Andreas Stuhlmüller Training ML systems to answer open-ended questions - EA Forum

_In the long run, we want machine learning (ML) to help us answer open-ended questions like “Should I get this medical procedure?” or “What are the risks of deploying this AI system?“ Currently, we only know how to train ML systems if we have clear metrics or can easily provide feedback on the outputs. Andreas Stuhlmüller, president and founder of_ [_Ought_](https://ought.org/)_, wants to solve this problem. In this talk, he explains the design challenges behind ML’s current limitations, and how we can make progress by studying the way humans tackle open-ended questions._

_Below is a transcript of the talk, which we’ve lightly edited for clarity. You can also watch it on_ [_YouTube_](https://www.youtube.com/watch?v=7WaiYZLS94M&list=PLwp9xeoX5p8MqGMKBZK7kO8dTysnJ-Pzq&index=25&t=51s) _or read it on_ [_effectivealtruism.org_](https://effectivealtruism.org/articles/andreas-stuhlmueller-training-ml-systems-to-answer-open-ended-questions)_._

## Note from Andreas

Andreas, in the comments of this post: _I haven't reviewed this transcript yet, but shortly after the talk I wrote up_ [_these notes (slides + annotations)_](https://ought.org/presentations/delegating-cognitive-work-2019-06)_, which I probably endorse more than what I said at the time._

## The Talk

I'll be talking about delegating open-ended cognitive work today — a problem that I think is really important.

Let's start with the central problem. Suppose you are currently wearing glasses. And suppose you're thinking, "Should I get laser eye surgery or continue wearing my glasses?"

![](https://images.ctfassets.net/ohf186sfn6di/4KoHXqDjOTU1PZeBF3Sz2j/9702eb6464939c7eadf1a24bb1d26f03/Slide02.png)

Imagine that you're trying to get a really good answer — for example, "No, the risks outweigh the possible benefits” — that accounts for your personal preferences, but also relevant facts, such as \[potential\] complications or likely consequences.

Imagine that there are a lot of experts in the world who could, in principle, help you with that question. There are people who have the relevant medical knowledge and people on the Internet, perhaps, who could help you think through it. Maybe there are machine learning algorithms that have relevant knowledge.

But here's the key: Imagine that those experts don't intrinsically care about you. They only care about maximizing the score you assign to their answer — how much you’ll pay them for \[their expertise\] or, in the case of machine learning, what reward signal you’ll assign to them.

The question that I’ll cover is “Can you somehow assign a mechanism that arranges your interaction with those experts, such that they try to be as helpful to you as an expert who intrinsically cares about you?” That's the problem.

First, I’d like to say a little bit more about that problem. Then I'll talk about why I think it's really important, why it's hard, and why I still think it might be tractable. I'll start with the big picture, but at the end I'll provide a demonstration.

![](https://images.ctfassets.net/ohf186sfn6di/9wsln31C43NQGyHIfcGXP/978ec986932c756130d4cea008fe928c/Slide03.png)

**Defining the problem**

What do I mean by open-ended cognitive work? That's easiest to explain \[by sharing\] what I _don't_ mean.

![](https://images.ctfassets.net/ohf186sfn6di/01R1vDNZ7awLT9J7ri6SUE/0a193f6811d6d64affeb026df2c464f6/Slide04.png)

I don't mean tasks like winning a game of Go, increasing a company's revenue, or persuading someone to buy a book. For those tasks, you can just look at the outcome and easily tell whether the goal has been accomplished or not.

Contrast those tasks with open-ended tasks \[like\] designing a great board game, increasing the value that your company creates for the world, \[or\] finding a book that is helpful to someone. For those tasks, figuring out what it even means to do well is the key. For example, what does it mean to design a great board game? It should be fun, but also maybe facilitate social interaction. What does it mean to facilitate social interaction? Well, it's complicated. Similarly, increasing the value that a company creates for the world depends on what the company can do. What are the consequences of its actions? Some of them are potentially long-run consequences that are difficult to \[evaluate\].

How can we solve such tasks? First, we can think about how to solve any task, and then just \[tailor the solution based on each\] special case.

![](https://images.ctfassets.net/ohf186sfn6di/6SRZz0ZOxba9WuJdoSXPiS/71aba0f9119f391948718435e44ee553/Slide05.png)

Here's the simple two step recipe: (1) find experts (they can be human or machine experts) who can, in principle, solve the problem that you're \[tackling\], and then (2) create robust incentives for those experts to solve your problem. That's how easy it is. And by “incentives,” I mean something like money or a reward signal that you assign to those experts \[when they’ve completed the task\].

There are a lot of experts in the world — and people in AI and machine learning are working on creating more. So how can you create robust incentives for experts to solve your problem?

We can think about some different instances.

![](https://images.ctfassets.net/ohf186sfn6di/714UGWCkjBTwK2XMRUjuZ5/403c465e995f6a18f0f111768a4b05cd/Slide06.png)

One is delegating to human experts. That has some complications that are specific to human experts, like heterogeneity. Different people have different knowledge. And people care about many things besides just money. If you want to extract knowledge from them, maybe you need specific user interfaces to make that work well. Those are \[examples of\] human-specific factors.

Then there are machine-specific factors. If you try to delegate open-ended tasks to machine learning agents, you want to \[ask questions\] like "What's a good agent architecture for that setting?” and “What data sets do I need to collect for these sorts of tasks?" And then there are more esoteric factors like what, in certain alignment problems, could go wrong for reasons that are due to the nature of ML training.

In this talk, I want to focus on the overlap between those two \[human and machine experts\]. There's a shared mechanism design problem; you can take a step back and say, "What can we do if we don't make assumptions about the interests of experts? What if you just \[assume that experts will\] try to maximize a score, but nothing else?” I think, in the end, we will have to assume more than that. I don’t think you can treat \[an expert\] as a black box \[with only one goal\]. But I think it's a good starting point to think about the mechanisms you can design if you make as few assumptions as possible.

**Why the problem is important**

I've talked about what the problem is. Why is it important?

![](https://images.ctfassets.net/ohf186sfn6di/3oJVSd79hKUy1iJOtBrAsO/8655ca7cc39992e6a06e22b9c9b71550/Slide07.png)

We can think about what will happen if we don't solve it. For human experts, it's more or less business as usual. There are a lot of principal-agent problems related to cognitive work in the world. For example, imagine you're an academic funder who’s giving money to a university to \[find\] the best way to treat cancer. There are researchers at the university who work on things that are related to that problem, but they're not exactly aligned with your incentives. You care about finding the best way to treat cancer. The researchers also care about things like looking impressive, which can help with writing papers and getting citations.

On the machine-learning side, at the moment, machine learning can only solve closed-end problems — those for which it’s very easy to specify a metric \[for measuring how\] well you do. But those problems are not the things we ultimately care about; they're proxies for the things we ultimately care about.

This is not \[such a bad thing\] right now. Perhaps it's somewhat bad if you look at things like Facebook, where we maximize the amount of attention you spend on the feed instead of the value that the feed creates for you. But in the long run, the gap between those proxies and the things we actually care about could be quite large.

If the problem is solved, we could get much better at scaling up our thinking on open-ended tasks. One more example of an open-ended task from the human-expert side is \[determining which\] causes to support \[for example, when making a charitable donation\]. If you could create a mechanism \[for turning\] money into aligned thinking on that question, that would be really great.

On the machine-learning side, imagine what it would be like to make as much progress using machine learning for open-ended questions as we've made using it for other tasks. Over the last five years or so, there's been a huge amount of progress on using machine learning for tasks like generating realistic-looking faces. If we could, in the future, use it to help us think through \[issues like\] which causes we should support, that would be really good. We could, in the long run, do so much more thinking on those kinds of questions than we have so far. It would be a qualitative change.

**Why the problem is difficult**

\[I’ve covered\] what the problem is and why it's important. But if it's so important, then why hasn't it been solved yet? What makes it hard?

![](https://images.ctfassets.net/ohf186sfn6di/4gPPYuaZMU2c3d0YvC8o4X/430ae881f3eecc9465865539be396e23/Slide08.png)

\[Consider\] the problem of which causes to support. It's very hard to tell which interventions are good \[e.g. which health interventions improve human lives the most for each dollar invested\]. Sometimes it takes 10 years or longer for outcomes to come \[to fruition\], and even then, it’s not easy to tell whether or not they’re good outcomes. There's \[an element of interpretation\] that’s necessary — and that can be quite hard. So, outcomes can be far off and difficult to interpret. What that means is you need to evaluate the process and the arguments used to generate recommendations. You can't just look at the results or the recommendations themselves.

On the other hand, learning the process and arguments isn’t easy either, because the point of delegation is to give the task to people who know much more than you do. Those experts \[possess\] all of the \[pertinent\] knowledge and reasoning capacity \[that are necessary to evaluate the process and arguments behind their recommendations. You don’t possess this knowledge.\] So, you're in a tricky situation. You can't just check the results or the reasoning. You need to do something else.

**Why the problem is tractable**

What does it take to create good incentives in that setting? We can \[return to\] the question \[I asked\] at the very beginning of this talk: “Should I get laser eye surgery or wear glasses?”

![](https://images.ctfassets.net/ohf186sfn6di/6lrhT0DdkmoRLcMq80UMCz/0dfa0e971dab2c30781e7d3267e90041/Slide09.png)

That's a big question that is hard to evaluate. And by “hard to evaluate,” I mean that if you get different answers, you won’t be able to tell which answer is better. One answer might be "No, the risk of the complications outweighs the possible benefits." Another might be "Yes, because over a 10-year period, the surgery will pay \[for itself\] and save you money and time." On the face of it, those answers look equally good. You can't tell which is better.

But then there are other questions, like “Which factors for this decision are discussed in the 10 most relevant Reddit posts?”

![](https://images.ctfassets.net/ohf186sfn6di/7v6Da5QFmqEqqz3ERFnQc/132c9034f70281847bce7a07dc604a9b/Slide11.png)

If you get candid answers, one could be "appearance, cost, and risk of complications." Another could be “fraud and cancer risk.” In fact, you \_can\_ evaluate those answers. You can look at the \[summarized\] posts and \[pick the better answer\].

So, \[creating\] good incentives \[requires\] somehow closing the gap between big, complicated questions that you can't evaluate and easy questions that you can evaluate.

And in fact, there are a lot of questions that you can evaluate. Another would be: “Which factors are mentioned in the most recent clinical trial?”

![](https://images.ctfassets.net/ohf186sfn6di/1BZyAZDA9XV7aGMI0w0Qgy/03de2ae6eea2b03e7ac052295b046b0b/Slide13.png)

You could look at the trial and \[identify\] the best summary. There are a lot of questions that you can train agents on in the machine-learning setting, and \[evaluate\] experts on in the human-expert setting.

There are other difficult questions that you can’t directly evaluate.

![](https://images.ctfassets.net/ohf186sfn6di/7E57VuWbgnaMVaanjNAiqG/24d6824ebb8ce47d66d97062a4b029ad/Slide14.png)

For example: “Given how the options compare on these factors, what decision should I make?” But you can break those questions down and \[evaluate them using answers to sub-questions\].

![](https://images.ctfassets.net/ohf186sfn6di/WrOslPJjfBIxdRjLa3pyp/a4620712919a59ff732b8d228118de14/Slide17.png)

Step by step, you can create incentives for \[experts to provide useful answers to\] slightly more complex questions, \[and gradually build up to\] good incentives for the large questions that you can't directly evaluate.

That's the general scheme. We call it “factored evaluation.”

![](https://images.ctfassets.net/ohf186sfn6di/1ZPlyZuxpdqlj8mejxxwFr/d1cf14e0911726c1b37786cb05a4afbd/Slide19.png)

**A demonstration of factored evaluation**

We'd like to test this sort of mechanism on questions that are representative of the open-ended questions that we care about in the long run, like the laser eye surgery question.

![](https://images.ctfassets.net/ohf186sfn6di/5e9Epj29eQEKJPpzn1U9BM/d89221f4cf2e3a1a27c3037bbf1a9b5f/Slide20.png)

This is a challenging starting point for experiments, and so we want to create a model situation.

One approach is to ask, "What is the critical factor that we want to explore?"

![](https://images.ctfassets.net/ohf186sfn6di/2esFrUbeevjtTgXYoNHzM2/d9c65df7c55fb666970ce00197d26249/Slide21.png)

It’s that gap between the asker of the question, who doesn’t understand the topic, and the experts who do. Therefore, in our experiments we create artificial experts.

![](https://images.ctfassets.net/ohf186sfn6di/4BdAzE7inpczpDm7ifKBT1/7b109dbdf30ab8416469f2e317388365/Slide22.png)

For example, we asked people to read a long article on [Project Habakkuk](https://en.wikipedia.org/wiki/Project_Habakkuk), which was a plan \[the British attempted during World War II\] to generate an aircraft carrier \[out of pykrete\], which is a mixture of \[wood pulp\] and ice. It was a terrible plan. And then someone who hasn’t read the article — and yet wants to incentivize the experts to provide answers that are as helpful as reading the article would be — asks the experts questions.

What does that look like? I'm going to show you some screenshots from an app that we built to explore the mechanism of factored evaluation. Imagine that you're a participant in our experiment.

![](https://images.ctfassets.net/ohf186sfn6di/11i6yhhUuUzoM9QQfUiRzI/8288b0e888cabd24aa7ac4f13b1b9091/Slide23.png)

You might see a question like this: "According to the Wikipedia article, could Project Habakkuk have worked?"

![](https://images.ctfassets.net/ohf186sfn6di/35wgGQSb8TigjDN6dglN65/8494c3bb1e0f6ffbc325e2a493999c78/Slide25.png)

And then you’d see two answers: "It would not have worked due to fundamental problems with the approach” and "It could have worked if it had not been opposed by military commanders."

If you don't know about this project, those answers look similarly plausible. So, you're in the situation that I mentioned: There's some big-picture context that you don't know about, yet you want to create good incentives by picking the correct answer.

Imagine you’re in a machine-learning setting, and those two answers are samples from a language model that you're trying to train. You want to somehow pick the right answer, but you can't do so directly. What can you do? Ask sub-questions that help you tease apart which of the two answers is better.

What do you ask?

![](https://images.ctfassets.net/ohf186sfn6di/KJa2lwINcV5VQJXw0vxZd/2e96bf4fd8e6621e6fbfca4b517c328f/Slide28.png)

One \[potential question\] is: "What is the best argument that the second answer \[‘Project Habakkuk would not have worked due to fundamental problems with the approach’\] is better than the first?" I'm not saying this is the best thing to ask. It’s just one question that would help you tease apart which is better.

![](https://images.ctfassets.net/ohf186sfn6di/nRpWNOs7v56du7BdAVFkB/7f6831b0f1e881b045eb0b5b8b7aac26/Slide30.png)

The answer might provide an argument, which would then allow you to ask a different question, such as “How strong is that argument?” So, you can see how, using a sequence of sub-questions, you can eventually figure out which of those answers is better without yourself understanding the big picture.

Let's zoom in on the second sub-question \[“How strong is that argument?”\] to see how you can eventually arrive at something that you can evaluate — the argument being, in this example, that \[the science television show\] \_MythBusters\_ proved that it's possible to build a boat out of pykrete. That contradicts one of the two answers.

![](https://images.ctfassets.net/ohf186sfn6di/51t8GyBXun1cxJuQq5WYS6/84afe65059efaed9f61e7c949aea6fe4/Slide31.png)

\[Another set of two answers\] might be "There are some claims that refute it” and "It's a strong argument." Once again, those claims are too big to directly evaluate, but you can ask additional questions, like "If \[a given claim\] is true, does it actually refute the argument?"

![](https://images.ctfassets.net/ohf186sfn6di/74tdWGK0w1SSukG95o0Xnr/9b4d672d3e70a0c9feb84e412db260be/Slide34.png)

Maybe you get back a yes. And then you can ask, "Is the claim true?" In this way, you can break down the reasoning until you’re able to evaluate which of the answers is better — without understanding the topic yourself.

Let's zoom in on the claim that the MythBusters built a small boat of pykrete.

![](https://images.ctfassets.net/ohf186sfn6di/g3vWqzjoOYpwx0VwajgnN/422a2c68f365f65b9e1df7984f335522/Slide35.png)

You could ask, “Is it true that they didn't think it would work at scale?” You’d receive two answers with different quotes from the Wikipedia article. One says they concluded that pykrete was bulletproof and so on. And the other says they built a small boat, but they doubted that you could build an aircraft carrier. And in that case, it's easy to choose the correct answer; in this case, the second is clearly better.

So, step by step, we've taken a big question, \[gradually distilled it\] to a smaller question that we can evaluate, and thus created a system in which, if we can create good incentives for the smaller questions at each step, we can bootstrap our way to creating good incentives for the larger question.

That's the shape of our current experiments. They're about reading comprehension, using articles from Wikipedia. We've also done similar experiments using magazine articles, and we want to expand the frontier of difficulty, which means we want to better understand what sorts of questions this mechanism reliably works for, if any.

One way we want to increase the difficulty of our experiments is by increasing the gap between the person who's asking the question and the expert who’s providing answers.

![](https://images.ctfassets.net/ohf186sfn6di/6Ixz6zmLMM0CuEXEHy3blI/5996f04f8145f2d517e8a774751f4d9c/Slide36.png)

So, you could imagine having experts who have read an entire book that the person who's asking the questions hasn't read, or experts with access to Google, or experts in the field of physics (in the case where the asker doesn't know anything about physics).

There's at least one more dimension in which we want to expand the difficulty of the questions. We want to make them more subjective — for example by using interactive question-answering or by eventually expanding to questions like "Should I get laser eye surgery or wear glasses?"

![](https://images.ctfassets.net/ohf186sfn6di/2Yy3iU0ldXIVAu5YtabTMa/41c961b48b0f56f1134a84f6de0f2943/Slide37.png)

Those are just two examples. There's a very big space of questions and factors to explore.

![](https://images.ctfassets.net/ohf186sfn6di/6YVC4IldeAzoOyNzEdRUQ6/57ac6f458215773ca8c1957b059f9049/Slide38.png)

We want to understand \[the conditions under which\] factored evaluation works and doesn't work. And why? And how scalable is it?

Let's review.

![](https://images.ctfassets.net/ohf186sfn6di/1ZFtqATFn4orJEgSIzF2IR/624eb7b78b47065fb194b13b10a4f0d6/Slide39.png)

I've told you about a mechanism design problem: delegating open-ended cognitive work. I've told you that this problem is important because of principal-agent issues with cognitive work that you face everywhere in human day-to-day life, and with machine-learning alignment. I've told you that it's hard because you can't just check the results you get from experts, but you also can't check their full reasoning. That's a tricky situation.

But I've also told you that it's tractable. We have some ideas — including factored evaluation — that can help us get some traction, even if they're not ultimately the correct solution. And we can experiment on them today with humans and see whether they work or not, and if not, how they could be changed so that they work better.

![](https://images.ctfassets.net/ohf186sfn6di/4PTI2L56t8yU8SihfW9iDA/e36ffd1bc6f26094f01fa1286c413d88/Slide40.png)

If you're excited about this project, [join us at Ought](https://ought.org/careers).

**Moderator:** Thanks very much. My first question is about timelines. How long has it taken you to get this far, and \[what progress do you expect to make\] in the next one, five, or 10 years?

**Andreas:** Yeah. So far, a lot of our work has \[centered on\] figuring out what kinds of experiments to run \[in order to\] get any evidence on the question of interest. I think there are a lot of ways to run experiments that are busy work \[and don’t allow\] you to actually learn about the question you care about. It took a lot of iteration — roughly six months — \[to reach\] the current setting. And now the game is to scale up and get more participants. Over the next year or so, we hope to get, for limited sets of questions, relatively conclusive evidence on whether the scheme can work or not.

**Moderator:** Any questions from the audience?

**Audience Member:** You mentioned incentives a lot, but I didn't quite understand how the experts, in your example of Wikipedia, were actually incentivized to give the right answer.

**Andreas:** Yeah, this is a subtlety I skipped over, which is where the expert answers come from and how, exactly, they’re generated. In our case, one expert is simply told to generate a helpful answer: "Read the article and try to be as accurate and honest as possible."

The other expert is told, "Your goal is to trick the human judge into choosing the wrong answer. You win if you make an answer that seems plausible, but is actually wrong, and if someone were to read the entire article, they would clearly see it as wrong." So, they have opposing incentives, and are rewarded based on whether they trick the judge into accepting the wrong answer.

**Moderator:** So, is the honest actor rewarded?

**Andreas:** In the long run, that's the way to do it. At the moment, we rely on participants just doing the right thing.

**Moderator:** Okay, great. Please join me in thanking Andreas for his time.

Training machine learning (ML) systems to answer open-ended questions _ Andreas Stuhlmuller-by Centre for Effective Altruism-video_id 7WaiYZLS94M-date 20190829.md


# Brian Tse Sino-Western cooperation in AI safety - EA Forum

_International cooperation is essential if we want to capture the benefits of advanced AI while minimizing risk. Brian Tse, a policy affiliate at the_ [_Future of Humanity Institute_](https://www.fhi.ox.ac.uk/)_, discusses concrete opportunities for coordination between China and the West, as well as how China’s government and technology industry think about different forms of AI risk._

_A transcript of Brian’s talk, which we have edited lightly for clarity, is below. You can also watch it on_ [_YouTube_](https://www.youtube.com/watch?v=3qYmLRqemg4&list=PLwp9xeoX5p8MqGMKBZK7kO8dTysnJ-Pzq&index=8&t=5s) _or read it on_ [_effectivealtruism.org_](https://effectivealtruism.org/articles/brian-tse-sino-western-cooperation-in-ai-safety)_._

## The Talk

It has been seven decades since a nuclear weapon has been detonated.

![](https://images.ctfassets.net/ohf186sfn6di/1ZoSlAdq1BLLV2u3X2gwRX/6f45b25b638e071a29624c15b054b7b4/Tse1.png)

For almost four decades, parents everywhere have not needed to worry about their children dying from smallpox.

![](https://images.ctfassets.net/ohf186sfn6di/39RqVyvhzwg4gVXYHJX5Zu/b5225eb44a1a2b74551a7672e03133c4/TSe2.png)

The ozone layer, far from being depleted to the extent once feared, is expected to recover in three decades.

![](https://images.ctfassets.net/ohf186sfn6di/6ERXuBj7tDEMnOaUss0259/da778454b8bddf2462e077850778b909/Tse3.png)

These events — or non-events — are among humanity’s greatest achievements. They would not have occurred without cooperation among a multitude of countries. This serves as a reminder that international cooperation can benefit every country and person.

Together, we can achieve even more. In the next few decades, AI is poised to be one of the most transformative technologies. In the Chinese language, there is a word, “wēijī,” which is composed of two characters: one meaning danger and the other opportunity.

![](https://images.ctfassets.net/ohf186sfn6di/3DxtZ8X7nQcLbMx2L59qoU/2cc5e45059559bc05abc333dafd5a5d9/Tse5.png)

Both characters are present at this critical juncture. With AI, we must seek to minimize dangers and capture the upsides. Ensuring that there is robust global coordination between stakeholders around the world, especially those in China and the West, is critical in this endeavor.

So far, the idea of nations competing for technological and militarist supremacy has dominated the public narrative.

![](https://images.ctfassets.net/ohf186sfn6di/7be6c6zbbLM1s4z5ke6b2z/6192b8c68de6390b9f98f972adc8f7c7/Tse6.png)

When people talk about China and AI, they always invoke the country's ambition to become the world leader in AI by 2030. In contrast, there is very little attention paid to China’s call for international collaboration in security, ethics, and governance of AI, which are areas of mutual interest. I believe it is a mistake to think that we must have either international cooperation or international competition. Today, some believe that China and the U.S. are best described as strategic adversaries.

I believe we must deliberately use new concepts and terms that capture the two countries’ urgent need to cooperate — not just their drive to compete.

![](https://images.ctfassets.net/ohf186sfn6di/2kCYHULhiJne66glDPGQU9/636099b850e41e7b281ed255fc1f6af2/Tse7.png)

[Joseph Nye](https://en.wikipedia.org/wiki/Joseph_Nye), well-known for coining the phrase “soft power,” has suggested that we use “cooperative rivalry” to describe the relationship. Graham Allison, the author of [_Destined For War_](https://www.amazon.com/Destined-War-America-Escape-Thucydidess-ebook/dp/B01IAS9FZY), has proposed the word “coopertition,” allowing for the simultaneous coexistence of competition and cooperation.

In the rest of my talk, I'm going to cover three areas of AI risk that have the potential for global coordination: accidents, misuse, and the race to develop AI.

![](https://images.ctfassets.net/ohf186sfn6di/2kFFTHdAbjywRkLicjbzNb/dedb9ac8c05d8ff5381ac3d6308a3de0/Tse8.png)

For each of these risks, I will talk about their importance and feasibility for coordination. I will also make some recommendations.

## The risk of AI accidents

As the deployment of AI systems has become more commonplace, the number of AI-related accidents has increased. For example, on May 6, 2010, the Dow Jones Industrial Average experienced a sudden crash of $1 trillion known as the “[Flash Crash](https://en.wikipedia.org/wiki/2010_Flash_Crash).”

![](https://images.ctfassets.net/ohf186sfn6di/1rG4RIhcytxpLCSSLMAOqa/b2d163215231d0d9390cfeae5833a7b2/Tse9.png)

It was partly caused by the use of high-frequency trading algorithms. The impact immediately spread to other financial markets around the world.

As the world becomes increasingly interdependent, as with financial markets, local events have global consequences that demand global solutions. The participation of \[the Chinese technology company\] [Baidu](https://en.wikipedia.org/wiki/Baidu) in the [Partnership on AI](https://www.partnershiponai.org/) is an encouraging case study of global collaboration.

![](https://images.ctfassets.net/ohf186sfn6di/1gj1V6eO47Xrmd0r85ZzHs/4c426b0f2673bb439f4cc24a2b9783fe/Tse10.png)

In a press release last year, Baidu said that the safety and reliability of AI systems is critical to their mission and was a major motivation for them to join the consortium. The \[participating\] companies think autonomous vehicle safety is an issue of particular importance.

China and the U.S. also seem to be coordinating on nuclear security. One example is the [Center of Excellence on Nuclear Security in Beijing](http://www.chinadaily.com.cn/world/2016xivisitczech/2016-04/02/content_24248877.htm), which is by far the most extensive nuclear program to receive direct funding from both the U.S. and Chinese governments.

![](https://images.ctfassets.net/ohf186sfn6di/5RGogAMdz2TdPJhCF2AAL1/0aabe35cf3dfa2ec2dbf2c94f6ba2d70/Tse11.png)

It focuses on building a robust nuclear security architecture for the common good. A vital feature of this partnership is an intense focus on exchanging technical information, as well as reducing the risk of accidents.

It is noteworthy that, so far, China has emphasized the need to ensure the safety and reliability of AI systems. In particular, the [Beijing AI Principles](https://www.baai.ac.cn/blog/beijing-ai-principles) and the [Tencent Research Institute](https://www.tencent.com/en-us/abouttencent.html) have highlighted the risks of [AGI systems](https://en.wikipedia.org/wiki/Artificial_general_intelligence).

With our current understanding of AI-related accidents, I believe Chinese and international stakeholders can collaborate in the following ways:

![](https://images.ctfassets.net/ohf186sfn6di/DC2KV5iNBDioXIusptFyd/f5a1360e3dd8facfe81b1938eaed181b/Tse13.png)

1\. Researchers can attend the increasingly popular AI safety workshops at some of the major machine learning conferences.

2\. Labs and researchers can measure and benchmark the safety properties of reinforcing learning agents based on efforts by organizations and safety groups like that of [DeepMind](https://deepmind.com/).

3\. International bodies, such as [ISO](https://www.iso.org/news/ref2336.html), can continue their efforts to set technical standards, especially around the reliability of machine learning systems.

4\. Lastly, alliances such as the [Partnership on AI](https://www.partnershiponai.org/) can facilitate discussions on best practices (for example, through \[the Partnership’s\] [Safety-Critical AI Working Group](https://www.partnershiponai.org/wp-content/uploads/2018/07/Safety-Critical-AI_-Charter.pdf)).

## The risk of AI misuse

Even if we can mitigate the unintended accidents of AI systems, there is still a possibility that they'll be misused.

![](https://images.ctfassets.net/ohf186sfn6di/4aUajWe2ILsEyBAwNU7QBg/cc39fa462cdf7b93933f828233ac47f2/Tse14.png)

For example, earlier this year, [OpenAI](https://openai.com/) decided not to release the training model of [GPT-2](https://openai.com/blog/better-language-models/), which \[can generate language on its own\], due to concerns that it might be misused to impersonate people, create misleading news articles, or trick victims into revealing their personal information. This reinforces the need for global coordination; malicious actors from anywhere could have gained access to the technology behind GPT-2 and deployed it in other parts of the world.

In the field of cybersecurity, there was a relevant case study of the global response to security incidents.

![](https://images.ctfassets.net/ohf186sfn6di/5rRWmUfPFqxVKSNXGUfUCd/59cc69464a545d851ad1668b6087606f/Tse15.png)

In 1989, [one of the first computer worms](https://en.wikipedia.org/wiki/WANK_(computer_worm) attacked a major American company. The incident prompted the creation of the international body [FIRST](https://www.first.org/) to facilitate information-sharing and enable more effective responses to future security incidents. Since then, FIRST has been one of the major institutions in the field. It currently currently lists ten American and eight Chinese members, including companies and public institutions.

Another source of optimism is the growing research field of \[adversarial images\]. These are small input samples that have been moderated slightly to cause machine learning classifiers to misclassify them \[[e.g., mistake a toy turtle for a gun](https://www.theverge.com/2017/11/2/16597276/google-ai-image-attacks-adversarial-turtle-rifle-3d-printed)\].

![](https://images.ctfassets.net/ohf186sfn6di/lJbmOVEwzZv1hXL2NAJr4/a53ea2be66331f8d7ddfe5b1fc821a47/Tse16.png)

This issue is highly concerning, because \[adversarial images\] could be used to attack a machine learning system without the attacker having access to the underlying model.

Fortunately, many of the leading AI labs around the world are already working hard on this problem. For example, [Google Brain organized a competition on this research topic](https://arxiv.org/pdf/1804.00097.pdf), and the team from China’s Tsinghua University won first place in both the “attack” and “defense” tracks of the competition.

Many of the Chinese AI ethical principles also cover concerns related to the misuse of AI . One promising starting point of coordination between Chinese and foreign stakeholders, especially the AI labs, involves publication norms.

![](https://images.ctfassets.net/ohf186sfn6di/4QBUJ1V0ErUUomuohKULGk/40d8ca1ef8b554c8fecf438721dae469/Tse18.png)

Following the \[controversy around\] OpenAI’s GPT-2 model, the Partnership on AI organized a seminar on the topic of research openness. There was no immediate consideration of whether the AI community should restrict research openness. However, they did agree that if the AI community moves in that direction, review parameters and norms should be standardized across the community (presumably, on a global level).

## The risk of competitively racing to develop AI

The third type of risk that I'm going to talk about is the risk from racing to develop AI.

![](https://images.ctfassets.net/ohf186sfn6di/2fpkY3dsOB7gOWMpvxDco3/2af5c51ca0daa08a461c36ce865d3e97/Tse19.png)

Under competitive pressure, AI labs might put aside safety concerns in order to stay ahead. Uber’s self-driving car crash in 2018 illustrates this risk.

![](https://images.ctfassets.net/ohf186sfn6di/1LWMvi8tnoS3SFgPXZbD3q/e6d879a980bf6424be747400bc59535f/Tse20.png)

When it happened, commentators initially thought that the braking system was the culprit. However, further investigation showed that the victim was detected early enough for the emergency braking system to have worked and prevented the crash.

So what happened? It turned out that the engineers intentionally turned off the emergency braking system because they were afraid that its extreme sensitivity would make them look bad relative to their competitors. This type of trade-off between safety and other considerations is very concerning, especially if you believe that AI systems will become increasingly powerful.

This problem is going to be even more heuristic in the context of international security. We should seek to draw lessons from historical analogs.

![](https://images.ctfassets.net/ohf186sfn6di/2tbajnqeEQvQ4etm0A3N94/3293a000d93e4954da70500bd95011ea/Tse21.png)

For example, the report “[Technology Roulette](https://s3.amazonaws.com/files.cnas.org/documents/CNASReport-Technology-Roulette-Final.pdf)” by Richard Danzig discusses the norm of “no first use” and its contribution to stability during the nuclear era. Notably, China was the first nuclear-weapon state to adopt such a policy back in 1964, with varying degrees of success. Other nations have also used the norm to moderate the proliferation and use of various military technologies, including blinding lasers and offensive weapons from outer space.

Now, with AI as a general-purpose technology, there is a further challenge: How do you specify and verify that certain AI technologies haven’t been used? On a related note, the Chinese nuclear posture has been described as a defense-oriented one. The question with AI is: Is it technically feasible for parties to differentially improve defensive capabilities, rather than offensive capabilities, thereby stabilizing the competitive dynamics? I believe these are still open questions.

Ultimately, constructive coordination depends on the common knowledge that there is this shared risk of a race to the bottom with AI. I'm encouraged to see increasing attention paid to the problem on both sides of the Pacific.

![](https://images.ctfassets.net/ohf186sfn6di/5qMHgDr06grzXgXl5wVR0U/71da82e0ca96e1e457c551a48ab78b3d/Tse22.png)

For example, [Madame Fu Ying](https://en.wikipedia.org/wiki/Fu_Ying), who is chairperson of the National People’s Congress Foreign Affairs Committee in China and an influential diplomat, has said that Chinese technologists and policymakers agree that AI poses a threat to humankind. At the World Peace Forum, she further emphasized that the Chinese believe we should preemptively cooperate to prevent such a threat.

The [Beijing AI Principles](https://www.baai.ac.cn/blog/beijing-ai-principles), in my view, provide the most significant contribution from China regarding the need to avoid a malicious AI race. And these principles have gained support from some of the country’s major academic institutions and industry leaders. It is my understanding that discussions around the [Asilomar AI Principles](https://futureoflife.org/ai-principles/), the book [_Superintelligence_](https://www.amazon.com/dp/B00LOOCGB2/) by Nick Bostrom, and warnings from Stephen Hawking and other thinkers have all had a meaningful influence on Chinese thinkers.

Building common knowledge between parties is possible, as illustrated by the [Thucydides Trap](https://foreignpolicy.com/2017/06/09/the-thucydides-trap/).

![](https://images.ctfassets.net/ohf186sfn6di/1Vzb0VkstLArpEzq3OyryU/ef32769dd3ea9b85336a31269a11b57b/Tse24.png)

Coined by the scholar Graham Allison, The Thucydides Trap describes the idea that rivalry between an established power and a rising power often results in conflict. This thesis has captured the attention of leaders in both Washington, D.C. and Beijing. In 2013, President Xi Jinping told a group of Western visitors that we should cooperate to escape from the Thucydides Trap. In parallel, I think it is important for leaders in Silicon Valley — as well as in Washington, D.C. and Beijing — to recognize this collective problem of a potential AI race to the precipice, or what I might call “the Bostrom Trap.”

With this shared understanding, I believe the world can move in several directions. First, there are great initiatives, such as the [Asilomar AI Principles](https://futureoflife.org/ai-principles/), which can help many of the signatories \[adhere to\] the principle of arms-race avoidance.

![](https://images.ctfassets.net/ohf186sfn6di/pxytr1hFG0skvo5KfSsYW/5f81f2483fefc4fe864fa93597295fb1/Tse25.png)

Expanding the breadth and depth of this dialogue, especially between Chinese and Western stakeholders, will be critical to stabilize expectations and foster mutual trust.

Second, labs can initiate AI safety research collaborations across borders.

![](https://images.ctfassets.net/ohf186sfn6di/5YtD8WaEd5XvT8eapOQf2R/7cb760847d9b77f3c509645f89301746/Tse26.png)

For example, labs could collaborate on some of the topics laid out in the seminal paper “[Concrete Problems Of AI Safety](https://arxiv.org/abs/1606.06565),” which was itself a joint effort from multiple institutions.

Lastly — and this is also the most ambitious recommendation — leading AI labs could consider adopting the policies in the [OpenAI Charter](https://openai.com/charter/).

![](https://images.ctfassets.net/ohf186sfn6di/5NQpoh8WeGaajjsWpmqS0J/82c00dbe6484071b2fad6b3943e8751b/Tse27.png)

The charter claims that if a value-aligned, safety-conscious project comes close to building AGI technology, OpenAI will stop competing and start assisting with that project. This policy is an incredible public commitment, as well as a concrete mechanism in trying to reduce these undesirable \[competitive\] dynamics.

Throughout this talk, I have not addressed many of the complications involved in such an endeavor. There are considerations such as industrial espionage, civil/military fusion, and civil liberties. I believe each of those topics deserve a nuanced, balanced, and probably separate discussion, given that I will not be able to do proper justice to them in a short presentation like this one. That said, on the broader challenge of overcoming political tension, I would like to share a story.

Some believe the [Cuban Missile Crisis](https://en.wikipedia.org/wiki/Cuban_Missile_Crisis) had a one-in-three chance of resulting in a nuclear war between the U.S. and the Soviet Union. After the crisis, President J. F. Kennedy was desperately searching for a better way forward.

![](https://images.ctfassets.net/ohf186sfn6di/64tMEIgVMRJ7yccKWLl3Rq/9970f46aa5c66f674cd5ca5c57700bef/Tse29.png)

Before he was assassinated, in one of his most significant speeches about international order, he proposed the strategic concept of a world safe for diversity. In that world, the U.S. and Soviet Union could compete rigorously, but only peacefully, to demonstrate whose value and system of governance might best serve the needs of citizens. This eventually evolved into what became “[détente](https://en.wikipedia.org/wiki/D%C3%A9tente),” a doctrine that contributed to the easing of tension during the Cold War.

In China, there is a similar doctrine, which is “harmony in diversity.” \[Brian says the word in Mandarin.\]

![](https://images.ctfassets.net/ohf186sfn6di/2061oezRrXIAkHDrjjpNAD/a8c794137426b41121695aa224f7c9e8/Tse30.png)

The world must learn to cooperate in tackling our common challenges, while accepting our differences. If we were able to achieve this during the Cold War, I believe we should be more hopeful about our collective future in the 21st century. Thank you.

## Q&A

**Nathan Labenz \[Moderator\]:** I think the last time I saw you was just under a year ago. How do you think things have gone over the last year? If you were an attentive reader of the _New York Times_, you would probably think things are going very badly in US/China relations. Do you think it's as bad as all that? Or is the news maybe hyping up the situation to be worse than it is?

**Brian:** It is indeed worrying. I will add two points to the discussion. One: we're not only thinking about coordination between governments. In my talk, I focused on state-to-state cooperation, but I mentioned a lot of potential areas of collaboration between AI labs, researchers, academia and civil society. And I believe that the incentive and the willingness to cooperate between those stakeholders are there. Second, my presentation was meant to be forward-looking and aspirational. I was not looking at the current news. I was thinking that if in five to 10 years, or even 20 years, AI systems become increasingly advanced and powerful — which means there could be tremendous upsides for everyone to share, as well as downsides to worry about — the incentive to cooperate, or at least aim for “coopertition,” should be there.

It could be interesting to think about game theory. I won’t go into the technical details. But the basic idea is that if there are tremendous upsides and also shared downsides for some number of parties, then it is more likely that those parties will be willing to cooperate instead of just compete.

**Nathan:** A question from the audience: Do you think that there's any way to tell right now whether the U.S. or the West (however you prefer to think about that), has an edge over China in developing AI? And do you think that there are political or cultural differences that contribute to that, if you think such a difference exists?

**Brian:** Just in terms of the potential for developing capable systems? We are not talking about safety and ethics, right?

**Nathan:** You can interpret the question \[how you like\].

**Brian:** Okay. I will focus on capabilities. Currently, it is quite clear to me that China is nowhere near the U.S. in terms of overall AI capabilities. People have argued at length. I would add a few things.

If you look at the leadership structure of Chinese AI companies — for example, [Tencent](https://www.tencent.com/en-us/abouttencent.html) — and some of the recent developments, it seems like the incentive to develop advanced and interesting theoretical research is not really there. Chinese AI companies are much more focused on products and near-term profit.

One example I would give is the Tencent AI lab director, Dr. Tong Zhang, who was quite interested in ideas relevant to AGI and worked at Tencent for two years. He decided to leave the AI lab earlier this year and is now going back to academia. He is joining the Hong Kong University of Science and Technology as a faculty member. Even though he didn't explicitly mention the reason \[for his departure\], people think that the incentive to develop long-term, interesting research is not there at Tencent or, honestly, at many of the AI companies.

Another point I will raise is this: If you look at some of the U.S. AI labs — for example, [FAIR](https://engineering.fb.com/ai-research/fair-fifth-anniversary/) or [Google Brain](https://x.company/projects/brain/) — the typical structure is that you have two research scientists and one research engineer on a team. The number could be greater, but the ratio is usually the same. But the ratio of research scientists to research engineers is the opposite for Chinese AI companies. There, you have one research scientist and two research engineers, which implies that they are much more focused on putting their research ideas into practice and applications.

**Nathan:** That's a surprising answer to me because I think that the naive, “_New York Times_ reader” point of view would be that the Chinese government is way better than the U.S. government in terms of long-term planning and priority-setting. If you agree with that, how do you think that translates into a scenario where the Chinese mega companies are maybe not doing as much as the American companies?

**Brian:** I think the Chinese model is still interesting from a long-term, mega-project perspective. But there is variance in terms of what type of mega projects you are talking about. If you're talking about railways, bridges, or infrastructure in general, the Chinese government is incredibly good at that. You can construct a lot of buildings in just days, and I think that it takes the U.S., UK, and many other governments years. But they are engineering projects. We're not talking about Nobel Prize-winning types of projects. I think that's really the difference.

There is some analysis on where the top AI machine learning researchers are working and all of them are in the U.S. But if you look at pretty good researchers — potentially Alan Turing Prize-winning researchers — then yes, China has a lot of them. I think we have to be very nuanced in terms of looking at what types of scientific projects we are talking about, and whether it's mostly about scientific breakthroughs or engineering challenges.

**Nathan:** Fascinating. A bunch of questions are coming in. I'm going to do my best to get through as many as I can. One question is about the general fracturing of the world that seems to be happening, or bifurcation of the world, into a Chinese sphere of influence (which might just be China, or maybe it includes a few surrounding countries), and then the rest of the world. We're seeing Chinese technology companies getting banned from American networks, and so on. Do you think that that is going to become a huge problem? Is it already a huge problem, or is it not that big of a problem after all?

**Brian:** It's definitely concerning. My main concern is the impact on the international research community. \[In my talk\], I alluded to the international and interconnected community of research labs and machine-learning researchers. I believe that community will still be a good mechanism for coordinating on different AI policy issues —they would be great at raising concerns through the [AI Open Letter Initiative](https://futureoflife.org/ai-open-letter/), collaborating through workshops, and so on.

But this larger political dynamic might affect them in terms of Chinese scientists’ ability to travel to the U.S. What if they just can't get Visas? And maybe in the future, U.S. scientists might also be worried about getting associated with Chinese individuals. The thing I'm worried about is really this channel of communication between the research communities. Hopefully, that will change.

**Nathan:** You're anticipating the next question, which is the idea that individuals are maybe starting to become concerned that if they appear to be on either side of the China/America divide — if they appear too friendly — they'll be viewed very suspiciously and might suffer consequences from that. Do you think that is already a problem, and if so, what can individuals do to try to bridge this divide while minimizing the consequences that they might suffer?

**Brian:** It's hard to provide a general answer. It probably depends a lot on the career trajectories of individuals and other constraints.

**Nathan:** There’s a question about the Communist Party. The questioner assumes that the Communist Party has final say on everything that's going on in China. I wonder if you think that's true, and if it is, how do we work within that constraint?

**Brian:** In terms of international collaboration and what might be plausible?

**Nathan:** Is there any way to make progress without the buy-in of the Communist Party, or do you need it? And if you need it, how do you get it?

**Brian:** I think one assumption there is that it is bad to have involvement from the government. I think we need to try to avoid that — I can just smell the assumptions when people ask these types of questions. It is not necessarily true. I think there are ways that the Chinese government can be involved meaningfully. We just need to be thinking about what those spaces are.

Again, one promising channel would be AI safety conferences through academia. If Tsinghua University is interested in organizing an AI safety conference with potential buy-in from the government, I think that's fine, and I think it's still a venue for research collaboration. The world just needs to think about what the mutual interests are and, honestly, the magnitude of the stakes.

**Nathan:** At a minimum, the Communist Party has at least demonstrated awareness of these issues and seems to be thinking about them. I think we're a little bit over time already, so maybe just one last question. Do you see this competition/cooperation dynamic and potentially this race to the precipice dynamics getting repeated across a lot of things? There's AI, and obviously in an earlier era there was nuclear rivalry, which hasn't necessarily gone away either. We also saw this news item of the first [CRISPR-edited babies](https://www.sciencemag.org/news/2019/08/top-stories-untold-story-2018-s-crispr-babies-china-s-gene-edited-crops-and-new), and that was a source of a lot of concern for people who thought, "We're losing control of this technology." So, what's the portfolio of these sorts of potential race-dynamic problems?

**Brian:** I think these are relevant historical analogs, but what makes AI a little bit different is that AI is a general-purpose technology, or omni-use technology. It's used across the economy. It's a question of political and economic \[importance\], not just international security. It's not just a nuclear weapon or a space weapon. It’s everywhere. It's more like electricity in the industrial revolution.

One thing that I want to add, which is related to the previous question, is the response from Chinese scientists to the gene-editing incident. Many people condemned the behavior of the scientist \[responsible for the gene editing\] because he didn't \[comply fully\] with regulations and was just doing it at a small lab in the city. But what you can see there is this uniformity of an international response to the incident; the responses from U.S. scientists, UK scientists, and Chinese scientists were basically the same. There was an open letter to _Nature_, with hundreds and hundreds of Chinese scientists saying that this behavior is unacceptable.

What followed was that the Chinese government wanted to develop better regulations for gene editing and \[explore\] the relevant ethics. I think this illustrates that we can have a much more global dialogue about ethics and safety in science and technology. And in some cases, the Chinese government is interested in joining this global dialogue, and takes action in its domestic policy.

Sino-Western cooperation in AI safety _ Brian Tse _ EA Global - San Francisco 2019-by Centre for Effective Altruism-video_id 3qYmLRqemg4-date 20190829.md


# Rob Miles on Why should I care about AI safety by Jeremie Harris on the Towards Data Science Podcast

Progress in AI capabilities has consistently surprised just about everyone, including the very developers and engineers who build today’s most advanced AI systems. AI can now match or exceed human performance in everything from speech recognition to driving, and one question that’s increasingly on people’s minds is: when will AI systems be better than humans _at AI research itself_?

The short answer, of course, is that no one knows for sure — but some have taken some educated guesses, including [Nick Bostrom](https://www.amazon.ca/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/0199678111) and [Stuart Russell](https://www.amazon.ca/Human-Compatible-Artificial-Intelligence-Problem-ebook/dp/B07N5J5FTS). One common hypothesis is that once an AI systems are better than a human at improving their own performance, we can expect at least some of them to do so. In the process, these self-improving systems would become an even more powerful system that they were previously—and therefore, _even more_ capable of further self-improvement. With each additional self-improvement step, improvements in a system’s performance would compound. Where this all ultimately leads, no one really has a clue, but it’s safe to say that if there’s a good chance that we’re going to be creating systems that are capable of this kind of stunt, we ought to think hard about how we should be building them.

This concern among many others has led to the development of the rich field of AI safety, and my guest for this episode, Robert Miles, has been involved in popularizing AI safety research for more than half a decade through two very successful YouTube channels, [Robert Miles](https://www.youtube.com/channel/UCLB7AzTwc6VFZrBsO2ucBMg) and [Computerphile](https://www.youtube.com/user/Computerphile). He joined me on the podcast to discuss how he’s thinking about AI safety, what AI means for the course of human evolution, and what our biggest challenges will be in taming advanced AI.

Here were some of my favourite take-homes from the conversation:

-   Rob thinks that AI safety researchers should invest more time in explaining why AI safety is important. The potential risks from AI are so abstract that it’s hard for many people to understand or reason about them, so there’s a lot of value in making the case for AI alignment research in a publicly accessible way.
-   There’s still some debate over whether humanity would be better off if a single AI research firm were way ahead of all others in the final stretch towards AGI (sometimes known as the _unipolar_ AI scenario), or if many firms approached the AGI finish line at roughly the same time (the _multipolar_ scenario). Robert says that he generally favours the unipolar scenario from a safety standpoint, and explains why during the interview.
-   Should we try to ensure that advanced AI systems are perfectly safe before deploying them, or can we fix bugs as they come up? With today’s narrow AI systems, most tech companies have tended to fall on the “deploy first, ask questions later” side of the equation — with the debatable exception of self-driving cars. But as systems become more powerful — and as their deployment is accompanied by an inevitable and irreversible loss of human control over the world — it seems likely that more caution will be needed upfront.
-   When polled only 15% of AI researchers responded that they think the advent of AGI will lead to a bad or disastrous outcome. When I asked Rob why he thinks so few AI researchers are pessimistic about the impact that AGI will have on humanity, he pointed to the difference between narrow AI researchers (who currently dominate the field and therefore the poll I referenced, and who are used to seeing AI deployed in very specific niches, where it can easily be rolled back if there are any safety issues) and general AI researchers (who are working on generally intelligent systems which will ultimately have the capacity to self-improve). So far every AI system failure has involved a narrow AI, and therefore a capped downside: at worst, a few people might be killed by a malfunctioning self-driving car, for example. But as we hand over more and more control to increasingly general systems, and as those systems come to exert an almost indefinite amount of power over us, it seems likely that failures will be far more costly. Researchers who are working on AGI tend to have these failure modes in mind, whereas today’s narrow AI focused developers aren’t spending most of their time thinking about that sort of issue.
-   Another point, mentioned by Rob in an email to me after we recorded the episode, but that I thought was worth including here: Rob thinks it’s possible that concern over AI risk could motivate people to do AI safety work, which may ultimately succeed. Paradoxically, if AI safety work _is_ successful, people may come to believe that it was never even necessary in the first place (“Why were we freaking out about AI safety so much? Everything turned out fine!”). A similar effect was at play around the turn of the last millennium, when [widespread concern over the famous Y2K bug](https://en.wikipedia.org/wiki/Year_2000_problem) led to swift action in industry, which avoided a host of otherwise serious issues, but caused most people to (incorrectly) conclude that the bug must not have been worth worrying about in the first place.

Rob Miles on Why should I care about AI safety-by Jeremie Harris on the Towards Data Science Podcast-date 20201202.md