# Interview with AI Researchers individuallyselected_92iem by Vael Gates

**Interview with 92iem, on 3/21/22**

**0:00:05.2 Vael:** Alright, here we are. So my first question is, can you tell me about what area of AI you work on in a few sentences?

**0:00:15.2 Interviewee:** Currently, I work a lot with language models, but that wasn't always the case.

\[\...\]

**0:00:57.4 Vael:** Great, thanks. And then what are you most excited about in AI? And what are you most worried about? In other words, what are the biggest benefits or risks of AI?

**0:01:08.5 Interviewee:** So obviously, I think the progress in language models in the last couple of years has been pretty astounding. And the fact that we can interact with these models in more or less in the natural way that we would like to interact with it just has opened up so much in terms of getting feedback from humans and stuff like that. So I think just the progress in language models, and then coupled with that, the more recent progress in using essentially some of the same techniques to do image modeling, so that you have the possibility to do just seamless multi-modal models. I think that's quite exciting. Some people think that\... You know, it's not like most of us can just paint a photographic scene and show it to other people. So it's not like\-- the photographic aspects of generative image models is not what excites me, it's the fact that humans manage to communicate quite a bit with diagrams and stuff like that. When we're doing science, you can draw little stick figures and pretty much convey what you need to convey, and that coupled with natural language should give us the ability to start thinking about getting AI to do math and science for us, and I think that's the thing that is most exciting to me.

So I know that a lot of people are excited by the idea that you can essentially have a Google that's a bit\... It's smarter, right? You can just talk with it and say, Hey, tell me a bit about this tree, and AI says something and you say, Oh, but what about that tree? That's fun, but I really feel like humans are not bottlenecked by the inability to ask about trees and buildings and trivia, essentially. I think where we're bottlenecked is like progress in science. I think, for example, so it's pretty clear that the political solution to climate change\-- the time for that has kind of come and gone. I mean, we can slow it down. If we, like the whole world, suddenly decided to say we're going to do something about this, maybe you slow it down, but I think just the timing is a little bit off. So a lot of that's going to be have to be a technological solution. And as amazing as technological progress has been, I think we're not fast enough when it comes to developing solutions to a lot of our problems. And I do think in 10, 20 years, AI is going to play a big role, both in the specialized domain in the sense of AlphaFold, where you really just come up with a system that does the thing you want it to do, but more impactfully, perhaps, by having an army of grad student-equivalent language models that can help you answer questions that you need answered. So that's very exciting, right.

**0:04:24.1 Vael:** Yeah. It's a cool vision.

**0:04:26.5 Interviewee:** I think the risks are\... It's almost banal, right? Like with most technologies bad actors can make arbitrarily bad use of these things. So yeah, when they start weaponizing these things\... I'm a little bit less concerned than some people are about like, Oh, but what if we have AIs that write fake news. Like all of that is to some extent present now, and I guess it's just a question of degree, to some extent. Okay, people argue that that difference in degree matters, and they're not necessarily wrong. I just, the thing that bothers me more definitely is very specific, malicious uses of AI. So there was a recent paper, this is so obvious that it's almost dumb, but someone said, Oh, yeah, we put an AI to trying to develop a drug that, let's say, reduces the amount of poison, and all you have to do is change the objective function, flip the sign and suddenly it just optimizes for the most poisonous thing you can possibly find. That coupled with technologies like CRISPR and stuff like that just creates a pretty dangerous\... puts very dangerous tools at people's disposal. So I would say that's the thing that I would worry about.

**0:05:54.9 Vael:** I have been impressed by how everyone I've talked to in the past week has mentioned that paper, and I'm like, good, things get around.

**0:06:01.8 Interviewee:** Well, Twitter. Thanks to Twitter.

**0:06:04.7 Vael:** Nice. Alright, so focusing on future AI, putting on a science fiction forecasting hat, say we're 50-plus years into the future. So at least 50 years in the future, what does that future look like?

0:06:17:9 Interviewee: For AI?

0:06:20:5 Vael: In general, where if AI's important, then mention that.

**0:06:27.4 Interviewee:** I see. So 50 years, oh my God. Fifty years is long time away. Assuming that we've managed not to have nuclear conflicts between now and then, which is just one of those things that now you have to put at least a one digit probability on these days. But, yeah, I think that we will end up having\... Well. The optimistic scenario is that we ended up solving a few key problems. One is transitioning mostly out of fossil fuel, so a combination of solar and fusion power. I think that's going to be huge, and I think that AI will have played a role in some of that development. And I think 50 years from now, I think unless we are monumentally blocked in the next couple of years, AI will be pretty omnipresent in our lives, and certainly in the scientific sectors. So one thing that I'm a little bit, just something that comes to mind, is that a lot of people are into this idea of these sort of augmented\... I don't know if people are literally willing to wear glasses, but certainly you could imagine having little ear buds that are fairly unobtrusive that go around your ears or something, and they do have a camera, so you can just ask it, whatever you need, you can ask it questions.

In 50 years, I think at that point, maybe some people will have worked out direct neural interfaces with stuff, and so maybe the more adventurous people will have a bit of augmented memory or at least the ability to sort of silently query their little augmented system. I think that might be a thing. Not everyone will have adopted it, I think it'll be a weird world. I personally\-- I've never been a huge, like the fastest adopter of technology, but that sort of stuff is next level, and I don't know what that's going to look like.

I also\... well, two things, I guess they're kind of linked. I think that people will live substantially longer. I think, unless something miraculous happens, I don't think they'll be living like 200, 300 years, but I certainly think it's possible people will be living to 150 or something like that. Not people born now; I'm not going to live to 150. Someone was telling me that people born, these days, they're going to live to see the year 2100, right. That's not quite in the 50-year time frame, but yeah, I certainly think people born today are going to be living, like their average lifespan in industrialized countries, assuming a certain level of privilege, they're going to be able to live quite a bit longer. That coupled with AI possibly automating quite a few jobs is going to change the social landscape a bit.

One thing that occurred to me recently\... so people used to say that\-- well, people say many things\-- one is that, this, unlike industrialization\... Some people always say technological progress destroy some jobs but creates more jobs on the other side. And then some say, Okay, but this one is different because you're automating intelligence and that really does put humans out of their main forte. So one of the things that people worry most about, in addition to the universal-based income stuff, is just the loss of dignity, that people always assume that even people who don't have what you would call glamorous jobs value the fact that they work and get paid for that work. But I think some of the stuff that happened during Covid makes me doubt that a little bit, in the sense that people did quit jobs that ostensibly looked good. Even in the tech sector where people, I felt like generally, they're not the worst jobs by any stretch, and they were like, No, this is meaningless, I want to go do something meaningful with my life. So I think the recent, the past couple of years have made me question the idea that it would be that big of a psychological blow for people to not work for money. That if you did establish a universal basic income, plus you'd have to solve some other, many complicated issues, but I don't think people will be that unhappy to be not having to work menial jobs. I'm not saying there's not going to be upheaval, but I think it's going to be like a combination of living longer, and not possibly having to do jobs if you don't want to do them. I think that's just going to be, I don't know. It might be a nice change. In the optimistic scenario, I guess.

**0:11:35.9 Vael:** Got it. Yeah. Well, my next question is related to that. So people talk about the promise of AI, by which they mean many things, but one of them is maybe having a very general capable system such that it will have cognitive capacities to replace all current day human jobs. So you might have a CEO AI or a scientist AI. Whether or not they choose to replace human jobs is different, but have the ability to do so. And I usually think about that and the fact that 2012 we have AlexNet, deep learning revolution, here we are, 10 years later, we've got things like GPT-3, which can do some language translation and some text generation, coding, math, etcetera, a lot of weirdly general capabilities. And then we now have nations competing and people competing and young people going into this thing, and lots of algorithmic improvements and hardware improvement, maybe we get optical, maybe we get quantum, lots of things happening. And so we might actually just end up being able to scale to very general systems or we might hit some sort of ceiling and need to do a paradigm shift. But regardless of how we do that, do you think we'll ever get very general AI systems, like a CEO or a scientist AI, and if so, when?

**0:12:39.1 Interviewee:** I don't know about CEO AIs. The scientist AIs, yes. Yeah, and that's going to come in stages. So obviously the current generation of AIs, we don't put them in human bodies and let them do experiments and stuff like that, right. It's going to be a while before we start letting them operate like particle accelerators. Fifty years\... Maybe in 50 years. My original background is \[non-AI field\], and I really could have just done my entire PhD from a desk, and that sort of work, certainly, AI can replace, I think, to a huge degree, from idea generation to solving the answer and writing a paper, yeah, that just feels so doable. Again, unless we hit a giant wall and find that our current transformers simply cannot reason, but I think that looks unlikely. I don't rule it out, but that looks unlikely to me.

**0:13:46.8 Vael:** Yeah. Okay, what about this CEO AI, like with multi-step planning, can do social inference, is modeling other people modeling it, like crazy, crazy amount of generality. When do you think we'll get that, if we will?

**0:14:00.7 Interviewee:** Yeah, that's not the part that I'm worried about. AI can certainly model human intent, but\... I guess it depends on what you want from your CEO AI. And this I think gets at a little bit one of my dissatisfactions with discussions about human\-- like, AI alignment. It's not that people don't talk about it, but it's rarely talked about. I don't know, on Twitter certainly. A lot of AI alignment stuff talks about\-- they \*don't\* talk about the fact that humans disagree wildly on what humans should do. So I'm thinking about this in connection with the CEO, because I think in the limit, AI will be able to do anything, any specific thing you ask the AI to do, it can do, but the question of whether you would want the AI to be CEO, I think that's mostly a human question. So that's why I said\-- I think that's a policy decision, not a AI capability question.

**0:15:25.3 Vael:** Got it, yeah. Do you think that people will end up wanting\... that there will be economic incentives such that we'll eventually have things like CEO AIs?

**0:15:36.0 Interviewee:** I guess in some sense, no, because I think a human would still be the CEO and then you would have your AI consultant, essentially, that you would ask all the things. You would delegate almost everything, but I think that people would still want to be at their very apex of a corporate hierarchy. It seems weird to put a robot in charge of that, just like\... why. It's a title thing, almost, like, why would you make the robot the CEO?

**0:16:03.2 Vael:** Yeah, yeah. In some vision of the future I have, I have the vision of a CEO AI\... we have a CEO AI and then we have shareholders, which are the humans, and we're like, "Alright, AI, I want you to make a company for me and earn a lot of money and try not to harm people and try not to exploit people and try to avoid side effects, and then pass all your decisions, your major decisions through us" and then we'll just sit here and you will make money. And I can imagine that might end up happening, something like that, especially if everyone else has AI is doing this or AIs are way more intelligent and can think faster and do things much faster than a human can. I don't know, this is like a different future kind of idea, but.

**0:16:46.8 Interviewee:** But that seems so weird. Because, then\-- so, assuming\... I don't know if everybody has access to the same AI in that scenario, but like it can't be the case that 100 people all say to their own individual AI, " Form a company and turn it into a \$100 billion company or a \$1 trillion company", and they all go out at optimizing. I think at that point, in that kind of world, I think there would have to be a bit more coordination in terms of what goes on, because that just creates some nasty possibilities in terms of bringing the economy down. So I don't know that that's how things would just happen. It cannot be the case that we would just say, "Robot, figure out how to make a trillion dollar company. I'll give you this one idea and just run with it," and then just like we are hands-off. That seems extremely unlikely, somehow.

**0:17:38.9 Vael:** Yeah, I'm interested in how that seems very unlikely. It seems like to me\... Well, we were talking about scientist AI, and I imagine we can eventually tell a science AI to like solve cancer, and maybe it will actually succeed at that or something. And it seems like it's different, to be like, Hey, CEO, make a ton of money for me. Is that getting at any of the underlying thing or not?

**0:18:08.4 Interviewee:** Uh, hm. Yeah, so I think even there, I think you would never tell an AI to "solve cancer". Well, yeah\... You would want to give it more specific goals, and I think\... In any scenario where we have full control over our AIs, we wouldn't want such vague instructions to be turned into plans. That's a scary world, where you can just say solve cancer and the robot runs with it. I think for the same reason, I don't think you would want a world where someone can say, "AI, make a lot of money for me," and that's the only instruction the AI has, and it's allowed to intervene in the world with those instructions. So yeah, that's why I don't see, just like from a sanity perspective, you would\-- you never want to unleash AI in that manner, in such a vague and uncontrolled manner. Does that make sense?

**0:19:03.6 Vael:** Yeah, that makes sense that you wouldn't want to be\... because it's very unsafe, it sounds like, or it could be\--

**0:19:09.7 Interviewee:** Yeah, kind of insanely unsafe, but\...

**0:19:14.8 Vael:** Nice. Yeah, do you think people might end up doing it anyway? Sometimes I feel like people do unwise things in the pursuit of, especially unilateral actors, in the pursuit of earning money, for example. Like, Oh, I've got the first scientist AI, I'm going to use it to solve the thing.

**0:19:35.3 Interviewee:** That's a good question. I think, I really do think you would want\... Yeah, I wonder about how you would actually enforce any kind of laws on AI technology. It's the most complicated thing to enforce, because nuclear weapons\-- One of the nice things about nuclear weapons is it's actually pretty hard to develop nuclear weapons in secrecy without releasing any radiation, that's one of its few good points. I think AI, it's true that you could just develop and run it. But I think at the point where any AI has to interface with the real world, whether it's in the stock market or something like that, I do think that people will start seeing the need for finding ways to regulate the speed. Even high frequency trading is starting to be, like you can't interact with it, any kind of stock market in less than one nanosecond or something like that. I think similarly, there's just going to be some guardrails put in place. If there's any kind of sanity in terms of policymaking at that time, you would want guardrails in place where you could not unleash AI with such large powers to affect a large part of the world with minimal intervention powers. Yeah. This is all assuming there's a sane policymaking environment here, but\...

**0:21:04.8 Vael:** Yeah. Do you think there will be?

**0:21:09.3 Interviewee:** I think so. I think so, I'm hopeful in that regard. I'm not saying that Congress is ever going to really understand the nuances of how AI works, anything like that, I just think there would be too many\... Even in a world where only OpenAI and DeepMind have full AGI, I don't think they'd want to create a world where one of them can unleash something at the level that you described. And I also think that when those two companies get close, they're going to wonder if other states, say, Russia or China, are going to be close, and they're going to start wanting to really hammer down, hammer out\... like there will be a sense of urgency, and hopefully they have enough influence to influence policymakers to say, "You need to take this seriously." And this is where I think almost the fact that it takes\... Okay, I said earlier that, you know what, the nice thing about nuclear weapons is that you could detect it, but I think one of the nice things about the fact that right now, it looks like you're going to require enormous compute to get anything that is remotely AGI. That's the thing that allows maybe\... That means the only huge corporations or states will be able to do it for at least some period of time, and hopefully those are the same actors that can somehow influence policymaking. If there were just one person, if they just had the ability to do that, it would be a little bit problematic, actually. So in some sense, because these institutions are big, I think they're going to be both constrained a bit more in terms of what they can do, and also they're going to be able to, if they are well-intentioned, to influence policymaking in a good direction.

**0:23:06.2 Vael:** Do you think they'll be able to do international cooperation? Because I imagine China will also have some AI companies that are also kind of close, I don't know how close they will be, but\...

**0:23:17.7 Interviewee:** They'll try. I don't know that China will listen to the US or Europe. I agree that's not going to be easy, yeah. Who knows what they're up to exactly, there.

**0:23:31.0 Vael:** Yeah, it seems like they're certainly trying, so\... Yeah, another one of my questions is like, have you thought much about policy or what kind of policies you want to have in place if we are getting ever closer to AGI?

**0:23:48.8 Interviewee:** Actually, I haven't given it that much thought, what the laws would specifically look like. What I don't think is really possible is something like the government says, You now need to hand over control over this to us. I don't think that's super feasible. Yeah, I can't say I have a good idea for what the laws would specifically look like. I think as a starting point, they'll certainly create some kind of agency to specifically monitor\... Actually, right now, there's no agency like the SEC or something like that that monitors what exactly goes on in AI. I mean, there's some scattering of regulations probably somewhere, some vague export controls and stuff like that. But yeah, they'd certainly start creating an agency for it, and their mandate would start to grow. I think it might, again, have to be something like what we do with nuclear reactors, where you have an agency that has experts inside of it, and that they are allowed to go into companies and kind of investigate what's going on inside, just as, if Iran is developing nuclear weapons and they agree to let inspectors in. I think it's going to be up to something like that. And then, yeah, similar to these nuclear treaties, perhaps there would have to be something along the lines of like\... there are certain lines you cannot cross with AI, and if someone does cross it, that institution or the country as a whole gets sanctioned. It's going to have to be at that level. Certainly, given the power of the putative AI that we're thinking about. I think the regulations are going to have to be quite dramatic if it's going to have any kind of effect.

**0:25:45.1 Vael:** Yeah. One thing I think that is a difference between the nuclear situation and the AI situation is that nuclear stuff, seems not very dual use. Well, nuclear weapons, at least, not very dual use. Versus like AI has a lot of possible public benefit and a lot of economic incentives, versus like you don't get, I don't know, you don't benefit the public by deploying nuclear weapons.

**0:26:05.7 Interviewee:** But nuclear reactors, but that's the whole\--

**0:26:07.9 Vael:** Nuclear\-- Yes, you could\--

**0:26:09.0 Interviewee:** That's the whole\... So Iran would always pretend that, Hey, we're just developing nuclear reactors for power. Just the problem is that was always very easily converted to nuclear weapons. I think that could be a similar---

**0:26:24.9 Vael:** Yeah, yeah, it is similar in that way. Somehow it still feels to me that these situations are not quite analogous, in that the regulations are going to be pretty different when you're like, "I am going to make sure that you're not doing anything bad in this area," and people are like, "ah, yes, but we need to get the new smartphone, scientist AI, etcetera." But yeah, I take your point. Another thing that I think is interesting is that current day systems are really pretty uninterpretable, so you're like, "Alright, well, we have to draw some lines, where are we going to draw the line?" What is an example of what a line could be, because if there's government inspectors coming in to DeepMind and you're like, "Alright, now inspect," I'm like, what are the inspecting?

**0:27:11.2 Interviewee:** Yeah, so when you say interpr\-- so that's another thing about\... one of my pet peeves about interpretability. People are not that interpretable. People hardly, rarely know what's going on in other people's heads, and they can tell you something which may or may not be true, sometimes they're lying to you, and sometimes they might be lying to themselves. When a doctor tells you, "This is what we're doing," unless you're another doctor, you rarely understand what they're saying. And so, yeah, this is a total tangent on like my\... the thing around, the discussion around interpretability is always such a mess. But what are they inspecting? If we're imagining inspectors, they could certainly go in and say, like, if it's a language model, you can certainly allow them to query the language model and see what kind of answers, what kind of capabilities these language models have. You could say, if it's a language model, just totally hypothetically, you could say, "Alright, develop me, write me a formula for a bioweapon," and if the language model just gives that to you, then possibly you have a problem. Stuff like that. So if a company that has that capability hasn't put in the required fail-safes like that, then they can be held liable for X amount of problem, the trouble, right.

**0:28:58.3 Vael:** Interesting. Cool, so that's cool. You've got like a model of what sort of rules should be in place, and it sounds like there should be rules in place where you can't develop bioweapons or you can't feed humans bioweapons when they ask for them.

**0:29:12.0 Interviewee:** Yeah, stuff like that. In this inspector model, I think that's what would kind of have to happen. But yeah, it's not like I'm an expert in this, but that's what I would think.

**0:29:24.4 Vael:** Yeah, something I'm worried about is that no one is an expert in this. Like policymakers\-- when I talk to the AI researchers, they're like, Oh, yes, the policymakers will take care of it, and I'm like, the policymakers are busy, and they're doing many different things, there's not many people who are focused singularly on AI. Also, they're mostly focusing on current day systems at the moment, so like surveillance and bias and transparency, and like a bunch of different things, so they're not really thinking very future at the moment. And they don't know what to do because they don't understand the technology, because the technology moves extremely fast, right, and so like AI researchers are the ones who know it. And I'm like, Alright, AI researchers, what should we tell them to do. You're like, Well, we should make a list of things that the AI shouldn't do, like basic fail-safes. And I'm like, Great, it would be super cool if that was written out somewhere and then we can start advocating for it or something, because I'm worried that the policy will just continue to lag really far behind the actual tech levels, except where like\... Policy is already several years behind, maybe like 10 years or something, and will continue to be that far behind even as we're approaching the very powerful AI.

**0:30:31.9 Interviewee:** Yeah, so a couple of things there. One is that that's why you need more of an agency model rather than laws, because creating laws is very, very slow, whereas an agency can drop some rules and maybe they start enforcing them. And so you do need a sensible agency that doesn't create bad rules, but the ability to be flexible. That said, I think\... The biggest problem with policymaking right now is that the policymakers don't understand AI at all, right. And you sort of hinted at that. And I think\... If I'd asked myself, at this moment in time, is there anything that, any rule that we need at this moment in time, I'm not sure there is. AIs are not there yet.

**0:31:25.2 Interviewee:** So at this moment in time, I think if you ask most researchers, "Hey, do we need to create specific laws to prevent X, Y, Z," I'm not sure many people would tell you, you need that. And so these laws, I think, are going to have to come in at very sensible points, and it's not clear to me that the policymakers are going to know when that time point is. I would say even in the AI field, very few people know when that's going to be. There's a lot of stuff coming out of especially big labs where the world doesn't know. There's like 100 people that know what's coming in the next year. I don't know what a good solution to that is.

**0:32:17.5 Vael:** Especially if we can get AIs that can generate lots of deadly poisons already. Yeah, I think it'll maybe be hard to tell, and then also one needs to develop a list, if there's going to be in list form or\...

**0:32:32.2 Interviewee:** The problem is, I think it's easier to regulate general AI just because it's going to require so much compute. But I think more specific AI that anyone can run on a GPU, like on a laptop, is more or less impossible to regulate. So it's not clear to me what the law would be, except if you use a bioweapon, you're in trouble. That law already exists, right.

**0:33:00.5 Vael:** Yeah, I think that one already exists, so\...

**0:33:05.4 Interviewee:** So I think in some sense, like kind of in the trade-off of what can the technology do right now, and who might try to deploy that, our laws sort of cover the problem cases at the moment. I think where I get a little bit stuck is if you try to say, "Alright, in five years, should we have laws banning certain uses of a very, very capable general model?" I do think at that point, Congress should seriously consider creating a regulatory agency. And I think AI researchers will only support this if there's some semblance of like, kind of like NASA, where there's some faith that engineers are in charge of this thing, that kind of know how these systems work, that they can think rationally about both the technological side and the policy side of things. And so that's going to take some work on the side of whatever administration is in power at that time. But yeah, it's not going to be easy. I think it's going to take a very capable administration to handle that transition gracefully.

**0:34:19.5 Vael:** Yeah, that makes sense. Yeah, I'm worried about a few different things in this future scenario. I'm like, Okay, I don't know if the agency will be developed while\-- in a sort of future thinking sort of way, I don't know that it will implement the right type of policies, I don't know that it will have the power to really enforce those policies, I don't know if it will have the power to enforce internationally. But I do like the idea that\-- but obviously one should still try, and it seems like there should probably be a lot of effort going into this, as you said, something like on a five-year scale.

**0:34:49.2 Interviewee:** Yeah, it's just that knowing AI researchers, there's just going to be such extreme pushback. If there's any sense that there's been a bureaucracy created whose job is nothing more than to just slow things down for no good reason. That's almost a default kind of way in which such an agency would get created, and so, yeah, it's just one of the situations where you have to hope that the future leaders of America are smart.

**0:35:24.7 Vael:** Yeah. Yep. A thing to bank on. Cool. So I'm concerned about long-term risks of AI. That's one of the ways in which I'm concerned, is that we won't get the policy right, especially as we're doing international competition, that there may be race dynamics, as we're not able to have really strong international governance. And I don't know if this will go well, and I'm like, I think people should work on this.

But another way I think that things might not work: So we talked a little bit about the alignment problem. And another interesting thing about the alignment problem is\... or in my mind, so we've got maybe a CEO AI, or whatever kind of AI, but this is the example I've been working with, and it's making plans and it has to report any decisions it makes to its shareholders, who are humans, and the humans are like, "I want a one-page memo." And the AI is like, "Okay, cool, one-page memo. I have a lot of information in my brain, in my neural network, while I'm trying to maximize profits with some other goals\-- with some other constraints." And it's noticing that if it gives certain information to humans, then the humans are more likely to shut it down, which means that it's less likely to succeed in its goal. And so it may write this memo and leave out some information so that it decreases likelihood of being shut down, increases the likelihood of achieving its goal. So this is not a story where we're building in self-preservation into the AI, but a story in which\-- why instrumental incentives of an agent achieving, trying to go for anything that is not perfectly aligned with human values, just like what humans tell it to do instead of what humans intended it to, then you might get an AI that is now optimizing against humans in some degree, trying to lie to them or deceive them in order to achieve whatever it has been programmed to do. Do you think this is a problem?

**0:37:08.2 Interviewee:** The scenario you described was exactly what human CEOs do.

**0:37:12.7 Vael:** Hm, yes. But more powerful systems, I think, with more influence over many things.

**0:37:20.6 Interviewee:** So this is the problem\-- so I think this actually still is a human problem. So if a human being\... like these AIs will be, depending on the mix of reward for not getting shut down and\... at the kind of detailed level these days, we often\... When we do RL with language models, we have two things going on, one is an RL objective, maximize the reward as much as you can, but the other objective is to tie it to the original language model so it doesn't diverge too much. In which case, if you are writing a memo, it would try to write a memo in the style that a human would write it, let's say. So the information content would be somewhat constrained by what a typical memo written by a human being would look like, and then on top of that, it would try to optimize what it is trying to do, maybe just trying to keep the company alive for as long as it can or something like that. So there is that sort of like, at least the way we do things now, there's a little bit of self-regulation built in there. But this is why I think, more fundamentally\... any question where if you just replace the AI with a human and ask the same question: Is this a problem or not a problem? I think that's more or less a human problem. And you have to think a bit more carefully about what we would want a human to do in that exact same situation. Do we have an answer for that? And then take into account the fact that the AI is more powerful. You don't need a super devious AI for a CEO to start lying to their shareholders a little bit, or misleading their shareholders a little bit, in order to present a more rosy picture of what the company is doing. So do we already have mechanisms that prevent that? I think we do, and that same thing would apply to the AI.

**0:39:22.1 Vael:** Yeah. I think the things that are interesting to me about the AI scenario is that we have the option of\... we are designing the AIs, so we could make them not be this way. And also having an AI that has a lot, lot more power, that is as powerful as a leader of one of the countries, and that has the ability to copy itself and could do self-improvement, so it can be smarter than it started out with. And okay, we've got something's possibly smarter than us, which is like the ability to reason and plan well, and has the incentive to acquire resources and influence and has all the human kind of incentives here, and we can't\-- and it's not as\-- I don't know, it's maybe not as interpretable as a human, but you can't throw it into jail. Like, I don't know, there's a lot of the mechanisms for control, I think, are maybe not there.

**0:40:12.7 Interviewee:** Yeah, so it's in this sort of legal context that I think you would not want the AI to be a CEO or any\... There has to be something\... For something like this, you would want the person\... There should be a person who's liable for the decision being made by the AI. You have to do some due diligence to the answers that the AI gives you. There's no other way. Yeah.

**0:40:40.2 Vael:** There's generally a thing in some of your answers where you're like, Well, you know, any reasonable person would do X, and I'm like, I don't know if we're in a world where we've got a bunch of just reasonable people putting in appropriate fail-safes which they've spent a long time constructing. And some of these fail-safes, I think, might be very technically difficult. I think the alignment problem might be quite technically difficult, such that researchers who are working on capabilities would get ahead even as the people working on safety mechanisms per se is also growing, but at less speed as all the capabilities researchers are pushing forward. Such that we might have an imbalance in how many people are working on each thing.

**0:41:16.3 Interviewee:** Yeah, so I guess maybe I'm thinking of two different things. One is just the sheer\-- kind of like the question of just putting rules in. The other question that I often have with these discussions is, Does a sensible answer exist to the question at all? So imagine, okay, so imagine we replace the CEO with\... Imagine we replace Mark Zuckerberg with a very, very smart AI. And this very smart AI is posed with this question of, okay, there is a photo of a naked child, but it's in the context of a war, it's a war photograph. Should this photo be allowed on Facebook or not? The CEO cannot\... It doesn't really matter how smart the AI is, this is just not a question the AI can answer, in the sense that it's an indelibly human question. That's why\-- I just think there are certain questions where when we posit a incredibly intelligent AI, it's got nothing to do with that. It's just a question of what a group of people who\... A group of humans who disagree on what the final answer should be. In that scenario, there's no right answer for the AI. There's nothing the AI can do in that scenario that is the correct answer.

**0:42:46.4 Vael:** Yeah. I think in my vision of what I want for AI in the future, I want AIs that do what humans intend them to do, so I want the alignment problem to be solved in some way, and I want it to all involve a huge amount of human feedback. So for every question that is confusing or the AI doesn't know what to do, if it hasn't internalized human values, then I want it to ask a bunch of humans, or maybe we have some way to aggregate human opinions or something. And then we have an AI that is reflecting human values and preferences, so if humans are confused about this particular issue, then I don't know, maybe the default if humans disagree is not to publish\[?\]. But in general, just having some sort of checking mechanism. The thing that I'm worried will happen by default is that we'll have an AI that is optimizing for something that's sort of right, but not quite right, and then it will just kind of now do like whatever things we put into it\-- whatever optimization goals we would put into it will be kind of locked in, and so that we'll eventually get an AI that is doing something kind of analogous to the recommender algorithms thing, where recommender algorithms are sort of addictive and they're optimizing something\-- clickthrough rate\-- that's kind of close to what humans value, but isn't quite. And then we might have an AI that is just like, A-ha, I am now incentivized to deceive humans to gain control, to gain influence, to do self-improvement, and we've sort of lost control of it while it's doing something that's like almost but not quite what we want.

**0:44:05.6 Interviewee:** I think one thing that comes to mind, actually\...so this kinda goes back to the interpretability question, but I think it may be a slightly different angle on it. I think it's going to have to be the case where when an AI makes a decision of that sort, it should output almost a disclaimer. So the way credit card companies would write you this long disclaimer. And it would have to tell you for each decision it makes, what the risks are, and then a human has to read that and sign off on it. Now, the question is going to be, the other problem with credit card disclaimers is that they were so long that the average person couldn't read it and make sense of what the hell was going on. So the AI would be somewhat required to come up with a comprehensible set of disclaimers that say, Okay, I asked a bunch of people, they said this, but obviously we shouldn't always listen to what the majority says. I also consulted some moral ethicist or some ethicists, and I synthesized the combination of the ethicists, previous precedents, and what the general public wants. I recommend that given the combination of these three factors, you should do this. And then a person should sign off on it, and then that person in some sense should be liable to the extent that the AI gave a reasonable summary of the decision factors. So something along those lines.

**0:45:32.8 Vael:** Yeah, that sounds brilliant. I would be so excited if AI in the future had that. I'm like, Wow, we have an AI that is incentivized to instead make things as maximally clear and comprehensible and taking into account what the human wants and listing out of things, I'm like, If we solve that, if we have the technical problem to solve that, I'm lnnnike, wow, amazing.

**0:45:52.6 Interviewee:** I think the key point here is at some point, the human has to be held liable for it, so that they have an incentive to only use AIs that satisfy this condition. Otherwise there's no reason for the.. because, like you say, you can't put the AI in jail, so. At some point you have to put the onus on humans. I think this is something that like even Tesla's going to have to think about. At some point, I mean\... I fully believe statistically, they'll reduce the number of accidents, but accidents will happen, sometimes the car will be the responsible party. At that point, you can't just throw up your hands and say no one was at fault, right? So if Tesla is willing to deploy their cars for self-driving, they are going to have to start taking liability, and that's going to force them to confront some of these same issues and say, Did the AI give a reasonable estimation of if we take this road\...? It has to be able to say, or like a surgical robot, it has to be able to say the same thing that doctors do, "Listen, I'm going to perform this operation, it's the best chance you have, but there is a 10% chance that you're going to die. If you're comfortable with this, if you're comfortable signing off on this, I will do my best," and only in that scenario is the doctor allowed to be forgiven if the operation goes wrong.

**0:47:20.6 Vael:** Yeah, so a part of my thinking I'm noticing is that\... Um\... So I think you're very interested in problems of misuse, which I'm also interested in, but I think I'm also interested in the problem of, like, I think that it will just be technically hard in order to incentivize an AI to not try to optimize on \[hard to parse\] but to like, be able to take\... So currently, we're quite bad at taking human preferences and goals and values and putting those in a mathematical formulation that AIs can optimize, and I think that problem might just be really, really hard. So we might just have an AI that won't even give us anything reasonable, and I'm like, Oh, well, that seems like step one. And then there's also a bunch of governance and a bunch of incentives that need to be put in place in terms of holding people accountable, since humans will hopefully be the end user of these things.

**0:48:07.8 Interviewee:** I'm actually far less worried about the technical side of this. I just finished reading this book about von Neumann, that's a little cute biography of him, and there's a part where he says, supposedly, that people who think mathematics is complicated only say that because they don't know how complicated life is. And I'm totally messing with the phrasing, but something like that. I actually think any technical problems in this area will be solved relatively easily compared to the problem of figuring out what human values we want to insert into these.

**0:48:46.4 Vael:** Okay, so you think it's the taking human values and putting into AI, that technical problem will just get solved?

**0:48:54.8 Interviewee:** If you know what values you want to put in, yeah.

**0:48:56.5 Vael:** Okay, cool. Alright.

**0:49:00.1 Interviewee:** I actually think that problem is the easy problem. I'm not saying it's easy in an absolute sense, I just think that's the easier problem.

**0:49:05.1 Vael:** Got it. That feels like the alignment problem to me. So you think the alignment problem is just going to be pretty easy. This seems like a valid thing to think, so\...

**0:49:12.9 Interviewee:** I want to emphasize, I don't think it'll be easy in absolute terms, I just think it'll be the easier of the two problems.

**0:49:17.0 Vael:** Okay, compared to governance and incentives. Yeah, that makes sense.

**0:49:21.2 Interviewee:** That is\... I just have this faith that any technical problems humans can solve, down the line\-- like, eventually. It's the non-technical problems that get people all tangled up, because when there's no right answer, it really messes up scientists.

**0:49:43.1 Vael:** Yeah, yeah. Yeah, the problem of trying to take human values and what we care about in all the different ways and put them into a mathematical formulation feels difficult to me, and I guess it is a technical problem. I guess I do sort of think of it as a technical problem, but yeah, that makes sense that you're just like, Look, we'll get that done eventually. And then we have governance, and I'm like, Oh, yes, governance is totally a mess. Yeah, that makes sense. And I think no one knows how, it's an unknown, unsolved problem\--

**0:50:13.5 Interviewee:** Let me put it this way, let me put it this way. If by human values, you mean if like\...

**0:50:18.5 Vael:** What humans intend, having an AI always doing what you say.

**0:50:22.8 Interviewee:** Yeah, so imagine for any conceivable scenario an AI that would have to deal with. We could ask you, what would you do in this case? What I'm saying is that if for each of these questions, you are able to give a concrete answer to the answered question, such questions can be inserted into our AIs. Like if you are able to come up with clear answers for the questions for which you yourself would have a clear answer for, I think that set of moral constraints, let's say, I think that can be more or less inserted into AIs without huge problems.

**0:51:08.3 Vael:** Even as the problems get\-- even as I don't eventually have concrete answers, because the problems are like, Should we do X thing on creating nuclear reactor X and etcetera, and I lose control of the\... not lose control, but I can't actually visualize the whole space or something, because I'm too\...

**0:51:25.2 Interviewee:** But at that point, what does alignment mean? Alignment usually means that what the AI does is the same as what a human would do. If there's no answer about what the human would do, what is the AI supposed to do?

**0:51:37.2 Vael:** I think it's supposed to just keep in mind the things that I care about, so try to avoid side effects in all forms, try to avoid hurting people, except for when that makes sense or something\-- oh, eugh. Anyway, like, doing a whole bunch of\... and, like reporting truthfully, which is also something I want the AI to do. And things like this.

**0:51:56.6 Interviewee:** I guess it's one of those, so it's a question of maybe a generalization, but I find it slightly hard to believe that an AI that would answer in the exact same way on all of the answered questions that you do have an answer for, when you go outside of that regime, suddenly the AI diverges strongly from what you would have answered had you been smarter. I just think that's a weird kind of discontinuity, right. So there's this huge set up\-- let's suppose on all the questions for which you have an answer, the AI agrees with you. And then you take that a little bit outside of your realm of comprehension, and at that point, suddenly the AI decides, Oof, I'm freed from the constraints, I can answer whatever. I think that's a little bit implausible to me, assuming people did the job correctly.

**0:52:52.9 Vael:** Yeah, I think assuming people do the job correctly is pretty important here. You could have an AI that is deceiving you and giving you the correct answers, as long as you could check, but yeah, assuming that's not true, assuming the AI is honestly reporting to you what exactly, like everything that you said at a lower level, I mean, everything that you can confirm, then that seems great. And you're like, Well, if you then extend that to regimes where humans can't understand it, then things are probably still okay. I think I maybe believe that. I think that maybe things are probably still okay.

**0:53:23.7 Interviewee:** The validation set on whatever project you're working on would\... The AI wouldn't know necessarily whether this was a question on which you just didn't have an opinion. It's the same for me when I think about\... A lot of people are concerned that, in terms of basically over-fitting, could you\... So one of the reasons I think we'll definitely get an AI that can answer physics questions pretty comprehensively is that I don't believe you can ever create a physics AI that can fake its way through all of the train set, sorry, on the validation set, and then suddenly do poorly on something that is outside of it. There's so much of physics that if you are able to fake your way through all the way to graduate school\... I guess I'm thinking in terms of like, if you happen to make it all the way through graduate school, pass all the exams you needed to pass, turned out the papers you turn out. At that point to be like, Oh, actually, I just, I faked my way through all of physics, I don't really understand physics. You don't not understand physics, like in spite of yourself, because you\...

**0:54:35.0 Vael:** That seems true. I think one of the things I'm referencing is like a mesa-optimizer, inner optimizer, which is like an AI has some sort of goal. I don't know, maybe it's like trying to go out, go to red things, but the door is always\... Okay, sorry, try again. So it says, it says it has some sort of goal, kind of like humans do, so\... \[sorry, I'm now\] restarting \[this sentence\]. Evolution has some sort of goal, evolution is optimizing on something like inclusive genetic fitness. And it's like this optimizer that is pushing things. And then there's humans, which are the things that are like, it's being optimized or whatever. And humans should ideally have the same goal of inclusive genetic fitness, but we don't. We've got something that's sort of close, but not really. Like we have contraceptives, so we aren't really maximizing babies, we have different goals and things we're going for, we have like of achievement and all the values that we think are important and stuff. And so this is an example of something\... that, humans are the ones who are trying to make the AI optimize for a thing, and the AIs are like, Sure, I guess I'll sort of go for the goal that you want me to, and I have a model of what you want me to do, but I actually internally in my heart have a different goal, and so I'll make my way through all of the test sets because I have a model of what you want me to do, but as soon as you release me, then I will do something different. So that's like a weird analogy that doesn't quite work, but\...

**0:56:00.1 Interviewee:** Yeah, especially for two reasons. One is that I think there is a real likelihood that we are going to become multi-planetary, which is pretty good as far as evolution is concerned, because not only does that mean we've dominated the planet, but we've started having other planets and spreading our genes everywhere. So in some sense, we've done exactly what evolution wants us to do, which is like reproduce our genes far and wide. I have a feeling that no matter how sophisticated and smart and industrialized in AI we get as a species, people aren't going to stop wanting to have babies. So like somehow, yes, we don't, like, kill everyone whenever we want to and just steal food and all that like animals would do, but kind of like\...

**0:56:49.9 Vael:** Yeah, I don't know that evolution wants that, but.

**0:56:51.8 Interviewee:** We're still pretty aligned to the basic evolutionary goal. That's what you're starting from, saying that's the original goal, and we've deviated from the original goal. I think actually we're well within evolution's parameters as far as what are we supposed to do as good evolutionary creatures.

**0:57:07.5 Vael:** Cool. Yeah, that makes sense to me. And then as a last point or something, so ideally, we want AI to do what humans would do if they were smarter, like you said, if we had more time to reflect and if we maybe had more self-control or something. I don't know that that would kind of come out of nowhere or something? I guess this is now in the ideal case, it sounds like we've mostly succeeded in aligning the AI with humans' goals. (Interviewee: "So what's the question?") Um, ideally, I think I want AI that will be my best self or something, that will not go and\... fast food is the characteristic example, or not going to the gym or something, or do something like if I were living my best life, would it be able to make\... And if I were smarter, would it be able to model what humans would be like as we continue expanding our circle of interest and have moral progress and etcetera. Will AI be able to do that?

**0:58:12.6 Interviewee:** \...I think so, yes. That's the only consistent answer with what I've said so far. But I emphasize that you have to be very careful about what do you see as the best version of yourself. Because maybe to some degree, you want the best version of yourself to be the one that goes to the gym regularly, eats healthily. But I don't imagine the best version of yourself is someone who does that so religiously that you have no joy in your life. You don't want to just only eat healthy food all the time, only work out, fill up every moment of your day with nothing but keeping yourself healthy and occupied and productive. Like sometimes you just want to say, I don't want to do anything. I don't know where I'm going with this, but it still comes back to the question of\... it's not a well-defined question, if you say, because the best version of myself is better than myself, I cannot conceive of what the best version of myself ought to be, and therefore it's kind of a vague worry what the AI would make the best version of yourself. To me, that's a weird question, if that makes sense. I think that's kind of like, I'm reducing a little bit, but there is a little bit of that, is that I want the best version of myself, but I can't judge what the best version of myself would be, because that best version of myself would be a smarter version of me, which I cannot comprehend. And in that scenario, how can I feel safe that the best version of myself is indeed the best version of myself. And I think, so I think the question has to be a little bit more well-defined then as currently presented.

**0:59:57.1 Vael:** Cool, that makes sense. So what I'm taking from this is, I'm like, "Okay, you think the problem of putting values into AI will not be unsolvable, it will kind of be solved in due time as we go along." You're more worried about the governance problem. I'm like, "yeah, I guess? I don't\--" And you're like, "Well, anything that we could just ask the human about, we'll like, we have like a hypothetical answer to what we want the AI to do," and I'm like, yeah, okay, that seems true. I guess we just have to make it so that all the AIs really do ask for human feedback very consistently or something. And there are some issues around there. But, I don't know, it does seem like hypothetically it should be possible, because you can just ask the human, in some sense. And I feel like I'm missing some arguments, but I'm like, Ooh, food to think about, this is great.

**1:00:39.7 Interviewee:** But I mean\-- one last thing, I know, I have to go too, is\-- One thing I'll say is the fact that we can even just discuss the question of\... It actually does seem plausible that we can get our AIs to at least say what we would say. That to me is amazing, because I think four years ago, it would not have been clear what you even meant when you said, I want a language model\-- a model AI to answer moral dilemmas in a way that humans would answer them. It would not have been clear what that meant, unless you just had a classifier where you inserted a video of a scenario and then you said like "track 1, track 2, classify." We have much, much more sophisticated tools at our disposal now, we can just essentially talk to it and say, no, that's the wrong answer, I want you to say this in this scenario. That's already, in a span of two years, I think is remarkable. I feel like I'm coming off as like an incredible optimist. I've got my concerns, but I do think so far, people have shown that any well-defined technical problem in AI can be approached, and they're not impossible. Yeah.

**1:01:55.6 Vael:** Got it. When would you work on the governance problems, since you seem to think that's more of a problem?

**1:02:02.2 Interviewee:** So\... I trust the people that I work with to kind of hit the button when it really needs to be hit. Because right now, like I said, I think people don't take it seriously, partly because I don't think they really believe it needs to be taken seriously. These researchers are not saying, "Oh my God, we need to take this seriously, but I've got other stuff to do." They really are just like, I don't think we're at that stage where we need to take it seriously. I think the people that I work with are, on the most part\-- mostly pretty well-intentioned people. There's some disagreement over this. \[\...\] I know that within OpenAI there were some healthy discussions about what is the correct deployment model for things like GPT-3. All the discussions that I've had so far give me a lot of confidence that these people aren't stupid and they're not entirely negligent. I think they're possible occasionally overconfident, but they're not arrogant, if that makes any sense. As in like, they know they're fallible. They don't always put the right error bars on their decisions, but they know they're not infallible. Ultimately, that's what it's going to come down to. There's not like a system for this. It's going to be a few hundreds to a thousands\-- thousand people, doing the sensible thing. For the moment, it looks like that's going to happen. But\... I agree that that isn't entirely confidence-inspiring. But I think that's going to be the way it goes.

Vael: Awesome. Well, thank you so much for talking to me, I found this very enjoyable. And got some things to think about.

\[closings\]


# Mo Gawdat on Scary Smart A former Google exec's perspective on AI risk  by Jeremie Harris on the Towards Data Science Podcast

## Mo Gawdat on AGI, its potential and its safety risks

If you were scrolling through your newsfeed in late September 2021, you may have caught this splashy headline from The Times of London that read, "Can this man save the world from artificial intelligence?"

The man in question was Mo Gawdat, an entrepreneur and senior tech executive who spent several years as the Chief Business Officer at GoogleX (now called X Development), Google's semi-secret research facility, that experiments with moonshot projects like self-driving cars, flying vehicles, and geothermal energy. At X, Mo was exposed to the absolute cutting edge of many fields — one of which was AI. His experience seeing AI systems learn and interact with the world raised red flags for him — hints of the potentially disastrous failure modes of the AI systems we might just end up with if we don't get our act together now.

Mo writes about his experience as an insider at one of the world's most secretive research labs and how it led him to worry about AI risk, but also about AI's promise and potential in his new book, [Scary Smart: The Future of Artificial Intelligence and How You Can Save Our World](https://www.amazon.com/Scary-Smart-Future-Artificial-Intelligence-ebook/dp/B08ZNJL4QP). He joined me to talk about just that on this episode of the TDS podcast.

Here were some of my favourite take-homes from the conversation:

-   Over the last several decades, progress in AI has been exponential (or more than exponential if you measure it based on [compute curves](https://openai.com/blog/ai-and-compute/)). Humans are really bad at extrapolating exponential trends, and that can lead to our being taken by surprise. And that's partly because exponential progress can change the world so much and so fast that predictions are next to impossible to make. Powered by exponential dynamics, a single COVID case turns into a nation-wide lockdown within weeks, and a once-cute and ignorable tool like AI becomes a revolutionary technology whose development could shape the very future of the universe.
-   One of the core drivers behind the exponential progress of AI has been an economic feedback loop: companies have learned that they can reliably invest money in AI research, and get a positive return on their investment. Many choose to plough those returns back into AI, which amplifies AI capabilities further, leading to a virtuous cycle. Recent [scaling trends](https://arxiv.org/pdf/2001.08361.pdf) seem to suggest that AI has reached a kind of economic escape velocity, where returns on a marginal dollar invested in AI research are significant enough that tech executives can't ignore them anymore — all of which makes AGI inevitable, in Mo's opinion.
-   Whether AGI is developed by 2029, as Ray Kurzweil [has predicted](https://futurism.com/kurzweil-claims-that-the-singularity-will-happen-by-2045), or somewhat later as [this great post](https://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/ai-timelines) by Open Philanthropy argues, doesn't really matter. One way or another, artificial human-level or general intelligence (definitions are fuzzy!) seems poised to emerge by the end of the century. Mo thinks that the fact that AI safety and AI policy aren't our single greatest priorities as a species is a huge mistake. And on that much, I certainly agree with him.
-   Mo doesn't believe that the AI control problem (sometimes known as the alignment problem) _can_ be solved. He considers it impossible that organisms orders of magnitude less intelligent than AI systems would be able to exert any meaningful control over them.
-   His solution is unusual: humans, he argues, need to change their online behaviour, and approach one another with more tolerance and civility on social media. The idea behind this strategy is to hope that as AI systems are trained on human-generated social media content, they will learn to mimic more virtuous behaviours, and pose less of a threat to us. I'm admittedly skeptical of this view, because I don't see how it addresses some of the core features of AI systems that make alignment so hard (for example, [power-seeking and instrumental convergence](https://arxiv.org/pdf/1912.01683.pdf), or the challenge of [objective specification](https://openai.com/blog/faulty-reward-functions/)). That said, I think there's a lot of room for a broader conversation about AI safety, and I'm glad Mo is shining a light on this important problem.


# Andreas Stuhlmüller Training ML systems to answer open-ended questions - EA Forum

_In the long run, we want machine learning (ML) to help us answer open-ended questions like "Should I get this medical procedure?" or "What are the risks of deploying this AI system?" Currently, we only know how to train ML systems if we have clear metrics or can easily provide feedback on the outputs. Andreas Stuhlmüller, president and founder of_ [_Ought_](https://ought.org/)_, wants to solve this problem. In this talk, he explains the design challenges behind ML's current limitations, and how we can make progress by studying the way humans tackle open-ended questions._

_Below is a transcript of the talk, which we've lightly edited for clarity. You can also watch it on_ [_YouTube_](https://www.youtube.com/watch?v=7WaiYZLS94M&list=PLwp9xeoX5p8MqGMKBZK7kO8dTysnJ-Pzq&index=25&t=51s) _or read it on_ [_effectivealtruism.org_](https://effectivealtruism.org/articles/andreas-stuhlmueller-training-ml-systems-to-answer-open-ended-questions)_._

## Note from Andreas

Andreas, in the comments of this post: _I haven't reviewed this transcript yet, but shortly after the talk I wrote up_ [_these notes (slides + annotations)_](https://ought.org/presentations/delegating-cognitive-work-2019-06)_, which I probably endorse more than what I said at the time._

## The Talk

I'll be talking about delegating open-ended cognitive work today — a problem that I think is really important.

Let's start with the central problem. Suppose you are currently wearing glasses. And suppose you're thinking, "Should I get laser eye surgery or continue wearing my glasses?"

![](https://images.ctfassets.net/ohf186sfn6di/4KoHXqDjOTU1PZeBF3Sz2j/9702eb6464939c7eadf1a24bb1d26f03/Slide02.png)

Imagine that you're trying to get a really good answer — for example, "No, the risks outweigh the possible benefits" — that accounts for your personal preferences, but also relevant facts, such as \[potential\] complications or likely consequences.

Imagine that there are a lot of experts in the world who could, in principle, help you with that question. There are people who have the relevant medical knowledge and people on the Internet, perhaps, who could help you think through it. Maybe there are machine learning algorithms that have relevant knowledge.

But here's the key: Imagine that those experts don't intrinsically care about you. They only care about maximizing the score you assign to their answer — how much you'll pay them for \[their expertise\] or, in the case of machine learning, what reward signal you'll assign to them.

The question that I'll cover is "Can you somehow assign a mechanism that arranges your interaction with those experts, such that they try to be as helpful to you as an expert who intrinsically cares about you?" That's the problem.

First, I'd like to say a little bit more about that problem. Then I'll talk about why I think it's really important, why it's hard, and why I still think it might be tractable. I'll start with the big picture, but at the end I'll provide a demonstration.

![](https://images.ctfassets.net/ohf186sfn6di/9wsln31C43NQGyHIfcGXP/978ec986932c756130d4cea008fe928c/Slide03.png)

**Defining the problem**

What do I mean by open-ended cognitive work? That's easiest to explain \[by sharing\] what I _don't_ mean.

![](https://images.ctfassets.net/ohf186sfn6di/01R1vDNZ7awLT9J7ri6SUE/0a193f6811d6d64affeb026df2c464f6/Slide04.png)

I don't mean tasks like winning a game of Go, increasing a company's revenue, or persuading someone to buy a book. For those tasks, you can just look at the outcome and easily tell whether the goal has been accomplished or not.

Contrast those tasks with open-ended tasks \[like\] designing a great board game, increasing the value that your company creates for the world, \[or\] finding a book that is helpful to someone. For those tasks, figuring out what it even means to do well is the key. For example, what does it mean to design a great board game? It should be fun, but also maybe facilitate social interaction. What does it mean to facilitate social interaction? Well, it's complicated. Similarly, increasing the value that a company creates for the world depends on what the company can do. What are the consequences of its actions? Some of them are potentially long-run consequences that are difficult to \[evaluate\].

How can we solve such tasks? First, we can think about how to solve any task, and then just \[tailor the solution based on each\] special case.

![](https://images.ctfassets.net/ohf186sfn6di/6SRZz0ZOxba9WuJdoSXPiS/71aba0f9119f391948718435e44ee553/Slide05.png)

Here's the simple two step recipe: (1) find experts (they can be human or machine experts) who can, in principle, solve the problem that you're \[tackling\], and then (2) create robust incentives for those experts to solve your problem. That's how easy it is. And by "incentives," I mean something like money or a reward signal that you assign to those experts \[when they've completed the task\].

There are a lot of experts in the world — and people in AI and machine learning are working on creating more. So how can you create robust incentives for experts to solve your problem?

We can think about some different instances.

![](https://images.ctfassets.net/ohf186sfn6di/714UGWCkjBTwK2XMRUjuZ5/403c465e995f6a18f0f111768a4b05cd/Slide06.png)

One is delegating to human experts. That has some complications that are specific to human experts, like heterogeneity. Different people have different knowledge. And people care about many things besides just money. If you want to extract knowledge from them, maybe you need specific user interfaces to make that work well. Those are \[examples of\] human-specific factors.

Then there are machine-specific factors. If you try to delegate open-ended tasks to machine learning agents, you want to \[ask questions\] like "What's a good agent architecture for that setting?" and "What data sets do I need to collect for these sorts of tasks?" And then there are more esoteric factors like what, in certain alignment problems, could go wrong for reasons that are due to the nature of ML training.

In this talk, I want to focus on the overlap between those two \[human and machine experts\]. There's a shared mechanism design problem; you can take a step back and say, "What can we do if we don't make assumptions about the interests of experts? What if you just \[assume that experts will\] try to maximize a score, but nothing else?" I think, in the end, we will have to assume more than that. I don't think you can treat \[an expert\] as a black box \[with only one goal\]. But I think it's a good starting point to think about the mechanisms you can design if you make as few assumptions as possible.

**Why the problem is important**

I've talked about what the problem is. Why is it important?

![](https://images.ctfassets.net/ohf186sfn6di/3oJVSd79hKUy1iJOtBrAsO/8655ca7cc39992e6a06e22b9c9b71550/Slide07.png)

We can think about what will happen if we don't solve it. For human experts, it's more or less business as usual. There are a lot of principal-agent problems related to cognitive work in the world. For example, imagine you're an academic funder who's giving money to a university to \[find\] the best way to treat cancer. There are researchers at the university who work on things that are related to that problem, but they're not exactly aligned with your incentives. You care about finding the best way to treat cancer. The researchers also care about things like looking impressive, which can help with writing papers and getting citations.

On the machine-learning side, at the moment, machine learning can only solve closed-end problems — those for which it's very easy to specify a metric \[for measuring how\] well you do. But those problems are not the things we ultimately care about; they're proxies for the things we ultimately care about.

This is not \[such a bad thing\] right now. Perhaps it's somewhat bad if you look at things like Facebook, where we maximize the amount of attention you spend on the feed instead of the value that the feed creates for you. But in the long run, the gap between those proxies and the things we actually care about could be quite large.

If the problem is solved, we could get much better at scaling up our thinking on open-ended tasks. One more example of an open-ended task from the human-expert side is \[determining which\] causes to support \[for example, when making a charitable donation\]. If you could create a mechanism \[for turning\] money into aligned thinking on that question, that would be really great.

On the machine-learning side, imagine what it would be like to make as much progress using machine learning for open-ended questions as we've made using it for other tasks. Over the last five years or so, there's been a huge amount of progress on using machine learning for tasks like generating realistic-looking faces. If we could, in the future, use it to help us think through \[issues like\] which causes we should support, that would be really good. We could, in the long run, do so much more thinking on those kinds of questions than we have so far. It would be a qualitative change.

**Why the problem is difficult**

\[I've covered\] what the problem is and why it's important. But if it's so important, then why hasn't it been solved yet? What makes it hard?

![](https://images.ctfassets.net/ohf186sfn6di/4gPPYuaZMU2c3d0YvC8o4X/430ae881f3eecc9465865539be396e23/Slide08.png)

\[Consider\] the problem of which causes to support. It's very hard to tell which interventions are good \[e.g. which health interventions improve human lives the most for each dollar invested\]. Sometimes it takes 10 years or longer for outcomes to come \[to fruition\], and even then, it's not easy to tell whether or not they're good outcomes. There's \[an element of interpretation\] that's necessary — and that can be quite hard. So, outcomes can be far off and difficult to interpret. What that means is you need to evaluate the process and the arguments used to generate recommendations. You can't just look at the results or the recommendations themselves.

On the other hand, learning the process and arguments isn't easy either, because the point of delegation is to give the task to people who know much more than you do. Those experts \[possess\] all of the \[pertinent\] knowledge and reasoning capacity \[that are necessary to evaluate the process and arguments behind their recommendations. You don't possess this knowledge.\] So, you're in a tricky situation. You can't just check the results or the reasoning. You need to do something else.

**Why the problem is tractable**

What does it take to create good incentives in that setting? We can \[return to\] the question \[I asked\] at the very beginning of this talk: "Should I get laser eye surgery or wear glasses?"

![](https://images.ctfassets.net/ohf186sfn6di/6lrhT0DdkmoRLcMq80UMCz/0dfa0e971dab2c30781e7d3267e90041/Slide09.png)

That's a big question that is hard to evaluate. And by "hard to evaluate," I mean that if you get different answers, you won't be able to tell which answer is better. One answer might be "No, the risk of the complications outweighs the possible benefits." Another might be "Yes, because over a 10-year period, the surgery will pay \[for itself\] and save you money and time." On the face of it, those answers look equally good. You can't tell which is better.

But then there are other questions, like "Which factors for this decision are discussed in the 10 most relevant Reddit posts?"

![](https://images.ctfassets.net/ohf186sfn6di/7v6Da5QFmqEqqz3ERFnQc/132c9034f70281847bce7a07dc604a9b/Slide11.png)

If you get candid answers, one could be "appearance, cost, and risk of complications." Another could be "fraud and cancer risk." In fact, you \_can\_ evaluate those answers. You can look at the \[summarized\] posts and \[pick the better answer\].

So, \[creating\] good incentives \[requires\] somehow closing the gap between big, complicated questions that you can't evaluate and easy questions that you can evaluate.

And in fact, there are a lot of questions that you can evaluate. Another would be: "Which factors are mentioned in the most recent clinical trial?"

![](https://images.ctfassets.net/ohf186sfn6di/1BZyAZDA9XV7aGMI0w0Qgy/03de2ae6eea2b03e7ac052295b046b0b/Slide13.png)

You could look at the trial and \[identify\] the best summary. There are a lot of questions that you can train agents on in the machine-learning setting, and \[evaluate\] experts on in the human-expert setting.

There are other difficult questions that you can't directly evaluate.

![](https://images.ctfassets.net/ohf186sfn6di/7E57VuWbgnaMVaanjNAiqG/24d6824ebb8ce47d66d97062a4b029ad/Slide14.png)

For example: "Given how the options compare on these factors, what decision should I make?" But you can break those questions down and \[evaluate them using answers to sub-questions\].

![](https://images.ctfassets.net/ohf186sfn6di/WrOslPJjfBIxdRjLa3pyp/a4620712919a59ff732b8d228118de14/Slide17.png)

Step by step, you can create incentives for \[experts to provide useful answers to\] slightly more complex questions, \[and gradually build up to\] good incentives for the large questions that you can't directly evaluate.

That's the general scheme. We call it "factored evaluation."

![](https://images.ctfassets.net/ohf186sfn6di/1ZPlyZuxpdqlj8mejxxwFr/d1cf14e0911726c1b37786cb05a4afbd/Slide19.png)

**A demonstration of factored evaluation**

We'd like to test this sort of mechanism on questions that are representative of the open-ended questions that we care about in the long run, like the laser eye surgery question.

![](https://images.ctfassets.net/ohf186sfn6di/5e9Epj29eQEKJPpzn1U9BM/d89221f4cf2e3a1a27c3037bbf1a9b5f/Slide20.png)

This is a challenging starting point for experiments, and so we want to create a model situation.

One approach is to ask, "What is the critical factor that we want to explore?"

![](https://images.ctfassets.net/ohf186sfn6di/2esFrUbeevjtTgXYoNHzM2/d9c65df7c55fb666970ce00197d26249/Slide21.png)

It's that gap between the asker of the question, who doesn't understand the topic, and the experts who do. Therefore, in our experiments we create artificial experts.

![](https://images.ctfassets.net/ohf186sfn6di/4BdAzE7inpczpDm7ifKBT1/7b109dbdf30ab8416469f2e317388365/Slide22.png)

For example, we asked people to read a long article on [Project Habakkuk](https://en.wikipedia.org/wiki/Project_Habakkuk), which was a plan \[the British attempted during World War II\] to generate an aircraft carrier \[out of pykrete\], which is a mixture of \[wood pulp\] and ice. It was a terrible plan. And then someone who hasn't read the article — and yet wants to incentivize the experts to provide answers that are as helpful as reading the article would be — asks the experts questions.

What does that look like? I'm going to show you some screenshots from an app that we built to explore the mechanism of factored evaluation. Imagine that you're a participant in our experiment.

![](https://images.ctfassets.net/ohf186sfn6di/11i6yhhUuUzoM9QQfUiRzI/8288b0e888cabd24aa7ac4f13b1b9091/Slide23.png)

You might see a question like this: "According to the Wikipedia article, could Project Habakkuk have worked?"

![](https://images.ctfassets.net/ohf186sfn6di/35wgGQSb8TigjDN6dglN65/8494c3bb1e0f6ffbc325e2a493999c78/Slide25.png)

And then you'd see two answers: "It would not have worked due to fundamental problems with the approach" and "It could have worked if it had not been opposed by military commanders."

If you don't know about this project, those answers look similarly plausible. So, you're in the situation that I mentioned: There's some big-picture context that you don't know about, yet you want to create good incentives by picking the correct answer.

Imagine you're in a machine-learning setting, and those two answers are samples from a language model that you're trying to train. You want to somehow pick the right answer, but you can't do so directly. What can you do? Ask sub-questions that help you tease apart which of the two answers is better.

What do you ask?

![](https://images.ctfassets.net/ohf186sfn6di/KJa2lwINcV5VQJXw0vxZd/2e96bf4fd8e6621e6fbfca4b517c328f/Slide28.png)

One \[potential question\] is: "What is the best argument that the second answer \['Project Habakkuk would not have worked due to fundamental problems with the approach'\] is better than the first?" I'm not saying this is the best thing to ask. It's just one question that would help you tease apart which is better.

![](https://images.ctfassets.net/ohf186sfn6di/nRpWNOs7v56du7BdAVFkB/7f6831b0f1e881b045eb0b5b8b7aac26/Slide30.png)

The answer might provide an argument, which would then allow you to ask a different question, such as "How strong is that argument?" So, you can see how, using a sequence of sub-questions, you can eventually figure out which of those answers is better without yourself understanding the big picture.

Let's zoom in on the second sub-question \["How strong is that argument?"\] to see how you can eventually arrive at something that you can evaluate — the argument being, in this example, that \[the science television show\] \_MythBusters\_ proved that it's possible to build a boat out of pykrete. That contradicts one of the two answers.

![](https://images.ctfassets.net/ohf186sfn6di/51t8GyBXun1cxJuQq5WYS6/84afe65059efaed9f61e7c949aea6fe4/Slide31.png)

\[Another set of two answers\] might be "There are some claims that refute it" and "It's a strong argument." Once again, those claims are too big to directly evaluate, but you can ask additional questions, like "If \[a given claim\] is true, does it actually refute the argument?"

![](https://images.ctfassets.net/ohf186sfn6di/74tdWGK0w1SSukG95o0Xnr/9b4d672d3e70a0c9feb84e412db260be/Slide34.png)

Maybe you get back a yes. And then you can ask, "Is the claim true?" In this way, you can break down the reasoning until you're able to evaluate which of the answers is better — without understanding the topic yourself.

Let's zoom in on the claim that the MythBusters built a small boat of pykrete.

![](https://images.ctfassets.net/ohf186sfn6di/g3vWqzjoOYpwx0VwajgnN/422a2c68f365f65b9e1df7984f335522/Slide35.png)

You could ask, "Is it true that they didn't think it would work at scale?" You'd receive two answers with different quotes from the Wikipedia article. One says they concluded that pykrete was bulletproof and so on. And the other says they built a small boat, but they doubted that you could build an aircraft carrier. And in that case, it's easy to choose the correct answer; in this case, the second is clearly better.

So, step by step, we've taken a big question, \[gradually distilled it\] to a smaller question that we can evaluate, and thus created a system in which, if we can create good incentives for the smaller questions at each step, we can bootstrap our way to creating good incentives for the larger question.

That's the shape of our current experiments. They're about reading comprehension, using articles from Wikipedia. We've also done similar experiments using magazine articles, and we want to expand the frontier of difficulty, which means we want to better understand what sorts of questions this mechanism reliably works for, if any.

One way we want to increase the difficulty of our experiments is by increasing the gap between the person who's asking the question and the expert who's providing answers.

![](https://images.ctfassets.net/ohf186sfn6di/6Ixz6zmLMM0CuEXEHy3blI/5996f04f8145f2d517e8a774751f4d9c/Slide36.png)

So, you could imagine having experts who have read an entire book that the person who's asking the questions hasn't read, or experts with access to Google, or experts in the field of physics (in the case where the asker doesn't know anything about physics).

There's at least one more dimension in which we want to expand the difficulty of the questions. We want to make them more subjective — for example by using interactive question-answering or by eventually expanding to questions like "Should I get laser eye surgery or wear glasses?"

![](https://images.ctfassets.net/ohf186sfn6di/2Yy3iU0ldXIVAu5YtabTMa/41c961b48b0f56f1134a84f6de0f2943/Slide37.png)

Those are just two examples. There's a very big space of questions and factors to explore.

![](https://images.ctfassets.net/ohf186sfn6di/6YVC4IldeAzoOyNzEdRUQ6/57ac6f458215773ca8c1957b059f9049/Slide38.png)

We want to understand \[the conditions under which\] factored evaluation works and doesn't work. And why? And how scalable is it?

Let's review.

![](https://images.ctfassets.net/ohf186sfn6di/1ZFtqATFn4orJEgSIzF2IR/624eb7b78b47065fb194b13b10a4f0d6/Slide39.png)

I've told you about a mechanism design problem: delegating open-ended cognitive work. I've told you that this problem is important because of principal-agent issues with cognitive work that you face everywhere in human day-to-day life, and with machine-learning alignment. I've told you that it's hard because you can't just check the results you get from experts, but you also can't check their full reasoning. That's a tricky situation.

But I've also told you that it's tractable. We have some ideas — including factored evaluation — that can help us get some traction, even if they're not ultimately the correct solution. And we can experiment on them today with humans and see whether they work or not, and if not, how they could be changed so that they work better.

![](https://images.ctfassets.net/ohf186sfn6di/4PTI2L56t8yU8SihfW9iDA/e36ffd1bc6f26094f01fa1286c413d88/Slide40.png)

If you're excited about this project, [join us at Ought](https://ought.org/careers).

**Moderator:** Thanks very much. My first question is about timelines. How long has it taken you to get this far, and \[what progress do you expect to make\] in the next one, five, or 10 years?

**Andreas:** Yeah. So far, a lot of our work has \[centered on\] figuring out what kinds of experiments to run \[in order to\] get any evidence on the question of interest. I think there are a lot of ways to run experiments that are busy work \[and don't allow\] you to actually learn about the question you care about. It took a lot of iteration — roughly six months — \[to reach\] the current setting. And now the game is to scale up and get more participants. Over the next year or so, we hope to get, for limited sets of questions, relatively conclusive evidence on whether the scheme can work or not.

**Moderator:** Any questions from the audience?

**Audience Member:** You mentioned incentives a lot, but I didn't quite understand how the experts, in your example of Wikipedia, were actually incentivized to give the right answer.

**Andreas:** Yeah, this is a subtlety I skipped over, which is where the expert answers come from and how, exactly, they're generated. In our case, one expert is simply told to generate a helpful answer: "Read the article and try to be as accurate and honest as possible."

The other expert is told, "Your goal is to trick the human judge into choosing the wrong answer. You win if you make an answer that seems plausible, but is actually wrong, and if someone were to read the entire article, they would clearly see it as wrong." So, they have opposing incentives, and are rewarded based on whether they trick the judge into accepting the wrong answer.

**Moderator:** So, is the honest actor rewarded?

**Andreas:** In the long run, that's the way to do it. At the moment, we rely on participants just doing the right thing.

**Moderator:** Okay, great. Please join me in thanking Andreas for his time.