# Mo Gawdat on Scary Smart A former Google exec’s perspective on AI risk  by Jeremie Harris on the Towards Data Science Podcast

## Mo Gawdat on AGI, its potential and its safety risks

If you were scrolling through your newsfeed in late September 2021, you may have caught this splashy headline from The Times of London that read, “Can this man save the world from artificial intelligence?”

The man in question was Mo Gawdat, an entrepreneur and senior tech executive who spent several years as the Chief Business Officer at GoogleX (now called X Development), Google’s semi-secret research facility, that experiments with moonshot projects like self-driving cars, flying vehicles, and geothermal energy. At X, Mo was exposed to the absolute cutting edge of many fields — one of which was AI. His experience seeing AI systems learn and interact with the world raised red flags for him — hints of the potentially disastrous failure modes of the AI systems we might just end up with if we don’t get our act together now.

Mo writes about his experience as an insider at one of the world’s most secretive research labs and how it led him to worry about AI risk, but also about AI’s promise and potential in his new book, [Scary Smart: The Future of Artificial Intelligence and How You Can Save Our World](https://www.amazon.com/Scary-Smart-Future-Artificial-Intelligence-ebook/dp/B08ZNJL4QP). He joined me to talk about just that on this episode of the TDS podcast.

Here were some of my favourite take-homes from the conversation:

-   Over the last several decades, progress in AI has been exponential (or more than exponential if you measure it based on [compute curves](https://openai.com/blog/ai-and-compute/)). Humans are really bad at extrapolating exponential trends, and that can lead to our being taken by surprise. And that’s partly because exponential progress can change the world so much and so fast that predictions are next to impossible to make. Powered by exponential dynamics, a single COVID case turns into a nation-wide lockdown within weeks, and a once-cute and ignorable tool like AI becomes a revolutionary technology whose development could shape the very future of the universe.
-   One of the core drivers behind the exponential progress of AI has been an economic feedback loop: companies have learned that they can reliably invest money in AI research, and get a positive return on their investment. Many choose to plough those returns back into AI, which amplifies AI capabilities further, leading to a virtuous cycle. Recent [scaling trends](https://arxiv.org/pdf/2001.08361.pdf) seem to suggest that AI has reached a kind of economic escape velocity, where returns on a marginal dollar invested in AI research are significant enough that tech executives can’t ignore them anymore — all of which makes AGI inevitable, in Mo’s opinion.
-   Whether AGI is developed by 2029, as Ray Kurzweil [has predicted](https://futurism.com/kurzweil-claims-that-the-singularity-will-happen-by-2045), or somewhat later as [this great post](https://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/ai-timelines) by Open Philanthropy argues, doesn’t really matter. One way or another, artificial human-level or general intelligence (definitions are fuzzy!) seems poised to emerge by the end of the century. Mo thinks that the fact that AI safety and AI policy aren’t our single greatest priorities as a species is a huge mistake. And on that much, I certainly agree with him.
-   Mo doesn’t believe that the AI control problem (sometimes known as the alignment problem) _can_ be solved. He considers it impossible that organisms orders of magnitude less intelligent than AI systems would be able to exert any meaningful control over them.
-   His solution is unusual: humans, he argues, need to change their online behaviour, and approach one another with more tolerance and civility on social media. The idea behind this strategy is to hope that as AI systems are trained on human-generated social media content, they will learn to mimic more virtuous behaviours, and pose less of a threat to us. I’m admittedly skeptical of this view, because I don’t see how it addresses some of the core features of AI systems that make alignment so hard (for example, [power-seeking and instrumental convergence](https://arxiv.org/pdf/1912.01683.pdf), or the challenge of [objective specification](https://openai.com/blog/faulty-reward-functions/)). That said, I think there’s a lot of room for a broader conversation about AI safety, and I’m glad Mo is shining a light on this important problem.

Mo Gawdat - Scary Smart - A former Google exec_s perspective on AI┬árisk-by Towards Data Science-video_id u2cK0_jUX_g-date 20220126.md


# Jade Leung and Seth Baum The role of existing institutions in AI strategy - EA Forum

_AI is very likely to make a huge impact on our world, especially as it grows more powerful than it is today. It’s hard for us to know exactly how that impact will look, but we do know many of the actors most likely to be involved. As AI gets stronger, what can we expect the world’s most powerful national governments to do? What about nongovernmental organizations, like the UN?_

_This advanced workshop from Effective Altruism Global: San Francisco 2018, presented by Jade Leung and Seth Baum, addresses these questions from multiple perspectives. A transcript of the workshop is below, which we have lightly edited for clarity. You can also watch the talk on_ [_YouTube_](https://www.youtube.com/watch?v=pgiwvmY3brg&list=PLwp9xeoX5p8P3cDQwlyN7qsFhC9Ms4L5W&index=3) _and read it on_ [_effectivealtruism.org_](https://www.effectivealtruism.org/articles/ea-global-2018-the-role-of-existing-institutions-in-ai-strategy/)_._

## The Talk

**Jade:** What we're going to do is we're going to introduce ourselves briefly so you kind of know where we're coming from. Then we've got two moots which we have just then decided were the two moots that we're going to talk about. We'll chuck them up on the board and we'll spend about half a session talking about one and then half a session talking about the other. This is a session where we'd both love for you guys to toss us your questions right throughout it basically so, yes, get ready to have your questions ready and we'll open it up pretty much soon after the intro.

Briefly intro to myself. I currently am based in the Future of Humanity Institute, and the work that I do specifically looks at the relationships between large multi-national technology firms and governments, specifically National Security and Defense components of governments in the US and China. And the questions that I ask are about how these actors should relate to each other, cooperate, coordinate, to steer us towards a future, or set of futures, that are more safe and beneficial than not, with transformative AI. My background is in engineering, I am masquerading as international relations person, but I'm not really that. I do a fair amount in the global governance space, in the IR space largely. That's me.

**Seth:** Cool. I'm Seth Baum, I was introduced with the Global Catastrophic Risk Institute, and as a think tank we try to sit in that classic think tank space of working at the intersection of, among other things, the world of scholarship and the world of policy. We spend a lot of time talking with people in the policy worlds, especially down in DC. For me, it's down in DC, I live in New York. I guess from here it would be over in DC. Is that what you say? You don't live here.

**Jade:** Sure.

**Seth:** Over in DC. And talking with people in policy. I work across a number of different policy areas, do a lot on nuclear weapons, little bit on biosecurity, and then also on AI, and especially within the last year or two there have been some more robust policy conversations about AI. The policy world has just started to take an interest in this topic and is starting to do some interesting things that have fallen on our radar, and so we'll be saying more about that. Do you want to?

**Jade:** Yeah, sure.

So the two institutions that we're going to chat about, is firstly the National Security and Defense. We might focus on the US National Security and Defense, and have a bit of a chat about what makes sense to engage them on in the space of our strategy, and how we should be thinking about their role in this space. That's the first moot. The second will turn to more international institutions, the kind of multilateral groups, e.g. the UN but not strictly so, and what role they could play in the space of AI strategy as well. We'll kind of go half and half there.

Just so I have a bit of a litmus test for who's in the audience, if I say AI strategy, who does that mean anything to? Ah, awesome. Okay, cool. Maybe we'll just start with getting Seth's quick perspective on this question. So the moot here is, this house believes that in the space of AI strategy, we should be actively engaging with National Security and Defense components of the US government. Do you want to speak quickly to what your quick take on that is?

**Seth:** Sure. So an interesting question here is engaging with, say the US government especially on the national security side, is this a good thing or a bad thing? I feel like opinions vary on this, maybe even within this room opinions vary on whether having these conversations is a good thing or a bad thing. The argument against it that I hear is essentially, you might tell them AI could take over the world and kill everyone, and they might hear, AI could take over the world, hear that and then go on to do harmful things.

I personally tend to be more skeptical of that sort of argument. The main reason for that is that the people who are in the government and working on AI, they've already heard this idea before. It's been headline news for a number of years now, some people from our communities including your organization caused some of those headlines.

**Jade:** I feel like you're asking me to apologize for them, and I'm not going to.

_Seth_: If one is concerned about the awareness of various people in government about runaway AI, you could ask questions like, was the publication of the Superintelligence book a good thing or a bad thing? You could maybe there make a case in either direction-

**Jade:** Could we do a quick poll actually? I'd be curious. Who thinks the publication of Superintelligence was on net, a net positive thing? On net, a negative thing? Hell yeah.

**Seth:** Doesn't mean that that's actually true.

**Jade:** Fair enough.

**Seth:** Just to be clear, I'm not arguing that it was a net negative, but the point is that the idea is out, and the people who work on AI, sure, they're mostly working on a narrow near term AI, but they've heard the idea before. They don't need us to put the thought into their heads. Now of course we could be kind of strengthening that thought within their heads, and that can matter, but at the same time when I interact with them, I actually tend to not be talking about superintelligence, general intelligence, that stuff anyway. Though more for a different reason, and that's because while they have heard of the idea, they're pretty skeptical about it. Either because they think it probably wouldn't happen or because if it would happen it would be too far in the future for them to worry about. A lot of people in policy have much more near term time horizons that they have to work with. They have enough on their plate already, nobody's asking them to worry about this, so they're just going to focus on the stuff that they actually need to worry about, which includes the AI that already exists and is in the process of coming online.

What I've found is then because they're pretty dismissive of it, I feel like if I talk about it they might just be dismissive of what I have to say, and that's not productive. Versus instead if the message is we should be careful about AI that acts unpredictably and causes unintended harms, that's not really about superintelligence. That same message applies to the AI that exists already: self driving cars, autonomous weapons. You don't want autonomous weapons causing unintended harm, and that's a message that people are very receptive to. By emphasizing that sort of message we can strengthen that type of thinking within policy worlds. That's for the most part the message that I've typically gone with, including in the National Security communities.

**Jade:** Cool. I've got a ton of questions for you, but maybe to quickly interject my version of that. I tend to agree with a couple of things that Seth said, and then disagree with a couple specific things.

I think generally the description of my perspective on this is that there's a very limited amount of useful engagement with National Security today, and I think the amount of potential to do wrong via engaging with them is large, and sufficiently large that we should be incredibly cautious about the manner in which we engage. That is a different thing to saying that we shouldn't engage with them at all, and I'll nuance that a little bit. I think, maybe to illustrate, I think the priors or assumptions that people hold when they're taking a stance on whether you should engage with National Security or not, is people I think disagree on maybe three axes. I said three because people always say three, I'm not entirely sure what the three are but we'll see how this goes.

So I think the first is people disagree on the competence of National Security to pursue the technology themselves, or at least to do something harmful with said information about capabilities of the technology. I think some people hold the extreme view that they're kind of useless and there's nothing that they can do in-house that is going to cause technology to be more unsafe than not, which is the thing that you're trying to deter. On the other hand, some people believe that NatSec at least have the ability to acquire control of this technology, or can develop it in-house sufficiently so, that an understanding of significant capabilities of AI would lead them to want to pursue it, and they can pursue it with competence, basically.

I think that kind of competence thing is one thing that people disagree on, and I would tend to land on them being more competent than people think. Even if that's not the case, I think it's always worth being conservative in that sense anyways.

So that's the first axis. Second axis I think is about whether they have a predisposition, or whether they have the ability to absorb this kind of risk narrative effectively, or whether that's just so orthogonal to the culture of NatSec that it's not going to be received in a nuanced enough way and they're always going to interpret whatever information with a predisposition to want to pursue unilateral military advantage, regardless of what you're saying to them. Some people on one end would hold that they are reasonable people with a broad open mind, and plausibly could absorb this kind of long-term risk narrative. Some other people would hold that information that is received by them will tend to just be received with the lens of how can we use this to secure a national strategic advantage.

I would tend to land on us having no precedent for the former, and having a lot more precedent for the latter. I think I'd like to believe that folks at DOD and NatSec can absorb, or can come around more to the long term risk narrative, but I don't think we've seen any precedent enough for that to place credence on that side of the spectrum. That's kind of where I sit on that second axis.

I said I had a third, I'm not entirely sure what the third is, so let's just leave it at two.

I think that probably describes the reasons why I hold that I think engaging with NatSec can be plausibly useful, but for every kind of one useful case, I can see many more reasons why engaging with them could plausibly be a bad idea, at least at this stage. So I'd encourage a lot more caution than I think Seth would.

**Seth:** That's interesting. I'm not sure how much caution… I would agree, first of all I would agree, caution is warranted. This is one reason why a lot of my initial engagement is oriented towards generically safe messages like, "avoid harmful unintended consequences." I feel like there are limits to how much trouble you can get in spreading messages like that. It's a message that they will understand pretty uniformly, it's just an easy concept people get that. They might or might not do much with it, but it's at least probably not going to prompt them to work in the wrong directions.

As far as their capability and also their tendency to take up the risk narrative, it's going to vary from person to person. We should not make the mistake of treating National Security communities even within one country as being some monolithic entity. There are people of widely varying technical capacity, widely varying philosophical understanding, ideological tendencies, interest in having these sorts of conversations in the first place, and so on.

A lot of the work that I think is important is meeting some people, and seeing what the personalities are like, seeing where the conversations are especially productive. We don't have to walk in and start trumpeting all sorts of precise technical messages right away. It's important to know the audience. A lot of it's just about getting to know people, building relationships. Relationships are really important with these sorts of things, especially if one is interested in a more deeper and ongoing involvement in it. These are communities. These are professional communities and it's important to get to know them, even informally, that's going to help. So I would say that.

**Jade:** I tend to agree with that sentiment in particular about building a relationship and getting trust within this community can take a fair amount of time. And so if there's any sort of given strategic scenario in which it's important to have that relationship built, then it could make sense to start some paving blocks there.

**Seth:** It is an investment. It is an investment in time. It's a trade off, right?

**Jade:** What's an example of a productive engagement you can think of having now? Say if I like put you in a room full of NatSec people, what would the most productive version of that engagement look like today?

**Seth:** An area that I have been doing a little bit of work on, probably will continue to do more, is on the intersection of artificial intelligence and nuclear weapons. This is in part because I happen to have also a background on nuclear weapons, a scenario where I have a track record, a bit of a reputation, and I know the lingo, know some of the people, can do that. AI does intersect with nuclear weapons in a few different ways. There is AI built into some of the vehicles that deliver the nuclear weapon from point A to point B, though maybe not as much as you might think. There's also AI that can get tied into issues of the cybersecurity of the command and control Systems, essentially the computer systems that tie the whole nuclear enterprise together, and maybe one or two other things. The National Security communities, they're interested in this stuff. Anything that could change the balance of nuclear power, they are acutely interested in, and you can have a conversation that is fairly normal from their perspective about it, while introducing certain concepts in AI.

**Seth:** So that's one area that I come in. The other thing I like about the nuclear weapons is the conversation there is predisposed to think in low frequency, high severity risk terms. That's really a hallmark of the nuclear weapons conversation. That has other advantages for the sorts of values that we might want to push for. It's not the only way to do it, but if you were to put me in a room, that's likely to be the conversation I would have.

**Jade:** So if you were to link that outcome to a mitigation of risk as an end goal, how does them understanding concepts better in AI translate into a mitigation of risk, broadly speaking? Assuming that's the end goal that you wanted to aim for.

**Seth:** One of the core issues with AI is this question of predictability and unintended consequences. You definitely do not want unpredictable AI managing your nuclear weapons. That is an easy sell. There is hyper-caution about nuclear weapons, and in fact if you look at the US procurement plans for new airplanes to deliver nuclear weapons, the new stealth bomber that is currently being developed, will have an option to be uninhabited, to fly itself. I think it might be remote controlled. The expectation is that it will not fly uninhabited on nuclear missions. That they want a human on board when there is also a nuclear weapon there, just in case something goes wrong. Even if the system is otherwise pretty reliable, that's just their… That's how they would look at this, and I think that's useful. So here we have this idea that AI might not do what we want it to, that's a good starting point.

**Jade:** Sure, cool. Let's toss it out to the audience for a couple of questions. We've got like 10 minutes to deal with NatSec and then we're going to move on into multilaterals. Yeah, go for it.

I didn't realize you were literally one behind the other. Maybe you first and then we'll go that way.

**Audience Member:** I was just in Washington, DC for grad school and had a number of friends who were working for think tanks that advise the military on technical issues like cybersecurity, or biosecurity, and I definitely felt like I had this sense of maybe the people in charge were pretty narrow-minded, but that there's this large non-homogenous group of people, some of whom were going to be very thoughtful and open-minded and some of whom weren't. And that there's definitely places where the message could fall on the right ears, and maybe something useful done about it, but it would be really hard to get it into the right ears without getting it into the wrong ears. I was wondering if you guys have any feelings about, is there a risk to giving this message or to giving a message to the wrong people? Or is that like very little risk, and it will just go in one ear and out the other if it goes to the wrong person? I feel like you could think about that either way.

**Jade:** Yeah, I'm curious to hear more about your experience actually, and whether there was a tendency for certain groups, or types of people to be the right ears versus the wrong ears. If you've got any particular trends that popped out to you, I'd love to hear that now or later or whenever.

But as a quick response, I think there's a couple of things to break down there. One is, what information are you actually talking about, what classifies as bad information to give versus good.

Two, is whether you have the ability to nuance the way that it's received, or whether it goes and is received in some way, and the action occurs without your control. I think, in terms of good information, that I would be positive about good ears receiving, and a bit meh about more belligerent ears received it, they couldn't actually do anything useful with the information anyway.

I think anything that nuances the technicality of what the technology does and doesn't do, generally is a good thing. I think also the element of introducing that risk narrative, if it falls on good ears, it can go good ways, if it falls on bad ears, they're just going to ignore it anyway.

You can't actually do anything actively bad with information about there being a risk, that maybe you don't have a predisposition to care about anyway. I'd say that's good information. I think the ability for you to pick the right ears for it to be received by, I'm skeptical about that.

I'm skeptical about the ability for you translate reliably up the hierarchy where it lands in a decision maker's hands, and actually translates into action that's useful. That would be my initial response to that, is that even if it exists and it's a more heterogeneous space than what would assume, I wouldn't trust that we have the ability to read into that well, is my response.

**Seth:** I would say I find it really difficult to generalize on this. In that, each point of information that we might introduce to a conversation is different. Each group that we would be interacting with can be different, and different in important ways. I feel, if we are actually in possession of some message that really is that sensitive then, to the extent that you can, do your homework on who it is that you're talking to, what the chain of command, the chain of conversation looks like.

If you're really worried, having people who you have a closer relationship with, where there may be at least some degree of trust, although, who knows what happens when you tell somebody something? Can you really trust me with what you say? Right? You don't know who else I'm talking to, right? So on for anyone else. At the end of the day, when decisions need to be made, I would want to look at the whole suite of factors, this goes for a lot of what we do, not just the transmission of sensitive information.

A lot of this really is fairly context specific and can come down to any number of things that may be seemingly unrelated to the thing that we think that we are talking about. Questions of bureaucratic procedure that get into all sorts of arcane minute details could end up actually being really decisive factors for some of these decisions.

It's good for us to be familiar, and have ways of understanding how it all works, that we can make these decisions intelligently. That's what I would say.

**Jade:** Cool.

**Audience Member:** All right, so from what I understand, a lot of people are new to this space. What sort of skills do you think would be good for people to learn? What sort of areas, like topics, should people delve into to prove themselves in AI strategy? What sort of thinking is useful for this space?

**Seth:** That's a good question. Should I start?

**Jade:** Yeah.

**Seth:** Okay. That's a good question. I feel for those who really want to have a strong focus on this, it helps to do a fairly deep dive into the worlds that you would be interacting with.

I can say from my own experience, I've gotten a lot of mileage out of fairly deep dives into a lot of details of international security.

I got to learn the distinction between a fighter plane and a bomber plane for example. The fighter plans are smaller and more agile, and maneuverable and the bombers are big sluggish beasts that carry heavy payloads and it's the latter that have the nuclear weapons, it's the former that benefit from more automation and a faster more powerful AI, because they're doing these really sophisticated aerial procedures, and fighting other fighter planes and that's… The more AI you can pack into that, the more likely you are to win, versus the bomber planes it just doesn't matter, they're slow and they're not doing anything that sophisticated in that regard.

That's just one little example of the sort of subtle detail that comes from a deeper dive into the topic that, in conversations, can actually be quite useful, you're not caught off guard, you can talk the lingo, you know what they're saying, you can frame your points in ways that they understand.

Along the way you also learn who is doing what, and get in that background. I would say it helps to be in direct contact with these communities. Like myself, I live in New York, I don't live in Washington, but I'm in Washington with some regularity attending various events, just having casual conversations with people, maybe doing certain projects and activities, and that has been helpful for positioning myself to contribute in a way that, if I want to, I can blend in.

They can think of me as one of them. I am one of them, and that's fine. That's normal. While also being here, and being able to participate in these conversations. So that's what I would recommend, is really do what you can to learn how these communities think and work and be able to relate to them on their level.

**Jade:** Addition to that would be, try to work on being more sensible, is the main thing I would say. It's one of those things where, a shout out to CFAR for example, those kind of methodologies… basically, I think the people that I think are doing the best work in this space, are the people who have the ability to A. Absorb a bunch of information really quickly, B. Figure out what is decision relevant quickly, and C. Cut through all the bullshit that is not decision relevant but that people talk about a lot.

I think those three things will lead you towards asking really good questions, and asking them in a sensible way, and coming to hypotheses and answers relatively quickly, and then knowing what to do with them.

Sorry, that's not a very specific answer, just work on being good at thinking, and figure out ways to train your mind to pick up decision relevant questions.

**Audience Member:** CFAR would be a good be a good organization for that, is that what you're saying?

**Jade:** CFAR would be epic, yeah. We've got a couple people from CFAR in the audience, I think. Do you want to put your hand up? If you're here. Nice. So, have a chat to them about how to get involved.

The other thing I'd say, is there is a ton of room for different types of skills, and figuring out where your comparative advantage is, is a useful thing.

I am not a white male, so I have a less comparative advantage in politics, I'm not a US citizen, can't do USG stuff, those are facts about me that I know will lead me toward certain areas in this space.

I am an entrepreneur by background, that leads me to have certain skills that maybe other people marginally don't have. Think about what you enjoy, what you're good at, and think about the whole pipeline of you doing useful stuff, which starts probably at fundamentally researching things, and ends at influencing decision makers/being a decision maker. Figure out where in that pipeline you are most likely to have a good idea.

Another shout out to 80k, who does a lot of good facilitation of thinking about what one's comparative advantage could be, and helps you identify those, too.

**Seth:** You mentioned the white male thing, and yeah sure, that's a thing.

**Jade:** That was genuinely not a dig at you being a white male.

**Seth:** No.

**Jade:** I promise. It's a dig at all of you for being white males. I just realized this is recorded, and this has gone so far downhill I just can't retract any of that. We're going to keep going.

**Seth:** So, for example, if I was attending a national security meeting instead of this, I might have shaved. Right? Because, it's a room full of a lot of people who are ex-military, or even active military or come from more… much of the policy culture in DC is more conservative, they're wearing suits and ties. Is there a single suit and tie in this room? I don't see one.

It's pretty standard for most of the events there that I go to. Simple things like that can matter.

**Jade:** Yeah.

**Seth:** You don't have to be a white male to succeed in that world. In fact, a lot of the national security community is actually pretty attentive to these sorts of things, tries to make sure that their speaking panels have at least one woman on them, for example.

There are a lot of very successful women in the national security space, very talented at it, and recognized as such. You don't have to look like me, minus the beard.

**Jade:** Nice. That's good to know. It's always useful having a token women's spot, actually. All right, one last question on NatSec, then we're going to move on. Yeah?

**Audience Member:** What do you think about the idea of measurements of algorithmic and hardware progress, and the amount of money going into AI and those kinds of measurements becoming public, and then NatSec becoming aware of?

**Jade:** That's a really interesting question.

I'm generally very, pro-that happening. I think those efforts are particularly good for serving a number of different functions. One is, the process of generating those metrics is really useful for the research community, to understand what metrics we actually care about measuring versus not. B, the measurement of them systematically across a number of different systems is very useful for at least starting conversations about which threshold points we care about superseding, and what changes about your strategy if you supersede certain metrics particularly quicker than you expected to.

I'm generally pro-those things, in terms of… I guess the pragmatic question is whether you can stop the publication of them anyway, and I don't think you can. I would say that if you had the ability to censor them, it would still be a net positive to have that stuff published for the things that I just mentioned.

I would also plausibly say that NatSec would have the ability to gather that information anyway. Yeah. I don't necessarily also think it's bad for them to understand progress better, and for them to be on the same page as everyone else about, specifically as the same as the technical research community, about how these systems are progressing. I don't think that's a bad piece of information necessarily, sorry, that was a really handwoven answer, but…

**Seth:** I feel like it is at least to an approximation reasonable to assume that if there's a piece of information and the US intelligence community would like that information, they will get it.

Especially if it's a relatively straightforward piece of information like that, that's not behind crazy locked doors and things of that sort. If it's something that we can just have a conversation about here, and they want it, they will probably get that information. There may be exceptions, but I think that's a reasonable starting point.

But I feel like what's more important than that, is the question of like, the interpretation of the information, right? It's a lot of information, the question is what does it mean?

I feel like that's where we might want to think more carefully about how things are handled. Even then there's a lot of ideas out there, and our own ideas on any given topic are still just another voice in a much broader conversation.

We shouldn't overestimate our own influence on what goes on in the interpretation of intelligence within a large bureaucracy. If it's a question of, do we communicate openly where the audience is mostly say, ourselves, right, and this is for our coordination as a community, for example?

Where, sure, other communities may hear this, whether in the US or anywhere around the world, but to them we're just one of many voices, right? In a lot of cases it may be fair to simply hide in plain sight. In that, who are we from their perspective, versus who are we from our perspective? We're paying attention to ourselves, and getting a lot more value of it.

Again, you can take it on a case by case basis, but that's one way of looking at it.

**Jade:** Cool. We're going to segue into talking about international institutions, maybe just to frame this chat a little bit. Specifically the type of institutions that I think we want to talk about, are probably multi-lateral state-based institutions.

That being, the UN and the UN's various children, and those other bodies that are all governed by the system. That assumes a couple of things: one, that states are the main actors at the table that mean anything, and two, that there are meaningful international coordination activities. Institutions are composed of state representatives and various things. The question here is, are they useful to engage with? I guess that's like a yes or no question.

Then if you want it nuance it a bit more, what are they useful for versus what are they not? Does that sound like a reasonable…

**Seth:** Yes.

**Jade:** My quick hot take on that, then I'll pass it over to Seth. I'll caveat this by saying, well I'll validate my statement by saying that I've spent a lot of my academic life working in the global governance space.

That field is fundamentally very optimistic about these institutions, so if anything I had the training to predispose me to be optimistic about them, and I'm not. I'm pessimistic about how useful they are for a number of reasons.

I think A is to do with the state-centric approach, B is to do with precedent, about what they're useful for versus not, and C it's also the pace at which they move.

To run through each one of those in turn, I think the assumption that a lot these institutions held, and they were built to rely on these assumptions, that states the core actors who are needing to be coordinated.

They are assumed to have the authority and legitimacy, to move the things that need to move, in order for this coordination to do the thing you want it to do. That is a set of assumptions that I think used to hold better, but almost certainly doesn't hold now, and almost certainly doesn't hold in the case of AI.

Particularly so, actors that I think are neglected and aren't conceptualized reasonably in these international institutions, large firms, and also military and security folks, or that component of government, doesn't tend to be the component of government that's represented in these institutions.

Those two are probably the most important actors, and they aren't conceptualized as the most important actors in that space. That's one reason to be skeptical, that by design they aren't designed to be that useful.

I think two, in terms of historically what they've been useful for, I think UN institutions have been okay at doing non-setting, non-building, non-proliferation stuff, I think they've been okay at doing things like standard setting, and instituting these norms and translating them into standards that end up proliferating across industries. That is useful as a function. I'll say particularly so in the case of technologies, the standardization stuff is useful, so I'm more optimistic about bodies like the ISO, which stands for the International Standards something, standards thing. Organization, I guess. Does that seem plausible? That seems plausible. I'm optimistic about them more so than I am about like the UN General Council or whatever. But, in any case, I think that's kind of a limited set of functions, and it doesn't really cover a lot of the coordination cooperation that we want it to do.

And then third is that historically these institutions have been so freaking slow at doing anything, and that pace is not anywhere close to where it needs to be. The one version of this argument is like if that's the only way that you can achieve the coordination activities that you want, then maybe that's the best that you have, but I don't think that's the best that we have. I think there are quicker arrangements between actors directly, and between small clubs of actors specifically, that will just be quicker at achieving the coordination that we need to achieve. So I don't think we need to go to the effort of involving slow institutions to achieve the ends that we want to. So, that's kind of why I'm skeptical about the usefulness of these institutions at all, with the caveat of them being useful for standard setting potentially.

**Seth:** I feel like people at those institutions might not disagree with what you just said. Okay, the standards thing, I think that's an important point. Also… so the UN. A lot of what the UN does operates on consensus across 200 countries. So yeah, that's not going to happen all that much. To the extent that it does happen, it's something that will often build slowly over time. There may be some exceptions like astronomers find an asteroid heading towards Earth, we need to do something now. Okay, yeah, you could probably get a consensus on that. And even then, who knows? You'd like to think, but… and that's a relatively straightforward one, because there's no bad guys. With AI, there's bad guys. There's benefits of AI that would be lost if certain types of AI that couldn't be pursued, and it plays out differently in different countries and so on, and that all makes this harder.

Same story with like climate change, where there are countries who have reasons to push back against action on climate change. Same thing with this. I'd say the point about states not necessarily being the key actors is an important one, and I feel like that speaks to this entire conversation, like is it worth our time to engage with national and international institutions? Well, if they're not the ones that matter, then maybe we have better things to do with our time. That's fair, because it is the case right now that the bulk of work of AI is not being done by governments. It's being done by the private corporate sector and also by academia. Those are, I would say, the two main sources, especially for the artificial general intelligence.

Last year, I published a survey of general intelligence R&D projects. The bulk of them were in corporations or academia. Relatively little in governments, and those, for the most part, tended to be smaller. There is something to be said for engaging with the corporations and the academic institutions in addition to, or possibly even instead of, the national government ones. But that's a whole other matter.

With respect to this, though, international institutions can also play a facilitation role. They might not be able to resolve a disagreement but they can at least bring the parties together to talk to them. The United Nations is unusually well-equipped to get, you know, pick your list of countries around the room together and talking. They might not be able to dictate the terms of that conversation and define what the outcome is. They might not be able to enforce whatever agreements, if any, were reached in that conversation. But they can give that conversation a space to happen, and sometimes just having that is worthwhile.

**Jade:** To what end?

**Seth:** To what end? In getting countries to work on AI in a more cooperative and less competitive fashion. So even in the absence of some kind of overarching enforcement mechanism, you can often get cooperation just through these informal conversations and norms and agreements and so on. The UN can play a facilitation role even if it can't enforce every country to do what they said they would do.

**Jade:** What's the best example you have of a facilitated international conversation changing what would have been the default state behavior without that conversation?

**Seth:** Oh, that's a good question. I'm not sure if I have a…

**Jade:** And if anyone actually in the audience actually has… yes.

**Audience Member:** Montreal Protocol.

**Jade:** Do you want to expand? I don't think that was not going to happen.

**Seth:** So the Montreal Protocol for ozone. Did you want to expand on that?

**Audience Member:** Yeah, it was a treaty that reduced emission… They got a whole bunch of countries to reduce emissions of greenhouse gases that would effectively destroy the ozone layer, and brought those emissions to very low levels, and now the ozone layer is recovering. Arguably, without that treaty, like maybe that wouldn't have happened. I don't know what the counterfactual would be.

**Jade:** Maybe. Yeah, and I think the Montreal… that's a good example. I think the Montreal Protocol… there was a clear set of incentives. There were barely any downsides for any state to do that. So put that alongside the Kyoto Protocol, for example, where the ask was somewhat similar, or similarly structured. Off the record, she says as this is being recorded live, I don't think the Kyoto Protocol had any win… as close as effective as the Montreal Protocol/wasn't even close to achieving whatever the goals were on paper. I think the reason was because the gas that was being targeted, there were very clear economic incentives for states to not mitigate those. In so far as the Montreal Protocol was a good example, it maybe like pointed out a really obvious set of incentives that just were going downhill anyways. But I don't know if it tweaked any of those, would be my response to that.

**Seth:** It is the case that some types of issues are just easier to get cooperation on than others. If there's a really clear and well-recognized harm from not cooperating, and the cost of cooperating is relatively low. I am not as much an expert on the Montreal Protocol but, superficially, my understanding is that addressing the ozone issue just happened to be easier than addressing the climate change issue, which has just proved to be difficult despite efforts. They might have gone about the Kyoto Protocol in a rather suboptimal fashion potentially but even with a better effort the climate change might just be harder to get collective action on, given the nature of the issue.

Then likewise, the question for us is so what does AI look like? Is it something that is easy to get cooperation on or not? Then what does that mean for how we would approach it?

**Jade:** Yeah, and I think, if anything… if you were to put the Montreal Protocol on one end of the spectrum where, I guess like the important things to abstract away from that particular case study is that you had a very clear set of incentives to mitigate this thing, and you had basically no incentive for anyone to keep producing the thing. So, that was easy. Then somewhere in the middle is the Kyoto Protocol where you've got pretty large incentives to mitigate the thing because climate, and then you've got some pretty complicated incentives to want to keep producing the thing, and the whole transition process is like hard and whatnot. And then we didn't sufficiently have sort of critical mass of believing that it was important to mitigate the thing, so it just became a lot harder. I think AI, I would put on that end of the spectrum, where you've got so many clear incentives to keep pursuing the thing. If anything, because you've got so many different uses that it's just economically very tasty for countries to pursue, not just countries but a number of other actors who want to pursue it. You've got people who don't even believe it's worth mitigating at all.

So I think, for that reason, I'd put it as astronomically bloody hard to do the cooperation thing on that side, at least in the format of international institutions. So I think the way to make it easier is to have a smaller number of actors and to align incentives and then to make clearer, sort of like binding mechanisms for that to have a shot in hell at working, in terms of cooperation.

**Seth:** But it could depend on which AI we're talking about. If you would like an international treaty to just stop the development of AI… yeah, I mean, good luck with that. That's probably not going to happen. But, that's presumably not what we would want in the first place because we don't need the restriction of all AI. There's plenty of AI that we're pretty confident can be a net positive for the world and we would not want that AI to be restricted. It would be in particular the types of AI that could cause major catastrophes and so on. That's what we would be especially interested in restricting. So an important question, this is actually more of like a technical computer science question than an international institutions question, but it feeds directly into this is, so which AI would we need to restrict? With an eye towards say future catastrophe scenarios, is it really like the core mainstream AI development that needs to be restricted, because all of that is a precursor to the stuff that could get out of hand? Or is it a fairly different, distinct branch of AI research that could go in that direction, such that the mainstream AI work can keep doing what it's doing? So there'll be some harms from it but they'll be more manageable, less catastrophic. How that question is answered, I think, really speaks to the viability of this.

**Jade:** Yeah. I guess what I'm skeptical of is the ability to segregate the two. Like I don't think there are clear delineations, and if people have ideas for this please tell me, but I don't think there are clear delineations for separating what are civilian, peaceful, good applications from military applications, at least in technical terms. So it becomes hard, if you want to design a thing, if you don't what the thing is that you're targeting, where you can't even specify what you're targeting to mitigate. So that's something that I'm currently skeptical of, and would love people to suggest otherwise.

**Seth:** Real quick, I would say it's not about civilian versus military, but about whether-

**Jade:** Good versus bad.

**Seth:** But I'm curious to see people's reactions to this.

**Jade:** Yes. Yeah.

**Audience Member:** Tangential, but coming back to the… you sort of were suggesting earlier the information asymmetry with national security is sitting very much on their side. That if they want the information, we're not keeping it from them. They're probably going to have. In a similar vein, do you think that in terms of the UN and the political machinery, that they're even necessarily going to have insight into what their own national security apparatus are working on, what the state of affairs is there? If that's sort of sitting in a separate part of the bureaucratic apparatus from the international agreements, how effective could that ever even be if you don't have that much interface between the two? Does that…

**Seth:** Essentially like, how can you monitor and enforce an agreement if you don't have access to the information that… with difficulty. This is a familiar problem, for example, with biological weapons. The technology there can also be used for vaccine development and things of that sort. It can cut both ways and a lot of it is dual-use, that's the catch phrase, and because of that, you have companies that have the right sort of equipment and they don't want other people knowing what they're doing because it's intellectual property. So the answer is with difficulty, and this is a challenge. The more we can be specific about what we need to monitor, the easier it becomes but that doesn't necessarily make it easy.

**Audience Member:** Something governments seem to hate is putting the brakes on anything that's like making them money, tax money. But something they seem to love is getting more control and oversight into corporations, especially if they think there's any sort of reputational risk or risk to them, and that the control and oversight is not going to pose any sort of economic slowdown in costs. Do you think there's a possibility of framing the message simply as, the countries should agree that non-state actors get to be spied on by states, and the states get some sort of oversight? And the states might all agree to that, even if the non-state actors don't like it very much. And the non-state actors might be okay if there was no… if it seemed like it was toothless at the start. So maybe if there was some sort of like slippery slope into government oversight to make things more safe that could be started with relatively low barrier.

**Jade:** Nice. I like the way you think. That's nice. Yeah, I think the short answer is yes. I think the major hurdle there is that firms will hate it. Firms, particularly multinational technology firms, that actually have a fair amount of sway in a number of different dimensions of sway, just won't be good with it and will threaten some things that states care about.

**Audience Member:** As someone who does AI research for a multinational firm, I really do actually feel a lot of friction when allowing certain sorts of code to cross national boundaries. So actually, I would like to say that state regulation is making more of an impact than you might realize, that there are certain sorts of things, especially around encryption protocols, where state agreements have made a big difference as to what can cross state boundaries, even with a lot of states not being in on the agreement. Just like the developed nations as of 30 years ago all agreeing, "Hey, we're going to keep the encryption to ourselves." Means that my coworkers in India don't get to see everything I get to work with because there's protocols in place. So, it does matter to international organizations, if you can get the laws passed in the first place.

**Jade:** Yeah, sure. Any other examples aside from encryption, out of curiosity? I know the encryption side of it relatively well but are there other-

**Seth:** Well, there's the privacy. My American nonprofit organization had to figure out if we needed to do anything to comply with Europe's new privacy law.

**Jade:** You sound very happy about that.

**Seth:** I say nothing. We are just about out of time, though, so maybe we should try to wrap up a little bit as far as take home messages. I feel like we did not fully answer the question of the extent to which engaging with national and international organizations is worth our time in the first place, to the question of like are these even the key actors? Superficially, noting we're basically out of time, I can say there are at least some reasons to believe they could end up being important actors and that I feel like it is worth at least some effort to engage with, though we should not put all our eggs in that basket, noting that other actors can be very important. Then, as far as how to pursue it, I would just say that we should try to do it cautiously and with skill, and by engaging very deeply and understanding the communities that we're working with.

**Jade:** I think the meta point maybe to point out as well is that these are very much… hopefully, illustratively, it's a very much alive debate on both of these questions. It's hard and there are a lot of strategic parameters that matter, and it's hard to figure out what the right strategy is moving forward and I hope you're not taking away that there are perspectives that are held strongly within this community. I hope you're mostly taking away that it's a hard set of questions that needs a lot more thought, but more so than anything it needs a lot more caution in terms of how we think about it because I think there are important things to consider. So, hopefully that's what you're taking away. If you're not, that should be what you're taking away. All right, thanks guys.

The role of existing institutions in AI strategy _ Jade Leung _ Seth Baum-by Centre for Effective Altruism-video_id pgiwvmY3brg-date 20181023.md


# Interview with AI Researchers individuallyselected_w5cb5 by Vael Gates

**Interview with w5cb5, on 3/18/22**
====================================

**0:00:02.2 Vael:** Alright, so my first question is, can you tell me about what area of AI you work on in a few sentences?

**0:00:08.5 Interviewee:** I worked in \[subfield\] originally, but I guess I branched out more broadly into AI research, because I\'m \[high-level research role\] now at an AI company.

**0:00:19.9 Vael:** Great, yeah. And then what are you most excited about in AI and what are you most worried about? In other words, what are the biggest benefits or risks of AI?

**0:00:28.3 Interviewee:** So I think, yeah, the world is going to change quite a lot with AI technology, and I think mostly in good ways, just because we\'re going to empower people with this technology. And it\'s going to be empowering I think in similar ways to the Internet, where people can do faster search, they have an assistant who can help them with all kinds of stuff. They have friends who maybe are not real, and all kinds of ways to make people happier, I think, or more efficient, or to give them time back and that sort of stuff. But obviously, there are also risks and the main risks are, I think that the field is too dominated by tech bros from Silicon Valley, so I guess I fall under that in a way. And so I think that\'s a real problem, so we need to take democratization of the technology much more seriously, that\'s also what my company is doing. And I think if we think about the ethical implications of our technology from first principles, and if we make them first-class citizens rather than just treating them as an afterthought, where you submit your paper and then, \"Oh, I also need to write a broader impact statement,\" but if you take that very seriously from the beginning as a core principle of your organization, then I think you can do much better research in a much more responsible way.

**0:01:56.5 Vael:** Interesting. Alright, so that was the question of \"what are you most excited about and what are you most worried about in AI\", okay. I heard\-- Lots of things they can go, lots of places they can go, lots of directions they can go, but you\'re worried about domination from specific areas and then people not caring about\... ethics enough? or---

**0:02:14.6 Interviewee:** Yeah, so misuse of technology. Do you want me to give you concrete examples? So I think very often, the technology that we develop, even if it\'s meant for benevolent purposes, can also be re-applied for not so benevolent purposes. And so like speech recognition or face recognition, things like that, you have to just be very careful with how you treat this technology. So that\'s why I think if people take responsible AI seriously from the beginning, that that is a good thing too.

**0:02:53.0 Vael:** Interesting. So you think if people incorporate responsible AI from the beginning of the process, then there will be less risk of misuse by any agent in the future?

**0:03:04.5 Interviewee:** Yeah, yeah. So you mentioned your IRB, so for a lot of technological research happening in industry, there is no real IRB. Some companies have sort of IRBs but most of them are so commercial and so driven by money in the end. And I think maybe we need an independent AI IRB for the broader research community, where anybody can go there and have somebody look at the potential applications of their work.

**0:03:39.6 Vael:** I see, cool. And then just having that sort of mindset seems good, in addition to the object-level effects. Alright. Makes sense. So focusing on future AI, putting on a science fiction forecasting hat, say we are 50 years, 50 plus years into the future. So at least 50 years into the future, what does that future look like?

**0:04:00.3 Interviewee:** At least 50 years in the future. So I still don\'t think we will have AGI, and that\'s I guess, I\'m probably unusual in the field because I think a lot of my colleagues would disagree, especially if they\'re at OpenAI or DeepMind because they think that it\'s like two years away. (Vael: \"Two years, huh!\") Yeah, well it depends on who you ask, they have some crazy people. \[chuckle\] I think in the next decade, we\'re going to realize what the limitations are of our current technology. I think what we\'ve been doing now has been very efficient in terms of scaling with data and scaling with compute, but it\'s very likely that we\'re just going to need entirely new algorithms that just require pure scientific breakthroughs. And so I don\'t think there\'s going to be another AI winter, but I do think that things are going to cool down a little bit again, because right now it\'s just been super hyped up. For good reason too, because we are really making really great progress. But there is still things that we really don\'t know how to do, so we have language models and they can do things and they\'re amazing, but we don\'t know how to make the language model do what we want it to do. So we\'re all just sort of hacking it a little bit, but it\'s not really anywhere close to being like a proper assistant, for example, who actually understands what you\'re saying, who actually understands the world. I think where we want to be 50 years from now is where we have machines who understand the world in the same way that humans understand it, so maybe something like Neuralink. So if I\'m being very futuristic, connecting AI to human brains and human perception of reality, that could be a way to get AI to have a much richer understanding of the world in the same way that humans understand it. So like dolphins are also very intelligent, but they also don\'t understand humans and they are not very useful assistants, right? I don\'t know if you\'ve ever had any dolphin assistant. So it\'s not really bad intelligence, it\'s specifically about human intelligence that makes AI potentially useful for us, and so that\'s something that I think is often overlooked.

**0:06:26.9 Vael:** So it sounds like, so you\'re thinking about when AGI will happen. And you said that you don\'t think we\'re gonna hit some sort of ceiling or slow down on the current deep learning paradigm or just like keep on scaling\--

**0:06:39.6 Interviewee:** Yeah, it\'s going to be asymptotic, and at some point, we\'re just going to hit the limits of what we can do with scaling data and scaling compute. And in order to get the next leap to real AGI I think we just need radically different ideas.

**0:06:55.1 Vael:** Yeah, when do you think we\'re going to\-- what kind of systems do you think we\'re going to have when we cap out on the current scaling paradigm?

**0:07:02.0 Interviewee:** Well, I think like the ones we have now, but yeah, in 50 years, I don\'t know. But in like 5 to 10 years, it will just be much bigger versions of this. And so what we have seen is that if you scale these systems, they generalize much better. If that keeps happening, then we would just have much better versions of what we have now. But still it\'s a language model that doesn\'t understand the world, and so still it\'s the component that is very limited in seeing only the training data that is in images on the internet, which is not all of the images that we have in the world, right? So I think the real problem is data, not so much scaling the compute.

**0:07:49.7 Vael:** What if we had a system that has cameras and can process auditory stuff that is happening all around it or something and it\'s not just using internet data, do you think that would eventually have enough data?

**0:08:03.3 Interviewee:** Yeah, so that\'s what I was just saying. If you have something that\'s embodied in the world in the same way as a human and where humans treat it as another human, sort of like cyborg style, things like that, that\'s a good way to get lots of very high quality data in the same way that humans get it. What are they called? Androids, right?

**0:08:24.9 Vael:** Yeah.

**0:08:25.3 Interviewee:** So if we actually had android robots walking around and being raised by humans and then we figured out how the learning algorithms would work in those settings, then you would get something that is very close to human intelligence. A good example I always like to use is the smell of coffee. So I know that you know what coffee smells like, but can you describe it to me in one sentence?

**0:08:54.2 Vael:** Probably not, no.

**0:08:55.7 Interviewee:** You can\'t, right? But the same goes for the taste of banana or things like that. I know that you know, so I\'ve never had to express this in words. So this is one of the fundamental parts of your brain; smell and taste are even older than sight and hearing. And so there\'s a lot of stuff happening in your brain that is just taken for granted. You can call this common sense or whatever you want, but it\'s like an evolutionary prior that all humans share with each other, and so that prior governs a lot of our behavior and a lot of our communication. So if you want machines to learn language but they don\'t have that prior, it becomes really, really hard for them to really understand what we\'re saying, right?

**0:09:38.7 Vael:** Yeah. I think when I think about AGI, I think about AGI that can do\-- or, just, generalizable systems that can do things that humans want them to do. So imagine we have like a CEO AI or a scientist AI. I don\'t think I need my CEO or scientist AI enough to know what coffee smells like per se, but I do need it to be able to like break down experiments and think kind of creative thoughts and figure out things.

**0:09:58.7 Interviewee:** Yeah, but I think what I\'m saying is that if they don\'t know what coffee smells like, that\'s just one example, but there are millions of these things that are just things we take for granted, that we don\'t really talk about. And so this will not be born out in the data in any way, so that means that a lot of the underlying assumptions are never really in the data, right? They\'re in our behavior, and so for an AI to pick up on those is going to be very difficult.

**0:10:27.6 Vael:** What if there were cameras everywhere, and it got to record everyone and process those?

**0:10:32.3 Interviewee:** Yeah, maybe. So the real question is, if you just throw infinite data at it, then will it work with current machine learning algorithms? Is I guess what you\'re asking, right? And so I don\'t know. I mean, I know that our learning algorithm is very different from a neural net, but I think if you look at it from a mathematical perspective, then gradient descent is probably more efficient than Hebbian learning anyway. So mathematically, it\'s definitely possible that if you have infinite data and infinite compute, then you can get something really amazing. Sure, we are the proof of that, right? So whether that also immediately makes it useful for us is a different question, I think.

**0:11:20.8 Vael:** Interesting. Yeah, I think I\'m trying to probe \"do we need something like embodied AI in order to get AGI\" or something. And then your last comment was like, whether that makes it useful for us. I\'m like, well, presumably we\'re going to\... feeding it a lot of data lets it do grounding, so like relationships between language and what actually exists in the world and how physics works. But presumably, we\'re going to be training them to do what we want, right? So that it will be useful to us?

**0:11:43.5 Interviewee:** Well, it depends, right? Can we do that? Probably the way they will learn this stuff is through self-supervised learning, not through us supervising them. We don\'t know how to specify reward signals and things like that anyway. I\'m not sure, if we actually are able to train up these huge systems that are actually intelligent through self-supervised learning, if they are then going to listen to us, right? Why would they?

**0:12:15.2 Vael:** Right. Okay, cool. Yeah, so this kind of leads right into my next question here. So imagine we\'re in the future and we\'ve got some AGIs and we\'ve got a CEO AI, and I\'m like, \"Okay, CEO AI, I want you to maximize profits and not run out of money and not try to exploit people and try to avoid side effects,\" and it seems like this would currently be extremely challenging for many reasons. But one is that we\'re not very good at taking human values and putting them\-- and like goals and preferences\-- and putting them in mathematical formulations that AI can currently work. And I worry that this is gonna happen in the future as well. So the question is: what do you think of the argument, \"Highly intelligence systems will fail to optimize exactly what their designers intended them to and this is dangerous\"?

**0:12:53 Interviewee:** Well, yeah. I agree with that. I don\'t think\... I think there are two separate questions here. So one you\'re asking about is the paperclip maximizer argument from Nick Bostrom. So like if you have a system and you tell it like \"you need to make as many paperclips as you possibly can\" then it\'s going to like destroy the earth to make as many paperclips as possible.

**0:13:15 Vael:** Well that would be doing maybe\-- oh, I see. Not quite what I intended. Yeah, all right.

**0:13:19.8 Interviewee:** Yeah, so\-- okay, so if that\'s not what the underlying question was, then\... We don\'t really\... I also think that we are\... some of us are fooling ourselves into believing that we know everything as humans and I think human values are changing all the time. I don\'t think we can capture correct human values. I don\'t think there is an absolute moral truth that we should all adhere to. I think that just morality itself is a very cultural concept. But I\'m \[interested in\] philosophy, so I\'m a bit different from most AI researchers, I guess. So I think that we could try to encode some very basic principles, so this is like Asimov\'s laws and things like that, but I don\'t think we can really go much further than that. And I think even in those cases, like you said, we don\'t know how to mathematically encode them in a way where you enforce whatever this dynamical system is that you\'re training, so a neural net, but then probably more complicated than the current neural nets\-- how do we impose a particular set of values? I don\'t think we know how to do that. I don\'t think there\'s a mathematical way to do that either actually, because it\'s all \[inaudible\]\--

**0:14:44.7 Vael:** Yeah, do you think we are eventually going to be able to?

**0:14:50.0 Interviewee:** So I think if you ask Yann LeCun or someone like that, he would say that probably, if we ever get to systems of this sort of level of intelligence, then they would be benevolent, because they\'re very smart and able to sort of understand how weak humans are.

**0:15:09.4 Vael:** Interesting. Yeah. So when I hear that argument, I\'m like, okay, it seems like Yann LeCun thinks that as you get more intelligent, you have morals that are very similar to humans, and this just kind of comes\--

**0:15:21.7 Interviewee:** No, not necessarily. No, but just better morals, right? So I think that the argument is sort of that if you look at human progress, then we\'ve also been getting better and better moral systems and a better understanding of what human values really matter. And like 100 years from now, probably everybody\'s gonna look back at us and say, \"They were eating meat. They were killing all these animals.\" So we are on the path of enlightenment. I don\'t know if I agree with this, but that\'s one way of saying it. And so a sign of an organism or a culture becoming more and more enlightened is also that you become more and more benevolent I think for others, but maybe that\'s a bit of a naive take.

**0:16:05.9 Vael:** Yeah. I think in my mind\-- certainly we have\-- well, actually, I don\'t know that we have the correlation that humans are getting smarter and also at the same rate, or, like\... Like humans are pretty smart. And we\'re getting better at IQ tests, but I don\'t know that we\'re vastly increasing our intelligence per se.

**0:16:20.4 Interviewee:** Yeah. That\'s for different reasons, right. Yeah.

**0:16:24.9 Vael:** Yeah. And meanwhile, we have, over\-- centuries, like not that many centuries, we\'ve been increasing our moral circle and putting in animals and people far away from us, etcetera. But I kind of think of the axes of intelligence and morality as kind of orthogonal, where if we have a system that is getting much smarter, I don\'t expect it to have\... I expect kind of a lot of human morality runs from evolutionary pressures and also coordination difficulties, such that you need to be able to not kill people, otherwise the species is gonna go extinct. And you know, there\'s a bunch of stuff that are kind of built into humans that I wouldn\'t expect to happen just natively with intelligence; where intelligence, I would think of something like\... the ability to solve problems well, to make multi-step plans, to think in the future, to take out correlations and figure out predictions, and I don\'t expect that to naively correlate with---

**0:17:19.9 Interviewee:** Yeah, so I think that\'s a very narrow definition of intelligence, and so I don\'t know if that definition of intelligence you have, if that actually is the most useful kind of intelligence for humans. So I think that in our society there is this concept where intelligence just means like mathematical reasoning capabilities almost, right? (Vael: \"Yeah.\") And that is a very, very narrow definition, and most of our intelligence is not that, right? (Vael: \"Yes.\") So for regimes to be useful to us\... so I think what you\'re talking about is sort of like this good old-fashioned AI concept of intelligence, where you have symbolic reasoners, and you\'re like\... you\'re very good at very fast symbol manipulation. And like, \"This is what computers are for.\" So we should just have super smart computers who can do the stuff that we don\'t want to do or can\'t do. It\'s possible that our intelligence is a direct consequence, not of our mathematical reasoning capabilities, but of something else, of our cultural interactions. So I definitely think if humans were not a multi-agent society, that we would not be nearly as intelligent. So a lot of our intelligence comes from sharing knowledge and communicating knowledge and having to abstract knowledge so that you can convey it to other agents and that sort of stuff.

**0:18:50.0 Vael:** Cool. Yeah. So when I think about how I define intelligence, I\'m like, \"What is the thing I care about?\" The thing I care about is how we develop AI. And I\'m like, \"How are we gonna develop AI?\" We\'re gonna develop it so that it completes economic incentives. So we want robots that do tasks that humans don\'t want to do. We want computers\--

**0:19:09.2 Interviewee:** Yeah. But is that AI or is that just machine learning? We\'re trying to have a\... like input-output black box, and we want that black box to be as optimal as possible for making money or whatever the goal is, right? So that\'s also a worry I have, is that a lot of people are conflating these different concepts. So artificial intelligence\...yeah, it depends on how you define it. Some people think of it more as like AGI. If you ask Yann again and all the old school deep learners, they would say, it used to be that they were explicitly not doing AI. So AI is like Simon and Newell and all that sort of stuff, so like pure symbol manipulation, symbolic AI. And pattern recognition is not AI. And now, since deep learning became very popular, some of the people were like, \"Oh yeah, this is AI now,\" but they used to be machine learning and not AI. So one thing is just like this black box. It can be anything and we just want to have the best possible black box for our particular problem mapping X to Y. And this could be any kind of problem, it could be like image recognition or whatever. In some cases, you want to have a symbolic approach, in other cases, you want to have a learning approach, it sort of just depends. So it\'s just software. Right? But in one case, the software is well defined, and in the other case, it\'s a bit fuzzier.

**0:20:37.9 Vael:** Yeah. So this all kind of depends on your frame, of course. I think my frame, or the reason why I care, is I\'m like, I think machine learning, AI, I don\'t know, whatever this thing is where humans are pouring a lot of investment and effort into making software better, and by better I mean better able to accomplish tasks that we want it to do\-- I think that this will be\-- it is very powerful, it has affected society a lot already and it will continue to affect society a lot. Such that like 50 years out, I expect this to be\... Whatever we developed to be very important in how\... Affect just a lot of things.

**0:21:10.8 Interviewee:** But we\'re notoriously bad at predicting the future, right? So if you asked in the \'60s, people would say like, there\'s flying cars, and like we\'re living on Mars and all that stuff. And we\'re getting a bit closer, but we\'re still not there yet. But none of these people would have seen the internet coming. And so I think maybe the next version of the internet is going to be more AI driven. So that is a sort of\... first use case that I would see for AI, which is like a better internet.

**0:21:50.0 Vael:** Interesting. Yeah, I think kind of\... people will find whatever economic niches will get them a lot of profit, is sort of how I expect things to continue to go, given that that seems to be \... Given that society works kind of the same way, and people have a lot of time and energy and have the capability to invest in this stuff, we will continue to develop machine learning, AI software, etcetera, such that it\--

**0:22:13.2 Interviewee:** We\'ve been doing that for like 30 years or even more. From the Perceptron, Rosenblatt. We\'ve been already doing this and so it\'s not really a question of like AI taking over the world, it\'s software taking over the world, and AI in some cases is better than like rule-based software. But it\'s still software taking over the world.

**0:22:35.8 Vael:** Yeah, yeah, certainly. And then the current paradigm of like, gigantic neural nets, seems to be better at doing things that we want it to do. And so we\'re continuing on in that direction, and at some point, as you say, it becomes less able to do what we want it to do, given the amount of resources that we\'re pouring into it, like that ratio trades off. Okay\--

**0:22:54.3 Interviewee:** Yeah. So there\'s other trade offs too, right? So as you become bigger as a neural net, you also become a lot more inefficient. This is already the case for something like GPT-3; latency is a big problem. For us to be able to talk like this to a machine, if the machine has 100 trillion parameters, it\'s going to be way too slow. It\'s going to take, I don\'t know, 10 minutes to generate an answer to a simple question. So it\'s not only a tradeoff of\... Best does not just mean accuracy. Best also is like, how efficient are you? How fair are you? How robust are you? How much environmental impact do you have? All of these different sort of metrics that all matter for choosing what defines \"best\" for a system. I think this is something we need to improve a lot on as a community, where we stop thinking beyond this pure accuracy thing, which is like an academic concept, to an actual\... like how can we deploy these systems in a responsible way, where we think about all the possible metrics that matter for deployment. So we want to be at the Pareto frontier of like 10 different metrics, not just accuracy.

**0:24:06.8 Vael:** Cool. Alright, that makes sense. So still thinking ahead in the future, do you think we\'ll ever get something like a CEO AI?

**0:24:14.0 Interviewee:** So, if\-- so a CEO AGI or a CEO AI?

**0:24:18.8 Vael:** Um, some sort of software system that can do the things that a CEO can do.

**0:24:25.6 Interviewee:** No.

**0:24:26.1 Vael:** No. Okay.

**0:24:28.6 Interviewee:** So not before we get AGI. So I think that is an AI complete problem. But I do think we\'ll get a very good CEO AI assistant. \[inaudible\] \...real human. It\'s like a plane, right? So like a plane is flown by a pilot but it\'s really flown by a computer. So I think the same could be true for a company where the company has like, a CEO pilot whose job is also to inspire people and do all of the human soft skills. And they have an assistant who does a lot of measurement stuff and tries to give advice for like where the company should be headed and things like that.

**0:25:05.1 Vael:** Okay, awesome. And you do think that you could have a CEO AGI, it sounds like.

**0:25:10.3 Interviewee:** Yeah, but if you have an AGI, then we don\'t need CEOs anymore.

**0:25:14.3 Vael:** What happens when we get AGI?

**0:25:16.9 Interviewee:** All the humans die.

**0:25:17.5 Vael:** All the humans die. Okay! \[laughs\]

**0:25:20.1 Interviewee:** \[laughs\] So I think it depends. I think actually the most likely scenario, as I said, for AGI to come into existence is when humans merge with AI. And so I don\'t think that it\'s a bad thing for AGI to emerge. So if there is an AGI, then it will be a beautiful thing, and we will have made it as a society. So yeah, if that thing takes over, then that thing is going to be insane, it\'s going to take over the universe, and then we will be sort of like the cute little people who made it happen. So either we become very redundant very quickly or we sort of merge with AI into this new species kind of.

**0:26:14.1 Vael:** Interesting, okay. And you don\'t necessarily see a connection between, like, the current\... \[you think\] if we just push really hard on the current machine learning paradigm for 50 years, we won\'t have an AGI. We need to do something different for an AGI, which sounds like embodiment / combination with humans, biological merging?

**0:26:31.7 Interviewee:** So it could be embodiment and combination with humans, but also just better, different learning algorithms. So probably more sparsity is something that scales better. More efficient learning. So the problem with gradient descent is that you need too much data for it. Maybe we need some like Bayesian things where we can very quickly update belief systems. But maybe that needs to happen at a symbolic level. I still think we have to fix symbolic processing happening on neural networks\-- so we\'re still very good at pattern recognition, and I think one of the things you see with things like GPT-3 is that humans are amazing at anthropomorphizing anything. I don\'t know if you\'ve ever read any Daniel Dennett, but what we do is we take an intentional stance towards things, and so we are ascribing intentionality even to inanimate objects. His theory is essentially that consciousness comes from that. So we are taking an intentional stance towards ourselves and thinking of ourselves as a rational agent and that loop is what consciousness is. But actually we\'re sort of biological machines who perceive their own actions and over time this became what we consider consciousness. So\... where was I going with this? \[laughs\] What was the question?

**0:27:57.2 Vael:** Yeah, okay. So I\'m like, alright, we\'ve got AI, we\'ve got lots of machine learning\--

**0:28:00.8 Interviewee:** \--oh yeah, so do you need new learning algorithms? Yeah. So I think what we need to solve is the sort of System 2, higher-level thinking and how to implement that on the neural net. The neural symbolic divide is still very much an open problem. There are lots of problems we need to solve, where I really don\'t think we can just easily solve them by scaling. And that\'s\-- like there is very little other research happening actually in field right now.

**0:28:35.3 Vael:** Alright. So say we do scaling, but we also have a bunch of software. Like algorithmic improvements at the rate we\'re seeing, and we\'ve got hardware improvements as well. I guess this is just more scaling, but we have optical, we have quantum computing. And then we have some sort of fast learning systems, we know how to do symbolic processing, we\'re much more efficient. Here we now have a system that generalizes very well and is pretty efficient, and I don\'t know, maybe we\'re hundred years out. Say maybe we\'re in a different paradigm, maybe we\'re kind of in the same paradigm. We now have a system that is\--

**0:29:05.5 Interviewee:** We would be in a different paradigm for sure.

**0:29:07.4 Vael:** Okay. We are in a different paradigm, because\... because all these learning algorithms\--?

**0:29:11.4 Interviewee:** Paradigms don\'t really last that long, if you look at the history of science.

**0:29:16.2 Vael:** Okay, cool. But are we still operating under like, here\'s software with faster learning algorithms, more efficient learning algorithms, like symbolic reasoning, Bayesian stuff\--

**0:29:24.7 Interviewee:** Maybe. But I mean it could be that neuromorphic hardware finally lives up to its promise, or that we can do photonic chips at the speed of light computation and things like that. We\'re also very good in AI at fooling ourselves into thinking that we are responsible for all of these amazing breakthroughs, but without hardware engineers at NVIDIA, none of this stuff would have happened, right? They are doing very different things.

**0:29:55.1 Vael:** Alright, so we\'ve got this AI system which is quite general, we\'re in maybe a different paradigm, but we\'re still like\-- faster learning systems. Here we are, these things are very capable, very general, when they generate stories, they model physics in the world and then use that to generate their stories. Maybe they can do a lot of social stuff, maybe they know how to interact with people. And here we are with our system. Is this now an AGI?

**0:30:18.0 Interviewee:** No, no, so\-- Okay, now I remember what I was gonna say about the Dennett thing. So we anthropomorphize everything, we take this intentional stance at everything. We do this to ourselves, we do this to everything, especially when it speaks language. So when we see a language model and it\'s like, \"whoa, it\'s amazing, it does this thing,\" but all it\'s really doing is negative log likelihood, maximum likelihood estimation. It\'s basically just trying to fit \"what is the most likely word to go here\". So you can ask yourself whether we are so impressed by this system because it\'s so amazing, or because we are sort of programmed to have a lot of respect for things that speak language, because things that speak language tend to be humans. What you were just saying made it sound like you were saying, when these systems are sort of like humans, when they can do this and when they do that, and when they understand the world. So how do you define \"understanding the world\" there\--

**0:31:18.7 Vael:** I mostly mean like they could sub in for human jobs, for example\--

**0:31:25.0 Interviewee:** Yeah, but that\'s not the same thing as\-- stepping in for a human, they can already do that. But it depends on the problem. They\'re very good at counting, but\--

**0:31:34.5 Vael:** Yeah, but I don\'t think we could have like a mathematician AI right now per se. I guess I forgot to define my interpretation of AGI, but like a system that is very capable of replacing all current human day jobs.

**0:31:51.6 Interviewee:** Including yours and mine?

**0:31:55.9 Interviewee:** Yup.

**0:31:57.8 Interviewee:** Okay. But then who would it be useful for? Would the president still have a job or not?

**0:32:09.7 Vael:** Uh\... It doesn\'t have to. I think you could just spend\-- humans wouldn\'t have to work anymore, for example, and they could just go around doing whatever they do.

**0:32:16.7 Interviewee:** Yeah. But that\'s not at all what humans do. We\'re all so programmed to compete with each other.

**0:32:24.7 Vael:** Yeah, we can have games, we can have competitions, we can do all sorts of things, we have sports.

**0:32:29.1 Interviewee:** I think it\'s gonna be very quickly my AI versus your AI, basically.

**0:32:33.9 Vael:** Okay, we can have big fights with AIs, that seems very dangerous.

**0:32:37.3 Interviewee:** Yeah, I know, yeah. So that is a more likely scenario, I think, than everybody being nice and friendly and playing games. (Vael: \"Yeah.\") If people want to have power, and whoever controls the AGI will have the most power, (Vael: \"That seems right,\") then I think we\'re going to be developing your own AGIs at the same time. And then those AGIs at some point are going to be fighting with each other.

**0:33:02.0 Vael:** Yeah, yeah, I think we might even get problems before that, where we\'re not able to get AIs aligned with us. Have you heard of AI alignment?

**0:33:10.9 Interviewee:** Yeah, so \[close professional relationship\] wrote a nice thesis about it. \[Name\], I don\'t know if you know \[them\] by any chance. So yeah, alignment is important, but my concern with all this alignment stuff is that it\'s very ill-defined, I think. Either it means the same as correctness, so is your system just correct, or good at what you want it to be good at\... alignment is sort of like a reinvention of just correctness. I can see why this is useful for some people to put a new name on it. But I think it\'s a very old concept where it\'s just, okay, we\'re measuring things on a very narrow static test set, but we should be thinking about all these other things. You want your system to be really good when you deploy it in the real world. So it needs to be a good system or a correct or an aligned system. And so alignment maybe is a useful concept, only in the sense that the systems are getting so good now that you can start thinking about different kinds of goodness that we didn\'t think about before, and we can call that alignment, like human value-style things. But I think the concept itself is very old; it\'s just like, is your system correct?

**0:34:40.0 Vael:** Yeah. And then it\'s nowadays being thought about in terms of very far future systems and aligning with all values and preferences. (Interviewee: Yeah.) Cool. Yeah, do you work on any sort of AI safety or what would convince you to work on this or not work on this, etcetera?

**0:34:56.5 Interviewee:** Yeah so, I\'m not sure. AI safety is a bit of a weird concept to me, but I do work on the responsible AI and ethical AI, yeah.

**0:35:06.9 Vael:** Hm. And what does that mean\--

**0:35:09.1 Interviewee:** So these are things like\... I\'m trying to get better fairness metrics for systems. So in \[company\] we built this provisional fairness metric where we do some heuristic swaps. And so right now we\'re working on a more sophisticated method for doing this where, let\'s say, you have something, a sentence or some sort of natural language inference example, so a premise and a hypothesis and it\'s about James, like if you change James to Jamal, that shouldn\'t change your prediction at all. Or if you change the gender from James and you turn into a woman, that shouldn\'t change anything there. And it does, actually, if you look at restaurant reviews, if you changed the restaurant to a Mexican restaurant and the person who\'s eating there to Jamal, then your sentiment goes down. So this is the sort of stuff that shouldn\'t happen in these systems that is direct consequence of us just scaling the hell out of our systems on as much data as we can, including all of the biases that exist in this data. So I\'m working on trying to do that better measurement for these sort of things. And so I think if we are not getting better at measurement, then all of this stuff is basically a pointless discussion.

**0:36:29.1 Vael:** Great, thank you. And then my last question is, have you changed your mind on anything during this interview and how was this interview for you?

**0:36:35.9 Interviewee:** It was fun. Yeah, I\'ve done a few of these with various people and it\'s always a bit like, I don\'t know. It feels a bit like\... we\'re getting ahead of ourselves a little bit. But maybe I\'m also just old. So when I talked to \[close professional relationship\] and how \[they\] think about stuff, I\'m like, I just don\'t understand how \[they\] think about AI.

**0:37:06.2 Vael:** Got it. \[They\'re\] like way out here, and we need to make sure that systems do our correct\--

**0:37:11.9 Interviewee:** Yeah, \[they\'re\] really.. Yeah, \[they\] put a lot more faith also in AI, which I think is very interesting. So I asked \[them\] like, \"Okay, so this alignment stuff, in the end who should we ask what is right or what is wrong? When we\'re trying to design the best AI systems, who should we ask for what\'s right and wrong?\" And then \[their\] answer was, \"We should ask the AI.\"

**0:37:38.7 Vael:** What? No, we should ask humans.

**0:37:41.0 Interviewee:** Yeah, no, so \[they\] think that basically AGI or AI is going to get so good, these language models are gonna get so good that they can tell us how we should think about our own moral philosophical values so that we can impose them onto AI systems. That to me just sounds crazy, like batshit crazy, but that\'s one way to think about it. I mean, I respect \[their\] opinion. I just can\'t understand it.

**0:38:11.7 Vael:** Interesting. Yeah, I think if I try to model what I would imagine \[they\] would be saying, under the alignment paradigm, I would say that you need to ask human feedback, but it\'s hard to get human feedback on very intelligent systems. And so you should ask AI to summarize human feedback, but it should always be ground down on a human otherwise we\'re in trouble, so.

\[ending comments\]

individuallyselected_w5cb5-by Vael Gates-date 20220318.md