# Eric Drexler Reframing Superintelligence - EA Forum

_When people first began to discuss advanced artificial intelligence, existing AI was rudimentary at best, and we had to reply on ideas about human thinking and extrapolate. Now, however, we've developed many different advanced AI systems, some of which outperform human thinking on certain tasks. In this talk from EA Global 2018: London, Eric Drexler argues that we should use this new data to rethink our models for how superintelligent AI is likely to emerge and function._

## The Talk

I've been working in this area for quite a while. The chairman of my doctoral committee was one Marvin Minsky. We had some discussions on AI safety around 1990. He said I should write them up. I finally got around to writing up some developed versions of those ideas just very recently, so that's some fairly serious procrastination. Decades of procrastination on something important.

For years, one couldn't talk about advanced AI. One could talk about nanotechnology. Now it's the other way around. You can talk about advanced AI, but not about advanced nanotechnology. So this is how the Overton window moves around.

What I would like to do is to give a very brief presentation which is pretty closely aligned with talks I've given at OpenAI, DeepMind, FHI, and Bay Area Rationalists. Usually I give this presentation to a somewhat smaller number of people, and structure it more around discussion. But what I would like to do, still, is to give a short talk, put up points for discussion, and encourage something between Q&A and discussion points from the audience.

Okay so, when I say "Reframing Superintelligence," what I mean is thinking about the context of emerging AI technologies as a process rolling forward from what we see today. And asking, "What does that say about likely paths forward?" Such that whatever it is that you're imagining needs to emerge from that context or make sense in that context. Which I think reframes a lot of the classic questions. Most of the questions don't go away, but the context in which they arise, the tools available for addressing problems, look different. That's what we'll be getting into.

Once upon a time, when we thought about advanced AI, we didn't really know what AI systems were likely to look like. It was very unknown. People thought in terms of developments in logic and other kinds of machine learning, different from the deep learning that we now see moving forward with astounding speed. And people reached for an abstract model of intelligent systems. And what intelligent systems do we know? Well, actors in the world like ourselves. We abstract from that very heavily and you end up with rational, utility-directed agents.

Today, however, we have another source of information beyond that abstract reasoning, which applies to a certain class of systems. And information that we have comes from the world around us. We can look at what's actually happening now, and how AI systems are developing. And so we can ask questions like, "Where do AI systems come from?" Well, today they come from research and development processes. We can ask, "What do AI systems do today?" Well, broadly speaking, they perform tasks. Which I think of, or will describe, as "performing services." They do some approximation or they do something that someone supposedly wants in bounded time with bounded resources. What will they be able to do? Well, if we take AI seriously, AI systems will be able to automate asymptotically all human tasks, and more, at a piecemeal and asymptotically general superintelligent level. So we said AI systems come from research and development. Well, what is research and development? Well, it's a bunch of tasks to automate. And, in particular, they're relatively narrow technical tasks which are, I think, uncontroversially automate-able on the path to advanced AI.

So the picture is of AI development moving forward broadly along the lines that we're seeing. Higher-level capabilities. More and more automation of the AI R&D process itself, which is an ongoing process that's moving quite rapidly. AI-enabled automation and also classical software techniques for automating AI research and development. And that, of course, leads to acceleration. Where does that lead? It leads to something like recursive improvement, but not the classic recursive improvement of an agent that is striving to be a more intelligent, more capable agent. But, instead, recursive improvement where an AI technology base is being advanced at AI speed. And that's a development that can happen incrementally. We see it happening now as we take steps toward advanced AI that is applicable to increasingly general and fast learning. Well, those are techniques that will inevitably be folded into the ongoing AI R&D process. Developers, given some advance in algorithms and learning techniques, and a conceptualization of how to address more and more general tasks, will pounce on those, and incorporate them into a broader and broader range of AI services.

So where that leads is to asymptotically comprehensive AI services. Which, crucially, includes the service of developing new services. So increasingly capable, increasingly broad, increasingly piecemeal and comprehensively superintelligent systems that can work with people, and interact with people in many different ways to provide the service of developing new services. And that's a kind of generality. That is a general kind of artificial intelligence. So a key point here is that the C in CAIS, C in Comprehensive AI Services does the work of the G in AGI. Why is it a different term? To avoid the implication... when people say AGI they mean AGI agent. And we can discuss the role of agents in the context of this picture. But I think it's clear that a technology base is not inherently in itself an agent. In this picture agents are not central, they are products. They are useful products of diverse kinds for providing diverse services. And so with that, I would like to (as I said, the formal part here will be short) point to a set of topics.

They kind of break into two categories. One is about short paths to superintelligence, and I'll argue that this is the short path. The topic of AI services and agents, including agent services, versus the concept of "The AI" which looms very large in people's concepts of future AI. I think we should look at that a little bit more closely. Superintelligence as something distinct from agents, superintelligent non-agents. And the distinction between general learning and universal competence. People have, I think, misconstrued what intelligence means and I'll take a moment on that. If you look at definitions of good from the 1960s, ultra-intelligence and more recent Bostrom and so on (I work across the hall from Nick) on superintelligence the definition is something like "a system able to outperform any person in any task whatsoever." Well, that implies general competence, at least as ordinarily read. But there's some ambiguity over what we mean by the word "intelligence" more generally. We call children intelligent and we call senior experts intelligent. We call a child intelligent because the child can learn, not because the child can perform at a high level in any particular area. And we call an expert who can perform at a high level intelligent not because the expert can learn - in principle you could turn off learning capacity in the brain - but because the expert can solve difficult problems at a high level.

So learning and competence are dissociable components of intelligence. They are in fact quite distinct in machine learning. There is a learning process and then there is an application of the software. And when you see discussion of intelligent systems that does not distinguish between learning and practice, and treats action as entailing learning directly, there's a confusion there. There's a confusion about what intelligence means and that's, I think, very fundamental. In any event, looking toward safety-related concerns, there are things to be said about predictive models of human concerns. AI-enabled solutions to AI-control problems. How this reframes questions of technical AI safety. Issues of services versus addiction, addictive services and adversarial services. Services include services you don't want. Taking superintelligent services seriously. And a question of whether faster development is better.

And, with that, I would like to open for questions, discussion, comment. I would like to have people come away with some shared sense of what the questions and comments are. Some common knowledge of thinking in this community in the context of thinking about questions this way.

## Discussion

_Question_: Is your model compatible with end-to-end reinforcement learning?

_Eric_: Yes.

To say a little bit more. By the way, I've been working on a collection of documents for the last two years. It's now very large, and it will be an FHI technical report soon. It's 30,000 words structured to be very skim-able. Top-down, hierarchical, declarative sentences expanding into longer ones, expanding into summaries, expanding into fine-grained topical discussion. So you can sort of look at the top level and say, hopefully, "Yes, yes, yes, yes, yes. What about this?" And not have to read anything like 30,000 words. So, what I would say is that reinforcement learning is a technique for AI system development. You have a reinforcement learning system. It produces through a reinforcement learning process, which is a way of manipulating the learning of behaviors. It produces systems that are shaped by that mechanism. So it's a development mechanism for producing systems that provide some service. Now if you turn reinforcement learning loose in the world open-ended, read-write access to the internet, a money-maximizer and did not have checks in place against that? There are some nasty scenarios. So basically it's a development technique, but could also be turned loose to produce some real problems. "Creative systems trying to manipulate the world in bad ways" scenarios are another sector of reinforcement learning. So not a problem per se, but one can create problems using that technique.

_Question_: What does asymptotic improvement of AI services mean?

_Eric_: I think I'm abusing the term asymptotic. What I mean is increasing scope and increasing level of capability in any particular task to some arbitrary limit. Comprehensive is sort of like saying infinite, but moving toward comprehensive and superintelligent level services. What it's intended to say is, ongoing process going that direction. If someone has a better word than asymptotic to describe that I'd be very happy.

_Question_: Can the tech giants like Facebook and Google be trusted to get alignment right?

_Eric_: Google more than Facebook. We have that differential. I think that questions of alignment look different here. I think more in terms of questions of application. What are the people who wield AI capabilities trying to accomplish? So there's a picture which, just background to the framing of that question, and a lot of these questions I think I'll be stepping back and asking about framing. As you might think from the title of the talk. So picture a rising set of AI capabilities: image recognition, language understanding, planning, tactical management in battle, strategic planning for patterns of action in the world to accomplish some goals in the world. Rising levels of capability in those tasks. Those capabilities could be exploited by human decision makers or could, in principle, be exploited by a very high-level AI system. I think we should be focusing more, not exclusively, but more on human decision makers using those capabilities than on high-level AI systems. In part because human decision makers, I think, are going to have broad strategic understanding more rapidly. They'll know how to get away with things without falling afoul of what nobody had seen before, which is intelligence agencies watching and seeing what you're doing. It's very hard for a reinforcement learner to learn that kind of thing.

So I tend to worry about not the organizations making aligned AI so much as whether the organizations themselves are aligned with general goals.

_Question_: Could you describe the path to superintelligent services with current technology, using more concrete examples?

_Eric_: Well, we have a lot of piecemeal examples of superintelligence. AlphaZero is superintelligent in the narrow domain of Go. There are systems that outperform human beings in playing these very different kinds of games, like Atari games. Face recognition recently surpassed human ability to map from human speech to transcriptive words. Just more and more areas piecemeal. A key area that I find impressive and important is the design of neural networks at the core of modern deep learning systems. The design of and learning to use appropriately, hyperparameters. So, as of a couple of years ago, if you wanted a new neural network, a convolutional network for vision, or some recurrent network, though recently they're going for convolution networks for language understanding and translation, that was a hand-crafted process. You had human judgment and people were building these networks. A couple of years ago people started in these, this is not AI in general but it's a chunk that a lot of attention went into, getting superhuman performance in neural networks by automated, AI-flavored like, for example, reinforcement learning systems. So developing reinforcement learning systems that learn to put together the building blocks to make a network that outperforms human designers in that process. So we now have AI systems that are designing a core part of AI systems at a superhuman level. And this is not revolutionizing the world, but that threshold has been crossed in that area.

And, similarly, automation of another labor-intensive task that I was told very recently by a senior person at DeepMind would require human judgment. And my response was, "Do you take AI seriously or not?" And, out of DeepMind itself, there was then a paper that showed how to outperform human beings in hyperparameter selection. So those are a few examples. And the way one gets to an accelerating path is to have more and more, faster and faster implementation of human insights into AI architectures, training methods, and so on. Less and less human labor required. Higher and higher level human insights being turned into application throughout the existing pool of resources. And, eventually, fewer and fewer human insights being necessary.

_Question_: So what are the consequences of this reframing of superintelligence for technical AI safety research?

_Eric_: Well, re-contexting. If in fact one can have superintelligent systems that are not inherently dangerous, then one can ask how one can leverage high-level AI. So a lot of the classic scenarios of misaligned powerful AI involve AI systems that are taking actions that are blatantly undesirable. And, as Shane Legg said when I was presenting this at DeepMind last Fall, "There's an assumption that we have superintelligence without common sense." And that's a little strange. So Stuart Russell has pointed out that machines can learn not only from experience, but from reading. And, one can add, watching video and interacting with people and through questions and answers in parallel over the internet. And we see in AI that a major class of systems is predictive models. Given some input you predict what the next thing will be. In this case, given a description of a situation or an action, you try to predict what people will think of it. Is it something that they care about or not? And, if they do care about it, is there widespread consensus that that would be a bad result? Widespread consensus that it would be a good result? Or strongly mixed opinion?

Note that this is a predictive model trained on many examples, it's not an agent. That is an oracle that, in principle, could operate with reasoning behind the prediction. That could in principle operate at a super intelligent level, and would have common sense about what people care about. Now think about having AI systems that you intend to be aligned with human concerns where, available for a system that's planning action, is this oracle. It can say, "Well, if such and such happened, what would people think of it?" And you'd have a very high-quality response. That's a resource that I think one should take account of in technical AI safety. We're very unlikely to get high-level AI without having this kind of resource. People are very interested in predicting human desires and concerns if only because they want to sell you products or brainwash you in politics or something. And that's the same underlying AI technology base. So I would expect that we will have predictive models of human concerns. That's an example of a resource that would reframe some important aspects of technical AI safety.

_Question_: So, making AI services more general and powerful involves giving them higher-level goals. At what point of complexity and generality do these services then become agents?

_Eric_: Well, many services are agent-services. A chronic question that arises, people will be at FHI or DeepMind and someone will say, "Well, what is an agent anyway?" And everybody will say, "Well, there is no sharp definition. But over here we're talking about agents and over here we're clearly not talking about agents." So I would be inclined to say that if a system is best thought of as directed toward goals and it's doing some kind of planning and interacting with the world I'm inclined to call it an agent. And, by that definition, there are many, many services we want, starting with autonomous vehicles, autonomous cars and such, that are agents. They have to make decisions and plan. So there's a spectrum from there up to higher and higher level abilities to do means-ends analysis and planning and to implement actions. So let's imagine that your goal is to have a system that is useful in military action and you would like to have the ability to execute tactics with AI speed and flexibility and intelligence, and have strategic plans for using those tactics that are superintelligent level.

Well, those are all services. They're doing something in bounded time with bounded resources. And, I would argue, that that set of systems would include many systems that we would call agents but they would be pursuing bounded tasks with bounded goals. But the higher levels of planning would naturally be structured as systems that would give options to the top level decision makers. These decision makers would not want to give up their power, they don't want a system guessing what they want. At a strategic level they have a chance to select, since strategy unfolds relatively slowly. So there would be opportunities to say, "Well, don't guess, but here's the trade off I'm willing to make between having this kind of impact on opposition forces with this kind of lethality to civilians and this kind of impact on international opinion. I would like options that show me different trade-offs. All very high quality but within that trade-off space. And here I'm deliberately choosing an example which is about AI resources being used for projecting power in the world. I think that's a challenging case, so it's a good place to go.

I'd like to say just a little bit about the opposite end, briefly. Superintelligent non-agents. Here's what I think is a good paradigmatic example of superintelligence and non-agency. Right now we have systems that do natural language translation. You put in sentences or, if you had a somewhat smarter system that dealt with more context, books, and out comes text in a different language. Well, I would like to have systems that know a lot to do that. You do better translations if you understand more about history, chemistry if it's a chemistry book, human motivations. Just, you'd like to have a system that knows everything about the world and everything about human beings to give better quality translations. But what is the system? Well, it's a product of R&D and it is a mathematical function of type character string to character string. You put in a character string, things happen, and out comes a translation. You do this again and again and again. Is that an agent? I think not. Is it operating at a superintelligent level with general knowledge of the world? Yes. So I think that one's conceptual model of what high-level AI is about should have room in it for that system and for many systems that are analogous.

_Question_: Would a system service that combines general learning with universal competence not be more useful or competitive than a system that displays either alone? So does this not suggest that agents might be more useful?

_Eric_: Well, as I said, agents are great. The question is what kind and for what scope. So, as I was saying, distinguishing between general learning and universal competence is an important distinction. I think it is very plausible that we will have general learning algorithms. And general learning algorithms may be algorithms that are very good at selecting algorithms that are good at selecting algorithms for learning a particular task and inventing new algorithms. Now, given an algorithm for learning, there's a question of what you're training it to do. What information? What competencies are being developed? And I think that the concept of a system being trained on and learning about everything in the world with some objective function, I don't think that's a coherent idea. Let's say you have a reinforcement learner. You're reinforcing the system to do what? Here's the world and it's supposed to be getting competence in organic chemistry and ancient Greek and, I don't know, control of the motion of tennis-playing robots and on and on and on and on. What's the reward function, and why do we think of that as one task?

I don't think we think of it as one task. I think we think of it as a bunch of tasks which we can construe as services. Including the service of interacting with you, learning what you want, nuances. What you are assumed to want, what you're assumed not to want as a person. More about your life and experience. And very good at interpreting your gestures. And it can go out in the world and, subject to constraints of law and consulting an oracle on what other people are likely to object to, implement plans that serve your purposes. And if the actions are important and have a lot of impact, within the law presumably, what you want is for that system to give you options before the system goes out and takes action. And some of those actions would involve what are clearly agents. So that's the picture I would like to paint that I think reframes the context of that question.

_Question_: So on that is it fair to say that the value-alignment problem still exists within your framework? Since, in order to train a model to build an agent that is aligned with our values, we must still specify our values.

_Eric_: Well, what do you mean by, "train an agent to be aligned with our values." See, the classic picture says you have "The AI" and "The AI" gets to decide what the future of the universe looks like and it had better understand what we want or would want or should want or something like that. And then we're off into deep philosophy. And my card says philosophy on it, so I guess I'm officially a philosopher or something according to Oxford. I was a little surprised. "It says philosophy on it. Cool!" I do what I think of as philosophy. So, in a services model, the question would instead be, "What do you want to do?" Give me some task that is completed in bounded time with bounded resources and we could consider how to avoid making plans that stupidly cause damage that I don't want. Plans that, by default, automatically do what I could be assumed to want. And that pursue goals in some creative way that is bounded, in the sense that it's not about reshaping the world; other forces would presumably try to stop you. And I'm not quite sure what value alignment means in that context. I think it's something much more narrow and particular.

By the way, if you think of an AI system that takes over the world, keep in mind that a sub-task of that, part of that task, is to overthrow the government of China. And, presumably, to succeed the first time because otherwise they're going to come after you if you made a credible attempt. And that's in the presence of unknown surveillance capabilities and unknown AI that China has. So you have a system and it might formulate plans to try to take over the world, well, I think an intelligent system wouldn't recommend that because it's a bad idea. Very risky. Very unlikely to succeed. Not an objective that an intelligent system would suggest or attempt to pursue. So you're in a very small part of a scenario space where that attempt is made by a high-level AI system. And it's a very small part of scenario space because it's an even smaller part of scenario space where there is substantial success. I think it's worth thinking about this. I think it's worth worrying about it. But it's not the dominant concern. It's a concern in a framework where I think we're facing an explosive growth of capabilities that can amplify many different purposes, including the purposes of bad actors. And we're seeing that already and that's what scares me.

_Question_: So I guess, in that vein, could the superintelligent services be used to take over the world by a state actor? Just the services?

_Eric_: Well, you know, services include tactical execution of plans and strategic planning. So could there be a way for a state actor to do that using AI systems in the context of other actors with, presumably, a comparable level of technology? Maybe so. It's obviously a very risky thing to do. One aspect of powerful AI is an enormous expansion of productive capacity. Partly through, for example, high-level, high quality automation. More realistically, physics-limited production technology, which is outside today's sphere of discourse or Overton window.

Security systems, I will assert, could someday be both benign and effective, and therefore stabilizing. So the argument is that, eventually it will be visibly the case that we'll have superintelligent level, very broad AI, enormous productive capacity, and the ability to have strategic stability, if we take the right measures beforehand to develop appropriate systems, or to be prepared to do that, and to have aligned goals among many actors. So if we distribute the much higher productive capacity well, we can have an approximately strongly Pareto-preferred world, a world that looks pretty damn good to pretty much everyone.

_Note: for a more thorough presentation on this topic, see Eric Drexler's_ [_other talk_](https://www.effectivealtruism.org/articles/ea-global-2018-paretotopian-goal-alignment/) _from this same conference._

_Question_: What do you think the greatest AI threat to society in the next 10, 20 years would be?

_Eric_: I think the greatest threat is instability. Sort of either organic instability from AI technologies being diffused and having more and more of the economic relationships and other information-flow relationships among people be transformed in directions that increase entropy, generate conflict, destabilize political institutions. Who knows? If you had the internet and people were putting out propaganda that was AI-enabled, it's conceivable that you could move elections in crazy directions in the interest of either good actors or bad actors. Well, which will that be? I think we will see efforts made to do that. What kinds of counter-pressures could be applied to bad actors using linguistically politically-competent AI systems to do messaging? And, of course, there's the perennial states engaging in an arms race which could tip into some unstable situation and lead to a war. Including the long-postponed nuclear war that people are waiting for and might, in fact, turn up some day. And so I primarily worry about instability. Some of the modes of instability are because some actor decides to do something like turn loose a competent hacking, reinforcement-learning system that goes out there and does horrible things to global computational infrastructure that either do or don't serve the intentions of the parties that released it. But take a world that's increasingly dependent on computational infrastructure and just slice through that, in some horribly destabilizing way. So those are some of the scenarios I worry about most.

_Question_: And then maybe longer term than 10, 20 years? If the world isn't over by then?

_Eric_: Well, I think all of our thinking should be conditioned on that. If one is thinking about the longer term, one should assume that we are going to have superintelligent-level general AI capabilities. Let's define that as the longer term in this context. And, if we're concerned with what to do with them, that means that we've gotten through the process to there then. So there's two questions. One is, "What do we need to do to survive or have an outcome that's a workable context for solving more problems?" And the other one is what to do. So, if we're concerned with what to do, we need to assume solutions to the preceding problems. And that means high-level superintelligent services. That probably means mechanisms for stabilizing competition. There's a domain there that involves turning surveillance into something that's actually attractive and benign. And the problems downstream, therefore, one hopes to have largely solved. At least the classic large problems and now problems that arise are problems of, "What is the world about anyway?" We're human beings in a world of superintelligent systems. Is trans-humanism in this direction? Uploading in this direction? Developing moral patients, superintelligent-level entities that really aren't just services, and are instead the moral equivalent of people? What do you do with the cosmos? It's an enormously complex problem. And, from the point of view of having good outcomes, what can I say? There are problems.

_Question_: So what can we do to improve diversity in the AI sector? And what are the likely risks of not doing so?

_Eric_: Well, I don't know. My sense is that what is most important is having the interests of a wide range of groups be well represented. To some extent, obviously, that's helped if you have in the development process, in the corporations people who have these diverse concerns. To some extent it's a matter of politics regulation, cultural norms, and so on. I think that's a direction we need to push in. To put this in the Paretotopian framework, your aim is to have objectives, goals that really are aligned, so, possible futures that are strongly goal-aligning for many different groups. For many of those groups, we won't fully understand them from a distance. So we need to have some joint process that produces an integrated, adjusted picture of, for example, how do we have EAs be happy and have billionaires maintain their relative position? Because if you don't do that they're going to maybe oppose what you're doing, and the point is to avoid serious opposition. And also have the government of China be happy. And I would like to see the poor in rural Africa be much better off, too. Billionaires might be way up here, competing not to build orbital vehicles but instead starships. And the poor in rural Africa of today merely have orbital space capabilities convenient for families, because they're poor. Nearly everyone much, much better off.

Reframing superintelligence _ Eric Drexler _ EA Global - London 2018-by Centre for Effective Altruism-video_id MircoV5LKvg-date 20190314.md


# Irina Rish on Out-of-distribution generalization by Jeremie Harris on the Towards Data Science Podcast

During training, AIs will often learn to make predictions based on features that are easy to learn, but deceptive.

Imagine, for example, an AI that’s trained to identify cows in images. Ideally, we’d want it to learn to detect cows based on their shape and colour. But what if the cow pictures we put in the training dataset always show cows standing on grass?

In that case, we have a _spurious correlation_ between grass and cows, and if we’re not careful, our AI might learn to become a grass detector rather than a cow detector. Even worse, we could only realize that’s happened once we’ve deployed it in the real world and it runs into a cow that isn’t standing on grass for the first time.

So how do you build AI systems that can learn robust, general concepts that remain valid outside the context of their training data?

That’s the problem of out-of-distribution generalization, and it’s a central part of the research agenda of Irina Rish, a core member of the Mila— Quebec AI Research institute, and the Canadian Excellence Research Chair in Autonomous AI. Irina’s research explores many different strategies that aim to overcome the out-of-distribution problem, from empirical AI scaling efforts to more theoretical work, and she joined me to talk about just that on this episode of the podcast.

Here were some of my favourite take-homes from the conversation:

-   To Irina, GPT-3 was an “AlexNet moment” for AI alignment and AI safety research. For the first time, we had built highly capable AIs without actually understanding their behaviour, or knowing how to steer it. As a result, Irina thinks that this is a great time to get into AI alignment research.
-   Irina thinks that out-of-distribution generalization is an area where AI capabilities research starts to merge with AI alignment and AI safety research. Getting systems to learn robust concepts is not only helpful for ensuring that they have rich representations of the world (which helps with capabilities), but also helps ensure that accidents don’t happen by tackling the problem of spurious correlations.
-   Irina has researched several strategies aimed at addressing the out-of-distribution sampling problem. One of them involves using the invariance principle: the idea that the features we want our AI models to learn from are going to be consistent (invariant) regardless of the environments our data come from. Consider for example the case of cow detection I mentioned earlier: the features we want our AI to lock onto (the shape and colour of cows, for example) are consistent across different environments. A cow is still a cow whether it’s in a pasture, indoors or in the middle of a desert. Irina is exploring techniques that allow AIs to distinguish between features that are invariant and desirable (like cow shape and colour) and features that are variable and unreliable predictors (like whether or not there’s grass on the ground).
-   Another approach Irina sees as promising is scaling. We’ve talked about scaling on the podcast before — but in a nutshell, it’s the idea that current deep learning techniques can more or less get us to AGI as-is, if only they’re used to train large enough neural networks with huge enough datasets, and an equally massive quantity of compute power. In principle, scaling certain kinds of neural nets could allow AIs to learn so much about their training data that their performance is limited only by the irreducible noise of the data itself.
-   That possibility raises another question: is there too much noise in language data (which was used to train GPT-3, and the first generation of massively scaled foundation models) for AIs trained on language alone to reach human-level capabilities across the board? It’s possible, Irina thinks — and that’s why she’s excited about the trend towards multi-modal learning: the practice of training AIs on multiple data types at the same time (for example, on image, text and audio data). The hope is that by combining these input data types, an AI can learn to transcend noise limits that may exist in any one data type alone.

Irina Rish - Out-of-distribution generalization-by Towards Data Science-video_id QjXFN4UWZCg-date 20220309.md


# Markus Anderljung and Ben Garfinkel Fireside chat on AI governance - EA Forum

**Getting involved in AI governance**

**Markus:** Ben, how did you get into the field of AI governance?

**Ben:** I majored in physics and philosophy at Yale, and was considering working in the philosophy of physics. I started thinking about it not being the most useful or employable field. At the same time, I got interested in EA \[effective altruism\]. Then, Allan Dafoe was transitioning to focus on a new Centre for the Governance of AI. He was looking for researchers. I seized the opportunity — it seemed important, with not enough people working in \[the field\]. That’s how I got involved.
**Markus:** What happened next?

**Ben:** I was a Research Assistant there for about a year, and, at the same time, held a job at the Centre for Effective Altruism. I then had the opportunity to transition to the AI governance area full-time.

**Markus:** Sounds a bit random — just having this opportunity pop up.

**Ben:** There was indeed an element of randomness, but it wasn’t a random thing to do. I got really interested in long-termism and EA, so AI governance was on my radar — and yes, an opportunity lined up at the same time.
**Markus:** How has your work in the field changed?

**Ben:** Quite a lot. I still have a broad focus in the area, but when I started, there was this sense that AI was going to be a very important field. It was around 2016. [AlphaGo](https://deepmind.com/research/case-studies/alphago-the-story-so-far) had just come out. [_Superintelligence_](https://www.amazon.com/dp/B00LOOCGB2/) had been written, so a good fraction of long-termist concern was on AI. AI seemed to be this really transformative technology, with risks we didn’t understand very well yet.

Almost no one was working on AI safety at the time or thinking about long-term AI governance challenges, and not that much AI governance was going on. So the early questions were “What is going on here?” and “What are we trying to do?” A lot of the early research was probably more naive, with people transitioning to the field and not knowing that much about AI.

**Markus:** And you’re now doing a DPhil at Oxford?

**Ben:** Yes, I just started \[working toward a DPhil\] in international relations. My highest credential so far in the area \[of AI governance\] remains an undergraduate degree in an unrelated field, so it seemed useful to have a proper degree in a domain more relevant to governance than physics.

Let’s turn the question on you, Markus. How did you get involved?

**Markus:** It’s a slightly convoluted story. I was first involved in EA issues at university, around 2011-2013. I got involved with [Giving What We Can](https://www.givingwhatwecan.org/) and \[other EA\] organizations. Then I spent a few years transitioning from \[the belief that\] long-termism — or the closest \[approximation\] of it back then — was true, but there wasn’t much to do about it in the emerging technology field, to becoming increasingly convinced that there _were_ things to do.

When I graduated from university in 2016, I moved back to Sweden. I considered building career capital, so I went into management consulting for a few years, which was very fun and interesting. After a while it felt that the trajectory had leveled off, and that I could do more in the cause areas I cared about.

I transitioned into work for EA Sweden. Building a community seemed like a good idea, and heading up the organization was a great opportunity. I got funding for that and did it for about a year. I then became convinced that I could do more with my career outside of Sweden.

I was looking for other options, thinking that my comparative advantage was “I’m a management consultant who understands the research, someone who _gets it_.” I applied to organizations where I thought my broader skill set of operations and recruitment could help. GovAI was one of those organizations.

It wasn’t like AI governance was the most important thing and I had to help there; it was just part of a broader class of areas that seemed useful.

**The relative importance of AI governance research**

**Markus:** Let’s talk about your recent research. At EA Global: London 2018, you gave a talk.

**Ben:** “[How sure are we about this AI stuff?](https://www.effectivealtruism.org/articles/ea-global-2018-how-sure-are-we-about-this-ai-stuff/)”

**Markus:** Right. So, I want to ask: How sure are we about this AI stuff?

**Ben:** There are two ways to be sure. One is robust. \[It centers on the question\] “If we had all of the facts and considerations, would this still be a top priority for EA?” The other \[centers on the question\] “Given our limited information and the expectations we have, are we sure it still makes sense to put a lot of resources into this area?”

With the first question, it’s really hard to be sure. There’s so much we don’t know: what future AI systems will look like, the rate of progress, which institutions will matter, the timelines. That’s true of any transformative technology; we don’t have a great picture.

**Markus:** Will it be different for AI versus other causes?

**Ben:** If you compare AI to climate change, there’s a lot of uncertainty in climate models. We don’t know everything about the feedback loops, or how to think about extreme events — is there a one-in-100 probability or a one-in-1,000 probability? We still have a basic sense of the parameters of the problem, such as how hot things will get (to some extent) and how bad it is.

With AI, if human labor becomes essentially unnecessary in the long term, as it’s replaced by AI systems, we don’t know what that world looks like or how it’s organized. It’s very difficult to picture. It’s like being in the 1500s and describing the internet in very rough terms, as \[something that will be\] more efficient \[and enable\] faster communication and information retrieval. You could have reasoned a bit about this future — maybe there would be larger businesses, since you can communicate on a larger scale. But you wouldn’t be visualizing the internet, but rather very fast carrier pigeons or something. You’re going to be way off. It’s hard to even find single dimensions where the answer is clear.

I think that’s about where we are with AI, which is a long-winded way of saying that it’s hard to be sure.

I actually feel pretty good about the weaker standard (“How sure are we, given these considerations, that a decent chunk of the EA movement should be focused on this?”). Overall, I think a smaller portion of the EA portfolio should be focused on this, but at least a few dozen people should be actively thinking about long-term applications of AI, and we’re not far from that number at the moment.

**Markus:** That sounds smaller than I expected. When you say a smaller portion of the EA portfolio should be focused on AI, what’s your current ballpark estimate of what that percentage is?

**Ben:** I think it’s really hard to think about the spread between things. Maybe it’s something along the lines of having one in five people who are fully engaged and oriented on the long term thinking about AI. That would be great.

**Markus:** Whereas now, you think the number is more like three or four in five people?

**Ben:** Yeah, it feels to me that it might be more than half, but I’m not sure that’s correct.

**Markus:** What interventions would you like these people to do instead?

**Ben:** There’s a lot of uncertainty. A lot of it is based on the skill set that these people have and the sort of work they’d be inclined to do. There being a lot of math and computer science \[in EA\] may justify the strong AI focus, but let’s imagine completely fungible people who are able to work on anything.

I really think fundamental cause prioritization research is still pretty neglected. There's a lot of good work being done at the [Global Priorities Institute](https://globalprioritiesinstitute.org/). There are some broad topics that seem relevant to long-term thinking that not many people are considering. They include questions like those I was working on: “Should we assume that we don’t have really great opportunities to influence the future now, relative to what future people might have if we save our money?” and “Should we pass resources on?” These seem crucial for the long-termist community.

**The importance and difficulty of meta-level research**

**Ben:** Even within AI, there are strangely not that many people thinking about, at the meta-level, the pathways for influence in AI safety and governance. What exactly is the nature of the risk? I think there’s a handful of people doing this sort of work, part-time, on the side of other things.

For example, Rohin Shah is doing a lot of good \[by thinking through\] “What exactly is the case for AI risk?” But there are not that many people on that list compared to the set of people working on AI safety. There’s an abstract argument to be made: Before you put many resources into object-level work, it’s quite useful, early on, to put them toward prioritizing different kinds of object-level work, in order to figure out what, exactly, is motivating the object-level work.

**Markus:** One of your complaints in \[your past\] talk was that people seemed to be putting a lot of research into this topic, but haven’t produced many proper writeups laying out the argument motivating people’s choices. Do you think we’ve seen improvement there? There are a few things that have been published since then.

**Ben:** Yeah, there’s definitely been an improvement since I gave the talk. I think the time at which I gave the talk was kind of a low point.

There was an initial period of time, after the publication of _Superintelligence_, when the motivation for AI governance, for a lot of people, corresponded to the nature of AI risk. Over time, there was some sort of transition; people have very different visions of this. The change happened along a lot of dimensions. One of them is that _Superintelligence_ focuses on a very discrete transition to advanced AI, in which not much is happening, and then we transition to quite advanced systems in a matter of days or weeks. A lot of people moved away from that.

Also, a lot of people, myself included, started thinking about risks that weren’t specifically safety-oriented. _Superintelligence_ discusses these but they’re not the main focus.

**Markus:** What do you mean by “not safety-oriented”?

**Ben:** There’s a lot of concern you might have about the future of AI not necessarily being great. For example, in a scenario in which human labor and government functions have been automated, it may not be a great world in terms of democracy and representation of the will of the people.

Another category is ethical \[and concerns\] the moral status of AI systems. Maybe those decisions are made wrongly, or are substantial deviations from the best possible case.

**Markus:** So these are risks that don’t \[involve\] accidents with very powerful systems.

**Ben:** We’ve had major technological transitions in history which haven’t been uniformly good. The classic one is the Neolithic Revolution — the introduction of agriculture — having a few aftereffects like the rise of the state. It’s difficult to do an overall assessment. Some of the results were positive, and some were very much not, like slavery becoming a massive institution, disease, and the establishment of hierarchies instead of decentralized decision making.

It’s not hard to imagine that if there’s another transition, in which human labor is replaced by capital, that \[transition\] may have various effects that aren’t exactly what we want.

**Markus:** Yes, and in these previous transitions, the bad consequences have been permanent structural effects, like slavery being more economically viable.

So \[the time of your EA Global talk\] — November 2018 — was the low point? In what sense?

**Ben:** It was the low point in the sense of people having changed their justifications quite a bit in a lot of different areas, \[without those changes being\] reflected in much published writing \[other than\] maybe some blog posts.

There hasn’t been a massive improvement, but there definitely has been useful stuff published since then. For example, Paul Christiano wrote a [series of blog posts](https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd) arguing for AI safety even in the context of a continuous transition; Richard Ngo did [some work](https://www.alignmentforum.org/posts/JbcWQCxKWn3y49bNB/disentangling-arguments-for-the-importance-of-ai-safety) to taxonomize different arguments and lay out \[current thinking in\] the space; Tom Sittler did [similar work](https://fragile-credences.github.io/prioritising-ai/); and Rohin Shah presented a case for AI risk in a good sequence called “[Value Learning](https://www.alignmentforum.org/s/4dHMdK5TLN6xcqtyc),” a series of essays that laid out the nature of the alignment problem.

I think that was after —

**Markus:** “[Reframing Superintelligence](https://www.fhi.ox.ac.uk/reframing/)”?

**Ben:** Yeah, Eric Drexler’s work at FHI \[the Future of Humanity Institute\] also came out framing his quite different picture of AI progress and the nature of the risks. Also MIRI \[the Machine Intelligence Research Institute\] put out [a paper](https://intelligence.org/embedded-agency/) on what they call “mesa-optimization,” which corresponds to one of their main arguments for why they’re worried about AI risk, and which wasn’t in _Superintelligence_.

There have been a decent amount of publications, but quite a bit fewer that I would ideally want. There are still a lot of viewpoints that aren’t captured in any existing writing, and a lot of writing is fairly short blog posts. Those are useful, but I’m not very comfortable with putting a lot of time and work into an area where justifications are short blog posts.

It’s obviously very difficult to communicate clearly about this. We don’t have the right picture of how things will go. It’s not uncommon to have arguments about what a given post is actually saying, which is not a great signal for our community being on the same page about the landscape of arguments and considerations.

**Markus:** Why do you think this is? Is it due to individual mistakes? Not spending enough time on this meta-level question?

**Ben:** To some extent, yes. There are complications. Working on this stuff is fairly difficult. It requires an understanding of the current landscape, of arguments in this area, of what’s going on in AI safety and in machine learning. It also requires the ability to do conceptual analysis and synthesis. There are perhaps not that many people right now \[who meet these criteria\].

Another complicating factor is that most people currently working in this area have just recently come into it, so there’s an unfortunate dynamic where people have the false sense that the problem framing is a lot more \[advanced\] than it actually is — that it just hasn’t been published, and that the people in the know have a good understanding.

When you enter an area, you aren’t usually in a great position to do this high-level framing work, because you don’t really know what exists in terms of unpublished Google Docs. It’s quite easy, and maybe sensible, when entering the area, to not do high-level \[thinking and writing\], and instead pick an object-level topic to work on.

Some of us might be better off dropping the object-level research program and \[addressing\] more high-level problems. Some have been doing this in their spare time, while their \[main area of study\] is object-level. It does seem like a difficult transition: to stop an object-level project and embark on a loose, “what-are-we-even-doing” project.

**Markus:** Are there particular viewpoints that you feel haven’t been accurately represented, or written up thoroughly enough?

**Ben:** Paul Christiano, for example, has written a few blog posts. One is called “[What Failure Looks Like](https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-failure-looks-like).” It shows what a bad outcome would look like, even in a scenario with a slow, gradual transition. However, there’s a limit on how thoroughly you can communicate in the form of a blog post. There is still a lot of ambiguity about what is being described — an active disaster? A lost opportunity? What is the argument for this being plausible? There’s a lot more that could be done there.

I feel similarly about this idea of mesa-optimization, which is now, I think, one of the primary justifications that MIRI has for assigning a high probability to AI safety risk. I saw a lot of ambiguity around what this concept exactly is. Maybe different people are trying to characterize it differently or are misunderstanding the paper, or the paper is ambiguous. It doesn’t seem like everyone is on the same page about what exactly mesa-optimization is. The paper argues that there might be this phenomenon called mesa-optimization, but doesn’t try to make the argument that, because this phenomenon might arise, then we should view it as a plausible existential risk. I think that work still hasn’t been done.

**Markus:** So the arguments that _are_ out there ought to be scrutinized more. Are there arguments or classes of viewpoints that you feel don’t even have an initial writeup?

**Ben:** I think there are a couple. For example, Allan \[Dafoe\] is thinking quite a lot about structural risks. Maybe the situation starts getting bad or disappointing in terms of our current values. It’s a bit nebulous like the Neolithic Revolution — not a concrete disaster. Some structural forces could push things in a direction you ideally wouldn’t want.

Similarly, there’s not much writing on whether AI systems will eventually have some sort of moral status. If they do, is that a plausible risk, and one that will be important enough for longtermists to focus on?

Those are probably two of the main things that stand out in my mind as plausible justifications for focusing on AI, but where I can’t point to a longer writeup.

**Markus:** What if I were to turn this around on you? You’re here, you’re working on these issues. What is your stab at a justification?

**Ben:** The main point is that it’s very likely that AI will be really transformative. We will eventually get to the point where human labor is no longer necessary for most things, and that world will look extremely different.

Then, there are around a half-dozen vaguely sketched arguments for why there might be some areas with EA leverage that would make the future much better or much worse — or why it could go either way.

It is hard to \[determine\] which topics may have long-term significance. I don’t think they \[comprise\] a large set, and there’s value in \[surfacing\] that information right now, in getting clearer on “what’s going on in AI.” It seems like one of the few places where there’s currently the opportunity to do useful, \[future-focused\] work.

**Markus:** So the argument is: “If you’re a long-termist, you should work in the areas that hold great leverage over the future, and this seems like one of the best bets.”

**Ben:** Yeah, that’s basically my viewpoint. The influence of historical events is extremely ambiguous. How plausible is it for us to know what our impact today will have 100 years from now? In the 1300s, people’s focus may have been on stopping Genghis Khan. I think that would have been ethically justified, but from a long-termist perspective, things are less clear. Genghis Khan’s impact may have ultimately been good because of the trade networks established and \[other such factors\]. We’re unable to discern good from bad from a long-termist perspective. Pick any century from more than five centuries ago, and you’ll be in the same position.

I think we should have a strong prior that justifies working on issues for their present influence, but for the long-term view, we shouldn’t prioritize issues where we can’t predict what difference they’ll make hundreds of years in the future.

There are not many candidates for relevant long-term interventions. Insofar as there are semi-plausible arguments for why AI could be one of them, I think it’s more useful to put resources into figuring out what is going on in the space and \[improving\] the value of information, rather than putting resources into object-level issues.

**The role of GovAI**

**Ben:** So, Markus, could you tell me what GovAI is currently up to in this space?

**Markus:** Broadly, we’re working on AI governance questions from a long-termist perspective, and I spend most of my time doing what I can to make sure we build an organization that does the best research.

In practice, we have a lot of different research projects going on. I personally spend a lot of time with recruiting, growing the organization. We run a [GovAI Fellowship](https://www.fhi.ox.ac.uk/governance-of-ai-fellowship/), where people spend three months doing research on topics that relate to the kinds of things we’re interested in. That’s a path into the AI governance space, and something we’ll continue doing for the foreseeable future. We \[award\] 10 fellowships every year. We’ll see whether we’ll be able to bring people \[on-site\] this summer. I’m pretty excited so far about this as a way of getting people into the field.

Since 2016, we’ve not been able to build up institutions in the field that provide clear pathways for people. I think this fellowship is one example of how we can do that.

My hope is to have, a few years down the line, a team of a dozen great researchers in Oxford doing great research. In terms of the research that we’ll do, we’re \[an unusual\] organization, in that we could define the very broad problem of AI governance as “everything required to make AI go well that isn’t technical AI safety.” That’s a _lot_.

It will span fields ranging from economics, to law, to policy — a tremendous number of different topics. I’m hoping that, over time, we’ll build narrower expertise as we get clearer on the meta picture and the specific fields that people need to work on.

A few years down the line, I’d really like for us and others in this space to have at least somewhat solid \[recommendations\] in \[situations\] like a corporation asking what their publication norms should be, or what sorts of internal mechanisms they should have to make sure they’re held accountable to the beautiful principles they’ve written up (e.g., “benefit humanity with our research”).

I don’t think we have that yet. Those are the kinds of questions I’m hoping we can make progress on in the next few years.

**Career recommendations**

**Ben:** Besides just applying for the GovAI Fellowship, do you have other career recommendations for people interested in \[entering or exploring whether to enter\] this space?

**Markus:** In general, I \[subscribe to the view\] that if you’re trying to figure out if you should be doing something, then do a bit of it. Find some bit of research that you can do and try it.

There aren’t a lot of opportunities that look like the GovAI Fellowship in the long-termist AI governance space, but others that are similar are at the [Center for Security and Emerging Technology](https://cset.georgetown.edu/), based in Washington, DC. There are also junior roles at DeepMind and OpenAI, where you might do \[some exploratory\] research. But there aren’t many such roles — probably fewer than a dozen a year.

I would encourage people to think much more broadly. You might try to work at a wider set of technology companies like Microsoft or Facebook as they start building up ethics and policy teams. It would be awesome to have people in these types of “council” roles.

Another good idea would be to use your studies to dip your toe into the water. You could do your bachelor’s or master’s dissertation on a relevant topic. Some are looking into PhDs in this area as well, and that may be a good way to expand the field.

The other main tip is to engage with a lot of the research. Read everything coming out of institutions like ours, or [CSET](https://cset.georgetown.edu/) \[the Center for Security and Emerging Technology\]. Try to really engage with it, keeping in mind that the authors may be smart and good at what they do, but aren’t oracles and don’t have that much knowledge. There’s a lot of uncertainty in this space, so read things with an open mind. Consider what may be wrong, stay critical, and try to form your own views instead of directly \[adopting\] other people’s conclusions. Form your own model of how you think the world could go.

**Ben:** Sounds good. Very wise.

**Markus:** Do you have any tips? What would you have told your past self?

**Ben:** I can’t think of anything beyond what you just said, other than to have checked my email in case Allan Dafoe was looking for research assistants in this area, with a fairly non-competitive process at that time. Not many people were interested in it back then.

**Markus:** Right, so _get really lucky_.

**Ben:** Get really lucky — that’s exactly what I would tell my past self.

**Markus:** Cool.

**Ben:** Well, it’s been fun chatting!

**Markus:** Yes, and I’ll add my new sign-off: Stay safe, stay sane, and see you later!

Fireside chat - AI governance _ Markus Anderljung _ Ben Garfinkel _ EA Global - Virtual 2020-by Centre for Effective Altruism-video_id bSTYiIgjgrk-date 20200321.md