# Jade Leung How can we see the impact of AI strategy research - EA Forum

_For now, the field of AI strategy is mostly focused on asking good questions and trying to answer them. But what comes next? In this talk, Jade Leung, Head of Research at the_ [_Center for the Governance of AI_](https://www.fhi.ox.ac.uk/govai/)_, discusses how we should think about practical elements of AI strategy, including policy work, advocacy, and branding._

_Below is a transcript of Jade’s talk, which we’ve lightly edited for clarity. You can also watch the talk on_ [_YouTube_](https://www.youtube.com/watch?v=8M3nIu7GIsA) _or read it on_ [_effectivealtruism.org_](https://effectivealtruism.org/articles/jade-leung-how-can-we-see-the-impact-of-ai-strategy-research)_._

## The Talk

Technology shapes civilizations. Technology has enabled us to hunt, gather, settle, and communicate. Technology today powers our cities, extends our lifespans, connects our ideas, and pushes the frontier of what it means to be human.

Technology has also fueled wars over power, ideology, prestige, history, and memories. Indeed, technology has pushed us to the precipice of risk in less than a decade — a fleeting moment in the timespan of human civilization. \[With just a few years of research,\] we equipped ourselves with the ability to wipe out the vast majority of the human population with the atomic bomb.

If we rewind to the emergent stage of these transformative technologies, we have to remember that we are far from being clear-eyed and prescient. Instead, we're some combination of greedy, clueless, confused, and reckless. But ultimately, how we choose to navigate between the opportunities and risks of transformative technologies will define what we gain from these transformative technologies — and also what risks we expose ourselves to in the process.

This is the canonical challenge of governance of these transformative technologies. Today, we're in the early stages of navigating a particular technology: artificial intelligence (AI). It may be one of the most consequential technologies of our time and the most important one for us to get right. But getting it right requires us to do something that we've never done before: Formulate a navigation strategy with deliberate caution and explicit altruistic intention. It requires us to have foresight and to orient \[ourselves\] toward the long-term future. This is the challenge of AI governance.

If we think about our history and track record, our baseline is pretty far from optimal. That’s a very kind way of saying that it sucks. We're not very good at governing transformative technologies. Sometimes we go down this path and \[the journey\] is somewhat safe and prosperous. Sometimes, we falter. We pursue the benefits of synthetic biology without thinking about how that affects biological weapons. Sometimes we stop ourselves at the starting line because of fear, failing to pursue opportunities like atomically precise manufacturing or STEM cell research. And sometimes we just fall into valleys.

![](https://images.ctfassets.net/ohf186sfn6di/5DCD4HJzhKUiuLdvynmunh/c89954c09a5a4cfdbeef94b0ece880cd/Slide02.png)

During the Cuban Missile Crisis, President John F. Kennedy estimated that the chance of nuclear war was one in three. _One in three._

The reality is we've been pretty damn lucky. We deserve no credit for \[avoiding any of these catastrophes\]. But as my swimming coach once said, “If you're really, really, really bad at something, you only need to try a little bit to become slightly better at it.” So here's to being slightly better at navigating these transformative technologies.

I think there are three goals in the AI strategy and governance space \[that can help us rise\] slightly \[above\] our currently awful baseline.

![](https://images.ctfassets.net/ohf186sfn6di/2D0pEJYB95BPTUn2JfPz4T/ca9b35b14b98956277464087358593a4/Slide05.png)

Goal number one: Gain a better understanding of what this landscape looks like. Where are the mountains? Where are the valleys, the slippery slopes, the utopias? This is super-hard to do. It's very speculative and uncertain, so we need to be humble. But we should try anyway.

The second thing we can try to do is equip ourselves with good heuristics for navigation. If uncertainty is an occupational hazard of working in this space, then we can try to figure out, in general terms, what might be good and bad \[to pursue\]. How should we orient ourselves? Which directions do we want to go in?

The last goal is to translate these heuristics into actual navigation strategies. How do we ensure that our heuristics make it into the hands of the people who are turning this boat in certain directions?

If you'll stick with my navigation metaphor for a bit longer, we can think of the first goal as a mapping exercise to determine where the mountains, valleys, water sources, and cafes with good wifi are. The second goal is about equipping ourselves with a compass. If we know that there are aggressive rhinos to the south and good vegan restaurants in the north, we’ll go north instead of south.

This metaphor is kind of falling apart, but the third goal is the steering wheel. You can’t use it if you’re in the back of the car. That's ultimately what I want to focus on today: How do we make sure that \[our map and compass will be used to steer — i.e., to make real-world decisions about AI\]?

\[I have two reasons for focusing\] on this. First, AI strategy and governance research is effective when it happens upstream of actionable, real-world tactics and strategies. They can be relatively far upstream. I think we would lose a lot of good research questions if \[we were always motivated by\] whether something could inform a decision today. But I think it would be a mistake for anyone who does AI strategy and governance research to \[avoid\] thinking about how they expect their research to \[play out\] in relevant, real-world decisions.

That leads me to the second reason for focusing: I don't think we know how to do this \[make our research actionable\] well. I think we invest far more effort into understanding how to do good research than we do into understanding how to \[come up with\] good tactics. Don't get me wrong: I don't think we know how to do good research yet. We're still trying to figure that out. And I find it hilarious that people think that I know how to do good research; if only you knew how little I know! But I think we need to invest far more proportional effort into \[asking ourselves\]: Once we’ve done our research and have some insights in place, what do we do to \[apply\] them and \[influence\] the direction in which we're going?

![](https://images.ctfassets.net/ohf186sfn6di/idyfmsAFAWL7iSMXvGvlw/d3e0dbbcbf2693c342158b6837950756/Slide06.png)

With that in mind, let's start at the end. What are the decisions that we want to influence in the real world? Another way to ask this question is: Who is making the decisions that we want to change?

They fall into two broad categories: (1) those developing and deploying AI and (2) those shaping the environment in which AI is developed and deployed.

![](https://images.ctfassets.net/ohf186sfn6di/7abtDrND6UQYIvZs55YHrn/0afb7092fddf02f791de60b03957fb28/Slide07.png)

Those developing and deploying AI include researchers, research labs, companies, and governments. those shaping the environment.

In terms of \[the second group\], there are a number of different environments to shape:

\* **The research environment** can be shaped by lab leaders, funders, universities, and CEOs. They shape the kind of research that is being invested in — i.e., the research considered within the [Overton window](https://en.wikipedia.org/wiki/Overton_window).

\* **The legislative environment**, which constrains what can be deployed and how, can be shaped by legislators, regulators, states, and the people \[being governed\].

\* **The market environment**, which can be shaped by investors, funders, consumers, and employees. They create incentives that drive certain forms of development and deployment, because of supply and demand.

Now, you can either become one of these decision-makers or you can become a person who influences them. This is in no way a commentary on your brilliance as human beings. But none of you will become important. I'm unlikely to become important. The reality is that’s how the world works. If you do end up becoming an important person, the recording of this talk is your voucher for a free drink on me. But if you assume that I'm right, most of you are going to fall into the category of people who influence decisions as opposed to making them.

Therefore, I’m going to \[spend the rest of this talk\] focusing on this question: How do we increase our ability to influence the decisions being made \[about AI\]?

![](https://images.ctfassets.net/ohf186sfn6di/w8TRFh3DPKsEoHpvMUSWB/a245963bccdae8366fa85babd6eaecdd/Slide08.png)

There are many steps, but I see them falling into two broad areas. The first step is having good things to say. The second step is making sure that the people who matter \[are made aware of\] these good things.

![](https://images.ctfassets.net/ohf186sfn6di/35KkwXeV291FVSe2D59ZKx/73049db077dedaa23f5e3b53e1654480/Slide09.png)

A quick note on what I mean by “good”: I'm broadly conceiving of all of us as good in the normative sense of steering our world in a direction that we want, and good in the pragmatic sense, in that a decision-maker will be likely to actually go in that direction because it's reasonable and falls within their timeframe.

Oftentimes these two definitions of good conflict. For example, things you think will be good for the long-term future won't \[necessarily\] be things that are tractable or reasonable from a decision-maker’s point of view. I acknowledge that these two things are in tension. It's hard to figure out how to compromise between them sometimes.

![](https://images.ctfassets.net/ohf186sfn6di/4hjmoCrfCkKmFHgrovH4Kx/e8f30f0f4ac753cba914fbf666a19e0d/Slide10.png)

That being said, I think AI strategy and governance research can aim to have good things to say about a given decision-maker’s (1) priorities, (2) strategies, and (3) tactics. Those are three broad buckets to dig into a bit more.

**Priorities:** I think priorities are basically people’s goals. What benefits are they incentivized to pursue, and what costs are they willing to bear in the process of pursuing those goals? For example, if you manage to convince a lab that safety leads to product excellence, that can make safety a goal for the lab. If you manage to convince a government that cooperation is necessary for technology leadership in an international world, that can make cooperation a goal.

**Strategies:** You may aim to have useful things to say about certain strategies that \[decision-makers adopt\]. For example, resource allocation is a pretty common strategy that one could aim to influence. How are they distributing their budgets? How are they investing in research and development efforts across various streams? You also may have things to say about what a given actor chooses to promote or advocate for versus \[ignore\]. For example, in the case of influencing a government, you might want them to pursue certain pieces of legislation that can help you achieve certain goals. In the case of labs, you might want them to invest in certain types of new programs or different workstreams.

**Tactics:** The third area is tactics. These include public relations tactics. What do they signal to the external world, and how does that affect their ability to achieve their goals? And what about relationship tactics — with whom do they coordinate and cooperate? Whom do they trust (and distrust)? Whom do they decide to invest in?

To make this a little bit more concrete, I'm going to pick on an actor who needs a lot of good \[advice\]: the U.S. government.

![](https://images.ctfassets.net/ohf186sfn6di/2JXNTSUGAIpxX5uKjvXQp1/c65a101927ec5d225dcaee983993594a/Slide11.png)

One of the biggest risks is that nation states will slide into techno-nationalist economic blocs. The framing of strategic competition that we have around AI now could exacerbate a number of AI risks. I won't go into detail now, but [we've written a fair amount about it at The Governance of AI](https://www.fhi.ox.ac.uk/govai/#publications). We want to prevent nations from sliding into various economic blocs and the nationalization of bits of AI research and development.

What would a caricature of the U.S. government's position look like? (I say “caricature” because it's not at all clear that they actually have a coherent strategy.) It looks something like sliding into these economic blocs. And that's a bad thing. Their \[overarching\] priorities are technology leadership, in both an economic and military sense, with a corollary of preserving and maintaining national security. Costs that they may be willing to bear in extreme circumstances include anything that is required to gain control of an R&D pipeline and secure it within national borders.

Now they are making moves in the strategy and tactics space — for example, announcements of export controls that the U.S. government made in November 2018 indicate that they want to preserve domestic capacity for R&D at the cost of investing in international efforts and transnational communities. They also indicate an explicit intention of shutting out foreign competitors and adversaries. \[Overall\], their AI strategies and tactics point in the direction of “America first.” And the footnotes there suggest that when America is first, over the long term the world suffers. That's too bad. So, those are the kinds of stances that the U.S. posture points toward.

If one has \[the chance to try persuading\] the U.S. government, one could aim to convince them that their priorities, strategies, and tactics should move in a different direction. For example, a desirable priority could be technology leadership, but leadership could mean leading with a global, cosmopolitan viewpoint. You \[could focus on influencing them to\] bear the cost of investing in things like safety research in order to pursue this priority in a responsible way. The strategies and tactics you could inform them of when they conduct this research could \[involve international outreach\]. With whom should they ally themselves and cooperate? What kinds of signals should they send externally to ensure that others with a similar view of technology leadership will \[take steps in\] the same direction?

This is the type of decision set that you want to influence when conducting upstream AI strategy and governance research. \[Once you\] have a broad sense of what you think is good, you have the mega-task of trying to make those good things happen in the real world.

I have a few suggestions for how to approach that.

![](https://images.ctfassets.net/ohf186sfn6di/6z2ZpuOh6qSqXPcuuDFUA2/fb430acad3de772a61ac199a49561042/Slide12.png)

The first is to \[focus on\]\] a few _tractable_ good things. I say “tractable” here to mean things that will make sense to, or sit well with, decision-makers, such that they are likely to do something about it.

One way to do that is to find hooks between things that you care about and things that a decision-maker cares about. Find that intersection or middle part of the Venn diagram. One canonical bifurcation — which I don't actually like all that much — is the bifurcation between near-term and long-term concerns. Near-term concerns are things that are politically salient. \[They allow you to\] have a discussion in Congress and not look nuts. Long-term concerns are often things that make you look a little bit wacky. But there are some things at the intersection that could lead you to talk about near-term concerns in a way that lays the foundation for long-term concerns that you actually care about and want to seed discussions around.

For example, the automation of manufacturing jobs is a huge discussion in the U.S. at the moment. It’s a microcosm of a much larger-scale problem \[involving\] massive labor displacement, economic disruptions, and the distribution of economic power in ways that could be undesirable. That’s a set of long-term concerns. But talking about it in the context of truck drivers in the U.S. could be an inroad into making those long-term concerns relevant.

A similar thing can be said about the U.S. and China. People in Washington, D.C. care about the U.S.’s posture toward China, and what the U.S. does and signals now will be relevant to how this particular bilateral relationship pans out in the future. And that's incredibly relevant for how certain race dynamics pan out.

Once you've filtered for these things that are tractable, then you need to do the work of translating them in a digestible way for decision-makers.

![](https://images.ctfassets.net/ohf186sfn6di/28dJJMsYjDwnPD60VSgHYP/45fd30db87e2be201b8fda4cf79d8cbd/Slide13.png)

The assumption here is that decision-makers are often very time-constrained and attention-constrained. They will \[be more likely to respond to messages that are\] easy to remember and \[relayed\] in the form of memes. And unfortunately, long, well-argued, epistemically robust pieces end up \[having less impact\] than we would hope.

Superintelligence is perhaps one of the best examples. This is an incredibly epistemically robust \[topic\]. But ultimately, the meme it was boiled down to for the vast majority of people was: “Smart Oxford academics think AI is going to kill us.” So don't try to beat them with nuance. Try to just play this meme game and come up with better memes.

Here are three examples of memes that are currently in danger of taking off:

![](https://images.ctfassets.net/ohf186sfn6di/733mHkA6KjER3FaIsl4Mi/b3c0460e02a13dfa39dbe39577b28302/Slide14.png)

1\. The U.S. and China are in an arms race.
2\. Whoever wins will have a decisive strategic advantage.
3\. AI safety is always orthogonal to performance.

It's not clear to me that all of these things are true. And for some of them I'm quite sure that I don't want them to be true. But they are being propagated in ways that are informing decisions that are currently being made. I think that's a bad thing.

One thing to focus on, in terms of trying to have good things to say and making those good things heard, it to translate them into messages that are similarly digestible.

![](https://images.ctfassets.net/ohf186sfn6di/EujxjpH5wbaTAZExHdTrK/08da0769129a91e57a6de0ae0106e312/Slide15.png)

Candidates for memes we might propagate are things like: “the equivalent of leading in the AI space is to care about safety and governance”; “the windfall distributions from a transformative AI should be distributed according to some common principles of good”; and “governance doesn't equal government regulation, so multiple actors carry the responsibility to govern well.” Unless we propagate our messages in easy ways, it's going to be very hard to compete with the bad narratives out there.

![](https://images.ctfassets.net/ohf186sfn6di/4ss4g3lGYNvcg0iGC48t8t/48865b05b5a7b1bbeafe610f799eb560/Slide16.png)

The last step is to ensure that \[our messages\] reach some circles of influence. To do that, model your actor well. For example, if you want to target a specific lab, try to figure out who the decision-makers are, what they care about, and whom they listen to. Then, target your specific messages and work with those particular circles of influence in order to get heard. That's my hot take on how \[research\] can be made slightly more relevant in a real-world sense.

Some final points that I want you to take away:

![](https://images.ctfassets.net/ohf186sfn6di/4r31D2owuDzQMIzfMW08Uc/ea15b45a3d6a36bd1ba545417b91cc25/Slide17.png)

Ultimately, the impact of this work is contingent on how good our tactics are. The claim that I've made today is that we need to put far more work into this. I’m uncertain how well we can do that — and how much effort we should put into it. But broadly speaking, as soon as we have relevant insights, we should be intentional about investing in propagating them.

![](https://images.ctfassets.net/ohf186sfn6di/5oPLXGvYTnXCp0vT22Ue2A/ddff5d0a887324bc75444b70bcba7a46/Slide18.png)

Second, exercising this influence is going to be a messy political game. The world’s \[approach to\] decision-making is muddled, parochial, and suboptimal. We can have a bit of a cry about how suboptimal it is. But ultimately, we need to work within that system. \[Using\] effective navigation strategies is going to require us to work within a set of politics that we may disagree with to some extent in terms of values. But we need to be tactical and do it.

![](https://images.ctfassets.net/ohf186sfn6di/6syDzhMJywbv59mL1IQi2D/f6969f592a6e8d567b96a5479dc97976/Slide19.png)

Finally, governance is a very hard navigation challenge. We have no track record of doing it well, so we should be humble about our ability to do it. At the moment we don't know that we can succeed, but we can try our best.

**Moderator:** Thank you for that talk. I’d like to start with something that you ended with. You said that we're dealing with systems that are difficult to operate in. To what extent do you even think it's possible to get people to think more clearly? Should we instead just be focusing on institutional change?

**Jade:** I think there are things that we need to try out. Institutional change is valuable. Attempting to communicate through existing decision-makers in existing institutions is valuable. But I don't think we know enough about what's necessary and how tractable certain things are in order to put all of our eggs in one basket.

So maybe one meta-point is that as a field, we need to diversify our strategies. For example, I think some people should be focusing on modeling existing decision-makers — particularly decision-makers that we think \[have enough credibility\] to be relevant. And I think others could take the view that existing institutions are insufficient, and that institutional change is ultimately what is required. And then that becomes a particular strategy that is pursued.

The field is shrouded in enough uncertainty about what's going to be relevant and tractable that I would encourage folks to diversify.

**Moderator:** You focused on the U.S. government as one of the actors that people might pay particular attention to. Are there others that you would recommend people pay attention to?

**Jade:** Yeah. I generally advocate for focusing on modeling governments \[based in places that are likely to be relevant\] more than particular private actors. For example, the Chinese government would be worth focusing on. I think we have a better shot at modeling them based on history. There are more variants and anomalies in private spaces.

Second, focus on organizations that are important developers of this technology \[AI\]. The canonical ones are [DeepMind](https://deepmind.com/) and [OpenAI](https://openai.com/). There are others worth focusing on too.

**Moderator:** Someone could construe your advice as trying to understand what's happening currently in the policy landscape and in a variety of academic disciplines that people spend their lives in, and then melding all of those together into a recommendation for policymakers. That can feel a little overwhelming as a piece of advice. If someone has to start somewhere and hasn't worked in this field before, what would you say is the minimum that they should be paying attention to?

**Jade:** Good question. If you're not going to try to do everything (which is good advice), I think one can narrow down the space of things to focus on based on competitive advantage. So think through which arenas of policy decisions you're likely to be able to influence the most. Then, focus specifically on the subset of actors in that space.

**Moderator:** And assuming a person doesn't have expertise in one area and is just trying to fill a vacuum of understanding somewhere in this AI strategy realm, what would you \[recommend\] somebody get some expertise in?

**Jade:** That’s a hard question. There are a lot of resources that are out there that can help orient you to the research space. Good places to start would be our [website](https://www.fhi.ox.ac.uk/govai/). There's a research agenda, which has a lot of footnotes and references that are very useful. And then there's also a [blog post](https://medium.com/@deepmindsafetyresearch/building-safe-artificial-intelligence-52f5f75058f1) by the safety research team at DeepMind — they’ve compiled a set of resources to help folks get started in this space.

If you're particularly interested in going deeper, you’re always welcome to [email me](https://www.fhi.ox.ac.uk/team/jade-leung/).

How can we see the impact of AI strategy research _ Jade Leung _ EA Global - San Francisco 2019-by Centre for Effective Altruism-video_id 8M3nIu7GIsA-date 20190829.md


# Rohin Shah: What’s been happening in AI alignment?

Interviewee: Rohin Shah
Date: 2020-03-21

While we haven’t yet built aligned AI, the field of alignment has steadily gained ground in the past few years, producing many useful outputs. In this talk, Rohin Shah, a sixth-year PhD student at UC Berkeley’s Center for Human-Compatible AI (CHAI), surveys conceptual progress in AI alignment over the last two years.

While Rohin started his PhD working on program synthesis, he became convinced that it was important to build safe, aligned AI, and so moved to CHAI at the start of his fourth year. He now thinks about how to provide specifications of good behavior in ways other than reward functions. He is best known for the Alignment Newsletter, a popular weekly publication with content relevant to AI alignment.

Below is a transcript of Rohin's talk, which we've lightly edited for clarity. You can also watch it on YouTube and discuss it on the EA Forum.

# The Talk

Hi, everyone. My name is Rohin Shah. I'm a sixth-year PhD student at the Center for Human-Compatible AI at UC Berkeley. My research is generally on what happens when you try to do deep reinforcement learning in environments that involve humans. More broadly, I work on technical AI safety. I also write the Alignment Newsletter.

Today, I'll cover what's been happening in AI alignment. I should warn you: While this talk doesn't assume any technical knowledge of AI, it does assume basic familiarity with the arguments for AI risk.

I'll be surveying a broad swath of work rather than focusing on my personal interests. I'm hoping that this will help you figure out which parts of AI alignment you find exciting and would like to delve into more deeply.

A lot of the talk is based on a literature review I wrote a few months ago. You can find references and details in that review.

[Flow Chart of AI Alignment Landscape]

With that, let's get started. Taking a high-level, outside view, the reason that most people work on AI safety is that powerful AI systems are going to be a big deal. They're going to radically transform the world that we live in. Therefore, we should probably put some effort into making sure that this transformation goes well.

In particular, if AI systems are smarter than we are, then they could become the dominant force on the planet, which could be bad for us — in the same way that gorillas probably aren’t [thrilled] about how we have taken over all of their habitats. This doesn't necessarily mean that [AI will create] be an x-risk [existential risk]. It just means that we should have a sound technical reason to expect that the powerful AI systems we build are actually beneficial for us. And I would argue that we currently do not have such a reason. Therefore, the case for working on AI alignment is that we really should be creating this reason.

I want to note that there’s a lot of disagreement over specific sub-questions in AI safety. That will become more evident over the rest of this talk. But my impression is that virtually everyone in the field agrees with the basic, high-level argument [that we should have a good reason for expecting AI systems to be beneficial].

What are the specific risks we're worried about with AI? One issue is that humans aren't ready to deal with the impacts of AI. People tend to be in conflict a lot, and the US-China relationship is a big concern [in the AI community]. AI will enable better and better ways of fighting. That seems pretty bad. Maybe our fights will lead to bigger and bigger impacts; at some point, that could result in extinction-level events. Or perhaps AI leads to technological progress at such a fast pace that we’re unable to [adjust]. As a result, we could lock in some suboptimal values [that AI would act on for the rest of humanity’s future]. In both of these scenarios, the AI system wouldn’t intentionally cause x-risk, but it nonetheless would happen.

I'm not going to focus too much on this, but will note that some people are talking about preference aggregation. This is the idea that the AI system aggregates preferences across all stakeholders and does its thing — and then everyone agrees not to [oppose] the results. Similarly, we could try to [arrive at a] better metaphilosophy to avoid problems like value lock-in.

Another outside view that people take, aside from “AI is powerful and a big deal,” is that optimization leads to extreme outcomes. To take a very simple example, men in the US are, on average, about five feet, 10 inches tall. But very few basketball players, who are selected for height, are five feet, 10 inches. Most are well over six feet. When you select for something and have optimization pressure, you tend to get extreme outcomes. And powerful AI systems are going to be powerful optimizers. As a result, we probably shouldn't expect our everyday reasoning to properly account for what these optimizers will do.

Therefore, we need to [cultivate] more of a security mindset and look for arguments that quantify every possibility, as opposed to the average possibility. This mindset inspires researchers, especially at MIRI [the Machine Intelligence Research Institute], to try to understand how intelligence really works, so that we can then make well-designed AI systems that we understand. This has led to research on embedded agency, partial agency, and abstraction.

A bit about embedded agency: This is one of MIRI’s main research programs. The basic idea is that, according to the standard model of reinforcement learning and [our understanding of] AI more generally, an environment takes in actions and produces [observable phenomena] and rewards. Then, completely separate from the environment, an agent [observes these phenomena] and takes actions as a result. But that’s not how agents work. I’m an agent, yet I am not separate from the environment; I am a part of it. This leads to many philosophical problems. I would love to go into more detail, but don't have too much time. There's a great sequence on the AI Alignment Forum that I strongly recommend.

——

The next problem I want to talk about is one that I call “the specification problem.” It's also called “outer alignment.” Basically, the way we build AI systems right now is by assuming that we have some infallible specification of the optimal behavior in all possible situations, as though it were handed down to us from God. Then, we must figure out how to meet that specification. But of course, we can never actually get such a specification. The classic paperclip maximizer thought experiment shows that it's quite hard to specify the behavior of an AI making paperclips in a reasonable and sane way. This is also the main problem that Stuart Russell discusses in his book Human Compatible. Organizations [whose work includes addressing] this specification problem include CHAI, OpenAI, DeepMind, and Ought.

The main proposed way of solving the specification problem is to do some form of value learning. One thing I want to note: Value doesn't necessarily mean “normative value.” You don't necessarily need to be thinking about population ethics. For example, a robot that learned how to clean your room, and then reliably did so, would count as [an example of] value learning. Maybe we should be calling it “specification learning,” but value learning seems to be the name that has stuck.

The types of value learning include CIRL (or “assistance games”). CIRL stands for “cooperative inverse reinforcement learning.” This is a particular formalization of how you could approach value learning, in which the world contains a single human who knows the reward function — the true specification — but, for some reason, can't communicate that explicitly to the agent. There is also an agent whose goal is to infer what the human’s specification is, and then optimize for it. And because the agent no longer has a definite specification that it's trying to optimize, and it's instead uncertain over what it's trying to optimize, this results in many nice properties.

For example, the agent might ask you about what you want; it may try to clarify what your preferences are. If you try to shut it down, it will reason that it must have been doing a poor job of helping you. Therefore, it's going to allow you to shut it down, unlike a classic unexpected utility maximizer, which will say, “No, I'm not going to shut down, because if I am shut down, then I can't achieve my goal.”

The unfortunate thing about assistance games is that they are [exceptionally] computationally intractable. It's very expensive to solve a CIRL game. In addition, it requires a good model of how human preferences relate to human behavior, which — as many of the social sciences show — is a very difficult problem. And there is a theorem that says it is impossible to prove in the super-general case. Although, of course, we don't actually need the super-general case; we only need the case that applies in the real world. Instead of being impossible to prove, [the real-world case] is merely very, very difficult.

Next, we have [strategies based on agents] learning human intent. This is a broad category of possible communication protocols that a human could use to communicate the specification to the agent. So perhaps a human could demonstrate the optimal behavior to the agent, and from that, the agent could learn what it's supposed to do. (This is the idea behind inverse reinforcement learning and imitation learning.) Alternatively, perhaps the human could evaluate proposed hypothetical behaviors that the agent might execute, and then the agent could reason out what it should be doing.

Now we come to intent alignment, or “corrigibility.” This is somewhat different. While the previous approaches try to specify an algorithm that learns values, with intent alignment we instead build an agent that tries to do what we want it to do. Put another way, we're trying to bake into the agent the motivation to be helpful to us. Then, if we have an agent [whose sole motivation] is to be helpful to [a human], that will naturally motivate it to do many other things that we want. For example, it's going to try to clarify what my [travel] preferences are in the same way that a good personal assistant would, so that it doesn’t have to bother me when I ask it to book me a flight.

That covers a broad spectrum of approaches to value learning. However, there are still a few problems that arise. Intuitively, one big one is that, since the agent is learning from our feedback, it's not going to be able to do better than we can; it won’t be able to scale to superhuman performance. If we demonstrate the task to the agent, it won’t be able to perform the task any better than we could, because it’s receiving no information on how to [go about that]. Similarly, if we're evaluating the agent's behavior, it won't be able to find good behaviors that we wouldn't recognize as good.

An example is AlphaGo's move 37 [in its match against Go champion Lee Sedol]. That was a famous move that AlphaGo made, which no human ever would have made. It seemed crazy. I think it was assigned a less than one-in-10,000 chance of succeeding, and yet that move ended up being crucial to AlphaGo's success. And why could AlphaGo do this? Because AlphaGo wasn't relying on our ability to determine whether a particular move was good. AlphaGo was just relying on a reward function to tell it when it had won and when it had lost, and that was a perfect specification of what counts as winning or losing in Go. So ideally, we would like to build superintelligent AI systems that can actually exceed human performance at tasks, but it's not clear how we do this with value learning.

The key idea that allows current approaches around this is: Our AI systems are never going to exceed the supervision that we give them, but maybe we can train our AI systems to approximate what we would do if we had an extremely long time to think. Imagine I had 1,000 years to think about what the best thing to do was in a certain scenario, and then I shared that with an AI system — and then the AI system properly approximated my suggestion, but could do so in a few minutes as opposed to 1,000 years. That would presumably be a superintelligent AI.

The details for how we take this insight and arrive at an algorithm so that we can try it soon — not in 1,000 years — are a bit involved. I'm not going to go into them. But the techniques to look for are iterated amplification, debate, and recursive reward modeling.

Another problem with value learning is the informed oversight problem: Even if we're smarter than the agent that we're training, we won’t be able to effectively supervise it in the event that we don't understand why it chose a certain action. The classic example is an agent tasked to write a new novel. Perhaps it has access to a library where it's supposed to learn about how to write books, and it can use this in order to write the novel, but the novel is supposed to be new; [the task requires more than] just memorizing a novel from the library and spitting it back out again. It’s possible that the agent will look at five books in the library, plagiarize chunks from all of them, and put those together into a book that reads very nicely to us, but doesn't really solve the task because [the novel is unoriginal]. How are we supposed to tell the agent that this was bad? In order to catch the agent looking at the five books and stealing sentences from them, we'd have to read the entire library — thousands of books — and search for evidence of plagiarism. This seems too expensive for oversight.

So, it may be significantly more costly for us to provide oversight than it is for the agent to take actions if we cannot see how the agent is taking those actions. The key to solving this is almost obvious. It's simply to make sure you know how the agent is taking their actions. Again, there are many details on exactly how we think about this, but the term to look for is “ascription universality.” Essentially, this means that the supervisor knows everything that the agent knows, including any facts about how the agent chose its output.

[In the novel-writing example], if we were ascription-universal with respect to the agent, then we would know that it had taken sentences from five books, because the agent knows that. And if we knew that, then we could appropriately analyze it and tell it not to plagiarize in the future.

How do we create this property? Sadly, I'm not going to tell you, because again, I have limited time. But there's a great set of blog posts and a summary in the Alignment Newsletter, and all of those items are in my literature review. Really, I just want you to read that link; I put a lot of work into it, and I think it's good.

——

Let's move on to another top-level problem: the problem of mesa optimization. I'm going to illustrate mesa optimization with a non-AI example. Suppose you're searching for a Python program that plays tic-tac-toe well. Initially you find some programs that have good heuristics. Maybe you find a program that always starts at the center square, and that one tends to win a little more often than the others. Later, you find a program that makes sure that anytime it has two spots in a row and the third spot is empty, it plays in that third spot and wins. One that does that in a single step starts to win a bit more.

Eventually, you come across the minimax algorithm, which plays optimally by searching for the best action to take in every situation. What happened here was that in your search for optimal Python programs, you ended up finding a program that was itself an optimizer that searched possible moves in tic tac toe.

This is mesa optimization. You have a base [or “outer”] optimizer — in this case, the search over Python programs — and in the course of running that base optimizer, you find a new optimizer, which in this case is the minimax algorithm.

Why is this weird example about programs relevant to AI? Well, often we think about AI systems that are trained using gradient descent. And gradient descent is an optimization algorithm that searches over the space of neural net parameters to find some set of parameters that performs well on a loss function.

Let's say that gradient descent is the outer optimizer. It seems plausible that mesa optimization could happen even with gradient descent, where gradient descent finds an instantiation of the neural net parameters, such that then the neural net itself, when it runs, performs some sort of optimization. Then the neural net would be a mesa optimizer that is optimizing some objective, which we would call the mesa objective. And while we know that the mesa objective should lead to similar behavior as the original objective on the training distribution, because that's what it was selected to do, it may be arbitrarily different [outside] the training distribution. For example, if you trained it on tic tac toe, then you know it's going to win at tic tac toe — but if you switch to Connect Four, it might do something crazy. Maybe in Connect Four, it will continue to look for three in a row instead of four in a row, and therefore it will lose badly at Connect Four, even though it was working well with tic tac toe.

Let’s say that this happened with gradient descent, and that we had a very powerful, intelligent neural net. Even if we had solved the specification problem, and had the ideal reward function to train this agent, it might be that the neural net model that we come up with optimizes for a different objective, which may once again be misaligned with what we want. The outer-inner distinction is why the specification problem is called “outer alignment,” and why mesa optimization is called “inner alignment.”

How do people solve mesa optimization? There's one main proposal: adversarial training. The basic idea is that in addition to training an AI system that's trying to perform well on your specifications, you also have an adversary — an AI system or AI human team that's trying to find situations in which the agent you're training would perform badly, or would optimize for something other than the specification problem.

In the case where you're trying to get a corrigible AI system, maybe your adversary is looking for situations in which the AI system manipulates you or deceives you into thinking something is true, when it is actually false. Then, if you can find all of those situations and penalize the agent for them, the agent will stop behaving badly. You'll have an agent that robustly does the right thing across all settings. Verification would [involve using] that agent to verify another property that you care about.

Ideally, we would like to say, “I have formally verified that the agent is going to reliably pursue the specification that I outlined.” Whether this is possible or not — whether people are actually optimistic or not — I'm not totally clear on. But it is a plausible approach that one could take.

There are also other areas of research related to less obvious solutions. Robustness to distributional shift is particularly important, because mesa optimization becomes risky with distributional shift. On your training distribution, your agent is going to perform well; it's only when the world changes that things could plausibly go badly.

——

A notable thing that I haven’t talked about yet is interpretability. Interpretability is a field of research which entails trying to make sure that we understand the AI systems we train. The reason I haven't included it yet is because it's useful for everything. For example, you could use interpretability to help your adversary [identify] the situations in which your agent will do bad things. This helps adversarial training work better. But interpretability is also useful for value learning. It allows you to provide better feedback to the agent; if you better understand what the agent is doing, you can better correct it. And it's especially relevant to informed oversight or description universality. So while interpretability is obviously not a solution in and of itself, it makes other solutions way better.

There's also the option of trying to prevent catastrophes. Someone else can deal with whether the AI system will be useful; we're just going to stop it from killing everybody. Approaches in this area include impact regularization, where the AI system is penalized for having large impacts on the world. Some techniques are relative reachability and attainable utility preservation. The hope here would be that you could create powerful AI systems that can do somewhat impactful things like providing advice on writing new laws, but wouldn't be able to do extremely impactful things like engineer a pandemic that kills everybody. Therefore, even if an AI system were motivated to harm us, the impact penalty would prevent it from doing something truly catastrophic.

Another [area of impact regularization] is oracles. The idea here is to restrict the AI system's action space so that all it does is answer questions. This doesn't immediately provide safety, but hopefully it makes it a lot harder for an AI system to cause a catastrophe. Alternatively, you could try to box the AI system, so that it can’t have much of an impact on the world. One example of recent work on this is BoMAI, or boxed myopic artificial intelligence. In that case, you put both the human and the AI system in a box so that they have no communication with the outside world while the AI system is operating. And then the AI system shuts down, and the human leaves the box and is able to use any information that the AI system gave them.

So that's most of [the material] I’ll cover in this problem-solution format. There's also a lot of other work on AI safety and alignment that's more difficult to categorize. For example, there's work on safe exploration, adversarial examples, and uncertainty. These all seem pretty relevant to AI alignment, but it’s not obvious to me where, exactly, they fit in the graph [above]. So I haven't put them in.

There's also a lot of work on forecasting, which is extremely relevant to [identifying] which research agendas you want to pursue. For example, there has been a lot of disagreement over whether or not there will be discontinuities in AI progress — in other words, whether at some point in the future, AI capabilities shoot up in a way that we couldn't have predicted by extrapolating from past progress.

Another common disagreement is over whether advanced AI systems will provide comprehensive services. Here’s a very short and basic description of what that means: Each task that you might want an AI system to do is performed by one service; you don't have a single agent that's doing all of the tasks. On the other hand, you could imagine a single monolithic AI agent that is able to do all tasks. Which of these two worlds are we likely to live in?

A third disagreement is over whether it is possible to get to powerful AI systems by just increasing the amounts of compute that we use with current methods. Or do we actually need some deep insights in order to get to powerful AI systems?

This is all very relevant to deciding what type of research you want to do. Many research agendas only make sense under some possible worlds. And if you find out that one world [doesn’t seem very likely], then perhaps you switch to a different research agenda.

That concludes my talk. Again, here’s the link to the literature review that I wrote. There is both a short version and a long version. I really encourage you to read it. It goes into more detail than I could in this presentation. Thank you so much.

Rohin Shah_ WhatΓÇÖs been happening in AI alignment_-by EA Global Virtual 2020-date 20200321.md