# Cullen O'Keefe The Windfall Clause — sharing the benefits of advanced AI - EA Forum

_The potential upsides of advanced AI are enormous, but there's no guarantee they'll be distributed optimally. In this talk, Cullen O'Keefe, a researcher at the_ [_Centre for the Governance of AI_](https://www.fhi.ox.ac.uk/govai/)_, discusses one way we could work toward equitable distribution of AI's benefits — the Windfall Clause, a commitment by artificial intelligence (AI) firms to share a significant portion of their future profits — as well as the legal validity of such a policy and some of the challenges to implementing it._

_Below is a transcript of the talk, which we've lightly edited for clarity. You can also watch it on_ [_YouTube_](https://www.youtube.com/watch?v=vFDL-NxY610) _or read it on_ [_effectivealtruism.org_](https://effectivealtruism.org/articles/cullen-okeefe-the-windfall-clause-sharing-the-benefits-of-advanced-ai)_._

## The Talk

Thank you. Today I'll be talking about a research project that I'm leading at the Centre for the Governance of Artificial Intelligence (AI) on [the Windfall Clause](https://www.fhi.ox.ac.uk/windfallclause/).

![](https://images.ctfassets.net/ohf186sfn6di/7kcc3peZb8bYEL1pUQiECC/b48ff8b72b17f453bb282d39deed89f4/Slide03.png)

Many \[people in the effective altruism movement\] believe that AI could be a big deal. As a result, we spend a lot of time focusing on its potential downsides — so-called x-risks \[existential risks\] and s-risks \[systemic risks\].

![](https://images.ctfassets.net/ohf186sfn6di/4eXmYnvvmLwYMr4uaA7Yu9/a987aec8a0a5d4ca71f308d30a47a1a4/Slide04.png)

But if we manage to avoid those risks, AI could be a very good thing. It could generate wealth on a scale never before seen in human history. And therefore, if we manage to avoid the worst downsides of artificial intelligence, we have a lot of work ahead of us.

Recognizing this opportunity (and challenge), Nick Bostrom, at the end of [_Superintelligence_](https://www.amazon.com/dp/B00LOOCGB2/), laid out what he called "the common good principle" — the premise that advanced AI should be developed only for the benefit of all humanity.

![](https://images.ctfassets.net/ohf186sfn6di/79oCHgmBoiaUgAvtjEnrmL/9690e46375dea4ba3780045b66288923/Slide05.png)

It's in service of the common good principle that we've been working on the Windfall Clause at the Center for the Governance of AI.

![](https://images.ctfassets.net/ohf186sfn6di/29XkeBrIdolcFRiuyD0AUU/27bd9ac6d0e9059b863edee6c91eb7f0/Slide06.png)

We've been working on it for about a year \[as of summer 2019\], and I'd like to share with you some of our key findings.

I'll start by defining the project's goal, then describe how we're going to pursue that goal, and end by sharing some open questions.

![](https://images.ctfassets.net/ohf186sfn6di/5hTsi3YTedlfp5v9KRDBaK/b96397af25551134bc6ef8efc32c6e3b/Slide07.png)

**The goal of the Windfall Clause project**

![](https://images.ctfassets.net/ohf186sfn6di/5tgfQ6gFIkvYs6Lqq2q4ow/92bae0439bd5a48cbc17f7699f544741/Slide09.png)

Our goal with this project is to work toward distributing the gains from AI optimally. Obviously, this is both easier said than done and underdefined, as it invokes deep questions of moral and political philosophy. We don't aim to answer those questions \[at the time of this update\], though hopefully our friends from the [Global Priorities Institute](https://globalprioritiesinstitute.org/) will help us do that \[in the future\]. But we do think that this goal is worth pursuing, and one we can make progress on for a few reasons.

First, it's not a goal that we expect will be achieved naturally. The gains from the current global economy are distributed very unequally, as graphs like \[the one below indicate\].

![](https://images.ctfassets.net/ohf186sfn6di/4BYytNk9tF0wynorZRCP7/27c4bd60ff58cd6866105b592defa971/Slide10.png)

AI could further exacerbate these trends by primarily benefiting the world's wealthiest economies, and also by devaluing human labor. Indeed, industrialization has been a path to development for a number of economies, including the one in which we sit today.

![](https://images.ctfassets.net/ohf186sfn6di/4GZzJympTdubjy4q4Dx7vr/4ec3b92dcdccb6b07703e2810724fcfe/Slide11.png)

And by eroding the need for human labor with complementary technologies like robotics, AI could remove that path to development.

Industry structure is also very relevant. The advanced tech industries tend to be quite concentrated \[with a few large companies dominating most major markets\].

![](https://images.ctfassets.net/ohf186sfn6di/3uUKMoXOLlZpr8tf9Jv6NW/481bd60af93ccd989e60b82cd0396f88/Slide12.png)

A number of people have speculated that due to increasing returns on data and other input factors, AI could be a natural monopoly or oligopoly. If so, we should expect oligopoly pricing to take effect, which would erode social surplus, transferring it from consumers to shareholders of technology producers.

![](https://images.ctfassets.net/ohf186sfn6di/4pHp4813aN9k31W3M39Vi2/07f3c76ddcc52529693c97cf8578a071/Slide13.png)

Working toward the common good principle could serve as a useful way to signal that people in the technology fields are taking the benefits of AI seriously — thereby establishing a norm of beneficial AI development. And on an international level, it could credibly signal that the gains from cooperative (or at least non-adversarial) AI development outweigh the potential benefits of an AI race.

One caveat: I don't want to fall victim to the Luddite fallacy, which is the prediction throughout history that new technologies would erode the value of human labor, cause mass unemployment, and more.

![](https://images.ctfassets.net/ohf186sfn6di/4O6BTdn8QfmsNBhKJ8zpMQ/caad04682d2983738371793ccfbb61a2/Slide14.png)

Those predictions have repeatedly been proven wrong, and could be proven wrong again with AI. The answer will ultimately turn on complex economic factors that are difficult to predict _a priori_. So instead of making predictions about what the impacts of AI will be, I merely want to assert that there are plausible reasons to worry about the gains from AI.

**How we intend to pursue the Windfall Clause**

Our goal is to optimally distribute the gains from AI, and I've shared some reasons why we think that goal is worth pursuing. Now I'll talk about our mechanism for pursuing it, which is the Windfall Clause.

![](https://images.ctfassets.net/ohf186sfn6di/6Rr66dR1h923zA7uAhNsol/6f0f0a231439461d32a17d0622ad950f/Slide17.png)

In a phrase, the Windfall Clause is an ex ante commitment to share extreme benefits from AI. I'm calling it an "ex ante commitment" because it's something that we want \[companies working on AI\] to agree to before \[any firm\] reaches, or comes close to reaching, extreme benefits.

![](https://images.ctfassets.net/ohf186sfn6di/2mSnMtTz9TQ2uz4DIF0V0g/9dc307fcc42290273b0ba17e26c5d288/Slide20.png)

It's a commitment mechanism, not just an agreement in principle or a nice gesture. It's something that, in theory, would be legally binding for the firms that sign it. So ultimately, the Windfall Clause is about distributing benefits in a way that's closer to the goal of optimal distribution than we would have without the Windfall Clause.

A lot turns on the phrase "extreme benefits." It's worth defining this a bit more.

![](https://images.ctfassets.net/ohf186sfn6di/6q4Wgt96ODGmckXcxCOHko/8fc85bac679bfd5578f05b4467907bb8/Slide21.png)

For the purposes of this talk, the phrase is synonymous with "windfall profits," or just "windfall" generally. Qualitatively, you can think of it as something like "benefits beyond what we would expect an AI developer to achieve without achieving a fundamental breakthrough in AI" — something along the lines of AGI \[artificial general intelligence\] or [transformative AI](https://www.youtube.com/watch?v=9GxVIf3FNJk). Quantitatively, you can think of it as on the order of trillions of dollars of annual profit, or as profits that exceed 1% of the world's GDP \[gross domestic product\].

A key part of the Windfall Clause is translating the definition of "windfall" into meaningful, firm obligations. And we're doing that with something that we're calling a "windfall function." That's how we translate the amount of money a firm is earning into obligations that are in accordance with the Windfall Clause.

![](https://images.ctfassets.net/ohf186sfn6di/6FHjdQxvcRMd6KAYcUr31j/0bdf008f90c3bdce600458542f94cd1b/Slide22.png)

We wanted to develop a windfall function that was clear, had low expected costs, scaled up with firm size, and was hard for firms to manipulate or game to their advantage. We also wanted to ensure that it would not \[competitively\] disadvantage the signatories, for reasons I'll talk about later.

![](https://images.ctfassets.net/ohf186sfn6di/Iw6xUDyCBpa9bSQTi6E7c/272c67a607c31979bd0797f2f6c85787/Slide24.png)

To make this more concrete, you can think of \[the windfall function\] like this: When firm profits are at normal levels (i.e., the levels that we currently see), obligations remain low, nominal, or nothing at all. But as a firm reaches windfall profits, over time their obligations would scale up on the margin. It's like income taxes; the more you earn, the more on the margin you're obligated to pay.

Just as a side note, this particular example \[in the slide above\] uses a step function on the margin, but you can also think of it as smoothly increasing over time. And there might be strategic benefits to that.

The next natural question is: Is this worth pursuing?

![](https://images.ctfassets.net/ohf186sfn6di/dgmAewzmLDuibqX1mpI97/c1050c00466bd40c9ad386b314741400/Slide26.png)

A key input to that question is whether it could actually work. Since we're members of the effective altruism community, we want \[the change we recommend to actually have an effect\].

![](https://images.ctfassets.net/ohf186sfn6di/6ZiavrVJKR1GvqBCBlNFS6/cb580d87907ae6857d4ff2b1b74f787f/Slide27.png)

One good reason to think that the Windfall Clause might work is that it's a matter of corporate law. This was a problem that we initially had to confront, because in American corporate law, firm directors — the people who control and make decisions for corporations — are generally expected to act in the best interests of their shareholders. After all, the shareholders are the ones who provide money to start the firm. At the same time, the directors have a lot of discretion in how they make decisions for the firm. The shareholders are not supposed to second-guess every decision that corporate directors make. And so traditionally, courts have been quite deferential to firm boards of directors and executives, applying a very high standard to find that they have violated their duties to shareholders. In addition, corporate philanthropy has traditionally been seen as an acceptable means of pursuing the best interests of shareholders.

In fact, in all seven cases in which shareholders have challenged firms on their corporate philanthropy, courts have upheld it as permissible. That's a good track record. Why is it permissible if firm directors are supposed to be acting in the best interest of shareholders? Traditionally, firms have noted that corporate philanthropy can bring benefits like public relations value, improved relations with the government, and improved employee relations.

![](https://images.ctfassets.net/ohf186sfn6di/4EdU6AE3FqJxUv0uAbBfDN/ba76bf3e4033a33d7e864904ca8ee75d/Slide32.png)

The Windfall Clause could bring all of these as well. We know that there is increasing scrutiny \[of firms' actions\] by the public, the government, and their own employees. We can think of several different examples. Amazon comes to mind. And the Windfall Clause could help \[advance this type of scrutiny\]. When you add in executives who are sympathetic to examining the negative implications of artificial intelligence, then there's a plausible case that they would be interested in signing the Windfall Clause.

![](https://images.ctfassets.net/ohf186sfn6di/53Q1e3vbW6WIC5fkZmJiuL/f2889d1971b3e4721b054943d91ccc26/Slide33.png)

Another important consideration: We think that the Windfall Clause could be made binding as a matter of contract law, at least in theory. Obviously, we're thinking about how a firm earning that much money might be able to circumvent, delay, or hinder performing its obligations under the Windfall Clause. And that invokes questions of internal governance and rule of law. The first step, at least theoretically, is making the Windfall Clause binding.

**Open questions**

We've done a lot of work on this project, but many open questions remain.

![](https://images.ctfassets.net/ohf186sfn6di/Y7ozpLEeLYW76N4tVChEM/d1b12062bb586dcb8e6177352d9b2507/Slide35.png)

Some of the hard ones that we've grappled with so far are:

![](https://images.ctfassets.net/ohf186sfn6di/1A8ijdO9Hg2gMywh9iqepd/eed74267e5848f0598d3ca35e90d9697/Slide37.png)

**1\. What's the proper measure of "bigness" or "windfall"?** I previously defined it in relation to profits, but there's also a good case to be made that market cap is the right measure of whether a firm has achieved windfall status, since market cap is a better predictor of a firm's long-term expected value. A related question: Should windfall be defined relative to the world economy or in absolute terms? We've made the assumption that it would be relative, but it remains an open question.

![](https://images.ctfassets.net/ohf186sfn6di/6hsvrKsUk5z1kOoI5f4ZVl/2a55fb05fb0297c0f5344ce72eb13f38/Slide39.png)

**2\. How do we make sure that the Windfall Clause doesn't disadvantage benevolent firms, competitively speaking?** This is a more important question. It would be quite bad if multiple firms were potential contenders for achieving AGI, but only the most benevolent ones signed onto the Windfall Clause. They'd be putting themselves at a competitive disadvantage by giving themselves less money to reinvest \[if they activated the Windfall Clause\], or making themselves less able to attract capital to invest in that goal \[because investor returns would probably be lower\]. Therefore, it would become more likely that amoral firms would achieve windfall profits or AGI \[before benevolent firms\]. We've had some ideas for how to prevent this \[see slide below\], and think these could go a long way toward solving the problem. But it's unclear whether these ideas are sufficient.

Those are questions that we have explored.

![](https://images.ctfassets.net/ohf186sfn6di/19aVwZZp56nndOGfDAOcaa/daee458a0d3c7b7557277e92a8098eec/Slide40.png)

Some questions that are still largely open — and that we intend to continue to pursue throughout the lifetime of this project — include:

**1\. How does the Windfall Clause interact with different policy measures that have been proposed to address some of the same problems?**

![](https://images.ctfassets.net/ohf186sfn6di/5zvKx0HuSyNt2wIbVjhplz/04d98c49adce9ae03a14164de19e54fe/Slide41.png)

**2\. How do we distribute the windfall?** I think that's a bigger question, and a pretty natural question for us to ponder as EAs. There are also some related questions: Who has input into this process? How flexible should this be to input at later stages in the lifetime of the Clause? How do we ensure that the windfall is being spent in accordance with the common good principle?

![](https://images.ctfassets.net/ohf186sfn6di/1NMK1LaaSIWJWLG6Vtqqes/fdb8bb272a4d46c279838e84a57e8c2f/Slide45.png)

Luckily, as members of the EA community, we have a lot of experience answering these questions through charity design and governance. I think that if this project goes well and we decide to pursue it further, this could be a relevant project for the EA community as a whole to undertake — to think about how to spend the gains from AI.

![](https://images.ctfassets.net/ohf186sfn6di/7s4xoF2XhdvRULjEnODCZN/278542e708a3d54ae19014a9db66bcb5/Slide46.png)

Accordingly, the Centre for the Governance of AI, in collaboration with partners like [OpenAI](https://openai.com/) and the [Partnership on AI](https://www.partnershiponai.org/), intends to make this a flagship policy investigation. It is one in a series of investigations on the general question of how to ensure that the gains from AI are spent and distributed well.

![](https://images.ctfassets.net/ohf186sfn6di/2nsLefzZnfeo2UGz7wJK2L/40cbfd2d30abf1d0b77bf117b08de958/Slide47.png)

In closing, I'd like to reiterate the common good principle and make sure that when we think about the potential downside risks of AI, we don't lose sight of the fact that \[a safe and powerful AI could address many of the world's problems\]. \[Keeping this fact in sight\] is a task worth pursuing on its own.

![](https://images.ctfassets.net/ohf186sfn6di/6VjKdUaaxKVUiVxNcdE6Mv/0ccb5b4b749d77671a1e3fe433691b05/Slide48.png)

If you're interested in learning more about this project and potentially contributing to it, I invite you to [email me](mailto:<EMAIL>). Thank you so much.

**Moderator:** Thanks for your talk. I think this is an \[overlooked\] area, so it is illuminating to \[hear about\]. Going back to the origins of the project, did you find historical case studies — other examples where companies came into a lot of wealth or power, and tried to distribute it more broadly?

**Cullen:** Yeah. We found one interesting one, which is in what is now the Democratic Republic of the Congo. When that area was under Belgian colonial rule, a mineral company came into so much wealth from its extractive activities there that it felt embarrassed about how much money it had. The company tried to do charitable work in the Congo with the money as a way of defusing tensions.

I don't think that turned out well for the company. But it is also quite common for firms throughout the world to engage in corporate social responsibility campaigns in the communities in which they work, as a means of improving community relations and ultimately mitigating the risk of expropriation, or activist action, or adverse governmental actions of other sorts. There's a range of cases — from those that are very analogous to more common ones.

**Moderator:** What mechanisms currently exist in large companies that are closest to the kind of distribution you're thinking about?

**Cullen:** A number of companies make commitments that are contingent on their profit levels. For example, Paul Newman has a line of foods. They give all of their profits to charity; it's \[Newman's\] charitable endeavor. That's a close analogy to this.

It's also quite common to see companies making commitments along the lines of a certain percentage of a product's purchase going to charity. That's similar, although it doesn't involve the firm's relative profit levels.

It's not super common to see companies make commitments contingent on profit levels. But OpenAI just restructured into a capped-profit model. That's somewhat similar to \[what we're proposing in the Windfall Clause project\]. They're giving all profits above a certain level to the nonprofit that continues to govern them.

**Moderator:** You mentioned toward the end of your talk that there are some legal binding commitments that a company could be held to, assuming that they decided to enter into this sort of social contract. But you could also imagine that they become so powerful from having come into so many resources that the law is a lot weaker. Can you say a little bit more on the mechanisms you have in mind that might be used to hold their feet to the fire?

**Cullen:** Yeah, absolutely. I think this is an outstanding problem in AI governance that's worth addressing not just for this project, but in general. We don't want companies to be above the law once they achieve AGI. So it's worth addressing for reasons beyond just the Windfall Clause.

There are very basic things you could do, like have the Windfall Clause set up in a country with good rule of law and enough of a police force to plausibly enforce the clause. But I don't expect this project to resolve this question.

Another point along the same lines is that this involves questions of corporate governance. Who in a corporation has the authority to tell an AGI, or an AGI agent, what to do? That will be relevant to whether we can expect AGI-enabled corporations to follow the rule of law. It also involves safety questions around whether AI systems are designed to be inherently constrained by the rule of law, regardless of what their operators tell them to do. I think that's worth investigating from a technical perspective as well.

**Moderator:** Right. Have you had a chance to speak with large companies that look like they might come into a windfall, as a result of AI, and see how receptive they are to an idea like this?

**Cullen:** We haven't done anything formal along those lines yet. We've informally pitched this at a few places, and have received generally positive reactions to it, so we think it's worth pursuing for that reason. I think that we've laid the foundation for further discussions, negotiations, and collaborations to come. That's how we see the project at this stage.

Presumably, the process of getting commitments (if we want to pursue it that far) will involve further discussions and tailoring to the specific receptiveness of different firms based on where they perceive themselves to be, what the executives' personal views are, and so forth.

**Moderator:** Right. One might think that if you're trying to take a vast amount of resources and make sure that they get distributed to the public at large — or at least distributed to a large fraction of the public — that you might want those resources to be held by a government whose job it is to distribute them. Is that part of your line of thinking?

**Cullen:** Yeah. I think there are reasons that the Windfall Clause could be preferable to taxation. But that is also not to say that we don't think governments should have a role in input for democratic accountability, and also pragmatic reasons to avoid nationalization.

One general consideration is that tax dollars tend not to be spent as effectively as charitable dollars, for a number of reasons. And, for quite obvious reasons involving stakeholder incentives, taxes tend to be spent primarily on the voters of that constituency. On the other hand, the common good principle, to which we're trying to stick with this project, demands that we distribute resources more evenly.

But as a pragmatic consideration, making sure that governments feel like they have influence in this process is something that we are quite attentive to.

**Moderator:** Right. And is there some consideration of internationalizing the project?

**Cullen:** Yeah. One thing that this project doesn't do is talk about control of AGI. And one might reasonably think that AGI should be controlled by humanity collectively, or through some decision-making body that is representative of a wide variety of needs. \[Our project\] is more to do with the benefits of AI, which is a bit different from control. I think that's definitely worth thinking about more, and might be very worthwhile. This project just doesn't address it.

**Moderator:** As a final question, the whole talk is predicated on the notion that there would be a windfall from having a more advanced AI system. What are the circumstances in which you wouldn't get such a windfall, and all of this is for naught?

**Cullen:** That's definitely a very good question. I'm not an economist, so what I'm saying here might be more qualitative than I would like. But if you're living in a post-scarcity economy, then money might not be super relevant. But it's hard to imagine this in a case where corporations remain accountable to their shareholders. Their shareholders are going to want to benefit in some way, and so there's going to have to be some way to distribute those benefits to shareholders.

Whether that looks like money as we currently conceive it, or vouchers for services that the corporation is itself providing, is an interesting question — and one that I think current corporate law and corporate governance are not well-equipped to handle, since money is the primary mode of benefit. But you can think of the Windfall Clause as capturing other sorts of benefits as well.

A more likely failure mode is just that firms begin to primarily structure themselves to benefit insiders, without meaningful accountability from shareholders. This is also a rule-of-law question. Because if that begins to happen, then the normal thing that you expect is for shareholders to vote out or sue the bad directors. And whether they're able to do that turns on whether the rule of law holds up, and whether there's meaningful accountability from a corporate-governance perspective.

If that fails to happen, you could foresee that the benefits might accrue qualitatively inside of corporations for the corporate directors.

**Moderator:** Right. Well, on that note, thank you so much for your talk.

**Cullen:** Thank you.


# Markus Anderljung and Ben Garfinkel Fireside chat on AI governance - EA Forum

**Getting involved in AI governance**

**Markus:** Ben, how did you get into the field of AI governance?

**Ben:** I majored in physics and philosophy at Yale, and was considering working in the philosophy of physics. I started thinking about it not being the most useful or employable field. At the same time, I got interested in EA \[effective altruism\]. Then, Allan Dafoe was transitioning to focus on a new Centre for the Governance of AI. He was looking for researchers. I seized the opportunity — it seemed important, with not enough people working in \[the field\]. That's how I got involved.
**Markus:** What happened next?

**Ben:** I was a Research Assistant there for about a year, and, at the same time, held a job at the Centre for Effective Altruism. I then had the opportunity to transition to the AI governance area full-time.

**Markus:** Sounds a bit random — just having this opportunity pop up.

**Ben:** There was indeed an element of randomness, but it wasn't a random thing to do. I got really interested in long-termism and EA, so AI governance was on my radar — and yes, an opportunity lined up at the same time.
**Markus:** How has your work in the field changed?

**Ben:** Quite a lot. I still have a broad focus in the area, but when I started, there was this sense that AI was going to be a very important field. It was around 2016. [AlphaGo](https://deepmind.com/research/case-studies/alphago-the-story-so-far) had just come out. [_Superintelligence_](https://www.amazon.com/dp/B00LOOCGB2/) had been written, so a good fraction of long-termist concern was on AI. AI seemed to be this really transformative technology, with risks we didn't understand very well yet.

Almost no one was working on AI safety at the time or thinking about long-term AI governance challenges, and not that much AI governance was going on. So the early questions were "What is going on here?" and "What are we trying to do?" A lot of the early research was probably more naive, with people transitioning to the field and not knowing that much about AI.

**Markus:** And you're now doing a DPhil at Oxford?

**Ben:** Yes, I just started \[working toward a DPhil\] in international relations. My highest credential so far in the area \[of AI governance\] remains an undergraduate degree in an unrelated field, so it seemed useful to have a proper degree in a domain more relevant to governance than physics.

Let's turn the question on you, Markus. How did you get involved?

**Markus:** It's a slightly convoluted story. I was first involved in EA issues at university, around 2011-2013. I got involved with [Giving What We Can](https://www.givingwhatwecan.org/) and \[other EA\] organizations. Then I spent a few years transitioning from \[the belief that\] long-termism — or the closest \[approximation\] of it back then — was true, but there wasn't much to do about it in the emerging technology field, to becoming increasingly convinced that there _were_ things to do.

When I graduated from university in 2016, I moved back to Sweden. I considered building career capital, so I went into management consulting for a few years, which was very fun and interesting. After a while it felt that the trajectory had leveled off, and that I could do more in the cause areas I cared about.

I transitioned into work for EA Sweden. Building a community seemed like a good idea, and heading up the organization was a great opportunity. I got funding for that and did it for about a year. I then became convinced that I could do more with my career outside of Sweden.

I was looking for other options, thinking that my comparative advantage was "I'm a management consultant who understands the research, someone who _gets it_." I applied to organizations where I thought my broader skill set of operations and recruitment could help. GovAI was one of those organizations.

It wasn't like AI governance was the most important thing and I had to help there; it was just part of a broader class of areas that seemed useful.

**The relative importance of AI governance research**

**Markus:** Let's talk about your recent research. At EA Global: London 2018, you gave a talk.

**Ben:** "[How sure are we about this AI stuff?](https://www.effectivealtruism.org/articles/ea-global-2018-how-sure-are-we-about-this-ai-stuff/)"

**Markus:** Right. So, I want to ask: How sure are we about this AI stuff?

**Ben:** There are two ways to be sure. One is robust. \[It centers on the question\] "If we had all of the facts and considerations, would this still be a top priority for EA?" The other \[centers on the question\] "Given our limited information and the expectations we have, are we sure it still makes sense to put a lot of resources into this area?"

With the first question, it's really hard to be sure. There's so much we don't know: what future AI systems will look like, the rate of progress, which institutions will matter, the timelines. That's true of any transformative technology; we don't have a great picture.

**Markus:** Will it be different for AI versus other causes?

**Ben:** If you compare AI to climate change, there's a lot of uncertainty in climate models. We don't know everything about the feedback loops, or how to think about extreme events — is there a one-in-100 probability or a one-in-1,000 probability? We still have a basic sense of the parameters of the problem, such as how hot things will get (to some extent) and how bad it is.

With AI, if human labor becomes essentially unnecessary in the long term, as it's replaced by AI systems, we don't know what that world looks like or how it's organized. It's very difficult to picture. It's like being in the 1500s and describing the internet in very rough terms, as \[something that will be\] more efficient \[and enable\] faster communication and information retrieval. You could have reasoned a bit about this future — maybe there would be larger businesses, since you can communicate on a larger scale. But you wouldn't be visualizing the internet, but rather very fast carrier pigeons or something. You're going to be way off. It's hard to even find single dimensions where the answer is clear.

I think that's about where we are with AI, which is a long-winded way of saying that it's hard to be sure.

I actually feel pretty good about the weaker standard ("How sure are we, given these considerations, that a decent chunk of the EA movement should be focused on this?"). Overall, I think a smaller portion of the EA portfolio should be focused on this, but at least a few dozen people should be actively thinking about long-term applications of AI, and we're not far from that number at the moment.

**Markus:** That sounds smaller than I expected. When you say a smaller portion of the EA portfolio should be focused on AI, what's your current ballpark estimate of what that percentage is?

**Ben:** I think it's really hard to think about the spread between things. Maybe it's something along the lines of having one in five people who are fully engaged and oriented on the long term thinking about AI. That would be great.

**Markus:** Whereas now, you think the number is more like three or four in five people?

**Ben:** Yeah, it feels to me that it might be more than half, but I'm not sure that's correct.

**Markus:** What interventions would you like these people to do instead?

**Ben:** There's a lot of uncertainty. A lot of it is based on the skill set that these people have and the sort of work they'd be inclined to do. There being a lot of math and computer science \[in EA\] may justify the strong AI focus, but let's imagine completely fungible people who are able to work on anything.

I really think fundamental cause prioritization research is still pretty neglected. There's a lot of good work being done at the [Global Priorities Institute](https://globalprioritiesinstitute.org/). There are some broad topics that seem relevant to long-term thinking that not many people are considering. They include questions like those I was working on: "Should we assume that we don't have really great opportunities to influence the future now, relative to what future people might have if we save our money?" and "Should we pass resources on?" These seem crucial for the long-termist community.

**The importance and difficulty of meta-level research**

**Ben:** Even within AI, there are strangely not that many people thinking about, at the meta-level, the pathways for influence in AI safety and governance. What exactly is the nature of the risk? I think there's a handful of people doing this sort of work, part-time, on the side of other things.

For example, Rohin Shah is doing a lot of good \[by thinking through\] "What exactly is the case for AI risk?" But there are not that many people on that list compared to the set of people working on AI safety. There's an abstract argument to be made: Before you put many resources into object-level work, it's quite useful, early on, to put them toward prioritizing different kinds of object-level work, in order to figure out what, exactly, is motivating the object-level work.

**Markus:** One of your complaints in \[your past\] talk was that people seemed to be putting a lot of research into this topic, but haven't produced many proper writeups laying out the argument motivating people's choices. Do you think we've seen improvement there? There are a few things that have been published since then.

**Ben:** Yeah, there's definitely been an improvement since I gave the talk. I think the time at which I gave the talk was kind of a low point.

There was an initial period of time, after the publication of _Superintelligence_, when the motivation for AI governance, for a lot of people, corresponded to the nature of AI risk. Over time, there was some sort of transition; people have very different visions of this. The change happened along a lot of dimensions. One of them is that _Superintelligence_ focuses on a very discrete transition to advanced AI, in which not much is happening, and then we transition to quite advanced systems in a matter of days or weeks. A lot of people moved away from that.

Also, a lot of people, myself included, started thinking about risks that weren't specifically safety-oriented. _Superintelligence_ discusses these but they're not the main focus.

**Markus:** What do you mean by "not safety-oriented"?

**Ben:** There's a lot of concern you might have about the future of AI not necessarily being great. For example, in a scenario in which human labor and government functions have been automated, it may not be a great world in terms of democracy and representation of the will of the people.

Another category is ethical \[and concerns\] the moral status of AI systems. Maybe those decisions are made wrongly, or are substantial deviations from the best possible case.

**Markus:** So these are risks that don't \[involve\] accidents with very powerful systems.

**Ben:** We've had major technological transitions in history which haven't been uniformly good. The classic one is the Neolithic Revolution — the introduction of agriculture — having a few aftereffects like the rise of the state. It's difficult to do an overall assessment. Some of the results were positive, and some were very much not, like slavery becoming a massive institution, disease, and the establishment of hierarchies instead of decentralized decision making.

It's not hard to imagine that if there's another transition, in which human labor is replaced by capital, that \[transition\] may have various effects that aren't exactly what we want.

**Markus:** Yes, and in these previous transitions, the bad consequences have been permanent structural effects, like slavery being more economically viable.

So \[the time of your EA Global talk\] — November 2018 — was the low point? In what sense?

**Ben:** It was the low point in the sense of people having changed their justifications quite a bit in a lot of different areas, \[without those changes being\] reflected in much published writing \[other than\] maybe some blog posts.

There hasn't been a massive improvement, but there definitely has been useful stuff published since then. For example, Paul Christiano wrote a [series of blog posts](https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd) arguing for AI safety even in the context of a continuous transition; Richard Ngo did [some work](https://www.alignmentforum.org/posts/JbcWQCxKWn3y49bNB/disentangling-arguments-for-the-importance-of-ai-safety) to taxonomize different arguments and lay out \[current thinking in\] the space; Tom Sittler did [similar work](https://fragile-credences.github.io/prioritising-ai/); and Rohin Shah presented a case for AI risk in a good sequence called "[Value Learning](https://www.alignmentforum.org/s/4dHMdK5TLN6xcqtyc)," a series of essays that laid out the nature of the alignment problem.

I think that was after —

**Markus:** "[Reframing Superintelligence](https://www.fhi.ox.ac.uk/reframing/)"?

**Ben:** Yeah, Eric Drexler's work at FHI \[the Future of Humanity Institute\] also came out framing his quite different picture of AI progress and the nature of the risks. Also MIRI \[the Machine Intelligence Research Institute\] put out [a paper](https://intelligence.org/embedded-agency/) on what they call "mesa-optimization," which corresponds to one of their main arguments for why they're worried about AI risk, and which wasn't in _Superintelligence_.

There have been a decent amount of publications, but quite a bit fewer that I would ideally want. There are still a lot of viewpoints that aren't captured in any existing writing, and a lot of writing is fairly short blog posts. Those are useful, but I'm not very comfortable with putting a lot of time and work into an area where justifications are short blog posts.

It's obviously very difficult to communicate clearly about this. We don't have the right picture of how things will go. It's not uncommon to have arguments about what a given post is actually saying, which is not a great signal for our community being on the same page about the landscape of arguments and considerations.

**Markus:** Why do you think this is? Is it due to individual mistakes? Not spending enough time on this meta-level question?

**Ben:** To some extent, yes. There are complications. Working on this stuff is fairly difficult. It requires an understanding of the current landscape, of arguments in this area, of what's going on in AI safety and in machine learning. It also requires the ability to do conceptual analysis and synthesis. There are perhaps not that many people right now \[who meet these criteria\].

Another complicating factor is that most people currently working in this area have just recently come into it, so there's an unfortunate dynamic where people have the false sense that the problem framing is a lot more \[advanced\] than it actually is — that it just hasn't been published, and that the people in the know have a good understanding.

When you enter an area, you aren't usually in a great position to do this high-level framing work, because you don't really know what exists in terms of unpublished Google Docs. It's quite easy, and maybe sensible, when entering the area, to not do high-level \[thinking and writing\], and instead pick an object-level topic to work on.

Some of us might be better off dropping the object-level research program and \[addressing\] more high-level problems. Some have been doing this in their spare time, while their \[main area of study\] is object-level. It does seem like a difficult transition: to stop an object-level project and embark on a loose, "what-are-we-even-doing" project.

**Markus:** Are there particular viewpoints that you feel haven't been accurately represented, or written up thoroughly enough?

**Ben:** Paul Christiano, for example, has written a few blog posts. One is called "[What Failure Looks Like](https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-failure-looks-like)." It shows what a bad outcome would look like, even in a scenario with a slow, gradual transition. However, there's a limit on how thoroughly you can communicate in the form of a blog post. There is still a lot of ambiguity about what is being described — an active disaster? A lost opportunity? What is the argument for this being plausible? There's a lot more that could be done there.

I feel similarly about this idea of mesa-optimization, which is now, I think, one of the primary justifications that MIRI has for assigning a high probability to AI safety risk. I saw a lot of ambiguity around what this concept exactly is. Maybe different people are trying to characterize it differently or are misunderstanding the paper, or the paper is ambiguous. It doesn't seem like everyone is on the same page about what exactly mesa-optimization is. The paper argues that there might be this phenomenon called mesa-optimization, but doesn't try to make the argument that, because this phenomenon might arise, then we should view it as a plausible existential risk. I think that work still hasn't been done.

**Markus:** So the arguments that _are_ out there ought to be scrutinized more. Are there arguments or classes of viewpoints that you feel don't even have an initial writeup?

**Ben:** I think there are a couple. For example, Allan \[Dafoe\] is thinking quite a lot about structural risks. Maybe the situation starts getting bad or disappointing in terms of our current values. It's a bit nebulous like the Neolithic Revolution — not a concrete disaster. Some structural forces could push things in a direction you ideally wouldn't want.

Similarly, there's not much writing on whether AI systems will eventually have some sort of moral status. If they do, is that a plausible risk, and one that will be important enough for longtermists to focus on?

Those are probably two of the main things that stand out in my mind as plausible justifications for focusing on AI, but where I can't point to a longer writeup.

**Markus:** What if I were to turn this around on you? You're here, you're working on these issues. What is your stab at a justification?

**Ben:** The main point is that it's very likely that AI will be really transformative. We will eventually get to the point where human labor is no longer necessary for most things, and that world will look extremely different.

Then, there are around a half-dozen vaguely sketched arguments for why there might be some areas with EA leverage that would make the future much better or much worse — or why it could go either way.

It is hard to \[determine\] which topics may have long-term significance. I don't think they \[comprise\] a large set, and there's value in \[surfacing\] that information right now, in getting clearer on "what's going on in AI." It seems like one of the few places where there's currently the opportunity to do useful, \[future-focused\] work.

**Markus:** So the argument is: "If you're a long-termist, you should work in the areas that hold great leverage over the future, and this seems like one of the best bets."

**Ben:** Yeah, that's basically my viewpoint. The influence of historical events is extremely ambiguous. How plausible is it for us to know what our impact today will have 100 years from now? In the 1300s, people's focus may have been on stopping Genghis Khan. I think that would have been ethically justified, but from a long-termist perspective, things are less clear. Genghis Khan's impact may have ultimately been good because of the trade networks established and \[other such factors\]. We're unable to discern good from bad from a long-termist perspective. Pick any century from more than five centuries ago, and you'll be in the same position.

I think we should have a strong prior that justifies working on issues for their present influence, but for the long-term view, we shouldn't prioritize issues where we can't predict what difference they'll make hundreds of years in the future.

There are not many candidates for relevant long-term interventions. Insofar as there are semi-plausible arguments for why AI could be one of them, I think it's more useful to put resources into figuring out what is going on in the space and \[improving\] the value of information, rather than putting resources into object-level issues.

**The role of GovAI**

**Ben:** So, Markus, could you tell me what GovAI is currently up to in this space?

**Markus:** Broadly, we're working on AI governance questions from a long-termist perspective, and I spend most of my time doing what I can to make sure we build an organization that does the best research.

In practice, we have a lot of different research projects going on. I personally spend a lot of time with recruiting, growing the organization. We run a [GovAI Fellowship](https://www.fhi.ox.ac.uk/governance-of-ai-fellowship/), where people spend three months doing research on topics that relate to the kinds of things we're interested in. That's a path into the AI governance space, and something we'll continue doing for the foreseeable future. We \[award\] 10 fellowships every year. We'll see whether we'll be able to bring people \[on-site\] this summer. I'm pretty excited so far about this as a way of getting people into the field.

Since 2016, we've not been able to build up institutions in the field that provide clear pathways for people. I think this fellowship is one example of how we can do that.

My hope is to have, a few years down the line, a team of a dozen great researchers in Oxford doing great research. In terms of the research that we'll do, we're \[an unusual\] organization, in that we could define the very broad problem of AI governance as "everything required to make AI go well that isn't technical AI safety." That's a _lot_.

It will span fields ranging from economics, to law, to policy — a tremendous number of different topics. I'm hoping that, over time, we'll build narrower expertise as we get clearer on the meta picture and the specific fields that people need to work on.

A few years down the line, I'd really like for us and others in this space to have at least somewhat solid \[recommendations\] in \[situations\] like a corporation asking what their publication norms should be, or what sorts of internal mechanisms they should have to make sure they're held accountable to the beautiful principles they've written up (e.g., "benefit humanity with our research").

I don't think we have that yet. Those are the kinds of questions I'm hoping we can make progress on in the next few years.

**Career recommendations**

**Ben:** Besides just applying for the GovAI Fellowship, do you have other career recommendations for people interested in \[entering or exploring whether to enter\] this space?

**Markus:** In general, I \[subscribe to the view\] that if you're trying to figure out if you should be doing something, then do a bit of it. Find some bit of research that you can do and try it.

There aren't a lot of opportunities that look like the GovAI Fellowship in the long-termist AI governance space, but others that are similar are at the [Center for Security and Emerging Technology](https://cset.georgetown.edu/), based in Washington, DC. There are also junior roles at DeepMind and OpenAI, where you might do \[some exploratory\] research. But there aren't many such roles — probably fewer than a dozen a year.

I would encourage people to think much more broadly. You might try to work at a wider set of technology companies like Microsoft or Facebook as they start building up ethics and policy teams. It would be awesome to have people in these types of "council" roles.

Another good idea would be to use your studies to dip your toe into the water. You could do your bachelor's or master's dissertation on a relevant topic. Some are looking into PhDs in this area as well, and that may be a good way to expand the field.

The other main tip is to engage with a lot of the research. Read everything coming out of institutions like ours, or [CSET](https://cset.georgetown.edu/) \[the Center for Security and Emerging Technology\]. Try to really engage with it, keeping in mind that the authors may be smart and good at what they do, but aren't oracles and don't have that much knowledge. There's a lot of uncertainty in this space, so read things with an open mind. Consider what may be wrong, stay critical, and try to form your own views instead of directly \[adopting\] other people's conclusions. Form your own model of how you think the world could go.

**Ben:** Sounds good. Very wise.

**Markus:** Do you have any tips? What would you have told your past self?

**Ben:** I can't think of anything beyond what you just said, other than to have checked my email in case Allan Dafoe was looking for research assistants in this area, with a fairly non-competitive process at that time. Not many people were interested in it back then.

**Markus:** Right, so _get really lucky_.

**Ben:** Get really lucky — that's exactly what I would tell my past self.

**Markus:** Cool.

**Ben:** Well, it's been fun chatting!

**Markus:** Yes, and I'll add my new sign-off: Stay safe, stay sane, and see you later!