# Ethan On Why Scale is All You Need on The Inside View Podcast

Interviewee: Ethan Caballero
Date: 2022-05-05

Ethan is known on Twitter as the edgiest person at MILA. We discuss all the gossips around scaling large language models in what will be later known as the Edward Snowden moment of Deep Learning. On his free time, Ethan is a Master's degree student at MILA in Montreal, and has published papers on out of distribution generalization and robustness generalization, accepted both as oral presentations and spotlight presentations at ICML and NeurIPS. Ethan has recently been thinking about scaling laws, both as an organizer and speaker for the 1st Neural Scaling Laws Workshop.

Outline

- 00:00 highlights

- 00:50 who is Ethan, scaling laws T-shirts

- 02:30 scaling, upstream, downstream, alignment and AGI

- 05:58 AI timelines, AlphaCode, Math scaling, PaLM

- 07:56 Chinchilla scaling laws

- 11:22 limits of scaling, Copilot, generative coding, code data

- 15:50 Youtube scaling laws, constrative type thing

- 20:55 AGI race, funding, supercomputers

- 24:00 Scaling at Google

- 25:10 gossips, private research, GPT-4

- 27:40 why Ethan was did not update on PaLM, hardware bottleneck

- 29:56 the fastest path, the best funding model for supercomputers

- 31:14 EA, OpenAI, Anthropics, publishing research, GPT-4

- 33:45 a zillion language model startups from ex-Googlers

- 38:07 Ethan's journey in scaling, early days

- 40:08 making progress on an academic budget, scaling laws research

- 41:22 all alignment is inverse scaling problems

- 45:16 predicting scaling laws, useful ai alignment research

- 47:16 nitpicks aobut Ajeya Cotra's report, compute trends

- 50:45 optimism, conclusion on alignment

## Introduction

**Michaël**: Ethan, you're a master's degree student at Mila in Montreal, you have published papers on out of distribution, generalization, and robustness generalization accepted as presentations and spotlight presentations at ICML and NeurIPS. You've recently been thinking about scaling laws, both as an organizer and speaker for the first neural scaling laws workshop in Montreal. You're currently thinking about the monotonic scaling behaviors for downstream and upstream task, like in the GPT-3 paper, and most importantly, people often introduce you as the edgiest person at Mila on Twitter, and that's the reason why you're here today. So thanks, Ethan, for coming on the show and it's a pleasure to have you.

**Ethan**: Likewise.

## Scaling Laws T-Shirts

**Michaël**: You're also well-known for publicizing some sweatshirt mentioning scale is all you need AGI is coming.

**Ethan**: Yeah.

**Michaël**: How did those sweatshirts appear?

**Ethan**: Yeah, there was a guy named Jordi Armengol-Estapé who interned at Mila, and he got really into scaling laws, apparently via me. And then he sent me the shirt and was like: look how cool this shirt is. Like, he's the person wearing the shirt in the picture, and he's like, look how cool this shirt I just made is. And so then I tweeted the shirt. And then Irina just turned it into a merchandising scheme to fund future scaling. So she just made a bunch and started selling it to people. Like apparently, like she sells like more than 10 to Anthropic already. Just scaling lot of t-shirts, that's the ultimate funding model for supercomputers.

## Scaling Laws, Upstream and Downstream tasks

**Michaël**: Maybe you can like explain intuitively for listeners that are not very familiar to what are scaling laws in general.

**Ethan**: Whatever your bottleneck compute data parameters, you can predict what the performance will be as that bottleneck is relieved. Currently, the thing most people know how to do is predict like the upstream performance. Like the thing people want though is to be able to predict the downstream performance and upstream is what you're like... It's like your literal loss function that you're optimizing and then downstream is just any measure that you have of, like something you care about, so just like a downstream dataset, or like, I mean, usually, it's just mean accuracy on a downstream dataset.

**Michaël**: And to take like concrete examples, like for GPT-3, the upstream task is just predict the next word. What are the downstream tasks?

**Ethan**: Like 190... a zillion like benchmarks that the NLP community has come up with over the years. Like they just evaluated like the accuracy and like things like F1 score on all those.

**Michaël**: And yeah, what should we care about upstream and downstream task?

**Ethan**: I mean, basically like up, well, we don't really care about upstream that much. Upstream's just the first thing that people knew how to predict, I guess, like predict the scaling of what we care about as downstream. I mean, basically, like downstream things that improve monotonically, they kind of can be interpreted as like capabilities or whatever, and then downstream stuff that doesn't necessarily improve monotonically often is stuff that is advertised as alignment stuff. So like toxicity or if you like speculate in the future, stuff like interpretability or controllability would be things that might not improve monotonically.

**Michaël**: So you don't get more interpretability as you scale your models?

**Ethan**: You do currently, but the class example is like CLIP. It gets more interpretable as it has representations that make more sense. But you can imagine at a certain point, it's less interpretable because then at a certain point, the concepts it comes up with are beyond human comprehension. Like now it's just how like dogs can't comprehend calculus or whatever.

## Defining Alignment and AGI

**Michaël**: Yeah, when you mention alignment, what's the easiest way for you to define it?

**Ethan**: I mean, the Anthropic definition's pretty practical. Like we want models that are helpful, honest, and harmless, and that seems to cover all the like weird edge cases that people can like come up with on the Alignment Forum or whatever.

**Michaël**: Gotcha, so it is not like a technical definition. It's more a theoretical one.

**Ethan**: Yeah, yeah.

**Michaël**: So would you consider yourself an alignment researcher or more like a deep learning researcher?

**Ethan**: I'd say just a beneficial AGI researcher. That seems to cover everything.

**Michaël**: What's AGI?

**Ethan**: The definition on NASA website's pretty good. Highly autonomous systems that outperform humans at most economically valuable tasks.

## AI Timelines

**Michaël**: When do you think we'll get AGI?

**Ethan**: I'll just say like, it depends mostly on just like compute stuff, but I'll just say 2040 is my median.

**Michaël**: What's your like 10% and 90% estimate?

**Ethan**: 10%, probably like 2035.

## Recent Progress: AlphaCode, Math Scaling

**Michaël**: I think there's been a week where we got DALL-E 2, Chinchilla, PaLM. Did that like update your models in any way?

**Ethan**: The one that I thought was the like... was the crazy day was the day that AlphaCode and the math-proving thing happened on the same day, because like, especially the math stuff, like Dan Hendricks has all those slides where he is like, oh, math has the worst scaling laws or whatever, but then like OpenAI has like the IMO stuff. So like at least according to like Dan Hendricks' slides, whatever, that would've been like, something that took longer than it did.

**Michaël**: So when you mentioned the IMO stuff, I think it was like at problem from maybe 20 years ago, and it was something that you can like do with maybe like two lines of math.

**Ethan**: I agree they weren't like super, super impressive, but it's more just the fact that math is supposed to have like the worst scaling supposedly, but like impressive stuff's already happened with math now.

**Michaël**: Why is math supposed to have the worst scaling?

**Ethan**: It's just an empirical thing. Like Dan Hendricks has that like math benchmark thing and then he tried to do some extrapolations based on the scaling of performance on that. But the amount of computing data we currently have, it's already like doing interesting stuff was kind of surprising for me.

**Michaël**: I think in the paper, they mentioned that the method would not really scale well because of, and some infinite actions base when trying to think of like actions.

**Ethan**: Yeah.

**Michaël**: So yeah, I didn't update it. I was like, oh yeah, scaling will be easy for math.

**Ethan**: I didn't update it as easy, but just easier than I had thought.

## The Chinchilla Scaling Law

**Michaël**: Okay, related to scaling, the paper by DeepMind about the Chinchilla model was the most relevant, right?

**Ethan**: Yeah, I thought it was interesting. Like, I mean, you probably saw me tweet it, like that person on Eleuther  Discord that was like, oh wait, Sam Altman already said this like six months ago, but they just didn't put it in a paper.

**Michaël**: Yeah, he said that on the Q&A, right?

**Ethan**: Yeah, yeah.

**Ethan**: Yeah, he said something like we shouldn't, our models will not be like much bigger.

**Ethan**: Yeah. He said they'll use way more compute, which is analogous to saying, there you'll train a smaller model, but on more data.

**Michaël**: Can you like explain the kind of insights from scaling laws between like compute model size, and then like what's called like the Kaplan Scaling law?

**Ethan**: It was originally something about computing. If your compute budget increase a billionfold, your model size increases a millionfold and your dataset size increases a thousandfold. And now it's something like, I know it's like one to one, but I don't remember like how big the model size to like compute ratio was. I know like the model-to-data ratio is one to one now, but I don't remember what the compute-to-model ratio is, the new compute-to-model ratio is.

**Michaël**: That's also what I remember, and I think like the main insight from the first thing you said from the Kaplan law is that like model size is all those matters compared to dataset and for a fixed compute budget.

**Ethan**: Yeah, the narrative with the Kaplan one was model size, like compute is the bottleneck for now until you get to the intersection point of the compute scaling and the data scaling, and at that point, data's gonna become more of a bottleneck.

**Michaël**: So compute is the bottleneck now. What about like having huge model?

**Ethan**: But yeah, yeah. That's like, because like they were saying that because model size grows so fast. So like to get the bigger models, you need more compute rather than like, you don't need more data 'cause like you don't even have enough compute to like train a large model on that data yet, with the current compute regime... was the narrative of the first of the original Kaplan paper. But it's different now because like the rate at which you should be getting data given, like the rate at which your data charge should be increasing given your compute budget is increasing is a lot faster now, like using the Chinchilla scaling law. For some increasing compute size, you're gonna increase your model by a certain amount, and the amount that you're dataset size increases is like a one-to-one relation to the amount that your model size increases. I don't remember what the relation between model and compute was, but I know that now the relation between model and dataset size is one to one, between model size and dataset size is one to one.

**Ethan**: And the main size is that now we can just have more data and more compute, but not like a lot of more compute. We just need the same amount as more compute. So we can just like have to scrap the internet and get more data.

**Ethan**: It just means like to use your compute budget optimally, the rate at which your dataset size grows is a lot faster.

**Michaël**: Does that make you more confident that we'll get like better performance for models quicker?

**Ethan**: Maybe for like YouTube stuff, because YouTube, we're not bottlenecked by data. We're bottlenecked by compute, whatever. But that implies the model sizes might not grow as fast for YouTube or whatever. But for text, we're probably gonna be bottlenecked by... It means we're probably gonna be bottlenecked like text and code by the dataset size earlier than we thought. But for YouTube, that might like speed up the unsupervised video on all of YouTube, like timeline stuff.

## Limits of Scaling: Data

**Michaël**: Yeah, so I'm curious when do you think about like how much are we bottlenecked by data for text?

**Ethan**: Yeah, I asked Jared Kaplan about this, and he said like, "Wait, okay. "It's 300 billion tokens for GP3." And then he said like, library of Congress, whatever, could be 10 trillion tokens or something like that. And so like the most pessimistic estimate of how much like the most capable organization could get is the 500 billion tokens. A more optimistic estimate is like 10 trillion tokens is how many tokens the most capable organization could get, like mostly English tokens.

**Michaël**: So how many like orders of magnitude in terms of like parameters does this give us?

**Ethan**: I don't remember what the... Like I haven't calculated it. Like I remember I kind of did it with the old one, but I haven't done it with the new Chinchilla one. But I mean, you said this in your thing today or whatever, like we probably are gonna be bottleneck by the amount of code.

**Michaël**: I was essentially quoting Jared Kaplan's video.

## Code Generation

**Ethan**: Yeah, yeah, but he, I mean, he's right. I'm kind of wondering what's philanthropic thinking of Adept, because Adept's like doing the training all the code thing, and Adept was gonna do all the train on all the code thing, and they're like, oh crap, we got another startup doing the train on all the code stuff.

**Michaël**: Yeah, so I think you said that if you remove the duplicates on GitHub, you get some amount of tokens, maybe like 50 billion tokens, 500, I'm not sure. Maybe 50 billion. Don't put me on that.

**Ethan**: Yeah.

**Michaël**: And yeah, so the tricks will be data augmentation... you're like applying the real things to make your model better, but it's not clear how do you improve performance? So my guess would be you do transfer learning, like you train on like all the different languages.

**Ethan**: That's definitely what they plan on doing, like you see the scaling lots for transfer paper is literally pre-train on English and then fine-tune on code.

**Michaël**: My guess is also that like, if you get a bunch of like the best programmers in the world to use co-pilot and then you get like feedback from what they accept, you get higher quality data. You get just like, oh yeah, this work just doesn't work. And so you have like 1 million people using your thing 100 times a day, 1,000 times a day, then that's data for free.

**Ethan**: I mean, I view that part kind of as like the human feedback stuff is kind like the alignment part is the way I view it. I mean, then there's some people who like say, oh, there might be ways to get like better pre-training scaling if you have like humans in the loop during the pre-training, but like, no one's really figured that out yet.

**Michaël**: Well, don't you think like having all this telemetric data from GitHub cooperatives is you can use it, right?

**Ethan**: Yeah, yeah, but I almost view it as like that it's like used for alignment, like for RL from human preferences.

**Michaël**: Okay. Gotcha. Yeah, I think the other thing they did for improving GPT-3 was just having a bunch of humans rate the answers from GPT-3 and then like that's the paper of instructivity. I think like they had a bit of humans and it kind of improved the robustness or not for business, but alignment of the answer somehow. Like it said less like non-ethical things.

**Ethan**: Yeah. I mean it's like people downvoted the non-ethical stuff, I think.

## Youtube Scaling, Contrastive Learning

**Michaël**: Exactly, yeah. And to go back to YouTube, why is scaling on YouTube interesting? Because there's unlimited data?

**Ethan**: Yeah, one, you're not banned, but I mean, the gist is YouTube's the most diverse, like simultaneously diverse and large source of like video data basically.

**Michaël**: And yeah. So for people who were not used to or thinking, what's the task in YouTube?

**Ethan**: Yeah, it could be various things. Like it might be like a contrastive thing or it might be a predict all the pixels thing. Like, I mean, so like at least places like Facebook seem to think like contrastive has better downstreams scaling laws, so it's gonna be a contrastive type thing.

**Michaël**: What's contrastive type thing?

**Ethan**: Like you want representations that have similar like semantic meaning to be close together, like have low cosign similarity, like in latent space. So basically, like maximize the mutual information between views. Like it's kind of hard to explain without pictures.

**Michaël**: So you'd say that your model takes a video, like all of the videos and views as input?

**Ethan**: Frames that were close together like in time, it tries to maximize the mutual information between them via maximizing cosign similarity between the latents of like a resonant encoder or whatever that encodes the images for both of those frames that were next to each other, like in time.

**Michaël**: So he tries to kind of predict correlations between frames in some kind of latent space from a resonance?

**Ethan**: Yeah, yeah. In the latent space, you want frames that were close to each other in time to have similar, like maximize the cosign similar between the latent space between the latent between the hidden layer output by the like resonance that took each of those in each of those frames in.

**Michaël**: And at the end of the day, you want something that is capable of predicting how many frames in lens.

**Ethan**: Kind of for, well, the like philosophy with like the contrastive stuff is we just want a good representation that's useful for downstream tasks or whatever. So like you don't actually like, there's no like output really. It's just you're training a latent space or whatever that can be fine-tuned to downstream tasks very quickly.

**Michaël**: What are the useful downstream tests, like robotics?

**Ethan**: Yeah, yeah. Like there's a zillion papers on like people pre-train on do some pre-train contrastive thing in like an Atari environment, and then they show like, oh, now we barely need any RL steps to like fine-tune it or whatever and it can like learn RL really quickly after we just did all this unsupervised contrastive, like pre-training or whatever.

**Michaël**: And yeah, wouldn't your model be kind of shocked by the real world when you just like show him like YouTube videos all the time and then you trust the robot with like a camera?

**Ethan**: Kind of not. I mean, 'cause there there's like everything on YouTube. They got like first person egocentric stuff, they got third person stuff. Like it'll just like realize which, like whether it's in first or third person pretty quickly. I feel like it just infers the context. Like now I saw GPT-3 just for the context, it's in, 'cause it seemed like every context ever.

**Michaël**: Gotcha. So I was mostly thinking about like entropy of language.

**Ethan**: If it's literally like a video generative model, then you can do like just the perfect analogies, GPT-3 or whatever. It gets a little trickier with like contrastive stuff, but yeah, I mean either one. I mean the analogies are pretty similar for either one.

**Michaël**: So one of the things about the scaling laws papers and the role of scaling laws, there was some different exponents for text.

**Ethan**: Yeah.

## Scaling Exponent for Different Modalities

**Michaël**: What do you think is the exponent for video? Would it be like much worse?

**Ethan**: I know the model size. The model size relation was the big point of the scaling laws. For autoregressive generative models, the paper says that the rate at which the model size grows, given your compute budget grows, is the same for every modality. So that was kind of like, that's like a big unexplained thing. Like that was the biggest part just of that paper and no one's been able to explain why that is yet.

**Michaël**: So there might be some universal law where scaling goes for all modality and nobody knows why.

**Ethan**: Just stuff. The rate at which your model size grows given your compute budget is increasing is the same for every modality, which is kind of weird and no one, like I haven't really heard a good explanation why.

**Michaël**: Who do you think will win the video prediction race?

## AGI Race: the Best Funding Model for Supercomputers

**Ethan**: The person who wins AGI is whoever has the best funding model for supercomputers. Whoever has the best funding model for supercomputers wins. Like, I mean yet to assume all entities are like, they have like the nerve, like we're gonna do the biggest training run ever, but then given that's your pre-filter, then it's just whoever has the best funding models for supercomputers.

**Michaël**: So who is able to spend the most money? So would it be USA, China, Russia?

**Ethan**: Yeah, yeah, it might be something. I mean, my guess is like China's already, like they already have this joint fusion of industry government and academia via the Beijing Academy of AI in China. So my guess is like at some point, like Beijing Academy of AI and be like, look, we just trained like a 10 to the 15 parameter model on all of YouTube and spent like $40 billion doing it. And then at that point, Jared Kaplan's gonna be in the White House press conference room, be like, look, see these straight lines on log log pots, we gotta do this in the USA now.

**Michaël**: Right, right. But how do you even spend that much money?

**Ethan**: By making people think if they don't, they'll no longer be the superpower of the world or whatever. Like China will take over the world or whatever. Like it's only like a fear. It's only a fear thing.

**Michaël**: From looking at the PaLM paper from Google, they seem pretty clever on how they use their compute.

**Ethan**: You mean the thing where they have like the two supercomputers that they split it across or whatever?

**Michaël**: Right. I think TPU pods or something, they call it.

**Ethan**: Yeah, yeah.

**Michaël**: So it didn't seem like they spent more money than OpenAI. So they tried to be more careful somehow. So my model of like people spending a lot of money is.

**Ethan**: Like most entities won't be willing to like do the largest training when they can, given their funding.

**Michaël**: So maybe China, but I see Google as being more helpful because of they do it on paper, but maybe I'm wrong.

**Ethan**: Jared Kaplan says like most like Anthropic and OpenAI are kind of unique in that they're like, okay. We're gonna like throw all our funding into this one big training run. But like Google and like 'cause Google and Amazon, they have like he said like at least, 10X or like 100X times the compute that OpenAI and Anthropic have, but they never like use all the compute for single training runs. They just have all these different teams that use to compute for these different things.

**Michaël**: Yeah, so they have like a different hypothesis. OpenAI is like scale is all that matters, somehow that they're secrets itself and-

**Ethan**: Yeah, it's something like that.

**Michaël**: You just let scale things and we are going to get better results, and Google is maybe there's more bureaucracy and it's maybe harder to get a massive budget.

## Private Research at Google and OpenAI

**Ethan**: Yeah, it's weird though, 'cause Jeff Dean's latest blog post, it summarizes all the Google's research progress mentions like scaling and scaling while it's a zillion times. So that almost implies that like they're on the scales. All you need bandwagon too. So I don't know.

**Michaël**: They probably know, but then the question is how like private things are and maybe there's stuff we don't really know.

**Ethan**: I know a bunch of Google said like, yeah, we have language models that are way bigger than GPT-3, but we just don't put 'em in papers.

**Michaël**: So you've talked to them like privately or is it just, they said online?

**Ethan**: I just I've heard things from people and that's feasible. I'm not just disclosing what I got that information from, but that's just what I've heard from people.

**Michaël**: So as we're on like gossip, I think like something that was around on the internet, like right when GPT-3 was launched was that Google was like reproduced it in a few months afterwards, but they didn't really talk about it publicly. I'm not sure about what to do with this information.

**Ethan**: I know like the DeepMind language models papers that they were a year old when they finally put 'em out on archive or whatever, like Gopher and Chinchilla. They had the language model finished training a year before the paper came out.

**Michaël**: So we should just like assume all those big companies are just like throwing papers when they're like not relevant anymore when they have like the other paper already?

**Ethan**: Maybe, but yeah. I don't know why it was delayed that much. Yeah, I don't know what the story is. Why it was delayed that long.

**Michaël**: People want to like keep their advantage, right?

**Ethan**: I guess, but I mean like I feel like GPT-3, they threw the paper on arXiv pretty soon after they finished training GPT-3.

**Michaël**: How do you know?

**Ethan**: Yeah, I don't, but I mean, yeah, I don't. But ice, it didn't. Yeah, maybe there was a big delay. I don't know.

**Michaël**: So I think you could just like retrace all Sam Altman tweet and then like you read the next paper like six months after and you're like, oh yeah, he tweeted about that. Like sometimes the tweets like, oh, AI is going to be wild, or oh, neural networks are really capable of understanding. I think you tweeted that like six months ago, like when they discovered GPT-4.

**Ethan**: OpenAI is like when Ilya tweeted the consciousness tweet, they're like, goddamn, GPT-4 must be crazy.

**Michaël**: Yeah, neural networks are in some ways slightly conscious.

**Ethan**: Yeah, yeah, that was the funniest quote.

**Michaël**: Yeah, I think people at OpenAI know things we don't know yet. They're all like super hyped. And I think you mentioned as well that at least privately that Microsoft has some deal with OpenAI and so they need to some amount of money before 2024, like.

**Ethan**: Oh yeah, yeah, yeah, yeah. I mean, right, right. When the Microsoft deal happened, like Greg Brockman said, "Our plan is to train "like a 100 trillion parameter model by 2024."

**Michaël**: Okay, so that's in two years?

**Ethan**: I mean, that was in 2019, but maybe they've changed their mind after like the Chinchilla scaling lot stuff, I don't know.

## Why Ethan did not update that much from PaLM

**Michaël**: Right. And so you were not like impressed by PaLM being able to predict to like do logic on airplane things and explain jokes?

**Ethan**: In my mind, like the video scaling was like a lot worse than text basically. That's the main reason why I like AGI will probably take longer in the five years or whatever in my mind.

**Michaël**: Okay, so we need, so if we just have text, it's not enough to have AGI. So if we're a like a perfect Oracle that can like talk like us, but it's not able to do robotic things, then we don't have AGI.

**Ethan**: Yeah.

**Michaël**: Well, I guess my main like is mostly like coding. So if we get like coding, like Codex or comparative, that gets really good, then everything accelerates and engineers become very productive, and then like.

**Ethan**: I guess if you could say like, engineers get really productive at making improvements in hardware, then like, maybe that would, like, I get how that would be like, okay. Then it's really fast. Like in my mind, at least at the current, I don't see the hardware getting fast enough to be far enough on the YouTube scaling lot in less than five years from now.

**Michaël**: Thinking about hardware, we're just like humans, Googling things and using.

**Ethan**: Yeah, yeah. I get what you're saying. Like you get the Codex thing and then we use Codex or whatever to design hardware faster.

**Michaël**: You mentioned you have like DALL-E, but like for designing chips.

**Ethan**: I mean, Nvidia already uses AI for designing their chips.

**Michaël**: That doesn't make you think of timelines of 10 years or closer.

**Ethan**: It may be 10 years, but not five years. The thing I'm trying to figure out is like, try to get like a student researcher gig at like someplace so that I can just get access to the big compute during the PhD.

**Michaël**: Oh, so that's your plan. Just get out of compute.

**Ethan**: Yeah, I mean, as long as I have big compute, it doesn't matter where I'm a PhD. I mean, it kind of matters if you're like trying to start an AGI startup or whatever, but safe, safe, safe AGI startup.

**Michaël**: We're kind of on record, but I'm not sure if I'm going to cut this part. So you can say unsafe, it's fine.

**Ethan**: Yeah, no, no, no. I mean, I don't even phrase. I just phrase it as beneficial AGI.

**Michaël**: You were spotted saying you wanted unsafe AGI the fastest possible.

## Thinking about the Fastest Path

**Ethan**: No, no, no. The way I phrase it is I think I explained this last time, you have to be thinking in terms of the fastest path, because there is like extremely huge economic and military incentives that are selecting for the fastest path, whether you want it to be that way or not. So like, you gotta be thinking in terms of, what is the fastest path and then how do you like minimize the alignment tax on that fastest path? 'Cause like the fastest path is the way it's probably gonna happen no matter what, like, so it's about minimizing the alignment techs on that fastest path.

**Michaël**: Or you can just throw nukes everywhere and try to make things slower?

**Ethan**: Yeah, I guess, but I mean the people who are on the fastest path will be like more powerful, such that like, I don't know, such that they'll deter all the nukes.

**Michaël**: So you want to be, okay, so you want to just like join the winners. Like if you join the skiing team at Google.

**Ethan**: Thing I've been trying to brainstorm about is who's gonna have the fastest, who's gonna have the best funding models for supercomputers, 'cause that's the place to go and you gotta try to minimize the alignment tax at that place.

**Michaël**: Makes sense. So everyone should infiltrate Google.

**Ethan**: Yeah, so whatever place ends up with the best funding model of supercomputers try to get as many weird alignment people to like infiltrate that place as possible.

**Michaël**: So I'm kind of happy having a bunch of EA people at OpenAI now, because they're kind of minimizing the text there, but...

**Ethan**: Yeah, I kind of viewed it as all the EA people left, like 'cause Anthropic was like the most extremist EA people at OpenAI. So I almost viewed when Anthropic happened a bunch of EA people. I view as that like EA almost leaving OpenAI when Anthropic happened.

**Michaël**: Some other people came, right?

**Ethan**: Like who?

**Michaël**: I don't know. Richard Ngo.

**Ethan**: Oh, okay, okay. Yeah, yeah.

**Michaël**: It's like a team on like predicting the future.

**Ethan**: Yeah, yeah. I wanna know what the Futures Team does 'cause that's like the most out there team. I'm really curious to what they actually do.

**Michaël**: Maybe they use their GPT-5 model and predict things.

**Ethan**: Right, 'cause I mean like DALL-E, like you know about the Foresight Team at OpenAI, right?

**Michaël**: They were trying to predict things as well, like forecasting.

**Ethan**: Yeah, that's where all this scaling lot stuff came from was on the Foresight Team at OpenAI. They're gone now because they became philanthropic. But like a team called like the Futures Team that almost has a similar vibe to like a team called the Foresight Team. So I'm kind of curious.

**Michaël**: But then there's just like doing more governance things and optimal governance and maybe economics.

**Ethan**: That's what it's about, governance and economics.

**Michaël**: The guy like Richard Ngo is doing governance there.

**Ethan**: Okay.

**Michaël**: Predicting how the future works, I think is in his Twitter bio.

**Ethan**: Yeah, yeah, but I mean, that's somewhat tangential to governance, like that almost sounds like something Mike Rick Kurtz, I would say, I'm predicting how the future.

**Michaël**: My model is like Sam Altman, as like they have GPT-4. Like they published GPT-3 in 2020. So it's been like two years.

**Ethan**: Yeah.

**Michaël**: And they've been talking about like in their Q & A about like treacherous results or something like one year ago. So now they must have access to something very crazy and they're just like trying to think like how do we operate with like DALL-E 2 and their GPT-4 they have in private and how they do something without like for him in the world? I don't know. Maybe they're just like trying to predict like how to make the most money with their API or.

**Ethan**: You're saying like if they release it, it's like an infohazard? 'Cause in my mind, GPT-4 still isn't like capable enough to F up the world, but you could argue, it's like capable enough to like be an infohazard or something.

**Michaël**: Imagine you have access to something that has the same kind of gap between GPT-2 and GPT-3, but like for GPT-4 on like understanding and being general. And you don't want everyone else to copy your work. So you're just going to keep it for yourself for sometime.

## A Zillion Language Model Startups from ex-Googlers

**Ethan**: Yeah, but I feel like that strategy is already kind of screwed. Like you know about how like a zillion large language model, like a zillion Googlers have left Google to start large language model startups. Like there's literally three large language model startups by ex-Googlers now. OpenAI is like a small actor in this now because there's like multiple large language model startups founded by ex-Googlers that all like that all were founded in the last like six months. Like there's a zillion VCs throwing money at large language model startups right now. The funniest thing, like Leo Gao, he's like, we need more large language model startups because the more startups we have, then it splits up all the funding so no organization can have all the funding to get the really big supercomputer. So we just need thousands of AI during its final startups. So no one can hoard all the funding to get the really big language model.

**Ethan**: That's the, yeah, with the AI model, you just like do open source. So like there's like more startups. And so all the funding gets splitted, I guess.

**Ethan**: Yeah, you could view OpenAI was like extra big brain. We need to do. We need to like release the idea of our joiners models onto the world such that no organization could have enough compute to be such that all the compute gets more split up, 'cause a zillion, our joiners model startups will show up all at once.

**Michaël**: That's yeah, that's the best idea ever. So do you have like other gossips besides like Google's? Did you post something on Twitter about people leaving Google?

**Ethan**: Yeah, I posted a bunch of stuff. Well, I mean, and also like you saw the... I mean it's three startups, adept.ai, character.ai, and inflection.ai. They're all large language model startups founded by ex-Googlers that got a zillion dollars in VC funding to scale large language models.

**Michaël**: What's a zillion dollars like?

**Ethan**: Like greater than 60 million. Each of them got greater than 60 million.

**Michaël**: So did they know about something we don't know? And they're just like get money to replicate what Google does?

**Ethan**: Well, I mean, most of 'em, they were famous people like founder of DeepMind scaling team. Another one is the inventor of The Transformer. Another one was founded by a different person on The Transformer paper. Like, so I mean, in some ways, they have more clout than like OpenAI had or whatever.

**Michaël**: But they don't have like the engineering and old infrastructure.

**Ethan**: No, they kind of do. Like, a lot of 'em, they were like the head of engineering for scaling teams at like DeepMind or Google.

**Michaël**: So there's like another game that is in private at Google and they've been scaling huge models for two years. and they're just like,

**Ethan**: Yeah, something like that.

**Michaël**: Starting startups with their knowledge and they're just scaling and we;re just like, like peasants like us talk about papers that are released one year after and then when you turn them out.

**Ethan**: Yeah, yeah. I guess that's, I mean, I don't know how long these delays are. I mean, in my mind, like, yeah. I guess you could view it as a delay thing, 'cause like in my mind it's just like, yeah, you're right, you're right. It's probably delayed by a year, yeah.

**Michaël**: So yeah, that makes me less confident about-

**Ethan**: Oh shit. You look like a clone of Lex Fridman from the side.

**Michaël**: What?

**Ethan**: When your face is like sideways, you look like a clone of Lex Fridman.

**Michaël**: Yeah.

**Ethan**: Like, 'cause your haircut's like identical to his when

**Michaël**: I'll take that as a compliment... I started working out. So yeah, Ethan Caballero, what's the meaning of life?

**Ethan**: Probably just maximize the flourishing of all sentient beings, like a very generic answer.

**Michaël**: Right. So I've done my Lex Fridman question. Now I'm just basically him.

**Ethan**: Yeah.

## Ethan's Scaling Journey

**Michaël**: Maybe we can just go back to like stuff we know more about like your work and because you've been doing some work on scaling.

**Ethan**: Yeah.

**Michaël**: So like more general, like why are you kind of interested in scaling and like how did you started on doing research on that?

**Ethan**: I mean, I knew about the body paper when it came out. Like I remember I was at this like Ian Goodfellow talking in 2017 and he was hyped about the body paper when it came out.

**Michaël**: Which paper?

**Ethan**: The deep burning scales, predictably, empirically, yeah, it came out 2017 and then I kind, I just, that was just on the back burner and I kind of just stopped paying attention to it after a while. And then like Aran Komatsuzaki was like, no, dude, this is the thing. Like this is gonna take over everything, and this was like in 2019 when he was saying that. And then, yeah. So then when the scaling laws papers got like re-popularized through like the OpenAI stuff, then I kind of like caught onto it a little bit early via like talking with Aran.

**Michaël**: I think in 2019 was also when GPT-2 was introduced.

**Ethan**: But that was kind of before, like that was before like the scaling law stuff kind of got popularized.

**Michaël**: Right, scaling laws paper is 2020.

**Ethan**: Yeah, the very end of 2020. All right. No, no, no. Oh no, no. The scaling law paper was the very end of 2000. It was the very beginning of 2020.

**Michaël**: And you were already on this killing train since 2017.

**Ethan**: I was aware of it, but I didn't, like, I was kind of just neutral about it until like 2000, like probably the middle of 2019.

## Making progress on an Academic budget, Scaling Laws Research

**Michaël**: And yeah, now you are kind of interested in scaling because it's useful to predict kind of what the whole field of AI is going.

**Ethan**: And also it just, it's I think people underestimate how easy it is to be contrived if you're not paying attention to scaling trends and trying to like extrapolate the compute budgets and data budgets that are like, well, the compute data and data budgets like five years from now.

**Michaël**: Yeah, if you're a huge company that does a lot of budget, but maybe if you're just a random company, you don't really care about scaling law that much.

**Ethan**: Yeah, yeah. Or if you're like in academia currently or whatever, like a zillion papers that like fancy conferences are like, here's our inducted bias that helps on like our punny academic budget. And we didn't test any of the scaling asso tos to see if it's like useful when you're training a trillion parameter model on all of YouTube or whatever.

**Michaël**: You're on an academic budget as far as I know. So how do you manage to do experiments in scaling?

**Ethan**: There's like the scaling on narrative. That's like, oh, you don't need the big budget to do because you can just predict what the outcomes will be for the large scale experiments, but that's at least current. At least when that narrative got popularized, it was mostly for upstream like scaling. But the thing everyone cares about is downstream scaling.

## AI Alignment as an Inverse Scaling Problem

**Michaël**: Yeah, so if we go back for a minute on like your work in alignment, how do you think your work on scaling or generalization like kind of fits with the alignment problem?

**Ethan**: Basically, all alignment, I guess this triggers the hell outta some people. But all alignment is inverse scaling problems. It's all downstream inverse scaling problems. So it's just in my mind, all of alignment is stuff that doesn't improve monotonically as compute data and parameters increase.

**Michaël**: There's a difference between not improving and inverse scaling. Inverse scaling goes badly, right?

**Ethan**: Yeah, yeah, yeah. But I just said not improved monotonically, 'cause like sometimes there's certain things where like it improves for a while, but then at a certain point, it gets worse. So like interpretability and controllability are the two like kind of thought experiment things where you could imagine they get more interpretable and more controllable for a long time until they get super intelligent. At that point. they're like less interpretable and less controllable.

**Michaël**: Do we have benchmarks for controllability or?

**Ethan**: Like just like just benchmarks that rely on prompting is a form of like a benchmark of controllability.

**Michaël**: And kinda to summarize your take, if we were able to just scale everything well and not have this inverse scaling problem, we would get like interpretability and controllability and everything else by just like good scaling of our models. And so we'd get like alignment kind of by defaults for free?

**Ethan**: Yeah. I mean, I guess, I mean like there's stuff besides interpretability, controllability, like those are just the examples. Like what you said, you asked like what's an example where like the reason I said, I phrased it as alignment is when I said inverse scaling, I said things that don't improve monotonically, 'cause I just wanted to say like, yes, there's obvious examples where it gets worse the entire time, but there's some you could imagine where it gets good for a long time, and then a certain point, then it starts getting drastically worse. I said, all of alignment can be viewed as a downstream scaling problem. The hard part is like Dan Hendricks and like Jacob Steinhardt say like, then the hard problem though is like measurement and like finding out what are the downstream evaluations 'cause say you got like some like fancy like deceptive AI that wants to like a treacherous turn or whatever. Like how do you even find the downstream evaluations to know whether it's gonna like try to deceive you or whatever? 'Cause like when I say, it's all a downstream scaling problem, that assumes like you have the downstream test, the downstream like thing that you're evaluating it on. But like if it's like some weird deceptive thing, that's like, it's hard to even find what's the downstream thing to evaluate it on to like know whether it's trying deceive or whatever.

**Michaël**: So there's no like test lost on this deception. We don't know for sure how to measure and have a clear benchmark from this.

**Ethan**: Yeah, it's tricky. I mean, and some people say like, well, that's why you need better interpretability. You need to like find the deception circuits or whatever.

**Michaël**: Knowing that we don't know yet, like all the different benchmarks and metrics for misalignment, don't you think that your work on scaling can be bad because you're actually like speeding up timelines?

## Predicting scaling laws, Useful AI Alignment research

**Ethan**: Yeah, I get the like infohazard point of view, but like in my mind, like whether you wanna do all capabilities or alignment stuff that stands the test of time, you need really good downstream scaling prediction. Like, say you came up with some like alignment method or whatever that mitigates inverse scaling, like you need the actual functional form to know whether that thing will like keep mitigating inverse scaling when you get to like a trillion parameters or whatever. You get what I mean?

**Michaël**: I get you but like on a differential progress mindset, like Jared Kaplan or someone else will come up with those functional forms without your work.

**Ethan**: I don't know, I don't know. I mean, that's the thing though, like Anthropics (ERRATUM: it's actually a gift, and the the merch was not sent at the time of the podcast) got that paper like predictability and surprise and generative models and they're just like, it's unpredictable. We can't predict it. And I'm like, ah, you guys, nah, I don't believe.

**Michaël**: Right, so you're kind of publishing papers when you're in advance because those companies are not publishing their results?

**Ethan**: I don't know. I don't. Yeah, I don't even, I don't know if Anthropic does the delay type stuff that OpenAI supposedly does, but maybe they do, I don't know.

**Michaël**: And you were just like drawing infohazard by publishing those laws?

**Ethan**: I mean, in my mind, whether or not, I get the argument, oh it, if you wanna do capabilities work that stands a test of time or alignment work that stands a test of time, in my mind, everything that people are doing in alignment will be very contrived without the functional form too though. So it's like alignment can't make progress without it either. So it's like, you get what I mean?

**Michaël**: Another kind of view on that is that if people do impressive deploying or ML board and they're also interested in alignment, it's still a good thing. Like let's take even through AI. Even if they open source their model because they did something impressive and they talk openly about alignment under Discord and gets like a lot of people that are very smart, interested in alignment. So if you publish something and you become like a famous researcher, something in two years and you talk about alignment in two years, then it's fine.

**Ethan**: I sort of tweet stuff about alignment, I think. Yeah, I mean, I retweet stuff about alignment at least.

## Ajeya Cotra's report, Compute Trends

**Michaël**: So if we go back to thinking about predicting future timelines and kind of scaling, I've read somewhere that you think that in the next few years, we might get billion or trillion times of more compute, like 12 orders of magnitude more.

**Ethan**: Yeah, I mean, so the Ajeya Cotra report said like, it's gonna max out probably at 10 to the 12 times as much compute as like the amount of compute in 2020, probably like 2070 or something like that. The one issue I have with the JS model is that like, she does, what does she do? It's like it's flops per dollar times willingness to spend its total flops that are allocated to pre-training runs. Problem is like, for the big like foundation models, like 10 of the 15 perimeter miles of the future or whatever, you're probably gonna need high pie like memory bandwidth between all like memory bandwidth and compute bandwidth between all the compute, which means it has to be on a supercomputer. So it's not just the flaps. It basically what really matters, at least if you're assuming it's like big, like 10 of the 15 parameter foundation models or whatever, like the speed of the fastest supercomputer is what matters, not just the total flaps that you can allocate, because if like all the flaps don't have good communication between them, then they aren't really useful for training like 10 of the 15 parameter model or whatever. Once you get to 10 of the 15 parameters, like there isn't much reason to go beyond that. And at that point, then you just have multiple models with 10 of the 15 parameters and they're like doing some crazy open ended, like Ken Stanley stuff and a multi-agent simulator after you do that. Like if they mentioned became like you do the 10 of the 15 parameter model feature and all YouTube, and then after that, you'll have like hundreds of 10 of the 15 parameter models that all just duke it out in like a Ken Stanley open-ended simulator to like, get the rest of the capabilities or whatever, like once they're in the Ken Stanley open-ended stimulator, then you don't need high compute bandwidth between all those individual, like 10 of the 15 parameter models, like duking it out in the simulator. They can just, each one, they only needs like 10. It only needs high compute bandwidth between like its own parameters. Like, it doesn't need high compute bandwidth between itself and the other like agents or whatever. And so in there, the flops where you could use all the flops for like the multi-agent simulation, but you only need high compute bandwidth within each agent.

**Michaël**: So you need a lot of bandwidths to train models because of the prioritization thing, but you only need flops to simulate on different things at the same time?

**Ethan**: Yeah, you only need high compute bandwidth within an individual brain, but like if you have multiple brains, then you don't need high compute bandwidth between the brains.

**Michaël**: And what was that kind of simulator you were talking about, the Kenley?

**Ethan**: Like Ken Stanley, the open-ended guy.

**Michaël**: I haven't seen that.

**Ethan**: Ken is like the myth day objective open-endedness, like Can Stanley's, Jeff Cones, like all that stuff. It's like, I don't know. Just Google, like Can Stanley open ended at some point. You've probably heard of it, but it's not like registering what I'm referencing.

# Optimism, conclusion on alignment

**Michaël**: Okay, so maybe one kind of last open-ended question. On a scale from Paul Christiano, Eliezer Yudkowsky, Sam Altman, how optimistic are you?

**Ethan**: Definitely not like Eliezer, or a doomer type person. I guess probably Paul Christiano is most similar. I mean, I feel like Paul Christiano is in the middle of the people you just said.

**Michaël**: Right. Yeah. So you are less optimistic than Sam Altman?

**Ethan**: Well, yeah, I mean, basically, I think deceptive AI is probably gonna be really hard.

**Michaël**: So do you have like one less monologue or sentence to say about why scaling is a solution for all alignment problems?

**Ethan**: Like just all alignment can be viewed as an inverse scaling problem. Like it all revolves on just mitigating inverse scaling, but also you have to make sure you have like the right downstream things that you're evaluating, like the inverse scaling and like part of what makes it hard is like you might need to do like fancy thought experiments on alignment, like counterintuitive thought experiments on alignment forum to find what are the downstream... to find what are the the downstream tests that you should be evaluating. Like whether or not there's inverse scaling behavior on those.

**Michaël**: Awesome, so we get the good version, as last sentence, and that's our conclusion. Thanks Ethan for being on the show.


# Ben Garfinkel How sure are we about this AI stuff - EA Forum

_It is increasingly clear that artificial intelligence is poised to have a huge impact on the world, potentially of comparable magnitude to the agricultural or industrial revolutions. But what does that actually mean for us today? Should it influence our behavior? In this talk from EA Global 2018: London, Ben Garfinkel makes the case for measured skepticism._

## The Talk

Today, work on risks from artificial intelligence constitutes a noteworthy but still fairly small portion of the EA portfolio.

![](https://images.ctfassets.net/ohf186sfn6di/ZsVNHCONV87govWeW1mzj/4a0ee1ce7ff02e0c9ee346f639d528e9/How_sure_are_we_about_this_AI_stuff_.jpg)

Only a small portion of donations made by individuals in the community are targeted at risks from AI. Only about 5% of the grants given out by the Open Philanthropy Project, the leading grant-making organization in the space, target risks from AI. And in surveys of community members, most do not list AI as the area that they think should be most prioritized.

![](https://images.ctfassets.net/ohf186sfn6di/6Y2EK3CXjTib0jmytnh6bn/51159295f2615db856b1342ce09f9dd7/How_sure_are_we_about_this_AI_stuff___1_.jpg)

At the same time though, work on AI is prominent in other ways. Leading career advising and community building organizations like 80,000 Hours and CEA often highlight careers in AI governance and safety as especially promising ways to make an impact with your career. Interest in AI is also a clear element of community culture. And lastly, I think there's also a sense of momentum around people's interest in AI. I think especially over the last couple of years, quite a few people have begun to consider career changes into the area, or made quite large changes in their careers. I think this is true more for work around AI than for most other cause areas.

![](https://images.ctfassets.net/ohf186sfn6di/5tGd4OLBwS1YILRm94IQrY/3d821b9e681e7cf5950d6c319f3a3017/How_sure_are_we_about_this_AI_stuff___2_.jpg)

So I think all of this together suggests that now is a pretty good time to take stock. It's a good time to look backwards and ask how the community first came to be interested in risks from AI. It's a good time look forward and ask how large we expect the community's bet on AI to be: how large a portion of the portfolio we expect AI to be five or ten years down the road. It's a good time to ask, are the reasons that we first got interested in AI still valid? And if they're not still valid, are there perhaps other reasons which are either more or less compelling?

![](https://images.ctfassets.net/ohf186sfn6di/4qQkD7b4iyiDcA9FJ2TUAH/3ae30c9b9373184785a4989ea6a84229/How_sure_are_we_about_this_AI_stuff___3_.jpg)

To give a brief talk roadmap, first I'm going to run through what I see as an intuitively appealing argument for focusing on AI. Then I'm going to say why this argument is a bit less forceful than you might anticipate. Then I'll discuss a few more concrete arguments for focusing on AI and highlight some missing pieces of those arguments. And then I'll close by giving concrete implications for cause prioritization.

## The intuitive argument

So first, here's what I see as an intuitive argument for working on AI, and that'd be the sort of, "AI is a big deal" argument.

![](https://images.ctfassets.net/ohf186sfn6di/7p0BlYuELpna1ITNlwyprB/4cdb5922dda5b7b270a62ad092bee27b/How_sure_are_we_about_this_AI_stuff___4_.jpg)

There are three concepts underpinning this argument:

1.  The future is what matters most in the sense that, if you could have an impact that carries forward and affects future generations, then this is likely to be more ethically pressing than having impact that only affects the world today.
2.  Technological progress is likely to make the world very different in the future: that just as the world is very different than it was a thousand years ago because of technology, it's likely to be very different again a thousand years from now.
3.  If we're looking at technologies that are likely to make especially large changes, then AI stands out as especially promising among them.

So given these three premises, we have the conclusion that working on AI is a really good way to have leverage over the future, and that shaping the development of AI positively is an important thing to pursue.

![](https://images.ctfassets.net/ohf186sfn6di/7mp0cWrXHVycsTrA5dYo8N/86be241a8b87bc02c026ecdfebdcf4a1/How_sure_are_we_about_this_AI_stuff___5_.jpg)

I think that a lot of this argument works. I think there are compelling reasons to try and focus on your impact in the future. I think that it's very likely that the world will be very different in the far future. I also think it's very likely that AI will be one of the most transformative technologies. It seems at least physically possible to have machines that eventually can do all the things that humans can do, and perhaps do all these things much more capably. If this eventually happens, then whatever their world looks like, we can be pretty confident the world will look pretty different than it does today.

![](https://images.ctfassets.net/ohf186sfn6di/2Itddi3KDH9et3k7fEfmaN/c2408b1dc8d01a6055b37ff1b3ead4cf/How_sure_are_we_about_this_AI_stuff___6_.jpg)

What I find less compelling though is the idea that these premises entail the conclusion that we ought to work on AI. Just because a technology will produce very large changes, that doesn't necessarily mean that working on that technology is a good way to actually have leverage over the future. Look back at the past and consider the most transformative technologies that have ever been developed. So things like electricity, or the steam engine, or the wheel, or steel. It's very difficult to imagine what individuals early in the development of these technologies could have done to have a lasting and foreseeably positive impact. An analogy is sometimes made to the industrial revolution and the agricultural revolution. The idea is that in the future, impacts of AI may be substantial enough that there will be changes that are comparable to these two revolutionary periods throughout history.

![](https://images.ctfassets.net/ohf186sfn6di/3zyPurKZmDrqEPr6ubFXbs/1d7b15e0f1644f740fd539ff2a053648/How_sure_are_we_about_this_AI_stuff___7_.jpg)

The issue here, though, is that it's not really clear that either of these periods actually were periods of especially high _leverage_. If you were, say, an Englishman in 1780, and trying to figure out how to make this industry thing go well in a way that would have a lasting and foreseeable impact on the world today, it's really not clear you could have done all that much. The basic point here is that from a long-termist perspective, what matters is leverage. This means finding something that could go one way or the other, and that's likely to stick in a foreseeably good or bad way far into the future. Long-term importance is perhaps a necessary condition for leverage, but certainly not a sufficient one, and it's a sort of flawed indicator in its own right.

## Three concrete cases

So now I'm going to move to three somewhat more concrete cases for potentially focusing on AI. You might have a few concerns that lead you to work in this area:

![](https://images.ctfassets.net/ohf186sfn6di/5cQPhQtAosME1MvnHP8zvr/1fe32c8a0ffe87221492dcf69108af88/How_sure_are_we_about_this_AI_stuff___8_.jpg)

1.  **Instability.** You might think that there are certain dynamics around the development or use of AI systems that will increase the risk of permanently damaging conflict or collapse, for instance war between great powers.
2.  **Lock-in.** Certain decisions regarding the governance or design of AI systems may permanently lock in, in a way that propagates forward into the future in a lastingly positive or negative way.
3.  **Accidents.** It might be quite difficult to use future systems safely. And that there may be accidents that occur in the future with more advanced systems that cause lasting harm that again carries forward into the future.

### Instability

![](https://images.ctfassets.net/ohf186sfn6di/5LTTWAR6vTDfNNip1cyJ8I/1262a65298d8e61959755c56a075aac6/How_sure_are_we_about_this_AI_stuff___9_.jpg)

First, the case from instability. A lot of the thought here is that it's very likely that countries will compete to reap the benefits economically and militarily from the applications of AI. This is already happening to some extent. And you might think that as the applications become more significant, the competition will become greater. And in this context, you might think that this all increases the risk of war between great powers. So one concern here is that there may be a potential for transitions in terms of what countries are powerful compared to which other countries.

A lot of people in the field of international security think that these are conditions under which conflict becomes especially likely. You might also be concerned about changes in military technology that, for example, increase the odds of accidental escalation, or make offense more favorable compared to defense. You may also just be concerned that in periods of rapid technological change, there are greater odds of misperception or miscalculation as countries struggle to figure out how to use the technology appropriately or interpret the actions of their adversaries. Or you could be concerned that certain applications of AI will in some sense damage domestic institutions in a way that also increases instability. That rising unemployment or inequality might be quite damaging, for example. And lastly, you might be concerned about the risks from terrorism, that certain applications might make it quite easy for small actors to cause large amounts of harm.

![](https://images.ctfassets.net/ohf186sfn6di/2TGpTLYHPCnXz7zcTmiROr/6054c97bcf3668966ba2ffaf3cae6631/How_sure_are_we_about_this_AI_stuff___10_.jpg)

In general, I think that many of these concerns are plausible and very clearly important. Most of them have not received very much research attention at all. I believe that they warrant much, much more attention. At the same time though, if you're looking at things from a long-termist perspective, there are at least two reservations you could continue to have. The first is just we don't really know how worried to be. These risks really haven't been researched much, and we shouldn't really take it for granted that AI will be destabilizing. It could be or it couldn't be. We just basically have not done enough research to feel very confident one way or the other.

You may also be concerned, if you're really focused on long term, that lots of instability may not be sufficient to actually have a lasting impact that carries forward through generations. This is a somewhat callous perspective. If you really are focused on the long term, it's not clear, for example, that a mid-sized war by historical standards would be sufficient to have a big long term impact. So it may be actually a quite high bar to achieve a level of instability that a long-termist would really be focused on.

### Lock-in

![](https://images.ctfassets.net/ohf186sfn6di/1Nrumzh71ltZw6f4zpR0Qt/2809264d0109c2314dc13e51a71cd752/How_sure_are_we_about_this_AI_stuff___11_.jpg)

The case from lock-in I'll talk about just a bit more briefly. Some of the intuition here is that certain decisions have been made in the past about, for instance the design of political institutions, software standards, or certain outcomes of military or economic competitions, which seem to produce outcomes that carry forward into the future for centuries. Some examples would be the design of the US Constitution, or the outcome of the Second World War. You might have the intuition that certain decisions about the governance or design of AI systems, or certain outcomes of strategic competitions, might carry forward into the future, perhaps for even longer periods of time. For this reason, you might try and focus on making sure that whatever locks in is something that we actually want.

![](https://images.ctfassets.net/ohf186sfn6di/308qz7pteWgeXxHeTGde0E/2f58fc3853fc0d81e516fbd29e892b49/How_sure_are_we_about_this_AI_stuff___12_.jpg)

I think that this is a somewhat difficult argument to make, or at least it's a fairly non-obvious one. I think the standard skeptical reply is that with very few exceptions, we don't really see many instances of long term lock-in, especially long term lock-in where people really could have predicted what would be good and what would be bad. Probably the most prominent examples of lock-in are choices around major religions that have carried forward for thousands of years. But it's quite hard to find examples that last for hundreds of years. Those seem quite few. It's also generally hard to judge what you would want to lock in. If you imagine fixing some aspect of the world, as the rest of world changes dramatically, it's really hard to guess what would actually be good under quite different circumstances in the future. I think my general feeling on this line of argument is that, I think it's probably not that likely that we should expect any truly irreversible decisions around AI to be made anytime soon, even if progress is quite rapid, although other people certainly might disagree.

### Accidents

![](https://images.ctfassets.net/ohf186sfn6di/6oUQxfjwmoB7VqDxvDLEgG/f71e293bd9ab4bd8325cb45025c4a5f5/How_sure_are_we_about_this_AI_stuff___13_.jpg)

Last, we have the case from accidents. The idea here is that, we know that there are certain safety engineering challenges around AI systems. It's actually quite difficult to design systems that you can feel confident will behave the way you want them to in all circumstances. This has been laid out most clearly in the paper 'Concrete Problems in AI Safety,' from a couple of years ago by Dario Amodei and others. I'd recommend for anyone interested in safety issues to take a look at that paper. Then we might think, given the existence of these safety challenges, and given the belief or expectation that AI systems will become much more powerful in the future or be given much more responsibility, we might expect that these safety concerns will become more serious as time goes on.

![](https://images.ctfassets.net/ohf186sfn6di/2sn6sL0PAjzmIhVnQg6no9/881544f686d1673dd547dcc363b49cf2/How_sure_are_we_about_this_AI_stuff___14_.jpg)

At the limit you might worry that these safety failures could become so extreme that they could perhaps derail civilization on the whole. In fact, there is a bit of writing arguing that we should be worried about these sort of existential safety failures. The main work arguing for this is still the book 'Superintelligence' by Nick Bostrom, published in 2014. Before this, essays by Eliezer Yudkowsky were the main source of arguments along these lines. And then a number of other writers such as Stuart Russell or, a long time ago, IJ Goods or David Chalmers have also expressed similar concerns, albeit more briefly. The writing on existential safety accidents definitely isn't homogeneous, but often there's a sort of similar narrative that appears in these essays expressing these concerns. There's this basic standard disaster scenario that has a few common elements.

![](https://images.ctfassets.net/ohf186sfn6di/1Wy6LheSrW6jOhKIus7xkW/cc39bc24ab78c4d6a20179a55c673ab7/How_sure_are_we_about_this_AI_stuff___15_.jpg)

First, the author imagines that a single AI system experiences a massive jump in capabilities. Over some short period of time, a single system becomes much more general or much more capable than any other system in existence, and in fact any human in existence. Then given the system, researchers specify a goal for it. They give it some input which is meant to communicate what behavior it should engage in. The goal ends up being something quite simple, and the system goes off and single-handedly pursues this very simple goal in a way that violates the full nuances of what its designers intended.

There's a classic sort of toy example, which is often used to illustrate this concern. We imagine that some poor paperclip factory owner receives a general super-intelligent AI on his doorstep. There's a slot that's to stick in a goal. He writes down the goal "maximize paperclip production," puts it in the AI system, and then lets it go off and do that. The system figures out the best way to maximize paperclip production is to take over all the world's resources, just to plow them all into paperclips. And the system is so capable that designers can do nothing to stop it, even though it's doing something that they actually really do not intend.

![](https://images.ctfassets.net/ohf186sfn6di/FUGgsz0vO3iVlUPhvQvuP/36a6d39bfbdcae267e65a9995d97595a/How_sure_are_we_about_this_AI_stuff___16_.jpg)

I have some general concerns about the existing writing on existential accidents. So first there's just still very little of it. It really is just mostly _Superintelligence_ and essays by Eliezer Yudkowsky, and then sort of a handful of shorter essays and talks that express very similar concerns. There's also been very little substantive written criticism of it. Many people have expressed doubts or been dismissive of it, but there's very little in the way of skeptical experts who are sitting down and fully engaging with it, and writing down point by point where they disagree or where they think the mistakes are. Most of the work on existential accidents was also written before large changes in the field of AI, especially before the recent rise of deep learning, and also before work like 'Concrete Problems in AI Safety,' which laid out safety concerns in a way which is more recognizable to AI researchers today.

Most of the arguments for existential accidents often rely on these sort of fuzzy, abstract concepts like optimization power or general intelligence or goals, and toy thought experiments like the paper clipper example. And certainly thought experiments and abstract concepts do have some force, but it's not clear exactly how strong a source of evidence we should take these as. Then lastly, although many AI researchers actually have expressed concern about existential accidents, for example Stuart Russell, it does seem to be the case that many, and perhaps most AI researchers who encounter at least abridged or summarized versions of these concerns tend to bounce off them or just find them not very plausible. I think we should take that seriously.

I also have some more concrete concerns about writing on existential accidents. You should certainly take these concerns with a grain of salt because I am not a technical researcher, although I have talked to technical researchers who have essentially similar or even the same concerns. The general concern I have is that these toy scenarios are quite difficult to map onto something that looks more recognizably plausible. So these scenarios often involve, again, massive jumps in the capabilities of a single system, but it's really not clear that we should expect such jumps or find them plausible. This is a wooly issue. I would recommend checking out writing by Katja Grace or Paul Christiano online. That sort of lays out some concerns about the plausibility of massive jumps.

Another element of these narratives is, they often imagine some system which becomes quite generally capable and then is given a goal. In some sense, this is the reverse of the way machine learning research tends to look today. At least very loosely speaking, you tend to specify a goal or some means of providing feedback. You direct the behavior of a system and then allow it to become more capable over time, as opposed to the reverse. It's also the case that these toy examples stress the nuances of human preferences, with the idea being that because human preferences are so nuanced and so hard to state precisely, it should be quite difficult to get a machine that can understand how to obey them. But it's also the case in machine learning that we can train lots of systems to engage in behaviors that are actually quite nuanced and that we can't specify precisely. Recognizing faces from images is an example of this. So is flying a helicopter.

It's really not clear exactly why human preferences would be so fatal to understand. So it's quite difficult to figure out how to map the toy examples onto something which looks more realistic.

## Caveats

Some general caveats on the concerns expressed. None of my concerns are meant to be decisive. I've found, for example, that many people working in the field of AI safety in fact list somewhat different concerns as explanations for why they believe the area is very important. There are many more arguments that I believe are shared individually, or inside people's heads and currently unpublished. I really can't speak exactly to how compelling these are. The main point I want to stress here is essentially that when it comes to the writing which has actually been published, and which is out there for analysis, I don't think it's necessarily that forceful, and at the very least it's not decisive.

![](https://images.ctfassets.net/ohf186sfn6di/7d0c2thDgSda4pA0UMa6eL/76337ea2edf400ed750e84801d4d66c5/How_sure_are_we_about_this_AI_stuff___17_.jpg)

So now I have some brief, practical implications, or thoughts on prioritization. You may think, from all the stuff I've just said, that I'm quite skeptical about AI safety or governance as areas to work in. In fact, I'm actually fairly optimistic. My reasoning here is that I really don't think that there are any slam-dunks for improving the future. I'm not aware of any single cause area that seems very, very promising from the perspective of offering high assurance of long-term impact. I think that the fact that there are at least plausible pathways for impact by working on AI safety and AI governance puts it head and shoulders above most areas you might choose to work in. And AI safety and AI governance also stand out for being pretty extraordinarily neglected.

Depending on how you count, there are probably fewer than a hundred people in the world working on technical safety issues or governance challenges with an eye towards very long-term impacts. And that's just truly, very surprisingly small. The overall point though, is that the exact size of the bet that EA should make on artificial intelligence, sort of the size of the portfolio that AI should take up will depend on the strength of the arguments for focusing on AI. And most of those arguments still just aren't very fleshed out yet.

![](https://images.ctfassets.net/ohf186sfn6di/7d4iE4NftlFqYUBEDvnNKA/c42643b32671fd613cb6616d8af8535b/How_sure_are_we_about_this_AI_stuff___18_.jpg)

I also have some broader epistemological concerns which connect to the concerns I've expressed. I think it's also possible that there are social factors relating to EA communities that might bias us to take an especially large interest in AI.

One thing is just that AI is especially interesting or fun to talk about, especially compared to other cause areas. It's an interesting, kind of contrarian answer to the question of what is most important to work on. It's surprising in certain ways. And it's also now the case that interest in AI is to some extent an element of community culture. People have an interest in it that goes beyond just the belief that it's an important area to work in. It definitely has a certain role in the conversations that people have casually, and what people like to talk about. I think these wouldn't necessarily be that concerning, except people sometimes also think that we can't really count on external feedback to push us back if we sort of drift a bit.

So first it just seems to be empirically the case that skeptical AI researchers generally will not take the time to sit down and engage with all of the writing, and then explain carefully why they disagree with our concerns. So we can't really expect that much external feedback of that form. People who are skeptical or confused, but not AI researchers, or just generally not experts may be concerned about sounding ignorant or dumb if they push back, and they also won't be inclined to become experts. We should also expect generally very weak feedback loops. If you're trying to influence the very long-run future, it's hard to tell how well you're doing, just because the long-run future hasn't happened yet and won't happen for a while.

Generally, I think one thing to watch out for is justification drift. If we start to notice that the community's interest in AI stays constant, but the reasons given for focusing on it change over time, then this would be sort of a potential check engine light, or at least a sort of trigger to be especially self-conscious or self-critical, because that may be some indication of motivated reasoning going on.

## Conclusion

![](https://images.ctfassets.net/ohf186sfn6di/1LHyv9s4vVgYCuBVvQnMUE/952fc5834362f2f6adb37202f2f79428/How_sure_are_we_about_this_AI_stuff___19_.jpg)

I have just a handful of short takeaways. First, I think that not enough work has gone into analyzing the case for prioritizing AI. Existing published arguments are not decisive. There may be many other possible arguments out there, which could be much more convincing or much more decisive, but those just aren't out there yet, and there hasn't been much written criticizing the stuff that's out there.

For this reason, thinking about the case for prioritizing AI may be an especially high impact thing to do, because it may shape the EA portfolio for years into the future. And just generally, we need to be quite conscious of possible community biases. It's possible that certain social factors will lead us to drift in what we prioritize, that we really should not be allowing to influence us. And just in general, if we're going to be putting substantial resources into anything as a community, we need to be especially certain that we understand why we're doing this, and that we stay conscious that our reasons for getting interested in the first place continue to be good reasons. Thank you.

## Questions

_Question_: What advice would you give to one who wants to do the kind of research that you are doing here about the case for AI potentially, as opposed to the AI itself?

_Ben_: Something that I believe would be extremely valuable is just basically talking to lots of people who are concerned about AI and asking them precisely what reasons they find compelling. I've started to do this a little bit recently and it's actually been quite interesting that people seem to have pretty diverse reasons, and many of them are things that people want to write blog posts on, but just haven't done. So, I think this is a low-hanging fruit that would be quite valuable. Just talking to people who are concerned about AI, trying to understand exactly why they're concerned, and either writing up their ideas or helping them to do that. I think that would be very valuable and probably not that time intensive either.

_Question_: Have you seen any of the justification drift that you alluded to? Can you pinpoint that happening in the community?

_Ben_: Yeah. I think that's certainly happening to some extent. Even for myself, I believe that's happened for me to some extent. When I initially became interested in AI, I was especially concerned about these existential accidents. I think I now place relatively greater prominence on sort of the case from instability as I described it. And that's certainly, you know, one possible example of justification drift. It may be the case that this was actually a sensible way to shift emphasis, but would be something of a warning sign. And I've also just spoken to technical researchers, as well, who used to be especially concerned about this idea of an intelligence explosion or recursive self improvement. These very large jumps. I now have spoken to a number of people who are still quite concerned about existential accidents, but make arguments that don't hinge on there being this one single massive jump into a single system.

_Question_: You made the analogy to the industrial revolution, and the 1780 Englishman who doesn't really have much ability to shape how the steam engine is going to be used. It seems intuitively quite right. The obvious counterpoint would be, well AI is a problem-solving machine. There's something kind of different about it. I mean, does that not feel compelling to you, the sort of inherent differentness of AI?

_Ben_: So I think probably the strongest intuition is, you might think that there will eventually be a point where we start turning more and more responsibility over to automated systems or machines, and that there might eventually come a point where humans have almost no control over what's happening whatsoever, that we keep turning over more and more responsibility and there's a point where machines are in some sense in control and you can't back out. And you might have some sort of irreversible juncture here. I definitely, to some extent, share that intuition that if you're looking over a very long time span, that that is probably fairly plausible. I suppose the intuition I don't necessarily have is that unless things go, I suppose quite wrong or if they happen in somewhat surprising ways, I don't necessarily anticipate that there will be this really irreversible juncture coming anytime soon. If let's say it takes a thousand years for control to be handed off, then I am not that optimistic about people having that much control over what that handoff looks like by working on things today. But I certainly am not very confident.

_Question_: Are there any policies that you think a government should implement at this stage of the game, in light of the concerns around AI safety? And how would you allocate resources between existing issues and possible future risks?

_Ben_: Yeah, I am still quite hesitant, I think, to recommend very substantive policies that I think governments should be implementing today. I currently have a lot of agnosticism about what would be useful, and I think that most current existing issues that governments are making decisions on aren't necessarily that critical. I think there's lots of stuff that can be done that would be very valuable, like having stronger expertise or stronger lines of dialogue between the public and private sector, and things like this. But I would be hesitant at this point to recommend a very concrete policy that at least I'm confident would be good to implement right now.

_Question_: You mentioned the concept of kind of a concrete decisive argument. Do you see concrete, decisive arguments for other cause areas that are somehow more concrete and decisive than for AI, and what is the difference?

_Ben_: Yeah. So I guess I tried to allude to this a little bit, but I don't think that really any cause area has an especially decisive argument for being a great way to influence the future. There's some that I think you can put sort of a lower bound on at least how likely it is to be useful that's somewhat clear. So for example, risk from nuclear war. It's fairly clear that it's at least plausible this could happen over the next century. You know, nuclear war has almost happened in the past, the climate effects are speculative, but at least somewhat well understood. And then there's this question of if there were nuclear war, how damaging is this? Do people eventually come back from this? And that's quite uncertain, but I think it'd be difficult to put above 99% chance that people would come back from a nuclear war.

So, in that case you might have some sort of a clean lower bound on, let's say working on nuclear risk. Or, quite similarly, working on pandemics. And I think for AI it's difficult to have that sort of confident lower bound. I actually tend to think, I guess as I alluded to, that AI is probably or possibly still the most promising area based on my current credences, and its extreme neglectedness. But yeah, I don't think any cause area stands out as especially decisive as a great place to work.

_Question_: I'm an AI machine learning researcher PhD student currently, and I'm skeptical about the risk of AGI. How would you suggest that I contribute to the process of providing this feedback that you're identifying as a need?

_Ben_: Yeah, I mean I think just a combination of in-person conversations and then I think even simple blog posts can be quite helpful. I think there's still been surprisingly little in the way of just, let's say something written online that I would point someone to who wants the skeptical case. This actually is a big part of the reason I suppose I gave this talk, even though I consider myself not extremely well placed to give it, given that I am not a technical person. There's so little out there along these lines that there's low hanging fruit, essentially.

_Question_: Prominent deep learning experts such as Yann Lecun and Andrew Ng do not seem to be worried about risks from superintelligence. Do you think that they have essentially the same view that you have or are they coming at it from a different angle?

_Ben_: I'm not sure of their specific concerns. I know this classic thing that Andrew Ng always says is he compares it to worrying about overpopulation on Mars, where the suggestion is that these risks, if they materialize, are just so far away that it's really premature to worry about them. So it seems to be sort of an argument from timeline considerations. I'm actually not quite sure what his view is in terms of, if we were like, let's say 50 years in the future, would he think that this is a really great area to work on? I'm really not quite sure.

I actually tend to think that the line of thinking that says, "Oh, this is so far away so we shouldn't work on it" just really isn't that compelling. It seems like we have a load of uncertainty about AI timelines. It seems like no one can be very confident about that. So yeah, it'd be hard to get under, let's say one percent that interesting things won't happen in the next 30 years or so. So I'm not quite sure about the extent of his concerns, but if they're based on timelines, I actually don't find them that compelling.