# Ethan On Why Scale is All You Need on The Inside View Podcast

Interviewee: Ethan Caballero
Date: 2022-05-05

Ethan is known on Twitter as the edgiest person at MILA. We discuss all the gossips around scaling large language models in what will be later known as the Edward Snowden moment of Deep Learning. On his free time, Ethan is a Master’s degree student at MILA in Montreal, and has published papers on out of distribution generalization and robustness generalization, accepted both as oral presentations and spotlight presentations at ICML and NeurIPS. Ethan has recently been thinking about scaling laws, both as an organizer and speaker for the 1st Neural Scaling Laws Workshop.

Outline

- 00:00 highlights

- 00:50 who is Ethan, scaling laws T-shirts

- 02:30 scaling, upstream, downstream, alignment and AGI

- 05:58 AI timelines, AlphaCode, Math scaling, PaLM

- 07:56 Chinchilla scaling laws

- 11:22 limits of scaling, Copilot, generative coding, code data

- 15:50 Youtube scaling laws, constrative type thing

- 20:55 AGI race, funding, supercomputers

- 24:00 Scaling at Google

- 25:10 gossips, private research, GPT-4

- 27:40 why Ethan was did not update on PaLM, hardware bottleneck

- 29:56 the fastest path, the best funding model for supercomputers

- 31:14 EA, OpenAI, Anthropics, publishing research, GPT-4

- 33:45 a zillion language model startups from ex-Googlers

- 38:07 Ethan's journey in scaling, early days

- 40:08 making progress on an academic budget, scaling laws research

- 41:22 all alignment is inverse scaling problems

- 45:16 predicting scaling laws, useful ai alignment research

- 47:16 nitpicks aobut Ajeya Cotra's report, compute trends

- 50:45 optimism, conclusion on alignment

## Introduction

**Michaël**: Ethan, you're a master's degree student at Mila in Montreal, you have published papers on out of distribution, generalization, and robustness generalization accepted as presentations and spotlight presentations at ICML and NeurIPS. You've recently been thinking about scaling laws, both as an organizer and speaker for the first neural scaling laws workshop in Montreal. You're currently thinking about the monotonic scaling behaviors for downstream and upstream task, like in the GPT-3 paper, and most importantly, people often introduce you as the edgiest person at Mila on Twitter, and that's the reason why you're here today. So thanks, Ethan, for coming on the show and it's a pleasure to have you.

**Ethan**: Likewise.

## Scaling Laws T-Shirts

**Michaël**: You're also well-known for publicizing some sweatshirt mentioning scale is all you need AGI is coming.

**Ethan**: Yeah.

**Michaël**: How did those sweatshirts appear?

**Ethan**: Yeah, there was a guy named Jordi Armengol-Estapé who interned at Mila, and he got really into scaling laws, apparently via me. And then he sent me the shirt and was like: look how cool this shirt is. Like, he's the person wearing the shirt in the picture, and he's like, look how cool this shirt I just made is. And so then I tweeted the shirt. And then Irina just turned it into a merchandising scheme to fund future scaling. So she just made a bunch and started selling it to people. Like apparently, like she sells like more than 10 to Anthropic already. Just scaling lot of t-shirts, that's the ultimate funding model for supercomputers.

## Scaling Laws, Upstream and Downstream tasks

**Michaël**: Maybe you can like explain intuitively for listeners that are not very familiar to what are scaling laws in general.

**Ethan**: Whatever your bottleneck compute data parameters, you can predict what the performance will be as that bottleneck is relieved. Currently, the thing most people know how to do is predict like the upstream performance. Like the thing people want though is to be able to predict the downstream performance and upstream is what you're like... It's like your literal loss function that you're optimizing and then downstream is just any measure that you have of, like something you care about, so just like a downstream dataset, or like, I mean, usually, it's just mean accuracy on a downstream dataset.

**Michaël**: And to take like concrete examples, like for GPT-3, the upstream task is just predict the next word. What are the downstream tasks?

**Ethan**: Like 190... a zillion like benchmarks that the NLP community has come up with over the years. Like they just evaluated like the accuracy and like things like F1 score on all those.

**Michaël**: And yeah, what should we care about upstream and downstream task?

**Ethan**: I mean, basically like up, well, we don't really care about upstream that much. Upstream's just the first thing that people knew how to predict, I guess, like predict the scaling of what we care about as downstream. I mean, basically, like downstream things that improve monotonically, they kind of can be interpreted as like capabilities or whatever, and then downstream stuff that doesn't necessarily improve monotonically often is stuff that is advertised as alignment stuff. So like toxicity or if you like speculate in the future, stuff like interpretability or controllability would be things that might not improve monotonically.

**Michaël**: So you don't get more interpretability as you scale your models?

**Ethan**: You do currently, but the class example is like CLIP. It gets more interpretable as it has representations that make more sense. But you can imagine at a certain point, it's less interpretable because then at a certain point, the concepts it comes up with are beyond human comprehension. Like now it's just how like dogs can't comprehend calculus or whatever.

## Defining Alignment and AGI

**Michaël**: Yeah, when you mention alignment, what's the easiest way for you to define it?

**Ethan**: I mean, the Anthropic definition's pretty practical. Like we want models that are helpful, honest, and harmless, and that seems to cover all the like weird edge cases that people can like come up with on the Alignment Forum or whatever.

**Michaël**: Gotcha, so it is not like a technical definition. It's more a theoretical one.

**Ethan**: Yeah, yeah.

**Michaël**: So would you consider yourself an alignment researcher or more like a deep learning researcher?

**Ethan**: I'd say just a beneficial AGI researcher. That seems to cover everything.

**Michaël**: What's AGI?

**Ethan**: The definition on NASA website's pretty good. Highly autonomous systems that outperform humans at most economically valuable tasks.

## AI Timelines

**Michaël**: When do you think we'll get AGI?

**Ethan**: I'll just say like, it depends mostly on just like compute stuff, but I'll just say 2040 is my median.

**Michaël**: What's your like 10% and 90% estimate?

**Ethan**: 10%, probably like 2035.

## Recent Progress: AlphaCode, Math Scaling

**Michaël**: I think there's been a week where we got DALL-E 2, Chinchilla, PaLM. Did that like update your models in any way?

**Ethan**: The one that I thought was the like... was the crazy day was the day that AlphaCode and the math-proving thing happened on the same day, because like, especially the math stuff, like Dan Hendricks has all those slides where he is like, oh, math has the worst scaling laws or whatever, but then like OpenAI has like the IMO stuff. So like at least according to like Dan Hendricks' slides, whatever, that would've been like, something that took longer than it did.

**Michaël**: So when you mentioned the IMO stuff, I think it was like at problem from maybe 20 years ago, and it was something that you can like do with maybe like two lines of math.

**Ethan**: I agree they weren't like super, super impressive, but it's more just the fact that math is supposed to have like the worst scaling supposedly, but like impressive stuff's already happened with math now.

**Michaël**: Why is math supposed to have the worst scaling?

**Ethan**: It's just an empirical thing. Like Dan Hendricks has that like math benchmark thing and then he tried to do some extrapolations based on the scaling of performance on that. But the amount of computing data we currently have, it's already like doing interesting stuff was kind of surprising for me.

**Michaël**: I think in the paper, they mentioned that the method would not really scale well because of, and some infinite actions base when trying to think of like actions.

**Ethan**: Yeah.

**Michaël**: So yeah, I didn't update it. I was like, oh yeah, scaling will be easy for math.

**Ethan**: I didn't update it as easy, but just easier than I had thought.

## The Chinchilla Scaling Law

**Michaël**: Okay, related to scaling, the paper by DeepMind about the Chinchilla model was the most relevant, right?

**Ethan**: Yeah, I thought it was interesting. Like, I mean, you probably saw me tweet it, like that person on Eleuther  Discord that was like, oh wait, Sam Altman already said this like six months ago, but they just didn't put it in a paper.

**Michaël**: Yeah, he said that on the Q&A, right?

**Ethan**: Yeah, yeah.

**Ethan**: Yeah, he said something like we shouldn't, our models will not be like much bigger.

**Ethan**: Yeah. He said they'll use way more compute, which is analogous to saying, there you'll train a smaller model, but on more data.

**Michaël**: Can you like explain the kind of insights from scaling laws between like compute model size, and then like what's called like the Kaplan Scaling law?

**Ethan**: It was originally something about computing. If your compute budget increase a billionfold, your model size increases a millionfold and your dataset size increases a thousandfold. And now it's something like, I know it's like one to one, but I don't remember like how big the model size to like compute ratio was. I know like the model-to-data ratio is one to one now, but I don't remember what the compute-to-model ratio is, the new compute-to-model ratio is.

**Michaël**: That's also what I remember, and I think like the main insight from the first thing you said from the Kaplan law is that like model size is all those matters compared to dataset and for a fixed compute budget.

**Ethan**: Yeah, the narrative with the Kaplan one was model size, like compute is the bottleneck for now until you get to the intersection point of the compute scaling and the data scaling, and at that point, data's gonna become more of a bottleneck.

**Michaël**: So compute is the bottleneck now. What about like having huge model?

**Ethan**: But yeah, yeah. That's like, because like they were saying that because model size grows so fast. So like to get the bigger models, you need more compute rather than like, you don't need more data 'cause like you don't even have enough compute to like train a large model on that data yet, with the current compute regime... was the narrative of the first of the original Kaplan paper. But it's different now because like the rate at which you should be getting data given, like the rate at which your data charge should be increasing given your compute budget is increasing is a lot faster now, like using the Chinchilla scaling law. For some increasing compute size, you're gonna increase your model by a certain amount, and the amount that you're dataset size increases is like a one-to-one relation to the amount that your model size increases. I don't remember what the relation between model and compute was, but I know that now the relation between model and dataset size is one to one, between model size and dataset size is one to one.

**Ethan**: And the main size is that now we can just have more data and more compute, but not like a lot of more compute. We just need the same amount as more compute. So we can just like have to scrap the internet and get more data.

**Ethan**: It just means like to use your compute budget optimally, the rate at which your dataset size grows is a lot faster.

**Michaël**: Does that make you more confident that we'll get like better performance for models quicker?

**Ethan**: Maybe for like YouTube stuff, because YouTube, we're not bottlenecked by data. We're bottlenecked by compute, whatever. But that implies the model sizes might not grow as fast for YouTube or whatever. But for text, we're probably gonna be bottlenecked by... It means we're probably gonna be bottlenecked like text and code by the dataset size earlier than we thought. But for YouTube, that might like speed up the unsupervised video on all of YouTube, like timeline stuff.

## Limits of Scaling: Data

**Michaël**: Yeah, so I'm curious when do you think about like how much are we bottlenecked by data for text?

**Ethan**: Yeah, I asked Jared Kaplan about this, and he said like, "Wait, okay. "It's 300 billion tokens for GP3." And then he said like, library of Congress, whatever, could be 10 trillion tokens or something like that. And so like the most pessimistic estimate of how much like the most capable organization could get is the 500 billion tokens. A more optimistic estimate is like 10 trillion tokens is how many tokens the most capable organization could get, like mostly English tokens.

**Michaël**: So how many like orders of magnitude in terms of like parameters does this give us?

**Ethan**: I don't remember what the... Like I haven't calculated it. Like I remember I kind of did it with the old one, but I haven't done it with the new Chinchilla one. But I mean, you said this in your thing today or whatever, like we probably are gonna be bottleneck by the amount of code.

**Michaël**: I was essentially quoting Jared Kaplan's video.

## Code Generation

**Ethan**: Yeah, yeah, but he, I mean, he's right. I'm kind of wondering what's philanthropic thinking of Adept, because Adept's like doing the training all the code thing, and Adept was gonna do all the train on all the code thing, and they're like, oh crap, we got another startup doing the train on all the code stuff.

**Michaël**: Yeah, so I think you said that if you remove the duplicates on GitHub, you get some amount of tokens, maybe like 50 billion tokens, 500, I'm not sure. Maybe 50 billion. Don't put me on that.

**Ethan**: Yeah.

**Michaël**: And yeah, so the tricks will be data augmentation... you're like applying the real things to make your model better, but it's not clear how do you improve performance? So my guess would be you do transfer learning, like you train on like all the different languages.

**Ethan**: That's definitely what they plan on doing, like you see the scaling lots for transfer paper is literally pre-train on English and then fine-tune on code.

**Michaël**: My guess is also that like, if you get a bunch of like the best programmers in the world to use co-pilot and then you get like feedback from what they accept, you get higher quality data. You get just like, oh yeah, this work just doesn't work. And so you have like 1 million people using your thing 100 times a day, 1,000 times a day, then that's data for free.

**Ethan**: I mean, I view that part kind of as like the human feedback stuff is kind like the alignment part is the way I view it. I mean, then there's some people who like say, oh, there might be ways to get like better pre-training scaling if you have like humans in the loop during the pre-training, but like, no one's really figured that out yet.

**Michaël**: Well, don't you think like having all this telemetric data from GitHub cooperatives is you can use it, right?

**Ethan**: Yeah, yeah, but I almost view it as like that it's like used for alignment, like for RL from human preferences.

**Michaël**: Okay. Gotcha. Yeah, I think the other thing they did for improving GPT-3 was just having a bunch of humans rate the answers from GPT-3 and then like that's the paper of instructivity. I think like they had a bit of humans and it kind of improved the robustness or not for business, but alignment of the answer somehow. Like it said less like non-ethical things.

**Ethan**: Yeah. I mean it's like people downvoted the non-ethical stuff, I think.

## Youtube Scaling, Contrastive Learning

**Michaël**: Exactly, yeah. And to go back to YouTube, why is scaling on YouTube interesting? Because there's unlimited data?

**Ethan**: Yeah, one, you're not banned, but I mean, the gist is YouTube's the most diverse, like simultaneously diverse and large source of like video data basically.

**Michaël**: And yeah. So for people who were not used to or thinking, what's the task in YouTube?

**Ethan**: Yeah, it could be various things. Like it might be like a contrastive thing or it might be a predict all the pixels thing. Like, I mean, so like at least places like Facebook seem to think like contrastive has better downstreams scaling laws, so it's gonna be a contrastive type thing.

**Michaël**: What's contrastive type thing?

**Ethan**: Like you want representations that have similar like semantic meaning to be close together, like have low cosign similarity, like in latent space. So basically, like maximize the mutual information between views. Like it's kind of hard to explain without pictures.

**Michaël**: So you'd say that your model takes a video, like all of the videos and views as input?

**Ethan**: Frames that were close together like in time, it tries to maximize the mutual information between them via maximizing cosign similarity between the latents of like a resonant encoder or whatever that encodes the images for both of those frames that were next to each other, like in time.

**Michaël**: So he tries to kind of predict correlations between frames in some kind of latent space from a resonance?

**Ethan**: Yeah, yeah. In the latent space, you want frames that were close to each other in time to have similar, like maximize the cosign similar between the latent space between the latent between the hidden layer output by the like resonance that took each of those in each of those frames in.

**Michaël**: And at the end of the day, you want something that is capable of predicting how many frames in lens.

**Ethan**: Kind of for, well, the like philosophy with like the contrastive stuff is we just want a good representation that's useful for downstream tasks or whatever. So like you don't actually like, there's no like output really. It's just you're training a latent space or whatever that can be fine-tuned to downstream tasks very quickly.

**Michaël**: What are the useful downstream tests, like robotics?

**Ethan**: Yeah, yeah. Like there's a zillion papers on like people pre-train on do some pre-train contrastive thing in like an Atari environment, and then they show like, oh, now we barely need any RL steps to like fine-tune it or whatever and it can like learn RL really quickly after we just did all this unsupervised contrastive, like pre-training or whatever.

**Michaël**: And yeah, wouldn't your model be kind of shocked by the real world when you just like show him like YouTube videos all the time and then you trust the robot with like a camera?

**Ethan**: Kind of not. I mean, 'cause there there's like everything on YouTube. They got like first person egocentric stuff, they got third person stuff. Like it'll just like realize which, like whether it's in first or third person pretty quickly. I feel like it just infers the context. Like now I saw GPT-3 just for the context, it's in, 'cause it seemed like every context ever.

**Michaël**: Gotcha. So I was mostly thinking about like entropy of language.

**Ethan**: If it's literally like a video generative model, then you can do like just the perfect analogies, GPT-3 or whatever. It gets a little trickier with like contrastive stuff, but yeah, I mean either one. I mean the analogies are pretty similar for either one.

**Michaël**: So one of the things about the scaling laws papers and the role of scaling laws, there was some different exponents for text.

**Ethan**: Yeah.

## Scaling Exponent for Different Modalities

**Michaël**: What do you think is the exponent for video? Would it be like much worse?

**Ethan**: I know the model size. The model size relation was the big point of the scaling laws. For autoregressive generative models, the paper says that the rate at which the model size grows, given your compute budget grows, is the same for every modality. So that was kind of like, that's like a big unexplained thing. Like that was the biggest part just of that paper and no one's been able to explain why that is yet.

**Michaël**: So there might be some universal law where scaling goes for all modality and nobody knows why.

**Ethan**: Just stuff. The rate at which your model size grows given your compute budget is increasing is the same for every modality, which is kind of weird and no one, like I haven't really heard a good explanation why.

**Michaël**: Who do you think will win the video prediction race?

## AGI Race: the Best Funding Model for Supercomputers

**Ethan**: The person who wins AGI is whoever has the best funding model for supercomputers. Whoever has the best funding model for supercomputers wins. Like, I mean yet to assume all entities are like, they have like the nerve, like we're gonna do the biggest training run ever, but then given that's your pre-filter, then it's just whoever has the best funding models for supercomputers.

**Michaël**: So who is able to spend the most money? So would it be USA, China, Russia?

**Ethan**: Yeah, yeah, it might be something. I mean, my guess is like China's already, like they already have this joint fusion of industry government and academia via the Beijing Academy of AI in China. So my guess is like at some point, like Beijing Academy of AI and be like, look, we just trained like a 10 to the 15 parameter model on all of YouTube and spent like $40 billion doing it. And then at that point, Jared Kaplan's gonna be in the White House press conference room, be like, look, see these straight lines on log log pots, we gotta do this in the USA now.

**Michaël**: Right, right. But how do you even spend that much money?

**Ethan**: By making people think if they don't, they'll no longer be the superpower of the world or whatever. Like China will take over the world or whatever. Like it's only like a fear. It's only a fear thing.

**Michaël**: From looking at the PaLM paper from Google, they seem pretty clever on how they use their compute.

**Ethan**: You mean the thing where they have like the two supercomputers that they split it across or whatever?

**Michaël**: Right. I think TPU pods or something, they call it.

**Ethan**: Yeah, yeah.

**Michaël**: So it didn't seem like they spent more money than OpenAI. So they tried to be more careful somehow. So my model of like people spending a lot of money is.

**Ethan**: Like most entities won't be willing to like do the largest training when they can, given their funding.

**Michaël**: So maybe China, but I see Google as being more helpful because of they do it on paper, but maybe I'm wrong.

**Ethan**: Jared Kaplan says like most like Anthropic and OpenAI are kind of unique in that they're like, okay. We're gonna like throw all our funding into this one big training run. But like Google and like 'cause Google and Amazon, they have like he said like at least, 10X or like 100X times the compute that OpenAI and Anthropic have, but they never like use all the compute for single training runs. They just have all these different teams that use to compute for these different things.

**Michaël**: Yeah, so they have like a different hypothesis. OpenAI is like scale is all that matters, somehow that they're secrets itself and-

**Ethan**: Yeah, it's something like that.

**Michaël**: You just let scale things and we are going to get better results, and Google is maybe there's more bureaucracy and it's maybe harder to get a massive budget.

## Private Research at Google and OpenAI

**Ethan**: Yeah, it's weird though, 'cause Jeff Dean's latest blog post, it summarizes all the Google's research progress mentions like scaling and scaling while it's a zillion times. So that almost implies that like they're on the scales. All you need bandwagon too. So I don't know.

**Michaël**: They probably know, but then the question is how like private things are and maybe there's stuff we don't really know.

**Ethan**: I know a bunch of Google said like, yeah, we have language models that are way bigger than GPT-3, but we just don't put 'em in papers.

**Michaël**: So you've talked to them like privately or is it just, they said online?

**Ethan**: I just I've heard things from people and that's feasible. I'm not just disclosing what I got that information from, but that's just what I've heard from people.

**Michaël**: So as we're on like gossip, I think like something that was around on the internet, like right when GPT-3 was launched was that Google was like reproduced it in a few months afterwards, but they didn't really talk about it publicly. I'm not sure about what to do with this information.

**Ethan**: I know like the DeepMind language models papers that they were a year old when they finally put 'em out on archive or whatever, like Gopher and Chinchilla. They had the language model finished training a year before the paper came out.

**Michaël**: So we should just like assume all those big companies are just like throwing papers when they're like not relevant anymore when they have like the other paper already?

**Ethan**: Maybe, but yeah. I don't know why it was delayed that much. Yeah, I don't know what the story is. Why it was delayed that long.

**Michaël**: People want to like keep their advantage, right?

**Ethan**: I guess, but I mean like I feel like GPT-3, they threw the paper on arXiv pretty soon after they finished training GPT-3.

**Michaël**: How do you know?

**Ethan**: Yeah, I don't, but I mean, yeah, I don't. But ice, it didn't. Yeah, maybe there was a big delay. I don't know.

**Michaël**: So I think you could just like retrace all Sam Altman tweet and then like you read the next paper like six months after and you're like, oh yeah, he tweeted about that. Like sometimes the tweets like, oh, AI is going to be wild, or oh, neural networks are really capable of understanding. I think you tweeted that like six months ago, like when they discovered GPT-4.

**Ethan**: OpenAI is like when Ilya tweeted the consciousness tweet, they're like, goddamn, GPT-4 must be crazy.

**Michaël**: Yeah, neural networks are in some ways slightly conscious.

**Ethan**: Yeah, yeah, that was the funniest quote.

**Michaël**: Yeah, I think people at OpenAI know things we don't know yet. They're all like super hyped. And I think you mentioned as well that at least privately that Microsoft has some deal with OpenAI and so they need to some amount of money before 2024, like.

**Ethan**: Oh yeah, yeah, yeah, yeah. I mean, right, right. When the Microsoft deal happened, like Greg Brockman said, "Our plan is to train "like a 100 trillion parameter model by 2024."

**Michaël**: Okay, so that's in two years?

**Ethan**: I mean, that was in 2019, but maybe they've changed their mind after like the Chinchilla scaling lot stuff, I don't know.

## Why Ethan did not update that much from PaLM

**Michaël**: Right. And so you were not like impressed by PaLM being able to predict to like do logic on airplane things and explain jokes?

**Ethan**: In my mind, like the video scaling was like a lot worse than text basically. That's the main reason why I like AGI will probably take longer in the five years or whatever in my mind.

**Michaël**: Okay, so we need, so if we just have text, it's not enough to have AGI. So if we're a like a perfect Oracle that can like talk like us, but it's not able to do robotic things, then we don't have AGI.

**Ethan**: Yeah.

**Michaël**: Well, I guess my main like is mostly like coding. So if we get like coding, like Codex or comparative, that gets really good, then everything accelerates and engineers become very productive, and then like.

**Ethan**: I guess if you could say like, engineers get really productive at making improvements in hardware, then like, maybe that would, like, I get how that would be like, okay. Then it's really fast. Like in my mind, at least at the current, I don't see the hardware getting fast enough to be far enough on the YouTube scaling lot in less than five years from now.

**Michaël**: Thinking about hardware, we're just like humans, Googling things and using.

**Ethan**: Yeah, yeah. I get what you're saying. Like you get the Codex thing and then we use Codex or whatever to design hardware faster.

**Michaël**: You mentioned you have like DALL-E, but like for designing chips.

**Ethan**: I mean, Nvidia already uses AI for designing their chips.

**Michaël**: That doesn't make you think of timelines of 10 years or closer.

**Ethan**: It may be 10 years, but not five years. The thing I'm trying to figure out is like, try to get like a student researcher gig at like someplace so that I can just get access to the big compute during the PhD.

**Michaël**: Oh, so that's your plan. Just get out of compute.

**Ethan**: Yeah, I mean, as long as I have big compute, it doesn't matter where I'm a PhD. I mean, it kind of matters if you're like trying to start an AGI startup or whatever, but safe, safe, safe AGI startup.

**Michaël**: We're kind of on record, but I'm not sure if I'm going to cut this part. So you can say unsafe, it's fine.

**Ethan**: Yeah, no, no, no. I mean, I don't even phrase. I just phrase it as beneficial AGI.

**Michaël**: You were spotted saying you wanted unsafe AGI the fastest possible.

## Thinking about the Fastest Path

**Ethan**: No, no, no. The way I phrase it is I think I explained this last time, you have to be thinking in terms of the fastest path, because there is like extremely huge economic and military incentives that are selecting for the fastest path, whether you want it to be that way or not. So like, you gotta be thinking in terms of, what is the fastest path and then how do you like minimize the alignment tax on that fastest path? 'Cause like the fastest path is the way it's probably gonna happen no matter what, like, so it's about minimizing the alignment techs on that fastest path.

**Michaël**: Or you can just throw nukes everywhere and try to make things slower?

**Ethan**: Yeah, I guess, but I mean the people who are on the fastest path will be like more powerful, such that like, I don't know, such that they'll deter all the nukes.

**Michaël**: So you want to be, okay, so you want to just like join the winners. Like if you join the skiing team at Google.

**Ethan**: Thing I've been trying to brainstorm about is who's gonna have the fastest, who's gonna have the best funding models for supercomputers, 'cause that's the place to go and you gotta try to minimize the alignment tax at that place.

**Michaël**: Makes sense. So everyone should infiltrate Google.

**Ethan**: Yeah, so whatever place ends up with the best funding model of supercomputers try to get as many weird alignment people to like infiltrate that place as possible.

**Michaël**: So I'm kind of happy having a bunch of EA people at OpenAI now, because they're kind of minimizing the text there, but...

**Ethan**: Yeah, I kind of viewed it as all the EA people left, like 'cause Anthropic was like the most extremist EA people at OpenAI. So I almost viewed when Anthropic happened a bunch of EA people. I view as that like EA almost leaving OpenAI when Anthropic happened.

**Michaël**: Some other people came, right?

**Ethan**: Like who?

**Michaël**: I don't know. Richard Ngo.

**Ethan**: Oh, okay, okay. Yeah, yeah.

**Michaël**: It's like a team on like predicting the future.

**Ethan**: Yeah, yeah. I wanna know what the Futures Team does 'cause that's like the most out there team. I'm really curious to what they actually do.

**Michaël**: Maybe they use their GPT-5 model and predict things.

**Ethan**: Right, 'cause I mean like DALL-E, like you know about the Foresight Team at OpenAI, right?

**Michaël**: They were trying to predict things as well, like forecasting.

**Ethan**: Yeah, that's where all this scaling lot stuff came from was on the Foresight Team at OpenAI. They're gone now because they became philanthropic. But like a team called like the Futures Team that almost has a similar vibe to like a team called the Foresight Team. So I'm kind of curious.

**Michaël**: But then there's just like doing more governance things and optimal governance and maybe economics.

**Ethan**: That's what it's about, governance and economics.

**Michaël**: The guy like Richard Ngo is doing governance there.

**Ethan**: Okay.

**Michaël**: Predicting how the future works, I think is in his Twitter bio.

**Ethan**: Yeah, yeah, but I mean, that's somewhat tangential to governance, like that almost sounds like something Mike Rick Kurtz, I would say, I'm predicting how the future.

**Michaël**: My model is like Sam Altman, as like they have GPT-4. Like they published GPT-3 in 2020. So it's been like two years.

**Ethan**: Yeah.

**Michaël**: And they've been talking about like in their Q & A about like treacherous results or something like one year ago. So now they must have access to something very crazy and they're just like trying to think like how do we operate with like DALL-E 2 and their GPT-4 they have in private and how they do something without like for him in the world? I don't know. Maybe they're just like trying to predict like how to make the most money with their API or.

**Ethan**: You're saying like if they release it, it's like an infohazard? 'Cause in my mind, GPT-4 still isn't like capable enough to F up the world, but you could argue, it's like capable enough to like be an infohazard or something.

**Michaël**: Imagine you have access to something that has the same kind of gap between GPT-2 and GPT-3, but like for GPT-4 on like understanding and being general. And you don't want everyone else to copy your work. So you're just going to keep it for yourself for sometime.

## A Zillion Language Model Startups from ex-Googlers

**Ethan**: Yeah, but I feel like that strategy is already kind of screwed. Like you know about how like a zillion large language model, like a zillion Googlers have left Google to start large language model startups. Like there's literally three large language model startups by ex-Googlers now. OpenAI is like a small actor in this now because there's like multiple large language model startups founded by ex-Googlers that all like that all were founded in the last like six months. Like there's a zillion VCs throwing money at large language model startups right now. The funniest thing, like Leo Gao, he's like, we need more large language model startups because the more startups we have, then it splits up all the funding so no organization can have all the funding to get the really big supercomputer. So we just need thousands of AI during its final startups. So no one can hoard all the funding to get the really big language model.

**Ethan**: That's the, yeah, with the AI model, you just like do open source. So like there's like more startups. And so all the funding gets splitted, I guess.

**Ethan**: Yeah, you could view OpenAI was like extra big brain. We need to do. We need to like release the idea of our joiners models onto the world such that no organization could have enough compute to be such that all the compute gets more split up, 'cause a zillion, our joiners model startups will show up all at once.

**Michaël**: That's yeah, that's the best idea ever. So do you have like other gossips besides like Google's? Did you post something on Twitter about people leaving Google?

**Ethan**: Yeah, I posted a bunch of stuff. Well, I mean, and also like you saw the... I mean it's three startups, adept.ai, character.ai, and inflection.ai. They're all large language model startups founded by ex-Googlers that got a zillion dollars in VC funding to scale large language models.

**Michaël**: What's a zillion dollars like?

**Ethan**: Like greater than 60 million. Each of them got greater than 60 million.

**Michaël**: So did they know about something we don't know? And they're just like get money to replicate what Google does?

**Ethan**: Well, I mean, most of 'em, they were famous people like founder of DeepMind scaling team. Another one is the inventor of The Transformer. Another one was founded by a different person on The Transformer paper. Like, so I mean, in some ways, they have more clout than like OpenAI had or whatever.

**Michaël**: But they don't have like the engineering and old infrastructure.

**Ethan**: No, they kind of do. Like, a lot of 'em, they were like the head of engineering for scaling teams at like DeepMind or Google.

**Michaël**: So there's like another game that is in private at Google and they've been scaling huge models for two years. and they're just like,

**Ethan**: Yeah, something like that.

**Michaël**: Starting startups with their knowledge and they're just scaling and we;re just like, like peasants like us talk about papers that are released one year after and then when you turn them out.

**Ethan**: Yeah, yeah. I guess that's, I mean, I don't know how long these delays are. I mean, in my mind, like, yeah. I guess you could view it as a delay thing, 'cause like in my mind it's just like, yeah, you're right, you're right. It's probably delayed by a year, yeah.

**Michaël**: So yeah, that makes me less confident about-

**Ethan**: Oh shit. You look like a clone of Lex Fridman from the side.

**Michaël**: What?

**Ethan**: When your face is like sideways, you look like a clone of Lex Fridman.

**Michaël**: Yeah.

**Ethan**: Like, 'cause your haircut's like identical to his when

**Michaël**: I'll take that as a compliment... I started working out. So yeah, Ethan Caballero, what's the meaning of life?

**Ethan**: Probably just maximize the flourishing of all sentient beings, like a very generic answer.

**Michaël**: Right. So I've done my Lex Fridman question. Now I'm just basically him.

**Ethan**: Yeah.

## Ethan's Scaling Journey

**Michaël**: Maybe we can just go back to like stuff we know more about like your work and because you've been doing some work on scaling.

**Ethan**: Yeah.

**Michaël**: So like more general, like why are you kind of interested in scaling and like how did you started on doing research on that?

**Ethan**: I mean, I knew about the body paper when it came out. Like I remember I was at this like Ian Goodfellow talking in 2017 and he was hyped about the body paper when it came out.

**Michaël**: Which paper?

**Ethan**: The deep burning scales, predictably, empirically, yeah, it came out 2017 and then I kind, I just, that was just on the back burner and I kind of just stopped paying attention to it after a while. And then like Aran Komatsuzaki was like, no, dude, this is the thing. Like this is gonna take over everything, and this was like in 2019 when he was saying that. And then, yeah. So then when the scaling laws papers got like re-popularized through like the OpenAI stuff, then I kind of like caught onto it a little bit early via like talking with Aran.

**Michaël**: I think in 2019 was also when GPT-2 was introduced.

**Ethan**: But that was kind of before, like that was before like the scaling law stuff kind of got popularized.

**Michaël**: Right, scaling laws paper is 2020.

**Ethan**: Yeah, the very end of 2020. All right. No, no, no. Oh no, no. The scaling law paper was the very end of 2000. It was the very beginning of 2020.

**Michaël**: And you were already on this killing train since 2017.

**Ethan**: I was aware of it, but I didn't, like, I was kind of just neutral about it until like 2000, like probably the middle of 2019.

## Making progress on an Academic budget, Scaling Laws Research

**Michaël**: And yeah, now you are kind of interested in scaling because it's useful to predict kind of what the whole field of AI is going.

**Ethan**: And also it just, it's I think people underestimate how easy it is to be contrived if you're not paying attention to scaling trends and trying to like extrapolate the compute budgets and data budgets that are like, well, the compute data and data budgets like five years from now.

**Michaël**: Yeah, if you're a huge company that does a lot of budget, but maybe if you're just a random company, you don't really care about scaling law that much.

**Ethan**: Yeah, yeah. Or if you're like in academia currently or whatever, like a zillion papers that like fancy conferences are like, here's our inducted bias that helps on like our punny academic budget. And we didn't test any of the scaling asso tos to see if it's like useful when you're training a trillion parameter model on all of YouTube or whatever.

**Michaël**: You're on an academic budget as far as I know. So how do you manage to do experiments in scaling?

**Ethan**: There's like the scaling on narrative. That's like, oh, you don't need the big budget to do because you can just predict what the outcomes will be for the large scale experiments, but that's at least current. At least when that narrative got popularized, it was mostly for upstream like scaling. But the thing everyone cares about is downstream scaling.

## AI Alignment as an Inverse Scaling Problem

**Michaël**: Yeah, so if we go back for a minute on like your work in alignment, how do you think your work on scaling or generalization like kind of fits with the alignment problem?

**Ethan**: Basically, all alignment, I guess this triggers the hell outta some people. But all alignment is inverse scaling problems. It's all downstream inverse scaling problems. So it's just in my mind, all of alignment is stuff that doesn't improve monotonically as compute data and parameters increase.

**Michaël**: There's a difference between not improving and inverse scaling. Inverse scaling goes badly, right?

**Ethan**: Yeah, yeah, yeah. But I just said not improved monotonically, 'cause like sometimes there's certain things where like it improves for a while, but then at a certain point, it gets worse. So like interpretability and controllability are the two like kind of thought experiment things where you could imagine they get more interpretable and more controllable for a long time until they get super intelligent. At that point. they're like less interpretable and less controllable.

**Michaël**: Do we have benchmarks for controllability or?

**Ethan**: Like just like just benchmarks that rely on prompting is a form of like a benchmark of controllability.

**Michaël**: And kinda to summarize your take, if we were able to just scale everything well and not have this inverse scaling problem, we would get like interpretability and controllability and everything else by just like good scaling of our models. And so we'd get like alignment kind of by defaults for free?

**Ethan**: Yeah. I mean, I guess, I mean like there's stuff besides interpretability, controllability, like those are just the examples. Like what you said, you asked like what's an example where like the reason I said, I phrased it as alignment is when I said inverse scaling, I said things that don't improve monotonically, 'cause I just wanted to say like, yes, there's obvious examples where it gets worse the entire time, but there's some you could imagine where it gets good for a long time, and then a certain point, then it starts getting drastically worse. I said, all of alignment can be viewed as a downstream scaling problem. The hard part is like Dan Hendricks and like Jacob Steinhardt say like, then the hard problem though is like measurement and like finding out what are the downstream evaluations 'cause say you got like some like fancy like deceptive AI that wants to like a treacherous turn or whatever. Like how do you even find the downstream evaluations to know whether it's gonna like try to deceive you or whatever? 'Cause like when I say, it's all a downstream scaling problem, that assumes like you have the downstream test, the downstream like thing that you're evaluating it on. But like if it's like some weird deceptive thing, that's like, it's hard to even find what's the downstream thing to evaluate it on to like know whether it's trying deceive or whatever.

**Michaël**: So there's no like test lost on this deception. We don't know for sure how to measure and have a clear benchmark from this.

**Ethan**: Yeah, it's tricky. I mean, and some people say like, well, that's why you need better interpretability. You need to like find the deception circuits or whatever.

**Michaël**: Knowing that we don't know yet, like all the different benchmarks and metrics for misalignment, don't you think that your work on scaling can be bad because you're actually like speeding up timelines?

## Predicting scaling laws, Useful AI Alignment research

**Ethan**: Yeah, I get the like infohazard point of view, but like in my mind, like whether you wanna do all capabilities or alignment stuff that stands the test of time, you need really good downstream scaling prediction. Like, say you came up with some like alignment method or whatever that mitigates inverse scaling, like you need the actual functional form to know whether that thing will like keep mitigating inverse scaling when you get to like a trillion parameters or whatever. You get what I mean?

**Michaël**: I get you but like on a differential progress mindset, like Jared Kaplan or someone else will come up with those functional forms without your work.

**Ethan**: I don't know, I don't know. I mean, that's the thing though, like Anthropics (ERRATUM: it's actually a gift, and the the merch was not sent at the time of the podcast) got that paper like predictability and surprise and generative models and they're just like, it's unpredictable. We can't predict it. And I'm like, ah, you guys, nah, I don't believe.

**Michaël**: Right, so you're kind of publishing papers when you're in advance because those companies are not publishing their results?

**Ethan**: I don't know. I don't. Yeah, I don't even, I don't know if Anthropic does the delay type stuff that OpenAI supposedly does, but maybe they do, I don't know.

**Michaël**: And you were just like drawing infohazard by publishing those laws?

**Ethan**: I mean, in my mind, whether or not, I get the argument, oh it, if you wanna do capabilities work that stands a test of time or alignment work that stands a test of time, in my mind, everything that people are doing in alignment will be very contrived without the functional form too though. So it's like alignment can't make progress without it either. So it's like, you get what I mean?

**Michaël**: Another kind of view on that is that if people do impressive deploying or ML board and they're also interested in alignment, it's still a good thing. Like let's take even through AI. Even if they open source their model because they did something impressive and they talk openly about alignment under Discord and gets like a lot of people that are very smart, interested in alignment. So if you publish something and you become like a famous researcher, something in two years and you talk about alignment in two years, then it's fine.

**Ethan**: I sort of tweet stuff about alignment, I think. Yeah, I mean, I retweet stuff about alignment at least.

## Ajeya Cotra's report, Compute Trends

**Michaël**: So if we go back to thinking about predicting future timelines and kind of scaling, I've read somewhere that you think that in the next few years, we might get billion or trillion times of more compute, like 12 orders of magnitude more.

**Ethan**: Yeah, I mean, so the Ajeya Cotra report said like, it's gonna max out probably at 10 to the 12 times as much compute as like the amount of compute in 2020, probably like 2070 or something like that. The one issue I have with the JS model is that like, she does, what does she do? It's like it's flops per dollar times willingness to spend its total flops that are allocated to pre-training runs. Problem is like, for the big like foundation models, like 10 of the 15 perimeter miles of the future or whatever, you're probably gonna need high pie like memory bandwidth between all like memory bandwidth and compute bandwidth between all the compute, which means it has to be on a supercomputer. So it's not just the flaps. It basically what really matters, at least if you're assuming it's like big, like 10 of the 15 parameter foundation models or whatever, like the speed of the fastest supercomputer is what matters, not just the total flaps that you can allocate, because if like all the flaps don't have good communication between them, then they aren't really useful for training like 10 of the 15 parameter model or whatever. Once you get to 10 of the 15 parameters, like there isn't much reason to go beyond that. And at that point, then you just have multiple models with 10 of the 15 parameters and they're like doing some crazy open ended, like Ken Stanley stuff and a multi-agent simulator after you do that. Like if they mentioned became like you do the 10 of the 15 parameter model feature and all YouTube, and then after that, you'll have like hundreds of 10 of the 15 parameter models that all just duke it out in like a Ken Stanley open-ended simulator to like, get the rest of the capabilities or whatever, like once they're in the Ken Stanley open-ended stimulator, then you don't need high compute bandwidth between all those individual, like 10 of the 15 parameter models, like duking it out in the simulator. They can just, each one, they only needs like 10. It only needs high compute bandwidth between like its own parameters. Like, it doesn't need high compute bandwidth between itself and the other like agents or whatever. And so in there, the flops where you could use all the flops for like the multi-agent simulation, but you only need high compute bandwidth within each agent.

**Michaël**: So you need a lot of bandwidths to train models because of the prioritization thing, but you only need flops to simulate on different things at the same time?

**Ethan**: Yeah, you only need high compute bandwidth within an individual brain, but like if you have multiple brains, then you don't need high compute bandwidth between the brains.

**Michaël**: And what was that kind of simulator you were talking about, the Kenley?

**Ethan**: Like Ken Stanley, the open-ended guy.

**Michaël**: I haven't seen that.

**Ethan**: Ken is like the myth day objective open-endedness, like Can Stanley's, Jeff Cones, like all that stuff. It's like, I don't know. Just Google, like Can Stanley open ended at some point. You've probably heard of it, but it's not like registering what I'm referencing.

# Optimism, conclusion on alignment

**Michaël**: Okay, so maybe one kind of last open-ended question. On a scale from Paul Christiano, Eliezer Yudkowsky, Sam Altman, how optimistic are you?

**Ethan**: Definitely not like Eliezer, or a doomer type person. I guess probably Paul Christiano is most similar. I mean, I feel like Paul Christiano is in the middle of the people you just said.

**Michaël**: Right. Yeah. So you are less optimistic than Sam Altman?

**Ethan**: Well, yeah, I mean, basically, I think deceptive AI is probably gonna be really hard.

**Michaël**: So do you have like one less monologue or sentence to say about why scaling is a solution for all alignment problems?

**Ethan**: Like just all alignment can be viewed as an inverse scaling problem. Like it all revolves on just mitigating inverse scaling, but also you have to make sure you have like the right downstream things that you're evaluating, like the inverse scaling and like part of what makes it hard is like you might need to do like fancy thought experiments on alignment, like counterintuitive thought experiments on alignment forum to find what are the downstream... to find what are the the downstream tests that you should be evaluating. Like whether or not there's inverse scaling behavior on those.

**Michaël**: Awesome, so we get the good version, as last sentence, and that's our conclusion. Thanks Ethan for being on the show.

Ethan Caballero-by The Inside View-date 20220505.md


# Interview with AI Researchers NeurIPSorICML_7oalk by Vael Gates

**Interview with 7oalk, on 3/20/22**

**0:00:03.2 Vael:** Alright, my first question is, can you tell me about what area of AI you work on in a few sentences?

**0:00:08.5 Interviewee:** Yeah, I study biologically-inspired artificial intelligence of building models of biological intelligence, mostly with the visual system, but also with cognitive functions, and using those models to understand differences between humans and machines. And upon finding those differences, the hope is that we can build more human-like artificial intelligence and in the process develop models that can better explain the brain.

**0:00:46.1 Vael:** Interesting, yep. I was in that space for a while. The dual\-- AI\--

**0:00:49.1 Interviewee:** Oh, cool. I saw you worked with Tom Griffiths.

**0:00:54.0 Vael:** Yeah, that\'s right, yep.

**0:00:55.0 Interviewee:** Yeah, very much so. Cool.

**0:00:56.5 Vael:** Alright, so my next question is, what are you most excited about in AI, and what are you most worried about? In other words, what are the biggest benefits or risks of AI?

**0:01:06.9 Interviewee:** So I\'m most excited about AI, not for most of the applications that are getting a lot of praise and press these days, but more so for convergences to biomedical science. I think that the main driver of progress in AI over the past decade has been representation learning from large-scale data. And this is still an untapped area for biomedical science, so we really don\'t know what elements of biology you can learn predictive mappings for. So, for instance, given an image of a cell\'s morphology, can you predict its genome? Can you predict its transcriptome? Etcetera. So I think that what I\'m most excited about is the potential for AI to completely transform the game of drug discovery, of our ability to identify the causes and identify targets for treating disease.

**0:02:18.4 Vael:** Nice.

**0:02:18.7 Interviewee:** What I\'m most afraid of is in the game of AI that is most popular right now. The NeurIPS CDPR game, which\... lots of people in the field have pointed out the issues with biases. And also\... it\'s late here, so I\'m not thinking of a good term, but this Clever Hans, Mechanical Turk-ish nature of AI, where the ability to solve a problem that seems hard to us can give this sheen of intelligence to something that\'s explained a really trivial solution. And sometimes those trivial solutions, which are available to these deep learning networks which exploit correlations, those can be really painful for people. So in different applications of using AI models for, let\'s say screening job applicants, there\'s all these ethical issues about associating features not relevant to a job, but rather just regularities in the data with a predictive outcome. So that\'s a huge issue. I have no expertise in that. It\'s definitely something I\'m worried about. Something that\'s related to my work that I\'m super worried about\... I mentioned I do this biological to artificial convergence, this two-way conversation between biological and artificial intelligence. The best way to get money, funding in academia, for that kind of research is going through defense departments. So one of the ARPA programs\-- IARPA, DARPA, Office of Naval Research. So I know that the algorithms built for object tracking, for instance, could be extremely dangerous. And so I build them to solve biological problems, and then I scale them to large-scale data sets, large data sets to get into conferences. But, you know, easily, somebody could just take one of those algorithms and shove them into a drone. And you do pretty bad stuff pretty easily.

**0:04:41.2 Vael:** Yeah.

**0:04:43.2 Interviewee:** Yeah. I guess one more point related to the biomedical research. There\'s this fascinating paper in Nature and Machine Intelligence. One of the trends right now for using AI in biology is to predict protein folding, or to predict target activity for a molecule or a small molecule. So when you do biomedical research, you want to screen small molecules for their ability to give you a phenotype in the cell. And so you can just use a multilayer perceptron or graph neural network, transformer, whatever, to learn that mapping, and do a pretty good job it turns out; it\'s shocking to me. But what you can do is you can either predict\-- let\'s find the molecule that\'s going to have the best target site activity, that\'s the best therapeutic candidate. But you can also flip it, you can optimize it in the opposite direction. (Vael: \"Yeah\--\") You saw that.

**0:05:48.2 Vael:** Yeah. I saw that \[paper\].

**0:05:49.0 Interviewee:** It\'s so obvious, but easily, easily, easily you could use any of these biomedical applications for the nefarious things.

**0:06:00.5 Vael:** Yeah, yeah, that\'s right. That was a very recent paper right?

**0:06:02.5 Interviewee:** Yeah, it just came out like last week.

**0:06:05.1 Vael:** Got it. Yeah. Alright, we\'ve got excited and risks. My next question is talking about future AI. So putting on a science fiction, forecasting hat. So we\'re 50 plus years into the future, so at least 50 years in the future, what does that future look like?

**0:06:22.6 Interviewee:** Okay. Optimistically, we\'re all still here. So the reason why I got into this field is because I think it\'s the biggest force multiplier there is. So all of the greatest problems that\-- the existential crises we face today, there\'s a potential for machine learning to be part of the solution to those problems. So focusing just for a moment on health, biomedical research, which is where I\'m mostly interested right now, 50 years, I think that we\'re going to be to a point where we know we have drugs that are completely uninterpretable to humans. We have solved disease, we have cured disease, but we don\'t know why. We\'ve also created a potential issue, byproducts from that, adversarial effects, for lack of a better phrase, by curing cancer through some black box method. That\'s going to yield other problems, kind of like putting your finger in the dam. So I think that in 50 years from now, we will have machine learning-based solutions to some of the greatest health problems: cancer, aging, neurodegenerative disease that face us, but we won\'t understand those solutions. And so that\'s potentially the next frontier, which is to develop interpretable drugs. So I imagine that science will change. The paradigm of science will shift where there are no longer bench scientists, but instead robotics is how science is done. So scientists ask high-level questions, and then biology becomes a search problem where you just, akin to a tricorder from Star Trek where it just solves, and you move onto the next question. You can imagine that framework being applied to other problems. Climate change, hopefully. I\'m not optimistic, but hopefully, you could have solutions to climate change. But there will be a black box. And when you have a black box, there\'s always issues with unintended effects. So I guess 50 years from now, we\'ll have, rather than thinking about AGI and all that stuff, we\'ll have solutions to some of our grandest challenges today, but those solutions may bear costs that are unexpected.

**0:09:40.8 Vael:** Yeah, that makes sense. I have few follow-ups. So you mentioned existential risks facing humanity. What are those?

**0:09:51.5 Interviewee:** So there\'s climate change. There\'s nuclear war, which is the byproduct of power. There is poverty and hunger, famine. With the risk of sounding like some VC, I think that there are chances to disrupt solutions to each of those problems using machine learning. For instance, for energy, already folks at DeepMind are using reinforcement learning to drive the development of new forms of fusion to search for better forms of fusion. Who knows if this is scalable, but I think with\... it\'s not going to be better algorithms, it\'s going to be better integration of the toolset for studying fusion and these machine learning methods to make it more rapid to test and explore those questions that are related to that field, which I don\'t know. I think that will lead to positive solutions. That would potentially yield lower-emission or no-emission, high-energy solutions that can power homes without any emissions. Similarly, with cars\... self-driving, as much as I think that Elon Musk is barking up the wrong tree\-- going LIDAR-free, \[inaudible\] approach to self-driving, that will be a solved problem within 10 years. And that\'s going to help a lot the emissions issue. So energy and global warming are kind of tied together. And I think that there\'s lots of different applications of machine learning and AI that can help address those problems. Likewise for famine, for world hunger. Genetically-engineered organisms are a good approach to that problem. What I mentioned about for solutions to biomedical problems, disease, etc.\-- treating those as search problems and bringing to bear the power of artificial intelligence. I think you could probably adopt a similar approach for engineering organisms that are tolerant to weather, especially the changing climate and other diseases and bugs, etcetera. I think there\'s going to be some overlap between the methods of work for each of these areas. I guess I didn\'t mention COVID and pandemics. But again, that falls into the same framework of treating these existential problems as search problems, no longer doing brute force, then science. And instead resorting to black box solutions that are discovered by large scale neural network algorithms. Did I miss any? Oh, war. Yeah, I don\'t know. Okay, so here\'s an answer. Whoever has these technologies, who\'s ever able to scale these technologies most rapidly, that\'s going to be almost like an arms race. So whoever has these technologies will be able to generate so much more value than countries that don\'t have these technologies that economies will be driven by the ability to wield appropriate artificial intelligence tools for searching for solutions to these existential issues.

**0:14:36.1 Vael:** Got it, thanks. Alright, so my next question is\... so this is kind of a spiel. Some people talk about the promise of AI, by which they mean many things, but that the thing I\'m referring to is something like a very general capable system. So the cognitive capacities that can be used to replace all current day human jobs. Whether or not we choose to replace human jobs is a different question, but having the kind of capabilities to do that. So like, imagine\... I mostly think about this in the frame of 2012, we have the deep learning revolution with AlexNet, and then 10 years later, here we are, and we have systems like GPT-3, which have kind of weirdly emergent capabilities, that can be used in language translation and some text generation and some coding and some math. And one might expect that if we can continue pouring all of the human effort that we\'ve been pouring into this, with all the young people and the nations competing and corporations competing and algorithmic improvements at the rate we\'ve seen, and like hardware improvements, maybe people get optical or quantum, that we might get\... scale to very general systems, or we might hit some sort of ceiling and need to do a paradigm shift. But my question is regardless of how we\'d get there, do you think we\'ll ever get very general systems like a CEO AI or a scientist AI and if so, when?

**0:15:55.3 Interviewee:** I don\'t think it\'s worth it. I see the AI as\... look, I grew up reading Asimov books, so I love the idea of \"I, Robot\", not with Will Smith, but solving all these galaxy conquest issues and questions. But I don\'t think it\'s worth it because\... I see this as a conversation between human experts, the main experts, and artificial intelligence models that are just going to force multiply the ability of human experts to explore problems. And maybe in a 100 years, when we\'ve explored all of these problems, then we\'ll be so bored that we say, \"Can we engineer something that contains all of these different domain expert AIs in one, in a way that can\-- those AIs can respond to a question and affect behavior in the world and have their own free will.\" It\'s more of a philosophical pursuit that I don\'t even, I don\'t\... okay, so from an economic standpoint, I don\'t know who\'s going to pay for that. Even the military would be like, \"All I want is a drone that\'s going to kill everybody.\"

**0:17:30.9 Vael:** Yeah, Yeah, cool. So I have some counterpoints to that. So I\'m not actually imagining necessarily that we have AI that is conscious or has free will or something. I think I\'m just imagining an AI that is, has a, very capable of doing tasks. If you have a scientist AI then\...We were talking about automating science, but in this case, it\'s not even relying on an expert anymore. Maybe it defers to an expert. It\'s like, do you want me\... it looks to the expert\'s goals, like, \"Do you want me to solve cancer?\" Or like a CEO, it has shareholders. And I think that there is actually economic incentive to do this. DeepMind and OpenAI have explicitly said\... I think that they\'re trying to make artificial general intelligence, or like things that can replace jobs, and more and more replace jobs. And I think we\'re going in that direction with Codex and stuff, which is currently an assistant tool, but I wouldn\'t be surprised if in the end, we didn\'t do much coding ourselves.

**0:18:24.9 Interviewee:** Okay, so yeah, I agree with what you said. Right, it\'s going to change the job\... So strike from the record what I just said. It\'s going to change the job requirements. That would be a good feature in my opinion, where we no longer have to code because instead we describe our program. And again, it\'s a search problem to find the right piece of code to execute the concept that we have. We describe the medical problem that we\'re interested in, and we search within whatever imaging modality or probe that we\'re using to study that system. We can search for that solution. Where I\'m not so sure is, we have this huge service industry, and you mentioned shareholders and I count that as one of them. I think when it comes to service, there\'s this quality, there\'s this subjectiveness where for instance, in medicine, talking about radiology or pathology or even family medicine. I think there will be tools to give you diagnoses, but they\'re always going to come with a human second opinion. So, from that standpoint, it\'s going to be a bit of a science fair, kind of a sideshow to the expert. I don\'t think that human experts in service, medical or business, etcetera, are going to be pushed aside by AI, because ultimately, I think if there\'s a different answer given by the artificial pathologist versus the human pathologist, the human pathologist will double check and say, \"You are wrong, AI\" or \"You are right, AI\". But the patient will never receive the artificial intelligence answer. So I think in some fields, there\'s going to be a complete paradigm shift in how research is done, or how work is done, research for one. Driving, supply chain, transportation, that\'s another example. Which will be great. I think that would be great, it will push down costs, push down fatalities, morbidity, and probably be a net good. But yeah, for a shareholder, I cannot imagine a scenario in which a CEO who has to figure out the direction of the company, has to respond to shareholders would ever say, like, \"This time around, I\'m just going to give an AI a GPT-6 report for the markets that we should enter, the new verticals we should work on, etcetera.\" Maybe it would be just be a tool to get feedback.

**0:21:54.2 Vael:** Yeah. So it seems pretty true to me that we started\... our current level of AI, which is machine learning, that they started as tools and we use them as tools. I do\-- it sort of feels to me that as we progr\--\... we\'ve been working on AI for under 100 years, and deep learning revolution is quite new. And it seems to me like there are economic incentives towards making things more convenient, towards more automation, towards freeing up human labor to do whatever it is that people could do. So I wouldn\'t be surprised if eventually we reached a point where we had systems that were capable enough to be CEO. We could be like, \"Alright AI, I want you to spin me up a company,\" and maybe you would be able to do that in some very different future, I would imagine.

**0:22:38.2 Interviewee:** Ah, so see\-- okay, so this is a question. Creativity versus capturing the regularities or the regulations of a field. So I can imagine an AI VC, easy. Or in finance, yeah. Finance, I can imagine that being overtaken by AI. I\'m thinking CEO, spinning up your own company, identifying weakness in a field, or maybe some potential tool that can monopolize a field. That requires creativity. And I know there\'s this debate about creativity versus just interpolation in high-dimensional space of what GPT-3 is doing, and yes, I\'ve seen some cool-looking art, but I don\'t know. That seems like a\... I think if you talk to me about this, if we talked about this for another week, maybe you would convince me. But I\'m not quite there yet that I can imagine deep learning being able to do that. Although, now that I say that, maybe I\'m just thinking about it wrong. Where most businesses just amount to, \"What has worked somewhere else? Can that frame of reference work in this field? Can you disrupt this field with what\'s worked there?\" So maybe, maybe. Okay, I agree with what you\'re saying. I\'m evolving here, on the spot. I can see it, I can see that. I guess the one field that completely, that I\'m going to push back on is medicine, where I think that part of service need a human to tell you what is up. If you don\'t have a human, then I can\'t imagine a machine diagnosis would ever fly. Maybe in 200 years, that would fly. There would have to be a lot of development of the goal for that to happen.

**0:24:45.3 Vael:** Yeah, I think there\'s a number of things going on here. So I don\'t know that we\'ll be able to achieve\... I do think a CEO is very high cognitive capabilities, you need to be modeling other people, you need to be talking to other people, you need to be modeling other people modeling you, you need to have multi-step planning, you need to be doing a whole bunch of very advanced\...

**0:25:03.6 Interviewee:** Yeah, convincing people, interpersonal relationships. Yeah.

**0:25:06.2 Vael:** Yeah, there\'s a whole bunch of stuff there, and I don\'t that we\'re\... we\'re not near there with today\'s systems for sure. And I don\'t know that we can get there with the current deep learning paradigm, maybe we\'ll need something different. I think that\'s a possibility. I think what I\'m doing is zooming way out where I\'m like, \"Okay, evolution evolved humans, we have intelligence.\" I think that humans are trying pretty hard to make this thing happen, make up whatever intelligence is happen. And like, if we just continue working on this thing, absent some sort of huge disaster that befalls humanity, I kind of expect that we\'ll get there eventually.

**0:25:39.6 Interviewee:** Yeah.

**0:25:40.2 Vael:** And it sounds like, you said that like, maybe eventually, in like 200 years or something?

**0:25:45.1 Interviewee:** Definitely, definitely. So, I agree with what you said, science is incremental. And I think we have not found any ceilings in the field of modeling intelligence. We\'ve continued to move. Even though we\'re kind of arguably asymptoting with the current paradigm of deep learning, we have a proof of concept that it\'s possible to do better, which is our own brains. And so, maybe we just need to have neuromorphic hardware or whatever, who knows, who knows? So I think you\'re right, and if you can operationalize, just like you did very quickly, what it takes to be a CEO. If you can build models of each of those problems and if you can do what people do, which is benchmark those models and just start to make them better and better, yeah, I can imagine having some\... It sounds crazy to even visualize that, but yeah, having an automated CEO, I think so. I think the part that\'s probably the most tractable is finding the emerging markets or the, like I said, the technologies that have proven to be flexible, flexibly applied, or the tricks, let\'s say, that have worked in multiple fields, and then identifying the field where it has just not been applied yet. Kind of that first mover type advantage. I think that\'s where AI could\... I can definitely imagine an AI identifying those opportunities. Almost like drug discovery, right? It\'s just a hit. And you\'d have a lot of bad hits, and like, \"This is bullshit. Nobody cares about knee pads on baby pants,\" which is like the WeWork founder before WeWork, but you also have some good hits. So the interpersonal stuff, yeah, that\'s like 200 years off.

**0:27:50.6 Interviewee:** Although you have chat bots. So I\'m working on my own startups, a lot of this is trying to convince people to spend their time, that this is worthwhile, that they should leave their current stable positions to do this, that they should work\... They should do the work for you. They should drop what they\'re doing and do what you need them to do right now, and that you\'re going to help them eventually. And a lot of that is empathy and connection, and there\'s no evidence that we have yet that we can build a model that can do that stuff. So yeah, that\'s a really interesting idea, though. And I do think if you go through and operationalize each of those problems, you could make progress. So, 200 years, yes, I agree with you.

**0:28:56.7 Vael:** Alright, so maybe we\'ll get something like this in 200 years.

**0:29:00.1 Interviewee:** Yeah.

**0:29:00.6 Vael:** This is mainly an argument about the belief in\... like, faith in humans to follow economic incentives towards making life more convenient for themselves. That\'s where it feels like it\'s coming from in my mind. Like 200 years\... Humans have existed for a long time, and things are moving very quickly. 10,000 years ago, nothing changed from year to year, from lifetime to lifetime. Now, things are moving very quickly, and I\'m like, \"Hm.\"

**0:29:23.1 Interviewee:** I\'m happy to talk with you, like, add time to this interview. But I\'m just curious, what do you think? Do you think it\'s going to be sooner than 200 years? 10 years? 20 years?

**0:29:37.1 Vael:** I think there\'s a possibility that it could happen\... sooner than 200 years. \[chuckle\] I\'m trying to figure out how much of my personal opinion I want to put in here. There\'s a study, which was like, \"Hello, AI researchers. What do you think timelines?\" And there\'s several other kind of models of this, and a lot of the models are earlier than 70 years. So, this is\... possibilities. I think it could be within the next 30 years or something. But I don\'t actually know, there\'s like probability estimates over all of these things. I don\'t know how these things work.

**0:30:13.2 Interviewee:** Okay. Yeah. Some McKinsey consultant put like little probability estimates.

**0:30:18.9 Vael:** Yeah, a little. It\'s a little bit more than that. There\'s a lot of surveying of AI researchers and then some people have some more fancy models, I can send you to them afterwards, you can see\...

**0:30:27.3 Interviewee:** Please, please. Yeah.

**0:30:28.6 Vael:** Great.

**0:30:29.4 Interviewee:** And like I said, I\'m happy to add time to this, so\...

**0:30:32.3 Vael:** Awesome. Alright, so my next question is talking about these very\... highly intelligent systems. So in your mind, maybe like 200 years in the future. And so I guess, yeah, this argument probably feels a little bit more close to home if your timelines are shorter, but like, regardless. So, imagine we\'re talking about the future. We have our CEO AI, and I\'m like, \"Alright, CEO AI, I wish for you to maximize profits and try not to run out of money or try not to exploit people and try to not have these side effects.\" And this currently, obviously, is very technically challenging for many reasons. But one of the reasons I think, is that we\'re currently not very good at taking human values and human preferences and human goals, and then putting them into mathematical formulations such that AIs can optimize over them. And I worry that AIs in the future will continue to do what we tell them to do and not as we intend them to do. So, we\'re not that we won\'t be able to take all of our preferences\... it will continue to be hard to take our preferences and put them into something that can optimize over. So what do you think of the argument, \"Highly intelligent systems will fail to optimize exactly what their designers intended them to, and this is dangerous\"?

**0:31:46.6 Interviewee:** \...Yeah. Well, so\... Okay. Exactly what they\'re intended to do, so presumably this means, you start to test generalization of your system and some of holdout set, and it does something funky on that set. Or there\'s some attack and it does something funky. The question is\... so presumably it\'s going to do superhuman\... well, let\'s just imagine it\'s superhuman everywhere else, except for this one set. Now it\'s a real issue if humans are better than this AI on that one problem that it\'s being tested on, this one holdout problem\...

**0:32:32.4 Vael:** I think in this argument, maybe we have something like human-level intelligence in our AI and we\'re just trying to get it to do any goal at all, and we\'re like, \"Hello, I would like you to do a goal, which is solve cancer.\" But we are unable to properly specify what we want and all the side effects and all the things.

**0:32:47.0 Interviewee:** Yeah, yeah, yeah. Yeah. No, that\'s exactly where I\'m thinking in. So imagine that you are studying a degenerative disease and you figure out, or you ask your AI, \"Can you tell me why ALS happens in focal cells\" So your AI gives you an answer, \"Well, it\'s because of some weird transcription that happens on these genes.\" So if you have a small molecule that addresses that, then you\'re going to have normal-acting genes and normal phenotype in those cells. Okay, so byproduct could be, after 10 years, some of what was happening in those genes is natural for aging, and so you introduce some cancer, because what you did to fix neurodegenerative disease is so unnatural that it has that kind of horrible side effect. Now, is that a problem? (Vael: \"Yeah, I think\--\") Yes. Yeah, but\--

**0:33:52.7 Vael:** I think in this particular scenario, maybe the AI would know that it was a problem. That like 10 years out, you would not be unhappy. I mean, you would be unhappy.

**0:34:03.6 Interviewee:** You would be unhappy. Yeah, so okay, even if it does know it. Is that\... So yes, that\'s an issue, but it\'s also significantly better than where we are today. And let\'s say the expected lifespan for ALS is five years, and now you\'re saying 10 years, but you\'re going to have this horrible cancer. That\'s a bad trade-off, bad decision to force somebody to make, but is undoubtedly progress. So I think black boxes.. and that would not be a black box, I believe. Black box would be like when\... you don\'t know, you find out about this 10-year cancer 10 years after it passes clinical trials, you know? So this would be like, almost a very good version to AI that can tell you like, \"I found a solution, but given the current technology, this is the best we can do.\" So I would think that is a fantastic future that we\...

**0:35:00.8 Vael:** I think that\'s right.

**0:35:02.0 Interviewee:** The black box version, I think is also a fantastic future because it would still represent meaningful progress that would hopefully continue to be built on by advances in biomedical technologies and AI. So for this specific domain, there\'s obvious issues with black box. Like, if you\'re going to use AI to make any decisions about people, whether they can pass a bar for admissions, this or that or the other thing, there\'s going to be problems there. But for medicine at least, I think black boxes should be embraced.

**0:35:45.2 Vael:** Yeah, I think the scenario you outlined is actually what\-- I would say that that system is doing exactly what the designers intended them to. Like the designers wanted a solution, and even if we currently don\'t have best solution, it gave it the best we had. I\'m like, \"That seems successful.\" I think an unsuccessful version would be something like, \"Alright, I want you to solve this disease.\" But you forget to tell it that it shouldn\'t cause any other side effects that it certainly knows about and just doesn\'t tell you, and then it makes a whole bunch of things worse. And I think that would be more of an example of an unsuccessful system doing something that didn\'t optimize exactly what they\'re intended.

**0:36:19.0 Interviewee:** Yeah, yeah, so that\'s just a failure. Okay, so here. Let\'s say it only does that on a certain type of people, racial profiles. Okay, so then the upshot of automating AI is that, sorry, automating science, is that it should make it a lot cheaper to test a far wider range of people. So just like, now the current state of the art for image\-- object classification is to pre-train on this 300 million image data set, this JFT-300 or whatever. Likewise, you could develop one-billion-person data sets of IPS cells, which are like stem cells. Cool science fiction there that had never happened, but then you would have this huge sample where you would avoid these kinds of horrible side effects.

**0:37:28.7 Vael:** Cool. I still think\-- yeah. I\'ll just\-- (Interviewee: \"No, tell me. Tell me, you still think what?\") Ah, no, no, I, well. I\'m trying to think of how I want to order this, and we\'re running out of time and I do actually need to go afterwards. (Interviewee: \"Oh, okay.\") So I think I want to get to my next question, which does relate to how I think this previous\... where I\'m like, \"Hmm\" about this previous question. So assume we have this powerful AI CEO system, and it is capable of multi-step planning and is modeling other people modeling it, so it has like a model of itself in the world, which I think is probably pretty important for anything to be deployed as a CEO AI, it needs to have that capability. And it\'s planning for\... And its goal, people have told it, \"Alright, I want you to optimize profit with a bunch of these constraints,\" and it\'s planning for the future. And built into this AI is the idea that we want\... That it needs to have human approval for making decisions, because that seems like a basic safety feature. And the humans are like, \"Alright, I want a one-page memo telling us about this potential decision.\" And so the AI is thinking about this memo and it is planning, and it\'s like, \"Hmm, I noticed that if I include some information about what\'s going to happen, then the humans will shut me down and that will make me less likely to succeed in the goal that\'s been programmed into me, and that they nominally want. So why don\'t I just lie a little bit, or not even a lie, just omit some information on this one-page memo, such that I\'m more likely to succeed in the plan.\" And so this is not like a story about an AI being programmed with something like self-preservation, but is a story about an agent trying to optimize any goal and then\...

**0:38:55.2 Interviewee:** Like shortcuts, yeah.

**0:38:56.5 Vael:** And then having an instrumental incentive to preserve itself. And this is sort of an example that I think is paired with the previous question I was asking, where you\'ve built an AI, but you haven\'t fully aligned it with human values, so it isn\'t doing exactly what the humans want, it\'s doing what the humans told it to do. And so, what do you think of the argument, \"Highly intelligent systems will have an incentive to behave in ways to ensure that they are not shut off or limited in pursuing their goals, and this is dangerous.\"

**0:39:38.5 Interviewee:** Okay, so if we are going to be using supervision to train these systems, what you\'re describing is a shortcut. So, highly intelligent systems exploiting a shortcut like you\'re talking about, which would be like, \"Let\'s achieve a high score by doing the wrong thing\...\"

**0:40:00.4 Vael:** Yeah, yep.

**0:40:00.5 Interviewee:** Can be corrected by having the right data set. Not saying that it would be this simple, input-output, learn the mapping between the two, but we\'ll just imagine that it is. If you have any kind of low-level bias in your data set, or high level bias, if this is the case, neural networks will learn to find that bias. So it\'s a matter of having a broad enough data set where that just basically, statistically, what you\'re talking about would not exist.

**0:40:35.4 Vael:** Interesting. I feel like\--

**0:40:37.3 Interviewee:** I don\'t think that would be a problem if you can design the system appropriately.

**0:40:41.2 Vael:** Yes, I totally agree this would not be a problem if you can design the system appropriately.

**0:40:44.9 Interviewee:** The data set appropriately.

**0:40:46.6 Vael:** Ah, the data set appropriately. Interestingly, on one of the examples that I know of where we have, like you set the AI, it\'s like a reinforcement learning agent, and you\'re like, \"Reinforcement learning agent, optimize the number of points.\" And then instead of trying to win the race, which is the thing you actually wanna do, it just like finds this weird local maximum.

**0:41:02.8 Interviewee:** Yeah, so for RL, that\'s like the ultimate example of shortcuts. Because there, what you have are extremely\... So, for video games, if you\'re talking about a race, you have extremely limited sensor data, sorry, low dimensional sensor data. And you have few ways of controlling, manipulating that sensor data. So, let\'s say there\'s two local minima that can be found via optimization, and one is, \"go for red, because red wins,\" and the other is, \"Drive your car to avoid these obstacles and pass the finish line,\" which just happens to be red. Of course, the shortest solution will be the one chosen by the model. So I agree, shortcuts are ubiquitous, and of course, they will become more and more advanced as we move into these other domains. But I think what you\'re describing is just a problem with shortcuts. And so it becomes a question of, can you induce enough biases in your model architecture to ignore those shortcuts? Can you design data sets such that those shortcuts wash out via noise? Or can you have human intervention in the training loop that says, \"Keep working, keep working. I don\'t accept this answer?\"

**0:42:30.3 Vael:** Great, yeah. Cool. Have you heard of AI alignment?

**0:42:37.1 Interviewee:** Yeah, yeah, I have. Well, I\'ve seen OpenAI talking about that a ton these days.

**0:42:42.3 Vael:** Great. Can you explain the term for me?

**0:42:46.5 Interviewee:** So this is like, you have some belief about how a system should work, and the model is going to do what it does, and so alignment means you\'re going to bring the system into alignment with your belief about how it should work. It\'s essentially what you\'re talking about with the shortcut problem. So, that\'s it, yeah.

**0:43:09.4 Vael:** Great, thanks. Yeah, I think I mostly think the problem of aligning systems with humans is going to be\... One of my central thing is that I think it will be more difficult than it seems like you think, which you were like, \"Oh well, we can do it with having better data sets or we can do it with denoising things,\" and I\'m like, I don\'t actually know if\...

**0:43:31.1 Interviewee:** I don\'t know about denoising\-- I don\'t mean to make this sound trivial, because it certainly is not. But to give you an example, MNIST is so confounded with low-level cues. You don\'t need to know \"six\" to recognize six, you just need to know this contour, right? Is that\... only in six do you have the top of the six and then a continuous contour. So, that\'s what I mean. So if you can design a data set where those don\'t exist, then you\'re golden. But usually\-- In the real world, that doesn\'t happen that much because we have these long-tailed distributions of stuff. So then you have to induce biases in your architecture, and this comes back to human vision, sorry, human intelligence. It\'s alignment with human decision-making, because we have all these biases through development through our natural\-- our genomes, through neural development that make us able to interact with the world in such a way where we don\'t just go for the red thing, where we\'re not vulnerable to adversarial examples. So I don\'t mean to trivialize it, I only mean to say computationally, the problem is biases within the data set, and when you use gradient descent, if that is the learning algorithm that you used to train the model that you\'re talking about, then that\'s what you\'re going to be fighting against.

**0:45:06.0 Vael:** Cool. Yep, makes sense.

**0:45:07.6 Interviewee:** Yeah.

**0:45:08.3 Vael:** Alright. I think my last question is just, how has this interview been for you and have you changed your mind on anything during it?

**0:45:19.1 Interviewee:** Yeah, well, it\'s great. I never thought of an AI CEO. I think that\'s super fascinating. Yeah, it was super fun. Yeah, it was great.

**0:45:33.2 Vael:** Great. Wonderful. I think I\'m going to call it now, just because we both have places to be. And I will send along the money, and then I\'ll also send along some resources since you said you were curious about my opinions on timelines.

**0:45:47.6 Interviewee:** I definitely am. Thanks so much for your time, I really appreciate it.

**0:45:51.1 Vael:** Yeah, thanks so much for yours.

**0:45:53.9 Interviewee:** Okay. All right, take care. Have a good night.

**0:45:55.7 Vael:** Bye.

NeurIPSorICML_7oalk-by Vael Gates-date 20220320.md