Conversation with Paul Christiano

AI Impacts talked to AI safety researcher Paul Christiano about his views on AI risk. With his permission, we have transcribed this interview.
Participants
Paul Christiano — OpenAI safety teamAsya Bergal – AI ImpactsRonny Fernandez – AI ImpactsRobert Long – AI Impacts
Summary
We spoke with Paul Christiano on August 13, 2019. Here is a brief summary of that conversation:
AI safety is worth working on because AI poses a large risk and AI safety is neglected, and tractable.Christiano is more optimistic about the likely social consequences of advanced AI than some others in AI safety, in particular researchers at the Machine Intelligence Research Institute (MIRI), for the following reasons:The prior on any given problem reducing the expected value of the future by 10% should be low.There are several 'saving throws'–ways in which, even if one thing turns out badly, something else can turn out well, such that AI is not catastrophic.Many algorithmic problems are either solvable within 100 years, or provably impossible; this inclines Christiano to think that AI safety problems are reasonably likely to be easy.MIRI thinks success is guaranteeing that unaligned intelligences are never created, whereas Christiano just wants to leave the next generation of intelligences in at least as good of a place as humans were when building them. 'Prosaic AI' that looks like current AI systems will be less hard to align than MIRI thinks: Christiano thinks there's at least a one-in-three chance that we'll be able to solve AI safety on paper in advance.A common view within ML is that that we'll successfully solve problems as they come up.Christiano has relatively less confidence in several inside view arguments for high levels of risk:Building safe AI requires hitting a small target in the space of programs, but building any AI also requires hitting a small target.Because Christiano thinks that the state of evidence is less clear-cut than MIRI does, Christiano also has a higher probability that people will become more worried in the future. Just because we haven't solved many problems in AI safety yet doesn't mean they're intractably hard– many technical problems feel this way and then get solved in 10 years of effort.Evolution is often used as an analogy to argue that general intelligence (humans with their own goals) becomes dangerously unaligned with the goals of the outer optimizer (evolution selecting for reproductive fitness). But this analogy doesn't make Christiano feel so pessimistic, e.g. he thinks that if we tried, we could breed animals that are somewhat smarter than humans and are also friendly and docile.Christiano is optimistic about verification, interpretability, and adversarial training for inner alignment, whereas MIRI is pessimistic.MIRI thinks the outer alignment approaches Christiano proposes are just obscuring the core difficulties of alignment, while Christiano is not yet convinced there is a deep core difficulty.Christiano thinks there are several things that could change his mind and optimism levels, including:Learning about institutions and observing how they solve problems analogous to AI safety.Seeing whether AIs become deceptive and how they respond to simple oversight.Seeing how much progress we make on AI alignment over the coming years.Christiano is relatively optimistic about his iterated amplification approach:Christiano cares more about making aligned AIs that are competitive with unaligned AIs, whereas MIRI is more willing to settle for an AI with very narrow capabilities.Iterated amplification is largely based on learning-based AI systems, though it may work in other cases.Even if iterated amplification isn't the answer to AI safety, it's likely to have subproblems in common with problems that are important in the future.There are still many disagreements between Christiano and the Machine Intelligence Research Institute (MIRI) that are messy and haven't been made precise.
This transcript has been lightly edited for concision and clarity.
 Transcript
Asya Bergal: Okay. We are recording. I'm going to ask you a bunch of questions related to something like AI optimism.
I guess the proposition that we're looking at is something like 'is it valuable for people to be spending significant effort doing work that purports to reduce the risk from advanced artificial intelligence'? The first question would be to give a short-ish version of the reasoning around that.
Paul Christiano: Around why it's overall valuable?
Asya Bergal: Yeah. Or the extent to which you think it's valuable.
Paul Christiano: I don't know, this seems complicated. I'm acting from some longtermerist perspective, I'm like, what can make the world irreversibly worse? There aren't that many things, we go extinct. It's hard to go extinct, doesn't seem that likely.
Robert Long: We keep forgetting to say this, but we are focusing less on ethical considerations that might affect that. We'll grant…yeah, with all that in the background….
Paul Christiano: Granting long-termism, but then it seems like it depends a lot on what's the probability? What fraction of our expected future do we lose by virtue of messing up alignment * what's the elasticity of that to effort / how much effort?
Robert Long: That's the stuff we're curious to see what people think about.
Asya Bergal: I also just read your 80K interview, which I think probably covered like a lot of the reasoning about this.
Paul Christiano: They probably did. I don't remember exactly what's in there, but it was a lot of words.
I don't know. I'm like, it's a lot of doom probability. Like maybe I think AI alignment per se is like 10% doominess. That's a lot. Then it seems like if we understood everything in advance really well, or just having a bunch of people working on now understanding what's up, could easily reduce that by a big chunk.
Ronny Fernandez: Sorry, what do you mean by 10% doominesss?
Paul Christiano: I don't know, the future is 10% worse than it would otherwise be in expectation by virtue of our failure to align AI. I made up 10%, it's kind of a random number. I don't know, it's less than 50%. It's more than 10% conditioned on AI soon I think.
Ronny Fernandez: And that's change in expected value.
Paul Christiano: Yeah. Anyway, so 10% is a lot. Then I'm like, maybe if we sorted all our shit out and had a bunch of people who knew what was up, and had a good theoretical picture of what was up, and had more info available about whether it was a real problem. Maybe really nailing all that could cut that risk from 10% to 5% and maybe like, you know, there aren't that many people who work on it, it seems like a marginal person can easily do a thousandth of that 5% change. Now you're looking at one in 20,000 or something, which is a good deal.
Asya Bergal: I think my impression is that that 10% is lower than some large set of people. I don't know if other people agree with that.
Paul Christiano: Certainly, 10% is lower than lots of people who care about AI risk. I mean it's worth saying, that I have this slightly narrow conception of what is the alignment problem. I'm not including all AI risk in the 10%. I'm not including in some sense most of the things people normally worry about and just including the like 'we tried to build an AI that was doing what we want but then it wasn't even trying to do what we want'. I think it's lower now or even after that caveat, than pessimistic people. It's going to be lower than all the MIRI folks, it's going to be higher than almost everyone in the world at large, especially after specializing in this problem, which is a problem almost no one cares about, which is precisely how a thousand full time people for 20 years can reduce the whole risk by half or something.
Asya Bergal: I'm curious for your statement as to why you think your number is slightly lower than other people.
Paul Christiano: Yeah, I don't know if I have a particularly crisp answer. Seems like it's a more reactive thing of like, what are the arguments that it's very doomy? A priori you might've been like, well, if you're going to build some AI, you're probably going to build the AI so it's trying to do what you want it to do. Probably that's that. Plus, most things can't destroy the expected value of the future by 10%. You just can't have that many things, otherwise there's not going to be any value left in the end. In particular, if you had 100 such things, then you'd be down to like 1/1000th of your values. 1/10 hundred thousandth? I don't know, I'm not good at arithmetic.
Anyway, that's a priori, just aren't that many things are that bad and it seems like people would try and make AI that's trying to do what they want. Then you're like, okay, we get to be pessimistic because of some other argument about like, well, we don't currently know how to build an AI which will do what we want. We're like, there's some extrapolation of current techniques on which we're concerned that we wouldn't be able to. Or maybe some more conceptual or intuitive argument about why AI is a scary kind thing, and AIs tend to want to do random shit.
Then like, I don't know, now we get into, how strong is that argument for doominess? Then a major thing that drives it is I am like, reasonable chance there is no problem in fact. Reasonable chance, if there is a problem we can cope with it just by trying. Reasonable chance, even if it will be hard to cope with, we can sort shit out well enough on paper that we really nail it and understand how to resolve it. Reasonable chance, if we don't solve it the people will just not build AIs that destroy everything they value.
It's lots of saving throws, you know? And you multiply the saving throws together and things look better. And they interact better than that because– well, in one way worse because it's correlated: If you're incompetent, you're more likely to fail to solve the problem and more likely to fail to coordinate not to destroy the world. In some other sense, it's better than interacting multiplicatively because weakness in one area compensates for strength in the other. I think there are a bunch of saving throws that could independently make things good, but then in reality you have to have a little bit here and a little bit here and a little bit here, if that makes sense. We have some reasonable understanding on paper that makes the problem easier. The problem wasn't that bad. We wing it reasonably well and we do a bunch of work and in fact people are just like, 'Okay, we're not going to destroy the world given the choice.' I guess I have this somewhat distinctive last saving throw where I'm like, 'Even if you have unaligned AI, it's probably not that bad.'
That doesn't do much of the work, but you know you add a bunch of shit like that together.
Asya Bergal: That's a lot of probability mass on a lot of different things. I do feel like my impression is that, on the first step of whether by default things are likely to be okay or things are likely to be good, people make arguments of the form, 'You have a thing with a goal and it's so hard to specify. By default, you should assume that the space of possible goals to specify is big, and the one right goal is hard to specify, hard to find.' Obviously, this is modeling the thing as an agent, which is already an assumption.
Paul Christiano: Yeah. I mean it's hard to run or have much confidence in arguments of that form. I think it's possible to run tight versions of that argument that are suggestive. It's hard to have much confidence in part because you're like, look, the space of all programs is very broad, and the space that do your taxes is quite small, and we in fact are doing a lot of selecting from the vast space of programs to find one that does your taxes– so like, you've already done a lot of that.
And then you have to be getting into more detailed arguments about exactly how hard is it to select. I think there's two kinds of arguments you can make that are different, or which I separate. One is the inner alignment treacherous turney argument, where like, we can't tell the difference between AIs that are doing the right and wrong thing, even if you know what's right because blah blah blah. The other is well, you don't have this test for 'was it right' and so you can't be selecting for 'does the right thing'.
This is a place where the concern is disjunctive, you have like two different things, they're both sitting in your alignment problem. They can again interact badly. But like, I don't know, I don't think you're going to get to high probabilities from this. I think I would kind of be at like, well I don't know. Maybe I think it's more likely than not that there's a real problem but not like 90%, you know? Like maybe I'm like two to one that there exists a non-trivial problem or something like that. All of the numbers I'm going to give are very made up though. If you asked me a second time you'll get all different numbers.
Asya Bergal: That's good to know.
Paul Christiano: Sometimes I anchor on past things I've said though, unfortunately.
Asya Bergal: Okay. Maybe I should give you some fake past Paul numbers.
Paul Christiano: You could be like, 'In that interview, you said that it was 85%'. I'd be like, 'I think it's really probably 82%'.
Asya Bergal: I guess a related question is, is there plausible concrete evidence that you think could be gotten that would update you in one direction or the other significantly?
Paul Christiano: Yeah. I mean certainly, evidence will roll in once we have more powerful AI systems.
One can learn… I don't know very much about any of the relevant institutions, I may know a little bit. So you can imagine easily learning a bunch about them by observing how well they solve analogous problems or learning about their structure, or just learning better about the views of people. That's the second category.
We're going to learn a bunch of shit as we continue thinking about this problem on paper to see like, does it look like we're going to solve it or not? That kind of thing. It seems like there's lots of sorts of evidence on lots of fronts, my views are shifting all over the place. That said, the inconsistency between one day and the next is relatively large compared to the actual changes in views from one day to the next.
Robert Long: Could you say a little bit more about evidence from once more advanced AI starts coming in? Like what sort things you're looking for that would change your mind on things?
Paul Christiano: Well you get to see things like, on inner alignment you get to see to what extent do you have the kind of crazy shit that people are concerned about? The first time you observe some crazy shit where your AI is like, 'I'm going to be nice in order to assure that you think I'm nice so I can stab you in the back later.' You're like, 'Well, I guess that really does happen despite modest effort to prevent it.' That's a thing you get. You get to learn in general about how models generalize, like to what extent they tend to do– this is sort of similar to what I just said, but maybe a little bit broader– to what extent are they doing crazy-ish stuff as they generalize?
You get to learn about how reasonable simple oversight is and to what extent do ML systems acquire knowledge that simple overseers don't have that then get exploited as they optimize in order to produce outcomes that are actually bad. I don't have a really concise description, but sort of like, to the extent that all these arguments depend on some empirical claims about AI, you get to see those claims tested increasingly.
Ronny Fernandez: So the impression I get from talking to other people who know you, and from reading some of your blog posts, but mostly from others, is that you're somewhat more optimistic than most people that work in AI alignment. It seems like some people who work on AI alignment think something like, 'We've got to solve some really big problems that we don't understand at all or there are a bunch of unknown unknowns that we need to figure out.' Maybe that's because they have a broader conception of what solving AI alignment is like than you do?
Paul Christiano: That seems like it's likely to be part of it. It does seem like I'm more optimistic than people in general, than people who work in alignment in general. I don't really know… I don't understand others' views that well and I don't know if they're that– like, my views aren't that internally coherent. My suspicion is others' views are even less internally coherent. Yeah, a lot of it is going to be done by having a narrower conception of the problem.
Then a lot of it is going to be done by me just being… in terms of do we need a lot of work to be done, a lot of it is going to be me being like, I don't know man, maybe. I don't really understand when people get off the like high probability of like, yeah. I don't see the arguments that are like, definitely there's a lot of crazy stuff to go down. It seems like we really just don't know. I do also think problems tend to be easier. I have more of that prior, especially for problems that make sense on paper. I think they tend to either be kind of easy, or else– if they're possible, they tend to be kind of easy. There aren't that many really hard theorems.
Robert Long: Can you say a little bit more of what you mean by that? That's not a very good follow-up question, I don't really know what it would take for me to understand what you mean by that better.
Paul Christiano: Like most of the time, if I'm like, 'here's an algorithms problem', you can like– if you just generate some random algorithms problems, a lot of them are going to be impossible. Then amongst the ones that are possible, a lot of them are going to be soluble in a year of effort and amongst the rest, a lot of them are going to be soluble in 10 or a hundred years of effort. It's just kind of rare that you find a problem that's soluble– by soluble, I don't just mean soluble by human civilization, I mean like, they are not provably impossible– that takes a huge amount of effort.
It normally… it's less likely to happen the cleaner the problem is. There just aren't many very clean algorithmic problems where our society worked on it for 10 years and then we're like, 'Oh geez, this still seems really hard.' Examples are kind of like… factoring is an example of a problem we've worked a really long time on. It kind of has the shape, and this is the tendency on these sorts of problems, where there's just a whole bunch of solutions and we hack away and we're a bit better and a bit better and a bit better. It's a very messy landscape, rather than jumping from having no solution to having a solution. It's even rarer to have things where going from no solution to some solution is really possible but incredibly hard. There were some examples.
Robert Long: And you think that the problems we face are sufficiently similar?
Paul Christiano: I mean, I think this is going more into the like, 'I don't know man' but my what do I think when I say I don't know man isn't like, 'Therefore, there's an 80% chance that it's going to be an incredibly difficult problem' because that's not what my prior is like. I'm like, reasonable chance it's not that hard. Some chance it's really hard. Probably more chance that– if it's really hard, I think it's more likely to be because all the clean statements of the problem are impossible. I think as statements get messier it becomes more plausible that it just takes a lot of effort. The more messy a thing is, the less likely it is to be impossible sometimes, but also the more likely it's just a bunch of stuff you have to do.
Ronny Fernandez: It seems like one disagreement that you have with MIRI folks is that you think prosaic AGI will be easier to align than they do. Does that perception seem right to you?
Paul Christiano: I think so. I think they're probably just like, 'that seems probably impossible'. Was related to the previous point.
Ronny Fernandez: If you had found out that prosaic AGI is nearly impossible to align or is impossible to align, how much would that change your-
Paul Christiano: It depends exactly what you found out, exactly how you found it out, et cetera. One thing you could be told is that there's no perfectly scalable mechanism where you can throw in your arbitrarily sophisticated AI and turn the crank and get out an arbitrarily sophisticated aligned AI. That's a possible outcome. That's not necessarily that damning because now you're like okay, fine, you can almost do it basically all the time and whatever.
That's a big class of worlds and that would definitely be a thing I would be interested in understanding– how large is that gap actually, if the nice problem was totally impossible? If at the other extreme you just told me, 'Actually, nothing like this is at all going to work, and it's definitely going to kill everyone if you build an AI using anything like an extrapolation of existing techniques', then I'm like, 'Sounds pretty bad.' I'm still not as pessimistic as MIRI people.
I'm like, maybe people just won't destroy the world, you know, it's hard to say. It's hard to say what they'll do. It also depends on the nature of how you came to know this thing. If you came to know it in a way that's convincing to a reasonably broad group of people, that's better than if you came to know it and your epistemic state was similar to– I think MIRI people feel more like, it's already known to be hard, and therefore you can tell if you can't convince people it's hard. Whereas I'm like, I'm not yet convinced it's hard, so I'm not so surprised that you can't convince people it's hard.
Then there's more probability, if it was known to be hard, that we can convince people, and therefore I'm optimistic about outcomes conditioned on knowing it to be hard. I might become almost as pessimistic as MIRI if I thought that the problem was insolubly hard, just going to take forever or whatever, huge gaps aligning prosaic AI, and there would be no better evidence of that than currently exists. Like there's no way to explain it better to people than MIRI currently can. If you take those two things, I'm maybe getting closer to MIRI's levels of doom probability. I might still not be quite as doomy as them.
Ronny Fernandez: Why does the ability to explain it matter so much?
Paul Christiano: Well, a big part of why you don't expect people to build unaligned AI is they're like, they don't want to. The clearer it is and the stronger the case, the more people can potentially do something. In particular, you might get into a regime where you're doing a bunch of shit by trial and error and trying to wing it. And if you have some really good argument that the winging it is not going to work, then that's a very different state than if you're like, 'Well, winging it doesn't seem that good. Maybe it'll fail.' It's different to be like, 'Oh no, here's an argument. You just can't… It's just not going to work.'
I don't think we'll really be in that state, but there's like a whole spectrum from where we're at now to that state and I expect to be further along it, if in fact we're doomed. For example, if I personally would be like, 'Well, I at least tried the thing that seemed obvious to me to try and now we know that doesn't work.' I sort of expect very directly from trying that to learn something about why that failed and what parts of the problem seem difficult.
Ronny Fernandez: Do you have a sense of why MIRI thinks aligning prosaic AI is so hard?
Paul Christiano: We haven't gotten a huge amount of traction on this when we've debated it. I think part of their position, especially on the winging it thing, is they're like – Man, doing things right generally seems a lot harder than doing them. I guess probably building an AI will be harder in a way that's good, for some arbitrary notion of good– a lot harder than just building an AI at all.
There's a theme that comes up frequently trying to hash this out, and it's not so much about a theoretical argument, it's just like, look, the theoretical argument establishes that there's something a little bit hard here. And once you have something a little bit hard and now you have some giant organization, people doing the random shit they're going to do, and all that chaos, and like, getting things to work takes all these steps, and getting this harder thing to work is going to have some extra steps, and everyone's going to be doing it. They're more pessimistic based on those kinds of arguments.
That's the thing that comes up a lot. I think probably most of the disagreement is still in the, you know, theoretically, how much– certainly we disagree about like, can this problem just be solved on paper in advance? Where I'm like, reasonable chance, you know? At least a third chance, they'll just on paper be like, 'We have nailed it.' There's really no tension, no additional engineering effort required. And they're like, that's like zero. I don't know what they think it is. More than zero, but low.
Ronny Fernandez: Do you guys think you're talking about the same problem exactly?
Paul Christiano: I think there we are probably. At that step we are. Just like, is your AI trying to destroy everything? Yes. No. The main place there's some bleed over–  the main thing that MIRI maybe considers in scope and I don't is like, if you build an AI, it may someday have to build another AI. And what if the AI it builds wants to destroy everything? Is that our fault or is that the AI's fault? And I'm more on like, that's the AI's fault. That's not my job. MIRI's maybe more like not distinguishing those super cleanly, but they would say that's their job. The distinction is a little bit subtle in general, but-
Ronny Fernandez: I guess I'm not sure why you cashed out in terms of fault.
Paul Christiano: I think for me it's mostly like: there's a problem we can hope to resolve. I think there's two big things. One is like, suppose you don't resolve that problem. How likely is it that someone else will solve it? Saying it's someone else's fault is in part just saying like, 'Look, there's this other person who had a reasonable opportunity to solve it and it was a lot smarter than us.' So the work we do is less likely to make the difference between it being soluble or not. Because there's this other smarter person.
And then the other thing is like, what should you be aiming for? To the extent there's a clean problem here which one could hope to solve, or one should bite off as a chunk, what fits in conceptually the same problem versus what's like– you know, an analogy I sometimes make is, if you build an AI that's doing important stuff, it might mess up in all sorts of ways. But when you're asking, 'Is my AI going to mess up when building a nuclear reactor?' It's a thing worth reasoning about as an AI person, but also like it's worth splitting into like– part of that's an AI problem, and part of that's a problem about understanding managing nuclear waste. Part of that should be done by people reasoning about nuclear waste and part of it should be done by people reasoning about AI.
This is a little subtle because both of the problems have to do with AI. I would say my relationship with that is similar to like, suppose you told me that some future point, some smart people might make an AI. There's just a meta and object level on which you could hope to help with the problem.
I'm hoping to help with the problem on the object level in the sense that we are going to do research which helps people align AI, and in particular, will help the future AI align the next AI. Because it's like people. It's at that level, rather than being like, 'We're going to construct a constitution of that AI such that when it builds future AI it will always definitely work'. This is related to like– there's this old argument about recursive self-improvement. It's historically figured a lot in people's discussion of why the problem is hard, but on a naive perspective it's not obvious why it should, because you do only a small number of large modifications before your systems are sufficiently intelligent relative to you that it seems like your work should be obsolete. Plus like, them having a bunch of detailed knowledge on the ground about what's going down.
It seems unclear to me how– yeah, this is related to our disagreement– how much you're happy just deferring to the future people and being like, 'Hope that they'll cope'. Maybe they won't even cope by solving the problem in the same way, they might cope by, the crazy AIs that we built reach the kind of agreement that allows them to not build even crazier AIs in the same way that we might do that. I think there's some general frame of, I'm just taking responsibility for less, and more saying, can we leave the future people in a situation that is roughly as good as our situation? And by future people, I mean mostly AIs.
Ronny Fernandez: Right. The two things that you think might explain your relative optimism are something like: Maybe we can get the problem to smarter agents that are humans. Maybe we can leave the problem to smarter agents that are not humans.
Paul Christiano: Also a lot of disagreement about the problem. Those are certainly two drivers. They're not exhaustive in the sense that there's also a huge amount of disagreement about like, 'How hard is this problem?' Which is some combination of like, 'How much do we know about it?' Where they're more like, 'Yeah, we've thought about it a bunch and have some views.' And I'm like, 'I don't know, I don't think I really know shit.' Then part of it is concretely there's a bunch of– on the object level, there's a bunch of arguments about why it would be hard or easy so we don't reach agreement. We consistently disagree on lots of those points.
Ronny Fernandez: Do you think the goal state for you guys is the same though? If I gave you guys a bunch of AGIs, would you guys agree about which ones are aligned and which ones are not? If you could know all of their behaviors?
Paul Christiano: I think at that level we'd probably agree. We don't agree more broadly about what constitutes a win state or something. They have this more expansive conception– or I guess it's narrower– that the win state is supposed to do more. They are imagining more that you've resolved this whole list of future challenges. I'm more not counting that.
We've had this… yeah, I guess I now mostly use intent alignment to refer to this problem where there's risk of ambiguity… the problem that I used to call AI alignment. There was a long obnoxious back and forth about what the alignment problem should be called. MIRI does use aligned AI to be like, 'an AI that produces good outcomes when you run it'. Which I really object to as a definition of aligned AI a lot. So if they're using that as their definition of aligned AI, we would probably disagree.
Ronny Fernandez: Shifting terms or whatever… one thing that they're trying to work on is making an AGI that has a property that is also the property you're trying to make sure that AGI has.
Paul Christiano: Yeah, we're all trying to build an AI that's trying to do the right thing.
Ronny Fernandez: I guess I'm thinking more specifically, for instance, I've heard people at MIRI say something like, they want to build an AGI that I can tell it, 'Hey, figure out how to copy a strawberry, and don't mess anything else up too badly.' Does that seem like the same problem that you're working on?
Paul Christiano: I mean it seems like in particular, you should be able to do that. I think it's not clear whether that captures all the complexity of the problem. That's just sort of a question about what solutions end up looking like, whether that turns out to have the same difficulty.
The other things you might think are involved that are difficult are… well, I guess one problem is just how you capture competitiveness. Competitiveness for me is a key desideratum. And it's maybe easy to elide in that setting, because it just makes a strawberry. Whereas I am like, if you make a strawberry literally as well as anyone else can make a strawberry, it's just a little weird to talk about. And it's a little weird to even formalize what competitiveness means in that setting. I think you probably can, but whether or not you do that's not the most natural or salient aspect of the situation.
So I probably disagree with them about– I'm like, there are probably lots of ways to have agents that make strawberries and are very smart. That's just another disagreement that's another function of the same basic, 'How hard is the problem' disagreement. I would guess relative to me, in part because of being more pessimistic about the problem, MIRI is more willing to settle for an AI that does one thing. And I care more about competitiveness.
Asya Bergal: Say you just learn that prosaic AI is just not going to be the way we get to AGI. How does that make you feel about the IDA approach versus the MIRI approach?
Paul Christiano: So my overall stance when I think about alignment is, there's a bunch of possible algorithms that you could use. And the game is understanding how to align those algorithms. And it's kind of a different game. There's a lot of common subproblems in between different algorithms you might want to align, it's potentially a different game for different algorithms. That's an important part of the answer. I'm mostly focusing on the 'align this particular'– I'll call it learning, but it's a little bit more specific than learning– where you search over policies to find a policy that works well in practice. If we're not doing that, then maybe that solution is totally useless, maybe it has common subproblems with the solution you actually need. That's one part of the answer.
Another big difference is going to be, timelines views will shift a lot if you're handed that information. So it will depend exactly on the nature of the update. I don't have a strong view about whether it makes my timelines shorter or longer overall. Maybe you should bracket that though.
In terms of returning to the first one of trying to align particular algorithms, I don't know. I think I probably share some of the MIRI persp– well, no. It feels to me like there's a lot of common subproblems. Aligning expert systems seems like it would involve a lot of the same reasoning as aligning learners. To the extent that's true, probably future stuff also will involve a lot of the same subproblems, but I doubt the algorithm will look the same. I also doubt the actual algorithm will look anything like a particular pseudocode we might write down for iterated amplification now.
Asya Bergal: Does iterated amplification in your mind rely on this thing that searches through policies for the best policy? The way I understand it, it doesn't feel like it necessarily does.
Paul Christiano: So, you use this distillation step. And the reason you want to do amplification, or this short-hop, expensive amplification, is because you interleave it with this distillation step. And I normally imagine the distillation step as being, learn a thing which works well in practice on a reward function defined by the overseer. You could imagine other things that also needed to have this framework, but it's not obvious whether you need this step if you didn't somehow get granted something like the–
Asya Bergal: That you could do the distillation step somehow.
Paul Christiano: Yeah. It's unclear what else would– so another example of a thing that could fit in, and this maybe makes it seem more general, is if you had an agent that was just incentivized to make lots of money. Then you could just have your distillation step be like, 'I randomly check the work of this person, and compensate them based on the work I checked'. That's a suggestion of how this framework could end up being more general.
But I mostly do think about it in the context of learning in particular. I think it's relatively likely to change if you're not in that setting. Well, I don't know. I don't have a strong view. I'm mostly just working in that setting, mostly because it seems reasonably likely, seems reasonably likely to have a bunch in common, learning is reasonably likely to appear even if other techniques appear. That is, learning is likely to play a part in powerful AI even if other techniques also play a part.
Asya Bergal: Are there other people or resources that you think would be good for us to look at if we were looking at the optimism view?
Paul Christiano: Before we get to resources or people, I think one of the basic questions is, there's this perspective which is fairly common in ML, which is like, 'We're kind of just going to do a bunch of stuff, and it'll probably work out'. That's probably the basic thing to be getting at. How right is that?
This is the bad view of safety conditioned on– I feel like prosaic AI is in some sense the worst– seems like about as bad as things would have gotten in terms of alignment. Where, I don't know, you try a bunch of shit, just a ton of stuff, a ton of trial and error seems pretty bad. Anyway, this is a random aside maybe more related to the previous point. But yeah, this is just with alignment. There's this view in ML that's relatively common that's like, we'll try a bunch of stuff to get the AI to do what we want, it'll probably work out. Some problems will come up. We'll probably solve them. I think that's probably the most important thing in the optimism vs pessimism side.
And I don't know, I mean this has been a project that like, it's a hard project. I think the current state of affairs is like, the MIRI folk have strong intuitions about things being hard. Essentially no one in… very few people in ML agree with those, or even understand where they're coming from. And even people in the EA community who have tried a bunch to understand where they're coming from mostly don't. Mostly people either end up understanding one side or the other and don't really feel like they're able to connect everything. So it's an intimidating project in that sense. I think the MIRI people are the main proponents of the everything is doomed, the people to talk to on that side. And then in some sense there's a lot of people on the other side who you can talk to, and the question is just, who can articulate the view most clearly? Or who has most engaged with the MIRI view such that they can speak to it?
Ronny Fernandez: Those are people I would be particularly interested in. If there are people that understand all the MIRI arguments but still have broadly the perspective you're describing, like some problems will come up, probably we'll fix them.
Paul Christiano: I don't know good– I don't have good examples of people for you. I think most people just find the MIRI view kind of incomprehensible, or like, it's a really complicated thing, even if the MIRI view makes sense in its face. I don't think people have gotten enough into the weeds. It really rests a lot right now on this fairly complicated cluster of intuitions. I guess on the object level, I think I've just engaged a lot more with the MIRI view than most people who are– who mostly take the 'everything will be okay' perspective. So happy to talk on the object level, and speaking more to arguments. I think it's a hard thing to get into, but it's going to be even harder to find other people in ML who have engaged with the view that much.
They might be able to make other general criticisms of like, here's why I haven't really… like it doesn't seem like a promising kind of view to think about. I think you could find more people who have engaged at that level. I don't know who I would recommend exactly, but I could think about it. Probably a big question will be who is excited to talk to you about it.
Asya Bergal: I am curious about your response to MIRI's object level arguments. Is there a place that exists somewhere?
Paul Christiano: There's some back and forth on the internet. I don't know if it's great. There's some LessWrong posts. Eliezer for example wrote this post about why things were doomed, why I in particular was doomed. I don't know if you read that post.
Asya Bergal: I can also ask you about it now, I just don't want to take too much of your time if it's a huge body of things.
Paul Christiano: The basic argument would be like, 1) On paper I don't think we yet have a good reason to feel doomy. And I think there's some basic research intuition about how much a problem– suppose you poke at a problem a few times, and you're like 'Agh, seems hard to make progress'. How much do you infer that the problem's really hard? And I'm like, not much. As a person who's poked at a bunch of problems, let me tell you, that often doesn't work and then you solve in like 10 years of effort.
So that's one thing. That's a point where I have relatively little sympathy for the MIRI way. That's one set of arguments: is there a good way to get traction on this problem? Are there clever algorithms? I'm like, I don't know, I don't feel like the kind of evidence we've seen is the kind of evidence that should be persuasive. As some evidence in that direction, I'd be like, I have not been thinking about this that long. I feel like there have often been things that felt like, or that MIRI would have defended as like, here's a hard obstruction. Then you think about it and you're actually like, 'Here are some things you can do.' And it may still be a obstruction, but it's no longer quite so obvious where it is, and there were avenues of attack.
That's one thing. The second thing is like, a metaphor that makes me feel good– MIRI talks a lot about the evolution analogy. If I imagine the evolution problem– so if I'm a person, and I'm breeding some animals, I'm breeding some superintelligence. Suppose I wanted to breed an animal modestly smarter than humans that is really docile and friendly. I'm like, I don't know man, that seems like it might work. That's where I'm at. I think they are… it's been a little bit hard to track down this disagreement, and I think this is maybe in a fresher, rawer state than the other stuff, where we haven't had enough back and forth.
But I'm like, it doesn't sound necessarily that hard. I just don't know. I think their position, their position when they've written something has been a little bit more like, 'But you couldn't breed a thing, that after undergoing radical changes in intelligence or situation would remain friendly'. But then I'm normally like, but it's not clear why that's needed? I would really just like to create something slightly superhuman, and it's going to work with me to breed something that's slightly smarter still that is friendly.
We haven't really been able to get traction on that. I think they have an intuition that maybe there's some kind of invariance and things become gradually more unraveled as you go on. Whereas I have more intuition that it's plausible. After this generation, there's just smarter and smarter people thinking about how to keep everything on the rails. It's very hard to know.
That's the second thing. I have found that really… that feels like it gets to the heart of some intuitions that are very different, and I don't understand what's up there. There's a third category which is like, on the object level, there's a lot of directions that I'm enthusiastic about where they're like, 'That seems obviously doomed'. So you could divide those up into the two problems. There's the family of problems that are more like the inner alignment problem, and then outer alignment stuff.
On the inner alignment stuff, I haven't thought that much about it, but examples of things that I'm optimistic about that they're super pessimistic about are like, stuff that looks more like verification, or maybe stepping back even for that, there's this basic paradigm of adversarial training, where I'm like, it seems close to working. And you could imagine it being like, it's just a research problem to fill in the gaps. Whereas they're like, that's so not the kind of thing that would work. I don't really know where we're at with that. I do see there are formal obstructions to adversarial training in particular working. I'm like, I see why this is not yet a solution. For example, you can have this case where there's a predicate that the model checks, and it's easy to check but hard to construct examples. And then in your adversarial training you can't ever feed an example where it'll fail. So we get into like, is it plausible that you can handle that problem with either 1) Doing something more like verification, where you say, you ask them not to perform well on real inputs but on pseudo inputs. Or like, you ask the attacker just to show how it's conceivable that the model could do a bad thing in some sense.
That's one possible approach, where the other would be something more like interpretability, where you say like, 'Here's what the model is doing. In addition to it's behavior we get this other signal that the paper was depending on this fact, its predicate paths, which it shouldn't have been dependent on.' The question is, can either of those yield good behavior? I'm like, I don't know, man. It seems plausible. And they're like 'Definitely not.' And I'm like, 'Why definitely not?' And they're like 'Well, that's not getting at the real essence of the problem.' And I'm like 'Okay, great, but how did you substantiate this notion of the real essence of the problem? Where is that coming from? Is that coming from a whole bunch of other solutions that look plausible that failed?' And their take is kind of like, yes, and I'm like, 'But none of those– there weren't actually even any candidate solutions there really that failed yet. You've got maybe one thing, or like, you showed there exists a problem in some minimal sense.' This comes back to the first of the three things I listed. But it's a little bit different in that I think you can just stare at particular things and they'll be like, 'Here's how that particular thing is going to fail.' And I'm like 'I don't know, it seems plausible.'
That's on inner alignment. And there's maybe some on outer alignment. I feel like they've given a lot of ground in the last four years on how doomy things seem on outer alignment. I think they still have some– if we're talking about amplification, I think the position would still be, 'Man, why would that agent be aligned? It doesn't at all seem like it would be aligned.' That has also been a little bit surprisingly tricky to make progress on. I think it's similar, where I'm like, yeah, I grant the existence of some problem or some thing which needs to be established, but I don't grant– I think their position would be like, this hasn't made progress or just like, pushed around the core difficulty. I'm like, I don't grant the conception of the core difficulty in which this has just pushed around the core difficulty. I think that… substantially in that kind of thing, being like, here's an approach that seems plausible, we don't have a clear obstruction but I think that it is doomed for these deep reasons. I have maybe a higher bar for what kind of support the deep reasons need.
I also just think on the merits, they have not really engaged with– and this is partly my responsibility for not having articulated the arguments in a clear enough way– although I think they have not engaged with even the clearest articulation as of two years ago of what the hope was. But that's probably on me for not having an even clearer articulation than that, and also definitely not up to them to engage with anything. To the extent it's a moving target, not up to them to engage with the most recent version. Where, most recent version– the proposal doesn't really change that much, or like, the case for optimism has changed a little bit. But it's mostly just like, the state of argument concerning it, rather than the version of the scheme.

Conversation with Paul Christiano


Paul Christiano on the safety of future AI systems

Paul Christiano
As part of our AI optimism project, we talked to Paul Christiano about why he is relatively hopeful about the arrival of advanced AI going well. Paul Christiano works on AI alignment on the safety team at OpenAI. He is also a research associate at FHI, a board member at Ought, and a recent graduate of the theory group at UC Berkeley.

Paul gave us a number of key disagreements he has with researchers at the Machine Intelligence Research Institute (MIRI), including:
Paul thinks there isn't good evidence now to justify a confident pessimistic position on AI, so the fact that people aren't worried doesn't mean they won't be worried when we're closer to human-level intelligence.Paul thinks that many algorithmic or theoretical problems are either solvable within 100 years or provably impossible.Paul thinks not having solved many AI safety problems yet shouldn't give us much evidence about their difficulty.Paul's criterion for alignment success isn't ensuring that all future intelligences are aligned, it's leaving the next generation of intelligences in at least as good of a place as humans were when building them.Paul doesn't think that the evolution analogy suggests that we are doomed in our attempts to align smarter AIs; e.g. if we tried, it seems likely that we could breed animals that are slightly smarter than humans and are also friendly and docile.Paul cares about trying to build aligned AIs that are competitive with unaligned AIs, whereas MIRI is going for a less ambitious goal of building a narrow aligned AI without destroying the world.Unlike MIRI, Paul is relatively optimistic about verification and interpretability for the inner alignment of AI systems.
Paul also talked about evidence that might change his views; in particular, observing whether future AIs become deceptive and how they respond to simple oversight. He also described how research into the AI alignment approach he's working on now, iterated distillation and amplification (IDA), is likely to be useful even if human-level intelligences don't look like the AI systems we have today.
A full transcript of our conversation, lightly edited for concision and clarity, can be found here.
By Asya Bergal


Soft takeoff can still lead to decisive strategic advantage

Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.
[Epistemic status: Argument by analogy to historical cases. Best case scenario it's just one argument among many. Edit: Also, thanks to feedback from others, especially Paul, I intend to write a significantly improved version of this post in the next two weeks.]
I have on several occasions heard people say things like this:
The original Bostrom/Yudkowsky paradigm envisioned a single AI built by a single AI project, undergoing intelligence explosion all by itself and attaining a decisive strategic advantage as a result. However, this is very unrealistic. Discontinuous jumps in technological capability are very rare, and it is very implausible that one project could produce more innovations than the rest of the world combined. Instead we should expect something more like the Industrial Revolution: Continuous growth, spread among many projects and factions, shared via a combination of trade and technology stealing. We should not expect any one project or AI to attain a decisive strategic advantage, because there will always be other projects and other AI that are only slightly less powerful, and coalitions will act to counterbalance the technological advantage of the frontrunner. (paraphrased)
Proponents of this view often cite Paul Christiano in support. Last week I heard him say he thinks the future will be "like the Industrial Revolution but 10x-100x faster."
In this post, I assume that Paul's slogan for the future is correct and then nevertheless push back against the view above. Basically, I will argue that even if the future is like the industrial revolution only 10x-100x faster, there is a 30%+ chance that it will involve a single AI project (or a single AI) with the ability to gain a decisive strategic advantage, if they so choose. (Whether or not they exercise that ability is another matter.)
Why am I interested in this? Do I expect some human group to take over the world? No; instead what I think is that (1) an unaligned AI in the leading project might take over the world, and (2) A human project that successfully aligns their AI might refrain from taking over the world even if they have the ability to do so, and instead use their capabilities to e.g. help the United Nations enforce a ban on unauthorized AGI projects.
National ELO ratings during the industrial revolution and the modern era
In chess (and some other games) ELO rankings are used to compare players. An average club player might be rank 1500; the world chess champion might be 2800; computer chess programs are even better. If one player has 400 points more than another, it means the first player would win with ~90% probability.
We could apply this system to compare the warmaking abilities of nation-states and coalitions of nation-states. For example, in 1941 perhaps we could say that the ELO rank of the Axis powers was ~300 points lower than the ELO rank of the rest of the world combined (because what in fact happened was the rest of the world combining to defeat them, but it wasn't a guaranteed victory). We could add that in 1939 the ELO rank of Germany was ~400 points higher than that of Poland, and that the ELO rank of Poland was probably 400+ points higher than that of Luxembourg.
We could make cross-temporal fantasy comparisons too. The ELO ranking of Germany in 1939 was probably ~400 points greater than that of the entire world circa 1910, for example. (Visualize the entirety of 1939 Germany teleporting back in time to 1910, and then imagine the havoc it would wreak.)
Claim 1A: If we were to estimate the ELO rankings of all nation-states and sets of nation-states (potential alliances) over the last 300 years, the rank of the most powerful nation-state at at a given year would on several occasions be 400+ points greater than the rank of the entire world combined 30 years prior.
Claim 1B: Over the last 300 years there have been several occasions in which one nation-state had the capability to take over the entire world of 30 years prior.
I'm no historian, but I feel fairly confident in these claims.
In naval history, the best fleets in the world in 1850 were obsolete by 1860 thanks to the introduction of iron-hulled steamships, and said steamships were themselves obsolete a decade or so later, and then those ships were obsoleted by the Dreadnought, and so on… This process continued into the modern era. By "Obsoleted" I mean something like "A single ship of the new type could defeat the entire combined fleet of vessels of the old type."A similar story could be told about air power. In a dogfight between planes of year 19XX and year 19XX+30, the second group of planes will be limited only by how much ammunition they can carry.Small technologically advanced nations have regularly beaten huge sprawling empires and coalitions. (See: Colonialism)The entire world has been basically carved up between the small handful of most-technologically advanced nations for two centuries now. For example, any of the Great Powers of 1910 (plus the USA) could have taken over all of Africa, Asia, South America, etc. if not for the resistance that the other great powers would put up. The same was true 40 years later and 40 years earlier.
I conclude from this that if some great power in the era kicked off by the industrial revolution had managed to "pull ahead" of the rest of the world more effectively than it actually did–30 years more effectively, in particular–it really would have been able to take over the world.
Claim 2: If the future is like the Industrial Revolution but 10x-100x faster, then correspondingly the technological and economic power granted by being 3 – 0.3 years ahead of the rest of the world should be enough to enable a decisive strategic advantage.
The question is, how likely is it that one nation/project/AI could get that far ahead of everyone else? After all, it didn't happen in the era of the Industrial Revolution. While we did see a massive concentration of power into a few nations on the leading edge of technological capability, there were always at least a few such nations and they kept each other in check.
The "surely not faster than the rest of the world combined" argument
Sometimes I have exchanges like this:
Me: Decisive strategic advantage is plausible!Interlocutor: What? That means one entity must have more innovation power than the rest of the world combined, to be able to take over the rest of the world!Me: Yeah, and that's possible after intelligence explosion. A superintelligence would totally have that property.Interlocutor: Well yeah, if we dropped a superintelligence into a world full of humans. But realistically the rest of the world will be undergoing intelligence explosion too. And indeed the world as a whole will undergo a faster intelligence explosion than any particular project could; to think that one project could pull ahead of everyone else is to think that, prior to intelligence explosion, there would be a single project innovating faster than the rest of the world combined!
This section responds to that by way of sketching how one nation/project/AI might get 3 – 0.3 years ahead of everyone else.
Toy model: There are projects which research technology, each with their own "innovation rate" at which they produce innovations from some latent tech tree. When they produce innovations, they choose whether to make them public or private. They have access to their private innovations + all the public innovations.
It follows from the above that the project with access to the most innovations at any given time will be the project that has the most hoarded innovations, even though the set of other projects has a higher combined innovation rate and also a larger combined pool of accessible innovations. Moreover, the gap between the leading project and the second-best project will increase over time, since the leading project has a slightly higher rate of production of hoarded innovations, but both projects have access to the same public innovations
This model leaves out several important things. First, it leaves out the whole "intelligence explosion" idea: A project's innovation rate should increase as some function of how many innovations they have access to. Adding this in will make the situation more extreme and make the gap between the leading project and everyone else grow even bigger very quickly.
Second, it leaves out reasons why innovations might be made public. Realistically there are three reasons: Leaks, spies, and selling/using-in-a-way-that-makes-it-easy-to-copy.
Claim 3: Leaks & Spies: I claim that the 10x-100x speedup Paul prophecies will not come with an associated 10x-100x increase in the rate of leaks and successful spying. Instead the rate of leaks and successful spying will be only a bit higher than it currently is.
This is because humans are still humans even in this soft takeoff future, still in human institutions like companies and governments, still using more or less the same internet infrastructure, etc. New AI-related technologies might make leaking and spying easier than it currently is, but they also might make it harder. I'd love to see an in-depth exploration of this question because I don't feel particularly confident.
But anyhow, if it doesn't get much easier than it currently is, then going 3 years to 0.3 years without a leak is possible, and more generally it's possible for the world's leading project to build up a 0.3-3 year lead over the second-place project. For example, the USSR had spies embedded in the Manhattan Project but it still took them 4 more years to make their first bomb.
Claim 4: Selling etc. I claim that the 10x-100x speedup Paul prophecies will not come with an associated 10x-100x increase in the budget pressure on projects to make money fast. Again, today AI companies regularly go years without turning a profit — DeepMind, for example, has never turned a profit and is losing something like a billion dollars a year for its parent company — and I don't see any particularly good reason to expect that to change much.
So yeah, it seems to me that it's totally possible for the leading AI project to survive off investor money and parent company money (or government money, for that matter!) for five years or so, while also keeping the rate of leaks and spies low enough that the distance between them and their nearest competitor increases rather than decreases. (Note how this doesn't involve them "innovating faster than the rest of the world combined.")
Suppose they could get a 3-year lead this way, at the peak of their lead. Is that enough?
Well, yes. A 3-year lead during a time 10x-100x faster than the Industrial Revolution would be like a 30-300 year lead during the era of the Industrial Revolution. As I argued in the previous section, even the low end of that range is probably enough to get a decisive strategic advantage.
If this is so, why didn't nations during the Industrial Revolution try to hoard their innovations and gain decisive strategic advantage?
England actually did, if I recall correctly. They passed laws and stuff to prevent their early Industrial Revolution technology from spreading outside their borders. They were unsuccessful–spies and entrepreneurs dodged the customs officials and snuck blueprints and expertise out of the country. It's not surprising that they weren't able to successfully hoard innovations for 30+ years! Entire economies are a lot more leaky than AI projects.
What a "Paul Slow" soft takeoff might look like according to me
At some point early in the transition to much faster innovation rates, the leading AI companies "go quiet." Several of them either get huge investments or are nationalized and given effectively unlimited funding. The world as a whole continues to innovate, and the leading companies benefit from this public research, but they hoard their own innovations to themselves. Meanwhile the benefits of these AI innovations are starting to be felt; all projects have significantly increased (and constantly increasing) rates of innovation. But the fastest increases go to the leading project, which is one year ahead of the second-best project. (This sort of gap is normal for tech projects today, especially the rare massively-funded ones, I think.) Perhaps via a combination of spying, selling, and leaks, that lead narrows to six months midway through the process. But by that time things are moving so quickly that a six months' lead is like a 15-150 year lead during the era of the Industrial Revolution. It's not guaranteed and perhaps still not probable, but at least it's reasonably likely that the leading project will be able to take over the world if it chooses to.
Objection: What about coalitions? During the industrial revolution, if one country did successfully avoid all leaks, the other countries could unite against them and make the "public" technology inaccessible to them. (Trade does something like this automatically, since refusing to sell your technology also lowers your income which lowers your innovation rate as a nation.)
Reply: Coalitions to share AI research progress will be harder than free-trade / embargo coalitions. This is because AI research progress is much more the result of rare smart individuals talking face-to-face with each other and much less the result of a zillion different actions of millions of different people, as the economy is. Besides, a successful coalition can be thought of as just another project, and so it's still true that one project could get a decisive strategic advantage. (Is it fair to call "The entire world economy" a project with a decisive strategic advantage today? Well, maybe… but it feels a lot less accurate since almost everyone is part of the economy but only a few people would have control of even a broad coalition AI project.)
Anyhow, those are my thoughts. Not super confident in all this, but it does feel right to me. Again, the conclusion is not that one project will take over the world even in Paul's future, but rather that such a thing might still happen even in Paul's future.

Thanks to Magnus Vinding for helpful conversation.
By Daniel Kokotajlo


Ernie Davis on the landscape of AI risks

Ernie Davis (NYU)
Earlier this month, I spoke with Ernie Davis about why he is skeptical that risks from superintelligent AI are substantial and tractable enough to merit dedicated work. This was part of a larger project that we've been working on at AI Impacts, documenting arguments from people who are relatively optimistic about risks from advanced AI.
Davis is a professor of computer science at NYU, and works on the representation of commonsense knowledge in computer programs. He wrote Representations of Commonsense Knowledge (1990) and will soon publish a book Rebooting AI (2019) with Gary Marcus. We reached out to him because of his expertise in artificial intelligence and because he wrote a critical review of Nick Bostrom's Superintelligence.
Davis told me, "the probability that autonomous AI is going to be one of our major problems within the next two hundred years, I think, is less than one in a hundred." We spoke about why he thinks that, what problems in AI he thinks are more urgent, and what his key points of disagreement with Nick Bostrom are. A full transcript of our conversation, lightly edited for concision and clarity, can be found here.
By Robert Long


Conversation with Ernie Davis

AI Impacts spoke with computer scientist Ernie Davis about his views of AI risk. With his permission, we have transcribed this interview.
Participants

Ernest Davis – professor of computer science at the Courant Institute of Mathematical Science, New York University
Robert Long – AI Impacts

Summary
We spoke over the phone with Ernie Davis on August 9, 2019. Some of the topics we covered were:

What Davis considers to be the most urgent risks from AI
Davis's disagreements with Nick Bostrom, Eliezer Yudkowsky, and Stuart Russell

The relationship between greater intelligence and greater power
How difficult it is to design a system that can be turned off
How difficult it would be to encode safe ethical principles in an AI system

Davis's evaluation of the likelihood that advanced, autonomous AI will be a major problem within the next two hundred years; and what evidence would change his mind
Challenges and progress towards human-level AI

This transcript has been lightly edited for concision and clarity.
Transcript
Robert Long: You're one of the few people, I think, who is an expert in AI, and is not necessarily embedded in the AI Safety community, but you have engaged substantially with arguments from that community. I'm thinking especially of your review of Superintelligence.1 2
I was hoping we could talk a little bit more about your views on AI safety work. There's a particular proposition that we're trying to get people's opinions on. The question is: Is it valuable for people to be expending significant effort doing work that purports to reduce the risk from advanced artificial intelligence? I've read some of your work; I can guess some of your views. But I was wondering: what would you say is your answer to that question, whether this kind of work is valuable to do now?
Ernie Davis: Well, a number of parts to the answer. In terms of short term—and "short" being not very short—short term risks from computer technology generally, this is very low priority. The risks from cyber crime, cyber terrorism, somebody taking hold of the insecurity of the internet of things and so on—that in particular is one of my bugaboos—are, I think, an awful lot more urgent. So there's urgency; I certainly don't see that this is especially urgent work.
Now, some of the approaches are being taken to long term AI safety seem to me extremely far fetched. On the one hand the fears of people like Bostrom and Yudkowsky and to a lesser extent Stuart Russell—seem to me misdirected and the approaches they are proposing are also misdirected. I have a book with Gary Marcus which is coming out in September, and we have a chapter which is called 'Trust' which gives our opinions—which are pretty much convergent—at length. I can send you that chapter.
Robert Long: Yes, I'd certainly be interested in that.
Ernie Davis: So, the kinds of things that Russell is proposing—Russell also has a book coming out in October, he is developing ideas that he's already published about: the way to have safe AI is to have them be unsure about what the human goals are.3 And Yudkowsky develops similar ideas in his work, engages with them, and tries to measure their success. This all seems to me too clever by half. And I don't think it's addressing what the real problems are going to be.
My feeling is that the problem of AIs doing the wrong thing is a very large one—you know, just by sheer inadvertence and incompetent design. And the solution there, more or less, is to design them well and build in safety features of the kinds that one has in engineering, one has throughout engineering. Whenever one is doing an engineering project, one builds in—one designs for failure. And one has to do that with AI as well. The danger of AI being abused by bad human actors is a very serious danger. And that has to be addressed politically, like all problems involving bad human actors.
And then there are directions in AI where I think it's foolish to go. For instance it would be very foolish to build—it's not currently technically feasible, but if it were, and it may at some point become technically feasible—to build robots that can reproduce themselves cheaply. And that's foolish, but it's foolish for exactly the same reason that you want to be careful about introducing new species. It's why Australia got into trouble with the rabbits, namely: if you have a device that can reproduce itself and it has no predators, then it will reproduce itself and it gets to be a nuisance.
And that's almost separate. A device doesn't have to be superintelligent to do that, in fact superintelligence probably just makes that harder because a superintelligent device is harder to build; a self replicating device might be quite easy to build on the cheap. It won't survive as well as a superintelligent one, but if it can reproduce itself fast enough that doesn't matter. So that kind of thing, you want to avoid.
There's a question which we almost entirely avoided in our book, which people always ask all the time, which is, at what point do machines become conscious. And my answer to that—I'm not necessarily speaking for Gary—my answer to that is that you want to avoid building machines which you have any reason to suspect are conscious. Because once they become conscious, they simply raise a whole collection of ethical issues like—"is it ethical turn them off?", is the first one, and "what are your responsibilities toward the thing?". And so you want to continue to have programs which, like current programs, one can think of purely as tools which we can use, which it is ethical to use as we choose.
So that's a thing to be avoided, it seems to me, in AI research. And whether people are wise enough to avoid that, I don't know. I would hope so. So in some ways I'm more conservative than a lot of people in the AI safety world—in the sense that they assume that self replicating robots will be a thing and that self-aware robots will be a thing and the object is to design them safely. My feeling is that research shouldn't go there at all.
Robert Long: I'd just like to dig in on a few more of those claims in particular. I would just like to hear a little bit more about what you think the crux of your disagreement is with people like Yudkowsky and Russell and Bostrom. Maybe you can pick one because they all have different views. So, you said that you feel that their fears are far-fetched and that their approaches are far-fetched as well. Can you just say a little bit about more about why you think that? A few parts: what you think is the core fear or prediction that their work is predicated on, and why you don't share that fear or prediction.
Ernie Davis: Both Bostrom very much, and Yudkowsky very much, and Russell to some extent, have this idea that if you're smart enough you get to be God. And that just isn't correct. The idea that a smart enough machine can do whatever it want—there's a really good essay by Steve Pinker, by the way, have you seen it?4
Robert Long: I've heard of it but have not read it.
Ernie Davis: I'll send you the link. A couple of good essays by Pinker, I think. So, it's not the case that once superintelligence is reached, then times become messianic if they're benevolent and dystopian if they're not. They're devices. They are limited in what they can do. And the other thing is that we are here first, and we should be able to design them in such a way that they're safe. It is not really all that difficult to design an AI or a robot which you can turn off and which cannot block you from turning it off.
And it seems to me a mistake to believe otherwise. With two caveats. One is that, if you embed it in a situation where it's very costly to turn off—it's controlling the power grid and the power grid won't work if you turn it off, then you're in trouble. And secondly, if you have malicious actors who are deliberately designing, building devices which can't be turned off. It's not that it's impossible to build an intelligent machine that is very dangerous.
But that doesn't require superintelligence. That's possible with very limited intelligence, and the more intelligent, to some extent, the harder it is. But again that's a different problem. It doesn't become a qualitatively different problem once the thing has exceeded some predefined level of intelligence.
Robert Long: You might be even more familiar with these arguments than I am—in fact I can't really recite them off the top of my head—but I suppose Bostrom and Yudkowsky, and maybe Russell too, do talk about this at length. And I guess they're they're always like, Well, you might think you have thought of a good failsafe for ensuring these things won't get un-turn-offable. But, so they say, you're probably underestimating just how weird things can get once you have superintelligence.
I suppose maybe that's precisely what you're disagreeing with: maybe they're overestimating how weird and difficult things get once things are above human level. Why do you think you and they have such different hunches, or intuitions, about how weird things can get?
Ernie Davis: I don't know, I think they're being unrealistic. If you take a 2019 genius and you put him into a Neolithic village, they can kill him no matter how intelligent he is, and how much he knows and so on.
Robert Long: I've been trying to trace the disagreements here and I think a lot of it does just maybe come down to people's intuitions about what a very smart person can do if put in a situation where they are far smarter than other people. I think this actually comes up in someone who responded to your review. They claim, "I think if I went back to the time of the Romans I could probably accrue a lot of power just by knowing things that they did not know."5
Ernie Davis: I missed that, or I forgot that or something.
Robert Long: Trying to locate the crux of the disagreement: one key disagreement is what the relationship is between greater intellectual capacity and greater physical power and control over the world. Does that seem safe to say, that that's one thing you disagree with them about?
Ernie Davis: I think so, yes. That's one point of disagreement. A second point of disagreement is the difficulty of—the point which we make in the book at some length is that, if you're going to have an intelligence that's in any way comparable to human, you're going to have to build in common sense. It's going to have to have a large degree of commonsense understanding. And once an AI has common sense it will realize that there's no point in turning the world into paperclips, and that there's no point in committing mass murder to go fetch the milk—Russell's example—and so on. My feeling is that one can largely incorporate a moral sense, when it becomes necessary; you can incorporate moral rules into your robots.
And one of the people who criticized my Bostrom paper said, well, philosophers haven't solved the problems of ethics in 2,000 years, how do you think we're going to solve them? And my feeling is we don't have to come up with the ultimate solution to ethical problems. You just have to make sure that they understand it to a degree that they don't do spectacularly foolish and evil things. And that seems to me doable.
Another point of disagreement with Bostrom in particular, and I think also Yudkowsky, is that they have the idea that ethical senses evolve—which is certainly true—and that a superintelligence, if well-designed, can be designed in such a way that it will itself evolve toward a superior ethical sense. And that this is the thing to do. Bostrom goes into this at considerable length: somehow, give it guidance toward an ethical sense which is beyond anything that we currently understand. That seems to me not very doable, but it would be a really bad thing to do if we could do it, because this super ethics might decide that the best thing to do is to exterminate the human population. And in some super-ethical sense that might be true, but we don't want it to happen. So the belief in the super ethics—I have no belief, I have no faith in the super ethics, and I have even less faith that there's some way of designing an AI so that as it grows superintelligent it will achieve super ethics in a comfortable way. So this all seems to me pie in the sky.
Robert Long: So the key points of disagreement we have so far are the relationship between intelligence and power; and the second thing is, how hard is what we might call the safety problem. And it sounds like even if you became more worried about very powerful AIs, you think it would not require substantial research and effort and money (as some people think) to make them relatively safe?
Ernie Davis: Where I would put the effort in is into thinking about, from a legal regulatory perspective, what we want to do. That's not an easy question.
The problem at the moment, the most urgent question, is the problem of fake news. We object to having bots spreading fake news. It's not clear what the best way of preventing that is without infringing on free speech. So that's a hard problem. And that is, I think, very well worth thinking about. But that's of course a very different problem. The problems of security at the practical level—making sure that an adversary can't take control of all the cars that are connected to the Internet and start using them as weapons—is, I think, a very pressing problem. But again that has nothing much to do with the AI safety projects that are underway.
Robert Long: Kind of a broad question—I was curious to hear what you make of the mainstream AI safety efforts that are now occurring. My rough sense is since your review and since Superintelligence, AI safety really gained respectability and now there are AI safety teams at places like DeepMind and OpenAI. And not only do they work on the near-term stuff which you talk about, but they are run by people who are very concerned about the long term. What do you make of that trend?
Ernie Davis: The thing is, I haven't followed their work very closely, to tell you the truth. So I certainly don't want to criticize it very specifically. There are smart and well-intentioned people on these teams, and I don't doubt that a lot of what they're doing is good work.
The work I'm most enthusiastic about in that direction is problems that are fairly near term. And also autonomous weapons is a pretty urgent problem, and requires political action. So the more that can be done about keeping those under control the better.
Robert Long: Do you think your views on what it will take before we ever get to human-level or more advanced AI, do you think that drives a lot of your opinions as well? For example, your own work on common sense and how hard of a problem that can be?6 7
Ernie Davis: Yeah sure, certainly it informs my views. It affects the question of urgency and it affects the question of what the actual problems are likely to be.
Robert Long: What would you say is your credence, your evaluation of the likelihood, that without significant additional effort, advanced AI poses a significant risk of harm?
Ernie Davis: Well, the problem is that without more work on artificial intelligence, artificial intelligence poses no risk. And the distinction between work on AI, and work on AI safety—work on AI is an aspect of work on AI safety. So I'm not sure it's a well-defined question.
But that's a bit of a debate. What we mean is, if we get rid of all the AI safety institutes, and don't worry about the regulation, and just let the powers that be do whatever they want to do, will advanced AI be a significant threat? There is certainly a sufficiently significant probability of that, but almost all of that probability has do with its misuse by bad actors.
The problem that AI will autonomously become a major threat, I put it at very small. The probability that people will start deploying AI in a destructive way and causing serious harm, to some extent or other, is fairly large. The probability that autonomous AI is going to be one of our major problems within the next two hundred years I think is less than one in a hundred.
Robert Long: Ah, good. Thank you for parsing that question. It's that last bit that I'm curious about. And what do you think are the key things that go into that low probability? It seems like there's two parts: odds of it being a problem if it arises, and odds of it arising. I guess what I'm trying to get at is—again, uncertainty in all of this—but do you have hunches or 'AI timelines' as people call them, about how far away we are from human level intelligence being a real possibility?
Ernie Davis: I'd be surprised—well, I will not be surprised, because I will be dead—but I would be surprised if AI reached human levels of capacity across the board within the next 50 years.
Robert Long: I suspect a lot of this is also found in your written work. But could you say briefly what you think are the things standing in the way, standing in between where we're at now in our understanding of AI, and getting to that—where the major barriers or confusions or new discoveries to be made are?
Ernie Davis: Major barriers—well, there are many barriers. We don't know how to give computers basic commonsense understanding of the world. We don't know how to represent the meaning of either language or what the computer can see through vision. We don't have a good theory of learning. Those, I think, are the main problems that I see and I don't see that the current direction of work in AI is particularly aimed at those problems.
And I don't think it's likely to solve those problems without a major turnaround. And the problems, I think, are very hard. And even after the field has turned around I think it will take decades before they're solved.
Robert Long: I suspect a lot of this might be what the book is about. But can you say what you think that turnaround is, or how you would characterize the current direction? I take it you mean something like deep learning and reinforcement learning?
Ernie Davis: Deep learning, end-to-end learning, is what I mean by the current direction. It is very much the current direction. And the turnaround, in one sentence, is that one has to engage with the problems of meaning, and with the problems of common sense knowledge.
Robert Long: Can you think of plausible concrete evidence that would change your views one way or the other? Specifically, on these issues of the problem of safety, and what if any work should be done.
Ernie Davis: Well sure, I mean, if, on the one hand, progress toward understanding in a broad sense—if there's startling progress on the problem of understanding then my timeline changes obviously, and that makes the problem harder.
And if it turned out—this is an empirical question—if it turned out that certain types of AI systems inherently turned toward single minded pursuit of malevolence or toward their own purposes and so on. And it seems to me wildly unlikely, but it's not unimaginable.
Or of course, if in a social sense—if people start uncontrollably developing these things. I mean it always amazes me the amount of sheer malice in the cyber world, the number of people who are willing to hack systems and develop bugs for no reason. The people who are doing it to make money is one thing, I can understand them. The people do it simply out of the challenge and out of the spirit of mischief making—I'm surprised that there are so many.
Robert Long: Can I ask a little bit more about what progress towards understanding looks like? What sort of tasks or behaviors? What does the arxiv paper that demonstrates that look like? What's it called, and what is is the program doing, where you're like, "Wow, this is this is a huge stride."
Ernie Davis: I have a paper called "How to write science questions that are easy for people and hard for computers."8 So once you get a response paper to that: "My system answers all the questions in this dataset which are easy for people and hard for computers." That would be impressive. If you have a program that can read basic narrative text and answer questions about it or watch a video and answer questions, a film and answer questions about it—that would be impressive.
Notes

Conversation with Ernie Davis


Evidence against current methods leading to human level artificial intelligence

This is a list of published arguments that we know of that current methods in artificial intelligence will not lead to human-level AI.
Details
Clarifications
We take 'current methods' to mean techniques for engineering artificial intelligence that are already known, involving no "qualitatively new ideas".1 We have not precisely defined 'current methods'. Many of the works we cite refer to currently dominant methods such as machine learning (especially deep learning) and reinforcement learning.
By human-level AI, we mean AI with a level of performance comparable to humans. We have in mind the operationalization of 'high-level machine intelligence' from our 2016 expert survey on progress in AI: "Say we have 'high-level machine intelligence' when unaided machines can accomplish every task better and more cheaply than human workers."2
Because we are considering intelligent performance, we have deliberately excluded arguments that AI might lack certain 'internal' features, even if it manifests human-level performance.3 4 We assume, concurring with Chalmers (2010), that "If there are systems that produce apparently [human-level intelligent] outputs, then whether or not these systems are truly conscious or intelligent, they will have a transformative impact."5
Methods
We read well-known criticisms of current AI approaches of which we were already aware. Using these as a starting point, we searched for further sources and solicited recommendations from colleagues familiar with artificial intelligence.
We include arguments that sound plausible to us, or that we believe other researchers take seriously. Beyond that, we take no stance on the relative strengths and weaknesses of these arguments.
We cite works that plausibly support pessimism about current methods, regardless of whether the works in question (or their authors) actually claim that current methods will not lead to human-level artificial intelligence.
We do not include arguments that serve primarily as undercutting defeaters of positive arguments that current methods will lead to human-level intelligence. For example, we do not include arguments that recent progress in machine learning has been overstated.
These arguments might overlap in various ways, depending on how one understands them. For example, some of the challenges for current methods might be special instances of more general challenges.
List of arguments
Inside view arguments
These arguments are 'inside view' in that they look at the specifics of current methods.
Innate knowledge: Intelligence relies on prior knowledge which it is currently not feasible to embed via learning techniques, recapitulate via artificial evolution, or hand-specify. — Marcus (2018)6Data hunger: Training a system to human level using current methods will require more data than we will be able to generate or acquire. — Marcus (2018)7
Capacities
Some researchers claim that there are capacities which are required for human-level intelligence, but difficult or impossible to engineer with current methods.8 Some commonly-cited capacities are:
Causal models: Building causal models of the world that are rich, flexible, and explanatory — Lake et al. (2016)9, Marcus (2018)10, Pearl (2018)11Compositionality: Exploiting systematic, compositional relations between entities of meaning, both linguistic and conceptual — Fodor and Pylyshyn (1988)12, Marcus (2001)13, Lake and Baroni (2017)14Symbolic rules: Learning abstract rules rather than extracting statistical patterns — Marcus (2018)15Hierarchical structure: Dealing with hierarchical structure, e.g. that of language — Marcus (2018)16Transfer learning: Learning lessons from one task that transfer to other tasks that are similar, or that differ in systematic ways — Marcus (2018)17, Lake et al. (2016)18Common sense understanding: Using common sense to understand language and reason about new situations — Brooks (2019)19, Marcus and Davis (2015)20
Outside view arguments
These arguments are 'outside view' in that they look at "a class of cases chosen to be similar in relevant respects"21 to current artificial intelligence research, without looking at the specifics of current methods.
Lack of progress: There are many tasks specified several decades ago that have not been solved, e.g. effectively manipulating a robot arm,  open-ended question-answering. — Brooks (2018)22, Jordan (2018)23Past predictions: Past researchers have incorrectly predicted that we would get to human-level AI with then-current methods. — Chalmers (2010)24Other fields: Several fields have taken centuries or more to crack; AI could well be one of them. — Brooks (2018)25
Contributions
Robert Long and Asya Bergal contributed research and writing.
Notes
Featured image from www.extremetech.com.


Historic trends in land speed records

Land speed records did not see any greater-than-10-year discontinuities relative to linear progress across all records. Considered as several distinct linear trends it saw discontinuities of 12, 13, 25, and 13 years, the first two corresponding to early (but not first) jet-propelled vehicles.
The first jet-propelled vehicle just predated a marked change in the rate of progress of land speed records, from a recent 1.8 mph / year to 164 mph / year.
Details
This case study is part of AI Impacts' discontinuous progress investigation.
Background
According to Wikipedia, the land speed record is "the highest speed achieved by a person using a vehicle on land."1 Wheel-driven cars, which supply power to their axles, held the records for land speed record through 1963, when the first turbojet powered vehicles arrived on the scene. No wheel-driven car has held the record since 1964.2
Figure 1: Three record-setting vehicles: Sunbeam, Sunbeam Blue Bird, and Blue Bird3
Trends
Land speed records
Data
We took data from Wikipedia's list of land speed records,4 which we have not verified, and added it to this spreadsheet. See Figure 2 below.
Figure 2: Historic land speed records in mph over time. Speeds on the left are an average of the record set in mph over 1 km and over 1 mile. The red dot represents the first record in a cluster that was from a jet propelled vehicle. The discontinuities of more than ten years are the third and fourth turbojet points, and the last two points.
Discontinuity measurement
If we treat the data as a linear trend across all time,5 then the land speed record did not contain any greater than 10-year discontinuities.
However we divide the data into several linear trends.6 Extrapolating based on these trends, there were four discontinuities of sizes 12, 13, 25, and 13 years, produced by different turbojet-powered vehicles.7In addition to the size of these discontinuities in years, we have tabulated a number of other potentially relevant metrics here.8
Changes in the rate of progress
There are several marked changes in the rate of progress in this history. The first two discontinuities are near the start of a sharp change, that seemed to come from the introduction of jet-propulsion (though note that the first jet-propelled vehicle in the trend is neither discontinuous with the previous trend, nor seemingly within the period of faster growth).
If we look at the rates of progress in the stretches directly before the second jet propelled vehicle in 1964, and the stretch directly after that through 1965, the rate of progress increases from 1.8 mph / year to 164 mph / year.9
Notes

Historic trends in land speed records