Comments on OpenAI’s "Planning for AGI and beyond"

Sam Altman shared me on a draft of his OpenAI blog post Planning for AGI and beyond, and I left some comments, reproduced below without typos and with some added hyperlinks. Where the final version of the OpenAI post differs from the draft, I’ve noted that as well, making text Sam later cut red and text he added blue.
My overall sense is that Sam deleted text and occasionally rephrased sentences so as to admit more models (sometimes including mine), but didn’t engage with the arguments enough to shift his own probability mass around on the important disagreements.
Our disagreements are pretty major, as far as I can tell. With my comments, I was hoping to spark more of a back-and-forth. Having failed at that, I’m guessing part of the problem is that I didn’t phrase my disagreements bluntly or strongly enough, while also noting various points of agreement, which might have overall made it sound like I had only minor disagreements.

To help with that, I’ve added blunter versions below, in a bunch of cases where I don’t think the public version of the post fully takes into account the point I was trying to make.
(Though I don’t want this to take away from the positive aspects of the post, since I think these are very important as well. I put in a bunch of positive comments on the original draft, in large part because I think it’s worth acknowledging and reinforcing whatever process/author drafted the especially reasonable paragraphs.)
I don’t expect Sam to hear me make a blunt claim and instantly update to agreeing with me, but I put more probability on us converging over time, and clearly stating our disagreements for the benefit of readers, if he understands my claims, knows I still disagree, and feels license to push back on specific things I’ve re-asserted.

Formatting note: The general format is that I include the text of Sam’s original draft (that I commented on), my comment, and the text of the final post. That said, I don’t want to go blasting someone’s old private drafts across the internet just because they shared that draft with me, so in some cases, the original text is redacted, at Sam’s request.

Sam’s draft: Our mission is to ensure that AGI benefits all of humanity. The creation of AGI should be a tremendous shared triumph that everyone contributes to and benefits from; it will be the result of the collective technological and societal progress of humanity over millennia.
My comment: +1
Sam’s post: Our mission is to ensure that artificial general intelligence— AI systems that are generally smarter than humans—benefits all of humanity.

Sam’s draft: Of course our current progress could hit a wall, but if AGI is successfully created, this technology could help us elevate humanity by increasing abundance, turbocharging our economy, and aiding in the discovery of new scientific knowledge.
My comment: seems to me an understatement :-p
(unlocking nanotech; uploading minds; copying humans; interstellar probes that aren’t slowed down by needing to cradle bags of meat, and that can have the minds beamed to them; energy abundance; ability to run civilizations on computers in the cold of space; etc. etc., are all things that i expect to follow from automated scientific & technological development)
(seems fine to avoid the more far-out stuff, and also fine to only say things that you personally believe, but insofar as you also expect some of this tech to be within-reach in 50 sidereal years after AGI, i think it’d be virtuous to acknowledge)
Sam’s post: If AGI is successfully created, this technology could help us elevate humanity by increasing abundance, turbocharging the global economy, and aiding in the discovery of new scientific knowledge that changes the limits of possibility.
Blunter follow-up: seems to undersell the technological singularity, and the fact that the large-scale/coarse-grain shape of the future will be governed by superintelligences.

Sam’s draft: On the other hand, AGI would also come with serious risk of misuse and drastic accidents. Because the upside of AGI is so great, we do not believe it’s possible or desirable for society to stop its development forever; instead, we have to figure out how to get it right. [1]
My comment: +1
Sam’s post: On the other hand, AGI would also come with serious risk of misuse, drastic accidents, and societal disruption. Because the upside of AGI is so great, we do not believe it is possible or desirable for society to stop its development forever; instead, society and the developers of AGI have to figure out how to get it right.C
Blunter follow-up: still +1, with the caveat that accident risk >> misuse risk

Sam’s draft: 1) We want AGI to empower humanity to maximally flourish in the universe. We don’t expect the future to be an unqualified utopia, but we want to maximize the good and minimize the bad, and for AGI to be an amplifier of human will.
My comment: i think i agree with the sentiment here, but i disagree with parts of the literal denotation
for one, i sure hope for an unqualified utopia, and think there’s a chance that superintelligent assistance could figure out how to get one (cf fun theory).
(it is ofc important to note that "superintelligences puppet the humans through the motions of a utopia" is not in fact a utopia, and that the future will undoubtedly include tradeoffs (including continuing to let people make their own mistakes and learn their own lessons), and so in that sense i agree that it wouldn’t be an "unqualified utopia", even in the best case)
…though i don’t currently expect us to do that well, so i don’t technically disagree with the literal phrasing you chose there.
i do have qualms about "we want AGI to be an amplifier of human will". there’s a bunch of ways that this seems off-kilter to me. my basic qualm here is that i think getting a wonderful future is more of a fragile operation than simply cranking up everybody’s "power level" simultaneously, roughly analogously to how spoiling a child isn’t the best way for them to grow up.
i’d stand full-throatedly behind "we want AGI to be an amplifier of all the best parts of humanity".
also, i ofc ultimately want AGI that are also people, to be humanity’s friends as we explore the universe and so on. (though, stating the obvious, i think we should aim to avoid personhood in our early AGIs, for various reasons.)
Sam’s post: 1. We want AGI to empower humanity to maximally flourish in the universe. We don’t expect the future to be an unqualified utopia, but we want to maximize the good and minimize the bad, and for AGI to be an amplifier of humanity.
Blunter follow-up: we can totally get an unqualified utopia. also this "amplifier of humanity" thing sounds like an applause light—though i endorse certain charitable interpretations that i can wring from it (that essentially amount to CEV (as such things usually do)), at the same time i disendorse other interpretations.

Sam’s draft: 2) We want the benefits of, access to, and governance of AGI to be widely and fairly shared.
My comment: +1 to benefits of. i have lots more qualms about "access to" and "governance of".
re "access to", my guess is that early AGIs will be able to attain a decisive strategic advantage over the rest of the world entire. saying "everyone should have equal access" seems to me like saying "a nuclear bomb in every household"; it just sounds kinda mad.
i’d agree that once the world has exited the acute risk period, it’s critical for access to AGI tech to be similarly available to all. but that is, in my book, a critical distinction.
(so access-wise, i agree long-term, but not short-term.)
governance-wise, my current state is something like: in the short term, using design-by-committee to avert the destruction of the world sounds like a bad idea; and in the long term, i think you’re looking at stuff at least as crazy as people running thousands of copies of their own brain at 1000x speedup and i think it would be dystopian to try to yolk them to, like, the will of the flesh-bodied American taxpayers (or whatever).
there’s something in the spirit of "distributed governance" that i find emotionally appealing, but there’s also lots and lots of stuff right nearby, that would be catastrophic, dystopian, or both, and that implementation would be likely to stumble into in practice. so i have qualms about that one.
Sam’s post: [unchanged]
Blunter follow-up: full-throated endorsement of "benefits of". widely sharing access and governance in the short-term seems reckless and destructive.

Sam’s draft: 3) We want to successfully navigate massive risks. In confronting these risks, we acknowledge that what seems right in theory frequently plays out more strangely than expected in practice. We believe we have to continuously learn and adapt by deploying less powerful versions of the technology in order to avoid a "one shot to get it right" scenario.
My comment: i don’t think that this "continuously deploy weak systems" helps avoid the "you have one shot"-type problems that i predict we’ll face in the future.
(this also strikes me as a rationalization for continuing to do the fun/cool work of pushing the capabilities envelope, which i currently think is net-bad for everyone)
Sam’s post: 3. We want to successfully navigate massive risks. In confronting these risks, we acknowledge that what seems right in theory often plays out more strangely than expected in practice. We believe we have to continuously learn and adapt by deploying less powerful versions of the technology in order to minimize "one shot to get it right" scenarios.
Blunter follow-up: i think it takes a bunch more than "continuously deploy weak systems" to address the "one shot to get it right" scenarios, and none of the leading orgs (OAI included) seem to me to be on track to acquire the missing parts.

Sam’s draft: a gradual transition to a world with AGI is better than a sudden one
My comment: for the record, i don’t think continuous deployment really smooths out the sharp changes that i expect in the future. (i’m not trying to argue the point here, just noting that there are some people who are predicting a sort of sharp change that they think is ~unrelated to your choice of continuous deployment.)
Sam’s post: [unchanged]
Blunter follow-up: insofar as this sentence is attempting to reply to MIRI-esque concerns, i don’t think it’s a very good reply (for the reasons alluded to in the original comment).

Sam’s draft: A gradual transition gives people, policymakers, and institutions time to understand what’s happening, personally experience the benefits and downsides of these systems, adapt our economy, and to put regulation in place.
My comment: i’m skeptical, similar to the above.
Sam’s post: [unchanged]

Sam’s draft: It also allows us to learn as much as we can from our deployments, for society and AI to co-evolve, and to collectively figure out what we want while the stakes are relatively low.
My comment: stating the obvious, other ways of learning as much as you can from the systems you have include efforts in transparency, legibility, and interpretability.
Sam’s post: It also allows for society and AI to co-evolve, and for people collectively to figure out what they want while the stakes are relatively low.

Sam’s draft: As our systems get closer to AGI, we are becoming increasingly more cautious with the creation and deployment of our models.
My comment: +1
Sam’s post: As our systems get closer to AGI, we are becoming increasingly cautious with the creation and deployment of our models.
Blunter follow-up: +1 as a thing y’all should do. my guess is that you need to do it even faster and more thoroughly than you have been.
a more general blunt note: me +1ing various statements does not mean that i think the corporate culture has internalized the corresponding points, and it currently looks likely to me that OpenAI is not on track to live up to the admirable phrases in this post, and are instead on track to get everyone killed.
i still think it’s valuable that the authors of this post are thinking about these points, and i hope that these sorts of public endorsements increase the probability that the corresponding sentiments actually end up fully internalized in the corporate culture, but i want to be clear that the actions are what ultimately matter, not the words.

Sam’s draft: Our decisions will require much more caution than society usually applies to new technologies, and more caution than many users would like. Some people in the AI field think the risks of AGI (and successor systems) are fictitious; we would be delighted if they turn out to be right, but we are going to operate as if these risks are existential.
My comment: hooray
Sam’s post: [unchanged]
Blunter follow-up: to be clear, my response to the stated sentiment is "hooray", and i’m happy to see this plainly and publicly stated, but i have strong doubts about whether and how this sentiment will be implemented in practice.

Sam’s draft: At some point, the balance between the upsides and downsides of deployments (such as empowering malicious actors, creating social and economic disruptions, and accelerating an unsafe race) could shift, in which case we would significantly change our plans.
My comment: this is vague enough that i don’t quite understand what it’s saying; i’d appreciate it being spelled out more
Sam’s post: At some point, the balance between the upsides and downsides of deployments (such as empowering malicious actors, creating social and economic disruptions, and accelerating an unsafe race) could shift, in which case we would significantly change our plans around continuous deployment.

Sam’s draft: Second, we are working towards creating increasingly aligned (i.e., models that reliably follow their users’ intentions) and steerable models. Our shift from models like the first version of GPT-3 to ChatGPT and InstructGPT is an early example of this.
My comment: ftr, i think there’s a bunch of important notkilleveryoneism work that won’t be touched upon by this approach
Sam’s post: Second, we are working towards creating increasingly aligned and steerable models. Our shift from models like the first version of GPT-3 to InstructGPT and ChatGPT is an early example of this

Sam’s draft: Importantly, we think we often have to make progress on AI safety and capabilities together (and that it’s a false dichotomy to talk about them separately; they are correlated in many ways). Our best safety work has come from working with our most capable models.
My comment: i agree that they’re often connected, but i also think that traditionally the capabilities runs way out ahead of the alignment, and that e.g. if capabilities progress was paused now, there would be many years’ worth of alignment work that could be done to catch up (e.g. by doing significant work on transparency, legibility, and interpretability). and i think that if we do keep running ahead with the current capabilities/alignment ratio (or even a slightly better one), we die.
(stating the obvious: this is not to say that transparency/legibility/interpretability aren’t also intertwined with capabilities; it’s all intertwined to some degree. but one can still avoid pushing the capabilities frontier, and focus on the alignment end of things. and one can still institute a policy of privacy, to further avoid burning the commons.)
Sam’s post: Importantly, we think we often have to make progress on AI safety and capabilities together. It’s a false dichotomy to talk about them separately; they are correlated in many ways. Our best safety work has come from working with our most capable models. That said, it’s important that the ratio of safety progress to capability progress increases.

Sam’s draft: We have a clause in our charter about assisting other organizations instead of racing them in late-stage AGI development. [ redacted statement that got cut ]
My comment: i think it’s really cool of y’all to have this; +1
Sam’s post: We have a clause in our Charter about assisting other organizations to advance safety instead of racing with them in late-stage AGI development.

Sam’s draft: We have a cap on the returns our shareholders can earn so that we aren’t incentivized to attempt to capture value without bound and risk deploying something potentially catastrophically dangerous (and of course as a way to share the benefits with society)
My comment: also rad
Sam’s post: [unchanged]

Sam’s draft: We believe that the future of humanity should be determined by humanity.  [ redacted draft version of "and that it’s important to share information aboutprogress with the public" ]
My comment: +1 to "the future of humanity should be determined by humanity".
My comment (#2): i agree with some of the sentiment of [redacted sentence in Sam’s draft], but note that things get weird in the context of a global arms race for potentially-civilization-ending tech. i, for one, am in favor of people saying "we are now doing our AGI research behind closed doors, because we don’t think it would be used wisely if put out in the open".
Sam’s post: We believe that the future of humanity should be determined by humanity, and that it’s important to share information about progress with the public.
Blunter follow-up: The +1 is based on a charitable read where "the future of humanity should be determined by humanity" irons out into CEV, as such things often do.
Blunter follow-up (#2): seems to me like a lot of weight is being put on "information about progress".
there’s one read of this claim that says something like "the average human should know that a tiny cluster of engineers are about to gamble with everybody’s fate", which does have a virtuous ring to it. and i wouldn’t personally argue for >hiding that fact from anyone. but this is not a difficult fact for savvy people to learn from public information today.
is Sam arguing that there’s some concrete action in this class that the field has an unmet obligation to do, like dropping flyers in papua new guinea? this currently strikes me as a much more niche concern than the possible fast-approaching deaths of everyone (including papua new guineans!) at the hands of unfriendly AI, so i find it weird to mix those two topics together in a post about how to ensure a positive singularity.
possibly Sam has something else in mind, but if so i encourage more concreteness about what that is and why it’s important.

Sam’s draft: [ redacted statement that got cut ] There should be great scrutiny of all efforts attempting to build AGI and public consultation for major decisions.
My comment: totally agree that people shouldn’t try to control the world behind closed doors. that said, i would totally endorse people building defensive technology behind closed doors, in attempts to (e.g.) buy time.
(ideally this would be done by state actors, which are at least semilegitimate. but if someone’s building superweapons, and you can build technology that thwarts them, and the state actors aren’t responding, then on my ethics it’s ok to build the technology that thwarts them, so long as this also does not put people at great risk by your own hands.)
(of course, most people building powerful tech that think they aren’t putting the world at great risk by their own hands, are often wrong; there’s various types of thinking on this topic that should simply not be trusted; etc. etc.)
[ Post-hoc note: It’s maybe worth noting that on my ethics there’s an enormous difference between "a small cabal of humans exerts direct personal control over the world" and "run a CEV sovereign", and i’m against the former but for the latter, with the extra caveat that nobody should be trying to figure out a CEV sovereign under time-pressure, nor launching an AGI unilaterally simply because they managed to convince themselves it was a CEV sovereign. ]
My comment (#2): on "There should be great scrutiny of all efforts attempting to build AGI and public consultation for major decisions", i agree with something like "it’s ethically important that the will of all humans goes into answering the questions of where superintelligence should guide the future". this is separate from endorsing any particular design-by-committee choice of governance.
(for instance, if everybody today casts a vote for their favorite government style that they can think of, and then the AGI does the one that wins the most votes, i think that would end up pretty bad.)
which is to say, there’s some sentiment in sentences like this that i endorse, but the literal denotation makes me uneasy, and feels kinda like an applause-light. my qualms could perhaps be assuaged by a more detailed proposal, that i could either endorse or give specific qualms about.
Sam’s post: There should be great scrutiny of all efforts attempting to build AGI and public consultation for major decisions.

Sam’s draft: The first AGI will be just a point along the continuum of intelligence. We think it’s likely that progress will continue from there, possibly sustaining the rate of progress we’ve seen over the past decade for a long period of time. If this is true, the world could become extremely different from how it is today, and the risks could be extraordinary. A misaligned superintelligent AGI could cause grievous harm to the world; an autocratic regime with a decisive superintelligence lead could do that too.
My comment: +1
Sam’s post: [unchanged]

Sam’s draft: AI that can accelerate science is a special case worth thinking about, and perhaps more impactful than everything else. It’s possible that AGI capable enough to accelerate its own progress could cause major changes to happen surprisingly quickly (and even if the transition starts slowly, we expect it to happen pretty quickly in the final stages).
My comment: +1
Sam’s post: [unchanged]

Sam’s draft: We think a slower takeoff is easier to make safe, and coordination among AGI efforts to slow down at critical junctures will likely be important (even in a world where we don’t need to do this to solve technical alignment problems, slowing down may be important to give society enough time to adapt).
My comment: (sounds nice in principle, but i will note for the record that the plan i’ve heard for this is "do continuous deployment and hope to learn something", and i don’t expect that to help much in slowing down or smoothing out a foom.)
Sam’s post: [unchanged]

Sam’s draft: [redacted version with phrasing similar to the public version]
My comment: +1
Sam’s post: Successfully transitioning to a world with superintelligence is perhaps the most important—and hopeful, and scary—project in human history. Success is far from guaranteed, and the stakes (boundless downside and boundless upside) will hopefully unite all of us.

Sam’s draft: [ redacted statement calling back to "Our approach to alignment research" ]
My comment: i think this basically doesn’t work unless the early systems are already very aligned. (i have various drafts queued about this, as usual)
Sam’s post: [sentence deleted]
The post Comments on OpenAI’s "Planning for AGI and beyond" appeared first on Machine Intelligence Research Institute.


Focus on the places where you feel shocked everyone’s dropping the ball

Writing down something I’ve found myself repeating in different conversations:
If you’re looking for ways to help with the whole “the world looks pretty doomed” business, here’s my advice: look around for places where we’re all being total idiots.
Look for places where everyone’s fretting about a problem that some part of you thinks it could obviously just solve.
Look around for places where something seems incompetently run, or hopelessly inept, and where some part of you thinks you can do better.
Then do it better.
For a concrete example, consider Devansh. Devansh came to me last year and said something to the effect of,  “Hey, wait, it sounds like you think Eliezer does a sort of alignment-idea-generation that nobody else does, and he’s limited here by his unusually low stamina, but I can think of a bunch of medical tests that you haven’t run, are you an idiot or something?” And I was like, “Yes, definitely, please run them, do you need money”.
I’m not particularly hopeful there, but hell, it’s worth a shot! And, importantly, this is the sort of attitude that can lead people to actually trying things at all, rather than assuming that we live in a more adequate world where all the (seemingly) dumb obvious ideas have already been tried.

Or, this is basically my model of how Paul Christiano manages to have a research agenda that seems at least internally coherent to me. From my perspective, he’s like, “I dunno, man, I’m not sure I can solve this, but I also think it’s not clear I can’t, and there’s a bunch of obvious stuff to try, that nobody else is even really looking at, so I’m trying it”. That’s the sort of orientation to the world that I think can be productive.
Or the shard theory folks. I think their idea is basically unworkable, but I appreciate the mindset they are applying to the alignment problem: something like, “Wait, aren’t y’all being idiots, it seems to me like I can just do X and then the thing will be aligned”.
I don’t think we’ll be saved by the shard theory folk; not everyone audaciously trying to save the world will succeed. But if someone does save us, I think there’s a good chance that they’ll go through similar “What the hell, are you all idiots?” phases, where they autonomously pursue a path that strikes them as obviously egregiously neglected, to see if it bears fruit.
Contrast this with, say, reading a bunch of people’s research proposals and explicitly weighing the pros and cons of each approach so that you can work on whichever seems most justified. This has more of a flavor of taking a reasonable-sounding approach based on an argument that sounds vaguely good on paper, and less of a flavor of putting out an obvious fire that for some reason nobody else is reacting to.
I dunno, maybe activities of the vaguely-good-on-paper character will prove useful as well? But I mostly expect the good stuff to come from people working on stuff where a part of them sees some way that everybody else is just totally dropping the ball.
In the version of this mental motion I’m proposing here, you keep your eye out for ways that everyone’s being totally inept and incompetent, ways that maybe you could just do the job correctly if you reached in there and mucked around yourself.
That’s where I predict the good stuff will come from.
And if you don’t see any such ways?
Then don’t sweat it. Maybe you just can’t see something that will help right now. There don’t have to be ways you can help in a sizable way right now.
I don’t see ways to really help in a sizable way right now. I’m keeping my eyes open, and I’m churning through a giant backlog of things that might help a nonzero amount—but I think it’s important not to confuse this with taking meaningful bites out of a core problem the world is facing, and I won’t pretend to be doing the latter when I don’t see how to.
Like, keep your eye out. For sure, keep your eye out. But if nothing in the field is calling to you, and you have no part of you that says you could totally do better if you deconfused yourself some more and then handled things yourself, then it’s totally respectable to do something else with your hours.

If you don’t have an active sense that you could put out some visibly-raging fires yourself (maybe after skilling up a bunch, which you also have an active sense you could do), then I recommend stuff like cultivating your ability to get excited about things, and doing other cool stuff.
Sure, most stuff is lower-impact than saving the world from destruction. But if you can be enthusiastic about all the other cool ways to make the world better off around you, then I’m much more optimistic that you’ll be able to feel properly motivated to combat existential risk if and when an opportunity to do so arises. Because that opportunity, if you get one, probably isn’t going to suddenly unlock every lock on the box your heart hides your enthusiasm in, if your heart is hiding your enthusiasm.
See also Rob Wiblin’s “Don’t pursue a career for impact — think about the world’s most important, tractable and neglected problems and follow your passion.”
Or the Alignment Research Field Guide’s advice to “optimize for your own understanding” and chase the things that feel alive and puzzling to you, as opposed to dutifully memorizing other people’s questions and ideas. “[D]on’t ask “What are the open questions in this field?” Ask: “What are my questions in this field?””
I basically don’t think that big changes come from people who aren’t pursuing a vision that some part of them “believes in”, and I don’t think low-risk, low-reward, modest, incremental help can save us from here.
To be clear, when I say “believe in”, I don’t mean that you necessarily assign high probability to success! Nor do I mean that you’re willing to keep trying in the face of difficulties and uncertainties (though that sure is useful too).
English doesn’t have great words for me to describe what I mean here, but it’s something like: your visualization machinery says that it sees no obstacle to success, such that you anticipate either success or getting a very concrete lesson.
The possibility seems open to you, at a glance; and while you may suspect that there’s some hidden reason that the possibility is not truly open, you have an opportunity here to test whether that’s so, and to potentially learn why this promising-looking idea fails.
(Or maybe it will just work. It’s been known to happen, in many a scenario where external signs and portents would have predicted failure.)

The post Focus on the places where you feel shocked everyone’s dropping the ball appeared first on Machine Intelligence Research Institute.


What I mean by “alignment is in large part about making cognition aimable at all”

(Epistemic status: attempting to clear up a misunderstanding about points I have attempted to make in the past. This post is not intended as an argument for those points.)
I have long said that the lion’s share of the AI alignment problem seems to me to be about pointing powerful cognition at anything at all, rather than figuring out what to point it at.
It’s recently come to my attention that some people have misunderstood this point, so I’ll attempt to clarify here.
In saying the above, I do not mean the following:

(1) Any practical AI that you’re dealing with will necessarily be cleanly internally organized around pursuing a single objective. Managing to put your own objective into this “goal slot” (as opposed to having the goal slot set by random happenstance) is a central difficult challenge. [Reminder: I am not asserting this]

Instead, I mean something more like the following:

(2) By default, the first minds humanity makes will be a terrible spaghetti-code mess, with no clearly-factored-out “goal” that the surrounding cognition pursues in a unified way. The mind will be more like a pile of complex, messily interconnected kludges, whose ultimate behavior is sensitive to the particulars of how it reflects and irons out the tensions within itself over time.
Making the AI even have something vaguely nearing a ‘goal slot’ that is stable under various operating pressures (such as reflection) during the course of operation, is an undertaking that requires mastery of cognition in its own right—mastery of a sort that we’re exceedingly unlikely to achieve if we just try to figure out how to build a mind, without filtering for approaches that are more legible and aimable.

Separately and independently, I believe that by the time an AI has fully completed the transition to hard superintelligence, it will have ironed out a bunch of the wrinkles and will be oriented around a particular goal (at least behaviorally, cf. efficiency—though I would also guess that the mental architecture ultimately ends up cleanly-factored (albeit not in a way that creates a single point of failure, goalwise)).
(But this doesn’t help solve the problem, because by the time the strongly superintelligent AI has ironed itself out into something with a “goal slot”, it’s not letting you touch it.)
Furthermore, insofar as the AI is capable of finding actions that force the future into some narrow band, I expect that it will tend to be reasonable to talk about the AI as if it is (more-or-less, most of the time) “pursuing some objective”, even in the stage where it’s in fact a giant kludgey mess that’s sorting itself out over time in ways that are unpredictable to you.
I can see how my attempts to express these other beliefs could confuse people into thinking that I meant something more like (1) above (“Any practical AI that you’re dealing with will necessarily be cleanly internally organized around pursuing a single objective…”), when in fact I mean something more like (2) (“By default, the first minds humanity makes will be a terrible spaghetti-code mess…”).

In case it helps those who were previously confused: the “diamond maximizer” problem is one example of an attempt to direct researchers’ attention to the challenge of cleanly factoring cognition around something a bit like a ‘goal slot’.
As evidence of a misunderstanding here: people sometimes hear me describe the diamond maximizer problem, and respond to me by proposing training regimes that (for all they know) might make the AI care a little about diamonds in some contexts.
This misunderstanding of what the diamond maximizer problem was originally meant to be pointing at seems plausibly related to the misunderstanding that this post intends to clear up. Perhaps in light of the above it’s easier to understand why I see such attempts as shedding little light on the question of how to get cognition that cleanly pursues a particular objective, as opposed to a pile of kludges that careens around at the whims of reflection and happenstance.

The post What I mean by “alignment is in large part about making cognition aimable at all” appeared first on Machine Intelligence Research Institute.


July 2022 Newsletter

MIRI has put out three major new posts:
AGI Ruin: A List of Lethalities. Eliezer Yudkowsky lists reasons AGI appears likely to cause an existential catastrophe, and reasons why he thinks the current research community—MIRI included—isn't succeeding at preventing this from happening
A central AI alignment problem: capabilities generalization, and the sharp left turn. Nate Soares describes a core obstacle to aligning AGI systems:
[C]apabilities generalize further than alignment (once capabilities start to generalize real well (which is a thing I predict will happen)). And this, by default, ruins your ability to direct the AGI (that has slipped down the capabilities well), and breaks whatever constraints you were hoping would keep it corrigible.
On Nate's model, very little work is currently going into this problem. He advocates for putting far more effort into addressing this challenge in particular, and making it a major focus of future work.
Six Dimensions of Operational Adequacy in AGI Projects. Eliezer describes six criteria an AGI project likely needs to satisfy in order to have a realistic chance at preventing catastrophe at the time AGI is developed: trustworthy command, research closure, strong opsec, common good commitment, alignment mindset, and requisite resource levels.
Other MIRI updates

I (Rob Bensinger) wrote a post discussing the inordinately slow spread of good AGI conversations in ML.
I want to signal-boost two of my forum comments: on AGI Ruin, a discussion of common mindset issues in thinking about AGI alignment; and on Six Dimensions, a comment on pivotal acts and "strawberry-grade" alignment.
Also, a quick note from me, in case this is non-obvious: MIRI leadership thinks that humanity never building AGI would mean the loss of nearly all of the future's value. If this were a live option, it would be an unacceptably bad one.
Nate discusses MIRI's past writing on recursive self-improvement (with good discussion in the comments).
Let's See You Write That Corrigibility Tag: Eliezer posts a challenge to write a list of "the sort of principles you'd build into a Bounded Thing meant to carry out some single task or task-class and not destroy the world by doing it".
From Eliezer: MIRI announces new "Death With Dignity" strategy. Although released on April Fools' Day (whence the silly title), the post body is an entirely non-joking account of Eliezer's current models, including his currently-high p(doom) and his recommendations on conditionalization and naïve consequentialism.

News and links

Paul Christiano (link) and Zvi Mowshowitz (link) share their takes on the AGI Ruin post.
Google's new large language model, Minerva, achieves 50.3% performance on the MATH dataset (problems at the level of high school math competitions), a dramatic improvement on the previous state of the art of 6.9%.
Jacob Steinhardt reports generally poor forecaster performance on predicting AI progress, with capabilities work moving faster than expected and robustness slower than expected. Outcomes for both the MATH and Massive Multitask Language Understanding datasets "exceeded the 95th percentile prediction".
In the wake of April/May/June results like Minerva, Google's PaLM, OpenAI's DALL-E, and DeepMind's Chinchilla and Gato, Metaculus' "Date of Artificial General Intelligence" forecast has dropped from 2057 to 2039. (I'll mention that Eliezer and Nate's timelines were already pretty short, and I'm not aware of any MIRI updates toward shorter timelines this year. I'll also note that I don't personally put much weight on Metaculus' AGI timeline predictions, since many of them are inconsistent and this is a difficult and weird domain to predict.)
Conjecture is a new London-based AI alignment startup with a focus on short-timeline scenarios, founded by EleutherAI alumni. The organization is currently hiring engineers and researchers, and is "particularly interested in hiring devops and infrastructure engineers with supercomputing experience".

The post July 2022 Newsletter appeared first on Machine Intelligence Research Institute.

July 2022 Newsletter


A central AI alignment problem: capabilities generalization, and the sharp left turn

(This post was factored out of a larger post that I (Nate Soares) wrote, with help from Rob Bensinger, who also rearranged some pieces and added some text to smooth things out. I’m not terribly happy with it, but am posting it anyway (or, well, having Rob post it on my behalf while I travel) on the theory that it’s better than nothing.)

I expect navigating the acute risk period to be tricky for our civilization, for a number of reasons. Success looks to me to require clearing a variety of technical, sociopolitical, and moral hurdles, and while in principle sufficient mastery of solutions to the technical problems might substitute for solutions to the sociopolitical and other problems, it nevertheless looks to me like we need a lot of things to go right.
Some sub-problems look harder to me than others. For instance, people are still regularly surprised when I tell them that I think the hard bits are much more technical than moral: it looks to me like figuring out how to aim an AGI at all is harder than figuring out where to aim it.[1]
Within the list of technical obstacles, there are some that strike me as more central than others, like “figure out how to aim optimization”. And a big reason why I’m currently fairly pessimistic about humanity’s odds is that it seems to me like almost nobody is focusing on the technical challenges that seem most central and unavoidable to me.
Many people wrongly believe that I’m pessimistic because I think the alignment problem is extraordinarily difficult on a purely technical level. That’s flatly false, and is pretty high up there on my list of least favorite misconceptions of my views.[2]
I think the problem is a normal problem of mastering some scientific field, as humanity has done many times before. Maybe it’s somewhat trickier, on account of (e.g.) intelligence being more complicated than, say, physics; maybe it’s somewhat easier on account of how we have more introspective access to a working mind than we have to the low-level physical fields; but on the whole, I doubt it’s all that qualitatively different than the sorts of summits humanity has surmounted before.
It’s made trickier by the fact that we probably have to attain mastery of general intelligence before we spend a bunch of time working with general intelligences (on account of how we seem likely to kill ourselves by accident within a few years, once we have AGIs on hand, if no pivotal act occurs), but that alone is not enough to undermine my hope.
What undermines my hope is that nobody seems to be working on the hard bits, and I don’t currently expect most people to become convinced that they need to solve those hard bits until it’s too late.
Below, I’ll attempt to sketch out what I mean by “the hard bits” of the alignment problem. Although these look hard, I’m a believer in the capacity of humanity to solve technical problems at this level of difficulty when we put our minds to it. My concern is that I currently don’t think the field is trying to solve this problem. My hope in writing this post is to better point at the problem, with a follow-on hope that this causes new researchers entering the field to attack what seem to me to be the central challenges head-on.

Discussion of a problem
On my model, one of the most central technical challenges of alignment—and one that every viable alignment plan will probably need to grapple with—is the issue that capabilities generalize better than alignment.
My guess for how AI progress goes is that at some point, some team gets an AI that starts generalizing sufficiently well, sufficiently far outside of its training distribution, that it can gain mastery of fields like physics, bioengineering, and psychology, to a high enough degree that it more-or-less singlehandedly threatens the entire world. Probably without needing explicit training for its most skilled feats, any more than humans needed many generations of killing off the least-successful rocket engineers to refine our brains towards rocket-engineering before humanity managed to achieve a moon landing.
And in the same stroke that its capabilities leap forward, its alignment properties are revealed to be shallow, and to fail to generalize. The central analogy here is that optimizing apes for inclusive genetic fitness (IGF) doesn’t make the resulting humans optimize mentally for IGF. Like, sure, the apes are eating because they have a hunger instinct and having sex because it feels good—but it’s not like they could be eating/fornicating due to explicit reasoning about how those activities lead to more IGF. They can’t yet perform the sort of abstract reasoning that would correctly justify those actions in terms of IGF. And then, when they start to generalize well in the way of humans, they predictably don’t suddenly start eating/fornicating because of abstract reasoning about IGF, even though they now could. Instead, they invent condoms, and fight you if you try to remove their enjoyment of good food (telling them to just calculate IGF manually). The alignment properties you lauded before the capabilities started to generalize, predictably fail to generalize with the capabilities.

Some people I say this to respond with arguments like: “Surely, before a smaller team could get an AGI that can master subjects like biotech and engineering well enough to kill all humans, some other, larger entity such as a state actor will have a somewhat worse AI that can handle biotech and engineering somewhat less well, but in a way that prevents any one AGI from running away with the whole future?”
I respond with arguments like, “In the one real example of intelligence being developed we have to look at, continuous application of natural selection in fact found Homo sapiens sapiens, and the capability-gain curves of the ecosystem for various measurables were in fact sharply kinked by this new species (e.g., using machines, we sharply outperform other animals on well-established metrics such as “airspeed”, “altitude”, and “cargo carrying capacity”).”
Their response in turn is generally some variant of “well, natural selection wasn’t optimizing very intelligently” or “maybe humans weren’t all that sharply above evolutionary trends” or “maybe the power that let humans beat the rest of the ecosystem was simply the invention of culture, and nothing embedded in our own already-existing culture can beat us” or suchlike.
Rather than arguing further here, I’ll just say that failing to believe the hard problem exists is one surefire way to avoid tackling it.
So, flatly summarizing my point instead of arguing for it: it looks to me like there will at some point be some sort of “sharp left turn”, as systems start to work really well in domains really far beyond the environments of their training—domains that allow for significant reshaping of the world, in the way that humans reshape the world and chimps don’t. And that’s where (according to me) things start to get crazy. In particular, I think that once AI capabilities start to generalize in this particular way, it’s predictably the case that the alignment of the system will fail to generalize with it.[3]
This is slightly upstream of a couple other challenges I consider quite core and difficult to avoid, including:

Directing a capable AGI towards an objective of your choosing.
Ensuring that the AGI is low-impact, conservative, shutdownable, and otherwise corrigible.

These two problems appear in the strawberry problem, which Eliezer’s been pointing at for quite some time: the problem of getting an AI to place two identical (down to the cellular but not molecular level) strawberries on a plate, and then do nothing else. The demand of cellular-level copying forces the AI to be capable; the fact that we can get it to duplicate a strawberry instead of doing some other thing demonstrates our ability to direct it; the fact that it does nothing else indicates that it’s corrigible (or really well aligned to a delicate human intuitive notion of inaction).
How is the “capabilities generalize further than alignment” problem upstream of these problems? Suppose that the fictional team OpenMind is training up a variety of AI systems, before one of them takes that sharp left turn. Suppose they’ve put the AI in lots of different video-game and simulated environments, and they’ve had good luck training it to pursue an objective that the operators described in English. “I don’t know what those MIRI folks were talking about; these systems are easy to direct; simple training suffices”, they say. At the same time, they apply various training methods, some simple and some clever, to cause the system to allow itself to be removed from various games by certain “operator-designated” characters in those games, in the name of shutdownability. And they use various techniques to prevent it from stripmining in Minecraft, in the name of low-impact. And they train it on a variety of moral dilemmas, and find that it can be trained to give correct answers to moral questions (such as “in thus-and-such a circumstance, should you poison the operator’s opponent?”) just as well as it can be trained to give correct answers to any other sort of question. “Well,” they say, “this alignment thing sure was easy. I guess we lucked out.”
Then, the system takes that sharp left turn,[4][5] and, predictably, the capabilities quickly improve outside of its training distribution, while the alignment falls apart.
The techniques OpenMind used to train it away from the error where it convinces itself that bad situations are unlikely? Those generalize fine. The techniques you used to train it to allow the operators to shut it down? Those fall apart, and the AGI starts wanting to avoid shutdown, including wanting to deceive you if it’s useful to do so.
Why does alignment fail while capabilities generalize, at least by default and in predictable practice? In large part, because good capabilities form something like an attractor well. (That’s one of the reasons to expect intelligent systems to eventually make that sharp left turn if you push them far enough, and it’s why natural selection managed to stumble into general intelligence with no understanding, foresight, or steering.)
Many different training scenarios are teaching your AI the same instrumental lessons, about how to think in accurate and useful ways. Furthermore, those lessons are underwritten by a simple logical structure, much like the simple laws of arithmetic that abstractly underwrite a wide variety of empirical arithmetical facts about what happens when you add four people’s bags of apples together on a table and then divide the contents among two people.
But that attractor well? It’s got a free parameter. And that parameter is what the AGI is optimizing for. And there’s no analogously-strong attractor well pulling the AGI’s objectives towards your preferred objectives.
The hard left turn? That’s your system sliding into the capabilities well. (You don’t need to fall all that far to do impressive stuff; humans are better at an enormous variety of relevant skills than chimps, but they aren’t all that lawful in an absolute sense.)
There’s no analogous alignment well to slide into.
On the contrary, sliding down the capabilities well is liable to break a bunch of your existing alignment properties.[6]
Why? Because things in the capabilities well have instrumental incentives that cut against your alignment patches. Just like how your previous arithmetic errors (such as the pebble sorters on the wrong side of the Great War of 1957) get steamrolled by the development of arithmetic, so too will your attempts to make the AGI low-impact and shutdownable ultimately (by default, and in the absence of technical solutions to core alignment problems) get steamrolled by a system that pits those reflexes / intuitions / much-more-alien-behavioral-patterns against the convergent instrumental incentive to survive the day.
Perhaps this is not convincing; perhaps to convince you we’d need to go deeper into the weeds of the various counterarguments, if you are to be convinced. (Like acknowledging that humans, who can foresee these difficulties and adjust their training procedures accordingly, have a better chance than natural selection did, while then discussing why current proposals do not seem to me to be hopeful.) But hopefully you can at least, in reading this document, develop a basic understanding of my position.
Stating it again, in summary: my position is that capabilities generalize further than alignment (once capabilities start to generalize real well (which is a thing I predict will happen)). And this, by default, ruins your ability to direct the AGI (that has slipped down the capabilities well), and breaks whatever constraints you were hoping would keep it corrigible. And addressing the problem looks like finding some way to either keep your system aligned through that sharp left turn, or render it aligned afterwards.
In an upcoming post, I’ll say more about how it looks to me like  ~nobody is working on this particular hard problem, by briefly reviewing a variety of current alignment research proposals. In short, I think that the field’s current range of approaches nearly all assume this problem away, or direct their attention elsewhere.

^

Furthermore, figuring where to aim it looks to me like more of a technical problem than a moral problem. Attempting to manually specify the nature of goodness is a doomed endeavor, of course, but that’s fine, because we can instead specify processes for figuring out (the coherent extrapolation of) what humans value. Which still looks prohibitively difficult as a goal to give humanity’s first AGI (which I expect to be deployed under significant time pressure), mind you, and I further recommend aiming humanity’s first AGI systems at simple limited goals that end the acute risk period and then cede stewardship of the future to some process that can reliably do the “aim minds towards the right thing” thing. So today’s alignment problems are a few steps removed from tricky moral questions, on my models.

^

While we’re at it: I think trying to get provable safety guarantees about our AGI systems is silly, and I’m pretty happy to follow Eliezer in calling an AGI “safe” if it has a <50% chance of killing >1B people. Also, I think there’s a very large chance of AGI killing us, and I thoroughly disclaim the argument that even if the probability is tiny then we should work on it anyway because the stakes are high.

^

Note that this is consistent with findings like “large language models perform just as well on moral dilemmas as they perform on non-moral ones”; to find this reassuring is to misunderstand the problem. Chimps have an easier time than squirrels following and learning from human cues. Yet this fact doesn’t particularly mean that enhanced chimps are more likely than enhanced squirrels to remove their hunger drives, once they understand inclusive genetic fitness and are able to eat purely for reasons of fitness maximization. Pre-left-turn AIs will get better at various ‘alignment’ metrics, in ways that I expect to build a false sense of security, without addressing the lurking difficulties.

^

“What do you mean ‘it takes a sharp left turn’? Are you talking about recursive self-improvement? I thought you said somewhere else that you don’t think recursive self-improvement is necessarily going to play a central role before the extinction of humanity?” I’m not talking about recursive self-improvement. That’s one way to take a sharp left turn, and it could happen, but note that humans have neither the understanding nor control over their own minds to recursively self-improve, and we outstrip the rest of the animals pretty handily. I’m talking about something more like “intelligence that is general enough to be dangerous”, the sort of thing that humans have and chimps don’t.

^

“Hold on, isn’t this unfalsifiable? Aren’t you saying that you’re going to continue believing that alignment is hard, even as we get evidence that it’s easy?” Well, I contend that “GPT can learn to answer moral questions just as well as it can learn to answer other questions” is not much evidence either way about the difficulty of alignment. I’m not saying we’ll get evidence that I’ll ignore; I’m naming in advance some things that I wouldn’t consider negative evidence (partially in hopes that I can refer back to this post when people crow later and request an update). But, yes, my model does have the inconvenient property that people who are skeptical now, are liable to remain skeptical until it’s too late, because most of the evidence I expect to give us advance warning about the nature of the problem is evidence that we’ve already seen. I assure you that I do not consider this property to be convenient.
As for things that could convince me otherwise: technical understanding of intelligence could undermine my “sharp left turn” model. I could also imagine observing some ephemeral hopefully-I’ll-know-it-when-I-see-it capabilities thresholds, without any sharp left turns, that might update me. (Short of “full superintelligence without a sharp left turn”, which would obviously convince me but comes too late in the game to shift my attention.)

^

To use my overly-detailed evocative example from earlier: Humans aren’t tempted to rewire our own brains so that we stop liking good meals for the sake of good meals, and start eating only insofar as we know we have to eat to reproduce (or, rather, maximize inclusive genetic fitness) (after upgrading the rest of our minds such that that sort of calculation doesn’t drag down the rest of the fitness maximization). The cleverer humans are chomping at the bit to have their beliefs be more accurate, but they’re not chomping at the bit to replace all these mere-shallow-correlates of inclusive genetic fitness with explicit maximization. So too with other minds, at least by default: that which makes them generally intelligent, does not make them motivated by your objectives.

The post A central AI alignment problem: capabilities generalization, and the sharp left turn appeared first on Machine Intelligence Research Institute.