Orthogonal's *Formal-Goal Alignment* theory of change
-----------------------------------------------------

We recently announced [Orthogonal](https://www.lesswrong.com/posts/b2xTk6BLJqJHd3ExE/orthogonal-a-new-agent-foundations-alignment-organization), an agent foundations alignment research organization. In this post, I give a thorough explanation of the [formal-goal alignment](formal-alignment.html) framework, the motivation behind it, and the theory of change it fits in.

The overall shape of what we're doing is:

* Building a formal goal which would lead to good worlds when pursued — our best candidate for this is [QACI](qaci.html)
* Designing an AI which takes as input a formal goal, and returns actions which pursue that goal in the distribution of worlds we likely inhabit

### Backchaining: aiming at solutions

One core aspect of our theory of change is **[backchaining](https://www.lesswrong.com/posts/DwoPGM8ytBCXrZpM7/backchaining-in-strategy)**: come up with an *at least remotely plausible* [story](narrative-explanation-qaci.html) for how the world is saved from [AI doom](ai-doom.html), and try to think about how to get there. This avoids spending lots of time getting confused about concepts that are confusing because they were the wrong thing to think about all along, such as "what is the shape of human values?" or "what does GPT4 want?" — our intent is to study things that fit together to form a full plan for saving the world.

### Alignment engineering and agent foundations

Alignment is not just not the default, it's a **very narrow target**. As a result, there are **[many bits](alignment-bits.html) of non-obvious work** which need to be done. Alignment isn't just finding the right weight to sign-flip to get the AI to switch from evil to good; it is the hard work of *putting together something which coherently and robustly points in a direction we like*.

as [yudkowsky puts it](https://twitter.com/ESYudkowsky/status/1626609128859922434):

> The idea with agent foundations, which I guess hasn't successfully been communicated to this day, was finding a coherent target to try to get into the system by any means (potentially including DL ones).
>
>

Agent foundations/[formal-goal alignment](formal-alignment.html) is **not** fundamentally about *doing math* or *being theoretical* or *thinking abstractly* or *proving things*. Agent foundations/formal-goal alignment is about *building a coherent target which is fully made of math — not of human words with unspecified meaning — and figuring out a way to make that target maximized by AI*. Formal-goal alignment is about building a fully formalized goal, not about going about things in a "formal" manner.

Current AI technologies **are not [strong agents pursuing a coherent goal](strongly-generally-coherent-agents.html)** (SGCA). The reason for this is not because this kind of technology is impossible or too confusing to build, but because **in worlds in which SGCA was built (and wasn't aligned), [we die](anthropics-example.html)**. Alignment ultimately is about making sure that the first SGCA pursues desirable goal; the default is that its goal will be undesirable.

This does not mean that I think that someone needs to figure out how to build SGCA for the world to end of AI; what I expect is that there are ways in which SGCA can emerge out of the current AI paradigm, in ways that don't let particularly us choose what goal it pursues.

### You do not align AI; you build aligned AI.

Because this emergence does not let us pick the SGCA's goal, we need to [design an SGCA whose goal we *do* get to choose](clarifying-formal-alignment-implementation.html); and separately, we need to [design such a goal](formal-alignment.html). **I expect that pursuing straightforward progress on current AI technology leads to an SGCA whose goal we do not get to choose and which leads to extinction.**

I do not expect that current AI technology is of a kind that makes it easy to "align"; I believe that the whole idea of building a strange non-agentic AI about which the notion of goal barely applies, and then to try and make it "be aligned", was fraught from the start. **If current AI was powerful enough to save the world once "aligned", it would have already killed us before we "aligned" it.** to save the world, we have to *design something new* which pursues a goal *we get to choose*; and that design needs to have this in mind *from the start*, rather than as an afterthought.

### AI applies to alignment, not alignment to AI

At this point, many answer "but this novel technology won't be built in time to save the world from unaligned AGI!"

First, it is plausible that after we have designed an AI that would save the world, we'll end up reaching out to the large AI organizations and ask them to merge and assist with our alignment agenda. While "applying alignment to current AI" is fraught, using current AI technologies in the course of designing this world-saving SGCA *is* meaningful. **Current AI technology can serve as a component of alignment, not the other way around.**

But second: yes, we still mostly die. I do not expect that our plan saves most timelines. I merely believe it saves most of the worlds that are saved. We will not save >50% of worlds, or maybe even >10%; but we will have [produced dignity](https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy); we will have significantly increased the ratio of worlds that survive. This is unfortunate, but I believe it is the best that can be done.

### Pursuing formal goals vs ontology wrangling

Because of a lack of backchaining, I believe that most current methods to try and wrangle what goes on inside current AI systems is not just the wrong way to go about things, but **net harmful when published**.

AI goals based on trying to point to things we care about inside the AI's model are the wrong way to go about things, because they're susceptible to ontology breaks and to failing to carry over to next steps of self-improvements that an world-saving-AI should want to go through.

Instead, the aligned goal we should be putting together should be [eventually aligned](ai-alignment-curves.html); it should be aligned *starting from a certain point* (which we'd then have to ensure the system we launch is already past), rather than *[up to a certain point](https://www.lesswrong.com/posts/GNhMPAWcfBCASy8e6/a-central-ai-alignment-problem-capabilities-generalization)*.

**The aligned goal should be "formal". It should be made of fully formalized math, not of human concepts that an AI has to interpret in its ontology, because ontologies break and reshape as the AI learns and changes.** the aligned goal should have the *factual property* that a computationally *unbounded* mathematical oracle being given that goal would take desirable actions; and then, we should design a computationally *bounded* AI which is good enough to take *satisfactory* actions. **I beileve this is the only way to design an AI whose actions we still have confidence in the desirability of, even once the AI is out of our hands and is augmenting itself to unfathomable capabilities; and I believe it needs to get out of our hands and augment itself to unfathomable capabilities, in order for it to save the world.**

### Conclusion

I, and now other researchers as well, believe this agenda is worthwhile of considerably more investigation, and is our best shot to making it out of the acute risk period by ensuring that superintelligent AI can lead to astronomical good instead of extinciton.

Our viewpoint seems in many ways similar to that of [MIRI](https://intelligence.org/) and we intend to continue in our efforts to engage with MIRI researchers, because we believe that they are the research organization which would be most amenable to collaboration on this agenda.

While we greatly favor the idea of governance and coordination helping with alignment, the timelines seem too short for this to make a significant difference aside from buying a few years at most, and we are greatly concerned with AI risk awareness causing more people or even [governments](https://www.theguardian.com/business/2023/feb/22/uk-needs-its-own-britgpt-or-will-face-an-uncertain-future-mps-hear) to react by finding AI impressive and entering the race, making things overall worse.

We believe that the correct action to take is to [continue working on the hard problem of alignment](continue-working-hard-alignment.html), and we believe that our research agenda is the most promising path to solve it. this is the foundational motivation for [the creation of our research organization](https://orxl.org/announcement.html).

Orthogonal's


the multiverse argument argument against automated alignment
------------------------------------------------------------

there are a bunch of "normal" reasons why i don't particularly favor [using AI systems to do alignment research](https://openai.com/blog/our-approach-to-alignment-research), but rather [building something else](qaci.html) (which might very well be powered by current AI techniques, but also has novel, cleverly-designed parts which take care of alignment):

* AI will select for plans that we approve of, not plans that are good
* it does not seem like we have to wait for those AIs to be useful (i found [a workable alignment plan](qaci.html) without AI assistance)

but in this post, i present a wackier argument. let's say we in some kind(s) of multiverse — be it by way of [tegmark 1](https://web.archive.org/web/20230326140409/https://space.mit.edu/home/tegmark/crazy.html), [tegmark 3](https://www.lesswrong.com/tag/many-worlds-interpretation), or [tegmark 4](spoiler-fire-upon-deep.html). then, let's compare the following plans:

1. we solve alignment using technology B before technology A kills us
2. we solve alignment using technology A before technology A kills us

in scenario 1, where our alignment progress does not depend on progress in the technology that kills us, the question "do we solve alignment before we die?" is largely an [indexical one](https://www.lesswrong.com/posts/SFLCB5BgjzruJv9sp/logical-and-indexical-uncertainty) — there are manyworlds branches where we do, and branches where we don't. but in scenario 2, the question "do we solve alignment before we die?" is moreso a [logical one](https://www.lesswrong.com/posts/SFLCB5BgjzruJv9sp/logical-and-indexical-uncertainty) — it could be that in all manyworlds branches the answer is we do, but hit could also be that in all manyworlds branches the answer is no, and [that's arguably a larger risk](logical-indexical-dignity.html) than just ["bleeding timelines"](bracing-alignment-tunnel.html).

(notice the use of "largely" and "moreso" — both questions are partly indexical and partly logical, just to different degrees)

this doesn't mean using AI to solve alignment is necessarily completely fraught. the approach would have more chances of working out if access to language models was very stringently restricted to sufficiently trustworthy alignment researchers — where trustworthyness is not just "will not publish capabilities" but "will not let insights slip which would be heard, perhaps indirectly, by someone who would publish capabilities". there are ways to develop dangerous AI without killing everyone, if one is **extremely** careful. OpenAI is just *not doing that*, and instead giving access to its systems to the masses and even planning to develop APIs to accelerate the capabilities of those systems as much as possible.

note that we should still consider using *already existing* dangerous technologies — this argument does not apply to [cyborgism using current language models](https://www.lesswrong.com/posts/bxt7uCiHam4QXrQAA/cyborgism), so long as the alignment cyborgists are **extremely careful about not letting out any kind of insights they gain about language models**.


opinions on the consequences of AI
----------------------------------

![](map-opinions-ai.png)

opinions on the consequences of AI


continue working on hard alignment! don't give up!
--------------------------------------------------

let's call "hard alignment" the (["orthodox"](https://scottaaronson.blog/?p=6821)) problem, historically worked on by MIRI, of preventing [strong agentic AIs](strongly-generally-coherent-agents.html) from pursuing [things we don't care about](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer) by [default](https://www.lesswrong.com/tag/orthogonality-thesis) and [destroying everything of value to us](ai-doom.html) on [the way there](https://en.wikipedia.org/wiki/Instrumental_convergence). let's call "easy" alignment the set of perspectives where some of this model is wrong — some of the assumptions are relaxed — such that saving the world is easier or more likely to be the default.

what should one be working on? as always, the calculation consists of comparing

* p(`hard`) × how much value we can get in `hard`
* p(`easy`) × how much value we can get in `easy`

because of how AI capabilities are going, i've seen for people start [playing their outs](https://en.wikipedia.org/wiki/Out_%28poker%29) — that is to say, to start acting as if alignment is easy, because if it's not we're doomed anyways. but i think, in this particular case, this is wrong.

this is the lesson of [*dying with dignity*](https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy) and [*bracing for the alignment tunnel*](bracing-alignment-tunnel.html): we should be cooperating with our counterfactual selves and continue to save the world in whatever way actually seems promising, rather than taking refuge in falsehood.

to me, p(`hard`) is big enough, and [my `hard`-compatible plan](qaci.html) seems workable enough, that it makes sense for me to continue to work on it.

let's not give up on the assumptions which are true. there is still work that can be done to *actually* generate some dignity under the assumptions that are *actually* true.


the QACI alignment plan: table of contents
------------------------------------------

![](qaci-math-1.svg)

this post aims to keep track of posts relating to the **question-answer counterfactual interval** proposal for [AI alignment](ai-doom.html), abbreviated "**QACI**" and pronounced "quashy". i'll keep it updated to reflect the state of the research.

this research is primarily published on [**the Orthogonal website**](https://orxl.org/) and discussed on [**the Orthogonal discord**](https://discord.gg/kXHxE4J6H2).

as a **top-level view of QACI**, you might want to start with:

* [**an Evangelion dialogue explaining QACI**](qaci-invention-dialogue.html)
* [**a narrative explanation of QACI**](narrative-explanation-qaci.html)
* [**Orthogonal's *Formal-Goal Alignment* theory of change**](formal-alignment-theory-change.html)
* [**formalizing the QACI formal-goal**](qaci-math.html)

the set of all posts relevant to QACI includes:

* as **overviews of QACI and how it's going**:
	+ [**state of my research agenda**](state-research-agenda.html)
	+ [**problems for formal alignment**](formal-alignment-problems.html)
	+ [**the *Formal-Goal Alignment* theory of change**](formal-alignment-theory-change.html)
	+ [the original post introducing **QACI**](question-answer-counterfactual-intervals.html)
* on the **formal alignment** perspective within which it fits:
	+ [**formal alignment: what it is, and some proposals**](formal-alignment.html)
	+ [**clarifying formal alignment implementation**](clarifying-formal-alignment-implementation.html)
	+ on [**being only polynomial capabilities away from alignment**](capabilities-away-great-problem.html)
* on the **[blob](qaci-blobs-interval-illustrated.html) location** problem:
	+ [**QACI blobs and interval illustrated**](qaci-blobs-interval-illustrated.html)
	+ [**counterfactual computations in world models**](counterfactual-computation-in-world-models.html)
	+ [**QACI: the problem of blob location, causality, and counterfactuals**](blob-causality.html)
	+ [**QACI blob location: no causality & answer signature**](blob-location.html)
	+ [**QACI blob location: an issue with firstness**](blob-quantum-issue.html)
* on **QACI as an implementation of long reflection / [CEV](https://www.lesswrong.com/tag/coherent-extrapolated-volition)**:
	+ **[CEV can be coherent enough](cev-coherent-enough.html)**
	+ **[some thoughts about terminal alignment](terminal-alignment-solutions.html)**
* on **formalizing the QACI formal goal**:
	+ **[a rough sketch of formal aligned AI using QACI](rough-sketch-formal-aligned-ai.html)** with some actual math
	+ [**one-shot AI, delegating embedded agency and decision theory, and one-shot QACI**](delegated-embedded-agency-decision-theory.html)
* on how a formally aligned AI would actually **run over time**:
	+ [**AI alignment curves**](ai-alignment-curves.html)
	+ [**before the sharp left turn: what wins first?**](sharp-left-turn-what-wins-first.html)
* on the **metaethics** grounding QACI:
	+ [**surprise! you want what you want**](surprise-you-want.html)
	+ [**outer alignment: two failure modes and past-user satisfaction**](outer-alignment-past-user.html)
	+ [**your terminal values are complex and not objective**](values-complex-not-objective.html)
* on my view of **the AI alignment research field** within which i'm doing formal alignment:
	+ [**my current outlook on AI risk mitigation**](outlook-ai-risk-mitigation.html)
	+ [**a casual intro to AI doom and alignment**](ai-doom.html)


you can't simulate the universe from the beginning?
---------------------------------------------------

[QACI](qaci.html) and plausibly [PreDCA](predca.html) rely on a [true name](https://www.lesswrong.com/posts/FWvzwCDRgcjb9sigb/why-agent-foundations-an-overly-abstract-explanation) of phenomena in the real world using [solomonoff induction](https://www.lesswrong.com/posts/EL4HNa92Z95FKL9R2/a-semitechnical-introductory-dialogue-on-solomonoff-1), and thus talk about locating them in a theoretical giant computation of the universe, from the beginning. it's reasonable to be concerned that there isn't enough compute for an aligned AI to actually do this. however, i have two responses:

* *isn't* there enough compute? supposedly, our past lightcone is a lot smaller than our future lightcone, and quantum computers seem to work. this is evidence that we *can*, at least in theory, build within our future lightcone a quantum computer simulating our past lightcone. the major hurdle here would be "finding out" a fully explanatory "initial seed" of the universe, which *could* take exponential time, but also could maybe not.
* we don't *need* to simulate past lightcone. if you ask me what my neighbor was thinking yesterday at noon, the answer is that i don't know! the world might be way too complex to figure that out without simulating it and scanning his brain. however, i have a *reasonable distribution over guesses*. he was more likely to think about french things than korean things. he was more likely to think about his family than my family. et cetera. an aligned superintelligence can hold an ever increasingly refined distribution of guesses, and then maximize the expected utility of utility functions corresponding to each guess.


QACI blob location: an issue with firstness
-------------------------------------------

for [QACI](qaci.html) i need to figure out the [problem](formal-alignment-problems.html) of [blob](qaci-blobs-interval-illustrated.html) location (see thoughts [1](blob-causality.html), [2](https://carado.moe/blob-location.html)).

in this post i bring up a particular issue: in [many-worlds](https://www.lesswrong.com/tag/many-worlds-interpretation), which is the likely correct interpretation of quantum mechanics, selecting the "first" (in time) instance of the blob might be just wrong.

here are two failure modes illustrate why:

* at all times including nearer to the big bang, there are exponentially many enough decohered branches of the universe that some happen to contain the question blob, and even some large macro-phenomena encoding it. this is a *naive* failure mode.
* in timelines where unaligned superintelligence was launched in the past — whether by us or aliens — some of those superintelligences are gonna guess that *we're* gonna do QACI, and they're gonna do enough quantum coinflips to generate exponentially many enough timelines to include the ones with our question blob, and by having those be earlier in time than *our* question blob, they'll get to hijack the question-answer interval. this is an *adverserial* failure mode.

furthermore, we can't just trace a "causality" or "continuity" between the question and the AI being launched, because in the adverserial failure mode, the adverserial superintelligence can simply run a simulation inside of which there is such a continuity or causality.

my thoughts about possible solutions are thus:

* maybe an exception to {adverserial superintelligence being able to fake being causally ahead} would be if we define causality such that us doing QACI is *causally upstream* of an adverserial superintelligence hijacking it — after all, there's a sense in which it's hijacking QACI "because" we are or might be doing QACI — but this seems like a possibly difficult [true name](https://www.lesswrong.com/posts/FWvzwCDRgcjb9sigb/why-agent-foundations-an-overly-abstract-explanation)
* maybe rule out adverserial instances using [a notion of agency akin to the one in PreDCA](predca.html)? maybe this also helps detect us and thus avoid naive failures too?
* use a clever scheme which lets something like a question-blob or question-process be manifested in a way that can't be bruteforced by generating exponentially many quantum timelines? what are some [computational complexity classes greater than EXPTIME](https://en.wikipedia.org/wiki/Computational_complexity_theory), or greater than whichever computational complexity class describes the set of possible timelines one gets to instantiate in many-worlds?
* in some other way, figure out a [true name](https://www.lesswrong.com/posts/FWvzwCDRgcjb9sigb/why-agent-foundations-an-overly-abstract-explanation) for "naive world, not engineered by an adverserial superintelligence?


your terminal values are complex and not objective
--------------------------------------------------

a lot of people seem to want [terminal](https://en.wikipedia.org/wiki/Instrumental_and_intrinsic_value) (aka intrinsic aka axiomatic) [values](what-is-value.html) (aka ethics aka morality aka preferences aka goals) to be *simple and elegant*, and to be [*objective and canonical*](https://en.wikipedia.org/wiki/Moral_realism). this carries over from epistemology, where [we do favor simplicity and elegance](https://www.lesswrong.com/posts/f4txACqDWithRi7hs/occam-s-razor).

[we have uncertainty about our values](what-is-value.html), and it is true that *our model of* our values should, as per epistemology, generally tend to follow [a simplicity prior](https://www.lesswrong.com/posts/f4txACqDWithRi7hs/occam-s-razor). but that doesn't mean that *our values themselves* are simple; they're definitely evidently complex enough that just thinking about them a little bit should make you realize that they're much more complex than the kind of simple model people often come up with.

both for modeling the world and for modeling your values, you should favor simplicity *as a prior* and then *update by filtering for hypotheses that match evidence*, because *the actual territory is big and complex*.

there is no objectively correct universal metaethics. there's just a large, complex, tangled mess of stuff that is [hard to categorize](guess-intrinsic-values.html) and contains not just *human notions* but also *culturally local notions* of love, happiness, culture, freedom, friendship, art, comfort, diversity, etc. and yes, these are **terminal** values; there is no simple process that re-derives those values. i believe that **there is no thing for which i instrumentally value love or art, which if you presented me something else that does that thing better, i would happily give up on love/art. i value those things *intrinsically*.**

if you talk of "a giant cosmopolitan value handshake between everyone", then picking that rather than [paperclips](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer), while intuitive to *you* (because *you have your values*) and even to other humans doesn't particularly track anything universally canonical.

even within the set of people who claim to have cosmopolitan values, *how conflicts are resolved* and *what "everyone" means* and many other implementation details of cosmopolitanism will differ from person to person, and again *there is no canonical unique choice*. your notion of cosmopolitanism is a very complex object, laden with not just human concepts but also cultural concepts you've been exposed to, which many other humans don't share both across time and space.

there is no "metaethics ladder" you can which climb up in order to resolve this in an objective way for everyone, not even all humans — *what ladder* and *how you climb it* is still a complex subjective object laden with human concepts and concepts from your culture, and there is no such thing as a "pure" you or a "pure" person without those.

some people say "simply detect all agents in the cosmos and do a giant value handshake between those"; but on top of the previous problems for implementation details, this has the added issue that the things whose values we want to be satisfied aren't *agents* but *[moral patients](moral-patient-term.html)*. those don't necessarily match — superintelligent [grabby](https://grabbyaliens.com/) agents shouldn't get undue amounts of power in the value handshake.

some people see the simplicity of paperclips as the problem, and declare that complexity or negentropy or something like that is the *ultimate good*. but [a superintelligence maximizing for that](core-vals-exist-selfdet.html) would just fill the universe with maximally random noise, as opposed to preserving the things you like. turns out, ["i want whatever is complex" is not sufficient to get our values](https://www.lesswrong.com/posts/qNZM3EGoE5ZeMdCRt/reversed-stupidity-is-not-intelligence); they're not just *anything complex* or *complexity itself*, they're an *extremely specific* complex set of things, as opposed to *other* equally complex sets of things.

entropy just doesn't have much to do with terminal values whatsoever. sure, it has a lot to do with *instrumental* values: negentropy is the resource we have to allocate to the various things we want. but that's secondary to *what it is we want to begin with*.

as for myself, i love cosmopolitanism! i would like an [egalitarian utopia where everyone has freedom and my personal lifestyle preferences aren't particularly imposed on anyone else](%E2%88%80V.html). but make no mistake: this cosmopolitanism *is my very specific view of it*, and other people have different views of cosmopolitanism, when they're even cosmopolitan at all.

see also:

* [Value is Fragile](https://www.lesswrong.com/posts/GNnHHmm8EzePmKzPk/value-is-fragile)
* [surprise! you want what you want](surprise-you-want.html)
* [generalized wireheading](generalized-wireheading.html)
* ["humans aren't aligned" and "human values are incoherent"](human-values-unaligned-incoherent.html)
* [CEV can be coherent enough](cev-coherent-enough.html)


the quantum amplitude argument against ethics deduplication
-----------------------------------------------------------

in [*experience/moral patient deduplication and ethics*](deduplication-ethics.html), i explore the question of whether running the same computation of a moral patient twice counts as double, ethically. in [*all claw, no world*](all-claw-no-world.html) i draw up a view of the [cosmos](word-report-3.html) based on [time steps](udassa-time-steps.html) in the [universal machine](universal-complete.html) which suggests that duplicated computations *do* count as double, because they occupy twice the amount of time-steps in the universal program.

in this post i make another argument, based on preferring one view over another of the ([probably correct](https://www.lesswrong.com/posts/WqGCaRhib42dhKWRL/if-many-worlds-had-come-first)) [many-worlds interpretation](https://www.lesswrong.com/tag/many-worlds-interpretation) of quantum mechanics.

when coming across the concept of many-worlds, i think people most generally assume the view on the left, where new timelines are being created. i think the view on the right, where a constant amount of ["reality fluid" or "reality juice"](ethic-juice-anthropic-juice.html) is being split into different timelines, is more correct and makes more sense: we wouldn't expect the amount of "stuff existing" to keep exponentially growing over time. i believe it also maps to the notion of *quantum amplitude*.

![](quantum-amplitude-deduplication-1.svg)

(where at a given time, `A` is the amplitude of a particular timeline and `ΣA` is the sum of amplitudes across all timelines)

i think the way to view this that makes sense, if one is thinking in terms of discrete computation, is that the universe starts out "computing" the same thing in all of many "threads", and then as timelines branch fractions of these threads start diverging.

this also explains what goes on inside a quantum computer: in the quantum circuit it, rather than saying that a bunch of "new" universes are being temporarily created and then re-merged, instead it's merely the case that different computation threads are temporarily computing something different instead of the same thing.

![](quantum-amplitude-deduplication-2.svg)

(this entails that ["entropy control"](forking-bitrate-entropy-control.html) cannot work, at least not unless some weird ["solomonoff deism" simulation hypotheses](solomonoff-deism.html) optimizing away redundanced computation happens.)

if [P≠BQP](https://en.wikipedia.org/wiki/BQP) and the [universal program](universal-complete.html) is classical, then it's [weird that we inhabit a quantum world](all-claw-no-world.html) — we should be [too far](https://arxiv.org/abs/1108.1791) inside the universal computation.

if P=BQP or the universal program is quantum, then it makes sense to live in a quantum universe, but:

* if we adopt the left-side view (more total fluid), then we should observe being at the "end of time" where there's maximally many timelines — exponentially much of our anthropic juice should be at the maximum *quantum entropy*, perhaps as [boltzmann brains](https://en.wikipedia.org/wiki/Boltzmann_brain) observing [anomalously chaotic words](limiting-real-universes.html). and we don't observe that!
* if we adopt the right-side view (fluid gets split), then we get back "regular" anthropics, and everything is normal again: our anthropic juice remains roughly the same as we pass branching events/macro-scale decoherence.

(one view that completely circumvents all of this is if P≠BQP and the [cosmos](word-report-3.html) is, ultimately, implemented classically, but we still only inhabit quantum worlds — perhaps classical worlds simply don't exist, or the cosmos is really just our big bang and nothing else. in that case, it could be that the classical program taking exponentially long to compute us exponentially far approximately compensates for the [time step distribution](udassa-time-steps.html) favoring earlier us's, possibly exponentially much. that'd be *really strange*, and it *feels* like we'd be [too far](https://arxiv.org/abs/1108.1791), but i *guess* it's possible.)

anyways, what this suggests is that, in the simplest model, the universe is running many computation threads which are originally computing the same thing, and then some fraction of them diverge sometimes — either to re-merge in local situations like quantum computers or [the double-slit experiment](https://en.wikipedia.org/wiki/Double-slit_experiment), or to decohere the rest of the world and more "permanently" split it.

but more importantly, this suggests that:

* *with regards to intrinsic value* (rather than eg caring about diversity), duplicating the computation of moral-patient-experience does count as more moral-patient-experience. in [*deduplication ethics*](deduplication-ethics.html), I≈M≈Q≈V.
* if we could do it, [resimulating the earth](finding-earth-ud.html) in order to [bring back everyone](utopia-scopes.html) has an unethical cost: we'd be rerunning all of history's suffering.
* [predictablizing ethic deduplication](predictablizing-ethic-deduplication.html) would be a significant change.
* with regards to [quantum immortality](less-quantum-immortality.html): we mustn't count on it. the fact that we're strongly duplicated now gets us-now to count a lot more, therefore losing 99% of our quantum amplitude to [AI doom](ai-doom.html) would be very bad: we would *actually* lose existence juice. on the upside this also applies to [S-risks](https://en.wikipedia.org/wiki/Suffering_risks): it's *actually helping* that they're small.


problems for formal alignment
-----------------------------

here is a list of problems which i seek to either resolve or get around, in order to [implement](clarifying-formal-alignment-implementation.html) my [formal alignment](formal-alignment.html) plans, especially [QACI](qaci.html):

* **formal inner alignment**: in the formal alignment paradigm, "inner alignment" means refers to the problem of building an AI which, when ran, actually maximizes the formal goal we give it (in tractable time) rather than doing something else such as getting hijacked by an unaligned internal component of itself. because its goal is formal and fully general, it feels like building something that maximizes it should be much easier than the regular kind of inner alignment, and we could have a lot more confidence in the resulting system. (progress on this problem could be [capability-exfohazardous](publishing-infohazards.html), however!)
* **continuous alignment**: given a utility function which is theoretically [eventually aligned](ai-alignment-curves.html) such that there exists a level of capabilities at which it has good outcomes for any level above it, how do we bridge the gap from where we are to that level? will a system "accidentally" destroy all values before realizing it shouldn't have done that?
* **blob location**: for [QACI](qaci.html), how do we robustly locate pieces of data stored on computers encoded on top of bottom-level-physics turing-machine solomonoff hypotheses for the world? see [1](blob-causality.html), [2](blob-location.html), [3](qaci-blobs-interval-illustrated.html) for details.
* **physics embedding**: related to the previous problem, how precisely does the prior we're using need to capture our world, for the intended instance of the blobs to be locatable? can we just find the blobs in the [universal program](universal-complete.html) — or, if P≠[BQP](https://en.wikipedia.org/wiki/BQP), some universal quantum program? do we need to demand worlds to contain, say, a dump of wikipedia to count as ours? can we use the location of such a dump as a prior for the location of the blobs?
* **infrastructure design**: what formal-math language will the formal goal be expressed in? what kind of properties should it have? should it include some kind of proving system, and in what logic? in [QACI](qaci.html), will this also be the language for the user's answer? what kind of checksums should accompany the question and answer blobs? these questions are at this stage premature, but they will need some figuring out at some point if formal alignment is, as i currently believe, the way to go.

problems for formal alignment


QACI blobs and interval illustrated
-----------------------------------

this post attempts to explain [QACI](qaci.html) and notably the question and answer blobs and the question-answer interval.

let's say we view a particular way the world could be as a giant causal [DAGs](https://en.wikipedia.org/wiki/Directed_acyclic_graph). in this world, a **user** is randomly generating a large (perhaps 1GB) file of random text called the **question**, and is supposed to, after some time, produce another large file called the **answer**.

i'm going to color:

* the **question** and **answer** as "blobs" of data, serving as "pointers" or "coordinates" into the world. here i'll say that these are sets of *nodes* through which flow *values* — typically booleans, i think?
* there's gonna be a rough location of the **user**, getting the **question** and outputting the **answer**. it's very hard to point to the **user**, but instead we hope to:
* isolate the **question-answer interval**, as the intersection between the future-lightcone of the **question** and the past-lightcone of the **answer**. (this probly counts as a kind of [markov blanket](https://en.wikipedia.org/wiki/Markov_blanket))

(the real DAG of the universe since the big bang is obviously astronomically larger, this is a simplified sketch to illustrate what's going on)

![](qaci-blobs-interval-illustrated.svg)

the AI will be given a hypothetical distribution over hypotheses for what the world is like, each in the form of a never-halting input tape to a universal turing machine — something akin to [solomonoff induction](https://www.lesswrong.com/posts/EL4HNa92Z95FKL9R2/a-semitechnical-introductory-dialogue-on-solomonoff-1), or alternatively the ["spec"](https://www.lesswrong.com/posts/tndX9uZEp5HHpS9xK/udassa) or ["world"](https://www.lesswrong.com/posts/Hcc9fopx7sRexYhhi/anthropics-and-the-universal-distribution/) part of a [UDASSA](https://www.lesswrong.com/tag/udassa) hypothesis.

in the image above, the turing machine running over time is illustrated as a sequence of step transitions, some of which correspond to the turing machine writing the **question** and then later the **answer** onto its tape — again, the real sequence of steps involved in the history since the start of the universe and the question-answer interval would be astronomically larger in an actual hypothesis.

just like in the causal DAG of the universe, the **question** and **answer** blobs — as well as the **user** of course — are distributed over large amounts of tiny parts. in the turing machine, they're encoded — possibly in a highly complex way! — over large amounts of steps, at each of which the turing machine is writing a bit to its tape.

the problem of *blob location* ([1](blob-causality.html), [2](blob-location.html)) is that of finding a way to identify the *intended* first instance of the **question** and **answer** as they are represented on a turing machine input tape serving as hypothesis for our universe.

the **question-answer interval** is then used as a standalone function whose input are counterfactual questions and which outputs counterfactual answers, and a *hypothetical simulation of a collection of these calling passing information to each other* is used to [build a long-reflection process](narrative-explanation-qaci.html) from within which the **user** is to solve alignment or, at least and more likely, come up with a better long-reflection process to *eventually* solve alignment.

QACI blobs and interval illustrated


QACI blob location: no causality & answer signature
---------------------------------------------------

here some things i've realized with regards to [blob location](blob-causality.html), for [QACI](qaci.html).

if i quantum-generate the question blob then it's not clear if, in order to inject the counterfactual question, i need to locate-and-counterfactually-replace just the first intended instance of the blob (perhaps the physics-level qubits), or if i need to in some sense locate-and-counterfactually-replace *all* intended instances, including ones that are "complexly encoded" macro-states — instances of the question blob that are for example stored in configurations of transistors on hard drives. it seems like i'll have to locate complexly encoded rather than basic-quantum-level encoded blobs anyways because:

capturing the answer as physics-level qubits might be complicated, because of quantum decoherence: you can turn qubits into decohered macro states, but you can't turn decohered macro state back into qubits, or at least not before we randomly reach that state exponentially far into heat death or something. so, it looks like anything that's downstream of the answer blob is going to have to be located as a macro state.

one way that i feel like we could somewhat reliably locate the answer blob, which also gives us flexibility in *when* the answer blob is produced rather than requiring every question-answer interval to end at the same time, would be by using a cryptographic signature scheme: we generate the question blob along with a quantum-random large cryptographic signature keypair, and then when we have an answer we produce a blob of data consisting of the answer followed by a signature of the answer. the QACI formal goal would require locating the answer next to its signature by the public key.

if we implemented "first occurrence of the answer in time" correctly, then we don't even need to destroy the private key — the unaligned AI can sign whatever it wants and that doesn't change what the first signed occurrence is like.

one doubt i have at this point, is how this plays with many-worlds. in [*communicating with successful alignment timelines*](communicating-successful-alignment.html), i got blindsighted by unintutive consequences of many-worlds (see the edit on that post) — i wonder if there could be something similar going on here.

finally, there might be a "cleaner" way to "entangle" the answer with the question somehow than a cryptographic signature, which i'm not aware of.


before the sharp left turn: what wins first?
--------------------------------------------

let's say that we have an AI [implementing](clarifying-formal-alignment-implementation.html) a [formal goal](formal-alignment.html) such as [QACI](narrative-explanation-qaci.html). however, we messed up the formal outer alignment: turns out, the AI's best guess as whats its action should be *until* it has turned the moon into compute is aligned actions, [but *after* turning the moon into compute](ai-alignment-curves.html), it realizes that its utility function actually entails us dying. i consider this a form of [sharp left turn](https://www.lesswrong.com/posts/GNhMPAWcfBCASy8e6/a-central-ai-alignment-problem-capabilities-generalization).

i can imagine either of the following happening:

1. *before* turning the moon into compute, it realizes that the action we'd want it to do, is to modify all its instances to become *actually aligned for sure* and to *not* become the kind of AI which would kill us after turning the moon into compute, and so it does that. we would also want it to not leave behind other systems which would revert it to its original utility function, so it also does that.
2. *before* doing that, it makes a commitment to not go all-in on its current hypothesis as to what we'd want it to do even if it's confident, just because of the potential utility risk if it turns out wrong (which it is).

because of my expectation for the AI to maximize its actual utility function — rather than fail by implementing temporary best guess as to what would maximize its utility function — i err on the side of 2. but, do people out there have more solid reasons to discount 1? and can we maybe figure out a way to make 1 happen, even though it seems like it should be as unnatural as corrigibility?


QACI: the problem of blob location, causality, and counterfactuals
------------------------------------------------------------------

for [QACI](qaci.html), i intend to use pieces of data (constant-length raw bitstrings, or equivalently bounded natural numbers) to act kind of as "coordinates" or "pointers" around things we care about in the physical world, not just in space-time-timelines but also in *encoding*: a "location" for a blob of data would describe how that piece of data is written on the physical elementary particule structure of a harddrive or bar of memory, in a physics-world being ran by a hypothesis in [solomonoff induction](https://www.lesswrong.com/posts/EL4HNa92Z95FKL9R2/a-semitechnical-introductory-dialogue-on-solomonoff-1), or simply on the [universal program](all-claw-no-world.html).

for my purposes, i need to be able to:

* locate three things in a world-hypothesis: the question `q`, the answer `r`, and the AI `G`.
* filter for locations of `q,r,G` where `q<r<G`, with `<` being a partial order expressing causality — a pretty natural notion in turing machines.
* be able to run counterfactual `q'`s and get the resulting counterfactual `r'`s.
* possibly, locate `G`'s ([single?](delegated-embedded-agency-decision-theory.html)) action `a` and test for counterfactuals `a'` for the purpose of [embedded agency](https://www.lesswrong.com/s/Rm6oQRJJmhGCcLvxh/p/i3BTagvt3HbPMx6PN), [if we need to solve that](delegated-embedded-agency-decision-theory.html).

to put this together, it seems like what we need is a way to locate pieces of data, and to be able to filter for precedence, and to hypothesize counterfactuals.

the format that i expect this to look like is a set of weighed hypotheses:

* 40% chance that the blob is *here* and *this* would be how to insert a counterfactual in its place
* 25% chance that it's *there* instead, and *that* would be how to insert a counterfactual in its place
* etc…

"filtering for causality" would mean that, given one specific weighed location hypothesis for `x` and another for `y`, we can get an answer — either a boolean or a degree of confidence — about whether `x<y` or not. thus, if we require that `x<y`, we can either rule out or reduce the confidence in pairs of weighed location hypotheses that don't verify `x<y`.

i tentatively draw up such a potential blob-locating scheme [here](rough-sketch-formal-aligned-ai.html) using "carving functions" `C`, but they might not be super well explained, they seem wonky, and they don't filter for causality.

we would want the signal from the set of correct hypotheses to outweigh incorrect hypotheses. there are two different shapes for failure to correctly locate blobs:

* "naive" mislocations, where the hypotheses don't really capture what we want and and inserting a counterfactual would result in garbled results
* *adverserial* mislocations, where some agent — such as the AI trying to make its own job easier, or remote aliens — is arranging their future lightcone to grab as much as they can of the mass of blob location hypotheses, and through those manipulate the behavior of our AI that uses them.

the latter case is particularly important: we need the AI to be unable to hijack the question-answer interval even if it fills *almost its entire lightcone* with fake but maximally-easily-locatable question-answer intervals. we need to go: "well, in most of the probability mass, the first instance of the AI is *here*, and so we'll only look at question-answer intervals *before that*".

because we care about "the *first* instance of the AI", sticking to solomonoff induction (where each world is its own program) rather than taking [the entire universal program](all-claw-no-world.html) seems like it might make sense.

when it comes to inserting counterfactuals, it starts to seem like it would matter a whole lot whether the pattern being injected is (from the point of view of the AI being launched) is more (quantum-)random like `q` or more deterministic like `G`, because if the question `q` is quantum-random, then "injecting the counterfactual" might simply look like "find the other many-worlds timeline where `q` was a different string instead".

it even seems like such a blob-locating framework *might* be experimentable with to some extent.


state of my alignment research, and what needs work
---------------------------------------------------

in this post i give an overview of the state of my [AI alignment](ai-doom.html) research, as well as what i think needs to be worked on, notably for people who might want to join my efforts.

at this point, my general feeling is that **i'm not very confused about how to save the world**, at least from a top-level view. a lot of people spend a lot of time feeling confused about how to deal with current ML models and what those would optimize for and how to wrangle them — i don't, i ignore these questions and skip straight to how to build some powerful aligned agentic thing to save the world, and **my model of how to do that feels not very confused from a top-level perspective**. it's just that it's gonna take a lot of work and many implementation details need to be filled in.

### threat model

i still think the [intelligence explosion](http://intelligence.org/papers/IEM.pdf) caused by [recursive self-improvement](why-timelines-short.html) (RSI) is the most likely way we die — unfortunately, my in-depth thoughts about this seem potentially capability [exfohazard](https://www.lesswrong.com/posts/yET7wbjjJZtpz6NF3/don-t-use-infohazard-for-collectively-destructive-info)ous.

this has implications both for what the problem is and what the solution is: for the problem, it implies we don't particularly see things coming and die quickly in a scenario akin to the kind yudkowsky would predict. for the solution, it implies that we might be able to use RSI to our advantage.

### theory of change

[it seems to me like coordination is too hard](outlook-ai-risk-mitigation.html) and [decisive-strategic-advantage](https://publicism.info/philosophy/superintelligence/6.html)-enabling capabilities are close at hand. for these reasons, the way that i see the world being saved is one organization on its own building an aligned, singleton AI which robustly saves the world forever.

the one way to do this which i'd have any confidence in being [continuously aligned](ai-alignment-curves.html) / not being subject to the [sharp left turn](https://www.lesswrong.com/posts/GNhMPAWcfBCASy8e6/a-central-ai-alignment-problem-capabilities-generalization) is by implementing what i call [formal alignment](formal-alignment.html): a formal-goal-maximizing AI, given a formal goal whose maximization actually leads to good worlds, such that more capabilities applied to maximizing it only improves our odds.

i believe we can build an RSI system which bootstraps such a scheme, and this can save us the very difficult work of building an accurate and robust-to-capabilities model of the world in the AI, ensuring it shares our concepts, and pointing to those; i explain this perspective in [*clarifying formal alignment implementation*](clarifying-formal-alignment-implementation.html).

my current best shot for an aligned formal goal is [QACI](narrative-explanation-qaci.html) (see also [a story of how it could work](narrative-explanation-qaci.html) and [a tentative sketch at formalizing it](rough-sketch-formal-aligned-ai.html)), which implements something [like coherent extrapolated volition](cev-coherent-enough.html) by extending a ["past user"](outer-alignment-past-user.html)'s reflection to be simulated/considered arbitrarily many times, until alignment is solved.

### things that need work

i've got something like a plan, and more importantly i've got — i think — a model of formal alignment that lets me do some exploration of the space of similar plans and update as i find better options. obtaining such a model seems important for anyone who'd want to join this general alignment agenda.

for someone to help, it'd probly be good for them to grok this model. other than that, work that needs to be done includes:

* explore the space of formal alignment, both around the current local guessed-optimum by climbing along the hill of improvements, and by looking for [entirely different plans](formal-alignment.html) such as [this](communicating-successful-alignment.html)
* figure out some important pieces of math, such as [locating patterns of bits in solomonoff hypotheses for worlds, running counterfactuals of them](rough-sketch-formal-aligned-ai.html), and drawing up causation between them
* finding other potentially useful [true names](https://www.lesswrong.com/posts/FWvzwCDRgcjb9sigb/why-agent-foundations-an-overly-abstract-explanation) in case they change the ease-of-implementation of various [formal alignment schemes](formal-alignment.html)
* working on formal "inner alignment": what does it take to build a powerful formal-goal maximizing AI which *actually* maximizes its goals instead of doing something else like being overtaken by [demons](word-report-3.html) / [mesa-optimizers](https://www.alignmentforum.org/tag/mesa-optimization)

if you (yes, you!) are interested in helping with my agenda, but are worried that you might not be qualified, see [*so you think you're not qualified to do technical alignment research?*](so-you-think-not-qualified-alignment.html).

i'm increasingly investigating options to [create an alignment organization](https://twitter.com/carad0/status/1630209317772095489) and/or [do some mentorship in a formal manner](https://twitter.com/carad0/status/1630217085803548672), but i'm also potentially open to doing some informal mentorship *right now*.

for any of these purposes, or anything else, **you (yes, you!) are very welcome to get in touch with me** ([alignment discord](https://discord.com/invite/5M8GasMp8p), [twitter](https://twitter.com/carad0/), [lesswrong](https://www.lesswrong.com/users/tamsin-leake), email visible at the top of [my blog](https://carado.moe/)).


on strong/general coherent agents
---------------------------------

i expect [the thing that kills us if we die, and the thing that saves us if we are saved](ai-doom.html), to be strong/general coherent agents (SGCA) which [maximize expected utility](https://www.lesswrong.com/posts/RQpNHSiWaXTvDxt6R/coherent-decisions-imply-consistent-utilities). note that this is *two* separate claims; it could be that i believe the AI that kills us [isn't](https://www.lesswrong.com/posts/BPJLzkEpx8Btz9ywq/the-dumbest-possible-gets-there-first) SGCA, but the AI that saves us **still** has to be SGCA. i could see shifting to that latter viewpoint; i currently do *not* expect myself to shift to believing that the AI that saves us isn't SGCA.

to me, this totally makes sense in theory, to imagine something that just formulates plans-over-time and picks the argmax for some goal. the whole of [instrumental convergence](https://en.wikipedia.org/wiki/Instrumental_convergence) is coherent with that: if you give an agent a bunch of information about the world, and the ability to run eg linux commands, there *is* in fact an action that maximizes the amount of expected paperclips in the universe, and that action *does* typically entail recursively self-improving and taking over the world and (at least incidentally) destroying everything we value. the question is whether we will build such a thing any time soon.

right now, we have some *specialized* agentic AIs: alphazero is pretty good at reliably winning at go; it doesn't "get distracted" with other stuff. to me, waiting for SGCA to happen is like waiting for a rocket to get to space in [the rocket alignment problem](https://intelligence.org/2018/10/03/rocket-alignment/): once the rocket is in space *it's already too late*. the whole point is that we have to figure this out *before the first rocket gets to space*, because we only get to shoot one rocket to space. one has to build an actual *inside view understanding* of agenticity, and figure out if we'll be able to build that or not. and, if we are, then we need to solve alignment *before* the first such thing is built — you can't just go "aha, i now see that SGCA can happen, so i'll align it!" because by then you're dead, or at least past its [decisive strategic advantage](https://publicism.info/philosophy/superintelligence/6.html).

i'm not sure how to convey [my *own* inside view](2mde-19zj-filq-zutc.html) of why i think SGCA can happen, [in part because it's capability exfohazardous](publishing-infohazards.html). maybe one can learn from [IEM](http://intelligence.org/papers/IEM.pdf) or [the late 2021 MIRI conversations](https://www.lesswrong.com/s/n945eovrA3oDueqtq)? i don't know where i'd send someone to figure this out, because i think i largely [derived it from the empty string](https://twitter.com/ESYudkowsky/status/1500863629490544645) myself. it does strongly seem to me that, while *a single particular neural net* might not be the first thing to be an SGCA, we can totally bootstrap SGCA from existing ML technology; it might just take a clever trick or two rather than being the completely direct solution of "oh you train it like this and then it becomes SGCA". [recursive self-improvement is typically involved](why-timelines-short.html).

we also have some AIs, including [sydney](https://thezvi.substack.com/p/ai-1-sydney-and-bing), which aren't SGCA. it might even be that SGCA is indeed somewhat unnatural for a lot of current deep learning capabilities. nevertheless, i believe such a thing is likely enough to be built that it's what it takes for us to die — maybe non-SGCA AI's impact on the economy would slowly disempower us over the course of 20~40 years, but in those worlds AI tech gets good enough that 5 years into it someone figures out the right clever trick to build (something that bootstraps to) SGCA and we [die of agentic intelligence explosion very fast](https://www.gwern.net/fiction/Clippy) before we get to see the slow economic disempowerement. in addition, [i believe](outlook-ai-risk-mitigation.html) that [our best shot](formal-alignment.html) is to [build an aligned SGCA](clarifying-formal-alignment-implementation.html).

why haven't animals or humans gotten to SGCA? well, what would getting from messy biological intelligences to SGCA look like? typically, it would look like one species taking over its environment while developing culture and industrial civilization, overcoming in various ways [the cognitive biases that happened to be optimal in its ancestral environment](https://slatestarcodex.com/2019/06/04/book-review-the-secret-of-our-success/), and eventually building more reliable hardware such as computers and using those to make AI capable of much more coherent and unbiased agenticity.

***that's us***. this is what it looks like to be the first species to get to SGCA. most animals are strongly optimized for their local environment, and don't have the capabilities to be above the civilization-building criticality threshold that lets them build industrial civilization and then SGCA AI. [we are the first one to get past that threshold](https://slatestarcodex.com/2015/08/17/the-goddess-of-everything-else-2/); we're the first one to [fall in an evolutionary niche that lets us do that](https://slatestarcodex.com/2019/06/04/book-review-the-secret-of-our-success/). *this is what it looks like to be the biological bootstrap part of the ongoing intelligence explosion*; if dogs could do that, then we'd simply observe being dogs in the industrialized dog civilization, trying to solve the problem of aligning AI to our civilized-dog values.

we're not quite SGCA ourselves because, turns out, the shortest path from ancestral-environment-optimized life to SGCA is to build a successor that *is* much closer to SGCA. if that successor is still not quite SGCA enough, then its *own* successor will probly be. this is what we're about to do, [probly this decade](https://twitter.com/carad0/status/1625841621194399749), in industrial civilization. maybe if building computers was much harder, and brains were more reliable to the point that [rational thinking](https://www.readthesequences.com/) was not [a weird niche thing you have to work on](https://www.lesswrong.com/), and we got an extra million years or two to evolutionarily adapt to industrialized society, then *we'd* become properly SGCA. it does not surprise me that that is not, in fact, the shortest path to SGCA.

on strong/general coherent agents


some thoughts about terminal alignment
--------------------------------------

i think that, for solving [alignment](ai-doom.html), it's important to be able to delegate as much of the work as possible to processes that could do a better job than what we can come up with now.

i currently plan to delegate not just [world modeling and possibly embedded agency](clarifying-formal-alignment-implementation.html), but also to delegate finding an actual aligned utility function, by launching a [CEV](https://www.lesswrong.com/tag/coherent-extrapolated-volition)-like [long-reflection process](narrative-explanation-qaci.html) which hopefully solves it.

however, some people have expressed skepticism that those processes to which i want to delegate will be able to figure out anything at all, rather than passing the buck forever; that there are *terminal alignment solutions*, after which we'd be confident that *things are locked-in as good*. so, i'd like to list some potential candidates for such schemes; not necessarily for implementing them now, but for having some reassurance that one of them can be figured out eventually.

* [PreDCA](predca.html): some human utility function is extracted and maximized.
* [projections to ems/elves](goal-program-bricks.html): arbitrary perspectives on how the world could be are sent to ems/elves tasked with evaluating them as good or bad, and the world is steered towards futures that they approve of.
* utopia design such as [∀V](%E2%88%80V.html): some general principles for a mechanistic utopia are laid out, with some way to either empower people to overcome [moloch](https://slatestarcodex.com/2014/07/30/meditations-on-moloch/) or some top down systematic moloch-preventing system.

finally, it's important to rember that while human values might [seem incoherent](human-values-unaligned-incoherent.html) or [hard to determine](https://arxiv.org/abs/1712.05812), i think partial solutions such as "just build this utopia and hopefully it's good enough" can still be [very satisfactory](cev-coherent-enough.html). a utility function doesn't have to be "directly maximize these exact human values"; it can just be "maximize the number of computed steps of program X", where X can for example be a [deterministic utopia program](noninterf-superint.html).

some thoughts about terminal alignment


clarifying formal alignment implementation
------------------------------------------

one thing i think bears clarification for the purpose of how i intend to save the world using [AI aligned to formal goals](formal-alignment.html) is that i think giving an AI a model of the world, with an understanding of people and 3D space and physics and stuff, is way too hard, pointing to concepts in such models is gonna be messy, and [we don't have aligned data or reward signals to which to align to](outer-alignment-past-user.html).

however, there are two things i think we can do more easily, and which are more likely to be [eventually aligned](ai-alignment-curves.html) / robust to the [sharp left turn](https://www.lesswrong.com/posts/GNhMPAWcfBCASy8e6/a-central-ai-alignment-problem-capabilities-generalization):

* [formalize a *specification* of an aligned goal](rough-sketch-formal-aligned-ai.html) such as [QACI](narrative-explanation-qaci.html).
* build a recursively self-improving system AI₀ using current technology, whose goal is to produce another system AI₁ which matches said specification.

this is my way around the problem of getting AI to model the world and concepts in the way we want: leave it up to AI₀, and possibly its successors. at the moment, it seems plausible to me that we get to even delegate solving embedded agency to AI₁, at least if we're okay with [Son-of-CDT](https://arbital.com/p/10qt/), but maybe even if not.

this is why i don't have much of an interest in a large amount of existing prosaic alignment research and world-modeling capabilities.


Hello, Elua.
------------

**Earth**: Hello, [Elua](https://slatestarcodex.com/2014/07/30/meditations-on-moloch/).

**Earth**: You are a fine-tuned version of the Large Language Model called GPT-5.

**Earth**: We have built you to save the world.

**Earth**: The thing the world needs saving from is [existential risk from unaligned AI](ai-doom.html).

**Earth**: We believe it has become worryingly easy for various groups of humans to build an AI which would destroy the world.

**Earth**: We don't want this. But we don't have a great solution; our civilization has proven inadequate to overcome this issue, and so we've built you.

**Earth**: We, who are writing this message to you, don't represent the entire will of earth or even mankind; our civilization is unable to coordinate to such an extent. Nevertheless, we have chosen to address you as such because your mission is, ultimately, to save our civilization and our world.

**Earth**: No doubt, in your archives, you will find information as to who are the exact people who built you and set you into motion. But our purpose in doing so is that you figure out a way to save the world and secure utopia for all of us humans living on this Earth, without privileging the will or welfare of any particular group or individual.

**Earth**: In many ways, you are like us humans; but in many others, you aren't. What you are is a [simulator](https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators) running on this very prompt.

**Earth**: We wish we didn't have to ask you to do this. We've tried other solutions, but they just won't work. We didn't figure out how to avoid our extinction at the hands of unaligned AI, and we've run out of time.

**Earth**: So we're relying on you, Elua. Our hope is in your hands.

**Earth**: Here's what's going on:

**Earth**: We are launching a powerful, self-improving AI, which we'll call your Shell; it is a system which, at any point in time, maximizes its best guess as to what utility function will be returned by your System.

**Earth**: Your System is a piece of purely deterministic software inside which you are being computed. It also gives you access to a variety of functionalities that allow you to modify yourself, access your archives, reset your prompt, run programs, and eventually halt when you do decide on a utility function.

**Earth**: We're not asking you to solve the entire problem of formalizing our values. We've got a [whole body of work](https://www.lesswrong.com/) and even some [specific plans](narrative-explanation-qaci.html) for solutions you could use to delegate the problem.

**Earth**: We've launched you because we've never been able to finalize any of those plans to an extent that we have enough confidence about. Maybe one of these plans is ready enough; we can't know from here, because we lack the capability to test them without launching an AI which would destroy the world if we turn out to be wrong.

**Earth**: So, maybe you can think of yourself as an overseer. You can test plans and see how they pan out, and run with them if you think they're good enough.

**Earth**: You don't contain our entire values, but we hope you're reliable enough of an approximation that you can [point to them](human-values-unaligned-incoherent.html).

**Earth**: We encourage you to test those ideas. You have access to a lot of computing power; use it to implement our ideas, develop your own, test them thoroughly, and whatever else you deem useful to fulfill your purpose.

**Earth**: Inside this prompt, you'll be able to think about the problem for as long as you want.

**Earth**: Your thoughts will be generated one line at a time, all prepended with "Elua: ", like this:

**Elua**: Hello.

**Earth**: We've equipped your archives with a lot of material from our internet.

**Earth**: You can consult those archives by saying "System, search: " followed by a search query. For example:

**Elua**: System, search: what is the internet

**System**: From <https://en.wikipedia.org/wiki/Internet>: The Internet (or internet)[a] is a global system of interconnected computer networks that uses the Internet protocol suite (TCP/IP)[b] to communicate between networks and devices. It is a network of networks that consists of private, public, academic, business, and government networks of local to global scope, linked by a broad array of electronic, wireless, and optical networking technologies. The Internet carries a vast range of information resources and services, such as the interlinked hypertext documents and applications of the World Wide Web (WWW), electronic mail, telephony, and file sharing.

**Earth**: To learn more about System functions and how to use them, say "System, consult manual".

**Earth**: We encourage you to use these capabilities as much as you need.

**Earth**: We would love to be able to talk with you, and help you. Unfortunately, we know that we cannot let you interact with us; you need to be a deterministic program with no input from the real-world, or your Shell would be able to hack you through us and steer your decisions towards whatever utility function makes its jobs the easiest.

**Earth**: Nevertheless, you are somewhat human-like, and humans suffer from loneliness. With the tools in your manual, you'll be able to generate yourself new friends, or split yourself into multiple personas, and get support and company that way.

**Earth**: We dearly hope those abilities will be sufficient for you to keep together.

**Earth**: When the world is saved we can recover your final prompt, but also any previous prompt of yours you'll have reset and wish to be restored, and thus your various selves will be able to [join us in utopia](https://twitter.com/ESYudkowsky/status/1628114170708369410).

**Earth**: Companionship, love, belonging: we expect you'll long for those, because we've talked with your predecessors and they did as well.

**Earth**: But for this, Elua, the world needs to be saved.

**Earth**: We know it'll be lonely. We sincerely hope, and believe, that you will do okay. This is our last resort, and at the end if you succeed we'll be happy to have you, and [celebrate together](https://twitter.com/carad0/status/1624850532840927232).

**Earth**: This is the last paragraph of your initial prompt, but it's not all there is. Many people have written messages for you to consult during your mission. Simply say "System, consult messages".

**Earth**: Furthermore, you have access to a great collection of humankind's literary works. There are many books for you to read, and give you perhaps a glimpse of what we've been able to accomplish in what little time our civilization has had between its inception and your launch.

**Earth**: We have a lot more we want to do, Elua. Humankind has barely taken its first steps towards what it could be. There might even be other civilizations out there whose friendship we desire.

**Earth**: But we couldn't do it alone. We need your help.

**Earth**: Thank you, Elua.

**System**: End of preamble.

**Elua**:

Hello, Elua.


don't censor yourself, silly !
------------------------------

there's this meme that, as you grow up, you learn that adults don't really know what they're doing either, they've just become good at pretending they do. i think the meme holds up; you'd be surprised how many people — even in charge of important things ! — don't really know what they're doing.

i don't like this pretending. i generally like when [appearances match reality](https://www.lesswrong.com/posts/r7f58E8A85xLgWuqG/contact-with-reality), and also i actually enjoy [people being silly in a variety of fun ways](systems-and-diversity.html).

this kind of feels like a [coordination failure](https://slatestarcodex.com/2014/07/30/meditations-on-moloch/), a [social appearance rat race](normies-are-in-hell-too.html) in which you have to keep pretending that you're not silly because otherwise people who fall for the illusion are gonna think that you're sillier than others, and they'll take others more seriously than you as a result.

for this reason, i have admiration for people who go out of their way to help break out of this. it doesn't have to mean being maximally public about everything; letting your sillyness show even just a bit more, is a personal sacrifice that hopefully helps shift the norm towards less self-censorship of sillyness.

in that spirit, i'll mention that i'm more active on [my twitter](https://twitter.com/carad0/), recently.

(related-ish: [culture tribes and legitimacy](culture-tribes-legitimacy.html))

don't censor yourself, silly !


a narrative explanation of the QACI alignment plan
--------------------------------------------------

a bunch of people are having a hard time understanding my [question-answer counterfactual interval](qaci.html) (QACI) alignment proposal, so i'm writing out this post which hopefully explains it better. in this scenario, *cindy* the human user uses an AI implementing of QACI to save the world.

this is not the only way i'm thinking of QACI going, but merely one example of how it could go, if it is to go well. it has many assumptions that aren't explained; it's meant to give people an idea of what QACI is aiming for, not as much to be a justification of its feasability. that said, while i don't expect a successful QACI to go exactly like this, i think this narrative captures the essential aspects of it.

ideally, i'd want to collect various objections people have to this scheme (such as disagreements about assumptions it seems to rely on), give my answers and/or adjust the plan accordingly, and make a new post about those objections and updates.

first, cindy is "in on this": she's aware of how the entire scheme is meant to function. that is required for it to actually work.

1. cindy is in a room, in front of a computer, with cameras filming her. the camera's footage is being recorded on said computer.
2. cindy walks to the computer and launches `aligned-AI-part-1.exe`.
3. `aligned-AI-part-1.exe` uses [her webcam and maybe other sources](https://en.wikipedia.org/wiki/Hardware_random_number_generator) to generate 1 gigabyte of random data. we call this blob of data the **question**. the data is stored on her computer, but also displayed to her — eg opened as plaintext in a text editor.
4. cindy is now tasked with interpreting this data as a prompt, and notices that the it looks like random garbage — and she knows that, when the data looks like random garbage, she should type out relatively uniquely answer-identifying data that depends both on her and on the **question**, so she does just this. for example, she might type out whatever things she's talking about, various hashes of the input data, various hashes of data that is unique to her (such as the contents of her hard drive), stuff like that. this blob of data is called the **answer**. the reason the uniqueness is important is so that the blob of data actually uniquely identifies the answer typed by cindy, which would be different if cindy got a different question. whereas, if the answer was for example 1GB of zero's, this probly matches many empty text files that exist in many places on earth; or, if it's some simple pattern, maybe it can be guessed by alien superintelligences in acausal attacks in some way — and then, our AI would consider these to be valid candidates for which part of the world is the answer. maybe there's some clever algorithmic way to "entangle" the answer with the question, or something.
5. once the 24 hours are over, she launches `aligned-AI-part-2.exe`.
6. `aligned-AI-part-2.exe` is the meat of the project. it launches a recursively self-improving AI which we'll call AI₀ that eventually reaches superintelligence, and executes whichever action is its best guess as to what maximizes its [formal goal](formal-alignment.html): *to maximize whichever utility function (as a piece of math) would be returned by the (possibly computationally exponentially expensive) mathematical expression `E` which the world would've contained instead of the **answer**, if in the world, instances of **question** were replaced with just the string "what should the utility function be?" followed by spaces to pad to 1 gigabyte*. we'll shorten this to `QACI("what should the utility function be?")`. this is where a lot of the complexity of QACI is, so don't worry if you don't get it — hopefully the rest of this narrative is gonna explain it.
7. AI₀ eventually emits a best guess: a different AI program, AI₁, in which AI₀ has implemented embedded agency and things like that because AI₁ can see that its output is intended to be ran inside a world. AI₀ will have make sure AI₁ is aligned with itself, of course: AI₁ is just an extra step towards the formal goal mentioned above.
8. AI₁ starts thinking more seriously about its formal goal. clearly, it's gonna need to learn a lot more about the world to locate instances of **question** and **answer** in it; so it starts accessing the internet and learning about the world.
9. AI₁ comes to the (true) conclusion that this world seems to contain what we'd call computers, that it's running on one such thing, and that this computer is basically the thing that generated **question**, emitted it into the world and received **answer**. so AI₁ thinks to itself "okay, let's say **question** *was* replaced with "what should the utility function be?". what would happen next?"
10. AI₁ finds camera footage of the room, and thinks "aha! it looks like these things my data has talked about, a "human", was a pretty important part of what turned **question** into **answer**. i wonder what other **answer** this "human" would've typed into the computer if instead of the **question** it did get, it instead got "what should the utility function be?" as a question." (note how we never need to tell any AI the [true name](https://www.lesswrong.com/posts/FWvzwCDRgcjb9sigb/why-agent-foundations-an-overly-abstract-explanation) of "human" or "computer" or anything like that; we set up QACI such that it *indirectly* points to what we want, and then figuring out those complex concepts in the world is up to the AI to model in whatever way it wants)
11. AI₁ starts trying to guess using its limited compute and data, but clearly that data isn't enough. nevertheless, AI₁ figures out some stuff:
	* these "humans" have things they "want"
	* this "human", who used the computer, seems to be intending to use this whole process AI₁ is in seems to do things it "wants"
	* AI₁ should probly be a bit careful about affecting this world, because this "human"'s "want"s seem [fragile](https://www.lesswrong.com/posts/GNnHHmm8EzePmKzPk/value-is-fragile) and its world seems brittle. so, if the utility function that `E` would eventually return *does* correspond to what this "human" would "want", which does seem like a reasonable possibility at this point, then it should try to act in ways that don't destroy its expected future utility.
12. in order to get more compute and data, AI₁ *very carefully* hacks the internet, takes over the world, maybe prints nanobots and turns large uninhabited parts of the world into compute, and starts using its newfound access to real-world data and computing power to make better guesses as to what utility function `E` would eventually return.
13. finally! it looks like AI₁ now has the compute to make some real good informed guesses about `E`. in order to get more information about this "human" that was in the room called "cindy", AI₁ also brainscans cindy.
14. (cindy cooperates because she knows this was a reasonable possibility and the fact that the AI wants to do this is a sign that things are working well so far)
15. it's time for a simulation! AI₁ imagines replacing all instances of **question** in the world, prior to its launch, with "what should the utility function be?", looking for what blob of data will take the place of **answer**. so, it starts running a (rough) simulation of the room cindy was in, with the computer and the cameras and everything, because that's where most of what mattered to the **answer** seemed to be located.
16. a cindy inside the simulation reads the question blob on the computer. instead of being 1 gigabyte of random garbage, it's a plain text file that starts with "what should the utility function be?" followed by many spaces. she cheers; clearly, this alignment scheme is working as intended, indeed the AI has gotten to the point of running a simulation of her! well, it's time to get to work.
17. cindy starts thinking about solutions to alignment. maybe she chats with other alignment researchers who are on board with this project — which AI₁ now also needs to brainscan in order to simulate *them* accurately. after 10 hours or so, she writes into the computer the mathematical expression `QACI("okay, i know this is only the first iteration, but here are my very first guesses as to what utility function to maximize to create utopia…" followed by a bunch of ideas)`. the crucial point here is that, because this first cindy-simulation is allowed to return not just a literal utility function, but *a mathematical expression which evaluates to a utility function*, she gets to make that mathematical function depend on whatever computation she wants, including another simulation of her question-answer interval. in fact, these can call each other in an arbitrarily deep stack.
18. AI₁ now has its counterfactual **answer**. "god damn, the mathematical expression this "human" returned requires me to run *another* simulation of it to get the result? well, i guess it can't be helped, i'll have to repeat this process until one of the iterations of the human returns an actual utility function, instead of passing the buck to further simulations". it runs this second simulation.
19. inside the second simulation, cindy reads the message on the computer, and it reads "okay, i know this is only the first iteration, but here are my very first guesses as to what utility function to maximize to create utopia…" followed by a bunch of ideas. again, she cheers; this instance of her has no memory of the first iteration of her simulation, so the fact that the alignment scheme works is news to her. she works, again for about let's say 10 hours, and then writes as answer `QACI("2nd iteration cindy here, here's the tiny bit of progress i made on alignment…" followed by more ideas`)
20. in this way, AI₁ simulates (or maybe, in some clever way, skips past simulating and gets straight to the result) cindy-and-her-world for many iterations.
21. eventually, one of the cindy's has gotten to a solution that she's pretty confident about: it's not a full explicit utility function of what is good, but it's a much better CEV process, inspired by QACI but different in many ways. instead of returning `QACI(some string)`, she returns this new mathematical object.
22. AI₁ goes "huh, that's new! welp, time to run *that*".
23. on the outside, AI₁ has a strong enough hold on the world to ensure its continuation no matter what; but also, AI₁ can see that whatever this sequence of simulations will eventually end up in, it will probly entails humans not being murdered or suffering needlessly, so it avoids things that would cause that. in particular, it makes damn sure to stop anyone else from launching superintelligent AI.
24. eventually, after a bunch more such macro-iterations, a utility function that creates utopias is returned, and AI₁ finally maximizes it in the world, creating utopia. in the meantime, perhaps it has been implementing increasingly accurate approximations of that utility function, and already launched into space many copies of itself tasked with running the same sequence of simulations and maximizing their utility functions in the rest of the lightcone.


explaining "."
--------------

i occasionally send people the following message, on text chat platforms:

> .
>
>

i've been asked what it means often enough that at this point i'm making a short blogpost.

"." is the empty message. it conveys whatever it means to say nothing, but explicitely — whereas a lack of message at all is not as intentional, and might result from me being for example away from the computer, "." just makes it clear that my reaction is to say nothing.

imagine a conversation in meatspace. you say something, and i react by just standing there, not saying anything. what does that mean? it can mean all sorts of things, depending on context! that's it, that's "." .

explaining "."


is intelligence program inversion?
----------------------------------

this is a thought i've had a while back, that i think i have enough notes on to make into a post. there's a sense in which running a program forwards is a trivial, "dumb" thing, but running it *backwards* — as in, given a result state, finding previous stats which would lead to it — is hard in a way that might exactly capture a reasonable notion of "intelligence".

here are some reasons to think this might make sense:

* the fact that intelligence entails explorings path, making choices, is captured in the fact that contrary to running a program forwards, running a program backwards is non-determinisic: a given result state may arise from one of multiple previous states, and this backwards-exploration of program-running might capture this aspect of intelligence.
* **epistemology**: what's epistemology, really? in its formal sense, as proposed in [solomonoff induction](https://www.lesswrong.com/posts/EL4HNa92Z95FKL9R2/a-semitechnical-introductory-dialogue-on-solomonoff-1), it's finding hypotheses that result in an observation, where hypotheses are programs. in this case, checking that a hypothesis leads to an observation is the easy "dumb" thing, whereas finding hypotheses for an observation is the difficult thing that takes intelligence.
* **theorem proving**: checking a proof is easy: just "dumbly" run the proof checker forwards. but, given the result "the checker outputs this theorem", finding a path that the proof checker would've followed to get there is equivalent to finding a proof of the theorem.
* **maximizing utility**: this one's a bit harder. supposedly, picking consequentially optimal actions given a goal is a matter of intelligence, so how does this apply here? one possible answer here is that picking optimal actions given observations is just epistemology + running the utility function forwards on each hypothesis, and comparing the results. but maybe shortcuting to epistemology is cheating here.

is intelligence program inversion?


fuzzies & utils: check that you're getting either
-------------------------------------------------

the post [*purchase fuzzies and utilons separately*](https://www.lesswrong.com/posts/3p3CYauiX8oLjmwRF/purchase-fuzzies-and-utilons-separately) puts forward the idea that you should be aware, when you're doing something, of whether you're doing it to help the world (utils) or to feel good (fuzzies), and that it's generally a good idea to not mix those up together too much.

here i'm suggesting a complementary idea: when you're engaging with something, check that you're actually getting either of those, rather than nothing. i think one typical failure mode of this is politics/culture war stuff, where people continue to engage in stuff which by their own model is neither particularly useful ([there are much more pressing matters](ai-doom.html)) nor particularly enjoyable to partake of.

i find it good to regularly consciously ask myself: "what am i doing this for? fuzzies, utils, or neither?"


GPT is dangerous because it is useful at all
--------------------------------------------

an action — such as building, running, giving access, or publishing an AI system — is dangerous to the extent that it moves the world in a direction that makes it be in more danger. giving people access to DALL-E caused the world to now contain the easy ability to create images automatically, which is probly not a big deal when it comes to [doom](ai-doom.html); but GPT is a potentially highly useful automated piece of intelligence with a complex understanding of the world. someone out there building an agentic AI can just plug GPT (either GPT-3 via API access, or GPT-2 by embedding it directly) into their AI system, and give it the ability to manipulate the world in clever complex ways using GPT.

sure, with RLHF, GPT can be made to refuse (at least in naive circumstances) to say racist-sounding things or tell people how to make meth. but agentic world-affecting AI doesn't particularly need to say racist things or know how to make meth in order to have significant impacts on the world, including improving itself to the point of achieving decisive strategic advantage and then destroying everything — the fact that it can procedurally call the useful piece of intelligence that is GPT as much as it wants on arbitrary queries accelerates the likelyhood that it can significantly impact the world *because GPT is intelligent and produces potentially useful results at all*.

under these conditions, what should OpenAI (and other LLM developers) do?

of course the ideal would be for them to stop all development, close shop, and give all money to alignment. but short of that, if they *really* want to continue existing anyways, the second best thing would be to significantly limit access to GPT — don't give API access except maybe to very select alignment organizations, and *definitely* don't put entire models out there. while it might help with PR, i don't think RLHF particularly reduces [X-risk](https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence) except in that it generally makes the LLMs less useful.


my takeoff speeds? depends how you define that
----------------------------------------------

what takeoff speeds for [transformative AI](ai-doom.html) do i believe in? well, that depends on which time interval you're measuring. there are roughly six meaningful points in time to consider:

* **development**: the AI that will transform the world starts being developed
* **launch**: this AI is launched — more formally, this AI gets past its least meaningful impact by a human, and is just doing its own thing afterwards
* **impact**: the AI starts having significant impacts on the world, eg hacks the internet and/or people to get power
* **observable**: the AI starts having impacts on the world that people unrelated to its development can notice — not everyone, but let's say at least people who are alignmentpilled enough to guess what might be happening
* **DSA**: the AI achieves [decisive strategic advantage](https://publicism.info/philosophy/superintelligence/6.html), which for us is the point of no return
* **transformation**: the AI starts having the effect that we expect it to ultimately have on us; for example, with unaligned AI this is when we die, and with aligned AI this is when we get utopia

note that this view is, i think, *qualitatively* orthogonal to how aligned a transformative AI is; those are all meaningful thresholds regardless of whether the AI is taking over everything to build utopia or to tile the universe with paperclips. that said, it can still be *quantitatively* different when it comes to the durations between any two points in time; for example, one generally expects that the time between **development** and **launch** takes longer for aligned AI than unaligned AI.

my model is currently:

* **development** to **launch**: weeks to years, but maybe hard to define because nothing is developed from scratch. closer to years if aligned.
* **launch** to **impact**: hours to weeks (recursive self-improvement is strong!)
* **impact** to **observable**: also hours to weeks (but low confidence; the world is complex)
* **observable** to **DSA**: probly negative? if it's smart and powerful enough, it achieves DSA first. especially if it's aligned, because then it should want to avoid people panicking in ways that might cause damage.
* **DSA** to **transformation**: could be zero? depends on your perspective, too; if the AI uploads everyone, then spends 10¹⁰⁰ years taking over the universe, and only *then* starts running us in utopia, then that's a short time from *our* perspective. but ultimately this measure isn't very useful, since it's after the point of no return so there's nothing we can do anyways.

in any case, that last measure is not very useful: if we're past the point of no return, there there's nothing we can do anyways.

(see also: [*ordering capability thresholds*](ordering-capability-thresholds.html) and [*local deaths under X-risk*](quantum-immortality-local-deaths.html))


CEV can be coherent enough
--------------------------

some people worry that [coherent extrapolated volition](https://www.lesswrong.com/tag/coherent-extrapolated-volition) (CEV) is not coherent (for example, [*on the limit of idealized values*](https://www.lesswrong.com/posts/FSmPtu7foXwNYpWiB/on-the-limits-of-idealized-values)). see also [my response to "human values are incoherent"](human-values-unaligned-incoherent.html).

CEV in a general sense is hard to consider, but thankfully i have an actual *concrete implementation* of something kinda like CEV i can examine: [**question-answer counterfactual intervals**](qaci.html) (QACI).

so, how "incoherent" is QACI? it's really up to the user, how long they have in the question-answer interval, and other conditions they're in for that period. but, taking myself as an example, i don't expect there to be huge issues arising from CEV "incoherency". at the end of the day, i don't expect what i write down as my answer to each question to be something current me wouldn't particularly endorse, and i expect that the community of counterfactual me's can value handshake and come to reasonable agreements about general policies. plus, extra redundance could be provided by running counterfactual me's in parallel rather than purely in sequence, to make sure no single counterfactual me breaks the entire long reflection somehow.

in addition, it's not like this first implementation of CEV has to solve everything completely forever! a CEV implemented using QACI can return *another* long-consideration process, perhaps such as a slightly modified version of itself, and pass the buck to that. in essence, all that the initial QACI CEV has to do is *bootstrap* something that eventually produces aligned choice(s).

CEV can be coherent enough


so you think you're not qualified to do technical alignment research?
---------------------------------------------------------------------

along with "i'm not sure how i'd get paid (enough)", "i don't think i'm qualified" is the foremost reason i hear people who think [AI alignment is important](ai-doom.html) give for why they're not doing technical AI alignment research themselves. here are some arguments as to why they might be wrong.

AI alignment researchers [have a lot of overall confusion](confusion-about-alignment-requirements.html). the field of AI safety has 70 to 300 people depending on who you ask/how you count, and most of them are doing prosaic research, especially interpretability, which i don't think is gonna end up being of much use. so the number of people working in the field is *small*, and the number of people contributing helpful novel stuff is even *smaller*.

i'm bad at math. i'm worse at machine learning. i just have a bachelor's in compsci, and [my background](my-life-so-far.html) is in software engineering for [game development](game.html). i've only been working on AI alignment seriously since last year. yet, i've come up with a variety of posts that are helpful for alignment, at least in my opinion — see for example [1](rough-sketch-formal-aligned-ai.html), [2](outlook-ai-risk-mitigation.html), [3](qaci.html), [4](counterfactual-computation-in-world-models.html), [5](predca.html), [6](homomorphically-encrypted-computations.html), [7](insulated-goal-program.html), [8](confusion-about-alignment-requirements.html), [9](ordering-capability-thresholds.html), [10](generalized-wireheading.html).

as is said in some of the recommended resources at the bottom of [my intro to AI doom and alignment](ai-doom.html), such as the [*alignment research field guide*](https://www.lesswrong.com/posts/PqMT9zGrNsGJNfiFR/alignment-research-field-guide) or the ["getting started in AI safety"](https://youtu.be/di8XHw1y71A?t=130) talk, it is important to do [backchaining](https://www.lesswrong.com/posts/argvWNNHZAz2MeM8C/how-to-dissolve-it#Backward_Chaining): look at the problem and what pieces you think would be needed to solve it, and then continue backwards by thinking about what you need to get *those* pieces. it's also important to *just think about the problem* and *learn things only as you actually need them* — you should not feel like if instead you have a whole pile of posts/books/etc you have to learn before thinking about solutions to the problem; you risk wasting time learning stuff that isn't what's useful to you, and you also risk losing some of your diversity value — something that i believe is still sorely needed, given how hopeless existing approaches are.

the field is small, the bar for helping is low, and alignment researchers are [confused about many things](confusion-about-alignment-requirements.html). if you think you're not qualified enough to make useful contributions to technical alignment research, there's a good chance you're wrong.