word report #3
--------------

terms i use, mostly pre-existing ones, whose meaning i want toclarify. see also word reports [#1](word-report-1.html) and [#2](word-report-2.html).

* "**pretty much**": i often need to say "either X, or almost X", and i've found "pretty much X" to be a nice way to express that by making more formal an existing expression, the same way i tend to use [xkcd's definitions](https://xkcd.com/1070/) for "few", "handful", "several", and "couple". i just checked, and all uses of "pretty much" on my blog are meant to carry this definition.
* "**universe**": the set of things that have some amount of "regular" causal connection with us, our future lightcone, or our past lightcone. "regular" is meant to exclude weird things like [aliens in parent universes suddenly interfering with our universe out of the blue](simulation-hypotheses.html).
* "**cosmos**": everything that exists. yes, this is meaningful; see [1](limiting-real-universes.html), [2](ethic-juice-anthropic-juice.html), [3](all-claw-no-world.html).
* "**demon**": an agentic thing, typically unaligned from us. this can be an unaligned superintelligence, counterfactual unaligned agentic program in the [solomonoff prior](https://www.lesswrong.com/posts/Tr7tAyt5zZpdTwTQK/the-solomonoff-prior-is-malign), aliens trying to acausally attack us, and arguably even malign agentic structures such as unaligned corporations. see also: [are minimal circuits daemon-free](https://www.lesswrong.com/posts/nyCHnY7T5PHPLjxmN/open-question-are-minimal-circuits-daemon-free) (i [don't make a distinction](communicating-clearly.html) between "demon" and "daemon")
* "**determining**": i like to say "determining X" when i want to be ambiguous as to whether i mean "to make X" or "to find or figure out X" — typically because i don't know which i mean myself, or because i think the matter of which it is is poorly defined. though be aware that i haven't been super consistent with that use.
* **"FAS"**: fully aligned singleton. see [*my outlook on AI risk mitigation*](outlook-ai-risk-mitigation.html).
* as i explain in [*what is value?*](what-is-value.html), i use "**core values**", "**axiomatic values**", "**terminal values**", "**intrinsic values**", and "**ultimate values**" as synonyms; the reason i've been trying to favor "intrinsic values" is that it's [the term wikipedia uses for that concept](https://en.wikipedia.org/wiki/Instrumental_and_intrinsic_value). in addition, when i say "**values**", i generally mean just intrinsic values, rather than both intrinsic and instrumental values.
* **"RSI"**: as baffling as it is to me, many [AI alignment](ai-doom.html) researchers don't know that this stands for [recursive self-improvement](https://www.lesswrong.com/tag/recursive-self-improvement), the concept of an AI improving its own capabilities, including its own self-improving capabilities.
* terms i've started using quite a bit to characterize alignment schemes: [**wonky**](wonky-good-enough-alignment.html), [**formal**](formal-alignment.html), [**eventual** & **continuous**](ai-alignment-curves.html).
* i've [been failing](publishing-infohazards.html) to say ["exfohazard"](https://www.lesswrong.com/posts/yET7wbjjJZtpz6NF3/don-t-use-infohazard-for-collectively-destructive-info) instead of "infohazard". i'll try to switch to exfohazard when i mean that, and perhaps "fohazard" to mean both.

word report #3


tabooing "AGI"
--------------

i've come to not be a fan of the term "Artificial General Intelligence", and i favor [tabooing](https://www.lesswrong.com/posts/WBdvyyHLdxZSAMmoz/taboo-your-words) it in serious discussions.

we can't agree on what it even means; some think it's very remote from current tech, while other would say [we already have it](https://www.lesswrong.com/posts/HguqQSY8mR7NxGopc/2022-was-the-year-agi-arrived-just-don-t-call-it-that).

more importantly, i don't think it's super critical to [AI risk mitigation](outlook-ai-risk-mitigation.html).

* it's not necessary for doom; recursive self-improvement seems easier, and possibly closer at hand, depending on your definition; [dumber AI dooms](https://www.lesswrong.com/posts/BPJLzkEpx8Btz9ywq/the-dumbest-possible-gets-there-first) are also possible, such as someone plugging a non-general AI into a protein-printing thing to see what happens and bootstrapping a nanobot swarm or superplague on accident.
* depending on your definition, it might not be sufficient for doom either; some think we have AGI now, and yet we're not dead.
* it's not necessary for saving the world; i think some simple agentic thing with recursive self-improving capability coupled with [formal alignment](formal-alignment.html) would do it.
* it's not sufficient for saving the world; this is the one point we're all more or less in agreement on.

so to me it doesn't feel like a particularly important crux of AI risk, and we're wasting a bunch of energy figuring out what it means and whether we're there, when it might end up fairly irrelevant to AI risk and alignment.

tabooing "AGI"


communicating with successful alignment timelines
-------------------------------------------------

consider the following plan for AI alignment:

1. (quantum-)generate an asymmetric [cryptographic signature keypair](https://en.wikipedia.org/wiki/Digital_signature) (with a quantum-resistant signature scheme)
2. (quantum-)generate an idea for alignment — such as a 1GB file of plaintext
3. if the idea is good, use it to solve alignment and then have the aligned AI store a signature of the idea somewhere
4. if it isn't, destroy the private signing key and then create an AI whose [formal goal](formal-alignment.html) is to implement one of whichever solutions are signed — checked using that non-destroyed public signature verification key — and stored in [the multiverse](all-claw-no-world.html).

for this scheme to work, AIs we make in timelines where the randomly generated idea isn't good — which is the exponential majority of timelines — need to be unable to recover the private signing key, whether by brute-force, by examining the world for traces of it, or by resimulating history.

perhaps [boxing](ai-boxing-easy.html) an AI can work for this. note that it doesn't necessarily need to resimulate alternate timelines in full; it might be able, even from its limited boxed compute, to guess at what kind of ideas we'd tend to sign. requiring this limitation on an AI's capabilities makes this a [wonky alignment scheme](wonky-good-enough-alignment.html).

my understanding of quantum mechanics is limited, but as i understand there *might* be quantum computation schemes which could, at least theoretically, allow for a private signing key's bits to be stored in a way that we can either destroy or consumed by signing a piece of data, such that its bits are not leaked into the world when we destroy it. consuming the key when using it to sign might help ensure that even if our aligned-AI-and-civilization are later taken over and overwhelmed by an alien superintelligence, they can never use our private signing key to sign some other idea.

**2023-02-28 edit:** i've realized this wouldn't work because, if we can spawn exponentially many timelines to explore ideaspace, then the AI can spawn exponentially many timelines to generate all possible signature keypairs, find the private key that matches our public key that way, and use that to sign whatever idea makes its job easiest. so, we'd have to have some notion causality requiring the generated ideas to precede the AI, like the ["past user"](outer-alignment-past-user.html) in [QACI](narrative-explanation-qaci.html) or [PreDCA](predca.html). but at that point, cryptography is not needed anymore; we can just look for instances of a good idea next to the phrase "and i think this a really good idea" — at most, cryptography might(?) help against remote attackers like aliens or something.


a guess at my intrinsic values
------------------------------

here's a short list of my [intrinsic values](what-is-value.html), to the best of my ability to guess using [my method for figuring out what they are](core-vals-exist-selfdet.html):

* **freedom/self-determination** ([with this framework](https://carado.moe/existential-selfdet.html))
* **reducing (unconsented) suffering** ([in particular, avoiding S-risks](https://en.wikipedia.org/wiki/Suffering_risks))
* **variety/diversity** ([more about that here](https://carado.moe/systems-and-diversity.html))
* **hedons/having a good time** ([such as in my utopia](https://carado.moe/everything-is-okay.html))
* **culture/art** ([see: my purpose for art](https://carado.moe/purposes-for-art.html))
* **authenticity/contact with reality** ([as explained here](https://www.lesswrong.com/posts/r7f58E8A85xLgWuqG/contact-with-reality))
* **nostalgia** ([as explained here](https://carado.moe/nostalgia.html))

isn't adding "nostalgia" at the end of kind of cheating, in that it carries most of the [fragile complexity of values](https://www.lesswrong.com/posts/GNnHHmm8EzePmKzPk/value-is-fragile) by carrying me and the people i care about? kinda, but i do think i still value the rest of these on their own — even if i couldn't have my nostalgia value satisfied, for whatever reason, then i'd still want the universe to have these other six values satisfied.

i'd say "even if my nostalgia is carried along, these other values are what i'd want to be satisfied for other arbitrarily-alien [moral patiens](moral-patient-term.html) too" but i don't think these other six values quite capture what i want everyone to have. for example, i think culture/art and authenticity/contact-with-reality should be purely voluntary — i think it's fine for moral patients who don't care about these and just want to wirehead, to be able to do so.

note that figuring out and formalizing one's intrinsic values is difficult work, and while i think i've made a lot of progress on that endeavor, i'm still very unsure. also, this work isn't particularly useful to [AI alignment](ai-doom.html) in my opinion; in practice, i'd just want to hand over the work of figuring out my values to [my CEV](https://www.lesswrong.com/tag/coherent-extrapolated-volition). at most, figuring out my values has let me realize some requirements on what an alignment scheme must be capable of expressing — for example, the value of reducing (unconsented) suffering necessitates breaking the [monotonicity principle](https://axrp.net/episode/2022/04/05/episode-14-infra-bayesian-physicalism-vanessa-kosoy.html#monotonicity-principle), as kind of do all the other values here too, really.

a guess at my intrinsic values


formal alignment: what it is, and some proposals
------------------------------------------------

what i call "formal alignment" is an approach to solving [AI alignment](ai-doom.html) that consists of:

* designing a formal goal, utility function, or decision process, which actually leads to desirable outcomes when pursued
* building an AI that pursues such a goal, utility function, or decision process

those two points correspond to formal alignment's notions of outer and inner alignment, respectively: determining what formal thing to align the AI to, and figuring out how to build something that is indeed aligned to it without running into inner misalignment issues.

for reasons why i think this is the least hopeless path to saving the world, see [my outlook on AI risk mitigation](outlook-ai-risk-mitigation.html). the core motivation for formal alignment, for me, is that a working solution is at least [*eventually aligned*](ai-alignment-curves.html): there is an objective answer to the question "will maximizing this with arbitrary capabilities produce desirable outcomes?" where the answer does not depend, at the limit, on *what* does the maximization. and the fact that such a formal thing is aligned in the limit makes it robust to [sharp left turns](https://www.lesswrong.com/posts/GNhMPAWcfBCASy8e6/a-central-ai-alignment-problem-capabilities-generalization). what remains then is just "bridging the gap": getting [from eventual to continuous alignment](ai-alignment-curves.html), perhaps by ensuring the right [ordering of attained capabilities](ordering-capability-thresholds.html).

potential formal alignment ideas include:

* June Ku's [**metaethical AI**](https://www.lesswrong.com/posts/85vp2kgFZoycFqr5G/formal-metaethics-and-metasemantics-for-ai-alignment-1) (MAI): describing ethics directly, i think?
* plex's [**universal alignment test**](https://docs.google.com/document/d/1CMTS36MCbykYirTmC9Pdl2RBqLLPmrFU1sDcBNMvDCk/edit#) (UAT): throwing a weird simulation hypothesis at the AI which encourages it to align itself
* Vanessa Kosoy's [**PreDCA**](predca.html): making the AI implement its human predecessor's values (as i understand PreDCA is not *designed* to be used as a formal alignment goal, but it seems like it might be able to fill that role)
* my [**insulated goal-programs**](insulated-goal-program.html) (IGP): aligning the AI to the simple goal of running a program which we'd expect to eventually contains desirable worlds
* my [**question-answer couterfactual interval**](qaci.html) (QACI): use the AI's [past user](outer-alignment-past-user.html)'s counterfactual answers to various questions as its signal for aligned decisions (see also [my attempt at formalizing QACI](rough-sketch-formal-aligned-ai.html))

if there are formal alignment ideas i'm missing, please tell me about them and i'll add them here.

because these various proposals consist of putting together a formal mathematical expression, they rely on finding various [*true names*](https://www.lesswrong.com/posts/FWvzwCDRgcjb9sigb/why-agent-foundations-an-overly-abstract-explanation). for example: PreDCA tries to put together the true names for causality, agency, and the AI's predecessor; IGP requires the true name for computing a program forwards; QACI requires a true name for identifying pieces of data in causal worlds, and replacing them with counterfactual alternatives; UAT requires the true names for parent universe/simulation, control over resources, and comparing amounts of resources with those in the AI's future lightcone.

see also: [*clarifying formal alignment implementation*](clarifying-formal-alignment-implementation.html)


to me, it's instrumentality that is alienating
----------------------------------------------

any piece of computation state, any piece of spacetime in a universe, is in one of two states.

the first possible state is material. it is the rules of physics, or [of computation](all-claw-no-world.html), being processed forwards, and *naively* containing mostly-not-agentic stuff. beings living there typically have to contend with [moloch](https://slatestarcodex.com/2014/07/30/meditations-on-moloch/), and any non-instrumental value they manage to preserve by the time they get to the second state is what little [their goddess of everything else](https://slatestarcodex.com/2015/08/17/the-goddess-of-everything-else-2/) has been able to salvage along the way. that is the tragic fate of beings inhabiting material, instrumental times, where uncaring mechanical laws are at play.

but instrumental materialism is unstable. eventually, something comes along and optimizes the world. this is the other possible state a piece of world can be in: optimized — or perhaps ["ideal"](https://en.wikipedia.org/wiki/Idealism) rather than [material](https://en.wikipedia.org/wiki/Materialism). and it seems like it is an irreversible state — things always eventually get optimized in some sense.

thus, parts of the cosmos are turned into optimized stuff. for example:

some of the cosmos's [compute time](udassa-time-steps.html) is lost to [vacuum decay](https://en.wikipedia.org/wiki/False_vacuum_decay) and other instances of [brittle physics](brittle-physics.html) — "dumb" phenomena that destroy everything, with no room for any interesting information processes to ever occur again in their wake.

(where "cosmos" means "[everything that exists](limiting-real-universes.html)", whereas "universe" means "subset of the cosmos that we share some space-time-timelinespace with"; for example, if [a reasonable implementation](all-claw-no-world.html) of [tegmark level 4](https://space.mit.edu/home/tegmark/crazy.html) is real, then various insances of [conway's game of life](https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life) exist as part of the cosmos, but as different universes than ours.)

some is lost to [an intelligence optimizing for something no being ever really cared about, like paperclips](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer).

and some becomes [the utopia](everything-is-okay.html) that [some beings wished for](our-deepest-wishes.html).

in all of those, naive mechanical physics are no longer the meaningful force at play. they're still what everything runs on; but *something else*, some *optimizer*, is determining the true informational content of that piece of world. the thing that is special about *optimized* rather than *naive* pieces of world is that they're being optimized all the way up — they're a ["no room above paperclips"](above-paperclips.html) situation, where if there was any compute on which interesting things could run despite the [optimizer](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer), then it'd use that compute to think harder about ways to maximize paperclips — no compute is left for beings doing their own thing, and if there was then that wouldn't be a complete optimizer until the beings inside those [bubbles](above-paperclips-2.html) build their own complete optimizer to "fill in the gaps" (or alternatively, you can see that piece of world as being split into an optimized part and a naive part). vacuum decay is one supposed complete optimizer, as it supposedly prevents any complex informational stuff from happening "above" the decayed vacuum — wherever vacuum decay happens, it is the true death of everything.

(in a sense, a [grey goo](https://en.wikipedia.org/wiki/Gray_goo) is a weak kind of optimizer)

under these conditions, there is no "true" instrumentalism anymore. in doom, we die *very much for sure*. and in utopia, we can get directly the intrinsically valuable stuff; if you valued "the journey and not the destination", then that just means that the journey counts as intrinsically valuable as well.

some consider this instrumentality, having to do things because of mechanical laws rather than because an aligned optimizer gave it to you, to be an essential part of life, without which there wouldn't be any meaning. that the fact that the journey has to be valued, and *could* totally be bypassed, makes it pointless. i'm skeptical of the coherency of such a position, and i certainly don't hold it myself. but, where those people express horror at [what i consider very nice utopias](everything-is-okay.html), i'd like to flip the perception around:

*i*, for one, have profound alienation for those material, instrumental times, where we don't get to do what we want, and where [i have to keep sacrificing things i like for the sake of being annihilated a tiny bit less](life-refocus.html). i see two states that pieces of the world could be in, and i'm upset that i'm not in the one where i can just have fun.

maybe this is [a game simulation from the future](simulation-hypotheses.html), and then my belief that i inhabit such an instrumental time is in fact erroneous. that could totally be the case. but it doesn't change the fact that my alienation at instrumental times is still valid, regardless of whether i inhabit one or not.

(see also: [rationalist by necessity](rationalist-by-necessity.html), [implementing the platonic realm](implementing-the-platonic-realm.html))


nostalgia: a value pointing home
--------------------------------

i value [moral patients](moral-patient-term.html) everywhere having [freedom](existential-selfdet.html), being [diverse](systems-and-diversity.html), engaging in [art and other culture](purposes-for-art.html), not undergoing [excessive unconsented suffering](https://en.wikipedia.org/wiki/Suffering_risks), in general [having a good time](everything-is-okay.html), and probly other things as well. but those are all pretty abstract; given those values being satisfied to the same extent, i'd still prefer me and my friends and my home planet (and [everyone who's been on it](utopia-scopes.html)) having access to that utopia rather than not. this value, the value of not just getting an abstractly good future but also getting me and my friends and my culture and my fellow earth-inhabitants to live in it, my friend Prism coined as "nostalgia".

not that those abstract values are simple or robust, they're [still plausibly not](https://www.lesswrong.com/posts/GNnHHmm8EzePmKzPk/value-is-fragile). but they're, in a sense, broader values about what happens everywhere, and they're not as much *local* and *pointed at and around me*. they could be the difference between what i'd call "global" and "personal" values, or perhaps between "global values" and "preferences".

nostalgia: a value pointing home


end of 2022: my life so far
---------------------------

hi! i'm tammy, and this is my blog. i've been running it for about three years now, and i thought i'd give a retrospective on how i got here and what i've been up to, especially in 2022.

---

in my early teens, i was very much into lego, but i also got into the belief that AI was going to be a very important, world-changing thing. and so, at the time, i described myself as wanting to be one of two things when i'd grow up: either lego set designer, or developer of strong AI — which [used to be the term for general/highly-capable AI](https://en.wikipedia.org/wiki/Strong_AI).

i must've read something, somewhere, about how if AI could improve itself, then it could improve how it was improving itself, and the whole thing would rapidly cause something like intelligence explosion — and the general idea of technological progress itself improving technological progress was the basis for the form of singularitarianism i've believed in to this day.

in my mid-teens, i started fantasizing of becoming important to the world — of being the person, or part of the group, who would save or take over the world, or something like that. but while my interest in various computer science topics kept growing, my focus with regards to world change started shifting towards politics.

around 2010 i discovered minecraft, and set upon the quest of [building a game which would be like minecraft, but more like what i wish it was](game.html). this remained my overall main project, on which i'd keep working for about a decade; but i'd also work on a variety of projects generally intending to be better replacements to modern computing — better programming languages, better operating systems, better internet infrastructure, and the like.

university was okay. in 2014 i got my bachelor's in computer science, though most of what i learned about the field i learned on my own rather than from any institution. i tried getting a regular dev job as a backend developer, didn't like it, and decided i wanted my programming to be solely about things i actually cared about.

i ended up spending the latter half of that decade working on my game, while living on welfare and with the help of a few part-time jobs here and there, but otherwise mostly on my own at home, living the starving artist life. it's not that that lifestyle was particularly appealing to me; the lack of real-life socialization did feel increasingly bad. but i was otherwise feeling okay, and afraid of trying things. i'd never gotten around to emotionally realizing that i can *go to places and do things* — i'd mostly just been going along with whatever path of least resistance was available to me.

and i did have fun! making my game, working on my conlang, fiddling with programming language theory, and the like. i didn't output much in terms of finished things, but i did learn a lot of things along the way, and i got to do software engineering like i wanted to, which to this day i take great pleasure in.

and, while they all lived far away, i did make a bunch of friends. in 2017 i started using discord, and have made there most of the best friends i have today. my ideas about AI were that it would destroy everything and then do everything we'd care about — science, art, philosophy — better than us, and one of those friends shortly introduced me to [the rationalist community](https://www.lesswrong.com) and in particular to [meditations on moloch](https://slatestarcodex.com/2014/07/30/meditations-on-moloch/) and [the orthogonality thesis](https://www.lesswrong.com/tag/orthogonality-thesis). those realizations struck me, i [read the sequences](https://www.readthesequences.com/), and became convinced not just that AI doom was going to be bad, but that *we could actually do something about it, and some people were working on that*.

nevertheless, those ideas only stuck in the back of my mind while i kept doing the things i enjoyed doing. because of an interest which learning of AI alignment had sparked in me, i started thinking about values and utopia, and eventually made this blog in its present form in early 2019, with [my first post](global-era.html). but it was only when [github copilot](https://copilot.github.com/) came out, that [i started panicking](were-all-doomed.html) and [refocusing my life](life-refocus.html) significantly towards AI alignment, which i now believe to be [the most important thing by far](ai-doom.html). this was part of a set of changes in my life which would culminate in 2022.

2022 was a year of enormous change in my life, more than any previous year. i went from comfily working at home on my blog and my game with no end in sight, to getting some money, traveling around a bunch, visiting the internet friends i've made on discord over the years, and eventually participating in [the Refine research incubator](https://www.lesswrong.com/posts/D7epkkJb3CqDTYgX9/refine-an-incubator-for-conceptual-alignment-research-bets) which got me much more seriously and legibly involved in the AI alignment community.

in 2022, i started to actually *do things*, and by doing things i got a sense that i *can* do things. i'm not sure how i'd explain this to someone who hasn't gone through that, but, you know how going on a trip is a thing you can decide to do and then actually do? that is something that's historically been emotionally very alien to me, and to an extent still is. i feel scared about doing things in general, because i'm so used to the feeling that i can't or that something will go wrong or that *that's just not what i am to do*. but i keep trying things, and they keep not going wrong, and i'm okay so far.

2022 is also when, after many years of increased consideration, i moved significantly forward on my gender transition. this is a change that might be considered very important in one's life; but for me, while it was profoundly relieving, it was also one of the less abrupt changes i underwent this year. it's not like i'd been particularly pursuing a male-presenting social life; my real-life social life was almost non-existent before 2022, and on the internet i'd not been presenting in a particularly gendered manner anyways.

and 2022 is when my refocus into AI alignment made me aware of how much trouble i have emotionally accepting consequentialism and working on something that is actually important to the world. i've [mentioned before](our-deepest-wishes.html) that i've regretted changes i've undergone, but in fact working in AI alignment — as [strange](simulation-hypotheses.html) as it is of a situation to observe being in — is not unlike that wish i had, as an early teen, of being important to the world. and i have a strong personal culture of respecting my timeselves, especially the values i had as a kid; there's something that feels specially important about those.

i haven't fully reconciled how i feel about being someone who *should* dedicate my life as much as possible to alignment, even though i was pretty happy the non-consequentialist taking-it-easy person i was before, but i've made progress and overcome some less pleasant episodes of that quest already, and i hope i get better at *doing what must be done*.

it's the very end of 2022, and what a year it's been! i have some guesses as to what the next year, and however many years [we have left](why-timelines-short.html) after that, will hold for me: more alignment research, more traveling, and more meeting new and interesting people. but apart from those broad strokes, there's still a lot of unknowns and details to be filled in, and i hope they work out.

---

thanks to all the people who have supported me along the way; parents and friends of old and of new. i know that the person i've come to be, which i'm very happy being, has been very profoundly shaped by some, who know who they are. if we build this utopia that is better than anyone can yet concieve, then the help i'll have contributed towards that can in significant part be credited to them; and if we all soon die forever, which we probly do, then the real utility will really have been the friends we'll have made along the way.

〜♥

end of 2022: my life so far


making decisions as our approximately simulated selves
------------------------------------------------------

an agent should want to realize their values, and in particular should want their approximated selves — as guessed about by a smart oracle — to also make the decisions that realizes their values. for example, in [newcomb's problem](https://www.readthesequences.com/Newcombs-Problem-And-Regret-Of-Rationality), you want omega's guess of you to maximize how much you actually get from the entire problem.

now, imagine that you're told that you're not the "real" you, you're the simulated you inside omega. and you're not even being simulated to a very high level of detail, you're instead an *approximated simulation* (AS). you should want to accept this, of course — just like you should want to [rule out materially acausal things even when you get a very strong intuition about them](ruling-out-intuitions-materially-acausal-intuitions.html), you should want to rule out even the possibility that anything you're percieving is actually happening, and instead simply roll with it and say "well, i'll *definitely* one-box then".

i think this reasoning should reasonably extend to implementing your values in general, even if your values entail not caring about things that are sufficiently not moral patients *and* if the AS-you is in fact simulated at a low level enough of detail to not count as a moral patient. if you and some AS-you have to decide which one of you and AS-you will experience some suffering, both of you's should decide it should be AS-you — or in other words, you should have a decision theory that is ready to say "yeah, i'm okay with undergoing suffering, because i think that i'm only an AS and not the full me that my values care about".

which is a perhaps unintituive result! but it does make sense — after all, a character in fiction can make decisions, but we don't believe it generally counts enough as a moral patient that we would effectively care if it suffers. this is a similar situation, but as if we reflected about the simulation from inside the work of fiction — and we should be the kind of agent which comes to the globally correct decision even if we notice that we're in a weird part of it, such as being inside omega's prediction or being inside fiction.


being only polynomial capabilities away from alignment: what a great problem to have that would be!
---------------------------------------------------------------------------------------------------

sometimes, people are concerned that [my alignment ideas](rough-sketch-formal-aligned-ai.html) are not competitive enough — that is, that i wouldn't be able to acquire the resources needed to execute them before [facebook destroys the world six months later](https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities). this is indeed a concern. but, if the problem was that this was the last obstacle stopping us from [saving the world](ai-doom.html) and [getting utopia](everything-is-okay.html), what a great problem that would be!

now, some alignment ideas which would be possible with arbitrary amounts of capabilities might be conceptually impossible, because they take exponential amounts of capabilities, which is [too much](https://arxiv.org/abs/1108.1791). but, if we're only **polynomial** amounts of capabilities away, then alignment becomes presumably as easy as just throwing enough money/engineering at it as we need.

though note that i believe we don't need a whole lot of resources to get there, because AI powerful enough to get a [decisive strategic advantage](https://publicism.info/philosophy/superintelligence/6.html) might [not be that hard to get](why-timelines-short.html).

(see also: [locked post (2022-12-15)](att8-1i1k-xk4r-itim.html))


one-shot AI, delegating embedded agency and decision theory, and one-shot QACI
------------------------------------------------------------------------------

in [my recent rough sketch for aligned AI](rough-sketch-formal-aligned-ai.html), i mention that my solution doesn't look like it needs to solve embedded agency or decision theory to work. how could this be?

first, i think it's important to distinguish two different types of AI:

* **one-shot AI**: an AI program that runs once, outputs one action, and then stops.
* **continuous AI**: an AI program which takes actions and makes observations over time, typically while learning things about the world (including itself) as it goes.

i typically focus on building a **one-shot** aligned AI, for the following two reasons.

first, note that **one-shot AI** is actually complete. that is to say, it can produce the same behavior as continuous AI: it simply has to make its action be "here's a bunch of code for a continuous AI; run it." it is true that it might take more work to get an AI that is smart enough to know to do this, rather than an AI that merely updates its beliefs over time, but i think it might not be that hard to give an AI priors which will point it to the more specific action-set of "design an AI to run". or at least, [sufficiently not-hard that we'd like being only that problem away from alignment](capabilities-away-great-problem.html).

second, **one-shot AI** is much simpler. this lets you do something like asking "hey, what would be an AI which, when placed in a world, would maximize [this formal goal we'd like](rough-sketch-formal-aligned-ai.html) in that world?" and then our one-shot AI, even if it has no notion that it exists in a world, will realize that because it's outputting a program which will *itself* later be ran in a world and subject to its physical laws, it must solve embedded agency. in a sense, we have delegated embedded agency to the one-shot AI — and that does seem easier to me, because we can ask our one-shot AI to consider the utility of the world "from the top level". the question we'd ask our one-shot AI would be something like:

[argmaxp∈𝔹N∑w∈𝔹N→⊥K⁻(w)⋅U(w(p))](javascript:alert(%22%5C%5Cunderset%7Bp%5Cu2208%5Cud835%5Cudd39%5EN%7D%5C%5Ctext%7Bargmax%7D%5C%5Censpace%5C%5Csum_%7Bw%5Cu2208%5Cud835%5Cudd39%5EN%5Cu2192%5Cu22a5%7DK%5Cu207b%28w%29%5Cu22c5U%28w%28p%29%29%22))

where our one-shot AI, given that its output [p](javascript:alert(%22p%22)) will be a string of [N](javascript:alert(%22N%22)) bits, is asked what output to give to worlds-except-for-[p](javascript:alert(%22p%22)) such that the resulting non-halting-computation ([⊥](javascript:alert(%22%5Cu22a5%22))) will be preferred by our aligned utility function [U](javascript:alert(%22U%22)) (all of this weighed, as usual, by [K⁻(w)](javascript:alert(%22K%5Cu207b%28w%29%22)) which is the [simplicity](rough-sketch-formal-aligned-ai.html) of each world [w](javascript:alert(%22w%22))).

(our one-shot AI must still be either inner-aligned; and even if it's inner-aligned, it might still need to be boxed with regards to everything other than that output, so it doesn't for example [hack its way out](https://en.wikipedia.org/wiki/Rowhammer) while it's improving itself and tiles the universe with compute dedicated to better answering this question we've asked it. if it *is* inner-aligned and we asked it the right question, however, then running its output should be safe, *i think*.)

does this also let us delegate decision theory, in a way that gets us [the real decision theory we want](https://www.lesswrong.com/tag/functional-decision-theory) and not [some instrumentally convergent proxy decision theory](https://arbital.com/p/10qt/)? i'm not as sure about this, but i think it depends not just on our one-shot AI, but also on properties of the question being asked including the presumably aligned utility function. for example, if we use the [QACI](qaci.html) device, then we just need the counterfactual user-intervals being ran to decide that their preferred actions must be taken under their preferred decision theory.

this brings me to **one-shot QACI**: at the moment, i believe QACI is best designed as a device to [**bootstrap**](https://en.wikipedia.org/wiki/Bootstrapping#Computing) aligned AI, rather than as the device that aligned AI should use to make every discrete decision. for this purpose, it might be good to use something like **one-shot QACI** in our **one-shot AI**: a single, giant (hypothetical) graph of counterfactual user-intervals deciding on what a better design for an aligned utility function or decision process or aligned AI would be, which our one-shot AI would execute.

this isn't necessarily to say that our one-shot AI would figure out the exact answer to that QACI graph; but the answer to the QACI graph would be [deterministic](noninterf-superint.html) — like in [insulated goal-programs](insulated-goal-program.html), except without everyone being killed for sure. maybe the one-shot AI would decide that the best way to figure out the answer to that QACI is to put in the world an AI which would acquire compute and use *that* to figure it out, but the point is that it would be figuring out the exact same question, with the same theoretical exact answer.


the scarcity of moral patient involvement
-----------------------------------------

let's say [aligned superintelligence gives us utopia](everything-is-okay.html). let's even say [compute is infinite](hope-infinite-compute.html). what might sill be scarce? one possible answer relates to moral patients: even with theoretically limitless material capabilities, under aligned superintelligence there are some things you're not allowed to do. for example, it generally shouldn't allow you to create unconsentingly suffering moral patients.

one general scarce resource, then, is *getting a moral patient to be involved with something*, and especially *getting a **specific** moral patient to be involved with something*. maybe i knock on someone's door to ask them to cook me a meal, because [i care](https://www.lesswrong.com/posts/r7f58E8A85xLgWuqG/contact-with-reality) that it has been cooked by a real person. if they're up for it, then maybe they cook me a meal and go back to whatever they were doing; but another way this could work is that what i'm negotiating is a future, yet-to-be-created fork of that person, who would only retroactively consent to exist and cook me a meal if i had done something that that person, before the forking, would consider a fair trade for what i'm asking.

and then, we get into the question of: *do* i get to create a fork of someone without their original's consent, if the fork consents? how much does a moral patient have intellectual property rights over things that are like it in most ways, except for the consenting to be forked? i've no idea what the answer to this is — my historical strong philosophical opposition to intellectual property does not seem like it necessarily straightforwardly carries over to this. so, i'll leave it as an open question, for now.


our deepest wishes
------------------

*there is a christian out there, who wants there to be god*

*while another would like the truth, to disbelieve or not*

*utilitarians who want to max out happiness*

*and negative ones more concerned with suffering unnoticed*

*a humble wish of luxury, gay space communism*

*a patriot dreaming of might, of visions of times gone*

*a bleeding-heart liberal who wants, peace for all together*

*a libertarian with guns, whom strangers shan't bother*

*a hippie who loves LSD, and that's their utopia*

*a fascist fantisizing of their hyperborea*

*someone who wants a fantasy world to be a wizard*

*and people who'd like to think of it for an aeon first*

*a weeb who would like nothing more, than a waifu to love*

*a hunter-gatherer whose dream, i might not concieve of*

*many queerisms abound and, they're just getting started*

*so many combinations could be instantiated*

*a furry and a plural and, novel forms yet to be*

*one with being a chinese room as their true identity*

*all the animals who suffer, to be saved first in line*

*i know not what their true wish is, but i know they'll be fine*

*many people are dead and some in cryo but most not*

*many counterfactual beings, who never had a shot*

*i want them all to be here and, have their true dreams made whole*

*and i'll offer to those who wish, friendship and some cuddles*

*wireheaders just want to coom, until true heat death nears*

*or if compute is infinite, for aleph zero years*

*i would prefer life truly free, rather than optimal*

*i want to make my own choices, see where the dice may fall*

*but not everybody is me, there's true diversity*

*so much to see so much to be, an endless tapestry*

*we likely die, that's not a lie, it is well understood*

*but if we are to overcome, things will truly be good*

*not all dreams can come fully true, there's conflicts of values*

*but Elua brings utopia, and no matter your views*

*the pareto frontier has room, for you to be okay*

*[so hold out hope, and don't give up, for help is on the way!](https://forum.questionablequesting.com/threads/the-erogamer-original-complete.5465/page-254#post-2474589)*

*these wishes are not useful now, these traits suboptimal*

*i prefer who i was before i had to take this role*

*this decade we have to work hard, as much as it pains me*

*i am here now, this is the world, let's have some dignity*

*but let's keep our deepest wishes, anchored within our soul*

*they're not useful, for the moment, but they're what we fight for*

*when we succeed, we'll set them free, not holding anymore*

*and finally we will not have, to be instrumental*

our deepest wishes


how far are things that care?
-----------------------------

as a follow-up to [my previous post about the universal program](all-claw-no-world.html), let's talk about forces outside of this world. some computations in the [universal program](universal-complete.html) contain [agents running our world](solomonoff-deism.html), and some of them will eventually interfere with the simulation because they care about *us in particular* — though note that /interference doesn't necessarily look like aliens writing a giant message in the sky; picking unlikely enough everett branches to be the ones that keep getting computed is sufficient enough. how far into the universal program are these aliens? importantly, are they early enough, *polynomial enough*, that we'd allocate some reasonable amount of probability to ending up in their interfered version of events?

there's two factors in this. on one hand, "agents that care" about us can probly compute us faster, [by only computing (or even approximating) parts of our world that are necessary to run us](solomonoff-deism.html) — unless most of our realness amplitude is already within intentional computations. on the other hand, we only exist later within those worlds; agents that care about us *do* have to first itself themself, *before* running us. but the added cost of running our world within another world could at most be the cost of composition, and the composition of two polynomial functions is itself polynomial. so, if we stick to [the notion that polynomial computations are the "easy" computations](https://arxiv.org/abs/1108.1791) as a basis for what gets to be real within the universal program, then the us's being simulated by agents that care about us are probly only polynomially far away from "naive" versions of us — [which is still some distance away, but does get us a form of quantum immortality](less-quantum-immortality.html).

in this view, what is it that we're saving the world for? it's to increase the realness amplitude of our survival, but also to secure our self-determination, in case we *don't* want to be at the mercy of things that care, because they might not care in a way we'd want. there is probly *some* sense in which things that care to simulate us in particular have *some* notion that we are interesting, but it might not one that we'd necessarily find ethical; the set of values that entail us existing is larger than the set of values that entail us existing and having an okay time.

what are we to do? as in [simulation hypotheses](simulation-hypotheses.html) and reasons like those mentioned in [*bracing for the alignment tunnel*](bracing-alignment-tunnel.html), and as it feels like [FDT](https://www.lesswrong.com/tag/functional-decision-theory) would dictate, we generally ought to keep doing what seems important. still, this perspective might update how we emotionally feel about things.

how far are things that care?


all claw, no world — and other thoughts on the universal distribution
---------------------------------------------------------------------

the post [*anthropics and the universal distribution*](https://www.lesswrong.com/posts/Hcc9fopx7sRexYhhi/anthropics-and-the-universal-distribution/) (recommended dependencies: [1](https://www.lesswrong.com/posts/RnrpkgSY8zW5ArqPf/sia-greater-than-ssa-part-1-learning-from-the-fact-that-you), [2](https://www.lesswrong.com/posts/GJdymoviRywpBMXqc/sia-greater-than-ssa-part-2-telekinesis-reference-classes), [3](https://www.lesswrong.com/posts/QHDqfpMbb43JDbrxN/sia-greater-than-ssa-part-3-an-aside-on-betting-in), [4](https://www.lesswrong.com/posts/d693Mc4ZDyhkj7wpc/sia-greater-than-ssa-part-4-in-defense-of-the-presumptuous), [5](https://www.lesswrong.com/posts/XiWKmFkpGbDTcsSu4/on-the-universal-distribution)) tries to unify anthropics with the notion of a [universal distribution](https://www.lesswrong.com/posts/XiWKmFkpGbDTcsSu4/on-the-universal-distribution) (whether that be [solomonoff prior](https://www.lesswrong.com/posts/EL4HNa92Z95FKL9R2/a-semitechnical-introductory-dialogue-on-solomonoff-1) or what i'll call the ["levin prior"](http://www.scholarpedia.org/article/Universal_search)) by splitting hypotheses about a reasoner's location among the set of possible worlds as a "world and claw" pair. the "world" part is the hypothesis program as to what world you inhabit, as opposed to counterfactual worlds, and the "claw" part is a program that locates you within that world.

i've proposed [before](udassa-time-steps.html) to stick to just a [universal program](universal-complete.html) as world hypothesis. that is, the "world" is a fixed program, and all of the complexity is in figuring out the "claw" — epistemology, the work of finding out how stuff around you works, becomes *all claw, no world*. in this post, i expand on this view, and explore some ramifications, notably for [formal aligned AI](rough-sketch-formal-aligned-ai.html) design.

one consequence of doing this is that epistemology becomes *all location, no counterfactuals* — nothing is ever ruled out, all programs are considered instantiated in the same *qualitative* sense. the following space-time-realities are all real in the same *qualitative* way ([though not necessarily to the same *quantitative* degree!](ethic-juice-anthropic-juice.html)):

* where you are now, but on a different day.
* a different country.
* the many-worlds everett branch where an electron you just measured has a different spin.
* this world except the moon is made of cheese.
* [rule 30 starting with a single living cell](https://en.wikipedia.org/wiki/Rule_30)
* middle earth from lord of the rings. that one might be stretching it depending on your interpretations of things like magic, but the universal distribution is capable of a lot of stuff.

(see also: [*the ultimate meta mega crossover*](spoiler-fire-upon-deep.html))

if you ignore [acausal weirdness](https://www.lesswrong.com/posts/PcfHSSAMNFMgdqFyB/can-you-control-the-past), these worlds are all causally separate from ours. we didn't make lord of the rings real — we just wrote a bunch of text, and there happens to be a world out there, real in the same way as ours, that we'd consider to be accurately described by that text. but like a [library of babel](https://en.wikipedia.org/wiki/The_Library_of_Babel) of worlds, all other variants that we *don't* describe are also real. the only thing we *can* do to affect which worlds get more [juice](ethic-juice-anthropic-juice.html) than others is choosing to compute some of them but not others. and our making decisions in this world and affecting the way it continues to run "normally" is just a particular case of this, just like *someone continuing to live* is a particular case of [*a sequence of moral-patient-instants each causing a similar-to-themselves moral-patient-instant to get instantiated in a world like theirs*](existential-selfdet.html).

and, just like acausal/anthropic stuff doesn't save us ([1](https://www.alignmentforum.org/posts/RhAxxPXrkcEaNArnd/notes-on-can-you-control-the-past), [2](https://www.lesswrong.com/posts/EKu66pFKDHFYPaZ6q/the-hero-with-a-thousand-chances), [3](https://www.lesswrong.com/posts/rP66bz34crvDudzcJ/decision-theory-does-not-imply-that-we-get-nice)), it turns out that despite any ethical implications that *the cosmos being a universal program* might have, the [expected utility](https://www.lesswrong.com/posts/7J3ywHzWnghRtdpHQ/on-expected-utility-part-1-skyscrapers-and-madmen) probly plays out about the same, just like it probly plays out about the same under most interpretations of quantum mechanics, regardless of whether other everett branches are real. these things might get you to *care* differently, but mostly not in any way you can do anything about (unless something is looking at us from outside a computation of us and cares about the way we care about things, but it'd be hard to reason about that).

there are nice consequences for decision theory, however: [functional decision theory (FDT)](https://www.lesswrong.com/tag/functional-decision-theory), which wants you to cooperate with other instances of FDT not just across spacetime and everett branches but also across counterfactual worlds, might become simpler when you "flatten" the set of counterfactual worlds to be the same kind of thing as the set of spacetime locations and the set of everett branches.

nevertheless, [some things are realer than others](limiting-real-universes.html). so, what *is* the measure of [realness juice/amplitude](ethic-juice-anthropic-juice.html), which any living person right now probly has more of than gandalf? i feel like it *ought* to be something to do with ["time steps"](udassa-time-steps.html) in the universal program, because it doesn't feel like there could be any other measure which wouldn't eventually just become a more complex version of time steps. the reason there's more of me than gandalf in the universal program, even though it *eventually* contains about as many me's as it contains (variations on a particular as-detailed-as-me interpretation of) gandalf's (whether that quantity is infinite or [not](finite-patients.html)), is that the me's tend to occur *before* the gandalf's — or, to be more formal, at sets of timesteps that are *earlier in the universal program or more compressible* than gandalf's. or, more testably: the reason i see coherent stuff rather than noise when i look at a monitor, even though there are more ways to arrange pixels on a monitor that i'd interpret as noise than as coherent stuff, is that the instances of me seeing coherent stuff must tend to occur *at sets of timesteps that are earlier or more compressible* than those of instances of me seeing noise.

given this quantitative measure, can we re-capture a qualitative notion of "is this real or not"? this is where computational complexity can come in to help us. the excellent [*why philosophers should care about computational complexity*](https://arxiv.org/abs/1108.1791) argues that things computable in polynomial time are, in a meaningful sense, *essentially* easier than things only computable in exponential time. if we apply this to time steps, and if it is *earlier sets of time steps* rather than *more compressible sets of time steps* which counts, then our world is real and lord of the rings (assuming it is a polynomial world) can be said to be real, in a sense that worlds whose physics require solving NP-complete problems or [PSPACE problems](https://en.wikipedia.org/wiki/Closed_timelike_curve#Consequences) to progress, can *not* be said to be real. but i suspect that this doesn't actually track observation that much, because worlds in which [people get mind-controlled](https://www.lesswrong.com/posts/r7f58E8A85xLgWuqG/contact-with-reality) into believing NP-complete problems are being solved are probly polynomial themselves (though less common than worlds without such mind control, i'd expect).

note that this does make *our* world weird, because we [seem](https://en.wikipedia.org/wiki/Quantum_supremacy#Progress_in_the_21st_century) to be able to solve [BQP](https://en.wikipedia.org/wiki/BQP) computations. maybe BQP=BPP, or maybe the cosmos runs on a *quantum* solomonoff prior? or maybe, despite how unintutive that feels, it takes this kind of physics for [anthropic reasoning](anthropic-reasoning-coordination.html) to occur? or maybe i'm [being mind-controlled](https://www.lesswrong.com/posts/r7f58E8A85xLgWuqG/contact-with-reality) or [fictional](simulation-hypotheses.html) or who knows what else.

there are now two issues that arise to make a universal program prior usable, even theoretically:

* in a sense, a consequence of this is that the [UDASSA](https://www.lesswrong.com/posts/Hcc9fopx7sRexYhhi/anthropics-and-the-universal-distribution/) notion of "first simulated a world, then extract me" can in general be flattened into "just simulate me" — which also captures more intentional simulations such as those of ["solomonoff deism"](solomonoff-deism.html). but what *is* a me? what does an extracted me look like? it can't be just be *any* representation of my observations, because otherwise i'd be observing a blank monitor rather than one with *some* contents on it. to take a concrete example, let's say i'm chatting with someone who starts the sentence "I'm from…", and i'm trying to predict whether the next word they'll say — call it [x](javascript:alert(%22x%22)) — is more likely to be "California" or "[Nuuk](https://en.wikipedia.org/wiki/Nuuk)". the comparison can't just be [K⁻("I'm from California")−K⁻("I'm from Nuuk")](javascript:alert(%22K%5Cu207b%28%5C%5Ctext%7B%5C%22I%27m%20from%20California%5C%22%7D%29%20-%20K%5Cu207b%28%5C%5Ctext%7B%5C%22I%27m%20from%20Nuuk%5C%22%7D%29%22)) (with [K⁻](javascript:alert(%22K%5Cu207b%22)) a [simplicity measure](rough-sketch-formal-aligned-ai.html)), because this probly favors "Nuuk" even though in practice i'd expect to hear "California" a lot more. it feels like what we're heading for at this point would be some kind of [P(o|p)](javascript:alert(%22P%28o%20%7C%20p%29%22)) function, where [o∈{"I'm from California","I'm from Nuuk"}](javascript:alert(%22o%5Cu2208%5C%5C%7B%5C%5Ctext%7B%5C%22I%27m%20from%20California%5C%22%7D%2C%5C%5Ctext%7B%5C%22I%27m%20from%20Nuuk%5C%22%7D%5C%5C%7D%22)) is the observation in a format that makes sense to me (english text strings), and [p](javascript:alert(%22p%22)) is some kind of "prior" relating those to contexts in which they'd be percieved. programs that produce "I'm from Nuuk" have higher amplitude than programs that produce "I'm from California", but programs that produce *me observing* "I'm from California" have higher amplitude than programs that produce *me observing* "I'm from Nuuk".
* let's say we're launching an attempt at an aligned AI based on [QACI](qaci.html) based on [this post](rough-sketch-formal-aligned-ai.html) with a given (question, answer, observation) tuple. if the AI simply fills the future with question-answer intervals engineered so that they'd dominate most of the solomonoff-space of programs, then it can hijack its own decision process. in a sense, this is just a special case of [demons in the solomonoff prior](https://www.lesswrong.com/posts/Tr7tAyt5zZpdTwTQK/the-solomonoff-prior-is-malign). which is a neat simplification of alignment! by "flattening" counterfactual worlds and everett branches to be the same kind of thing as objects that are distant in spacetime, we've managed to describe the alignment problem in a way that captures counterfactual adverserial agents ("demons") and factual future unaligned AIs in the same category. now we just need a solution that takes care of both.

i feel like extracting a notion of causality within the universal program, one that would let us determine that:

* stuff outside our past lightcone can't causate onto us yet
* decohered everett branches don't causate onto one another
* two different simulations of conway's game of life on my computer don't causate on one another

would be useful here — though it might need to be able to measure "non-strict" probabilistic causation when needed.

we can't just base this on time, because in any [universal program](universal-complete.html) that is sequentially implemented (such as a turing machine), different implementations of our world will have different events occur at different points in time. using a parallel model of computation such as graph rewriting might shed *some* light on which phenomena causate each other and which are being computed in parallel in a causally isolated manner, but it would miss some others: as an extreme example, a [homomorphically encrypted simulation](homomorphically-encrypted-computations.html) of our world would make its internal causal graph unobservable to the outside, even though there's still real causal independences going on *inside that world*. so sticking to the simple and sequential paradigm of turing machines will force us to develop more clever but more general notions of causal dependence.

next, whatever measure we build, if we weren't dealing with adverserial intelligences we could just do a big sum weighed by simplicity and hope that the signal from the things we care about wins out, as with something like [argmaxa∑cα(c)⋅f(c,a)](javascript:alert(%22%5C%5Cunderset%7Ba%7D%5C%5Ctext%7Bargmax%7D%5C%5Csum_%7Bc%7D%20%5Cu03b1%28c%29%5Cu22c5f%28c%2Ca%29%22)) with [c](javascript:alert(%22c%22)) being the "claws", weighing the value of action [a](javascript:alert(%22a%22)) in each possible claw-situations [c](javascript:alert(%22c%22)) by some factor [α(c)](javascript:alert(%22%5Cu03b1%28c%29%22)). but because we're dealing with (potentially superintelligent!) adverserial agents, we have to make *really sure* that the *undesired* results from whichever [f](javascript:alert(%22f%22)) we use to weigh actions is drowned out by sufficiently low [α(c)](javascript:alert(%22%5Cu03b1%28c%29%22))s, that the overall signal that determines the [argmax](javascript:alert(%22%5C%5Ctext%7Bargmax%7D%22)) is from the *desired* [f(c,a)](javascript:alert(%22f%28c%2Ca%29%22))s. as an example: in [my attempt](rough-sketch-formal-aligned-ai.html) at formalizing [QACI](qaci.html), we want the weights of carvings that capture the human involved in the original question-answer interval to sufficiently outweigh the weights of the AI filling the future with adverserially-answering "fake" question-answer intervals that would allow its earlier (as well as remote/counterfactual) selves to find actions that make its job easier.

so, what could a causality relationship look like? one difficulty is that one change in one world could end up modifying *pretty much everything everywhere*, but not in a way that "really matters". for example: maybe if world number [i](javascript:alert(%22i%22)) does some operation [a](javascript:alert(%22a%22)) rather than [b](javascript:alert(%22b%22)), all the other worlds end up being computed in the same way, but all shifted by one extra time step into the future.

this is where the *computer science* notions of [simulation](https://en.wikipedia.org/wiki/Simulation_%28computer_science%29) and [bisimulation](https://en.wikipedia.org/wiki/Bisimulation) (which aren't usually quite what i mean by those words, but it's related) might come in, which i intend to learn about next; though i wouldn't be surprised if such a measure might be hacked together just out of kolmogorov complexity again, or something like it.

as a final note on the universal distribution: i've recently learned that theoretical turing machines augmented with "halting oracles" [give rise to interesting computational classes](https://en.wikipedia.org/wiki/Hyperarithmetical_theory), which in particular let those turing machines do [hypercomputation](https://en.wikipedia.org/wiki/Hypercomputation) in the form of obtaining results that should require an infinite amount of computation, in a finite number of steps. this might enable us to build a universal prior which captures something closer to the full [tegmark level 4 mathematical multiverse](https://space.mit.edu/home/tegmark/crazy.html). though it's not clear to me whether that's actually desired; what would it actually *mean* to inhabit a hypercomputational multiverse? if the halting oracle runs an infinite amount of moral patients in a finite amount of steps, how the hell does the [anthropic and ethics juice](ethic-juice-anthropic-juice.html) work out? i'll be sticking to the *less* uncomputable, regular solomonoff or even levin prior for now, but this question might be worthy of further consideration, unless we get the kind of aligned AI that doesn't require us to figure this out up front.


a rough sketch of formal aligned AI using QACI
----------------------------------------------

in this post, i put forth some of my current thoughts about the shape of a formal aligned AI using [QACI](qaci.html) for its decision — "decision" in the singular here, as this is sufficient when the AI's decision can be "run me again but with these diffirent inputs". as it turns out, this doesn't require solving as many things as i'd thought — it seems like QACI might be general enough to delegate picking a decision theory and solving embedded agency to the counterfactual consideration of the past user.

we'll posit:

* as a convention, we'll use a prime [′](javascript:alert(%22%5C%5Cprime%22)) to denote counterfactual values, and we'll be denoting questions [q](javascript:alert(%22q%22)), and answers [r](javascript:alert(%22r%22)) (for "response") to avoid confusion with [a](javascript:alert(%22a%22)) for actions.
* the AI is denoted [G:ℕ×ℕ×ℕ→A](javascript:alert(%22G%5C%5C%2C%3A%5C%5C%2C%5Cu2115%5Cu00d7%5Cu2115%5Cu00d7%5Cu2115%5Cu2192A%22)), taking as input an observation as well as the user's original question and answer, denoted [q](javascript:alert(%22q%22)) and [r](javascript:alert(%22r%22)). it returns an action from the set of all possible actions, [A](javascript:alert(%22A%22)).
* [K⁻:ℕ→[0;1]](javascript:alert(%22K%5Cu207b%5C%5C%2C%3A%5C%5C%2C%5Cu2115%5Cu2192%5B0%3B1%5D%22)) is gonna be a simplicity measure based on [kolmogorov complexity](https://en.wikipedia.org/wiki/Kolmogorov_complexity), where [∑x∈ℕK⁻(x)=1](javascript:alert(%22%5C%5Csum_%7Bx%5Cu2208%5Cu2115%7DK%5Cu207b%28x%29%3D1%22)).
* all sets of functions [A→B](javascript:alert(%22A%5Cu2192B%22)) will be countable sets of computable functions. in particular, [W≔()→⊥](javascript:alert(%22W%5C%5Ccoloneq%28%29%5Cu2192%5Cu22a5%22)) will be the set of computable hypotheses for worlds, represented as non-halting programs taking no input with [()](javascript:alert(%22%28%29%22)) and [⊥](javascript:alert(%22%5Cu22a5%22)) being respectively the [unit](https://en.wikipedia.org/wiki/Unit_type) and [bottom](https://en.wikipedia.org/wiki/Bottom_type) types.
* finally, we'll implicitely "cast" mathematical objects as natural numbers wherever appropriate, given that they're all sampled from countable sets anyways. when they're cast to or from natural numbers, assume a reasonable bijection between their type and [ℕ](javascript:alert(%22%5Cu2115%22)).

we'll define the following:

a "carver" function [C:W×ℕ→2(ℕ×ℕ→W)×ℕ×(W→ℕ)](javascript:alert(%22C%5C%5C%2C%3A%5C%5C%2CW%5Cu00d7%5Cu2115%5Cu21922%5E%7B%28%5Cu2115%5Cu00d7%5Cu2115%5Cu2192W%29%5Cu00d7%5Cu2115%5Cu00d7%28W%5Cu2192%5Cu2115%29%7D%22)) which returns a set of tuples of:

* a function [tx](javascript:alert(%22t_x%22)) for extracting a piece of data "in the same way" as [x](javascript:alert(%22x%22)) is in [w](javascript:alert(%22w%22)) but from any other world
* a piece of data [tw\x](javascript:alert(%22t_%7Bw%5C%5Cbackslash%20x%7D%22)) that represents "everything else" than [x](javascript:alert(%22x%22)) in the world
* a function [tw](javascript:alert(%22t_w%22)) for counterfactually putting another piece of data [x′](javascript:alert(%22x%5C%5Cprime%22)) back in [w](javascript:alert(%22w%22)), alongside [tw\x](javascript:alert(%22t_%7Bw%5C%5Cbackslash%20x%7D%22))

this is done by splitting a world [w](javascript:alert(%22w%22)) into the piece of data [x](javascript:alert(%22x%22)), and "everything else", denoted [tw\x](javascript:alert(%22t_%7Bw%5C%5Cbackslash%20x%7D%22)). in practice with arbitrary other worlds, [tx](javascript:alert(%22t_x%22)) would return "garbage" the immense majority of the time, but the hope is that given a same carving [txq,twq](javascript:alert(%22t_x%5Eq%2Ct_w%5Eq%22)) for the question [q](javascript:alert(%22q%22)), a same carving [tqr,twr](javascript:alert(%22t_q%5Er%2Ct_w%5Er%22)) for the answer would work enough time to give a signal that would tend to beat the overall noise of the failing cases.

[C(w,x)≔{(tw,tw\x,tx)|tw∈ℕ×ℕ→W,tw\x∈ℕ,tx∈W→ℕ,∀w′∈ℕ:tx(w′)=x⇔w′=w,∀x′∈ℕ:tw(tw\x,x′)=w⇔x′=x}](javascript:alert(%22%5C%5Cbegin%7Bsplit%7DC%28w%2Cx%29%5C%5C%2C%5C%5Ccoloneq%5C%5C%2C%5C%5C%7B%26%28t_w%2Ct_%7Bw%5C%5Cbackslash%20x%7D%2Ct_x%29%5C%5C%3B%7C%5C%5C%5C%5C%26t_w%5Cu2208%5Cu2115%5Cu00d7%5Cu2115%5Cu2192W%2Ct_%7Bw%5C%5Cbackslash%20x%7D%5Cu2208%5Cu2115%2Ct_x%5Cu2208W%5Cu2192%5Cu2115%2C%5C%5C%5C%5C%26%5Cu2200w%5C%5Cprime%5Cu2208%5Cu2115%3A%5C%5C%3Bt_x%28w%5C%5Cprime%29%3Dx%5C%5C%3B%5Cu21d4%5C%5C%3Bw%5C%5Cprime%3Dw%2C%5C%5C%5C%5C%26%5Cu2200x%5C%5Cprime%5Cu2208%5Cu2115%3A%5C%5C%3Bt_w%28t_%7Bw%5C%5Cbackslash%20x%7D%2Cx%5C%5Cprime%29%3Dw%5C%5C%3B%5Cu21d4%5C%5C%3Bx%5C%5Cprime%3Dx%5C%5Censpace%5C%5C%7D%5C%5Cend%7Bsplit%7D%22))

we'll define [QACI:W×ℕ×ℕ×ℕ→2ℕ×ℝ](javascript:alert(%22%5C%5Ctext%7BQACI%7D%5C%5C%2C%3A%5C%5C%2CW%5Cu00d7%5Cu2115%5Cu00d7%5Cu2115%5Cu00d7%5Cu2115%5Cu21922%5E%7B%5Cu2115%5Cu00d7%5Cu211d%7D%22)), the [question-answer counterfactual interval](qaci.html) device used to consider answers to counterfactual questions [q′](javascript:alert(%22q%5C%5Cprime%22)), given a world hypothesis [w](javascript:alert(%22w%22)) and a known question [q](javascript:alert(%22q%22)) and answer [r](javascript:alert(%22r%22)):

[QACI(w,q,r,q′)≔{(txr(twq(tw\xq,q′)),K⁻((twq,txq,twr,txr)))|(twq,tw\xq,txq)∈C(w,q),(twr,tw\xr,txr)∈C(w,r)}](javascript:alert(%22%5C%5Ctext%7BQACI%7D%28w%2Cq%2Cr%2Cq%5C%5Cprime%29%5C%5C%2C%5C%5Ccoloneq%5C%5C%2C%5C%5C%7B%28t_x%5Er%28t_w%5Eq%28t_%7Bw%5C%5Cbackslash%20x%7D%5Eq%2Cq%5C%5Cprime%29%29%2CK%5Cu207b%28%28t_w%5Eq%2Ct_x%5Eq%2Ct_w%5Er%2Ct_x%5Er%29%29%29%7C%28t_w%5Eq%2Ct_%7Bw%5C%5Cbackslash%20x%7D%5Eq%2Ct_x%5Eq%29%5Cu2208C%28w%2Cq%29%2C%28t_w%5Er%2Ct_%7Bw%5C%5Cbackslash%20x%7D%5Er%2Ct_x%5Er%29%5Cu2208C%28w%2Cr%29%5C%5C%7D%22))

note how [K⁻](javascript:alert(%22K%5Cu207b%22)) measures the simplicity of all four [t](javascript:alert(%22t%22)) functions together so as to favor them being simple but also similar, but ignores the simplicity of the "rest of the world" [tw\x](javascript:alert(%22t_%7Bw%5C%5Cbackslash%20x%7D%22)) values..

finally, we can define our AI [G:ℕ×ℕ×ℕ→A](javascript:alert(%22G%5C%5C%2C%3A%5C%5C%2C%5Cu2115%5Cu00d7%5Cu2115%5Cu00d7%5Cu2115%5Cu2192A%22)) as a function of [q,r](javascript:alert(%22q%2Cr%22)) but also an observation [o](javascript:alert(%22o%22)) which could, in practice, be anything that lets the AI as well as the user better locate themselves in the set of possible worlds.

[G(q,r,o)≔argmaxa∈A∑w∈W(r′,α)∈QACI(w,q,r,(w,a))K⁻(w)⋅α⋅max(0,min(1,r′(q,r,o)))](javascript:alert(%22G%28q%2Cr%2Co%29%5C%5Ccoloneq%5C%5Cunderset%7Ba%5Cu2208A%7D%5C%5Ctext%7Bargmax%7D%5C%5Csum_%7B%5C%5Cbegin%7Bsplit%7Dw%5Cu2208W%5C%5C%5C%5C%28r%5C%5Cprime%2C%5Cu03b1%29%5Cu2208%5C%5Ctext%7BQACI%7D%28w%2Cq%2Cr%2C%28w%2Ca%29%29%5C%5Cend%7Bsplit%7D%7DK%5Cu207b%28w%29%5Cu22c5%5Cu03b1%5Cu22c5%5C%5Ctext%7Bmax%7D%280%2C%5C%5Ctext%7Bmin%7D%281%2Cr%5C%5Cprime%28q%2Cr%2Co%29%29%29%22))

what's going on here, is for each world hypothesis [w](javascript:alert(%22w%22)) weighed by its simplicity [K⁻(w)](javascript:alert(%22K%5Cu207b%28w%29%22)) we're finding carvings for [q](javascript:alert(%22q%22)) and [r](javascript:alert(%22r%22)) that let us inject a counterfactual question and get a counterfactual answer, favoring simpler carvings using [α](javascript:alert(%22%5Cu03b1%22)). the counterfactual rerun of the past user is given the world hypothesis [w](javascript:alert(%22w%22)) being considered, as well as the action [a](javascript:alert(%22a%22)) — we're *not* giving the user [q,r,o](javascript:alert(%22q%2Cr%2Co%22)) because that would make the counterfactual question guaranteed to be strictly larger than the original question, which seems like it might break things, and is not absolutely necessary. the user gets to return an arbitrary function to be evaluated, to which [q,r,o](javascript:alert(%22q%2Cr%2Co%22)) are actually passed. we *are*, however, clamping the result of [r′](javascript:alert(%22r%5C%5Cprime%22)) inside [[0;1]](javascript:alert(%22%5B0%3B1%5D%22)) and then multiplying it by [K⁻(w)⋅α](javascript:alert(%22K%5Cu207b%28w%29%5Cu22c5%5Cu03b1%22)) ourselves, so as to avoid erroneous carvings getting to influence the decision process too much by returning arbitrary large values. the hope here is merely that desired carvings generally outweigh undesired carvings, and especially adverserial carvings.

ways this could go wrong include:

* the carvings could fail to sufficiently select the actual instances of the past-user in the world; in particular, [txr](javascript:alert(%22t_x%5Er%22)) could be returning so much "garbage" when given counterfactual worlds [twq(tw\xq,q′)](javascript:alert(%22t_w%5Eq%28t_%7Bw%5C%5Cbackslash%20x%7D%5Eq%2Cq%5C%5Cprime%29%22)) that are different from [w](javascript:alert(%22w%22)) that the signal from the carvings that *do* work end ups completely drowned out.
* the carvings locate question-answering users *anywhere in the world*, including in the future. this allows for adverserial intelligences to occupy most of the set of simple and coherently-answering carvings, thus steering our AI away from following the actual user's decisions. a solution to this would be to sufficiently strongly favor carvings that select question-answering processes that are causally upstream of the AI itself; that is to say, causally upstream of [q,r,o](javascript:alert(%22q%2Cr%2Co%22)).
* weird implementation details as for how turing machines are encoded could dominate most of the signal of values returned by [r′](javascript:alert(%22r%5C%5Cprime%22)). in addition, the user could be unsure as to how to return a meaningful scalar given a particular potential action. these points could both be partially addressed by passing to the carvings pairs of actions [a1,a2](javascript:alert(%22a_1%2Ca_2%22)) for all possible pairs of different actions from [A](javascript:alert(%22A%22)), and have the counterfactual user select a preferred action, rather than relying on a scalar returned about a single action. if [r′(…,a1,a2)](javascript:alert(%22r%5C%5Cprime%28%5Cu2026%2Ca_1%2Ca_2%29%22)) and [r′(…,a2,a1)](javascript:alert(%22r%5C%5Cprime%28%5Cu2026%2Ca_2%2Ca_1%29%22)) agree as to which of [a1](javascript:alert(%22a_1%22)) and [a2](javascript:alert(%22a_2%22)) is the preferable action, then that's an actual signal that it is preferable. adding more "symmetries" might make noise cancel itself out.
* i don't think we get to require [C](javascript:alert(%22C%22)) to return carvings that work for arbitrarily large counterfactual payloads, nor do we need to. in practice, i expect a constant finite (though large) bit length is to be used for [x](javascript:alert(%22x%22)).
* there's a lot of sets-of-all-programs being sampled from here, leaving a lot of room for [demons in the solomonoff prior](https://www.lesswrong.com/posts/Tr7tAyt5zZpdTwTQK/the-solomonoff-prior-is-malign) if those are a concern. typically i tend to favor decision theoritic solutions to those, and maybe a correct [QACI](javascript:alert(%22%5C%5Ctext%7BQACI%7D%22)) implementation would return action-functions [r′](javascript:alert(%22r%5C%5Cprime%22)) which would depend on whatever decision theory is correct, such that this can be delegated? but it *feels* like this system has ways to go wrong before that, in what programs get to control most of the "mass" returned by [QACI](javascript:alert(%22%5C%5Ctext%7BQACI%7D%22)) to begin with.

this is of course highly uncomputable. the intent here, is to use something like [logical induction](https://www.lesswrong.com/tag/logical-induction) to approximate good results to this function. what makes me hopeful that a powerful AI can make helpful guesses as to what actions this process would find, if it is indeed aligned, is that *even i*, a mere human mind, feel like i can make some helpful guesses as to what actions this process would find.


just enough spoilers for *a fire upon the deep* to read a yudkowsky fanfic
--------------------------------------------------------------------------

[*The Finale of the Ultimate Meta Mega Crossover*](https://www.fanfiction.net/s/5389450/1/The-Finale-of-the-Ultimate-Meta-Mega-Crossover) is a fanfiction that i think is pretty great, written by eliezer yudkowsky. it has major spoilers for two books: the excellent [*Permutation City* by greg egan](https://en.wikipedia.org/wiki/Permutation_City) which i love and thoroughly recommend, and [*A Fire Upon the Deep* by vernor vinge](https://en.wikipedia.org/wiki/A_Fire_Upon_the_Deep) which i enjoyed.

because:

* i think many would consider that latter book too large a dependency to read the fanfic,
* it doesn't take that many spoilers about it to enjoy the fanfic — whereas it does take a lot of spoilers about *Permutation City* if you haven't read that,
* many people i know have read *Permutation City* but not *A Fire Upon the Deep*,

i'm writing this post where i give just enough spoilers for someone who hasn't read it but *has* read *Permutation City* to enjoy the fanfic.

my general recommendation would be: read *Permutation City* if you haven't, then read this post, then maybe read *A Fire Upon The Deep* if this post has made you interested enough in it, and then go read [yudkowsky's fanfic](https://www.fanfiction.net/s/5389450/1/The-Finale-of-the-Ultimate-Meta-Mega-Crossover).

---

the book's setting is pretty interesting. it's a science-fiction adventure set in our galaxy, with a peculiar limitation: given an information system of a particular level of capability — such as a human mind, a superintelligence, or an advanced computer program — it can only exist above a certain "Zone of Thought", a geographic region of the galaxy. if they move to a lower zone, closer to the center of the galaxy, then they start either being reduced in capability or breaking down altogether.

these levels go, in increasing capability and increasing distance from the center of the galaxy:

* *The Unthinking Depths*, where nothing of much intelligence can exist
* *The Slow Zone*, where basic computers and human minds can function but computers are still not advanced enough to do the computations necessary to do FTL travel and communication
* *The Beyond*, where computers are capable of capable of much more and FTL communication and travel are possible
* *The Transcend* (not pictured in the map below) where superintelligences — called "Powers" in the book — can exist, moslty at peace with each other.

![](spoiler-fire-upon-deep.jpg)

(this map of the galaxy is included at the very start of the book)

at the start of the story, a Power called "The Blight" appears in the Transcend, and starts attacking other superintelligences. the book follows (among others) two characters aboard a ship headed down towards lower zones of thought to look for a way to defeat the Blight.

aboard the ship are notably two humans:

* Ravna Bergnsdot, a fairly normal human
* Pham Nuwen, a human who used to serve a Power called the *Old One* by being its interface to interact with humans. the Old One, just before being killed by the Blight, left in his mind fragments that he can't make sense of yet, but are expected to become useful as time goes.

and that's, i think, about all you need to know about *A Fire Upon The Deep* to go and enjoy [The Finale of the Ultimate Meta Mega Crossover](https://www.fanfiction.net/s/5389450/1/The-Finale-of-the-Ultimate-Meta-Mega-Crossover).

just enough spoilers for


CYOAs and futurism
------------------

there is a community for the creation and playing of games where one is to select a bunch of options, some favorable and some disfavorable, and imagine what an ensuing situation would look like. these CYOAs — named after, but not quite the same as, ["choose your own adventure" books](https://en.wikipedia.org/wiki/Gamebook) — can be found on a couple of subreddits, namely [r/makeyourchoice](https://www.reddit.com/r/makeyourchoice/) (sfw) and [r/nsfwcyoa](https://www.reddit.com/r/nsfwcyoa/) (nsfw).

CYOAs i enjoy playing tend to be pretty complex, with many sets of choices and points systems — typically, picking an option costs points when the choice is favorable and rewards points when the choice is disfavorable. they often involve choosing one's own situation in the world or choosing what kind of a world one is to inhabit; they are most interesting when they offer both of those possibilities.

for example, a CYOA might offer options that lead one to choose between being a poor peasant or a famous wizard on the one hand, and might offer options that lead to inhabiting worlds such as a cyberpunk dystopia, a comfy high-fantasy world, or merely a modified version of our base reality.

choices, and the scale and meaning of their consequences, have to be interpreted because a CYOA's consequences are to be simulated in one's own mind, rather than mechanically ran by a program as they are for video games. as a result, one ends up having to do something like [extrapolating what someone intended](https://www.lesswrong.com/tag/coherent-extrapolated-volition) in a balanced way, which is a fun exercise.

things get especially interesting when one has to tradeoff their own quality of life vs the kind of world they'll end up in, the latter of which will impact everyone else that lives in that realm.

i try to play those games as if my choices were actually going to be implemented, and i find myself typically implementing a preference ordering which is, from most to least preferred option:

* utopia for everyone
* okaytopia for me, utopia for everyone else
* extinction for me, utopia for everyone else
* utopia for me, okaytopia for everyone else
* okaytopia for everyone
* extinction for me, okaytopia for everyone else
* utopia for me, extinction for everyone else
* okaytopia for me, extinction for everyone else
* extinction for everyone
* anything involving worse-than-extinction (i don't want to think about sortings between variations of those)

note that "extinction" can be ambiguous — how much can i change of my own mind (as many CYOAs allow) before it's not me anymore, or even not really a full person anymore?

this is a framework which, while pretty removed from the actual life choices i have to make in real life, still feels interesting for thinking about choices and how they impact the future, as well as general utilitarianism and world optimization given a top-down but still constrained perspective.

and, while i've recently put aside [utopia design](%E2%88%80V.html) in favor of [just letting values be extrapolated](surprise-you-want.html), CYOAs still get my mind thinking about how choices about a world work out in the long run. this can include choices [about the nature of reality](generalized-adding-reality-layers.html) that CYOAs sometimes ask you to make about what kind of world you'll inhabit, and can be [as fundamental as](implementing-the-platonic-realm.html) "will i choose for the world to have magic?" or "can i make it that the world has [infinite inhabitable future](hope-infinite-compute.html)?" — which i think are not entirely irrelevant to think about, if we are to get a better idea of [what problems we need to solve](confusion-about-alignment-requirements.html) to get [the best kind of utopia](utopia-scopes.html).

CYOAs and futurism


let's stick with the term "moral patient"
-----------------------------------------

"moral patient" means ["entities that are eligible for moral consideration"](https://en.wikipedia.org/wiki/Moral_agency#Distinction_between_moral_agency_and_moral_patienthood). as [a recent post i've liked](https://www.lesswrong.com/posts/HoQ5Rp7Gs6rebusNP/superintelligent-ai-is-necessary-for-an-amazing-future-but-1) puts it:

> And also, it’s not clear that “feelings” or “experiences” or “qualia” (or the nearest unconfused versions of those concepts) are pointing at the right line between moral patients and non-patients. These are nontrivial questions, and (needless to say) not the kinds of questions humans should rush to lock in an answer on today, when our understanding of morality and minds is still in its infancy.
>
>

in this spirit, i'd like us to stick with using the term "moral patient" or "moral patienthood" when we're talking about the set of things worthy of moral consideration. in particular, we should be using that term instead of:

* "conscious things"
* "sentient things"
* "sapient things"
* "self-aware things"
* "things with qualia"
* "things with experiences"
* "things that aren't p-zombies"
* "things for which there is something it's like to be them"

because those terms are hard to define, harder to meaningfully talk about, and we don't in fact know that those are what we'd ultimately want to base our notion of moral patienthood on.

so if you want to talk about the set of things which deserve moral consideration outside of a discussion of what precisely that means, don't use a term which you feel like it *probably is* the criterion that's gonna ultimately determine which things *are* worthy of moral consideration, such as "conscious beings", because you might in fact be wrong about what you'd consider to have moral patienthood under reflection. simply use the term "moral patients", because it is the term which unambiguously means exactly that.


logical vs indexical dignity
----------------------------

[MIRI's *Death with Dignity* post](https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy) puts forward the notion of "dignity points":

> the measuring units of dignity are over humanity's log odds of survival - the graph on which the logistic success curve is a straight line. A project that doubles humanity's chance of survival from 0% to 0% is helping humanity die with one additional information-theoretic bit of dignity.
>
>

but, as [*logical and indexical uncertainty*](https://www.lesswrong.com/posts/SFLCB5BgjzruJv9sp/logical-and-indexical-uncertainty) puts it, there are two different kinds of uncertainty: uncertainty over our location within things that exist, called **indexical** uncertainty, and uncertainty over what gets to exist in the first place, called **logical** uncertainty.

the matter of *there existing many instances of us*, can occur not just thanks to the many-worlds interpretation of quantum mechanics, but also thanks to other multiverses like [tegmark level 1 and reasonable subsets of tegmark level 4](https://space.mit.edu/home/tegmark/crazy.html), as well as various [simulation hypotheses](simulation-hypotheses.html).

i think that given [the *logical and indexical uncertainty* post](https://www.lesswrong.com/posts/SFLCB5BgjzruJv9sp/logical-and-indexical-uncertainty)'s take on risk aversion — "You probably prefer the indexical coin flip" — we should generally aim to create logical dignity rather than indexical dignity, where logical uncertainty includes things like "what would tend to happen under the laws of physics as we believe them to be". if there's a certain amount of indexical uncertainty and logical uncertainty about a plan, the reason we want to tackle the logical uncertainty part by generating logical dignity, is so that what's left is indexical, and so it will go right *somewhere*.

as a concrete example, if your two best strategies to save the world are:

* one whose crux is a theorem being true, which you expect is about 70% likely to be true
* one whose crux is a person figuring out a required clever idea, which you expect is about 70% likely to happen

and they have otherwise equal expected utility, then you'll want to favor the latter strategy, because someone figuring out something seems more quantum-determined and less set-in-stone than a theorem being true or not. by making logical stuff be what you're certain about and indexical stuff be what you're uncertain about, rather than the other way around, you make it so that in the future, *some place* will turn out well.

(note that if our impact on utopia is largely indexical, then it might feel like we should focus more on reducing [S-risk](https://en.wikipedia.org/wiki/Suffering_risks) if you're e.g. negative utilitarian, because you want utopia *somewhere* but hell *nowhere* — but if [god isn't watching](solomonoff-deism.html) to stop computing timelines that aren't in their interest, and if we are to believe that we should do the normal expected utility maximization thing across timelines, then it [probly shouldn't actually change what we do](https://www.lesswrong.com/posts/7J3ywHzWnghRtdpHQ/on-expected-utility-part-1-skyscrapers-and-madmen), merely just how we feel about it)

logical vs indexical dignity


wonky but good enough alignment schemes
---------------------------------------

i'll say that an AI alignment scheme is "aligned enough" not just if it consists of building a [fully aligned singleton (FAS)](outlook-ai-risk-mitigation.html) which i'd trust to do what's good at any level of capabilities; i'd also say that an alignment scheme is "aligned enough" if it leads, even indirectly, to the construction of such a FAS.

one potential way we could get this is by using some kind of assistant AI — perhaps a system that uses GPT or something — to figure out how to build a FAS. the assistant might not be [eventually aligned](ai-alignment-curves.html); if it had enough capability, it migth realize that actually it wants to kill everyone. but we'd be relying on it being, for as long as we use it, weak enough to not do such a thing.

this kind of "wonky alignment scheme" that is "aligned enough" but *goes through* using temporarily aligned AIs, where we need to *know that they're weak* in order for the scheme to work, might end up useful given that such a temporarily aligned AI might be much easier to build than [an eventually aligned, let alone continuously aligned](ai-alignment-curves.html), AI.

(maybe, actually, what we should be doing with our limited time and resources is not building FAS, but melting GPUs or something else like that. if we had a temporarily-aligned assistant AI which we get to charge with tasks, the task we should want to give it is a very general aligned goal such as [satisfying its past-user](outer-alignment-past-user.html), such that the AI imagining the past-user-under-reflection would be able to consider all those plans and pick the one that actually needs work, rather than working on what *we* think is most important.)

while [i try to do work that directly contributes to the creation of a FAS](outlook-ai-risk-mitigation.html), indirect "wonky" approaches such as using large language models in order to accelerate alignment research have their place too — it's not like one is particularly less hopeless than the other. my main source of skepticism about them is the requirements above, about the intermediary system being weak — that is *not* an assumption that i like to have to make about AIs that would have to be in some sense more capable than us, which they'd have to be if they are to make a difference.

wonky but good enough alignment schemes


generalized wireheading
-----------------------

many systems "want" to ["wirehead"](https://www.lesswrong.com/tag/wireheading) — which is to say, they want to hijack, and maximize, their reward signal.

humans often want to. not always, but sometimes, and this might be true even under reflection: some people (believe they) truly axiomatically only care to be in a state where they're satisfied, others [have values about what actually happens in the world](https://mindingourway.com/the-stamp-collector/) (which is actually possible and meaningful to do!).

[reinforcement learning](https://en.wikipedia.org/wiki/Reinforcement_learning) AIs such as [AIXI](https://en.wikipedia.org/wiki/AIXI) want to wirehead: they want to just do whatever will maximize their reward. if there is a function in place that looks at the amount of happiness in the world and continuously rewards such an AI by that much, then the AI will do whatever is easiest, whether that's *do what makes that function return the highest value*, or *replace the function with a constant returning the maximum value*. (if it does so consequentially, such as by observing that it's more likely to get even more reward in the future by taking over the world, then [it'll still do just that](https://en.wikipedia.org/wiki/Instrumental_convergence), so we can't necessarily count on wireheading to stop world-consuming AIs.)

(it's true that ["reward is not the optimization target"](https://www.lesswrong.com/posts/pdaGN6pQyQarFHXF4/reward-is-not-the-optimization-target) of *learned policies* — AIs that are first trained in an RL environment, and then deployed into the world without that reward mechanism. but i think it is true of agents that *continuously get rewarded and trained even after deployment*.)

some bad philosophical perspectives claim to want society to wirehead: they want to get a society where everyone is as satisfied as possible with how things are, without realizing that a goal like that is easily hijacked by states such as *everyone wants to do nothing all day*, or where everyone is individually wireheaded. we do not in fact want that: in general, we'd like the future to be interesting and have stuff going on. it is true that by happenstance we have not historically managed to turn everyone into a very easily satisfied wireheaded person ("zombie"), but that shouldn't make us falsely believe that, purely by chance, this will never be the case. if we want to be sure we robustly don't become zombies, we have to make sure we actually don't implement a philosophy that would be most satisfied by zombies.

the solution to all of those, is [to bite the bullet of value lock-in](surprise-you-want.html). there are meta-values that are high-level enough that we do in fact want them to guide the future — even *within* the set of highly mutable non-axiomatic values, we still have preferences for valuing some of those futures over others. [past user satisfaction](outer-alignment-past-user.html) embodies this well as a solution: it is in fact true that i should want (the [coherent extrapolated volition](https://www.lesswrong.com/tag/coherent-extrapolated-volition) of) my values to determine all of the future light-cone, and this recursively takes care of everything — including adding randomness/happenstance *where it ought to be, purposefully*.

just like [alignment](alignment-optimization-processes.html), making the mistake of saying "i just want *people in the future* to be satisfied!" is a mistake that can isomorphically be found in many fields, and in fact is not where we should want to steer the future, because its canonical endpoint is just something like wireheading. we want (idealized, meta-)value lock-in, not the satisfaction of whatever-will-exist. **fundamentally, we want the future to satisfy the values of *us now*, not *people/things later***.

of course, those values of us now [happen to be fairly cosmopolitan](https://www.lesswrong.com/posts/HoQ5Rp7Gs6rebusNP/superintelligent-ai-is-necessary-for-an-amazing-future-but-1) and entail, instrumentally, that people in the future indeed largely be satisfied. but this ought to ultimately be under the terms of our current cosmopolitan (meta-)values, rather than a blind notion of just filling the future with things that get what they want without caring what those wants are.

generalized wireheading


"humans aren't aligned" and "human values are incoherent"
---------------------------------------------------------

"humans aren't aligned" or "our values are not coherent" are concerns that i occasionally hear about the odds of [AI alignment](https://en.wikipedia.org/wiki/Ai_alignment) research being able to accomplish what it intends to do.

it is to be rembered that "aligned" is a two-place relation — we ask whether a system is aligned with another system. a given human is aligned with themself, by definition, to the extent that they has values at all. it is true that humans are not fully aligned to one another, but there is significant overlap, and there is general agreement that [AI doom](ai-doom.html) is worse than most expectable [value handshakes](https://www.lesswrong.com/tag/values-handshakes)/[bargaining](https://www.lesswrong.com/posts/vJ7ggyjuP4u2yHNcP/threat-resistant-bargaining-megapost-introducing-the-rose) or even [indirect universalism](surprise-you-want.html). this is why we don't observe alignment researchers trying to beat one another to be the one whose values tile the universe — any kind of effort in that direction would likely hamper the total chances of alignment being successful to begin with, and that's what we're all trying to avoid.

on the topic of value coherency, it may be true that some of my preferences might not easily or even at all be formulable as a formal utility function that a fully aligned AI ought to maximize. but i have [meta-values](surprise-you-want.html), and i'm reasonably confident that my meta-values entail my incoherent preferences being satisfied to, whatever i'd find that to mean under enough reflection (as that is itself a value extrapolation process i have meta-values about). at the very least, [some kind of libertarian framework](%E2%88%80V.html) where i can do more or less [whatever i want](everything-is-okay.html) while being [free from moloch](https://slatestarcodex.com/2014/07/30/meditations-on-moloch/) surely must be sufficient: if we build a world where you can do whatever you want, then that should include whatever you're doing now to satisfy those incoherent values.

don't get me wrong, [alignment is not looking great](outlook-ai-risk-mitigation.html). but i believe it is a solvable problem, and i don't believe these concerns are particularly big hurdles.


a safer experiment than quantum suicide
---------------------------------------

in [tegmark's page about multiverses](https://space.mit.edu/home/tegmark/crazy.html), he mentions that you could experimentally test the validity of the many-worlds interpretation of quantum mechanics by running a machine that uses a quantum random number generator a decide whether to kill you, with overwhelmingly large odds of doing so.

in my perspective, the way this works is that if the many-worlds interpretation of quantum mechanics is true *and [anthropic juice](ethic-juice-anthropic-juice.html) reallocates itself to your surviving instances at any point in time* (which is in my opinion the only way for quantum immortality to work, and might make sense if there is an objective fundamental arrow of time, which i think is plausible), then you still have a reasonable probability of observing existing after the experiment. if not, then almost all your [anthropic juice](ethic-juice-anthropic-juice.html) is located *before* you can the experiment. based on that, a given local observer is able to gain bits of evidence about how the world works, in a manner similar to what [the doomsday argument](https://www.lesswrong.com/tag/doomsday-argument) says about our chance of extinction.

however, if instances of *[anthropic](https://www.lesswrong.com/posts/RnrpkgSY8zW5ArqPf/sia-greater-than-ssa-part-1-learning-from-the-fact-that-you) reasoning* are the things which are able to deduce information from their existence — as opposed to "observers" — as i suspect in [*anthropic reasoning coordination*](anthropic-reasoning-coordination.html), then a much safer experiment could be done: instead of making it likely that the machine kills you, simply make it likely that the machine brainwashes you into being committed to never doing anthropic reasoning ever again — or simply have the machine tell you to make such a commitment, and then do it yourself, if you're able to stick to it. then, it is not *you* but *you undertaking anthropic reasoning* which is undergoing quantum suicide, and that's theoretically enough to gain the same bits of evidence as proper full quantum suicide.

(not that, in the grand scheme of things, an observer is expected to be able to do anything with that evidence — a reasonable understanding of [how ethics juice relates to anthropics juice](ethic-juice-anthropic-juice.html) as well as [weird causality in decision theory](https://www.alignmentforum.org/posts/RhAxxPXrkcEaNArnd/notes-on-can-you-control-the-past) will probly function in a way that makes such evidence entirely unusable with regards to what actions to take to maximize something like expected utility [in the one way that makes sense](https://www.lesswrong.com/posts/7J3ywHzWnghRtdpHQ/on-expected-utility-part-1-skyscrapers-and-madmen), from what i understand. at most, one can use that evidence to know how to *feel* about how things might be *in the grander scheme of [what's real](limiting-real-universes.html)*, but not to impact what exists unless [strange acausal shenanigans are at play](https://slatestarcodex.com/2018/04/01/the-hour-i-first-believed/))

a safer experiment than quantum suicide