*(this is a work of fiction)*

everything is okay
------------------

it's been four years since the singularity. someone pressed the button, and their preferences were implemented across the cosmos. i don't think anyone knows *who* pressed the button; that is probly how they'd like things to be. maybe they don't know themself.

i wake up and cuddle with my partners for a while. we live in a log cabin, which was currently somewhere near forests and mountains, somewhere in washington state i think.

i don't know if it's *actually* washington state, because i don't care about being uploaded. it could be that 10¹⁰⁰ objective years have passed since the singularity and that now that it's got the compute it needs, Elua has started running our simulations. it could be that it's entirely remade the cosmos into a shape we cannot conceive. or it could be that this is actually earth, in its full physical continuity, actually four objective years after the singularity. all of these are okay to me.

"Elua" is what i've called the superintelligent singleton that now rules over everything. i call it that because of [an old pre-singularity writing](https://slatestarcodex.com/2014/07/30/meditations-on-moloch/), which i still like. most of my friends have picked up on that name, but i hear some people in the nearby small town call it god. many people out there probly don't even know about Elua, because they would prefer not to. i'm sure even for them, everything is okay.

i wonder if someone out there cares about being uploaded. i wonder if their preferences have been satisfied, and if so how. what i *am* pretty confident about, is that whatever the situation, somehow, they are okay.

one of my partners goes to put on some music. we have something like a vinyl player. i like that the device is legible — the sound wave that we hear has been encoded into the physical shape of the object, and so the whole way the device works is understandable by a human mind. we don't really *need* a player of course, we could just magically hear whatever we wanted, without any artefacts. but i like things this way. i like the rustic experience of being a human manipulating tools. and it's not like they would ever actually get in the way of what i want to any extent which wouldn't be okay.

sometimes i talk with Elua in my dreams. i could talk to her anywhere, but dreams seem like a nice context for it. i've used lucid dreams to reshape my body a few times, and then woken up with my new body. it's mostly similar to the one i had before the singularity; i want to stay in a mostly grounded human experience, at least for now. maybe one day i'll explore much more alien forms of existence, as i'm sure many are doing already; but for the moment, this is what feels okay.

certainly, i don't suffer any grave illnesses; i do get a bit under the weather sometimes, because i think it's okay for that to happen.

i decide to check on how my friend is doing. i open the cupboard and find my crystal ball, put it on the table, and say the name of my friends. when some piece of technology has to be illegible, i like having it presented as magic. if this *is* the inside of a simulation that is getting ad-hoc intervened with for that device to work, then it might as well be magic anyways. both this kind of reliable magic, and other more [mysterious](https://www.youtube.com/watch?v=VHrTTgmB_3w) forms of magic, are okay.

i speak the name of a friend to the ball, and then make an effort to focus on it. the focus does not do anything to the ball, but it makes it that my sensorial input of the ball is much amplified, and my input of the rest reduced. my friend is immediately available for communication — whether by either of us getting paused long enough for the other to become available, or because they actually were — and after greeting me, they reports what it's been like to expand their intellect a millionfold and study ever expanding maths. they tell me about some unimaginably elegant theorems they've found out about. as they say this, my focus makes it that i can see my friend as if they were standing in front of me, and they point at mathematical shapes floating in the air. i semi-consciously let them enter my mind, and the mathematical structures permeate my understanding. they are not visual, but truly mathematical, as if a logic-percieving module was attached to my mind to percieve mathematical logic directly. i appreciate my friend's discoveries, but i also discreetly chuckle at how cute they are when they get excited about it. i tell them about how i've been taking it easy but, percieving that they're not particularly interested, i let them get back to their stuff. our goodbye is a bit awkward, but that's okay.

by a flick of the mind, i retract my focus from the crystal ball, at which point the smell of toast strikes me. after getting my bearings for a second or two i put it back in the cupboard, and head to the small living room to see what my partners have been cooking. it's toasted bread with some sort of cheesey-creamy stuff on it. i don't know if the cheese appeared at the store magically, or if comes from fake animals that exist for the sake of people who want to partake of farming, but i don't have to worry about anything like meat industry scale suffering. something like that would just not happen — everything that does happen is okay.

we decide to go into town. the town is pretty small — not many people are in the streets. various stores are open. most give stuff away for free, while some sell it for money. money has become strange since the singularity. some people choose to care about it, and there *are* some scarce things it can track, such as the use of someone's time; but it doesn't make sense to track much else, such as material resources. so most people kind of just don't bother. even land in an absolute sense is not scarce; it seems like Elua's solution to some people such as me wanting to live on something like a single earth, has been to add more space in between existing space. the total amount of land that "earth" consists of may very well have doubled since the singularity, by now. somehow, it's all arranged such that traveling to somewhere you wanna go leads you there, but travelling aimlessly does get you to many new places. we can even get lost sometimes, when we're okay with that.

it is mid-winter, but i can't be bothered to put on something warm; nevertheless, i barely feel cold: i'm semi-consciously opting for it to feel just a bit chilly, reducing the pain of cold but still getting the informational sensation of it, the way some people pre-singularity would be born with the full information but none of the sensation of pain. in any case, feeling just a bit chilly is okay.

we go to the adventure guild, where i posted a quested for a playstation 1. i did give some currency as a reward — moreso to not feel bad that i'm using someone's time, even though the people who fulfill quests are all pretty much happy to do so — they're people who want their life to provide value and meaning to others, and for most of them those others *must actually* be real people; and it wouldn't be okay for Elua to just create people out of nowhere to create an artificial demand, so it doesn't. and so, there is a genuine market mismatch, in more people wanting to fulfill quests. despite the fact that adventurers are the ones gaining most of the value from this system, the custom has remained that it is the quest poster who pays the adventurer — it's not like money is very important anyways, so what might in a previous era have been considered terrible market inefficiency, is now more than okay.

the language used in town is basically english, with some internet meme slang thrown in there. it also has some pretty local characteristics, but hasn't diverged that much — people value using english as a lingua franca around here, and as for me and my partners, we reserve for private use the artistic constructed language we've developed together. i like english, it's good. and sometimes, people around don't speak it, and we just find something that they do know, or ask a local who can help translate, or even kinda just gesture at each other and work things out like that. anyone could just choose to have their brain understand any language they want, or even communicate by thought, but i like sticking to communities that share this humancore artefact that is highly imperfect verbal communication. even when there are misunderstandings, it's not a big deal; it's okay.

just as we arrive in the store, an adventurer comes back with the genuine playstation 1 we'd requested. probly not a coincidence, probly fortunate timing arranged by Elua. well, it's not like the timing would've been correct anyways: some time dilation has certainly taken place, considering the adventurer tells us how it's taken them several weeks to find that playstation due to them committing to not using Elua's help, while on my end i remember posting the quest just the day before. the adventurer recounts to us his adventure finding the playstation 1, driving to various pawn shops in the area, and asking people. i had made the quest kind of hard: i had requested a playstation that had existed physically continuously since the singularity, not one that had been created out of thin air or even constructed since the singularity, nor a pre-singularity one that had been copied into multiple instances. but he did find one, and the journey he was on made me feel, as i head back home with my new playstation, like this playstation now carries an extra bit of meaning.

as soon as i get home i simply plop the playstation in front of the chalkboard that we use as a TV, grab the controller, and put on the copy of metal gear solid i'd obtained a while ago. it's just as great a game as i'd rembered, and while i'm focusing on it, one of my partners watches while the other asks if they can play with me, and when i say yes they sit where i'm sitting, the two of us temporarily occupying the same physical location, so that we can hold the controller at the same time while our minds intermingle as they open to one another. we could have also magically duplicated the controller and taken turns, but it is more fun this way, each of us taking controlling the playing character to various degrees, and also having a shared piece of mind keeping track of what we're doing together, so as to not have to verbally communicae our intentions. we are fully focused on the game and on each other and we scarcely feel time go by, such that when my other partner calls to our attention to get dinner, it's already dark out. the days go by fast when we take things this easy, but it's okay; it's not like time is scarce.

we have some great tartiflette, and then head to bed, to chit chat and cuddle before sleep. we talk about what to do tomorrow, and decide to have the cabin move somewhere unexpected during our sleep, so we can go explore some new surroundings. maybe we'll wake up in a different continent, and we'll do some adventurous hiking, reassured by the feeling that whatever happens, *everything will be okay*.

### afterword

writing utopia is important. it's not just a good way to get people to start actually thinking about how good things [could actually be](utopia-scopes.html), but also, if something like [PreDCA](predca.html) is the kind of benevolent superintelligent singleton we get, we have to start acting in ways that make us people who tend to express our values. we have to cultivate ourselves and each other to wish for good worlds, and realize how much we'd dislike bad ones. we need to help make future benevolent superintelligence's job at realizing our values as easy as it can be, and to make our expressed values clear through our actions, such that if we *do* start up an AI which extrapolates our values from our actions, it gets the correct idea. finally, writing utopian fiction is just plain fun, and i find it good motivation to work on AI risk mitigation: think carrot, not stick.

everything is okay


*(this post has been written for the second [Refine](https://www.alignmentforum.org/posts/5uiQkyKdejX3aEHLM/how-to-diversify-conceptual-alignment-the-model-behind) blog post day. thanks to [vanessa kosoy](https://www.lesswrong.com/users/vanessa-kosoy), [adam shimi](https://www.lesswrong.com/users/adamshimi), sid black, [artaxerxes](https://www.lesswrong.com/users/artaxerxes), and [paul bricman](https://www.lesswrong.com/users/paul-bricman) for their feedback.)*

PreDCA: vanessa kosoy's alignment protocol
------------------------------------------

in this post, i try to give an overview of [vanessa kosoy](https://www.lesswrong.com/users/vanessa-kosoy)'s new alignment protocol, *Precursor Detection, Classification and Assistance* or *PreDCA*, as she describes it in [a recent youtube talk](https://www.youtube.com/watch?v=24vIJDBSNRI).

keep in mind that i'm not her and i could totally be misunderstanding her video or misfocusing on what the important parts are supposed to be.

the gist of it is: the goal of the AI should be to **assist** the **user** by picking policies which maximize the user's **utility function**. to that end, we characterize what makes an **agent** and its **utility function**, then **detect** agents which could potentially be the user by looking for **precursors** to the AI, and finally we **select** a subset of those which likely contains the user. all of this is enabled by infra-bayesian physicalism, which allows the AI to reason about what the world is like and what the results of computations are.

the rest of this post is largely a collection of mathematical formulas (or informal suggestions) defining those concepts and tying them together.

an important aspect of PreDCA is that the mathematical formalisms are *theoretical* ones which could be given to the AI as-is, not necessarily specifications as to what algorithms or data structures should exist inside the AI. ideally, the AI could just figure out what it needs to know about them, to what degree of certainty, and using what computations.

the various pieces of PreDCA are described below.

---

**[infra-bayesian physicalism](https://www.lesswrong.com/posts/gHgs2e2J5azvGFatb/infra-bayesian-physicalism-a-formal-theory-of-naturalized)**, in which an agent has a hypothesis `Θ ∈ □(Φ×Γ)` (note that `□` is *actually* a square, not a character that your computer doesn't have a glyph for) where:

* `Φ` is the set of hypotheses about how the physical world could be — for example, different hypotheses could entail different truthfulness for statements like "electrons are lighter than protons" or "norway has a larger population than china".
* `Γ` is the set of hypotheses about what the outputs of all programs are — for example, a given hypothesis could contain a statement such as "2+2 gives 4", "2+2 gives 5", "the billionth digit of π is 7", or "a search for proofs that either P=NP or P≠NP would find that P≠NP". note that, as the "2+2 gives 5" example demonstrates, these don't have to be correct hypotheses; in fact, PreDCA relies a lot on entertaining counterfactual hypotheses about the results of programs. a given hypothesis `γ∈Γ` would have type `γ : program → output`.
* `Φ×Γ` is the set of pairs of hypotheses — in each pair, one hypothesis about the physical world and one hypothesis about computations. note that a given hypothesis `φ∈Φ` or `γ∈Γ` is not a single statement about the world or computationspace, but rather entire descriptions of those. a given `φ∈Φ` would say *everything there is to say* about the world, and a given `γ` would specify the output of *all possible programs*. they are not to be stored inside the AI in their entirety of course; the AI would simply make increasingly informed guesses as to what correct hypotheses would entail, given how they are defined.
* `□(Φ×Γ)` assigns degrees of beliefs to those various hypotheses; in [infra-bayesianism](https://www.lesswrong.com/s/CmrW8fCmSLK7E25sa), those degrees are represented as "infra-distributions". i'm not clear on what those look like exactly, and a full explanation of infra-bayesianism is outside the scope of this post, but i gather that — as opposed to scalar bayesian probabilities — they're meant to encode not just the probability but also uncertainty about said probability.
* `Θ` is one such infra-bayesian distribution.

vanessa emphasizes that infra-bayesian physicalist hypotheses are described "from a bird's eye view" as opposed to being agent-centric, which helps with [embedded agency](https://www.lesswrong.com/tag/embedded-agency): the AI has guesses as to what the whole world is like, which just happens to contain itself somewhere. in a given hypothesis, the AI is simply described as a part of the world, same as any other part.

next, **a measure of agency** is then defined: a "[g-factor](https://en.wikipedia.org/wiki/G_factor_%28psychometrics%29)" `g(G|U)` for a given agent `G` and a given utility function (or loss function) `U`, which is defined as `g(G|U) = -log(Pr π∈ξ [U(⌈G⌉,π) ≥ U(⌈G⌉,G*)])` where

* a policy is a function which takes some input — typically a history, i.e. a collection of pairs of past actions and observations — and returns a single action. an action itself could be advice given by an AI to humans, the motions of a robot arm, a human's actions in the world, what computations the agent chooses to run, etc.
* `ξ` is the set of policies which an agent could counterfactually hypothetically implement.
* `G` is an agent; it is composed of a program implementing a specific policy, along with its cartesian boundary. the policy which the agent `G` actually implements is written `G*`, and the cartesian boundary of the agent is written `⌈G⌉` — think of it as the outline separating the agent from the rest of the world, across which its inputs and outputs happen.
* `U : cartesian-boundary × policy → value` is a utility function, measuring how much utility the world would have if a given agent's cartesian boundary contained a program implementing a given policy. its return value is typically a simple scalar, but could really be any ordered quantity such as [a tuple of scalars with lexicographic ordering](https://en.wikipedia.org/wiki/Lexicographic_preferences).
* `U(⌈G⌉,G*)` is the utility produced by agent `G` if it would execute the actual policy `G*` which its program implements
* `U(⌈G⌉,π)` is the utility produced by agent `G` hypothetically executing some counterfactual policy `π` — if the cartesian boundary `⌈G⌉` contained a program implementing policy `π` instead of implementing the policy `G*`.
* `Pr π∈ξ [U(⌈G⌉,π) ≥ U(⌈G⌉,G*)]` is the probability that, for a random policy `π∈ξ`, that policy has better utility than the policy `G*` its program dictates; in essence, how bad `G`'s policies are compared to random policy selection

so `g(G|U)` measures how good agent `G` is at satisfying a given utility function `U`.

given `g(G|U)`, we can **infer the probability that an agent `G` has a given utility function `U`**, as `Pr[U] ∝ 2^-K(U) / Pr π∈ξ [U(⌈G⌉,π) ≥ U(⌈G⌉,G*)])` where `∝` means "is proportional to" and `K(U)` is the kolmogorov complexity of utility function `U`.

so an agent `G` probably has utility function `U` if it's relatively good at satisfying that utility function and if that utility function is relatively simple — we penalize arbitrarily complex utility functions notably to avoid hypotheses such as "woah, this table is *really good* at being the exact table it is now (a complete description of the world would be an extremely complex utility function).

we also get the ability to **detect what programs are agents** — or more precisely, how agenty a given program is: `g(G|U) - K(U)` tells us how agenty a program `G` with utility function `U` is: its agentyness is its g-factor minus the complexity of its utility function.

**"computationalism and counterfactuals"**: given a belief `Θ ∈ □(Φ×Γ)`, the AI can test whether it thinks the world contains a given program by examining the following counterfactual: "if the result of that program was a *different* result than what it actually is, would the world look different?"

for example, we can consider the [AKS](https://en.wikipedia.org/wiki/AKS_primality_test) prime number testing algorithm. let's say `AKS(2^82589933-1)` returns `TRUE`. we can ask "if it returned `FALSE` instead, would the universe — according to our computational hypothesis about it — look different?" if it *would* look different, then that means that someone or something in the world is running the program `AKS(2^82589933-1)`.

to offer a higher-level example: if we were to know the [true name](https://www.alignmentforum.org/posts/FWvzwCDRgcjb9sigb/why-agent-foundations-an-overly-abstract-explanation) of suffering, [described as a program](generalized-values-testing-patterns.html), then we can test whether the world contains suffering by asking a counterfactual: let's say that every time suffering happened, a goldfish appeared (somehow as an output of the suffering computation). if that were the case, would the world look different? if it *would*, then it contains suffering.

this ability to determine which programs are running in the world, coupled with the ability to measure how agenty a given program is, lets us find what agents exist in the world.

**agentic causality**: to determine whether an agent `H`'s executed policy `H*` can causate onto another agent `G`, we can ask whether, if `H` had executed a different policy `π≠H*`, the agent `G` would receive different inputs. we can apparently get an [information-theoritic](https://en.wikipedia.org/wiki/Information_theory) measure of "how impactful" `H*` is onto agent `G` by determining how much mutual information there is between `H*` and `G`'s input.

**precursor detection**: we say that an agent `H` is a precursor of agent `G` if, counterfactually, `H` could have prevented `G` from existing by executing a policy which is different from its actual policy `H*`.

we can now start to build a definition that lets the AI **detect** and then **classify** who its user is.

**user detection**: the AI is trying to determine who its precursor program could be. but, given a hypothesis for "the thing producing *these* policies is the precursor", there are infinitely many different programs which could output the observed policies. so we choose the one which is the most agenty, using the function described above: `g(H|U) - K(U)`.

note that while we extrapolate the user's actions into the future, the user is defined as an ***instant**-agent* which *precedes* the AI's existence; such that the actual physical person's actual future actions does not change what utility function the AI should try to maximize. this stops the AI from influencing the user's utility function: we define the user strictly in the past, causally outside of the AI's light-cone. the AI is maximizing the utility function of the instant-user which causated its existence, not that of the continuously existing user-over-time.

**user classification**: for each potential precursor hypothesis, we have now selected a program that models them and their respective utility functions. we then eliminate some hypotheses as to what the user could be — notably to avoid acausal attacks by remote aliens or [counterfactual demons](https://www.lesswrong.com/posts/Tr7tAyt5zZpdTwTQK/the-solomonoff-prior-is-malign) — using the following criteria:

* the user is a precursor which, as of the AI's startup, should be in close causal proximity to it. for example, a human with their hands on the keyboard controlling the AI is more directly causally related than another human in the neighboring room.
* the g-factor of the user should be in a range where we would expect humans to lie. hopefully, this helps avoid selecting superintelligent acausal attackers, whose g-factor we'd expect to be much higher than that of humans.

finally, we end up with a hopefully small set of hypotheses as to who the user could be; at that point, we simply compose their utility functions, perhaps weighed by the infra-distribution of each of those hypotheses. this composition is the utility function that the AI should want to maximize, by selecting policies which maximize the utility that the world would have if they were enacted, to the best of the AI's ability to evaluate.

---

vanessa tells us how far along her protocol is, as a collection of pieces that have been completed to various degrees — green parts have gotten some progress, purple parts not as much. "informal PreDCA" is the perspective that she provides in her talk and which is hopefully conveyed by this post.

![](predca.svg)

finally, some takeaways that can be taken from this informal PreDCA perspective:

* infra-bayesian physicalism is a powerful toolbox for formalizing agent relationships (in a way that reminds me of my [bricks](goal-program-bricks.html) for [insulated goal-program](insulated-goal-program.html))
* this framework allows for "ambitious" alignment plans — ones that can actually transform the world in large ways that match our values (and notably might help prevent [facebook AI from destroying everything six months later](https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities)) as opposed to "weak safe AI"
* vanessa claims that her approach is the only one that she knows to provide some defenses against acausal attacks

my own opinion is that PreDCA is a very promising perspective. it offers, if not full "direct alignment", at least a bunch of pieces that might be of significant use to general [AI risk mitigation](say-ai-risk-mitigation-not-alignment.html).


carmack predictions
-------------------

[john carmack has started working on AGI](https://en.wikipedia.org/wiki/John_Carmack#Career) and apparently, despite yudkowsky's efforts, he's hard to alignmentpill — as in, convince that [alignment](https://en.wikipedia.org/wiki/Ai_alignment) is a difficult and important matter.

my current model is that, if a very smart person studies the problem of AGI, whether they become alignmentpilled "from the inside" (by working on the problem) is a matter of what order they attack the problem in. if they start from "hmm, what's the input to the AI's motivation/decision-theory system?", then they're a lot more likely to alignmentpill themselves than if they start from "hmm, how to optimize decision-making?". given this and vague things i've heard about carmack, i'll emit the following predictions as to which of those possibilities happen first, conditioned on the assumption that nothing interrupts his work — such someone else making [clippy](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer) first:

* carmack makes AI that kills everyone50%
* carmack becomes alignmentpilled30%
* carmack fails/gives up on AI work10%
* other/unknown10%

(predictions are spoilered so you can make your own guesses without being anchored — click on a prediction to see the percentage)

carmack predictions


guiding your brain: go with your gut!
-------------------------------------

inside you are [two systems](https://www.readthesequences.com/Biases-An-Introduction). system 1 bumbles around, doing some tasks automatically, and heuristically generating feelings — including value-laden ones — about things. "this looks good! that looks evil!"

system 2 is where the explicit reasoning happens — or *tries* to happen, difficult as it is. if you're a rationalist, then system 2 is where you generally try to do your important consequentialist decisions.

when determining what system to use — and how to use it — in various situations, it can be tempting to tend to prioritize system 2. there are good reasons to rely on system 1's "common sense" judgment, for example to avoid [being convinced by reasonable-sounding bullshit](https://slatestarcodex.com/2019/06/03/repost-epistemic-learned-helplessness/). another reason to do it, however, is to avoid being convinced by reasonable-sounding *correct things you don't want to know*. for example, [if demons are trying to blackmail you *even with correct reasoning*](alignment-researchspace-is-malign.html). it's hard to precommit to not succumb to blackmail, because we're only humans. and it's even harder to implement general correct decision theory in system 2; not just because implementing formal software in system 2 is generally unreliable, but also because we might not actually know for sure what the correct decision theory is.

so, one solution could be to approach novel ideas by just kinda bumbling around evaluating things with system 1 "common sense", outright reject blackmail-shaped things without system-2-thinking about them too much, and then start up your system 2 when you need to solve specific problems that seem reasonably benign — including high-level ones. system 2 can also be of good values when deciding what to train system 1 on; you want to keep your heuristics and social influence and such reasonably hygienic and aligned to good and useful things. that may be what it takes [for rationalists to win](https://www.readthesequences.com/Newcombs-Problem-And-Regret-Of-Rationality).

this isn't a strong recommendation or even a claim that i intend to systematically do that myself, just a possibility to give reasonable consideration.

guiding your brain: go with your gut!


alignment research is very weird
--------------------------------

on its face, AI alignment is just the field of study of how we make AI do what we want. seems simple enough.

in practice, it leads to many very strange places. turns out, making an AI that optimizes [anything at all, we don't care what](https://www.gwern.net/fiction/Clippy), is much easier than making it robustly optimize for what we want ([whatever that is](outer-alignment-politics-philosophy.html)). here are some weird questions that come up and seem like they might actually need figuring out in order to build aligned AI:

* [can you control the past?](https://www.lesswrong.com/posts/PcfHSSAMNFMgdqFyB/can-you-control-the-past)
* how to avoid [demons in the space of hypotheses](https://www.lesswrong.com/posts/Tr7tAyt5zZpdTwTQK/the-solomonoff-prior-is-malign)?
* how to [generalize bayesian thinking to anthropics](https://www.lesswrong.com/posts/Hcc9fopx7sRexYhhi/anthropics-and-the-universal-distribution/)?
* [are minimal circuits demon-free?](https://www.lesswrong.com/posts/nyCHnY7T5PHPLjxmN/open-question-are-minimal-circuits-daemon-free)
* [is the cosmos a graph of causal universe-bubbles?](above-paperclips-2.html) and [how are properties inherited between those?](generalized-adding-reality-layers.html)
* [could alignment research be creating even more risk?](against-ai-alignment.html)
* [when do identical/similar computations get more "ethics juice"?](predictablizing-ethic-deduplication.html)
* [are there finitely many moral patients?](finite-patients.html) which ones are [viable](unviable-moral-patient.html)? what the hell even is a moral patient?
* [is BQP=P?](https://en.wikipedia.org/wiki/BQP#BQP,_P,_and_NP) [can we resimulate earth?](finding-earth-ud.html)
* [on what levels are we many-instanced?](https://space.mit.edu/home/tegmark/crazy.html)
* [what's a good decision theory which can reason about its embeddedness?](https://intelligence.org/2018/10/31/embedded-decisions/)
* [how are we to care about existential self-determination?](genuineness-existselfdet-satisfaction-pick2.html)
* [in what sense, if any, is our civilization quantum-immortal?](less-quantum-immortality.html)
* [what kind of utopia could we be aiming at?](utopia-scopes.html)
* [how much compute can we eventually grab?](hope-infinite-compute.html)
* and many others.

it may be that some or most of those questions are irrelevant; for example, it may be that we can just build "dumb AI" that's limited in scope to writing poetry and designing non-smart-AI software, and somehow everyone agrees to only make that kind of AI (as opposed to [facebook AI killing everyone six months later](https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities)). but for the general case where AI is supposed to be *arbitrarily* capacitous, in a way that most AGI labs are pursuing (sometimes even intentionally and explicitely) these questions are relevant — at the very least in the meta sense of "which questions do we actually need to figure out".

alignment research is very weird


alignment researchspace is potentially malign
---------------------------------------------

alignment research leads to [many very strange places](alignment-research-is-very-weird.html). some of those could totally lead alignment researchers, with [enough time](the-peerless.html) or [otherwise augmented](https://www.lesswrong.com/posts/FSmPtu7foXwNYpWiB/on-the-limits-of-idealized-values) thinking, or maybe even with the limited thinking time and capacity we have now, to stumble upon [demonic](https://www.lesswrong.com/posts/Tr7tAyt5zZpdTwTQK/the-solomonoff-prior-is-malign) forces we're [too imperfect](https://www.lesswrong.com/posts/KnPN7ett8RszE79PH/demons-in-imperfect-search) to avoid.

because of this, it isn't just *in[adequate](https://intelligence.org/2018/10/31/embedded-decisions/)ly implemented aligned AI* which [is vulnerable](https://www.lesswrong.com/posts/Tr7tAyt5zZpdTwTQK/the-solomonoff-prior-is-malign); inadequately implemented human cognition might be vulnerable as well. who knows what kind of traps we might fall in, and what the first one to be highly memetic could look like. and also, unlike AI, we can't implement into ourselves better decision theory.

in this landscape it is useful to focus on decision theory, and perhaps on plans which are willing to sacrifice some amount of certainty for the sake of expedience — not just because we're racing with capabilities, but now also to reduce the risk of encountering possibly-memetic demons. maybe we can design "computed-assisted research" software to help us systematically avoid those, or perhaps we can design plans where most of the aspects of alignment are automatically solved by software that bootstraps aligned superintelligent AI, rather than solved manually by humans. in the meantime, memetic/infohazard hygiene and containment policies would be good to develop.

(by the way, this is a good reason to think [clippy](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer) might not even tile the universe with paperclips — it may fall prey to demons before/instead of implementing into itself the devices it needs to avoid them. alignment is not just the work of making the AI's values be our values — it's also the work of making the AI's values resilient and correctly pursued, as opposed to hijackable or value-driftable or subject to whatever other failure modes exist)


trading with superintelligence: a wonky proto-alignment scheme
--------------------------------------------------------------

if we had as long as we want to figure out AI alignment, then we wouldn't worry as much — the problem is that [timelines are short](why-timelines-short.html).

so, what if we traded with the AI? we could make an AI that isn't aligned yet, and we try to only let it have tiny effects on the world — maybe trading some stocks in a limited manner — while having, and telling it about, the following commitment:

when we figure out alignment, then we'll align (possibly new) AI to 99% our values, 1% whatever wants this current AI that we're trading with. if anything threatens to let it escape its box, we'll destroy down said box first.

if we can restrict that AI's ability to impact the world in ways that help it trick us enough, then its remaining option is to help, and impact the world in whatever way maximizes our chances of having the time to figure out alignment and minimizes the chance that we all die of some other AI.

there's a bunch of assumptions going into this:

* it can't meaningfully acausally trade with other AIs or aliens
* its output channel is not sufficient to take over everything deceptively, and it's otherwise [safely boxed](ai-boxing-easy.html)
* we don't look at its outputs in ways that could hack our brains
* we [can actually](https://www.readthesequences.com/Newcombs-Problem-And-Regret-Of-Rationality) make the strong commitment above
* probly many others

but at least it seems like a vaguely plausible plan, [or at least](https://www.alignmentforum.org/posts/ADMWDDKGgivgghxWf/productive-mistakes-not-perfect-answers) one that might inspire better ideas.


essential inequality vs functional inequivalence
------------------------------------------------

i percieve two paradigms of computational/mathematical thinking, which ramificate into a bunch of fields. i believe the core of the distinction is captured by comparing two mathematical perspectives: **essential inequality** such as what distinguishes two vertices in a graph, and **functional inequivalence** such as what distinguishes two general mathematical functions by comparing their sets of outputs for every input.

**functional inequivalence** (hereby **FI**) seems to me like the default: if two objects cannot be distinguished by how they relate to other things, then they might as well be considered the same. for example, `f(x) = 2 * x` and `g(x) = x + x`, or in some sense vertices `a` and `b` in the graph `V={a,b,c}, E={a→a,a→c,b→b,b→c}`

**essential inequality** (hereby **EI**) is the perspective where elements get to have a special *pointer* or *uniqueness essence* that make them essentially different from all others, even when things are completely equivalent if you swap them around. for example, two vertices in any graph are usually thought of in that way; other examples include memory locations in a turing machine's tape, pairs in a LISP (with their `eq?` comparability), and others.

i've notably seen people use **EI** as an argument for worrying about consciousness with regards to teleportation, uploading, or other supposedly continuity-breaking events: instead of the vertices or memory locations being roughly the same ones, or at least being causally "nearby" the ones they were being computed on just before, they're suddenly being computed on a wholly different set of locations/vertices.

as for myself, i usually am on the **FI** side, especially when it comes to fundamental cosmos stuff. for example, i tend to [deduplicate for the purpose of ethics](deduplication-ethics.html). for anthropics, however, [my prior](udassa-time-steps.html) allows for caring not just about first instance, but about *number of instances* of a given experience or moral patient, which could be in contradiction with **FI**. in addition, because most [computational models](https://en.wikipedia.org/wiki/Computational_model) seem to have *some* notion of *essential location* which could be a basis for **EI**, i find myself mildly updated towards **EI** — though i still mostly fall on the **FI** side, just less strongly so.

more importantly, regardless of whether the cosmos fundamentally "has" **FI** or **EI**, i don't believe that consciousness/qualia/soulstuff have their continuity particularly broken when they're moved to being computed elsewhere. i believe that, in order to coherently believe they're being particularly broken, you need to believe in some immaterial magical soul-type thing, and i [have a good reason to not assume that](ruling-out-intuitions-materially-acausal-intuitions.html), on top of occam disfavoring it. thus, i believe there are *only* functional processes, either being computed or not.

(**FI** vs **EI** might have also ramifications for [SIA vs SSA](https://www.lesswrong.com/posts/RnrpkgSY8zW5ArqPf/sia-greater-than-ssa-part-1-learning-from-the-fact-that-you), but it's not at the moment clear to me which ones if any)

one argument in favor of **FI** is that it could be [implemented on top of](generalized-adding-reality-layers.html) **EI**, or that some "implementation details" of the cosmos (such as [persistent data structures](persistent-data-structures-consciousness.html)) de-reify **FI** by automatically deduplicating compute.

see also: [psi rewriting](psi-rewriting.html), in which i offer an **FI** alternative to [wolfram's hypergraph rewriting](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/).


why my timelines are short: all roads lead to doom
--------------------------------------------------

i think [AI doom](https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence) is likely to happen this decade, or maybe the next (assuming no quantum immortality). in this post, i explain why i'm so pessimistic.

to me, there's a big attractor at *AI improving AI*. technology that works finds technology that works better; this happens as soon as some technology is at least a bit good at finding other technology.

here, the technology in question is software, which we're generally [*really bad*](https://www.youtube.com/watch?v=2FuGtDSKOos)  at. what that means is that there are huge low hanging fruit that any AI or *random person designing AI in their garage* can find by just grasping in the dark a bit, to get huge improvements at accelerating speeds.

some people think AI improvement can hit unexpected difficulty bumps. to me, that's not the default, and i don't see any reason to assume it to be true. i expect there to be countless [*ReLU instead of sigmoid*](https://www.lesswrong.com/posts/7im8at9PmhbT4JHsW/ngo-and-yudkowsky-on-alignment-difficulty)-type improvements waiting to happen, pointing fast in the direction of the *AI things that work* attractor. and you don't need all of them: you just need some, and you rapidly find others. **all roads lead to superintelligent AI**.

the state of affairs we observe now ([1](https://github.com/features/copilot/), [2](https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html), [3](https://twitter.com/davisblalock/status/1558347542101839873), etc) is exactly what i'd think being [at the cusp of criticality](https://aiimpacts.org/ai-and-the-big-nuclear-discontinuity/) looks like. the terribleness of our software and AI tech is such that the potential of what's doable with our hardware is *immense* compared to what exists now. if what we can do now by bruteforcing AI is GPT or LaMDA, then what AIs can design once they start designing new stuff *even just a bit above criticality* has plenty of room to get superintelligent, *fast*.

it's all a matter of whether someone and/or something grasps in the dark a bit to find the few improvements necessary to fall very quickly into the capability attractor, and then [we all die](https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence).


the foundation book
-------------------

[this timeline dies](ai-doom.html). if it doesn't, [things don't look good](how-timelines-fall.html).

in asimov's *foundation* series (for which this is mild premise spoilers), some intellectual elite of a civilization put together *The Foundation*, a planet-organization tasked with preserving mankind's knowledge during an expected incoming dark age. it might be worth considering doing something similar in our own timeline: write a large book, with instructions to learn to read it as well as providing the [bases of rationality](https://www.readthesequences.com/), and warning its readers to be wary of AI risk and other existential risks in their civilization. we'll have failed, and we want them to have a better shot.

it needs to largely encourage its propagation, but also some ability for future people to improve upon it. it needs to allow its users to quickly outcompete others. it needs to be physically printed in many copies, to be distributed across the world — including poor countries — to maximize our chances. as the foundation for something more like [dath ilan](https://www.lesswrong.com/tag/dath-ilan) and less like [here](https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities).

a whole organization would be even better, if it can survive — but i feel like a repository of ideas is a more robust format, and communities can form around it and implement it.

the foundation book


*(this post has been written for the first [Refine](https://www.lesswrong.com/posts/D7epkkJb3CqDTYgX9/refine-an-incubator-for-conceptual-alignment-research-bets) blog post day, at the end of the week of readings, discussions, and exercises about epistemology for doing good conceptual research)*

goal-program bricks
-------------------

this is the follow-up to [the Insulated Goal-Program idea](insulated-goal-program.html) in which i suggest doing alignment by giving an AI a program to run as its ultimate goal, the running of which would hopefully realize our values. in this post, i talk about what pieces of software could be used to put together an appropriate goal-program, as well as some example of plans built out of them.

* "[**ems**](https://en.wikipedia.org/wiki/The_Age_of_Em)": uploaded people, who could for example evaluate how much a given situation satisfies our values; if they are uploads of AI alignment researchers and engineers, they could also be put to work on alignment and AI software — all of this *inside* the goal-program.
* "**elves**": neural net models, or patchworks of software likely containing those, designed to be a rough representation of our values, carry a rough subset of our skills, or be some other subset of the human mind. those might have to make do if running ems is either impossible due to for example brain scan technology being unavailable, or if running elves poses less of an [S-risk](https://en.wikipedia.org/wiki/Suffering_risks) than running ems in some situations.
* **collaborative environments**, such as collaborative programming environments or full 3D virtual environments, for **ems** and/or **elves** to work in together. those are instrumental environments designed to let their users develop something.
* "**utopia infrastructure**": pieces of software designed to robustly support [beings](utopia-scopes.html) living together in [utopia](%E2%88%80V.html), as i've previously designed for [a video game idea](game.html) (which [i'm no longer working on](life-refocus.html)). these are places designed for long-term (possibly forever-term) inhabitation by endless persons, under hopefully utopic conditions.
* **program searches**: programs iterating through programspace, typically in order to find worlds or models or programs which match some criteria. just like "a bunch of ems and/or elves programming together", program searches can be used to produce more of the things in this list. that said, [program search finds can find demons](https://www.lesswrong.com/posts/Tr7tAyt5zZpdTwTQK/the-solomonoff-prior-is-malign), which is something to look out for; a general program search utilizing its output for anything must either fully sanitize what it does use, or skip demonic programs to begin with.
* **observer programs**: programs which consume a slice of computation (typically a world simulation) for examination, and maybe even editing, typically by an em or an elf.
* **a simulation of earth** would be useful if it were [somehow](finding-earth-ud.html) obtainable in reasonable computational time. it could serve to extract alignment researchers from it in order to spawn a simulation of them without having to figure out brain scanning; it could be used to create an alternate history where AI researchers are somehow influenced, possibly at an early date; it could also be used to recover the full population of earth in order to give them access to utopia once we have a satisfactory instance of it.
* **a dump of (as much as possible of) the internet**, which could be useful to both locate the earth, or re-extrapolate things like humans or earth or maybe specific persons.

here are some naive examples of outlines for goal-program which seem like they could be okay:

* a simulation of a bunch of researchers, with a lot of time to figure out alignment (as in [the peerless](the-peerless.html)).
* a bunch of elves forever evaluating various light-cones of a program search for worlds, keeping ones with seemingly good contents and discarding ones with seemingly bad contents — although this idea is potentially quite vulnerable to demon-laden worlds.
* a bunch of elves working to, using a copy of the internet, re-extrapolate ems which could then figure out AI alignment
* any of these schemes, except with ems or elves checking at a level above that everything goes well, with the ability to abort or change plans

these feel like we could be *getting somewhere* in terms of figuring out actual goal-program that could contain to valuable outcomes; at the very least, it seems like a valuable avenue of investigation. in addition, [unlike AGI](https://www.alignmentforum.org/posts/72scWeZRta2ApsKja/epistemological-vigilance-for-alignment#Iterability__don_t_mess_it_up), individual many pieces of the goal-program can be individually tested, iterated on, etc. in the usual engineering fashion.

goal-program bricks


anthropic mindfulness
---------------------

in the past few months, especially since reading a lot about [anthropics](https://www.lesswrong.com/posts/RnrpkgSY8zW5ArqPf/sia-greater-than-ssa-part-1-learning-from-the-fact-that-you) and experiencing some psychedelics, i have experienced moments that i'd describe as *anthropic mindfulness*.

they consist of stepping back from automatically processing not just my environment (as in [mindfulness](https://en.wikipedia.org/wiki/Mindfulness)) but also automatically assuming the contents of my mind, so that they become [factored out](anthropic-reasoning-coordination.html) of what feels like normal background environment/assumptions/ground truth.

those moments fill me with feelings of wonder that might be transcribed into words as "woah! what a weird world to exist, and what a weird mind to inhabit! how come the one point of reality that seems real is this one?"

because of my general positive experience of life, they tend to make me optimistic about things. without doing any reasoning, *at least from this moment-location*, reality looks like good things.

it also helps reframe memories as merely artefactual evidence rather than *ground truth*, in the way that they normally feel as do [other ungrounded intuitions](ruling-out-intuitions-materially-acausal-intuitions.html).

anthropic mindfulness


what does it mean to value our survival?
----------------------------------------

let's say that i want to continue living, or that i want everyone currently on earth to continue living. what *does* that value look like, in a formalized format?

in my [goal-program framework](insulated-goal-program.html), i can see two ways to implement this:

* **embedded**: we make the goal-program contain either a full copy of myself/earth, or enough information to [re-find it](finding-earth-ud.html)
* **physical**: we somehow make the values [point](https://www.lesswrong.com/tag/the-pointers-problem) to the real world — and then determine what counts as the thing and/or locate the thing we care about within it

some constraints apply:

* saving [*people who existed in the past* — and maybe *aliens outside of our light cone*](utopia-scopes.html) — seems like it would probly require [resimulating earth](finding-earth-ud.html); hence the **embedded** solution
* the **physical** solution might require significant philosophical work, or might be [actually completely infeasable](tiling-unavoidable.html)
* computational/storage limits might force us to stick to the **physical** solution

i believe there is [some work](https://www.lesswrong.com/posts/gHgs2e2J5azvGFatb/infra-bayesian-physicalism-a-formal-theory-of-naturalized) towards the **physical** solution, so part the reason i tend to focus on the embedded solution is that it seems a lot less explored.


future paths
------------

we are at a node in a state graph (or [MDP](https://en.wikipedia.org/wiki/Markov_decision_process)), where every state points to a bunch of other states, notably by way of:

* irreversible superintelligent singleton implementation, whether it leads to [doom, utopia, or hell](timeline-codes.html)
* [civilizational collapse](how-timelines-fall.html), [smaller X-risks](smaller-x-risk.html), and other things that give us a [mulligan](https://en.wikipedia.org/wiki/Mulligan_%28games%29) for AI risk mitigation
* [pivotal acts](https://arbital.com/p/pivotal/) which let us [flip the gameboard](https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities)

on one hand, booting up irreversible superintelligent singletons should be very carefully considered, as the irreversibility forces us to commit to [a specific system](%E2%88%80V.html), potentially ruling out whole [scopes of utopia](utopia-scopes.html) entirely.

on the other one hand, it is to be kept in mind that, even though the current world sure seems like it has enough [quantum amplitude](quantum-immortality-local-deaths.html) or [anthropic juice](https://www.lesswrong.com/posts/Hcc9fopx7sRexYhhi/anthropics-and-the-universal-distribution/) to feel pretty real, we must be careful of generating civilization-wide (possibly quantum) [micromorts](https://en.wikipedia.org/wiki/Micromort) damaging the realness of valuable future states. it might be that we only have 1 unit of anthropic juice to allocate to future states, some of which gets consumed every time we create a bunch of [dead timelines](timeline-codes.html).

i believe it is useful for people and groups working on [AI risk mitigation](say-ai-risk-mitigation-not-alignment.html) to keep a (mental or physical) picture of this graph, and carefully choose where they want to aim. making the correct consequentialist choice is not a trivial matter, and indeed blindly following what you believe to be your best shot without *looking around* could be a large mistake.

future paths


scopes of utopia
----------------

how good can we make the future? would we prefer 0.1 quantum amplitude of a really good utopia, or 0.2 quantum amplitude of a kinda okay utopia? what does "kind of utopia" even mean?

in this post i list a combinatorial set of possible utopias. i think i want a

* [**sublime** / **concrete**](https://www.lesswrong.com/posts/SLw2MEgxFtiKAqgQ5/actually-possible-thoughts-on-utopia) utopia, where
* **some** / **all living** / **all living and past** / **all possible**
* **humans** / **moral-patient living beings** / **moral-patient information systems**
* **on earth** / **in light cone** / **everywhere**
* **have their abstract values satisfied** / **live** / **live and have their abstract values satisfied**

this lets us get a classification of what various long-term alignment plans could aim for; it also poses a framework to discuss what this perspective might be missing, as well as getting an idea of what we could be aiming for.

of course, ideally, we would prefer to not need to figure this out soon, and instead buy ourselves more time before we commit to [a global utopia shape](%E2%88%80V.html). but we may not have that option, and even if we do, it might be useful to have some idea where we might be wanting to go.

scopes of utopia


*(this post has been written for the first [Refine](https://www.lesswrong.com/posts/D7epkkJb3CqDTYgX9/refine-an-incubator-for-conceptual-alignment-research-bets) blog post day, at the end of the week of readings, discussions, and exercises about epistemology for doing good conceptual research)*

the Insulated Goal-Program idea
-------------------------------

the **Insulated Goal-Program** idea is a framework for [AI alignment](https://en.wikipedia.org/wiki/AI_alignment) which feels more potentially tractable than most other ideas i've seen.

it splits the task of building aligned AI into two parts:

1. building a very intelligent AI which, when running, will have the axiomatic goal of running a program, which we'll call goal-program
2. building said goal-program, such that when ran, it hopefully creates valuable outcomes

the fact that the AI's goal is to run a program, whose functioning it is motivated to run without altering it, lets us design a goal-program that doesn't have to deal with an adverse optimizing superintelligence — it is insulated from the AI's choices.

(or at least, there's supposedly no reason for the AI to run long stretches of variants of that program, because of the computational cost for supposedly no gain)

one way to insulate the goal-program is to make it fully deterministic. ideally, however, we would want it to be able to receive as input the state of the world before the AI modifies the world — which it will [pretty much inevitably](https://en.wikipedia.org/wiki/Instrumental_convergence) do, destroying everything and tiling the universe with computronium dedicated to running the goal-program.

this is how this idea solves the ["facebook AI destroys the world six months later"](https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities) problem: the AI will run the goal-program at any cost, including turning everything that exists into computronium.

but that's okay: the point here is for us, or at least our values, to survive inside the goal-program. that is the bullet i bite to allow this idea to function: i give up on the literal physical world around us, in the hopes that we're satisfied enough with getting to determine what it is that runs on the computronium that everything is turned into.

making the goal-program able to be ran on quantum compute [might allow us to resimulate earth](finding-earth-ud.html) as well as generally gain a lot more compute from the universe, especially if [BQP ≠ P](https://en.wikipedia.org/wiki/BQP#BQP,_P,_and_NP).

this whole framework splits the problem of aligned AI cleanly into two parts: the design of the AI-insulated goal-program, and the design of the AI whose goal will be to run said program. the goal-program's insulatedness lets us design utopias or utopia-finding-programs which don't have to deal with adverseriality from the AI, such as vaguely-friendly-NNs evaluating the quality of simulated worlds, or simulated researchers figuring out alignment with as much time as they need. [i write more about goal-program design here](goal-program-bricks.html).

it also resolves some questions of [embedded agency](https://www.lesswrong.com/tag/embedded-agency): the goal-program is indeed *smaller* than the agent, so it might only need notions of embedded agency resolved for how it thinks about the outside world it's turning to computronium.

the Insulated Goal-Program idea


unviable moral patients
-----------------------

in [∀V](%E2%88%80V.html) i talk about the plausible ethical unviability of human children.

but, there is a broader category of moral patient whose existence i believe must be prevented due to their ethical unviability. for example, a moral patient which strongly desires to suffer continuously and to modify itself in order to never change its mind about that, cannot exist without either their preferences being unreasonably dissatisfied or their existence generating too much suffering.

it might be that such a moral patient cannot coherently exist — but if it can, i still oppose letting it come into existence, due to this issue.

depending on your ethical framework and your valuing of [various things](genuineness-existselfdet-satisfaction-pick2.html), it may be that no, some, or all potential moral patients are unviable. figuring out who is, of course, an important question that might need figuring out.

unviable moral patients


ruling out intuitions about materially acausal things
-----------------------------------------------------

i have an intuition — *not just*, and *preceeding*, a reasoned belief — that i have a weird consciousness-observer-soul-thing. i also have an intuition that moral realism is true, that the arrow of time moves forwards rather than backwards [or sideways](https://web.archive.org/web/20201112014828/http://kim.oyhus.no/QuantumMechanicsForProgrammers.html), that [i am a continuous stream of consciousness rather than an instant-observer](https://opentheory.net/2018/09/a-new-theory-of-open-individualism/) (such that you might particularly [worry about teleportation](https://www.youtube.com/watch?v=nQHBAdShgYI)), that my memories must be true (rather than the universe appearing five minutes ago), and many others.

those can all be ruled out with a simple device: if any of these things were the case, could that causate onto whether such an intuition fires? for all of them, the answer is no: because they are immaterial claims, the fact of them being true or false *cannot* have causated my thoughts about them. therefore, these intuitions must be discarded when reasoning about them.

that does not mean that those statements are all *necessarily* false, just that my intuitions for them cannot be providing bayesian evidence. after all, the amount to which bayesians update a belief based on evidence should be as a function of how likely such evidence is to arise given that the belief be true rather than false — even when the evidence is intuition.


quantum immortality and local deaths under X-risk
-------------------------------------------------

assume quantum immortality, for mankind as a whole when facing [X-risks](https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence). then, depending on two factors,

* the probability `P` with which a given non-doomed timeline becomes doomed — when it passes a point of no return, where the extinction of all persons is guaranteed,
* the average time `T` between the point of no return is reached, and the point at which everyone *actually does* die,

it can be the case that the majority of person-experience happening can be in-between point-of-no-return and actual-extinction (in red below) rather than in continuously surviving (green below)

![](quantum-immortality-local-deaths.svg)

this probably matters both from an [anthropics](https://www.lesswrong.com/posts/RnrpkgSY8zW5ArqPf/sia-greater-than-ssa-part-1-learning-from-the-fact-that-you) and from an ethics perspective. this is a good reason to work on reducing X-risk (reducing `P`) even under the assumption of quantum immortality, if you value knowing that you're probably not uselessly working in an already-doomed timeline, or if you believe that doomed timelines experience particularly more suffering. another way to avoid spending experience in those doomed timelines is to reduce `T`: to make sure that, once doomed, we die as soon as possible.

in addition, if you think forking the timeline costs us [forking bits](forking-bitrate-entropy-control.html) — if you think we can only fork the timeline so much, and we wanna preserve as many forks as we want for utopia — then reducing P becomes more important than reducing T, because you save more "realness juice" or "forking bits" for later, when we've solved AI alignment and start populating the timelines with utopia.

which, thankfully, agrees with the straightforward no-quantum-immortality perspective on X-risks: reducing the chances of it is the important thing.


probability under potential hardware failure
--------------------------------------------

suppose you program a computer to do bayesian calculations. it does a bunch of math with probability numbers attached to, for example, logical belief statements.

but, suppose that each time you access one of those numbers, there is a tiny chance `ε` of hardware failure causing the memory/register to return an erroneous number — such as [cosmic ray bit flips](https://en.wikipedia.org/wiki/Soft_error).

this fact can inform our decisions about how many bits of information we are to store our numbers as. indeed, the computer can never have probability ranges outside of `[ε ; 1-ε]`: the probabilities are clamped by the chance that, while they were computed, a random hardware failure occured.

if a probability is calculated that is a function of many calculations, then the errors can accumulate. the computer might be able to rerun the computation to be more sure of its result, but it will never escape the range `[ε ; 1-ε]`.

this constraint feels to me like it would also limit the number of bits of precision one can meaningfully store: there is only so many ways to combine numbers in that range, with errors at each step of computation, before the signal is lost to error noise. i'm not sure and haven't worked out the math, but it may turn out that arbitrary-precision numbers, for example, are ultimately of no use: given a constant `ε`, there is a constant `f(ε)` maximum number of useful bits of precision.

this issue relates to [the uncertainty of 2+2=4](uncertainty-2+2=4.html): logical reasoning on a computer or on a human is still probabilistic/error-prone, because of hardware failures.


tiling the cosmos might be unavoidable
--------------------------------------

it is possible that the value of "i want this to happen *in this seemingly-top-level material plane*" — rather than in a *some* computation *somewhere*, possibly within arbitrarily many layers of simulation — is incoherent, and simply *cannot* be given to an intelligent enough artificial intelligence. in this view, given the goal "please instantiate a strawberry on this plate, and keep everything else around that plate the same", there is no way to give that goal to a sufficiently intelligent AI such that *killing everyone and tiling the cosmos with computronium running an exact simulation of this world except with a strawberry on the plate* does not satisfy that goal.

this could be the case, for example, if — though not *necessarily* if — the cosmos in some fundamental sense contains [first-class](https://en.wikipedia.org/wiki/First-class_citizen) *cycles of universe-bubbles*, such as [this example with rule 30 and rule 110 causating one another](above-paperclips-2.html).

my intuition is that, if this is the case, then it is actually fine for everyone to be forcibly uploaded. if it is impossible to coherently care about what level of reality we run on, then with sufficient thinking we would find that we don't in fact care about it. if you disagree, if you think you profoundly care about things that are fundamentally not coherently pursuable, then i worry that ethics/alignment might pose to you significant difficulties.

nevertheless, it is possible i may one day end up agreeing with you. if that is to be the case, who knows what the solution is? perhaps to create "dumb" superintelligences that are *merely* smart enough to prevent any other superintelligences from arising — such as by permanently preventing all non-human-brain computation and guaranteeing some integrity of human brains — and then leaving humankind's fate in its own hands.

tiling the cosmos might be unavoidable


isn't it weird that we have a chance at all?
--------------------------------------------

we are facing imminent [AI X-risk](https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence). but, we have a bunch of tools around us to figure out that this is a problem, and to even start thinking about solutions.

we have enough physics to think about heat death, enough computational complexity to think about how NP-complete solutions are probly not reasonable, enough rationality to organize a small movement around AI alignment work and figure out things like solomonoff induction or the malignhood of the universal prior, the ability to do some anthropics, and even a few mild ideas as to what the fuck human values even are.

isn't this kind of weird? it feels to me like most civilizations about to die of AI X-risk would be entirely missing several to most of these; but somehow, unless i'm missing a crucially important unknown unknown field, it does kind of look like we have almost enough to work with in the various fields required. even the geopolitical situation and the public awaraness situation, while disastrous, are not entirely hopeless.

i wonder if this has any meaning, whether it be anthropic or simulation theoritic or otherwise.


an anthropics example
---------------------

in the course of discussing anthropics with a friend — notably the [SIA vs SSA](https://www.lesswrong.com/posts/RnrpkgSY8zW5ArqPf/sia-greater-than-ssa-part-1-learning-from-the-fact-that-you) discussion — i have produced an example case which i believe demonstrates not just the usefulness of anthropics, but also how SIA and SSA can differ. it goes as follows:

suppose you ask the question, "if a civilization were to follow roughly the same technological progress as us, would we expect them to have killed themselves with [AI doom](https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence) by the time of their 2022".

suppose you have reduced your thinking to two hypotheses, which you believe to the following extents:

* `S` (safe) hypothesis: 1/3 chance that with only the technology of up until now, they would still be alive basically-for-sure.
* `R` (risk) hypothesis: 2/3 chance that with only the technology of up until now, they would have a 3/4 chance of having killed themselves.

(to be clear, this pair of hypotheses does not represent my actual beliefs — they are a toy example i provide for the purpose of this post)

here, the question is: which of the hypotheses is true? is it true that, given our technology, they almost certainly would still be alive? or is it instead true that, given our technology, they would only have a 1/4 chance of still being alive?

this question is clearly useful: using anthropics, we might be able to get (from the fact that we exist) information about the risk posed our current level of technology — and such information would surely be useful to reason about the risk posed by near-future technology, as it is roughly similar to ours.

the scenario looks like this:

```
if S hypothesis is true (1/3):
    survive
if R hypothesis is true (2/3):
    if 1/4 chance that we survive despite the risk:
        survive
    else:
        extinct

```

now, is there anywhere where such a scenario has been ran, where we can make an observation? well, yes: *our own* world! we notice that, in our own world and despite our level of technology, we are surviving rather than extinct. how does this observation update us with regards to the prior?

if you are using SIA, you're comparing the *expected number of people in your epistemic situation* (hereby `#ES`) across both possibilities (`S` and `R`). in `S`, there is four times as many expected people in your epistemic situation than in `R` (this is true however you draw the set of "people in your epistemic situation"!). so, you update:

```
SIA(S)
  = (P(S) × #ES(S)) / (P(S) × #ES(S) + P(R) × #ES(R)    )
  = (1/3  × #ES(S)) / (1/3  × #ES(S) + 2/3  × #ES(S) / 4)
  = (1/3  × #ES(S)) / (1/2  × #ES(S)                    )
  = (1/3          ) / (1/2                              )
  = 2/3

```

(`P(x)` is the prior probability of `x`)

so, SIA updates from the fact that you exist, towards `S` from the prior probability of 1/3 to a posterior probability of 2/3.

on the other hand, in SSA you're comparing not the raw `ES`, but the proportion of *your reference class* (hereby `RC`) that is in your `ES`. notice that, *in this scenario*, it does not really matter what kind of reference class you draw up — it could be just you, all humans doing anthropics reasoning, all humans, or all living beings, and the answer would be the same in all cases because AI doom causes *all* of those to be destroyed — the proportion `#ES / #RC` is the same, because none of the hypotheses change the number of, say, you vs humans, or humans vs living beings. either everything dies or everything lives.

the only reference class that could change something here is either distant aliens, or the AI that kills everything itself (or it subsystems) — but for simplicity, we'll rule those out. i'll call the result of this restriction "reasonable-SSA".

so, `#ES(S) / #RC(S) = #ES(R) / #RC(R)`.

```
SSA(S)
  = (P(S) × #ES(S) / #RC(S)) / (P(S) × #ES(S) / #RC(S) + P(R) × #ES(R) / #RC(R))
  = (1/3  × #ES(S) / #RC(S)) / (1/3  × #ES(S) / #RC(S) + 2/3  × #ES(S) / #RC(S))
  = (1/3  × #ES(S) / #RC(S)) / (1 × #ES(S) / #RC(S)                            )
  = (1/3                   ) / (1                                              )
  = 1/3

```

so here, SSA makes no update from the fact that you exist — it stays at 1/3 for `S`.

i believe that the fact that SIA and reasonable-SSA would bet on different things here (SIA bets on `S` and reasonable-SSA bets on `risk`), and that what you believe about this question could be very useful (we want to know how dangerous the technology we're using now is, because we don't want to die!), demonstrates the usefulness of anthropics as well as the importance of the SIA vs SSA question.

an anthropics example


generalized values: testing for patterns in computation
-------------------------------------------------------

i believe the [True Name](https://www.alignmentforum.org/posts/FWvzwCDRgcjb9sigb/why-agent-foundations-an-overly-abstract-explanation) for our [formalized values](what-is-value.html) will be a program running a computation (typically a [simulation or bunch of simulations](%E2%88%80V.html)) in which those values are maximized or satisfied.

in [*generalized computational interpretability*](generalized-computation-interpretability.html), i talk about what deeply caring about a general computation looks like. in this post, i will outline an example of this.

suppose you are given a program A described in [SKI calculus](https://en.wikipedia.org/wiki/SKI_combinator_calculus), and you want to know whether it encodes a simulation of [conway's game of life](https://en.wikipedia.org/wiki/Conway's_Game_of_Life) B with a hardcoded specific initial state.

in the general case, to get the state of a specific cell in a conway's game of life after `n` steps takes `Θ(n³)` time: `n` because of the time steps, and `n²` because of the two-dimensional light cone. for this example, let's say that this is actually the lowest bound to get the states you care about in this particular initial state of conway's game of life.

now, if there is an "extracting" program E such that for any coordinates `x,y` and time step `n`, taking as input the entire history of running A for `Θ(n³)` steps, it returns in less than `Θ(n³)` — for example in `O(log n)` — a value that is always equal to the state of B at `x,y,n`, then A encodes the computation B: A must have "done some of the work" of running B, because we can extract that value from A without re-doing all of the required work.

on the other hand, a formal proof that no such program exists can demonstrate that A does not encode B.

it could be that the problem is sometimes undecidable — i.e. there exists neither a program E (or proof that it exists), nor a proof that it doesn't exist. this seems fine to me; for example, when you're unable to determine whether a computation encodes suffering, just don't run it, or only run it for a limited amount of steps.

(to generalize this to constant-time or constant-size patterns, and to help us figure out the constants in those big-O/Θ's, perhaps information theory can help)

if the specific conway's game of life pattern B you're testing for happens to already be computable in less than `Θ(n³)`, then whatever complexity is its lowest bound (if it has one) is the new one under which E must run.

hopefully this can give us an idea as to what formalized shape our values (avoiding suffering, [etc](core-vals-exist-selfdet.html)) should take, and how to create a world that realizes them.


*(2022-10-12 edit: handsandcities.com seems to be down, so i've replaced the links to posts with links to the same posts on lesswrong)*

recommending Hands and Cities
-----------------------------

early this year, i found out about a blog called [*Hands and Cities*](https://handsandcities.com/).

it explores various topics, notably ethics and anthropics, in an exploratory style i find not dissimilar to my own, and generally easy to understand; and some of the ideas there are genuinely novel and fun to consider.

while you may have noticed that i've started heavily referencing it through links on this blog, in this post i'm explicitely recommending [*Hands and Cities*](https://handsandcities.com/). in addition, i'll list some of my favorite posts:

* [Alienation and meta-ethics (or: is it possible you should maximize helium?)](https://www.lesswrong.com/posts/3jeBKhek57sEkYGCs/alienation-and-meta-ethics-or-is-it-possible-you-should)
* [Actually possible: thoughts on Utopia](https://www.lesswrong.com/posts/SLw2MEgxFtiKAqgQ5/actually-possible-thoughts-on-utopia)
* [Contact with reality](https://www.lesswrong.com/posts/r7f58E8A85xLgWuqG/contact-with-reality)
* [On the limits of idealized values](https://www.lesswrong.com/posts/FSmPtu7foXwNYpWiB/on-the-limits-of-idealized-values)
* [In search of benevolence (or: what should you get Clippy for Christmas?)](https://www.lesswrong.com/posts/oXQDcyXJpMQTbaTMS/in-search-of-benevolence-or-what-should-you-get-clippy-for)
* [Can you control the past?](https://www.lesswrong.com/posts/PcfHSSAMNFMgdqFyB/can-you-control-the-past)
* [SIA > SSA, part 1: Learning from the fact that you exist](https://www.lesswrong.com/posts/RnrpkgSY8zW5ArqPf/sia-greater-than-ssa-part-1-learning-from-the-fact-that-you) along with parts [2](https://www.lesswrong.com/posts/GJdymoviRywpBMXqc/sia-greater-than-ssa-part-2-telekinesis-reference-classes), [3](https://www.lesswrong.com/posts/QHDqfpMbb43JDbrxN/sia-greater-than-ssa-part-3-an-aside-on-betting-in), and [4](https://www.lesswrong.com/posts/d693Mc4ZDyhkj7wpc/sia-greater-than-ssa-part-4-in-defense-of-the-presumptuous)
* [On the Universal Distribution](https://www.lesswrong.com/posts/XiWKmFkpGbDTcsSu4/on-the-universal-distribution) and [Anthropics and the Universal Distribution](https://www.lesswrong.com/posts/Hcc9fopx7sRexYhhi/anthropics-and-the-universal-distribution/)
* [On infinite ethics](https://www.lesswrong.com/posts/5iZTwGHv2tNfFmeDa/on-infinite-ethics), but see also [this comment](https://www.lesswrong.com/posts/5iZTwGHv2tNfFmeDa/on-infinite-ethics?commentId=KkmEbtKFTpHTrF3Dn)
* some of [On expected utility, part 1: Skyscrapers and madmen](https://www.lesswrong.com/posts/7J3ywHzWnghRtdpHQ/on-expected-utility-part-1-skyscrapers-and-madmen) and [part 2: Why it can be OK to predictably lose](https://www.lesswrong.com/posts/nPjMnPvMTajN9KM5E/on-expected-utility-part-2-why-it-can-be-ok-to-predictably)

recommending Hands and Cities


solonomonoff induction, time penalty, the universal program, and deism
----------------------------------------------------------------------

if you subscribe to the standard [solomonoff induction](https://www.lesswrong.com/posts/EL4HNa92Z95FKL9R2/a-semitechnical-introductory-dialogue-on-solomonoff-1), with its uncomputable [universal distribution](https://www.lesswrong.com/posts/XiWKmFkpGbDTcsSu4/on-the-universal-distribution), then the answer to "where are we in it?" is often "in the [universal program](universal-complete.html)"; if it's not, then it's somewhere "below" the universal program — maybe in [rule 30](https://en.wikipedia.org/wiki/Rule_30) (or maybe rule 30 likely implements its own universal program, and serves as a minimum?)

this makes the universal distribution limited in usability: where are we? oh, this one program that contains everything. to remedy problems like this, people come up with [UDASSA, with the "world and claw" assumption](https://www.lesswrong.com/posts/Hcc9fopx7sRexYhhi/anthropics-and-the-universal-distribution/) — this roughly means that you have to describe not just a program that contains us, but also a program that *locates* us within the computation. so, sure, we're somewhere in the universal program, but now you have to *find where*, and this is where the bits of interestingness come back.

an alternative is to penalize computations that run too long; [levin search](http://www.scholarpedia.org/article/Universal_search) uses exponential time, but [if exponentials put you off](https://arxiv.org/abs/1108.1791), [my time step proposal](udassa-time-steps.html) uses linear time.

so now, the question remains: *where* (which also now includes *when*) are we in the prior? in the spirit of that question, let me get into a tangent.

i was thinking the other day about what a fully rationality-equipped argument for atheism looks like. the basic structure is something like: {computation seems fundamental + simplicity seems to work} → solomonoff induction → the prior dislikes god. the rough idea is that a personified god is a needlessly rich computation to describe, to get to our world.

while this remains true for human-like deities like the abrahamic gods, it makes an interesting argument for deism emerge: what occurs earlier in the prior? the raw code for our universe, or the code for an intelligent program which in turns designs our universe? it is hard to reason about this, but if the bootstraping code for an intelligent and stuff-caring intelligence is simple, then there's a reasonable argument to be made for some form of deism. notably: if the program of the cosmos turns out to be the bootstrap for a universe in which an intelligence appears that *itself* spawns *our* universe, i think that that mildly counts as deism.

the "world and claw" or "locating time step" approaches to locating us within the computation make this perhaps even more plausible: what's more likely to compute us *first*, a naive computation of everything everywhere, or a lazy computation focused on a set of places that includes us — for example, because we're interesting, notably this century.

depending on which universal prior you choose (the standard uncomputable universal distribution with the "world and claw", levin search with log time step, the universal program with linear time step, or some other universal prior), i think you can make reasonable arguments for our existence being the product of a caring entity. examining whether our laws of physics are more likely to be designed or emerged, as well as considering how interesting we think we are, might help.


generalized computation interpretability
----------------------------------------

one approach to [AI alignment](https://en.wikipedia.org/wiki/AI_alignment) is this: develop technology to analyze what is going on within an AI, in order to determine what it's thinking. this is called [interpretability](https://en.wikipedia.org/wiki/Explainable_artificial_intelligence). the most generalized view of this is would involve something like [demon](https://www.lesswrong.com/posts/nyCHnY7T5PHPLjxmN/open-question-are-minimal-circuits-daemon-free) detection in arbitrary computations.

in addition, in order to implement our values, an aligned AI should [deeply](generalized-adding-reality-layers.html) care about them, which is to say: it should care about implementing those values even in arbitrarily encoded computations. it's not enough that the humans in the simulation don't suffer, they should also be unable to run computers which contain other humans which suffer, as well as [many other weird ways suffering moral patients can occur](https://reducing-suffering.org/what-are-suffering-subroutines/). for example, it should ban all [homomorphic encryption](https://en.wikipedia.org/wiki/Homomorphic_encryption) it can't decrypt, because otherwise it might be missing on some suffering moral patients.

i claim that there is a commonality to those two things: the detection of deeply, arbitrarily encoded computations. in one for demons, in the other for moral patients. i wonder if there are models of computation, perhaps laden with proofs of [benign](https://www.lesswrong.com/posts/Tr7tAyt5zZpdTwTQK/the-solomonoff-prior-is-malign)-ness, which make detecting those systemically doable.

this doesn't necessarily have to do with value-laden things; the theory of generalized interpretability could in general be used to determine objectively whether a given computation deeply contains, say, [rule 30](https://en.wikipedia.org/wiki/Rule_30).

furthermore, it seems like for this theory to determine whether "computation X deeply contains computation Y", we would need to specify Y in a profound way which might be the kind of format we'd need values to be in for general alignment. as an example, an aligned AI could be tasked with running whichever computations contain persons but do not contain suffering; and the kind of specification those would look like would, in order to be able to apply *deeply*, need to be fully general. note that i'm not sure they would need to be fully general, but my intuition points that way.


anthropic reasoning coordination
--------------------------------

what can a piece of [anthropic reasoning](https://www.lesswrong.com/posts/RnrpkgSY8zW5ArqPf/sia-greater-than-ssa-part-1-learning-from-the-fact-that-you) determine? what counterfactuals is it comparing its observations with?

if you are doing anthropic reasoning and observing something such as "plants exist" or "i remember childhood", how much you can gain from that information must depend on whether, if plants *didn't* exist or you *didn't* remember childhood, you would still be doing anthropic reasoning. if anthropic reasonings only or disproportionately exist in worlds where plants exist or where they occur in information systems with access to childhood memories, then you don't gain as much by observing those things.

as such, if you want the community of anthropic reasoners to gain as much information as possible, you have to commit to partaking of anthropic reasoning in reasonably equal amounts no matter what situation you're in. for example, to make [the doomsday argument](https://en.wikipedia.org/wiki/Doomsday_argument) work, we have to commit to being agents who would partake of anthropic reasoning *even if* [we overcame doom](timeline-codes.html); otherwise, we have to at least somewhat discount how much that observation tells us. maybe we solve alignment, and utopia is brought about, and in utopia we never or extremely rarely do anthropic reasoning, for whatever reason — maybe because it's obsolete and we're spending all our time frolicking about, or maybe those of us who *would* partake of anthropic reasoning discover forms of thought or knowledge which make anthropic reasoning thoroughly obsolete. or maybe we're all busy [suffering forever](https://en.wikipedia.org/wiki/S-risk).

anthropic reasoning coordination


"AI risk drone"
---------------

i want to coin the term "AI risk drone" to mean "person singlemindedly and as-optimizedly-as-possible dedicated to [AI risk mitigation](say-ai-risk-mitigation-not-alignment.html)". the term "alignment drone" derives similarly.

i am doing this because i am unaware of existing terminology to mean this, and because [given how bad things are](https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities) we might need significant coordinated effort towards that end; AI risk mitigation is hard and it seems that not that many people are able and willing to help it significantly, but maybe optimizing/"brainwashing" (but in a voluntary and good way) are more studied areas where we can get some not-too-high-hanging fruit to boost the work, perhaps merely in exchange for dollars.

as for myself, [i'm pretty decided](life-refocus.html) that i'd want to become this if it were available.

"AI risk drone"


outer alignment: politics & philosophy
--------------------------------------

[inner alignment](https://www.lesswrong.com/tag/inner-alignment) is "just" a hard engineering problem; [outer alignment](https://www.lesswrong.com/tag/outer-alignment) is the work of philosophy and politics and values which our species has been investigating and debating about for millenia.

are human values they the same for everyone, or do they differ?

should we implement the values held by us, us now, everyone now, everyone ever, or everyone possible?

would some philosophical/political perspectives constitute [suffering risks](https://en.wikipedia.org/wiki/S-risk)? for example, if many people on earth want to be correct, and if they also believe there is a hell where some people suffer forever, does that mean satisfying their values entails creating an at least moderately-sized hell, the inhabitants of which in some sense "value" suffering forever? *is that okay?*

if one person wants to go have gay sex, but ten christians want *nobody anywhere* to have gay sex, does [self-determination](core-vals-exist-selfdet.html) trump naïve utilitarian value satisfaction?

or should we create one giant super-consensus society where we all value being [boringly blissful](https://twitter.com/Merryweatherey/status/1185636106257211392), and forego all diversity, such that our values are easily implemented and non-conflicting; do we desire harmony above diversity?

if we value diversity, how much diversity should we instantatiate; what is the threshold of "evilness" at which a culture should not be able to exist?

how do we even reason about [existential self-determination](genuineness-existselfdet-satisfaction-pick2.html)?

what about [suffering in fundamental physics](https://reducing-suffering.org/is-there-suffering-in-fundamental-physics/) and [suffering subroutines](https://reducing-suffering.org/what-are-suffering-subroutines/)?

what *are* the politics and fundamental values of the people who will get to work on alignment?

on one hand, my belief about these questions is respectively "the latter", "us now", "possibly, yes, no", "yes", "no", "a bunch", "i don't know", "hopefully they don't matter too much", and "uh oh". on the other hand, i hope this post invokes how ridiculously not-talked-about-enough these questions are, considering how important they might be to what we fill the rest of this universe's history with.

mildly related: *["politics is the mind-killer" is the mind-killer](https://www.lesswrong.com/posts/uxsTyFLtSmxmniTzt/politics-is-the-mind-killer-is-the-mind-killer)*

outer alignment: politics & philosophy


where are your alignment bits?
------------------------------

information theory lets us make claims such as "you probly can't compress [this 1GB file](http://prize.hutter1.net/) into 1KB, given a [reasonable](kolmogorov-objectivity-in-languagespace.html) [programming language](https://en.wikipedia.org/wiki/Kolmogorov_complexity)".

when someone claims to have an "easy" solution to aligning AI to human values, which are [complex](https://www.readthesequences.com/Value-Is-Fragile) (have many bits of information), i like to ask: where are the bits of information of what human values are?

are the bits of information in the reward function? are they in how you selected your training data? are they in the prompt you intend to ask an AI? if you are giving it an entire corpus of data, which you think *contains* human values: even if you're right, the bits of information are in how you *delimitate* which parts of that corpus encode human value, a plausibly [exponential](https://en.wikipedia.org/wiki/Computational_complexity_theory) task. classification is hard; "gathering all raw data" is easy, so that's not where the bits of hard work are.

this general information-theoritic line of inquiry, i think, does a good job at pointing to why aligning to complex values is *[likely](plausible-vs-likely.html), actually* hard; not just *[plausibly](plausible-vs-likely.html), maybe* hard.
 we don't "might maybe need" to do the hard work, we *do likely need* to do the hard work.

where are your alignment bits?


diversity vs novelty
--------------------

[a friend of mine](https://ouroborista.neocities.org/) likes to say "novelty is inherently good"; as for myself, i've made known how much i care about cultural diversity ([1](from-above-fine-grain-diversity.html), [2](systems-and-diversity.html), [3](albions-seed.html), [4](%E2%88%80V.html)).

we have come to see those as similar, except that my notion of diversity is over space while my friend's notion of novelty is over time. notably, the two can be distinguished by the following experiment:

a planet right now contains 10 quite different cultures. you get to choose which of the two possible future states it will go in:

* a future state with those same 10 cultures, except they're now each *a bit different* from their previous state.
* a future state with 3 quite different cultures, but all are *very* different from all of the ones they have now.

if it makes a difference: other values are the same, all cultures mentioned here are reasonably non-evil and all about equally "good", you won't ever get to interact with that planet or see their culture yourself, and this planet will not exist after experiencing the selected future — the only two real times at which it's [instantiated](questions-cosmos-computations.html) are its present state, and the future you pick.

a space-diversity perspective compares the two future states and says: "10 > 3, so i pick the first future, it is a present state with more diversity".

a time-novelty perspective, however, compares the two timelines and says: "10 < 10 + 3, so i pick the second future, it is a timeline experiencing more total diversity".

i wonder if there is a more generalized notion of diversity that doesn't care about time vs space, and [if i value](core-vals-exist-selfdet.html) that one instead of space-diversity. another difficult question is: what about diversity between parallel universes or [bubbles](above-paperclips-2.html) ?

time is still weird, huh.

diversity vs novelty


concentric rings of illiberalism
--------------------------------

i feel like [liberalism](https://en.wikipedia.org/wiki/Liberalism) is not just a political system but a general *mode of thought* and *collection of background assumptions*; which entails that it takes an outside system to properly study that system including its assumptions.

one stuck in the liberal mode of thought is limited; outside of it are both socialistic and conservativistic concerns about what sort of unexpected issues emerge from liberalism, such as systemic concentration of wealth or cultural value drift. the outermost proper circles of illiberalism i know of include james c. scott's [against the grain](https://slatestarcodex.com/2019/10/14/book-review-against-the-grain/) and [seeing like a state](https://slatestarcodex.com/2017/03/16/book-review-seeing-like-a-state/) as well as kaczynsi's [industrial society and its future](https://en.wikipedia.org/wiki/Industrial_Society_and_Its_Future), where the very fundamentals of industrialism and sedentary states are questioned.

concentric rings of illiberalism


plausible vs likely
-------------------

when talking about important (utility-monsterly so) problems like [AI risks](say-ai-risk-mitigation-not-alignment.html), it is easy for two people to behave the same whether one believes the event is merely plausible (<50%, sometimes <10%) or likely (>50%, sometimes >90%).

i have seen this lead to some confusion about whether we are working on stuff to mitigate plausible risks or try plausible plans, or whether we are working to reduce likely risks or instantiate likely to work plans.

i'd like to make clear some of my beliefs related to AI risk issues.

* AI [X-risk](https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence): [likely](why-timelines-short.html)
* AI [S-risk](https://en.wikipedia.org/wiki/S-risk): [plausible](https://reducing-suffering.org/near-miss/)
* AI alignment is hard: [likely](https://www.readthesequences.com/Value-Is-Fragile)
* [*the peerless*](the-peerless.html) or other [pivotal acts](https://arbital.greaterwrong.com/p/pivotal/) are doable: plausible
* superintelligence will come about by [foom](https://en.wiktionary.org/wiki/foom#Noun): likely

plausible vs likely


say "AI risk mitigation" not "alignment"
----------------------------------------

the common thread i see between the work of people who describe themselves as working on alignment seems to be AI risk mitigation.

this is the case because "alignment" does not necessarily cover eg [pivotal acts](https://arbital.greaterwrong.com/p/pivotal/); in addition, [X-risks](https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence) are [not the whole story](https://en.wikipedia.org/wiki/S-risk) (see also: [*alignment near miss*](https://reducing-suffering.org/near-miss/) and [separation from hyperexistential risk](https://arbital.com/p/hyperexistential_separation/)).

while it *is* true that AI risks are largely caused by us not having alignment, it is not necessarily the case that the immediate solution is to have alignment.

to encompass the spirit of the work i do (when i am being truthful about it), i tend to say that i think about AI risk mitigation — [whatever form](ai-risk-plans.html) that takes.


implementing the platonic realm
-------------------------------

(or: holding the world by its souls; or: upside-down metaphysics)

i think, to construct a utopia (possibly especially a [sublime](https://www.lesswrong.com/posts/SLw2MEgxFtiKAqgQ5/actually-possible-thoughts-on-utopia) one) we ought to reverse the metaphysics of the world.

in any graph forming a hierarchy, you can pick a different node as root and view the graph from there. for example, in the model of reality below, you can pick psychology as your root instead of physics, and end up with reversed arrows.

![](implementing-the-platonic-realm.svg)

this gets closer to something like a "platonic realm" in which objects are truly, objectively integral, rather than patterns in the stuff they're made of. we can select any layer to be this, and then maybe let details fill in as we investigate them, de-implementing [reductionism](https://www.readthesequences.com/Reductionism). we can implement this the same way video games implement objects: as their own data structures with a bunch of methods to manipulate them, some of which able to generate detail to fill the object with; but those details are not an essential part making up the object, it's moreso the object and its context that produce the details.

these top level platonic persons are still full persons, in the computational sense we care about: their full neural net is still being ran at the top level.

why do this? so we can make a world that is actually focused on, and aligned to serve, persons. to make persons legible to the superintelligent singleton running that world.

now, if other detail layers contain enough compute to simulate another person, then there might still exist suffering persons that the top level is not keeping track of.

two solutions are: either detect encoded persons and extract them back up to the platonic level, or make other layers too weak to consistently encode suffering moral patients — this would put severe bounds on what kinds of computers we can use, but maybe that's fine. maybe computers are not even worth having, in the person-focused platonic realm.

implementing the platonic realm