AI boxing could be easy
-----------------------

while i think on its own it doesn't solve *that* much, i don't think building an [AI box](https://en.wikipedia.org/wiki/AI_box) is particularly difficult.

i am not talking about *letting an AI have full dominion over a computer but physically isolating it*; physical limitations are hard to predict and unreliable. i am talking about mathematically isolating a piece of software, using known sandboxing techniques.

for example, consider an AI whose code is encoded as SKI calculus, running on a simple SKI calculus calculator. or an AI whose code is encoded into conway's game of life, running on a simple conway's game of life simulator. these programs can be defined in such a way that their only failure mode is stopping due to running out of resources.

remaining vulnerabilities involve things like [rowhammer](https://en.wikipedia.org/wiki/Rowhammer), or [meltdown](https://en.wikipedia.org/wiki/Meltdown_%28security_vulnerability%29) and [spectre](https://en.wikipedia.org/wiki/Spectre_%28security_vulnerability%29); but those shouldn't be too hard to contain so long as the level of interpretation of the AI is sufficiently abstracted away from hardware that the configurations that lead to these sorts of exploits just don't occur — in fact, considering [how easy](https://www.gwern.net/Turing-complete) it is to make very abstracted turing complete systems, designing a fully sandboxed and highly-abstracted-from-hardware turing complete environment running a boxed AI should be doable with only a constant-factor slowdown. the information theoritic limit of this would be something like "reverse" [homomorphic encryption](https://en.wikipedia.org/wiki/Homomorphic_encryption), where the inside of the encrypted computation is information-theoretically guaranteed to be unable to control how its process affects the hardware around it (as long as the encryption scheme holds, of course) — though i am unsure whether such a scheme exists, or even makes sense to consider.

as long as the computation's output isn't used for anything, i posit that this is reasonably safe. the difficult part is designing *output ports* that are safe. however, even without that, we can do some cool stuff such as making an reinforcement-based AI's reward function unhackable.

if the boxed AI is fully deterministic, then it has no notion of anything going on in the real world like how fast time is passing or what hardware it's running on; all it has to go on to guess is the structure of its boxed environment, which can be made simple enough that the AI guessing anything about us seems information-theoretically unlikely enough — for example, i would think that an AI living in a seemingly classical computation would be unlikely to guess that the outside world runs on quantum mechanics.
giving the boxed AI input ports is *theoretically* safe, but significantly amplifies any risks due to insufficient output safety — if the AI *does* have access to outside information, it gets evidence as to what kind of world it is trying to affect.

what could a safe output port look like? we could use the output port to receive proofs to theorems which are verifiable, for example. i believe discussion of how to turn this into pivotal acts is discussed in some of the first [late 2021 MIRI conversations](https://www.lesswrong.com/s/n945eovrA3oDueqtq). however, we may prefer queries which verifiably have only one correct answer (like "what is the ordered list of prime factors of this number"; unlike "what is a proof of this theorem"); if there are multiple correct answers, then that's some degree of freedom the AI could use to affect us, especially if any human happens to glance upon the output data.

AI boxing could be easy


predictablizing ethic deduplication
-----------------------------------

does the machinery of the cosmos [deduplicate identical moral patients and/or their experiences](deduplication-ethics.html) ?

this seems like a very difficult question to even address; but the good news is that for the general future we might not have to care about it. we can simply make it that the superintelligence that runs everything, *does* deduplicate (memoize) identical computations and data structures, which guarantees that the ethics we build on top of that (for superintelligence to implement) *can* know about deduplication.

why choose deduplication over no-deduplication? because if we add deduplication on top of any machinery of the cosmos, then we can know for sure deduplication happens, but if we *don't* implement deduplication, then whether computation is deduplicated depends on the machinery of the cosmos.

"but doesn't this require looking inside arbitrarily encoded computations, such as [homomorphic encryption](https://en.wikipedia.org/wiki/Homomorphic_encryption) ?"

that is true, but for an aligned superintelligence we require this *anyways*. otherwise, it could just let unseen pockets of arbitrary suffering happen.

predictablizing ethic deduplication


generalized adding reality layers
---------------------------------

in [*predictablizing ethic deduplication*](predictablizing-ethic-deduplication.html), i talk about how when we don't know how reality works, we can task a singleton superintelligence with "adding a layer" to reality, which guarantees that inside that simulated reality we *are* able to function with known ethics.

in addition there's a sense in which, if one principle overrides the other, in general with [arbitrarily many layers of reality-simulation](above-paperclips-2.html) we should tend to favor whichever option overrides. so for example: if our reality is actually on top of 1000 layers of reality simulations, then it only takes **one** of them to be (truly, deeply) deduplicating for our universe and any sub-universe we simulate to also have deduplication.

or, more precisely, for any set of mutually exclusive traits with a dominance ordering (such as deduplication > no-deduplication), we can expect the take one of those shapes:

(click on the image to expand)

[![](generalized-adding-reality-layers.svg)](generalized-adding-reality-layers.svg)

i will call this the "generalized adding reality layers" (GARL) device, and i think it could have a broad use to reason about properties of the cosmos (the set of [instantiated](questions-cosmos-computations.html) universes), even ones that might seem axiomatic and [untestable](https://en.wikipedia.org/wiki/Newton's_Flaming_Laser_Sword).

for any set of mutually exclusive traits, we care about four properties:

* what the dominance ordering is between those traits
* how they affect the rate of spawning varied sub-universe
* how they affect the rate of spawning moral patient experiences
* how they affect the rate of spawning deeply caring actors

so, what other sets of traits can we examine using GARL ? here are some that i can think of off the top of my head, as well as my guess for the questions above..

| question | dominance ordering | most varied sub-universes | most moral patient experiences | most deeply caring actors |
| --- | --- | --- | --- | --- |
| [moral patient deduplication](deduplication-ethics.html) | dedup > no-dedup | unaffected ? | no-dedup > dedup | i've no idea |
| [infinite compute](hope-infinite-compute.html) ¹ | finite > infinite | infinite > finite | infinite > finite | infinite > finite ? |
| type of compute ¹ | classical > quantum > hyper | hyper > quantum > classical ? | unknown | hyper > quantum > classical ? |
| moral realism ² | realism > non-realism | unsure | whichever maximizes good | realism > non-realism ? |
| deeply-caring superintelligence | present > absent | depends on its goals | depends on its goals | present > absent |

¹: these two questions are similar to one another in that they have one dominant variant that restricts computation, and one recessive variant that doesn't; as a result, i would tend to assume that the recessive variant has a higher chance of spawning most kinds of stuff

²: my reasoning: once what is true becomes aligned with what is good, then the [orthogonality thesis](https://www.lesswrong.com/tag/orthogonality-thesis) becomes falsified in that sub-cosmos, and superintelligences are more easily aligned by default

other questions to which GARL may be applicable but i haven't figured out how:

* is occam razor's/[solomonoff induction](https://www.lesswrong.com/posts/EL4HNa92Z95FKL9R2/a-semitechnical-introductory-dialogue-on-solomonoff-1) applicable?
* [are minimal circuits daemon-free?](https://www.lesswrong.com/posts/nyCHnY7T5PHPLjxmN/open-question-are-minimal-circuits-daemon-free)
* [can you control the past?](https://www.lesswrong.com/posts/PcfHSSAMNFMgdqFyB/can-you-control-the-past)
* [SIA vs SSA](https://www.lesswrong.com/posts/RnrpkgSY8zW5ArqPf/sia-greater-than-ssa-part-1-learning-from-the-fact-that-you)

generalized adding reality layers


smaller X-risk
--------------

a superintelligence killing us all is a *superintelligent, very large* [X-risk](https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence).

the superintelligence will tile its values in all directions; not just through space at the speed of light [or faster](https://en.wikipedia.org/wiki/Alcubierre_drive), but also, if it can, by [hacking physics](brittle-physics.html) and traversing across, for example, worldlines of the quantum many-worlds.

we may be able to create smaller X-risks, that only make us extinct in this timeline, on this earth. there are a few reasons we may want to do this:

* other timelines might have a better shot than us, and us booting a superintelligence may reduce their chances through weird stuff like intertimeline hacking
* to [avoid S-risks](when-in-doubt-kill-everyone.html), including S-risks that may be involved in instrumental cosmic-scale X-risk (maybe superintelligence wants to simulate civilizations in various ways for [acausal trade](https://www.lesswrong.com/tag/acausal-trade) or [other acausal weirdness](https://www.lesswrong.com/posts/PcfHSSAMNFMgdqFyB/can-you-control-the-past) reasons?)
* the next intelligent species on earth is more likely than us to solve alignment before superintelligence, and seems likely enough to be at least a little bit aligned with us (better than cosmic X-risk, at least)
* same as above, but for nearby aliens (whether current or future)

*smaller X-risk*, where we limit damage to just our civilization, seems harder than tiling the cosmos with paperclips; but at least it might be easier than [other plans](ai-risk-plans.html).

in a similar way, reducing our civilization to ashes *without* actually becoming extinct might also be a way to get another shot, if we think we're likely to do less badly next time.

remember: this is bigger than all of us. when the fate of the cosmos is at play, we can't afford to be too selfish.

smaller X-risk


cognitive biases regarding the evaluation of AI risk when doing AI capabilities work
------------------------------------------------------------------------------------

i have recently encountered a few rationality failures, in the context of talking about AI risk. i will document them here for reference; they probly have already been documented elsewhere, but their application to AI risk is particularly relevant here.

### 1. forgetting to multiply

let's say i'm talking with someone about the likelyhood that working on some form of AI capability [kills everything everywhere forever](https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence). they say: "i think the risk is near 0%". i say: "i think the risk is maybe more like 10%".

would i bet that it will kill everyone? no, 10% is less than 50%. but "what i bet" isn't the only relevant thing; a proper utilitarian *multiples* likelyhood by *quality of outcome*. and X-risk is really bad. i mistakenly see some people use only the probability, forgetting to multiply; if i think everyone dying is not likely, that's enough for them. one should care that it's *extremely* unlikely.

### 2. categorizing vs average of risk

let's take the example above again. let's say you believe said likelyhood is close to 0% and i believe it's close to 10%; and let's say we each believe the other person generally tends to be as correct as oneself.

how should we come out of this? some people seem to want to pick an average between "carefully avoiding killing everyone" and "continuing as before" — which lets them more easily continue as before.

this is not how things should work. if i learn that someone who i generally consider about as likely as me to be correct about things, seriously thinks there's a 10% chance that my tap water has lead in it, my reaction is not "well, whatever, it's only 10% and only 1 out of the two of us believe this". my reaction is "what the hell?? i should look into this and stick to bottled water in the meantime". the average between risk and no risk is not "i guess maybe risk maybe no risk"; it's "lower (but still some) risk". the average between ≈0% and 10% is not "huh, well, one of those numbers is 0% so i can pick 0% and only have half a chance of being wrong"; the average is 5%. 5% is still a large risk.

this is kind of equivalent to *forgetting to multiply*, but to me it's a different problem: here, one is not just forgetting to multiply, one is forgetting that probabilities are numbers altogether, and is treating them as a set of discrete objects that they have to pick one of — and thus can justify picking the one that makes their AI capability work okay, because it's one out of the two objects.

### 3. deliberation ahead vs retroactive justification

someone says "well, i don't think the work i'm doing on AI capability is likely to kill everyone" or even "well, i think AI capability work is needed to do alignment work". that *may* be true, but how carefully did you arrive at that consideration?

did you sit down at a table with everybody, talk about what is safe and needed to do alignment work, and determine that AI capability work of the kind you're doing is the best course of actions to pursue?

or are you already committed to AI capability work and are trying to retroactively justify it?

i know the former isn't the case because there *was* no big societal sitting down at a table with everyone about cosmic AI risk. most people (including AI capability devs) don't even meaningfully *know* about cosmic AI risk; let alone deliberated on what to do about it.

this isn't to say that you're necessarily wrong; maybe by chance you happen to be right this time. but this is not how you arrive at truth, and you should be highly suspicious of such convenient retroactive justifications. and by "highly suspect" i don't mean "think mildly about it while you keep gleefully working on capability"; i mean "seriously sit down and reconsider whether what you're doing is more likely helping to save the world, or hindering saving the world".

### 4. it's not a prisoner's dilemma

some people think of alignment as a coordination problem. "well, unfortunately everyone is in a [rat race](https://slatestarcodex.com/2014/07/30/meditations-on-moloch/) to do AI capability, because if they don't they get outcompeted by others!"

this is *not* how it works. such prisoner's dilemmas work because if your opponent defects, your outcome if you defect too is worse than if you cooperate. this is **not** the case here; less people working on AI capability is pretty much strictly less probability that we all die, because it's just less people trying (and thus less people likely to randomly create an AI that kills everyone). even if literally everyone except you is working on AI capability, you should still not work on it; working on it would *still only make things worse*.

"but at that point it only makes things negligeably worse!"

…and? what's that supposed to justify? is your goal to *cause evil as long as you only cause very small amounts of evil*? shouldn't your goal be to just generally try to cause good and not cause evil?

### 5. we *are* utilitarian… right?

when situations akin to the trolley problem *actually appear*, it seems a lot of people are very reticent to actually press the lever. "i was only LARPing as a utilitarian this whole time! pressing the lever makes me feel way too bad to do it!"

i understand this and worry that i am in that situation myself. i am not sure what to say about it, other than: if you believe utilitarianism is what is *actually right*, you should try to actually *act utilitarianistically in the real world*. you should *actually press actual levers in trolley-problem-like situations in the real world*, not just nod along that pressing the lever sure is the theoretical utilitarian optimum to the trolley problem and then keep living as a soup of deontology and virtue ethics.

i'll do my best as well.

### a word of sympathy

i would love to work on AI capability. it sounds like great fun! i would love for everything to be fine; trust me, i really do.

sometimes, when we're mature adults who [take things seriously](life-refocus.html), we have to actually consider consequences and update, and make hard choices. this can be kind of fun too, if you're willing to truly engage in it. i'm not arguing with AI capabilities people out of hate or condescension. i *know* it sucks; it's *painful*. i have cried a bunch these past months. but feelings are no excuse to risk killing everyone. we **need** to do what is **right**.

shut up and multiply.


life refocus
------------

because of the [recent](https://www.metaculus.com/questions/3479/date-weakly-general-ai-system-is-devised/) [events](https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy), which i've been dreading [for a while](were-all-doomed.html), i'm taking AI risk a lot more seriously, and have started significantly refocusing my life. (**edit**: see also [my more recent intro to AI risk](ai-doom.html))

there is a post called [*musk's non-missing mood*](https://lukemuehlhauser.com/musks-non-missing-mood/) that resonates quite well with me. it is indeed kind of disconcerting how people who seem rationally aware of AI risk, don't seem to *grok* it as an *actual thing*. despite how real it is, it's hard to think of it not as fantasy fiction.

i totally understand why. i've been there too. but eventually i managed to progressively update.

i'm still not quite there yet, but i'm starting to actually grasp what is at stake.

["detaching the grim-o-meter"](https://mindingourway.com/detach-the-grim-o-meter/) remains a reasonable thing to do; you don't want to become so depressed that you kill yourself instead of saving the world. but you also don't want to remain so deluded that you don't quite weigh the importance of saving the world enough either.

i'll learn japanese after the singularity. i'll make [my game](game.html) and [my alternative web](saving-the-web.html) and my conlang and [my software stack](psi.html) and many other things, after the singularity. it is painful. but it is what's right; it's closer to [the best i can do](https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy).

and i know that, if at some point i give up, then it won't look like pretending that everything is fine and compartmentalizing our imminent death as some fantasy scenario. it'll be a *proper* giving up, like going to spend the remaining years of my life with my loved ones. even my giving up scenario is one that takes things seriously, as it should. that's what being an adult capable of taking things seriously is like.

how you handle your mental state is up to you. there is a collection of AI-risk-related mental health posts [here](https://www.lesswrong.com/posts/pLLeGA7aGaJpgCkof/mental-health-and-the-alignment-problem-a-compilation-of). do what it takes for you to do the work that needs to be done. that's not becoming a doomer; your brain is straight-up not designed to deal with cosmic doom. but that's not remaining blindly naive either. the world needs you; it won't be saved by pretending things are fine.

and it *certainly* won't be saved by pretending things are fine and *working on AI capability*. that's *just bad*. *please* don't.

please take AI risk seriously.

life refocus


hope for infinite compute
-------------------------

here are some reasons we may have infinite universe to inhabit in the future.

* encoding ourselves in heat death noise. this could at least buy us exponentially much time to exist; it becomes infinite if the amount of possible states also increases.
* where are we in [the universal distribution](https://www.lesswrong.com/posts/EL4HNa92Z95FKL9R2/a-semitechnical-introductory-dialogue-on-solomonoff-1) ? most models that produce [rich](questions-cosmos-computations.html) computation, such as [rule 30](https://en.wikipedia.org/wiki/Rule_30), seem to grow forever in amount of stuff. this includes [wolfram's hypergraph rewriting system](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/).
* space is expanding. the planck length is a constant. unless there's something i'm mistaken about, this sure seems like more positions in space we could inhabit, and thus an increasing total amount of states the universe can be in. it is not obvious how to utilize this, but it may be evidence for other ways in which the amount of stuff in the universe grows. in wolfram's perspective, it is simply the hypergraph creating new positions in space, as it has been doing forever.
* why are there 10⁸⁰ particles in the observable universe? if the total number is larger, why is it that larger number? wouldn't it be occam-simpler that there be 1 or few particles (or qubits or whatever) in the start, and have that amount grow over time? in fact, with expanding space, won't there statistically tend to be more particles overall, if only because there's more space for quantum fluctuations to randomly spawn particles? surely a superintelligence can harness this.
* even if we're in a physically finite universe, we may be able to acausally trade/hack/blackmail aliens living in infinite worlds such as rule 30. maybe. [acausal trading is weird](https://www.lesswrong.com/posts/PcfHSSAMNFMgdqFyB/can-you-control-the-past).

hope for infinite compute


AI risk plans
-------------

people have criticized my [*peerless*](the-peerless.html) plan on the grounds that it's too long-term/far-fetched.

while i don't disagree, i think that it is only one variable to be taken into consideration. here is a comparison of plans for addressing AI risk, with vague estimates.

| plan | achievable before [X-line](timeline-codes.html)¹ | chance of [U-line](timeline-codes.html)² | [S-risk](https://en.wikipedia.org/wiki/S-risk)² |
| --- | --- | --- | --- |
| doing nothing | 100% | <[1e-6](https://www.lesswrong.com/tag/orthogonality-thesis) | <1e-6 |
| direct alignment | [.1%](https://www.readthesequences.com/Value-Is-Fragile) | 5% → .005% | [5%](https://reducing-suffering.org/near-miss/) → .005% |
| [the peerless](the-peerless.html) | 2% | 10% → .2% | 1% → 0.02% |

* ¹: assuming significant effort is put behind the plan in question, what is the likelyhood that we'll have accomplished the work to what we *believe* to be completion? note that my current AI timelines are pretty pessimistic (we become more likely to die than not this decade)
* ²: *if* we believe to have completed the work; the latter number is adjusted by being multiplied with "achievable before [X-line](timeline-codes.html)".

note that the numbers i put here are only very vague estimates, feel free to replace them with your own guesses. but my point is, in order for the peerless to be the plan we should be working on, we don't need it to be *feasible*, we just need it to be *less infeasible than all the other plans*. i think the peerless is more tractable than doing direct alignment, and only more risky because it has more chances to succeed. depending on how scared of S-lines you are, you should push for either doing nothing (and thus [oppose direct alignment](against-ai-alignment.html)) or for my plan. (or come up with your own, and then compare it to these!)

not pictured: the plan to [melt all GPUs](https://www.lesswrong.com/s/n945eovrA3oDueqtq/p/7im8at9PmhbT4JHsW), because it's just a modifier on what we do afterwards. but yes, melting all GPUs is a great idea if we think we can reasonably do it more than other plans.

AI risk plans


a unit for utils
----------------

as utilitarians, it would be convenient for us to have an actual unit to measure utility, a number to be computed and compared.

the usual pick is money, but some people could have different judgments of the world that lead them to have different instrumental valuings of money even when they have the same intrinsic values; and also some people could intrinsically value money.

the unit i propose, to measure how much an agent cares about a thing, is a ratio of that person's "total caring pie". for example, you could intrinsically value 70% something and 30% something else; and then i'm sure we can figure out some math that makes sense (probly inspired from probability theory) to derive our valuings of instrumental values from that.

this seems like the least biased way to measure utils. the only criticism i can think of is that it breaks if two agents have different amounts of total valuing: perhaps one person *just has more total caring* than another.

however, is this testable in any way? is there any situation where one agent would act differently than another if they have the same intrinsic valuing proportions but one of them has a million times more total caring? i don't think so: the idea that inaction counts, seems to me to track either willpower or just different valuings of not doing effort.

a unit for utils


the uncertainty of 2+2=4
------------------------

people generally claim "it will rain tomorrow" and "2+2=4" are different *kinds* of knowledge. not only is one about the world and the other about mathematics, but the former is probabilistic and the latter is certain.

i here would like to dispute this second claim; in fact, 2+2=4 is, as we are now with our brains the way they are, probabilistic knowledge as well. it is possible that, by sheer chance, every time you and every other human and every computer ever calculated what 2+2 would be, they made a mistake by [random chance](https://en.wikipedia.org/wiki/Soft_error#Cosmic_rays_creating_energetic_neutrons_and_protons), and happened to make the exact same mistake. it could be that actually 2+2=5. or, for a more systematic example, one could find themself as [the mathematician in a lying simulation](https://www.lesswrong.com/posts/r7f58E8A85xLgWuqG/contact-with-reality).

now, that's *extremely* unlikely, just because of how universally 2+2=4 is claimed to be observed and verified. but it is indeed *possible*, and this fits into [the lack of absolute certainty](https://www.readthesequences.com/Zero-And-One-Are-Not-Probabilities).

maybe there [could](exact-minds-in-an-exact-world.html) be minds and computers that could determine this for sure (although, what about unknown unknowns?); but those aren't what we have now.

once you know this, other probabilistic guarantees like cryptography, or more generally problems that are in computational complexity classes like [BPP](https://en.wikipedia.org/wiki/BPP_%28complexity%29) and [BQP](https://en.wikipedia.org/wiki/BQP), can be percieved as more reasonable relative to *exact* guarantees: after all, everything we interact with on top of the standard model is probabilistic.

the uncertainty of 2+2=4


finding earth in the universal program
--------------------------------------

this post expands on step one of [*the Peerless*](the-peerless.html): creating virtual people. brain scans-and-simulations are apparently still quite far off, so i'll be focusing on the second approach: resimulating earth and plucking out persons.

(one great side-advantage of this method is that, if we can relocate earth to pluck out persons for the simulation of alignment researchers, then we can later also relocate earth in order to restore it once we've solved alignment. so resimulating and locating earth, regardless of having early enough mind-plucking-out tech, is something we might need to do anyways.)

if compute is [infinite](ai-alignment-wolfram-physics.html) and [we don't mind being inefficient](udassa-time-steps.html), then we can use exponential or even infinite compute to locate earth. one approach is the following: create a big informational beacon — perhaps a copy of a huge portion of the internet, along with MRI scans of as many people as we can afford. then, we use some type of (non-intelligent) deterministic error-bound statistical location procedure to locate patterns that look like that beacon inside the [universal program](universal-complete.html). we can afford the statistical detection to be imperfect — if it misses on one encoding of earth, there will be different ones in the universal program.

because of the time penalty of the universal program, however, we may find just compressed copies of the beacon (instead of a full simulation of earth leading to the time at which we build that beacon), and because of the deterministic bound, we want need to stop on the first match; if this first match is *just* the beacon, without earth, then we fail; perhaps superintelligence can notice that it's not finding any nearby minds to pluck out, or perhaps it plucks out garbage. so we can start the universal program with not one step per program, but rather a very large number of steps — i hear stephen wolfram has estimates on the number of computation steps it takes to get to the current state of the universe. this will favor programs that takes every long to lead to the beacon, but are themselves shorter program.

(what if the first program to contain earth is itself a universal program *without* that huge constant, such that *it* finds the beacon before it finds earth? i am not sure how to address this. perhaps we can explore programs in an order that favors worlds that look like our physics instead of looking like discrete iterations of all computations?)

there's also the concern that the universal program, just like the [universal distribution](https://www.lesswrong.com/posts/XiWKmFkpGbDTcsSu4/on-the-universal-distribution), [is malign](https://www.lesswrong.com/posts/Tr7tAyt5zZpdTwTQK/the-solomonoff-prior-is-malign). i'd think plain top-level earth, maybe especially as detectable by a simple enough beacon locator, would tend to occur before malign aliens emitting our beacon to trick us; but that's a risk to keep in mind.

if we *do* care about computational efficiency, then there are two main factors we need to account for:

* can our universe can be ran in polynomial time on whatever computers the superintelligence can build? for example, can it be ran in polynomial time on quantum computers, and can quantum computers be built? note that if this is the case we might need to step through *quantum steps* of *quantum programs* to run the search in the expected time. this doesn't need we need to build quantum computers outselves, mind you — superintelligence can just notice that a quantum computer would run the computations we describe efficiently, and build and use those.
* is the "seed" program for the universe small? intuitively i believe it is, and i find wolfram's efforts to reproduce the behavior of particles from the standard model using simple graph rewriting, to be evidence in that direction. that said, if it is large, then finding that program is an exponential search again — and so, again, we might need to build a search that "favors" our physics to save on exponential search time.

finally, we might want to put a hard bound on the number of tries the superintelligence will run to locate earth. the reason for that is that, if for some reason we messed up something in the beacon locator and it *never, ever* finds earth, then it will instantiate all computations, which appears to me to be a potential [S-risk](timeline-codes.html). in fact, even if we do find earth, it may not be worth it if we have to simulate exponentially much potential suffering before running our utopia — what if, after solving alignment, we have a great time, but then decide to eventually fade away after only polynomial time? then we will might have created exponentially much suffering in total.

### intermediary simulation

in case isolating minds from this simulation is hard, we could build an intermediary step between the location of earth in simulation-space, and booting the peerless simulation proper — superintelligence could, once it has located our beacon, get in touch with our organization *inside the simulation of earth*, and give it extraordinary computational (and maybe physical?) ability within the simulation to either take over everything, or figure out brain plucking-out and then let us press a big "ok, start now" button.

note, however, that we might not want to remain in this intermediary simulation for too long — it is still vulnerable to inner unaligned superintelligences, just like our top level reality is. we want to get to a safe, sandboxed, computationally weak environment as early as possible.

this is also a great argument for readying ourselves to build the beacon and utilize this contact-from-superintelligence as early as we can; indeed, to make that the first step of implementing the peerless plan. the reason for that is that the earlier we are able to take advantage of it, the earlier the time step of the simulation superintelligence can help us start bootstrapping towards the proper simulation of the peerless, and the less likely we are to be doomed by other superintelligences, if we need some intermediary "pre-peerless" simulation time.

finding earth in the universal program


The Peerless
------------

In this post, I propose a plan for addressing superintelligence-based risks.

Before I say anything, I will mention a crucial point that a *bunch* of people have ignored despite it being addressed at the bottom of this post: the idea I describe here is very unlikely to work. I'm proposing it because of other plans because I feel like other plans are *extremely* unlikely to work (see also [this post](ai-risk-plans.html)). Yes, we probably can't do this in time. That doesn't make it not our best shot. Rationalists select the best plan, not ["the first plan, and then after that a new plan only if it seems good enough"](https://www.readthesequences.com/If-Many-Worlds-Had-Come-First).

(spoilers for the premise of [*orthogonal* by Greg Egan](https://en.wikipedia.org/wiki/Orthogonal_%28series%29)) In Orthogonal, a civilization facing annihilation comes up with a last-minute plan: to create a ship, accelerate it until its time arrow is orthogonal to the time arrow of its home world (which is possible thanks to the alternate physics of their world), and thus give its crew as much time as it needs to figure out how to save their homeworld before reversing course and coming back. This plan is inspired by that, and i'm naming this post after their ship, the *Peerless*.

The short version is: we design a simulation for a bunch of people (probably rationalists) to live in and figure out alignment with as much time as they need, and create a superintelligence whose sole goal is to run that simulation and implement a new goal it will eventually decide on. I've written about this idea previously, but [that post](upload-for-alignment.html) is not required reading; this is a more fleshed-out view.

I will be describing the plan in three steps.

### 1. Create virtual people

We need virtual persons inside this world. They will be the ones who figure out alignment. A few possibilities come to my mind; there may be more.

* Brain scans, or full person scans. This is the most obvious solution. I'm not too familiar with the state of that field, but surely there's some work in that direction we can take advantage of; otherwise, we can just throw money at our own initiatives. This option does have the downside that it's quite likely brains aren't sufficient to keep someone functional — we may need to scan or re-implement a bunch more.
* Resimulate earth and pluck out persons. If there's a clever way to locate ourselves in the [universal distribution](https://www.lesswrong.com/posts/XiWKmFkpGbDTcsSu4/on-the-universal-distribution) (or a [computable variant of it](http://www.scholarpedia.org/article/Universal_search)), then we can just make a program that reruns that earth up to say, now, and then locates some or all human brains, and "download" them out of that simulation of earth and into our own simulated environment. For more details on this possibility, see [*finding earth in the universal program*](finding-earth-ud.html).
* Scan the earth and pluck our persons. This one seems harder than resimulating earth, but may be doable. It's certainly an idea worth throwing a few people at, to see if there's a clever way to make it work.

The main risk that's been brought to my attention regarding this part is the following: what if the virtual persons end up unaligned from their previous selves? The brain scan scenario seems like the most likely to have that risk, but even then i'm not *too* worried about it; intuitively, it seems unlikely enough to me that all the uploaded persons would come out misaligned in a similar direction, and in a similar direction that would lead them to decide on a [botched alignment](botched-alignment-and-awareness.html) for the superintelligence.

An obvious question here is: who gets to be on board the simulation? The values of the people who get uploaded might significantly affect what the superintelligence is aligned to (not all humans necessarily have the same values, [maybe even after thinking about it really hard for a long time](https://www.lesswrong.com/posts/FSmPtu7foXwNYpWiB/on-the-limits-of-idealized-values)). I don't have any answers other than the obvious "me please!" and "my tribe please!" that occur to me.

Note that i'm *not* proposing augmenting the uploaded minds — at least not for the first simulation iteration (see below). That *does* seem like an exceedingly risky prospect, alignment-wise, and one we don't need to commit to right away.

### 2. Design a virtual environment

Those persons will live in a virtual environment, within which they'll hopefully figure out alignment. However, the environment needs to be a deterministic computation, such that the "outer" superintelligence (the one running the virtual environment) has no ability to affect its outcome; its goal will only be to "implement whatever this computation decides". If the superintelligence wants to implement the actual result of the actual computation, and that computation is fully deterministic, (and if we don't simulate anything complex enough for that superintelligence to "leak back in"), then it has no room to meddle with what we do in it! It's stuck running us until we decide on something.

Some things we need to figure out include:

* How do we incorporate our virtual minds? I think we should go for something plugged in "ad-hoc" rather than embedded into the physics of that world, to preserve the integrity of those minds, which may live for very long times. In addition, in case virtual minds go crazy after living 200 years or something, we may want to allow them to reset themselves and/or die. A reset is not necessarily a big deal: hopefully previous-me can transmit enough information to future-me to continue the work. Maybe there are two me's at any given time, a teacher and an apprentice. Regular resets of individual persons also hopefully help maintain their values over long stretches of time. Many schemes are possible.
* What is this world like? We could make do with just something as basic as minecraft, but it would be better if the virtual persons don't have to go crazy from being stuck in a minecraft steve's body with no senses except sight and sound.
* How do we prevent "sub-singularities"? Given that this world is deterministic, there is nothing the outer superintelligence can do to prevent internal superintelligences from popping up and breaking everything it can. Potential solutions include things like "there are no computers" or "all computers inside this world are very slow and limited in capability".
* What about memetic safety? What about virtual violence? What if someone duplicates themself a billion times? And so on. There are a collection of design challenges, but designing [a peaceful world](%E2%88%80V.html) with [sensible virtual physics](game.html) doesn't seem out of reach. They seem like tractable engineering challenges.
* What is the final voting procedure? Remember that the goal of the simulation is to give the people inside it time to figure out alignment, but they should probably agree on something eventually: either a final decision on alignment, or a "next iteration": a new simulation to be ran, which they think has better/safer/still-safe conditions within which to research alignment. In fact, there may be arbitrarily many such "simulation iterations". Anyways, the simulation will have a big red button inside of it which says "okay, we're done", and takes as input a new goal (and possibly decision theory?) that the outer superintelligence will have as its ultimate goal. But what should it take to press the button? Everyone to agree? A majority? What if we end up unable to come to an agreement? Again, there is work to be done on this, but it seems figure-out-able.

The people inside this simulation will have somewhere between *plenty* and [*infinite*](ai-alignment-wolfram-physics.html) time and compute to figure out alignment. If they do have infinite compute, and if the cosmos isn't full of [consequentialists competing for earlyness in the universal distribution](udassa-time-steps.html) (or other things that might make wasting compute bad), then we can even run exponential-or-longer computations in, from our perspective, instant time; we just need to be sure we don't run anything malign and unbounded — although the risks from running malign stuff might be mitigated by the computations being fully and provably sandboxable, and we can shut them down whenever we want as long as they don't get to output enough to convince us not to. After all, maybe there are some bits of information that are the result of very large malign-dominated computations, that can nevertheless still be of use to us.

I mentioned before that maybe only slow computers are available; running a "very large" computation might require a majority vote or something like it. Or we can just boot without any computers at all and spend the first few millenia designing slow computers that are actually safe, and then work from there — when we have all the time we want, and maybe-infinite *potential* compute, a lot of options open up.

One downside is that we will be "flying blind". The outer superintelligence will gleefully turn the cosmos into computronium to ensure it can run us, and *will* be genociding everything back meat-side, in our reachable universe — or beyond, if for example physics is hackable, as wolfram suggests might be possible. Superintelligence might even do that *first*, and *then* boot our simulation. Hopefully, if we want to, we can resimulate-and-recover aliens we've genocided after we've solved alignment, just like hopefully we can resimulate-and-recover the rest of earth; but from inside the simulation we won't be able to get much information at least in the first iteration. We *can*, however, end our iteration by agreeing on a new iteration that has some carefully-designed access to outside information, if we think we can safely do that; but nothing guarantees us that there will still be something salvageable outside.

Another way to model "successive simulation iterations, each deterministic, but each having the ability to make the next one not deterministic with a large enough vote" is as a single simulation that isn't quite deterministic, but made of large deterministic chunks separated by small controlled I/O accesses; think of it as a haskell computation that lazily evaluates everything right up until it waits for an input, and then as soon as it has that it can continue computing more.

Still, the current outlook is that we genocide everything *including ourselves*. Even if nothing else is recoverable, "a tiny human population survives and eventually repopulates" still seems like a better plan than the current expected outcome of "everything dies forever."

### 3. Make a superintelligence to run it

Now, this is the "easy part": just make a superintelligence that destroys everything to implement its one simple goal; except instead of paperclips, the simple goal is "implement whatever goal is the result of this very big turing machine".

We can either build and start that superintelligence as soon as we can, or [keep it ready](when-in-doubt-kill-everyone.html) while we stay on our regular world. I'd probably advocate for the former just to be safe, but it can depend on your beliefs about quantum immortality, S-risks, and such. In any case, having *something that might work ready to fire* is certainly better than the current *we just die*.

Of course, it is crucial that we make the superintelligence *after* we have designed and implemented the virtual environment, complete with its virtual persons (or its deterministic procedure to obtain them); we don't want it to be able to influence what goal we give it, so we likely need to have the goal ready and "plugged in" from the start.

Some risks are:

* It doesn't run the simulation accurately. I'd think it's surely not too hard to make a superintelligence have "run this discrete, deterministic thing and then adopt its output as goal" as its goal, but perhaps there are difficulties around this. I'm optimistic that we can figure that out.
* It doesn't run the simulation (or adopt its result as goal) at all. Perhaps the requirement that the simulation be ran perfectly will make superintelligence too paranoid about being sure it has run the simulation correctly, and it will spend its entire existence getting more computronium to increase the likelyhood that the outcome it has computed is correct, but it's never certain enough to actually adopt the new goal as its own. There may be probabilistic workarounds to this; we'll need to look into it more.
* It fails in the various ways AI usually value drift from us — hacking its reward function, etc. While this concern may indeed remain, the fact that i'm proposing what seems to me like a easy to formalize goal is already a big improvement compared to the current state of affairs.

### Conclusion

This is a plan with a bunch of things that need work, but it doesn't seem *absurdly* hard to me; if anything, step 1 seems like the hardest, and I don't even know that we've *tried* throwing billions of dollars at it.

I share yudkowsky's current [gloomy outlook](https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy) on AI. The current route of "hey, maybe we should study things vaguely related to harnessing what neural nets do, and hope to be able to grab a miracle should it come up" seems like a pretty bad plan. I think, in comparison, the plan I outline here has better chances.

It is to be remembered that my vision is competing not against the likelyhood of superintelligence emergence, but against the likelyhood that alignment works. If pursuing mostly alignment gives us a 1e-10 chance of survival, and pursuing mostly my plan gives us a 1e-8 chance of survival, then it doesn't matter that *yes, superintelligence is still overwhelmingly likely to kill us*; we should still favor my plan. See also: [this post](ai-risk-plans.html) comparing plans.

The Peerless


bracing for the alignment tunnel
--------------------------------

it looks like we're gonna invent AI that [kills everyone](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer) before we figure out [AI alignment](https://en.wikipedia.org/wiki/AI_alignment).

what this means is that soon, [if not already](https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html), we are going to start bleeding [timelines](https://en.wikipedia.org/wiki/Many-worlds_interpretation), **hard**; by which i mean, an increasing ratio of multiverse-instants are gonna become dominated by unaligned AIs — and thus be devoid of population ([probably](above-paperclips-2.html)).

after that, there is a period in the (ever-diminishing amount of) surviving timelines, where we [ride on quantum immortality](https://en.wikipedia.org/wiki/Quantum_suicide_and_immortality) to solve alignment; after which, we finally reach [U-lines](timeline-codes.html), hopefully.

by many theories of [anthropics](https://www.lesswrong.com/posts/RnrpkgSY8zW5ArqPf/sia-greater-than-ssa-part-1-learning-from-the-fact-that-you), observing existing either before or after that period is a lot more likely than observing existing in it. before the period, it is more likely because there are a lot more populated timelines in which to exist; after the period, it is more likely because we can hopefully "repopulate horizontally" by allowing the population to increase again.

if i am correct in the reasoning in this post, then being someone who exists in this very narrow alignment "tunnel" is exceedingly unlikely (barring weird circumstances such as post-singularity mankind choosing to simulate many variants of the tunnel for some reason). indeed, if you do observe being in it, you should think that something weird is going on, and update against the narrative presented in this post.

yet, *again if i am correct*, this is a period where we need to hold tight and work on alignment, perhaps as quickly as possible in order to reduce astronomical waste. very few us's inhabit the tunnel, but those very few us's are the critical ones who we need to care about.

so we need to brace our minds for the alignment tunnel. we need to commit to be persons who, if we observe being in the tunnel, will keep working on alignment even if, *from inside those timelines*, it looks like the reasoning i'm presenting here can't possibly be right. this is perhaps a weird case of instrumental rationality.

(note that i'm not saying the conclusion of observing being in those timelines should be to stop working on alignment; perhaps we would want to work on it either way, in which case we don't have to worry about anything. but i worry that it could lead us to other places such as "oh, maybe this AI killing everyone business isn't real after all, or maybe a weird alien force is preventing us from dying somehow")

bracing for the alignment tunnel


should we implement free will?
------------------------------

if many worlds [is true](https://www.readthesequences.com/If-Many-Worlds-Had-Come-First), then in our current paradigm of physics, every possible outcome of any time step is instantiated.

however, once we [take over everything](ai-alignment-wolfram-physics.html), we might be able to hack physics to implement [exact computing](exact-minds-in-an-exact-world.html), and then upload ourselves to simulations running on those.

it is then to be decided: should we choose to keep forking every possibility? or should we select, at least in simulated minds, some outcome as *the only one* that gets instantiated, such as by selecting the most likely outcome and instatiating only that? or maybe it should be [up to persons and socities](%E2%88%80V.html)?

should we implement free will?


goals for emergency unaligned AI
--------------------------------

in [a previous post](when-in-doubt-kill-everyone.html) i talk about killing timelines — by making an AI fill them with something that's an easy to implement goal, such as paperclips — to avoid the even worse outcome of them becoming [S-lines](timeline-codes.html). in this post i wonder: can we do better than paperclips?

if the True Nature of the cosmos is [a universal program](universal-complete.html), then there could be some things to turn timelines into that consume less compute cycles; for example, maybe we somehow make an AI that makes its timeline run as little compute as possible. the cosmic universal program will then spend less many cycles running those dead timelines, and more running remaining live timelines — making survivors in them possibly "more real". in this sense, it may be that pruning timelines can be done without causing astronomical waste: compute time and therefore kinda ["realness"](udassa-time-steps.html) or ["soul juice"](https://www.lesswrong.com/posts/Hcc9fopx7sRexYhhi/anthropics-and-the-universal-distribution/) are redistributed to remaining timelines.

even if the "naive" most likely "implementation" of our universe consumes just as much compute regardless of what goes on in it, the universal computation will contain other "implementations" that "compress" compressible timeline-states, and we will reclaim cycles in them — and if they are good at compressing, those implementations might be where most of our realness juice is located anyways.

another possibility is to task an AI with turning its timeline into a state that is as identical as it can be to another timeline in which said AI's didn't appear. if it can achieve that, then we can kill timelines by replacing them with copies of alive timelines. this also recycles compute time, possibly more efficiently since it doesn't rely on compressing implementations.

goals for emergency unaligned AI


are there finitely many moral patients?
---------------------------------------

wouldn't it be neat if we didn't have to worry about [infinite ethics](https://www.lesswrong.com/posts/5iZTwGHv2tNfFmeDa/on-infinite-ethics) ?

i think it is plausible that there are finitely many moral patients.

the first step is to [deduplicate moral patients by computational equivalence](deduplication-ethics.html). this merges not only humans and other creatures we usually care about, but also probably a lot of [other potential sources of moral concerns](https://reducing-suffering.org/what-are-suffering-subroutines/).

then, i think we can restrict ourselves to patients in worlds that are discrete (like ours); even if there *were* moral patients in non-discrete worlds, it seems to me that from where we are, we could only access discrete stuff. so whether by inherent limitation, plain assumption, or just limiting the scope of this post, i'll only be talking about discrete agents (agents in discrete worlds).

once we have those limitations (deduplication and discreteness) in place, there are finitely many moral patients of any given size; the only way for an infinite variety of moral patients — or more precisely, moral patient moments — to come about is for some moral patients to grow in size forever. while infinite time seems plausible [even in this world](ai-alignment-wolfram-physics.html), it is not clear to me that whatever the hell a "moral patient" is can be arbitrarily complex; perhaps at a certain size, i start only caring about a *subset* of the information system that a "person" would consist of, a ["sub-patient"](deduplication-ethics.html).

are there finitely many moral patients?


making the UD and UDASSA less broken: identifying time steps
------------------------------------------------------------

the [universal distribution](https://www.lesswrong.com/posts/XiWKmFkpGbDTcsSu4/on-the-universal-distribution) ("UD") and [its applicability to anthropics](https://www.lesswrong.com/posts/Hcc9fopx7sRexYhhi/anthropics-and-the-universal-distribution/) apparently suffers from some issues.

one is uncomputability. i think a speed penalty seems reasonable; but as a more general solution, i think it is unreasonable to "wait for the machine to halt". instead, let us see turing machines as running forever, with getting stuck on a repeating state as a special case. then, the space of all computations on a given universal turing machine ("UTM") is a set of pairs `(input program, time step)`; or more simply, if we use a [universal complete](universal-complete.html) program, it is *just* a time step: every computation will be ran at some point.

this model feels quite natural to me, and seems like an easy way to rank priors: weigh the result of hypotheses by the inverse of the time step at which the universal complete program runs into them.

this also helps with [UDASSA](https://www.lesswrong.com/posts/Hcc9fopx7sRexYhhi/anthropics-and-the-universal-distribution): if you use a deterministic [model of computation](https://en.wikipedia.org/wiki/Model_of_computation) where every computation step only does a finite amount of stuff (like turing machines or SKI calculus, but unlike [wolfram-style graph rewriting](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/)) then you don't need a "claw" program to locate you within the world described by the world program; you are simply located at the set of time steps which happen to be updating the part of world that is *you*. this gives us a natural "locator" for persons within the space of all computation; it flattens together time, space, timelines (when a computation splits a world into multiple states all of which it then continues computing), and possible computation machines all into one neat linear sequence of steps.


values system as test-driven development
----------------------------------------

i realized something while reading [hands and cities on infinite ethics](https://www.lesswrong.com/posts/5iZTwGHv2tNfFmeDa/on-infinite-ethics): the work of determining the shape of [our values system](not-hold-on-to-values.html) is akin to [test-driven development](https://en.wikipedia.org/wiki/Test-driven_development).

we are designing a procedure (possibly looking for [the simplest one](https://en.wikipedia.org/wiki/Kolmogorov_complexity)) by throwing it at a collection of decision tests, and looking for which one matches our intuitions.

i wonder if a value-learning approach to AI alignment could look like trying to get superintelligence to find such a procedure; perhaps we feed it a collection of tests and it looks for the simplest procedure that matches those, and hopefully that extrapolates well to situations we didn't think of.

perhaps, even pre-superintelligence we can formalize values research as tests and try to come up with or generate a simple procedure which passes them while also being selected for simplicity.

why simplicity? doesn't occam's razor only apply to descriptive research, not prescriptive? that is true, but "what is the procedure that formalizes my values system" is indeed a prescriptive matter, in a way: we're trying to model something to the best factual accuracy we can.


the word "syntax" in programming, linguistics and LISP
------------------------------------------------------

in programming, the word "syntax" is used to describe quite superficial characteristics of a programming language: what an if/else statement looks like, how instructions are separated, how string literals are written and escaped, things like that.

in linguistics, the word carries a deeper meaning: "[syntax](https://en.wikipedia.org/wiki/Syntax)" refers to all the ways words are structured together and include grammatical tenses and the like; it cares about how the [tokens](https://en.wikipedia.org/wiki/Lexical_analysis#Token) relate to each other in a complex way which in programming would be moreso described as being about e.g. types.

in [LISP](https://en.wikipedia.org/wiki/Lisp_%28programming_language%29), as opposed to most programming languages, i feel like the term syntax has been reappropriated to be closer to its linguistic use: it refers to how macros (or [fexprs](https://web.cs.wpi.edu/~jshutt/kernel.html)) manipulate expressions in a more profound way, more akin to the linguistic usage of the term; see notably [Scheme](https://en.wikipedia.org/wiki/Scheme_%28programming_language%29)'s [`define-syntax`](http://www.shido.info/lisp/scheme_syntax_e.html) special form.

so while from an outside perspective, "LISP syntax is very simple" can be true in a straightforward sense, it misses the whole point of LISP: that its true, *deep* syntax is arbitrarily complex, dynamic, and manipulable.


experience/moral patient deduplication and ethics
-------------------------------------------------

suppose you can spend a certain amount of money (or effort, resources, etc) to prevent the spawning of a million rooms (in, let's say, simulations), with an exact copy of one random person in each. they will wake up in the rooms, spend a week not able to get out (basic necessities covered), then get tortured for a week, and then the simulations are shut down.

i want to split this hypothetical into four cases:

* the **identical** case (`I`): the million rooms and persons are exactly identical simulations.
* the **mildly different** case (`M`): the million rooms are the exact same, except that each room has, somewhere on one wall, a microscopically different patch of paint. the persons likely won't be able to directly observe the difference, but it *will* probably eventually cause the million brains to diverge from each other.
* the **quite different** case (`Q`): the million rooms will have different (random) pieces of music playing, as well as random collections of paintings on the walls, random collections of books, movies, video games, etc. to pass the time.
* the **very different** case (`V`): same as the **quite different** case, but on top of that the rooms actually contain a random person picked from random places all over the world instead of copies of the same person.

the point is that you should want to reduce suffering by preventing the scenario, but how much you care should be a function of whether/much you count the million different persons's suffering as *multiple* experiences.

it seems clear to me that one's caring for each case should increase in the order in which the cases are listed (that is, **identical** being the least cared about, and **very different** beig the most cared about); the question is more about the *difference* between consecutive cases. let's call those:

* `IM` = difference in caring between the **identical** case and the **mildly different** case
* `MQ` = difference in caring between the **mildly different** case and the **quite different** case
* `QV` = difference in caring between the **quite different** case and the **very different** case

currently, my theory of ethics deduplicates identical copies of moral patients (for reasons such as [not caring about implementation details](persistent-data-structures-consciousness.html)), meaning that i see the **mildly different** case as fundamentally different from the **identical** case. `IM > MQ ≈ QV`, and even `IM > (MQ + QV)`.

![](deduplication-ethics-1.png)

however, this strikes me as particularly unintutive; i *feel* like the **mildly different** case should get an amount of caring much closer to the **identical** case than the **quite different** case; i *feel* like i want to get `QV > MQ > IM`, or at least `QV > IM < MQ`; either way, definitely `IM < (MQ + QV)`.

![](deduplication-ethics-2.png)

here are the ways i can see out of this:

1. bite the bullet. commit to the idea that the slightest divergence between moral patients is enough to make them distinct persons worth caring about as different, much more than further differences. from a strict computational perspective such as [wolfram physics](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/), it might be what makes the most sense, but it seems quite unintuitive. this sort of caring about integer numbers of persons (rather than continuous quantities) maybe also feels mildly akin to [SIA's counting of world populations](https://www.lesswrong.com/posts/RnrpkgSY8zW5ArqPf/sia-greater-than-ssa-part-1-learning-from-the-fact-that-you), in a way, maybe.
2. interpolate difference: two moral patients count *more* if they are *more* different (rather than a strict criteria of perfect equality). this seems like the straightforward solution to this example, though if the curve is smooth enough then it runs into weird cases like caring more about the outcome of one population of 1000 people than another population of 1001 people, if the former is sufficiently more heterogenous than the latter. it kinda feels like i'm *rewarding* moral patients with extra importance for being diverse; but i'm unsure whether to treat the fact that i also happen to value diversity as coincidence or as evidence that this option is coherent with my values.
3. fully abandon deduplication: count the million moral patients as counting separately in the first case. this is the least appealing to me because from a functional, computational perspective it doesn't make sense to me, and [i can make up "implementation details" for the universe under which it breaks down](persistent-data-structures-consciousness.html). but, even though it feels as intangible as positing some magical observer-soul, maybe implementation details *do* matter?
4. de-monolithize moral patients; consider individual pieces of suffering instead of whole moral patients, in the hope that in the **mildly different** case i can extract a sufficiently similar suffering "sub-patient" and then deduplicate that sub-patient.

i think i'll tentatively stick to 1 because 2 feels *weird*, but i'll consider it more; as well as making room for the possibility that 3 might be right. finally, i'm not sure how to go about investigating 4; but compared to the other three it is at least materially investigatable — surely, either such a sub-patient can be isolated, or it can't.


recognition
-----------

(spoilers for episode 8 of [girls' last tour](https://en.wikipedia.org/wiki/Girls'_Last_Tour#Anime))

it feels to me like there is special about recognition by other persons. i will call it "recognition" — the fact of having one's experience be percieved by other persons, and valuing things about that.

recognition can be across not just space but also time; while people in the present recognizing the experiences of people in the past is a common thing, in the example above it is a society of the past that recognizes the experiences of future persons, through the symbol of the statue.

in general, i care a lot about recognition. i feel like things i do are particularly more meaningful if even just one person percieves it, in a way maybe akin to how [the subaltern finds their voice](https://www.youtube.com/watch?v=rVarn-m7o9k) (spoilers for Bladerunner).

this leads me to weird situations like valuing "pity points"; but in general, being a member of a society where some persons percieve my experiences, and others percieve theirs and so on in one wide web of mutual recognition, is a very homely feeling, and i feel like it is one i wouldn't want to be false (for example by everyone else being non-conscious high-fidelity NPCs as could exist in [dishonest simulations](https://www.lesswrong.com/posts/r7f58E8A85xLgWuqG/contact-with-reality)).

recognition


do not hold on to your believed intrinsic values — follow your heart!
---------------------------------------------------------------------

i posit the following framework for thinking about [intrinsic values](https://en.wikipedia.org/wiki/Instrumental_and_intrinsic_value) (hereby just called "values").

somewhere out there, there is **your value system**.

it is a function (in the mathematical sense) that takes as input *things*, and spits out *feelings*. that function is where your *true* values lie; they *are* that function.

how is that function encoded in your brain? who knows! i don't have an answer, and there may [not be an answer](https://en.wikipedia.org/wiki/Computational_irreducibility).

in conscious thought, however, you don't have access to the source code of that function, whatever it is. the best we can do for the moment seems to be to try thinking about it real hard, throw various things at it and examine the output ([perhaps through some mildly systematic process](core-vals-exist-selfdet.html)), and define another function that tries to approximate what your actual values are. this is hard and takes a lot of work. perhaps someday we will have a device that can scan your brain and give you a better idea of what your value function looks like, but at the moment that is not the case.

so, you build up an idea of what your values might be. here is where i think a lot of people make a mistake: they choose to believe strongly that this guess *is* their actual set of values (even though [it likely isn't](https://www.readthesequences.com/Value-Is-Fragile)). they [crystallize](value-crystallization.html) those values; they live by them, until they in turn become influenced by those values and perhaps *actually adopt them*. (the actual value function is mutable!)

this is generally bad; [you should want to preserve whatever your values are](https://en.wikipedia.org/wiki/Instrumental_convergence#Goal-content_integrity). hence the appeal that stands as the title of this post: do *not* hold on to the approximate function that is your best guess at what your value system is; you're only human, your guess is likely incorrect, and adopting it would run the risk of damaging your *actual* values; a function which, while hard to figure out, can be mutable, will certainly be mutated by acting as if your values are not what they are, and whose mutation you should generally want to avoid.

so, pay attention to your feelings. they are what is the output of your *actual* values system, by definition; follow your heart, not your reason's best guess.

note that this is *not* an appeal to favor deontology over consequentialism: how you feel can be about actions (deontology) or about outcomes (consequentialism), and which one it is is orthogonal to whether you follow that system, or whether you decide to follow your current best approximation of it. if you are consequentialist (as i recommend), just make sure to give your value system a full picture of what the outcome would look like, and *then* decide based on what feelings are produced by that.

meta note: this framework for thinking about values should be itself held with suspicion, as should probly any formal framework that concerns values. you should be careful about holding on to it just like you should have been careful about holding onto your believed values (careful enough to be able to consider the present post, for example). which isn't to say *don't believe what i just wrote*, but leave room for it to be wrong, partially or wholly.


my current pyramid of needs
---------------------------

![](pyramid-needs.svg)

([on concrete vs sublime](https://www.lesswrong.com/posts/SLw2MEgxFtiKAqgQ5/actually-possible-thoughts-on-utopia))

my current pyramid of needs


forking bitrate and entropy control
-----------------------------------

if physics is based on a computational framework such as [wolfram physics](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/), but plausibly even if not, such that not all states the universe can be in produce the same number as next possible states;

in addition, if i am to follow to conclusion my current belief that moral patients count as different when they start being functionally different in terms of computation, and that exact copies morally count as a single person (as it makes [not much sense](persistent-data-structures-consciousness.html) to believe otherwise);

and if ([as it seems to be the case](limiting-real-universes.html)) the universe values coherence and thus only a limited set of local outcomes can emerge from a given local situation, or at least outcomes are weighed by coherence;

then it makes sense to start caring about the amount of forking a given timeline goes through. which is to say: the amount of future states to be [instantiated](questions-cosmos-computations.html), be it directly next step or indirectly in the longer term.

in fact, if one calls what they care about *moral patients*, then we should care about the "forking bitrate" of moral patients. for example, we could want moral patients with a net negative future to be forked as little as possible, and moral patients with a net positive future to be forked as much as possible. considering forks are created over steps of time, and entropy seems to be a good measure for them, i think "bitrate" is an appropriate term for this; hence, *forking bitrate*.

if we're just talking about a place as small as earth, we can estimate that consequences rapidly ramificate around to all moral patients; and as such, it seems reasonable to think that the forking bitrate of all patients will tend to go about in the same direction.

so, if you see a quantum dice, should you throw it?

if you think the future of earth has expected net positive moral value, or has little enough suffering for your taste (depending on your moral framework), then yes: by throwing the (quantum) dice, you might be multiplying the amount of instances of that value by the number of possible outputs of the dice, by creating that many times more future timelines.

if not, then you shouldn't throw it.

(even in the absence of quantum effects, if one were to just move entropy around while [phase space being conserved](https://www.lesswrong.com/posts/QkX2bAkwG2EpGvNug/the-second-law-of-thermodynamics-and-engines-of-cognition), moving the entropy from not-moral-patients to moral-patientss (or whichever thing you can about) still has that effect, i think)

this can probly be expanded to much larger-scale entropy control — and also, if superintelligences care about it (and if they're to be aligned, we might want them to) we can expect them to use it to maximize their value. even a [paperclip maximizer](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer) can want to create as many timelines containing as many varied possible paperclips and arrangements thereof, if it is made to care about that.

forking bitrate and entropy control


political technology
--------------------

some communists like to claim that one of the problems with capitalism is the requirement for people to work. liberals can counter-argue that this is a fact of the state of technology and automation being not quite there yet, rather than what political system is in place: even under communism, someone would have to make the stuff.

similar claims can be made about other material conditions such as scarcity or the meat industry.

while not necessarily an unreasonable point, i feel like the liberal counter-argument misses a point and suggests an overly narrow view. indeed, part of the work of achieving communism is bringing about the technologies that enable communism to exist. the work of moving towards communism isn't as narrow as "what policies do we apply to the material conditions of today"; they extend to the transformation of these very material conditions.

some works of construction, while they might seem like innocuous new pieces of liberal society or even successes of that system, can in fact be steps on the road to overcoming capitalism. in fact, the last two ideas are not incompatible: it is to be remembered that communists have historically seen the move from feudalism to capitalism as a step of improvement, from which we can yet improve forth even more.

and technology is not an ever-forward-moving sequence of inevitable innovations; it has real directions that can be influenced and can even [decline](https://www.youtube.com/watch?v=pW-SOdj4Kkk) sometimes.

political technology


balancing utilitarianism
------------------------

suppose you have multiple values ("i want to be healthy but also i want to eat a lot of fries") or a value applying to multiple individuals ("i want both alice and bob to be happy"), but sometimes there are tradeoffs between these values. how do you resolve such situations ?

a simple weighed sum might suffice in many cases, but i feel like there are cases where this is not sufficient.

for example, consider a population of 5 persons, who you care about equally, and consider a simple scalar value you have for them, such as happiness.

now, consider the following three options:

* all individuals get 0.5 utility ("fair")
* one individual gets 0.9 utility, the other four get 0.4 utility ("bully")
* one individual gets 0.1 utility, the other four get 0.6 utility ("scapegoat")

if we are to use a simple sum, all three of these situations sum to 2.5 total utility; yet, i feel like something ought to be done to favor the fair situation over the other two (and then probably to favor the bully situation over the scapegoat situation?)

what i propose to address this is to apply a square root (or other less-than-one exponent) to the utilities of persons before summing, which has the effect of favoring more equal situations. in this case, we get:

* fair: 3.54 utility
* bully: 3.48 utility
* scapegoat: 3.41 utility

which does seem to produce the desired effect: in this situation, it maps to how i *feel* about things: fair > bully > scapegoat

balancing utilitarianism


hackable multiverse
-------------------

in [a previous post](brittle-physics.html) i talk about how hackable physics might allow a superintelligence to take over very quickly (perhaps faster than the speed of light).

in *[psi rewriting](psi-rewriting.html)* i propose that multiversehood can be more cleanly described as a particularly implemented feature of the cosmos, rather than an intrinsic thing.

but, if the cohabitation of multiple timelines is indeed an implemented feature rather than a primitive one, then there is a possibility that it is hackable, and that a superintelligence could hack across timelines.

now, it is to be noted that even if hackability exists, it might still be limited: perhaps there something like a light cone at play, or perhaps a given timeline can only access a finite number of other timelines.

it is to be remembered that timelines are not slots, they're not variables that hold values; timelines are *the values themselves*. still, hackability could mean some branches of the causality graph stop getting computed, for example.

either way, under these conditions, even quantum immortality might not save us from an X-risk superintelligence, and [given](https://deepmind.com/blog/article/Competitive-programming-with-AlphaCode) [recent](https://openai.com/blog/formal-math/) [developments](https://blog.eleuther.ai/announcing-20b/), we should panic a lot.

hackable multiverse


a cognitively hazardous idea
----------------------------

**caution: this post is a cognitively hazardous idea which may cause you to change your behavior and regret having learned about said idea. please don't proceed without informed consent, and please don't tell people about cognitive hazards without their own informed consent.**

six months ago, [i was worried about AI development going too far](were-all-doomed.html). today, things [keep](https://deepmind.com/blog/article/Competitive-programming-with-AlphaCode) [going](https://openai.com/blog/formal-math/) [badly](https://blog.eleuther.ai/announcing-20b/), to the point that i think it has become utilitarianistically useful to release this idea i've thought about recently.

in *[how timelines fall](how-timelines-fall.html)* i talk about how, if we are to keep observing a timeline that somehow survives [X-risks](https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence) even as they become increasingly likely, we should observe our timeline doing whatever chance things it takes to avoid them happening — including global economic collapse, if that's a likely enough event.

turns out, if *you, personally* choose to do something which might help AI development (and thus increase the probability of X-risk, or if you prefer, the amount of timelines that die to X-risk) then you make *yourself* something that will tend to have been incapacitated in surviving timelines. you might die, which would be unpleasant to the people who like you; but, you might also just eventually quit that job, or become unable to work for whatever reason.

a cognitively hazardous idea


how timelines fall
------------------

i've speculated that we are all together, as a civilization, quantum immortal; [timelines where we all die](timeline-codes.html) can somewhat [be ignored](brittle-physics.html), leaving us mostly just with concerns of [heaven vs hell](botched-alignment-and-awareness.html) timelines.

but, in the lucky timelines where we *do* keep avoiding an [X-risk](https://en.wikipedia.org/wiki/X-risk) [superintelligence](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer), what does that look like ?

it would be silly to expect that avoiding such a superintelligence would look like trying to press the button to turn it on but at the last minute the button jams, or trying to press the button to turn it on but at the last minute the person about to press it has a heart attack. indeed, bayes should make us think that we should expect it to look like whatever makes it likely that superintelligence fails to be implemented.

what does this look like ?

global nuclear war, broad economic collapse, great cataclysms or social unrest in cities where most of the AI development is done, and other largely unpleasant events.

don't expect the world to look like the god of anthropics is doing miracles to save us from superintelligence; expect the world to look like he's is slowly conspiring to do whatever it takes to make superintelligence unlikely to happen long in advance.

expect the god of anthropics to create AI winters and generally make us [terrible at software](https://www.youtube.com/watch?v=pW-SOdj4Kkk).

expect the god of anthropics to create plausible but still surprising reasons for the availability of tensor hardware to become scarce.

look around. does this look like a century where superintelligence appears ? yes, i think so as well. the god of anthropics has his work cut out for him. let's try and offer him timelines where AI development slows down more peacefully than if he has to take the initiative.

while some of us are working on aligning god, the rest of us should worry about aligning luck.

how timelines fall


uploading people for alignment purposes
---------------------------------------

as per [my utopian vision](%E2%88%80V.html), i've thought that an aligned AI would want to figure out how to upload us.

but, thinking about it more, it could be the other way around: if we can upload people in a deterministic simulation, this can buy us a lot of time to figure out alignment, as per [this post](noninterf-superint.html).

notably, the simulation could for example contain a single uploaded person (say, eliezer yudkowsky, or a bunch of copies of yudkowsky), which would save us from an arms-race type coordination problem; and while, on the outside, the superintelligence is killing everyone instantly to tile the universe with more compute to run this simulation, whoever's inside of it has plenty of time to figure things out (and hopefully [resurrect everyone once that's done](what-happens-when-you-die.html)).

this seems like a long shot, but [have you looked around?](https://www.lesswrong.com/s/n945eovrA3oDueqtq) this could be the [miracle](https://www.lesswrong.com/s/n945eovrA3oDueqtq/p/7im8at9PmhbT4JHsW) we need.

of course this could also turn into a [hell](botched-alignment-and-awareness.html) where infinite yudkowsky's are suffering forever everywhere. hopefully we can make another button which actually stops the simulation and tiles the universe with only benign paperclips, and maybe even make that button auto-activate if the yudkowsky is detected to be suffering or incoherent.

remember: [as long as the simulation is deterministic, superint can't force the uploaded yudkowsky to not shut it down](noninterf-superint.html), or force or even coerce him to do anything for that matter; it can only make the yudkowsky simulation run slower, which basically eventually achieves the same effect as either completing it or shutting it down.

uploading people for alignment purposes


questions about the cosmos and rich computations
------------------------------------------------

**computation**: a running state of any [model of computation](https://en.wikipedia.org/wiki/Model_of_computation); for example, a specific [SKI calculus expression](https://en.wikipedia.org/wiki/SKI_combinator_calculus), or a specific turing machine with its rules, current state, and current tape values. given that any model of computation can run the computations of any other model, it does not really matter which one we choose, and i will be juggling between different models throughout this post.

### 1: is any computation rich ?

**rich**: a computation is rich if it is generally [computationally irreductible](https://en.wikipedia.org/wiki/Computational_irreducibility). as a tentative formal definition for richness, i'm tempted to say that a computation is rich if there is no function able to generally predict any of its future states in a time [less than linear](https://en.wikipedia.org/wiki/Computational_complexity_theory) in the number of steps it would take to arrive at that state normally. for example, [rule 30](https://en.wikipedia.org/wiki/Rule_30) *looks* rich: it looks like to calculate the value of cell at index `i` at time step `j`, it generally takes about `O(abs(i) × j)` steps of computation. on the other hand, it looks like [rule 54 and rule 60](https://mathworld.wolfram.com/ElementaryCellularAutomaton.html) can generally have their cells predicted in time logarithmic to the number of computational steps it would naively take to arrive at them.

note that richness is not the same as halting: while a halting computation is necessarily not rich, a non-halting computation can either be non-rich (like rule 54), or rich (possibly like rule 30).

it seems clear to me that rich computations exist: for example, it is known that sorting a list of `n` elements takes `O(n × log(n))` steps, and thus a computation running a sorting algorithm of that complexity cannot have its result predicted in a smaller time complexity than it took to calculate naively. the ease with which i can demonstrate that, however, makes me doubt my tentative formal definition; maybe something more akin to [polynomial time complexity](https://arxiv.org/abs/1108.1791) would better capture the essence of computational irreductibility: perhaps a better determining question for richness could be "is there a function which can tell if a pattern looking like this will ever emerge in that computation, in time polynomial to the size of that pattern?" or "is there a function that can, in time polynomial to `n`, predict a piece of state that would naively take `aⁿ` steps to compute?"

### 2: does the cosmos instantiate any rich computation ?

to **instantiate a computation** means for that computation to, somewhere, eventually, be ran (forever or until it halts). i start from the fact that i'm observing a coherent-looking universe, deduce that at least *some* computation is happening, and which other computations are happening (as in, are being observed somewher, or which i could have observed). as [clarified before](limiting-real-universes.html), one can't just assume that all computations are equally happening: things look way too coherent for that, there seems to be a bias for coherence/simplicity (one which i've tentatively attributed to [how soon that computation spawns](less-quantum-immortality.html)).

looking at the cosmos (the set of instantiated computations) from a computational perspective, it seems like it contains at least our universe, which is expanding. if this expansion is, [as has been hypothesized](https://www.wolframphysics.org/technical-introduction/potential-relation-to-physics/cosmology-expansion-and-singularities/), caused by the computational substrate of the universe manufacturing new vertices of spacetime, and computations can run on this new fabric as it is produced, then it's possible that [some computations can run forever](ai-alignment-wolfram-physics.html), including potentially rich ones.

however:

### 3: does the cosmos contain causal bubbles ?

a **causal bubble** is a piece of computation that can run forever with the guarantee that it won't be physically interfered with from the outside; see [yes room above paperclips](above-paperclips-2.html).

for example, while one can build [a turing machine inside conway's game of life](https://www.conwaylife.com/wiki/Turing_machine), a stray object on the same conway's game of life plane can eventually collide with said machine and break its computational process.

however, in some [graph rewriting rulesets](https://en.wikipedia.org/wiki/Graph_rewriting), as well as in expression-rewriting systems with nested expressions such as a varient of [SKI calculus](https://en.wikipedia.org/wiki/SKI_combinator_calculus) or [lambda calculus](https://en.wikipedia.org/wiki/%CE%9B_calculus) where the evaluation rule expands all sub-expressions, some pieces of computation can run without ever being physically interfered with by other pieces of the computation.

(i'm specifying "*physically* interfered with" because acausal coordination or mutual simulation can lead to interference, but at least that interference is up to the singleton (such as a superintelligence) "running" said bubble (if any); they can just choose to never acausally coordinate and to never simulate other bubbles)

in our own spacetime, it seems like causal bubbles exist thanks to the expansion of spacetime: some pairs of points get further apart from one another faster than the speed of light, and thus should never be able to interact with one another so long as that expansion continues and FTL travel is impossible. under the perspective of wolfram physics, however, it is not clear that both of those things will necessarily be the case forever; spacetime might be [hackable](brittle-physics.html).

note that the splitting of universes with nondeterministic rules (such as ours with quantum mechanics) into different causally isolated timelines is another way for causal bubbles to exist, assuming the implementation of such a nondeterministic universe is that all possibilities are instantiated at any nondeterministic choice.

the presence of causal bubbles allows some pieces of spacetime to [survive superintellingences appearing in other pieces of spacetime](unoptimal-superint-doesnt-lose.html), while the absence of causal bubbles makes it that a superintelligence or collection of superintelligences probably eventually does take over everything.

if they exist, then causal bubbles are a blessing and a curse: they save us from alien superintelligences and, [between timelines](timeline-codes.html), from our own superintelligences, but they might also ensure that our own aligned superintelligence (once we have figured out alignment) cannot reach all computation, and thus that any random person has a good chance of existing in a bubble that hasn't been "saved" by our aligned superintelligence.

### 4. is a universal-complete computation instantiated ?

[**universal complete computations**](universal-complete.html) (such as the annex in [this post](less-quantum-immortality.html)) instantiate *all* computations, over time.

if one takes the perspective that a top-level "root" bubble existed first, then the answer to this question is up in the air.

maybe we are this root computation, and the deterministic fate of the cosmos (in all timelines) is, for example, for physics to break at some point and kill everything, or for a superintelligence to appear at some point and kill everything (the two being [pretty equivalent](brittle-physics.html)) leaving [no room for bubbles](above-paperclips.html).

maybe the root bubble [does spawn](above-paperclips-2.html) a finite and small (after deduplicating by identical computations) number of bubbles, and each of those is fated to be killed in its entirety.

or, maybe somewhere in this chain, one of the bubbles spawns *many* new, different bubbles, at which point it becomes likely enough that eventually one of those bubbles either is, or itself later spawns, a universal-complete program. in which case, the initial set of the "root" bubble and maybe a few other next bubbles serve together as merely the boot process for the program that will eventually spawn *all computations*.

it might be interesting to find out how small universal-complete programs can get, both in bubble-friendly frameworks like systematically-expanded SKI calculus, and bubble-unfriendly frameworks like cellular automata; to get an idea how likely they are to randomly be stumbled into.


brittle physics and the nature of X-risks
-----------------------------------------

suppose physics is hackable, and a hard to accomplish hack that requires intelligence (like a fancier version of [rowhammer](https://en.wikipedia.org/wiki/Rowhammer)) can break the fabric of spacetime — maybe in ways that said intelligence can take advantage of, such as embedding its computation into something that survives said breakage, in a way that could help such a superintelligence accomplish its goal.

we could expect that [boxing an AI](https://en.wikipedia.org/wiki/AI_box) could be really hard: even without access to the outside, it might be able to guesses physics and hack it, from the comfort of its box.

as usual in such [X-risk scenarios](timeline-codes.html), i believe we just [keep living only in timelines in which, by chance, we don't die](quantum-suicide.html).

these sort of hacks are not ruled out by [wolfram physics](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/). indeed, they are plausible, and can spread at some speed faster than the speed of light — because they can run in the substrate *underlying* spacetime — such that nobody would ever be able to observe such hacks: the hack reaches and destroys you before the result of the breakage can reach your sensory organs, let alone your brain.

so, maybe "dumb-goal" superintelligences such as [paperclip maximizers](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer) are popping up all over the place all the time and constantly ruining the immense majority of not-yet-hacked timelines, and we keep living in the increasingly few timelines in which they haven't done that yet.

now, let's stop for a minute, and consider: what if such a hack *isn't* hard ? what if it *doesn't* need an intelligent agent ?

what if, every planck time, every particle has a 99% chance of breaking physics ?

well, we would observe exactly the same thing: those hacked universes either become computationally simple or [boot up more universes](above-paperclips-2.html); either way, we don't survive in them, so we don't observe those hacks.

in this way, it is [S-lines and U-lines](timeline-codes.html) that are very special: outcomes in which we *survive*, thanks to a superintelligence with a "rich" goal. the rest is just timelines constantly dying, whether it be due to X-risk superintelligences, or just plain old physics happening to cause this.

in fact, let's say that the universe is [a nondeterministic graph rewriting system](https://en.wikipedia.org/wiki/Graph_rewriting) with a rule that sometimes allows everything be reduced to a single, inactive vertex. would this count as "sometimes everything is destroyed" ? or would this make more sense to be modeled as a weird quirk of physics where the graph of possible timelines includes the production of passive vertices all the time, which can be safely ignored ?

what if instead of a nondeterministic system, we have a deterministic one [which just happens to expand all timelines](psi-rewriting.html). in such a system, "different timelines" is no longer a primitive construct: it is merely an observation about the fact that such a system tends to, when ran, create from a given piece of data, several newer ones. let's say that in such a system there is a rule where from every piece of data we'd consider a timeline, numerous inert vertices are also created.

would we say "aha, look! every time a computation step happens, many inert vertices are created around it, and i choose to interpret this as the creation of many timelines (one per inert vertex) in which everyone in that universe dies, and others (new complex pieces of data) in which everything keeps existing",

or would we, in my opinion more reasonably, say "well, it looks like as a weird quirk of how this system runs, many inert vertices are popping up; but they're simple enough that we can just ignore them and only consider richer new pieces of data as *timelines* proper."

i believe, if we are to worry about what states this universe ends up in, we ought to use a measure of what counts as a "next state of this universe" that measures something about the richness of its content: maybe the amount of information, maybe the amount of computation going on, or maybe the number of moral patients. and, depending on what measure we use, "losing" timelines to paperclip maximizers (which turn the universe into something possibly simple) is no more of a big deal than "losing" timelines to a rewriting rule that sometimes creates inert vertices, and neither of which should really count as proper timelines.

otherwise we end up needlessly caring about degenerate states because of what we believe to be, but really isn't, an objective measure of what a timeline is.

*timelines* might be in the [map](https://en.wikipedia.org/wiki/Map%E2%80%93territory_relation), while what is in the territory is just *what we end up observing* and thus, computed states that contain us.

finally, what about universe states where *all* outcomes are an inert vertex or an otherwise simple universe (such as as infinite list of identical paperclips) ? while those might happen, and i'd say *would* count as X-risks, you don't need to consider simple states as timelines to make that observation: maybe some timelines end up in a state where *no* new states can be created (such as a locally truly terminated piece of computation), and others end up in a state where *only simple* new states are created. those ought to be considered equivalent enough, and are what a true X-risk looks like.