finding-earth-ud

2022-04-13

## finding earth in the universal program

this post expands on step one of [*the Peerless*](https://carado.moe/the-peerless.html): creating virtual people. brain scans-and-simulations are apparently still quite far off, so i'll be focusing on the second approach: resimulating earth and plucking out persons.

(one great side-advantage of this method is that, if we can relocate earth to pluck out persons for the simulation of alignment researchers, then we can later also relocate earth in order to restore it once we've solved alignment. so resimulating and locating earth, regardless of having early enough mind-plucking-out tech, is something we might need to do anyways.)

if compute is [infinite](https://carado.moe/ai-alignment-wolfram-physics.html) and [we don't mind being inefficient](https://carado.moe/udassa-time-steps.html), then we can use exponential or even infinite compute to locate earth. one approach is the following: create a big informational beacon — perhaps a copy of a huge portion of the internet, along with MRI scans of as many people as we can afford. then, we use some type of (non-intelligent) deterministic error-bound statistical location procedure to locate patterns that look like that beacon inside the [universal program](https://carado.moe/universal-complete.html). we can afford the statistical detection to be imperfect — if it misses on one encoding of earth, there will be different ones in the universal program.

because of the time penalty of the universal program, however, we may find just compressed copies of the beacon (instead of a full simulation of earth leading to the time at which we build that beacon), and because of the deterministic bound, we want need to stop on the first match; if this first match is *just* the beacon, without earth, then we fail; perhaps superintelligence can notice that it's not finding any nearby minds to pluck out, or perhaps it plucks out garbage. so we can start the universal program with not one step per program, but rather a very large number of steps — i hear stephen wolfram has estimates on the number of computation steps it takes to get to the current state of the universe. this will favor programs that takes every long to lead to the beacon, but are themselves shorter program.

(what if the first program to contain earth is itself a universal program *without* that huge constant, such that *it* finds the beacon before it finds earth? i am not sure how to address this. perhaps we can explore programs in an order that favors worlds that look like our physics instead of looking like discrete iterations of all computations?)

there's also the concern that the universal program, just like the [universal distribution](https://handsandcities.com/2021/10/29/on-the-universal-distribution/), [is malign](https://www.lesswrong.com/posts/Tr7tAyt5zZpdTwTQK/the-solomonoff-prior-is-malign). i'd think plain top-level earth, maybe especially as detectable by a simple enough beacon locator, would tend to occur before malign aliens emitting our beacon to trick us; but that's a risk to keep in mind.

if we *do* care about computational efficiency, then there are two main factors we need to account for:

* can our universe can be ran in polynomial time on whatever computers the superintelligence can build? for example, can it be ran in polynomial time on quantum computers, and can quantum computers be built? note that if this is the case we might need to step through *quantum steps* of *quantum programs* to run the search in the expected time. this doesn't need we need to build quantum computers outselves, mind you — superintelligence can just notice that a quantum computer would run the computations we describe efficiently, and build and use those.
* is the "seed" program for the universe small? intuitively i believe it is, and i find wolfram's efforts to reproduce the behavior of particles from the standard model using simple graph rewriting, to be evidence in that direction. that said, if it is large, then finding that program is an exponential search again — and so, again, we might need to build a search that "favors" our physics to save on exponential search time.

finally, we might want to put a hard bound on the number of tries the superintelligence will run to locate earth. the reason for that is that, if for some reason we messed up something in the beacon locator and it *never, ever* finds earth, then it will instantiate all computations, which appears to me to be a potential [S-risk](https://carado.moe/timeline-codes.html). in fact, even if we do find earth, it may not be worth it if we have to simulate exponentially much potential suffering before running our utopia — what if, after solving alignment, we have a great time, but then decide to eventually fade away after only polynomial time? then we will might have created exponentially much suffering in total.

### intermediary simulation

in case isolating minds from this simulation is hard, we could build an intermediary step between the location of earth in simulation-space, and booting the peerless simulation proper — superintelligence could, once it has located our beacon, get in touch with our organization *inside the simulation of earth*, and give it extraordinary computational (and maybe physical?) ability within the simulation to either take over everything, or figure out brain plucking-out and then let us press a big "ok, start now" button.

note, however, that we might not want to remain in this intermediary simulation for too long — it is still vulnerable to inner unaligned superintelligences, just like our top level reality is. we want to get to a safe, sandboxed, computationally weak environment as early as possible.

this is also a great argument for readying ourselves to build the beacon and utilize this contact-from-superintelligence as early as we can; indeed, to make that the first step of implementing the peerless plan. the reason for that is that the earlier we are able to take advantage of it, the earlier the time step of the simulation superintelligence can help us start bootstrapping towards the proper simulation of the peerless, and the less likely we are to be doomed by other superintelligences, if we need some intermediary "pre-peerless" simulation time.


utils-unit

2022-04-30

## a unit for utils

as utilitarians, it would be convenient for us to have an actual unit to measure utility, a number to be computed and compared.

the usual pick is money, but some people could have different judgments of the world that lead them to have different instrumental valuings of money even when they have the same intrinsic values; and also some people could intrinsically value money.

the unit i propose, to measure how much an agent cares about a thing, is a ratio of that person's "total caring pie". for example, you could intrinsically value 70% something and 30% something else; and then i'm sure we can figure out some math that makes sense (probly inspired from probability theory) to derive our valuings of instrumental values from that.

this seems like the least biased way to measure utils. the only criticism i can think of is that it breaks if two agents have different amounts of total valuing: perhaps one person *just has more total caring* than another.

however, is this testable in any way? is there any situation where one agent would act differently than another if they have the same intrinsic valuing proportions but one of them has a million times more total caring? i don't think so: the idea that inaction counts, seems to me to track either willpower or just different valuings of not doing effort.


rationalist-by-necessity

2020-12-22 ★

## Rationalist by necessity

in [The Sequences](https://www.readthesequences.com/), Eliezer Yudkowsky [describes rationality](https://www.readthesequences.com/What-Do-I-Mean-By-Rationality) as

1. **Epistemic rationality**: systematically improving the accuracy of your beliefs.
2. **Instrumental rationality**: systematically achieving your values.

now, personally, i [intrinsically value](https://carado.moe/core-vals-exist-selfdet.html) a bunch of things, but having accurate beliefs isn't necessarily one of them; for me, rationality is an [instrumental value](https://en.wikipedia.org/wiki/Instrumental_and_intrinsic_value) in that it helps me achieve my other values better.

in general, i value people being able to do whatever they want, and as such they shouldn't necessarily have to form accurate beliefs if they don't care to. in fact, forming inaccurate beliefs is a great source of culture, and culture is something that i *do* personally intrinsically value.

but we live in the era of liberal democracies, where society requires people to form accurate beliefs, because they're the ones directing society through elections. i see the need for people to be rationalist as an unfortunate necessity; hopefully a need we can be rid of when we [reach a topia where human decisions are no longer the pillar of civilization](https://carado.moe/two-principles-for-topia.html).

not, of course, that there's anything wrong with any individual or even group choosing to intrinsically value rationality. the part i care about is that it being a choice.


rough-ai-risk-estimates

2022-05-18

## my rough AI risk estimates

the numbers are spoilered so you can make your own guess without getting anchored. click on the text to reveal the number.

these are my pretty rough estimates for what happens to our current civilization:

* we robustly [get past](https://en.wikipedia.org/wiki/AI_alignment) AI risk0.01%
* our civilization dies in a non-AI-existential-risk way, leaving room for future civilization0.1%
* we [die](https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence) of AI99.89%

if we die of AI, when does it happen?

* 2020s60%
* 2030s30%
* 2040s5%
* the remaining % get spilled over the remaining future

rember that if we get past those decades without dying, that's not evidence that i got those wrong; that's evidence that either i got those wrong, or our civilization benefits from some amount of quantum immortality.

part of my reasons for believing in those odds is private information.


prototype-realities

2020-11-18

*(this post may contain some very vague spoileryness about the video game Outer Wilds)*

## A Prototypeness Hierarchy of Realities

one property of many video games that i felt the most when playing the excellent [Outer Wilds](https://store.steampowered.com/app/753640/Outer_Wilds/) was *prototypeyness*.

many games, and especially that one, feel like they are prototypes for reality to some extent; they try to extract some essence of what is interesting about this world, without having the ability to implement all of it in a fully dynamic way, and thus hardcoding the rest.

now, this aspect of prototypeyness is sufficiently present in Outer Wilds that i ended up asking myself the question: what would real life (this universe where earth is) be a prototype for ? and i think the answer is:

real life is a prototype for living in virtual realities/cyberspace.

once we upload ourselves to computers (a good thing!) we will be able to make the entirety of the substrate that individuals interact with way more flexible; inhabit spaces of any number of dimensions or maybe not even spaces at all and just graphs (as is the shape of the web), modify our minds in ways meat brains wouldn't support, basically utilize any type of computational constructs we want with no regard for most limitations, depending on reality only as a substrate to run the computronium for it all.

like the step between prototypey video games and reality, it is one of a nearly definitional boundary in scale of computing power, and one whose non-prototype side i'm very interested in.


cosmic-missing-outs

2021-10-14

## cosmic missing outs

this might be a complete waste of brainflops, but sometimes i wonder about "cosmic missing outs".

my typical example for those is the culture of modern japan.

imagine timelines where japan never became the country it did, and we never got its culture. that'd be a huge thing to miss out on, right? the second best thing might be korean culture or something like that.

but, now that you've imagined this timeline that is missing out on modern japan culture, imagine the opposite: there are timelines out there that have those great cultures of countries that we're missing out on, that us missing out on is kind of on the same scale as those other timelines missing out on japan's culture.

i'm talking about this because i just thought of some other things kind of of this type:

what are some unknown things that we are missing out on, that us missing out on is kind of like if other timelines were missing out on music?

what are some unknown things that we are missing out on, that us missing out on is kind of like if other timelines were missing out on philosophy, science, or math?

these speculations are the closest i can get to putting human minds into perspective and considering the existence of things entirely outside of human conception, the way many things are entirely outside of a mouse or ant's ability to conceive.

to be clear: i still can't have that consideration, this is only the closest i get, but it's not quite there.


deduplication-ethics

2022-03-06

## experience/moral patient deduplication and ethics

suppose you can spend a certain amount of money (or effort, resources, etc) to prevent the spawning of a million rooms (in, let's say, simulations), with an exact copy of one random person in each. they will wake up in the rooms, spend a week not able to get out (basic necessities covered), then get tortured for a week, and then the simulations are shut down.

i want to split this hypothetical into four cases:

* the **identical** case (`I`): the million rooms and persons are exactly identical simulations.
* the **mildly different** case (`M`): the million rooms are the exact same, except that each room has, somewhere on one wall, a microscopically different patch of paint. the persons likely won't be able to directly observe the difference, but it *will* probably eventually cause the million brains to diverge from each other.
* the **quite different** case (`Q`): the million rooms will have different (random) pieces of music playing, as well as random collections of paintings on the walls, random collections of books, movies, video games, etc. to pass the time.
* the **very different** case (`V`): same as the **quite different** case, but on top of that the rooms actually contain a random person picked from random places all over the world instead of copies of the same person.

the point is that you should want to reduce suffering by preventing the scenario, but how much you care should be a function of whether/much you count the million different persons's suffering as *multiple* experiences.

it seems clear to me that one's caring for each case should increase in the order in which the cases are listed (that is, **identical** being the least cared about, and **very different** beig the most cared about); the question is more about the *difference* between consecutive cases. let's call those:

* `IM` = difference in caring between the **identical** case and the **mildly different** case
* `MQ` = difference in caring between the **mildly different** case and the **quite different** case
* `QV` = difference in caring between the **quite different** case and the **very different** case

currently, my theory of ethics deduplicates identical copies of moral patients (for reasons such as [not caring about implementation details](https://carado.moe/persistent-data-structures-consciousness.html)), meaning that i see the **mildly different** case as fundamentally different from the **identical** case. `IM > MQ ≈ QV`, and even `IM > (MQ + QV)`.

![](deduplication-ethics-1.png)

however, this strikes me as particularly unintutive; i *feel* like the **mildly different** case should get an amount of caring much closer to the **identical** case than the **quite different** case; i *feel* like i want to get `QV > MQ > IM`, or at least `QV > IM < MQ`; either way, definitely `IM < (MQ + QV)`.

![](deduplication-ethics-2.png)

here are the ways i can see out of this:

1. bite the bullet. commit to the idea that the slightest divergence between moral patients is enough to make them distinct persons worth caring about as different, much more than further differences. from a strict computational perspective such as [wolfram physics](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/), it might be what makes the most sense, but it seems quite unintuitive. this sort of caring about integer numbers of persons (rather than continuous quantities) maybe also feels mildly akin to [SIA's counting of world populations](https://handsandcities.com/2021/09/30/sia-ssa-part-1-learning-from-the-fact-that-you-exist/), in a way, maybe.
2. interpolate difference: two moral patients count *more* if they are *more* different (rather than a strict criteria of perfect equality). this seems like the straightforward solution to this example, though if the curve is smooth enough then it runs into weird cases like caring more about the outcome of one population of 1000 people than another population of 1001 people, if the former is sufficiently more heterogenous than the latter. it kinda feels like i'm *rewarding* moral patients with extra importance for being diverse; but i'm unsure whether to treat the fact that i also happen to value diversity as coincidence or as evidence that this option is coherent with my values.
3. fully abandon deduplication: count the million moral patients as counting separately in the first case. this is the least appealing to me because from a functional, computational perspective it doesn't make sense to me, and [i can make up "implementation details" for the universe under which it breaks down](https://carado.moe/persistent-data-structures-consciousness.html). but, even though it feels as intangible as positing some magical observer-soul, maybe implementation details *do* matter?
4. de-monolithize moral patients; consider individual pieces of suffering instead of whole moral patients, in the hope that in the **mildly different** case i can extract a sufficiently similar suffering "sub-patient" and then deduplicate that sub-patient.

i think i'll tentatively stick to 1 because 2 feels *weird*, but i'll consider it more; as well as making room for the possibility that 3 might be right. finally, i'm not sure how to go about investigating 4; but compared to the other three it is at least materially investigatable — surely, either such a sub-patient can be isolated, or it can't.


questions-cosmos-computations

2022-01-07 ★

## questions about the cosmos and rich computations

**computation**: a running state of any [model of computation](https://en.wikipedia.org/wiki/Model_of_computation); for example, a specific [SKI calculus expression](https://en.wikipedia.org/wiki/SKI_combinator_calculus), or a specific turing machine with its rules, current state, and current tape values. given that any model of computation can run the computations of any other model, it does not really matter which one we choose, and i will be juggling between different models throughout this post.

### 1: is any computation rich ?

**rich**: a computation is rich if it is generally [computationally irreductible](https://en.wikipedia.org/wiki/Computational_irreducibility). as a tentative formal definition for richness, i'm tempted to say that a computation is rich if there is no function able to generally predict any of its future states in a time [less than linear](https://en.wikipedia.org/wiki/Computational_complexity_theory) in the number of steps it would take to arrive at that state normally. for example, [rule 30](https://en.wikipedia.org/wiki/Rule_30) *looks* rich: it looks like to calculate the value of cell at index `i` at time step `j`, it generally takes about `O(abs(i) × j)` steps of computation. on the other hand, it looks like [rule 54 and rule 60](https://mathworld.wolfram.com/ElementaryCellularAutomaton.html) can generally have their cells predicted in time logarithmic to the number of computational steps it would naively take to arrive at them.

note that richness is not the same as halting: while a halting computation is necessarily not rich, a non-halting computation can either be non-rich (like rule 54), or rich (possibly like rule 30).

it seems clear to me that rich computations exist: for example, it is known that sorting a list of `n` elements takes `O(n × log(n))` steps, and thus a computation runnig a sorting algorithm of that complexity cannot have its result predicted in a smaller time complexity than it took to calculate naively. the ease with which i can demonstrate that, however, makes me doubt my tentative formal definition; maybe something more akin to [polynomial time complexity](https://arxiv.org/abs/1108.1791) would better capture the essence of computational irreductibility: perhaps a better determining question for richness could be "is there a function which can tell if a pattern looking like this will ever emerge in that computation, in time polynomial to the size of that pattern?" or "is there a function that can, in time polynomial to `n`, predict a piece of state that would naively take `aⁿ` steps to compute?"

### 2: does the cosmos instantiate any rich computation ?

to **instantiate a computation** means for that computation to, somewhere, eventually, be ran (forever or until it halts). i start from the fact that i'm observing a coherent-looking universe, deduce that at least *some* computation is happening, and which other computations are happening (as in, are being observed somewher, or which i could have observed). as [clarified before](https://carado.moe/limiting-real-universes.html), one can't just assume that all computations are equally happening: things look way too coherent for that, there seems to be a bias for coherence/simplicity (one which i've tentatively attributed to [how soon that computation spawns](https://carado.moe/less-quantum-immortality.html)).

looking at the cosmos (the set of instantiated computations) from a computational perspective, it seems like it contains at least our universe, which is expanding. if this expansion is, [as has been hypothesized](https://www.wolframphysics.org/technical-introduction/potential-relation-to-physics/cosmology-expansion-and-singularities/), caused by the computational substrate of the universe manufacturing new vertices of spacetime, and computations can run on this new fabric as it is produced, then it's possible that [some computations can run forever](https://carado.moe/ai-alignment-wolfram-physics.html), including potentially rich ones.

however:

### 3: does the cosmos contain causal bubbles ?

a **causal bubble** is a piece of computation that can run forever with the guarantee that it won't be physically interfered with from the outside; see [yes room above paperclips](https://carado.moe/above-paperclips-2.html).

for example, while one can build [a turing machine inside conway's game of life](https://www.conwaylife.com/wiki/Turing_machine), a stray object on the same conway's game of life plane can eventually collide with said machine and break its computational process.

however, in some [graph rewriting rulesets](https://en.wikipedia.org/wiki/Graph_rewriting), as well as in expression-rewriting systems with nested expressions such as a varient of [SKI calculus](https://en.wikipedia.org/wiki/SKI_combinator_calculus) or [lambda calculus](https://en.wikipedia.org/wiki/Λ_calculus) where the evaluation rule expands all sub-expressions, some pieces of computation can run without ever being physically interfered with by other pieces of the computation.

(i'm specifying "*physically* interfered with" because acausal coordination or mutual simulation can lead to interference, but at least that interference is up to the singleton (such as a superintelligence) "running" said bubble (if any); they can just choose to never acausally coordinate and to never simulate other bubbles)

in our own spacetime, it seems like causal bubbles exist thanks to the expansion of spacetime: some pairs of points get further apart from one another faster than celerity, and thus should never be able to interact with one another so long as that expansion continues and FTL travel is impossible. under the perspective of wolfram physics, however, it is not clear that both of those things will necessarily be the case forever; spacetime might be [hackable](https://carado.moe/brittle-physics.html).

note that the splitting of universes with nondeterministic rules (such as ours with quantum mechanics) into different causally isolated timelines is another way for causal bubbles to exist, assuming the implementation of such a nondeterministic universe is that all possibilities are instantiated at any nondeterministic choice.

the presence of causal bubbles allows some pieces of spacetime to [survive superintellingences appearing in other pieces of spacetime](https://carado.moe/unoptimal-superint-doesnt-lose.html), while the absence of causal bubbles makes it that a superintelligence or collection of superintelligences probably eventually does take over everything.

if they exist, then causal bubbles are a blessing and a curse: they save us from alien superintelligences and, [between timelines](https://carado.moe/timeline-codes.html), from our own superintelligences, but they might also ensure that our own aligned superintelligence (once we have figured out alignment) cannot reach all computation, and thus that any random person has a good chance of existing in a bubble that hasn't been "saved" by our aligned superintelligence.

### 4. is a universal-complete computation instantiated ?

[**universal complete computations**](https://carado.moe/universal-complete.html) (such as the annex in [this post](https://carado.moe/less-quantum-immortality.html)) instantiate *all* computations, over time.

if one takes the perspective that a top-level "root" bubble existed first, then the answer to this question is up in the air.

maybe we are this root computation, and the deterministic fate of the cosmos (in all timelines) is, for example, for physics to break at some point and kill everything, or for a superintelligence to appear at some point and kill everything (the two being [pretty equivalent](https://carado.moe/brittle-physics.html)) leaving [no room for bubbles](https://carado.moe/above-paperclips.html).

maybe the root bubble [does spawn](https://carado.moe/above-paperclips-2.html) a finite and small (after deduplicating by identical computations) number of bubbles, and each of those is fated to be killed in its entirety.

or, maybe somewhere in this chain, one of the bubbles spawns *many* new, different bubbles, at which point it becomes likely enough that eventually one of those bubbles either is, or itself later spawns, a universal-complete program. in which case, the initial set of the "root" bubble and maybe a few other next bubbles serve together as merely the boot process for the program that will eventually spawn *all computations*.

it might be interesting to find out how small universal-complete programs can get, both in bubble-friendly frameworks like systematically-expanded SKI calculus, and bubble-unfriendly frameworks like cellular automata; to get an idea how likely they are to randomly be stumbled into.


against-ai-alignment

2021-11-08

## against AI alignment ?

[i usually consider AI alignment to be pretty critical](https://carado.moe/were-all-doomed.html). that said, there are some ways in which i can see the research that is generally associated with alignment to have more harmful potential than not, if it is applied.

this is a development on my idea of [botched alignment](https://carado.moe/botched-alignment-and-awareness.html): just like AI tech is dangerous if it's developed before alignment because unaligned AI might lead to [X-lines](https://carado.moe/timeline-codes.html), alignment is dangerous because it lets us align AI to things we think we want, but aren't actually good; which sounds like it could lead to an increase in the ratio of S-lines to U-lines.

with this comes a sort of second [orthogonality thesis](https://www.lesswrong.com/tag/orthogonality-thesis), if you will: one between what we think we want and what is actually good. note that in both cases, the orthogonality thesis is a *default* position: it could be wrong, but we shouldn't assume that it is.

determining what is good is very hard, and in fact has been the subject of the field of *ethics*, which has been a work in progress for millenia. and, just like we must accomplish alignment before we accomplish superintelligence if we are to avoid X-risks, we might want to consider getting ethics accomplished before we start using alignment if we are to avoid S-risks, which should be a lot more important. or, at least, we should heavily consider the input of ethics into alignment.

things like [my utopia](https://carado.moe/∀V.html) are merely patches to try and propose a world that *hopefully* doesn't get *too bad* even after a lot of time has passed; but they're still tentative and no doubt a billion unknown and unknown unknown things can go wrong in them.

it is to be emphasized that both parts of the pipeline are important: we must make sure that what we think is good is what is actually good, and then we must ensure that that is what AI pursues. maybe there's a weird trick to implementing what is good directly without having to figure it out ourselves, but i'm skeptical, and in any case we shouldn't go around assuming that to be the case. in addition, i remain highly skeptical of approaches of "value learning"; that would seem like it would be *at most* as good as aligning to what we think is good.

so, it is possible that just as i have strongly opposed doing AI tech research until we've figured out AI alignment, i might now raise concerns about researching AI alignment without progress on, and input from, ethics. in fact, there's a possibility that putting resources into AI tech over alignment could be an improvement: [we should absolutely avoid S-risks, even at the cost of enormously increased X-risks](https://carado.moe/when-in-doubt-kill-everyone.html).


fermi-paradox

2021-06-16

## my answer to the fermi paradox

the [fermi paradox](https://en.wikipedia.org/wiki/Fermi_paradox) asks, if aliens are supposedly so statistically prevalent, why we haven't received any radio signals from them.

here is my independently-developed (though probly not original) answer:

statistically, it seems reasonable that civilizations would [accidentally invent unaligned superintelligence](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer) not long after inventing radio signals (in our case, a couple centuries). in order to percieve those signals, you would need to exist *after* your planet receives those signals, but *before* your planet receives that unaligned superintelligence's [expanding sphere of death](https://carado.moe/moral-cost-of-unaligned-ai.html), which might very well travel at celerity or near-celerity.

thus, given the low probability, it is not surprising that we haven't percieved those; for any given alien civilization, in a given timeline, we either haven't received their radio signals, or have already been killed by them. seeing as we're alive, this timeline must be one of the former.


exact-minds-in-an-exact-world

2021-10-13

## exact minds in an exact world

[in the sequences](https://www.readthesequences.com/Zero-And-One-Are-Not-Probabilities) it is argued that 0 and 1 are not probabilities; that these "certainty ratios" aren't meaningful. but, i can think a situation that challenges this.

imagine a fully deterministic world — for example, running on [a cellular automaton](https://en.wikipedia.org/wiki/Cellular_automata) — and imagine that in this world there are some intelligences (either artificial or natural) that utilize this determinism to have the ability to make flawless logical deductions (for example, [automated theorem proving](https://en.wikipedia.org/wiki/Automated_theorem_proving) algorithms running on computers that cannot ever have undetected [hardware failures](https://en.wikipedia.org/wiki/Soft_error)). for example, if they think about mathematics, under the axioms under which they work, 2 + 2 will always equal to 4, and doing any mathematical computation will either result in them knowing they don't have the computational resources to do the operation, or the result being guaranteedly true with the same certainty as that the cellular's automaton's rules will be applied next tick.

now, these beings still have a use for probability and statistics: those can be used to talk about parts of the world that they don't have complete information about. but, there will be some contexts, both purely in their minds (such as logic or math) or sometimes in the real world (they could make assessments like "this box cannot contain any [spaceship](https://en.wikipedia.org/wiki/Spaceship_%28cellular_automaton%29) of a certain size") that *will* be, functionally, certain.

it could be argued that they *should* still be weighing everything by the probability that there might be unknown unknowns; for example, their cellular automaton might have rules that apply only very rarely, and that they never got a chance observe yet but might yet observe later. but, let's say that they *assume* the rules of their world are exactly as they think, and let's say that they happen to be correct in that assessment. does that not make some of their deductions actually entirely certain?


above-paperclips

2021-11-20

## no room above paperclips

(edit: see also [*yes room above paperclips?*](https://carado.moe/above-paperclips-2.html))

when presented with the idea of a [paperclip-maximizing unaligned superintelligence](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer), people sometimes mention the possibility that sure, the universe gets tiled with paperclips, but maybe there's [slack](https://thezvi.wordpress.com/2017/09/30/slack/) in how paperclips are arranged, and that maybe nice things can exist again "above" paperclips.

(note: this relates to the idea of ["daemon-free"ness in "minimal circuints"](https://www.lesswrong.com/posts/nyCHnY7T5PHPLjxmN/open-question-are-minimal-circuits-daemon-free))

i think it's a reasonable line of thinking, but it's short-sighted: let's think about what happens next. eventually, above those paperclips, some evolutionary process may take place, leading (possibly, such as in our case, through the step of a technological species) eventually to a superintelligence taking over everything. given that *the entire cosmos* gets tiled with paperclips [*possibly forever*](https://carado.moe/ai-alignment-wolfram-physics.html), and that a superintelligent singleton taking over everything is irreversible (short of everything dying forever), in all likelyhood in the long term in any piece of universe not already actively managed by a superintelligence, eventually either everything dies forever, or a superintelligence takes over everything forever.

and then what? either this new superintelligence cares about "upwards", and has some plan for how its own paperclips are arranged (such as into more "macro"-paperclips), or it doesn't and the cycle begins again.

given that the outcome of an "alien" superintelligence's takeover is probly a worse outcome than the takeover of a superintelligence of our own (we should expect them to be about as incompetent as us at alignment, but to have values [less aligned to ours](https://www.lesswrong.com/posts/HawFh7RvDM4RyoJ2d/three-worlds-collide-0-8)), we need to care about our own iteration first, it's our best bet.

the point is, eventually for any given local patch of spacetime, either a superintelligence explosion is reached or everything dies forever. this can't be avoided, even by "climbing up" on substrates, so we should care about alignment now; we can't just hope that things are okay despite paperclips.


alignment-optimization-processes

2021-10-23

## alignment is an optimization processes problem

i like to talk about [AI](https://www.lesswrong.com/tag/ai) alignment a lot, but the matter of alignment is really general to *optimization processes* in general.

here are some ways it applies to some other areas:

### natural selection

natural selection is an optimization process that improves the survival and duplication of inheritable traits (genes) in living beings.

it is not intelligent: there is no agent involved in this process which is able to make decisions by looking ahead into the future at what consequences those decisions will have; with the possible exception of humans making rational decisions about what will maximize their amount offspring.

it is completely multipolar: basically no agents in this process (either genes themselves, or individuals or populations carrying those genes) have the ability to coordinate decisions with one another, since they're not even intelligent.

the default of natural selection is genes whose only purpose is to be better at duplicating themseles.

one way in which we've aligned this process is by breeding: by selecting the individuals we like best among, for example, crops, cattle, or dogs, we've been able to align the process of gene selection to respond to what we value rather than the default.

### economics

the economy is an optimization process that improves economic efficiency.

it is intelligent: actors in the economy, ranging from individuals to states and giant conglomerates, have the ability to make intelligent decisions about the long term.

it is fairly multipolar: while they don't use it much, states do have overriding power over companies (they determine what's legal or not, after all), and also economic agents are able to coordinate to an extent using contracts and trusts. nevertheless, it is still largely multipolar, with agents overall competing with one another.

the default of economics is the optimization out of anything that doesn't generate maximally much resources: the optimizing out of people when they become the unoptimal form of labor because of automation, and the strip-mining of the universe to acquire ever more resources with which to create more machines to mine even more resources, and so on.

the way we align economics is through taxes, redistribution, and the like. redistribution like [UBI](https://carado.moe/ubi.html) aligns the economy to serves the demand of people, while tax externalities can align economic agents to take steps to preserve nice things, such as avoiding pollution.

### electoralism

electoral representative democracy is an optimization process that improves voter satisfaction.

it is intelligent: the agents competing for the reward, here political parties, are able to make decisions about the future. some organisms even plan for the very long term, taking steps to improve their chances when they become parties, long before they do.

it is fairly multipolar: like economics, while parties can coordinate and ally with one another, they are still competing agents at the end of the day, with no central authority to guide them and solve coordination.

the default of electoralism is parties throwing all values under the bus to do whatever gets them and keeps them in office for as long as possible.

the way we align electoralism is by having universal sufferage on the one hand, which makes it that it is the population that parties must try to satisfy; and the various apparatii of liberal democracies (journalism and free press, public debate, education of the voting public, etc), which we'd hope would help that voting population determine which parties do indeed implement policies that satisfy their demand.


balancing-utilitarianism

2022-02-04

## balancing utilitarianism

suppose you have multiple values ("i want to be healthy but also i want to eat a lot of fries") or a value applying to multiple individuals ("i want both alice and bob to be happy"), but sometimes there are tradeoffs between these values. how do you resolve such situations ?

a simple weighed sum might suffice in many cases, but i feel like there are cases where this is not sufficient.

for example, consider a population of 5 persons, who you care about equally, and consider a simple scalar value you have for them, such as happiness.

now, consider the following three options:

* all individuals get 0.5 utility ("fair")
* one individual gets 0.9 utility, the other four get 0.4 utility ("bully")
* one individual gets 0.1 utility, the other four get 0.6 utility ("scapegoat")

if we are to use a simple sum, all three of these situations sum to 2.5 total utility; yet, i feel like something ought to be done to favor the fair situation over the other two (and then probably to favor the bully situation over the scapegoat situation?)

what i propose to address this is to apply a square root (or other less-than-one exponent) to the utilities of persons before summing, which has the effect of favoring more equal situations. in this case, we get:

* fair: 3.54 utility
* bully: 3.48 utility
* scapegoat: 3.41 utility

which does seem to produce the desired effect: in this situation, it maps to how i *feel* about things: fair > bully > scapegoat


two-principles-for-topia

2020-11-15

(edit: this post is *sort of* superceded by [∀V](https://carado.moe/∀V.html))

## Two Principles For Topia

the more i think about it, the less i think the solution to [Moloch](https://web.archive.org/web/20140730043944/http://slatestarcodex.com/2014/07/30/meditations-on-moloch/) is a single benevolent Elua; or, in other terms, we shouldn't implement Elua, but we should enact reasonable principles which Elua might want to implement herself.

here are what i currently believe to be the two principles that form the basis of a largely [freedom-conserving](https://carado.moe/core-vals-exist-selfdet.html) utopia:

* the first principle, Voluntaryism, consists of NAP, UBI, and population control.

	* the systematic enforcement of the [non-aggression principle](https://en.wikipedia.org/wiki/Non-aggression_principle) (NAP) to guarantee agency and freedom of association,
	* mandatory redistribution enough for every individual to be guaranteed a reasonable-with-[slack](https://thezvi.wordpress.com/2017/09/30/slack/) living (UBI) (where living includes basic resources and healthcare up to immortality), and
	* enough population control to guarantee this redistribution can even happen in the first place in a world with (even locally) limited resources,

	are to be the basis of a reasonable [voluntary](https://en.wikipedia.org/wiki/Voluntaryism) world.

	secondary notions like taxation on [externalities](https://en.wikipedia.org/wiki/Externality) and usage of [the commons](https://en.wikipedia.org/wiki/Commons) help make that UBI tangible ("why does the UBI currency have value ?" → because it's what eventually one must pay those taxes with) and reasonably redistribute ressources so as to help all persons benefit from growth.

* the second principle is the dismantlement of non-person forces (DNPF).

	what i mean by a non-person force is any phenomenon that interacts with mankind in a way that isn't answerable to persons; this goes, in order of scale, from gravity and kinetics, to cancer, to publicly-owned corporations and states. these all keep abusing persons (by which i here mean [moral patient](https://en.wikipedia.org/wiki/Moral_agency#Distinction_between_moral_agency_and_moral_patienthood)) in many ways, and just generally keep us from being in control of our lives.

	the example of corporations is particularly insidious: though they would be (under UBI) aligned to benefit the values of persons, they still outcoordinate those persons and thus in many ways outsmart them through the abuse of discoordination and cognitive biases; and not only that, but they are, in the petri dish of capitalism, bred so as to maximize their ability to do this. that said, at least fully top-down autocratic corporations have a person agent at the top, who is able to enforce the values of persons; publicly-owned corporations are even worse in that even their top-level direction is uncoordinated enough that valuing nice things is guaranteedly out of the equation (this could perhaps be addressed with better and maybe more society-distributed shareholder voting, but those shareholders probably get outcoordinated).

	(the argument above, by the way, is my largest criticism of non-[distributist](https://en.wikipedia.org/wiki/Distributism) capitalism)

	in effect, this principle turns the world we inhabit from a world of cold natural and emergent laws inside which reside some minds located in brains (materialism), into a world of ad-hoc minds determining everything else ([panpsychism](https://en.wikipedia.org/wiki/Panpsychism) ?).

	the easiest way to implement this principle is probably to move everyone to a virtual world (which saves resources too, which helps the population control cap be way higher)

in my current opinion, those two principles **must be enforced** for the basis of a utopia to be form. the rest can be done through the voluntary action of persons (hopefully), but these two principles are what Elua/the singularity is to **enforce** for the continued free and valueful life of persons to be guaranteed.

Voluntaryism alone is not enough, and this is largely missed by what i'm tempted to call right-wing utopians; not just abusive structures, but systematically self-reinforcing abusive structures, can and will still happen even under a complete voluntary society. [Meditations on Moloch](https://web.archive.org/web/20140730043944/http://slatestarcodex.com/2014/07/30/meditations-on-moloch/) addresses this largely with coordination, but coordination only *hopefully wins battles*; the addition of DNPF permanently wins the war.

DNPF alone is not enough either, and this is what is largely missed by what i'm tempted to call left-wing utopians; in a virtual world of minds where resources are fairly allocated between persons, there can still be abuse, plagues, [malthusian traps](https://en.wikipedia.org/wiki/Malthusian_trap), and so on; and ultimately abusive structures, just of a different kind. the common left-wing answer of organizing people (and the scarier "changing culture to make people systematically organize against those" which, if voluntary, is largely wishful thinking, and if not, insanely violates self-determination and the values of persons) only wins battles; the addition of Voluntaryism permanently wins the war.


were-all-doomed

2021-06-30

## we're all doomed

[a major tech company is now explicitly invested in getting AI to write code](https://copilot.github.com/).

this is a major warning sign; a first step on the explicit path to [superintelligence](https://en.wikipedia.org/wiki/Superintelligence) [explosion](https://en.wikipedia.org/wiki/Technological_singularity#Intelligence_explosion), an event [already considered relatively likely](https://intelligence.org/faq/#imminent) and, [in the absence of sufficient AI alignment progress](https://intelligence.org/2018/10/03/rocket-alignment/), is overwhelmingly likely to [permanently end all life at least in the observable universe](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer).

the time scale probably lies somewhere between a few years and a few decades, but in any case it's becoming to seem increasingly unlikely that [the only organization trying to actually figure out AI alignment](https://intelligence.org/) is gonna accomplish that in time.

if you can, go and [help them out](https://intelligence.org/get-involved/), or at least [donate everything you can to them](https://intelligence.org/donate/).

if you're currently working in AI development in any way, *please stop*. whether anything on earth survives this century is gonna be a matter of whether AI alignment is figured out by the time we get enough AI development; by helping the latter, you're making it even more likely that it happens before the former.

on a gloomier note, if you have all the philosophical beliefs required to think it can work, you may want to start preparing to [abandon this timeline](https://carado.moe/quantum-suicide.html) if singularity starts happening and looks like it's not gonna go well.

edit: see also: [are we in an AI overhang?](https://www.lesswrong.com/posts/N6vZEnCn6A95Xn39p/are-we-in-an-ai-overhang)


life-refocus

2022-05-13

## life refocus

because of the [recent](https://www.metaculus.com/questions/3479/date-weakly-general-ai-system-is-devised/) [events](https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy), which i've been dreading [for a while](https://carado.moe/were-all-doomed.html), i'm taking AI risk a lot more seriously, and have started significantly refocusing my life.

there is a post called [*musk's non-missing mood*](https://lukemuehlhauser.com/musks-non-missing-mood/) that resonates quite well with me. it is indeed kind of disconcerting how people who seem rationally aware of AI risk, don't seem to *grok* it as an *actual thing*. despite how real it is, it's hard to think of it not as fantasy fiction.

i totally understand why. i've been there too. but eventually i managed to progressively update.

i'm still not quite there yet, but i'm starting to actually grasp what is at stake.

["detaching the grim-o-meter"](https://mindingourway.com/detach-the-grim-o-meter/) remains a reasonable thing to do; you don't want to become so depressed that you kill yourself instead of saving the world. but you also don't want to remain so deluded that you don't quite weigh the importance of saving the world enough either.

i'll learn japanese after the singularity. i'll make [my game](https://carado.moe/game.html) and [my alternative web](https://carado.moe/saving-the-web.html) and my conlang and [my software stack](https://carado.moe/psi.html) and many other things, after the singularity. it is painful. but it is what's right; it's closer to [the best i can do](https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy).

and i know that, if at some point i give up, then it won't look like pretending that everything is fine and compartmentalizing our imminent death as some fantasy scenario. it'll be a *proper* giving up, like going to spend the remaining years of my life with my loved ones. even my giving up scenario is one that takes things seriously, as it should. that's what being an adult capable of taking things seriously is like.

how you handle your mental state is up to you. there is a collection of AI-risk-related mental health posts [here](https://www.lesswrong.com/posts/pLLeGA7aGaJpgCkof/mental-health-and-the-alignment-problem-a-compilation-of). do what it takes for you to do the work that needs to be done. that's not becoming a doomer; your brain is straight-up not designed to deal with cosmic doom. but that's not remaining blindly naive either. the world needs you; it won't be saved by pretending things are fine.

and it *certainly* won't be saved by pretending things are fine and *working on AI capability*. that's *just bad*. *please* don't.

please take AI risk seriously.


upload-for-alignment

2022-01-11

## uploading people for alignment purposes

as per [my utopian vision](https://carado.moe/∀V.html), i've thought that an aligned AI would want to figure out how to upload us.

but, thinking about it more, it could be the other way around: if we can upload people in a deterministic simulation, this can buy us a lot of time to figure out alignment, as per [this post](https://carado.moe/noninterf-superint.html).

notably, the simulation could for example contain a single uploaded person (say, eliezer yudkowsky, or a bunch of copies of yudkowsky), which would save us from an arms-race type coordination problem; and while, on the outside, the superintelligence is killing everyone instantly to tile the universe with more compute to run this simulation, whoever's inside of it has plenty of time to figure things out (and hopefully [resurrect everyone once that's done](https://carado.moe/what-happens-when-you-die.html)).

this seems like a long shot, but [have you looked around?](https://www.lesswrong.com/s/n945eovrA3oDueqtq) this could be the [miracle](https://www.lesswrong.com/s/n945eovrA3oDueqtq/p/7im8at9PmhbT4JHsW) we need.

of course this could also turn into a [hell](https://carado.moe/botched-alignment-and-awareness.html) where infinite yudkowsky's are suffering forever everywhere. hopefully we can make another button which actually stops the simulation and tiles the universe with only benign paperclips, and maybe even make that button auto-activate if the yudkowsky is detected to be suffering or incoherent.

remember: [as long as the simulation is deterministic, superint can't force the uploaded yudkowsky to not shut it down](https://carado.moe/noninterf-superint.html), or force or even coerce him to do anything for that matter; it can only make the yudkowsky simulation run slower, which basically eventually achieves the same effect as either completing it or shutting it down.


hackable-multiverse

2022-02-03

## hackable multiverse

in [a previous post](https://carado.moe/brittle-physics.html) i talk about how hackable physics might allow a superintelligence to take over very quickly (perhaps faster than celerity).

in *[psi rewriting](https://carado.moe/psi-rewriting.html)* i propose that multiversehood can be more cleanly described as a particularly implemented feature of the cosmos, rather than an intrinsic thing.

but, if the cohabitation of multiple timelines is indeed an implemented feature rather than a primitive one, then there is a possibility that it is hackable, and that a superintelligence could hack across timelines.

now, it is to be noted that even if hackability exists, it might still be limited: perhaps there something like a light cone at play, or perhaps a given timeline can only access a finite number of other timelines.

it is to be remembered that timelines are not slots, they're not variables that hold values; timelines are *the values themselves*. still, hackability could mean some branches of the causality graph stop getting computed, for example.

either way, under these conditions, even quantum immortality might not save us from an X-risk superintelligence, and [given](https://deepmind.com/blog/article/Competitive-programming-with-AlphaCode) [recent](https://openai.com/blog/formal-math/) [developments](https://blog.eleuther.ai/announcing-20b/), we should panic a lot.


brittle-physics

2022-01-05

## brittle physics and the nature of X-risks

suppose physics is hackable, and a hard to accomplish hack that requires intelligence (like a fancier version of [rowhammer](https://en.wikipedia.org/wiki/Rowhammer)) can break the fabric of spacetime — maybe in ways that said intelligence can take advantage of, such as embedding its computation into something that survives said breakage, in a way that could help such a superintelligence accomplish its goal.

we could expect that [boxing an AI](https://en.wikipedia.org/wiki/AI_box) could be really hard: even without access to the outside, it might be able to guesses physics and hack it, from the comfort of its box.

as usual in such [X-risk scenarios](https://carado.moe/timeline-codes.html), i believe we just [keep living only in timelines in which, by chance, we don't die](https://carado.moe/quantum-suicide.html).

these sort of hacks are not ruled out by [wolfram physics](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/). indeed, they are plausible, and can spread at some speed faster than celerity — because they can run in the substrate *underlying* spacetime — such that nobody would ever be able to observe such hacks: the hack reaches and destroys you before the result of the breakage can reach your sensory organs, let alone your brain.

so, maybe "dumb-goal" superintelligences such as [paperclip maximizers](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer) are popping up all over the place all the time and constantly ruining the immense majority of not-yet-hacked timelines, and we keep living in the increasingly few timelines in which they haven't done that yet.

now, let's stop for a minute, and consider: what if such a hack *isn't* hard ? what if it *doesn't* need an intelligent agent ?

what if, every planck time, every particle has a 99% chance of breaking physics ?

well, we would observe exactly the same thing: those hacked universes either become computationally simple or [boot up more universes](https://carado.moe/above-paperclips-2.html); either way, we don't survive in them, so we don't observe those hacks.

in this way, it is [S-lines and U-lines](https://carado.moe/timeline-codes.html) that are very special: outcomes in which we *survive*, thanks to a superintelligence with a "rich" goal. the rest is just timelines constantly dying, whether it be due to X-risk superintelligences, or just plain old physics happening to cause this.

in fact, let's say that the universe is [a nondeterministic graph rewriting system](https://en.wikipedia.org/wiki/Graph_rewriting) with a rule that sometimes allows everything be reduced to a single, inactive vertex. would this count as "sometimes everything is destroyed" ? or would this make more sense to be modeled as a weird quirk of physics where the graph of possible timelines includes the production of passive vertices all the time, which can be safely ignored ?

what if instead of a nondeterministic system, we have a deterministic one [which just happens to expand all timelines](https://carado.moe/psi-rewriting.html). in such a system, "different timelines" is no longer a primitive construct: it is merely an observation about the fact that such a system tends to, when ran, create from a given piece of data, several newer ones. let's say that in such a system there is a rule where from every piece of data we'd consider a timeline, numerous inert vertices are also created.

would we say "aha, look! every time a computation step happens, many inert vertices are created around it, and i choose to interpret this as the creation of many timelines (one per inert vertex) in which everyone in that universe dies, and others (new complex pieces of data) in which everything keeps existing",

or would we, in my opinion more reasonably, say "well, it looks like as a weird quirk of how this system runs, many inert vertices are popping up; but they're simple enough that we can just ignore them and only consider richer new pieces of data as *timelines* proper."

i believe, if we are to worry about what states this universe ends up in, we ought to use a measure of what counts as a "next state of this universe" that measures something about the richness of its content: maybe the amount of information, maybe the amount of computation going on, or maybe the number of moral patients. and, depending on what measure we use, "losing" timelines to paperclip maximizers (which turn the universe into something possibly simple) is no more of a big deal than "losing" timelines to a rewriting rule that sometimes creates inert vertices, and neither of which should really count as proper timelines.

otherwise we end up needlessly caring about degenerate states because of what we believe to be, but really isn't, an objective measure of what a timeline is.

*timelines* might be in the [map](https://en.wikipedia.org/wiki/Map–territory_relation), while what is in the territory is just *what we end up observing* and thus, computed states that contain us.

finally, what about universe states where *all* outcomes are an inert vertex or an otherwise simple universe (such as as infinite list of identical paperclips) ? while those might happen, and i'd say *would* count as X-risks, you don't need to consider simple states as timelines to make that observation: maybe some timelines end up in a state where *no* new states can be created (such as a locally truly terminated piece of computation), and others end up in a state where *only simple* new states are created. those ought to be considered equivalent enough, and are what a true X-risk looks like.


universal-complete

2021-07-16

## universal complete

under [a turing-complete model of computation](https://en.wikipedia.org/wiki/Model_of_computation), there are some initial-states or initial-states-and-rulesets which eventually contain an algorithm that iterates over all possible algorithms and runs them.

in single-threaded models, it can do this by having an increasingly long list of algorithms that it runs by one step each; it's not an issue if each algorithm runs increasingly slowly, as long as it keep running.

i choose to call such initial-states[-and-rulesets] *Universal Complete*.

they contain all turing computation based universes (and thus each other, if indirectly); so, for example, if [Rule 30 with one alive cell](https://en.wikipedia.org/wiki/Rule_30) is Universal Complete, then it contains all computable universes (including ours).

this could be interesting because proving that property about some frameworks means that programming a particular algorithm starting from that initial-state[-and-ruleset] is just a matter of *locating* it.

it could also be interesting because it might turn out that many things that *look* sufficiently chaotic (such as Rule 30 with one alive cell) are effectively universal complete, and so [Wolfram's quest](https://www.youtube.com/watch?v=0bMYtEKjHs0) for the rule that describes our universe [in his hypergraph-rewrite system](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/) might be reductible to "whichever simplest initial-state-and-ruleset starts all algorithms"; though his idea of *running every rule at every step* might kind of functionally do that.

### appendix: a simple universal-complete program

here is a simple algorithm implemeting this, iterating over the countable set of turing machines.

x ← simplest turing machine
l ← empty list
loop:
	for machine in l:
		update machine by one step of computation

	append x to l
	x ← next simplest turing machine after x


values-tdd

2022-03-21

## values system as test-driven development

i realized something while reading [hands and cities on infinite ethics](https://handsandcities.com/2022/01/30/on-infinite-ethics/): the work of determining the shape of [our values system](https://carado.moe/not-hold-on-to-values.html) is akin to [test-driven development](https://en.wikipedia.org/wiki/Test-driven_development).

we are designing a procedure (possibly looking for [the simplest one](https://en.wikipedia.org/wiki/Kolmogorov_complexity)) by throwing it at a collection of decision tests, and looking for which one matches our intuitions.

i wonder if a value-learning approach to AI alignment could look like trying to get superintelligence to find such a procedure; perhaps we feed it a collection of tests and it looks for the simplest procedure that matches those, and hopefully that extrapolates well to situations we didn't think of.

perhaps, even pre-superintelligence we can formalize values research as tests and try to come up with or generate a simple procedure which passes them while also being selected for simplicity.

why simplicity? doesn't occam's razor only apply to descriptive research, not prescriptive? that is true, but "what is the procedure that formalizes my values system" is indeed a prescriptive matter, in a way: we're trying to model something to the best factual accuracy we can.


what-happens-when-you-die

2021-08-25

## what happens when you die?

contrary to popular (secular) belief, i don't believe nothing happens.

considering that [you are your information system](https://carado.moe/you-are-your-information-system.html) and [nothing else](https://carado.moe/persistent-data-structures-consciousness.html), any future occurence of your information system *is you*. so, the meaning of the question "what happens when you die?" is really "what are some next things your information system will percieve after facing what should be fatal events in its original body?"

this is very likely [not nothing](https://carado.moe/quantum-suicide.html). somewhere, in some timeline,  your information system is probably being redundanced.

first, your body can miraculously avoid death. this would be a weird kind of immortality, where there is almost always a timeline where you somehow avoid death. it is, however, pretty unlikely to persist.

second, your mind could arise somewhere by accident. this could be as simple as random fluctuations in space producing something that runs your mind's information system by pure chance. this is *extremely* unlikely.

in fact, the most likely scenario is that someone in the far future reproduces your mind on purpose. for example, this could be a society in a [u-line](https://carado.moe/timeline-codes.html) being able to, and deciding, to run an accurate enough simulation of the entire earth up to some point, and downloading people from this simulation into their world, to allow them to avoid death. as it'd probly take a bunch of effort, and sounds like a pretty nice thing to do, i expect that to happen mostly in u-lines; however, there could be some [s-lines](https://carado.moe/timeline-codes.html) where this happens too. and while getting resurrected seems more likely in u-lines than s-lines, s-lines seem more likely than u-lines, and i don't know if the probabilities cancel out.

so, what happens when you die? you wake up either in heaven or hell, depending not on your personal actions in particular but in how likely it is we figure out AI alignment (a probability which you do have, if small, an impact on).


estimating-populated-intelligence-explosions

2021-07-10

(edit 2021-07-18: this post is probly not very good, as there's some anthropic principle research out there and i haven't read any and just gone off thinking about it on my own.)

## estimating the amount of populated intelligence explosion timelines

the [imminent](https://carado.moe/were-all-doomed.html) [intelligence explosion](https://en.wikipedia.org/wiki/Technological_singularity#Intelligence_explosion) is likely to [go wrong](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer).

how likely?

if you imagine that you live pretty much at the cusp of such an event, you should expect as per the [anthropic principle](https://en.wikipedia.org/wiki/Anthropic_principle) that there are about as many observer-instants before you, as there are after you. (an observer-instant being an instant at which you have a chance of making observations about that fact; see [this](https://www.greaterwrong.com/posts/uSMa6Fj5nMgntpxfo/are-coincidences-clues-about-missed-disasters-it-depends-on) and notably Nick Bostrom's Self-Sampling Assumption)

i've previously calculated that the future from now until heat death has room for roughly 10^200 human lifespans (of 80 years) (an estimation based on the number of particles in the observable universe,  the amount of time until heat death, and the computational cost of running a human brain).

the past, on the other hand, holds about 10^11 human lifespans (most of them not full 80-year lifespans, but such details will get amortized by using orders of magnitude).

if intelligence explosion is, as i believe, likely to result either in [total death](https://carado.moe/were-all-doomed.html) or in well-populated futures (whether good or [bad](https://en.wikipedia.org/wiki/Suffering_risks)), then the fact that i'm observing being right next to the event (in time) rather than observing being one of the (in well-populated timelines) countless observers to exist *after* the event, must be compensated by such well-populated timelines being particularly rare within the set of future possible timelines.

how rare? about 1 in (10^200 / 10^11), which is 1 in 10^189.

factors which may make this calculation wrong:

* my 10^200 estimate might be wrong (for example: if each person comes to eat a *lot* of computation resources, then the number of future observers is drastically reduced).
* the 10^11 estimate for the past might be wrong: what if there have beings in earth's past smart enough to make this observation? it may seem unlikely, but if i am to encompass the immense amount of forms future observerc might take, i should account for a wide variety of forms of past observers too.
* because entropy increases, there are (possibly a lot) more future universe states than past universe states. accounting these "timeline splits" for the number of future observers even more massively decreases the expected ratio of well-populated timeline-states, though i'm not sure by how much.


systems-and-diversity

2021-07-21

## systems and diversity

as i've said in [a previous post](https://carado.moe/lets-not-generalize-politics.html): i really like culture; and, to that end, i like diversity (by which i mean people being more weird and different from one another).

there are many systems that exist today that affect diversity. most of them punish it; not as a coincidence, but because diversity is [a fragile value](https://www.readthesequences.com/Value-Is-Fragile): if you optimize for something else, it will tend to get optimized out.

if you optimize for economic efficiency, diversity gets optimized out because the most easily served economy is one in which demand is relatively uniform.

in general, if you optimize for people having their values satisfied, diversity gets optimized out because the most easily satisfied set of values is relatively uniform and easy to satisfy values; if you tell a superintelligence to "make a world where everyone has their values satisfied", the simplest way to achieve that (other than killing everyone) is to make sure everyone has very simple values like doing nothing all day or dying as soon as possible.

the scary thing about such an optimization is that it "works": at no point does an economy headed towards uniformity need to collapse; on the contrary, the more it has optimized out diversity, the more efficient and stable it'll be! so, we need to *[near-intrinsically](https://en.wikipedia.org/wiki/Instrumental_and_intrinsic_value)* care about preserving diversity, even when all else seems fine. this makes diversity preservation probably my largest concern with capitalism; at least, a system that wouldn't care about efficiency, wouldn't necessarily be aligned against diversity (though it might be aligned against it for other reasons).

social pressures such as [generalizations and expectations](https://carado.moe/lets-not-generalize-politics.html) punish diversity by rewarding conformity.

democracy and general consensus enforcment systems punish diversity by generally letting majority lifestyles be better supported by society than minority lifestyles.

i do know of one force of human nature which encourages diversity: [fetishism](https://en.wikipedia.org/wiki/Sexual_fetishism#Definitions). fetishism tends to make people prefer things specifically because they go against the norm. as such, i propose that if we value rich culture, we should want to cultivate fetishism.

the takeaway is: in any long-term societal plan, we need to care not just about values being satisfied, but about what values people have to begin with. a clear example in modern capitalism is advertising: it's okay that companies are aligned to satisfy values, but [with advertising they get to affect what values people have to begin with](https://carado.moe/unfair-feedback-loops.html).

(one could argue we could encourage people to [crystallize](https://carado.moe/value-crystallization.html) and [conserve](https://en.wikipedia.org/wiki/Instrumental_convergence#Goal-content_integrity) their values, as well as forbid the creation of new persons; but [i'd rather that not be required](https://carado.moe/rationalist-by-necessity.html))


unoptimal-superint-doesnt-lose

2021-12-09

## unoptimal superintelligence doesn't lose

i [previously](https://carado.moe/unoptimal-superint-loses.html) wrote a post about how a superintelligence with an unoptimal decision system likely loses to alien superintelligences that are more optimal, at the scale of cosmic wars between those superints.

i don't think this is necessarily true: maybe physics *does* look like a funny graph à la wolfram, and then maybe we can carve out pieces of space that still grow but are causally isolated from the rest of the universe; and then, whether a given causally isolated bubble ever has to encounter an alien superint is purely up to whether it decides to generate alien space that leads to those, which is prevented easily enough.


the-peerless

2022-04-12 ★

## The Peerless

In this post, I propose a plan for addressing superintelligence-based risks.

Before I say anything, I will mention a crucial point that a *bunch* of people have ignored despite it being addressed at the bottom of this post: the idea I describe here is very unlikely to work. I'm proposing it because of other plans because I feel like other plans are *extremely* unlikely to work (see also [this post](https://carado.moe/ai-risk-plans.html)). Yes, we probably can't do this in time. That doesn't make it not our best shot. Rationalists select the best plan, not ["the first plan, and then after that a new plan only if it seems good enough"](https://www.readthesequences.com/If-Many-Worlds-Had-Come-First).

(spoilers for the premise of [*orthogonal* by Greg Egan](https://en.wikipedia.org/wiki/Orthogonal_%28series%29)) In Orthogonal, a civilization facing annihilation comes up with a last-minute plan: to create a ship, accelerate it until its time arrow is orthogonal to the time arrow of its home world (which is possible thanks to the alternate physics of their world), and thus give its crew as much time as it needs to figure out how to save their homeworld before reversing course and coming back. This plan is inspired by that, and i'm naming this post after their ship, the *Peerless*.

The short version is: we design a simulation for a bunch of people (probably rationalists) to live in and figure out alignment with as much time as they need, and create a superintelligence whose sole goal is to run that simulation and implement a new goal it will eventually decide on. I've written about this idea previously, but [that post](https://carado.moe/upload-for-alignment.html) is not required reading; this is a more fleshed-out view.

I will be describing the plan in three steps.

### 1. Create virtual people

We need virtual persons inside this world. They will be the ones who figure out alignment. A few possibilities come to my mind; there may be more.

* Brain scans, or full person scans. This is the most obvious solution. I'm not too familiar with the state of that field, but surely there's some work in that direction we can take advantage of; otherwise, we can just throw money at our own initiatives. This option does have the downside that it's quite likely brains aren't sufficient to keep someone functional — we may need to scan or re-implement a bunch more.
* Resimulate earth and pluck out persons. If there's a clever way to locate ourselves in the [universal distribution](https://www.lesswrong.com/posts/XiWKmFkpGbDTcsSu4/on-the-universal-distribution) (or a [computable variant of it](http://www.scholarpedia.org/article/Universal_search)), then we can just make a program that reruns that earth up to say, now, and then locates some or all human brains, and "download" them out of that simulation of earth and into our own simulated environment. For more details on this possibility, see [*finding earth in the universal program*](https://carado.moe/finding-earth-ud.html).
* Scan the earth and pluck our persons. This one seems harder than resimulating earth, but may be doable. It's certainly an idea worth throwing a few people at, to see if there's a clever way to make it work.

The main risk that's been brought to my attention regarding this part is the following: what if the virtual persons end up unaligned from their previous selves? The brain scan scenario seems like the most likely to have that risk, but even then i'm not *too* worried about it; intuitively, it seems unlikely enough to me that all the uploaded persons would come out misaligned in a similar direction, and in a similar direction that would lead them to decide on a [botched alignment](https://carado.moe/botched-alignment-and-awareness.html) for the superintelligence.

An obvious question here is: who gets to be on board the simulation? The values of the people who get uploaded might significantly affect what the superintelligence is aligned to (not all humans necessarily have the same values, [maybe even after thinking about it really hard for a long time](https://handsandcities.com/2021/06/21/on-the-limits-of-idealized-values/)). I don't have any answers other than the obvious "me please!" and "my tribe please!" that occur to me.

Note that i'm *not* proposing augmenting the uploaded minds — at least not for the first simulation iteration (see below). That *does* seem like an exceedingly risky prospect, alignment-wise, and one we don't need to commit to right away.

### 2. Design a virtual environment

Those persons will live in a virtual environment, within which they'll hopefully figure out alignment. However, the environment needs to be a deterministic computation, such that the "outer" superintelligence (the one running the virtual environment) has no ability to affect its outcome; its goal will only be to "implement whatever this computation decides". If the superintelligence wants to implement the actual result of the actual computation, and that computation is fully deterministic, (and if we don't simulate anything complex enough for that superintelligence to "leak back in"), then it has no room to meddle with what we do in it! It's stuck running us until we decide on something.

Some things we need to figure out include:

* How do we incorporate our virtual minds? I think we should go for something plugged in "ad-hoc" rather than embedded into the physics of that world, to preserve the integrity of those minds, which may live for very long times. In addition, in case virtual minds go crazy after living 200 years or something, we may want to allow them to reset themselves and/or die. A reset is not necessarily a big deal: hopefully previous-me can transmit enough information to future-me to continue the work. Maybe there are two me's at any given time, a teacher and an apprentice. Regular resets of individual persons also hopefully help maintain their values over long stretches of time. Many schemes are possible.
* What is this world like? We could make do with just something as basic as minecraft, but it would be better if the virtual persons don't have to go crazy from being stuck in a minecraft steve's body with no senses except sight and sound.
* How do we prevent "sub-singularities"? Given that this world is deterministic, there is nothing the outer superintelligence can do to prevent internal superintelligences from popping up and breaking everything it can. Potential solutions include things like "there are no computers" or "all computers inside this world are very slow and limited in capability".
* What about memetic safety? What about virtual violence? What if someone duplicates themself a billion times? And so on. There are a collection of design challenges, but designing [a peaceful world](https://carado.moe/∀V.html) with [sensible virtual physics](https://carado.moe/game.html) doesn't seem out of reach. They seem like tractable engineering challenges.
* What is the final voting procedure? Remember that the goal of the simulation is to give the people inside it time to figure out alignment, but they should probably agree on something eventually: either a final decision on alignment, or a "next iteration": a new simulation to be ran, which they think has better/safer/still-safe conditions within which to research alignment. In fact, there may be arbitrarily many such "simulation iterations". Anyways, the simulation will have a big red button inside of it which says "okay, we're done", and takes as input a new goal (and possibly decision theory?) that the outer superintelligence will have as its ultimate goal. But what should it take to press the button? Everyone to agree? A majority? What if we end up unable to come to an agreement? Again, there is work to be done on this, but it seems figure-out-able.

The people inside this simulation will have somewhere between *plenty* and [*infinite*](https://carado.moe/ai-alignment-wolfram-physics.html) time and compute to figure out alignment. If they do have infinite compute, and if the cosmos isn't full of [consequentialists competing for earlyness in the universal distribution](https://carado.moe/udassa-time-steps.html) (or other things that might make wasting compute bad), then we can even run exponential-or-longer computations in, from our perspective, instant time; we just need to be sure we don't run anything malign and unbounded — although the risks from running malign stuff might be mitigated by the computations being fully and provably sandboxable, and we can shut them down whenever we want as long as they don't get to output enough to convince us not to. After all, maybe there are some bits of information that are the result of very large malign-dominated computations, that can nevertheless still be of use to us.

I mentioned before that maybe only slow computers are available; running a "very large" computation might require a majority vote or something like it. Or we can just boot without any computers at all and spend the first few millenia designing slow computers that are actually safe, and then work from there — when we have all the time we want, and maybe-infinite *potential* compute, a lot of options open up.

One downside is that we will be "flying blind". The outer superintelligence will gleefully turn the cosmos into computronium to ensure it can run us, and *will* be genociding everything back meat-side, in our reachable universe — or beyond, if for example physics is hackable, as wolfram suggests might be possible. Superintelligence might even do that *first*, and *then* boot our simulation. Hopefully, if we want to, we can resimulate-and-recover aliens we've genocided after we've solved alignment, just like hopefully we can resimulate-and-recover the rest of earth; but from inside the simulation we won't be able to get much information at least in the first iteration. We *can*, however, end our iteration by agreeing on a new iteration that has some carefully-designed access to outside information, if we think we can safely do that; but nothing guarantees us that there will still be something salvageable outside.

Another way to model "successive simulation iterations, each deterministic, but each having the ability to make the next one not deterministic with a large enough vote" is as a single simulation that isn't quite deterministic, but made of large deterministic chunks separated by small controlled I/O accesses; think of it as a haskell computation that lazily evaluates everything right up until it waits for an input, and then as soon as it has that it can continue computing more.

Still, the current outlook is that we genocide everything *including ourselves*. Even if nothing else is recoverable, "a tiny human population survives and eventually repopulates" still seems like a better plan than the current expected outcome of "everything dies forever."

### 3. Make a superintelligence to run it

Now, this is the "easy part": just make a superintelligence that destroys everything to implement its one simple goal; except instead of paperclips, the simple goal is "implement whatever goal is the result of this very big turing machine".

We can either build and start that superintelligence as soon as we can, or [keep it ready](https://carado.moe/when-in-doubt-kill-everyone.html) while we stay on our regular world. I'd probably advocate for the former just to be safe, but it can depend on your beliefs about quantum immortality, S-risks, and such. In any case, having *something that might work ready to fire* is certainly better than the current *we just die*.

Of course, it is crucial that we make the superintelligence *after* we have designed and implemented the virtual environment, complete with its virtual persons (or its deterministic procedure to obtain them); we don't want it to be able to influence what goal we give it, so we likely need to have the goal ready and "plugged in" from the start.

Some risks are:

* It doesn't run the simulation accurately. I'd think it's surely not too hard to make a superintelligence have "run this discrete, deterministic thing and then adopt its output as goal" as its goal, but perhaps there are difficulties around this. I'm optimistic that we can figure that out.
* It doesn't run the simulation (or adopt its result as goal) at all. Perhaps the requirement that the simulation be ran perfectly will make superintelligence too paranoid about being sure it has run the simulation correctly, and it will spend its entire existence getting more computronium to increase the likelyhood that the outcome it has computed is correct, but it's never certain enough to actually adopt the new goal as its own. There may be probabilistic workarounds to this; we'll need to look into it more.
* It fails in the various ways AI usually value drift from us — hacking its reward function, etc. While this concern may indeed remain, the fact that i'm proposing what seems to me like a easy to formalize goal is already a big improvement compared to the current state of affairs.

### Conclusion

This is a plan with a bunch of things that need work, but it doesn't seem *absurdly* hard to me; if anything, step 1 seems like the hardest, and I don't even know that we've *tried* throwing billions of dollars at it.

I share yudkowsky's current [gloomy outlook](https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy) on AI. The current route of "hey, maybe we should study things vaguely related to harnessing what neural nets do, and hope to be able to grab a miracle should it come up" seems like a pretty bad plan. I think, in comparison, the plan I outline here has better chances.

It is to be remembered that my vision is competing not against the likelyhood of superintelligence emergence, but against the likelyhood that alignment works. If pursuing mostly alignment gives us a 1e-10 chance of survival, and pursuing mostly my plan gives us a 1e-8 chance of survival, then it doesn't matter that *yes, superintelligence is still overwhelmingly likely to kill us*; we should still favor my plan. See also: [this post](https://carado.moe/ai-risk-plans.html) comparing plans.

I have cross-posted this [on lesswrong](https://www.lesswrong.com/posts/PATFQm6hPN3Wycq4W/the-peerless); feel free to discuss the idea with me there.


quantum-suicide

2021-04-28

**DISCLAIMER: the idea described here stands or tenuous philosophysical ground and should generally _not_ be considered worth the risk attempting because it may be wrong; in addition, this plan should _not_ be utilized to retroactively justify depression-based suicide — retroactive justification is erroneous; if you are feeling suicidal, contact [your local suicide crisis line](https://en.wikipedia.org/wiki/List_of_suicide_crisis_lines).**

## Plausible Quantum Suicide

in this post, i make a collection of arguments and follow them to what seems to me like what should be their conclusion. i don't have strong confidence in every argument, but i'd consider using this plan worth it to avoid sufficiently bad scenarios, such as a singularity gone wrong (which it probably will).

### 1. The No Obtainable Evidence Argument For Materialism

by materialism i mean here something maybe closer to [physicalism](https://en.wikipedia.org/wiki/Physicalism), but maybe even a stronger version of it:

there is no special soul that people have, [you are your information system](https://carado.moe/you-are-your-information-system.html).

i make an other strong claim: time isn't particularly "moving" in any metaphysical sense, there is no "special present time". the universe can be seen as a graph of probabilistically connected states, and a random walk through those states matches the notion of entropy pretty well (which can be seen as defining the direction of time, because we happen to have memories of universe states with generally lower entropy), but that's a local notion.

the *illusion* that the present is particularly present, that we have a particular soul, or even that morality/ethics is in some sense objective, stems from the fact that we *inhabit our brain's model*: we don't get to see our brain from the outside as modeling its environment, we live right inside it, and we don't spawn with a clear distinction between normative ideas (morality/ethics) and descriptive ideas (statements of fact about the world).

but those illusions *must* be wrong, and here is the argument: as far as we can tell, there is no possible way for a brain to obtain evidence that his present time is particularly real; therefore, it must be erroneous for any brain to generate rationally the idea that its present is particularly real. same goes for having what i call a "read-only soul" that some people believe in (a special observer thing that observes a person's mind state from outside the material universe, but cannot causate anything upon it). see also [these](https://www.readthesequences.com/Zombies-Zombies) [three](https://www.readthesequences.com/Zombie-Responses) [posts](https://www.readthesequences.com/The-Generalized-Anti-Zombie-Principle).

### 2. Limiting Real Universes

my post ["Limiting Real Universes"](https://carado.moe/limiting-real-universes.html) isn't that good, so i'll try to explain it more clearly here:

if for some reason all possible states of our universe were equally real, then you should expect to observe widely anomalous phenomena around you, because most randomly picked states our universe can be in don't have to be coherent.

but the fact that we seem to observe a continuously very coherent universe tells us that there must be some sense in which coherent universe states, that stem from a continuous history following an increasing entropy timeline, must be particularly more real.

it's not that your magical soul has been blessed with inhabiting universe states: as explained in argument 1, you shouldn't have any reason to think you have such a thing.

it's not that [you can only exist to observe universe states that are coherent, because you wouldn't exist in incoherent ones](https://en.wikipedia.org/wiki/Anthropic_argument): there are still way more possible universe states where everything is incoherent except your brain, than possible universe states where everything is coherent including your brain. for any amount of state of stuff you require to say you can exist to observe the world, the rest of the universe should still generally seem incoherent if all possible universe states are equally real.

it's not that you have been following your own special arrow of time: even though i debunk that you should even think this makes sense in argument 1, another reason is that, even if some of your brain states have a past-arrow-of-time and not others, there's no reason for you to think you're one of the former. if all possible universe states were equally real, you'd likely be a version of your brain that *thinks* it has a past history but doesn't, than one that does.

### 3. Many-Worlds Is True

[Eliezer Yudkosky makes a good argument](https://www.readthesequences.com/If-Many-Worlds-Had-Come-First) that we should currently believe in the many-worlds [interpretation of quantum mechanics](https://en.wikipedia.org/wiki/Interpretations_of_quantum_mechanics); but even if that turned out wrong, [Max Tegmark makes another good argument](https://space.mit.edu/home/tegmark/crazy.html) that even just with a single history universe, all possible initial states of the universe are represented each in an infinite amount of instances by just being variations of initial conditions and random quantum determinations at different places of the infinite universe.

what matters here is that basically one should expect every reasonably possible outcome to be a real instance of universe that exists somewhere. because of argument 2, some possibilities are particularly real, and because of argument 3 (this one), that set or fuzzy set of coherent possibilities should be widely instanced: at each possible fork (they're not *really* forks, but that's a good enough analogy from our point of view), every plausible outcome is realized as a real or fairly real universe.

### 4. Quantum Suicide Works

if the previous 3 arguments stand, then a more general version [quantum suicide](https://en.wikipedia.org/wiki/Quantum_suicide_and_immortality) should be achievable: by dying instantly in one timeline, there is no version of you in that timeline able to experience it, and the only remaining future you's able to experience anything are the you's in other timelines.

because of argument 1, we know that saving a backup of your brain, and then later dying and restoring another copy of yourself from backup, is equivalent to just losing memories of the time after the backup: it's unfortunate that that time and those memories were "lost", but it's not a big deal, you can just keep going.

given that, even non-instantaneous, after-the-event suicide works: if you commit yourself to committing suicide in all timelines where an event goes wrong, then the only future you's able to experience any time after that suicide will be the ones in the timelines in which that event went well (or at least in which you think it did); you lose a bit of time and memories from those timelines in which you didn't kill yourself *literally instantly* after the thing went wrong, but it's just equivalent to a restoration from backup: the backups are automatically saved by the universe as forks of that previous universe state before the event's outcome was determined.

### ramifications

if this is true, then every person is strongly empowered: by committing themselves to committing suicide in every timeline in which even the slightest thing goes wrong, they are able to restrict the set of instances of them purely to timelines in which everything goes the way they want.

but, it also creates a problem if the practice becomes widespread: every person will end up observing a timeline in which increasingly greater amounts of people who they don't particularly care about, have committed suicide to go to other timelines. if i play the lottery and commit suicide if i lose, then you have as many timelines as players, each with 1 alive lottery winner, and all the others players having committed suicide. even if you don't care about living in such a world, economics cares: pre-automation, you *want* other people in your society to keep living so they can help create together the value that you can enjoy.

you can choose to commit suicide in all timelines in which too many *other* people also have committed suicide, in an acausally-collaborative effort to search for a timeline in which everyone is happy; but if no such timeline exists, then everyone will just have *truly* committed suicide out of existence.

pre-automation, this creates a [coordination problem](https://slatestarcodex.com/2014/07/30/meditations-on-moloch/), where each person wants to be able to commit suicide, but doesn't want other people to be able to. there is much ethical and political discourse to be had on the right to commit suicide; i generally lean on the libertarian side of things, but if quantum suicide becomes enough of a problem pre-automation that society looks like it's not gonna be able to get to post-automation, then we might need to consider at least disincentivizing it somehow.

post-automation, there is still a problem for people who want to live in a world which has other people in it, but the problem is much milder. it might be bringing the [end of the global era](https://carado.moe/global-era.html) even earlier than would have happened otherwise, but that's not necessarily *that* big of a deal, and there's an option for people who want to inhabit a more densely populated timeline: just lower your standard for non-population-based outcomes, such that you commit suicide less often and thus exist in more timelines. if many people do this, they should be able to find each other in many densely populated timelines.

this *does* explain the anthropic argument of, "if things go well in the future and the population booms, why are we happening to experience a particularly early age of human existence?"; other than the extinction of able-to-observe beings, this can be explained by able-to-observe beings just become really trigger-happy about quantum suicide, such that each civilization of able-to-observe beings' "observedspace" is condensed to their pre-finding-out-about-quantum-suicide; their population after that is much more sparsely distributed across timelines, even without extinction events.

as for me, i don't intend to start committing quantum suicide any time soon. i don't have strong enough confidence in the arguments posted here to take the risk of actually permanently dying. but it is definitely a possibility i'll consider, *especially* as we get closer to the singularity happening, and the existential risks that it poses.