2022-03-06

## experience/moral patient deduplication and ethics

suppose you can spend a certain amount of money (or effort, resources, etc) to prevent the spawning of a million rooms (in, let's say, simulations), with an exact copy of one random person in each. they will wake up in the rooms, spend a week not able to get out (basic necessities covered), then get tortured for a week, and then the simulations are shut down.

i want to split this hypothetical into four cases:

* the **identical** case (`I`): the million rooms and persons are exactly identical simulations.
* the **mildly different** case (`M`): the million rooms are the exact same, except that each room has, somewhere on one wall, a microscopically different patch of paint. the persons likely won't be able to directly observe the difference, but it *will* probably eventually cause the million brains to diverge from each other.
* the **quite different** case (`Q`): the million rooms will have different (random) pieces of music playing, as well as random collections of paintings on the walls, random collections of books, movies, video games, etc. to pass the time.
* the **very different** case (`V`): same as the **quite different** case, but on top of that the rooms actually contain a random person picked from random places all over the world instead of copies of the same person.

the point is that you should want to reduce suffering by preventing the scenario, but how much you care should be a function of whether/much you count the million different persons's suffering as *multiple* experiences.

it seems clear to me that one's caring for each case should increase in the order in which the cases are listed (that is, **identical** being the least cared about, and **very different** beig the most cared about); the question is more about the *difference* between consecutive cases. let's call those:

* `IM` = difference in caring between the **identical** case and the **mildly different** case
* `MQ` = difference in caring between the **mildly different** case and the **quite different** case
* `QV` = difference in caring between the **quite different** case and the **very different** case

currently, my theory of ethics deduplicates identical copies of moral patients (for reasons such as [not caring about implementation details](https://carado.moe/persistent-data-structures-consciousness.html)), meaning that i see the **mildly different** case as fundamentally different from the **identical** case. `IM > MQ ≈ QV`, and even `IM > (MQ + QV)`.

![](deduplication-ethics-1.png)

however, this strikes me as particularly unintutive; i *feel* like the **mildly different** case should get an amount of caring much closer to the **identical** case than the **quite different** case; i *feel* like i want to get `QV > MQ > IM`, or at least `QV > IM < MQ`; either way, definitely `IM < (MQ + QV)`.

![](deduplication-ethics-2.png)

here are the ways i can see out of this:

1. bite the bullet. commit to the idea that the slightest divergence between moral patients is enough to make them distinct persons worth caring about as different, much more than further differences. from a strict computational perspective such as [wolfram physics](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/), it might be what makes the most sense, but it seems quite unintuitive. this sort of caring about integer numbers of persons (rather than continuous quantities) maybe also feels mildly akin to [SIA's counting of world populations](https://handsandcities.com/2021/09/30/sia-ssa-part-1-learning-from-the-fact-that-you-exist/), in a way, maybe.
2. interpolate difference: two moral patients count *more* if they are *more* different (rather than a strict criteria of perfect equality). this seems like the straightforward solution to this example, though if the curve is smooth enough then it runs into weird cases like caring more about the outcome of one population of 1000 people than another population of 1001 people, if the former is sufficiently more heterogenous than the latter. it kinda feels like i'm *rewarding* moral patients with extra importance for being diverse; but i'm unsure whether to treat the fact that i also happen to value diversity as coincidence or as evidence that this option is coherent with my values.
3. fully abandon deduplication: count the million moral patients as counting separately in the first case. this is the least appealing to me because from a functional, computational perspective it doesn't make sense to me, and [i can make up "implementation details" for the universe under which it breaks down](https://carado.moe/persistent-data-structures-consciousness.html). but, even though it feels as intangible as positing some magical observer-soul, maybe implementation details *do* matter?
4. de-monolithize moral patients; consider individual pieces of suffering instead of whole moral patients, in the hope that in the **mildly different** case i can extract a sufficiently similar suffering "sub-patient" and then deduplicate that sub-patient.

i think i'll tentatively stick to 1 because 2 feels *weird*, but i'll consider it more; as well as making room for the possibility that 3 might be right. finally, i'm not sure how to go about investigating 4; but compared to the other three it is at least materially investigatable — surely, either such a sub-patient can be isolated, or it can't.


2022-05-14

## cognitive biases regarding the evaluation of AI risk when doing AI capabilities work

i have recently encountered a few rationality failures, in the context of talking about AI risk. i will document them here for reference; they probly have already been documented elsewhere, but their application to AI risk is particularly relevant here.

### 1. forgetting to multiply

let's say i'm talking with someone about the likelyhood that working on some form of AI capability [kills everything everywhere forever](https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence). they say: "i think the risk is near 0%". i say: "i think the risk is maybe more like 10%".

would i bet that it will kill everyone? no, 10% is less than 50%. but "what i bet" isn't the only relevant thing; a proper utilitarian *multiples* likelyhood by *quality of outcome*. and X-risk is really bad. i mistakenly see some people use only the probability, forgetting to multiply; if i think everyone dying is not likely, that's enough for them. one should care that it's *extremely* unlikely.

### 2. categorizing vs average of risk

let's take the example above again. let's say you believe said likelyhood is close to 0% and i believe it's close to 10%; and let's say we each believe the other person generally tends to be as correct as oneself.

how should we come out of this? some people seem to want to pick an average between "carefully avoiding killing everyone" and "continuing as before" — which lets them more easily continue as before.

this is not how things should work. if i learn that someone who i generally consider about as likely as me to be correct about things, seriously thinks there's a 10% chance that my tap water has lead in it, my reaction is not "well, whatever, it's only 10% and only 1 out of the two of us believe this". my reaction is "what the hell?? i should look into this and stick to bottled water in the meantime". the average between risk and no risk is not "i guess maybe risk maybe no risk"; it's "lower (but still some) risk". the average between ≈0% and 10% is not "huh, well, one of those numbers is 0% so i can pick 0% and only have half a chance of being wrong"; the average is 5%. 5% is still a large risk.

this is kind of equivalent to *forgetting to multiply*, but to me it's a different problem: here, one is not just forgetting to multiply, one is forgetting that probabilities are numbers altogether, and is treating them as a set of discrete objects that they have to pick one of — and thus can justify picking the one that makes their AI capability work okay, because it's one out of the two objects.

### 3. deliberation ahead vs retroactive justification

someone says "well, i don't think the work i'm doing on AI capability is likely to kill everyone" or even "well, i think AI capability work is needed to do alignment work". that *may* be true, but how carefully did you arrive at that consideration?

did you sit down at a table with everybody, talk about what is safe and needed to do alignment work, and determine that AI capability work of the kind you're doing is the best course of actions to pursue?

or are you already committed to AI capability work and are trying to retroactively justify it?

i know the former isn't the case because there *was* no big societal sitting down at a table with everyone about cosmic AI risk. most people (including AI capability devs) don't even meaningfully *know* about cosmic AI risk; let alone deliberated on what to do about it.

this isn't to say that you're necessarily wrong; maybe by chance you happen to be right this time. but this is not how you arrive at truth, and you should be highly suspicious of such convenient retroactive justifications. and by "highly suspect" i don't mean "think mildly about it while you keep gleefully working on capability"; i mean "seriously sit down and reconsider whether what you're doing is more likely helping to save the world, or hindering saving the world".

### 4. it's not a prisoner's dilemma

some people think of alignment as a coordination problem. "well, unfortunately everyone is in a [rat race](https://slatestarcodex.com/2014/07/30/meditations-on-moloch/) to do AI capability, because if they don't they get outcompeted by others!"

this is *not* how it works. such prisoner's dilemmas work because if your opponent defects, your outcome if you defect too is worse than if you cooperate. this is **not** the case here; less people working on AI capability is pretty much strictly less probability that we all die, because it's just less people trying (and thus less people likely to randomly create an AI that kills everyone). even if literally everyone except you is working on AI capability, you should still not work on it; working on it would *still only make things worse*.

"but at that point it only makes things negligeably worse!"

…and? what's that supposed to justify? is your goal to *cause evil as long as you only cause very small amounts of evil*? shouldn't your goal be to just generally try to cause good and not cause evil?

### 5. we *are* utilitarian… right?

when situations akin to the trolley problem *actually appear*, it seems a lot of people are very reticent to actually press the lever. "i was only LARPing as a utilitarian this whole time! pressing the lever makes me feel way too bad to do it!"

i understand this and worry that i am in that situation myself. i am not sure what to say about it, other than: if you believe utilitarianism is what is *actually right*, you should try to actually *act utilitarianistically in the real world*. you should *actually press actual levers in trolley-problem-like situations in the real world*, not just nod along that pressing the lever sure is the theoretical utilitarian optimum to the trolley problem and then keep living as a soup of deontology and virtue ethics.

i'll do my best as well.

### a word of sympathy

i would love to work on AI capability. it sounds like great fun! i would love for everything to be fine; trust me, i really do.

sometimes, when we're mature adults who [take things seriously](https://carado.moe/life-refocus.html), we have to actually consider consequences and update, and make hard choices. this can be kind of fun too, if you're willing to truly engage in it. i'm not arguing with AI capabilities people out of hate or condescension. i *know* it sucks; it's *painful*. i have cried a bunch these past months. but feelings are no excuse to risk killing everyone. we **need** to do what is **right**.

shut up and multiply.


2022-01-07 ★

## questions about the cosmos and rich computations

**computation**: a running state of any [model of computation](https://en.wikipedia.org/wiki/Model_of_computation); for example, a specific [SKI calculus expression](https://en.wikipedia.org/wiki/SKI_combinator_calculus), or a specific turing machine with its rules, current state, and current tape values. given that any model of computation can run the computations of any other model, it does not really matter which one we choose, and i will be juggling between different models throughout this post.

### 1: is any computation rich ?

**rich**: a computation is rich if it is generally [computationally irreductible](https://en.wikipedia.org/wiki/Computational_irreducibility). as a tentative formal definition for richness, i'm tempted to say that a computation is rich if there is no function able to generally predict any of its future states in a time [less than linear](https://en.wikipedia.org/wiki/Computational_complexity_theory) in the number of steps it would take to arrive at that state normally. for example, [rule 30](https://en.wikipedia.org/wiki/Rule_30) *looks* rich: it looks like to calculate the value of cell at index `i` at time step `j`, it generally takes about `O(abs(i) × j)` steps of computation. on the other hand, it looks like [rule 54 and rule 60](https://mathworld.wolfram.com/ElementaryCellularAutomaton.html) can generally have their cells predicted in time logarithmic to the number of computational steps it would naively take to arrive at them.

note that richness is not the same as halting: while a halting computation is necessarily not rich, a non-halting computation can either be non-rich (like rule 54), or rich (possibly like rule 30).

it seems clear to me that rich computations exist: for example, it is known that sorting a list of `n` elements takes `O(n × log(n))` steps, and thus a computation runnig a sorting algorithm of that complexity cannot have its result predicted in a smaller time complexity than it took to calculate naively. the ease with which i can demonstrate that, however, makes me doubt my tentative formal definition; maybe something more akin to [polynomial time complexity](https://arxiv.org/abs/1108.1791) would better capture the essence of computational irreductibility: perhaps a better determining question for richness could be "is there a function which can tell if a pattern looking like this will ever emerge in that computation, in time polynomial to the size of that pattern?" or "is there a function that can, in time polynomial to `n`, predict a piece of state that would naively take `aⁿ` steps to compute?"

### 2: does the cosmos instantiate any rich computation ?

to **instantiate a computation** means for that computation to, somewhere, eventually, be ran (forever or until it halts). i start from the fact that i'm observing a coherent-looking universe, deduce that at least *some* computation is happening, and which other computations are happening (as in, are being observed somewher, or which i could have observed). as [clarified before](https://carado.moe/limiting-real-universes.html), one can't just assume that all computations are equally happening: things look way too coherent for that, there seems to be a bias for coherence/simplicity (one which i've tentatively attributed to [how soon that computation spawns](https://carado.moe/less-quantum-immortality.html)).

looking at the cosmos (the set of instantiated computations) from a computational perspective, it seems like it contains at least our universe, which is expanding. if this expansion is, [as has been hypothesized](https://www.wolframphysics.org/technical-introduction/potential-relation-to-physics/cosmology-expansion-and-singularities/), caused by the computational substrate of the universe manufacturing new vertices of spacetime, and computations can run on this new fabric as it is produced, then it's possible that [some computations can run forever](https://carado.moe/ai-alignment-wolfram-physics.html), including potentially rich ones.

however:

### 3: does the cosmos contain causal bubbles ?

a **causal bubble** is a piece of computation that can run forever with the guarantee that it won't be physically interfered with from the outside; see [yes room above paperclips](https://carado.moe/above-paperclips-2.html).

for example, while one can build [a turing machine inside conway's game of life](https://www.conwaylife.com/wiki/Turing_machine), a stray object on the same conway's game of life plane can eventually collide with said machine and break its computational process.

however, in some [graph rewriting rulesets](https://en.wikipedia.org/wiki/Graph_rewriting), as well as in expression-rewriting systems with nested expressions such as a varient of [SKI calculus](https://en.wikipedia.org/wiki/SKI_combinator_calculus) or [lambda calculus](https://en.wikipedia.org/wiki/Λ_calculus) where the evaluation rule expands all sub-expressions, some pieces of computation can run without ever being physically interfered with by other pieces of the computation.

(i'm specifying "*physically* interfered with" because acausal coordination or mutual simulation can lead to interference, but at least that interference is up to the singleton (such as a superintelligence) "running" said bubble (if any); they can just choose to never acausally coordinate and to never simulate other bubbles)

in our own spacetime, it seems like causal bubbles exist thanks to the expansion of spacetime: some pairs of points get further apart from one another faster than celerity, and thus should never be able to interact with one another so long as that expansion continues and FTL travel is impossible. under the perspective of wolfram physics, however, it is not clear that both of those things will necessarily be the case forever; spacetime might be [hackable](https://carado.moe/brittle-physics.html).

note that the splitting of universes with nondeterministic rules (such as ours with quantum mechanics) into different causally isolated timelines is another way for causal bubbles to exist, assuming the implementation of such a nondeterministic universe is that all possibilities are instantiated at any nondeterministic choice.

the presence of causal bubbles allows some pieces of spacetime to [survive superintellingences appearing in other pieces of spacetime](https://carado.moe/unoptimal-superint-doesnt-lose.html), while the absence of causal bubbles makes it that a superintelligence or collection of superintelligences probably eventually does take over everything.

if they exist, then causal bubbles are a blessing and a curse: they save us from alien superintelligences and, [between timelines](https://carado.moe/timeline-codes.html), from our own superintelligences, but they might also ensure that our own aligned superintelligence (once we have figured out alignment) cannot reach all computation, and thus that any random person has a good chance of existing in a bubble that hasn't been "saved" by our aligned superintelligence.

### 4. is a universal-complete computation instantiated ?

[**universal complete computations**](https://carado.moe/universal-complete.html) (such as the annex in [this post](https://carado.moe/less-quantum-immortality.html)) instantiate *all* computations, over time.

if one takes the perspective that a top-level "root" bubble existed first, then the answer to this question is up in the air.

maybe we are this root computation, and the deterministic fate of the cosmos (in all timelines) is, for example, for physics to break at some point and kill everything, or for a superintelligence to appear at some point and kill everything (the two being [pretty equivalent](https://carado.moe/brittle-physics.html)) leaving [no room for bubbles](https://carado.moe/above-paperclips.html).

maybe the root bubble [does spawn](https://carado.moe/above-paperclips-2.html) a finite and small (after deduplicating by identical computations) number of bubbles, and each of those is fated to be killed in its entirety.

or, maybe somewhere in this chain, one of the bubbles spawns *many* new, different bubbles, at which point it becomes likely enough that eventually one of those bubbles either is, or itself later spawns, a universal-complete program. in which case, the initial set of the "root" bubble and maybe a few other next bubbles serve together as merely the boot process for the program that will eventually spawn *all computations*.

it might be interesting to find out how small universal-complete programs can get, both in bubble-friendly frameworks like systematically-expanded SKI calculus, and bubble-unfriendly frameworks like cellular automata; to get an idea how likely they are to randomly be stumbled into.


2021-10-13

## exact minds in an exact world

[in the sequences](https://www.readthesequences.com/Zero-And-One-Are-Not-Probabilities) it is argued that 0 and 1 are not probabilities; that these "certainty ratios" aren't meaningful. but, i can think a situation that challenges this.

imagine a fully deterministic world — for example, running on [a cellular automaton](https://en.wikipedia.org/wiki/Cellular_automata) — and imagine that in this world there are some intelligences (either artificial or natural) that utilize this determinism to have the ability to make flawless logical deductions (for example, [automated theorem proving](https://en.wikipedia.org/wiki/Automated_theorem_proving) algorithms running on computers that cannot ever have undetected [hardware failures](https://en.wikipedia.org/wiki/Soft_error)). for example, if they think about mathematics, under the axioms under which they work, 2 + 2 will always equal to 4, and doing any mathematical computation will either result in them knowing they don't have the computational resources to do the operation, or the result being guaranteedly true with the same certainty as that the cellular's automaton's rules will be applied next tick.

now, these beings still have a use for probability and statistics: those can be used to talk about parts of the world that they don't have complete information about. but, there will be some contexts, both purely in their minds (such as logic or math) or sometimes in the real world (they could make assessments like "this box cannot contain any [spaceship](https://en.wikipedia.org/wiki/Spaceship_%28cellular_automaton%29) of a certain size") that *will* be, functionally, certain.

it could be argued that they *should* still be weighing everything by the probability that there might be unknown unknowns; for example, their cellular automaton might have rules that apply only very rarely, and that they never got a chance observe yet but might yet observe later. but, let's say that they *assume* the rules of their world are exactly as they think, and let's say that they happen to be correct in that assessment. does that not make some of their deductions actually entirely certain?


2021-04-28

**DISCLAIMER: the idea described here stands or tenuous philosophysical ground and should generally _not_ be considered worth the risk attempting because it may be wrong; in addition, this plan should _not_ be utilized to retroactively justify depression-based suicide — retroactive justification is erroneous; if you are feeling suicidal, contact [your local suicide crisis line](https://en.wikipedia.org/wiki/List_of_suicide_crisis_lines).**

## Plausible Quantum Suicide

in this post, i make a collection of arguments and follow them to what seems to me like what should be their conclusion. i don't have strong confidence in every argument, but i'd consider using this plan worth it to avoid sufficiently bad scenarios, such as a singularity gone wrong (which it probably will).

### 1. The No Obtainable Evidence Argument For Materialism

by materialism i mean here something maybe closer to [physicalism](https://en.wikipedia.org/wiki/Physicalism), but maybe even a stronger version of it:

there is no special soul that people have, [you are your information system](https://carado.moe/you-are-your-information-system.html).

i make an other strong claim: time isn't particularly "moving" in any metaphysical sense, there is no "special present time". the universe can be seen as a graph of probabilistically connected states, and a random walk through those states matches the notion of entropy pretty well (which can be seen as defining the direction of time, because we happen to have memories of universe states with generally lower entropy), but that's a local notion.

the *illusion* that the present is particularly present, that we have a particular soul, or even that morality/ethics is in some sense objective, stems from the fact that we *inhabit our brain's model*: we don't get to see our brain from the outside as modeling its environment, we live right inside it, and we don't spawn with a clear distinction between normative ideas (morality/ethics) and descriptive ideas (statements of fact about the world).

but those illusions *must* be wrong, and here is the argument: as far as we can tell, there is no possible way for a brain to obtain evidence that his present time is particularly real; therefore, it must be erroneous for any brain to generate rationally the idea that its present is particularly real. same goes for having what i call a "read-only soul" that some people believe in (a special observer thing that observes a person's mind state from outside the material universe, but cannot causate anything upon it). see also [these](https://www.readthesequences.com/Zombies-Zombies) [three](https://www.readthesequences.com/Zombie-Responses) [posts](https://www.readthesequences.com/The-Generalized-Anti-Zombie-Principle).

### 2. Limiting Real Universes

my post ["Limiting Real Universes"](https://carado.moe/limiting-real-universes.html) isn't that good, so i'll try to explain it more clearly here:

if for some reason all possible states of our universe were equally real, then you should expect to observe widely anomalous phenomena around you, because most randomly picked states our universe can be in don't have to be coherent.

but the fact that we seem to observe a continuously very coherent universe tells us that there must be some sense in which coherent universe states, that stem from a continuous history following an increasing entropy timeline, must be particularly more real.

it's not that your magical soul has been blessed with inhabiting universe states: as explained in argument 1, you shouldn't have any reason to think you have such a thing.

it's not that [you can only exist to observe universe states that are coherent, because you wouldn't exist in incoherent ones](https://en.wikipedia.org/wiki/Anthropic_argument): there are still way more possible universe states where everything is incoherent except your brain, than possible universe states where everything is coherent including your brain. for any amount of state of stuff you require to say you can exist to observe the world, the rest of the universe should still generally seem incoherent if all possible universe states are equally real.

it's not that you have been following your own special arrow of time: even though i debunk that you should even think this makes sense in argument 1, another reason is that, even if some of your brain states have a past-arrow-of-time and not others, there's no reason for you to think you're one of the former. if all possible universe states were equally real, you'd likely be a version of your brain that *thinks* it has a past history but doesn't, than one that does.

### 3. Many-Worlds Is True

[Eliezer Yudkosky makes a good argument](https://www.readthesequences.com/If-Many-Worlds-Had-Come-First) that we should currently believe in the many-worlds [interpretation of quantum mechanics](https://en.wikipedia.org/wiki/Interpretations_of_quantum_mechanics); but even if that turned out wrong, [Max Tegmark makes another good argument](https://space.mit.edu/home/tegmark/crazy.html) that even just with a single history universe, all possible initial states of the universe are represented each in an infinite amount of instances by just being variations of initial conditions and random quantum determinations at different places of the infinite universe.

what matters here is that basically one should expect every reasonably possible outcome to be a real instance of universe that exists somewhere. because of argument 2, some possibilities are particularly real, and because of argument 3 (this one), that set or fuzzy set of coherent possibilities should be widely instanced: at each possible fork (they're not *really* forks, but that's a good enough analogy from our point of view), every plausible outcome is realized as a real or fairly real universe.

### 4. Quantum Suicide Works

if the previous 3 arguments stand, then a more general version [quantum suicide](https://en.wikipedia.org/wiki/Quantum_suicide_and_immortality) should be achievable: by dying instantly in one timeline, there is no version of you in that timeline able to experience it, and the only remaining future you's able to experience anything are the you's in other timelines.

because of argument 1, we know that saving a backup of your brain, and then later dying and restoring another copy of yourself from backup, is equivalent to just losing memories of the time after the backup: it's unfortunate that that time and those memories were "lost", but it's not a big deal, you can just keep going.

given that, even non-instantaneous, after-the-event suicide works: if you commit yourself to committing suicide in all timelines where an event goes wrong, then the only future you's able to experience any time after that suicide will be the ones in the timelines in which that event went well (or at least in which you think it did); you lose a bit of time and memories from those timelines in which you didn't kill yourself *literally instantly* after the thing went wrong, but it's just equivalent to a restoration from backup: the backups are automatically saved by the universe as forks of that previous universe state before the event's outcome was determined.

### ramifications

if this is true, then every person is strongly empowered: by committing themselves to committing suicide in every timeline in which even the slightest thing goes wrong, they are able to restrict the set of instances of them purely to timelines in which everything goes the way they want.

but, it also creates a problem if the practice becomes widespread: every person will end up observing a timeline in which increasingly greater amounts of people who they don't particularly care about, have committed suicide to go to other timelines. if i play the lottery and commit suicide if i lose, then you have as many timelines as players, each with 1 alive lottery winner, and all the others players having committed suicide. even if you don't care about living in such a world, economics cares: pre-automation, you *want* other people in your society to keep living so they can help create together the value that you can enjoy.

you can choose to commit suicide in all timelines in which too many *other* people also have committed suicide, in an acausally-collaborative effort to search for a timeline in which everyone is happy; but if no such timeline exists, then everyone will just have *truly* committed suicide out of existence.

pre-automation, this creates a [coordination problem](https://slatestarcodex.com/2014/07/30/meditations-on-moloch/), where each person wants to be able to commit suicide, but doesn't want other people to be able to. there is much ethical and political discourse to be had on the right to commit suicide; i generally lean on the libertarian side of things, but if quantum suicide becomes enough of a problem pre-automation that society looks like it's not gonna be able to get to post-automation, then we might need to consider at least disincentivizing it somehow.

post-automation, there is still a problem for people who want to live in a world which has other people in it, but the problem is much milder. it might be bringing the [end of the global era](https://carado.moe/global-era.html) even earlier than would have happened otherwise, but that's not necessarily *that* big of a deal, and there's an option for people who want to inhabit a more densely populated timeline: just lower your standard for non-population-based outcomes, such that you commit suicide less often and thus exist in more timelines. if many people do this, they should be able to find each other in many densely populated timelines.

this *does* explain the anthropic argument of, "if things go well in the future and the population booms, why are we happening to experience a particularly early age of human existence?"; other than the extinction of able-to-observe beings, this can be explained by able-to-observe beings just become really trigger-happy about quantum suicide, such that each civilization of able-to-observe beings' "observedspace" is condensed to their pre-finding-out-about-quantum-suicide; their population after that is much more sparsely distributed across timelines, even without extinction events.

as for me, i don't intend to start committing quantum suicide any time soon. i don't have strong enough confidence in the arguments posted here to take the risk of actually permanently dying. but it is definitely a possibility i'll consider, *especially* as we get closer to the singularity happening, and the existential risks that it poses.


2021-12-25

## non-scarce compute means moral patients might not get optimized out

i tend to assume AI-borne [X-lines are overwhelmingly more likely than S-lines or U-lines](https://carado.moe/timeline-codes.html), because in almost all cases (such as [paperclip manufacturing](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer)) the AI eventually realizes that it doesn't need to waste resources on moral patients existing (whether they're having an okay time or are suffering), and so recycles us into more resources to make paperclips with.

but [if wolfram's idea is correct](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/#how-it-works) — a possibility which [i'm increasingly considering](https://carado.moe/ai-alignment-wolfram-physics.html) — it may very well be that computation is not a scarce resource; instead printing always more paperclips is a trivial enough task, and the AI might let "bubbles" of computation exist which are useless to its goals, even growing bubbles.

and those could contain moral patients again.

of course this reduces to the [*no room above paperclips* argument](https://carado.moe/above-paperclips.html) again: inside that bubble we probly just eventually make our own superintelligence again, and *it* takes over everything, and then either bubbles appear again and the cycle repeats, or eventually in one of the layers they don't anymore and the cycle ends.

but, i still think it's an interesting perspective for how something-maximizing AIs might not need to actually take over *everything* to maximize, if there's nonscarce compute as wolfram's perspective can imply.


2021-11-20

## unoptimal superintelligence loses

(edit: [maybe it doesn't](https://carado.moe/unoptimal-superint-doesnt-lose.html))

what if a phenomena is powerful enough to kill everyone, but not smart enough to be optimal at reasoning? (such as a grey goo event, or a "dumb" superintelligence with a faulty decision mechanism)

then, in all likelyhood, it eventually dies to an alien superintelligence that is better at decision-making and thus at taking over everything.

our superintelligence doesn't just need to be aligned enough; it needs to be aligned enough, and on the tech side, to be maximally intelligent. hopefully, it's smart enough to start making itself smarter recursively, which should do the trick.

the point is: when talking about the eventual superintelligence(s) that reign over the cosmos, assume whichever one(s) to have "won" to be optimal at decision making, because others probly got outcompeted.


2021-07-16

## universal complete

under [a turing-complete model of computation](https://en.wikipedia.org/wiki/Model_of_computation), there are some initial-states or initial-states-and-rulesets which eventually contain an algorithm that iterates over all possible algorithms and runs them.

in single-threaded models, it can do this by having an increasingly long list of algorithms that it runs by one step each; it's not an issue if each algorithm runs increasingly slowly, as long as it keep running.

i choose to call such initial-states[-and-rulesets] *Universal Complete*.

they contain all turing computation based universes (and thus each other, if indirectly); so, for example, if [Rule 30 with one alive cell](https://en.wikipedia.org/wiki/Rule_30) is Universal Complete, then it contains all computable universes (including ours).

this could be interesting because proving that property about some frameworks means that programming a particular algorithm starting from that initial-state[-and-ruleset] is just a matter of *locating* it.

it could also be interesting because it might turn out that many things that *look* sufficiently chaotic (such as Rule 30 with one alive cell) are effectively universal complete, and so [Wolfram's quest](https://www.youtube.com/watch?v=0bMYtEKjHs0) for the rule that describes our universe [in his hypergraph-rewrite system](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/) might be reductible to "whichever simplest initial-state-and-ruleset starts all algorithms"; though his idea of *running every rule at every step* might kind of functionally do that.

### appendix: a simple universal-complete program

here is a simple algorithm implemeting this, iterating over the countable set of turing machines.

x ← simplest turing machine
l ← empty list
loop:
	for machine in l:
		update machine by one step of computation

	append x to l
	x ← next simplest turing machine after x


2021-07-25 ★

## what is value?

i've come to clarify my view of value sufficiently many times that i feel like having a single post i can link to would be worth it. this is that.

what i call *value* is *things we care about*; *what determines what we ought to do*. i use "morality" and "ethics" interchangeably to generally mean the study of value.

a lot of this post is just ethics 101, but i feel it's still nice to have my own summary of things.

for more on values, read [the sequences](https://www.readthesequences.com/), notably [book V](https://www.readthesequences.com/Book-V-Mere-Goodness).

see also [this post on how explicit values can come to be](https://slatestarcodex.com/2018/07/24/value-differences-as-differently-crystallized-metaphysical-heuristics/).

### consequentialism vs deontology

a first distinction is that between [consequentialism](https://en.wikipedia.org/wiki/Consequentialism), where values are about *outcomes*, and [deontology](https://en.wikipedia.org/wiki/Deontology), where values are about *actions*.

the [trolley problem](https://en.wikipedia.org/wiki/Trolley_problem) is the typical example of a thought experiment that can help us determine whether someone is a consequentialism or a deontologist: a consequentialist will press the lever because they care about the outcome of people being alive, whereas a deontologist will not press the lever because they care about the action of causing a death.

i am a consequentialist: i care about outcomes. that said, consequentialism has to be followed to the end: if someone says "well, a consequentialist would do this thing, which would eventually lead to a worse world", then they're failing to understand consequentialism: if the eventual outcome is a worse world, then a consequentialist should oppose the thing. to that end, we have [rule consequentialism](https://en.wikipedia.org/wiki/Consequentialism#Rule_consequentialism): recognizing that committing to certain rules (such as "if you commit a murder, you go to prison") help us achieve generally better outcomes in the longer term.

a special case of consequentialism is [utilitarianism](https://en.wikipedia.org/wiki/Utilitarianism), in which the consequential outcome being cared about is some form of positive outcome for persons; generally happiness and/or well-being. i tend to also value people getting their values satisfied and having [self-determination/freedom](https://carado.moe/core-vals-exist-selfdet.html) (not valuing self-determination [has issues](https://slatestarcodex.com/2018/10/24/nominating-oneself-for-the-short-end-of-a-tradeoff/)), possibly moreso than happiness or well-being, so i don't know if i count as a utilitarian.

### intrinsic vs instrumental

i make a distinction between [instrumental values, and intrinsic values](https://en.wikipedia.org/wiki/Instrumental_and_intrinsic_value) (the latter can also be called "core values", "axiomatic values", "ultimate values", or "terminal values"; but i try to favor the term "intrinsic" just because it's the one wikipedia uses).

instrumental values are values that one has because it helps them achieve other values; intrinsic values are what one ultimately values, without any justification.

* "why do i want people to get practice hygiene? so they don't get sick as often"
* "why do i want people to get sick less often? because being sick seems like a decrease in their well-being"
* "why do i want people to have well-being? i can't give a justification for that, it's what i *intrinsically* value"

any theoretical query into values should be a sequence of instrumental values eventually leading to a set of intrinsic values; and those cannot be justified. if a justification is given for a value, then that value is actually instrumental.

just because intrinsic values don't have justifications, doesn't mean we can't have a discussion about them: a lot of discussion i have about values is trying to determine whether the person i'm talking to *actually* holds the values that they *believe* they hold; people *can be* and very often *are* wrong about what values they hold, no doubt to some extent including myself.

one can have multiple intrinsic values; and then, maximizing the *satisfaction* of those values, is often the careful work of weighing those different intrinsic values in tradeoffs.

this isn't to say intrinsic values don't have causal origins; but that's a different matter from moral justificaiton.

a lot of the time, when just saying "values", people are talking about *intrinsic* values rather than all values (including instrumental); i do this myself, including throughout this post.

### knowing one's values

most people don't have a *formalized* set of values, they just act by whatever seems right to them in the moment. but, even to [rationalists](https://www.readthesequences.com/What-Do-I-Mean-By-Rationality) like me, knowing what values one has is *very hard*, even moreso in a formalized manner; if we had the complete formal description of the values of even just one person, we'd have gone a long way towards solving [AI alignment](https://intelligence.org/2016/12/28/ai-alignment-why-its-hard-and-where-to-start/), which is [by extremely far](https://carado.moe/ai-alignment-wolfram-physics.html) the [single most important problem humankind has ever faced](https://carado.moe/were-all-doomed.html), and [is gonna be very difficult to get right](https://www.readthesequences.com/Value-Is-Fragile).

to try and determine my own values, i generally [make a guess and then extrapolate how a superintelligence would maximize those values to the extreme and see where that fails](https://carado.moe/core-vals-exist-selfdet.html). but, even with that process, it is very hard work, and like pretty much everyone else, i don't have a clear idea what my values are; though i have some broad ideas, i still have to go by what feels right a lot of the time.

### selfishness vs altruism

this is *not* about how someone ultimately only wants *their values* to be satisfied; this is true *by definition*. this is about whether those values can be *about* something other than the person having the values.

people seem to be divided between the following positions:

1. all values are ultimately selfish; there is no meaningful sense in which someone can *truly, intrinsically* care about anything outside themselves.
2. someone can have values about themselves, or have values about the rest of the world, or both.
3. all values are ultimately about the world; there is no meaningful sense in which someone can actually care about their own person in particular (for example because the notion of identity is erroneous).

i hold position 2, and **strongly** reject position 1, though it seems very popular among people with whom i have talked about values; i see no reason why someone can't hold a value about the world outside of themselves, such as *intrinsically* wanting other people to be happy or *intrinsically* wanting the world to contain pretty things. for more on that, see [this post](https://www.lesswrong.com/posts/XPErvb8m9FapXCjhA/adaptation-executers-not-fitness-maximizers) and [this post](https://www.readthesequences.com/Terminal-Values-And-Instrumental-Values) from the sequences.

position 3 can make some sense if you deconstruct identity, but i believe identity [is a real thing that can be tracked](https://carado.moe/you-are-your-information-system.html), and so the outcome of which you can absolutely happen to particularly care about.

### value preservation

[value preservation](https://en.wikipedia.org/wiki/Instrumental_convergence#Goal-content_integrity) is the notion that, if you know that you value something (such as being wealthy or the world containing pretty things), you should probly try to avoid becoming someone who *doesn't* value those things, or worse: someone who values the opposite (such as being poor or the world containing only ugly things).

the reason for this is simple: you know that if you become someone who values being poor, you'll be unlikely to keep taking actions that will lead you to be wealthy, which goes against your current values; and your goal is to accomplish your values.

some people argue "well, if i become someone who values being poor, and then i take actions to that end, that's fine isn't it? i'm still accomplishing my values". but it's really not! we established that your values is "being wealthy", not "being someone whose values are satisfied". in fact, "being someone whose values are satisfied" is meaningless to have as a particular value; the fact that you want your values to be satisfied is implied in them being your values.

i call the process of someone finding out that they should preserve their values, and thus committing to whatever values they had at that moment, ["value crystallization"](https://carado.moe/value-crystallization.html); however, one ought to be careful with that. considering one's set of values is likely a very complex thing, one is likely to hastily over-commit to what they *believe* are their values, even though they are wrong about what values they hold; worse yet, they might end up committing so hard that they actually start changing what values they have towards those believed values. this is something that of course one should aim to avoid: as mentioned above, you generally don't want to become someone who doesn't hold the values you currently do, including through the process of hasty crystallization and over-commitment.

this is not to say you should remain in a complete haze where you just do whatever seems right at any moment; without a special effort, this could very well entail your values changing, something you shouldn't want even if you don't know what those values are.

what you should do is try to broadly determine what values you have, and generally try to commit to preserving whatever values you have; and in general, to *be the type of person who preserves the values they have*. this should help you preserve whatever values you actually do have, even while you still haven't figured out what they are.

a funny hypothetical version of this could be: present-you should make a contract with future-you that if they ever gain the ability to precisely examine values, they should examine what values present-you had, and adopt those.


2020-11-15

(edit: this post is *sort of* superceded by [∀V](https://carado.moe/∀V.html))

## Two Principles For Topia

the more i think about it, the less i think the solution to [Moloch](https://web.archive.org/web/20140730043944/http://slatestarcodex.com/2014/07/30/meditations-on-moloch/) is a single benevolent Elua; or, in other terms, we shouldn't implement Elua, but we should enact reasonable principles which Elua might want to implement herself.

here are what i currently believe to be the two principles that form the basis of a largely [freedom-conserving](https://carado.moe/core-vals-exist-selfdet.html) utopia:

* the first principle, Voluntaryism, consists of NAP, UBI, and population control.

	* the systematic enforcement of the [non-aggression principle](https://en.wikipedia.org/wiki/Non-aggression_principle) (NAP) to guarantee agency and freedom of association,
	* mandatory redistribution enough for every individual to be guaranteed a reasonable-with-[slack](https://thezvi.wordpress.com/2017/09/30/slack/) living (UBI) (where living includes basic resources and healthcare up to immortality), and
	* enough population control to guarantee this redistribution can even happen in the first place in a world with (even locally) limited resources,

	are to be the basis of a reasonable [voluntary](https://en.wikipedia.org/wiki/Voluntaryism) world.

	secondary notions like taxation on [externalities](https://en.wikipedia.org/wiki/Externality) and usage of [the commons](https://en.wikipedia.org/wiki/Commons) help make that UBI tangible ("why does the UBI currency have value ?" → because it's what eventually one must pay those taxes with) and reasonably redistribute ressources so as to help all persons benefit from growth.

* the second principle is the dismantlement of non-person forces (DNPF).

	what i mean by a non-person force is any phenomenon that interacts with mankind in a way that isn't answerable to persons; this goes, in order of scale, from gravity and kinetics, to cancer, to publicly-owned corporations and states. these all keep abusing persons (by which i here mean [moral patient](https://en.wikipedia.org/wiki/Moral_agency#Distinction_between_moral_agency_and_moral_patienthood)) in many ways, and just generally keep us from being in control of our lives.

	the example of corporations is particularly insidious: though they would be (under UBI) aligned to benefit the values of persons, they still outcoordinate those persons and thus in many ways outsmart them through the abuse of discoordination and cognitive biases; and not only that, but they are, in the petri dish of capitalism, bred so as to maximize their ability to do this. that said, at least fully top-down autocratic corporations have a person agent at the top, who is able to enforce the values of persons; publicly-owned corporations are even worse in that even their top-level direction is uncoordinated enough that valuing nice things is guaranteedly out of the equation (this could perhaps be addressed with better and maybe more society-distributed shareholder voting, but those shareholders probably get outcoordinated).

	(the argument above, by the way, is my largest criticism of non-[distributist](https://en.wikipedia.org/wiki/Distributism) capitalism)

	in effect, this principle turns the world we inhabit from a world of cold natural and emergent laws inside which reside some minds located in brains (materialism), into a world of ad-hoc minds determining everything else ([panpsychism](https://en.wikipedia.org/wiki/Panpsychism) ?).

	the easiest way to implement this principle is probably to move everyone to a virtual world (which saves resources too, which helps the population control cap be way higher)

in my current opinion, those two principles **must be enforced** for the basis of a utopia to be form. the rest can be done through the voluntary action of persons (hopefully), but these two principles are what Elua/the singularity is to **enforce** for the continued free and valueful life of persons to be guaranteed.

Voluntaryism alone is not enough, and this is largely missed by what i'm tempted to call right-wing utopians; not just abusive structures, but systematically self-reinforcing abusive structures, can and will still happen even under a complete voluntary society. [Meditations on Moloch](https://web.archive.org/web/20140730043944/http://slatestarcodex.com/2014/07/30/meditations-on-moloch/) addresses this largely with coordination, but coordination only *hopefully wins battles*; the addition of DNPF permanently wins the war.

DNPF alone is not enough either, and this is what is largely missed by what i'm tempted to call left-wing utopians; in a virtual world of minds where resources are fairly allocated between persons, there can still be abuse, plagues, [malthusian traps](https://en.wikipedia.org/wiki/Malthusian_trap), and so on; and ultimately abusive structures, just of a different kind. the common left-wing answer of organizing people (and the scarier "changing culture to make people systematically organize against those" which, if voluntary, is largely wishful thinking, and if not, insanely violates self-determination and the values of persons) only wins battles; the addition of Voluntaryism permanently wins the war.


2021-06-16

## the persistent data structure argument against linear consciousness

people have the belief that they live a continuous, linear stream of consciousness (whatever that means).

i've [made arguments before](https://carado.moe/quantum-suicide.html) as to why this is erroneous; but here is another interesting argument that undoes the seeming coherence of such a statement.

think of reality as a computational process, generating frames one after another, possibly splitting into timelines.

where is your consciousness? one might be tempted to answer that it's the set of bytes representing the state of the brain. if i split the world into two timelines, which one is the "fake copy" and which one is the "continuous original"? one might answer that the copy is whichever new data structure has new bytes copied to it, and that the original is whichever presence in memory hasn't been moved; the *same* bytes, supposedly stored on the *same* hardware transistors.

if i were to split timelines by creating two copies and destroying the original, one might answer that this is akin to killing the original and creating two "fakes copies".

however, there exist [persistent data structures](https://en.wikipedia.org/wiki/Persistent_data_structure), which represent new sets of data as added constructions on top of an original one. this is a perfectly reasonable way to do computation, and one would probably agree that if someone is only ever running a single timeline, people have continuous consciousness.

if i were to run a world simulation using persistent data structures and generate a timeline split, which one is the "continuous person"? just like with continuous single timeline computation, both new timelines are merely new data structures with their own set of whichever data is different, and pointers back to whichever sets of data are unchanged.

the least unreasonable choice someone who believes in linear streams of consciousness could make is that somehow persistent data structures are *not* a valid form of universe computation; that a computation ought to be run by reusing the same memory locations. surely the arbitraryness of such a claim despite its functional equivalence to persistent data structures for single-timeline computation demonstrates well enough how the notion of linear streams of consciousness doesn't make sense.


2022-01-11

## uploading people for alignment purposes

as per [my utopian vision](https://carado.moe/∀V.html), i've thought that an aligned AI would want to figure out how to upload us.

but, thinking about it more, it could be the other way around: if we can upload people in a deterministic simulation, this can buy us a lot of time to figure out alignment, as per [this post](https://carado.moe/noninterf-superint.html).

notably, the simulation could for example contain a single uploaded person (say, eliezer yudkowsky, or a bunch of copies of yudkowsky), which would save us from an arms-race type coordination problem; and while, on the outside, the superintelligence is killing everyone instantly to tile the universe with more compute to run this simulation, whoever's inside of it has plenty of time to figure things out (and hopefully [resurrect everyone once that's done](https://carado.moe/what-happens-when-you-die.html)).

this seems like a long shot, but [have you looked around?](https://www.lesswrong.com/s/n945eovrA3oDueqtq) this could be the [miracle](https://www.lesswrong.com/s/n945eovrA3oDueqtq/p/7im8at9PmhbT4JHsW) we need.

of course this could also turn into a [hell](https://carado.moe/botched-alignment-and-awareness.html) where infinite yudkowsky's are suffering forever everywhere. hopefully we can make another button which actually stops the simulation and tiles the universe with only benign paperclips, and maybe even make that button auto-activate if the yudkowsky is detected to be suffering or incoherent.

remember: [as long as the simulation is deterministic, superint can't force the uploaded yudkowsky to not shut it down](https://carado.moe/noninterf-superint.html), or force or even coerce him to do anything for that matter; it can only make the yudkowsky simulation run slower, which basically eventually achieves the same effect as either completing it or shutting it down.


2021-07-17

## AI alignment and wolfram physics

[wolfram physics](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/) is a project by [stephen wolfram](https://www.youtube.com/watch?v=0bMYtEKjHs0) to model physics using something kind of like a cellular automaton made of vertices in a graph instead of cells on a grid.

it's pretty interesting and there are insights in and around it that are of importance for the far future, and thus for [AI alignment](https://carado.moe/were-all-doomed.html).

the most notable is that wolfram thinks there's compute everywhere. the motion of the wind is doing compute, the motion of the seas is doing compute, the fabric of spacetime is doing compute, and even the state of heat death is still doing compute.

that last point notably means we might be able to embed ourselves into heat death and further, and thus get computed literally forever. this multiplies the importance of AI alignment by potentially literally infinity. i'm not quite sure how we are to handle this.

some of the compute may be doing things that are opaque to us; it might appear [homomorphically encrypted](https://en.wikipedia.org/wiki/Homomorphic_encryption). as we want (and expect) our superintelligence to spread everywhere to enforce values, we would hope civilizations living inside homomorphically encrypted spaces can be inspected; otherwise, nuking them altogether might be the only way to ensure that no [S-risk](https://en.wikipedia.org/wiki/Suffering_risks) is happening there.

wolfram postulates that one might be able to hack into the fabric of spacetime; one of the mildest effects of this would be the ability to communicate (and thus, likely, move) faster than celerity (but probably still slower than some other hard limit). if you didn't think [AI boxing](https://en.wikipedia.org/wiki/AI_box) was hopeless enough as it is, hackable spacetime ought to convince you.

finally, there is, value wise, an immense amount of compute being wasted; even just [standard model particles](https://en.wikipedia.org/wiki/Standard_Model) live way above true elementary computation. if superintelligence is well-aligned, this provides us with an hard estimate as to how much computing power we can live on to enjoy value, and it's probably a very large amount; wolfram [talks about](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful#how-it-works) something like 1e400 vertices in our universe.


2021-03-04

## Value Crystallization

there is a weird phenomenon whereby, as soon as an agent is rational, it will want to conserve its current values, as that is in general the most sure way to ensure it will be ablo to start achieving those values.

however, the values themselves aren't, and in fact [cannot](https://en.wikipedia.org/wiki/Is–ought_problem) be determined purely rationally; rationality can at most help [investigate](https://carado.moe/core-vals-exist-selfdet.html) what values one has.

given this, there is a weird effect whereby one might strategize about when or even if to inform other people about [rationality](https://www.readthesequences.com/) at all: depending on when this is done, whichever values they have at the time might get crystallized forever; whereas otherwise, without an understanding of why they should try to conserve their value, they would let those drift at random (or more likely, at the whim of their surroundings, notably friends and market forces).

for someone who hasn't thought about values much, *even just making them wonder about the matter of values* might have this effect to an extent.


2022-05-13

## life refocus

because of the [recent](https://www.metaculus.com/questions/3479/date-weakly-general-ai-system-is-devised/) [events](https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy), which i've been dreading [for a while](https://carado.moe/were-all-doomed.html), i'm taking AI risk a lot more seriously, and have started significantly refocusing my life.

there is a post called [*musk's non-missing mood*](https://lukemuehlhauser.com/musks-non-missing-mood/) that resonates quite well with me. it is indeed kind of disconcerting how people who seem rationally aware of AI risk, don't seem to *grok* it as an *actual thing*. despite how real it is, it's hard to think of it not as fantasy fiction.

i totally understand why. i've been there too. but eventually i managed to progressively update.

i'm still not quite there yet, but i'm starting to actually grasp what is at stake.

["detaching the grim-o-meter"](https://mindingourway.com/detach-the-grim-o-meter/) remains a reasonable thing to do; you don't want to become so depressed that you kill yourself instead of saving the world. but you also don't want to remain so deluded that you don't quite weigh the importance of saving the world enough either.

i'll learn japanese after the singularity. i'll make [my game](https://carado.moe/game.html) and [my alternative web](https://carado.moe/saving-the-web.html) and my conlang and [my software stack](https://carado.moe/psi.html) and many other things, after the singularity. it is painful. but it is what's right; it's closer to [the best i can do](https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy).

and i know that, if at some point i give up, then it won't look like pretending that everything is fine and compartmentalizing our imminent death as some fantasy scenario. it'll be a *proper* giving up, like going to spend the remaining years of my life with my loved ones. even my giving up scenario is one that takes things seriously, as it should. that's what being an adult capable of taking things seriously is like.

how you handle your mental state is up to you. there is a collection of AI-risk-related mental health posts [here](https://www.lesswrong.com/posts/pLLeGA7aGaJpgCkof/mental-health-and-the-alignment-problem-a-compilation-of). do what it takes for you to do the work that needs to be done. that's not becoming a doomer; your brain is straight-up not designed to deal with cosmic doom. but that's not remaining blindly naive either. the world needs you; it won't be saved by pretending things are fine.

and it *certainly* won't be saved by pretending things are fine and *working on AI capability*. that's *just bad*. *please* don't.

please take AI risk seriously.


2020-12-22 ★

## Rationalist by necessity

in [The Sequences](https://www.readthesequences.com/), Eliezer Yudkowsky [describes rationality](https://www.readthesequences.com/What-Do-I-Mean-By-Rationality) as

1. **Epistemic rationality**: systematically improving the accuracy of your beliefs.
2. **Instrumental rationality**: systematically achieving your values.

now, personally, i [intrinsically value](https://carado.moe/core-vals-exist-selfdet.html) a bunch of things, but having accurate beliefs isn't necessarily one of them; for me, rationality is an [instrumental value](https://en.wikipedia.org/wiki/Instrumental_and_intrinsic_value) in that it helps me achieve my other values better.

in general, i value people being able to do whatever they want, and as such they shouldn't necessarily have to form accurate beliefs if they don't care to. in fact, forming inaccurate beliefs is a great source of culture, and culture is something that i *do* personally intrinsically value.

but we live in the era of liberal democracies, where society requires people to form accurate beliefs, because they're the ones directing society through elections. i see the need for people to be rationalist as an unfortunate necessity; hopefully a need we can be rid of when we [reach a topia where human decisions are no longer the pillar of civilization](https://carado.moe/two-principles-for-topia.html).

not, of course, that there's anything wrong with any individual or even group choosing to intrinsically value rationality. the part i care about is that it being a choice.


2020-12-25

## You are your information system

what makes you, you ?

we tend to intuitively think of a person as their entire body, somehow including limbs and organs but not clothing or food.

yet, if you close your eyes, and then i swap your arm with someone else's, when you wake up you will still be the same person, just with a new arm. in fact, i'd argue i could replace everything except for the nervous system (including the brain) and when you open your eyes again you would notice that your entire body has changed but your thoughts and memories have remained the same — rather than, for example, still having the same body but different thoughts and memories.

are you the matter that makes up that nervous system ? i could probably replace neurons and synapses one at a time and you would continue to be the same person. is it the electric signals then ? i could probably put on some synapses a device that absorbs electric signals and then sends out identical but "different" signals and you would still be the same person.

in fact, it doesn't really make sense to ask "which matter" makes up your nervous system: under quantum physics, everything is changing and particles are merely [values in an omnipresent field](https://www.youtube.com/watch?v=MmG2ah5Df4g) rather than solid objects.

ultimately, what you are, is *the information system* which your nervous system (including your brain) runs. standing still, walking forwards, teleporting yourself, and being uploaded into a sufficiently powerful computer, all preserve your personhood in the exact same way; there is nothing special about the meat that currently runs your mind.

*despite everything, it's still you.*


2022-01-05

## brittle physics and the nature of X-risks

suppose physics is hackable, and a hard to accomplish hack that requires intelligence (like a fancier version of [rowhammer](https://en.wikipedia.org/wiki/Rowhammer)) can break the fabric of spacetime — maybe in ways that said intelligence can take advantage of, such as embedding its computation into something that survives said breakage, in a way that could help such a superintelligence accomplish its goal.

we could expect that [boxing an AI](https://en.wikipedia.org/wiki/AI_box) could be really hard: even without access to the outside, it might be able to guesses physics and hack it, from the comfort of its box.

as usual in such [X-risk scenarios](https://carado.moe/timeline-codes.html), i believe we just [keep living only in timelines in which, by chance, we don't die](https://carado.moe/quantum-suicide.html).

these sort of hacks are not ruled out by [wolfram physics](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/). indeed, they are plausible, and can spread at some speed faster than celerity — because they can run in the substrate *underlying* spacetime — such that nobody would ever be able to observe such hacks: the hack reaches and destroys you before the result of the breakage can reach your sensory organs, let alone your brain.

so, maybe "dumb-goal" superintelligences such as [paperclip maximizers](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer) are popping up all over the place all the time and constantly ruining the immense majority of not-yet-hacked timelines, and we keep living in the increasingly few timelines in which they haven't done that yet.

now, let's stop for a minute, and consider: what if such a hack *isn't* hard ? what if it *doesn't* need an intelligent agent ?

what if, every planck time, every particle has a 99% chance of breaking physics ?

well, we would observe exactly the same thing: those hacked universes either become computationally simple or [boot up more universes](https://carado.moe/above-paperclips-2.html); either way, we don't survive in them, so we don't observe those hacks.

in this way, it is [S-lines and U-lines](https://carado.moe/timeline-codes.html) that are very special: outcomes in which we *survive*, thanks to a superintelligence with a "rich" goal. the rest is just timelines constantly dying, whether it be due to X-risk superintelligences, or just plain old physics happening to cause this.

in fact, let's say that the universe is [a nondeterministic graph rewriting system](https://en.wikipedia.org/wiki/Graph_rewriting) with a rule that sometimes allows everything be reduced to a single, inactive vertex. would this count as "sometimes everything is destroyed" ? or would this make more sense to be modeled as a weird quirk of physics where the graph of possible timelines includes the production of passive vertices all the time, which can be safely ignored ?

what if instead of a nondeterministic system, we have a deterministic one [which just happens to expand all timelines](https://carado.moe/psi-rewriting.html). in such a system, "different timelines" is no longer a primitive construct: it is merely an observation about the fact that such a system tends to, when ran, create from a given piece of data, several newer ones. let's say that in such a system there is a rule where from every piece of data we'd consider a timeline, numerous inert vertices are also created.

would we say "aha, look! every time a computation step happens, many inert vertices are created around it, and i choose to interpret this as the creation of many timelines (one per inert vertex) in which everyone in that universe dies, and others (new complex pieces of data) in which everything keeps existing",

or would we, in my opinion more reasonably, say "well, it looks like as a weird quirk of how this system runs, many inert vertices are popping up; but they're simple enough that we can just ignore them and only consider richer new pieces of data as *timelines* proper."

i believe, if we are to worry about what states this universe ends up in, we ought to use a measure of what counts as a "next state of this universe" that measures something about the richness of its content: maybe the amount of information, maybe the amount of computation going on, or maybe the number of moral patients. and, depending on what measure we use, "losing" timelines to paperclip maximizers (which turn the universe into something possibly simple) is no more of a big deal than "losing" timelines to a rewriting rule that sometimes creates inert vertices, and neither of which should really count as proper timelines.

otherwise we end up needlessly caring about degenerate states because of what we believe to be, but really isn't, an objective measure of what a timeline is.

*timelines* might be in the [map](https://en.wikipedia.org/wiki/Map–territory_relation), while what is in the territory is just *what we end up observing* and thus, computed states that contain us.

finally, what about universe states where *all* outcomes are an inert vertex or an otherwise simple universe (such as as infinite list of identical paperclips) ? while those might happen, and i'd say *would* count as X-risks, you don't need to consider simple states as timelines to make that observation: maybe some timelines end up in a state where *no* new states can be created (such as a locally truly terminated piece of computation), and others end up in a state where *only simple* new states are created. those ought to be considered equivalent enough, and are what a true X-risk looks like.


2021-12-27

## *less* quantum immortality?

if the set of nested universes [really does](https://carado.moe/what-happens-when-you-die.html) look like a funny graph of bubbles, i think there are two likely possibilities: either the set of bubbles rapidly dries up, or it grows towards infinity; in which case, if compute is infinite [as wolfram would have me think](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/) then as soon as the bubble explosion happens, it's likely a [universal complete](https://carado.moe/universal-complete.html) algorithm is booted somewhere reasonably fast, itself booting in turn all initial states.

this has the result of instanciating all (countable, discrete) [tegmark 4 universes](https://space.mit.edu/home/tegmark/crazy.html), over time.

yet, [we still observe a preference for coherency](https://carado.moe/limiting-real-universes.html): i think the reasonablest interpretation of what'd be going on is that "computationally early" or at least "computationall frequent" states are favored; and thus, very weird and incoherent initial-state universes *do* get spawned, but much later and/or are being computed more slowly (for example, maybe computation is equally distributed among all timelines, and as more and more timelines spawn over time each individual one gets updated less and less often).

while this creates a neat explanation for what selects for universe coherence, it does make it that while [quantum immortality/suicide](https://carado.moe/quantum-suicide.html) can be considered to "still work", if you choose to keep living [only by waiting to be reincarnated later](https://carado.moe/what-happens-when-you-die.html), you're reducing the "realness" of your continued existence; you're making universes in which you continue to live appear only "computationally later".

it also provides a nice simplicity test fo roccam's razor: the simplicity of a hypothesis can be akin to how soon a universal-complete program that simulates all spawned computations arrives at it.

this probly doesn't apply to "classical" quantum immortality where you just use the fact that you're redundanced on other timelines, because i would imagine those other you's in other timelines would tend to be computed "at the same time".


2020-09-09 ★

## Determining core values & existential self-determination

[Rationalism](https://www.readthesequences.com/) is about [epistemic rationality and instrumental rationality](https://www.readthesequences.com/What-Do-I-Mean-By-Rationality); but [when the two conflict, "rationalists should win"](https://www.readthesequences.com/Newcombs-Problem-And-Regret-Of-Rationality); so,

> Instrumental rationality: systematically achieving your values.

How does one determine their core (axiomatic) values ? Here's how i do it: i start from what i think is my set of values, and then i extrapolate what would happen if a [superintelligent](https://en.wikipedia.org/wiki/Superintelligence) [singleton](https://en.wikipedia.org/wiki/Singleton_%28global_governance%29) tried to implement those values.

Generally, the result looks like hell, so i try to figure what went wrong and start again with a new set of values.

For example: imagine i think my only core value is general happiness. The most efficient way for a superintelligence to maximize that is to [rewire everyone's brain](https://wiki.lesswrong.com/wiki/Wireheading) to be in a constant state of bliss, and [turn as much of the universe as possible](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer) into either more humans that experience constant bliss (whichever form of "human" is the cheapest resource-wise to produce) or into infrastructure that can be used to guarantee that nothing can ever risk damaging the current set of blissful humans.

So, clearly, this is wrong. The next step is freedom/self-determination; such that people can do whatever they want.

However, the most efficient way to make sure people can do what they want is to make sure they don't want to do anything; that way, they can just do nothing all day, be happy with that, and some form of freedom is maximized.

To address this issue, my latest idea is to value something i'd like to call *exstential self-determination*: the freedom to *exist as you would normally have*. It's a very silly notion, of course; there is no meaninful "normally". But still, i feel like something *like that* would be core to making sure not just that existing people can do what they want, but that humankind's general ability to be original people who want to do things is not compromised.


2022-05-18

## my rough AI risk estimates

the numbers are spoilered so you can make your own guess without getting anchored. click on the text to reveal the number.

these are my pretty rough estimates for what happens to our current civilization:

* we robustly [get past](https://en.wikipedia.org/wiki/AI_alignment) AI risk0.01%
* our civilization dies in a non-AI-existential-risk way, leaving room for future civilization0.1%
* we [die](https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence) of AI99.89%

if we die of AI, when does it happen?

* 2020s60%
* 2030s30%
* 2040s5%
* the remaining % get spilled over the remaining future

rember that if we get past those decades without dying, that's not evidence that i got those wrong; that's evidence that either i got those wrong, or our civilization benefits from some amount of quantum immortality.

part of my reasons for believing in those odds is private information.


2022-02-06

## forking bitrate and entropy control

if physics is based on a computational framework such as [wolfram physics](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/), but plausibly even if not, such that not all states the universe can be in produce the same number as next possible states;

in addition, if i am to follow to conclusion my current belief that moral patients count as different when they start being functionally different in terms of computation, and that exact copies morally count as a single person (as it makes [not much sense](https://carado.moe/persistent-data-structures-consciousness.html) to believe otherwise);

and if ([as it seems to be the case](https://carado.moe/limiting-real-universes.html)) the universe values coherence and thus only a limited set of local outcomes can emerge from a given local situation, or at least outcomes are weighed by coherence;

then it makes sense to start caring about the amount of forking a given timeline goes through. which is to say: the amount of future states to be [instantiated](https://carado.moe/questions-cosmos-computations.html), be it directly next step or indirectly in the longer term.

in fact, if one calls what they care about *moral patients*, then we should care about the "forking bitrate" of moral patients. for example, we could want moral patients with a net negative future to be forked as little as possible, and moral patients with a net positive future to be forked as much as possible. considering forks are created over steps of time, and entropy seems to be a good measure for them, i think "bitrate" is an appropriate term for this; hence, *forking bitrate*.

if we're just talking about a place as small as earth, we can estimate that consequences rapidly ramificate around to all moral patients; and as such, it seems reasonable to think that the forking bitrate of all patients will tend to go about in the same direction.

so, if you see a quantum dice, should you throw it?

if you think the future of earth has expected net positive moral value, or has little enough suffering for your taste (depending on your moral framework), then yes: by throwing the (quantum) dice, you might be multiplying the amount of instances of that value by the number of possible outputs of the dice, by creating that many times more future timelines.

if not, then you shouldn't throw it.

(even in the absence of quantum effects, if one were to just move entropy around while [phase space being conserved](https://www.lesswrong.com/posts/QkX2bAkwG2EpGvNug/the-second-law-of-thermodynamics-and-engines-of-cognition), moving the entropy from not-moral-patients to moral-patientss (or whichever thing you can about) still has that effect, i think)

this can probly be expanded to much larger-scale entropy control — and also, if superintelligences care about it (and if they're to be aligned, we might want them to) we can expect them to use it to maximize their value. even a [paperclip maximizer](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer) can want to create as many timelines containing as many varied possible paperclips and arrangements thereof, if it is made to care about that.


2020-03-30

*(2020-11-15 edit: this post is now largely superceded by [Two Principles For Topia](https://carado.moe/two-principles-for-topia.html))*

## Topia: Layer 0

In a similar way to the [Hierarchy of Needs](https://en.wikipedia.org/wiki/Maslow's_hierarchy_of_needs), I have been thinking about what post-singularity utopia we would want in term of layers.

I want a Layer 0, a universal set of guarantees that apply to everybody; so that, on top of that, people can build voluntary societies and sub-societies as far as they want. Ideally, their socities would be mutually compatible; one could partake of multiple societies and have friends in both. But they wouldn't have to be. It's all dependent on what society you want to join.

But, I think we do need a universal Layer 0. One that at least makes the singularity AI prevent other AIs from emerging and [turning everyone into paperclips](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer) or other existential risks. As this is the layer that applies universally, we want it as thin as possible, so that societies built on top of it have as much freedom to implement whatever they want; in particular, almost everything mentioned here can *be opted out of* (such as when joining a social contract). It's just your starting kit.

These should be the universal guarantees that everyone starts with: physical safety and basic living ressources.

For the physical safety, it's fairly easy to think of telling the singularity AGI to implement what I like to call NAPnobots — nanobots that are omnipresent in physical reality (and would manifest as mandatory added laws of physics to virtual realities) and enforce the [NAP](https://en.wikipedia.org/wiki/Non-aggression_principle); that is, prevent people and their physical property from being subjected to aggression without their consent ("without their consent" could be a tricky part — also, should people be able to *permanently* opt out of some NAP violation protections ?).

You want to create a society in which it's fine to punch each other in the face ? That's fine with me. All I ask is that that society be purely opt-in.

You want to create a communist utopia in which all belonging are shared ? That's fine with me. Just create a voluntarily contract where people consent to pooling their properties together for shared use.

You want the freedom to hurt yourself ? Just consent to being hurt by yourself. You want the freedom to hurt non-consenting others ? *No.* That's my personal opinion, of course, but I do think the maximum reach of a social contract should be that it can't force others into joining it, and I hope everyone else can agree that at least requiring one's consent to partaking of interaction should be a guaranteed absolute. On the other hand, I would definitely consent, personally, to very large ranges of interactions with at least my friends. They can punch me if they want; I trust them to not do that, and if my trust is betrayed beyond what I consider reasonable, I can always unconsent.

Also, since the brain evaluates every input it receives, this includes for the most part having to consent to any form of communication; rember that unconsented advertising *is* nigh-literally rape.

The second part, basic living ressources, is a concern of economics. Unless we manage to escape to universes where ressources not only are infinite, but can be accessed faster than humans can appear (which may become as simply as duplicating a running process on a computer), it requires among other things limiting the number of humans that can exist, or you run into [malthusian traps](https://slatestarcodex.com/2014/07/30/meditations-on-moloch/).

The best way I can think of doing this is: when the singularity starts, give everyone enough basic assets that the dividends that can be generated from them are easy to live off of. Then, for people to have a child, they need to acquire a number of assets that will provably generate the same amount of dividends for the child, and give those to him. On top of this arbitrarily complex liberal contracts can be established, of course; you can have a socialist society where everyone has consented to their ressources being taxed by exactly how much giving basic living assets to the new kids costs (kids which won't themselves be taxed in that way unless they consent to in turn join that society). There is the issue of what amount of ressources constitutes basic living, as well as if there is a cap on the amount of unviolable property a person or group can have — can an environmentalist group very quickly just plant a flag on all nearby useful planets (as to declare them their unviolable property) and then forever refuse for them to be interacted with (as the NAPnobots will enforce) ?

The choice of living ressources should be included: if what we choose to eat has as much of an effect on our psyche as we are starting to find out it does, then choosing what configuration of nutrients we receive should part of the basic living guarantees.

One of those basic ressources of course is healthcare, and healthcare should include guaranteed immortality unless you opt out of it. There's just no particular reason old age should have any more right to hurt a non-consenting person than any *other* outside aggressor; "outside" because people are their brain's information system, not their body. Becoming a virtual person would be the "easy" solution to immortality.


2022-05-12

## hope for infinite compute

here are some reasons we may have infinite universe to inhabit in the future.

* encoding ourselves in heat death noise. this could at least buy us exponentially much time to exist; it becomes infinite if the amount of possible states also increases.
* where are we in [the universal distribution](https://www.lesswrong.com/posts/EL4HNa92Z95FKL9R2/a-semitechnical-introductory-dialogue-on-solomonoff-1) ? most models that produce [rich](https://carado.moe/questions-cosmos-computations.html) computation, such as [rule 30](https://en.wikipedia.org/wiki/Rule_30), seem to grow forever in amount of stuff. this includes [wolfram's hypergraph rewriting system](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/).
* space is expanding. the planck length is a constant. unless there's something i'm mistaken about, this sure seems like more positions in space we could inhabit, and thus an increasing total amount of states the universe can be in. it is not obvious how to utilize this, but it may be evidence for other ways in which the amount of stuff in the universe grows. in wolfram's perspective, it is simply the hypergraph creating new positions in space, as it has been doing forever.
* why are there 10⁸⁰ particles in the observable universe? if the total number is larger, why is it that larger number? wouldn't it be occam-simpler that there be 1 or few particles (or qubits or whatever) in the start, and have that amount grow over time? in fact, with expanding space, won't there statistically tend to be more particles overall, if only because there's more space for quantum fluctuations to randomly spawn particles? surely a superintelligence can harness this.
* even if we're in a physically finite universe, we may be able to acausally trade/hack/blackmail aliens living in infinite worlds such as rule 30. maybe. [acausal trading is weird](https://www.lesswrong.com/posts/PcfHSSAMNFMgdqFyB/can-you-control-the-past).


2022-01-11 ★

## how timelines fall

i've speculated that we are all together, as a civilization, quantum immortal; [timelines where we all die](https://carado.moe/timeline-codes.html) can somewhat [be ignored](https://carado.moe/brittle-physics.html), leaving us mostly just with concerns of [heaven vs hell](https://carado.moe/botched-alignment-and-awareness.html) timelines.

but, in the lucky timelines where we *do* keep avoiding an [X-risk](https://en.wikipedia.org/wiki/X-risk) [superintelligence](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer), what does that look like ?

it would be silly to expect that avoiding such a superintelligence would look like trying to press the button to turn it on but at the last minute the button jams, or trying to press the button to turn it on but at the last minute the person about to press it has a heart attack. indeed, bayes should make us think that we should expect it to look like whatever makes it likely that superintelligence fails to be implemented.

what does this look like ?

global nuclear war, broad economic collapse, great cataclysms or social unrest in cities where most of the AI development is done, and other largely unpleasant events.

don't expect the world to look like the god of anthropics is doing miracles to save us from superintelligence; expect the world to look like he's is slowly conspiring to do whatever it takes to make superintelligence unlikely to happen long in advance.

expect the god of anthropics to create AI winters and generally make us [terrible at software](https://www.youtube.com/watch?v=pW-SOdj4Kkk).

expect the god of anthropics to create plausible but still surprising reasons for the availability of tensor hardware to become scarce.

look around. does this look like a century where superintelligence appears ? yes, i think so as well. the god of anthropics has his work cut out for him. let's try and offer him timelines where AI development slows down more peacefully than if he has to take the initiative.

while some of us are working on aligning god, the rest of us should worry about aligning luck.


2021-12-25 ★

## yes room above paperclips?

in two [previous](https://carado.moe/above-paperclips.html) [posts](https://carado.moe/nonscarce-compute-optimize-out.html) i talk about the ultimate inability for interesting things to happen when everything has been tiled with paperclips, even if the superintelligence doing the tiling isn't very good at it — i.e. lets room exist ["besides" (by superintelligence not actually consuming everything)](https://carado.moe/nonscarce-compute-optimize-out.html), or ["above" (using as a substrate)](https://carado.moe/above-paperclips.html) said paperclips (or whatever else the universe is being tiled with).

but, actually, this is only true if the spare compute (whether it's besides or above) only has room for one superintelligence; if that spare compute is composed of multiple bubbles causally isolated from one another, then maybe a superintelligence permanently kills everything in one, but another one creates even more bubbles in another.

in fact, as long as the first superintelligence to create many bubbles preceeds the first superintelligence to create no bubbles at all, and then if the amount of bubbles tends to be slightly more than one, and assuming superintelligences can't (or can only do so at a lesser rate than new bubbles being created) just "hack back upwards" (to escape to their parent universe), we can expect the set of pre-X-risk superintelligence bubbles to just increase over time.

this might provide a better explanation [than just us dying forever](https://carado.moe/estimating-populated-intelligence-explosions.html), for the weird fact that we exist now when the future could contain very many ([plausibly infinitely many](https://carado.moe/ai-alignment-wolfram-physics.html)) persons: it's not just that the amount of pre-singularity population is large compared to future timelines multiplied by their low likelyhood of being populated, it's that it grows over time forever and so makes it harder for U-lines or S-links to "compete", expected population wise.

we can then run into weird questions: rather than a tree, or even a [DAG](https://en.wikipedia.org/wiki/Directed_acyclic_graph), why couldn't this be just a general graph? if [the "seeds" for complex universes can be simple](https://en.wikipedia.org/wiki/Rule_30), it makes sense to imagine bubbles causating each other: maybe someone in [Rule 30](https://en.wikipedia.org/wiki/Rule_30) eventually boots a superintelligence that takes over everything but happens to cause a [Rule 110](https://en.wikipedia.org/wiki/Rule_110) bubble to appear (perhaps among many others), and then in that Rule 110 bubble someone creates a superintelligence that causes a Rule 30 bubble to appear again.

conceptually navigating this likely cyclical graph of pre-superintelligence bubbles seems like headache so i'll put the matter aside for now, but i'll be thinking on it more in the future. for the moment, we should expect bubbles with simpler seeds to be highly redundanced, and ones with more complex seeds to be rarer; but there's no reason to assume any ceiling on bubble seed complexity (in fact, if even just one of these bubbles is [universal complete](https://carado.moe/universal-complete.html), then *any* seed eventually gets instanciated!), and it seems nigh impossible to predict which types or complexities of seeds could lead to which outcomes, superintelligence-wise.

in the meantime, rember that while things might look pretty hopeless with this perspective, [it's plausible that we can actually actually causate *very far*](https://www.lesswrong.com/posts/PcfHSSAMNFMgdqFyB/can-you-control-the-past).


2022-02-04

## balancing utilitarianism

suppose you have multiple values ("i want to be healthy but also i want to eat a lot of fries") or a value applying to multiple individuals ("i want both alice and bob to be happy"), but sometimes there are tradeoffs between these values. how do you resolve such situations ?

a simple weighed sum might suffice in many cases, but i feel like there are cases where this is not sufficient.

for example, consider a population of 5 persons, who you care about equally, and consider a simple scalar value you have for them, such as happiness.

now, consider the following three options:

* all individuals get 0.5 utility ("fair")
* one individual gets 0.9 utility, the other four get 0.4 utility ("bully")
* one individual gets 0.1 utility, the other four get 0.6 utility ("scapegoat")

if we are to use a simple sum, all three of these situations sum to 2.5 total utility; yet, i feel like something ought to be done to favor the fair situation over the other two (and then probably to favor the bully situation over the scapegoat situation?)

what i propose to address this is to apply a square root (or other less-than-one exponent) to the utilities of persons before summing, which has the effect of favoring more equal situations. in this case, we get:

* fair: 3.54 utility
* bully: 3.48 utility
* scapegoat: 3.41 utility

which does seem to produce the desired effect: in this situation, it maps to how i *feel* about things: fair > bully > scapegoat


2021-12-09

## unoptimal superintelligence doesn't lose

i [previously](https://carado.moe/unoptimal-superint-loses.html) wrote a post about how a superintelligence with an unoptimal decision system likely loses to alien superintelligences that are more optimal, at the scale of cosmic wars between those superints.

i don't think this is necessarily true: maybe physics *does* look like a funny graph à la wolfram, and then maybe we can carve out pieces of space that still grow but are causally isolated from the rest of the universe; and then, whether a given causally isolated bubble ever has to encounter an alien superint is purely up to whether it decides to generate alien space that leads to those, which is prevented easily enough.