2021-07-18

## when in doubt, kill everyone

one thing that is way worse than [mere existential risks](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer), possibly [by a factor of infinity](https://carado.moe/ai-alignment-wolfram-physics.html), is [suffering risks, or S-risks](https://en.wikipedia.org/wiki/Suffering_risks).

i could see (though going by what i could see [is not a reliable apparatus](https://carado.moe/overcoming-narratives.html)) someone make an AI and, while trying to align it to human values, accidentally misaligns it to something that happens to tile the universe with suffering humans. this would be an instance of S-risk.

whereas, an AI that merely wants to accomplish a relatively simple goal will probly just tile the universe with something simple that doesn't contain suffering persons; and given that [we're all probly quantum immortal](https://carado.moe/quantum-suicide.html), we just "escape" to the timeline where that didn't happen.

considering this, a 99% chance of X-risk 1% chance of utopia is preferable to a 1% chance of S-risk 99% chance of utopia. so, one thing we might want to do if we figure out superintelligence before we do alignment (which [seems pretty likely at this point](https://carado.moe/were-all-doomed.html); see also "Zero percent" on [this page](https://intelligence.org/2018/10/03/rocket-alignment/)), we might want to keep a ready-to-fire paperclip AI on standby and boot it up in case we start seeing S-risks on the horizon, just to terminate dangerous timelines before they evolve into permanent exponential hell.

in fact, just to be sure, we might want to give many people the trigger, to press as soon as someone even *suggests* doing any kind of AI work that is not related to figuring out goddamn alignment.


2020-04-27

(2020-04-27 edit: actually Greg Egan already made this argument previously ([see Q5 if you have read Permutation City](https://www.gregegan.net/PERMUTATION/FAQ/FAQ.html)))

(2021-04-28 edit: this post might not be the best job at explaining its idea; see an alternate explanation in [this other post](https://carado.moe/quantum-suicide.html))

## Limiting Real Universes

The following is an argument for thinking that the set of universes that can "be real" (what that means is covered) is limited.

Notably, not all [Tegmark 4](https://space.mit.edu/home/tegmark/crazy.html) (i.e. mathematically possible universes) are real, nor even all considerable states of universes based on our current physics.

### I. Limiting Many-Worlds

A universe being "real" is defined here as "one could observe being in it".

Suppose all possible configurations of particles under our current laws of physics are *real*.

Then, out of all the universes that contain an exact physical copy of you, the vast majority of them should be universes that *do not* descend from a coherent history and thus everything that surrounds the copy of you should look like random particle soup.

(If such a distinction even makes sense), then You cannot tell if you're "the original" that comes from a coherent history or if you were "just created" as-is, because your memory could also "just have been created" as-is.

Yet, when you look around, everything looks very coherent.

Therefore, either you're *extremely* lucky, or only universe-states that descend from a coherent history are *real*. As per bayesianism, you should think the latter.

### II. Limiting Computing Ability

Suppose all universes based on our current physics, but with arbitrary amounts of "computing power" (i.e. how of its stuff can be turned into computers i.e. how much it has stuff) are "real".

Then some of those universes would end up making simulations of random universe-states, some of which happen to contain an exact copy of you.

However, if that were possible, because of the number of amounts of computing powers possible, *you* should be more likely to exist in one of these randomly created simulations with a universe with much more computing power than ours.

Yet, when you look around, everything looks very coherent.

Therefore, there must be *some* limit on the amount of computing power universes can have; and then, so that the sheer number of these universe can't compete with the meagre set of history-coherent from which our reality descends, there must either be a limit in the number of initial configurations other universes can have, or on the total computing power allocated to all universes.

Even better: suppose all states of [Conway's Game of Life](https://en.wikipedia.org/wiki/Conway's_Game_of_Life) are *real*. Then out of this infinity, a smaller infinity should happen to be running perfect simulations of subsets of this universe that happen to have you in them. But, reusing my argument again, you observe probably not being in them; therefore there must be a limit on the amount of computing power *even universes with other physics* have (and, again, a limit on either the number of configurations these other universes would be in, or the total computing power allocated to them all collectively).

### III. Conclusion

At this point it seems easier to *just assume that only universe-states that descend from our history* exist, or at least that the number of such histories is limited.

That certainly seems simpler than imagining there being a set of various systems by which other universes with other rules of physics would have a fixed amount of computing power allocated amongst themselves.

Not that all of this is mostly true even if you're a dualist: even if you have a soul (or equivalent), if there were infinite universes with infinite computing power, there's no reason the soul of *you reading this right now* should happen to be the soul of the original you and not a soul "created in a just-then created universe-state", unless you also assume complex soul mechanics.


2021-08-31 ★

## ∀V: A Utopia For Ever

-->

∀V (read "universal voluntaryism" or "univol") is my utopia proposal. for people who are familiar with me or my material, this may serve as more of a clarification than an introduction; nevertheless, for me, this will be the post to which i link people in order to present my general view of what i would want the future to look like.

### what's a person?

you'll notice that throughout this post i've stuck to the word "person" instead of, for example, "human". this isn't just in case we eventually encounter aliens who we consider to be persons just like us, but it's also possible some existing animals might count, or even beings whose existence we largely don't even envision. who knows what kind of computational processes take place inside the sun, for example?

by person i mean something deserving of moral patienthood; though this is still hard for me to even start to determine. it probably requires some amount of information system complexity, as well as a single point of explicit consideration and decision, but apart from that i'm not quite sure.

i do know that pretty much all currently living humans count as moral patients. other than that, we should probably err on the safe side and consider things moral patients when in doubt.

### systems within systems

all top-level systems are, in the long term, permanent.

if you want society to "settle what it wants later", then your top-level permanent system is what they'll eventually settle on.

if you want society to never be stuck in any system and always have a way out, then the top-level permanent system is "going from system to system, being forever unable to settle" and you better hope that it spends more time in utopian systems than dystopian systems.

if your view is "whatever happens happens", then your top-level permanent system is whatever happens to happen. by not caring about what the future looks like, you don't make the future more free, you only are less likely to make sure it's one you'd find good.

if there is going to be a top-level system no matter what, no matter how flexible its internals are, we ought to care a lot about what that system is.

### enforcement: superintelligence

even if you don't think [superintelligence explosion](https://en.wikipedia.org/wiki/Technological_singularity#Intelligence_explosion) is [imminent](https://carado.moe/were-all-doomed.html), you should think it will happen eventually. given this, what may very well be [the "infinite majority" of remaining time](https://carado.moe/ai-alignment-wolfram-physics.html) will be one where a superintelligence is the ultimate decider of what happens; it is the top-level.

i find this reassuring: there *is* a way to have control over what the eternal top-level system is, and thus ensure we avoid possibilities such as unescapable dystopias.

### generalized alignment

in AI development, "alignment" refers to [the problem of ensuring that AI does what we *actually* want](https://en.wikipedia.org/wiki/AI_control_problem) ([rather than](https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml), for example, what we've explicitely instructed it to do, or just maximizing its reward signal).

when we think about how to organize future society, we actually care not just about the alignment of the top-level superintelligence, but also "societal alignment" before (and within) that. i will call "generalized alignment" the work of making sure future society will be in a state we think is good, whether that be by aligning the top-level superintelligence, or aligning the values of the population.

so, even if you don't think a superintelligence is particularly imminent, you should want society to start worrying about it sooner rather than later, given what you should consider being the amount of unknown variables surrounding the time and circumstances at which such an event will occur. you want to align society *now*, to your values as well as the value of figuring out superintelligence alignment, hopefully not too late.

### not just values

at this point, one might suggest directly loading values into superintelligence, and letting it implement whatever maximizes those values. while this may seem like a reasonable option, i would kind of like there to be hard guarantees. technically, from a utilitarian perspective, there exists a number N sufficiently large that, if N people really want someone to genuinely be tortured, it is utilitarianly preferable for that person to be tortured than not; my utopia instead proposes a set of hard guarantees for everyone, and *then, within the bounds of those guarantees*, lets people do what they want (including "i just want superintelligence to accomplish my values please").

one might consider the solution to that to be "just make it that people never want others to be tortured", but that's a degree of freedom on people's thoughts i'd rather keep if i can. i want persons to be as free as possible, including the freedom to want things that can't ethically (and thus, in my utopia, can't) be realized.

### a substrate for living

i am increasingly adopting [wolfram's computational perspective](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/) on the foundation of reality; beyond offering great possibilities such as [overcoming heat death](https://carado.moe/ai-alignment-wolfram-physics.html), i feel like it strongly supports the [informational view of persons](https://carado.moe/you-are-your-information-system.html) and the ability for people and socities to live on any form of computation framework; and those aren't particularly less or more real than our current, [standard model](https://en.wikipedia.org/wiki/Standard_Model)-supported reality.

given this, the most efficient (in terms of realizable value per unit of resource) way for superintelligence to run a world is to extract the essence of valuable computations (notably [the information systems of persons](https://carado.moe/you-are-your-information-system.html)) into a more controllable substrate in which phenomena such as aging, attachment to a single physical body, vulnerability to natural elements, or vulnerability to other persons, can be entirely avoided by persons who wish to avoid them. this extraction process is often referred to as "uploading", though that implies uploading into a nested world (such as computers running in this world); but if wolfram's perspective is correct, a superintelligence would probably be able to run this computation at a level parallel to or replacing standard model physics rather than on top of that layer.

this is not to say that people should all be pure orbs of thought floating in the void. even an existence such as a hunter-gatherer lifestyle can be extracted into superintelligence-supervised computation, allowing people to choose superintelligence-assisted lifestyles such as "hunter-gathering, except no brutal injury please, and also it'd be nice if there were unicorns around".

### universal voluntaryism

at this point, we come to the crux of this utopia, rather than its supporting foundation: ultimately, in this framework, the basis of the existence of persons would be for each of them to have a "computation garden" with room to run not just their own mind but also virtual environments. the amount of computational resource would be like a form of universal basic income: fixed per person, but amounts of it could be temporarily shared or transferred.

note that if resources are potentially infinite over time, as wolfram's perspective suggests, then there is no limit to the amount of raw computation someone can use: if they need more and it's not available, superintelligence can just put either their garden or *everyone's gardens* on pause until that amount of computation resource becomes available, and then resume things. from the point of view of persons, that pause would be imperceptible, and in fact functionally just an "implementation detail" of this new reality.

persons would have the ability to transform their mind as they want (though having a bunch of warnings would probably be a reasonable default) and experience anything that their garden can run; *except for computing the minds of other persons*, even within their own mind: you wouldn't want to be at the mercy of someone just because you happen to be located within their mind.

persons would be able to consent to interact with others, and thus [have the ultimate say on what information reaches their mind](https://carado.moe/cultural-and-memetic-hygiene.html). they could consent to visit parts of each other's gardens, make a shared garden together, and all manner of other possibilities, so long as all parties consent to all interactions, as determined by superintelligence — and here we're talking about [explicit consent](https://carado.moe/defining-freedom.html), not inferred desires even though superintelligence would probably have the ability to perfectly determine those.

for a perspective on what a society of "uploaded" persons might look like, see for example [Diaspora by Greg Egan](https://en.wikipedia.org/wiki/Diaspora_%28novel%29).

### rationale and non-person forces

the goal of this structure is to allow people to live and associate with each other in the most free way possible, making the least possible restrictions on lifestyle, while retaining some strong guarantees about consent requirements.

in a previous post i talk about [non-person forces](https://carado.moe/two-principles-for-topia.html); those being for example social structures that act with an agenthood of their own, running on other people as their own substrate.

at the moment, i simply don't know how to address this issue.

the problem with the "dismantlement" of such forces is that, if every person is consenting to the process, it's hard to justify superintelligence coming in and intervening. on the other hand, it does feel like not doing anything about them, short of being able to align sufficiently many people *forever*, will tend to make people dominated by such structures, as a simple process of natural selection: if there is room for such structures and they can at least slightly causate their own growth or reproduction, then they will tend to exist more than not. this may be thought of as [moloch](https://slatestarcodex.com/2014/07/30/meditations-on-moloch/) "attacking from above".

one major such potential non-person force is superintelligence itself trying to make people tend to want to live in ways that are easier to satisfy. if everyone wants to sit in their garden forever and do nothing computationally costly, that makes superintelligence's job a lot "easier" than if they wanted to, for example, communicate a lot with each other and live computationally expensive to run lifestyles; and the reason superintelligence will want to make its job easier is to increase the probability that it succeeds at that job (which it *should* want).

if informationally insulating people from superintelligence except when they outright consent to it intervening in their decisions is *not* sufficient, then maybe we can add the rule that people can never ask superintelligence to intervene in their life unless there is one single optimal way to intervene, and hopefully *that's* enough. the idea there being: if, for any request to superintelligence, there is only a single optimal way to accomplish that request, then superintelligence has no degree of freedom to influence people and thus what they want.

### on new persons

there are some reasons to be worried about the creation of new persons.

one is [malthusian traps](https://slatestarcodex.com/2014/07/30/meditations-on-moloch/): if the amount of resources is either finite or growing but bounded, or if it's unknown whether the amount of resources will end up being finite or not, then you *have* to cap population growth to at most the speed at which the amount of resource grows (if the amount of resources grows, the maximum speed of population growth should preferably be lower, so that the amount of resource each person has can grow as well). while it does seem like in current society people tend to have less kids when they have a higher quality of life, in a system where persons can live forever and modify their minds, one can't make such a guarantee over potentially infinite time.

another is replicator cultures: if there is no limit on creating new persons, and if people can influence even just a bit the values of new persons they create, then soon the world is overrun by people whose values are to create kids. or: making a world in which "new person slots" are filled by whoever wants to fill them first, will just select for people who want to fill those slots the most.

there might also be weird effects such as, even if resources were infinite, allowing arbitrary amounts of persons to be created could "stretch" the social network of consenting-to-interact-with-each-other persons such that, even if someone has registered an automatic consent to interact *even just a bit* with the kids of persons they already consent to interact with, they are soon flooded with a potentially exponentially growing network of kid interactions; though this probably can be addressed by that person by revoking this automatic consent.

beyond various resource and network effects, new persons create an ethical dilemma: does a person consent to living? or, for a child, do they consent to being taken care of for some amount of years after they are born — a time during which we often consider them to require affecting them in ways they might be unable to consent to?

if such philosophical quandries don't have a solution, then the safest route is to simply forbid the haphazard creation of new persons, whether that be through conventional human infants, [headmates](https://en.wikipedia.org/wiki/Multiplicity_%28psychology%29) and [tulpas](https://en.wikipedia.org/wiki/Tulpa#21st_century) if those are "real" enough to count as persons, and potentially other ways of creating new persons that can't consent to future interactions because they don't exist yet.

on the other hand, one way to increase the population *with* consent, is simply to "fork" existing persons: create a duplicate of them. because both are a continuation of the original single person, the original person's consent counts for both resulting persons, and there is no issue. the "merging" of consenting persons together might be possible *if* it can be reasonably estimated that their shared consent "carries through" to the new, merged person; i am currently undecided about how to even determine this.

finally, if resources are finite, creating a new person (whatever the means) should require permanently transferring one "universal basic computation amount"'s worth of computation garden to them, as no person should start out without this guarantee. this could be done by a person consenting to die and give up their own computation garden, it could be done by several "parents" consenting to give up a ratio of their gardens to the new person, it could be done by reclamating the redistribution of persons who die and don't make any decisions about what should be done with their computation garden, etc.


2021-11-20

## no room above paperclips

(edit: see also [*yes room above paperclips?*](https://carado.moe/above-paperclips-2.html))

when presented with the idea of a [paperclip-maximizing unaligned superintelligence](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer), people sometimes mention the possibility that sure, the universe gets tiled with paperclips, but maybe there's [slack](https://thezvi.wordpress.com/2017/09/30/slack/) in how paperclips are arranged, and that maybe nice things can exist again "above" paperclips.

(note: this relates to the idea of ["daemon-free"ness in "minimal circuints"](https://www.lesswrong.com/posts/nyCHnY7T5PHPLjxmN/open-question-are-minimal-circuits-daemon-free))

i think it's a reasonable line of thinking, but it's short-sighted: let's think about what happens next. eventually, above those paperclips, some evolutionary process may take place, leading (possibly, such as in our case, through the step of a technological species) eventually to a superintelligence taking over everything. given that *the entire cosmos* gets tiled with paperclips [*possibly forever*](https://carado.moe/ai-alignment-wolfram-physics.html), and that a superintelligent singleton taking over everything is irreversible (short of everything dying forever), in all likelyhood in the long term in any piece of universe not already actively managed by a superintelligence, eventually either everything dies forever, or a superintelligence takes over everything forever.

and then what? either this new superintelligence cares about "upwards", and has some plan for how its own paperclips are arranged (such as into more "macro"-paperclips), or it doesn't and the cycle begins again.

given that the outcome of an "alien" superintelligence's takeover is probly a worse outcome than the takeover of a superintelligence of our own (we should expect them to be about as incompetent as us at alignment, but to have values [less aligned to ours](https://www.lesswrong.com/posts/HawFh7RvDM4RyoJ2d/three-worlds-collide-0-8)), we need to care about our own iteration first, it's our best bet.

the point is, eventually for any given local patch of spacetime, either a superintelligence explosion is reached or everything dies forever. this can't be avoided, even by "climbing up" on substrates, so we should care about alignment now; we can't just hope that things are okay despite paperclips.


2020-11-18

*(this post may contain some very vague spoileryness about the video game Outer Wilds)*

## A Prototypeness Hierarchy of Realities

one property of many video games that i felt the most when playing the excellent [Outer Wilds](https://store.steampowered.com/app/753640/Outer_Wilds/) was *prototypeyness*.

many games, and especially that one, feel like they are prototypes for reality to some extent; they try to extract some essence of what is interesting about this world, without having the ability to implement all of it in a fully dynamic way, and thus hardcoding the rest.

now, this aspect of prototypeyness is sufficiently present in Outer Wilds that i ended up asking myself the question: what would real life (this universe where earth is) be a prototype for ? and i think the answer is:

real life is a prototype for living in virtual realities/cyberspace.

once we upload ourselves to computers (a good thing!) we will be able to make the entirety of the substrate that individuals interact with way more flexible; inhabit spaces of any number of dimensions or maybe not even spaces at all and just graphs (as is the shape of the web), modify our minds in ways meat brains wouldn't support, basically utilize any type of computational constructs we want with no regard for most limitations, depending on reality only as a substrate to run the computronium for it all.

like the step between prototypey video games and reality, it is one of a nearly definitional boundary in scale of computing power, and one whose non-prototype side i'm very interested in.


2021-07-18

## AI alignment timeline codes

this is a small post proposing simple one-letter codes for identifying timelines depending on their status relative to [AI alignment](https://en.wikipedia.org/wiki/AI_control_problem) and the appearance of [superintelligence](https://en.wikipedia.org/wiki/Superintelligence):

* **P-line**: a pre-intelligence explosion and pre-figuring out AI alignment timeline. we are in a P-line.
* **X-line**: a timeline where an [existential risk (or X-risk)](https://en.wikipedia.org/wiki/X-risk) has been realized by an unaligned superintelligence. everything is dead, forever.
* **S-line**: a timeline where a [suffering risk (or S-risk)](https://en.wikipedia.org/wiki/S-risk) has been realized by an unaligned superintelligence; the universe from then on contains net suffering on immense scales for all remaining time, [which is possibly infinite](https://carado.moe/ai-alignment-wolfram-physics.html). we should want to avoid this pretty much at all costs (including by [opting for an X-line instead](https://carado.moe/when-in-doubt-kill-everyone.html)).
* **A-line**: AI alignment has been figured out, and no superintelligence has been deployed yet. from that point on, we have the means to reach a U-line; though this isn't guaranteed. this is where we want to get as soon as possible.
* **U-line**: an aligned or [somehow otherwise](https://www.lesswrong.com/tag/orthogonality-thesis) benevolent superintelligence has been deployed, and we are guaranteed a relatively utopian world forever. this is the ultimate goal. while not strictly necessary, going through an A-line is almost certainly required to get there.

U-line, X-line, and S-line all have deployed superintelligences and are therefore terminal outcomes; they are unescapable. P-line and A-line are transitionary; they likely lead to one of the three terminal outcomes mentioned here.

other terminal might exist, but they seem unlikely enough to not warrant listing here; for example, even if everyone dies from, say, a meteor impact, life on earth or nearby will probably evolve another civilization *eventually*, which will also probably face the AI alignment challenge and end up in one of the terminal timelines.


2022-04-30

## a unit for utils

as utilitarians, it would be convenient for us to have an actual unit to measure utility, a number to be computed and compared.

the usual pick is money, but some people could have different judgments of the world that lead them to have different instrumental valuings of money even when they have the same intrinsic values; and also some people could intrinsically value money.

the unit i propose, to measure how much an agent cares about a thing, is a ratio of that person's "total caring pie". for example, you could intrinsically value 70% something and 30% something else; and then i'm sure we can figure out some math that makes sense (probly inspired from probability theory) to derive our valuings of instrumental values from that.

this seems like the least biased way to measure utils. the only criticism i can think of is that it breaks if two agents have different amounts of total valuing: perhaps one person *just has more total caring* than another.

however, is this testable in any way? is there any situation where one agent would act differently than another if they have the same intrinsic valuing proportions but one of them has a million times more total caring? i don't think so: the idea that inaction counts, seems to me to track either willpower or just different valuings of not doing effort.


2022-04-13

## finding earth in the universal program

this post expands on step one of [*the Peerless*](https://carado.moe/the-peerless.html): creating virtual people. brain scans-and-simulations are apparently still quite far off, so i'll be focusing on the second approach: resimulating earth and plucking out persons.

(one great side-advantage of this method is that, if we can relocate earth to pluck out persons for the simulation of alignment researchers, then we can later also relocate earth in order to restore it once we've solved alignment. so resimulating and locating earth, regardless of having early enough mind-plucking-out tech, is something we might need to do anyways.)

if compute is [infinite](https://carado.moe/ai-alignment-wolfram-physics.html) and [we don't mind being inefficient](https://carado.moe/udassa-time-steps.html), then we can use exponential or even infinite compute to locate earth. one approach is the following: create a big informational beacon — perhaps a copy of a huge portion of the internet, along with MRI scans of as many people as we can afford. then, we use some type of (non-intelligent) deterministic error-bound statistical location procedure to locate patterns that look like that beacon inside the [universal program](https://carado.moe/universal-complete.html). we can afford the statistical detection to be imperfect — if it misses on one encoding of earth, there will be different ones in the universal program.

because of the time penalty of the universal program, however, we may find just compressed copies of the beacon (instead of a full simulation of earth leading to the time at which we build that beacon), and because of the deterministic bound, we want need to stop on the first match; if this first match is *just* the beacon, without earth, then we fail; perhaps superintelligence can notice that it's not finding any nearby minds to pluck out, or perhaps it plucks out garbage. so we can start the universal program with not one step per program, but rather a very large number of steps — i hear stephen wolfram has estimates on the number of computation steps it takes to get to the current state of the universe. this will favor programs that takes every long to lead to the beacon, but are themselves shorter program.

(what if the first program to contain earth is itself a universal program *without* that huge constant, such that *it* finds the beacon before it finds earth? i am not sure how to address this. perhaps we can explore programs in an order that favors worlds that look like our physics instead of looking like discrete iterations of all computations?)

there's also the concern that the universal program, just like the [universal distribution](https://handsandcities.com/2021/10/29/on-the-universal-distribution/), [is malign](https://www.lesswrong.com/posts/Tr7tAyt5zZpdTwTQK/the-solomonoff-prior-is-malign). i'd think plain top-level earth, maybe especially as detectable by a simple enough beacon locator, would tend to occur before malign aliens emitting our beacon to trick us; but that's a risk to keep in mind.

if we *do* care about computational efficiency, then there are two main factors we need to account for:

* can our universe can be ran in polynomial time on whatever computers the superintelligence can build? for example, can it be ran in polynomial time on quantum computers, and can quantum computers be built? note that if this is the case we might need to step through *quantum steps* of *quantum programs* to run the search in the expected time. this doesn't need we need to build quantum computers outselves, mind you — superintelligence can just notice that a quantum computer would run the computations we describe efficiently, and build and use those.
* is the "seed" program for the universe small? intuitively i believe it is, and i find wolfram's efforts to reproduce the behavior of particles from the standard model using simple graph rewriting, to be evidence in that direction. that said, if it is large, then finding that program is an exponential search again — and so, again, we might need to build a search that "favors" our physics to save on exponential search time.

finally, we might want to put a hard bound on the number of tries the superintelligence will run to locate earth. the reason for that is that, if for some reason we messed up something in the beacon locator and it *never, ever* finds earth, then it will instantiate all computations, which appears to me to be a potential [S-risk](https://carado.moe/timeline-codes.html). in fact, even if we do find earth, it may not be worth it if we have to simulate exponentially much potential suffering before running our utopia — what if, after solving alignment, we have a great time, but then decide to eventually fade away after only polynomial time? then we will might have created exponentially much suffering in total.

### intermediary simulation

in case isolating minds from this simulation is hard, we could build an intermediary step between the location of earth in simulation-space, and booting the peerless simulation proper — superintelligence could, once it has located our beacon, get in touch with our organization *inside the simulation of earth*, and give it extraordinary computational (and maybe physical?) ability within the simulation to either take over everything, or figure out brain plucking-out and then let us press a big "ok, start now" button.

note, however, that we might not want to remain in this intermediary simulation for too long — it is still vulnerable to inner unaligned superintelligences, just like our top level reality is. we want to get to a safe, sandboxed, computationally weak environment as early as possible.

this is also a great argument for readying ourselves to build the beacon and utilize this contact-from-superintelligence as early as we can; indeed, to make that the first step of implementing the peerless plan. the reason for that is that the earlier we are able to take advantage of it, the earlier the time step of the simulation superintelligence can help us start bootstrapping towards the proper simulation of the peerless, and the less likely we are to be doomed by other superintelligences, if we need some intermediary "pre-peerless" simulation time.


2021-06-30

## we're all doomed

[a major tech company is now explicitly invested in getting AI to write code](https://copilot.github.com/).

this is a major warning sign; a first step on the explicit path to [superintelligence](https://en.wikipedia.org/wiki/Superintelligence) [explosion](https://en.wikipedia.org/wiki/Technological_singularity#Intelligence_explosion), an event [already considered relatively likely](https://intelligence.org/faq/#imminent) and, [in the absence of sufficient AI alignment progress](https://intelligence.org/2018/10/03/rocket-alignment/), is overwhelmingly likely to [permanently end all life at least in the observable universe](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer).

the time scale probably lies somewhere between a few years and a few decades, but in any case it's becoming to seem increasingly unlikely that [the only organization trying to actually figure out AI alignment](https://intelligence.org/) is gonna accomplish that in time.

if you can, go and [help them out](https://intelligence.org/get-involved/), or at least [donate everything you can to them](https://intelligence.org/donate/).

if you're currently working in AI development in any way, *please stop*. whether anything on earth survives this century is gonna be a matter of whether AI alignment is figured out by the time we get enough AI development; by helping the latter, you're making it even more likely that it happens before the former.

on a gloomier note, if you have all the philosophical beliefs required to think it can work, you may want to start preparing to [abandon this timeline](https://carado.moe/quantum-suicide.html) if singularity starts happening and looks like it's not gonna go well.

edit: see also: [are we in an AI overhang?](https://www.lesswrong.com/posts/N6vZEnCn6A95Xn39p/are-we-in-an-ai-overhang)


2022-02-03

## hackable multiverse

in [a previous post](https://carado.moe/brittle-physics.html) i talk about how hackable physics might allow a superintelligence to take over very quickly (perhaps faster than celerity).

in *[psi rewriting](https://carado.moe/psi-rewriting.html)* i propose that multiversehood can be more cleanly described as a particularly implemented feature of the cosmos, rather than an intrinsic thing.

but, if the cohabitation of multiple timelines is indeed an implemented feature rather than a primitive one, then there is a possibility that it is hackable, and that a superintelligence could hack across timelines.

now, it is to be noted that even if hackability exists, it might still be limited: perhaps there something like a light cone at play, or perhaps a given timeline can only access a finite number of other timelines.

it is to be remembered that timelines are not slots, they're not variables that hold values; timelines are *the values themselves*. still, hackability could mean some branches of the causality graph stop getting computed, for example.

either way, under these conditions, even quantum immortality might not save us from an X-risk superintelligence, and [given](https://deepmind.com/blog/article/Competitive-programming-with-AlphaCode) [recent](https://openai.com/blog/formal-math/) [developments](https://blog.eleuther.ai/announcing-20b/), we should panic a lot.


2022-04-10

## bracing for the alignment tunnel

it looks like we're gonna invent AI that [kills everyone](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer) before we figure out [AI alignment](https://en.wikipedia.org/wiki/AI_alignment).

what this means is that soon, [if not already](https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html), we are going to start bleeding [timelines](https://en.wikipedia.org/wiki/Many-worlds_interpretation), **hard**; by which i mean, an increasing ratio of multiverse-instants are gonna become dominated by unaligned AIs — and thus be devoid of population ([probably](https://carado.moe/above-paperclips-2.html)).

after that, there is a period in the (ever-diminishing amount of) surviving timelines, where we [ride on quantum immortality](https://en.wikipedia.org/wiki/Quantum_suicide_and_immortality) to solve alignment; after which, we finally reach [U-lines](https://carado.moe/timeline-codes.html), hopefully.

by many theories of [anthropics](https://handsandcities.com/2021/09/30/sia-ssa-part-1-learning-from-the-fact-that-you-exist/), observing existing either before or after that period is a lot more likely than observing existing in it. before the period, it is more likely because there are a lot more populated timelines in which to exist; after the period, it is more likely because we can hopefully "repopulate horizontally" by allowing the population to increase again.

if i am correct in the reasoning in this post, then being someone who exists in this very narrow alignment "tunnel" is exceedingly unlikely (barring weird circumstances such as post-singularity mankind choosing to simulate many variants of the tunnel for some reason). indeed, if you do observe being in it, you should think that something weird is going on, and update against the narrative presented in this post.

yet, *again if i am correct*, this is a period where we need to hold tight and work on alignment, perhaps as quickly as possible in order to reduce astronomical waste. very few us's inhabit the tunnel, but those very few us's are the critical ones who we need to care about.

so we need to brace our minds for the alignment tunnel. we need to commit to be persons who, if we observe being in the tunnel, will keep working on alignment even if, *from inside those timelines*, it looks like the reasoning i'm presenting here can't possibly be right. this is perhaps a weird case of instrumental rationality.

(note that i'm not saying the conclusion of observing being in those timelines should be to stop working on alignment; perhaps we would want to work on it either way, in which case we don't have to worry about anything. but i worry that it could lead us to other places such as "oh, maybe this AI killing everyone business isn't real after all, or maybe a weird alien force is preventing us from dying somehow")


2022-05-12

## AI risk plans

people have criticized my [*peerless*](https://carado.moe/the-peerless.html) plan on the grounds that it's too long-term/far-fetched.

while i don't disagree, i think that it is only one variable to be taken into consideration. here is a comparison of plans for addressing AI risk, with vague estimates.

|plan|achievable before [X-line](https://carado.moe/timeline-codes.html)¹|chance of [U-line](https://carado.moe/timeline-codes.html)²|[S-risk](https://en.wikipedia.org/wiki/S-risk)²|
|---|---|---|---|
|doing nothing|100%|<[1e-6](https://www.lesswrong.com/tag/orthogonality-thesis)|<1e-6|
|direct alignment|[.1%](https://www.readthesequences.com/Value-Is-Fragile)|5% → .005%|[5%](https://reducing-suffering.org/near-miss/) → .005%|
|[the peerless](https://carado.moe/the-peerless.html)|2%|10% → .2%|1% → 0.02%|

* ¹: assuming significant effort is put behind the plan in question, what is the likelyhood that we'll have accomplished the work to what we *believe* to be completion? note that my current AI timelines are pretty pessimistic (we become more likely to die than not this decade)
* ²: *if* we believe to have completed the work; the latter number is adjusted by being multiplied with "achievable before [X-line](https://carado.moe/timeline-codes.html)".

note that the numbers i put here are only very vague estimates, feel free to replace them with your own guesses. but my point is, in order for the peerless to be the plan we should be working on, we don't need it to be *feasible*, we just need it to be *less infeasible than all the other plans*. i think the peerless is more tractable than doing direct alignment, and only more risky because it has more chances to succeed. depending on how scared of S-lines you are, you should push for either doing nothing (and thus [oppose direct alignment](https://carado.moe/against-ai-alignment.html)) or for my plan. (or come up with your own, and then compare it to these!)

not pictured: the plan to [melt all GPUs](https://www.lesswrong.com/s/n945eovrA3oDueqtq/p/7im8at9PmhbT4JHsW), because it's just a modifier on what we do afterwards. but yes, melting all GPUs is a great idea if we think we can reasonably do it more than other plans.


2021-10-14

## cosmic missing outs

this might be a complete waste of brainflops, but sometimes i wonder about "cosmic missing outs".

my typical example for those is the culture of modern japan.

imagine timelines where japan never became the country it did, and we never got its culture. that'd be a huge thing to miss out on, right? the second best thing might be korean culture or something like that.

but, now that you've imagined this timeline that is missing out on modern japan culture, imagine the opposite: there are timelines out there that have those great cultures of countries that we're missing out on, that us missing out on is kind of on the same scale as those other timelines missing out on japan's culture.

i'm talking about this because i just thought of some other things kind of of this type:

what are some unknown things that we are missing out on, that us missing out on is kind of like if other timelines were missing out on music?

what are some unknown things that we are missing out on, that us missing out on is kind of like if other timelines were missing out on philosophy, science, or math?

these speculations are the closest i can get to putting human minds into perspective and considering the existence of things entirely outside of human conception, the way many things are entirely outside of a mouse or ant's ability to conceive.

to be clear: i still can't have that consideration, this is only the closest i get, but it's not quite there.


2021-08-23

## right to death, therefore

because i like [freedom](https://carado.moe/defining-freedom.html) so much, i think people should be generally able to do what they want. but this immediately raises a conundrum: should someone be able to do an action that hampers their *future* freedom?

one relatively extreme case is the ability to commit suicide: it's about as committal as you can get, in terms of actions with future ramifications to oneself. if you choose to get in debt or cut off a limb, that can be pretty hard to get out of, but it still seems less impactful and less inescapable than suicide.

so, should suicide be allowed? (i am of course only talking about reasonable, clear-minded suicide, *informedly* consented; not coerced suicide, nor suicide out of compromised ability to make decisions)

in my opinion, *obviously yes*. the alternative, that people be forced to live until their biology kills them (which we may very well find ways to prevent indefinitely), seems abhorrent to me. given this guarantee, then it makes sense to me that any lesser commitments should also be fine.

there are some criticisms one can make about this argument. bad but non-death commitments could tend to increase the amount of suffering people in society at any given moment; and, if people change over time (as they tend to do), then commitments can ramificate into a future person who is sufficiently different from the person making the commitment that it might be considered unreasonable for them to be subject to some excessive amounts of "locally" unconsented negative effects. a cap on the time duration of commitments, and/or the requirement for people to guarantee that they remain the same "enough" over time until the commitment is expired (a technology we currently don't have, but will become easier to make once we're uploaded and we understand the human mind better), might be reasonable patches for these issues.


2021-07-10

(edit 2021-07-18: this post is probly not very good, as there's some anthropic principle research out there and i haven't read any and just gone off thinking about it on my own.)

## estimating the amount of populated intelligence explosion timelines

the [imminent](https://carado.moe/were-all-doomed.html) [intelligence explosion](https://en.wikipedia.org/wiki/Technological_singularity#Intelligence_explosion) is likely to [go wrong](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer).

how likely?

if you imagine that you live pretty much at the cusp of such an event, you should expect as per the [anthropic principle](https://en.wikipedia.org/wiki/Anthropic_principle) that there are about as many observer-instants before you, as there are after you. (an observer-instant being an instant at which you have a chance of making observations about that fact; see [this](https://www.greaterwrong.com/posts/uSMa6Fj5nMgntpxfo/are-coincidences-clues-about-missed-disasters-it-depends-on) and notably Nick Bostrom's Self-Sampling Assumption)

i've previously calculated that the future from now until heat death has room for roughly 10^200 human lifespans (of 80 years) (an estimation based on the number of particles in the observable universe,  the amount of time until heat death, and the computational cost of running a human brain).

the past, on the other hand, holds about 10^11 human lifespans (most of them not full 80-year lifespans, but such details will get amortized by using orders of magnitude).

if intelligence explosion is, as i believe, likely to result either in [total death](https://carado.moe/were-all-doomed.html) or in well-populated futures (whether good or [bad](https://en.wikipedia.org/wiki/Suffering_risks)), then the fact that i'm observing being right next to the event (in time) rather than observing being one of the (in well-populated timelines) countless observers to exist *after* the event, must be compensated by such well-populated timelines being particularly rare within the set of future possible timelines.

how rare? about 1 in (10^200 / 10^11), which is 1 in 10^189.

factors which may make this calculation wrong:

* my 10^200 estimate might be wrong (for example: if each person comes to eat a *lot* of computation resources, then the number of future observers is drastically reduced).
* the 10^11 estimate for the past might be wrong: what if there have beings in earth's past smart enough to make this observation? it may seem unlikely, but if i am to encompass the immense amount of forms future observerc might take, i should account for a wide variety of forms of past observers too.
* because entropy increases, there are (possibly a lot) more future universe states than past universe states. accounting these "timeline splits" for the number of future observers even more massively decreases the expected ratio of well-populated timeline-states, though i'm not sure by how much.


2021-12-09

## non-interfering superintelligence and remaining philosophical progress: a deterministic utopia

[in a previous post](https://carado.moe/against-ai-alignment.html) i talk about the need to accomplish philosophical progress at determining what we value, before alignment. i wouldn't be the first to think of "what if we boot superintelligence now, and decide later?" as an alternative: it would indeed be nice to have this possibility, especially given the seeming imminentness of superintelligence.

alas, typically, making this proposition goes like this:

* A: "we should boot superintelligence now, and make it that we can adjust it later when we figure out more of philosophy."
* B: "yeah, but superintelligence isn't gonna just wait: it's gonna want to try to make us figure out whichever philosophy would make its job simpler, such as actually we value all dying immediately so that there's nothing to protect"
* A: "well, in that case, we need to make sure superintelligence can't interfere with our decision process."
* B: "and *how* do you ensure that the new being running all things in the world, has no interference into human affairs, exactly?"

which is a pretty good point, and usually a reasonably A concedes at that point.

today, however, i am here to offer a continuation to this conversation, from A's side.

my idea is to implement a deterministic computational utopia for people to be uploaded in, whose internals are disconnected from the outside world, such as [∀V](https://carado.moe/∀V.html); if we have infinite compute, then it can be even more free from outside interference.

the trick is to have that utopia's principles be *deontological*, or at least to make them absolute rather than able to be weighed against decisions outside of it: as it largely is in ∀V, ensure everything about utopia has a definite okay or not-okay status, evaluable without knowing anything about the "outside" of this utopia. either someone's consent is being violated, or it's not. with a set of decisions based only on the state of the utopia being simulated, every decision of the superintelligence about what it does in ∀V is unique: all superintelligence is doing is calculating the next step of this deterministic computation, including ethical principles, and thus there is nothing superintelligence can do to bias that decision in a way that is helpful to it. all it can do is run the computation and wait to see what it is that persons inside of it will decide to reprogram it to value or do; on the outside/before the singularity, all we need to ensure is that superintelligence does indeed eventually run this computation and apply the changes we decide on once it finds them out.

under these conditions, a device could be set up for us to later reprogram superintelligence somehow when/if we ever figure out what values we *actually* want, and it wouldn't be able to meaningfully interfere with our decision process, because every decision it takes regarding how our utopia is ran is fully deterministic.

not that i think being able to reprogram a superintelligence after boot is necessarily a good idea, but at least, i think it can be a possibility.


2021-07-21

## systems and diversity

as i've said in [a previous post](https://carado.moe/lets-not-generalize-politics.html): i really like culture; and, to that end, i like diversity (by which i mean people being more weird and different from one another).

there are many systems that exist today that affect diversity. most of them punish it; not as a coincidence, but because diversity is [a fragile value](https://www.readthesequences.com/Value-Is-Fragile): if you optimize for something else, it will tend to get optimized out.

if you optimize for economic efficiency, diversity gets optimized out because the most easily served economy is one in which demand is relatively uniform.

in general, if you optimize for people having their values satisfied, diversity gets optimized out because the most easily satisfied set of values is relatively uniform and easy to satisfy values; if you tell a superintelligence to "make a world where everyone has their values satisfied", the simplest way to achieve that (other than killing everyone) is to make sure everyone has very simple values like doing nothing all day or dying as soon as possible.

the scary thing about such an optimization is that it "works": at no point does an economy headed towards uniformity need to collapse; on the contrary, the more it has optimized out diversity, the more efficient and stable it'll be! so, we need to *[near-intrinsically](https://en.wikipedia.org/wiki/Instrumental_and_intrinsic_value)* care about preserving diversity, even when all else seems fine. this makes diversity preservation probably my largest concern with capitalism; at least, a system that wouldn't care about efficiency, wouldn't necessarily be aligned against diversity (though it might be aligned against it for other reasons).

social pressures such as [generalizations and expectations](https://carado.moe/lets-not-generalize-politics.html) punish diversity by rewarding conformity.

democracy and general consensus enforcment systems punish diversity by generally letting majority lifestyles be better supported by society than minority lifestyles.

i do know of one force of human nature which encourages diversity: [fetishism](https://en.wikipedia.org/wiki/Sexual_fetishism#Definitions). fetishism tends to make people prefer things specifically because they go against the norm. as such, i propose that if we value rich culture, we should want to cultivate fetishism.

the takeaway is: in any long-term societal plan, we need to care not just about values being satisfied, but about what values people have to begin with. a clear example in modern capitalism is advertising: it's okay that companies are aligned to satisfy values, but [with advertising they get to affect what values people have to begin with](https://carado.moe/unfair-feedback-loops.html).

(one could argue we could encourage people to [crystallize](https://carado.moe/value-crystallization.html) and [conserve](https://en.wikipedia.org/wiki/Instrumental_convergence#Goal-content_integrity) their values, as well as forbid the creation of new persons; but [i'd rather that not be required](https://carado.moe/rationalist-by-necessity.html))


2022-05-16

## smaller X-risk

a superintelligence killing us all is a *superintelligent, very large* [X-risk](https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence).

the superintelligence will tile its values in all directions; not just through space at celerity [or faster](https://en.wikipedia.org/wiki/Alcubierre_drive), but also, if it can, by [hacking physics](https://carado.moe/brittle-physics.html) and traversing across, for example, worldlines of the quantum many-worlds.

we may be able to create smaller X-risks, that only make us extinct in this timeline, on this earth. there are a few reasons we may want to do this:

* other timelines might have a better shot than us, and us booting a superintelligence may reduce their chances through weird stuff like intertimeline hacking
* to [avoid S-risks](https://carado.moe/when-in-doubt-kill-everyone.html), including S-risks that may be involved in instrumental cosmic-scale X-risk (maybe superintelligence wants to simulate civilizations in various ways for [acausal trade](https://www.lesswrong.com/tag/acausal-trade) or [other acausal weirdness](https://www.lesswrong.com/posts/PcfHSSAMNFMgdqFyB/can-you-control-the-past) reasons?)
* the next intelligent species on earth is more likely than us to solve alignment before superintelligence, and seems likely enough to be at least a little bit aligned with us (better than cosmic X-risk, at least)
* same as above, but for nearby aliens (whether current or future)

*smaller X-risk*, where we limit damage to just our civilization, seems harder than tiling the cosmos with paperclips; but at least it might be easier than [other plans](https://carado.moe/ai-risk-plans.html).

in a similar way, reducing our civilization to ashes *without* actually becoming extinct might also be a way to get another shot, if we think we're likely to do less badly next time.

remember: this is bigger than all of us. when the fate of the cosmos is at play, we can't afford to be too selfish.


2021-06-16

## my answer to the fermi paradox

the [fermi paradox](https://en.wikipedia.org/wiki/Fermi_paradox) asks, if aliens are supposedly so statistically prevalent, why we haven't received any radio signals from them.

here is my independently-developed (though probly not original) answer:

statistically, it seems reasonable that civilizations would [accidentally invent unaligned superintelligence](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer) not long after inventing radio signals (in our case, a couple centuries). in order to percieve those signals, you would need to exist *after* your planet receives those signals, but *before* your planet receives that unaligned superintelligence's [expanding sphere of death](https://carado.moe/moral-cost-of-unaligned-ai.html), which might very well travel at celerity or near-celerity.

thus, given the low probability, it is not surprising that we haven't percieved those; for any given alien civilization, in a given timeline, we either haven't received their radio signals, or have already been killed by them. seeing as we're alive, this timeline must be one of the former.


2022-03-21

## are there finitely many moral patients?

wouldn't it be neat if we didn't have to worry about [infinite ethics](https://handsandcities.com/2022/01/30/on-infinite-ethics/) ?

i think it is plausible that there are finitely many moral patients.

the first step is to [deduplicate moral patients by computational equivalence](https://carado.moe/deduplication-ethics.html). this merges not only humans and other creatures we usually care about, but also probably a lot of [other potential sources of moral concerns](https://reducing-suffering.org/what-are-suffering-subroutines/).

then, i think we can restrict ourselves to patients in worlds that are discrete (like ours); even if there *were* moral patients in non-discrete worlds, it seems to me that from where we are, we could only access discrete stuff. so whether by inherent limitation, plain assumption, or just limiting the scope of this post, i'll only be talking about discrete agents (agents in discrete worlds).

once we have those limitations (deduplication and discreteness) in place, there are finitely many moral patients of any given size; the only way for an infinite variety of moral patients — or more precisely, moral patient moments — to come about is for some moral patients to grow in size forever. while infinite time seems plausible [even in this world](https://carado.moe/ai-alignment-wolfram-physics.html), it is not clear to me that whatever the hell a "moral patient" is can be arbitrarily complex; perhaps at a certain size, i start only caring about a *subset* of the information system that a "person" would consist of, a ["sub-patient"](https://carado.moe/deduplication-ethics.html).


2021-07-19

2022-05-09 edit: i have found out that this idea is more thoroughly explored [here](https://reducing-suffering.org/near-miss/).

## botched alignment and alignment awareness

[AI alignment](https://en.wikipedia.org/wiki/AI_control_problem) is [hard](https://intelligence.org/2018/10/03/rocket-alignment/).

an AI developer who doesn't know about the problem of alignment to general human values might accidentally develop a superintelligence which optimizes for something largely unrelated to humans, leading us to an [X-line](https://carado.moe/timeline-codes.html); on the other hand, if they make a botched attempt at alignment to human values, it seems like there's more of a chance (compared to if they don't try) at booting a superintelligence which cares about enough aspects of human existence to tile the universe with some form of humans, but not enough to make those humans' lives actually worth living (goals such as "humans must not die"), resulting in S-lines.

considering this, raising awareness of AI alignment issues may be a very bad idea: it might be much better to let everyone develop not-human-caring-at-all AI and cause X-lines rather than risk them making imperfect attempts resulting in S-lines. or: we shouldn't try to *implement* alignment to human values until we *really* know what we're doing.

contrary to a [previous post of mine](https://carado.moe/were-all-doomed.html), this is a relatively hopeful position: no matter how many timelines end in X-risk, inhabited P-lines can continue to exist and research alignment, hopefully without too many S-lines being created. on the other hand, while it increases the chance of the [singularity](https://en.wikipedia.org/wiki/Technological_singularity#Intelligence_explosion) turning out good by leaving us more time to figure out alignment, it also means that it might take longer than i'd've otherwise expected.


2022-04-12 ★

## The Peerless

In this post, I propose a plan for addressing superintelligence-based risks.

Before I say anything, I will mention a crucial point that a *bunch* of people have ignored despite it being addressed at the bottom of this post: the idea I describe here is very unlikely to work. I'm proposing it because of other plans because I feel like other plans are *extremely* unlikely to work (see also [this post](https://carado.moe/ai-risk-plans.html)). Yes, we probably can't do this in time. That doesn't make it not our best shot. Rationalists select the best plan, not ["the first plan, and then after that a new plan only if it seems good enough"](https://www.readthesequences.com/If-Many-Worlds-Had-Come-First).

(spoilers for the premise of [*orthogonal* by Greg Egan](https://en.wikipedia.org/wiki/Orthogonal_%28series%29)) In Orthogonal, a civilization facing annihilation comes up with a last-minute plan: to create a ship, accelerate it until its time arrow is orthogonal to the time arrow of its home world (which is possible thanks to the alternate physics of their world), and thus give its crew as much time as it needs to figure out how to save their homeworld before reversing course and coming back. This plan is inspired by that, and i'm naming this post after their ship, the *Peerless*.

The short version is: we design a simulation for a bunch of people (probably rationalists) to live in and figure out alignment with as much time as they need, and create a superintelligence whose sole goal is to run that simulation and implement a new goal it will eventually decide on. I've written about this idea previously, but [that post](https://carado.moe/upload-for-alignment.html) is not required reading; this is a more fleshed-out view.

I will be describing the plan in three steps.

### 1. Create virtual people

We need virtual persons inside this world. They will be the ones who figure out alignment. A few possibilities come to my mind; there may be more.

* Brain scans, or full person scans. This is the most obvious solution. I'm not too familiar with the state of that field, but surely there's some work in that direction we can take advantage of; otherwise, we can just throw money at our own initiatives. This option does have the downside that it's quite likely brains aren't sufficient to keep someone functional — we may need to scan or re-implement a bunch more.
* Resimulate earth and pluck out persons. If there's a clever way to locate ourselves in the [universal distribution](https://www.lesswrong.com/posts/XiWKmFkpGbDTcsSu4/on-the-universal-distribution) (or a [computable variant of it](http://www.scholarpedia.org/article/Universal_search)), then we can just make a program that reruns that earth up to say, now, and then locates some or all human brains, and "download" them out of that simulation of earth and into our own simulated environment. For more details on this possibility, see [*finding earth in the universal program*](https://carado.moe/finding-earth-ud.html).
* Scan the earth and pluck our persons. This one seems harder than resimulating earth, but may be doable. It's certainly an idea worth throwing a few people at, to see if there's a clever way to make it work.

The main risk that's been brought to my attention regarding this part is the following: what if the virtual persons end up unaligned from their previous selves? The brain scan scenario seems like the most likely to have that risk, but even then i'm not *too* worried about it; intuitively, it seems unlikely enough to me that all the uploaded persons would come out misaligned in a similar direction, and in a similar direction that would lead them to decide on a [botched alignment](https://carado.moe/botched-alignment-and-awareness.html) for the superintelligence.

An obvious question here is: who gets to be on board the simulation? The values of the people who get uploaded might significantly affect what the superintelligence is aligned to (not all humans necessarily have the same values, [maybe even after thinking about it really hard for a long time](https://handsandcities.com/2021/06/21/on-the-limits-of-idealized-values/)). I don't have any answers other than the obvious "me please!" and "my tribe please!" that occur to me.

Note that i'm *not* proposing augmenting the uploaded minds — at least not for the first simulation iteration (see below). That *does* seem like an exceedingly risky prospect, alignment-wise, and one we don't need to commit to right away.

### 2. Design a virtual environment

Those persons will live in a virtual environment, within which they'll hopefully figure out alignment. However, the environment needs to be a deterministic computation, such that the "outer" superintelligence (the one running the virtual environment) has no ability to affect its outcome; its goal will only be to "implement whatever this computation decides". If the superintelligence wants to implement the actual result of the actual computation, and that computation is fully deterministic, (and if we don't simulate anything complex enough for that superintelligence to "leak back in"), then it has no room to meddle with what we do in it! It's stuck running us until we decide on something.

Some things we need to figure out include:

* How do we incorporate our virtual minds? I think we should go for something plugged in "ad-hoc" rather than embedded into the physics of that world, to preserve the integrity of those minds, which may live for very long times. In addition, in case virtual minds go crazy after living 200 years or something, we may want to allow them to reset themselves and/or die. A reset is not necessarily a big deal: hopefully previous-me can transmit enough information to future-me to continue the work. Maybe there are two me's at any given time, a teacher and an apprentice. Regular resets of individual persons also hopefully help maintain their values over long stretches of time. Many schemes are possible.
* What is this world like? We could make do with just something as basic as minecraft, but it would be better if the virtual persons don't have to go crazy from being stuck in a minecraft steve's body with no senses except sight and sound.
* How do we prevent "sub-singularities"? Given that this world is deterministic, there is nothing the outer superintelligence can do to prevent internal superintelligences from popping up and breaking everything it can. Potential solutions include things like "there are no computers" or "all computers inside this world are very slow and limited in capability".
* What about memetic safety? What about virtual violence? What if someone duplicates themself a billion times? And so on. There are a collection of design challenges, but designing [a peaceful world](https://carado.moe/∀V.html) with [sensible virtual physics](https://carado.moe/game.html) doesn't seem out of reach. They seem like tractable engineering challenges.
* What is the final voting procedure? Remember that the goal of the simulation is to give the people inside it time to figure out alignment, but they should probably agree on something eventually: either a final decision on alignment, or a "next iteration": a new simulation to be ran, which they think has better/safer/still-safe conditions within which to research alignment. In fact, there may be arbitrarily many such "simulation iterations". Anyways, the simulation will have a big red button inside of it which says "okay, we're done", and takes as input a new goal (and possibly decision theory?) that the outer superintelligence will have as its ultimate goal. But what should it take to press the button? Everyone to agree? A majority? What if we end up unable to come to an agreement? Again, there is work to be done on this, but it seems figure-out-able.

The people inside this simulation will have somewhere between *plenty* and [*infinite*](https://carado.moe/ai-alignment-wolfram-physics.html) time and compute to figure out alignment. If they do have infinite compute, and if the cosmos isn't full of [consequentialists competing for earlyness in the universal distribution](https://carado.moe/udassa-time-steps.html) (or other things that might make wasting compute bad), then we can even run exponential-or-longer computations in, from our perspective, instant time; we just need to be sure we don't run anything malign and unbounded — although the risks from running malign stuff might be mitigated by the computations being fully and provably sandboxable, and we can shut them down whenever we want as long as they don't get to output enough to convince us not to. After all, maybe there are some bits of information that are the result of very large malign-dominated computations, that can nevertheless still be of use to us.

I mentioned before that maybe only slow computers are available; running a "very large" computation might require a majority vote or something like it. Or we can just boot without any computers at all and spend the first few millenia designing slow computers that are actually safe, and then work from there — when we have all the time we want, and maybe-infinite *potential* compute, a lot of options open up.

One downside is that we will be "flying blind". The outer superintelligence will gleefully turn the cosmos into computronium to ensure it can run us, and *will* be genociding everything back meat-side, in our reachable universe — or beyond, if for example physics is hackable, as wolfram suggests might be possible. Superintelligence might even do that *first*, and *then* boot our simulation. Hopefully, if we want to, we can resimulate-and-recover aliens we've genocided after we've solved alignment, just like hopefully we can resimulate-and-recover the rest of earth; but from inside the simulation we won't be able to get much information at least in the first iteration. We *can*, however, end our iteration by agreeing on a new iteration that has some carefully-designed access to outside information, if we think we can safely do that; but nothing guarantees us that there will still be something salvageable outside.

Another way to model "successive simulation iterations, each deterministic, but each having the ability to make the next one not deterministic with a large enough vote" is as a single simulation that isn't quite deterministic, but made of large deterministic chunks separated by small controlled I/O accesses; think of it as a haskell computation that lazily evaluates everything right up until it waits for an input, and then as soon as it has that it can continue computing more.

Still, the current outlook is that we genocide everything *including ourselves*. Even if nothing else is recoverable, "a tiny human population survives and eventually repopulates" still seems like a better plan than the current expected outcome of "everything dies forever."

### 3. Make a superintelligence to run it

Now, this is the "easy part": just make a superintelligence that destroys everything to implement its one simple goal; except instead of paperclips, the simple goal is "implement whatever goal is the result of this very big turing machine".

We can either build and start that superintelligence as soon as we can, or [keep it ready](https://carado.moe/when-in-doubt-kill-everyone.html) while we stay on our regular world. I'd probably advocate for the former just to be safe, but it can depend on your beliefs about quantum immortality, S-risks, and such. In any case, having *something that might work ready to fire* is certainly better than the current *we just die*.

Of course, it is crucial that we make the superintelligence *after* we have designed and implemented the virtual environment, complete with its virtual persons (or its deterministic procedure to obtain them); we don't want it to be able to influence what goal we give it, so we likely need to have the goal ready and "plugged in" from the start.

Some risks are:

* It doesn't run the simulation accurately. I'd think it's surely not too hard to make a superintelligence have "run this discrete, deterministic thing and then adopt its output as goal" as its goal, but perhaps there are difficulties around this. I'm optimistic that we can figure that out.
* It doesn't run the simulation (or adopt its result as goal) at all. Perhaps the requirement that the simulation be ran perfectly will make superintelligence too paranoid about being sure it has run the simulation correctly, and it will spend its entire existence getting more computronium to increase the likelyhood that the outcome it has computed is correct, but it's never certain enough to actually adopt the new goal as its own. There may be probabilistic workarounds to this; we'll need to look into it more.
* It fails in the various ways AI usually value drift from us — hacking its reward function, etc. While this concern may indeed remain, the fact that i'm proposing what seems to me like a easy to formalize goal is already a big improvement compared to the current state of affairs.

### Conclusion

This is a plan with a bunch of things that need work, but it doesn't seem *absurdly* hard to me; if anything, step 1 seems like the hardest, and I don't even know that we've *tried* throwing billions of dollars at it.

I share yudkowsky's current [gloomy outlook](https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy) on AI. The current route of "hey, maybe we should study things vaguely related to harnessing what neural nets do, and hope to be able to grab a miracle should it come up" seems like a pretty bad plan. I think, in comparison, the plan I outline here has better chances.

It is to be remembered that my vision is competing not against the likelyhood of superintelligence emergence, but against the likelyhood that alignment works. If pursuing mostly alignment gives us a 1e-10 chance of survival, and pursuing mostly my plan gives us a 1e-8 chance of survival, then it doesn't matter that *yes, superintelligence is still overwhelmingly likely to kill us*; we should still favor my plan. See also: [this post](https://carado.moe/ai-risk-plans.html) comparing plans.

I have cross-posted this [on lesswrong](https://www.lesswrong.com/posts/PATFQm6hPN3Wycq4W/the-peerless); feel free to discuss the idea with me there.


2021-11-21 ★

## Genuineness, Existential Selfdetermination, Satisfaction: pick 2

imagine you have a world where one person wants the moon to be painted blue, and another wants the moon to be painted red.

they both mean the current actual physical moon as it exists now; they both refuse any "cheating" option such as duplicating the moon or duplicating reality, and they don't want their minds changed, nor to compromise.

there's three ways to resolve situations like this:

* you sacrifice **genuineness**: you somehow make both of them believe, mistakenly, that what they want is satisfied. maybe by unknowingly giving them eye implants that change what color they see the moon.
* you sacrifice **[existential selfdetermination](https://carado.moe/core-vals-exist-selfdet.html)**: you ensure the situation never happens to begin with, that no two persons will ever want the moon to be painted different colors; or maybe you brainwash one of them after the fact.
* or, you sacrifice **satisfaction**: you let them want what they want, and let them see what the moon looks like, such that at most only one of them will ever be satisfied.

you can't have all three of those things.

many hedonists will happily sacrifice **genuineness**; authoritarians like to sacrifice **existential selfdetermination**.

as for me, for [∀V](https://carado.moe/∀V.html), i ultimately sacrifice **satisfaction**: while people can choose to become mistaken about things, the default is that they get to access the actual true state of things.


2021-11-21

## the two-vtable problem

when programming, there are in general two types of interfaces: "static, known" interfaces and "dynamic, unknown" interfaces.

in the former, the possibilities are well known. maybe an object has a bunch of public methods that can be used, or maybe even public fields; maybe an API has a known call endpoints.

when the behavior or contents of an object are unknown or inaccessible, someone can still implement how it interacts with another known-interface object: just send the known object to the unknown object, and let the unknown object manipulate the known object however it wants.

however, there is no general way to make two (or more) objects interact with each other, when they both have a dynamic/unknown interface.

this is what i call the **two-vtable problem**.

one approach is to implement all n² behaviors: implement the behavior for any possible concrete type of one object and any possible concrete type of the other. the rust ecosystem is kind of like that with types and traits: if you have `N` types and `K` traits, unless one of those traits has a blanket implementation for all types, you'll need to write `N×K` implementations to have proper coverage in the general case.

but this is hardly scalable; and doesn't work well, for example, in environments in which objects are expected to be implemented by different parties that don't necessarily coordinate with one another, where those objects are then expected to work together without putting in extra effort afterwards. i'm sure this probly has been encountered a lot for example in video game modding communities, regarding the interaction between mods created by different people.

an answer can be taken from other fields that have already solved that problem on their own, however. i can think of two: how natural selection solved negotiation between dynamic persons, and how liberalism solved negotiation between dynamic private actors.

the general solution to the two-vtable is to have the two objects have a language —as tells us the evolution of humans— that they can use to communicate and negotiate an outcome. liberalism tells us that the shape of negotiated outcomes is contracts, and cryptocurrencies tell us that the formalized form of contracts is programs.

and so, here is my proposed solution to the two-vtable problem: when two dynamic objects want to interact with one another, that interaction must take the shape of the two of them building a program together, which will be executed once they both agree on it. perhaps this can take the shape of both of them sending an initial contract to the other which is ideal from the perspective of the sender, and from there try to incrementally build up programs that implement a compromise between the two ideals, until they meet somewhere in the middle; like haggling.

this framework generalizes nicely enough that it could be used for arbitrary informatic agents, such as a bot and an uploaded person, or two uploaded persons. in fact, contract negotiation of that kind, when understood enough by both agents partaking of it, can be the formalized form of consent, a matter i've [grappled with formalizing](https://carado.moe/defining-freedom.html) for [my utopia](https://carado.moe/∀V.html).

this could also be useful for negotiation between different [compilation stacks for portable programs](https://carado.moe/portable-programs.html), or even for the negotiation between [different wasms running on a liberal server market](https://carado.moe/saving-server-internet.html).


2021-08-25

## what happens when you die?

contrary to popular (secular) belief, i don't believe nothing happens.

considering that [you are your information system](https://carado.moe/you-are-your-information-system.html) and [nothing else](https://carado.moe/persistent-data-structures-consciousness.html), any future occurence of your information system *is you*. so, the meaning of the question "what happens when you die?" is really "what are some next things your information system will percieve after facing what should be fatal events in its original body?"

this is very likely [not nothing](https://carado.moe/quantum-suicide.html). somewhere, in some timeline,  your information system is probably being redundanced.

first, your body can miraculously avoid death. this would be a weird kind of immortality, where there is almost always a timeline where you somehow avoid death. it is, however, pretty unlikely to persist.

second, your mind could arise somewhere by accident. this could be as simple as random fluctuations in space producing something that runs your mind's information system by pure chance. this is *extremely* unlikely.

in fact, the most likely scenario is that someone in the far future reproduces your mind on purpose. for example, this could be a society in a [u-line](https://carado.moe/timeline-codes.html) being able to, and deciding, to run an accurate enough simulation of the entire earth up to some point, and downloading people from this simulation into their world, to allow them to avoid death. as it'd probly take a bunch of effort, and sounds like a pretty nice thing to do, i expect that to happen mostly in u-lines; however, there could be some [s-lines](https://carado.moe/timeline-codes.html) where this happens too. and while getting resurrected seems more likely in u-lines than s-lines, s-lines seem more likely than u-lines, and i don't know if the probabilities cancel out.

so, what happens when you die? you wake up either in heaven or hell, depending not on your personal actions in particular but in how likely it is we figure out AI alignment (a probability which you do have, if small, an impact on).


2022-03-02

## do not hold on to your believed intrinsic values — follow your heart!

i posit the following framework for thinking about [intrinsic values](https://en.wikipedia.org/wiki/Instrumental_and_intrinsic_value) (hereby just called "values").

somewhere out there, there is **your value system**.

it is a function (in the mathematical sense) that takes as input *things*, and spits out *feelings*. that function is where your *true* values lie; they *are* that function.

how is that function encoded in your brain? who knows! i don't have an answer, and there may [not be an answer](https://en.wikipedia.org/wiki/Computational_irreducibility).

in conscious thought, however, you don't have access to the source code of that function, whatever it is. the best we can do for the moment seems to be to try thinking about it real hard, throw various things at it and examine the output ([perhaps through some mildly systematic process](https://carado.moe/core-vals-exist-selfdet.html)), and define another function that tries to approximate what your actual values are. this is hard and takes a lot of work. perhaps someday we will have a device that can scan your brain and give you a better idea of what your value function looks like, but at the moment that is not the case.

so, you build up an idea of what your values might be. here is where i think a lot of people make a mistake: they choose to believe strongly that this guess *is* their actual set of values (even though [it likely isn't](https://www.readthesequences.com/Value-Is-Fragile)). they [crystallize](https://carado.moe/value-crystallization.html) those values; they live by them, until they in turn become influenced by those values and perhaps *actually adopt them*. (the actual value function is mutable!)

this is generally bad; [you should want to preserve whatever your values are](https://en.wikipedia.org/wiki/Instrumental_convergence#Goal-content_integrity). hence the appeal that stands as the title of this post: do *not* hold on to the approximate function that is your best guess at what your value system is; you're only human, your guess is likely incorrect, and adopting it would run the risk of damaging your *actual* values; a function which, while hard to figure out, can be mutable, will certainly be mutated by acting as if your values are not what they are, and whose mutation you should generally want to avoid.

so, pay attention to your feelings. they are what is the output of your *actual* values system, by definition; follow your heart, not your reason's best guess.

note that this is *not* an appeal to favor deontology over consequentialism: how you feel can be about actions (deontology) or about outcomes (consequentialism), and which one it is is orthogonal to whether you follow that system, or whether you decide to follow your current best approximation of it. if you are consequentialist (as i recommend), just make sure to give your value system a full picture of what the outcome would look like, and *then* decide based on what feelings are produced by that.

meta note: this framework for thinking about values should be itself held with suspicion, as should probly any formal framework that concerns values. you should be careful about holding on to it just like you should have been careful about holding onto your believed values (careful enough to be able to consider the present post, for example). which isn't to say *don't believe what i just wrote*, but leave room for it to be wrong, partially or wholly.


2022-03-21

## making the UD and UDASSA less broken: identifying time steps

the [universal distribution](https://handsandcities.com/2021/10/29/on-the-universal-distribution/) ("UD") and [its applicability to anthropics](https://handsandcities.com/2021/11/28/anthropics-and-the-universal-distribution/) apparently suffers from some issues.

one is uncomputability. i think a speed penalty seems reasonable; but as a more general solution, i think it is unreasonable to "wait for the machine to halt". instead, let us see turing machines as running forever, with getting stuck one a repeating state as a special case. then, the space of all computations on a given universal turing machine ("UTM") is a set of pairs `(input program, time step)`; or more simply, if we use a [universal complete](https://carado.moe/universal-complete.html) program, it is *just* a time step: every computation will be ran at some point.

this model feels quite natural to me, and seems like an easy way to rank priors: weigh the result of hypotheses by the inverse of the time step at which the universal complete program runs into them.

this also helps with [UDASSA](https://handsandcities.com/2021/11/28/anthropics-and-the-universal-distribution): if you use a deterministic [model of computation](https://en.wikipedia.org/wiki/Model_of_computation) where every computation step only does a finite amount of stuff (like turing machines or SKI calculus, but unlike [wolfram-style graph rewriting](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/)) then you don't need a "claw" program to locate you within the world described by the world program; you are simply located at the set of time steps which happen to be updating the part of world that is *you*. this gives us a natural "locator" for persons within the space of all computation; it flattens together time, space, timelines (when a computation splits a world into multiple states all of which it then continues computing), and possible computation machines all into one neat linear sequence of steps.


2022-03-22

## goals for emergency unaligned AI

in [a previous post](https://carado.moe/when-in-doubt-kill-everyone.html) i talk about killing timelines — by making an AI fill them with something that's an easy to implement goal, such as paperclips — to avoid the even worse outcome of them becoming [S-lines](https://carado.moe/timeline-codes.html). in this post i wonder: can we do better than paperclips?

if the True Nature of the cosmos is [a universal program](https://carado.moe/universal-complete.html), then there could be some things to turn timelines into that consume less compute cycles; for example, maybe we somehow make an AI that makes its timeline run as little compute as possible. the cosmic universal program will then spend less many cycles running those dead timelines, and more running remaining live timelines — making survivors in them possibly "more real". in this sense, it may be that pruning timelines can be done without causing astronomical waste: compute time and therefore kinda ["realness"](https://carado.moe/udassa-time-steps.html) or ["soul juice"](https://handsandcities.com/2021/11/28/anthropics-and-the-universal-distribution/) are redistributed to remaining timelines.

even if the "naive" most likely "implementation" of our universe consumes just as much compute regardless of what goes on in it, the universal computation will contain other "implementations" that "compress" compressible timeline-states, and we will reclaim cycles in them — and if they are good at compressing, those implementations might be where most of our realness juice is located anyways.

another possibility is to task an AI with turning its timeline into a state that is as identical as it can be to another timeline in which said AI's didn't appear. if it can achieve that, then we can kill timelines by replacing them with copies of alive timelines. this also recycles compute time, possibly more efficiently since it doesn't rely on compressing implementations.


2022-03-21

## values system as test-driven development

i realized something while reading [hands and cities on infinite ethics](https://handsandcities.com/2022/01/30/on-infinite-ethics/): the work of determining the shape of [our values system](https://carado.moe/not-hold-on-to-values.html) is akin to [test-driven development](https://en.wikipedia.org/wiki/Test-driven_development).

we are designing a procedure (possibly looking for [the simplest one](https://en.wikipedia.org/wiki/Kolmogorov_complexity)) by throwing it at a collection of decision tests, and looking for which one matches our intuitions.

i wonder if a value-learning approach to AI alignment could look like trying to get superintelligence to find such a procedure; perhaps we feed it a collection of tests and it looks for the simplest procedure that matches those, and hopefully that extrapolates well to situations we didn't think of.

perhaps, even pre-superintelligence we can formalize values research as tests and try to come up with or generate a simple procedure which passes them while also being selected for simplicity.

why simplicity? doesn't occam's razor only apply to descriptive research, not prescriptive? that is true, but "what is the procedure that formalizes my values system" is indeed a prescriptive matter, in a way: we're trying to model something to the best factual accuracy we can.


2021-11-08

## against AI alignment ?

[i usually consider AI alignment to be pretty critical](https://carado.moe/were-all-doomed.html). that said, there are some ways in which i can see the research that is generally associated with alignment to have more harmful potential than not, if it is applied.

this is a development on my idea of [botched alignment](https://carado.moe/botched-alignment-and-awareness.html): just like AI tech is dangerous if it's developed before alignment because unaligned AI might lead to [X-lines](https://carado.moe/timeline-codes.html), alignment is dangerous because it lets us align AI to things we think we want, but aren't actually good; which sounds like it could lead to an increase in the ratio of S-lines to U-lines.

with this comes a sort of second [orthogonality thesis](https://www.lesswrong.com/tag/orthogonality-thesis), if you will: one between what we think we want and what is actually good. note that in both cases, the orthogonality thesis is a *default* position: it could be wrong, but we shouldn't assume that it is.

determining what is good is very hard, and in fact has been the subject of the field of *ethics*, which has been a work in progress for millenia. and, just like we must accomplish alignment before we accomplish superintelligence if we are to avoid X-risks, we might want to consider getting ethics accomplished before we start using alignment if we are to avoid S-risks, which should be a lot more important. or, at least, we should heavily consider the input of ethics into alignment.

things like [my utopia](https://carado.moe/∀V.html) are merely patches to try and propose a world that *hopefully* doesn't get *too bad* even after a lot of time has passed; but they're still tentative and no doubt a billion unknown and unknown unknown things can go wrong in them.

it is to be emphasized that both parts of the pipeline are important: we must make sure that what we think is good is what is actually good, and then we must ensure that that is what AI pursues. maybe there's a weird trick to implementing what is good directly without having to figure it out ourselves, but i'm skeptical, and in any case we shouldn't go around assuming that to be the case. in addition, i remain highly skeptical of approaches of "value learning"; that would seem like it would be *at most* as good as aligning to what we think is good.

so, it is possible that just as i have strongly opposed doing AI tech research until we've figured out AI alignment, i might now raise concerns about researching AI alignment without progress on, and input from, ethics. in fact, there's a possibility that putting resources into AI tech over alignment could be an improvement: [we should absolutely avoid S-risks, even at the cost of enormously increased X-risks](https://carado.moe/when-in-doubt-kill-everyone.html).