two-vtable

2021-11-21

## the two-vtable problem

when programming, there are in general two types of interfaces: "static, known" interfaces and "dynamic, unknown" interfaces.

in the former, the possibilities are well known. maybe an object has a bunch of public methods that can be used, or maybe even public fields; maybe an API has a known call endpoints.

when the behavior or contents of an object are unknown or inaccessible, someone can still implement how it interacts with another known-interface object: just send the known object to the unknown object, and let the unknown object manipulate the known object however it wants.

however, there is no general way to make two (or more) objects interact with each other, when they both have a dynamic/unknown interface.

this is what i call the **two-vtable problem**.

one approach is to implement all n² behaviors: implement the behavior for any possible concrete type of one object and any possible concrete type of the other. the rust ecosystem is kind of like that with types and traits: if you have `N` types and `K` traits, unless one of those traits has a blanket implementation for all types, you'll need to write `N×K` implementations to have proper coverage in the general case.

but this is hardly scalable; and doesn't work well, for example, in environments in which objects are expected to be implemented by different parties that don't necessarily coordinate with one another, where those objects are then expected to work together without putting in extra effort afterwards. i'm sure this probly has been encountered a lot for example in video game modding communities, regarding the interaction between mods created by different people.

an answer can be taken from other fields that have already solved that problem on their own, however. i can think of two: how natural selection solved negotiation between dynamic persons, and how liberalism solved negotiation between dynamic private actors.

the general solution to the two-vtable is to have the two objects have a language —as tells us the evolution of humans— that they can use to communicate and negotiate an outcome. liberalism tells us that the shape of negotiated outcomes is contracts, and cryptocurrencies tell us that the formalized form of contracts is programs.

and so, here is my proposed solution to the two-vtable problem: when two dynamic objects want to interact with one another, that interaction must take the shape of the two of them building a program together, which will be executed once they both agree on it. perhaps this can take the shape of both of them sending an initial contract to the other which is ideal from the perspective of the sender, and from there try to incrementally build up programs that implement a compromise between the two ideals, until they meet somewhere in the middle; like haggling.

this framework generalizes nicely enough that it could be used for arbitrary informatic agents, such as a bot and an uploaded person, or two uploaded persons. in fact, contract negotiation of that kind, when understood enough by both agents partaking of it, can be the formalized form of consent, a matter i've [grappled with formalizing](https://carado.moe/defining-freedom.html) for [my utopia](https://carado.moe/∀V.html).

this could also be useful for negotiation between different [compilation stacks for portable programs](https://carado.moe/portable-programs.html), or even for the negotiation between [different wasms running on a liberal server market](https://carado.moe/saving-server-internet.html).


right-to-death-therefore

2021-08-23

## right to death, therefore

because i like [freedom](https://carado.moe/defining-freedom.html) so much, i think people should be generally able to do what they want. but this immediately raises a conundrum: should someone be able to do an action that hampers their *future* freedom?

one relatively extreme case is the ability to commit suicide: it's about as committal as you can get, in terms of actions with future ramifications to oneself. if you choose to get in debt or cut off a limb, that can be pretty hard to get out of, but it still seems less impactful and less inescapable than suicide.

so, should suicide be allowed? (i am of course only talking about reasonable, clear-minded suicide, *informedly* consented; not coerced suicide, nor suicide out of compromised ability to make decisions)

in my opinion, *obviously yes*. the alternative, that people be forced to live until their biology kills them (which we may very well find ways to prevent indefinitely), seems abhorrent to me. given this guarantee, then it makes sense to me that any lesser commitments should also be fine.

there are some criticisms one can make about this argument. bad but non-death commitments could tend to increase the amount of suffering people in society at any given moment; and, if people change over time (as they tend to do), then commitments can ramificate into a future person who is sufficiently different from the person making the commitment that it might be considered unreasonable for them to be subject to some excessive amounts of "locally" unconsented negative effects. a cap on the time duration of commitments, and/or the requirement for people to guarantee that they remain the same "enough" over time until the commitment is expired (a technology we currently don't have, but will become easier to make once we're uploaded and we understand the human mind better), might be reasonable patches for these issues.


you-are-your-information-system

2020-12-25

## You are your information system

what makes you, you ?

we tend to intuitively think of a person as their entire body, somehow including limbs and organs but not clothing or food.

yet, if you close your eyes, and then i swap your arm with someone else's, when you wake up you will still be the same person, just with a new arm. in fact, i'd argue i could replace everything except for the nervous system (including the brain) and when you open your eyes again you would notice that your entire body has changed but your thoughts and memories have remained the same — rather than, for example, still having the same body but different thoughts and memories.

are you the matter that makes up that nervous system ? i could probably replace neurons and synapses one at a time and you would continue to be the same person. is it the electric signals then ? i could probably put on some synapses a device that absorbs electric signals and then sends out identical but "different" signals and you would still be the same person.

in fact, it doesn't really make sense to ask "which matter" makes up your nervous system: under quantum physics, everything is changing and particles are merely [values in an omnipresent field](https://www.youtube.com/watch?v=MmG2ah5Df4g) rather than solid objects.

ultimately, what you are, is *the information system* which your nervous system (including your brain) runs. standing still, walking forwards, teleporting yourself, and being uploaded into a sufficiently powerful computer, all preserve your personhood in the exact same way; there is nothing special about the meat that currently runs your mind.

*despite everything, it's still you.*


hope-infinite-compute

2022-05-12

## hope for infinite compute

here are some reasons we may have infinite universe to inhabit in the future.

* encoding ourselves in heat death noise. this could at least buy us exponentially much time to exist; it becomes infinite if the amount of possible states also increases.
* where are we in [the universal distribution](https://www.lesswrong.com/posts/EL4HNa92Z95FKL9R2/a-semitechnical-introductory-dialogue-on-solomonoff-1) ? most models that produce [rich](https://carado.moe/questions-cosmos-computations.html) computation, such as [rule 30](https://en.wikipedia.org/wiki/Rule_30), seem to grow forever in amount of stuff. this includes [wolfram's hypergraph rewriting system](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/).
* space is expanding. the planck length is a constant. unless there's something i'm mistaken about, this sure seems like more positions in space we could inhabit, and thus an increasing total amount of states the universe can be in. it is not obvious how to utilize this, but it may be evidence for other ways in which the amount of stuff in the universe grows. in wolfram's perspective, it is simply the hypergraph creating new positions in space, as it has been doing forever.
* why are there 10⁸⁰ particles in the observable universe? if the total number is larger, why is it that larger number? wouldn't it be occam-simpler that there be 1 or few particles (or qubits or whatever) in the start, and have that amount grow over time? in fact, with expanding space, won't there statistically tend to be more particles overall, if only because there's more space for quantum fluctuations to randomly spawn particles? surely a superintelligence can harness this.
* even if we're in a physically finite universe, we may be able to acausally trade/hack/blackmail aliens living in infinite worlds such as rule 30. maybe. [acausal trading is weird](https://www.lesswrong.com/posts/PcfHSSAMNFMgdqFyB/can-you-control-the-past).


ai-capability-risk-biases

2022-05-14

## cognitive biases regarding the evaluation of AI risk when doing AI capabilities work

i have recently encountered a few rationality failures, in the context of talking about AI risk. i will document them here for reference; they probly have already been documented elsewhere, but their application to AI risk is particularly relevant here.

### 1. forgetting to multiply

let's say i'm talking with someone about the likelyhood that working on some form of AI capability [kills everything everywhere forever](https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence). they say: "i think the risk is near 0%". i say: "i think the risk is maybe more like 10%".

would i bet that it will kill everyone? no, 10% is less than 50%. but "what i bet" isn't the only relevant thing; a proper utilitarian *multiples* likelyhood by *quality of outcome*. and X-risk is really bad. i mistakenly see some people use only the probability, forgetting to multiply; if i think everyone dying is not likely, that's enough for them. one should care that it's *extremely* unlikely.

### 2. categorizing vs average of risk

let's take the example above again. let's say you believe said likelyhood is close to 0% and i believe it's close to 10%; and let's say we each believe the other person generally tends to be as correct as oneself.

how should we come out of this? some people seem to want to pick an average between "carefully avoiding killing everyone" and "continuing as before" — which lets them more easily continue as before.

this is not how things should work. if i learn that someone who i generally consider about as likely as me to be correct about things, seriously thinks there's a 10% chance that my tap water has lead in it, my reaction is not "well, whatever, it's only 10% and only 1 out of the two of us believe this". my reaction is "what the hell?? i should look into this and stick to bottled water in the meantime". the average between risk and no risk is not "i guess maybe risk maybe no risk"; it's "lower (but still some) risk". the average between ≈0% and 10% is not "huh, well, one of those numbers is 0% so i can pick 0% and only have half a chance of being wrong"; the average is 5%. 5% is still a large risk.

this is kind of equivalent to *forgetting to multiply*, but to me it's a different problem: here, one is not just forgetting to multiply, one is forgetting that probabilities are numbers altogether, and is treating them as a set of discrete objects that they have to pick one of — and thus can justify picking the one that makes their AI capability work okay, because it's one out of the two objects.

### 3. deliberation ahead vs retroactive justification

someone says "well, i don't think the work i'm doing on AI capability is likely to kill everyone" or even "well, i think AI capability work is needed to do alignment work". that *may* be true, but how carefully did you arrive at that consideration?

did you sit down at a table with everybody, talk about what is safe and needed to do alignment work, and determine that AI capability work of the kind you're doing is the best course of actions to pursue?

or are you already committed to AI capability work and are trying to retroactively justify it?

i know the former isn't the case because there *was* no big societal sitting down at a table with everyone about cosmic AI risk. most people (including AI capability devs) don't even meaningfully *know* about cosmic AI risk; let alone deliberated on what to do about it.

this isn't to say that you're necessarily wrong; maybe by chance you happen to be right this time. but this is not how you arrive at truth, and you should be highly suspicious of such convenient retroactive justifications. and by "highly suspect" i don't mean "think mildly about it while you keep gleefully working on capability"; i mean "seriously sit down and reconsider whether what you're doing is more likely helping to save the world, or hindering saving the world".

### 4. it's not a prisoner's dilemma

some people think of alignment as a coordination problem. "well, unfortunately everyone is in a [rat race](https://slatestarcodex.com/2014/07/30/meditations-on-moloch/) to do AI capability, because if they don't they get outcompeted by others!"

this is *not* how it works. such prisoner's dilemmas work because if your opponent defects, your outcome if you defect too is worse than if you cooperate. this is **not** the case here; less people working on AI capability is pretty much strictly less probability that we all die, because it's just less people trying (and thus less people likely to randomly create an AI that kills everyone). even if literally everyone except you is working on AI capability, you should still not work on it; working on it would *still only make things worse*.

"but at that point it only makes things negligeably worse!"

…and? what's that supposed to justify? is your goal to *cause evil as long as you only cause very small amounts of evil*? shouldn't your goal be to just generally try to cause good and not cause evil?

### 5. we *are* utilitarian… right?

when situations akin to the trolley problem *actually appear*, it seems a lot of people are very reticent to actually press the lever. "i was only LARPing as a utilitarian this whole time! pressing the lever makes me feel way too bad to do it!"

i understand this and worry that i am in that situation myself. i am not sure what to say about it, other than: if you believe utilitarianism is what is *actually right*, you should try to actually *act utilitarianistically in the real world*. you should *actually press actual levers in trolley-problem-like situations in the real world*, not just nod along that pressing the lever sure is the theoretical utilitarian optimum to the trolley problem and then keep living as a soup of deontology and virtue ethics.

i'll do my best as well.

### a word of sympathy

i would love to work on AI capability. it sounds like great fun! i would love for everything to be fine; trust me, i really do.

sometimes, when we're mature adults who [take things seriously](https://carado.moe/life-refocus.html), we have to actually consider consequences and update, and make hard choices. this can be kind of fun too, if you're willing to truly engage in it. i'm not arguing with AI capabilities people out of hate or condescension. i *know* it sucks; it's *painful*. i have cried a bunch these past months. but feelings are no excuse to risk killing everyone. we **need** to do what is **right**.

shut up and multiply.


botched-alignment-and-awareness

2021-07-19

2022-05-09 edit: i have found out that this idea is more thoroughly explored [here](https://reducing-suffering.org/near-miss/).

## botched alignment and alignment awareness

[AI alignment](https://en.wikipedia.org/wiki/AI_control_problem) is [hard](https://intelligence.org/2018/10/03/rocket-alignment/).

an AI developer who doesn't know about the problem of alignment to general human values might accidentally develop a superintelligence which optimizes for something largely unrelated to humans, leading us to an [X-line](https://carado.moe/timeline-codes.html); on the other hand, if they make a botched attempt at alignment to human values, it seems like there's more of a chance (compared to if they don't try) at booting a superintelligence which cares about enough aspects of human existence to tile the universe with some form of humans, but not enough to make those humans' lives actually worth living (goals such as "humans must not die"), resulting in S-lines.

considering this, raising awareness of AI alignment issues may be a very bad idea: it might be much better to let everyone develop not-human-caring-at-all AI and cause X-lines rather than risk them making imperfect attempts resulting in S-lines. or: we shouldn't try to *implement* alignment to human values until we *really* know what we're doing.

contrary to a [previous post of mine](https://carado.moe/were-all-doomed.html), this is a relatively hopeful position: no matter how many timelines end in X-risk, inhabited P-lines can continue to exist and research alignment, hopefully without too many S-lines being created. on the other hand, while it increases the chance of the [singularity](https://en.wikipedia.org/wiki/Technological_singularity#Intelligence_explosion) turning out good by leaving us more time to figure out alignment, it also means that it might take longer than i'd've otherwise expected.


when-in-doubt-kill-everyone

2021-07-18

## when in doubt, kill everyone

one thing that is way worse than [mere existential risks](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer), possibly [by a factor of infinity](https://carado.moe/ai-alignment-wolfram-physics.html), is [suffering risks, or S-risks](https://en.wikipedia.org/wiki/Suffering_risks).

i could see (though going by what i could see [is not a reliable apparatus](https://carado.moe/overcoming-narratives.html)) someone make an AI and, while trying to align it to human values, accidentally misaligns it to something that happens to tile the universe with suffering humans. this would be an instance of S-risk.

whereas, an AI that merely wants to accomplish a relatively simple goal will probly just tile the universe with something simple that doesn't contain suffering persons; and given that [we're all probly quantum immortal](https://carado.moe/quantum-suicide.html), we just "escape" to the timeline where that didn't happen.

considering this, a 99% chance of X-risk 1% chance of utopia is preferable to a 1% chance of S-risk 99% chance of utopia. so, one thing we might want to do if we figure out superintelligence before we do alignment (which [seems pretty likely at this point](https://carado.moe/were-all-doomed.html); see also "Zero percent" on [this page](https://intelligence.org/2018/10/03/rocket-alignment/)), we might want to keep a ready-to-fire paperclip AI on standby and boot it up in case we start seeing S-risks on the horizon, just to terminate dangerous timelines before they evolve into permanent exponential hell.

in fact, just to be sure, we might want to give many people the trigger, to press as soon as someone even *suggests* doing any kind of AI work that is not related to figuring out goddamn alignment.


emergency-unaligned-ai-goals

2022-03-22

## goals for emergency unaligned AI

in [a previous post](https://carado.moe/when-in-doubt-kill-everyone.html) i talk about killing timelines — by making an AI fill them with something that's an easy to implement goal, such as paperclips — to avoid the even worse outcome of them becoming [S-lines](https://carado.moe/timeline-codes.html). in this post i wonder: can we do better than paperclips?

if the True Nature of the cosmos is [a universal program](https://carado.moe/universal-complete.html), then there could be some things to turn timelines into that consume less compute cycles; for example, maybe we somehow make an AI that makes its timeline run as little compute as possible. the cosmic universal program will then spend less many cycles running those dead timelines, and more running remaining live timelines — making survivors in them possibly "more real". in this sense, it may be that pruning timelines can be done without causing astronomical waste: compute time and therefore kinda ["realness"](https://carado.moe/udassa-time-steps.html) or ["soul juice"](https://handsandcities.com/2021/11/28/anthropics-and-the-universal-distribution/) are redistributed to remaining timelines.

even if the "naive" most likely "implementation" of our universe consumes just as much compute regardless of what goes on in it, the universal computation will contain other "implementations" that "compress" compressible timeline-states, and we will reclaim cycles in them — and if they are good at compressing, those implementations might be where most of our realness juice is located anyways.

another possibility is to task an AI with turning its timeline into a state that is as identical as it can be to another timeline in which said AI's didn't appear. if it can achieve that, then we can kill timelines by replacing them with copies of alive timelines. this also recycles compute time, possibly more efficiently since it doesn't rely on compressing implementations.


ai-alignment-wolfram-physics

2021-07-17

## AI alignment and wolfram physics

[wolfram physics](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/) is a project by [stephen wolfram](https://www.youtube.com/watch?v=0bMYtEKjHs0) to model physics using something kind of like a cellular automaton made of vertices in a graph instead of cells on a grid.

it's pretty interesting and there are insights in and around it that are of importance for the far future, and thus for [AI alignment](https://carado.moe/were-all-doomed.html).

the most notable is that wolfram thinks there's compute everywhere. the motion of the wind is doing compute, the motion of the seas is doing compute, the fabric of spacetime is doing compute, and even the state of heat death is still doing compute.

that last point notably means we might be able to embed ourselves into heat death and further, and thus get computed literally forever. this multiplies the importance of AI alignment by potentially literally infinity. i'm not quite sure how we are to handle this.

some of the compute may be doing things that are opaque to us; it might appear [homomorphically encrypted](https://en.wikipedia.org/wiki/Homomorphic_encryption). as we want (and expect) our superintelligence to spread everywhere to enforce values, we would hope civilizations living inside homomorphically encrypted spaces can be inspected; otherwise, nuking them altogether might be the only way to ensure that no [S-risk](https://en.wikipedia.org/wiki/Suffering_risks) is happening there.

wolfram postulates that one might be able to hack into the fabric of spacetime; one of the mildest effects of this would be the ability to communicate (and thus, likely, move) faster than celerity (but probably still slower than some other hard limit). if you didn't think [AI boxing](https://en.wikipedia.org/wiki/AI_box) was hopeless enough as it is, hackable spacetime ought to convince you.

finally, there is, value wise, an immense amount of compute being wasted; even just [standard model particles](https://en.wikipedia.org/wiki/Standard_Model) live way above true elementary computation. if superintelligence is well-aligned, this provides us with an hard estimate as to how much computing power we can live on to enjoy value, and it's probably a very large amount; wolfram [talks about](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful#how-it-works) something like 1e400 vertices in our universe.


timeline-codes

2021-07-18

## AI alignment timeline codes

this is a small post proposing simple one-letter codes for identifying timelines depending on their status relative to [AI alignment](https://en.wikipedia.org/wiki/AI_control_problem) and the appearance of [superintelligence](https://en.wikipedia.org/wiki/Superintelligence):

* **P-line**: a pre-intelligence explosion and pre-figuring out AI alignment timeline. we are in a P-line.
* **X-line**: a timeline where an [existential risk (or X-risk)](https://en.wikipedia.org/wiki/X-risk) has been realized by an unaligned superintelligence. everything is dead, forever.
* **S-line**: a timeline where a [suffering risk (or S-risk)](https://en.wikipedia.org/wiki/S-risk) has been realized by an unaligned superintelligence; the universe from then on contains net suffering on immense scales for all remaining time, [which is possibly infinite](https://carado.moe/ai-alignment-wolfram-physics.html). we should want to avoid this pretty much at all costs (including by [opting for an X-line instead](https://carado.moe/when-in-doubt-kill-everyone.html)).
* **A-line**: AI alignment has been figured out, and no superintelligence has been deployed yet. from that point on, we have the means to reach a U-line; though this isn't guaranteed. this is where we want to get as soon as possible.
* **U-line**: an aligned or [somehow otherwise](https://www.lesswrong.com/tag/orthogonality-thesis) benevolent superintelligence has been deployed, and we are guaranteed a relatively utopian world forever. this is the ultimate goal. while not strictly necessary, going through an A-line is almost certainly required to get there.

U-line, X-line, and S-line all have deployed superintelligences and are therefore terminal outcomes; they are unescapable. P-line and A-line are transitionary; they likely lead to one of the three terminal outcomes mentioned here.

other terminal might exist, but they seem unlikely enough to not warrant listing here; for example, even if everyone dies from, say, a meteor impact, life on earth or nearby will probably evolve another civilization *eventually*, which will also probably face the AI alignment challenge and end up in one of the terminal timelines.


unoptimal-superint-loses

2021-11-20

## unoptimal superintelligence loses

(edit: [maybe it doesn't](https://carado.moe/unoptimal-superint-doesnt-lose.html))

what if a phenomena is powerful enough to kill everyone, but not smart enough to be optimal at reasoning? (such as a grey goo event, or a "dumb" superintelligence with a faulty decision mechanism)

then, in all likelyhood, it eventually dies to an alien superintelligence that is better at decision-making and thus at taking over everything.

our superintelligence doesn't just need to be aligned enough; it needs to be aligned enough, and on the tech side, to be maximally intelligent. hopefully, it's smart enough to start making itself smarter recursively, which should do the trick.

the point is: when talking about the eventual superintelligence(s) that reign over the cosmos, assume whichever one(s) to have "won" to be optimal at decision making, because others probly got outcompeted.


forking-bitrate-entropy-control

2022-02-06

## forking bitrate and entropy control

if physics is based on a computational framework such as [wolfram physics](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/), but plausibly even if not, such that not all states the universe can be in produce the same number as next possible states;

in addition, if i am to follow to conclusion my current belief that moral patients count as different when they start being functionally different in terms of computation, and that exact copies morally count as a single person (as it makes [not much sense](https://carado.moe/persistent-data-structures-consciousness.html) to believe otherwise);

and if ([as it seems to be the case](https://carado.moe/limiting-real-universes.html)) the universe values coherence and thus only a limited set of local outcomes can emerge from a given local situation, or at least outcomes are weighed by coherence;

then it makes sense to start caring about the amount of forking a given timeline goes through. which is to say: the amount of future states to be [instantiated](https://carado.moe/questions-cosmos-computations.html), be it directly next step or indirectly in the longer term.

in fact, if one calls what they care about *moral patients*, then we should care about the "forking bitrate" of moral patients. for example, we could want moral patients with a net negative future to be forked as little as possible, and moral patients with a net positive future to be forked as much as possible. considering forks are created over steps of time, and entropy seems to be a good measure for them, i think "bitrate" is an appropriate term for this; hence, *forking bitrate*.

if we're just talking about a place as small as earth, we can estimate that consequences rapidly ramificate around to all moral patients; and as such, it seems reasonable to think that the forking bitrate of all patients will tend to go about in the same direction.

so, if you see a quantum dice, should you throw it?

if you think the future of earth has expected net positive moral value, or has little enough suffering for your taste (depending on your moral framework), then yes: by throwing the (quantum) dice, you might be multiplying the amount of instances of that value by the number of possible outputs of the dice, by creating that many times more future timelines.

if not, then you shouldn't throw it.

(even in the absence of quantum effects, if one were to just move entropy around while [phase space being conserved](https://www.lesswrong.com/posts/QkX2bAkwG2EpGvNug/the-second-law-of-thermodynamics-and-engines-of-cognition), moving the entropy from not-moral-patients to moral-patientss (or whichever thing you can about) still has that effect, i think)

this can probly be expanded to much larger-scale entropy control — and also, if superintelligences care about it (and if they're to be aligned, we might want them to) we can expect them to use it to maximize their value. even a [paperclip maximizer](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer) can want to create as many timelines containing as many varied possible paperclips and arrangements thereof, if it is made to care about that.


topia-layer-0

2020-03-30

*(2020-11-15 edit: this post is now largely superceded by [Two Principles For Topia](https://carado.moe/two-principles-for-topia.html))*

## Topia: Layer 0

In a similar way to the [Hierarchy of Needs](https://en.wikipedia.org/wiki/Maslow's_hierarchy_of_needs), I have been thinking about what post-singularity utopia we would want in term of layers.

I want a Layer 0, a universal set of guarantees that apply to everybody; so that, on top of that, people can build voluntary societies and sub-societies as far as they want. Ideally, their socities would be mutually compatible; one could partake of multiple societies and have friends in both. But they wouldn't have to be. It's all dependent on what society you want to join.

But, I think we do need a universal Layer 0. One that at least makes the singularity AI prevent other AIs from emerging and [turning everyone into paperclips](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer) or other existential risks. As this is the layer that applies universally, we want it as thin as possible, so that societies built on top of it have as much freedom to implement whatever they want; in particular, almost everything mentioned here can *be opted out of* (such as when joining a social contract). It's just your starting kit.

These should be the universal guarantees that everyone starts with: physical safety and basic living ressources.

For the physical safety, it's fairly easy to think of telling the singularity AGI to implement what I like to call NAPnobots — nanobots that are omnipresent in physical reality (and would manifest as mandatory added laws of physics to virtual realities) and enforce the [NAP](https://en.wikipedia.org/wiki/Non-aggression_principle); that is, prevent people and their physical property from being subjected to aggression without their consent ("without their consent" could be a tricky part — also, should people be able to *permanently* opt out of some NAP violation protections ?).

You want to create a society in which it's fine to punch each other in the face ? That's fine with me. All I ask is that that society be purely opt-in.

You want to create a communist utopia in which all belonging are shared ? That's fine with me. Just create a voluntarily contract where people consent to pooling their properties together for shared use.

You want the freedom to hurt yourself ? Just consent to being hurt by yourself. You want the freedom to hurt non-consenting others ? *No.* That's my personal opinion, of course, but I do think the maximum reach of a social contract should be that it can't force others into joining it, and I hope everyone else can agree that at least requiring one's consent to partaking of interaction should be a guaranteed absolute. On the other hand, I would definitely consent, personally, to very large ranges of interactions with at least my friends. They can punch me if they want; I trust them to not do that, and if my trust is betrayed beyond what I consider reasonable, I can always unconsent.

Also, since the brain evaluates every input it receives, this includes for the most part having to consent to any form of communication; rember that unconsented advertising *is* nigh-literally rape.

The second part, basic living ressources, is a concern of economics. Unless we manage to escape to universes where ressources not only are infinite, but can be accessed faster than humans can appear (which may become as simply as duplicating a running process on a computer), it requires among other things limiting the number of humans that can exist, or you run into [malthusian traps](https://slatestarcodex.com/2014/07/30/meditations-on-moloch/).

The best way I can think of doing this is: when the singularity starts, give everyone enough basic assets that the dividends that can be generated from them are easy to live off of. Then, for people to have a child, they need to acquire a number of assets that will provably generate the same amount of dividends for the child, and give those to him. On top of this arbitrarily complex liberal contracts can be established, of course; you can have a socialist society where everyone has consented to their ressources being taxed by exactly how much giving basic living assets to the new kids costs (kids which won't themselves be taxed in that way unless they consent to in turn join that society). There is the issue of what amount of ressources constitutes basic living, as well as if there is a cap on the amount of unviolable property a person or group can have — can an environmentalist group very quickly just plant a flag on all nearby useful planets (as to declare them their unviolable property) and then forever refuse for them to be interacted with (as the NAPnobots will enforce) ?

The choice of living ressources should be included: if what we choose to eat has as much of an effect on our psyche as we are starting to find out it does, then choosing what configuration of nutrients we receive should part of the basic living guarantees.

One of those basic ressources of course is healthcare, and healthcare should include guaranteed immortality unless you opt out of it. There's just no particular reason old age should have any more right to hurt a non-consenting person than any *other* outside aggressor; "outside" because people are their brain's information system, not their body. Becoming a virtual person would be the "easy" solution to immortality.


how-timelines-fall

2022-01-11 ★

## how timelines fall

i've speculated that we are all together, as a civilization, quantum immortal; [timelines where we all die](https://carado.moe/timeline-codes.html) can somewhat [be ignored](https://carado.moe/brittle-physics.html), leaving us mostly just with concerns of [heaven vs hell](https://carado.moe/botched-alignment-and-awareness.html) timelines.

but, in the lucky timelines where we *do* keep avoiding an [X-risk](https://en.wikipedia.org/wiki/X-risk) [superintelligence](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer), what does that look like ?

it would be silly to expect that avoiding such a superintelligence would look like trying to press the button to turn it on but at the last minute the button jams, or trying to press the button to turn it on but at the last minute the person about to press it has a heart attack. indeed, bayes should make us think that we should expect it to look like whatever makes it likely that superintelligence fails to be implemented.

what does this look like ?

global nuclear war, broad economic collapse, great cataclysms or social unrest in cities where most of the AI development is done, and other largely unpleasant events.

don't expect the world to look like the god of anthropics is doing miracles to save us from superintelligence; expect the world to look like he's is slowly conspiring to do whatever it takes to make superintelligence unlikely to happen long in advance.

expect the god of anthropics to create AI winters and generally make us [terrible at software](https://www.youtube.com/watch?v=pW-SOdj4Kkk).

expect the god of anthropics to create plausible but still surprising reasons for the availability of tensor hardware to become scarce.

look around. does this look like a century where superintelligence appears ? yes, i think so as well. the god of anthropics has his work cut out for him. let's try and offer him timelines where AI development slows down more peacefully than if he has to take the initiative.

while some of us are working on aligning god, the rest of us should worry about aligning luck.


what-is-value

2021-07-25 ★

## what is value?

i've come to clarify my view of value sufficiently many times that i feel like having a single post i can link to would be worth it. this is that.

what i call *value* is *things we care about*; *what determines what we ought to do*. i use "morality" and "ethics" interchangeably to generally mean the study of value.

a lot of this post is just ethics 101, but i feel it's still nice to have my own summary of things.

for more on values, read [the sequences](https://www.readthesequences.com/), notably [book V](https://www.readthesequences.com/Book-V-Mere-Goodness).

see also [this post on how explicit values can come to be](https://slatestarcodex.com/2018/07/24/value-differences-as-differently-crystallized-metaphysical-heuristics/).

### consequentialism vs deontology

a first distinction is that between [consequentialism](https://en.wikipedia.org/wiki/Consequentialism), where values are about *outcomes*, and [deontology](https://en.wikipedia.org/wiki/Deontology), where values are about *actions*.

the [trolley problem](https://en.wikipedia.org/wiki/Trolley_problem) is the typical example of a thought experiment that can help us determine whether someone is a consequentialism or a deontologist: a consequentialist will press the lever because they care about the outcome of people being alive, whereas a deontologist will not press the lever because they care about the action of causing a death.

i am a consequentialist: i care about outcomes. that said, consequentialism has to be followed to the end: if someone says "well, a consequentialist would do this thing, which would eventually lead to a worse world", then they're failing to understand consequentialism: if the eventual outcome is a worse world, then a consequentialist should oppose the thing. to that end, we have [rule consequentialism](https://en.wikipedia.org/wiki/Consequentialism#Rule_consequentialism): recognizing that committing to certain rules (such as "if you commit a murder, you go to prison") help us achieve generally better outcomes in the longer term.

a special case of consequentialism is [utilitarianism](https://en.wikipedia.org/wiki/Utilitarianism), in which the consequential outcome being cared about is some form of positive outcome for persons; generally happiness and/or well-being. i tend to also value people getting their values satisfied and having [self-determination/freedom](https://carado.moe/core-vals-exist-selfdet.html) (not valuing self-determination [has issues](https://slatestarcodex.com/2018/10/24/nominating-oneself-for-the-short-end-of-a-tradeoff/)), possibly moreso than happiness or well-being, so i don't know if i count as a utilitarian.

### intrinsic vs instrumental

i make a distinction between [instrumental values, and intrinsic values](https://en.wikipedia.org/wiki/Instrumental_and_intrinsic_value) (the latter can also be called "core values", "axiomatic values", "ultimate values", or "terminal values"; but i try to favor the term "intrinsic" just because it's the one wikipedia uses).

instrumental values are values that one has because it helps them achieve other values; intrinsic values are what one ultimately values, without any justification.

* "why do i want people to get practice hygiene? so they don't get sick as often"
* "why do i want people to get sick less often? because being sick seems like a decrease in their well-being"
* "why do i want people to have well-being? i can't give a justification for that, it's what i *intrinsically* value"

any theoretical query into values should be a sequence of instrumental values eventually leading to a set of intrinsic values; and those cannot be justified. if a justification is given for a value, then that value is actually instrumental.

just because intrinsic values don't have justifications, doesn't mean we can't have a discussion about them: a lot of discussion i have about values is trying to determine whether the person i'm talking to *actually* holds the values that they *believe* they hold; people *can be* and very often *are* wrong about what values they hold, no doubt to some extent including myself.

one can have multiple intrinsic values; and then, maximizing the *satisfaction* of those values, is often the careful work of weighing those different intrinsic values in tradeoffs.

this isn't to say intrinsic values don't have causal origins; but that's a different matter from moral justificaiton.

a lot of the time, when just saying "values", people are talking about *intrinsic* values rather than all values (including instrumental); i do this myself, including throughout this post.

### knowing one's values

most people don't have a *formalized* set of values, they just act by whatever seems right to them in the moment. but, even to [rationalists](https://www.readthesequences.com/What-Do-I-Mean-By-Rationality) like me, knowing what values one has is *very hard*, even moreso in a formalized manner; if we had the complete formal description of the values of even just one person, we'd have gone a long way towards solving [AI alignment](https://intelligence.org/2016/12/28/ai-alignment-why-its-hard-and-where-to-start/), which is [by extremely far](https://carado.moe/ai-alignment-wolfram-physics.html) the [single most important problem humankind has ever faced](https://carado.moe/were-all-doomed.html), and [is gonna be very difficult to get right](https://www.readthesequences.com/Value-Is-Fragile).

to try and determine my own values, i generally [make a guess and then extrapolate how a superintelligence would maximize those values to the extreme and see where that fails](https://carado.moe/core-vals-exist-selfdet.html). but, even with that process, it is very hard work, and like pretty much everyone else, i don't have a clear idea what my values are; though i have some broad ideas, i still have to go by what feels right a lot of the time.

### selfishness vs altruism

this is *not* about how someone ultimately only wants *their values* to be satisfied; this is true *by definition*. this is about whether those values can be *about* something other than the person having the values.

people seem to be divided between the following positions:

1. all values are ultimately selfish; there is no meaningful sense in which someone can *truly, intrinsically* care about anything outside themselves.
2. someone can have values about themselves, or have values about the rest of the world, or both.
3. all values are ultimately about the world; there is no meaningful sense in which someone can actually care about their own person in particular (for example because the notion of identity is erroneous).

i hold position 2, and **strongly** reject position 1, though it seems very popular among people with whom i have talked about values; i see no reason why someone can't hold a value about the world outside of themselves, such as *intrinsically* wanting other people to be happy or *intrinsically* wanting the world to contain pretty things. for more on that, see [this post](https://www.lesswrong.com/posts/XPErvb8m9FapXCjhA/adaptation-executers-not-fitness-maximizers) and [this post](https://www.readthesequences.com/Terminal-Values-And-Instrumental-Values) from the sequences.

position 3 can make some sense if you deconstruct identity, but i believe identity [is a real thing that can be tracked](https://carado.moe/you-are-your-information-system.html), and so the outcome of which you can absolutely happen to particularly care about.

### value preservation

[value preservation](https://en.wikipedia.org/wiki/Instrumental_convergence#Goal-content_integrity) is the notion that, if you know that you value something (such as being wealthy or the world containing pretty things), you should probly try to avoid becoming someone who *doesn't* value those things, or worse: someone who values the opposite (such as being poor or the world containing only ugly things).

the reason for this is simple: you know that if you become someone who values being poor, you'll be unlikely to keep taking actions that will lead you to be wealthy, which goes against your current values; and your goal is to accomplish your values.

some people argue "well, if i become someone who values being poor, and then i take actions to that end, that's fine isn't it? i'm still accomplishing my values". but it's really not! we established that your values is "being wealthy", not "being someone whose values are satisfied". in fact, "being someone whose values are satisfied" is meaningless to have as a particular value; the fact that you want your values to be satisfied is implied in them being your values.

i call the process of someone finding out that they should preserve their values, and thus committing to whatever values they had at that moment, ["value crystallization"](https://carado.moe/value-crystallization.html); however, one ought to be careful with that. considering one's set of values is likely a very complex thing, one is likely to hastily over-commit to what they *believe* are their values, even though they are wrong about what values they hold; worse yet, they might end up committing so hard that they actually start changing what values they have towards those believed values. this is something that of course one should aim to avoid: as mentioned above, you generally don't want to become someone who doesn't hold the values you currently do, including through the process of hasty crystallization and over-commitment.

this is not to say you should remain in a complete haze where you just do whatever seems right at any moment; without a special effort, this could very well entail your values changing, something you shouldn't want even if you don't know what those values are.

what you should do is try to broadly determine what values you have, and generally try to commit to preserving whatever values you have; and in general, to *be the type of person who preserves the values they have*. this should help you preserve whatever values you actually do have, even while you still haven't figured out what they are.

a funny hypothetical version of this could be: present-you should make a contract with future-you that if they ever gain the ability to precisely examine values, they should examine what values present-you had, and adopt those.


ΓêÇV

2021-08-31 ★

## ∀V: A Utopia For Ever

-->

∀V (read "universal voluntaryism" or "univol") is my utopia proposal. for people who are familiar with me or my material, this may serve as more of a clarification than an introduction; nevertheless, for me, this will be the post to which i link people in order to present my general view of what i would want the future to look like.

### what's a person?

you'll notice that throughout this post i've stuck to the word "person" instead of, for example, "human". this isn't just in case we eventually encounter aliens who we consider to be persons just like us, but it's also possible some existing animals might count, or even beings whose existence we largely don't even envision. who knows what kind of computational processes take place inside the sun, for example?

by person i mean something deserving of moral patienthood; though this is still hard for me to even start to determine. it probably requires some amount of information system complexity, as well as a single point of explicit consideration and decision, but apart from that i'm not quite sure.

i do know that pretty much all currently living humans count as moral patients. other than that, we should probably err on the safe side and consider things moral patients when in doubt.

### systems within systems

all top-level systems are, in the long term, permanent.

if you want society to "settle what it wants later", then your top-level permanent system is what they'll eventually settle on.

if you want society to never be stuck in any system and always have a way out, then the top-level permanent system is "going from system to system, being forever unable to settle" and you better hope that it spends more time in utopian systems than dystopian systems.

if your view is "whatever happens happens", then your top-level permanent system is whatever happens to happen. by not caring about what the future looks like, you don't make the future more free, you only are less likely to make sure it's one you'd find good.

if there is going to be a top-level system no matter what, no matter how flexible its internals are, we ought to care a lot about what that system is.

### enforcement: superintelligence

even if you don't think [superintelligence explosion](https://en.wikipedia.org/wiki/Technological_singularity#Intelligence_explosion) is [imminent](https://carado.moe/were-all-doomed.html), you should think it will happen eventually. given this, what may very well be [the "infinite majority" of remaining time](https://carado.moe/ai-alignment-wolfram-physics.html) will be one where a superintelligence is the ultimate decider of what happens; it is the top-level.

i find this reassuring: there *is* a way to have control over what the eternal top-level system is, and thus ensure we avoid possibilities such as unescapable dystopias.

### generalized alignment

in AI development, "alignment" refers to [the problem of ensuring that AI does what we *actually* want](https://en.wikipedia.org/wiki/AI_control_problem) ([rather than](https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml), for example, what we've explicitely instructed it to do, or just maximizing its reward signal).

when we think about how to organize future society, we actually care not just about the alignment of the top-level superintelligence, but also "societal alignment" before (and within) that. i will call "generalized alignment" the work of making sure future society will be in a state we think is good, whether that be by aligning the top-level superintelligence, or aligning the values of the population.

so, even if you don't think a superintelligence is particularly imminent, you should want society to start worrying about it sooner rather than later, given what you should consider being the amount of unknown variables surrounding the time and circumstances at which such an event will occur. you want to align society *now*, to your values as well as the value of figuring out superintelligence alignment, hopefully not too late.

### not just values

at this point, one might suggest directly loading values into superintelligence, and letting it implement whatever maximizes those values. while this may seem like a reasonable option, i would kind of like there to be hard guarantees. technically, from a utilitarian perspective, there exists a number N sufficiently large that, if N people really want someone to genuinely be tortured, it is utilitarianly preferable for that person to be tortured than not; my utopia instead proposes a set of hard guarantees for everyone, and *then, within the bounds of those guarantees*, lets people do what they want (including "i just want superintelligence to accomplish my values please").

one might consider the solution to that to be "just make it that people never want others to be tortured", but that's a degree of freedom on people's thoughts i'd rather keep if i can. i want persons to be as free as possible, including the freedom to want things that can't ethically (and thus, in my utopia, can't) be realized.

### a substrate for living

i am increasingly adopting [wolfram's computational perspective](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/) on the foundation of reality; beyond offering great possibilities such as [overcoming heat death](https://carado.moe/ai-alignment-wolfram-physics.html), i feel like it strongly supports the [informational view of persons](https://carado.moe/you-are-your-information-system.html) and the ability for people and socities to live on any form of computation framework; and those aren't particularly less or more real than our current, [standard model](https://en.wikipedia.org/wiki/Standard_Model)-supported reality.

given this, the most efficient (in terms of realizable value per unit of resource) way for superintelligence to run a world is to extract the essence of valuable computations (notably [the information systems of persons](https://carado.moe/you-are-your-information-system.html)) into a more controllable substrate in which phenomena such as aging, attachment to a single physical body, vulnerability to natural elements, or vulnerability to other persons, can be entirely avoided by persons who wish to avoid them. this extraction process is often referred to as "uploading", though that implies uploading into a nested world (such as computers running in this world); but if wolfram's perspective is correct, a superintelligence would probably be able to run this computation at a level parallel to or replacing standard model physics rather than on top of that layer.

this is not to say that people should all be pure orbs of thought floating in the void. even an existence such as a hunter-gatherer lifestyle can be extracted into superintelligence-supervised computation, allowing people to choose superintelligence-assisted lifestyles such as "hunter-gathering, except no brutal injury please, and also it'd be nice if there were unicorns around".

### universal voluntaryism

at this point, we come to the crux of this utopia, rather than its supporting foundation: ultimately, in this framework, the basis of the existence of persons would be for each of them to have a "computation garden" with room to run not just their own mind but also virtual environments. the amount of computational resource would be like a form of universal basic income: fixed per person, but amounts of it could be temporarily shared or transferred.

note that if resources are potentially infinite over time, as wolfram's perspective suggests, then there is no limit to the amount of raw computation someone can use: if they need more and it's not available, superintelligence can just put either their garden or *everyone's gardens* on pause until that amount of computation resource becomes available, and then resume things. from the point of view of persons, that pause would be imperceptible, and in fact functionally just an "implementation detail" of this new reality.

persons would have the ability to transform their mind as they want (though having a bunch of warnings would probably be a reasonable default) and experience anything that their garden can run; *except for computing the minds of other persons*, even within their own mind: you wouldn't want to be at the mercy of someone just because you happen to be located within their mind.

persons would be able to consent to interact with others, and thus [have the ultimate say on what information reaches their mind](https://carado.moe/cultural-and-memetic-hygiene.html). they could consent to visit parts of each other's gardens, make a shared garden together, and all manner of other possibilities, so long as all parties consent to all interactions, as determined by superintelligence — and here we're talking about [explicit consent](https://carado.moe/defining-freedom.html), not inferred desires even though superintelligence would probably have the ability to perfectly determine those.

for a perspective on what a society of "uploaded" persons might look like, see for example [Diaspora by Greg Egan](https://en.wikipedia.org/wiki/Diaspora_%28novel%29).

### rationale and non-person forces

the goal of this structure is to allow people to live and associate with each other in the most free way possible, making the least possible restrictions on lifestyle, while retaining some strong guarantees about consent requirements.

in a previous post i talk about [non-person forces](https://carado.moe/two-principles-for-topia.html); those being for example social structures that act with an agenthood of their own, running on other people as their own substrate.

at the moment, i simply don't know how to address this issue.

the problem with the "dismantlement" of such forces is that, if every person is consenting to the process, it's hard to justify superintelligence coming in and intervening. on the other hand, it does feel like not doing anything about them, short of being able to align sufficiently many people *forever*, will tend to make people dominated by such structures, as a simple process of natural selection: if there is room for such structures and they can at least slightly causate their own growth or reproduction, then they will tend to exist more than not. this may be thought of as [moloch](https://slatestarcodex.com/2014/07/30/meditations-on-moloch/) "attacking from above".

one major such potential non-person force is superintelligence itself trying to make people tend to want to live in ways that are easier to satisfy. if everyone wants to sit in their garden forever and do nothing computationally costly, that makes superintelligence's job a lot "easier" than if they wanted to, for example, communicate a lot with each other and live computationally expensive to run lifestyles; and the reason superintelligence will want to make its job easier is to increase the probability that it succeeds at that job (which it *should* want).

if informationally insulating people from superintelligence except when they outright consent to it intervening in their decisions is *not* sufficient, then maybe we can add the rule that people can never ask superintelligence to intervene in their life unless there is one single optimal way to intervene, and hopefully *that's* enough. the idea there being: if, for any request to superintelligence, there is only a single optimal way to accomplish that request, then superintelligence has no degree of freedom to influence people and thus what they want.

### on new persons

there are some reasons to be worried about the creation of new persons.

one is [malthusian traps](https://slatestarcodex.com/2014/07/30/meditations-on-moloch/): if the amount of resources is either finite or growing but bounded, or if it's unknown whether the amount of resources will end up being finite or not, then you *have* to cap population growth to at most the speed at which the amount of resource grows (if the amount of resources grows, the maximum speed of population growth should preferably be lower, so that the amount of resource each person has can grow as well). while it does seem like in current society people tend to have less kids when they have a higher quality of life, in a system where persons can live forever and modify their minds, one can't make such a guarantee over potentially infinite time.

another is replicator cultures: if there is no limit on creating new persons, and if people can influence even just a bit the values of new persons they create, then soon the world is overrun by people whose values are to create kids. or: making a world in which "new person slots" are filled by whoever wants to fill them first, will just select for people who want to fill those slots the most.

there might also be weird effects such as, even if resources were infinite, allowing arbitrary amounts of persons to be created could "stretch" the social network of consenting-to-interact-with-each-other persons such that, even if someone has registered an automatic consent to interact *even just a bit* with the kids of persons they already consent to interact with, they are soon flooded with a potentially exponentially growing network of kid interactions; though this probably can be addressed by that person by revoking this automatic consent.

beyond various resource and network effects, new persons create an ethical dilemma: does a person consent to living? or, for a child, do they consent to being taken care of for some amount of years after they are born — a time during which we often consider them to require affecting them in ways they might be unable to consent to?

if such philosophical quandries don't have a solution, then the safest route is to simply forbid the haphazard creation of new persons, whether that be through conventional human infants, [headmates](https://en.wikipedia.org/wiki/Multiplicity_%28psychology%29) and [tulpas](https://en.wikipedia.org/wiki/Tulpa#21st_century) if those are "real" enough to count as persons, and potentially other ways of creating new persons that can't consent to future interactions because they don't exist yet.

on the other hand, one way to increase the population *with* consent, is simply to "fork" existing persons: create a duplicate of them. because both are a continuation of the original single person, the original person's consent counts for both resulting persons, and there is no issue. the "merging" of consenting persons together might be possible *if* it can be reasonably estimated that their shared consent "carries through" to the new, merged person; i am currently undecided about how to even determine this.

finally, if resources are finite, creating a new person (whatever the means) should require permanently transferring one "universal basic computation amount"'s worth of computation garden to them, as no person should start out without this guarantee. this could be done by a person consenting to die and give up their own computation garden, it could be done by several "parents" consenting to give up a ratio of their gardens to the new person, it could be done by reclamating the redistribution of persons who die and don't make any decisions about what should be done with their computation garden, etc.


persistent-data-structures-consciousness

2021-06-16

## the persistent data structure argument against linear consciousness

people have the belief that they live a continuous, linear stream of consciousness (whatever that means).

i've [made arguments before](https://carado.moe/quantum-suicide.html) as to why this is erroneous; but here is another interesting argument that undoes the seeming coherence of such a statement.

think of reality as a computational process, generating frames one after another, possibly splitting into timelines.

where is your consciousness? one might be tempted to answer that it's the set of bytes representing the state of the brain. if i split the world into two timelines, which one is the "fake copy" and which one is the "continuous original"? one might answer that the copy is whichever new data structure has new bytes copied to it, and that the original is whichever presence in memory hasn't been moved; the *same* bytes, supposedly stored on the *same* hardware transistors.

if i were to split timelines by creating two copies and destroying the original, one might answer that this is akin to killing the original and creating two "fakes copies".

however, there exist [persistent data structures](https://en.wikipedia.org/wiki/Persistent_data_structure), which represent new sets of data as added constructions on top of an original one. this is a perfectly reasonable way to do computation, and one would probably agree that if someone is only ever running a single timeline, people have continuous consciousness.

if i were to run a world simulation using persistent data structures and generate a timeline split, which one is the "continuous person"? just like with continuous single timeline computation, both new timelines are merely new data structures with their own set of whichever data is different, and pointers back to whichever sets of data are unchanged.

the least unreasonable choice someone who believes in linear streams of consciousness could make is that somehow persistent data structures are *not* a valid form of universe computation; that a computation ought to be run by reusing the same memory locations. surely the arbitraryness of such a claim despite its functional equivalence to persistent data structures for single-timeline computation demonstrates well enough how the notion of linear streams of consciousness doesn't make sense.


noninterf-superint

2021-12-09

## non-interfering superintelligence and remaining philosophical progress: a deterministic utopia

[in a previous post](https://carado.moe/against-ai-alignment.html) i talk about the need to accomplish philosophical progress at determining what we value, before alignment. i wouldn't be the first to think of "what if we boot superintelligence now, and decide later?" as an alternative: it would indeed be nice to have this possibility, especially given the seeming imminentness of superintelligence.

alas, typically, making this proposition goes like this:

* A: "we should boot superintelligence now, and make it that we can adjust it later when we figure out more of philosophy."
* B: "yeah, but superintelligence isn't gonna just wait: it's gonna want to try to make us figure out whichever philosophy would make its job simpler, such as actually we value all dying immediately so that there's nothing to protect"
* A: "well, in that case, we need to make sure superintelligence can't interfere with our decision process."
* B: "and *how* do you ensure that the new being running all things in the world, has no interference into human affairs, exactly?"

which is a pretty good point, and usually a reasonably A concedes at that point.

today, however, i am here to offer a continuation to this conversation, from A's side.

my idea is to implement a deterministic computational utopia for people to be uploaded in, whose internals are disconnected from the outside world, such as [∀V](https://carado.moe/∀V.html); if we have infinite compute, then it can be even more free from outside interference.

the trick is to have that utopia's principles be *deontological*, or at least to make them absolute rather than able to be weighed against decisions outside of it: as it largely is in ∀V, ensure everything about utopia has a definite okay or not-okay status, evaluable without knowing anything about the "outside" of this utopia. either someone's consent is being violated, or it's not. with a set of decisions based only on the state of the utopia being simulated, every decision of the superintelligence about what it does in ∀V is unique: all superintelligence is doing is calculating the next step of this deterministic computation, including ethical principles, and thus there is nothing superintelligence can do to bias that decision in a way that is helpful to it. all it can do is run the computation and wait to see what it is that persons inside of it will decide to reprogram it to value or do; on the outside/before the singularity, all we need to ensure is that superintelligence does indeed eventually run this computation and apply the changes we decide on once it finds them out.

under these conditions, a device could be set up for us to later reprogram superintelligence somehow when/if we ever figure out what values we *actually* want, and it wouldn't be able to meaningfully interfere with our decision process, because every decision it takes regarding how our utopia is ran is fully deterministic.

not that i think being able to reprogram a superintelligence after boot is necessarily a good idea, but at least, i think it can be a possibility.


limiting-real-universes

2020-04-27

(2020-04-27 edit: actually Greg Egan already made this argument previously ([see Q5 if you have read Permutation City](https://www.gregegan.net/PERMUTATION/FAQ/FAQ.html)))

(2021-04-28 edit: this post might not be the best job at explaining its idea; see an alternate explanation in [this other post](https://carado.moe/quantum-suicide.html))

## Limiting Real Universes

The following is an argument for thinking that the set of universes that can "be real" (what that means is covered) is limited.

Notably, not all [Tegmark 4](https://space.mit.edu/home/tegmark/crazy.html) (i.e. mathematically possible universes) are real, nor even all considerable states of universes based on our current physics.

### I. Limiting Many-Worlds

A universe being "real" is defined here as "one could observe being in it".

Suppose all possible configurations of particles under our current laws of physics are *real*.

Then, out of all the universes that contain an exact physical copy of you, the vast majority of them should be universes that *do not* descend from a coherent history and thus everything that surrounds the copy of you should look like random particle soup.

(If such a distinction even makes sense), then You cannot tell if you're "the original" that comes from a coherent history or if you were "just created" as-is, because your memory could also "just have been created" as-is.

Yet, when you look around, everything looks very coherent.

Therefore, either you're *extremely* lucky, or only universe-states that descend from a coherent history are *real*. As per bayesianism, you should think the latter.

### II. Limiting Computing Ability

Suppose all universes based on our current physics, but with arbitrary amounts of "computing power" (i.e. how of its stuff can be turned into computers i.e. how much it has stuff) are "real".

Then some of those universes would end up making simulations of random universe-states, some of which happen to contain an exact copy of you.

However, if that were possible, because of the number of amounts of computing powers possible, *you* should be more likely to exist in one of these randomly created simulations with a universe with much more computing power than ours.

Yet, when you look around, everything looks very coherent.

Therefore, there must be *some* limit on the amount of computing power universes can have; and then, so that the sheer number of these universe can't compete with the meagre set of history-coherent from which our reality descends, there must either be a limit in the number of initial configurations other universes can have, or on the total computing power allocated to all universes.

Even better: suppose all states of [Conway's Game of Life](https://en.wikipedia.org/wiki/Conway's_Game_of_Life) are *real*. Then out of this infinity, a smaller infinity should happen to be running perfect simulations of subsets of this universe that happen to have you in them. But, reusing my argument again, you observe probably not being in them; therefore there must be a limit on the amount of computing power *even universes with other physics* have (and, again, a limit on either the number of configurations these other universes would be in, or the total computing power allocated to them all collectively).

### III. Conclusion

At this point it seems easier to *just assume that only universe-states that descend from our history* exist, or at least that the number of such histories is limited.

That certainly seems simpler than imagining there being a set of various systems by which other universes with other rules of physics would have a fixed amount of computing power allocated amongst themselves.

Not that all of this is mostly true even if you're a dualist: even if you have a soul (or equivalent), if there were infinite universes with infinite computing power, there's no reason the soul of *you reading this right now* should happen to be the soul of the original you and not a soul "created in a just-then created universe-state", unless you also assume complex soul mechanics.


udassa-time-steps

2022-03-21

## making the UD and UDASSA less broken: identifying time steps

the [universal distribution](https://handsandcities.com/2021/10/29/on-the-universal-distribution/) ("UD") and [its applicability to anthropics](https://handsandcities.com/2021/11/28/anthropics-and-the-universal-distribution/) apparently suffers from some issues.

one is uncomputability. i think a speed penalty seems reasonable; but as a more general solution, i think it is unreasonable to "wait for the machine to halt". instead, let us see turing machines as running forever, with getting stuck one a repeating state as a special case. then, the space of all computations on a given universal turing machine ("UTM") is a set of pairs `(input program, time step)`; or more simply, if we use a [universal complete](https://carado.moe/universal-complete.html) program, it is *just* a time step: every computation will be ran at some point.

this model feels quite natural to me, and seems like an easy way to rank priors: weigh the result of hypotheses by the inverse of the time step at which the universal complete program runs into them.

this also helps with [UDASSA](https://handsandcities.com/2021/11/28/anthropics-and-the-universal-distribution): if you use a deterministic [model of computation](https://en.wikipedia.org/wiki/Model_of_computation) where every computation step only does a finite amount of stuff (like turing machines or SKI calculus, but unlike [wolfram-style graph rewriting](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/)) then you don't need a "claw" program to locate you within the world described by the world program; you are simply located at the set of time steps which happen to be updating the part of world that is *you*. this gives us a natural "locator" for persons within the space of all computation; it flattens together time, space, timelines (when a computation splits a world into multiple states all of which it then continues computing), and possible computation machines all into one neat linear sequence of steps.


not-hold-on-to-values

2022-03-02

## do not hold on to your believed intrinsic values — follow your heart!

i posit the following framework for thinking about [intrinsic values](https://en.wikipedia.org/wiki/Instrumental_and_intrinsic_value) (hereby just called "values").

somewhere out there, there is **your value system**.

it is a function (in the mathematical sense) that takes as input *things*, and spits out *feelings*. that function is where your *true* values lie; they *are* that function.

how is that function encoded in your brain? who knows! i don't have an answer, and there may [not be an answer](https://en.wikipedia.org/wiki/Computational_irreducibility).

in conscious thought, however, you don't have access to the source code of that function, whatever it is. the best we can do for the moment seems to be to try thinking about it real hard, throw various things at it and examine the output ([perhaps through some mildly systematic process](https://carado.moe/core-vals-exist-selfdet.html)), and define another function that tries to approximate what your actual values are. this is hard and takes a lot of work. perhaps someday we will have a device that can scan your brain and give you a better idea of what your value function looks like, but at the moment that is not the case.

so, you build up an idea of what your values might be. here is where i think a lot of people make a mistake: they choose to believe strongly that this guess *is* their actual set of values (even though [it likely isn't](https://www.readthesequences.com/Value-Is-Fragile)). they [crystallize](https://carado.moe/value-crystallization.html) those values; they live by them, until they in turn become influenced by those values and perhaps *actually adopt them*. (the actual value function is mutable!)

this is generally bad; [you should want to preserve whatever your values are](https://en.wikipedia.org/wiki/Instrumental_convergence#Goal-content_integrity). hence the appeal that stands as the title of this post: do *not* hold on to the approximate function that is your best guess at what your value system is; you're only human, your guess is likely incorrect, and adopting it would run the risk of damaging your *actual* values; a function which, while hard to figure out, can be mutable, will certainly be mutated by acting as if your values are not what they are, and whose mutation you should generally want to avoid.

so, pay attention to your feelings. they are what is the output of your *actual* values system, by definition; follow your heart, not your reason's best guess.

note that this is *not* an appeal to favor deontology over consequentialism: how you feel can be about actions (deontology) or about outcomes (consequentialism), and which one it is is orthogonal to whether you follow that system, or whether you decide to follow your current best approximation of it. if you are consequentialist (as i recommend), just make sure to give your value system a full picture of what the outcome would look like, and *then* decide based on what feelings are produced by that.

meta note: this framework for thinking about values should be itself held with suspicion, as should probly any formal framework that concerns values. you should be careful about holding on to it just like you should have been careful about holding onto your believed values (careful enough to be able to consider the present post, for example). which isn't to say *don't believe what i just wrote*, but leave room for it to be wrong, partially or wholly.


genuineness-existselfdet-satisfaction-pick2

2021-11-21 ★

## Genuineness, Existential Selfdetermination, Satisfaction: pick 2

imagine you have a world where one person wants the moon to be painted blue, and another wants the moon to be painted red.

they both mean the current actual physical moon as it exists now; they both refuse any "cheating" option such as duplicating the moon or duplicating reality, and they don't want their minds changed, nor to compromise.

there's three ways to resolve situations like this:

* you sacrifice **genuineness**: you somehow make both of them believe, mistakenly, that what they want is satisfied. maybe by unknowingly giving them eye implants that change what color they see the moon.
* you sacrifice **[existential selfdetermination](https://carado.moe/core-vals-exist-selfdet.html)**: you ensure the situation never happens to begin with, that no two persons will ever want the moon to be painted different colors; or maybe you brainwash one of them after the fact.
* or, you sacrifice **satisfaction**: you let them want what they want, and let them see what the moon looks like, such that at most only one of them will ever be satisfied.

you can't have all three of those things.

many hedonists will happily sacrifice **genuineness**; authoritarians like to sacrifice **existential selfdetermination**.

as for me, for [∀V](https://carado.moe/∀V.html), i ultimately sacrifice **satisfaction**: while people can choose to become mistaken about things, the default is that they get to access the actual true state of things.


above-paperclips-2

2021-12-25 ★

## yes room above paperclips?

in two [previous](https://carado.moe/above-paperclips.html) [posts](https://carado.moe/nonscarce-compute-optimize-out.html) i talk about the ultimate inability for interesting things to happen when everything has been tiled with paperclips, even if the superintelligence doing the tiling isn't very good at it — i.e. lets room exist ["besides" (by superintelligence not actually consuming everything)](https://carado.moe/nonscarce-compute-optimize-out.html), or ["above" (using as a substrate)](https://carado.moe/above-paperclips.html) said paperclips (or whatever else the universe is being tiled with).

but, actually, this is only true if the spare compute (whether it's besides or above) only has room for one superintelligence; if that spare compute is composed of multiple bubbles causally isolated from one another, then maybe a superintelligence permanently kills everything in one, but another one creates even more bubbles in another.

in fact, as long as the first superintelligence to create many bubbles preceeds the first superintelligence to create no bubbles at all, and then if the amount of bubbles tends to be slightly more than one, and assuming superintelligences can't (or can only do so at a lesser rate than new bubbles being created) just "hack back upwards" (to escape to their parent universe), we can expect the set of pre-X-risk superintelligence bubbles to just increase over time.

this might provide a better explanation [than just us dying forever](https://carado.moe/estimating-populated-intelligence-explosions.html), for the weird fact that we exist now when the future could contain very many ([plausibly infinitely many](https://carado.moe/ai-alignment-wolfram-physics.html)) persons: it's not just that the amount of pre-singularity population is large compared to future timelines multiplied by their low likelyhood of being populated, it's that it grows over time forever and so makes it harder for U-lines or S-links to "compete", expected population wise.

we can then run into weird questions: rather than a tree, or even a [DAG](https://en.wikipedia.org/wiki/Directed_acyclic_graph), why couldn't this be just a general graph? if [the "seeds" for complex universes can be simple](https://en.wikipedia.org/wiki/Rule_30), it makes sense to imagine bubbles causating each other: maybe someone in [Rule 30](https://en.wikipedia.org/wiki/Rule_30) eventually boots a superintelligence that takes over everything but happens to cause a [Rule 110](https://en.wikipedia.org/wiki/Rule_110) bubble to appear (perhaps among many others), and then in that Rule 110 bubble someone creates a superintelligence that causes a Rule 30 bubble to appear again.

conceptually navigating this likely cyclical graph of pre-superintelligence bubbles seems like headache so i'll put the matter aside for now, but i'll be thinking on it more in the future. for the moment, we should expect bubbles with simpler seeds to be highly redundanced, and ones with more complex seeds to be rarer; but there's no reason to assume any ceiling on bubble seed complexity (in fact, if even just one of these bubbles is [universal complete](https://carado.moe/universal-complete.html), then *any* seed eventually gets instanciated!), and it seems nigh impossible to predict which types or complexities of seeds could lead to which outcomes, superintelligence-wise.

in the meantime, rember that while things might look pretty hopeless with this perspective, [it's plausible that we can actually actually causate *very far*](https://www.lesswrong.com/posts/PcfHSSAMNFMgdqFyB/can-you-control-the-past).


less-quantum-immortality

2021-12-27

## *less* quantum immortality?

if the set of nested universes [really does](https://carado.moe/what-happens-when-you-die.html) look like a funny graph of bubbles, i think there are two likely possibilities: either the set of bubbles rapidly dries up, or it grows towards infinity; in which case, if compute is infinite [as wolfram would have me think](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/) then as soon as the bubble explosion happens, it's likely a [universal complete](https://carado.moe/universal-complete.html) algorithm is booted somewhere reasonably fast, itself booting in turn all initial states.

this has the result of instanciating all (countable, discrete) [tegmark 4 universes](https://space.mit.edu/home/tegmark/crazy.html), over time.

yet, [we still observe a preference for coherency](https://carado.moe/limiting-real-universes.html): i think the reasonablest interpretation of what'd be going on is that "computationally early" or at least "computationall frequent" states are favored; and thus, very weird and incoherent initial-state universes *do* get spawned, but much later and/or are being computed more slowly (for example, maybe computation is equally distributed among all timelines, and as more and more timelines spawn over time each individual one gets updated less and less often).

while this creates a neat explanation for what selects for universe coherence, it does make it that while [quantum immortality/suicide](https://carado.moe/quantum-suicide.html) can be considered to "still work", if you choose to keep living [only by waiting to be reincarnated later](https://carado.moe/what-happens-when-you-die.html), you're reducing the "realness" of your continued existence; you're making universes in which you continue to live appear only "computationally later".

it also provides a nice simplicity test fo roccam's razor: the simplicity of a hypothesis can be akin to how soon a universal-complete program that simulates all spawned computations arrives at it.

this probly doesn't apply to "classical" quantum immortality where you just use the fact that you're redundanced on other timelines, because i would imagine those other you's in other timelines would tend to be computed "at the same time".


smaller-x-risk

2022-05-16

## smaller X-risk

a superintelligence killing us all is a *superintelligent, very large* [X-risk](https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence).

the superintelligence will tile its values in all directions; not just through space at celerity [or faster](https://en.wikipedia.org/wiki/Alcubierre_drive), but also, if it can, by [hacking physics](https://carado.moe/brittle-physics.html) and traversing across, for example, worldlines of the quantum many-worlds.

we may be able to create smaller X-risks, that only make us extinct in this timeline, on this earth. there are a few reasons we may want to do this:

* other timelines might have a better shot than us, and us booting a superintelligence may reduce their chances through weird stuff like intertimeline hacking
* to [avoid S-risks](https://carado.moe/when-in-doubt-kill-everyone.html), including S-risks that may be involved in instrumental cosmic-scale X-risk (maybe superintelligence wants to simulate civilizations in various ways for [acausal trade](https://www.lesswrong.com/tag/acausal-trade) or [other acausal weirdness](https://www.lesswrong.com/posts/PcfHSSAMNFMgdqFyB/can-you-control-the-past) reasons?)
* the next intelligent species on earth is more likely than us to solve alignment before superintelligence, and seems likely enough to be at least a little bit aligned with us (better than cosmic X-risk, at least)
* same as above, but for nearby aliens (whether current or future)

*smaller X-risk*, where we limit damage to just our civilization, seems harder than tiling the cosmos with paperclips; but at least it might be easier than [other plans](https://carado.moe/ai-risk-plans.html).

in a similar way, reducing our civilization to ashes *without* actually becoming extinct might also be a way to get another shot, if we think we're likely to do less badly next time.

remember: this is bigger than all of us. when the fate of the cosmos is at play, we can't afford to be too selfish.


bracing-alignment-tunnel

2022-04-10

## bracing for the alignment tunnel

it looks like we're gonna invent AI that [kills everyone](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer) before we figure out [AI alignment](https://en.wikipedia.org/wiki/AI_alignment).

what this means is that soon, [if not already](https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html), we are going to start bleeding [timelines](https://en.wikipedia.org/wiki/Many-worlds_interpretation), **hard**; by which i mean, an increasing ratio of multiverse-instants are gonna become dominated by unaligned AIs — and thus be devoid of population ([probably](https://carado.moe/above-paperclips-2.html)).

after that, there is a period in the (ever-diminishing amount of) surviving timelines, where we [ride on quantum immortality](https://en.wikipedia.org/wiki/Quantum_suicide_and_immortality) to solve alignment; after which, we finally reach [U-lines](https://carado.moe/timeline-codes.html), hopefully.

by many theories of [anthropics](https://handsandcities.com/2021/09/30/sia-ssa-part-1-learning-from-the-fact-that-you-exist/), observing existing either before or after that period is a lot more likely than observing existing in it. before the period, it is more likely because there are a lot more populated timelines in which to exist; after the period, it is more likely because we can hopefully "repopulate horizontally" by allowing the population to increase again.

if i am correct in the reasoning in this post, then being someone who exists in this very narrow alignment "tunnel" is exceedingly unlikely (barring weird circumstances such as post-singularity mankind choosing to simulate many variants of the tunnel for some reason). indeed, if you do observe being in it, you should think that something weird is going on, and update against the narrative presented in this post.

yet, *again if i am correct*, this is a period where we need to hold tight and work on alignment, perhaps as quickly as possible in order to reduce astronomical waste. very few us's inhabit the tunnel, but those very few us's are the critical ones who we need to care about.

so we need to brace our minds for the alignment tunnel. we need to commit to be persons who, if we observe being in the tunnel, will keep working on alignment even if, *from inside those timelines*, it looks like the reasoning i'm presenting here can't possibly be right. this is perhaps a weird case of instrumental rationality.

(note that i'm not saying the conclusion of observing being in those timelines should be to stop working on alignment; perhaps we would want to work on it either way, in which case we don't have to worry about anything. but i worry that it could lead us to other places such as "oh, maybe this AI killing everyone business isn't real after all, or maybe a weird alien force is preventing us from dying somehow")


core-vals-exist-selfdet

2020-09-09 ★

## Determining core values & existential self-determination

[Rationalism](https://www.readthesequences.com/) is about [epistemic rationality and instrumental rationality](https://www.readthesequences.com/What-Do-I-Mean-By-Rationality); but [when the two conflict, "rationalists should win"](https://www.readthesequences.com/Newcombs-Problem-And-Regret-Of-Rationality); so,

> Instrumental rationality: systematically achieving your values.

How does one determine their core (axiomatic) values ? Here's how i do it: i start from what i think is my set of values, and then i extrapolate what would happen if a [superintelligent](https://en.wikipedia.org/wiki/Superintelligence) [singleton](https://en.wikipedia.org/wiki/Singleton_%28global_governance%29) tried to implement those values.

Generally, the result looks like hell, so i try to figure what went wrong and start again with a new set of values.

For example: imagine i think my only core value is general happiness. The most efficient way for a superintelligence to maximize that is to [rewire everyone's brain](https://wiki.lesswrong.com/wiki/Wireheading) to be in a constant state of bliss, and [turn as much of the universe as possible](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer) into either more humans that experience constant bliss (whichever form of "human" is the cheapest resource-wise to produce) or into infrastructure that can be used to guarantee that nothing can ever risk damaging the current set of blissful humans.

So, clearly, this is wrong. The next step is freedom/self-determination; such that people can do whatever they want.

However, the most efficient way to make sure people can do what they want is to make sure they don't want to do anything; that way, they can just do nothing all day, be happy with that, and some form of freedom is maximized.

To address this issue, my latest idea is to value something i'd like to call *exstential self-determination*: the freedom to *exist as you would normally have*. It's a very silly notion, of course; there is no meaninful "normally". But still, i feel like something *like that* would be core to making sure not just that existing people can do what they want, but that humankind's general ability to be original people who want to do things is not compromised.


finite-patients

2022-03-21

## are there finitely many moral patients?

wouldn't it be neat if we didn't have to worry about [infinite ethics](https://handsandcities.com/2022/01/30/on-infinite-ethics/) ?

i think it is plausible that there are finitely many moral patients.

the first step is to [deduplicate moral patients by computational equivalence](https://carado.moe/deduplication-ethics.html). this merges not only humans and other creatures we usually care about, but also probably a lot of [other potential sources of moral concerns](https://reducing-suffering.org/what-are-suffering-subroutines/).

then, i think we can restrict ourselves to patients in worlds that are discrete (like ours); even if there *were* moral patients in non-discrete worlds, it seems to me that from where we are, we could only access discrete stuff. so whether by inherent limitation, plain assumption, or just limiting the scope of this post, i'll only be talking about discrete agents (agents in discrete worlds).

once we have those limitations (deduplication and discreteness) in place, there are finitely many moral patients of any given size; the only way for an infinite variety of moral patients — or more precisely, moral patient moments — to come about is for some moral patients to grow in size forever. while infinite time seems plausible [even in this world](https://carado.moe/ai-alignment-wolfram-physics.html), it is not clear to me that whatever the hell a "moral patient" is can be arbitrarily complex; perhaps at a certain size, i start only caring about a *subset* of the information system that a "person" would consist of, a ["sub-patient"](https://carado.moe/deduplication-ethics.html).


value-crystallization

2021-03-04

## Value Crystallization

there is a weird phenomenon whereby, as soon as an agent is rational, it will want to conserve its current values, as that is in general the most sure way to ensure it will be ablo to start achieving those values.

however, the values themselves aren't, and in fact [cannot](https://en.wikipedia.org/wiki/Is–ought_problem) be determined purely rationally; rationality can at most help [investigate](https://carado.moe/core-vals-exist-selfdet.html) what values one has.

given this, there is a weird effect whereby one might strategize about when or even if to inform other people about [rationality](https://www.readthesequences.com/) at all: depending on when this is done, whichever values they have at the time might get crystallized forever; whereas otherwise, without an understanding of why they should try to conserve their value, they would let those drift at random (or more likely, at the whim of their surroundings, notably friends and market forces).

for someone who hasn't thought about values much, *even just making them wonder about the matter of values* might have this effect to an extent.


nonscarce-compute-optimize-out

2021-12-25

## non-scarce compute means moral patients might not get optimized out

i tend to assume AI-borne [X-lines are overwhelmingly more likely than S-lines or U-lines](https://carado.moe/timeline-codes.html), because in almost all cases (such as [paperclip manufacturing](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer)) the AI eventually realizes that it doesn't need to waste resources on moral patients existing (whether they're having an okay time or are suffering), and so recycles us into more resources to make paperclips with.

but [if wolfram's idea is correct](https://writings.stephenwolfram.com/2020/04/finally-we-may-have-a-path-to-the-fundamental-theory-of-physics-and-its-beautiful/#how-it-works) — a possibility which [i'm increasingly considering](https://carado.moe/ai-alignment-wolfram-physics.html) — it may very well be that computation is not a scarce resource; instead printing always more paperclips is a trivial enough task, and the AI might let "bubbles" of computation exist which are useless to its goals, even growing bubbles.

and those could contain moral patients again.

of course this reduces to the [*no room above paperclips* argument](https://carado.moe/above-paperclips.html) again: inside that bubble we probly just eventually make our own superintelligence again, and *it* takes over everything, and then either bubbles appear again and the cycle repeats, or eventually in one of the layers they don't anymore and the cycle ends.

but, i still think it's an interesting perspective for how something-maximizing AIs might not need to actually take over *everything* to maximize, if there's nonscarce compute as wolfram's perspective can imply.