The Professional's Dilemma

In most cases we should expect Alan's wife to feel safer if Alan keeps up appearances. And that's why vast swathes of our professional society are underperforming their potential: because playing it safe means covering things up. Can organizations ever support new ideas? An organization that's funding outside parties to accomplish certain goals (e.g. a charitable donor or foundation giving to charities) generally won't fund ideas, even ideas from those people who are both famous and established experts, unless those ideas start off with a critical amount of their own money attached. Either the idea has to be spearheaded by someone who has enough personal funds, or it has to have raised enough money from donors/investors who no longer have the in-practice authority to dictate decision making. (For more detail on this, see Samo Burja's essay Borrowed versus Owned Power . Borrowed power is granted conditionally --if your charity is given a "donation" under the condition that you continue to make the donor happy, that's borrowed power. If you have the money in your own personal savings account, or if the charity raised it from so many small donors that none of them are entitled to make demands about how you spend it, that's owned power. Borrowed power is always weaker than owned power.) It's not generally considered "credible", by big donors, that an initiative will be effective enough to be worth funding unless the initiative has managed to cross a certain threshold in terms of acquiring resources. Moreover, if the grantee meets that threshold, gets funded, and does a good job with that money, this doesn't necessarily "unlock" disproportionately more resources from the big donor; the grantee will still have to hustle the same amount, or nearly so, to get the second grant. They're still running on borrowed power; they still need to keep looking over their shoulder, worrying about what the funders might think. We know it's possible to do better than this, because this paradigm would never have allowed the Manhattan Project to get funded. Einstein didn't come with his own money; he was merely a renowned physicist with a credible proposal. Somebody in the government had to literally believe his claims that a specific thing that had never been done before could be done. This kind of belief is crucially different from the sense usually used when we talk about believing in someone, in that it is about the trusted person's relationship to their field of expertise, not their relationship to the person trusting them. Not everyone can evaluate every idea, but if a person reaches a certain threshold of expertise, eminence, reputation for honesty, etc., the funder has to be able to come to believe that the content of their ideas is more likely than baseline to be valid and worth pursuing. As a funder, you have to be able to believe someone --not everyone, but someone --when they say an initiative is worthwhile, enough that you are willing to take the initiative to make it possible, and you don't expect to regret that decision. Otherwise, you are not looking for ideas outside your institution . You are not a "hub", you are not reachable, you are just doing your thing with no outside input. Another way of putting this is that funders need to be able to see potential grantees as peers . Organizations having "friends" --other organizations or consultants/free-agents that they trust, are mission-aligned with, and communicate openly with --is often considered unprofessional ("incestuous"), but is actually a good thing! There needs to be someone outside you (or your org) whom you trust to be well-informed and value-aligned. Without the capacity to process genuinely new ideas, it's less effective to have grantees than to just have employees and decide everything top-down. If you're doing something in a decentralized fashion, it should be because you actually value the decentralization --you want to get an outside perspective, or on-the-ground data, or expertise, or something.
 Principal-Agent Problems How can the principal (the funder) trust the agent (the grantee)? Most of the incentive systems in the what we hear about in charity are rather primitive; it boils down to "demand more documentation and proof from grantees." This is primitive because it tries to enforce honesty by spending resources on detecting dishonesty, which can be very costly, relative to other ways of incentivizing honesty. It's not even asking the question "what's the most efficient way to get the grantee to tell me the truth?" The aesthetic underlying this attitude is called authoritarian high modernism; just trying to add metrics, not even engaging with the fact that metrics can and will be gamed. The people who survive in positions of power in such a system are not the ones who naively try to answer questions they're asked as accurately as possible; they're the ones who keep up appearances. Conversely, when interacting with someone who keeps up appearances and who has survived in a position of power in such a system, there is common knowledge only 'professional' efforts will be supported, and that 'professional' efforts are efforts that don't freely reveal information. You can enforce quality control in a top-down fashion if you have an impartial investigative process and an enforcement mechanism, but who investigates the investigators? Who judges the judges? Ultimately somebody has to WANT to give an honest report of what's going on. And those impartial investigators have to be incentive-aligned such that they benefit more from being honest than lying . Otherwise, even if initially most of the investigators are highly selected for honesty, by organizational maturity, all will have been selected for 'discretion'. To get honest reports, you need a mechanism designed in such a way as to systematically favor truth, much like auctions designed so the most advantageous price to bid is also each person's true price. How do you do that?
 How to solve the problem Let's work through a concrete example: Suppose you're considering giving a grant to a charitable organization. They send you a budget and ask for a dollar amount. How can you incentivize them NOT to pad the budget? For instance, if they expect that they might be able to get this grant but they'll have a hard time getting a second grant, they have a strong incentive to ask for enough money up front to last them several years, BUT to say that they need it all for this year. This will happen UNLESS they have reason to believe that you'll frown on budget-padding, BUT are willing to look favorably on orgs that do well in the first year for subsequent grants. This requires opening up lines of communication MUCH more for successful/trusted grantees than for random members of the public or arbitrary grant applicants. It needs to be POSSIBLE to earn your trust, and for that to unlock a certain amount of funding security. Otherwise, grantees must seek funding security --for themselves, their families, their employees, their charitable work --because without doing so they won't be able to do their job at all. They'll seek it through deceiving you, because that will feel like, and will actually be, and will be seen by all observers to be, the responsible thing for them to do . If you make it seem like "nobody should be able to count on my support, I'll keep 'em on their toes", they'll find something else they can count on, namely your ignorance of how their project works. The agent ALWAYS knows more than the principal about the operational details of the project! They can always keep you in the dark more effectively than you can snoop on them. So you have to make earning your trust p ossible, empowering and rewarding . You can of course revoke those rewards if they betray your trust, but ultimately you have to be much less suspicious of your trusted friends than you are of randos. Yes, this means you take on risk; but the grantees ALSO take on risk by being honest with you. It's symmetric. Here's one common failure mode in the conventional paradigm: you can't appear competent if you reveal the truth that something in your project isn't working at the current funding level and needs more money. You can't seem "needy" for money, or you'll look incompetent, and you won't get the money. So instead you try to get the money by inflating your accomplishments and hiding your needs and trying to appear "worthy." This is counterproductive from the funder's point of view. As a donor, you want to give money where it'll do the most good. This means you need accurate info about what it'll be spent on. But the grantee doesn't necessarily trust you to believe that what they actually need money for is worthy. For a well-known instance, many donors think operational costs are "waste", so charities fudge the accounting to make it seem like all donations go to program expenses, and still underspend on operational costs like paying their employees. Or, sometimes part of a charity's actual budget is somewhat unsavory, like bribes to officials in corrupt countries being a necessary cost of actually operating there. Or, some costs can be embarrassing to admit, like the costs of learning/mistakes/R&D, if there's a perceived expectation that you have to get things right the first time. So the onus is on the funder to make it clear that you want to know what they ACTUALLY need, what their REAL constraints are, and that you will not pull the plug on them if they have an awkward funding crunch, once they have earned your trust to an adequate degree. It has to be clear that CANDOR is an important part of earning your trust. In particular, it needs to be clear enough to the third parties in the life of the individual you're funding, that being honest with you will pay off, that they pressure the grantee to be more honest with you, the funder. How does their family feel? Is the spouse of the charity's executive director nagging them to be more honest when fundraising, because that's what'll put their kids through college? Because, if not, you can bet their spouse is nagging them to be less honest while fundraising, to squeeze more money out of the donors, because that's the responsible thing to do for their family. What about their other dependents --employees, collaborators, beneficiaries of charitable programs --would THEY put pressure on the executive director to be more forthcoming with the funder? Or would they say "squeeze those rich morons harder so we can keep the lights on and help people who need it?" (Or, perhaps, more discretely but synonymously, "don't make waves, jump through the hoops they're asking you to jump through and tell them what they want to hear.") People don't exist in a vacuum; they want to gain the esteem and meet the needs of the people they care about. We have to be not only rewarding honesty -not only rewarding honesty more than smoothing things over -but OBVIOUSLY ENOUGH rewarding honesty that even third parties know our reputation. Otherwise everyone who tries to be honest with us will receive a continuous stream of pressure to stop being so irresponsible . "Why are you favoring a rich guy or a huge foundation over your own family and an important cause?" It's a fair question! And you need to flip it on its head --you want everyone to be asking "Why are you forgoing HUGE OPPORTUNITIES for your family and an important cause, just because you're too insecure to be candid with a rich guy?

Abstract: Alan, the executive director of a small nonprofit, is sitting down to dinner with his family. Responsibility weighs heavily on Alan. It's nearly time to reapply for a grant that's been his organization's largest source of funds. Alan has carefully cultivated a relationship with Beth, the program officer for the foundation funding the grant. Beth knows and likes Alan, and is excited about the work he's doing. If Alan's organization applies for a grant to continue their existing work, it will almost certainly be approved. But Alan isn't sure what to do. In the past few months, it's become increasingly clear that his fledgling organization's main program -the one they were able to write grants for -isn't the best fit for the needs of the people it's supposed to serve. A bright young intern, Charles, proposed a new program that, as far as Alan can tell, would be a better fit both for the team and the people they are trying to help. On the merits, it seems like Charles's idea is the right thing to do. Alan could, in reapplying, explain the limitations of the existing program, and why the new program would be better. But that would rock the boat. It would mean challenging the very story that got last year's grant. There's no guarantee Beth would like the new idea. No one else has done it. There's no track record to point to. Alan thinks of what might happen if the grant isn't approved. He'd have to lay people off, good people, and his organization might not survive. Even if it does, he'd have to take a pay cut, maybe move to a house in a less expensive neighborhood, pull his kids out of school. It's not certain this would happen, but can he afford to take that chance? Let's look across the dinner table at Alan's wife, Dana. If Alan tells her about his dilemma, what do we expect her to feel. What would be the safe option? Disclosing the potentially unsettling truth to the funder in the hopes of being able to do better work in the future? Or smoothing things over, tweaking the existing program to try to patch some of the worse gaps, and muddling on? What will feel to her like the responsible choice for Alan to make, for his family and the other people depending on him?


Andrew E Snyder-Beattie
Toby Ord
Michael B Bonsall

An upper bound for the background rate of human extinction

Bounding the extinction Rate Based on Age of Humanity Anatomically modern human fossils in Ethiopia have been dated to 195 ± 5 thousand years ago (kya)  13  . A more recent fossil discovery in Morocco of an anatomically modern human has been dated to 315 ± 34 kya  14, 15  (though the fossil may exhibit more primitive neurocranial and endocranial morphology). Given that Homo sapiens has existed for hundreds of thousands of years, what can we infer about our background rate of extinction? Assuming that we share a common extinction rate with our predecessors, we can rule out rates that are too high to be compatible with this track record of survival. As our aim is to construct an upper bound, we can set aside the possibility that modern human technology, habitat range, and population size have reduced a number of natural extinction risks. The upper bound is only violated if we have reason to believe current extinction rates are higher than those our predecessors faced. Since we exclude anthropogenic risks from our analysis, we also set aside the majority of the ways in which this could be the case, although we acknowledge there exist boundary cases between purely natural and anthropogenic risks (e.g. a naturally emerging disease could be spread further by modern technology). Ultimately the scope of the upper bound is limited to all risks that have remained constant (or have been reduced) over the past few hundred thousand years. Likelihood of extinction rates. Analysis of taxonomic survivorship curves and temporal ranges for a wide variety of taxa suggest that extinction probabilities can be approximated well by assuming a constant risk of extinction over time  [16] [17] [18]  . Under this model, extinction can be represented by the exponential distribution with constant extinction rate μ. The probability that humanity goes extinct before time t is given by the cumulative distribution function P(T ≤ t) = 1 − e −μt , where T is the random variable denoting the longevity of our species. Conversely, the probability that humanity makes it beyond time t is P(T ≥ t) = e −μt . We want to evaluate the likelihood of an extinction rate μ, given the observation that humanity has lasted up to time t (so we know that the total longevity of humanity T ≥ t). This can be evaluated as the likelihood function  μ| ≥ = μ − T t e ( ) t . We compute the likelihood of extinction rates between 10 −8 and 10 −4 given a number of different plausible starting dates for Homo sapiens outlined in Fig.  1  and Table  1 . Assuming a 200 thousand year (kyr) survival time, we can be exceptionally confident that rates do not exceed 6.9 × 10 −5 . This corresponds to an annual extinction probability below roughly 1 in 14,000. The relative likelihood for such high extinction rates are below 10 −6 (one in a million) when compared to a rate of 10 −8 . If we assume that our track record extends further, this upper bound becomes stronger. Using the fossil dated to 315 ka as a starting point for humanity gives an upper bound of μ < 4.4 × 10 −5 , corresponding to an annual extinction probability below 1 in 22,800. Using the emergence of Homo as our starting point pushes the initial bound back a full order of magnitude, resulting in an annual extinction probability below 1 in 140,000. We can also relax the one in million relative likelihood constraint and derive less conservative upper bounds. An alternative bound would be rates with relative likelihood below 10 −1 (1 in 10) when compared to the baseline rate of 10 −8 . If we assume humanity has lasted 200 kyr, we obtain a bound of μ < 1.2 × 10 −5 , corresponding to an annual extinction probability below 1 in 87,000. Using the 2 Myr origin of Homo strengthens the bound by an order of magnitude in a similar way and produces annual extinction probabilities below 1 in 870,000. It is worth noting that this model can be generalised to allow for a varying extinction rate over time μ(t), so that the probability of surviving past time t is given by P(T ≥ t) = e −Θ(t)t , where ∫ μ Θ = t t s ds ( ) (1/ ) ( ) t 0 . The upper bound on Θ(t), the average extinction rate over the interval, can then be calculated in the same way as for the constant rate model.
 Observation Selection Effects The data on humanity's survival time could be subject to survivorship bias. If early Homo sapiens requires a long period of time to develop the intellectual machinery needed to make scientific observations, then such observations could not include short evolutionary histories, regardless of the extinction rate. The amount of information we could derive from a long track record of survival would therefore be limited due to this observation selection effect. Such a track record could indicate a low extinction rate, or be the byproduct of lucky ancestors surviving high extinction rates long enough to beget progeny capable of making scientific observations. One might therefore object that the bounds on the extinction rate we have estimated are too low  12, 23  . Here, we examine and respond to this concern.
 Models to quantify potential sample bias. To model observation selection bias, let us assume that after Homo sapiens first arises another step must be reached. This could represent the origin of language, writing, science, or any relevant factor that would transition early humans into the reference class of those capable of making observations (we call this step 'observerhood'). Let this step be a random variable denoted S, with cumulative distribution function F S (t). As we are examining natural risks, we assume that S and T are independent. The probability that humanity survives long enough to reach observerhood status (via intelligence, language, writing, science, etc) can be found with the following integral: ∫ > = ∞ P T S f t F t dt ( ) ( ) ( ) (1) T S 0 www.nature.com/scientificreports www.nature.com/scientificreports/ where f T (t) = μe −μt , the probability of extinction at time t. We evaluate an adjusted likelihood function  μ| > ⁎ T t ( ) , denoting that we are taking the likelihood of an extinction rate μ given that humanity has survived to time t, and the fact that we are conditioning on the existence of observers such that T > S. This results in the adjusted likelihood function: μ μ | > = > | > ⁎ T t P T t T S ( ) ( , ) (2)  ∫ = ∞ c f s F s ds 1 ( ) ( ) (3) t T S where c = P(T > S) is a normalising constant. We evaluate a model with four variations for the observerhood step: a model in which observerhood occurs as a single event that has a constant rate over time, a model with an increasing rate over time, a model with multiple steps, and a model where observerhood simply requires a fixed amount of time. If desired, we could more crisply define this observerhood property as the ability for a species to collect reliable data on its own track record of survival (e.g. via fossil dating) and analyse it. When correcting for observation selection effects, we are simply conditioning on the fact that our species has developed the ability to conduct this analysis. The observerhood property need not invoke consciousness or be the property of a biological species-a machine estimating a parameter would need to account for observer selection bias if its ability to make such estimates were correlated with the parameter in question. Model 1: Single step, constant rate. Our first model assumes that observerhood has a constant rate of occurrence θ, so that S is exponentially distributed with cumulative distribution function: F S (t) = 1 − e −θt . This model describes a process in which the transition from early humans into observers occurs by chance as a single step. This could represent the hypothesis that hierarchical language emerged in humans as the byproduct of a chance mutation  24  . With this model, the probability that observers arrive before extinction is P(T > S) = θ(θ + μ) −1 . Our likelihood function can be analytically derived:   ∫ μ θ μ θ μ | > =      +      − μ θ ∞ − − ⁎ T t e e ds ( ) (1 ) (4) t s s θ μ θ μ θ =      +      −          μ μθ − −+ e e ( 5 ) t t ( ) Model 2: single step, increasing rate. Our second model similarly assumes that a single step is needed but that the rate of observerhood increases over time. This model could represent increasing population size or population density, which could in turn drive cultural evolution and increase the probability of such a step  25  . We represent this with a Weibull distribution with cumulative distribution function = − θ − F t e ( ) 1 S t ( ) k where k > 1 indicates increasing rate over time (when k = 1, this is the same as the exponential in Model 1). We use numerical integration to evaluate the likelihood function. Model 3: multiple steps, constant rate. Our third model assumes that there are multiple steps that need to occur in a sequence in order to get observers. This could represent more incremental development of tools, culture, or language. We assume that each step is exponentially distributed with rate θ, so that the timing of the final kth step follows an Erlang distribution with cumulative distribution function: ∑ θ = − . θ = − − F t n e t ( ) 1 1 ! ( ) (6) S n k t n 0 1 Note that when k = 1, the distribution is the same as the exponential in Model 1. We use numerical integration to evaluate the likelihood function. Model 4: fixed time requirement. Our final model assumes that it takes a fixed amount of time τ to reach observerhood. This is an extreme model that allows for no chance, but could represent a gradual and deterministic accumulation of traits. The probability that observerhood has been reached before time t is therefore F S (t) = 1 [t>τ] , the characteristic function that takes the value 1 when t > τ and 0 otherwise. The probability that humanity survives past time τ is 1 − F T (τ) = e −μτ . Our likelihood function of μ is: ∫ μ μ | > = μτ μ τ − ∞ − > ⁎ T t e e d s ( ) 1 1 (7) t s s [ ]  = . μ τ − − e (8) t ( ) This likelihood expression can also be derived using the memoryless property of the exponential. It is worth noting that the fixed time model is a limiting case for both the increasing rate model and the multiple steps model. Taking the limit of Model 2 as k → ∞ results in a fixed time model with τ = θ −1 . Similarly, Model 3 converges to a fixed time model as the number of steps increases and the expected time of each step decreases (having infinitely many steps in the limit, each of which is infinitely short).
 Results of sample bias models. We evaluate the likelihood of extinction rates between 10 −8 and 10 −2 , given a human survival time of 200 kyr and a wide range of different rates at which observers could originate (Fig.  2 ). The first thing to note about the first three models is that when the observerhood rates are sufficiently rapid, the likelihood function converges to the unbiased version in the previous section. This can be verified by taking limits: for all of the models as θ → ∞ (or τ → 0 in the case of the fixed time model ),  μ| > → μ − ⁎ T t e ( ) t . If observerhood is expected to occur quickly, then we can take a 200 kyr track record of survival at face value and estimate the extinction rate without observation selection bias. However, as the observerhood rates decrease to the point where the expected observerhood time approaches an order of magnitude close to 200 kyr, observer selection bias emerges. Rates that were previously ruled out by our track record of survival are assigned higher likelihoods, since a portion of the track record is a necessity for observers (Fig.  2 ). For example in Model 1, when θ = 2 × 10 −4 (corresponding to an expected observerhood time of 20 kyr), the relative likelihood of μ = 6.9 × 10 −5 is increased by a factor of 2.3 (from 10 −6 to 2.3 × 10 −6 ). To get a likelihood of 10 −6 (corresponding to the most conservative upper bound), the rate must be set at 7.3 × 10 −5 (see all edited bounds in Table  2 ). Interestingly though, this effect is limited. Even as observerhood rates slow to the point where expected observerhood time greatly exceeds 200 kyr (for example exceeding 20 billion years), the revised upper bounds remain within a factor of 2 of the original bounds. The stricter the bound, the weaker the potential bias: for example the 10 −6 likelihood bound is only changed by a factor of about 1.2 in the limit as θ → 0. Although there would be some sample bias, there is a hard ceiling on how much our track record of survival can be distorted by observation selection effects. The reason slow rates of observerhood have a limited impact on our estimates is because if the extinction rate were exceptionally high, the lucky humans that do successfully survive to observerhood will have achieved such a status unusually quickly, and therefore will still observe a very short track record of survival. A long track record of survival is therefore still sufficient to rule out high extinction rates paired with low observerhood rates. We can demonstrate this by examining the typical time it takes for lucky survivors to reach observerhood, assuming a high extinction rate and a low observerhood rate. For example, in the single step constant rate model when www.nature.com/scientificreports www.nature.com/scientificreports/ θ = 10 −6 (corresponding to an expected observerhood time of 1 Myr) and μ = 10 −3 (corresponding to a typical extinction time of 1000 years), the expected observerhood time conditional on these high extinction rates is 1000 years. A typical observer will thus still have a very short track record of survival. Models with increasing rates or multiple steps exhibit the same property, although the bias is larger depending on parameter k. For both model 2 and 3 with θ = 10 −6 , μ = 10 −3 , and k = 2 (parameters normally corresponding to an expected observerhood time of 830 kyr for Model 2 and 2 Myr for model 3), the high extinction rates will still result in a typical observer emerging unusually early and having only about a 2000 year track record of survival. This can be also seen in Fig.  2  where for Models 1, 2, and 3, the likelihood of high extinction rates exceeding 10 −4 are still assigned low likelihood regardless of θ. However, severe observer selection bias can occur in Models 2 and 3 as k becomes larger, shaping the observerhood distribution such that early observerhood is vanishingly unlikely and late observerhood almost guaranteed. In the most extreme case this is represented by the fixed time model, where the probability of observerhood jumps from 0 to 1 when t = τ (the fixed time model is also the limiting case when k → ∞). If that fixed amount of time is long enough (say, exceeding 190 or 195 kyr), a 200 kyr track record of survival is no longer sufficient to rule out extinction rates greater than 10 −4 . This result occurs as the fixed time model prohibits any possibility of observerhood occurring unusually quickly. Any lineage of Homo sapiens lucky enough to survive long enough to obtain observer status must necessarily have a survival time greater than τ, which means that being an observer with a survival time of τ conveys zero information about the extinction rate. For numerous reasons, we find the fixed time model to be implausible. Virtually all biological and cultural processes involve some degree of contingency, and there is no fundamental reason to think that gaining the ability to make scientific observations would be any different. To illustrate a comparison, let us consider a world in which For the first three models, the unbiased model is recovered for large θ, and results start to become biased as the expected observerhood time approaches humanity's track record of survival. However, even as θ → 0, the bias is limited, and the likelihood of rates exceeding 10 −4 remains at zero. This is only violated in the final fixed time model, or in models 2 and 3 when k is sufficiently large.  www.nature.com/scientificreports www.nature.com/scientificreports/ the extinction rate is −4 (averaging one extinction every 10,000 years), but observerhood status takes a fixed 200 kyr. Under this model, humanity successfully surviving long enough to reach observer status is an event with 1 in 200 million chance. Given observation selection bias, we cannot rule out the possibility of rare events that are required for our observations. But we could ask why a 1 in 200 million chance event could not also include the possibility that modern human observers would emerge unusually rapidly. Language, writing, and modern science are perhaps highly unlikely to develop within ten thousand years of the first modern humans, but it seems exceptionally overconfident to put the odds at fewer than 1 in 200 million. A similar line of reasoning can be applied to determine whether the increasing rate and multiple step models with high k are reasonable. We test this by asking what parameters would be needed to expect a 200 kyr track record of survival with an extinction rate at our conservative upper bound of μ = 6.9 × 10 −5 . For the increasing rate model, observerhood is expected after 203 kyr with θ = 10 −7 and k = 14 and for the multiple step model, observerhood is expected after 190 kyr with θ = 10 −7 and k = 16. Although these models do not assign strictly zero probability to early observerhood times, the probabilities are still vanishingly small. With an increasing rate and these parameters, observerhood has less than a one in a trillion chance of occurring within 10,000 years (3.4 × 10 −14 ), and about 1% chance of occurring within 100,000 years. With multiple steps and these parameters, observerhood has less than one in a trillion chance of occurring within 10,000 years (5.6 × 10 −17 ), and less than a 0.02% chance of occurring within 100,000 years. In a similar fashion to the fixed time model, we feel that these models exhibit unrealistic levels of confidence in late observerhood times. Although the plausibility of the fixed time (or nearly fixed time) models is hard to test directly, the wide variance in the emergence of modern human behavior across geography offers one source of data that can test their plausibility. The Upper Palaeolithic transition occurred about 45 kya in Europe and Western Asia, marked by the widespread emergence of modern human behaviour  25  (e.g. symbolic artwork, geometric blades, ornamentation). But strong evidence exists for the sporadic appearance of this modern human behaviour much earlier in parts of Africa  26, 27  , including evidence of artwork and advanced tools as early as 164 kya  28  . Although numerous factors could have prevented the Upper Palaeolithic transition from occurring quickly, the fact that some human communities made this transition more than 100 kyr earlier than the rest of humanity indicates that a much earlier development trajectory is not entirely out of the question. In summary, observer selection effects are unlikely to introduce major bias to our track record of survival as long as we allow for the possibility of early observers. Deceptively long track records of survival can occur if the probability of early observers is exceptionally low, but we find these models implausible. The wide variance in modern human behavior is one source of data that suggests our track record is unlikely to be severely biased. We can also turn to other sources of indirect data to test for observer selection bias.
 testing the Bound with indirect Data We cross check our upper bound against four other sources of data: mammalian extinction rates, survival times of other human species, rates of potential catastrophes, and mass extinction rates. Although these alternative data do not directly predict the background extinction rate of Homo sapiens per se, the rates of extinction are likely generated by similar processes and thus enable an indirect test of the upper bound. If our upper bound is sound (not biased by observer selection effects or otherwise flawed), we can make testable predictions that it will be (A) broadly consistent with the extinction rates for similar species, and (B) not exceeded by the rate of potential catastrophes or mass extinctions. As the extinction rate of other species and catastrophes many millions of years ago have little bearing on our ability to make scientific observations, these data are also less subject to potential observer selection bias.
 Mammalian extinction rates. We first evaluate whether the upper bound is consistent with extinction rates for a typical mammalian species. Using fossil record data, median extinction rates for mammals have been estimated as high as 1.8 extinctions per million species years (E/MSY) 2 , or equivalently μ = 1.8 × 10 −6 . Other estimates using fossil record data range from 0.165 extinctions per million genus years 17 to 0.4 E/MSY for Cenozoic mammals  18  . Alternative methods using molecular phylogeny suggest a much lower rate of 0.023 E/MSY for mammals  29  and rates of 0.219-0.359 E/MSY for primates  30  , although these methods have been criticized  31  . All of these estimated background rates are consistent with our upper bound. It is worth noting that Homo sapiens may be at lower extinction risk than a typical mammalian species due to a large habitat range, large population size, and having a generalist diet, which are all traits that militate against extinction risk (whereas long generation times and large body mass are sometimes correlated with increased extinction risk)  32, 33  . Hominin survival times. Next, we evaluate whether the upper bound is consistent with the broader hominin fossil record. There is strong evidence that Homo erectus lasted over 1.7 Myr and Homo habilis lasted 700 kyr  21  , indicating that our own species' track record of survival exceeding 200 kyr is not unique within our genus. Fossil record data indicate that the median hominin temporal range is about 620 kyr, and after accounting for sample bias in the fossil record this estimate rises to 970 kyr  22  . Although it is notable that the hominin lineage seems to have a higher extinction rate than those typical of mammals, these values are still consistent with our upper bound. It is perhaps also notable that some hominin species were likely driven to extinction by our own lineage  34  , suggesting an early form of anthropogenic extinction risk. individual sources of extinction risk. The upper bound can also be evaluated against the frequency of events that could pose extinction risks (examples provided in Table  3 ). If any particular risk (such as those from asteroid impacts) is known to have a higher rate than our bound of 6.9 × 10 −5 , this could undermine and potentially falsify our hypothesis. We evaluate the frequencies of four types of potential disasters for which credible quantitative estimates exist: asteroid impacts, supervolcanic eruptions, stellar explosions, and vacuum collapse. www.nature.com/scientificreports www.nature.com/scientificreports/ All of these risks have estimated to occur with a frequency well below our bound (Table  3 ), with the exception of smaller supervolcanic eruptions. Recent work has suggested the frequency of eruptions ejecting >10 3 km 3 of material exceeds our upper bound of 6.9 × 10 −5 with a recurrence time of 17 kyr  35  . However, it is important to note that the smaller eruptions within this category do not necessarily have a high probability of causing human extinction. The most severe eruption of the past 2 million years occurred just 74 kya, and it is unclear whether the human population at the time was at risk of extinction. Some argue that the human population suffered a major bottleneck at the same time as the eruption  43  , although this theory remains controversial  44  . Some climate records averaged over decades fail to observe a severe volcanic winter in Africa at the time  45  and archaeological evidence shows that human communities in South Africa thrived both before and after the eruption  46  (although these data are not sufficient to rule out a severe short-lived catastrophe followed by a fast recovery in population). More conclusively, most members of the Hominidae family did not suffer population bottlenecks around the time, with the possible exception of Eastern chimpanzees and Sumatran orangutans  47  . The lack of dramatic evidence suggesting other species extinctions or bottlenecks undercuts the possibility that humanity's survival was highly improbable and is observed only due to observation selection effects. However, a handful of substantially larger flood basalt events have taken place over the past 250 Myr that have been linked to mass extinctions  39, 48  . These events occur with a frequency of roughly once every 20-30 Myr, much more infrequently than smaller eruptions. If we assume that human extinction is threatened only from larger volcanic eruptions well exceeding 10 3 km 3 , then none of the risk frequencies we have catalogued come within an order of magnitude of the conservative upper bound. Similarly, impacts from smaller asteroid around 1 km in diameter may not have a high probability of causing human extinction. Although it is hard to estimate the consequences of such impacts, some researchers have argued that such impacts would fall below the threshold for a global catastrophe  49  . Impacts that disperse enough dust and sulphites to significantly disrupt photosynthesis occur much more rarely, with an estimated frequency of about 15 Myr years  49  . If we assume human extinction is only threatened by these more severe impacts exceeding 5 km, each of these catastrophe frequencies falls well below even our most optimistic bound of 1 in 870,000 chance of extinction per year.
 Mass extinction frequency. A mass extinction is marked by substantially increased extinction of multiple geographically widespread taxa over a relatively short period of time  50  . There have been five major mass extinctions in the past 541 Myr  51, 52  , with many arguing that human activity is currently causing a sixth  2  . In a similar way to our previous analysis of catastrophe rates, we should expect our upper bound to be consistent with the frequency of non-anthropogenic mass extinctions. Using only the big five extinctions produces a frequency of less than one per 100 Myr, far below our upper bound. In addition to the big five, there have been 13 other mass extinctions in the fossil record  53  . Using these numbers for 18 mass extinctions over 541 Myr still results in a frequency of about one per 30 Myr, many orders of magnitude below our upper bound.
 conclusions Using the fact that humans have survived at least 200 kyr, we can infer that the annual probability of human extinction from natural causes is less than 1 in 87,000 with modest confidence (0.1 relative likelihood) and less than 1 in 14,000 with near certainty (10 −6 relative likelihood). These are the most conservative bounds. Estimates based on older fossils such as the ones found in Morocco dated to 315 kya result in annual extinction probabilities of less than 1 in 137,000 or 1 in 23,000 (for relative likelihood of 0.1 and 10 −6 , respectively). Using the track record of survival for the entire lineage of Homo, the annual probability of extinction from natural causes falls below 1 in 870,000 (relative likelihood of 0.1). We also conclude that these data are unlikely to be biased by observer selection effects, especially given that the bounds are consistent with mammalian extinction rates, the temporal range of other hominin species, and the frequency of potential catastrophes and mass extinctions. The bounds are subject to important limitations. Most importantly, they only apply to extinction risks that have either remained constant or declined over human history. Our 200 kyr track record of survival cannot rule out much higher extinction probabilities from modern sources such as nuclear weapons or anthropogenic climate change. Some naturally occurring risks could be also be worsened by anthropogenic factors: a minor asteroid impact could be interpreted as a nuclear attack and lead to retaliation  54  , or a naturally occurring disease which previously may have only been a local extinction risk could spread much further due to modern travel  23  . In the cases where a natural risk is amplified by modern conditions, we can still derive some partial information from the upper bound by evaluating how much the risk would need to change from the purely natural baseline. For  www.nature.com/scientificreports www.nature.com/scientificreports/ example, the claim that a natural disease poses a than 1 in 1,000 chance of extinction per year would require that anthropogenic conditions have increased the risk of natural disease by a factor of more than 14 to 870 (under our most conservative and optimistic upper bounds, respectively). In general, for a naturally occurring risk to violate our upper bounds via human activity by more than a factor of two, the majority of the risk would still need to come from anthropogenic circumstances. In general, we conclude that anthropogenic extinction risks are likely greater than natural ones. We do not have a long track record of data for anthropogenic risks, so evaluating this relies far more on speculation. But despite the paucity of data, the little evidence we do have seems to be indicative of rates greatly exceeding our upper bounds. During the Cuban Missile Crisis of 1962, John F Kennedy put the odds of nuclear war at 'somewhere between one out of three and even'  55  . If 0.1% of nuclear wars result in human extinction via nuclear winter, taking Kennedy's odds that year would surpass our most conservative bound by more than a factor of four (and surpass our most optimistic bound by a factor of more than 250). Anthropogenic climate change could pose existential risks as well if warming is much worse than expected. A ballpark suggestion for the probability of 20 degrees of anthropogenic climate change was placed at 1%  56  , which would make the planet largely uninhabitable for humans due to heat stress  57  . And these are not the only risks we may face. One century ago, the existential risks posed by nuclear weapons or climate change may have seemed extremely implausible. We should therefore be cautious before dismissing the potential risks that future centuries of technological development could bring, such as those stemming from biotechnology  58  or artificial general intelligence  59  . Despite the low probability of human extinction from natural causes, it may still be prudent to reduce these risks. Existential risks jeopardize not only the lives of those currently present, but also the existence of all future generations. Depending on how much value we place on such generations, it may still be cost-effective to reduce existential risks from natural sources  60  . However, given limited resources to spend on reducing existential risks, one may be better off focusing on greater risks from our own design. Figure 1 . 1 Figure1. Likelihood of extinction rates given our track record of survival so far, with estimated ranges of Hominin extinction rates, mammalian extinction rates, and mass extinction frequency included for reference. Blue horizontal lines indicate likelihood of 10% and 1%. Rates exceeding 6.9 × 10 −5 are ruled out even with the most conservative data. Extending humanity's track record of survival to match older fossils, the divergence with Homo neanderthalensis, or the origin of Homo creates even stricter bounds.
 Figure 2 . 2 Figure 2. Models of observer selection bias. Surface plots show likelihood for combinations of μ and θ (where k = 3 for Models 2 and 3) or τ in Model 4. Upper righthand plots show how likelihood shifts when θ → 0 in Model 1, and for a variety of k values in Models 2 and 3.For the first three models, the unbiased model is recovered for large θ, and results start to become biased as the expected observerhood time approaches humanity's track record of survival. However, even as θ → 0, the bias is limited, and the likelihood of rates exceeding 10 −4 remains at zero. This is only violated in the final fixed time model, or in models 2 and 3 when k is sufficiently large.
 Table 1 . 1 Survival times and resulting upper bounds. www.nature.com/scientificreports www.nature.com/scientificreports/ 
 Table 2 . 2 Upper bounds of μ with model 1 bias.
 Table 3 . 3 Catastrophe frequency estimates.

Abstract: We evaluate the total probability of human extinction from naturally occurring processes. Such processes include risks that are well characterized such as asteroid impacts and supervolcanic eruptions, as well as risks that remain unknown. Using only the information that Homo sapiens has existed at least 200,000 years, we conclude that the probability that humanity goes extinct from natural causes in any given year is almost guaranteed to be less than one in 14,000, and likely to be less than one in 87,000. Using the longer track record of survival for our entire genus Homo produces even tighter bounds, with an annual probability of natural extinction likely below one in 870,000. These bounds are unlikely to be affected by possible survivorship bias in the data, and are consistent with mammalian extinction rates, typical hominin species lifespans, the frequency of well-characterized risks, and the frequency of mass extinctions. no similar guarantee can be made for risks that our ancestors did not face, such as anthropogenic climate change or nuclear/biological warfare. Out of all species that have existed, over 99% are now extinct 1 . Although human activity is dramatically increasing extinction rates for many species 2 , species extinctions were regular occurrences long before humanity emerged. Many of these extinctions were caused by gradual environmental shifts, evolutionary arms races, or local interspecific competition 3,4 , while others were abrupt, being part of global mass extinctions caused by asteroid impacts, volcanism, or causes as of yet to be identified  5, 6  . Could such a catastrophe befall our own species? If so, are the risks greater from natural or anthropogenic sources? Here, we evaluate the natural 'background' extinction rate for Homo sapiens. This means considerations of anthropogenic risks such as climate change and nuclear weapons are excluded from our estimates, although these clearly pose existential threats to our own species as well as others. Indeed, it has been hypothesized that the great majority of human extinction risk comes from anthropogenic sources  7, 8  . But by limiting our analysis to natural risks that our predecessors also faced, we can draw on data spanning many thousands (or millions) of years. Obtaining bounds on natural extinction rates also enables an indirect and partial test of the hypothesis that anthropogenic risks are greater than natural ones, as sufficiently low natural extinction risk will imply higher relative risks from anthropogenic sources. Estimating such an extinction rate directly is impossible. We have no examples of Homo sapiens extinction, so the most directly relevant data are non-existent. An alternative approach would be to enumerate the different types of naturally occurring hazards (e.g. asteroids, supervolcanoes), estimate their independent probability of causing extinction, and then use these probabilities to derive an aggregate extinction rate. However, this method has its own shortcomings. Beyond the great uncertainties around the probabilities of each risk, there could also be unknown risks that fail to be included. It would be hard to say with confidence that any list of risks had captured all natural hazards to humanity. We can bypass these problems by instead considering the length of time that humanity has survived so far 9,10 . This survival time can be used to estimate an upper bound on the extinction rate from all natural sources combined, including from sources for which we remain unaware. However, this approach could be subject to a particular form of sample bias known as an observation selection bias. These observer selection biases occur when a sample is not representative of all outcomes, but rather a subset of outcomes that are compatible with the existence of the observers 11 . For example, if human existence required a 10 million year (Myr) period of evolution free from asteroid impacts, any human observers will necessarily find in their evolutionary history a period of 10 Myr that is free of asteroid impacts, regardless of the true impact rate. Inferring a rate based on those 10 Myr could therefore be misleading, and methods must to be used to correct for this bias 12 .


The Ethics of Sustainability for Artificial Intelligence

Introduction A basic attribute of modern human civilization is that the stock of natural resources steadily decreases, whereas the stock of artificial resources steadily increases. For example, artificial intelligence (AI) research is commonly powered by the burning of fossil fuels, and in the process produces new technologies that civilization can benefit from. Will the increases in artificial resources be sufficient to offset the loss of natural resources, such that civilization can be sustained into the future? That is one important perspective on the ethics of sustainability as it relates to AI, though, as this paper discusses, it is not the only one. Sustainability is not an inherently ethical concept. In its essence, "sustainability" refers to a particular characteristic of systems as they change over time. The term can be used in many ways that do not have any particular ethical significance. For example, sprinters run at an unsustainable speed; eventually, their muscles will fatigue and they will be unable to continue. This is a basic characteristic of human physiology and not a matter of ethical significance. In common usage, however, sustainability takes on ethical significance. Sustainability is widely treated as a good thing and something worth pursuing  [1] . It is in that spirit that there have been initiatives on AI and sustainability, including conferences such as "Sustainable AI",  1  "Towards Sustainable AI",  2  and "AI for the Planet",  3  as well as groups such as AI4Good that work on AI in support of the United Nations Sustainable Development Goals (SDGs).  4  It is also in that spirit that sustainability is one of the principles found in some AI ethics guidelines  [2] . The ethical dimensions of sustainability warrant ethical analysis. Important ethics questions include: What exactly should be sustained, or rather, be able to be sustained? Why should it be able to be sustained? For how long? How much emphasis should be placed on sustainability relative to other goals? These ethics questions can be answered in a variety of ways. How they are answered has important implications for ongoing human activity, including for the development, use, and governance of AI technology. Prior literature on AI and sustainability (reviewed below) has not considered the ethical dimensions to any significant extent. Therefore, this paper analyzes the ethics of sustainability as it relates to AI. The paper proceeds in three parts. First, we explain the ethics of sustainability as a general concept (Section 2). Second, we describe the current usage of the concept of sustainability within work on AI (Section 3). Third, we present an argument for a long-term, non-anthropocentric conception of sustainability and explain the implications of this for AI (Section 4). Section 5 concludes. Much of the discussion in this paper covers ethical dimensions of sustainability that are not specific to AI. This is a feature, not a bug. To a large extent, the ethical dimensions of AI and sustainability are the same as those for sustainability in general. Furthermore, aspects of the topic that are specific to AI build on more general sustainability concepts. Therefore, to understand the ethics of sustainability for AI, it is essential to first understand the ethics of sustainability. The paper contributes to the growing literature on AI and sustainability. Most of this literature is on AI in relation to the sustainability of human civilization and its environmental underpinnings; this includes reviews by Nishant et al.  [3]  and Liao and Wang  [4] , the environmental politics of AI  [5] , AI in relation to systemic risk and sustainability  [6] , the environmental footprint of AI systems  [7] , and the role of AI in meeting the SDGs  [8] [9] [10] . Some literature focuses on the sustainability of the AI systems themselves  [11] , including for consumer autonomy  [12] , in global health initiatives  [13] , in certain economic mechanisms  [14] , and in decision-making applications  [15] . Additionally, there is a broader field of computational sustainability that applies computer science methods to advance environmental and social sustainability  [16] [17] [18] . Overall, the literature on AI and sustainability offers a variety of important contributions, but it provides limited discussion of the ethics of sustainability. The paper additionally contributes to some adjacent literatures. Prior studies have considered the application of AI for environmental protection  [19] [20] , for climate change mitigation  [21] [22] , and the energy consumption of AI systems  [23] ; these are relevant for environmental conceptions of sustainability. Also relevant are debates on the relative importance of near-term, medium-term, and long-term AI  [24] [25] [26] [27] ; as this paper discusses, the future-orientation of sustainability can imply an emphasis on long-term AI. Finally, of more general relevance is prior work on the ethics of sustainability  [1] , especially sustainability over 1 https://www.uni-bonn.de/en/news/120-2021 2 http://www.wikicfp.com/cfp/servlet/event.showcfp?eventid=140164&copyownerid=158619 3 https://aifortheplanet.org/ 4 https://ai4good.org/ long time scales  [28] , and the ethics of AI  [2, [29] [30] , especially regarding AI and the future  [31]  and AI and nonhumans  [32] .
 The Ethics of Sustainability The word "sustainability" is commonly traced to the German word Nachhaltigkeit, and specifically to Hans Carl von Carlowitz's 1713 treatise on sustainable yield forestry  [33] . The tension between using resources now and preserving or cultivating them for future use is of course much older. Modern analysis traces to the environmental economics work of Hotelling  [34] , which remains relevant today  [35] . Hotelling's work did not use the term "sustainability". Use of the term primarily traces to the 1987 report Our Common Future, which was led by former Norwegian Prime Minister Gro Harlem Brundtland and is commonly known as the Brundtland Report. Like von Carlowitz's treatise, the Brundtland Report specifically conceptualizes sustainability in socio-environmental terms. The report's definition of sustainable development, "Development that meets the needs of the present without compromising the ability of future generations to meet their needs", has been widely influential and serves as a foundation for the UN SDGs. Since Our Common Future, there has been a proliferation of definitions of sustainability and sustainable development  [36] [37] [38] . Widespread and imprecise usage of the term "sustainability" has watered it down. Critics argue that "sustainability" has become "a concept that is equivalent to 'good' and thus devoid of any specific meaning-a blanket concept to assure stakeholders of the policy's good intentions"  [39, p.3439 ]. Likewise, the term is said to have been appropriated by self-interested actors to continue "business as usual" activities that drive environmental destruction and social inequity  [37] . One notable point of criticism is the so-called "three pillars" of sustainability: the social, the economic, and the environmental-or "people, profit, and planet". The pillars construct is an attempt to represent the major components of socio-environmental sustainability. However, these categories have been criticized for being fuzzy and overlapping, for excluding of other relevant categories such as the cultural and the political, and for not being essential to the core matter of whether civilization can be sustained over time  [39] . Indeed, economic transactions are an inherently social activity, and all human activity is inherently environmental because humans are part of nature. Furthermore, sustainability is commonly associated with environmental protection, but some environmental disturbances, such as many types of air or water pollution, rapidly dissipate and have a negligible effect on future generations' ability to sustain themselves. Likewise, some matters that could classify as social and/or economic, such as social justice within the contemporary population, are often of limited relevance to the ability of future generations to sustain themselves, even if these matters may be important for other reasons. Other socio-economic matters, such as education and investment in future economic growth, may be of greater importance to the ability of future generations to sustain themselves. For these reasons, the "three pillars" provide a weak foundation for sustainability. Work on AI and sustainability that incorporates the "three pillars", such as the "AI for People: Towards Sustainable AI" conference, 5 should take note. We now turn to three fundamental questions for the ethics of sustainability (Sections 2.1-2.3) followed by a comparison between sustainability and optimization (Section 2.4). The reasons something should be able to be sustained typically derive from what are, in moral philosophy, referred to as intrinsic value and instrumental value. Roughly speaking, something is intrinsically valuable if it is valuable for its own sake, or valuable as an ultimate end in itself; something is instrumentally valuable if it is valuable because it promotes something else that is valuable  [40] . For example, we might suppose that sunlight is valuable because it can (among other things) be converted into electricity, and that electricity is valuable because it can (among other things) be used to power AI systems, and that AI systems are valuable because they can (among other things) enable humans to have enjoyable lives, and that enjoyable human lives are just good things on their own. In that case, sunlight, electricity, and AI systems are instrumentally valuable, while enjoyable human lives are intrinsically valuable. In many conceptions of ethics, what should be done-including what should be sustained-is defined with reference to some conception of what is intrinsically valuable.
 Conceptions of sustainability can vary in terms of what they intrinsically value and what sorts of instrumental values they focus on. One can seek to sustain X either because X is intrinsically valuable or because X is instrumentally valuable. For example, one can seek to sustain natural ecosystems either because one considers them to be intrinsically valuable or because one considers them to be instrumentally valuable for other things, such as for human welfare. Common conceptions of sustainability are anthropocentric, meaning that they only intrinsically value humans  [41] . For example, although the Brundtland Report's emphasis on future generations could conceivably be interpreted to mean future generations of something other than humans, the Report clearly focuses on humans. Likewise, sustainable management of natural resources generally treats the resources as instrumental values for human benefit. In contrast, some conceptions of sustainability are ecocentric, meaning that they intrinsically value ecosystems. Examples include the Earth Charter 6 and the Earth Manifesto  [42] . Though less common, the concept of sustainability can also be used with other notions of intrinsic value, such as the idea that there is intrinsic value in the welfare of sentient nonhuman animals or (if possible) sentient AI systems. The distinction between intrinsic and instrumental value does not always matter, but it often is important. For example, reducing greenhouse gas emissions is generally good on both anthropocentric and ecocentric grounds. However, some biodiversity protection is worth pursuing mainly on ecocentric grounds, because, for better or worse, certain species could go extinct with little impact on human welfare. Therefore, it is important for discussions of sustainability to explicitly specify what they intrinsically value.
 For how long should something be able to be sustained? There is a big difference between sustaining something for a few days and sustaining it for decades, centuries, or even indefinitely into the distant future. Unfortunately, discussions of sustainability are often not precise in their consideration of time scales. For example, the 1987 Brundtland Report's emphasis on future generations implies a time scale of at least decades, assuming the generations are of humans. But how many future generations? The actions needed to enable the next few generations to be sustained often differ significantly from the actions needed to enable the same for every generation that could ever exist.
 How much effort should be made for sustainability? The sustainability of intrinsic or instrumental values may be a good thing, but how good? The world has many competing values and opportunities. Indeed, the Brundtland definition was specifically crafted to acknowledge the competing values of present and future generations; while the report aspires to promote actions that support both the present and future generations, many choices involve tradeoffs between them. For example, depleting natural resources often benefits present generations at the expense of future generations. Basic research often benefits future generations at the expense of present generations, especially where the same resources could instead be used for applied research. The relative importance of the present and future can be operationalized in a variety of ways, such as through discount rates or other weighting functions  [43] [44] . This sort of intergenerational evaluation is generally made within the context of anthropocentric conceptions of sustainability, but similar approaches can be taken with other conceptions. One way or another, moral guidance about sustainability must consider how to evaluate tradeoffs between sustainability and other moral goals. This point fits within the broader issue of tensions between different AI ethics principles  [45] . Also relevant are fundamental questions about the appropriate degree of effort to take to achieve ethical goals. In moral philosophy, the term "supererogation" refers to actions that "go beyond the call of duty", meaning that they are good but not strictly required  [46] . One common question in moral philosophy is whether some moral frameworks are too demanding. This question is especially acute for consequentialist moral frameworks that call for moral agents to maximize some conception of intrinsic value, because maximization is a demanding task  [47] . These are important questions for any moral debate, certainly including those involving sustainability. Setting aside tradeoffs between sustainability and other moral goals, one can ask: How much effort should a person or an organization make to advance sustainability? Should they "give it everything they've got"? Or would just a little effort be acceptable? Conversely, is it enough to work to advance sustainability? Or is it important to also work toward more ambitious goals, such as intertemporal optimization of intrinsic value?
 Sustainability vs. Optimization Sustainability can be an optimization criterion-that would mean seeking to optimize the ability for something to be sustained over time. However, this is distinct from optimizing intrinsic value. Sustainability means enabling something to be sustained in at least some minimal form; optimization means making something be the best that it can be. Ensuring sustainability is perhaps best understood as a basic minimum standard of intertemporal conduct, whereas the intertemporal optimization of intrinsic value may be understood as a loftier ideal to aspire for. This distinction can be seen, for example, in the Brundtland Report call for the present generation to act "without compromising the ability of future generations to meet their needs". The basic needs of human life are, to an approximation, food, clothing, and shelter. Following the Report's call could result in "a society living forever at a minimum subsistence level of consumption"  [48, p.327] . Future society would be able to meet its needs, but it may not be able to do anything more. If the present generation does not act so as to enable future generations to do much better than meeting their needs, then the present generation will, quite arguably, have squandered a massive opportunity. Of course, if the present generation fails to enable future generations to meet their needs, that would be, quite arguably, a massive loss. AI is advanced technology. AI research and development is often oriented toward enabling higher standards of living instead of enabling basic future needs. Human lives do not strictly need, for example, AI systems to steer vehicles or search the internet. Such work generally falls outside the scope of sustainability, but could fall within the scope of optimizing intrinsic value.
 Prior Work on AI and Sustainability With the ethics of sustainability in mind, we now survey prior work on AI and sustainability. Sections 3.1 and 3.2 present quantitative analysis of trends in the ethics of sustainability found in AI ethics principles and AI research. Both analyses characterize sustainability in terms of the three ethics dimensions presented in Section 2: intrinsic value, time scale, and degree of effort. Section 3.3 presents overarching trends across Sections 3.1-3.2.
 AI Ethics Principles Jobin et al.  [2]  compiles 84 sets of AI ethics principles. 11 of these sets of principles include some reference to sustainability.  7  This indicates that sustainability is a small but nonzero priority in AI ethics. We examined the ethical basis of the 11 sets of principles. We found that 7 refer to some form of environmental sustainability, 3 refer to sustainability of the AI system itself, and 1 refers to sustainable social development. We further found that 3 intrinsically value humans, 5 intrinsically value humans and nonhumans including ecosystems, all life, biodiversity, and the planet, and 5 are ambiguous in terms of what is intrinsically valued. Regarding time scales, 2 refer to "future generations" and 9 do not specify time scales. Finally, none of the principles specify degree of effort. Some examples of the AI ethics principles are presented in Appendix A.
 AI Sustainability Research We performed a systematic mapping review  [49]  of research at the intersection of AI and sustainability. Our review maps this literature in terms of the ethical attributes presented in Section 2 and also used in Section 3.1, with some additional nuances to catch the diversity of the research literature. Specifically, we analyzed results from a Google Scholar search for ["artificial intelligence" "sustainability"] and ["AI" "sustainability"] conducted between Sept 16 and 21, 2021. The Google Scholar search engine was selected over other academic databases due to its inclusivity. Whereas databases such as Web of Science concentrate on 7 Table  3  of Jobin et al.  [2]  states that 14 sets if principles include sustainability, but only 12 are referenced in the text and we were unable to identify the other 2. Of the 12 referenced sets of principles, we found that 1 did not cover sustainability, leaving a total of 11 for our analysis. peer-reviewed journals, artificial intelligence research is often published in other spaces such as arXiv. The searches returned 229,000 total results for ["artificial intelligence" "sustainability"] and 1,490,000 total results for ["AI" "sustainability"], respectively. We examined the first ten pages of each of the two searches. We observed that after ten pages of each search, the search results became repetitive and less relevant. These two sets of ten pages contained 200 total results, or 153 results after duplicates were extracted. Of these 153 publications, we were unable to access 11. For the remaining 142 publications, we examined the text in the degree of detail needed to categorize its treatment of sustainability. For most of the publications, this involved looking at the abstract and introduction, and skimming the text for discussion of sustainability. In some cases, we examined the entire publication in more detail. Out of the 142 publications, 60 were found not to be relevant because they were not sufficiently on the nexus of AI and sustainability, leaving a data set of 82 publications. 66 publications were on environmental sustainability, 7 were on the sustainability of the AI system itself, and 9 were on the sustainability of something else, including 2 on the sustainability of organizations, 2 on social sustainability in human-robot/AI interactions, 1 on the sustainability of group decision-making processes, 1 on sustainable curriculum planning, 1 on sustainable healthcare systems, 1 on the social sustainability of AI, and 1 on sustainable industrial development. Of the 66 environmental sustainability publications, 43 were on environmental and social sustainability and 23 were exclusively on environmental sustainability. 29 of the 66 environmental sustainability publications referred to the Brundtland definition and/or the SDGs. 56 publications intrinsically valued humans only, 10 intrinsically valued humans and nonhumans including nonhuman species, life on Earth, biodiversity, ecosystems, the biosphere, and the planet. 16 were too ambiguous to interpret any notion of intrinsic value. Regarding time scales, 7 refer to "future generations" and 1 refers to a time frame from 1990 to 2028. The other 74 do not specify time scales. None of the publications specify degree of effort. Some examples of the AI sustainability publications are presented in Appendix B.
 Overarching Trends In consideration of the work presented in the two preceding subsections, the following overarching trends in the ethics of existing work on AI and sustainability can be identified. First, most work on AI and sustainability is focused on some form of environmental sustainability, with substantial minorities focused on the sustainability of AI systems or on the sustainability of miscellaneous other things. The environmental sustainability work mainly intrinsically values humans and sometimes intrinsically values nonhumans. These trends are consistent with wider usage of sustainability outside the context of AI. Indeed, work on AI and sustainability often explicitly links to broader treatments of sustainability, such as the Brundtland Report and the UN SDGs. Work on the sustainability of AI systems treats AI systems as instrumentally valuable, such as in decision-making applications  [15]  and in global health initiatives  [13] . Outside the context of sustainability, some research considers that AI systems could be intrinsically valuable  [50] [51] [52] . We find that the sustainability of intrinsically valuable AI systems has not yet been addressed. Second, work on AI and sustainability is imprecise on its ethical dimensions. As illustrated in Appendices A-B, our classification of AI ethics principles and AI sustainability research involved frequent parsing of ambiguous phrasings. Indeed, outside references to the SDGs or the Brundtland definition, few publications explicitly define sustainability. Within treatments of environmental sustainability, the term "sustainability" was commonly equated with environmental protection, especially efforts to minimize energy and resource consumption, even though environmental issues do not necessarily have sustainability implications. Some work equated "sustainability" with "good for people/and or society" or even just "good", which further drains "sustainability" of its meaning. As discussed in Section 2, these are common problems with sustainability discourse. Our analysis finds that these problems have been reproduced in the AI literature.
 The Moral Case for Long-Term, Non-Anthropocentric Sustainability and Optimization In this section, we present our own views on the ethics of sustainability as it relates to AI. Specifically, we make the case for sustainability that is non-anthropocentric and long-term oriented. We further argue for substantial effort for this conception of sustainability, or, preferably, for the optimization of long-term intrinsic value. Finally, we explain the implications of such sustainability for AI. Each of our arguments depends on certain positions on underlying ethical principles. As with all ethical principles, there is no universal consensus on which position to take. One can disagree with the positions we take, but doing so requires taking a different position on the underlying ethical principles. This is important to bear in mind, especially when considering the implications for AI.
 Non-Anthropocentric Sustainability First, we call for non-anthropocentric conceptions of sustainability. By non-anthropocentric, we mean that humans are not the only entities that are intrinsically valued. Modern science unambiguously shows that humans are members of the animal kingdom and part of nature. Morally significant attributes such as the innate drive to live and flourish or to experience pleasure and pain are not unique to the human species.  8  For purposes of this paper, we set aside important debates about which nonhuman entities to intrinsically value, but some potential examples include sentient nonhuman animals or (if possible) sentient AI systems, natural ecosystems, and biodiversity. We, the authors of this paper, also happen to disagree among ourselves as to which nonhumans are intrinsically valuable, but we agree that some are. There can be legitimate reasons to sometimes intrinsically value humans more than other entities-for example, a human can have a longer and richer life than a spider. However, we see no morally sound reasons to refuse to intrinsically value any nonhuman lives or entities. This means that sustainability should be defined as to also sustain some nonhumans for their own sake: it is not enough to sustain nonhumans for their instrumental role for humans.  9  8 Additionally, some conceptions of intrinsic value are rooted in the attributes of systems, such as interdependencies of biotic and abiotic entities within ecosystems. These holistic conceptions of intrinsic value are also not specific to humans  [53] . 9 For a more detailed argument for non-anthropocentrism advanced within AI ethics, see  [32] . This non-anthropocentrism is at odds with common conceptions of sustainability, including that of the Brundtland Report. In failing to intrinsically value anything other than humans, we believe these conceptions of sustainability are in moral error. It is unfortunate but not surprising that similar anthropocentric tendencies are found within existing work on AI and sustainability (Section 3). Future work on AI and sustainability should be more inclusive of the intrinsic value of nonhumans.
 Long-Term Sustainability Second, we call for sustainability over long time scales. Our motivation for this is an ethical principle of equality across time. In essence, this means that no one or no thing of moral significance should be disadvantaged because of the time in which they happen to exist. A person (for example) is of the same intrinsic value regardless of whether they live in the year 2021 or 2051 or 2151 or even 22021 or any other future time  [54] [55] . This perspective can be justified, for example, by a "veil of ignorance" thought experiment in which one does not know in advance which time period one would exist in  [56] . Under such hypothetical circumstances, it would only be fair to value each time period equally. Taking temporal equality seriously means including attention to all future time periods, including the astronomically distant future. Combined with our call for non-anthropocentrism, this means sustainability should aim to sustain that which is intrinsically valuable into the distant future. This long-termism is broadly consistent with common conceptions of sustainability, though it is different in emphasis. Common conceptions do not specify precise time scales; this is seen, for example, in the Brundtland Report's emphasis on an unspecified number of future generations. With no clear time limit, these conceptions could include the astronomically distant future, though in practice, they focus on matters that are short-term in comparison. We believe this is a moral error, an unjustified exclusion of distant-future generations and distant-future instances of anyone and anything else of intrinsic value.
 Substantial Effort for Sustainability or Long-Term Optimization Third, we call for a high degree of effort toward sustainability, or, preferably, for the optimization of long-term intrinsic value. The astronomically distant future offers astronomically large opportunities for advancing intrinsic value. These opportunities are vastly larger than those available for the present time and the near-term future. This point suggests a high degree of priority for actions oriented toward the long-term. That does not mean ignoring the present. As members of the present time period, we have special opportunities to help with present circumstances. The present also sets the stage for the future. Nonetheless, if the principle of equality across time is to be taken seriously, it requires a major focus on long-term outcomes. We further believe that people should make great efforts to advancing moral progress of all types, including sustainability, balanced mainly by the need for reasonable selfcare, and that organizations and institutions should likewise be oriented accordingly. An important perspective comes from the physics of the long-term future. Earth will become uninhabitable in roughly one billion years due to the gradual warming and expanding of the Sun  [57] . Survival beyond this time can only occur in outer space. For Earth-originating entities, this will require an advanced technological civilization capable of settling in outer space. Human civilization is already positioned to accomplish this task, given its ongoing space missions and general technological progress. As long as human civilization remains intact, the ability to sustain Earth-originating entities will persist. Long-term sustainability requires resettling in outer space  [28] . For the present generation, that means keeping human civilization intact. In many contexts, there is no significant distinction between sustainability and intertemporal optimization for the distant future. Both goals require maintaining the basic functionality of civilization, including by sustaining sufficient resources and by handling major threats such as global warming, pandemics, and nuclear warfare. They likewise entail evaluating environmental threats in terms of their implications for the continuity of human civilization and not in terms of biogeophysical disturbances or smaller-scale human consequences  [58] [59] . However, looking ahead, the goals point in different directions. Sustaining Earth-originating entities into the distant future only requires some minimal space settlement over very long timescales. In contrast, optimization of long-term intrinsic value entails space expansion sooner and at larger scales, in order to fill the universe with whatever is intrinsically valuable. The distinction between sustainability and optimization of long-term intrinsic value is also important in terms of what is intrinsically valuable. Long-term sustainability can entail the same course of action for both anthropocentric and non-anthropocentric conceptions of intrinsic value: If humanity fails to settle in outer space, then other Earth-originating entities would also die out in a billion or so years, if not sooner  [28, 60] . However, long-term optimization generally entails expansion into outer space-but expansion in what way? Anthropocentrism would entail expansion of human populations, whereas nonanthropocentrism would entail expansion of something else.
 Implications for AI AI has several important roles to play in the story outlined above. First, current and near-term forms of AI can be applied to addressing certain immediate threats to global civilization. For example, AI is in active use for addressing global warming and environmental protection in a variety of ways, and additional ways have been identified and called for  [21] . AI is also in active use for addressing the ongoing COVID-19 pandemic by supporting tasks such as medical analysis  [61]  and robotics to support social distancing  [62] . Further work along these lines could be of value for improving the resilience pf human civilization to COVID-19 and future pandemics. Whereas global warming is a traditional environmental sustainability topic, pandemics are not, though pandemics can derive from environmental activities, in particular those that put humans in contact with novel zoonotic pathogens. Nonetheless, both issues threaten the ability of global human civilization to be sustained into the long-term future. Second, future forms of AI could be particularly consequential. The field of AI has long entertained notions of extreme future AI that could be "the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control"  [63, p.33] . Recently, there has been debate on the extent to which people in AI should focus on near-term or long-term AI. To some extent, this debate may be unnecessary, due to the existence of activities that are good to do for both near-term and long-term AI  [24, [26] [27] . Nonetheless, to the extent that each type of AI merits some distinct attention, a case can be made for attention to long-term AI due to its potential importance for long-term sustainability. Long-term AI can play three roles of relevance to long-term sustainability. First, it could bolster efforts to address threats such as global warming and pandemics. Second, it could pose a threat of its own, especially for runaway AI scenarios in which the AI effectively takes over the world. Third, it could play an instrumental role in space expansion. A significant dilemma exists for the dual status of long-term AI as both threat and tool for addressing other threats. Ideally, long-term AI would be designed slowly and carefully to ensure a high standard for safety and ethics. However, delaying the deployment of long-term AI reduces the potential for its use to address other threats. One implication of this is that other work to address other threats can be of value for "buying time" to safely and ethically develop long-term AI  [64] . This includes work using near-term AI to address these other threats. If there is extended time available to slowly and carefully design long-term AI, then that also buys time to reflect on what should be done with respect to space expansion and related opportunities  [65] . On the other hand, there is no guarantee that an extended time will be available. Indeed, AI research and development is proceeding at brisk pace, prompting concerns about a race to develop long-term AI  [66] . One proposed means of buying some time is to deploy a moderately powerful AI "nanny" who can protect and support humanity while it reflects on what to do next  [67] . That possibility is not without its own risks, such as the risk of a poorly designed nanny AI that steers the world in a bad or even catastrophic direction. These are all among the AI issues that can be of profound importance for long-term sustainability.
 Conclusion In this paper, we have surveyed the ethics of sustainability, analyzed the ethical basis of existing work on AI and sustainability, and presented an argument for a non-anthropocentric, long-term conception of sustainability and an accompanying argument for favoring optimization over sustainability. Taken together, the paper provides some guidance on how ongoing work on AI and sustainability can and should proceed. First, work on AI and sustainability should precisely specify its ethical basis, in particular on what it seeks to sustain, for how long, and how much effort should be made on sustainability. Second, work on AI and sustainability should consider adopting the non-anthropocentric, long-term conception of sustainability and optimization that this paper argues for. In practice, that entails a focus on applying AI to addressing major global threats such as global warming and pandemics, to ensuring the long-term sustainability of the resource base needed for civilization, and for pursuing opportunities to expand human civilization into outer space. In closing, we wish to emphasize the ethical principle of equality across time. A fundamental aspect of sustainability is its future-orientation. The principle of equality across time means that all future times should be treated equally, or rather that there should be no bias against something just because of when it exists. This is a compelling moral principle. If it is to be taken seriously, it demands an attention to the big-picture distant future of the universe and to the ways in which near-term actions can affect it. Actions involving AI are among the most significant ways to affect the distant future. The field of AI has special opportunities to make an astronomically large positive difference-to make the universe a better place. It should make pursuit of these opportunities a major priority. strongly indicate that the natural environment is valued instrumentally to advance human wellbeing. Nothing in the statement clarifies the time scales of sustainability or degree of effort. The UNI Global Top 10 Principles for Ethical AI includes the principle, "Make AI serve people and planet: This includes codes of ethics for the development, application and use of AI so that throughout their entire operational process, AI systems remain compatible and increase the principles of human dignity, integrity, freedom, privacy and cultural and gender diversity, as well as with fundamental human rights. In addition, AI systems must protect and even improve our planet's ecosystems and biodiversity." This principle is classified as intrinsically valuing humans, ecosystems and biodiversity as the final line calls for not only protection but improvement of nonhuman entities, and the phrasing "make AI serve people and planet" strongly suggest moral consideration for also nonhumans.
 Appendix B: Examples of AI Sustainability Publications Section 3.2 presents data on publications on the nexus of AI and sustainability. This appendix presents some illustrative examples of these data. Theodorou et al.  [68]  developed a single-player game designed to simulate a sustainable world based on accurate ecological models and behavior economics principles. In the game, individual people must aim to act toward a sustainable society. The paper describes the game in terms implying that natural resources play an instrumental role to advance individual people's survival and wellbeing. Other indications of what is meant by a sustainable world is discussed in social and economic terms. This publication was, therefore, classified as implying intrinsic value of humans and instrumental value of nonhumans. Yigitcanlar & Cugurullo  [69]  define smart and sustainable cities as "an urban locality functioning as a robust system of systems with sustainable practices, supported by community, technology, and policy, to generate desired outcomes and futures for all humans and nonhumans". This publication is classified as intrinsically valuing humans and nonhumans as it explicitly refers to their benefits. van Wysnberghe  [7]  calls for a "third wave" in AI ethics and states that "This third wave must place sustainable development at its core" (emphasis original). Sustainable development is in turn defined as in the Brundtland report. This suggests an anthropocentric conception of sustainability, in which only humans are intrinsically valuable. It further suggests placing a high degree of effort on sustainability, though the emphasis on sustainable development leaves open the question of the relative importance of present vs. future generations. Gomes et al.  [70]  argue that "computational sustainability harnesses computing and artificial intelligence for human well-being and the protection of our planet" and that "planning for sustainable development encompasses complex interdisciplinary decisions spanning a range of questions concerning human well-being, infrastructure (…) and the environmental protection of the Earth and its species". All of this is strongly suggestive of intrinsic value of nonhumans as well as humans, so this publication is classified as such. Zhang et al.  [71]  studies the contributions of big data analytics capability and artificial intelligence capability to the sustainability of organizational development. The publication does not define sustainability and apply the following uses of "sustainability during the abstract and introduction only: "sustainable innovation and performance", "sustainability development projects", "sustainability design and commercialization processes", "sustainable growth and performance", "sustainable organizational growth", "sustainable competitive advantages", "sustainable investment", "sustainable development goals", "sustainable positional advantages", and big data as "sustainable resources". This publication is therefore classified as ambiguous. Larsson et al.  [72]  by the AI Sustainability Center defines sustainable AI as follows: "The AI Sustainability Center supports an approach in which the positive and negative impacts of AI on people and society are as important as the commercial benefits or efficiency gains. We call it Sustainable AI." This publication is classified as intrinsically valuing humans as it discusses and defines sustainability as pertaining to human and social aspects only, including sustainable AI which is defined by accountability, bias, malicious use, and transparency. The publication is further noteworthy for presenting an extensive literature review of AI ethics as a review of literature on sustainable AI, thus equating AI ethics or ethical AI with sustainable AI, which arguably drains "sustainability" of meaning. 5 2 . 1 21 https://aiforpeople.org/conference/ What should be able to be sustained, and why?
			 https://earthcharter.org/

Abstract: Sustainability is widely considered a good thing and is therefore a matter of ethical significance. This paper analyzes the ethical dimensions of existing work on AI and sustainability, finding that most of it is focused on sustaining the environment for human benefit. The paper calls for sustainability that is not human-centric and that extends into the distant future, especially for advanced future AI as a technology that can advance expansion beyond Earth.