Abstract: Lewis's Principal Principle says that one should usually align one's credences with the known chances. In this paper I develop a version of the Principal Principle that deals well with some exceptional cases related to the distinction between metaphysical and epistemic modal ity. I explain how this principle gives a unified account of the Sleeping Beauty problem and chancebased principles of anthropic reasoning. In doing so, I defuse the Doomsday Argument that the end of the world is likely to be nigh.

Doomsday and objective chance

Introduction It's often the case that one should align one's credences with what one knows of the objective chances.  1    Lewis (1980)  calls this the Principal Principle. For example, it is often the case that if one knows that a fair coin has been tossed, then one should have credence 1/2 that heads came up. The standard caveatthe reason for the 'often'-is that one sometimes knows too much to simply defer to the chances. A trivial example: once one sees that the coin has landed tails, one should no longer have credence 1/2 in heads. In such cases, one has what Lewis calls 'inadmissible evidence'. In this paper, I develop a version of the Principal Principle that handles two subtler kinds of exceptions, both related to the distinction between epis temic and metaphysical modality. The first arises because one can know some contingent truths a priori. The second is related to the fact that even an ideal thinker may be ignorant of certain necessary truths-in particular, one may not know who one is. The second type of case is my main focus, and I will illustrate it with two wellknown examples: the Sleeping Beauty puzzle  (Elga, 2000)  and the Doomsday Argument  (Leslie, 1992) . My version of the Principal Principle, labelled simply PP, yields standard views about both these cases: it yields the thirder solution to Sleeping Beauty, and denies that Doomsday is especially close at hand. These conclusions are well represented in the literature; my contribution is to present them as an attractive package deal, following from a single principle about the conceptual role of objective chance. The Doomsday Argument, in particular, is usually analysed in quite different terms, using an thropic principles like the Strong SelfSampling Assumption (which is used in the Doomsday Argument itself ) and the SelfIndication Assumption (which is used to resist it). I will explain how PP leads to chancebased versions of these assumptions, unified in a principle I call Proportionality. I will especially urge the merits of Proportionality over the SelfIndication Assumption. In §2, I introduce the existing version of the Principal Principle that will be my starting place. In §3, I explain the problem that arises from a priori contingencies, and suggest a preliminary solution. In §4, I explain why this preliminary solution is unsatisfactory: it applies only in the very unusual cir cumstance that one has no selflocating information. I then state my preferred principle, PP, and show how it handles Sleeping Beauty and Doomsday. In §5, I state the principle of Proportionality, which follows from PP, and com pare it to the standard anthropic principles. (The proof of the main result is in the appendix.) Then, in §6, I briefly consider what my chancebased prin ciples suggest about reasoning based simply on a priori likelihood, rather than chance. Section 7 sums up and points out one remaining difficulty for my theory. Along the way, I will use the framework of epistemic twodimensionalism  (Chalmers, 2004)  to model the connection between epistemic and metaphys ical modality. I won't be defending epistemic twodimensionalism in this pa per, but it does conveniently represent the phenomena with which I am con cerned. My hope is that critics of twodimensionalism can find equivalent (or better!) things to say in their own frameworks.
 Background I will think of the Principal Principle as a constraint of rationality on an agent's ur prior. This ur prior, which I denote Cr, is a probability measure reflecting the agent's judgments of a priori probability and evidential support. Or, bet ter, not a probability measure but a Popper function, a twoplace function directly encoding conditional probabilities. 2 I'll refer to the arguments of Cr as 'hypotheses'. 'Propositions' would also do, but I use different terminology to emphasize that hypotheses are individuated hyperintensionally: the hypoth esis that water is H 2 O is distinct from the hypothesis that water is water, and someone could reasonably have different credences in them. What's the relationship between ur priors and credences? Suppose that at time t one has total evidence E and credence function Cr t . Then one should (I suggest) satisfy the norm of Ur Prior Conditionalization. Cr t (H ) = Cr E (H ) := Cr(H | E ). As is well known, Ur Prior Conditionalization entails ordinary Bayesian Con ditionalization: if one's evidence strengthens from E to E & E ′ , then one's credences change from Cr E (H ) to Cr E (H | E ′ ). However, Ur Prior Condi tionalization has the advantage that it handles situations where one's evidence changes in other ways, like cases of forgetting: whatever happened in the past, the appropriate thing now is to conditionalize one's ur prior on one's current evidence. The question of whether Ur Prior Conditionalization handles such cases correctly will be relevant later on, but for the most part I will treat this as a working hypothesis to which I do not know any comparably adequate alternative.  3  Now, as to the Principal Principle, I will start from a version developed by  Meacham (2010)  and (as he notes) in unpublished work by Arntzenius. Let 〈ch(H | E ) = p〉 stand for the hypothesis that the chance of H , given E , is p. Then Arntzenius's formulation of the principle is Cr(H | E & 〈ch(H | E ) = p〉) = p. A more general claim will also be useful. Let 〈ch = f 〉 stand for the hypothesis that the chances agree with the (perhaps only partially defined) Popper func tion f ; thus 〈ch = f 〉 is effectively a conjuction of hypotheses of the form 〈ch(H | E ) = p〉. I will write Cr f for the Popper function obtained by condi tionalizing Cr on 〈ch = f 〉: Cr f (H | E ) := Cr(H | E & 〈ch = f 〉). Then the general principle I attribute to Meacham and Arntzenius is PP1. Cr f (H | E ) = f (H | E ). More precisely, the two sides should be equal when both are defined, but from now on I'll always leave out this type of qualification.  4  I defer to Meacham for a careful explanation of the connection between PP1 and Lewis's classic version of the Principal Principle, but two points are especially relevant. First, PP1 is compatible with the existence of nontrivial chances even in worlds where the fundamental physics is deterministic. For example, if E is a suitable macroscopic specification of the initial conditions of a fair coin toss, and H is the hypothesis that the coin lands heads, we may well have ch(H | E ) = 1/2. This doesn't contradict the claim of determinism that, if E ′ is a complete microphysical specification of the initial conditions, then either ch(H | E ′ ) = 1 or ch(H | E ′ ) = 0. So I won't hesitate to treat coin tosses as genuinely chancy. The second important point is that, unlike one of Lewis's formulations, PP1 does not need an exception for inadmissible evidence. Continuing the example from the previous paragraph, suppose that the agent learns that H is true. Then, for any Popper function f , PP1 gives Cr f (H | H & E ) = f (H | H & E ) = 1. So after one learns the result of the coin toss, one is no longer bound to give credence 1/2 to heads.
 The Problem of A Priori Contingents
 The Problem The first problem for PP1 arises from the distinction between epistemic and metaphysical modality, and in particular from the phenonenon of a priori con tingents. Example 1: Topper Comes Up. Suzy is about to flip a coin, which she knows to be fair. She introduces 'Topper' to rigidly designate whichever side of the coin will come up. Because of the way she introduces the term, she can be certain that Topper comes up. However, Topper is either heads or tails. Suzy knows that, either way, there is a 1/2 chance that Topper comes up. So her credence that Topper comes up should not equal the known chance that Topper comes up.  5  If E is a suitable specification of the cointossing setup, and Top is the hypoth esis that Topper comes up, then Cr(Top | E & 〈ch(Top | E ) = 1/2〉) = 1 contradicting PP1 . This example trades on the idea that chance has to do with metaphysical or nomological modality, whereas credence is a matter of epistemic modality. It's essentially a priori for Suzy that Topper comes up, and that's why Suzy gives it credence one. But it's not necessary that Topper comes up, and so too it's not chance one. Similar problems can arise for natural kind terms. Suppose that 'water' rigidly designates what one might describe for short as the predominant wet stuff (which turns out to be H 2 O). Then we can dream up a case in which it's a priori that the predominant wet stuff is water, and yet there's a 1/2 chance that the predominant wet stuff is H 2 O. This would again enable a counterexample to PP1. Finally-and most importantly for this paper-similar cases arise for in dexicals. Example 2: The Sheds. I'm Carlos; Ramon is my twin. There are two windowless sheds. A fair coin is tossed. If heads, Ramon goes in Shed 1 and I go in Shed 2; if tails, the other way around. We sit there in the dark. Just before noon, partial amnesia is induced: although we both remember the general setup, neither of us is sure whether he is Carlos or Ramon, nor how the coin landed, nor whether he is in Shed 1 or Shed 2. If I'm Carlos and this is Shed 1, then the chance that I'm in this shed is the chance that Carlos is in Shed 1, i.e. 1/2. Similarly if I'm Ramon and this is Shed 2, and so on. In any case, there's a 1/2 chance that I'm in this shed. And yet I'm certain that I'm in this shed.
 A FirstPass Solution To avoid the problems raised by these examples, we could simply restrict the Principal Principle to cases in which the relevant hypotheses do not involve proper names, or natural kind terms, or indexicals, or anything of the sortin short, to the kind of hypotheses that  Chalmers (2011)    (H | E ) = f (H | E ). While this basic proposal will require some amendment in §4, its meaning and limitations will be clearer if we pause, first to explain how the neutrality restric tion works within the framework of epistemic twodimensionalism  (Chalmers, 2004 (Chalmers, , 2011 , arguably its natural home; and second to explain how PP2 pur ports to give a full account of Topper Comes Up and similar cases. I especially want to introduce the standard idea of a primary intension, and the less stan dard idea of a neutral paraphrase, which will play an important role in what follows. Recall that the intension of a hypothesis is a set of possible worlds-the worlds in which the hypothesis would be true. Two hypotheses are necessar ily equivalent if and only if they have they same intension. My assumption that chance is a form of metaphysical modality amounts to the claim that the chance of a hypothesis depends only on its intension. However, rational credences can distinguish between necessarily equivalent hypotheses. For ex ample, suppose that Suzy's coin in fact lands headsup. Then the hypothesis that Topper comes up is necessarily equivalent to the hypothesis that heads comes up. Yet Suzy gives them different credences. To represent the distinctions made by rational credences, twodimension alists introduce a second dimension of epistemic scenarios. These are like pos sible worlds, but individuated by epistemic criteria. For example, there are some scenarios in which Topper is heads, and others in which Topper is tails; as Suzy's uncertainty attests, these are distinct and genuine epistemic possibil ities. The primary intension of a hypothesis is the set of scenarios (rather than possible worlds) in which it is true; two hypotheses are a priori (rather than necessarily) equivalent if and only if they have the same primary intension. For my purposes, the key point is that, according to Chalmers, each sce nario picks out (i) a possible world as actual; and (ii) an intension, i.e. a set of possible worlds, for each hypothesis. For example, some scenarios pick out a world in which heads comes up. In such a scenario, Topper is heads, and the intension of Top is the set of worlds in which heads comes up. Other scenar ios pick out a world in which tails comes up. Then Topper is tails, and the intension of Top is the set of worlds in which tails comes up.  7  Crucially, a scenario s is in the primary intension of H if and only if the intension of H in s contains the world that is actual in s . Chalmers calls a hypothesis is neutral if and only if, unlike Top, it has the same intension in every scenario. We can now see why the restriction to neutral hypotheses, understood in this way, avoids the problems raised by Topper Comes Up and similar cases. Ur priors can't distinguish between a priori equivalent hypotheses, like 'Topper comes up' and 'Whichever side comes up, comes up'. PP1 is bound to fail insofar as such hypotheses are not necessarily equivalent, so that they can have different chances. This problem cannot arise for the class of neutral hypotheses, however: it is easy to see that two neutral hypotheses that are a priori equivalent are also a priori necessarily equivalent (they have the same intension in each and every scenario), and therefore a priori have the same chance.  8  Even if the restricted principle PP2 avoids these problems, one might worry that it applies too rarely to constrain credences in all the expected ways. In one respect, which I'll discuss in §4, this turns out to be a very serious worry, but I think it is worth having a firstpass explanation of how PP2 could give the desired results in a case like Topper Comes Up. Let us focus on Suzy's credence that Topper is heads. Since this hypothesis is not neutral, PP2 does not directly tell us the right credence. However, we can reason in two stages. First, the hypothesis that heads comes up is (more plausibly) neutral, and, if so, PP2 does require Suzy's credence in heads to be 1/2. Second, it's a priori for Suzy that Topper is heads if and only if heads comes up. Therefore Suzy must also have credence 1/2 that Topper is heads. Not only do we get the right conclusion, the explanation for it strikes me as perspicacious. At any rate, it illustrates that the restriction to neutral hy potheses is not debilitating insofar as there are what I'll call neutral paraphrases of more general hypotheses. Here, E • is a neutral paraphrase of E if and only if E • is neutral and E and E • have the same primary intension (they are a pri ori equivalent). Because of this last condition, E and E • are interchangeable when it comes to ideal ur priors. For example, 'heads comes up' is a neutral paraphrase of 'Topper is heads'. Suzy's credence in the latter is determined by her credence in the former, which is in turn determined by PP2.
 The Problem of SelfLocation
 The Problem While it is arguable that a wide range of hypotheses about the world do ad mit neutral paraphrases, it is unfortunately impossible to maintain that our evidence in ordinary circumstances-circumstances in which we expect the Principal Principle to be binding-is of that type. The reason is that neutral hypotheses exclude the use of indexicals. Because of this-as I'm about to ex plain in more detail-a neutral hypothesis, or one with a neutral paraphrase, can arguably give an adequate thirdpersonal, qualitative description of the world, but it can do nothing to identify one's own situation. As a result, PP2 only applies if one has essentially no knowledge whatsoever of what one is like or where one is in space and time. This is an unaccceptable result. To see the problem more formally, consider, for example, the hypothesis S that I am sitting-a perfectly ordinary thing for me to know. In a scenario where I am Carlos, the intension of S consists of the worlds in which Carlos is sitting; in a scenario where I am Ramon, it consists of the worlds in which Ramon is sitting. So S is not neutral. Nor, I claim, does it admit a neutral paraphrase. Remember that a neutral paraphrase would be a neutral hypothe sis with the same primary intension as S . Consider a world w in which Carlos is sitting but Ramon is not, and consider two scenarios: in the first, w is actual and I am Carlos; in the second, w is actual and I am Ramon. The primary in tension of S contains the first of these scenarios, but not the second. A neutral hypothesis, in contrast, must have the same intension in both scenarios, and this intension either contains w or it doesn't. Correspondingly, the primary in tension of a neutral hypothesis either contains both scenarios or neither. Since the primary intension of S contains only one of the scenarios, it doesn't have a neutral paraphrase. (Note, for example, that someone is sitting, though neutral, is not a neutral paraphrase of S : its primary intension contains both of the scenarios I described.) Generalizing from this example, my evidence in ordinary circumstances distinguishes between different scenarios in which one and the same world is actual: it distinguishes me from most of the other people in the actual world. Neutral hypotheses cannot do this. Therefore my evidence in ordinary cir cumstances does not admit a neutral paraphrase, and therefore PP2 does not ordinarily apply. The twodimensionalist framework represents this point in the following way. Following Chalmers again, we can identify each epistemic scenario with a centered possible world: a triple (w, x , t ) where w is a possible world, and x is an individual and t a time in w. I'll refer to (w, x , t ) as a centering of w, and (x , t ) as a center. Thus the primary intension of a hypothesis is a set of centered worlds. For example, the primary intension of the hypothesis I'm sitting consists of the centered worlds (w, x , t ) such that, a priori, if I'm x at t in w, then I'm sitting.  9  Now, let w be some possible world. For a hypothesis to be neutral, it must have the same intension with respect to every scenario, and in particular with respect to every centering of w. It follows that if the primary intension contains one centering of w, it must contain them all. This makes precise the idea that neutral hypotheses (or those with neutral paraphrases) completely fail to locate the subject in the world.
 A Solution One strategy would be to supplement PP2 by principles of a different kind that together constrain one's credences in the right way. I will consider some such principles in §5. However, this strategy seems backtofront. The Prin cipal Principle, whatever the details, is supposed to express a platitude about how knowledge of the chances ordinarily constrains our credences. It hardly matters what it says about bizarre cases of complete selflocating ignorance; it ought to apply directly in situations that (at worst) idealize what we take to be the ordinary case. I propose instead to formulate a modification of PP2 that applies directly when one does have fully selflocating evidence: that is, more carefully, when the primary intension of one's evidence contains at most one center for each possible world. Lest this appear a radical move, let me emphasize that it is a natural de velopment of what  Lewis (1980)  himself says. He states the Principal Princi ple in a setting where one's credences assign probabilities to possible worlds, and therefore do not explicitly distinguish different centers within each world. However, the point is not that his principle applies only in bizarre cases of complete selflocating ignorance! He applies it to ordinary cointossing cases, after all. Rather, uncentered possible worlds usually suffice because each such world comes with an implicit center, picked out by the agent's selflocating evidence. As Lewis says, we only need to use centers explicitly if we want to handle cases in which 'one's credence might be divided between different pos sibilities within a single world'  (Lewis, 1980, p. 268) . So Lewis's principle applies when one's credences are not so divided, i.e. when one has fully self locating evidence. Moreover, it applies no matter what the implicit centerings may be. That's exactly the picture I want to spell out. To emphasize the role of indexicals, I will often represent potentially non neutral hypotheses in the form 〈I am G 〉. Then 〈I am F G 〉 means 〈I am F 〉& 〈I am G 〉, and so on. I'll say that a hypothesis is fully selflocating if and only if its primary intension contains at most one centering of each possible world. The proposal is to restrict the Principal Principle to cases of fully selflocating evidence. However, there is a more convenient way to put this. Say that 〈I am G 〉 is merely selflocating relative to background evidence E if and only if it picks out exactly one centering of each world compatible with E . More carefully, I am talking about primary intensions, so the condition is that, if the primary intension of E contains a centering of w, then the primary intension of E & 〈I am G 〉 contains exactly one centering of w. It follows that E & 〈I am G 〉 is fully selflocating. In these terms, the main proposal of this paper is that the chances bind credences conditional on any merely selflocating hypothesis: PP. If E and H are neutral hypotheses, and 〈I am G 〉 is merely selflocating relative to E , then Cr f (H | E & 〈I am G 〉) = f (H | E ). The restriction to neutral E and H is still important here, but in §5 I will develop a less restricted principle-Proportionality-as a consequence of PP. One might worry that ordinary evidence is never fully selflocating: per haps it does not narrow things down to exactly one individual and one time in each world compatible with one's evidence. I'll consider a troubling form of this worry in §7, but for now I will just address the most mundane form: one's evidence may not often pin down a precise time. There are two basic responses. The first is that one can have fully selflocating evidence even if one does not know what time it is in the ordinary sense that one does not know what clocks are saying right now. Clockfaces are only one way of picking out an instant in each world. However, one may still worry that one's evidence is coarsegrained in a way that just can't pin the present down exactly. There may be some deep issues here about perception and even about the metaphysics of time, but the short answer is that we are allowed, as far as PP goes, to count times in a coarsegrained way. We need not take 'one time' to mean 'one instant' rather than 'one interval of unit length', where the units are adjustable and we count nonoverlapping unitlength intervals as different times. What's crucial in applying PP is that the precision with which 〈I am G 〉 locates me in the world is independent of how the world turns out, conditional on E . Now I'll illustrate PP with two important applications.
 Application: Sleeping Beauty First, consider this famous example: 10 Example 3: Sleeping Beauty. On Sunday night, Beauty knows she is in the following situation. After she goes to sleep, a fair coin will be tossed. She will be awakened on Monday. A few minutes later, she will be told it is Monday. Then she will go back to sleep. If the coin landed heads, she will sleep through Tuesday. But if it landed tails, her memories of Monday will be erased, and she will be awakened on Tuesday morning. Thus, when Beauty wakes on Monday, she does not know whether the coin landed heads or tails, nor, supposing the coin landed tails, whether it is Monday or Tuesday. What should Beauty's credence in heads be (1) on Sunday night; (2) after learning it is Monday; (3) on first waking? PP allows us to analyse the case as follows. (1) On Sunday night, Beauty's evidence, as normal, is fully selflocating. Therefore PP applies and tells us she should have credence 1/2 in heads. But let me spell this out, to make clear how the formalism works. Let E be a neutral, thirdperson specification of the setup; Beauty knows E throughout, and she also knows the chance proposition 〈ch = f 〉 to the effect that heads (H ) and tails (T ) both have chance 1/2 conditional on E . Let 〈I am G 〉 be the hypothesis that I'm Beauty and it's Sunday night. With some minor simplifications, the primary intension of E & 〈I am G 〉 effectively contains just two centered worlds: the world in which the coin lands heads, centered on Beauty on Sunday night, and the world in which the coin lands tails, centered on Beauty on Sunday night. The hypothesis 〈I am G 〉 is merely selflocating relative to E . PP then yields Cr f (H | E & 〈I am G 〉) = 1/2, as expected. (2) A very similar application of PP shows that Beauty should have cre dence 1/2 in heads after she learns it's Monday: at that point, too, she has fully selflocating evidence. (3) On first waking, however, her evidence is not fully selflocating, and PP does not apply. Nevertheless, we can argue from PP that she must have credence 1/3 in heads. Consider the three hypotheses HM (heads, which im plies it's Monday), TM (tails and it's Monday), and TT (tails and it's Tuesday). Assuming that Beauty will update by conditionalization when she learns it's Monday (i.e. HM ∨ TM), she must, when first waking, already give HM and TM equal credence. Now consider what would happen were she instead to learn HM∨TT. She would again have fully selflocating evidence, and should again have credence 1/2 in heads. So she must already give HM and TT equal credence. All together, she gives equal credence to each of the three hypothe ses HM, TM, and TT. Since these are mutually exclusive and exhaust the possibilities open to her, she must give credence 1/3 to each. This pattern of credences is called the 'thirder' position in the literature on Sleeping Beauty. I find the extant arguments for thirderism quite compelling, and I am happy to refer to them as corroboration for my view. However, the analysis I've presented is slightly different from the most common way of understanding thirderism.  Elga (2000)  appeals to a principle of indiffer ence: Beauty should, on waking, consider the hypotheses TM and TT equally likely, since her evidence is fully symmetric between them. But this suggestion invites standard worries about indifference reasoning, including the thought that Beauty might have symmetrical but only imprecise credences in these hy potheses  (Weatherson, 2005) . My argument is different, and isn't directly sus ceptible to such worries. Instead of appealing to evidential symmetry, I claim that Beauty should align her credences with the known chances, not only after she learns it's Monday, but also if she were instead to learn HM ∨ TT, on the basis that these are both merely selflocating hypotheses relative to her other evidence E . Of course, thirderism is not the only standard position when it comes to Sleeping Beauty. As Elga explains, the main rivals to thirders are halfers, who claim that Beauty should give credence 1/2 to heads when she wakes up, as well as on Sunday night. I can't do justice to the whole literature, and want to focus on my positive proposal, but it seems significantly more difficult to do for halferism what I've done for thirderism here: to embed it in a package that includes systematic norms for updating (as in Ur Prior Conditionalization) and a natural version of the Principal Principle (as in PP). For, on the one hand, Beauty's evidence after learning it's Monday is structurally very similar to her evidence on Sunday night, so it's hard to see why the Principal Principle would apply in the second case but not the first. On the other hand, if, as 'double halfers' claim, Beauty should have credence 1/2 in heads at both these times and on first waking, then she must not apply Bayesian conditionalization when she learns it's Monday. 11
 Application: The Doomsday Argument Here is another example. It is very similar to Sleeping Beauty, but it will be useful to consider it separately, because it is commonly analysed using quite different tools, which I will contrast with PP in section 5. Example 4: Doomsday. There's a 1/2 chance that humanity goes extinct at an early stage, resulting in a total of 100 billion people who ever live (call this outcome early doom); and a 1/2 chance that humanity hangs on much longer, resulting in 100 quadrillion people who ever live (call this outcome late doom). Against this evidential background, I learn that I am the 70 bil lionth person to be born. What should my credence be in early doom? As the Doomsday Argument notes, knowing I am the 70 billionth person rules out many possibilities that are compatible with both early doom and late doom, but vastly many more that are only compatible with late doom (for example, the possibility that I am the 200 billionth person). So, for any reasonable priors, that piece of evidence should result in a dramatic shift in credence to wards early doom. Unless I was antecedently ridiculously confident in late doom-far more confident than the stated 1/2 chance-I should now be al most certain of early doom.  12  The example is practically significant because our actual evidential situ ation is stylistically similar to the one described. We have some idea about the various kinds of extinction risks we face (either as a species or as a global ecosystem), and a fairly precise idea of how far along we are since life began. The basic logic of the Doomsday Argument generalizes to more complicated cases, and seems to show that an early doom for humanity is much more likely (epistemically speaking) than the chances would on their face suggest. However, in parallel to my analysis of Sleeping Beauty, PP implies that my posterior credence in early doom should be 1/2. At least, it does so for reasonable ways of filling in the details. Most simply, assume that everyone has the same lifespan; then the hypothesis that I'm the 70 billionth person is fully selflocating by the criteria sketched at the end of §4.2. Thus it is after, not before, learning that I am the 70 billionth person that PP binds my credence to 1/2. This, along with Ur Prior Conditionalization, commits me to having been 'ridiculously' confident in late doom prior to gaining the new evidence. But, then again, prior to that evidence I was in the ridiculous epistemic state of having essentially no selflocating information. There is a general worry about Bayesianism that extremely high confidence rarely seems warranted, but, beyond that, we shouldn't be too worried about getting surprising results about such exotic epistemic positions.
 The Principal Principle and Anthropic Reasoning PP only directly constrains the credences of agents with fully selflocating evi dence. But, as already hinted in my analysis of Sleeping Beauty and Doomsday, it has broader implications. I'll now draw out some of those implications, and show how they relate to-and in some ways improve upon-the anthropic principles that are commonly used to analyse Doomsday.
 Proportionality The main technical result of this paper is that, given a sufficiently rich domain of hypotheses, PP entails a superficially stronger principle that I call Proportion ality. I'll give the argument from PP to Proportionality in the appendix, and focus on its formulation here. Proportionality is a sophisticated version of the intuitive idea that my credence that I'm F , given that I'm G , should be high to the extent that most G s are F s. In many cases, it sets that credence equal to the chanceexpected number of F andG s divided by the chanceexpected number of G s.  13  However, stating it in full generality requires a little work. So, consider a neutral hypothesis E and an arbitrary hypothesis 〈I am G 〉. Recall that the primary intension of 〈I am G 〉 contains zero or more centerings of each possible world. I define N f (G | E ) to be the expected number of such centerings, according to the probability measure f (− | E ). To spell it out: for each world w, we take the number of centerings of w in the primary intension of 〈I am G 〉, we multiply that by the probability (according to f , conditional on E ) that w is actual, and then we sum over worlds.  14  Thus N f (G | E ) = 1 if 〈I am G 〉 is merely selflocating with respect to E , and will be higher insofar as 〈I am G 〉 fails to pin down my location in th worlds where E is true. Here is the key principle: Proportionality. Suppose E is a neutral hypothesis. Then Cr f (〈I am F 〉 | 〈I am G 〉 & E ) = N f (F G | E ) N f (G | E ) . Note that (unlike in PP2) the restriction to neutral E is not onerous, since the overall evidence 〈I am G 〉 & E is effectively arbitrary. One can get a good idea of how Proportionality works by applying it to Sleeping Beauty. (And one can get a good idea of the argument for Propor tionality by pondering the argument for thirderism I gave in section 4.) As in my earlier discussion of the case, let E be a neutral specification of the setup, but now let 〈I am G 〉 be the hypothesis that I'm Beauty and I've just woken up on one of the relevant days. The primary intension of E & 〈I am G 〉 effec tively consists of three centered worlds: the heads world centered on Beauty on Monday, the tails world centered on Beauty on Monday, and the tails world centered on Beauty on Tuesday. It follows that N f (G | E ) = 3/2. On the other hand, let 〈I am F 〉 be the same as the hypothesis H that the coin landed heads. Then the primary intension of 〈I am F G 〉 consists of a single centered world: the heads world, centered on Beauty on Monday. It follows that N f (F G | E ) = 1/2. Therefore, according to Proportionality, Cr f (〈I am F 〉 | 〈I am G 〉 & E ) = 1/3, as the thirder claims.
 The SelfSampling Assumption Discussions of the Doomsday Argument usually turn on two 'anthropic' prin ciples: the SelfSampling Assumption and the SelfIndication Assumption. I'll now explain how these principles are related to Proportionality. One goal is to shed more light on Proportionality, but I'll also argue that-at least in some ways-Proportionality provides a more satisfactory picture of the Doomsday Argument. The first of the two main anthropic principles is, in Bostrom's influential formulation, The Strong SelfSampling Assumption (SSSA). One should rea son as if one's present observermoment were a random sample from the set of all observermoments in its reference class.  15  Here an 'observermoment' is what I have been calling a centered possible world: one's present observer moment is the actual world centered on oneself and the present time. Although SSSA is not precisely stated, the basic idea is that one should consider different merely selflocating hypotheses to be equally likely. So, for example, Beauty should be indifferent between Monday and Tuesday, conditional on tails. And, in Doomsday, I should initially give equal credence to different hypotheses about my birthrank in each world separately. Because of this, my initial credence that I'm the 70 billionth person is a million times higher conditional on early doom than on late doom. This determines how strongly I should update in favour of early doom upon learning my birth rank: the subjective odds of early doom increase by a factor of one million. My own analysis of Doomsday used PP to determine my posterior cre dence in early doom directly. It is unnecessary to adduce SSSA as a separate principle, since the following version of it is a simple application of Propor tionality: Uniformity. Suppose that 〈I am G 〉 and 〈I am G ′ 〉 are merely selflocating relative to a neutral hypothesis E . Then Cr f (〈I am G 〉 | E & 〈I am G or G ′ 〉) = Cr f (〈I am G ′ 〉 | E & 〈I am G or G ′ 〉). In short, the merely selflocating hypotheses 〈I am G 〉 and 〈I am G ′ 〉 get equal credence. Besides being much more precise, Uniformity differs from SSSA in several important respects. First, Uniformity only applies conditional on an appropriate chance hy pothesis 〈ch = f 〉. I'll say more about this limitation in §6. It is a limitation, but it also points to a key strength: Uniformity emerges from a unifying story about objective chance. Second, Uniformity makes sense even when some worlds compatible with E include infinitely many observermoments, whereas there is no entirely rea sonable way to randomly sample from an infinite set.  16  This is the effect of conditionalizing on 〈I am G or G ′ 〉: it narrows consideration to at most two centers in each world. Third, as usually conceived, SSSA is a principle of indifference between different merely selflocating hypotheses, similar to the indifference principle Elga used to analyse Sleeping Beauty. In contrast, Uniformity is based on a claim about the applicability of the chancecredence link. Of course, PP does include a kind of indifference claim, to the effect that all merely selflocating hypotheses are equally good from the point of view of the Principal Principle. A fourth, closely related difference is that SSSA appeals to the the idea of a 'reference class' of observermoments. Uniformity treats all merely self locating hypotheses as equally good, without limitation to a narrower reference class (but with the understanding that centered worlds include only genuine a priori possibilities-see fn. 9). Bostrom uses flexibility in the choice of ref erence class to resolve various problems that arise from his theory, including the Doomsday Argument. This flexibility seems unnecessary when it comes to Uniformity: PP defuses the Doomsday Argument without further recourse to reference classes.
 The SelfIndication Assumption The second, more controversial anthropic principle is The SelfIndication Assumption (SIA). Given the fact that you exist, you should (other things equal) favor hypotheses accord ing to which many observers exist over hypotheses on which few observers exist.  (Bostrom, 2002, p. 66)  This is again rather imprecise, but SIA is commonly understood as a claim about the evidential import of the fact that one exists: conditionalising one's ur prior on that evidence increases the relative likelihood of worlds with large pop ulations. This idea is especially clearly stated by  Bartha & Hitchcock (1999a) , but goes back to  Dieks (1992) . One post hoc motivation for SIA is that it provides a way of blocking the Doomsday Argument. Suppose that we think the chancecredence link is properly given by PP2. Then, knowing only the chance hypothesis stated in Doomsday, I should have a 1/2 credence in early doom and 1/2 in late doom. Next, I conditionalize on two pieces of evidence: (E 1 ) that I exist and (E 2 ) that I am, specifically, the 70 billionth person. The Doomsday Argument really shows us that given E 1 , E 2 shifts my credences dramatically towards early doom. But SIA suggests that conditionalizing on E 1 itself shifts my credences towards late doom-towards worlds with lots of people. (For simplicity, I as sume that 'person' and 'observer' mean the same thing.) So if we interpret SIA in exactly the right way, these two shifts will cancel out, and the net effect of learning E 1 and E 2 is to leave my credence in early doom at the original 1/2. Is there any independent reason to think that E 1 has exactly the eviden tial significance required?  Bartha & Hitchcock (1999a, p. 349)  provide what they call a 'justso story': if the 100 billion people in the early doom world and the 100 quadrillion people in late doom world were chosen separately and uniformly at random from a stock of possible people, then any one of those possible people would have a greater chance (and greater to just the right de gree!) of being selected into the late doom world. But even if we managed to take this justso story seriously as a piece of cosmology, the upshot would be unclear. How does it help with cases of selflocation within a life, as in Sleeping Beauty? And notice that the metaphysical claim that the population is chosen at random is compatible with the not unreasonable epistemic claim that it is a priori, for me, that I exist. But if it is a priori, then it has no eviden tial weight for me at all, contrary to SIA. As this indicates, the justso story equivocates between metaphysical and epistemic modality in the way I have been trying to avoid. Motivationally, then, the SelfIndication Assumption is on shaky ground. Nevertheless, there is a precise sense in which Proportionality, somewhat like SIA, requires one to give higher credence to largepopulation hypotheses than the chances naively suggest. It entails: Weighting. If E and H are neutral hypotheses, then Cr f (H | E & 〈I am G 〉) = N f (G | H & E ) N f (G | E ) f (H | E ). According to Weighting, my credence in H conditional on E & 〈I am G 〉 should equal the chance of H conditional on E , but weighted by a factor that reflects how H is correlated with the number of G s. The factor is large insofar as the worlds in which H is true tend to have many centerings in the primary intension of 〈I am G 〉. Weighting is a precise generalization of the claims that, before learning it's Monday, Beauty should be quite confident in tails, and that, before learning I'm the 70 billionth person, I should be extremely confident in late doom-in both cases, more confident than the 1/2 chance. Some readers might like to see in more detail how Weighting applies in Sleeping Beauty. As in §5.1, let E &〈I am G 〉 be Beauty's nonchance evidence on first waking; as before, its primary intension contains three centered worlds, and N f (G |E ) = 3/2. Let T (rather than H ) be the hypothesis that tails comes up. The primary intension of T & E & 〈I am G 〉 consists of two centerings of the tails world, and it follows that N f (G | T & E ) = 2. Thus, Weighting says, Beauty's credence in tails should be 2/(3/2) = 4/3 times the 1/2 chance, i.e. it should be 2/3, just as the thirder claims.
 Beyond Chance This paper has been about objective chance, and the anthropic principles de veloped in §5 are formulated in terms of a chance hypothesis 〈ch = f 〉. As I mentioned in §2, my understanding of chancetalk is pretty broad: it's not just limited to indeterministic interpretations of quantum mechanics, or anything like that. Still, I agree that there are situations where talk of chances would seem misplaced, including cases in which we are considering the relative plau sibility of different scientific theories. So I don't claim to have recovered the full scope of the anthropic principles that have been proposed in the literature. But I have shown that one can get pretty far with chances, and the results are suggestive of a more general analysis.  (H | E & 〈ch = f 〉) = f (H | E ). And the argument for Proportionality given in the appendix supports the more general claim Cr(〈I am F 〉 | 〈I am G 〉 & E ) = N Cr 0 (F & G | E ) N Cr 0 (G | E ) This generalization of Proportionality does not involve any chance hypothe sis; it instead involves the judgments of evidential support represented by the Popper function Cr 0 . The point of this innovation is that sometimes our judgments of a priori evidential support plausibly relate to Cr 0 rather than to Cr. We just don't usu ally consider the case of complete selflocating ignorance; we take selflocation for granted, as does most of the literature in epistemology that is not specif ically concerned with Sleeping Beauty or Doomsdaylike cases. So the loose thought that H and ¬H are equally likely conditional on E may well suggest that Cr 0 (H |E ) = 1/2 rather than Cr(H |E ) = 1/2. Note that Cr 0 (H |E ) = 1/2 is what we'd expect from PP if one knew a priori that ch(H | E ) = 1/2. In that sense, the judgements of evidential support represented by Cr 0 are calibrated to the chances. Some other ways of measuring a priori likelihood are at least compatible with chancecalibration. For example, one might attempt to gauge the relative likelihood of H and ¬H by imagining what an angel in heaven would find plausible without having looked out to see how the universe is going. 17 But of course the angel knows perfectly well where he is, so judgments arrived at in this way must already take selflocating evidence into account. For illustration, consider a version of Doomsday in which early doom and late doom are supposed to be 'equally likely a priori', but this isn't cashed out in terms of chances. If 'equally likely' is understood in terms of Cr, then (set ting aside SIA and other shenanigans) the Doomsday Argument does seem to show that someone with fully selflocating evidence will be dramatically more confident in early doom than in late. But this point is not very inter esting unless we have a decent grip on what is epistemically likely given the exotic evidential background of complete selflocating ignorance. In contrast, if 'equally likely' is understood in a chancecalibrated sense, or just against an implicit evidential background that already includes selflocating information, then the Doomsday Argument does not go through.
 A Final Problem I've shown how to formulate a version of the Principal Principle that is bet ter insulated against the problem of a priori contingencies and which works even in the context of selflocating ignorance. The main ideas are that one should to stick to neutral hypotheses, and that chances bind credences relative to fully selflocating evidence. The resulting picture, including Ur Prior Con ditionalization, fits cleanly with the thirder view of Sleeping Beauty. It also yields chancebased versions of some wellknown anthropic principles (Uni formity, Weighting, and most fundamentally Proportionality) while blocking the chancebased Doomsday Argument. Finally, one can generalize these prin ciples beyond chances to chancecalibrated judgments of a priori likelihood. The aspect of this picture that I ultimately find least satisfying is that, when it comes down to it, our ordinary evidence may not be fully selflocating. Given the immense size of the universe, we should take seriously the possibil ity that there are qualitative duplicates, or near enough, of ourselves and our surroundings somewhere else. (More carefully, the issue is that my total evi dence includes in its primary intension some epistemic scenarios centered on sufficiently close duplicates of myself.) As a stylized case, consider a version of Doomsday in which the 100 quadrillion people in the late doom world consist of a million distantly separated groups of duplicates of the 100 billion people who would exist given early doom. Against that background, it would be hard for me to get fully selflocating evidence; reasonable evidence could at best narrow down one's identity to a million qualitatively identical people, condi tional on late doom. By Weighting, I should then be extremely confident in late doom. And, to emphasise, I need not be unusually uninformed: I could be well acquainted with my environment as far as telescopes can see. I think I have to bite the bullet here: compared to the chances, my cre dences should favour worlds that contain many clones of myself and my en vironment.  18  The consolation is that this won't interfere with ordinary appli cations of the Principal Principle. For example, when it comes to a fair coin toss, one should still give heads credence 1/2, so long as the expected num ber of one's clones doesn't depend on the toss. It is true that Proportionality, rather than PP, is more directly applicable. So once we take into account the possibility of clones, Proportionality may be the best way to think about the chancecredence link.
 Appendix: Derivation of Proportionality The argument will assume that there is a sufficiently rich space of hypotheses. Instead of formulating general conditions, I'll just state exactly what I'll use in terms of the hypothesis E and the predicates F and G . (a) E is nonatomic: there is a neutral hypothesis A such that f (A|E ) ̸ = 0, 1. (It will be convenient to write A ′ for ¬A.) This first condition is a substantive restriction on the class of cases in which Proportionality follows formally from PP. Informally, though, it would be odd if Proportionality held for nonatomic E but not for E ; I also note that nonatomicity assumptions are common in axiomatic decision theory. Any way, in contrast, the following two conditions, while messier to formulate, are more vacuous: they say that there are enough neutral hypotheses and fully self locating hypotheses to carve up modal space in the ways one would expect. For example-recalling that each hypothesis has an intension in each scenariothe next two conditions hold if there is a hypothesis for every function from scenarios to intensions. (b) For all integers k ≥ j ≥ 0, there is a neutral hypothesis E j k whose intension contains a world w iff the primary intension of 〈I am G 〉& E calls neutral: 6 PP2. If E and H are neutral hypotheses, then Cr f
 How so? Starting from an ur prior Cr, we can (partially) define a Popper function Cr 0 that encodes judgements of evidential support given an arbitrary background of merely selflocating evidence. Restricting ourselves to neutral hypotheses H and E , the idea is that Cr 0 (H | E ) = p holds if and only if Cr(H | E & 〈I am G 〉) = p whenever 〈I am G 〉 is merely selflocating relative to E . With this definition, we can reformulate PP more simply as the claim that Cr 0
			 This paper is mainly a project in Bayesian epistemology, and I'll speak throughout about what one 'knows' as a shorthand for what evidence one has in the sense relevant to Bayesian conditionalization. This is a natural way of speaking, but nothing turns on the identification of evidence with knowledge.
			  See Hájek (2003)  for reasons one might take conditional probabilities as primitive. Un conditional probabilities can be recovered as probabilities conditional on a tautology.3 See e.g.Moss (2015, pp. 174-176)  for discussion of Ur Prior Conditionalization, and Titelbaum (2016) for some relevant alternatives.
			 To clarify the connection to Meacham's work: the hypothesis 〈ch = f 〉 takes the place of what he calls a 'chancegrounding' proposition.
			 This example is inspired by a similar one inHawthorne & LasonenAarnio (2009, pp. 95- 96).
			 Because of the conditionalization, it really suffices that E and H & E are neutral. And, while I won't focus on this issue here,Lewis (1980, pp. 268-9)  essentially points out that the Popper function f must also be given in a suitably neutral form. If I know that the chance of heads is x , and, unknown to me, x equals 1/4, then I'm under no compulsion to set my credence in heads to 1/4.
			 If heads actually comes up, there is no possible world in which Topper is tails. However, there are possible worlds in which tails comes up, and the thought is that, in any scenario that picks out such a world as actual, Topper is tails.
			 This depends on a seemingly harmless assumption, adopted by Chalmers, that every pos sible world is actual in some scenario.
			 The identification of scenarios with centered worlds, and the question of whether this is fully appropriate, are somewhat delicate; I defer to Chalmers (2011)  for discussion. The use of centred possible worlds to model selflocating ignorance is standard since at least Lewis (1979) , and most of the rest of this paper could be written in a Lewisian framework. Note though that Lewis claims the objects of belief are properties, whose intensions are sets of centred worlds. In contrast, for twodimensionalists, the (ordinary, not primary) intension of a hypothesis is still a set of possible worlds. See Magidor (2015)  for critique especially of the Lewisian tradition. By the way, it may be that some formally possible centered worlds do not represent genuine epistemic possibilities, even a priori. Perhaps it is a priori for me that I am not a rock; then (w, x , t ) does not correspond to an epistemic scenario, if x is a rock at time t in w. When I talk about centered worlds, I only mean those that represent genuine epistemic possibilities.
			 The example was made popular by Elga (2000) ; see his first footnote for its history.
			 On the first point, Lewis (2001)  claims that Beauty has inadmissible evidence once she learns it's Monday, but it seems hard to independently justify this claim, or to fit it into a systematic account of admissibility. On the second, see Titelbaum (2016)  for a survey of alternative updating methods and their problems.
			 This is a simple version of the Doomsday Argument treated explicitly by Leslie (1992)  and attributed to Brandon Carter. See Bostrom (2002)  for a discussion of its history. Note that your current evidence may well be fully selflocating even if you have little idea of your birthrank (cf. my discussion of knowing the time in §4.2). So this Doomsday Argument says nothing about what should happen if you were to learn your birthrank in real life. In fact, if you know that the difference between early doom and late doom is some future extinction event, then you've already ruled out having a birth rank incompatible with early doom, and learning your birth rank shouldn't have much (or any) interesting effect.
			 Proportionality is closely related to what Manley (2014) calls 'Typicality', but importantly different from what Arntzenius & Dorr (2017) call 'Proportion': roughly, the latter requires the stated credence to equal the expected proportion of G s that are F s.
			 This recipe is a little rough for the usual reason that there may be uncountably many relevant worlds, and we can't just sum over them; I'll give a more formal definition in the appendix.
			  Bostrom (2002, p. 162). The (not 'Strong') SelfSampling Assumption applies to ob servers, rather than observer moments, but that won't help with Sleeping Beauty cases, and is actually incompatible with SSSA.
			 I don't claim to solve all the related problems that arise from infinite worlds, for discussion of which see Bartha & Hitchcock (1999b) , Weatherson (2005) , and especially Arntzenius & Dorr (2017) . It's worth mentioning in this context that Popper functions need not be countably additive.
			  See Bostrom (2002, pp. 32ff) for a similar heuristic.
			  See Elga (2004); Weatherson (2005)  for a discussion of related problems.
			 A partition A 1 , . . . , A n of a hypothesis A is a collection of hypotheses such that, a priori, the A i are mutually exclusive and their disjunction is equivalent to A.


Abstract: This paper is the first installment in a series on "AI safety," an area of machine learning research that aims to identify causes of unintended behavior in machine learning systems and develop tools to ensure these systems work safely and reliably. Below, we introduce three categories of AI safety issues: problems of robustness, assurance, and specification. Other papers in this series elaborate on these and further key concepts.

Key Concepts in AI Safety: An Overview

Introduction The past decade has seen the emergence of modern artificial intelligence and a variety of AI-powered technological innovations. This rapid transformation has predominantly been driven by machine learning, a subfield of AI in which computers learn patterns and form associations based on data. Machine learning has achieved success in application areas including image classification and generation, speech and text generation, and decision making in complex environments such as autonomous driving, video games, and strategy board games. However, unlike the mathematical and computational tools commonly used in engineering, modern machine learning methods do not come with safety guarantees. While advances in fields such as control theory have made it possible to build complex physical systems, like those found in various types of aircraft and automobiles, that are validated and guaranteed to have an extremely low chance of failure, we do not yet have ways to produce similar guarantees for modern machine learning systems. As a result, many machine learning systems cannot be deployed without risking the system encountering a previously unknown scenario that causes it to fail. The risk of system failures causing significant harm increases as machine learning becomes more widely used, especially in areas where safety and security are critical. To mitigate this risk, research into "safe" machine learning seeks to identify potential causes of unintended behavior in machine learning systems and develop tools to reduce the likelihood of such behavior occurring. This area of research is referred to as "AI safety" 1 and focuses on technical solutions to ensure that AI systems operate safely and reliably. Many other challenges related to the safe deployment of AI systems-such as how to integrate them into existing networks, how to train operators to work effectively with them, and so onare worthy of substantial attention, but are not covered here. Problems in AI safety can be grouped into three categories: robustness, assurance, and specification. Robustness guarantees that a system continues to operate within safe limits even in unfamiliar settings; assurance seeks to establish that it can be analyzed and understood easily by human operators; and specification is concerned with ensuring that its behavior aligns with the system designer's intentions. 2
 Modern Machine Learning Machine learning methods are designed to learn patterns and associations from data. 3 Typically, a machine learning method consists of a statistical model of the relationship between inputs and outputs (for example, the relationship between an audio recording and a text transcription of it) and a learning algorithm specifying how the model should change as it receives more information about this input-output relationship. The process of updating the model as more data is made available is called "training," and recent advances in fundamental research and engineering have enabled efficient training of highly complex models from large amounts of data. Once trained successfully, a machine learning system can be used to make predictions (such as whether or not an image depicts an object or a human), to perform actions (such as autonomous navigation), or to generate synthetic data (such as images, videos, speech, and text). Many modern machine learning systems use deep neural networks-statistical models that can represent a wide range of complex associations and patterns and that work particularly well with large amounts of data. Examples of useful application areas for deep neural networks include image classification and sequential decision-making in autonomous systems, as well as text, speech, and image generation. Machine learning systems derive associations and patterns from data rather than from a prespecified set of rules. As a result, these systems are only as good as the data they were trained on. While modern machine learning systems usually work remarkably well in settings similar to those encountered during training, they often fail in settings that are meaningfully different. For example, a deep neural network trained to classify images of cats and dogs in black and white is likely to succeed at classifying similar images of cats and dogs in color. However, it will not be able to correctly classify a fish if it has never encountered an image of one during training. While machine learning systems do not use explicit rules to represent associations and patterns, they do use rules to update the model during training. These rules, also called "learning algorithms," encode how the human designer of a machine learning system wants it to "learn" from data. For example, if the goal is to correctly classify images of cats and dogs, the learning algorithm should include a set of steps that update the model to gradually become better at classifying cats and dogs. This goal can be encoded in a learning algorithm in many ways, and it is the task of the human designer of such a system to do so.
 Robustness In order to be reliable, a machine learning system must operate safely under a wide range of conditions. Building into the system the ability to quantify whether or not it is confident about a prediction may reduce the chance of failure in situations it is not well-prepared to handle. The system, upon recognizing it is in a setting it was not trained for, could then revert to a safe fallback option or alert a human operator. Challenging inputs for machine learning systems can come in many shapes and guises, including situations a system may never have encountered before (as in the fish classification example above). Operating safely in such scenarios means that a system must, first, recognize that it has not been trained for such a situation and, second, have a way to act safely-for example, by notifying a human operator to intervene. An active area of research around this problem seeks to train machine learning models to estimate confidence levels in their predictions. These estimates, called predictive uncertainty estimates, would allow the system to alert a human operator if it encounters inputs meaningfully different from those it was trained on. Consider, for example, a machine learning system tasked to identify buildings in satellite imagery. If trained on satellite imagery of a certain region, the system learns to identify buildings that look similar to those in the training data. If, when deployed, it encounters an image of a building that looks meaningfully unlike anything it has seen during training, a robust system may or may not classify the image as showing a building, but would invariably alert a human operator about its uncertainty, prompting manual human review.
 Assurance To ensure the safety of a machine learning system, human operators must understand why the system behaves the way it does, and whether its behavior will adhere to the system designer's expectations. A robust set of assurance techniques already exist for previous generations of computer systems. However, they are poorly suited to modern machine learning systems such as deep neural networks. Interpretability (also sometimes called explainability) in AI refers to the study of how to understand the decisions of machine learning systems, and how to design systems whose decisions are easily understood, or interpretable. This way, human operators can ensure a system works as intended and, in the case of unexpected behavior, receive an explanation for said behavior. It is worth noting that researchers and engineers working with and developing modern machine learning systems do understand the underlying mathematical operations inside so-called "black-box" models and how they lead from inputs to outputs. But this type of understanding is difficult to convert into typical human explanations for decisions or predictions-say, "I liked the house because of its large kitchen," or "I knew that dog was a Dalmatian because it had spots." Interpretability, then, seeks to understand how trained machine learning systems "reason"-that is, how certain types of inputs or input characteristics inform a trained system's predictions. Some of the best tools we have for this so far include generating visualizations of the mathematical operations inside a machine learning system or indicating which input characteristics are most responsible for a model's outputs. In high-stakes settings where humans interact with machine learning systems in real time, interpretability will be crucial in giving human operators the confidence to act on predictions obtained from such systems.
 Specification "Specification" of machine learning systems refers to defining a system's goal in a way that ensures its behavior aligns with the human operator's intentions. Machine learning systems follow a pre-specified algorithm to learn from data, enabling them to achieve a specific goal. Both the learning algorithm and the goal are usually provided by a human system designer. Examples of possible goals include minimizing a prediction error or maximizing a reward. During training, a machine learning system will try to reach the given goal, regardless of how well it reflects the designer's intent. Therefore, designers must take special care to specify an objective that will lead to the desired behavior. If the goal set by the system designer is a poor proxy for the intended behavior, the system will learn the wrong behavior and be considered "misspecified." This outcome is likely in settings where the specified goal cannot fully capture the complexities of the desired behavior. Poor specification of a machine learning system's goal can lead to safety hazards if a misspecified system is deployed in a high-stakes environment and does not operate as intended. Misspecification has already arisen as a problem in YouTube's video recommendation algorithms. This algorithm was designed to optimize for engagement-the length of time a user spends watching videos-to maximize ad revenue. However, an unintended side effect manifested: To maximize viewing time, in some cases, the recommendation algorithm gradually steered users toward extremist content-including videos from white supremacist and other political and religious extremist channelsbecause it predicted these recommendations would cause the user to stay engaged longer. The extent of this phenomenon remains disputed, and YouTube has changed its algorithms since this issue first gained considerable attention. Yet the underlying idea-that optimizing for engagement could have unintended effectsdemonstrates the hazards of goal misspecification. 4
 Conclusion Safety considerations must precede the deployment of modern machine learning systems in high-stakes settings. Robustness, assurance, and specification are key areas of AI safety that can guide the development of reliably safe machine learning systems. While all three are the subjects of active and ongoing research, it remains uncertain when we will be able to consider machine learning systems reliably safe.