Suffering-focused AI safety: Why "fail-safe" measures might be our top intervention

Classical AI safety looks very difficult. In order to ensure a truly "utopian" outcome, the following things need to happen: 1. The AI that gains a decisive strategic advantage over the competition needs to be built by the right group of people. 2. These people would need to have figured out how to program favorable values into an AI (or program the AI to learn these values). 3. They would also need to have come up with a satisfying description of what these values are, either directly, or indirectly with the help of a suitable extrapolation procedure (e.g.  Muehlhauser & Helm, 2012) . 4. Finally, the creators of the first superintelligence might need to get decision theory right (see  Soares & Fallenstein, 2015)  and safeguard the AI from a lot of hardto-anticipate failure modes. Succeeding with all of this together is the only safe way of bringing about a flourishing utopia mostly free of suffering. (There might be other paths to utopia, provided enough luck.) But what about outcomes where something goes wrong? It is important to consider that things can go wrong to very different degrees. For value systems that place primary importance on the prevention of suffering, this aspect is crucial: the best way to avoid bad-case scenarios specifically may not be to try and get everything right. Instead, it makes sense to focus on the worst outcomes (in terms of the suffering they would contain) and on tractable methods to avert them. As others are trying to shoot for a best-case outcome (and hopefully they will succeed!), it is important that some people also take care of addressing the biggest risks. This perspective to AI safety is especially promising both because it is currently neglected and because it is easier to avoid a subset of outcomes rather than to shoot for one highly specific outcome. Finally, it is something that people with many different value systems could get behind.
 How AI outcomes might contain suffering From a suffering-focused perspective, the main reason to be concerned about the risks from artificial intelligence is not the possibility of human extinction or the corresponding failure to build a flourishing, intergalactic civilization. Rather, it is the thought of misaligned or "illaligned" AI as a powerful but morally indifferent optimization process which, in the pursuit of its goals, may transform galactic resources into (among other things) suffering. Either like the suffering we see on Earth today, or by bringing about optimized structures that possibly also contain novel forms of suffering. The things a superintelligent AI would build to pursue its goals might include a fleet of "worker bots", factories, supercomputers, space colonization machinery, etc. It is possible that all of these end up being built in a way that does not permit suffering. However, given that evolution has produced plenty of sentient minds, this weakly suggests that some of the easiest ways to implement mind architectures do come with the capacity for suffering. In the absence of an explicit antisuffering preference, even the slightest benefit to the AI's objectives would lead to the instantiation of suffering minds. What makes this especially worrying is that the stakes involved will be huge: Space colonization is an attractive subgoal for almost any powerful optimization process, as it leads to control over the largest amount of resources  (Omohundro, 2008) . Even if only a small portion of these resources were used for purposes that include suffering, the resulting disvalue would sadly be astronomical. To find the best interventions to prevent suffering, it is crucial to study how AI outcomes differ in their expected amount of suffering and whether there are ways to favorably affect the likely outcomes. This paper aims to provide an overview on "bad-case scenarios" in the AI context and some general suggestions for how they could be avoided. Future work will include checking fundamental assumptions and looking at the most promising proposals in more detail.
 Type I: Controlled AI gone bad In these scenarios, the team that builds the first superintelligence knew what it was doing and managed to successfully shape the resulting AI's goals exactly the way they wanted it to go. Unfortunately, the goals are not what we would wish them to be and thus lead to bad or very bad consequences. Things that could go wrong include: a) Anthropocentrism: perhaps the resulting AI would not care about the suffering of "weird" and/or voiceless nonhuman minds such as suffering subroutines or animal minds in ancestor simulations. b) Retributivism: perhaps the AI gets built by people who want to punish members of an outgroup (e.g. religious fundamentalists punishing sinners). c) Uncooperative: perhaps the AI's goal is something like classical utilitarianism (or any other "monotone" maximizing function) with no additional regards for cooperation with other value systems. Even though such an AI would all else equal prefer to not create suffering, it seems possible that the anti-suffering concern would in practice be overridden by opportunity costs: if happiness simulations contain much more utility for the AI than the disutility produced by the accidental suffering created in the buildup, then such an AI would behave similarly "recklessly" as e.g. a paperclipmaximizing AI. 2 Another concern lies in how strongly the AI would value robustness to deterioration. d) Libertarianism regarding computations: perhaps the creators of the first superintelligence instruct the AI to give every human alive at the time of the singularity control of a planet or galaxy, with no additional rules to govern what goes on within those territories. Some of these human rulers are curious and amused by seeing others hurt, or worse, might be psychopaths.
 Type II: "Near misses" In this scenario, the first superintelligence is built by people who were aware of the risks, and who tried to get things right. Unfortunately, mistakes happened and the resulting outcome with "nearly-controlled AI" is bad or very bad. a) Miserable creatures: perhaps the AI's goal function includes terms that attempt to specify sentient or human-like beings and conditions that are meant to be good for these beings. However, because of programming mistakes, unanticipated loopholes or side-effects, the conditions specified actually turn out to be bad for these beings. Worse still, the AI has a maximizing function and wants to fill as many regions of the universe as possible with these poor creatures. b) Black swans: perhaps the AI cares about sentient or human-like minds in "proper" ways, but has bad priors, ontology, decision theory, or other fundamental con-2 An AI that values happiness simulations but takes cooperation with suffering reducers into account would in expectation still create some amount of suffering, but depending on just how costly it is to use more "sufferingproof" algorithms in the colonization process, or to do fewer or less fine-grained ancestor simulations, it might be possible to cut down on most of the instrumentally produced suffering at a cost that isn't very high (e.g. not higher than 15% of total happiness simulations produced). stituents that would make it act in unfortunate and unpredictable ways.
 Type III: Uncontrolled AI Uncontrolled AI is stipulated to have goals that were never intended by its creators, or at least were not intended to take over control in the form of a singleton  (Bostrom, 2006) . Unless the AI in question has a neuromorphic design, it is unlikely that its goals would (directly) have something to do with sentient beings. However, even non-neuromorphic uncontrolled AI might instrumentally instantiate suffering minds in the process of achieving its goal, as a side-effect of the computations it performs. The following presents a list of ways how uncontrolled AI might create vast amounts of suffering: • Suffering subroutines: For tasks like inventing and developing advanced technologies, as well as for coordinating its expansion into space for resource accumulation, the AI would have to rely on a fleet of "robot workers" or "robot scientists" of various kinds (which, of course, might work and function very unlike human workers or scientists). It is possible that these processes would be capable of suffering  (Tomasik, 2014) , although it is unclear whether would perform at their best if they (occasionally) suffer. What is clear is that if they do suffer under useful circumstances, an AI that is not explicitly aligned with compassionate goals would not have any qualms about instantiating astronomical numbers of them. • Ancestor simulations  (Bostrom, 2003; Bostrom, 2014, Ch.8) : in order to gather information relevant to its goals, e.g. for improving its understanding of human psychology or sociology  (Bostrom, 2014, pp. 125-26)  or for studying the density of aliens in the universe and what their likely values are, a superintelligence might simulate many runs of Darwinian evolution in planet-sized supercomputers  (Sandberg, 1999) . It is possible that these simulations would be fine-grained enough to contain sentient minds. • Warfare: it may be that the density of life in the universe is high enough for colonizing AIs to eventually encounter one another. If so, uncontrolled AI might end up clashing with other superintelligences, including AIs that build expanding civilizations of happy sentient beings. Perhaps they could agree to a compromise, but such an encounter could also lead to fighting over resources and warfare at the points of contact. It is interesting to note that none of the aforementioned scenarios involve large quantities of suffering in the types of structures that the AI directly optimizes for. It seems likely that what makes uncontrolled AI bad is what it builds for instrumental reasons, including colonization machinery, science simulations, and other strategic computations. This suggests that opportunity costs are relevant: If an uncontrolled AI's goal function does not have diminishing returns on resources (like the infamous "paperclip maximizer"), then the room for instrumentally important computations might be small. By contrast, an AI with goals that are easy to fulfill 3 -e.g. a "paperclip protector" that only cares about protecting a single paperclip, or an AI that only cares about its own reward signal -would have much greater room pursuing instrumentally valuable computations. This line of thought suggests that outcomes from uncontrolled AI should be steered away from "paperclip-protector types." 4 3 Suffering-focused AI safety: Some proposals Bad-case scenarios can be avoided in several ways. Using the typology from above, we can distinguish three classes of interventions: • Influencing outcomes with controlled AI: influence the values implemented in outcomes with controlled AI either through value spreading or by supporting the AI safety project with the best values The following can be labelled "fail-safe" measures: • Targeted AI safety: a) Differential progress: push classical AI safety differentially in ways that best insulates it from the worst "near misses" b) AI safety where it is most needed: identify the paradigm in general AI that would create the worst outcomes in the event of a control failure, and work towards identifying (general) control or shut-down mechanisms in those areas • Graceful fails: research ways to make AI fail in a "benign" way conditional on it failing ("It might be the end of the world, but it could have been (much) worse") The following sections present some preliminary ideas for the type of interventions that could be done in those areas.
 Influencing outcomes with controlled AI This class of interventions is primarily targeted at "controlled AI gone bad" outcomes. It is something FRI and others in the EA community have already thought about a great deal. • Improve values: raise awareness of antispeciesism, anti-substratism, concern for weird minds or suffering-focused ethics in general. • Cooperation: promote the idea that the gains from cooperation make it important for different value-systems to work together. • Cooperative AI-safety efforts: by funding the AI-safety project with the best values or the best approach to cooperation, we make it more likely that controlled AI won't create vast amounts of suffering.
 Differential progress in AI safety This intervention is targeted at the worst outcomes in the "near misses" category. The goal is to figure out solutions to the worst problems first, such that any additional progress in AI safety is unlikely to lead to "near-misses," which may be worse than outcomes with uncontrolled AI in terms of suffering produced. This intervention seems comparatively more neglected than interventions in the category above. • Secure value implementation: Look into the pros and cons of various approaches to specifying an AI's values and work on the one least likely to lead to really bad outcomes. Questions to study may include: • What's more likely to go wrongsomething like indirect normativity, or an explicit specification of a goal function? • What are other promising approaches to AI safety, and how bad would the outcome be if they fail according to their weak points? • Would corrigibility reduce risks of astronomical suffering? • Foundational research: Which unanticipated failure modes could there be for controlled AI? Once they are identified, how can we prevent them?
 AI safety where it is most needed This intervention is targeted at preventing the worst outcomes involving uncontrolled AI. Not all uncontrolled AIs are expected to behave equally; there are different paradigms in AI research, i.e. different basic architectures for attempting to build smarter-than-human intelligence. Examples include reinforcement learning  (Sutton & Barto, 1998) , proof-based seed AI, and neuromorphic AIs  (Hasler & Marr, 2013) . Classical AI safety is necessarily concerned with the tradeoff between success probability and "controllability;" if a particular approach to building AI is likely to lead to superintelligence, but in a way where it is hard to control the values of the resulting agent, then this poses a likely dead end. By contrast, a suffering-focused "fail-safe" approach does not have to worry about controllability to the same extent, as long as there's enough controllability to predictably avoid outcomes with the most suffering. This suggests that, conditional on uncontrolled AI being worse than (nearly-)controlled AI in expectation, suffering reducers can be less constrained by issues with controllability, and can thus focus more on applying AI safety to the paradigm/architecture with the highest probability of bringing about superintelligence. In addition, we could try to identify differences among the types of uncontrolled AI each AI-paradigm would produce, in order to then focus on AI safety in the domain where failures of control result in the worst outcomes. Suppose for instance that one way of building AI appears to show promising results and rapid progress, yet control seems hard and likely failure modes are particularly worrying. In this case, we should consider putting efforts into AI safety applied to that particular domain. Specific examples of this more general idea include: • AI safety for machine learners: With DeepMind's recent success, it seems not unlikely that the first superintelligent AI may be all reinforcement learning as the top control structure (as opposed to GO-FAI, theorem proving or logic-based AI). In contrast to AI architectures where goals refer to the state of the world  (Hibbard, 2011)  rather than to internal or observational signals, pure reinforcement learners may be at a higher risk of wireheading, i.e. of hacking their input signals to continuously attain the maximum reward. If the wireheading AI is smart enough to plan into the future, it might patiently avoid doing so and develop plans for takeover first: As Bostrom notes in Superintelligence  (Bostrom, 2014, 122-23) , smart wireheaders would still want to colonize other galaxies for option value, protection, research, etc. However, unlike paperclip maximizers, wireheaders would more generally function like "paperclip protectors" in that they would have almost no opportunity costs  (Ring & Orseau, 2011) . If the observation that lower opportunity costs make outcomes with uncontrolled AI worse is accurate (see the discussion under "Type III: Uncontrolled AI), this would make AI safety applied to reinforcement learning architectures particularly promising. Specific interventions could include work on Interruptibility  (Armstrong & Orseau, 2016) , but also methods of value-loading (examples here and here). • AI safety for "em-first" scenarios: Conditional on whole brain emulation  (Sandberg & Bostrom, 2008; Hanson, 2016)  becoming technologically feasible before the advent of de-novo AI, de-novo AI as eventually developed will more likely exhibit a neuromorphic design. 5 This would make it more likely that the values are humanlike, which also comes with the possi-bility of "bad values" or "near misses." AI safety applied to "em-first" scenarios could thus be a promising intervention (although with early arrival timelines being longer, it may not be favored by haste considerations). Caveat: Pursuing any of the interventions above would require that we make sure to not speed up progress in AI development -both in general and in "unsafe" or "hard-to-control" approaches specific to those interventions.
 Graceful fails This class of interventions is targeted at preventing the worst outcomes involving "nearlycontrolled" or uncontrolled AI. Instead of trying to make fails less likely, the idea is to come up with safety nets or mechanisms such that, if control fails, the outcome will be as good as it gets under the circumstances. Graceful fails are easiest to bring about in the domain of near misses, as the researchers working on AI safety are aware of the risks and thus open to the idea of safety nets. • Backup goal functions: Research ways to implement multi-layered goal functions, with a "backup goal" that kicks in if the implementation of the top layer does not fulfill certain safety criteria. The backup would be a simpler, less ambitious goal that is less likely to result in bad outcomes. Difficulties would lie in selecting the safety criteria in ways that people with different values could all agree on, and in making sure that the backup goal gets triggered under the correct circumstances. In the context of AI research that is not guided by safety precautions, graceful fails proposals seem harder to implement. The challenge is to get an AI project that might result in uncontrolled AI to care about whatever proposal there is to implement. Perhaps there are proposals to consider in rather early stages of AI development where researchers think hard takeoffs are very unlikely. Should the AI design in question nevertheless undergo an intelligence explosion, there might be ways to make the result counterfactually less bad. • Dummy goals: We could come up with a goal function with the following properties: • Easy to program • Interesting (in the sense that it's easy to create problems for an AI with this goal function to solve, such that researchers can track their progress on general intelligence) • Leads to decent outcome in the unlikely case of hard takeoff If AI research and testing at early stages is always conducted with goals of this type, an unanticipated hard takeoff would at least be comparatively benign.
 Concluding thoughts Suffering-focused AI safety, and "fail-safe" measures in particular, make up a large and promising area of interventions for FRI (and perhaps other organizations) to investigate further. Advantages are that they are neglected and often "less ambitious" than classical AI safety, which means they might end up being more tractable. Moreover, "fail-safe" measures are a promising project to focus on because the interventions discussed are positive or at worst "unobjectionable" from the perspectives of virtually all other value systems. The main reason we had not explicitly zoomed in on this category of interventions earlier was that FRI was initially too focused on answering the more general, complicated question whether AI safety on the whole is net positive for suffering reducers. Upon reflection, this question may be less important. AI safety is a very broad category where we should expect a lot of room for targeted efforts. The general lesson to draw is that partitioning broad categories can be an important step towards making progress. More research into the feasibility of suffering-focused AI safety carries high information value for this very reason: As we look into more proposals and their practical feasibility, we might uncover more relevant distinctions among AI outcomes, valuetransfer proposals, or general approaches to AI -a process which, in turn, will make it easier to identify effective interventions. Influencing outcomes with controlled AI . . . . . . . . . . . . . . . . . . . . . . 3.2 Differential progress in AI safety . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 AI safety where it is most needed . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Graceful fails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Introduction Contents 1 Introduction 2 How AI outcomes might contain suffering 2.1 Type 3 Suffering-focused AI safety: Some proposals 3.1 4 Concluding thoughts I: Controlled AI gone bad . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Type II: "Near misses" . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Type III: Uncontrolled AI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
			 "Fail-safe" in the sense that if control fails, at least the AI causes less suffering than would have been the case without fail-safe measures.
			 Thanks to Carl Shulman for bringing this idea to our attention. 4 However, it may be that paperclip protectors are easier to compromise with, and thus more likely to also pursue maximization of sorts in exchange for benefits from "paperclip-maximizer types."
			 If only because neuroscience research would be very advanced in such a world.

Abstract: AI-safety efforts focused on suffering reduction should place particular emphasis on avoiding risks of astronomical disvalue. Among the cases where uncontrolled AI destroys humanity, outcomes might still differ enormously in the amounts of suffering produced. Rather than concentrating all our efforts on a specific future we would like to bring about, we should identify futures we least want to bring about and work on ways to steer AI trajectories around these. In particular, a "fail-safe" 1 approach to AI safety is especially promising because avoiding very bad outcomes might be much easier than making sure we get everything right. This is also a neglected cause despite there being a broad consensus among different moral views that avoiding the creation of vast amounts of suffering in our future is an ethical priority.


Extracting Money from Causal Decision Theorists

Introduction In Newcomb's problem  [Nozick, 1969; Ahmed, 2014] , a "being" offers two boxes, A and B. Box A is transparent and contains $1,000. Box B is opaque and may contain either $1,000,000 or nothing. An agent is asked to choose between receiving the contents of both boxes, or of box B only. However, the being has put $1,000,000 in box B if and only if the being predicted that the agent would choose box B only. The being's predictions are uncannily accurate. What should the agent do? Causal decision theory (CDT) recommends that the agent reason as follows: I cannot causally affect the content of the boxes -whatever is in the boxes is already there. Thus, if I choose both boxes, regardless of what is in box B, I will end up with $1,000 more than if I choose one box. Hence, I should choose both boxes. Evidential decision theory (EDT), on the other hand, recommends that the agent reason as follows: if I choose one box, then in all likelihood the being predicted that I would choose one box, so I can expect to walk away with $1,000,000. (Even if the being is wrong some small percentage of the time, the expected value will remain at least close to $1,000,000.) If I choose both, then I can expect to walk away with (close to) $1,000. Hence, I should choose one box. While Newcomb's problem itself is far-fetched, it has been argued that the difference between CDT and EDT matters in various game-theoretic settings. First, it has been pointed out * Contact Author that Newcomb's problem closely resembles playing a Prisoner's Dilemma against a similar opponent  [Brams, 1975; Lewis, 1979] . Whereas CDT recommends defecting even in a Prisoner's Dilemma against an exact copy, EDT recommends cooperating when facing sufficiently similar opponents. What degree of similarity is required and whether EDT and CDT ever come apart in this way in practice has been the subject of much discussion  [Ahmed, 2014, Section 4.6] . A common view is that CDT and EDT come apart only under fairly specific circumstances that do not include most human interactions.  Still, Hofstadter [1983] , for example, argues that "superrational" humans should cooperate with each other in a real-world one-shot Prisoner's Dilemma, reasoning in a way that resembles the reasoning done under EDT. Economists have usually been somewhat dismissive of such ideas, sometimes referring to them as "(quasi-)magical thinking" when trying to explain observed human behavior  [Shafir and Tversky, 1992; Masel, 2007; Daley and Sadowski, 2017] . Indeed, standard game-theoretic solution concepts are closely related to ratificationism  [Joyce and Gibbard, 1998, Section 5 ], a variant of CDT which we will revisit later in this paper (see Part 3 of Section 4). Second, even if the players of a game are dissimilar, it is in the nature of strategic interactions that each player's behavior is predicted by the other players. Newcomb's problem itself (as well as the ADVERSARIAL OFFER discussed in this paper) differs from strategic interactions as they are usually considered in game theory in its asymmetry. However, one might still expect that CDT and EDT offer different perspectives on how rational agents should deal with the mutual prediction inherent in strategic interactions  [Gauthier, 1989, Section XI] . Third, CDT and EDT differ in their treatment of situations with imperfect recall. Seminal discussions of such games are due to  Piccione and Rubinstein [1997 ] and Aumann et al. [1997 ] [cf. Bostrom, 2010 , for an overview from a philosopher's perspective]. While these problems originally were not associated with Newcomb's problem, the relevance of different decision theories in this context has been pointed out by  Briggs [2010 ] [cf. Armstrong, 2011 Schwarz, 2015; Conitzer, 2015] . The importance of the differences in decision theory is amplified if, instead of humans, we consider artificial agents. After all, it is common that multiple copies of a software sys-tem are deployed and other parties are often able to obtain the source code of a system to analyze or predict its behavior. As some software systems choose more autonomously, we might expect their behavior will be (approximately) describable by CDT or EDT (or yet some other theory). If either of these theories has serious flaws, we might worry that if a system implements the wrong theory, it will make unexpected, suboptimal choices in some scenarios. Such scenarios might arise naturally, e.g., as many copies of a system are deployed. We might also worry about adversarial problems like the one in this paper. One argument against CDT is that causal decision theorists (tend to) walk away with less money than evidential decision theorists, but this argument has not proved decisive in the debate. For instance, one influential response has been that CDT makes the best out of the situation -fixing whether the money is in box B -which EDT does not  [Joyce, 1999, Section 5.1] . It would be more convincing if there were Newcomb-like scenarios in which a causal decision theorist volunteers to lose money (in expectation or with certainty). 1 Constructing such a scenario from Newcomb's problem is non-trivial. For example, in Newcomb's problem, a causal decision theorist may realize that box B will be empty. Hence, he would be unwilling to pay more than $1,000 for the opportunity to play the game. In this paper, we provide Newcomb-like decision problems in which the causal decision theorist voluntarily loses money to another agent. We first give a single-decision scenario in which this is true only in expectation (Section 2). We then extend the scenario to create a diachronic Dutch book against CDT -a two-step scenario in which the causal decision theorist is sure to lose money (Section 3). Finally, we discuss the implications of the existence of such scenarios (Section 4). 1 Walking away with the maximum possible (expected) payoff under any circumstances is not a realistic desideratum for a decision theory: any decision theory X has a lower expected payoff than some other decision theory Y in a decision problem that rewards agents simply for using decision theory Y [cf.  Skalse, 2018 , for a harderto-defuse version of this point]. However, such a setup does not allow one to devise a generic scenario in which an agent voluntarily loses money, i.e. loses money in spite of having the option to walk away losing nothing. Furthermore, scenarios with voluntary loss appear significantly more problematic for pragmatic reasons. Regardless of what you think is the right option in Newcomb's problem, you might not view Newcomb's problem as relevant ground for decision-theoretical argument because it is so unlikely that one would ever face Newcomb's problem in the real world. For instance, even if you thought that one-boxing is rational (and two-boxing is not), you might stick with CDT nonetheless because your real-world expected opportunity costs from two-boxing in Newcomb's problem are negligible. [For some discussion of this deflationary argument, see, e.g.,  Gauthier, 1989, Section XI; Ahmed, 2014, Section 7.1.iv; Oesterheld, 2019 , Section 1, and references therein.] However, if there is a Newcomb-like problem in which the causal decision theorist voluntarily loses money to some other agent, this generates a significant incentive to place him in such a situation.
 Extracting a profit in expectation from causal decision theorists Consider the following scenario: ADVERSARIAL OFFER: Two boxes, B 1 and B 2 , are on offer. A (risk-neutral) buyer may purchase one or none of the boxes but not both. Each of the two boxes costs $1. Yesterday, the seller put $3 in each box that she predicted the buyer not to acquire. Both the seller and the buyer believe the seller's prediction to be accurate with probability 0.75. No randomization device is available to the buyer (or at least no randomization device that is not predictable to the seller). 2 If the buyer takes either box B i , then the expected money gained by the seller is $1 − P ($3 in B i | buyer chooses B i ) • $3 = $1 − 0.25 • $3 = $0.25. Hence, the buyer suffers an expected loss of $0.25 (if he buys a box). The best action for the buyer therefore appears to be to not purchase either box. Indeed, this is the course of action prescribed by EDT as well as other decision theories that recommend one-boxing in Newcomb's problem [e.g., those proposed by  Spohn, 2012; Poellinger, 2013; Soares and Levinstein, 2017] . In contrast, CDT prescribes that the buyer buy one of the two boxes. Because the agent cannot causally affect yesterday's prediction, CDT prescribes to calculate the expected utility of buying box B i as P ($3 in box B i ) • $3 − $1, (1) where P ($3 in box B i ) is the buyer's subjective probability that the seller has put money in box B i , prior to updating this belief based on his own decision. For i = 1, 2, let p i be the probability that the buyer assigns to the seller having predicted him to buy B i . Similarly, let p 0 be the probability the buyer assigns to the seller having predicted him to buy nothing. These beliefs should satisfy p 0 + p 1 + p 2 = 1. Because p 0 ≥ 0, we have that (p 0 +p 1 )+(p 0 +p 2 ) = 2p 0 +p 1 +p 2 ≥ 1. Hence, it must be the case that p 0 + p 1 ≥ 1 2 or p 0 + p 2 ≥ 1 2 (or both). Because P ($3 in box B i ) = p 0 + p 3−i for i = 1, 2, it is P ($3 in box B i ) ≥ 1 2 for at least one i ∈ {1, 2}. Thus, the expected utility in eq. 1 of at least one of the two possible purchases is at least 1 2 • $3 − $1 = $0.50, which is positive. Any seller capable of predicting the causal decision theorist sufficiently well will thus have an incentive to use this scheme to exploit CDT agents. (It does not matter whether the seller subscribes to CDT or EDT.) It should be noted that even if the buyer uses CDT, his view of the deal matches the seller's as soon as the dollar is paid. That is, after observing his action, he will realize that the box he bought is empty with probability 0.75 and thus worth less than a dollar. CDT knows that it will regret its choice [see  Joyce, 2012; Weirich, 1985  for discussions of the phenomenon of anticipated regret a.k.a. decision instability in CDT].
 A diachronic Dutch book against causal decision theory ADVERSARIAL OFFER results in a loss in expectation for the causal decision theorist. It is natural to ask whether it is possible to set up the scenario so that the causal decision theorist ends up with a sure loss; effectively, a Dutch book. Arguably, Dutch books are more convincing than scenarios with expected losses since the very meaning of "expectations" is the subject of the debate about EDT and CDT. Of course, if the seller could perfectly predict the buyer in ADVERSARIAL OFFER (instead of being right only 75% of the time), then ADVERSARIAL OFFER would become a Dutch book. But can we construct a Dutch book without perfect prediction? We have already observed that in ADVERSARIAL OFFER the causal decision theorist always regrets his decision after observing its execution. This suggests the following simple approach to constructing a Dutch book. After the box is sold, the seller allows the buyer to reverse his decision for a small fee (ending up without any box and having lost only the fee). However, a CDT buyer may then anticipate eventually undoing his choice and therefore not buy a box in the first place  [Ahmed, 2014, Section 3.2; though cf. Skyrms, 1993; Rabinowicz, 2000 ].  3  To get our Dutch book to work, we add another choice before ADVERSARIAL OFFER. ADVERSARIAL OFFER WITH OPT-OUT: It is Monday. The buyer is scheduled to face the AD-VERSARIAL OFFER on Tuesday. He also knows that the seller's prediction was already made on Sunday. As a courtesy to her customer, the seller approaches the buyer on Monday. She offers to not offer the boxes on Tuesday if the buyer pays her $0.20. Note that the seller does not attempt to predict whether the buyer will pay to opt out. Also, we assume that the buyer cannot, on Monday, commit himself to a course of action to follow on Tuesday. It seems that a rational agent should never feel compelled to accept the Monday offer. After all, doing so loses him money with certainty, whereas simply refusing both offers (on Monday and on Tuesday) guarantees that he loses no money. CDT, however, recommends opting out on Monday, for the following reasons. A CDT buyer knows on Monday that if he does not opt out, he will buy a box on Tuesday (though he may not yet know which one). Further, he believes that whatever box he will take on Tuesday will contain $3 with only 25% probability, thus implying an overall expected payoff of 0.25 • $3 − $1 = −$0.25. This is because, on Monday, CDT treats the decision on Tuesday in the same way as it treats any other random variable in the environment. So the causal expected utility of not opting out is just what an outside observer would expect the payoff of a CDT agent facing ADVERSARIAL OFFER to be. Because this expected payoff of −$0.25 is less than the certain payoff of −$0.20 that can be obtained by opting out, CDT recommends opting out. In fact, for the argument in the previous paragraph to succeed, it is only necessary that CDT is used on Tuesday; other decision theories would also recommend accepting the Monday offer, if they anticipate that the agent will use CDT on Tuesday. For instance, if the agent followed EDT on Monday and CDT on Tuesday (and is aware on Monday that he will use CDT on Tuesday), then he would still accept the Monday offer. Similarly, if the seller believes that the buyer will pick one of the boxes on Tuesday, then she will hope that he rejects the Monday offer. Thus, it seems that what creates the opportunity for a Dutch book is the prospect of buying a box on Tuesday (as CDT recommends), not the use of CDT on Monday.
 Discussion We differentiate four types of responses to these scenarios available to supporters of causal decision theory: 1. They could claim that these scenarios are irrelevant for evaluating decision theories, in the sense that they are impossible to set up or otherwise out of scope, and therefore unpersuasive. 2. They could concede that these scenarios are relevant for evaluating decision theories, but claim that CDT's recommendations in them are acceptable. 3. They could concede that our analysis obliges them to give up on certain specific formulations of CDT, but try to modify CDT to get these scenarios right while maintaining some of its essence, in particular two-boxing and the causal dominance principle. 4. They could concede that these scenarios show that the very core of CDT (two-boxing and the causal dominance principle) is implausible. We will discuss these options in turn. 1 Surely, if one could show that a CDT agent will or can never face these scenarios -despite the seller having an obvious incentive to set them up -that would be the most convincing defense of CDT. In particular, a causal decision theorist might claim that sufficiently accurate prediction of a CDT agent is simply impossible. 4 However, not much accuracy is required, for the following reasons. The CDT agent will take one of the two boxes. Even if the seller picks the box to fill with money uniformly at random, she would therefore be right half of the time. If she can do any better than that, predicting correctly with probability 1/2+ , then she can extract money from the CDT agent by putting (instead of $3) some amount between $2/(1 − 2 ) and $2 in the box predicted not to be taken. Thus, the CDT agent needs to be completely unpredictable in order to avoid being taken advantage of in these examples. Most human beings are, generally speaking, at least somewhat predictable in their actions even when such predictability can be used against them. For example, in rock-paperscissors -which structurally resembles the ADVERSARIAL OFFER -most people follow exploitable patterns in what moves they select [see, e.g.,  Farber, 2015, and references therein] .  5  Consider such a somewhat predictable person who aims to be a causal decision theorist. It seems that he would indeed be vulnerable to the examples discussed earlier. The only defense for the supporter of causal decision theory would seem to then be that if so, the person in question is not truly acting in the way that CDT describes. That is, acting according to CDT also requires being unpredictable to the seller, either by succeeding at out-thinking the seller sufficiently often, or by acting sufficiently randomly. Is it reasonable to consider this a requirement of acting according to CDT? CDT does not suggest any strict preference for choosing randomly across options, as opposed to just deterministically choosing one of the options that is best according to the buyer's beliefs. Hence, the unpredictability would have to emerge from the buyer attempting to out-think the seller. But it does not seem that this is always an attainable goal. For example, imagine that the buyer is a deterministic computer program whose source code is known to the seller. Then regardless of how exactly the agent works, the seller can predict the buyer's behavior perfectly [cf. Soares and Fallenstein, 2014, Section 2; Cavalcanti, 2010, Section 5]. We would thus be forced to conclude that such a program cannot possibly follow CDT, which to us is an unsatisfactory conclusion. Plausibly any other physically realized agent that chooses deterministically can at least in principle (if not with current technology) be predicted by creating or emulating an atom-by-atom copy of that agent [cf.  Yudkowsky, 2010, pp. 85ff.] . Even if the supporter of CDT acknowledges that these scenarios are possible, he might nevertheless argue that they are irrelevant, in the sense that the decision theory is not intended to be used for such scenarios and hence nothing that one could show about its performance in such a scenario is of significance for evaluating the theory. "It is as if one evaluated a car by testing how it performs underwater." There is little we can say about this response. Still, we expect it to be unattractive to most decision theorists. After all, our scenarios (in particular the ADVERSARIAL OFFER) resemble Newcomb's problem -the problem that has led to the development of CDT in the first place. Further, if our scenarios were out of CDT's scope, then we (and presumably most other decision theorists) would still be interested in identifying a decision theory that does make good recommendations for predictable agents (such as artificial intelligent agents whose behavior is 5 There are multiple rock-paper-scissors bots available online which attempt to predict their opponent's future moves based on past moves (using data from other players). As of July 2019, the bot at http://www.essentially.net/rsp/ has reportedly played about 2 million rounds and won 57% more often than it lost. determined by a computer program) facing a wide range of scenarios including the ones given in this paper. 2 If our scenarios are within the scope of causal decision theory, then the supporter of causal decision theory has to contend with the fact that one can extract expected money from, and even Dutch-book, CDT agents in them. But he might question the significance of Dutch-book arguments and other money extraction schemes, either in general or in this particular context. For some general discussion of whether (diachronic) Dutch books are conclusive decision-theoretic arguments, see, e.g.,  Vineberg [2016]  or  Hájek [2009] . Note, though, that some of the most influential arguments in favor of expected utility maximization (EUM) -of which CDT is a refinement -are Dutch books. Of course, one may use different arguments to justify EUM. But it would seem odd to follow Dutch-book arguments to EUM but no further. Instead of rehashing some of the more generic reasons for and against the persuasiveness of Dutch books and loss of money in expectation, we here discuss a response that is specific to CDT and ADVERSARIAL OFFER WITH OPT-OUT.  6  A causal decision theorist may argue that it is not generally fair to expect any kind of coherence from CDT's recommendations when multiple decisions are to be made across time, due to the different perspectives that the decision maker adopts (and, arguably, has to adopt) at different points in time. Consider Newcomb's problem. Let t 0 be the time at which the predictor observes the agent (perhaps using fMRI or the like) in order to make a prediction. Then, before t 0 , CDT recommends committing -and if needed paying money to commit -to one-boxing [cf.  Barnes, 1997; Joyce, 1999, pp. 153f.; Meacham, 2010] . After t 0 , CDT recommends two-boxing. However, most decision theorists do not consider this to be a compelling argument against CDT. The causal decision theorist can easily justify the difference in the decision made by the fact that, before t 0 , the commitment decision has a causal effect on what is in the boxes, and after t 0 , it does not. It would be hypocritical for an evidential decision theorist to disagree, since EDT is dynamically inconsistent in analogous ways. For instance, consider a version of Newcomb's problem in which both boxes are transparent  [Gibbard and Harper, 1981, Section 10 ; also discussed by  Gauthier, 1989; Drescher, 2006, Section 6.2; Arntzenius, 2008, Section 7; Meacham, 2010, Section 3.2.2] . Let t 0 be the time at which the EDT agent sees the content of both boxes. Then before t 0 , EDT recommends committing -and if needed paying money to commit -to one-boxing. After t 0 , EDT recommends twoboxing.  7  The evidential decision theorist can easily justify this along similar lines: before t 0 , her commitment is evidence about what is in the boxes, and after t 0 it no longer is. Thus, at least some types of dynamic inconsistency do not constitute strong arguments against a decision theory. However, in our opinion, the dynamic inconsistency displayed by CDT in the ADVERSARIAL OFFER WITH OPT-OUT is much more problematic. For one, it leads to a Dutch book. Often, the main argument that is given for why a particular inconsistency is problematic is precisely that it allows for a Dutch book. Conversely, defenses of dynamic inconsistencies  [Ahmed, 2014, Section 3.2 , for an example in a Newcomb-like scenario] often focus on arguing that they do not allow for Dutch Books. Further, it seems that some of the reasons for (or defenses of) dynamic inconsistency in the above decision problems do not apply to CDT's dynamic inconsistency in ADVERSARIAL OFFER WITH OPT-OUT. For CDT in Newcomb's problem, there is a particular event at time t 0 that splits the decision perspectives: the loss of causal control at t 0 over the content of box B. Similarly, for EDT in the Newcomb's problem with transparent boxes, that event is the loss of evidential control [cf. Almond, 2010, Section 4.5] at t 0 over the content of box B. It is thus easy to argue for defenders of the respective theories that the perspectives from before and after t 0 or t 0 should diverge  [Ahmed and Price, 2012, pp. 22-23, Section 4] . In sharp contrast, the ADVERSARIAL OFFER WITH OPT-OUT lacks any such event between the decision points. The difference in perspectives for CDT appears to be purely a result of CDT viewing its current choice differently than it views past and future decisions. All that being said, we agree that caution should be taken when evaluating a decision theory based on scenarios with multiple decisions across time. In general, more research on what conclusions can be drawn from such scenarios is needed  [Steele and Stefánsson, 2016, Section 6 ]. Nevertheless, we do not see any clear path by which such research would justify CDT's recommendations in the ADVERSARIAL OFFER WITH OPT-OUT. In any case, even if one is at this point unwilling to consider scenarios with multiple decision points at all for the purpose of evaluating decision theories, one would still have to contend with the simpler ADVERSARIAL OFFER scenario, in which there is only one decision point. 3 If a straightforward interpretation of CDT cannot be defended against our scenarios, one may look to modify it to avoid expected or sure loss while preserving some of CDT's core tenets. In particular, in response to other alleged counterexamples, some authors have tried to modify CDT while maintaining the causal dominance  [Joyce, 1999, Section 5 .1] a.k.a. sure thing  [Gibbard and Harper, 1981, Section 7]  principle [though see  Ahmed, 2012 , for an argument against the motivation behind some of these approaches]. For example, one may turn to the concept of ratifiability. In Newcomb-like scenarios such as those under discussion here, for any choice a, we can consider the beliefs about what is in the boxes that would result from knowing that one will choose a. Then, a choice a is ratifiable if it is an optimal choice -as judged by CDT -under those beliefs. For example, in Newcomb's problem only two-boxing is ratifiable, precisely because it is causally dominant. For an overview of ratification and its relation to CDT, see  Weirich [2016, Section 3.6 ]. Unfortu-nately, this concept is of no help in the ADVERSARIAL OF-FER, because none of the three options (buying B 1 , buying B 2 or declining) is ratifiable. For instance, under the beliefs that would result from knowing that you will take box B i , it would be better to buy the other box B 3−i . The ratificationist may respond by claiming that unpredictable randomization should always be possible. If that were true, then the only ratifiable option would be to take each box with probability 50%, thus gaining money in expectation. But again, we would like to have a decision theory that works in a broad variety of scenarios, including ones where the agent expects to be somewhat predictable. Furthermore, even if a true random number generator (TRNG) (e.g., one based on nuclear decay) is in fact available, this does not settle the issue. For example, consider a variant of the AD-VERSARIAL OFFER in which the seller refrains from putting money in any box if she predicts the buyer to make different choices depending on the output of the TRNG. In this ANTI-RANDOMIZATION ADVERSARIAL OFFER, again no option is ratifiable: under the beliefs that would result from knowing that you will make different choices depending on the TRNG's output (and therefore choose a box with some positive probability), you would rather not pick any box. To circumvent this example, the ratificationist could argue that the decision maker should be able to randomize in such a way that whether he is randomizing is unpredictable. However, at this point, one might just as well assert the impossibility (or irrelevance) of Newcomb-type scenarios altogether, which we have addressed in 1. A different strategy for modifying CDT to avoid the Dutch book in the ADVERSARIAL OFFER WITH OPT-OUT is the following. The Dutch book arises from a disagreement between CDT on Monday and CDT on Tuesday (cf. the discussion under 2). A tempting possibility is to modify CDT so that it considers all decisions to be made at once. That is, such a version of CDT -let us refer to it as policy-CDT -prescribes that one decide on one's general policy all at once.  8  In the ADVERSARIAL OFFER WITH OPT-OUT, there are four possible policies: opt out, buy B 1 , buy B 2 , and buy nothing (where the last three possibilities include declining the opt-out offer). When considering these policies, buy nothing dominates opt out. Hence, policy-CDT will decline the opt-out offer and thereby avoid the Dutch book. (Note, however, that such a modification of CDT will make no difference to the choices it prescribes in ADVERSARIAL OFFER, which has only one decision point. Hence, it will still lose money in expectation.) While this appears to be a promising approach, it is nontrivial to flesh out, because on other examples it is less clear what policy-CDT should prescribe. For illustration, consider the following interpretation of policy-CDT: follow the policy to which CDT would like to commit ex ante, where "ex ante" refers to some point in time before the first decision of the scenario. Now, let us consider a version of Newcomb's problem which is supplemented by another trivial and unrelated decision -say, whether to eat a peppermint -that takes place when the agent still has a causal influence over the prediction. Then the ex-ante-commitment interpretation of policy-CDT would recommend one-boxing. To the causal decision theorist, this may be unacceptable, especially given that adding the peppermint decision is such a minor modification of Newcomb's problem. Perhaps there is a way to define policy-CDT that avoids such dependence on irrelevant decisions while also prescribing two-boxing, but it is not immediately obvious how to do so. Many other ways of modifying CDT are worth considering. For instance, in the ADVERSARIAL OFFER, it may be unrealistic for the buyer to form a single probability distribution over box contents. Instead, he may consider multiple different probability distributions, including one under which box B 1 is probably empty and one under which box B 2 is probably empty. He could then evaluate each option pessimistically, i.e., w.r.t. the probability distribution that is worst under that option. Such a version of CDT would prescribe declining to buy a box. At the same time, it would recommend twoboxing in Newcomb's problem and more generally obey the causal dominance principle. For a discussion of this maxmin criterion for choice under multiple probability distributions, see, e.g.,  Gilboa and Schmeidler [1989]  and in particular game-theoretic interpretations such as that of  Grünwald and Halpern [2011] . A more general discussion of how using sets of probability distributions (while potentially decision rules other than the maxmin criterion) is offered by  Bradley [2012] . In our setting, B 1 are B 2 are, roughly, complementary bets in the causalist's beliefs. In all worlds in which B i is empty, B 3−i is full. As discussed by Bradley, it has been argued that a rational agent should accept one of a pair of complementary bets. Indeed, expected utility maximization for a single probability distribution satisfies this complementarity criterionto the causalist's detriment in the Adversarial Offer.  Bradley [2012]  argues that in general, an agent with imprecise probabilities should not satisfy the complementarity criterion and that this allows him to avoid Dutch books -though, of course, he considers Dutch books of a very different type. 4 Finally, one may view at least one of the scenarios in this paper as supporting a persuasive argument against the very core of CDT. EDT is the obvious alternative. However, depending on how problematic we find EDT's prescriptions in other cases -such as the Smoking lesion  [Ahmed, 2014, Section 4.1-4.3]  or cases of dynamic inconsistency like Newomb's problem with transparent boxes (and the problems listed in footnote 7) -we may also look to various other decision theories that have been proposed  [Gauthier, 1989; Spohn, 2012; Poellinger, 2013; Soares and Levinstein, 2017] . 			 This decision problem resembles the widely discussed Death in Damascus scenario [introduced to the decision theory literature byGibbard and Harper, 1981, Section 11]  and even more closely the Frustrater case proposed by Spencer and Wells[2017], though these are not set up to result in an expected financial loss.Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
			 This, of course, requires that the reversal offer does not come as a surprise. Throughout, we insist that the buyer knows all the rules of the game.
			 For a general discussion of such unpredictability claims in defense of CDT, seeAhmed [2014, Chapter 8].Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
			 For a discussion of similar arguments about other diachronic Dutch books, see, e.g.,Rabinowicz [2008].7 Parfit's (1984)  hitchhiker [Barnes, 1997] , XOR Blackmail [Soares and Levinstein, 2017, Section 2] and Yankees vs. Red Sox[Arntzenius, 2008, pp. 22-23; Ahmed and Price, 2012]  similarly expose dynamic inconsistencies in EDT. Conitzer [2015]  gives a somewhat different type of scenario -based on the Sleeping Beauty problem -in which EDT is dynamically inconsistent.Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
			  Policy-CDT resembles Fisher's [nd]  disposition-based decision theory. Compare Meacham [2010]  for a discussion of explicit precommitment. Similarly, Gauthier [1989]  has argued for evaluating "plans" not decisions in Newcomb-like problems (without basing this argument on any particular theory like CDT or EDT). A few authors have also proposed policy versions of other, more EDT-like decision theories[Drescher, 2006, Section 6.2; Yudkowsky and Soares,  2018, Section 4].Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
			 Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

Abstract: Newcomb's problem has spawned a debate about which variant of expected utility maximization (if any) should guide rational choice. In this paper, we provide a new argument against what is probably the most popular variant: causal decision theory (CDT). In particular, we provide two scenarios in which CDT voluntarily loses money. In the first, an agent faces a single choice and following CDT's recommendation yields a loss of money in expectation. The second scenario extends the first to a diachronic Dutch book against CDT.


Knowledge and Implicature: Modeling Language Understanding as Social Cognition

To what extent does language understanding rely on extra-linguistic knowledge and processes? One view of language processing suggests that it consists of largely separate, special-purpose faculties; another view suggests that it depends critically on domain-general inference mechanisms, and even on intuitive theories that are not specific to language. Indeed, many thinkers have viewed speech as an action with communicative goals, such as informing a listener  (Clark, 1996; Grice, 1975; Levinson, 2000) . A listener making this assumption can make stronger inferences than an utterance would allow from its literal meaning-pragmatic effects can strengthen, or change, the interpreted meaning. Recent work has aimed to formalize the social inference view of pragmatics using tools of Bayesian statistics and information theory  (Frank & Goodman, 2012) ; we refer to this formal framework as the rational speech-act theory of language understanding. It views pragmatic competence as following naturally from an intuitive theory of speech production, which in turn is a special case of intuitive theory of mind. More precisely, listeners have an internal model that describes speakers as choosing their utterances approximately optimally on the basis of certain social goals, such as conveying information to the listener; listeners then interpret an utterance by using Bayesian inference to "invert" this model of the speaker, drawing conclusions about the world state and speaker's intention from the utterance and any other relevant world knowledge. These two rationality assumptions, for listener and speaker, have a role similar to those made in ideal observer models of perception  (Geisler, 2003) : They provide a starting point for a quantitative understanding of the complex interactions involved in language understanding. Indeed, this view has provided good quantitative models for pragmatic inference in a number of simple settings  (Frank & Goodman, 2012; Frank, Goodman, Lai, & Tenenbaum, 2009) . Because rational decision making predicts that action selection is related to the expected utility-a quantity that depends on the actor's belief distribution-a listener who views the speaker as rational should be sensitive to the speaker's belief state. The rational speech-act theory thus predicts an interaction between (shared) knowledge about a speaker's knowledge state and a listener's interpretation of his utterance. This is a very general prediction of the framework, which could easily prove to be false-if pragmatic inference is a highly modularized computation, for instance, we would not expect such general knowledge to affect it. Deriving and testing precise predictions about this interaction thus provide an important test of the rational speech-act theory. If you hear "some of the apples are red," you will infer that not all of the apples are. Pragmatic effects of this sort are called scalar implicatures  (Horn, 2004) , and they provide a window on the interactions between the language faculty and general cognition. Consider Fig.  1 : If the speaker has seen all the apples, his utterance would be interpreted as "some, but not all, of the apples are red." However, if the speaker had only looked at two of the apples, the listener might draw a different conclusion. Indeed, we show below that the implicature "not all" can be canceled by facts about the speaker's perceptual access. This interaction between language understanding and general knowledge is not predicted by strongly modular theories that place scalar implicature within a semantics module  (Chierchia, Fox, & Spector, 2011) . We show further that the interaction of knowledge and implicature is fine grained: The details of a speaker's belief distribution affect the details of an implicature. This article provides a formal model of the pragmatic inference that leads to scalar implicatures, building on the rational speech-act framework. We directly model the possibility that the speaker may have incomplete knowledge and the effects this has on the listener's interpretation. We derive predictions of this model for interpretation of the quantifier "some" and the number words ("one," "two," etc.). The model both explains the standard implicatures for these words and predicts that these implicatures can be canceled, completely or partially, when the speaker has incomplete knowledge. We test these predictions in two experiments and find good fits to human judgements, both qualitatively and quantitatively.
 A rational speech-act model We view language comprehension as a rational inference based on an intuitive theory of language production. Our setting is illustrated in Fig.  1 . The listener infers the world state, s, given the speaker's utterance, w, and shared information about the speaker's (possibly incomplete) information access, a. By Bayes' rule: P listener ðs j w; aÞ / P speaker ðw j s; aÞPðsÞ; ð1Þ where P(s) captures the listener's prior beliefs about the world state and P speaker ðw j s; aÞ describes the listener's intuitive theory of how the speaker chooses words. The speaker chooses an utterance in accord with Bayesian decision theory  (Berger, 1985) : She acts to approximately optimize expected utility. Imagine a speaker who makes observation o about the true state of the world. (For instance, in Fig.  1 , the speaker has perceptual access to two of three apples and observes that those two are red). The speaker selects an utterance w to convey information about the world state to a listener and does so by soft-max optimizing expected utility: P speaker ðw j o; aÞ / expðaE Pðs j o;aÞ ½Uðw; sÞÞ: The speaker's utility function, U(w;s), captures the value of saying w if the world is actually s. The expectation is taken over the speaker's belief state, P(s | o,a), because the speaker may still be uncertain about the state of the world. The parameter a controls the deviation from optimality. So far, nothing in the model is unique to language-indeed, similar models have been used to model social cognition more generally  (Baker, Saxe, & Tenenbaum, 2009; Ullman et al., 2009) . To capture a motivation to be informative, utility must be related to the information conveyed in the utterance.  1  More specifically, utility is related to the amount of information that a literal listener would not yet know about state s after hearing it described by utterance w-this is the negative surprisal  (Cover & Thomas, 1991) : Uðw; sÞ ¼ lnðP lex ðs j wÞÞ; ð3Þ where the literal interpretation probability P lex ðs j wÞ is determined by the lexicon-here, we will assume that each utterance has a truth function, F w : s7 !f0; 1g, and the distribution is otherwise uninformative: P lex ðs j wÞ / d F w ðsÞ . We assume that the speaker's access a is common knowledge of speaker and listener, but the listener still does not know what observation the speaker made, hence: P speaker ðw j s; aÞ ¼ X o P speaker ðw j o; aÞPðo j a; sÞ: ð4Þ In the experiments below (as in Fig.  1 ), the state of the world is always a set of objects that may have a given property and observations consist of looking at a subset of the objects. Thus, the observation probability P(o | a,s) is a hyper-geometric distribution (i.e., the probability of drawing N balls of a given color, without replacement, from an urn containing a given set of colored balls). In this setting, it is also reasonable to assume that the prior probability, P(s), is a binomial distribution (i.e., draws with replacement); we will initially assume so and will later measure the distribution empirically. Overall, the above equations describe the inferences that a rational listener will make to comprehend a speaker that she believes to be approximately rational and have a goal to be informative. Importantly, these inferences depend on shared knowledge about the aspects of the world to which the speaker has access. Thus, the rational speech-act theory predicts that the speaker's access affects utterance interpretation; we test this prediction below. To derive more specific predictions, we must describe the set of alternative utterances.
 The alternatives We have assumed that the interpretation of an utterance is made with respect to a set of alternative utterances. These alternatives could be all possible utterances or could be a limited set generated by replacing key words in the actual utterance with related words. The alternatives may, for instance, be generated by a grammatical mechanism as in  Fox and Katzir (2011) . For our results, the details of this process are unimportant; what is important is that there exists a set of alternative expressions and a (truth-functional) literal meaning for each. We make standard assumptions in both respects. Consider the case of the quantifier words "some" and "all." Under the standard semantics, "all the balls are red" is true exactly when N of the N balls are red, while "some of the balls are red" is true when M ! 1 of the N balls are red. In particular, the literal meaning of "some" allows the state where all N balls are red. We use a lexicon that consists of the standard meanings for "none," "some," and "all." Model predictions are shown in Fig.  2A  for the listener's interpretation of "some of the balls are red" when there are three objects, under varying conditions of speaker access. When the speaker has perceptual access to three of the three objects (hence complete knowledge) and says "some," there is a lower probability on three than two-this is the standard "some but not all" implicature. To understand why the model predicts this implicature, we can first simplify the speaker model: In a complete knowledge situation, the SOME OF THE APPLES ARE RED. ALL SOME Fig.  1 . How will the listener interpret the speaker's utterance? How will this change if she knows that he can see only two of the objects? speaker's belief distribution is concentrated on the true state, so Eqs. 2, 3, and 4 become: P speaker ðw j s; aÞ / expðalnðP lex ðs j wÞÞÞ / ðP lex ðs j wÞÞ a : ð5Þ Using the standard meanings described above, we see that if the state were 3, the speaker would be more likely to say "all" (P lex ðs j wÞ ¼ 1) than "some" (P lex ðs j wÞ ¼ 1 3 ); conversely, if the state were 2, the speaker would say "some," as the probability for "all" is now P lex ðs j wÞ ¼ 0. Now, consider Eq. 1 and imagine for the moment a uniform prior over states. In this case, the listener will infer each state s in proportion to how likely the speaker was to say "some" given this state. Overall, this leads to the implicature that "some" is unlikely to be interpreted as 3-"some but not all." In contrast, when the speaker has only partial access, the calculation is more complex, involving the inferred belief distribution of the speaker. Comparing across the three panels of Fig.  2A , we see the probability of 3 is much higher when access is 1 or 2 than when it is 3. When access is 1, no implicature is predicted (the probability of 3 is approximately the same as the probability of 2); when access is 2, only a very slight implicature. Overall, we predict that incomplete speaker knowledge can cancel the standard "some but not all" implicature. The case of numerals ("one," "two," ...) is similar but more subtle. It has been argued that number words have a lower bound meaning  (Horn, 1972 ) (e.g., "two balls are red" means M ! 2 of the balls are red), and the intuitive, exact, meanings arise as a pragmatic implicature-"one but not two, three, etc." In Fig.  2B , we show model predictions based on the lower bound semantics for number words, varying speaker's access. We see that exact meanings do arise as an inference when the speaker has complete access, but there is an interaction: Number words do not receive an exact interpretation when the speaker has incomplete knowledge. Of particular interest is the case where the speaker has seen two of three objects and says "one": here a partial implicature is predicted, with the probability of 3 low, but 1 and 2 high.
 Experiment 1 Because a rational speaker chooses actions based on expected utility, the rational speech-act model predicts an effect of speaker's knowledge on the listener's interpretation of "some" statements. We tested the predictions of the model by putting participants in the role of the listener and asking them to judge the state of the world in scenarios where perceptual access (and hence knowledgeability) of the speaker varied.
 Participants, materials, and methods Fifty participants were recruited through Amazon's Mechanical Turk crowd-sourcing service and completed the experiment for a small payment. We constructed six scenarios in which a speaker had three objects that could have (or not) a given property. The speaker then made a statement indicating the number of objects he or she had looked at and a quantified ("some") statement. We split each scenario into setup and speech-act phases. The setup phase named the speaker and described the objects and the relevant property. For example: Letters to Laura's company almost always have checks inside. Today Laura received 3 letters. Because our model predicts greater effects when the a priori base rate of the property is high (otherwise it is difficult to tell an implicature from an a prior belief that it is unlikely for all objects to have the property), we describe all properties as "almost always" holding. To make sure participants attended to the setup, we asked them to report the a priori probability that 0, 1, 2, or 3 objects had the property: How many of the 3 letters do you think have checks inside? The speech-act phase introduced a speech act in which the speaker both declared how many of the objects he or she had observed and stated that some objects had the property: Laura tells you on the phone: "I have looked at 2 of the 3 letters. Some of the letters have checks inside." We then again elicited judgements about how many objects had the property: Now how many of the 3 letters do you think have checks inside? Finally, because the speech act might not be a perfect manipulation of speaker's knowledgeability (for instance, the speaker may have gained knowledge by another route), we elicited this directly: Do you think Laura knows exactly how many of the 3 letters have checks inside? Each response was given by a betting measure: Participants were instructed to divide "$100" among the options, betting to indicate their confidence in each option. For the first two questions, there were four options (0-3 of the objects have the property), and for the final question, there were two options (the speaker does/does not have complete knowledge). Before the experiment began, participants were given a brief warm-up, using unrelated questions, to familiarize them with the betting measure. Each scenario existed in forms varying speaker access (the number of objects the speakers had looked at) from 1 to 3. Each participant saw each access condition once, in random order, presented using randomly chosen scenarios (with no duplicate scenarios). In terms of our predictions, we have two partial-knowledge conditions (where we expect cancelation of the implicature) and one complete-knowledge "control" (where we expect the standard implicature). 2
 Qualitative results There was no effect of scenario, so we collapse across this factor in all analyses. As expected based on related work  (Goodman et al., 2009) , the speaker's access statement (e.g., "I have looked at 2 of the 3 letters") was not a perfect manipulation of knowledgeability: In the partial access conditions, some participants judged that the speaker was likely to know exactly how many objects had the property. (The bet that the speaker had complete knowledge when access = 1 was M = 27.1, SD = 4.9; when access = 2, M = 34.8, SD = 5.7; when access = 3, M = 93.0, SD = 2.7.) As we were interested in the effects of varying knowledgeability, we initially exclude trials in which the knowledge judgement was less than 70 in the expected direction (we come back to the full data set in the quantitative analysis below). Fig.  2C  shows the mean of bets on each option, as access varied. As predicted, there was an effect of speaker's access on listener's interpretation (one-way ANOVA with bets on 3 as dependent variable, F(2, 102) = 10.18, p < .001). We next performed our preplanned comparisons to check that an implicature was drawn when the speaker had complete knowledge, but not when the speaker had partial knowledge. In the complete access condition, bets on 3 were less than bets on 2 (paired, directional t test, t(43) = À10.2, p < .001). In the partial access conditions, the implicature was canceled: Bets on 2 did not exceed bets on 3 when speaker had access to 1 object (paired, directional t test, t(31) = 0.77, p = .78) or when access was 2 (paired, directional t test, t(28) = À0.82, p = .21). If we look at just bets on 3, we see significantly lower bets in the complete access condition than the access 1 condition (unpaired, directional t test, t(47) = À4.0, p < .001) or the access 2 condition (unpaired, directional t test, t(43) = À3.5, p < .001). While there was no significant implicature in either partial-access condition, there is a slightly greater tendency toward implicature in the access 2 condition than the access 1 condition, as predicted by the model (two-way ANOVA with bet as dependent variable, and access (1 or 2) and state (2 or 3) as independent variables, F(2, 294) = 3.77, p < .05). Thus, the knowledge of the speaker affected listener's interpretation of "some" in the way predicted by the rational speech-act model. We examine the quantitative fit of the model below.
 Experiment 2 In Experiment 2, we tested the predictions of the rational speech-act model for interpretation of numerals. We again expect to find an effect of speaker's knowledge, but in this case, there is a more detailed interaction: The implicature should be canceled when the speaker says "one" after seeing only one object and when the speaker says "two" after seeing two objects, but it should only be partially canceled when the speaker says "one" after seeing two objects-this implies a fine-grained interplay between the speaker's knowledge state and the interpretation of her utterance.
 Participants, materials, and methods Fifty participants were recruited through Amazon's Mechanical Turk crowd-sourcing service and completed the experiment for a small payment. We used the same stimuli as in Experiment 1, modifying the scenarios only in the speech act: The speaker now made a statement indicating the number of objects he or she had looked at and the number that had the property. For instance: Laura tells you on the phone: "I have looked at 2 of the 3 letters. 1 of the letters has checks inside." Each scenario existed in forms varying the speaker's access, from 1 to 3, and the number word the speaker used, from 1 to 3; we limited to sensible situations where the word used was no greater than the number of objects seen. Hence, we had six conditions, with access/word: 1/1, 2/1, 2/2, 3/1, 3/2, and 3/3. In terms of our predictions, we have three partial-knowledge conditions (where we expect partial or complete cancelation of the implicature) and three complete-knowledge "controls" (where we expect the standard implicatures). The order of scenarios and the order of conditions were randomized between participants.
 Qualitative results There was again no effect of scenario, so we collapse across this factor. As in Experiment 1, the speaker's access statement was not a perfect manipulation of knowledgeability (bets that speaker had complete knowledge in partial-access conditions: M = 42.0, SD = 3.4, in complete access conditions: M = 92.1, SD = 1.6). We once again limit to trials with the expected judgements of knowledgeability (with a threshold of 70). The mean of participants' bets are shown in Fig.  2D . To evaluate the overall effect of access, we performed an ANOVA with access and word as independent measures and bet on 3 as dependent measure. We find a main effect of access (F(2, 205) = 6.57, p < .01), an interaction between word and access (F(1,205) = 34.7, p < .001), and a main effect of word (F(2, 205) = 269.8, p < .001). We then explored the results in more detail using planned comparisons to test whether implicatures were drawn (only) when predicted. We found an implicature in the complete access conditions: When the speaker said "two," bets on state 3 were less than on state 2 (paired, directional t test, t(43) = À10.2, p < .001). When the speaker said "one," bets on state 1 were greater than on state 3 (paired, directional t test, t(42) = À13.1, p < .001) or state 2 (paired, directional t test, t(42) = À17.1, p < .001). In contrast, there was no implicature when access was 1 and the speaker says "one"-bets on 1 were not greater than on 2 (paired, directional t test, t(24) = 1.9, p = .96) or on 3 (paired, directional t test, t(24) = 3.2, p = 1.0)-and no implicature when access is 2 and the speaker says "two"-bets on 2 were not greater than on 3 (paired, directional t test, t(24) = 1.1, p = .87). When access is 2 and the speaker says "one" we found the predicted partial implicature: Bets on state 1 were significantly greater than on state 3 (paired, directional t test, t(25) = À3.9, p < .001), but not on state 2 (paired, directional t test, t(25) = 1.5, p = .92). These results again support the predictions of the rational speech-act model, showing not merely an interaction between the speaker's knowledge and the listener's tendency to draw an implicature, but a fine-grained interaction that is unlikely to result from a modular process of language understanding. In addition, these results support the standard, but controversial  (Barner & Bachrach, 2010; Huang, Snedeker, & Spelke, 2004) , view that number words have a "lower-bound" semantics which is only strengthened into an exact meaning by pragmatic inference.
 Quantitative model comparison To evaluate the quantitative model predictions, we first compare model predictions with mean human ratings for the subset of trials in which participants gave knowledgeability ratings in the expected direction (>70, as above). As in the model description, we assume a binomial prior distribution. We fit the prior base rate parameter and the a parameter by minimizing the root mean squared error (RMSE) of the model predictions and mean data for both experiments. The resulting model fit is RMSE = 9.01 and r = .96. The model predictions with the best-fitting parameters are shown in Fig.  2A,B  and show striking correspondence with the human data in Fig.  2C,D . To consider all responses, including those previously removed due to unexpected knowledge judgements, we extend the model by including an additional knowledge parameter: the probability that the speaker did in fact have complete knowledge (regardless of perceptual access). As we measured prior expectations and knowledgeability in each trial, we compute model predictions, trial by trial, using these values. Because the betting interface encouraged participants to round small bets to 0, but probability 0 is very different than a non-zero small probability to the model, we changed 0 and 100 bets to 1 and 99. In addition, we assumed a simple power-law relationship between subjective probability and bets (equivalent to the soft-max used to model the speaker, Eq. 2). The parameter of this power law and the speaker optimality, a, were then fit by minimizing the RMSE of mean model predictions to mean human judgements in both experiments. The resulting fit was again good: RMSE = 8.36, r = .95. Looking at individual participants, we find median correlation of r = .75 between a participant's judgements and the model predictions based on his or her prior and knowledge scores. These results suggest that the rational speech-act model is able to capture the quantitative pattern of people's judgements both across the population and within individual participants.
 Conclusion We have described a rational speech-act model of scalar implicatures and their interaction with speaker knowledge. This model formalizes language understanding as social cognition, with language-specific goals and actions, using the tools of Bayesian statistics. In addition to predicting the standard implicatures ("some but not all") as an inference that depends on the alternative utterances, this model predicted that these implicatures could be canceled, completely or in part, when it was common knowledge that the speaker had incomplete knowledge. Experiments 1 and 2 verified these qualitative predictions and showed tight quantitative fits with the model. The predicted interaction between interpretation and speaker's knowledge was not a peculiarity of this set of words or of scalar lexical items; instead, it follows from the general fact that rational decision makers must choose actions based on their expected utility. In contrast, a strongly modular model of implicatures would predict no such interactions. One could amend these theories  (Chierchia et al., 2011)  to allow a speaker's ignorance to affect the implicature mechanism. To predict the fine-grained interactions demonstrated for number words, additional machinery would be required, describing how the details of a speaker's knowledge state influence a listener's interpretation. In contrast, we have shown that the rational speech-act framework parsimoniously predicts these effects from high-level assumptions about the goals of listener and speaker. The rational speech-acts framework we have used here is closely related to that of game-theoretic pragmatics (J€ ager, in press), and particularly to the use of lifted games to capture epistemic effects  (Franke, 2009) . There are two principal differences of game-theoretic approaches from ours: In those approaches, it is assumed that speakers fully optimize (a ? ∞ in our framework), and that they carry out deeply recursive social reasoning-"I think, that you think, that I think, that..."-while we assume only one such level of reasoning. The quantitative fits we have shown suggest that limited recursion and optimization are psychologically realistic assumptions. Future work will be needed to explore all the possible models in this space. Our results support the rational speech-act framework for modeling pragmatics. More generally, they further boost the momentum building for quantitative models of language as a branch of rational social cognition. In the words of  Grice (1975) : "One of [our] avowed aims is to see talking as a special case or variety of purposive, indeed rational, behavior..." Fig. 2. (A and B)Model prediction for probability of each world state (number of objects with property), varying the word the speaker used and the speaker's perceptual access. The prior is assumed to be binomial with base rate 0.62, and the speaker optimality parameter is set to a = 3.4. (C and D) Mean participant bet on each world state, varying the word the speaker used and the speaker's perceptual access. Data have been filtered to include only trials where the participant's bet that the speaker had complete knowledge was greater than 70 in the expected direction. Error bars are standard error of the mean.
			 N. D. Goodman, A. Stuhlm€ uller / Topics in Cognitive Science 5 (2013)

Topics in Cognitive Science - 2013 - Goodman - Knowledge and Implicature Modeling Language Understanding as Social.tei.xml

Abstract: Is language understanding a special case of social cognition? To help evaluate this view, we can formalize it as the rational speech-act theory: Listeners assume that speakers choose their utterances approximately optimally, and listeners interpret an utterance by using Bayesian inference to "invert" this model of the speaker. We apply this framework to model scalar implicature ("some" implies "not all," and "N" implies "not more than N"). This model predicts an interaction between the speaker's knowledge state and the listener's interpretation. We test these predictions in two experiments and find good fit between model predictions and human judgments.